1. Introduction
In this tutorial, we’re going to talk about Kubernetes‘s probes and demonstrate how we can leverage Actuator‘s HealthIndicator to have an accurate view of our application’s state.
For the purpose of this tutorial, we’re going to assume some pre-existing experience with Spring Boot Actuator, Kubernetes, and Docker.
2. Kubernetes Probes
Kubernetes defines two different probes that we can use to periodically check if everything is working as expected: liveness and readiness.
2.1. Liveness and Readiness
With Liveness and Readiness probes, Kubelet can act as soon as it detects that something’s off and minimize the downtime of our application.
Both are configured the same way, but they have different semantics and Kubelet performs different actions depending on which one is triggered:
- Readiness – Readiness verifies if our Pod is ready to start receiving traffic. Our P**od is ready when all of its containers are ready
- Liveness – Contrary to readiness, liveness checks if our Pod should be restarted. It can pick up use cases where our application is running but is in a state where it’s unable to make progress; for example, it’s in deadlock
We configure both probe types at the container level:
apiVersion: v1
kind: Pod
metadata:
name: goproxy
labels:
app: goproxy
spec:
containers:
- name: goproxy
image: k8s.gcr.io/goproxy:0.1
ports:
- containerPort: 8080
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 1
successThreshold: 1
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
timeoutSeconds: 2
failureThreshold: 1
successThreshold: 1
There are a number of fields that we can configure to more precisely control the behavior of our probes:
- initialDelaySeconds – After creating the container, wait n seconds before initiating the probe
- periodSeconds – How often this probe should be run, defaulting to 10 seconds; the minimum is 1 second
- timeoutSeconds – How long we wait before timing out the probe, defaulting to 1 second; the minimum is again 1 second
- failureThreshold – Try n times before giving up. In the case of readiness, our pod will be marked as not ready, whereas giving up in case of liveness means restarting the Pod. The default here is 3 failures, with the minimum being 1
- successThreshold – This is the minimum number of consecutive successes for the probe to be considered successful after having failed. It defaults to 1 success and its minimum is 1 as well
In this case, we opted for a tcp probe, however, there are other types of probes we can use, too.
2.2. Probe Types
Depending on our use case, one probe type may prove more useful than the other. For example, if our container is a web server, using an http probe could be more reliable than a tcp probe.
Luckily, Kubernetes has three different types of probes that we can use:
- exec – Executes bash instructions in our container. For example, check that a specific file exists. If the instruction returns a failure code, the probe fails
- tcpSocket – Tries to establish a tcp connection to the container, using the specified port. If it fails to establish a connection, the probe fails
- httpGet – Sends an HTTP GET request to the server that is running in the container and listening on the specified port. Any code greater than or equal to 200 and less than 400 indicates success
It’s important to note that HTTP probes have additional fields, besides the ones we mentioned earlier:
- host – Hostname to connect to, defaults to our pod’s IP
- scheme – Scheme that should be used to connect, HTTP or HTTPS, with the default being HTTP
- path – The path to access on the web server
- httpHeaders – Custom headers to set in the request
- port – Name or number of the port to access in the container
3. Spring Actuator and Kubernetes Self-Healing Capabilities
Now that we have a general idea on how Kubernetes is able to detect if our application is in a broken state, let’s see how we can take advantage of Spring’s Actuator to keep a closer eye not only on our application but also on its dependencies!
For the purpose of these examples, we’re going to rely on Minikube.
3.1. Actuator and Its HealthIndicators
Considering that Spring has a number of HealthIndicators ready to use, reflecting the state of some of our application’s dependencies over Kubernetes‘s probes is as simple as adding the Actuator dependency to our pom.xml:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
3.2. Liveness Example
Let’s begin with an application that will boot up normally and, after 30 seconds will transition to a broken state.
We’re going to emulate a broken state by creating a HealthIndicator that verifies if a boolean variable is true. We’ll initialize the variable to true, and then we’ll schedule a task to change it to false after 30 seconds:
@Component
public class CustomHealthIndicator implements HealthIndicator {
private boolean isHealthy = true;
public CustomHealthIndicator() {
ScheduledExecutorService scheduled =
Executors.newSingleThreadScheduledExecutor();
scheduled.schedule(() -> {
isHealthy = false;
}, 30, TimeUnit.SECONDS);
}
@Override
public Health health() {
return isHealthy ? Health.up().build() : Health.down().build();
}
}
With our HealthIndicator in place, we need to dockerize our application:
FROM openjdk:8-jdk-alpine
RUN mkdir -p /usr/opt/service
COPY target/*.jar /usr/opt/service/service.jar
EXPOSE 8080
ENTRYPOINT exec java -jar /usr/opt/service/service.jar
Next, we create our Kubernetes template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: liveness-example
spec:
...
spec:
containers:
- name: liveness-example
image: dbdock/liveness-example:1.0.0
...
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 10
timeoutSeconds: 2
periodSeconds: 3
failureThreshold: 1
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 20
timeoutSeconds: 2
periodSeconds: 8
failureThreshold: 1
We’re using an httpGet probe pointing to Actuator’s health endpoint. Any change to our application state (and its dependencies) will be reflected on the healthiness of our deployment.
After deploying our application to Kubernetes, we’ll be able to see both probes in action: after approximately 30 seconds, our Pod will be marked as unready and removed from rotation; a few seconds later, the Pod is restarted.
We can see the events of our Pod executing kubectl describe pod liveness-example:
Warning Unhealthy 3s (x2 over 7s) kubelet, minikube Readiness probe failed: HTTP probe failed ...
Warning Unhealthy 1s kubelet, minikube Liveness probe failed: HTTP probe failed ...
Normal Killing 0s kubelet, minikube Killing container with id ...
3.3. Readiness Example
In the previous example, we saw how we could use a HealthIndicator to reflect our application’s state on the healthiness of a Kubernetes deployment.
Let’s use it on a different use case: suppose that our application needs a bit of time before it’s able to receive traffic. For example, it needs to load a file into memory and validate its content.
This is a good example of when we can take advantage of a readiness probe.
Let’s modify the HealthIndicator and Kubernetes template from the previous example and adapt them to this use case:
@Component
public class CustomHealthIndicator implements HealthIndicator {
private boolean isHealthy = false;
public CustomHealthIndicator() {
ScheduledExecutorService scheduled =
Executors.newSingleThreadScheduledExecutor();
scheduled.schedule(() -> {
isHealthy = true;
}, 40, TimeUnit.SECONDS);
}
@Override
public Health health() {
return isHealthy ? Health.up().build() : Health.down().build();
}
}
We initialize the variable to false, and after 40 seconds, a task will execute and set it to true.
Next, we dockerize and deploy our application using the following template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: readiness-example
spec:
...
spec:
containers:
- name: readiness-example
image: dbdock/readiness-example:1.0.0
...
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 40
timeoutSeconds: 2
periodSeconds: 3
failureThreshold: 2
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 100
timeoutSeconds: 2
periodSeconds: 8
failureThreshold: 1
While similar, there are a few changes in the probes configuration that we need to point out:
- Since we know that our application needs around 40 seconds to become ready to receive traffic, we increased the initialDelaySeconds of our readiness probe to 40 seconds
- Similarly, we increased the initialDelaySeconds of our liveness probe to 100 seconds to avoid being prematurely killed by Kubernetes
If it still hasn’t finished after 40 seconds, it still has around 60 seconds to finish. After that, our liveness probe will kick in and restart the Pod.
4. Conclusion
In this article, we talked about Kubernetes probes and how we can use Spring’s Actuator to improve our application’s health monitoring.
The full implementation of these examples can be found over on Github.