1. Introduction
The rising popularity of cloud-native applications and micro-services generate an increased demand for embedded servlet containers. Spring Boot allows developers to easily build applications or services using the 3 most mature containers available: Tomcat, Undertow, and Jetty.
In this tutorial, we'll demonstrate a way to quickly compare container implementations using metrics obtained at startup and under some load.
2. Dependencies
Our setup for each available container implementation will always require that we declare a dependency on spring-boot-starter-web in our pom.xml.
In general, we want to specify our parent as spring-boot-starter-parent, and then include the starters we want:
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.0</version>
<relativePath/>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
</dependencies>
2.1. Tomcat
No further dependencies are required when using Tomcat because it is included by default when using spring-boot-starter-web.
2.2. Jetty
In order to use Jetty, we first need to exclude spring-boot-starter-tomcat from spring-boot-starter-web.
Then, we simply declare a dependency on spring-boot-starter-jetty:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
</dependency>
2.3. Undertow
Setting up for Undertow is identical to Jetty, except that we use spring-boot-starter-undertow as our dependency:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-undertow</artifactId>
</dependency>
2.4. Actuator
We'll use Spring Boot's Actuator as a convenient way to both stress the system and query for metrics.
Check out this article for details on Actuator. We simply add a dependency in our pom to make it available:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
2.5. Apache Bench
Apache Bench is an open source load testing utility that comes bundled with the Apache web server.
Windows users can download Apache from one of the 3rd party vendors linked here. If Apache is already installed on your Windows machine, you should be able to find ab.exe in your apache/bin directory.
If you are on a Linux machine, ab can be installed using apt-get with:
$ apt-get install apache2-utils
3. Startup Metrics
3.1. Collection
In order to collect our startup metrics, we'll register an event handler to fire on Spring Boot's ApplicationReadyEvent.
We'll programmatically extract the metrics we're interested in by directly working with the MeterRegistry used by the Actuator component:
@Component
public class StartupEventHandler {
// logger, constructor
private String[] METRICS = {
"jvm.memory.used",
"jvm.classes.loaded",
"jvm.threads.live"};
private String METRIC_MSG_FORMAT = "Startup Metric >> {}={}";
private MeterRegistry meterRegistry;
@EventListener
public void getAndLogStartupMetrics(
ApplicationReadyEvent event) {
Arrays.asList(METRICS)
.forEach(this::getAndLogActuatorMetric);
}
private void processMetric(String metric) {
Meter meter = meterRegistry.find(metric).meter();
Map<Statistic, Double> stats = getSamples(meter);
logger.info(METRIC_MSG_FORMAT, metric, stats.get(Statistic.VALUE).longValue());
}
// other methods
}
We avoid the need to manually query Actuator REST endpoints or to run a standalone JMX console by logging interesting metrics on startup within our event handler.
3.2. Selection
There are a large number of metrics that Actuator provides out of the box. We selected 3 metrics that help to get a high-level overview of key runtime characteristics once the server is up:
- jvm.memory.used – the total memory used by the JVM since startup
- jvm.classes.loaded – the total number of classes loaded
- jvm.threads.live – the total number of active threads. In our test, this value can be viewed as the thread count “at rest”
4. Runtime Metrics
4.1. Collection
In addition to providing startup metrics, we'll use the /metrics endpoint exposed by the Actuator as the target URL when we run Apache Bench in order to put the application under load.
In order to test a real application under load, we might instead use endpoints provided by our application.
Once the server has started, we'll get a command prompt and execute ab:
ab -n 10000 -c 10 http://localhost:8080/actuator/metrics
In the command above, we've specified a total of 10,000 requests using 10 concurrent threads.
4.2. Selection
Apache Bench is able to very quickly give us some useful information including connection times and the percentage of requests that are served within a certain time.
For our purposes, we focused on requests-per-second and time-per-request (mean).
5. Results
On startup, we found that the memory footprint of Tomcat, Jetty, and Undertow was comparable with Undertow requiring slightly more memory than the other two and Jetty requiring the smallest amount.
For our benchmark, we found that the performance of Tomcat, Jetty, and Undertow was comparable but that Undertow was clearly the fastest and Jetty only slightly less fast.
Metric
Tomcat
Jetty
Undertow
jvm.memory.used (MB)
168
155
164
jvm.classes.loaded
9869
9784
9787
jvm.threads.live
25
17
19
Requests per second
1542
1627
1650
Average time per request (ms)
6.483
6.148
6.059
Note that the metrics are, naturally, representative of the bare-bones project; the metrics of your own application will most certainly be different.
6. Benchmark Discussion
Developing appropriate benchmark tests to perform thorough comparisons of server implementations can get complicated. In order to extract the most relevant information, it's critical to have a clear understanding of what's important for the use case in question.
It's important to note that the benchmark measurements collected in this example were taken using a very specific workload consisting of HTTP GET requests to an Actuator endpoint.
It's expected that different workloads would likely result in different relative measurements across container implementations. If more robust or precise measurements were required, it would be a very good idea to set up a test plan that more closely matched the production use case.
In addition, a more sophisticated benchmarking solution such as JMeter or Gatling would likely yield more valuable insights.
7. Choosing a Container
Selecting the right container implementation should likely be based on many factors that can't be neatly summarized with a handful of metrics alone. Comfort level, features, available configuration options, and policy are often equally important, if not more so.
8. Conclusion
In this article, we looked at the Tomcat, Jetty, and Undertow embedded servlet container implementations. We examined the runtime characteristics of each container at startup with the default configurations by looking at metrics exposed by the Actuator component.
We executed a contrived workload against the running system and then measured performance using Apache Bench.
Lastly, we discussed the merits of this strategy and mentioned a few things to keep in mind when comparing implementation benchmarks. As always, all source code can be found over on GitHub.