The Extra Kotlin Daemon. Again

In previous articles, I've shared different experiments around Android build performance, from understanding memory pressure in CI builds to terminating the Kotlin daemon before R8. Most of them started with a small observation and ended up with more experiments than I expected.

For many of these experiments I use Telltale, a framework I built to run build performance comparisons. I enjoy collecting and analyzing the data, but preparing projects, executing experiments, and reviewing results can consume a considerable amount of time.

About a year ago, I started automating more of that process. The idea was simple: whenever a new version of a major component such as Gradle, AGP, or KGP is released, I wanted to generate a project, execute both variants, and analyze the results automatically. The website contains all the different results:

The automated workload mainly runs assembleDebug on synthetic projects. It won't catch every performance change, and a slower result doesn't necessarily mean there's a regression. But it's a good starting point.

A possible Kotlin regression

Last week, Gradle 9.8.1 and Kotlin 2.4.21 were released. In the automated comparison for Kotlin 2.4.21, the generated analysis reported the following:

The overall build time for Kotlin 2.4.21 was approximately 19.6 seconds longer, an 8.2% increase over the baseline. The P50 and P90 build times increased by 20.2 seconds (7.8%) and 20.6 seconds (7.5%), respectively. Configuration time increased by about 1.3 seconds (3.1%). Several task types, including KotlinCompile and DexMergingTask, were slower. Peak memory usage also increased from 9.34 GB to 9.65 GB.

An 8% difference was enough to investigate further.

I checked the Kotlin 2.4.21 release notes, but couldn't find anything that would explain the difference.

The Kotlin Build Reports also showed increases across several compilation metrics:

Increasing the size of the experiment

Instead of the original 100-module project, I generated a project with 350 modules and varied the number of Kotlin compilation units across those modules. I also increased the number of executions to 50 iterations per variant, comparing Kotlin 2.4.20 against Kotlin 2.4.21.

The older version was still faster:

Kotlin 2.4.21 was 36 seconds slower at the median and 46 seconds slower on average. I then checked the process metrics to understand what was happening.

The Kotlin daemon problem. Again.

While reviewing the Kotlin process metrics, I noticed that one process wasn't reporting the GC metrics I expected. That made me check how many Kotlin compiler processes were running.

I had already written about this problem. Different Kotlin versions in build logic and application compilation can result in additional compiler processes, and I'd measured their memory footprint in my article about RSS.

And somehow I'd managed to hit the same problem again.

Why can we end up with two Kotlin daemons?

Gradle includes an embedded Kotlin version for Kotlin DSL build scripts. But compiling .gradle.kts scripts isn't the same as compiling Kotlin sources in convention plugins or in the Android project.

In my case, the mismatch was between Kotlin compilation in an included build using kotlin-dsl and compilation in the main project.

When the included build and main project use incompatible Kotlin compiler versions, they may end up using separate Kotlin compiler daemons. One daemon can finish its early compilation work and then remain alive, while the other handles the rest of the project's Kotlin compilation.

The extra daemon can remain alive after finishing its work, still consuming RSS. But was it also affecting build duration?

Checking the Kotlin versions

To understand where the extra process came from, I explicitly aligned the Kotlin version used in the included build with the version used by the main project:

plugins {
    `kotlin-dsl`
    kotlin("jvm") version "2.4.21"
}

After aligning the versions, I checked the running processes and confirmed that the build used one Kotlin compiler daemon instead of two. This configuration depends on the build logic and Gradle version, so it's worth checking rather than assuming.

Here was the problem with my original comparison: the Kotlin 2.4.20 baseline was running one compiler daemon, while the 2.4.21 variant was running two because of the mismatch in the included build. I wasn't comparing the two Kotlin versions under equivalent conditions.

The performance difference could be related to the KGP update, the additional daemon, or both. I couldn't use those results to conclude that KGP 2.4.21 introduced a regression.

For the following analysis, I kept the original comparison: KGP 2.4.20 with one daemon versus KGP 2.4.21 with two daemons. Aligning the included build helped me identify the mismatch, but it wasn't a controlled experiment that isolated daemon count. I wanted to understand where the slowdown was happening.

First hypothesis: JVM warm-up and JIT compilation

Each Kotlin daemon has its own JVM, so JIT-compiled code isn't shared between them. I wondered whether the main project's daemon in the 2.4.21 configuration was spending extra time warming up, while the baseline configuration could potentially reuse code compiled earlier by the same daemon.

On a four-core GitHub Actions runner, that extra work could matter. I compared the JIT activity of the main daemon in the 2.4.21 configuration with the main daemon in the 2.4.20 baseline.

Here's how the JIT compilation activity evolved during both executions:

The cumulative JIT compilation time represents time spent compiling methods, not elapsed build duration. The normalized values compare that cumulative time with the lifetime of the daemon, so they are ratios rather than additional seconds of build time. The number of compiled methods differed by less than 1%. Both runs had JIT activity at the beginning, but the 2.4.21 configuration didn't show more normalized JIT activity in its main daemon. The extra daemon also did very little JIT compilation after its initial work.

I couldn't find evidence that JIT compilation explained the difference in build duration.

Second hypothesis: CPU contention

The runner had four CPU cores, and the build could run Kotlin, Java, KSP, dex, and Android resource tasks concurrently. I wanted to see whether CPU contention could help explain the difference.

I sampled /proc/stat to calculate system-wide CPU utilization. I also checked CPU PSI, the runnable queue, and CPU throttling. PSI some measures the percentage of time during which at least one task is stalled waiting for CPU.

The aggregated utilization looked like this:

Other system-level measurements:

• CPU PSI some was around 57–59%. During more than half the measured time, at least one runnable task was waiting for CPU time.
• The runnable queue was around 8 tasks on a four-core runner.
• I found no significant CPU quota throttling.
• Memory pressure was negligible.

Both runs were close to full CPU utilization, and the differences were small. I couldn't explain the slower build from these averages, although they could hide a problem during a shorter part of the execution.

Third hypothesis: Gradle scheduling and synchronization

I also used Java Flight Recorder to look for waiting, lock contention, GC, or overhead in the communication with the Kotlin daemon.

The extra daemon was mostly idle after its initial compilation. I didn't find meaningful differences in Gradle locks or GC. There was slightly more waiting in Kotlin daemon connection handling, but I couldn't establish whether it contributed to the difference in build duration.

The process metrics didn't explain the difference, so I went back to the Kotlin compilation tasks. Maybe only some tasks were getting slower.

Looking at individual Kotlin compilation tasks

I sorted the KotlinCompile tasks by mean duration in the 2.4.21 variant and looked at the five slowest:

All five were in :core:, one of the first layers in the dependency graph. Four were more than six seconds slower in the 2.4.21 variant. The fifth was only about half a second slower, but it was still among the slowest tasks by duration.

These tasks were among the first Kotlin compilations in the build, potentially overlapping with the startup of the second daemon. I wanted to understand whether the beginning of the build was particularly affected.

Comparing the compilation layers

Instead of selecting a few tasks, I grouped Kotlin compilation durations by project layer. The mean and median values are for individual Kotlin compilation tasks, not the total execution time of each layer:

The :core: layer had the biggest difference, around 14%. The other layers were between 3% and 7%.

I went back to the beginning of the build. On a four-core GitHub Actions runner, many :core: modules could start Kotlin compilation at the same time, just as a compiler daemon was starting. The CPU and JFR measurements hadn't shown me what was happening in that specific window.

So I changed the experiment to reduce the number of Kotlin tasks that could start together.

Changing the dependency graph

For the next experiment, I generated a different project with 250 modules, but with only one module in the initial core layer. Instead of many Kotlin compilations starting together, the downstream modules would have to wait for that single core module.

The task-level results were different. As in the previous table, these are individual Kotlin compilation durations grouped by layer:

The single core compilation was almost twice as slow in the 2.4.21 variant: 20.9 seconds versus 10.6 seconds. But this time it was only one task. In the downstream layers, mean Kotlin compilation durations were around 9–11% lower in the 2.4.21 variant.

The overall build was also faster with KGP 2.4.21, unlike the 350-module experiment. That variant was still running two Kotlin daemons.

One important difference between the projects was the number of Kotlin compilations that could start at the beginning of the build. In the first experiment, many :core: tasks could execute concurrently. In the second, there was only one.

However, I also changed the number of modules and the dependency graph, so I can't say that the initial compilation concurrency explains the difference.

The performance difference between the two configurations changed with the workload. Since the configurations differ in both KGP version and daemon count, I still can't isolate what caused the difference.

Final thoughts

What initially looked like a Kotlin 2.4.21 regression turned out to be a comparison where the 2.4.21 variant was also running an additional Kotlin compiler daemon. That means I can't attribute the slower builds directly to the KGP update.

The results also changed depending on the workload. With many Kotlin compilations starting together, the 2.4.21 variant was slower. With a different dependency graph, it was faster, even though it was still running two Kotlin daemons. I couldn't isolate the cause of that difference.

The included build in these experiments was lightweight. In projects with heavier build logic, an additional daemon could remain idle while consuming substantial RSS, potentially increasing memory pressure on CI runners.

Happy Building!