A Java Architect Mapped AI Into Four Layers: Where Java Wins Today and Where It Still Hurts

If you work in the JVM ecosystem, you have heard the verdict repeated all year: AI belongs to Python, and Java should quietly keep running the billing service while the interesting work happens somewhere else.

At JavaOne 2026, Paul Sandoz of Oracle's Java team gave a fifty-minute answer to that claim in a session called "How does Java fit into the world of AI, and how can it become an even better fit?" Sandoz has spent years inside OpenJDK's core libraries, and he is one of the people behind Project Babylon, the OpenJDK effort to run Java code on GPUs. His talk is the most structured take on Java and AI I have found: not hype, not denial, but an actual map of which AI workloads Java handles well today, which ones hurt, and which OpenJDK projects are supposed to fix the pain.

One disclosure before going further: this article is based on the recorded session, the OpenJDK project pages, and the JEP drafts it references. I have not run Project Babylon myself. Everything below is sourced from the talk and the linked public documents.

The prediction he is willing to make

Sandoz opened by admitting predictions are dangerous right now, then made one anyway: some form of agentic AI will become an essential tool in the software development toolbox, and the Java community needs to adapt both technically and culturally.

He borrowed his framing from Mark Reinhold's 2018 talk "Change is the Only Constant," quoting HG Wells ("adapt or perish, now as ever, is nature's inexorable imperative") and Andy Grove's blunter version, "adapt or die." The historical argument is that Java has already survived five punctuated shifts: object-oriented programming, the internet, open source, mobile, and cloud. Companies that failed to adapt along the way are gone. Where's Netscape? Where's Sun? Java adapted through all of it, and he believes AI is the next shift of the same kind, not an exception that ends the story.

Whether you find that reassuring or complacent probably depends on how much of your career is invested in the JVM. But it sets up the most interesting idea in the talk.

"Compressed insight": why LLMs work well with Java

Sandoz's central concept is one he credits to reading across the industry: over roughly thirty years, the JVM and its ecosystem have compressed an enormous amount of engineering insight into a platform that still behaves the same way with respect to its specifications. The source code, the specifications, the compatibility discipline, the refusal to fork the language the way Python split between 2 and 3. He called that accumulated compatibility and knowledge a kind of asset, "rocket fuel for LLMs."

His argument, in three parts:

LLMs prefer tools with compressed insight. A model works best when the surrounding platform already encodes decades of decisions, so the model does not have to re-derive them from scratch.
Context is finite. He joked about a Douglas Adams line that the human brain has "seven memory registers," and argued LLMs hit the same wall. That is why agent frameworks obsess over what context to feed the model, and why a platform that answers questions in fewer tokens wins.
Do not resynthesize what already exists. On the idea of regenerating a mature platform with an LLM: "There are better ways to spend your money on tokens. The OpenJDK is just a git clone away for a fraction of the price."

That last point cuts against a real pattern in 2026, where teams point coding agents at greenfield rewrites of systems that already work. His claim is that a stable, spec-abiding, thirty-year-old platform is exactly the kind of surface an LLM is good at working with, because the platform constrains the model instead of the model having to invent conventions.

The four-layer map

The core of the talk is a decomposition of AI into four layers, ranked by how easy the Java platform makes each one today. He drew it as two axes: how hard the problem is in any language, and how much the platform helps.

Training. Computing model weights from data. The hardest problem, and the one where Java support is weakest. It requires automatic differentiation, GPUs at scale, and skills very few teams have.
Inference. Running a trained model to get output. He believes aggregate inference demand is why the industry is pouring money into data centers, more than training. In Java this is possible today but rougher than it should be.
Agent. Composition of inference: a system that orchestrates an LLM, tools, and information sources.
Agentic. Composition of agents with dynamic context, orchestrating work at scale, which is what modern coding tools do.

His verdict on the top two layers is blunt: they are not a terribly hard problem for Java, because they are not a new kind of problem. Agents send HTTP requests carrying JSON or SSE events. They compose services, manage retries, handle failures, and orchestrate workflows. "There's nothing magical going on with these systems," he said. If you have built distributed systems in Java before, you already have the skills. That is why he puts agent and agentic development in the "easy on the platform today" bucket, and why each release makes it easier.

The pain starts as you move down the stack toward inference and training, and most of the second half of the talk is about what the JDK is doing to move that pain line.

What you can actually use today

Working from the talk and the OpenJDK pages, here is the current toolbox for Java teams doing AI work, layer by layer:

Calling an LLM: The JDK's HttpClient is enough to talk to any OpenAI-compatible API. A small JSON library is on the way: JEP 540, the Simple JSON API, is targeted as an incubator for JDK 28, which removes one of the last reasons a one-off LLM call needs a third-party dependency.
Native model runtimes: Project Panama's Foreign Function and Memory API is the supported path to native code, and Sandoz demonstrated that running ONNX models through ONNX Runtime from Java works well, with none of the JNI fragility of the old approach.
CPU inference: The Vector API lets you implement model math on the CPU directly, which matters if you do not want to pay for GPUs. He was explicit that this is a real option when the implementation is aligned with how memory caches and vectorization actually work.
The Python escape hatch: Project Detroit, announced at JavaOne 2026, embeds the CPython and V8 runtimes inside the JVM process over the FFM API. When a Java equivalent of a Python library does not exist, Detroit lets you call the real CPython runtime from Java instead of shelling out to a subprocess.
GPU code in Java: Project Babylon's Heterogeneous Accelerator Toolkit (HAT), built on code reflection, compiles plain Java into GPU kernels.

The GPU demo: normal Java, 200 frames per second

The demos are what make the talk worth watching rather than just reading about. Sandoz ran shaders written as ordinary-looking Java methods, with no annotation-driven DSL and no embedded GLSL strings, on his laptop's GPU. Code reflection translates the Java into GPU kernel code, and HAT dispatches it.

The numbers from the session:

A shader running on one CPU thread: about 2 frames per second.
The same shader multithreaded on the CPU: about 25 frames per second. His comment: "HotSpot is really damn good at compiling Java code on the CPU."
A more expensive shader multithreaded on the CPU: over 200 frames per second.
A ray-tracing demo: roughly 30 frames per second on the GPU, against more than 3 seconds per frame on the CPU.

The point is not that Java beats CUDA. The point is the workflow: you write normal Java, debug it on the CPU where stepping through data is easy, then run the same code on the GPU where it is fast. Debugging on GPUs today is, in his words, "really hard." Being able to develop and debug on CPU and deploy the identical code to the GPU is the part he called a big deal, and it works across GPU hardware because the translation targets a portable model rather than one vendor's stack.

Valhalla feeds into this too. The demo code uses vector types like float2 and float3; once value classes land, those become flat, headerless values in memory, which should reduce garbage and speed up the CPU path as well.

Where the status actually stands

Grounding the talk against the public record:

Code reflection is JEP draft 8361105, owned by Sandoz, with a status of Submitted. He said in the session he expects it to move to Candidate soon and then target a JDK release, where it will incubate in the jdk.incubator.code module and iterate every six months. Nothing is final, and the @Reflect marker annotation is explicitly described as a placeholder until a language-level approach is settled.
HAT currently lives in the Babylon repository and is expected to get its own repository and identity once code reflection incubates in the JDK.
JEP 540 (Simple JSON API, incubator) is targeted at JDK 28.
Project Detroit is an official OpenJDK project since March 2026, with the detroit and detroit-python repositories public.

The honest part: training is still the wall

To his credit, Sandoz did not pretend the map ends well. Training is the layer where he said he "bashes his head against the wall." The hard problem is not raw compute; it is reverse-mode automatic differentiation, the mechanism that makes stochastic gradient descent scale to models with billions of parameters. Very few people on the planet can implement it correctly, and nobody has done it in Java.

He also offered a cautionary tale from his own ecosystem's history: Swift for TensorFlow put differentiation into the language specification itself, and he called maintaining that "a nightmare for the language maintainers," because language teams are not autodiff experts. Babylon's bet is the opposite: keep Java the language untouched, expose the code as a transformable model, and let libraries do the differentiation and the GPU translation. The same code reflection mechanism can express type-safe SQL, eBPF programs, or plain Java-to-Java transforms, which is why he called it a Swiss Army knife rather than an AI feature.

And on the question of whether Java should even try to compete with PyTorch, he gave an answer that is either vision or hubris depending on your priors: "I think we can do a better job than Python in the long run," pointing at the scaling heritage of Apache Spark and Lucene, and at the possibility that smaller models trained on smaller data could put the tooling in the hands of learners. He conceded nobody will believe it until it exists. That honesty is the right tone: this is a multi-year bet by the OpenJDK community, not a shipped capability.

If you are deciding what to do with this

The takeaway matrix from the talk, compressed:

Building agents or agentic workflows: Start now. This is HTTP, JSON, and orchestration, and your existing Java skills transfer completely. No new platform features are required.
Running inference in JVM services: Use ONNX Runtime over Panama today, watch JEP 540 for the dependency-free JSON piece, and consider the Vector API if GPU spend is the blocker.
Needing a Python-only library: Track Project Detroit rather than bolting subprocess calls into your deployment.
Curious about GPU compute from Java: Clone the Babylon repo and experiment with HAT, ideally with a coding assistant pointed at it. Sandoz himself noted the tooling is "remarkably adept" at navigating the repository, and the team wants feedback from exactly that kind of experimentation.
Hoping to train models in Java: Realistically, wait. Code reflection needs to incubate first, and the autodiff story does not exist yet.

I write about Java, the JVM, and AI every week. Subscribe, it's free.

Have you built agents or inference pipelines in Java, or did you reach for Python by default? I am curious how the "Java is fine for agents" claim matches what people are actually shipping.