Introducing Hearth v0.3.0: Scale Long-Tail LLMs to Zero on Private Kubernetes

** An open-source, composable control plane for teams that want to stop paying the always-on cost of rarely used models.
**
Hearth v0.3.0(https://github.com/hearthproject/hearth/releases/tag/v0.3.0) is now available.

Most conversations about LLM serving focus on high-traffic models

large fleets, complex routing, distributed inference, and maximum throughput. But many private Kubernetes clusters face a different problem.

They have a limited pool of GPUs or NPUs, a handful of frequently used models, and a much longer tail of models that may receive only occasional requests. Keeping every model warm wastes expensive accelerator capacity. Scaling them down sounds simple—until the next request arrives.

Setting a Deployment to zero replicas is easy. Preserving the first client request while Kubernetes schedules a Pod, pulls an image, loads model weights, passes readiness probes, and begins serving tokens is not. Safe scale-down is equally difficult when requests are still streaming.

This is the lifecycle problem Hearth is designed to solve.

What is Hearth?

Hearth (https://github.com/hearth-project/hearth) is a minimal, composable LLM serving control plane for private Kubernetes clusters.

It is not an inference engine and does not implement accelerator kernels. It does not install device plugins or replace Kubernetes schedulers. It is also not trying to become a fleet-level AI serving platform.

Instead, Hearth owns the Kubernetes lifecycle between a model declaration and a working, scale-to-zero inference endpoint.

** Application owners create a namespaced LLMService describing:
**

• the model source;
• the desired runtime;
• accelerator, CPU, and memory requirements;
• caching and prewarming;
• scaling behavior; and
• cold-start endpoint behavior.

** Cluster administrators publish reusable, cluster-scoped InferenceRuntime profiles describing:
**

• the serving image and arguments;
• the device-plugin resource name;
• node selectors and tolerations;
• optional scheduler and queue configuration;
• health probes; and
• graceful termination behavior.

This separation keeps serving intent portable while leaving hardware and cluster details under administrator control.

From these two resources, Hearth reconciles the serving backend, a stable gateway, Kubernetes Services, model-cache resources, an optional prewarm Job, and KEDA autoscaling objects.

** Scale-to-zero that understands cold starts
**
Hearth keeps a lightweight gateway available while the model backend scales between zero and the configured maximum number of replicas.

The lifecycle works as follows:

1. When idle, KEDA keeps the model backend at zero replicas.
2. A request arrives at the stable Hearth gateway.
3. The gateway records demand and either holds the request or returns 503 with Retry-After, according to policy.
4. KEDA activates the backend through the default metrics API or the opt-in external-push scaler.
5. The backend loads its model from cache and becomes Ready only after the serving runtime passes its health checks.
6. The gateway forwards the request and streams the model response.
7. Sustained queue demand can scale the backend from one replica to multiple replicas.
8. When demand disappears, graceful draining protects in-flight streams before the backend returns to zero.

The gateway also provides bounded admission. When the queue is full, excess requests receive 429 instead of silently exhausting memory or connections. Cold-start activation has a deadline, and streaming requests can receive SSE heartbeats while waiting for the model.

In v0.3.0, the new external-push mode removes KEDA’s polling interval from the initial activation path. An activation lease preserves demand even when reject mode immediately returns a 503 or when KEDA reconnects its scaler stream.

The default metrics-based mode remains available for compatibility.

** Caching is part of the scale-to-zero design
**
Releasing an accelerator is useful only if restarting the model remains practical.

Hearth supports HostPath and NodeLocalPVC caches, optional prewarming for Hugging Face and ModelScope models, and pvc:// sources for model weights that are already staged in the cluster.

Prewarm Jobs inherit the runtime’s scheduling constraints but do not request an accelerator. This lets a cluster download model weights before an inference Pod claims a GPU or NPU.

Hearth also exposes stable gateway and backend metrics, while monitoring remains independent. Teams can deploy kube-prometheus-stack, another Prometheus-compatible system, or no monitoring stack at all without coupling it to the core reconciler.

** Real hardware, not only rendered YAML
**
Hearth’s adapters and controllers are covered by automated tests, including a CPU-only vLLM stub and complete scale-to-zero suites on Kind. But rendered manifests are not treated as proof of hardware support.

The project records physical validation separately.

On two NVIDIA A10 GPUs, the v0.3.0 release candidate completed the full 0→1→2→0 lifecycle using vLLM v0.25.1 and KEDA external-push. The run covered real inference, cache prewarming and persistence, bounded admission, reject mode, graceful draining, Volcano placement, component replacement, and host reboot recovery.

Six pending requests drove the backend from one to two replicas, with the two Pods placed on different physical GPUs. A 105-client cold burst admitted 100 requests and rejected the remaining five with the expected 429 responses.

** Hearth and Kthena: hot models and the long tail
**
Hearth does not need to replace a broader serving platform.

The project includes a real-hardware demo in which Kthena keeps a frequently used model warm while Hearth manages a long-tail model on the same Kubernetes cluster. The Hearth backend starts at zero, activates when a request arrives, serves real tokens, and returns to zero afterward.

This suggests a practical division of responsibility:

• keep high-traffic, latency-sensitive models on a platform designed for fleet-level serving; and
• place occasional or long-tail models behind Hearth’s smaller scale-to-zero lifecycle.

Volcano can schedule both workloads, but it remains an external integration rather than a dependency embedded in Hearth.

Watch the Hearth and Kthena operational demo (https://github.com/hearth-project/hearth/blob/main/docs/demo.md).

** Where Hearth fits today
**
Hearth v0.3.0 is alpha software with a serving.hearth.dev/v1alpha1 API.

It is best suited to internal, development, laboratory, and staging environments where:

• accelerator capacity is limited or expensive;
• some models receive intermittent traffic;
• cold-start latency is acceptable;
• workloads run inside a trusted private boundary; and
• teams want a small Kubernetes-native control plane rather than a complete serving platform.

Hearth is not yet intended for shared multi-tenant clusters or public customer-facing endpoints. The gateway does not currently provide built-in authentication, and external-push mode requires one gateway replica until demand aggregation is implemented.

Fractional accelerators, HAMi sharing, MIG, and general multi-node topology have not been established by the current whole-device validation.

GitHub: github.com/hearth-project/hearth (https://github.com/hearth-project/hearth)
Release: v0.3.0 (https://github.com/hearth-project/hearth/releases/tag/v0.3.0)
Roadmap: ROADMAP.md (https://github.com/hearth-project/hearth/blob/main/ROADMAP.md)
Contributing: CONTRIBUTING.md (https://github.com/hearth-project/hearth/blob/main/CONTRIBUTING.md)