Debugging a 6-Second JVM Death Loop: A Bytecode Decompilation Story

The Symptom

A secondary server in a High Availability (HA) failover cluster kept dying. Not crashing loudly — dying quietly, exactly 6-7 seconds after every startup attempt, with a log message that told me absolutely nothing:

System going to Shutdown --- received process interrupt

No stack trace. No exception. No hint about why. Just... gone.

This is the story of how I traced that message all the way down to a single PostgreSQL configuration parameter — by decompiling the vendor's own Java bytecode when documentation, logs, and community forums all came up empty.

The Setup

I was building a proof-of-concept for a commercial network monitoring product's HA failover feature, testing a leaner 3-VM topology instead of the vendor's standard 4-VM recommendation:

• VM 1: Primary application server
• VM 2: Secondary application server (the one that kept dying)
• VM 3: Combined PostgreSQL database + shared filesystem host

Nothing exotic. Standard failover pattern. The vendor's docs didn't explicitly forbid combining the DB and shared-folder roles on one VM, so I built it that way to reduce the deployment footprint.

The primary server worked flawlessly. The secondary would not stay alive.

Round 1: The Obvious Suspects (all wrong)

I worked through every config-level hypothesis I could think of, and found four genuinely broken things along the way — none of which were the actual problem:

1. A missing config directory (pgsql/ext_conf/) that got silently excluded from server-to-server replication, causing a FileOutputStream error deep in a startup utility class.
2. A database superuser with no password set, while the encrypted credential file expected one — classic drift between "what the config says" and "what the database actually has."
3. A missing row in a server-status tracking table — the secondary node had never registered itself, so an internal health check found literally nothing to report on.
4. Wrong startup script order — a "prerequisite verification" script was being treated as if it performed activation, when it was actually meant to run before the main service launcher, not instead of it.

I fixed all four. The secondary still died at the 6-7 second mark, every single time.

Round 2: The Internet Has Nothing

At this point I did what any reasonable engineer does — I went looking for prior art. Vendor community forums, official HA/failover documentation, even documentation for a sibling product built on the same underlying framework.

I confirmed one useful thing: "received process interrupt" is a generic message from the Java Service Wrapper (the process supervisor wrapping the JVM) — it fires on any JVM exit, whether from a clean System.exit() call or an uncaught exception. It's not a product-specific error code. It's the software equivalent of a shrug.

I found zero precedent for my exact combination of symptoms. Time to go deeper than logs.

Round 3: Wrong Database Tables

I widened my database inspection beyond the tables mentioned in visible error messages, and found two tables the startup code actually reads that I hadn't considered:

• One tracking node registration status — had zero rows for the secondary, meaning it had never successfully told the cluster "I exist."
• One holding the shared folder path configuration — still pointing at a stale value from an earlier test topology, never updated despite multiple rounds of config-file fixes.

I fixed both directly in the database. The secondary died at the exact same 6-7 second mark. Same message. No change whatsoever.

That's when I knew I was debugging the wrong layer entirely.

Round 4: Decompiling the Vendor's Own Code

When a vendor's error output tells you nothing, and public documentation has nothing, there's one place left with the actual answer: the compiled code itself.

I downloaded CFR, a solid open-source Java decompiler, straight onto the VM, and pointed it at the product's own JAR files using its bundled JRE:

java -jar cfr.jar SomeVendorClasses.jar --outputdir /tmp/decompiled

Then I started tracing the actual startup call chain by reading real source code instead of guessing from logs:

StartupHooks.preStartServer()
  → StartupCheckHandler.doPreCheck()
  → Preprocessor.initialize(coldStart)
      → moduleInit()          [ruled out — its log markers never appeared]
      → StartupCheckHandler.doDBMemoryCheck()
          → sqlChecks()       [ruled out — returns true immediately for this DB type]

I traced through five separate classes across three different JARs. Every single Failover-specific code path I could find had zero log evidence of ever running on the failing secondary. Not "ran and failed" — never executed at all. Which meant the JVM was dying before it ever reached any of the HA-specific logic I'd spent days chasing.

That redirected me somewhere much more basic: connection pool bootstrap.

The Actual Root Cause

A full read of the raw stderr log (not the application-level logs I'd been checking) turned up this:

Could not instantiate RelationalAPI in NmsUtil. Server quitting
Check for the NmsStorageException :
CreateConnectionException

Grepping the decompiled source for that exact string led me straight to the responsible method: a catch block wrapped around database connection pool creation, which — on failure — logs this generic message and calls System.exit(1).

The secondary was dying during the most basic operation possible: trying to open its own database connection pool. Before any failover logic. Before any node registration. Before anything I'd spent four rounds of debugging on.

So why would connection pool creation fail?

SHOW max_connections;
-- 100

SELECT count(*) FROM pg_stat_activity;
-- 57 (all from the primary server)

The secondary's connection pool was configured to request 50 connections on startup — same as the primary. The primary was already using 57. 57 + 50 = 107, against a ceiling of 100.

The secondary's connection pool creation failed. The generic exception handler caught it, logged a message that gave no hint of the actual cause, and killed the JVM. The process supervisor reported this as "received process interrupt" — a message so generic it actively pointed me in the wrong direction for days.

The Fix

sudo sed -i 's/max_connections = 100/max_connections = 250/' /etc/postgresql/17/main/postgresql.conf
sudo systemctl restart postgresql

The secondary started on the first attempt after this. Verified through the application's own audit log:

The service is now in standby mode.

Four rounds of legitimate-but-wrong fixes, and the actual cause was a single default configuration value that never scales past one connection pool.

Why This Took So Long

A few things stacked up to make this unusually hard:

1. The error message carried zero diagnostic information. A generic wrapper-level message masked an application-level exception, which masked a database-level exception, which masked the real cause.
2. All my available logs were downstream of the actual failure. They weren't broken — they were just logs for code that never got a chance to run.
3. The exception's own message field was empty. Only the exception class name in the following log line gave a usable clue — everything else was silence.
4. The bug is topology-dependent, not a product defect. A single-server deployment, or a properly-sized deployment from day one, would never hit this. It only surfaces the moment a second full-size connection pool gets added against a database whose capacity was never re-planned for two consumers.

The Bigger Lesson

If you're running any HA/failover setup against PostgreSQL, and your database's max_connections was left at the default 100, do the arithmetic before you add that second node:

required = (primary pool size) + (secondary pool size) + headroom

Most application connection pools default to something in the 20-50 range. Two servers pointed at one database can burn through the PostgreSQL default embarrassingly fast — and the failure mode you'll see almost never mentions max_connections directly.

And more generally: when a vendor's error output is genuinely uninformative, and public search turns up nothing, decompiling the vendor's own compiled classes (for diagnostic purposes, not license circumvention) is a legitimate escalation path. It took me from "four plausible-but-wrong fixes and no resolution" to "exact root cause, quantified, fixed on the first retry" — faster than waiting on a support ticket, though vendor support is still the right call when you don't have the time or tooling to go this deep.

Diagram: The Investigation Funnel

flowchart TD
    A["🔴 Symptom: JVM dies every 6-7 seconds<br/>'received process interrupt'"] --> B["Round 1: Config-level fixes<br/>4 real bugs found and fixed"]
    B -->|"Still dies the same way"| C["Round 2: Deep public research<br/>Confirmed: generic wrapper message"]
    C -->|"No precedent found"| D["Round 3: Correct DB tables<br/>fosnodedetails, fosparams"]
    D -->|"Still dies the same way"| E["Round 4: Decompile bytecode with CFR<br/>Trace real call chain across 5 classes"]
    E --> F["🎯 Found: JVM dies before reaching<br/>any Failover-specific logic"]
    F --> G["Read the raw stderr log in full"]
    G --> H["🎯 Root Cause: CreateConnectionException<br/>PostgreSQL max_connections exhausted"]
    H --> I["✅ Fix: max_connections 100→250<br/>Secondary starts on first retry"]

    style A fill:#ff6b6b,color:#fff
    style H fill:#51cf66,color:#fff
    style I fill:#51cf66,color:#fff

Diagram: The Call Chain Traced From Decompiled Source

flowchart LR
    A[StartupHooks<br/>preStartServer] --> B[StartupCheckHandler<br/>doPreCheck]
    B --> C[Preprocessor<br/>initialize coldStart]
    C --> D["moduleInit()<br/>❌ Ruled out - log never appeared"]
    C --> E[StartupCheckHandler<br/>doDBMemoryCheck]
    E --> F["sqlChecks()<br/>❌ Ruled out - returns true immediately"]
    C -.->|"JVM dies before reaching this point"| G["Connection Pool<br/>Bootstrap"]
    G --> H["🎯 CreateConnectionException<br/>max_connections exhausted"]

    style D fill:#868e96,color:#fff
    style F fill:#868e96,color:#fff
    style G fill:#ffd43b,color:#000
    style H fill:#ff6b6b,color:#fff

Tools used: CFR decompiler v0.152, PostgreSQL 17, standard Linux server tooling. No proprietary vendor names in this writeup by design — the pattern generalizes to any Java-based HA product backed by PostgreSQL.