Concurrency Without a Parallel Parser: Splitting PCAPs by Session

tshark needs linear state, so I moved the concurrency upstream — and hit a corrupted-output problem on the way.

Where the last post left off

In my first post about the 2.5 GB wall, streaming fixed memory. It didn't fix throughput.

The pipeline was still one file, one tshark process, one core. Piping stdout to stdin doesn't parallelize anything — it just stops the pipeline from holding the whole file in memory at once. Processing time dropped, but the shape of the work didn't change: still sequential, still bottlenecked on a single core no matter how large the machine underneath it was.

A few people asked the obvious question in the comments: why not just add goroutines?

I tried. It didn't work, and the reason it didn't work is more interesting than the fix.

Why the parser can't parallelize

Goroutines help when work can be split into independent pieces. tshark's dissection isn't independent — it's linear state.

It reads a capture packet by packet, and what it reads in one packet changes how it decodes the next. TCP stream reassembly needs the packets in order to rebuild the byte stream. Connection state tracks which flow a packet belongs to. Dissectors that depend on earlier context — retransmission detection, duplicate ACK detection, anything under tcp.analysis.* — read and update shared conversation tables as they go.

None of that is optional. tshark wasn't written single-threaded because nobody got around to parallelizing it — the dissection model requires strict ordering. Skip packets out of order and the state tshark is tracking for that stream goes wrong.

So wrapping the consuming side in goroutines doesn't help, because the problem was never on the consuming side. My Go program reading tshark's output line by line was already fast. The producing side — tshark itself, walking the file front to back — is sequential no matter what I do to the code around it.

If I wanted concurrency, it had to happen before tshark ever saw the file.

Move the concurrency upstream

If one tshark process can't go faster, run several over pieces of the file at once.

The obvious version of that is wrong. You can't cut a pcap into equal-sized chunks — a TCP stream that straddles a cut point loses the packets on the other side of the boundary, and the dissector on that chunk has no idea the connection existed before the cut. Retransmission flags, sequence tracking, anything stateful breaks the moment a conversation is split across two files.

The fix is to split along conversation boundaries instead of byte boundaries. Group packets by session — same 5-tuple, same connection — so every chunk holds complete conversations and nothing crosses a chunk boundary mid-stream. No state is lost, because no state ever needed to survive past a single chunk.

Once the split is done that way, the chunks are independent by construction. Each one goes to its own tshark process, running in parallel, merged back together at the end. The dissection itself stays linear — it just becomes N linear passes running at once instead of one linear pass running alone.

That was the plan. Getting there took one extra tool falling apart on me.

The bug

I reached for PcapSplitter, part of PcapPlusPlus, running in connection mode — split by 5-tuple, exactly what I needed.

In connection mode it holds one output file open per distinct flow for the length of the pass. On the files that mattered, that was 95 to 125 distinct flows open at once. Under that load, the output came back corrupted.

Two distinct failure signatures:

• total block length N of an EPB is too small for M bytes of packet data
• total block length 0 of an unknown block type is less than the minimum block size 12

I reproduced both on the master build and on the v25.05 stable release, so this wasn't something already fixed between versions with me sitting on a stale checkout. I reproduced both on pcapng and on legacy pcap after converting with editcap -F pcap, so it wasn't specific to the pcapng container format either. And I ran the source file through pcapfix before assuming the input was the problem — it came back clean. The corruption was being introduced by the split, not inherited from the source.

Same root cause both times: too many output file handles held open at once by one process. I didn't dig further into PcapSplitter's internals to find the exact line — I didn't need to, because the fix wasn't going to be a patch. I replaced it.

Reimplementing it in-process

I rewrote the session split using gopacket, inside my own program, instead of shelling out to a separate tool. No external process, no file handles held open by something I don't control — the splitting and the writing happen where I can see and bound exactly what's open at once.

Ran it 20 times across both files that had triggered the original corruption, in both the Go and Python versions of the pipeline. 20 out of 20 succeeded, including full concurrency on the file that had reproducibly failed before.

The honest ending

Here's the part worth sitting with.

Session splitting only kicks in above 100,000 packets — below that, one tshark process handles the whole file fine. I went back through the real corpus this pipeline processes day to day: 57 files. Exactly 3 of them cross that threshold.

I built session splitting, and everything downstream of it, for a problem that shows up in about 1 out of every 19 files. It was the right thing to build — the 3 files that need it would otherwise take hours or fail outright — but it's worth saying plainly: most of the corpus never touches this code path at all.