Understanding the ORC File Format: Why Is It So Fast

Every article about ORC says the same thing.

"Columnar storage, built-in indexes, predicate pushdown - so it's fast."

That's not an explanation. That's a slogan.

So I wrote a small ORC parser from scratch, generated 5 million rows, and opened the file with my hands.

The question I actually wanted answered:

What happens inside an ORC file when a query asks for 2 columns out of 100?

What a Columnar File Actually Does

Strip away the marketing, and it comes down to three things.

• Store each column's values together, so you can read one column without touching the others.
• Encode those values in ways that only work when neighbours are similar.
• Keep a small map of what's inside, so a query can skip most of the file without opening it.

Everything else - compression codecs, bloom filters, ACID support - is built around that core job.

The Setup

An orders table. Deliberately boring.

order_id     int64    sequential
customer_id  int64    250k distinct
country      string   10 distinct
product      string   2,000 distinct
quantity     int32    1-19
price        double   40k distinct
order_date   date     730 days
status       string   5 distinct

5,000,000 rows, written with PyArrow:

orc.write_table(
    tbl, "orders.orc",
    compression="zlib",
    stripe_size=8 * 1024 * 1024,      # small on purpose
    row_index_stride=10_000,
    dictionary_key_size_threshold=0.8,
)

Production ORC uses 64-256 MB stripes. My whole file is 53 MB, so a default stripe would swallow the entire dataset and there'd be nothing to skip. Shrinking it is how you see multi-stripe behaviour on a laptop.

Same data also went out as a 310 MB CSV, for scale.

ORC Is Read Backwards

The last byte of the file is a single number: the length of the PostScript.

Here are the final 26 bytes of my file:

08 c0 03 10 01 18 80 80 04 22 02 00 0c 28 97 05 30 06 82 f4 03 03 4f 52 43 19

That's a Protocol Buffers message. Decoded:

So a query engine does three reads: last byte, PostScript, then footer + metadata. A few hundred bytes on a 53 MB file.

After that it knows the schema, the row count, every stripe's byte offset, and min/max stats for every column.

It hasn't touched a single row of data yet.

Note the ordering: the codec lives in the PostScript, and the PostScript is never compressed. It can't be - you'd need the codec to decompress the thing that tells you the codec.

Stripes

A stripe is a self-contained slab of rows. Its own index, its own data, its own footer.

 #       offset     index         data  footer       rows
 0            3     8,412    8,544,070     159    687,104
 1    8,552,644     8,487    8,556,939     158    688,128
 ...
 7    59,917,580     2,796    2,315,836    152    186,176

8,412 bytes of index for 8.5 MB of data. 0.1% overhead for the ability to skip.

That ratio is the entire trade.

Because each stripe carries its own footer, you can hand a Spark executor one byte range and it can decode that stripe with zero knowledge of the rest of the file. That's the basis of parallel scans.

Where a Column Physically Lives

Inside a stripe, a column isn't "a column." It's a set of streams.

col name       stream                   bytes
  3 country    DICTIONARY_DATA             23
  3 country    LENGTH                       5
  3 country    DATA                   310,404
  6 price      DATA                 2,374,052

All streams for one column sit contiguously. "Read country from stripe 3" is one range read, not 687,104 scattered seeks.

There's also a PRESENT stream for nulls - and ORC omits it entirely when a column has none. My columns are all non-null, so null tracking cost zero bytes. A row format pays for nullability on every row whether you use it or not.

Encodings Do More Than Compression Does

People credit ORC's size to compression. Mostly wrong.

Bytes per column, whole file:

Five million 64-bit integers in 11 KB.

That's ORC's RLE v2 picking its DELTA sub-encoding: for a monotonic sequence it stores a base, a delta width, and a run length. zlib never saw 40 MB of integers, because the encoder never produced them.

Same effect on sorted dates:

order_date, 5M values:
  random order : 7,390,184 bytes
  sorted       :     4,898 bytes    <- 1,500x smaller

The PyArrow gotcha

dictionary_key_size_threshold defaults to 0.0, which disables dictionary encoding entirely.

14% of the file, from one keyword argument.

Compression Sits On Top

Only after encoding does the codec run - in independent 64 KB chunks, each with a 3-byte header. Bit 0 says "stored raw", because if compressing a chunk made it bigger, ORC just doesn't.

Chunking is what makes seeking possible. One big compressed blob would mean inflating 8 MB to read the last value.

zstd wins on all three axes. zlib is only still the default because ORC is old.

Also worth noting: uncompressed ORC is still 3.6x smaller than CSV. That gap is pure layout and encoding.

I Actually Measured the Skipping

ORC keeps min/max statistics at three zoom levels: file (448 B), stripe (663 B), and row group of 10,000 rows (8.4 KB per stripe).

Query:

SELECT price FROM orders
WHERE order_date BETWEEN '2024-06-01' AND '2024-06-30'

One month out of 24. About 4% of rows. Same query, same data, two files - one in insertion order, one sorted by order_date.

Unsorted:

stripe 0   2023-01-01 .. 2024-12-30   READ
stripe 1   2023-01-01 .. 2024-12-30   READ
...
stripes to read: 8/8
bytes read: 24,430,256

Every stripe spans the whole date range, so every min/max says "maybe". Zero pruning.

Sorted by order_date:

stripe 0   2023-01-01 .. 2023-04-10   SKIP
stripe 4   2024-02-02 .. 2024-05-11   SKIP
stripe 5   2024-05-11 .. 2024-08-19   READ
stripe 6   2024-08-19 .. 2024-11-26   SKIP
...
stripes to read: 1/8
bytes read: 2,335,435          <- 13.7% of the unsorted read

And inside that one surviving stripe, the row-group index eliminated another 68%: 22 of 69 row groups matched. The engine ends up decoding roughly 220,000 of 5,000,000 rows.

The uncomfortable conclusion:

ORC's statistics are a summary, not an index. Their usefulness is entirely determined by how you wrote the data.

Sorting by your dominant filter column isn't a tuning detail. It's the difference between skipping 87% of the file and skipping nothing.

A predicate stats can't help with

country = 'IN'
stripe ranges: ('AU','US'), ('AU','US'), ('AU','US'), ...
stripes that could contain 'IN': 8/8

Scattered values mean every range straddles the target. That's the gap bloom filters fill - they cost 1.4% of file size on product, and are a waste on country.

Does Column Projection Actually Pay Off?

I wrapped the file handle in a counter so I could see every byte the reader actually pulled off disk.

8 columns, 53 MB:

100 columns, 1M rows, 153 MB - the original question:

2 columns out of 100 = 2.23% of the bytes. Not "less I/O" in a hand-wavy sense. 45x less, measured.

Reading 2 columns from the 310 MB CSV still requires pulling all 310 MB off disk and parsing every field to find the commas.

How Big Should a Stripe Be?

Same sorted data, same June predicate:

Halve the stripe, halve the rows you're forced to scan - until pruning bottoms out at the real selectivity. Total file size moved 0.3% across that whole range, because more stripes cost more metadata, not more data.

So why not 2 MB stripes everywhere? Because a stripe is also the unit of parallelism, and on object storage each one is a separate GET. The row-group index already gives you 10,000-row granularity inside a big stripe - that's the layer meant to do fine pruning. Keep stripes big.

What I Couldn't Test

• A real cluster. Everything here is one machine, local NVMe. On S3, request count matters more than byte count, and my numbers say nothing about that.
• lz4 in this build. It produced 86.02 MB - essentially identical to uncompressed. Either the PyArrow build isn't wiring it up or something else is wrong. I'm reporting it rather than quietly dropping the row, but don't trust that number.
• Bloom filter effectiveness. I measured what they cost (1.4% of file size), not what they save, because PyArrow's reader doesn't expose filter pushdown for ORC. You'd need Hive or Trino for that.

Where This Doesn't Help

Columnar layout is a bet, and it loses on some workloads.

• Not for OLTP. Inserting one row means rewriting a stripe. Fetching one full record means touching every column's streams separately.
• Small files kill it. All this metadata machinery amortizes over large files. A thousand 2 MB ORC files is the classic way to make a fast format slow.

One Thing a Different Columnar Format Does Differently

ORC isn't the only format built on these ideas - Parquet uses the same stripe/row-group/statistics playbook. The clearest place they diverge is encoding scope.

Same data, same codec, same row grouping:

order_id (sequential)    ORC 7.8 KB    vs   Parquet 7.2 MB     (912x)
price (40k distinct)     ORC 16.3 MB   vs   Parquet 10.2 MB    (0.62x)

ORC's dictionary encoding applies only to string columns - a DOUBLE always goes out DIRECT, full width, and only the compressor gets a shot at it. Parquet dictionary-encodes any physical type, so a double with modest cardinality compresses to narrow indices where ORC can't.

That's a real, measured trade-off, not a verdict. It's also its own rabbit hole - I measured it properly in a follow-up post.

So, Why Is It Fast?

Not one reason. A chain, where each link multiplies the next.

columnar layout      -> read 2 columns, not 100          (45x fewer bytes)
  x encodings        -> delta/dictionary before codec    (up to 3,467x on one column)
  x compression      -> chunked, seekable                (~2x on top)
  x statistics       -> skip stripes that can't match    (8x when sorted)
  x row-group index  -> skip 10k-row blocks inside those (3x more)
  ---------------------------------------------------------------
  = read 2.2% of the file, decode 4% of the rows you touched

Every layer is doing the same thing from a different angle: making it possible to not read something.

ORC isn't fast because it's clever with the bytes it reads. It's fast because it spends 0.1% of the file on a map detailed enough to avoid reading almost all of them.

And the part that's actually in your control: sorting by your filter column moved my query from 24 MB to 2.3 MB. No format setting in this entire post came close to that.