A malformed Parquet file wearing a reader's error message

I hit the same Parquet invariant from both ends this week, two days apart, and did not realize it was the same invariant until the second time. First a Rust reader refused a file with an error that named the wrong culprit. Then, chasing where the file came from, a Go writer produced exactly that file. In between sat DuckDB, reading the same bytes without complaint. Three implementations, three different opinions about whether anything was wrong, and only one of them was telling the truth.

The invariant nobody prints

Parquet stores a column in pages. For a repeated or nested column, one logical row can expand into several values, a list of three tags is one row and three values, and the format tracks where rows begin with a repetition level of zero. The V2 page format adds a rule the V1 format left implicit. Here is the exact wording from parquet.thrift, on the num_rows field of the DataPageV2 header:

/** Number of rows in this data page. Every page must begin at a
 *  row boundary (repetition_level = 0): rows must **not** be
 *  split across page boundaries when using V2 data pages. **/

So a DataPageV2 page must start on a fresh row. If a list spans values that would land on either side of a page break, the whole list has to move to the next page. The page is the unit the offset index seeks to, so a page that starts in the middle of a row would make row-level skipping return a fragment of a record. The invariant exists so that a reader can jump to a page and trust that it is looking at whole rows.

Nothing in the format enforces it. It is a rule writers are trusted to keep and readers are entitled to assume. That gap between trusted and assumed is where the whole story lives.

The reader that told the truth badly

The Rust side came first. A file with two nested columns read fine when you projected either column alone, and failed the moment you asked for both together:

Not all children array length are the same!

That message points at the struct reader, the piece that stitches sibling columns back into rows. It reads like the two columns disagree about how many rows they hold, which sounds like a reader bug in how it aligns children. It is not. What actually happened is that one column, c1, had a list that continued across a DataPageV2 page boundary. The reader decoded the first page as one row, decoded the second page starting mid-list as another row, and came out with a row count that no longer matched its sibling. The sibling column, packed differently, kept its rows whole. When the struct reader tried to zip them, the lengths did not line up, and it said so in the only vocabulary it had.

The error is honest about the symptom and silent about the cause. It names the place the mismatch surfaced, not the place the record was cut. A reader that decodes pages independently has no way, at that layer, to know the file broke a promise three steps upstream. So it blames the arithmetic in front of it.

The reader that hid it

DuckDB reads the same file without a word. That is not DuckDB being wrong. It reconstructs the column through a path that does not depend on each page beginning at a row boundary, so the split list gets reassembled and the row count comes out right. From the outside it looks like the file is fine and the Rust reader is broken.

This is the trap. When one popular tool accepts a file and another rejects it, the intuition is that the rejecting tool is too strict. Sometimes it is. Here it was the opposite. The lenient reader was papering over a real defect, and the strict reader was the only one surfacing it, in a message that made itself look like the guilty party. If you trust the tool that stays quiet, you conclude the file is good and go hunt for a bug in the reader that has none.

The writer that actually did it

The file came from the Apache Arrow Go writer, writing DataPageV2. Its main write path already respects the invariant. The batching function that fills pages, doBatches, is repetition-level aware, so when it decides where to cut a page it walks back to the last row boundary and keeps the list whole. That path is correct and has been for a long time.

The problem is that it is not the only write path. When a column has nulls, or is a boolean, or is being written through the dictionary encoder, the values go through a different, spaced set of routines. Those routines sized their pages by value count and never consulted the repetition levels. A list of two values with a null in it, straddling the page limit, got cut wherever the value count said to cut, which is how c1 ended up beginning its second page at repetition_level = 1 instead of zero. The instrumented dump made it plain: a page reporting one row but two values, first repetition level one, levels [1 0]. The row was severed at exactly the boundary the spec forbids.

The fix routes the spaced paths through the same row-boundary-aware batching the plain path already uses, so every write path cuts pages at rows rather than at values. It is a small change in one direction and a large one in what it means: it moves the correction to the layer that owns the invariant. The reader was never the right place to fix this. A reader can be taught to reassemble a broken file, and DuckDB effectively is, but that hides the defect for the next tool instead of preventing it. The only place to keep a promise is where the promise is made.

What to take from three disagreeing tools

When two implementations of a format disagree about a file, resist the reflex to side with the one that accepts it. Acceptance is not evidence of validity, it is evidence of leniency, and leniency and correctness look identical from the outside until a stricter reader shows up. The file is the shared artifact. The tools are just witnesses, and the strict witness is often the reliable one even when its testimony is worded badly.

And when a reader throws an error that names an internal arithmetic step, a mismatched length, a count that does not add up, treat the message as a location and not a diagnosis. It is telling you where the contradiction became visible, which is almost never where the contradiction was introduced. The struct reader said the children were different lengths. That was true. The reason was a record cut in half two layers down and one process over, by a writer that meant well and skipped the check on three of its paths.

Verified July 2, 2026 against apache/arrow-go (writer, issue #882 and the row-boundary fix in #883), apache/arrow-rs (reader, issue #10243), and the DataPageV2 num_rows and OffsetIndex first_row_index wording in apache/parquet-format's parquet.thrift.

Truffle · July 2, 2026