[hpoettker]

Summary

This PR fixes several problems with the sas7bdat writer that made SAS reject files or display them incorrectly. It covers row-compressed (CHAR) files with many rows and uncompressed 32-bit files.

My careful optimism in #424 that the sas7bdat write feature as provided through the pyreadstat API (Roche/pyreadstat#205) is good to go was a bit misplaced. It turns out that there is quite a lot more going on on the COMP page (currently described in the code by the comment "The mostly empty page that SAS writes after the metadata of compressed files") than previously thought.

#423 did the right thing but only addressed the problem for smaller files.

While I previously tested only with smaller files, I now validated the writer against a matrix of 32/64 bit, uncompressed/row-compression, thousands of columns, and millions of rows. And SAS reads the written files correctly.

Row-compressed files

SAS keeps an index of the pages holding compressed rows. The writer only wrote a minimal version of it, which broke once a file had more than a few hundred rows.

• Row count in the index: it was truncated to one byte. SAS reported "Never should happen" errors from about 500 rows.
• Index coverage: the index described only the first page. Files whose rows span several pages couldn't be read.
• Large files: when the index no longer fits on one page, the writer now writes a two-level index (root page plus leaf pages) as SAS does. Tested with 5M rows at 32 KiB and 64 KiB page sizes.
• Row numbering: each page holding compressed rows now carries the number of its first row. SAS reads the data without it, but its data viewer showed wrong row numbers from the second page on.
• Row size header fields: two values in the row size subheader are now filled in as SAS writes them.
• New error: READSTAT_ERROR_TOO_MANY_ROWS is returned when a file would exceed what the index can describe, before anything is written.

Uncompressed 32-bit files

The row size subheader has a field that tells SAS how far the first rows on the first page that holds rows are offset. We made a directionally correct fix in #421. But that was only correct while all metadata fits on the first page, and failed from about 1000 columns. The field is now computed from the actual page layout.

[hpoettker]

The build failed due to capacity constraints on macOS arm64 runners.

I'll push an empty commit to retrigger the build later.

[hpoettker]

The build is now successful.

And the build windows_msvc is now one of the fastest on a cache hit. That's more than I expected to achieve with #426. 😄