Skip to content

Read remote FLAC files with fewer requests - #621

Merged
tompollard merged 1 commit into
MIT-LCP:mainfrom
TheNetherWatcher:flac-remote-reads
Nov 1, 2026
Merged

tompollard merged 1 commit into
MIT-LCP:mainfrom
TheNetherWatcher:flac-remote-reads

Conversation

@TheNetherWatcher

@TheNetherWatcher TheNetherWatcher commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #426.

soundfile reads FLAC files 8 kB at a time, in order. NetFile buffers 32 kB per request, so a remote FLAC file took one HTTP request per 32 kB. PhysioNet takes a second or more to answer each request, so most of the time went into waiting.

NetFile now doubles the size of the next request (up to 8 MB) when a read continues where the buffer ends. Reads anywhere else in the file go back to the normal 32 kB request, so random access works as before, and at most about twice the data actually read is downloaded. This helps any reader that reads a remote file in small pieces, not only FLAC. buffering=0 and buffering=-2 are unchanged.

Measurements:

Before After
81739927_0008 from mimic4wdb (3 FLAC files, 9.4 MB): requests 292 20
81739927_0008e.dat alone (7.4 MB): requests 229 8
5.4 MB FLAC from a local test server with 0.2 s per request 165 requests, 34.7 s 8 requests, 2.0 s

The PhysioNet timings were 385 s before and 241 s after for that segment. The whole record from the issue (81739927, 22 segments) now reads in about 4 minutes with 209 requests. Before this change my run took over 10 minutes and then failed with LibsndfileError: Internal psf_fseek() failed (I didn't keep the full traceback, so I can't say whether that was a network error). The rest of the time is the download itself, and PhysioNet's speed varied a lot while I was testing (the same file took 14–80 s with plain requests), so I've left the transfer itself alone.

Tests: the dummy server now records the Range header of each request. test_sequential_reads reads a 2 MB file in 8 kB pieces and checks that it takes at most 8 requests (64 before this change). test_readahead_reset checks that a read after a seek uses a normal 32 kB request again.

soundfile reads FLAC files 8 kB at a time, so reading a remote FLAC
file took one HTTP request per 32 kB buffer: 229 requests for a 7.4 MB
MIMIC-IV signal file, each waiting a second or more for PhysioNet to
respond. NetFile now doubles the size of the next request (up to 8 MB)
when a read continues where the buffer ends, so the same file takes 8
requests. Reads elsewhere in the file still start with a small request.

Fixes MIT-LCP#426.

@tompollard tompollard left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks great! I tried it with a 2.7 MB FLAC record on the test server: a full read went from 82 requests to 7 with the same bytes downloaded, reads from the middle of the file are unchanged, and partial reads download well under twice what's needed (20 minutes of signal: 0.52 MB instead of 0.49 MB, in 5 requests instead of 15).

@tompollard
tompollard merged commit 95553bb into MIT-LCP:main Nov 1, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance issue while reading the data with FLAC file format via HTTP.

2 participants