Repository navigation
Read remote FLAC files with fewer requests - #621
Merged
Merged
Conversation
soundfile reads FLAC files 8 kB at a time, so reading a remote FLAC file took one HTTP request per 32 kB buffer: 229 requests for a 7.4 MB MIMIC-IV signal file, each waiting a second or more for PhysioNet to respond. NetFile now doubles the size of the next request (up to 8 MB) when a read continues where the buffer ends, so the same file takes 8 requests. Reads elsewhere in the file still start with a small request. Fixes MIT-LCP#426.
tompollard
approved these changes
Nov 1, 2026
tompollard
left a comment
Member
There was a problem hiding this comment.
Thanks, this looks great! I tried it with a 2.7 MB FLAC record on the test server: a full read went from 82 requests to 7 with the same bytes downloaded, reads from the middle of the file are unchanged, and partial reads download well under twice what's needed (20 minutes of signal: 0.52 MB instead of 0.49 MB, in 5 requests instead of 15).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #426.
soundfile reads FLAC files 8 kB at a time, in order.
NetFilebuffers 32 kB per request, so a remote FLAC file took one HTTP request per 32 kB. PhysioNet takes a second or more to answer each request, so most of the time went into waiting.NetFilenow doubles the size of the next request (up to 8 MB) when a read continues where the buffer ends. Reads anywhere else in the file go back to the normal 32 kB request, so random access works as before, and at most about twice the data actually read is downloaded. This helps any reader that reads a remote file in small pieces, not only FLAC.buffering=0andbuffering=-2are unchanged.Measurements:
81739927_0008from mimic4wdb (3 FLAC files, 9.4 MB): requests81739927_0008e.datalone (7.4 MB): requestsThe PhysioNet timings were 385 s before and 241 s after for that segment. The whole record from the issue (
81739927, 22 segments) now reads in about 4 minutes with 209 requests. Before this change my run took over 10 minutes and then failed withLibsndfileError: Internal psf_fseek() failed(I didn't keep the full traceback, so I can't say whether that was a network error). The rest of the time is the download itself, and PhysioNet's speed varied a lot while I was testing (the same file took 14–80 s with plainrequests), so I've left the transfer itself alone.Tests: the dummy server now records the Range header of each request.
test_sequential_readsreads a 2 MB file in 8 kB pieces and checks that it takes at most 8 requests (64 before this change).test_readahead_resetchecks that a read after a seek uses a normal 32 kB request again.