Repository navigation
Cache upload checksums to skip uploaded files without reading them - #943
Merged
Merged
Conversation
upload looks up the upload history by the checksums of the files, so it read every file in full before it could skip an already uploaded sequence or video: a run that skips 10,000 images or a long video read all of them first. Cache the checksum of each file in the upload history folder, keyed by its path, size and modification time, as rsync's quick check is. A file whose key is unchanged is not read again: 10,000 cached files are looked up in about 0.1 s. Entries expire after 30 days, which only bounds the size of the cache. The cache is used only when the upload history is, and is written in one transaction per sequence. process does not compute checksums, as mapillary#725 removed that from it to keep it fast.
caglarpir
force-pushed
the
cache-upload-checksums
branch
from
October 30, 2026 09:20
43b3814 to
c3e3813
Compare
caglarpir
added a commit
to caglarpir/mapillary_tools
that referenced
this pull request
Oct 30, 2026
- A track assumed to start with a video without GPS is never refused for its length, in process or in sampling, and the creation time is never used to place it: a MapillaryGPXAlignmentWarning says when the track and the video differ in length (mapillary#944). Only --interpolation_use_gpx_start_time still refuses a long track. - A file that a GPX or NMEA track missed fails with the miss when no later source geotags it (mapillary#938). - The process report lists the geotagging warnings (mapillary#945). - upload caches the checksums of the files, and shows progress while computing them (mapillary#942, mapillary#943).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
uploadlooks up the upload history by the checksums of the files, so it read every file in full before it could skip an already uploaded sequence or video. A run that skips 10,000 images, or a long video, read all of it first.This caches each file's checksum in the upload history folder (
checksum_cache), keyed by its path, size and modification time, like rsync's quick check:processstill doesn't compute checksums: #725 removed that from it to keep it fast.Example,
upload --dry_runtwice with history enabled. The second run skips everything at once, with no checksum bars:Tests: unit tests for
KeyValueStore.set_many(one transaction),PersistentCache.get_many/set_many, and_update_md5sumswith a cache (unchanged file taken from the cache without reading it; changed size read again). Also checked by hand with images and a GoPro video,processthenupload --dry_runtwice.