A phone backup that dedups by content hash and keeps running with the screen off

An Android client and a backend that together backed up 19 GB of photos and 8K video — and the three bugs that hid behind green tests.

2026 · mobilestorage

files

2,329

one person's camera roll

backed up

18.98 GB

64 videos among them

hash throughput

731.8 MB/s

against a 50 MB/s gate

backend tests

929

203 more on the client

Context

MemberHub is a Django monolith giving a member organisation nineteen tools, one of which is a file manager on S3-compatible storage. This sits on the same bucket and the same models: an Android client that copies a person's camera roll into a private space of their own, and the endpoints that receive it.

Private is the whole design. A backed-up photo still carries the organisation foreign key, because that is how storage spend stays attributable — but every organisation-wide listing excludes rows with a personal owner, so an organisation's own administrator cannot see their own backup through the Files tool. That is why the browser viewer had to be a new door, not a relaxed filter.

The other constraint was that I do not have the phone, so the rule became: pick gates I can measure myself, from the database or the access log, and write the rest down as owed. The roll it had to survive was roughly 2,300 items and about 36 GB, with single clips of 8K video running 202 MB for 20.7 seconds.

What I built

The client is a state machine over one item at a time: scan the camera folder, hash the file, ask the server whether that hash is already held, take a presigned PUT if not, upload the bytes, confirm. A local ledger row records where each item got to. It scans the camera folder, not the whole device: the first version filled the backup with chat images and screenshots, and on Android an album is just a folder.

1. Scanthe camera folder, over a window the ledger decides
2. Hashstreaming SHA-256 on the device, counted in bytes read
3. Checkis this hash already held for this person?
4. Presigna PUT URL for one new object key
5. PUTthe bytes go straight to the bucket
6. Confirmthe server HeadObjects the key, then marks it ready

Hashing is a small Kotlin module streaming SHA-256 off the file, built first because the dedup key depended on it: 4152.0 MB in 5674 ms, or 731.8 MB/s, against a design gate of 50 MB/s, and digests matching sha256sum on device. It reports bytes read, not the file's stat size: a stream that ends early yields a well-formed SHA-256 of a prefix and an inflated throughput number, so the failure looks better than the success. And a prefix hash collides — two clips from one camera share a header, so the second is never backed up.

The server trusts nothing the client says. Confirm HeadObjects the key before recording the real size and content type, so a row marked ready is evidence the bytes exist rather than a claim they were sent; it accepts only a row still pending, so a replay cannot rewrite the quota.

Background execution is a foreground service whose only job is to stop Android reclaiming the process. The premise was tested before anything was built on it: React Native's JS thread keeps running on Android while the process lives, unlike iOS. Then it was measured from the server, not reported from the phone: with the app closed and the screen locked, 63 files and 479 MB arrived in four minutes.

Two ends make it usable. A Photos tab in the member portal gives a month-grouped grid, a lightbox, download and a trash. And a transcode worker, because backup uploads skip the derivative pipeline web uploads run through: the clips are 8K HEVC, which almost no browser decodes, so they uploaded perfectly and would not play. A timer claims one video every five minutes at the lowest priority and writes a 1080p copy plus a poster frame.

Decisions

Measure the hash before designing around it. Content hashing is the better dedup key by a distance, but only if it can run over a whole roll on a phone. The alternative was size, capture date and filename — cheap, and wrong in the cases that matter. Measuring first cost one spike, not a backend redesign.

Wi-Fi-only defaults on, and anything not definitely Wi-Fi counts as not Wi-Fi. The failure directions are not symmetrical: guessing toward Wi-Fi spends 36 GB of somebody's mobile data, guessing the other way costs one wake.

Inject every native and network dependency. The client's core may not import platform modules or call fetch, and a grep enforces it, so two hundred tests were possible with no device in the room. It is also their limit: that environment renders no components, which is how one build shipped black text on a dark background with a green suite.

What went wrong

The backup locked itself out, and the screen said nothing was wrong. A run ended by stamping a watermark with the current clock unless an authentication failure had stopped it, and the scan asks the device for assets created after that value. Once the watermark reached the present, nothing already on the phone could be listed again — and the scan is the only way an item enters a run. "Not stopped" is not "clean": a dropped connection arrives as status 0, counts as retryable and increments no counter, so a run could drain with 4,118 items unfinished and look perfect.

The field that diagnosed it was 0 already backed up. Photos were provably in the backup, so a scan returning even those would have counted them as known; zero meant the asset list was empty, not unproductive. Read which counter is zero, not that all of them are. Underneath was a worse version of the same fault: the ledger was documented as the resume point and nothing read it back, because its branches were reachable only for assets the scan returned. An unreachable resume path reads like a working one. Now the ledger decides the window.

A lock wired into one of two callers. A run lock existed so a background wake could not collide with a foreground run, and only the periodic path acquired it. The test that proved the property was green because it pre-acquired the lock by hand — a test that sets up the state production was supposed to create is not a test of that code. Nor was it a rare race: a wake fires only while the activity is backgrounded, which is the screen-off drain the feature exists for. My plan text had wired the lock into one caller and never mentioned the other.

One 8K frame took 24.7 seconds to decode, against a 20-second limit shared with the other subprocess helpers, so poster frames kept timing out. Frame extraction now has its own 90-second budget and reads the 1080p copy the worker just wrote. The related discovery: those thumbnails had never existed. Upload pre-allocated a thumbnail key for every file while the client deliberately skips video thumbnails, so sixty-odd rows named an object nobody had uploaded. A pre-allocated key is a claim; the rule now is that a stored key implies the object exists, and the stale keys had to be cleared before deploying or blank tiles would have become broken ones.

Outcome

The backup holds 2,329 files and 18.98 GB of one person's camera roll. Of 64 videos, 60 are transcoded and carry poster frames; the four that do not are rows whose bytes never finished uploading.

The backend is at 929 tests, the client at 203 across 22 suites, every new guard mutation-checked. What those numbers do not cover is written down rather than implied: eleven on-device checks no test here can reach, and a set of Japanese strings that are correct, keyed, and unreachable because the locale is never set.