Short answer: test private object storage as a recovery boundary, not as a bucket with an attractive upload number. For a media app that serves generated reports to authenticated customers, the winning design is the one that can prove three things during the same drill: the right customer receives the right generation, a deleted application index can be rebuilt, and a complete EU or US recovery has an owner and an observable stopping point.
That changes the experiment. Instead of starting with a rate sheet, start with a broken application: the report database is unavailable, the object store still contains immutable report bundles, and a customer needs one report now. The simple approach is to expose a signed download link and call the upload successful. The useful approach treats the manifest, customer authorization, and delivery path as separate evidence.
I build RAG and agent features in Python, so I am suspicious of demos that pass one happy-path prompt and skip the eval harness. Storage deserves the same discipline. The notebook should contain a fixed report corpus and fixed assertions; production should run the same checks against the chosen region and retention policy. Small test. Big difference.
Cost starts with the workload shape
For app backups, restore costs are an input to the experiment, not its conclusion. Separate stored data, restored data, requests, transfer, retention, and the engineering time for a recovery drill. One archive size cannot stand in for a media application's request pattern.
That is why the test corpus needs a database export, several large report PDFs, many small thumbnails, and a manifest. A workload dominated by small objects can exercise a different request shape from one dominated by a few archive files, even when both show the same stored gigabytes. Your mileage may vary.
Treat the manifest as the interface
The most useful design decision is to make a report generation a portable artifact with a manifest. The manifest records object keys, checksums, media types, and a generation identifier; the application record stores customer ownership and current-generation state. That contract gives a notebook, a worker, a recovery job, and a customer-facing service something stable to test while the object-storage backend changes.
The failure this exposes is ordinary: a report generation completes, a customer requests it, and the application cannot tell which bytes belong to that customer's generation. A second version may have replaced the first. A URL may have outlived the intended authorization. A bundle may contain its main PDF but omit one generated image. In a media workflow, that last case is easy to miss: the PDF opens, the page count looks right, and only a customer viewing a chart discovers that the thumbnail object was absent. The recovery job then needs to distinguish a missing object from an unauthorized request, while the delivery service needs to distinguish a stale generation from a current one. Those are separate decisions with different evidence, and folding them into one “download succeeded” metric makes the runbook look healthier than the system is.
Model the report as a generation with a manifest, rather than as one convenient filename. A manifest records the object keys, checksums, media types, and generation identifier. The application database records who owns the report and which generation is current. The private object store holds the bytes. During recovery, the manifest is the source for reconstructing the application-side index.
Here is the smallest useful domain model. It does not know which object-storage service is underneath, and it does not let a browser-supplied key decide ownership.
from dataclasses import dataclass from datetime import datetime @dataclass(frozen=True) class ReportGeneration: report_id: str customer_id: str generation_id: str object_keys: tuple[str, ...] checksums: tuple[str, ...] def may_download( generation: ReportGeneration, authenticated_customer_id: str, requested_generation_id: str, ) -> bool: return ( generation.customer_id == authenticated_customer_id and generation.generation_id == requested_generation_id ) The check is intentionally dull. Dull checks are easier to evaluate. The server looks up the report by an internal identifier, compares the authenticated subject with the stored customer identifier, and only then creates a short-lived delivery capability for the selected generation. The browser never receives a storage credential that can enumerate a customer's archive.
This is the boundary.
The first assertion is a negative one: customer B must receive no bytes for customer A's generation. I would keep that failed authorization attempt in the test evidence. It tells us more about the access boundary than a successful download does. It can't be reduced to a storage-rate comparison.
What should app backups prove before an EU or US private object storage restore?
Treat the restore as a state transition with a clear beginning and end. A scheduled worker writes a bundle and manifest. A verifier checks every listed object and checksum. Only then does the application mark that generation complete. A recovery worker reads the manifest and reconstructs the index. A customer-facing service authorizes a download separately.
This ordering prevents a plausible-looking archive from becoming a false success. If the report index is rebuilt before checksum verification, an incomplete bundle can look available. If authorization is performed after a URL is minted, the URL may become a second identity system. If the current generation is selected by filename alone, a retry can return an older or newer report without making that choice visible.
The drill should use the shape of the real media data: one database export, several large report PDFs, many small thumbnails, a manifest, and a report regenerated while the test is running. Do not replace this with one large zero-filled file. Object count, request count, interrupted downloads, and generation races are part of the application behavior.
I keep the assertions explicit in Python so a backend change does not silently change the experiment.
from dataclasses import dataclass @dataclass(frozen=True) class RestoreEvidence: manifest_verified: bool object_count_matches: bool checksums_match: bool customer_is_authorized: bool expired_capability_denied: bool index_rebuilt: bool def restore_passed(evidence: RestoreEvidence) -> bool: return all( ( evidence.manifest_verified, evidence.object_count_matches, evidence.checksums_match, evidence.customer_is_authorized, evidence.expired_capability_denied, evidence.index_rebuilt, ) ) Run this once for a routine report request and once after deleting the application-side index in a test environment. I would also retry an expired capability, interrupt a large download, and request an old generation while a new one is being written. The expected result for each case belongs in the harness before the storage service is compared.
Measure recovery completion time, time to first byte, object count, request count, transferred bytes, checksum mismatches, and operator actions. Record the selected region, retention period, and object layout beside the result. Your mileage may vary: a workload dominated by small thumbnails has a different request shape from one dominated by a few archive files, even when both show the same stored gigabytes.
Put the storage boundary behind a migration-sized interface
The application should need a compact contract: write an object, read an object for a trusted recovery worker, list objects for a known prefix, inspect metadata, and create a temporary customer download. The adapter can translate those operations to the interfaces of Backblaze B2, Cloudflare R2, AWS S3, Wasabi, or a self-hosted object store. The report service should not import provider response types into its authorization and manifest code.
This is a developer-experience decision with a reliability payoff. A narrow interface makes it possible to run the same Python corpus against a staging region, a migration target, and a recovery copy. It also keeps prompt-cost-aware application code from growing a new storage-specific branch every time an agent feature needs an artifact. I want the backend choice visible in configuration and test output, not scattered through notebook cells.
There is a limit to this abstraction. A common interface cannot erase differences in lifecycle controls, residency commitments, request semantics, transfer behavior, or recovery tooling. Keep those differences in the decision record and in integration tests. An adapter that hides an important boundary is less portable, not more.
For customer delivery, use ordinary HTTP metadata as part of the contract. Content-Disposition lets the application suggest a filename for a generated report; the application still needs to choose the content type and avoid trusting a browser-provided value. The delivery path should log the report identifier, customer identifier, generation, authorization result, and expiry time, while redacting the temporary capability itself.
Compare evidence after the drill
The familiar comparison set for app backups includes Backblaze B2, Cloudflare R2, AWS S3, and Wasabi. Naming the candidates is less useful than giving each one the same test. A storage option should earn a place in the architecture by passing the application's access, recovery, and operations assertions in the required EU or US location. The table below is a decision record, not a leaderboard.
Use a matrix like this after the drills:
| Boundary | Evidence from the drill | A reason to reject the option |
|---|---|---|
| Customer access | Cross-customer denial, generation match, and expiry denial | Authorization cannot remain in the application |
| Recovery | Manifest rebuild, object count, checksums, and completion time | A restore requires an undocumented manual sequence |
| Residency | Selected EU or US location and current contractual terms | The required data boundary cannot be demonstrated |
| Delivery | Large-file metadata, interruption behavior, and retry result | The customer path needs an opaque proxy with no clear purpose |
| Lifecycle | Retention and transition rules exercised in a test account | Deletion or retention behavior cannot be explained to operators |
| Operations | Logs, alerts, runbook owner, and last successful drill | Nobody owns the next recovery exercise |
The catch is that private object storage does not solve customer authorization, immutable backup policy, or recovery ownership by itself. It is not suitable when the surrounding architecture cannot meet a required retention or residency control. Stick with the option and design that make the missing control demonstrable, even when its upload path is less familiar.
A rate-sheet result without the measured restore shape is an estimate, not a decision.
Make governance part of the release
Before copying the choice into production, make two drills a release requirement: a routine authenticated report download and a complete recovery from the manifest. The second drill should begin with the application index unavailable. The operator should know which objects to validate, which generation to rebuild, how to verify checksums, and how to close customer access until authorization records are restored. Put the owner, retention rule, selected region, and evidence location in the service record so those decisions survive a team change.
The final artifact is not a vendor score. It is a repeatable record containing the dataset version, region, object layout, assertions, timings, failures, and owner. That record lets a team compare an EU deployment with a US deployment without pretending they have identical constraints.
Three words I want in the acceptance note: authorized, recoverable, observable. If one is missing, the backup is only partially tested.

