Full App Backup: User Uploads, Database Dump, and Node.js Nightly Restore

A nightly archive is acceptable only when its recovery-point objective can tolerate a full day of data loss; choose it for a whole-application recovery unit, then prove it by restoring the database dump and user uploads together in an isolated environment. The operational constraint is the restore objective, not whether the scheduled process exits cleanly.

The job should publish one immutable-looking prefix per run and make a manifest the final object. That gives an operator a clear answer during an incident: pick a completed recovery unit, validate it, restore it somewhere new, and switch traffic only after application checks pass.

What should an object storage backup include for user uploads, database dumps, and a restore?

Treat a backup as a named recovery unit rather than a collection of files that happened to arrive overnight. A practical layout uses a UTC identifier, with a separate namespace for the environment:

recovery/prod/2026-08-07T020000Z/database.dump recovery/prod/2026-08-07T020000Z/uploads.tar.gz recovery/prod/2026-08-07T020000Z/manifest.json 

The manifest contains the selected backup ID, creation time, object names, byte counts, and SHA-256 checksums. Upload it last. A restore tool must ignore a prefix without that marker, because partial uploads and interrupted retries are normal operational states, not candidate recovery points.

Avoid making latest authoritative. A convenience pointer can help an operator discover a run, but the incident record should name the dated prefix actually selected. Otherwise a delayed retry can move the pointer while someone is restoring. Keep production and non-production in different access boundaries where the policy burden is justified: the application writes uploads, the backup job writes its recovery namespace, and the restore identity reads it without delete permission. Those duties should not share one broad credential.

The consistency boundary needs a written choice. A database export and an upload archive can be individually valid while disagreeing about an object created during the archive window. A low-write application may briefly pause writes; a busier one may rely on versioned upload objects and reconcile against database metadata. The right choice depends on write rate, tolerated RPO, and whether an uploaded object can be regenerated. There is no safe generic answer.

A green scheduled run is not a recovery signal

A scheduler proves that a process ran. It does not prove the dump loads, the archive contains the intended directory, or the identity used during a restore has the permissions it needs.

Track three separate signals: completion within the job window, publication of a valid manifest under the expected prefix, and the age of the last successful restore drill. Only the final signal exercises the service outcome. An SLO should describe that outcome in terms such as maximum recovery time and maximum acceptable data age; “nightly job succeeded” is supporting telemetry.

Done means restored.

Capacity planning belongs in the runbook because recovery work consumes more than retained storage. Estimate retained bytes, daily change, compression behavior, staging-space demand, request volume, and measured restore throughput. The retention calculation needs an explicit owner and a review date: a one-day increase in retained copies can be harmless for a small export, but a material commitment once uploads dominate the archive, and the restore staging volume must hold the incoming objects as well as the extracted form. Check the size of the largest expected dump, the uncompressed upload tree, and concurrent restore headroom against the actual worker disk instead of assuming the archive's compressed byte count will fit. Object storage cost commonly has storage, request, and data-transfer dimensions, so recalculate from current provider pricing documentation during planning reviews rather than freezing a price into operational prose. Don't let an apparently cheap retention decision turn into a missed restore window because the only worker with enough disk is unavailable. A warning before staging disk is too small for the next dump plus archive is much more useful than an alert after the job has already missed its window.

Retries also need transaction semantics. Give one invocation one backup ID; retries for that invocation write the same intended keys, while a new source snapshot gets a new ID. Build data artifacts privately, close them, calculate checksums, upload them, and publish the manifest only after the data objects are present. This is deliberately boring. Boring recovery paths are easier to audit at 03:00.

Safe nightly implementation with a commit marker

Run backup work outside the request-serving process so its credentials, CPU use, and failure budget do not become part of application request handling. The database dump below is assumed to have been produced by the database engine's documented consistent-export mechanism. Copying live database files is a different operation and should not be substituted for an export without engine-specific evidence that it is recoverable.

This Go example creates an upload archive, computes digests, sends the data objects to pre-authorized generic PUT URLs, then sends manifest.json last. URL creation is intentionally outside the job; the pattern keeps the storage protocol independent of a particular SDK and makes the expected object names explicit.

package main import ( "archive/tar" "compress/gzip" "crypto/sha256" "encoding/hex" "encoding/json" "fmt" "io" "net/http" "os" "path/filepath" "time" ) type object struct { Name string `json:"name"` Bytes int64 `json:"bytes"` SHA256 string `json:"sha256"` } type manifest struct { BackupID string `json:"backup_id"` Created string `json:"created_at"` Objects []object `json:"objects"` } func archive(src, dst string) error { out, err := os.Create(dst) if err != nil { return err } defer out.Close() gz := gzip.NewWriter(out) defer gz.Close() tw := tar.NewWriter(gz) defer tw.Close() return filepath.Walk(src, func(path string, info os.FileInfo, err error) error { if err != nil { return err } rel, err := filepath.Rel(src, path) if err != nil || rel == "." { return err } header, err := tar.FileInfoHeader(info, "") if err != nil { return err } header.Name = filepath.ToSlash(rel) if err = tw.WriteHeader(header); err != nil || !info.Mode().IsRegular() { return err } in, err := os.Open(path) if err != nil { return err } _, copyErr := io.Copy(tw, in) closeErr := in.Close() if copyErr != nil { return copyErr } return closeErr }) } func inspect(path string) (object, error) { f, err := os.Open(path) if err != nil { return object{}, err } defer f.Close() h := sha256.New() n, err := io.Copy(h, f) return object{Name: filepath.Base(path), Bytes: n, SHA256: hex.EncodeToString(h.Sum(nil))}, err } func put(path, url string) error { f, err := os.Open(path) if err != nil { return err } defer f.Close() req, err := http.NewRequest(http.MethodPut, url, f) if err != nil { return err } res, err := http.DefaultClient.Do(req) if err != nil { return err } defer res.Body.Close() if res.StatusCode < 200 || res.StatusCode >= 300 { return fmt.Errorf("unexpected upload status %s", res.Status) } return nil } func main() { id := os.Getenv("BACKUP_ID") dump := os.Getenv("DB_DUMP_PATH") uploads := os.Getenv("UPLOADS_DIR") if id == "" || dump == "" || uploads == "" { panic("backup inputs are required") } dir, err := os.MkdirTemp("", "backup-") if err != nil { panic(err) } defer os.RemoveAll(dir) archivePath := filepath.Join(dir, "uploads.tar.gz") if err = archive(uploads, archivePath); err != nil { panic(err) } items := make([]object, 0, 2) for _, path := range []string{dump, archivePath} { item, err := inspect(path) if err != nil { panic(err) } items = append(items, item) if err = put(path, os.Getenv("PUT_"+item.Name)); err != nil { panic(err) } } body, err := json.Marshal(manifest{BackupID: id, Created: time.Now().UTC().Format(time.RFC3339), Objects: items}) if err != nil { panic(err) } manifestPath := filepath.Join(dir, "manifest.json") if err = os.WriteFile(manifestPath, body, 0600); err != nil { panic(err) } if err = put(manifestPath, os.Getenv("PUT_manifest.json")); err != nil { panic(err) } } 

Set a deadline, emit the chosen prefix and byte counts, and alert on a missing completed manifest rather than every intermediate retry. The catch is that a single compressed upload archive simplifies transfer but makes selective file recovery slow. Use individually addressable, versioned objects when support work regularly needs one-file restoration.

Restore verification and rollback checklist

Restore into an isolated account or environment. Do not overwrite production to find out if the archive is usable.

1. Record the dated prefix and fetch its manifest.
2. Download only the named objects, then compare their byte counts and SHA-256 values with the manifest.
3. Load the dump into an empty database and extract uploads into an empty location.
4. Start a disposable application release against those targets and test representative authentication, records linked to uploads, sampled file reads, and the schema path most likely to drift.
5. Measure selection-to-verified-service time and record the backup ID, operator, checksum result, application checks, duration, and procedure differences.

For an actual recovery, create a new database target and upload location, validate both, then make a controlled configuration switch after application checks. Retain the displaced targets according to incident policy so the switch has a rollback boundary. In-place overwrite is a conscious risk acceptance because it removes that boundary.

A nightly archive is not suitable when the required RPO is shorter than the schedule, and a single archive is not suitable when individual-file restoration is a frequent support path. Use a database recovery method that meets the database RPO and a file layout that meets the file-recovery workflow. Re-run the drill after changes to schema, upload paths, credentials, retention, or the backup image, and choose its recurring cadence from business impact.

Sources

• https://aws.amazon.com/s3/pricing/
• https://docs.digitalocean.com/products/spaces/