Hosting Is a Trust Problem When Production Is Real

A production system I operate has more than 17 million attachments and more than 4 TB of media, so at that size hosting decides which failures I can see, reverse or accidentally turn into corruption.

Price matters, but a cheaper VM is not cheaper if a bad queue decision delays signed webhooks until a provider retries them or a migration leaves a document URL pointing at an unreadable bucket; I have run this Rails workload on Hatchbox and DigitalOcean and a different airspace system on Cloudflare Workers, R2 and KV, and neither model is universally better because the operational model must fit the expected failure and tell the truth when that prediction is wrong.

The host is part of the data-integrity design

The Rails application is an operations workspace and authenticated portal for a UK property inventory agency, holding property and inspection history, reports, invoices, images and document bundles in a database of about 100 GB, with inspections dating to 2011 - the working record for people looking for an inspection, PDF or photograph while their day is moving.

For an early product I can accept rough edges, so a single managed database, a conventional deploy and a basic error tracker are usually enough, because there isn't much state to lose and an hour of downtime annoys a handful of people.

This platform is different because every attachment has a history - it might have come from the source system, or been adopted from older object storage, or been downloaded later after an earlier import missed it - and a report PDF can be regenerated separately from its body, but a portal user must never receive a provider URL that works today and exposes the wrong object tomorrow.

Storage boundaries, access boundaries and job execution are therefore application architecture, while the Hatchbox and DigitalOcean deployment is conventional Rails, PostgreSQL, Active Storage and Solid Queue with explicit data services, which lets me inspect the deploy path, database, storage credentials and responsible process - legibility that beats a clever abstraction during an incident.

Two DigitalOcean Spaces services run at once: current is read-write and legacy is read-only, because a migration is not complete when the new bucket uploads; the old bucket may still hold media referenced by a live record, to be adopted only after identity checks or freshly downloaded, and write access would turn its preservation boundary into an accident waiting for an admin task or bad code path, so current uploads use one service and historical reads another until the data has earned deletion.

One queue is an outage waiting to happen

Heavyweight media work does not share a queue with webhook processing, because the platform receives signed webhooks from the inspection source, Airtable and Xero that are time-sensitive, may retry or show failure in someone else's dashboard, and may arrive while an import pressures the same external API.

Media work can download a large image, find an absent URL, retry a transient response, compare a filename or add an item to a durable missing-download worklist, and it is slow because it is large; a single queue looks tidy until a photograph backfill occupies workers needed for signature verification, persistence and follow-up work.

The platform has nearly 7 million observations, more than 11 million Active Storage blobs and more than 17 million attachments, so a repair touching a small fraction is a workload; Solid Queue is database-backed, not a substitute for deciding which work gets a lane, because webhook-critical jobs need protected capacity and visibility, media downloads need concurrency limits and progress measures, and reconciliation needs a lane that can run without posing as a customer action.

Hosting comparisons of memory, vCPUs and regions skip the question that matters under load: can I run and supervise separate worker groups after a deploy, then throttle a noisy job without starving external state changes? One worker and a short queue can be fine on a small system, but for years of documents they gamble that the next repair arrives on a quiet day.

The topology is explicit: web serves requests, a webhook lane handles integration work, a media lane moves and verifies objects at a controlled rate, and scheduled and reconciliation work can be slowed or stopped independently; before choosing a host, I ask whether it can run those processes with those controls without every adjustment becoming an SSH session and a guess.

Storage migrations are a long production state

Current and legacy Spaces remove the false choice between a big-bang migration and permanent duplication, because the application can read historical objects from legacy and write new objects only to current - a configuration that represents the migration instead of copying once, changing DNS or an environment variable, then discovering an old report reference a year later.

The hard part is proving identity: a historical image may already be in legacy, be adopted after a filename match, or need fetching from the inspection provider, while ordinary source names collide because two unrelated records can be kitchen.jpg; a wrong adoption attaches the wrong photograph to a property and looks trustworthy, worse than a visible missing image, which is why collision reporting and the missing-download worklist make storage a migration process with evidence, failure states and controls.

Before trusting a host with more than 4 TB, I check that the runtime role writes only to current, legacy credentials are read-only and development and production prefixes differ so a test deployment cannot reach the production set, because IAM features are insufficient if the policy does not make a mistaken code path fail.

I also check retrieval through real application permissions - an authenticated request, a signed or proxied response where appropriate, the outcome when the source is unavailable, and accessibility after a retry or deploy - then check inventory, checksums or recorded counts and exceptions, deleted-object and lifecycle-rule recovery, and egress during repair, because an emergency copy that takes days or hits a provider limit and becomes a week of partial failures matters more than the bill.

Source deletion was blocked by verification, not optimism: a per-property manifest from read-only production queries and live source-system reads found cover images and meter photographs without UUIDs missed by an earlier download path; remediation verified all 17,595 distinct cover-image URLs and all 42,494 distinct meter-photo URLs in imported storage, but an actions backfill remained incomplete, so the source set was not deleted - a trustworthy host permits that awkward state rather than forcing a tidy cutover story with no place for a missing object.

Observability must answer the next operational question

AppSignal, Sentry and Rack::Attack observe different failures: AppSignal shows which endpoint or job slowed and whether database time, external HTTP or object storage caused it, while CPU at 42 percent does not explain a six-second authenticated request or whether media work occupied every worker.

Sentry makes an exception traceable by route, release, exception and request shape, and for integrations I need to distinguish an expected retry after a temporary provider response from a programming error that fails until code or configuration changes; Rack::Attack preserves capacity before a crawler or abusive endpoint creates database pressure, but its limits must fit real portal and integration behaviour rather than block a legitimate workflow.

For Hatchbox and DigitalOcean, I want web success and duration, queue depth and job age by queue, database connection saturation, slow queries and available disk, and storage errors divided by current and legacy; ten thousand media jobs can be acceptable, but a webhook waiting ten minutes is not, while a failed legacy read is a migration concern and a failed current write interrupts live work.

I also need proof of progress, because a live job process may do nothing, an importer may retry the same root record and an acknowledged webhook may never project; green processes without age, retries, dead letters or cursor movement are decoration, and source-system rate limits, paced Airtable pagination and expiring OAuth tokens belong in the signals, so when retries rise I need to know whether we respect Retry-After, are stuck on authentication or make bad requests before support notices.

Finally, I perform a restore exercise, because a backup is not protection until a fresh database restores, the application connects and a representative document serves, and object storage deserves the same test.

Cloudflare is a different trust model, not a shortcut

The airspace app retrieves two official GeoJSON feeds, normalises them and publishes a current zone bundle, while a Cloudflare Worker syncs every ten minutes, stores raw and prepared bundles in R2, keeps the ETag in KV and serves location checks at the edge; it fits a scheduled transformation and read-heavy public API, not a large relational operations database or years of private documents, and if a prepared response is missing after deployment the Worker reconstructs it from the raw bundle.

Trust questions change rather than disappear: I need the last successful upstream feed, malformed-feature count and current-bundle availability, while a failed bundle load or geometry check returns 503 UNAVAILABLE, not CLEAR, because for safety-relevant data a false clear is worse than visible unavailability; the iOS client can cache zones for browsing, but a failed current check becomes unavailable after the defined cache window.

Workers, R2 and KV remove servers to patch and supervise but add dependence on bindings, key naming, scheduled execution and platform visibility, so I check environment prefixes, failed-cron inspection, raw-bundle survival after a formatting or deployment mistake, and missing-data responses; whether Rails runs on a VPS, managed container platform or Worker, I trust it only after I can describe, find, limit and recover from failure.

The price comparison happens after the recovery plan

A host is cheap if it fits the system and expensive if ordinary recovery is mysterious, so before choosing one I write actions for a bad deploy, saturated database, queue of slow jobs, changed storage credential, provider outage and restore from last night: where I look first, what I do first and when I stop guessing and roll back.

That list exposes costs faster than a calculator: a platform that needs an unreachable expert to restore a database has a support cost, storage that cannot stay read-only during a migration has a data-integrity cost, and metrics that cannot distinguish blocked webhooks from a healthy busy media queue have an incident cost.

The Rails system needs legible long-running processes, isolated queues, durable database state and controlled access to two storage services, while the airspace system benefits from a scheduled Worker, edge reads and a small durable bundle; I would not force either into the other's model to make an architecture diagram consistent.

For the work behind these decisions, including Rails operations, migrations and production recovery, my CV and rates are here. For the related question of what actually makes a workflow replacement expensive, read what drives MVP development cost. And if you are assessing existing technical work before taking it on, I have written a technical due diligence checklist.