OpenAI API Integration Checklist for Production Apps

The request has 25 seconds to finish, after which the server stops waiting for the model and tells the user it could not analyse their meal.

That is the timeout on a food-logging feature I shipped for iPhone and Android, where a person types "chicken wrap, chips and a Coke", or sends a photograph, and a Cloudflare Worker calls gpt-4o-mini to turn it into calories, protein, carbohydrates, fat, and food items before the user sees an estimate and chooses whether to save it.

The model call is the small part: the work is deciding what result the application may accept, protecting a metered endpoint, making a slow upstream service unsurprising, and admitting that structured JSON is not necessarily correct, while some of this checklist exists in the feature and some does not, and the distinction between them matters.

The response has to be a contract before it can be useful

If output feeds a database, payment decision, permission change, or typed screen, it needs an explicit shape, so the food feature's text and photo routes both return meal totals and an array of items, each with a name, calories, protein grams, carbohydrate grams, and fat grams, and iOS, Android, and the Worker share one contract.

The Worker requests strict JSON Schema at temperature 0.3 and disallows extra properties, which prevents explanation before JSON, carbs becoming carbohydrates_g, or an extra confidence field breaking a strict client, but it is structure rather than validation: a schema cannot show that 1,400 calories is plausible, item totals equal the meal total, or a pasta photo was recognised correctly.

Syntax validation: can the response be parsed and conform to the schema?
Domain validation: do values make sense for this product and agree?
Decision validation: should the application act automatically or ask for confirmation?

The shipped feature has provider-schema validation and review before Save, but no independent Worker validator, nutritional plausibility bounds, or reconciliation of item and meal totals; it casts shaped JSON to its response type, so next it should reject negative macros and materially inconsistent totals, limit a single input, and return an editable draft rather than quietly recording a precise-looking number.

The same applies elsewhere: reject an unrecognised support-ticket category, check extracted dates, currency amounts, and identifiers against a document, and put a destructive proposal before a person, because valid JSON is not permission to do damage.

Timeouts and retries need a defined failure, not hope

The upstream API will be slow, rate-limit, return a server error, or disappear, so this Worker uses a 25-second timeout and turns upstream rate limits and server failures into 503; iOS and Android retry retryable server failures once after two seconds, never malformed requests, and show the failure. Interactive requests can justify one retry, while a queued report may need more attempts and exponential backoff and checkout may need none, because retry policy is product behaviour rather than a generic HTTP helper retrying every error.

State-changing paths also need idempotency: a repeated draft request may spend twice but is usually safe, while a request that writes a record or triggers a downstream action needs an idempotency key, request fingerprint, final result, and enough state to return the original result after reconnect, so a mobile retry cannot create two food entries, messages, or invoices.

The food path is stateless request-response, with the client saving an accepted result locally rather than the Worker creating a remote entry, which reduces duplicate writes but not duplicate calls or local saves if the flow later changes; I would send a client-generated identifier and deduplicate short-window repeats. For images, hash compressed bytes and decide whether the same image may be analysed again - yes for a changed prompt may be right, while no for a transport retry may be right - because timing cannot tell those events apart.

Never invent a result when the provider is unavailable: saved entries work locally without a network, while new AI analysis and barcode lookup require it, and there is no offline analysis queue, cached-analysis fallback, or connectivity-specific native path. The next version should keep the typed description as an unsaved draft, say analysis is unavailable, and offer retry later or manual values; if it adds a queue, it should ask before background submission.

A request limit is a cost control, not an abuse solution

The Worker rate-limits before calling the model, hashing a platform-scoped device identifier and storing daily counters in Cloudflare KV: attested callers get 100 AI requests daily, while unattested callers or IP-identity fallbacks get 10; food photos are compressed to JPEG, initially targeting at most 1 MiB before base64 submission, which caps one identity's daily spend, reduces avoidable vision cost, and helps mobile requests finish.

They are not complete abuse prevention, because App Attest is not cryptographically verified in this Worker: a present, non-sentinel header changes the tier, which is friction rather than proof of a genuine device, and a per-device cap does not answer the maximum total spend I will accept.

Cost control needs four numbers:

A per-request limit for input, output, tools, image count, and model.
An identity limit for a user, device, workspace, or tenant over a sensible period.
A product limit tying premium or high-cost work to the purchased plan.
A global limit that alerts and degrades deliberately at daily or monthly budget.

The implementation partly has the first through image compression and a fixed model request, and the second through daily KV counters, but it has no recorded token telemetry, per-request cost calculation, cache-hit metric, or global spend budget; the code cannot tell me production call volume, latency distribution, or model bill, so I will not invent them.

Add Worker-boundary telemetry for model, prompt version, route, input class, provider token counts, elapsed time, status, and estimated cost, without raw meal text or image bytes, then aggregate daily and by route and alert at a real budget boundary. Cache only after deciding what is safe: a system prompt and static reference material can be shared, while a typed meal is personal and may mean a different portion today; an analysis-cache key should include normalised input, prompt and model versions, input method, and relevant user choices, cache only an editable draft, and measure hits first.

Prompts need versions and changes need evidence

A prompt is production behaviour, so the food prompt's instruction to use standard portions unless specified is a product decision: "I had lasagne" gets an assumption even when the model follows instructions exactly.

Give prompts stable identifiers such as food-text-v1, store the version with the analysis result, and include it in telemetry and eval output; the device already retains original description, totals, input method, and encoded AI response, so a prompt version would make support work less speculative.

Treat model versions the same way: before switching because a dashboard calls one cheaper or smarter, run known inputs and compare output structure, acceptance rate, domain checks, latency, and estimated cost, then release by percentage or feature flag if traffic justifies it.

Evals are a release gate, so build a small, consented, scrubbed corpus of short descriptions, ambiguous portions, mixed meals, regional foods, implausible user instructions, blurry photos, and no-food images, with each case needing an expected property rather than a supposedly true calorie number: JSON parses; required values are non-negative; totals roughly reconcile; uncertainty is identified rather than a brand or portion invented; a non-food image has the intended failure shape; prompt injection remains user content rather than changing the contract.

The project has no committed evaluation corpus, golden-response suite, prompt-injection benchmark, model-quality set, or cost-budget test, while Worker tests cover the health endpoint and malformed or missing bodies - useful input validation but not proof that successful model output is usable. Start with 30 to 50 anonymised cases, a controlled-provider fixture for contract behaviour, and a separate manual or provider-backed model-quality run; keep metered network calls out of ordinary unit tests, but run real candidates before deployment, record results, and block a known regression.

For when a model needs retrieved source material rather than a changed prompt, see RAG and fine-tuning. First define the fact you need to verify.

Logs should explain a failure without collecting the meal

The food Worker does not retain meal data: the native app stores the accepted entry locally, including original description and encoded AI response, because it must present and retain the log, while the processing service does not need a permanent second copy.

Server logs should retain request ID; one-way or rotating caller identifier; route, model, prompt version, input class, image-byte bucket, status class, duration, provider request ID when available, token usage, and estimated cost, while logging a failed validation rule rather than its raw value. Content capture needs explicit, time-limited user consent and access controls, so never quietly log everyone.

API keys never reach the client; device identifiers are not logged in plain form; authentication and attestation headers are not debugging text, while app errors should explain the user's action and internal events retain code-level context. A 200 is not success for an estimate: edits, rejections, retries, and manual entry are weak but better product-quality signals, and the current code shows none of that telemetry, so I would add it before claiming accuracy.

The feature must fail as an honest part of the product

A model API is an external dependency with a variable answer. Before shipping:

The output has a schema, independent runtime validation, and a defined review step when the result can be wrong.
The endpoint has a timeout, narrow retry rules, and an idempotency design for every state-changing path.
The service has per-request, per-identity, product, and global cost boundaries backed by measured usage.
Prompts and model choices have versions. A change runs against known cases before it reaches users.
Logs identify the request and the failure without retaining raw user content by default.
The unavailable state preserves user work and offers a real fallback rather than acting as if the model answered.

The shipped feature has strict response shape, a 25-second timeout, one retry for retryable failures, daily request tiers, compressed image input, client review before Save, and no Worker retention of meal data, but it lacks independent domain validation, idempotency tracking, cost telemetry, production latency measurement, caching policy, prompt versions, evaluation coverage, and a usable offline fallback - that is the work that turns a successful API call into dependable product behaviour.

I build the API boundary and the unglamorous controls around it. My CV and rates are here.