How to Hire an AI Engineer Who Can Verify the Work

Every candidate says they have AI experience now, but the claim has stopped carrying information.

It is not - the hard part starts when a believable answer changes a customer's decision, which is why, when you hire an AI engineer, you should not screen for whether they can make a model respond but whether they can prove when it is useful, detect when it is not, and give the user a way out when it is wrong.

I built a food logging flow where people type a meal description or submit a photo for an estimated calorie and macro entry, using gpt-4o-mini, strict provider JSON Schema, and temperature 0.3, which shapes the response rather than nutritional accuracy; the user chooses whether to save it, and that distinction is the work.

AI proficiency is no longer a useful hiring signal

"AI engineer" can mean building around a hosted model, training or serving one, or fixing inconsistent, inaccessible product data owned by six disagreeing systems. Calling all three AI work starts a bad hire.

Most product teams hiring in 2026 need a product engineer for a dependable model workflow - input handling, permissions, data boundaries, prompt changes, application code, interface, error paths, and output inspection after release - because it is product engineering with one extra problem: the dependency can return fluent falsehoods.

Ask about the last feature they shipped, then interrupt at the model call: what could the user do with a bad response, what could it change, how did they know it improved after a prompt or model change, who could inspect a disputed result, and what data was kept and for how long?

Useful answers are concrete: "This assistant only drafts the reply, it cannot send it." "We require a source citation before showing a policy answer." "We compare extracted invoice totals to the original document and send mismatches to a review queue." "The user confirms the meal before it becomes a record."

Confidence words are bad answers: "The prompt is very good." "The new model is smarter." "We tell it not to hallucinate." A model instruction is one input to a probabilistic service, not verification.

Do not make this a keyword test, because an experienced person may say evaluation, retrieval, guardrails, tracing, or none of them, but they must describe a falsifiable contract: what passes, what fails, and what happens next.

Hiring an AI engineer is an exercise in verification

Put a real feature in front of the candidate - drafted support replies from account history, fields pulled from invoices, or answers from a limited policy set - then ask for the first evaluation set before the system diagram.

A serious answer starts with ordinary inputs, known trouble, expected output, who decides it, and how to grade reasonable disagreement, because thirty carefully selected real examples beat 3,000 generated questions that all look alike.

For document extraction, separate easy invoices from image-only PDFs, credit notes, two currencies, missing reference numbers, changing supplier layouts, and wrong languages; for an assistant, include stale policies, absent answers, conflicting documents, account-boundary requests, and badly phrased questions.

Each example needs an outcome rule: a purchase order number matches or does not, while a support reply may be acceptable, unsafe, or wrong beyond automatic scoring, and the engineer must know the difference.

Ask what they log - prompt or version, model and exposed version, retrieval identifiers, latency, token or request cost, output, and collectable eventual outcome - because "We would add observability" is not an answer.

Prompt text is production code with a worse diff: a one-line change can improve one answer class and damage another, and without its exact prompt and model configuration, they cannot investigate a regression.

Ask how they roll out change - a small internal cohort, sampled traffic split with approval, or shadow mode recording proposed output unseen by users - because they must compare old and new behaviour before making the latter permanent.

Some responses cannot be verified directly - a coach suggesting a meal idea is not document extraction - so reduce the cost of error with advice rather than automatic action, visible uncertainty, a correction path, and the original input beside the suggestion; that is product judgement.

Plausible and wrong is the failure mode that matters

Ordinary software fails noisily with a 500, inert button, or validation message, but model features can fail with a clean answer containing one invented detail.

Structured output is not proof: my food logging Worker requests calorie and macro totals plus named food items, with no extra fields, then parses that shape into native clients' models; it prevents prose where the app needs numbers, but valid JSON can still estimate 200 grams of chicken when the user ate 80 or invent items in a photographed mixed meal, because schema enforcement solves shape rather than truth.

The result is displayed for review and stored only if the person saves it, while barcode scans instead use public nutrition fields; those choices limit the consequence of a bad answer, but they do not make the model safe unattended.

The limits are real: there is no committed nutrition evaluation corpus, plausibility bounds, reconciliation proving item totals equal meal totals, prompt-injection evaluation, or production token or cost telemetry in the codebase, while Worker tests cover malformed request bodies and its health endpoint rather than model accuracy, so hiding those gaps behind strict schemas is not hireable.

Give a plausible but wrong result, then wait for a good person to ask whether it is advisory or automated, its harm, whether a reliable source can check it, and who corrects it; they may say it should not ship, whereas a weak person changes the prompt and calls it solved.

Treat the model as an untrusted collaborator that can propose, classify, extract, rank, and draft, but must not silently move money, publish legal claims, delete records, change permissions, or make health decisions without action-specific controls, and the candidate must make that boundary explicit.

Cost and latency belong in the feature brief

The model call is neither free nor instant, so ask for a latency budget before choosing a model or chaining calls: how long may the user wait, is it blocking or background work, and what does the application show after five seconds, fifteen, and a timeout? "We will see" means it is not designed.

My food-analysis Worker gives its upstream request 25 seconds, maps provider rate limits and server failures to 503, and has native clients retry one retryable server failure after two seconds - availability mechanics rather than a performance result, since the repository has no measured production latency.

It keeps daily request counts in Cloudflare KV and permits 100 AI requests per day for attested callers, or 10 for other callers and IP fallbacks, which caps casual use rather than cost because no token use, model spend, cache hit rate, or production request volume is recorded, and there is no cache.

A candidate should turn assumptions into arithmetic: what does an average request cost at the provider's current price, what does a large image cost, what if retrieval adds ten large text chunks, and how many calls can a customer make before it stops making commercial sense? They need working, unknowns, and a measurement replacing the guess.

Ask where the budget is enforced: per user, workspace, account, feature, or provider account; and what happens at the limit: smaller model, queue, upgrade request, or clear failure.

Ask about input size: the iOS photo path JPEG-compresses images and initially targets 1 MiB before base64 submission, which protects request size and avoids sending the original camera file by default but does not establish image quality; photo input without permitted images, cost, and latency skips the product work.

A good portfolio includes the bad cases

A notebook demo proves somebody can make a model answer a question in a controlled environment, not ship it in your product.

A useful portfolio item shows input and response contracts, representative examples, a failure case and its user path, data leaving and retained, prompt changes without guessing, and, with retrieval, a question it should refuse.

The strongest write-ups contain an unflattering paragraph: "We found that this document type had poor extraction quality, so we routed it to review." "The first prompt passed our examples but failed when users used abbreviations." "We removed automatic sending because nobody could explain a wrong reply after the fact." Those matter more than model logos.

Ask for test artefacts rather than a screen recording: candidates can show redacted examples, a synthetic reproduction of the failure shape, a grading rubric, or a small test program without revealing client data, but if they show only a notebook and cheerful answer box, assume that is the system.

Do not mistake AI-assisted development for runtime AI experience: I worked on a Rails platform for a youth education non-profit where agent-oriented planning and project memory supported engineering, but the inspected product has no model-provider SDK, prompt store, model endpoint, retrieval flow, or runtime AI feature, so calling it an AI learning product would be dishonest. "Built with AI" can mean ordinary application code written with an assistant.

For production integration questions, read the OpenAI API integration checklist, and if the candidate says you need retrieval or fine-tuning, make them justify it against your data; RAG versus fine-tuning is a start.

A paid slice is the shortest route to the truth

A one-week paid trial can reveal more than four interviews when it uses a real product seam, not a toy task.

Choose a workflow with real constraints, provide a small approved dataset or safe sample preserving the original mess, define permitted action, unacceptable failures, and who judges ambiguous answers, then ask for the narrowest complete version.

Require the feature, evaluation set with expected outcomes, known failures, request logging for prompt and model configuration, cost estimate with assumptions, latency budget and timeout behaviour, and next steps if evaluation is poor.

Do not require a grand platform: one endpoint, small interface, and visible review or fallback route suffice, so watch their sequence - user and error cost or five libraries, baseline or retrieval, a bad case found, and what remains unverified.

Pay for the trial because it has value, and unpaid auditions select for the wrong thing; scope it for a capable person, then read notes before the demo, because they reveal whether the candidate knows what they built.

If you need help defining the slice before hiring, MVP development cost in 2026 covers decisions that change an estimate before anybody builds.

I build AI features as product workflows, with explicit limits rather than magic claims. My CV, past work, and day rate are here.