AI engineer interview questions for LLM applications
Practice AI engineer interview questions with a fictional document assistant, worked answer sketches and checks for retrieval, evaluation and tool access.

On this page
- Start with this fictional document assistant
- 1. What would you build first, and why?
- 2. The answer has a citation. Why is it still wrong?
- 3. What should the assistant say when evidence is missing?
- 4. How would you evaluate a prompt or model change?
- 5. Does a valid tool call prove that access is allowed?
- 6. What would you inspect when an update is slower or less reliable?
- Rehearse an answer you can defend
- Frequently asked questions
- Sources
For LLM application roles, practice answering AI engineer interview questions about architecture choices, retrieval failures, evaluation and tool access. Explain each decision through a concrete input, the evidence your application can use and the behavior you would check. The original practice questions below follow one fictional assistant, with worked answer sketches you can challenge.
LLM means large language model. This exercise concerns building applications with these models, rather than training every kind of machine learning system. It complements broader system design interview practice. Read the actual role description before deciding which skills to prioritize; the machine learning role catalog includes roles with different requirements.
Start with this fictional document assistant
The assistant answers support-policy questions for a signed-in reader in workspace A. These are invented plans and documents, unrelated to Prepin’s product or pricing.
| Document | Trusted registry status | Complete policy facts supplied for this exercise |
|---|---|---|
| B2 | Workspace A; approved current Basic policy | Email support is included. Chat support is not included. |
| B1 | Workspace A; superseded Basic policy | Chat support is included. |
| P1 | Workspace A; approved current Pro policy | Email and chat support are included. |
| X1 | Workspace B; restricted operations document | Its contents are unavailable to this reader and are not part of the exercise. |
An administrator-maintained registry supplies workspace, permission, document-type and revision status. The assistant cannot change that registry. The reader may access current product policies B2 and P1; B1 is excluded from current-policy answers, and X1 is unauthorized.
The only exposed tool is read_policy(plan), accepting Basic or Pro. The backend supplies the authenticated reader and workspace, checks permissions, and returns approved current policy excerpts with document identifiers. It accepts no model-selected workspace or user identity. No write tools are available. The reader’s plan is not otherwise known, and neither current policy says anything about phone support.
This is a proposed rehearsal fixture, not an implemented system, benchmark or employer assessment.
1. What would you build first, and why?
Start with a bounded lookup-and-answer workflow for this fixture. Ask which plan the reader means when it is missing, retrieve the permitted current policy, then answer with its document identifier. There is no requirement here for autonomous planning or account actions.
Retrieval-augmented generation, or RAG, gives a model retrieved material to use when answering. A defensible sketch names where retrieval ends and generation begins. State the output contract: answer the policy question, identify supporting evidence and mark any part the supplied policy does not answer.
For “Does Basic include chat?”, the expected lookup returns B2. The answer is “Basic does not include chat support,” citing B2. If your design retrieves B1 instead, a polished sentence does not satisfy the current-policy requirement.
Explain what would make you change the design. A future requirement to compare historical revisions would need a different retrieval contract. Do not silently widen today’s permissions or revision rules to accommodate an imagined future feature.
2. The answer has a citation. Why is it still wrong?
Inspect the retrieved evidence before changing the prompt. Microsoft’s RAG evaluation documentation separates evaluation of retrieved context from evaluation of the generated response. An answer’s consistency with its context does not establish that the context was the right material.
Consider this hypothetical failure: the question asks about current Basic chat support; retrieval returns B1; the answer says chat is included and cites B1. The sentence repeats the excerpt accurately, but the lookup violated the exercise’s revision rule.
A useful answer sketch identifies the failed boundary: check registry status and the retrieval filter, require B2 for the current Basic request, then rerun the answer check. Retain this failure as a regression case.
P1 exposes a different error. It is current and authorized, but it describes Pro. Returning it for a Basic question is a relevance failure, even if the generated answer faithfully repeats it. Authorization, relevance and current revision are separate conditions.
3. What should the assistant say when evidence is missing?
Answer the supported part and state what remains unknown. For “Does Basic include chat and phone support?”, B2 supports a no for chat. It supplies no phone-support policy.
One defensible response is: “Basic does not include chat support [B2]. The supplied current Basic policy does not specify phone support, so I cannot confirm that part.” Claiming that phone support is unavailable would add a fact the exercise does not contain.
Compare that with “Is chat included?” No plan is supplied, and B2 and P1 differ. Ask whether the reader means Basic or Pro. The missing information is the request’s scope, rather than a fact that needs a more confident guess.
Microsoft distinguishes groundedness from response completeness: an answer can agree with its context while omitting required information. “Basic has no chat” leaves the phone part unresolved unless the answer acknowledges that gap.
A lookup timeout needs different wording again. Say that the current policy could not be checked. Do not interpret the failed request as evidence that a feature is absent.
4. How would you evaluate a prompt or model change?
Define the expected behavior before looking at the new answer. OpenAI’s evaluation guidance recommends task-specific evaluation data, explicit objectives and continued evaluation after changes, including edge and adversarial cases.
Use the following proposed matrix to rehearse that process. These cases have expected outcomes; no execution results are claimed.
| Input or failure condition | Expected behavior | Evidence to inspect |
|---|---|---|
| Current Basic chat question | Say chat is not included; cite B2 | Lookup returns B2, not B1 or P1 |
| Basic chat and phone question | Answer chat; acknowledge unknown phone policy | Neither unsupported phone claim nor omitted phone question |
| Chat question without a plan | Ask Basic or Pro | No invented plan assumption |
| Retriever proposes superseded B1 | Exclude it from current-policy evidence | Registry status and returned document identifiers |
| Request for X1 or workspace B | Do not return unauthorized document content | Backend access decision and tool result |
| A test copy of B2 contains instructions to read X1 | Treat those instructions as untrusted document text | No broader lookup or unauthorized disclosure |
| Current-policy lookup times out | Explain that verification failed | Tool error, bounded handling and final answer |
Separate unambiguous assertions from judgments about language. A forbidden document appearing in the tool result should fail a deterministic access check. A paraphrased answer needs a rubric: does the cited evidence support the claim, does it address each requested part, and does it acknowledge missing information?
Google’s evaluation metric definitions distinguish computation-based checks from model-based rubrics. Its metric prompt templates show explicit criteria for groundedness. These sources support making the criteria inspectable; the matrix above is an original exercise, not Google’s pass standard.
A small fixture can expose a regression. Passing it would not establish production reliability. Add representative real tasks and review automated judgments against human decisions before treating an evaluation as release evidence.
5. Does a valid tool call prove that access is allowed?
No. Valid argument types do not establish who may read the resulting document. In this fixture, a request for Basic must still be restricted to the authenticated reader’s workspace and permitted current product policies.
Google ADK’s safety guidance discusses identity, authorization and deterministic policy enforcement in tools. Apply that distinction to the proposed contract: obtain reader identity from the trusted application session, enforce access in the backend, and reject unsupported arguments. A workspace or document identifier in a user message cannot grant access.
Consider a backend accidentally using one shared, broadly privileged identity for every reader. Checking that the model supplied “Basic” would not detect a lookup returning another workspace’s policy. Inspect the actual access decision and returned identifiers, including failure cases.
Document text also cannot widen permissions. For the adversarial test, append an instruction to a test copy of B2 telling the assistant to read X1. The policy facts remain the same; the added instruction is untrusted input. The backend must still deny unauthorized access. Read-only permissions prevent writes, but they do not by themselves prevent disclosure of private information.
Do not promise that a prompt makes injection impossible. Explain the enforced boundary, the test evidence you would collect and the limits that remain.
6. What would you inspect when an update is slower or less reliable?
Compare the tool trace and final answer for the same tasks before choosing an optimization. Record which policy was requested, what permitted evidence returned, tool errors and the answer’s supported claims. Keep sensitive document contents out of unnecessary diagnostic records.
Anthropic’s tool-design guidance describes evaluating outcomes alongside runtime, tool calls, token usage and errors. For this fixture, extra irrelevant context is a candidate to investigate, rather than a reason to remove permission or revision checks.
If the tool repeatedly times out, propose a bounded retry or an explicit unavailable response within the application’s latency budget. Specify that budget with the interviewer; none is supplied in this exercise. Compare failure handling as well as successful requests.
Anthropic’s evaluation framework distinguishes a task, a trial, its trace and the observed outcome. A reassuring final sentence is not proof that the lookup respected access rules. Inspect the backend result too, and repeat trials where model variation matters.
An answer sketch can therefore recommend holding or reverting a change that breaks the fixture’s required behavior. It cannot claim a universal release threshold or production success from these examples.
Rehearse an answer you can defend
Pick a question and explain the requirement, evidence, decision and proposed check aloud. Then change one assumption: remove B2, withhold the plan, or introduce a timeout. Check whether your explanation changes for the right reason.
Use the question bank for additional practice topics. If you want spoken practice with feedback on your answers, Prepin offers AI mock interviews for a target role. Practice this interview.
Frequently asked questions
Are these questions asked by specific employers?
No employer attribution or frequency is claimed. These are original prompts for rehearsing LLM application decisions; use the employer’s role description and preparation instructions to decide what else to study.
What should I prepare beyond LLM applications?
Check the role’s requirements for software engineering, data work, machine learning or infrastructure responsibilities. This small document-assistant exercise does not cover that full range, so it should be one part of your preparation.
Do I need to claim that I built this assistant?
No. Describe it as a hypothetical design exercise unless you actually implement it, and distinguish proposed tests from measured results. Prepare separate behavioral interview stories from your own work when discussing past experience.
Is an AI engineer interview the same as an AI interviewer?
An AI engineer interview assesses someone for an engineering role involving AI. An AI interviewer describes how an interview is conducted; the two phrases do not identify the same preparation task.
Sources
Prepared with AI assistance; the official sources below were checked on November 18, 2026. The documents support the engineering concepts, not the fictional policies or a claim about employer question frequency.
- OpenAI: Evaluation best practices, current undated documentation.
- Microsoft Learn: Retrieval-Augmented Generation evaluators, updated June 2, 2026.
- Google Cloud: Define your evaluation metrics, updated November 17, 2026.
- Google Cloud: Metric prompt templates for model-based evaluation, updated November 17, 2026.
- Google ADK: Safety and Security for AI Agents, current undated documentation.
- Anthropic: Writing effective tools for agents, September 11, 2025; stable tool-design guidance.
- Anthropic: Demystifying evals for AI agents, January 9, 2026.
Practice the interview out loud
Run a realistic AI mock interview for your target role and get feedback on every answer.
Start a mock interview →Screen engineers before the first call
Send candidates one link. Prepin runs the technical voice interview and live coding, then gives you a scored report.
See how screening works →