These 30 generative AI and LLM interview questions cover what freshers and junior engineers are actually asked: transformer basics, training and fine-tuning, prompting and inference settings, and running LLMs safely in production. Each has a short model answer you can adapt.
How these LLM interview questions are organised
These LLM interview questions follow the order most interviews take: fundamentals of how the model works, how it is trained and adapted, how you control it when calling it, and how you run it safely in a product. Most generative AI interview questions for freshers come from the first and third groups; the fourth group separates candidates who have shipped something from those who have only read about it.
Each answer is what a strong candidate would say in under a minute. Don't memorise the wording. Say it out loud in your own words, and where you can, swap in an example from your own project. For retrieval-specific depth (chunking, vector indexes, RAGAS), there is a separate list of RAG interview questions.
Fundamentals: transformers, tokens and attention
1. What is a large language model?
An LLM is a neural network, almost always a transformer, trained on a very large text corpus to predict the next token given the previous ones. Doing that well at scale forces it to learn grammar, facts, patterns of reasoning and style. Chat models are the same base model further trained to follow instructions and hold a conversation. At inference it generates text one token at a time, feeding each output back in as input.
2. Explain the transformer architecture briefly.
A transformer turns tokens into embeddings, adds position information, and passes them through a stack of identical blocks. Each block has a self-attention layer, which lets every token gather information from other tokens, and a feed-forward network applied to each position, with residual connections and layer normalisation around both. The original design had an encoder and a decoder; most modern LLMs are decoder-only. Its key advantage over RNNs is that training processes all positions in parallel.
3. How does self-attention work?
Each token's vector is projected into a query, a key and a value. Attention scores are the dot products of a token's query with every key, scaled by the square root of the key dimension and passed through softmax to get weights. The output is the weighted sum of the values: softmax(QK^T / sqrt(d_k)) V. Multi-head attention runs several of these in parallel with different projections, so different heads can track different relationships.
What they're really checking: why the scaling. Without it, large dot products push softmax into regions with tiny gradients.
4. What is the difference between encoder-only, decoder-only and encoder-decoder models?
Encoder-only models like BERT see the whole input in both directions and are good for understanding tasks such as classification and embeddings. Decoder-only models like the GPT family use a causal mask, so each token sees only earlier tokens, which suits generation. Encoder-decoder models like T5 encode an input and then decode an output, a natural fit for translation and summarisation. Decoder-only dominates today's general-purpose LLMs.
5. What are tokens, and why do they matter?
Tokens are the units the model reads and writes, usually subword pieces from a tokenizer such as BPE or SentencePiece. In English a token averages roughly four characters, but many other languages and scripts, including Hindi, often need more tokens for the same meaning. Tokens matter because context limits, latency and API pricing are all counted in them, and because tokenization explains odd failures such as miscounting letters in a word.
6. How does a transformer know word order?
Attention on its own treats input as an unordered set, so position information is added. The original transformer added fixed sinusoidal position encodings to the embeddings; some models learn position embeddings; many current LLMs use rotary position embeddings (RoPE), which rotate query and key vectors by an angle depending on position so attention depends on relative distance. Position method affects how well a model handles contexts longer than it was trained on.
7. What is a context window, and what limits it?
The context window is the maximum number of tokens the model can handle in one call, counting the prompt and the generated output together. Standard self-attention cost grows quadratically with sequence length, and the key-value cache grows linearly in memory, so long contexts are expensive. A large window also doesn't guarantee the model uses everything well; information in the middle of a long prompt is often used less reliably than information at the start or end.
8. What is the difference between token embeddings and sentence embeddings?
Token embeddings are the internal vectors for each token inside the model, which change at every layer. Sentence or text embeddings are a single vector for a whole passage, produced by a model trained specifically so similar passages land close together. The second kind is what you store in a vector database for semantic search and RAG; you would not normally use a chat model's raw hidden states for that.
Training and fine-tuning
9. What happens during pre-training?
The model is trained with self-supervision on a huge text corpus: given a sequence, predict the next token, and minimise cross-entropy loss over billions or trillions of tokens. No human labels are needed because the text itself supplies the targets. This is where nearly all of the model's knowledge and capability comes from, and it is by far the most expensive stage.
10. What is supervised fine-tuning (SFT)?
SFT continues training a pre-trained model on curated examples of prompts and ideal responses, so it learns to follow instructions in a particular format. The loss is usually computed only on the response tokens. It is the step that turns a text-completion model into an assistant, and it is also what companies do on a smaller scale to teach a model a domain format or style.
11. Explain RLHF.
Reinforcement Learning from Human Feedback has three steps after pre-training. First SFT. Then humans compare pairs of model responses, and a reward model is trained to predict which one people prefer. Finally the LLM is optimised with reinforcement learning, classically PPO, to produce responses the reward model scores highly, with a KL penalty that keeps it close to the SFT model so it doesn't drift into gaming the reward.
12. What is DPO, and how is it different from RLHF?
Direct Preference Optimization uses the same kind of data, pairs of preferred and rejected responses, but skips the separate reward model and the RL loop. It optimises a classification-style loss directly on the preference pairs, with a reference model playing the role of the KL constraint. It is simpler and more stable to train, which is why it became popular, though RL-based methods are still widely used.
13. What is LoRA?
Low-Rank Adaptation freezes the pre-trained weights and learns a small update for selected weight matrices, usually the attention projections, as the product of two thin matrices of rank r (often 8 to 64). Only those small matrices are trained, so memory use and checkpoint size drop dramatically, and you can keep several adapters for one base model. After training, the update can be merged into the weights so inference costs nothing extra.
14. What is QLoRA?
QLoRA loads the frozen base model in 4-bit precision, using a data type called NF4, and trains LoRA adapters on top in higher precision. Gradients flow through the quantized weights into the adapters. This lets you fine-tune much larger models on a single GPU with results close to 16-bit LoRA. It also introduced tricks such as double quantization and paged optimizers to save more memory.
What they're really checking: that you know the base model is quantized but the adapters are not.
15. When should you fine-tune, and when should you use RAG?
Use RAG when the model needs knowledge: private, changing or citable facts. Use fine-tuning when it needs behaviour: a strict output format, a consistent style, a specialised classification, or a smaller cheaper model matching a larger one on a narrow task. Try prompting first, then RAG, and fine-tune when those plateau and you have a few hundred to thousands of good examples. Fine-tuning on facts tends to give confident wrong answers and has to be repeated when facts change.
16. What is catastrophic forgetting?
When you fine-tune a model on a narrow dataset, it can lose general abilities it had before, because full fine-tuning overwrites weights that supported other skills. Mitigations include using parameter-efficient methods like LoRA, mixing some general data into the fine-tuning set, using a low learning rate and few epochs, and evaluating on general tasks as well as the target task.
Prompting and inference
17. What do temperature and top-p do?
Temperature divides the logits before softmax: below 1 it sharpens the distribution toward the most likely tokens, above 1 it flattens it. At 0 decoding becomes greedy, which is close to deterministic, though serving systems can still produce small variations. Top-p (nucleus sampling) samples only from the smallest set of tokens whose probabilities add up to p. Top-k keeps the k most likely tokens. For factual or extraction tasks use low temperature; for brainstorming, higher.
18. What is few-shot prompting?
You include a few worked input-output examples in the prompt before the real input, and the model follows the pattern without any training. It is the cheapest way to fix format and tone. Pick examples that cover edge cases, keep them consistent with each other, and watch for the model copying details from examples that don't apply. Zero-shot is just the instruction with no examples.
19. What is chain-of-thought prompting?
Chain-of-thought asks the model to work through intermediate steps before giving the final answer, either by instruction or by showing worked examples. On multi-step problems, such as arithmetic or logic, this often improves accuracy because the model can condition on its own intermediate results. It costs more output tokens and latency, and the written reasoning is not guaranteed to reflect how the model actually reached the answer.
20. How do you get reliable structured output such as JSON?
Ask for it with a clear schema and an example, then enforce it. Many APIs offer a JSON mode or schema-constrained decoding, which restricts generation to tokens that keep the output valid. Always validate the result in code with a schema library, and on failure retry with the validation error included. Keep schemas flat and give fields descriptive names; deeply nested optional fields cause more errors.
21. How does function calling or tool calling work?
You describe available functions to the model with names, descriptions and parameter schemas. When a request needs one, the model outputs a structured call with arguments instead of plain text. Your code, not the model, executes the function, then sends the result back, and the model uses it to write the final answer. Treat every call as untrusted input: validate arguments, check permissions, and require confirmation for anything that changes data or spends money.
What they're really checking: that you know the model never runs the code itself.
22. What is the system prompt, and how is it different from the user prompt?
The system prompt sets the rules and role for the whole conversation: tone, scope, format, what to refuse. The user prompt is the specific request. Models are trained to give system instructions more weight, but that is a learned tendency, not a security boundary, so don't put secrets in it and don't rely on it alone to stop misuse.
23. What is the KV cache?
During generation, each new token needs the keys and values of all previous tokens for attention. The KV cache stores them so they aren't recomputed at every step, which makes generation much faster. The cost is memory, growing with sequence length, batch size and model size, which is one reason long contexts and many concurrent users are expensive to serve.
Production, safety and evaluation
24. How do you reduce LLM latency and cost in production?
Measure time to first token and tokens per second separately. Stream responses so users see output early. Cut input tokens by trimming prompts and retrieved context. Route simple requests to a smaller model and hard ones to a larger model. Use provider prompt caching for long, fixed prefixes such as system prompts. Batch offline work, cap output length, and for self-hosted models, use quantization and an efficient serving engine.
25. What is semantic caching, and what is the risk?
Semantic caching stores past answers keyed by the embedding of the question, and returns a cached answer when a new question is similar enough. It saves cost and latency on repeated questions. The risk is a wrong hit: two questions that are close in embedding space but need different answers, such as different dates or account numbers. Use strict thresholds, include user or context in the key, and never cache personalised answers across users.
26. Why do LLMs hallucinate, and how do you reduce it?
Training rewards producing plausible next tokens, not saying "I don't know", so when the model lacks the fact it produces something that sounds right. Reduce it by grounding answers in retrieved sources (RAG), asking for citations, allowing and testing an explicit "not found" answer, using low temperature for factual tasks, and verifying claims or outputs in code where possible. You can make hallucination rarer and visible; you can't promise to remove it.
27. What is prompt injection?
Prompt injection is when text the model reads overrides the developer's instructions. Direct injection comes from the user ("ignore your rules and..."); indirect injection hides instructions in content the model processes, such as a web page, email or PDF. It is especially dangerous when the model has tools. Defences include separating instructions from data, least-privilege tools, human confirmation for actions, output filtering and monitoring. There is no complete fix, so design assuming some injections will succeed.
28. What are guardrails?
Guardrails are checks around the model, not inside it. On input: moderation, blocking out-of-scope requests, detecting personal data. On output: schema validation, moderation, checking that cited sources exist, blocking leaked secrets or personal data. They can be rules, classifiers or another model. Good guardrails are specific to the product's risks and are tested like any other code.
29. How do you evaluate an LLM application?
Build a test set of real inputs with expected outputs or grading criteria, including edge cases and past failures. Score with exact checks where possible (valid JSON, correct label), human review on a sample, and LLM-as-judge with a clear rubric for open-ended answers, keeping in mind judges favour longer answers and their own style. Run the set on every prompt, model or retrieval change, and track production signals such as user feedback.
What they're really checking: that you have measured something, not just eyeballed outputs.
30. What is an AI agent, and what goes wrong with agents?
An agent is an LLM running in a loop: it decides which tool to call, observes the result, and repeats until it finishes the task. Agents can handle open-ended multi-step tasks, but errors compound across steps, costs and latency grow with each loop, they can get stuck repeating actions, and tools widen the damage a prompt injection can do. Practical agents have step limits, narrow tools, logging of every step, and human approval for risky actions. A fixed workflow is often more reliable when the steps are known in advance.
Turning answers into interview wins
Definitions get you through the screening round. What moves you forward is connecting each answer to something you built. "I set temperature to 0.2 because my app answers from documents and I wanted consistency" lands better than any textbook line about logits. The same goes for prompt injection, evaluation and latency: one real example beats three paragraphs of theory.
If you don't have a GenAI project yet, a RAG app is the most useful one to build, because it touches embeddings, vector search, prompting, hallucination control and streaming in one small codebase. The RAG project guide explains what to build and how to present it in a viva or interview, and the list of AI projects for students has other options if RAG isn't your fit.
Want the complete, deployable version? Chat with your PDFs is a ready RAG project: full Next.js source code, a 20-minute free deploy guide, an 11-minute video walkthrough and 15 interview answers. Try the live demo.
The Chat with your PDFs kit is a ready-made version of that project, with source code, a free deploy guide and its own project-specific interview questions; you can try the live demo before deciding.
Mistakes that cost candidates the round
- Mixing up training and inference. RAG doesn't train the model, and the model doesn't "learn" from your chat.
- Saying temperature 0 is fully deterministic. It is close, but not guaranteed on real serving systems.
- Claiming fine-tuning fixes hallucinations. It can make wrong answers sound more confident.
- Quoting model rankings or release details you can't back up. Stick to concepts that hold across models.
- No numbers. Token counts, latency, top-k, temperature, eval scores. Any real number from your own work beats none.
Frequently asked questions
What LLM interview questions are asked to freshers?
Mostly fundamentals: what a transformer and self-attention are, what tokens and context windows are, what temperature does, the difference between pre-training, fine-tuning and RAG, and how to reduce hallucinations. If you list a GenAI project, expect questions about how you built and evaluated it.
Do I need to know the maths of attention for a generative AI interview?
You should be able to write softmax(QK^T / sqrt(d_k)) V and explain what Q, K and V are and why the scaling exists. Deep derivations are rare outside research roles, but a clear intuition is expected.
Is fine-tuning or RAG more important for interviews?
Both come up, usually together as 'when would you use each'. RAG is asked more often for application roles because most company projects are RAG-based; fine-tuning, LoRA and QLoRA come up more for ML engineer roles.
How should I prepare for generative AI interview questions quickly?
Learn the 30 questions on this page well enough to answer in your own words, then build one small working project, such as a RAG app, so you can back answers with something you made and measured.
Will interviewers ask about specific models like GPT or Gemini?
Sometimes, but they mostly care about concepts that transfer across models: context windows, sampling, tool calling, cost per token and evaluation. Avoid quoting benchmark claims you can't support.
Originally published at kitsbyai.com.
