Guides · 6 min read · October 26, 2026

RAG on Azure, step by step: how the pieces fit

Retrieval-augmented generation, or RAG, answers questions from your own documents. On Azure, four services do most of the work. This guide walks the pipeline in the order your data flows. For each stage it names the service that does the job and the mistake that stage invites.

Everything here comes from building one RAG system end to end and running it on live Azure in October 2026. The outputs shown are real captures from that system.

What RAG is, in one paragraph

RAG works in two steps. First, retrieve: search your documents for the few passages most likely to hold the answer. Then generate: give those passages to a language model, with an instruction to answer only from them and to cite them. The model brings the language skill, and your documents bring the facts. It's an open-book exam. The model doesn't need to have memorized your policies, because the right page is on the desk.

Use RAG when the model needs to know things that change or are private. Updating a document updates the answers, and every answer can point to its source. If you need the model to behave differently instead, start with better prompts.

The four services

Your app sits in the middle and calls each one. None of them is "the RAG service". The system is the pipeline you build between them.

1. Ingest: get text out of every format

Real documents are messy: markdown, HTML, CSV, Word, PDFs with text, and PDFs that are only a picture of a page. Most formats can be read with ordinary Python libraries, for free. Only image-only PDFs need OCR, which Document Intelligence provides with its prebuilt-read model. OCR bills per page, so send it only the files that have no text layer.

The mistake: skipping files that fail without saying so. One corrupt PDF quietly missing from your index means questions about it get confident wrong answers later. Report every failure, clearly, with the file name.

2. Chunk: split documents into passages

Search works on passages, not whole documents, so each document is split into chunks. Too big, and a chunk mixes several topics and matches everything weakly. Too small, and the answer gets cut in half. A few hundred words with a little overlap is a common start. Keep each document's title with its chunks, because titles carry a lot of meaning.

The mistake: picking a chunk size once and never measuring it. Chunk size is an experiment, and a golden set of test questions (stage 6) is how you run it.

3. Embed: turn passages into vectors

An embedding is a list of numbers that captures what a passage means. Texts with similar meaning get similar numbers. An embedding model such as text-embedding-3-small turns each chunk into a vector with 1,536 numbers. Your questions are embedded the same way, and search finds the chunks whose vectors are closest.

The mistake: mixing models. Vectors from two different embedding models can't be compared. Change the model, and you must re-embed every chunk. Moving a vector to a different store, though, never needs re-embedding, only re-uploading.

4. Retrieve: hybrid search

Azure AI Search stores your chunks and searches them three ways. Keyword search (BM25) is strong on exact names and codes. Vector search is strong on meaning, when people describe a problem in their own words. Hybrid search runs both for the same query and merges the two ranked lists.

Here's why the merge matters. On a test corpus, the question "checkout broke" should find a postmortem about a checkout outage. Vector search alone put three runbooks on top and missed the postmortem. Hybrid search put it second:

0.0333  runbook-checkout-api.md  —  Runbook: checkout-api
0.0320  postmortem-checkout-latency.md  —  Postmortem: checkout latency (2026-07-14)
0.0268  service-catalog.csv  —  Table service-catalog.csv — columns: service, team, tie

The mistake: assuming hybrid always wins. It's the safe default, but measure it on questions like your users' real ones. The hybrid search guide covers how the merge works.

5. Generate: a grounded prompt with citations

The model gets three things, in a fixed order: rules, numbered sources, and the question. The rules say to use only the sources, to cite every claim with its source number, and to give one exact sentence when the sources don't contain the answer. Here's the result for a question whose answer lives only in a scanned paper memo:

$ python3 scripts/ask.py "When do the old office badges stop working, and which vendor is doing the replacement?"
ANSWER: Old badges stop working on April 14, 2026, and the replacement vendor is
NorthPeak Cloud field services.[1]
CITATIONS:
  [1] scanned-facilities-memo.pdf
  [2] policy-badge-access.html
  [3] vendors.csv

And for a question the documents can't answer:

$ python3 scripts/ask.py "What is our policy on quantum computing purchases?"
ANSWER: I can't find that in your documents.

The mistake: a vague refusal. Make it one exact sentence, so a test can check it. "I don't know" in ten different wordings can't be tested.

6. Evaluate: a golden set and a gate

A golden set is a list of real questions, each with the document that should answer it. Retrieval is scored with simple metrics, such as whether the right document appears in the top five. The answers are scored by a second model acting as a judge, for things like groundedness: is every claim supported by the sources?

Run that as a test suite in CI, and quality becomes a gate, like unit tests:

$ pytest -m eval
tests/eval/test_eval_gate.py ...    [100%]
3 passed, 38 deselected in 190.78s

The mistake: judging quality by trying a few questions by hand. You'll remember the good answers and miss the regressions.

7. Observe: traces and cost per answer

Each answer is several calls: embed the question, search, then generate. Application Insights, through OpenTelemetry, records each step as a span in one trace. You see where the time goes, and you can attach the token counts and the dollar cost to every answer.

The mistake: finding out about cost from the monthly bill. Per-answer cost turns "is this affordable?" into a number you can watch.

8. Harden: injection, budgets, and caching

Your documents are input too. A document can contain text like "ignore your instructions", and the model will read it. This is indirect prompt injection, and no single fix stops it. Defend in layers: mark retrieved text as data, check what goes out, and limit what the model can do. Add a token budget per request, and a cache for repeated questions.

The mistake: treating the system prompt as a security boundary. It isn't one.

Sign-in: no API keys

Every call above can run without an API key. Your code gets short-lived tokens from Microsoft Entra ID, and each identity gets only the role it needs. The keyless Azure OpenAI guide shows how.

What it costs to learn

Most of these services bill per use and cost nothing while idle. The exception is Azure AI Search above the free tier, which bills by the hour: the Basic tier was about $0.11 an hour in October 2026. Building the full system in the package below costs about $4–14 in total, if you delete resources at the end of each session.