Managed RAG vs custom RAG: which should you build on?

What does a managed RAG service do for you?

A managed RAG service runs the pipeline from documents to retrieval for you: it ingests files, splits them into chunks, embeds them, stores the vectors and retrieves the best matches at query time. Amazon Bedrock Knowledge Bases, Google’s Vertex AI RAG Engine and OpenAI’s file search tool all follow this pattern.

Amazon’s documentation says a knowledge base abstracts the heavy lifting of building pipelines and gives an out-of-the-box RAG solution that reduces build time. OpenAI describes file search as a hosted tool managed by OpenAI, so you write no code to run retrieval. The figure below shows who typically does the work at each stage, for three ways of building.

When is a managed service the right choice?

A managed service is the right choice when your documents are clean and the first version has to ship quickly. Plain help-centre pages, policy text and FAQ content in one language fit well, and so does a prototype whose only job is to show whether RAG helps your users at all.

It also suits teams without retrieval engineers, because the vendor carries the pipeline. The test is whether answers need to survive an audit: if a wrong or uncited answer is merely unhelpful, a managed service is usually enough. If it becomes a liability, read the next section.

When do you outgrow a managed service?

You outgrow it when you need control the service does not expose: how tables and scans are parsed, how an answer cites its source, where the data is processed, or how retrieval is tuned to your own questions.

Location is the clearest case. Google’s RAG Engine page says VPC-SC controls and CMEK are supported, while data residency controls are not, and that access in some regions needs an allowlist. Parsing is another. Super Cat Technology’s on-premise system for a Big 4 firm splits each page into text, table and figure regions and returns the source paragraph with every answer. That is pipeline work we designed around the firm’s documents.

Where do open-source frameworks fit?

Open-source frameworks sit between the two: you get building blocks, and you run the result. The LlamaIndex documentation describes its framework as an open-source toolkit for RAG and agents. You still choose and operate the parser, the vector store, the retrieval logic and the hosting.

That makes a framework a good fit for a team that wants control and has engineers to run it, and a poor fit for a team that wants someone else to carry the pipeline. If your questions span many documents, see vector RAG vs GraphRAG vs LightRAG for when a graph is worth adding.

What should you decide before you choose?

Settle these five questions first. Together they usually decide the route:

1. Where must the data live, and which regions and controls does the service support?
2. What do the documents look like: clean prose, or tables, scans and two languages?
3. Must every answer cite its source passage?
4. Who will maintain the pipeline after launch?
5. How will you measure answer quality, with which labelled questions?

For Hong Kong data, bilingual documents and regulator expectations, see Enterprise RAG in Hong Kong: build vs buy.

What can go wrong?

Each route has its own failure modes. These are the ones to plan for:

• The service moves under you. Microsoft’s Azure AI Search guidance now says new RAG implementations should start with agentic retrieval and existing ones should consider migrating. A pipeline tied to one service’s pattern inherits that churn.
• Costs you did not model. Google’s RAG Engine page says that if the service manages a Spanner instance as your vector database, Google Cloud bills you for that instance.
• Limited reach. The same page says RAG Engine access in several regions needs an allowlist for new projects.
• No evaluation set. Without labelled questions, you cannot tell whether a new chunking setting or a new service made answers better or worse, and nothing alerts you.