# How to build AI search over company knowledge: RAG architecture

URL: https://pixelactstudio.com/blog/ai-search-over-company-knowledge

> How to build AI search over company knowledge: chunking, Postgres hybrid search, permission filters, cited answers, RAG evaluation, and when to skip RAG.

By Dev Talan. Published 2026-11-01. Updated 2026-11-01. Type: System breakdown.

## Key takeaways

- Write the eval set before any code and measure keyword search alone as the baseline. Keep vectors, fusion, and reranking only if each one raises the hit rate.
- Filter by permission inside the retrieval query, before ranking. Once restricted text reaches the prompt, treat it as already disclosed.
- Check citations in code on every answer. If the answer cites a passage that was never retrieved, or cites nothing, show plain search results instead.
- Start the eval set with 50 to 100 real questions, including ones the documents can't answer and ones asked as a restricted user, and block the merge when a score drops below the last accepted baseline.
- Answer counts and live records with a database query. If the whole corpus fits in one prompt and everyone may read it, send all of it with prompt caching and skip retrieval.

AI search over company knowledge is a retrieval-augmented generation (RAG) system: it finds the passages in your internal documents that answer a question, keeps only those the person asking may see, and has a language model answer from them with citations. To build AI search over company knowledge that people trust, focus on retrieval, permissions, and evals.

## How do you build AI search over company knowledge?

You build it as two pipelines that share one index. The ingestion pipeline runs in the background: it pulls documents from your sources, splits them into chunks, and indexes them with their permissions. The query pipeline runs on every question: it retrieves the chunks the user may see, ranks them, and asks a model to answer from the best few.

**The parts of a RAG architecture for company knowledge**

| Part | What it does | Where it usually goes wrong |
| --- | --- | --- |
| Connectors | Pull documents, edits, deletions, and access rules from sources such as Google Drive, Confluence, or a helpdesk | Deleted or unshared documents stay in the index |
| Parsing and chunking | Turn files into clean text and split it into passages along headings and sections | Tables, scanned PDFs, and page headers and footers turn into noise |
| Embeddings and index | Store each chunk as a vector for meaning and as text for keyword search | Exact terms such as product codes and error messages go missing |
| Retrieval and reranking | Find candidate chunks, merge the result lists, and keep the best few | The right document ranks twelfth, and the model never sees it |
| Permission filter | Drop every chunk the asking user may not open, before ranking | An answer quotes a document the user cannot access |
| Answer generation | Write a short answer from the retrieved passages and cite each one | Fluent answers with no source, or with a source that says something else |
| Evals and feedback | Score retrieval and answers on real questions, and turn bad answers into test cases | Changes ship on gut feel, and quality drifts without anyone noticing |

Most answer quality is decided before the model writes a word. If retrieval returns the wrong passages, a better model or a cleverer prompt only produces a more convincing wrong answer.

## When is RAG the wrong tool?

RAG is the wrong tool when the answer is a number in a database, when the documents fit in one prompt, when nobody keeps the content current, or when one tool already holds everything. Check your real questions against this list first.

- **Counts, totals, and live records.** “How many customers renewed last quarter?” needs a query against your database, because passages that look similar can't count. That answer belongs in a dashboard or an [internal tool](https://pixelactstudio.com/services/internal-tools), or with a model that writes queries against a read-only view.
- **A small, stable set of documents.** If the whole corpus fits comfortably in the model's context window and everyone who asks may read all of it, send all of it with each question and use prompt caching to cut the cost. You skip chunking and indexing, and the retrieval bugs that come with them.
- **Content nobody owns.** RAG cites whatever it finds. If three versions of the travel policy disagree and nobody can say which is current, the system will quote the wrong one with a citation attached. Fix ownership first.
- **Knowledge that lives in one tool.** If everything sits in one wiki or helpdesk, and its built-in knowledge base AI respects your permissions, use that. Build when questions span several sources or need your own access rules.

## How should you ingest and chunk internal documents?

Ingest each source incrementally, keep its access rules and metadata with every chunk, and split documents along their own structure rather than at a fixed character count.

Connectors should sync only what changed. Most sources expose a modified timestamp or a webhook for new and edited documents, and a periodic reconciliation removes chunks whose source was deleted or unshared.

Good chunks follow headings and sections, at a few hundred tokens each with a small overlap. Prepend the document title and heading path, so a passage that says “the limit is 30 days” still carries “Refund policy, annual plans” when retrieved. Keep tables whole, and run scanned PDFs through OCR, or they index as empty pages.

### What to store with every chunk

- A stable chunk ID and the ID of its source document.
- The source URL with a section anchor, so a citation opens the right place.
- Title, heading path, owner, and last-updated date.
- Tenant ID and the groups or users allowed to read the source.
- The embedding model and version that produced the vector.
- A content hash, so unchanged text is skipped on the next sync.

## Do you need vector search, keyword search, or both?

Use both. Vector search finds passages with the same meaning in different words, and keyword search finds exact terms that embeddings blur, such as product codes, error messages, names, and acronyms. Hybrid search runs the two and merges the results.

A common merge is reciprocal rank fusion: each chunk scores by its position in each list, so a chunk ranked well by both searches rises to the top. A reranker can then score the top few dozen candidates against the question and keep the best five to eight for the model. Reranking adds a model call, so keep it only if your evals show it helps.

If you already run Postgres, all of this fits in one database. The [pgvector README](https://github.com/pgvector/pgvector) documents hybrid search with Postgres full-text search, and the function below keeps the permission filter inside both searches.

```sql
CREATE FUNCTION search_chunks(
  user_tenant_id uuid,
  user_group_ids uuid[],
  query_embedding vector,
  query_text text
)
RETURNS TABLE (chunk_id bigint, fused_score numeric)
LANGUAGE sql STABLE
AS $$
  WITH semantic AS (
    SELECT id, row_number() OVER (ORDER BY embedding <=> query_embedding) AS rank
    FROM chunks
    WHERE tenant_id = user_tenant_id AND allowed_group_ids && user_group_ids
    ORDER BY embedding <=> query_embedding
    LIMIT 40
  ),
  keyword AS (
    SELECT id, row_number() OVER (ORDER BY ts_rank_cd(search_vector, query) DESC) AS rank
    FROM chunks, websearch_to_tsquery('english', query_text) AS query
    WHERE tenant_id = user_tenant_id AND allowed_group_ids && user_group_ids
      AND search_vector @@ query
    ORDER BY ts_rank_cd(search_vector, query) DESC
    LIMIT 40
  )
  SELECT id, sum(1.0 / (60 + rank)) AS fused_score
  FROM (SELECT id, rank FROM semantic UNION ALL SELECT id, rank FROM keyword) AS ranked
  GROUP BY id
  ORDER BY fused_score DESC
  LIMIT 8;
$$;
```

With an approximate index such as HNSW, pgvector applies the filter after scanning the index, so a strict permission filter can return too few rows. Turn on iterative index scans with the `hnsw.iterative_scan` setting, available from pgvector 0.8.0, or partition large tenants. Move to a dedicated engine such as OpenSearch only when load tests show one database is no longer enough.

## How do you keep retrieval inside each user's permissions?

Copy each source's access rules onto its chunks at ingestion, resolve the asking user's groups on the server, and filter inside the retrieval query before anything is ranked. Permissions-aware retrieval means a restricted chunk never reaches the model, the prompt, the logs, or the cache.

> **The model is not an access control layer.** Retrieving everything and telling the model to withhold what the user may not see fails the first time someone phrases a question cleverly. Once restricted text is in the prompt, treat it as already disclosed.

- **Resolve identity on the server.** Look up the user's groups for every request, from your identity provider or a directory synced from it, never from a value the browser sends.
- **Sync permissions faster than content.** Treat an unshare or a role change as its own event, and agree the maximum lag in writing. For the most sensitive sources, check permissions at query time.
- **Enforce the tenant boundary twice.** In a multi-tenant product, Postgres row-level security on the chunks table blocks cross-tenant reads even if an application query forgets its filter.
- **Scope caches and logs.** Key answer caches by the question and the permission set, and restrict who can read traces, which contain retrieved text.
- **Treat document text as data.** A document can contain instructions aimed at the model. Give the search step read-only access and no tool that can send, delete, or pay.

The [OWASP entry on vector and embedding weaknesses](https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/) lists permission-aware vector stores and partitioned datasets among its prevention steps. Our [startup security checklist](https://pixelactstudio.com/resources/startup-security-checklist) covers the access-control basics underneath them.

## How should the answer cite its sources?

Send the model numbered passages, tell it to answer only from them and cite a passage for every claim, and let it say it found nothing. Then check the citations in code before the answer reaches the user.

Ask for structured output, such as JSON with the answer and the cited passage IDs. Link each citation to its source section and show the document's last-updated date, so a reader can check a claim in one click. If a cited ID was never retrieved, or the answer cites nothing, show plain search results instead.

```ts
type Passage = { id: string; sourceUrl: string };
type DraftAnswer = { text: string; citedIds: string[] };

export function findUnsupportedCitations(draft: DraftAnswer, passages: Passage[]) {
  const retrievedIds = new Set(passages.map((passage) => passage.id));
  return draft.citedIds.filter((citedId) => !retrievedIds.has(citedId));
}

export function hasValidCitations(draft: DraftAnswer, passages: Passage[]) {
  return draft.citedIds.length > 0 && findUnsupportedCitations(draft, passages).length === 0;
}
```

## How do you evaluate a RAG system?

RAG evaluation scores retrieval and generation separately, on a fixed set of real questions, every time the chunking, embedding model, prompt, or model changes. A single quality score hides which half broke.

Start with 50 to 100 real questions from support tickets, Slack threads, and onboarding notes. For each one, record the documents that answer it and an answer a domain expert would accept. Add questions the documents cannot answer, and questions that only a restricted group may ask.

- **Retrieval hit rate at k:** The share of questions with a correct source in the top k chunks. If this is low, no prompt change will help.
- **Faithfulness:** Whether every claim is supported by the retrieved passages, usually scored by a model acting as judge, with a person checking a sample.
- **Answer correctness:** Whether the answer matches the accepted one. A faithful answer can still be wrong if it was built on an outdated document.
- **Correct refusal:** Whether the system says it found nothing when the documents hold no answer, instead of guessing.
- **Permission leakage:** Questions run as a restricted user, failing if any restricted chunk is retrieved. The target is zero on every run.

Run the suite in CI on every pull request that touches chunking, prompts, or models, and block the merge when a score falls below the last accepted baseline, as our [engineering standards](https://pixelactstudio.com/about/engineering-standards) require for any failing test. Model-judged scores vary from run to run, so set each threshold a small agreed margin below the baseline.

## How do feedback and human review keep answers accurate?

Log every question with the retrieved chunk IDs, answer, model version, latency, and cost, and let users flag a bad answer. A person labels why it failed, and the fix goes where the failure started.

- **Missing document.** No indexed source holds the answer. Add the source, or ask its owner to write the page.
- **Retrieval miss.** The right document exists but did not rank. Adjust chunking, keyword fields, or reranking.
- **Stale or conflicting source.** The system cited an old version. Send it to the document owner.
- **Generation error.** The right passages were retrieved, and the answer still misread them. Adjust the prompt or the model.

Every labeled case joins the eval set. Many fixes land in the documents rather than the code, so the system also shows you which knowledge is missing or out of date.

Where a wrong answer is expensive, such as policy answers sent to customers, the system drafts a cited reply and an agent approves or edits it before it goes out. Widen automatic answers only where review data supports it.

## What does AI search cost to run, and how fast is it?

Running cost is mostly the model call on each question, and latency is mostly the time the model takes to write. Retrieval in a well-indexed database is rarely the slow or expensive part. Measure both per question on your documents and expected volume before you commit to a model.

- **Send fewer, better passages.** Five good passages after reranking cost less than twenty mediocre ones, and usually give a better answer.
- **Embed only what changed.** After the first run, embed only chunks whose content hash changed. A new embedding model means re-embedding everything, so test it on the eval set first.
- **Cache what repeats.** Use prompt caching for the fixed instructions, and serve repeated questions from the permission-scoped answer cache.
- **Match the model to the step.** A smaller model can rewrite queries, leaving the larger one for the answer.
- **Stream the answer.** Show the sources first and stream the text, so reading starts sooner.
- **Cap spend.** Set per-user rate limits and provider spend alerts from the first day.

## What happens when AI search fails?

Plan each failure before launch. Most fallbacks show ranked search results, which still beats an error page.

**Failure handling for AI search over company knowledge**

| Failure | What the user sees | What the system does |
| --- | --- | --- |
| No passage answers the question | “I couldn't find this in the documents you can access,” with the closest matches | Logs the question as a possible gap for the document owner |
| Model provider slow or down | Ranked search results without a written answer | Times out the call, retries once, falls back to keyword search if embedding also fails, and alerts on the error rate |
| Sources disagree | Both passages with their dates | Flags the conflict to both document owners |
| Ingestion job fails | Answers as before, with the last sync time visible | Alerts the team and retries from the last checkpoint |
| A document contains injected instructions | A normal answer, or no answer | Has no write tools for the injected text to trigger, and logs the document for its owner to clean up |

## In what order should you build it?

Build retrieval first and measure it before anything else. The first four steps answer whether AI search works on your documents at all, which makes them a sensible scope for a [pilot sprint](https://pixelactstudio.com/services/pilot-sprints).

1. **Pick one audience and one job.** For example, support agents answering product questions from one or two sources. Company-wide search can be a later phase.
2. **Write the eval set first.** Collect real questions, source documents, and accepted answers before any code, so every later choice has a score.
3. **Ingest and measure keyword search alone.** Index the sources with metadata and permissions, then measure hit rate with keyword search only. That score is the baseline every addition must beat.
4. **Add vectors, fusion, reranking, and answers.** Keep each addition only if it raises the score. Then add cited answers and measure faithfulness and cost per question.
5. **Test permissions and failures.** Before any real user, run the permission leakage cases as a restricted user and trigger every fallback in the table above.
6. **Launch to a small group with feedback on.** Review flagged answers every week, add them to the eval set, and widen access while the scores hold.

If a pilot is new to you, our guide to [prototypes, pilots, and MVPs](https://pixelactstudio.com/blog/prototype-vs-pilot-vs-mvp) explains what each one proves.

## What should you do next?

Collect 50 real questions your team asked last month and the documents that answer them. If you can't find the documents, fix the knowledge before you build search over it. If you can, work through the [AI feature readiness checklist](https://pixelactstudio.com/resources/ai-feature-readiness-checklist) to check data access, accuracy targets, review, and running cost before you commit.

AI work at Pixelact starts with a fixed-price pilot from $4,500 that measures retrieval, answer accuracy, and cost per question on your documents. The [AI systems and automation](https://pixelactstudio.com/services/ai-systems) page sets out what is included and what isn't.

## Frequently asked questions

### How long does it take to build AI search over company knowledge?

It depends on how many sources you connect, how complex their permissions are, and how far the first measurements land from your accuracy target, so an honest estimate comes after you measure retrieval on your documents. Run the first four steps of the build order as a pilot for one audience and one or two sources. Its hit rate and cost per question set the build milestones, and company-wide search becomes a later phase.

### Which vector database should we use for RAG?

If you already run Postgres, start with pgvector and Postgres full-text search in the same database. You get one system to operate, transactions with the rest of your data, and row-level security for tenant isolation. Move to a dedicated vector database or search engine when load tests or corpus size show a real need. Chunking and evals usually matter more than the choice of store.

### Should AI search index Slack and email?

Be selective. Chat and email hold a lot of knowledge, but also noise, private conversation, and decisions that were later reversed, and their permissions are set per channel or per mailbox. Start with curated sources such as docs, wikis, and resolved tickets. Add public channels later if eval questions show the answers live there, and keep private messages out of a shared index unless your policy explicitly allows it.

### Can AI search over company knowledge be accurate enough to trust?

For a defined job, yes, but no RAG system is right every time. Before building, agree what accuracy the job needs and what a wrong answer costs. Internal search that links every source can launch below the bar for answers sent to customers, where an agent approves or edits each cited draft before it goes out. Track hit rate, faithfulness, and correct refusals against that target on the eval set, and hold permission leakage at zero.

### Do we need an AI agent, or is one retrieval step enough?

One retrieve-then-answer step covers most questions over company knowledge. An agent loop, where the model runs several searches or calls a tool before answering, helps with multi-part questions that span sources. It costs more per question, takes longer, and is harder to evaluate. Start with one step, and add a loop only when eval cases show single-pass retrieval keeps missing.

### What happens to the index when a document is deleted or someone leaves the company?

Deletions should reach the index on the next sync, either from the source's delete events or from a periodic check that removes chunks whose source no longer exists. A departure needs no reindexing. Because groups are looked up on the server for every request, search access ends as soon as the person's removal from your identity provider reaches your app and their sessions are revoked.