My Copilot Studio agent kept giving confident wrong answers because the retrieval layer was pulling correct-sounding fragments from two unrelated documents and gluing them into one answer. I fixed what the agent was reading rather than how I was asking: bloated documents split into one topic each, headings rewritten to say what sits underneath them.
At a glance: before you touch a single instruction, open the test pane's Debug Insights. It names which knowledge source and which passage got used for the answer you are staring at, and it will say plainly when a search came back empty, when a file was skipped, or when permissions blocked something.
The symptom: a confident answer built from two documents
The tell was always the same shape. I'd ask a plain question and the agent answered fast, in full sentences, with the right tone. Nothing about the answer looked broken. Then I'd check it against the system itself and find it wrong in a specific way. Half the answer was right and came straight from the correct document. The other half was true of something else entirely, bolted on like it belonged there.
A question about one process would come back with a real step from the right document, followed by a caveat or a condition that belonged to a completely different process, filed several documents away. Read on its own, every sentence in the answer was true somewhere in the knowledge base. Read together, the answer described a process that didn't exist.
Why rewriting the prompt was the wrong first move
My first instinct was the instinct most people have. Add more instructions. Tell it to only answer from verified sources. Tell it to say "I don't know" rather than guess. Tell it to double check itself before responding. I spent a good chunk of a session down that path, and the wrong answers kept coming, just with more hedging language wrapped around them.
It didn't work because the model was reporting accurately on what it had been handed. A prompt governs how the agent behaves with the material it retrieves, and it has no say over which material gets retrieved in the first place. Hand a fluent model two unrelated passages and ask it to answer a question with them, and you get a fluent, wrong answer no matter how firmly you've told it to be careful.
What the retrieval was returning when I checked
So I stopped editing the prompt and opened the test pane instead. Copilot Studio's Debug Insights view shows which topic got triggered and which knowledge source got pulled for a given question. That's the place to look. The chat window only ever shows you the polished end of the process.
What I found was retrieval doing exactly what it was built to do with what I'd given it. The source documents were large, and each one covered several loosely related topics under a single file. One document could run through setup, everyday use, and troubleshooting for three different scenarios, all filed under generic headings like "Notes" or "Other information." When you upload files straight into Copilot Studio, you don't get much say in how they're chunked for retrieval, so a document like that gets sliced up in ways that don't respect where one topic ends and the next begins. Ask a specific question and the search can easily return a chunk that straddles two topics, or two chunks from two different scenarios that both happened to match the same keyword. Those chunks were what came back, so those chunks were what got used.
Fixing the source content: one topic per document, headings that mean something
Once I could see that, the fix was obvious rather than clever. I went back through the source material and split anything covering more than one topic into its own document. A file called "Notes," covering five scenarios, became five files, each about one scenario.
Then I rewrote the headings. "Other information" became the actual question someone would type, in the words a person in that situation would use. That matters more than it sounds like it should, because a document's headings and structure are most of what retrieval has to work with when it decides whether a passage is relevant. A generic label gives the search nothing to match against.
I also pulled documents that hadn't been accurate for a while. A stale document competes for the same search results as the current one, and sometimes wins.
| Before | After |
|---|---|
| One file, five topics, headed "Notes" | Five files, one topic each |
| "Other information" | The actual question a user would ask, as the heading |
| Whatever heading I typed first | Wording that matches how people phrase the question |
Your agent's answers don't add up?
Free 30-minute call. Send me a few examples and I'll say whether it's retrieval, the prompt, or the source content underneath both.
How I test an agent before anyone else sees it
I don't publish an agent off a handful of questions that happened to work. Before anyone else sees it, I run three passes.
- The obvious questions, worded the way I'd word them, to confirm the easy cases work.
- The same questions worded the way a stressed, non-technical user would type them, because that's who ends up using it.
- The awkward ones, on purpose. Questions that sit right on the boundary between two topics, or that could plausibly pull from two documents at once, because that boundary is exactly where I saw it go wrong.
For every answer in that third pass, I check Debug Insights before I check the answer itself. If the wrong source got pulled, no amount of prompt tuning fixes it and the source content needs another look. Once the right sources were coming back, the confident wrong answers mostly stopped on their own. Better material in, better answers out.
Testing before anyone else sees it is one gate. There are several more, and how I run a squad of coding agents covers the rest of them.
Cleaning up the source documents fixes what the agent can find. The other half is the process rules that were never written down anywhere, which now have a place to live in Dataverse.
Where this ends up
This is the same discipline behind the Copilot Studio agent I built for troubleshooting on a large internal Power App, where getting the source content right was most of the work. The same "check what it retrieved before you trust it" habit carried over into the voice command layer I built for my own machine, a local model with a much narrower job than an agent's, but the same discipline of testing what it heard rather than assuming it. More of what I've built sits on the work page, and the mechanics of what a Copilot Studio agent needs to run well are on the Power Platform services page.
If you're looking at a support or knowledge agent for your own business and want a second opinion on whether the source content will hold up, book a free 30-minute call. If you want it reviewed, with the gaps written up, that's what a paid Workflow Review is for.
Common questions
Why does a Copilot Studio agent give a confident wrong answer?
Almost always because of what got retrieved. The agent pulls a passage or two from its knowledge sources and answers from them. If the search returns the wrong passage, or two passages from unrelated topics, the model still answers fluently and with confidence, because it is reporting accurately on the material it was handed.
Should I fix a Copilot Studio agent's prompt or its knowledge sources first?
Check the knowledge sources first. Open the test pane's Debug Insights and see which source and passage actually got pulled for a wrong answer. Prompt instructions govern how the agent behaves with material it already has, and they do not change which material gets retrieved, so a retrieval problem stays put no matter how the prompt is worded.
How should I structure documents for better retrieval in Copilot Studio?
Keep one topic per document rather than bundling several loosely related topics into one long file. Write headings as the actual question someone would ask, in the words they'd use, rather than generic labels like 'Notes' or 'Other information'. Remove documents that are no longer accurate, because a stale document still competes for search results against the current one.