Module 14: Agent Builder
Lesson 14.5: Adding Knowledge and Testing Responses
Lesson Promise
Choose knowledge sources and test the agent before users trust it.
Real-World Scenario
The team has ten files, but only three are current. The agent must answer from the current files and avoid outdated process notes.
Core Concept
Agent quality depends heavily on knowledge hygiene. Approved, current, well-named sources produce better answers than messy folders.
Testing should include source-backed questions, edge cases, missing information, ambiguous wording, and sensitive prompts.
A test log turns agent improvement from vibes into evidence.
Step-By-Step Workflow
- Audit candidate files for ownership, currency, permissions, and sensitivity.
- Use only approved sources for the first version.
- Create test prompts from real user questions.
- Record expected answer, actual answer, source behavior, and needed fix.
- Adjust instructions or knowledge and retest.
- Create a feedback path for users after launch.
Prompt Lab
Bad Prompt
Use all these files.
Better Prompt
Use only the 2026 renewal playbook, approved FAQ, and escalation matrix. Ignore archived process notes.
Expert Prompt
Create a knowledge and testing plan for this agent. Include source inventory, approved sources, excluded sources, permission checks, test prompts, expected answers, edge cases, sensitive prompts, failure criteria, remediation actions, and launch readiness score.
Hands-On Exercise
Build a ten-prompt test set for your agent.
Deliverable
A source inventory and agent test log.
Agent Builder Review Checklist
Common Mistakes
- Building a broad general helper instead of a focused role-based agent.
- Adding messy or outdated files and expecting the agent to infer what matters.
- Skipping refusal instructions for policy, legal, financial, HR, or customer-impacting questions.
- Testing only the happy path and missing vague, hostile, or sensitive prompts.
- Sharing an agent before ownership, feedback, and update routines are clear.
Quiz / Checkpoint
What should be tested besides correct answers?
Source behavior, missing information, ambiguous questions, sensitive requests, refusal behavior, and escalation.
Official Sources To Verify
Want your AI product explained to a large AI-native audience?
Kingy AI helps AI companies turn complex products into clear, useful YouTube videos that drive awareness, product understanding, demos, clicks, and search visibility.
