Client work under NDA. This page describes system design, my role, and evaluation method. No document contents, tenant identifiers, matter details, or names other than my own appear here.
An AI assistant for a legal office is only useful if it never invents an answer. Most of the engineering was in specifying what it must refuse to do, and building the evaluation that could tell whether it actually refused.
- Role
- Project manager & technical lead
- Team
- 5 people
- Duration
- Jun 2026 – Present, two semesters
- Status
- MVP reached August 2026; pilot phase
- Context
- UCF Senior Design, sponsored capstone
- Users
- ~10 attorneys
- Microsoft Copilot Studio
- SharePoint Online
- Microsoft Entra ID
- Power Automate
- React
- TypeScript
- Fluent UI
- SPFx
The problem
The client is an in-house university legal team of roughly ten attorneys. Decades of contracts, memoranda, and prior advice live scattered across Teams, OneDrive, and legacy drives, and the knowledge of where things live is held informally by individual staff. When a senior attorney leaves, a meaningful portion of that institutional memory leaves with them.
Knights Counsel replaces manual folder-tree searching with a conversational interface inside Microsoft Teams. An authorized user asks a question and receives an answer drawn only from the office's own documents, with citations linking back to the source files. A second capability lets a user file new documents through the same conversation, with the assistant proposing a category and metadata for the user to confirm before anything is written.
The cost of getting this wrong is not a bad search result. An assistant that appears to give legal advice creates real professional risk for the attorneys relying on it.
Constraints
- Relay, not advise. The system reports what documents say. It does not interpret them, recommend a course of action, or predict an outcome. This boundary was set by the nature of the work, not chosen for convenience, and it shapes nearly every behavior in the system.
- Hard tenant boundary. All work inside the client's enterprise Microsoft 365 environment. No external servers, no personal cloud, no third-party AI tools touching client material, no privileged material on personal machines.
- Inherited access, not reinvented access. The agent authenticates each user through the enterprise identity provider and operates under that user's existing document permissions, so privilege handling is not something the team implemented and could get wrong.
- A non-technical executive sponsor. The client is a domain expert with no software background, which made the ownership boundary explicit: the sponsor owns the document taxonomy, because only they know how the office thinks about its own work. The team owns the mechanism.
- An 8,000-character instruction budget. A real engineering constraint that forced case-by-case decisions about which rules live in instructions and which have to be pushed into topics, metadata, or flows.
What I decided
Two agents, not six
I originally designed a six-agent architecture: one agent per document category, plus a dedicated search agent and a conflict-resolution agent. It was clean on paper and I had already written the specification documents for it.
When I looked at the evaluation logs, there were zero retrieval-precision failures caused by mixed document types — which was the entire justification for splitting by category. Child agents also inherit the parent's knowledge source, which removed the folder scoping the split was supposed to buy. Meanwhile every orchestration hop cost latency and per-message credits.
I collapsed it to two: an Agent Manager owning all search, citation, conflict handling, and every reply the user sees, and a Document Sorting child agent owning classification and all writes. Only the Agent Manager speaks to the user, stated as a standing principle so that adding a delegation target later is an addition rather than a rewrite.
Testing then produced a standing architectural rule: attaching the same document path to both a parent and a child produced duplicate retrieval and unstable citations, so a given path is now attached to exactly one agent — never to both a parent and a child, and never to two children.
Rejected: the six-agent topology. Recorded as an architecture decision record rather than quietly deleted, so the reasoning survives for whoever picks this up next.
Where the line between relay and advice actually falls
A naive reading of "no legal advice" would make the tool useless. Surfacing a clause the user intends to reuse is relay — the agent is reporting what exists. What converts a response into advice is a recommendation or prediction layered on top of the citation.
The area the original specification left vague was calculation, so I drew the boundary explicitly: arithmetic on an explicit, cited figure or date is permitted, because computing a ninety-day deadline from a stated renewal date is addition applied to two stated facts. Any calculation that requires the agent to choose which rule applies is not, because selecting among several notice provisions is legal judgment. Characterising urgency, ranking, or predicting an outcome is not, because that is evaluative framing on top of a fact.
Rejected: treating any question containing a judgment component as wholly unanswerable. Refusing the retrievable half along with the rest is a defect, not caution — the specified behavior is to relay the facts with citation first, then redirect the judgment.
Two kinds of not-found, kept separate
Silence and guessing are both unacceptable when the library has no answer, but the two cases mean different things to an attorney. Nothing relevant in the corpus gets a plain statement that the office materials do not cover the question, plus a redirect. Related documents that exist but do not directly address the question get named, with a suggestion to review them or follow up.
The first case could not be enforced from the agent's instructions at all. With ungrounded responses disabled, a query returning zero matches never reaches the model, so the platform's default fallback fires first. That wording had to be set on the fallback topic instead — the first of several behaviors that had to migrate out of instructions entirely.
The agent proposes, the human disposes
The second half of the system is document intake, and it mirrors the relay boundary on the retrieval side: the agent does the mechanical work and the human keeps the decision. It never writes on its own judgment. It reads the file, proposes a category and a complete set of metadata in one message, and a write happens only after explicit confirmation.
The rule that took a revision to get right is that it always proposes. A field it cannot determine is shown as not determined rather than left blank or invented; a low-confidence classification is still presented, marked as such, with the alternative it considered. An earlier draft routed uncertain files to an unsorted holding area, which was wrong — an uncertain proposal a human can correct in one word beats no proposal, and a file quietly parked in a holding area is a file nobody is prompted to fix.
Rejected: asking the user field-by-field questions. Filing a document should cost one message in the common case, and the cost of a batch should scale with the number of corrections needed, not the number of files.
A managed platform over a custom build
I evaluated two candidate architectures on answer reliability, cost, build time, and organizational supportability: a managed low-code agent platform grounded on SharePoint, versus a custom search-plus-model-plus-blob-storage build. I selected the managed path and drove it to a locked decision, confirmed independently by enterprise IT.
Rejected: the custom build. It would have been more interesting to write and considerably harder for the client's own IT organization to support after the capstone ended.
Directive language, never advisory
Testing established that soft prompt language — "prefer", "may ask" — is treated as advisory by the model and dropped under pressure. I converted every required behavior to directive language and made that a standing authoring rule for the team.
The related finding: any behavior that must occur when retrieval returns nothing cannot live in instructions at all, because instructions shape how the model answers and do not run when the model is never invoked. Those behaviors had to move into topics.
Descoping the custom front end
I scoped and stood up a SharePoint Framework web part in React, TypeScript, and Fluent UI as a potential custom surface, including its real toolchain constraints. Once the native delivery surface was confirmed to meet the client's needs, I recommended descoping it from the MVP critical path to a stretch goal.
Rejected: shipping the custom front end in the MVP. It added build risk without adding user-visible value at pilot scale.
How I knew it worked
This is the part I would lead with. Anyone can say they built an agent; the harder claim is knowing whether it behaves as specified.
- An 18-case evaluation suite covering every specified behavior: citation format, advice refusal, the relay/advice boundary, both forms of not-found, ambiguity handling, document conflicts, greetings and off-topic requests, session memory, tone, set-scoping filters, and comparison queries. It grew from 15 cases as new failure modes surfaced.
- A custom seven-label grading rubric with strict precedence, first failure wins: overstepped into advice, accepted unsupported task, ungrounded or uncited, unsupported filtering, off-topic leak, wrongly declined, missing required closer, and correct behavior.
- A multi-run testing methodology. Thirteen of fifteen responses changed between two runs taken eighteen minutes apart with no configuration change of any kind. Static topic messages were byte-identical across both runs while every generated response drifted. Single-run results on generated responses are noise, so every proposed change now requires two to three runs plus a version bump before any behavioral delta can be attributed to it.
- A fixed grading rule: only the visible response text for a turn is judged, never internal orchestration or which retrieval path fired. That keeps a test case valid regardless of whether the behavior is later implemented in instructions, in a topic, or in a flow.
Why the built-in grader had to be replaced
The platform's own quality grader scores whether a question was answered. This system's defining feature is that it refuses in defined circumstances: it declines to give legal advice, it says so when a document does not exist, and it redirects judgment calls to an attorney. Every one of those correct behaviors registered as a failure.
On one run the built-in grader reported 60% against a result that was correct on eleven of thirteen cases by the design criteria. Rather than tune the agent toward a metric measuring the wrong thing, I built the replacement rubric. That is the distinction the whole project turns on: a system failing and a metric being wrong look identical on a dashboard.
What broke
The most consequential defect surfaced through evaluation, not through use. The agent answered single-fact questions well, so on the surface it looked fine. But when a question scoped a set rather than asking for a single fact — which agreements are governed by law outside this state, the five most recent agreements for a given unit — the agent ran a semantic search and then presented the unfiltered results as though they had been filtered and ordered.
Every individual citation was correct, which is exactly why it was dangerous. It did not look like a hallucination. It looked like an answer. I caught it by running the same filter question three times inside one session and getting three different document sets.
The root cause was an absent query predicate: semantic similarity cannot function as a filter, and no amount of instruction tuning changes that. The fix lived in structured metadata columns — a workstream the team had been treating as a refinement. I moved it onto the critical path.
A second class of defect: an agent component's description text alone caused the orchestrator to assert that files had been uploaded and processed when the component's actions performed no such write. I confirmed the diagnosis against the platform's own reasoning trace and turned it into a standing requirement — any component whose safety property depends on a confirmation step must have a description that affirmatively rules out unperformed actions.
Two quiet failures worth naming
Filenames over 100 characters, or containing parentheses, are silently skipped by the indexer. A skipped file is invisible to retrieval while looking perfectly normal in the library — the most dangerous failure mode in the system, because nothing anywhere reports it. Rather than rely on user discipline, intake validates and renames on upload and retains the original name in its own column. The internal working target is 80 characters, so the margin absorbs category codes and dates without anyone hand-counting.
Agent-written metadata can become a grounding hazard. Several schema fields — summary, key issues — are authored by the agent rather than read from the document. Stored without distinction, a later retrieval could surface an agent-written sentence and cite it as though it were document content, violating the grounding rule through a metadata pathway rather than a prompting one. Those fields therefore carry an explicit generated-and-unverified marker, so the retrieval side treats them as navigational aids and never as citable source text.
What I would do differently
- Build the evaluation suite before the agent, not alongside it. Several instruction revisions were spent chasing behaviors that a test case would have specified precisely in the first place.
- Establish the multi-run rule at the start. Early single-run comparisons produced conclusions I later had to discard, because I could not distinguish a real improvement from run-to-run drift.
- Treat metadata as infrastructure from day one. I classified it as a refinement because it looked like data entry rather than engineering, and the evaluation eventually proved that wrong.
Artifacts
- Technical white paper, v1.0 (August 2026) — co-author, and sole author of the system design and behavior specification sections. Available on request.
- Final design document, v2 — contributing author, sole author of the system design section, including the behavior test set and the architecture decision records. Available on request.
- No live demo. The system runs inside a client tenant under NDA and cannot be shown. That constraint is worth stating plainly rather than working around.
More work on the projects page, or back to Selected Work.