Demos are built to look good. The only reliable way to know whether an AI review tool helps your team is to test it on your own work, against a result you already trust. Here's a simple pilot that works for any tool, including ours.
Step 1. Pick a set you've already reviewed
Choose a completed plan set where you know what was found: internal comments, agency comments, and any issues that surfaced later. That history is your answer key. A set with a mix of finding types (grading, drainage, code, cross-sheet) will tell you more than a clean one.
Step 2. Decide what counts as a "catch" before you run it
Write down your scoring rules first, so the result isn't argued after the fact. A useful split:
- Confirmed catches: real issues your team had also found.
- New real finds: real issues nobody had flagged. These matter most.
- False positives: flagged items that aren't problems.
- Misses: issues in your answer key the tool didn't surface.
Step 3. Run it, then verify a sample of findings yourself
Count how many findings you can verify quickly. A finding with a sheet reference, a location, and a source for any code requirement takes minutes to check. One without those details sends your reviewer hunting, which cancels out the time saved. Track how long verification takes per finding.
Step 4. Test the jurisdiction question
Run it on a set from a jurisdiction you know well. Does the tool identify the right authority and the codes actually adopted there? Can you correct or add to that basis? What does it do when it can't find a requirement? A tool that fills gaps with plausible-sounding citations is a risk.
Step 5. Check the non-technical basics
Before anyone uploads a client set, get clear answers on data handling (training use, retention, access), who approves findings, and where responsibility sits. The engineer of record remains the engineer of record, so every finding should require human approval before anything is acted on.
How to read the results
- High confirmed catches and few misses means it covers what your team already covers.
- New real finds are the strongest signal of added value.
- False positives are the cost. Weigh them against verification time.
- If findings are hard to verify, the time savings won't materialize.
We'd rather you test us than take our word for it. If you want to run this pilot with CivilPlanAI, we're glad to set it up.