Better models give us more to build with. At Colabra, we compare diligence work done with our software against work done with direct access to the same files. That helps us identify where our product makes the review better and which tasks we can hand over to the model.
Answering a question about one contract and reviewing a whole data room require different tests. For each, I'd compare the findings, the evidence behind them, and the work a person has to do to check and finish the review.
Give the model the tools it needs
Direct file access might include a directory listing, search, scripts, and a request list describing which documents should exist. An application might offer structured queries, citations, and saved review state. The evaluation needs to describe the tools available in each setup.
For example, a model can look for missing documents by comparing an inventory with a request list. We need to check whether it makes that comparison and whether the inventory covers all the files. A missing-document test would be misleading if we withheld the request list from one setup.
If the model can use scripts for calculations or comparisons across documents, it should be able to use them in the test. Restricting those tools would limit what we could conclude from the result.
Score the work a reviewer receives
Consider an agreement changed by a later amendment. An answer can quote the original clause accurately and still be wrong because it ignores the amendment. Checking the citation alone would miss the error.
I would assess these parts of a diligence review separately:
- Coverage: Which documents and questions were examined? What was skipped or could not be processed?
- Source support: Does the cited passage support the finding? Is it the operative agreement or a superseded version?
- Materiality: Does the system distinguish an important exception from ordinary contract language?
- Review: Can a person correct the finding, record a decision, and return to it later?
- Cost and completion: Did the review finish within a useful time and cost, including any work the person had to redo?
A single average can hide those differences. I want to see which scenarios failed and read the sources for those findings, especially when the answer sounds convincing.
What synthetic tests can establish
We have used synthetic data rooms with known issues to compare Colabra with direct file access. We can check whether each system finds the issues we planted without using confidential customer documents.
The results depend on the documents and tasks we chose, the model's tools, and the grading method. An automated judge can disagree with another judge or reward a well-written answer that misses an important issue. I want to inspect those disagreements and check the grading against human review.
Synthetic tests guide product development. Production reliability and customer adoption require evidence from actual use. For a buyer evaluating Colabra, I'd propose a historical deal with an agreed scope and success criteria, so the team could compare our findings with its own review.
Let the result change the roadmap
If direct access works better, the individual failures should guide what we change. A missing source may call for better retrieval. A review that takes too much setup may call for fewer steps in the interface.
We can also let a customer use their preferred AI interface. Colabra's MCP connection lets an external AI client work with project evidence it has permission to access. Colabra maintains the project and access controls; the available tools depend on the connection and its permissions.
A team can bring Colabra's document tools into its own AI workflow, and we can evaluate the result by the same standards: what the AI reviewed, whether its findings hold up, and how much work it saves the reviewer. Those are the questions I bring to buyers considering an evaluation, and the results help us decide what to build next.