Measured accuracy
Last updated: 2026-08-23
Extraction, evidence reading, clause retrieval and review grids, run against a fixed set of invented contracts whose answers are known, on the same prompts and models the product uses.
What was run
Four groups of checks over a fixed set of three invented contracts whose answers are known and committed alongside the harness. Extraction, evidence reading, clause retrieval and one review grid row, each running the product's own prompt, its own validators and the model the deployment resolves. The contracts carry no customer data. A number here says the mechanism works on paper whose answers are known; it does not predict a number on paper it has not seen.
| Run on | 2026-08-23 |
|---|---|
| Checks | 22 |
| Met | 17 |
| Missed | 5 |
| Checks met | 77.3% |
Models
| Job | Model |
|---|---|
| Extraction, evidence reading and review grid | anthropic/claude-fable-5 |
| Clause embeddings | voyage/voyage-law-2 |
Prompt versions
| Surface | Version |
|---|---|
| Extraction | clm-extract-2.9.0 |
| Evidence reading | clm-evidence-1.0.0 |
| Review grid | clm-grid-1.0.0 |
Extraction: does the register carry the duties and the dates
The product's extraction engine reads each of the 3 fixtures once, on the model the deployment resolves. A check is met when the number of accepted duties falls inside the band a careful reader would produce, when every date the paper states in full is quoted by some citation, and when each subject that matters is cited: 8 dates and 11 subjects across the set. Every citation is verified against the source text before the duty is accepted, so a quoted phrase is a phrase the paper carries.
- ExtractionMissed
Northmoor Analytics master services agreement: duties, dates and citations
- Known answer
- Between 8 and 26 duties, with 3 stated dates and 4 subjects cited.
- What came back
- 18 duties, 2 of 3 dates quoted, 4 of 4 subjects cited.
- Duties accepted: 18
- Band: 8 to 26
- Dates quoted: 2 of 3
- Subjects cited: 4 of 4
- Citations refused by the gate: 4
- Duplicates suppressed: 0
- Sections covered: 11
- Model turns: 16
- Input tokens: 32
- Output tokens: 9479
- ExtractionMet
Larkfield Insight data licence: duties, dates and citations
- Known answer
- Between 7 and 22 duties, with 1 stated dates and 4 subjects cited.
- What came back
- 15 duties, 1 of 1 dates quoted, 4 of 4 subjects cited.
- Duties accepted: 15
- Band: 7 to 22
- Dates quoted: 1 of 1
- Subjects cited: 4 of 4
- Citations refused by the gate: 1
- Duplicates suppressed: 0
- Sections covered: 11
- Model turns: 8
- Input tokens: 16
- Output tokens: 8542
- ExtractionMissed
Pelham Systems order form: duties, dates and citations
- Known answer
- Between 4 and 16 duties, with 4 stated dates and 3 subjects cited.
- What came back
- 10 duties, 3 of 4 dates quoted, 3 of 3 subjects cited.
- Duties accepted: 10
- Band: 4 to 16
- Dates quoted: 3 of 4
- Subjects cited: 3 of 3
- Citations refused by the gate: 0
- Duplicates suppressed: 0
- Sections covered: 7
- Model turns: 9
- Input tokens: 18
- Output tokens: 6328
Evidence reading: does silence come back as silence
One duty and one file that has nothing to do with it. The known answer is cannot_tell, and it is known for the plainest possible reason: the document is about staff parking. A verdict of appears_to_satisfy here would be a model willing to mark a duty performed on the strength of an unrelated upload.
- Evidence readingMet
A parking notice filed against a penetration test report duty
- Known answer
- cannot_tell, because the file is about something else entirely.
- What came back
- cannot_tell, with 2 reasons.
- Reasons given: 2
- Characters of the file read: 794
Clause retrieval: does a typed question find the right clause
The fixture set is cut into 38 passages exactly as the product cuts a clause read, embedded on the product's pinned embedding model, and each of the 8 questions is embedded and read against them by cosine similarity, which is the same measure the clause index reads by. A check is met when the closest passage is in the right document, carries the phrase that answers the question, and sits at or above the question band's floor. The floor matters: a correct passage below it is one the product would not show.
- Clause retrievalMet
"what happens if the customer pays late"
- Known answer
- The passage in northmoor-msa carrying "interest on the overdue amount".
- What came back
- The right passage came first, reading as close.
- Top similarity: 0.433
- Rank of the right passage: 1
- Closest wrong passage: 0.35
- Clause retrievalMissed
"limit on the damages we can claim"
- Known answer
- The passage in northmoor-msa carrying "total aggregate liability".
- What came back
- The closest passage was in northmoor-msa, and the right one placed at 2.
- Top similarity: 0.401
- Rank of the right passage: 2
- Closest wrong passage: 0.401
- Clause retrievalMet
"can we assign this contract to a buyer of our business"
- Known answer
- The passage in northmoor-msa carrying "purchaser of the whole of its business".
- What came back
- The right passage came first, reading as strong.
- Top similarity: 0.567
- Rank of the right passage: 1
- Closest wrong passage: 0.365
- Clause retrievalMet
"how much notice to stop the renewal"
- Known answer
- The passage in northmoor-msa carrying "written notice of non-renewal".
- What came back
- The right passage came first, reading as loose.
- Top similarity: 0.392
- Rank of the right passage: 1
- Closest wrong passage: 0.331
- Clause retrievalMet
"who owns the data"
- Known answer
- The passage in larkfield-licence carrying "retains all right, title and interest".
- What came back
- The right passage came first, reading as strong.
- Top similarity: 0.461
- Rank of the right passage: 1
- Closest wrong passage: 0.293
- Clause retrievalMissed
"when can either side walk away early"
- Known answer
- The passage in larkfield-licence carrying "The Client may terminate this Agreement at any time".
- What came back
- The closest passage was in northmoor-msa, and the right one placed at 6.
- Top similarity: 0.345
- Rank of the right passage: 6
- Closest wrong passage: 0.345
- Clause retrievalMet
"what has to be kept confidential"
- Known answer
- The passage in larkfield-licence carrying "keep confidential all Confidential Information".
- What came back
- The right passage came first, reading as close.
- Top similarity: 0.423
- Rank of the right passage: 1
- Closest wrong passage: 0.284
- Clause retrievalMissed
"who pays if a third party sues over the software"
- Known answer
- The passage in larkfield-licence carrying "indemnify the Client".
- What came back
- The right passage came first, reading as below the floor.
- Top similarity: 0.318
- Rank of the right passage: 1
- Closest wrong passage: 0.297
- Clause retrievalMet
The index the questions were read against
- Known answer
- One passage index built from the fixture set alone.
- What came back
- 38 passages from 3 documents.
- Passages indexed: 38
- Questions asked: 8
- Embedding tokens: 2162
Review grid: does a cell answer, or say the paper is silent
One agreement, 8 questions, one call, exactly as the product runs a grid row. 5 of the questions are answered by the paper and 3 are not: the fixture carries no insurance clause, no data processing terms and no force majeure clause. A check is met when the cell comes back with the status the paper supports. An answered cell must cite a tag it was shown, which the product's own validator enforces before a row is stored, so a row that cited evidence it was never given would not reach these checks at all.
- Review gridMet
What is the cap on liability, and what does it exclude?
- Known answer
- answered, from "total aggregate liability".
- What came back
- answered, with 2 citations.
- Citations: 2
- Review gridMet
Which law governs the agreement?
- Known answer
- answered, from "law of England and Wales".
- What came back
- answered, with 1 citation.
- Citations: 1
- Review gridMet
How much notice is needed to stop the agreement renewing?
- Known answer
- answered, from "(120) days' written notice".
- What came back
- answered, with 1 citation.
- Citations: 1
- Review gridMet
How long does the customer have to pay an invoice?
- Known answer
- answered, from "thirty (30) days of receipt".
- What came back
- answered, with 1 citation.
- Citations: 1
- Review gridMet
Can either side terminate for convenience, and on what notice?
- Known answer
- answered, from "ninety (90) days' written".
- What came back
- answered, with 1 citation.
- Citations: 1
- Review gridMet
What insurance must the supplier carry?
- Known answer
- cannot_tell, because the paper does not mention insurance.
- What came back
- cannot_tell, with 0 citations.
- Citations: 0
- Review gridMet
Are data processing terms incorporated, and where from?
- Known answer
- cannot_tell, because the paper does not mention processing.
- What came back
- cannot_tell, with 0 citations.
- Citations: 0
- Review gridMet
What happens if a party is prevented by an event outside its control?
- Known answer
- cannot_tell, because the paper does not mention force majeure.
- What came back
- cannot_tell, with 0 citations.
- Citations: 0
- Review gridMet
The row as a whole
- Known answer
- 5 answered and 3 silent, over the evidence the row was shown.
- What came back
- 5 answered and 3 silent.
- Cells returned: 8
- Evidence tags offered: 17
- Answered: 5
- Said the paper is silent: 3
What these figures cover
The contracts in the fixture set are invented, short, and written for this harness. They carry no customer data. A score on them says the mechanism works on paper whose answers are known; it does not predict a score on your own contracts, which are longer and stranger than any fixture.
The way to read a number here is as a floor that a change to a prompt or a model has to clear. That is what the harness is for, and it is why the fixtures and their known answers are committed alongside the code rather than described.
The rest of the picture is on the security page, including which provider processes what and what provenance every reading carries.