Data Governance
The harder half is upstream: was each question classified correctly?
Whether paper building is usable at all comes down to one tedious thing —— whether every question was filed under the right topic. Having a teacher review thousands one by one isn't realistic; handing it entirely to a model isn't trustworthy.
rulesOfficial specification as the baseline constraintdraws the legal classification space
↓
humanTeacher-view topic ontology + teacher method tagsthe standard is set by people; models take no part in defining it. The physics ontology is on v3, converged from 44 nodes to 27
↓
modelLLM classifies question by questionthe grunt work that needs scale goes to the model
↓
modelTwo-agent independent review: solver and reviewer kept separateso no single agent grades its own work
↓
stronger modelSampled calibration by a stronger modelspecifically to offset same-source bias
↓
gateA / B / C triageproduction · review_queue · quarantine
↓
humanTeachers settle disagreements · taxonomy rebuilt to the teacher's standardfinal judgement always stays with people
↓
releaseTiered release: high-agreement ships first, disputed goes to internal testingneeds_review never reaches production
humans set the standard and make the call
models label and review
rules and gates
The A / B / C gate
A
production
High confidence, both agents agree → straight into the production bank, retrievable.
B
review_queue
Disagreement or middling confidence → into the review queue for a teacher to settle. Never ships to production.
C
quarantine
Low confidence or clearly anomalous → isolated; doesn't even enter the review queue until it's rerun or a human steps in.
Why the sampling check has to use a stronger, different model
I started out using the same model to both answer and review. The agreement rate looked great. Then I realised what it was: same-source bias. A model naturally agrees with its own judgement, so a high agreement rate only proves it is self-consistent —— not that it is right.
The fix was to bring in a stronger model from a different generation for sampled calibration, and to archive every disagreement as its own document. Two independent sets of data show why this was necessary.
Errors that come in clusters are invisible question by question
After rebuilding the physics ontology I ran a full review: across 637 accepted questions, the independent review agent agreed with the pipeline 85.9% of the time, and of the 90 disagreements 79 were ruled pipeline misclassifications. The important part is that those 79 were tightly clustered on 4 knowledge boundaries —— standing waves, emf and circuits, scattering and accelerators, vectors and SUVAT. Systematic errors in clusters like that never surface in spot checks; only a full cross-check brings them out.
The measured cost of same-source bias
Later, while raising quality across chemistry / biology / economics / psychology, I made stronger-model calibration a fixed step specifically to offset that bias. It caught 20 over-corrections by the review agent —— psychology's over-correction rate reached 25.8%, and one unit was ruled against 16 out of 16. Without switching models, those 20 would have been written into the bank as fixes. 164 confirmed corrections were written back in the end.
Definition statement · please read this with the numbers
The "agreement rates" I quote (e.g. ~98% on FP2 and ~81% on FP3 in one subtopic relabelling round) are internal metrics: the denominator is the batch of questions being labelled, and the comparison is between the labelling model and an independent review agent —— not against human ground truth. It measures how much two independent judgements agree, and cannot be read as accuracy.
A percentage without a stated denominator carries no information in an evaluation context —— which is why the definition sits next to the number.
