“A human reviews everything” is the most common sentence in AI project proposals and the least examined. If the review is slow, people skim. If people skim, the review is theatre.
We built a platform that generates exam questions and learning cards for Tavis, a Malaysian online tuition provider, in English and Bahasa Melayu. Wrong content reaches students, so review is not negotiable. Most of the engineering went into making it cheap. These are the patterns that did the work.
1. Let AI review AI first
Before any person sees a generated item, a review job casts independent AI votes on it. The number of votes is a setting. Each vote returns a verdict and, where it disagrees, a proposed correction. The votes are tallied.
- A strict majority judging the item good: it is approved automatically.
- Anything else, a tie included: it is marked as needing human review and goes to an editor.
This does not replace people. It sorts their queue. An editor’s attention goes to the minority of items where something is genuinely uncertain, instead of being spread evenly across everything.
One deliberate choice: an automatic approval is only ever local. It never publishes. Publishing is always a separate, explicit human action.
2. Show a diff, not a document
When a review pass proposes a correction, the editor does not get a new version to reread. They get the original with additions and deletions highlighted word by word.
Checking “the model wants to capitalise NOT in this stem” takes two seconds. Rereading a full question to work out what changed takes thirty, and people stop doing it carefully after the first fifty.
The rule we follow: never apply an AI suggestion silently. Every change is visible and attributable to either a person or a named review job.
3. Ground generation in source material
Review is cheaper when there is less to catch. The platform does not ask a model what it knows about a topic. Subject teams upload the actual textbooks and papers, the system extracts them into a per-chapter knowledge base, and generation draws on that.
This removes the largest class of error, plausible content that is off-syllabus, before it reaches review. It also gives the reviewer something to check against.
Figures get the same treatment. When a diagram is redrawn, an automated fidelity check compares the redraw against the source figure and reports discrepancies.
4. Put prompts in the hands of the people who know the subject
Every stage of the pipeline has its own prompt and model setting: extraction, chapter mapping, generation, tagging, review and so on. They are stored as prompt sets, assignable per subject, and editable by the content team in the application.
When the mathematics editors notice the model writing ambiguous distractors, they adjust the instruction themselves and see the effect on the next job. No developer, no deployment.
5. Make long jobs visible
Generating and reviewing fifty questions takes minutes. A spinner for minutes teaches people to open another tab and forget.
Jobs run on a queue and stream their status to the browser as they progress: how many done, how many passed, how many escalated. A failed review job can be retried, and it resumes: votes already cast are kept and only the missing ones are re-cast. People stay with the task because they can see it moving.
The parts nobody demos
Underneath are the things that make the difference between a prototype and a system a content team relies on:
- Invariants. One learning card per chapter and term, for example, so the same concept is not generated twice.
- Tests. More than 1,150 automated tests on the backend. When a prompt or model changes, the contract around it holds.
- A one-way publish. Approved content is pushed to the client’s production platform by an explicit action, with images copied alongside.
- A history. Who approved what, when, from which job.
How to tell if your review step is real
Ask three questions of any AI workflow.
How long does it take to review one item? If the answer is more than a few seconds for the common case, it will not be done properly at volume.
What does the reviewer see? If it is the raw output with no source and no diff, they are being asked to redo the work.
What happens to the reviewer’s correction? If it is not recorded, you cannot learn from it or prove it happened.
A person in the loop is only a safeguard if the loop was designed for a person.