The problem
Named entity recognition, the task of finding and labelling things such as names, organisations, dates and amounts in text, looks solved on benchmark data. Real-world documents are a different matter. They have inconsistent formatting, domain-specific terminology, abbreviations, and entities that only make sense in context.
A model that scores well on a clean benchmark can still miss what matters in the documents a business actually handles. And because extraction usually feeds other systems, its errors do not stay contained: a missed or mislabelled entity becomes a wrong field in a record.
What we built
An NER pipeline tuned for messy real-world documents, with evaluation and human review built in, so accuracy is measured and corrected rather than assumed. The work covered the pipeline design and the stacking of models.
How it works
Two ways of finding an entity
The pipeline stacks two complementary approaches.
- BERT-CRF models read the text and label each word. BERT provides a rich understanding of each word in context, and a conditional random field (CRF) on top enforces that the sequence of labels makes sense, for example that an entity spanning several words is labelled consistently from start to end.
- Reverse question answering approaches the task from the other side. Instead of labelling every word, it asks the document a direct question, such as what the contract date is, and extracts the answer.
The two methods fail in different ways. Sequence labelling is thorough but can miss an entity that is phrased unusually. Question answering is targeted and handles unusual phrasing well, but only finds what it is asked for. Combining them catches cases that either would miss alone.
Iterative fine-tuning
RoBERTa models are fine-tuned in rounds. After each round, errors are examined, and the next round is informed by where the previous one fell short. This steadily adapts the models to the particular documents and entities that matter.
Evaluation harnesses
Performance is measured on real documents, not only on benchmark data, and broken down by entity type. An overall accuracy figure can hide a model that is excellent on common entities and poor on the rare ones that matter most.
Human review loops
Cases where the pipeline is uncertain are routed to a person. Their corrections are fed back into training, so the system improves from exactly the cases it found hardest.
The pipeline is built with Transformers and spaCy.
Why the review loop matters most
For extraction that feeds real decisions, a pipeline that knows where it is uncertain and routes those cases to a person is worth more than one that is slightly more accurate but silent about its mistakes. The review loop converts uncertainty from a hidden risk into a managed workflow.
Where it can go wrong
- Confident errors. The dangerous mistakes are the ones the model is sure about, because they bypass review. Calibration of confidence matters as much as accuracy.
- Label drift. Reviewers must label consistently, or corrections will teach the model contradictory lessons.
- New document types. A pipeline tuned for one kind of document may perform poorly on another. New formats need evaluation before they are trusted.
Questions to ask before building one
If you are considering a system like this, these are the questions we would want answered first.
- Which entities actually matter? List them, with examples, and rank them by the cost of getting them wrong.
- What does your real data look like? Gather a representative sample, including the messiest documents, before choosing an approach.
- Who will review uncertain cases? A review loop needs people, time and consistent labelling guidance.
- Where does the output go? Know which systems consume the extracted data, because that defines the accuracy you need.