The Definitions Index Problem

Large language models have made it tractable to extract structured rules from natural language policy documents. But the dominant approach — chunking a document into segments and processing each independently — contains a fundamental flaw that becomes catastrophic on documents longer than roughly thirty pages. We call this the definitions index problem: the systematic failure of chunked extraction pipelines to resolve cross-references and defined terms that are declared in one part of a document and used throughout the rest.

This is an engineering case study, not a discipline-level claim — it documents one system's extraction pipeline and the failure mode it hit. The underlying structural problem, however, generalizes to any pipeline processing legal or regulatory text.

1. Where we started

The premise is straightforward: take a governance document — a lending policy, a regulatory compliance handbook, an operational procedure — and produce a set of machine-evaluable rules. Each rule should be an abstract syntax tree node: a condition expressed as a concrete field, operator, and value, paired with a deterministic action.

The first version of the extraction pipeline was what any reasonable engineer would build: split the document into chunks of roughly 5,000 characters with some overlap, send each chunk to a language model with a carefully designed prompt, collect the extracted rules, deduplicate by name, and return the merged set. This worked well on short documents — a ten-page credit policy, a five-page fraud escalation procedure. Then a real 100-page lending policy from a major financial institution was processed. The pipeline produced 34 rules. Roughly 40% of them were unusable.

2. The failure mode

The document had a structure typical of legal and regulatory writing. Section 1 — Definitions: four pages of precise definitions (“Default means the failure by the Borrower to pay three or more consecutive installments without prior written approval from the Bank.”). Sections 2 through 9 — Operational clauses: forty pages of rules that used the defined terms as if their meaning was self-evident (“Upon Default, the Bank may accelerate the outstanding balance.”).

The chunked pipeline processed each section independently. The chunk containing the Default definition was in chunk 1. The clause using it was in chunk 7, which went to the model without chunk 1. The model did what any reasonable system would do given incomplete context: it produced a partially-specified rule referencing an invented field, borrower_in_default — a concept it understood but had no definition to resolve into a real data field. The condition was not evaluable. The rule could not be sealed.

This wasn't a failure of the language model. It was a failure of the pipeline architecture. We gave the model an incomplete document and expected a complete answer.

3. Why obvious fixes don't work

Larger chunks help marginally but do not solve the problem: a 100-page document is roughly 200,000 characters, and no matter where the chunk boundary falls, a definition in section 1 will be separated from its usage in sections 6 through 8 — the problem is structural, not a matter of chunk size.

Overlap between chunks helps when adjacent chunks need shared context, but does nothing for a definition in section 1 referenced in section 9; the overlap window would need to be the entire preceding document, defeating the purpose of chunking.

Sending the full document fails for two reasons: extraction quality degrades significantly on very long inputs as attention dilutes, and a single very-long-context call is expensive and slow — more importantly, it does not solve the underlying architectural problem, it just papers over it with a larger window.

4. The insight: separate indexing from extraction

The breakthrough was recognising that the naive pipeline collapses two distinct phases into one. Phase 1: understanding the document's vocabulary — a reading comprehension task across the entire document, before extracting any rules. Phase 2: extracting evaluable conditions from individual clauses — once every term's meaning is known, each clause can be processed independently with the definitions available as context.

Legal documents are written this way deliberately: a well-drafted contract front-loads definitions so operational clauses can be written precisely without repetition, assuming the reader has already internalised the definitions section. The original pipeline violated this assumption by processing sections as if independent.

The solution is to replicate the human reading process architecturally: read the definitions first, build an index, then extract from clauses with that index in hand.

5. The five-stage pipeline

Stage 1: Definitions index extraction

Before any chunking begins, a full pass over the entire document builds a definitions index, collecting three categories of content: explicit definition sections (Definitions, Glossary, Interpretation, Defined Terms), the terminal portion of the document (schedules and annexures, where the last 20% of most legal documents defines supplementary terms), and inline definition patterns (“X means...”, “X shall mean...”) captured via a regex pass over the full text. The output is a flat dictionary of term to definition, built once and persisting for the lifetime of the extraction run.

Stage 2: Section-aware chunking

Rather than chunking by raw character count, the revised pipeline chunks by document structure, splitting at heading boundaries from a layout analysis layer — keeping semantically related content together and respecting the document's own logical divisions.

Stage 3: Parallel extraction with context injection

Each chunk is sent to the extraction model in parallel, with the critical change that every call receives the full definitions index injected as additional context. When the model now encounters “upon Default, the Bank may accelerate”, it has the precise definition available and produces a fully evaluable condition instead of an invented field.

Stage 4: Coherence pass

Parallel extraction across chunks produces a new problem — the same rule extracted multiple times from adjacent sections. A name-based deduplication pass removes exact duplicates but misses semantic ones. The coherence pass sends the full merged rule set to a single model call tasked with identifying semantic duplicates and flagging any rules that still contain unresolved cross-references.

Stage 5: Targeted cross-reference resolution

For rules still flagged as containing unresolved references, the pipeline parses the reference, locates the referenced section in the full document text, appends it to the rule's context, and calls the extraction model again. Where the referenced content is itself ambiguous or requires human judgment, the rule is flagged as non-evaluable and routed to human review. The system does not guess.

6. What this produces

On the same 100-page lending policy that initially produced 34 rules with 40% unusable, the revised pipeline produces 61 rules, 94% fully evaluable on first pass. The remaining 6% are flagged with explicit reasons — genuinely ambiguous language, references to external documents not included in the upload, or judgment calls requiring human discretion.

The improvement is not primarily in the number of rules extracted. It is in the quality: conditions that previously produced vague boolean flags now express precise numeric thresholds against fields that exist in the data schema. The output is not approximately right. It is evaluable or it is flagged.

7. The broader implication

The definitions index problem is not specific to any one system. It is a structural property of legal and regulatory documents that any extraction pipeline will encounter. Legal documents separate definitions from operational clauses deliberately, for precision and internal consistency — which makes them opaque to chunked machine processing, because the precision of the operational clauses depends entirely on the reader having internalised the definitions section.

Most extraction pipelines built on language models process short documents where this problem does not arise. Applied to regulatory compliance documents, the failure is silent: the output looks structured, the fields look reasonable, the conditions look evaluable — but the fields are invented, the thresholds are guessed, and the conditions fail at runtime when the invented field is not present in the decision context.

The silence of this failure is what makes it dangerous. A governance system that silently produces wrong rules is worse than one that produces no rules, because it creates the appearance of compliance without the substance.

8. Implementation notes

Three architectural choices are worth stating explicitly. Layout-aware PDF parsing: text is not extracted from raw character streams; a layout analysis layer understands headings, sections, tables, and columns, and the resulting structural markers are the map the pipeline navigates. Structured output enforcement: the extraction model is constrained at the output layer to a typed AST schema — rules that cannot be coerced into a valid AST are flagged and routed to human review, never silently dropped or partially stored. Model allocation by task: the definitions and coherence passes are comprehension and classification tasks handled by a smaller, faster model without quality loss; the primary extraction pass uses a larger model — a deliberate architectural decision, not a cost optimisation applied after the fact.

9. After publication: three further refinements

9.1 Conditional definitions

The original Stage 1 prompt extracted definitions as flat strings, which failed on conditional definitions — “Default means the failure to pay three or more consecutive installments, unless prior written approval has been obtained.” The unless branch is often the entire difference between a rule applying and not applying. The fix is a prompt change: the extraction model now captures both branches via an explicit exception annotation, which Stage 3 reads to produce an AND condition encoding both branches. No schema change was required.

9.2 Cross-document reference resolution

Stage 5 originally resolved references only against the definitions index and the current document's text. Users often upload a parent regulation alongside a child policy, but the child's cross-references couldn't reach the parent. Stage 5 was extended with a session corpus, resolving in order: the definitions index, the current document's full text, sibling documents in the same session, then flag for human review.

9.3 Evaluability transparency

The original claim that 94% of rules were fully evaluable did not explain how evaluability was measured. The metric is structural: a rule is evaluable if its condition contains a non-empty field, a defined operator, and a non-null value — the gate every rule must pass before being sealed. An EvaluabilityReport is now computed and exposed for every extraction run, shown in the interface as a quality signal on each policy.

One deeper gap remains. Structural completeness — field, operator, value all present — is necessary for a rule to be evaluable. It is not sufficient to guarantee the rule correctly represents the policy intent. Validating semantic correctness requires domain expert annotation against a ground-truth rule set for each document type. That is a research problem, not an engineering fix, and it applies equally to any extraction pipeline. We state it here explicitly rather than leaving it implicit.

10. Conclusion

Chunked extraction works. It does not work on legal documents where defined terms are declared in one location and used throughout the rest. The failure is structural, not a limitation of the underlying model. The solution is architectural: separate the indexing pass from the extraction pass, build the document's vocabulary before processing its clauses, inject that vocabulary into every extraction call, and verify evaluability at the field level, not just the structural level.

This is not an optimisation. It is the difference between a pipeline that produces evaluable governance rules and one that produces plausible-looking output that will fail silently in production. The definitions index is not a clever trick. It is the minimum viable architecture for extraction on real regulatory documents.