Rules Engines vs. AI for Document Analysis: Strengths, Weaknesses, and When to Use Each

The question of whether to use a rules engine or an AI model for document analysis is usually asked as though one must replace the other. The more useful framing is: what does each approach find, and what does each one miss?

What rules engines are good at

A rules engine is a set of deterministic tests. For each rule, the input is a document property and the output is a finding or no finding. Rules engines excel at:

**Structural checks.** Does every internal reference resolve? Are all defined terms used? Do dates form a consistent timeline? These are well-defined tests with clear pass/fail outcomes.

**Pattern matching.** Does the document contain placeholder text? Does it include hedge words in obligation clauses? These are pattern recognition problems that a well-designed rule handles with complete reliability.

**Numerical validation.** Does the total match its components? Are values within expected ranges? These are arithmetic problems — rules engines answer them without the probabilistic errors that affect AI models.

**Consistency checks.** Does terminology drift across sections? Do figures stated multiple times agree? These are comparison problems that rules handle reliably and at speed.

The advantages of rules engines for these tasks are significant: they are fast, explainable, deterministic, and do not hallucinate. Every finding has a specific rule behind it, and that rule can be inspected, challenged, and verified.

What rules engines cannot do

Rules engines fail at tasks that require semantic understanding — grasping the meaning of text rather than its structure.

**Semantic contradiction.** "The report contains all required information" followed five pages later by "the information in Section 3 is not yet available" is a contradiction. But detecting it requires understanding what "required information" refers to and that Section 3's information was being described as required. Rules cannot do this reliably.

**Contextual appropriateness.** A rules engine can detect that "should" appears in an obligation clause. It cannot determine whether that particular instance of "should" is appropriate given the broader context of the agreement.

**Implicit risk signals.** A clause that is technically correct but creates an unusual commercial risk requires understanding of commercial norms. Rules engines cannot encode "this is unusual for a contract of this type."

**Quality and clarity assessment.** Whether a sentence is clear, whether an argument is well-structured, whether a paragraph achieves its purpose — these require judgment that rules cannot replicate.

What AI models are good at

Large language models (LLMs) trained on large text corpora develop representations of language meaning that allow them to:

**Detect semantic contradictions.** An LLM can read two statements that use different surface language but describe incompatible states of affairs, and identify them as contradictory.

**Assess writing quality.** Ambiguous phrasing, awkward construction, inappropriate tone — these are pattern recognition problems over semantic space that LLMs handle well.

**Surface implicit risk.** An LLM can identify language that suggests an unmanaged risk even when no specific rule was written for it, because it has learned from a corpus that includes risk analysis.

**Summarise and contextualise.** LLMs can explain why a finding matters, what the implication is, and what an appropriate remediation would be.

What AI models struggle with

LLMs are not reliable for the tasks that rules engines excel at.

**Deterministic correctness.** LLMs can make arithmetic errors. They can miss a cross-reference inconsistency that a rules engine would catch immediately. They are probabilistic, not deterministic, and their accuracy on structural checks is lower than a rules engine's.

**Explainability.** An LLM finding is harder to inspect than a rules-engine finding. "The model flagged this as a contradiction" is less actionable than "these two specific sentences contain conflicting claims because sentence A uses 'increased' and sentence B uses 'decreased' for the same metric."

**Hallucination.** LLMs occasionally generate findings that have no basis in the document. Rules engines do not hallucinate.

**Cost and speed at scale.** LLM inference is more expensive and slower than rule execution. For large document sets, the economics favour rules where rules are adequate.

The hybrid approach

The strongest document analysis systems use both:

• Rules engines for structural, numerical, and pattern checks where determinism and speed matter
• AI models for semantic, contextual, and quality checks where meaning matters

The rules engine catches what it is designed to catch with complete reliability. The AI model catches what the rules engine cannot — but its findings require a higher degree of human validation.

This is not a compromise — it is a recognition that the two approaches have complementary strengths. A document that passes both the rules engine and the AI review has been examined from two fundamentally different perspectives, which is more robust than either alone.