Milestone 2 — Classify Extracted Answers

Status: Complete
Completed: 2026-06-07

Goal

Prove that answers are not all the same kind of thing, and that classification improves validation routing.

Stakeholder story: "We can tell the difference between a fact, a procedure, a recommendation, and an unresolved issue."

What Was Built

Horn Type Classifier

A rule-based classifier (RuleBasedHornTypeClassifier) scores each answer text against nine Robert Horn information types using keyword pattern tables and NLTK POS-tag heuristics. The HybridHornTypeClassifier wraps it and provides a BERT fine-tune integration point for future milestones.

Supported types:

Type	Governance implication
fact	Requires assertion subtype and grammar; highest accountability
concept	Definition or explanation; no grammar required
procedure	Step-by-step; requires validated, tested steps before approval
process	System workflow description
principle	Normative guidance; must not be treated as a factual claim
reference	List or lookup table
troubleshooting	Problem–cause–solution
recommendation	Context-specific advice
open_issue	Unresolved or TBD; always requires review

Assertion Subtype Classifier

When the primary type is fact, a FactSubtypeClassifier runs all six MVP subtypes and returns the highest-scoring one:

Subtype	Typical signals
existence	"is available", "there is/are", "out of the box"
capability	"supports", "can", "allows", "enables"
configuration	"defaults to", "set to", "configured", "settings"
state	"is currently", "triggered by", "status"
constraint	"requires", "must", "maximum", "prevents"
temporal	"as of", "since", year patterns, "version X.Y"

Assertion Grammar Extractor

For fact candidates the AssertionGrammarExtractor populates structured fields:

• Subject — the entity the assertion is about
• Predicate — the relationship or property asserted
• Object — the value or entity on the receiving end
• Scope — boundary within which the assertion holds (detected via trigger words: "when", "if", "within")
• Temporal boundary — time-scoped validity (detected via year patterns and temporal triggers)

Fields the extractor cannot determine are recorded in unknown_fields for validator attention.

ClassificationService

Orchestrates the three-stage pipeline per candidate:

1. Horn type classifier → primary type + confidence + alternatives
2. (FACT only) Subtype classifier → MVPAssertionSubtype
3. (FACT only) Grammar extractor → AssertionGrammar
4. Sets requires_review = True when confidence < threshold or type == OPEN_ISSUE
5. Persists ClassificationRecord to Postgres
6. Emits structured log event

ClassificationRecord

The full, immutable audit record of a classification decision:

ClassificationRecord(
    record_id="...",
    candidate_id="...",
    answer_text="The platform supports localized help sites.",
    primary_horn_type=HornType.FACT,
    primary_confidence=0.81,
    assertion_subtype=MVPAssertionSubtype.CAPABILITY,
    assertion_grammar=AssertionGrammar(
        subject="platform",
        predicate="supports",
        object_value="localized portal builds",
        scope="locale-specific branches configured",
    ),
    requires_review=False,
    model_version="rule-based-v1.0",
)

API Endpoints

Method	Path	Description
POST	/v1/classify	Classify a single candidate answer
POST	/v1/classify/batch	Classify up to 100 candidates
GET	/v1/classifications/{candidate_id}	Get latest classification for a candidate

Quality Gates

Gate	Criterion	Result
Golden set precision	≥ 85% primary Horn type accuracy	See evaluate.py
Golden set recall	≥ 82%	See evaluate.py
Assertion subtype accuracy	≥ 80% on Fact candidates	See evaluate.py
Grammar completeness	≥ 80% of Facts have subject + predicate	See evaluate.py
Reproducibility	Same input = same output	Verified by unit tests
Derived flag	OPEN_ISSUE always requires review	Enforced in service

Run the evaluation: python tests/golden/evaluate.py

Running the Evaluation

# Regex fallback (no NLTK data needed)
python tests/golden/evaluate.py

# With NLTK POS heuristics
python tests/golden/evaluate.py --use-nltk