Skip to content

How Raw to Knowledge Builds Governed Knowledge

Raw to Knowledge transforms raw information artifacts into structured, validated, reusable knowledge through a multi-stage pipeline. Each stage adds a distinct layer of governance — privacy protection, semantic typing, human review, or provenance tracking. Understanding the full lifecycle helps every role interact with the system purposefully.

The complete lifecycle

Stage 1: Intake

Everything starts with a source artifact. Raw to Knowledge accepts VTT, SRT, plain text, Markdown, HTML, PDF-derived text, and structured JSON. Transcripts remain the primary use case, but the intake boundary is designed for any text-bearing raw material that can carry provenance.

The intake pipeline performs three operations on every file:

Segmentation. The raw source is split into TranscriptSegment records. Each segment carries the available offsets, speaker identifiers when present, and the segment text. Segmentation is the foundation for everything that follows — all downstream analysis operates on segments, not the full source file.

Privacy tagging. NLTK named-entity recognition and regex pattern matching scan each segment for personal information. The pipeline detects PERSON names, EMAIL addresses, US phone numbers, SSNs, and credit card numbers. Every SourceArtifact record receives a privacy_class and a declared legal_basis (drawn from GDPR Article 6 categories: CONSENT, LEGITIMATE_INTEREST, CONTRACT, LEGAL_OBLIGATION, or NOT_ASSESSED). If legal_basis is NOT_ASSESSED, the artifact is held and does not advance to extraction.

Flow A/B detection. The extraction stage identifies question-answer pairs. When both a question and an answer are present in the source material, the candidate is marked Flow A — the answer exists in the source. When a question was raised but not answered in the source, the candidate is marked Flow B — an answer must be generated from internal knowledge sources. See Flow A vs Flow B for the full distinction.

Stage 2: Classification

Each candidate question-answer pair enters the classification engine. The classifier assigns a Horn information type — one of nine categories defined by Robert Horn's Information Mapping methodology (FACT, CONCEPT, PROCEDURE, PROCESS, PRINCIPLE, REFERENCE, TROUBLESHOOTING, RECOMMENDATION, OPEN_ISSUE).

For FACT-typed candidates, the classifier also assigns an assertion subtype (EXISTENCE, CAPABILITY, CONFIGURATION, STATE, CONSTRAINT, or TEMPORAL) and populates the assertion grammar: subject, predicate, object, and scope. This structured representation is what makes FACT answers searchable, comparable, and contradiction-detectable in the graph layer.

A ClassificationRecord is created for each candidate, storing the assigned type, confidence score, and grammar fields. If confidence falls below 0.75, or if the assigned type is OPEN_ISSUE, requires_review is set to True — guaranteeing human attention regardless of confidence.

Stage 3: Validation

Validated candidates travel through a 9-state machine. The candidate starts at PENDING, moves to IN_REVIEW when a validator claims it, and then receives one of four decisions: APPROVED, REVISED, REJECTED, or ESCALATED.

The role of the validator matters: the KGS authority table maps each Horn type to the roles permitted to approve it. A validator without authority for a given type can still review a candidate and escalate it, but cannot approve it.

REVISED candidates return a corrected text and re-enter the classification pipeline before coming back to the queue. REJECTED candidates stay in the candidates table for audit but never enter the registry. ESCALATED candidates go to a specialist queue managed by the governance lead.

OPEN_ISSUE candidates can never be approved. The system enforces this constraint at the validation layer.

See Validation Authority and Lifecycle for the complete state diagram and decision definitions.

Stage 4: Registry

When a candidate is APPROVED, a new ApprovedAnswer record is created in PostgreSQL. This record is immutable — the approved answer text cannot be edited in place. If new information supersedes an approved answer, a new version is created and the previous record's status changes to SUPERSEDED. The version chain is preserved and queryable.

Every registry entry carries: validation_record_id, owner_role, evidence_links, and review_due. Missing any of these fields raises a ProvenanceError and blocks registry insertion. These requirements are not optional governance preferences — they are enforced by the application layer.

Stage 5: Downstream surfaces

Once an answer enters the registry, it becomes available across four output surfaces:

Readouts surface approved answers scoped to a specific customer context. Readouts are assembled on demand, filtering by customer scope, Horn type, and recency. Answers within 30 days of their review-due date are flagged.

Retrieval exposes a semantic search endpoint (GET /v1/retrieval/search) backed by the Weaviate vector index. When a query matches an indexed answer at similarity ≥ 0.92, the existing approved answer is returned directly. Results can be exported as Markdown or JSON-LD.

Graph analysis runs on the Neo4j projection of approved answers. The ContradictionDetector identifies pairs where the same subject+predicate+scope pair has conflicting object values across different approved answers. Contradiction flags appear in readouts and are queryable via the graph API. The supersession chain is also traversable in the graph.

Dashboard aggregates pipeline metrics: queue depth by Horn type, classification confidence distribution, answer aging, validation throughput, and contradiction counts. Prometheus metrics are available at GET /metrics.

Full lifecycle diagram

flowchart TD
    TS([Source Artifact\nVTT · SRT · TXT · MD · HTML · PDF · JSON])

    subgraph intake["Stage 1: Intake"]
        SEG[Segmentation\nSpeaker · Timestamp · Text]
        PII[Privacy Tagging\nNLTK NER · Regex · Legal Basis]
        FAB{Flow A or B?}
    end

    subgraph classif["Stage 2: Classification"]
        CLAS[Horn Type Assignment\n9 Types · Confidence Score]
        GRAM[Assertion Grammar\nSubject · Predicate · Object · Scope]
        CR[ClassificationRecord]
    end

    subgraph valid["Stage 3: Validation"]
        VQ[Validation Queue\nPending → In Review]
        DEC{Validator Decision}
        APR[APPROVED]
        REV[REVISED → Re-classify]
        REJ[REJECTED\nAudit only]
        ESC[ESCALATED\nSpecialist queue]
    end

    subgraph registry["Stage 4: Registry"]
        AA[(ApprovedAnswer\nImmutable Record)]
        VC[Version Chain\nSupersedes ID]
    end

    subgraph surfaces["Stage 5: Surfaces"]
        RD[Readouts\nCustomer-scoped]
        RT[Retrieval\nWeaviate Search]
        GR[Graph\nNeo4j · Contradictions]
        DB[Dashboard\nMetrics · Quality]
    end

    subgraph flowB["Flow B: Generation"]
        FBQ[Open Question\nNo answer in source]
        AGWMP[AGWMP Pipeline\nRAG · Weaviate · CircuitBreaker]
        GC[GeneratedCandidate]
        KAD{Known answer\nsim ≥ 0.92?}
        BYPASS[Return existing\napproved answer]
    end

    TS --> SEG --> PII --> FAB
    FAB -->|Flow A: answer present| CLAS
    FAB -->|Flow B: no answer| FBQ

    FBQ --> KAD
    KAD -->|Yes| BYPASS
    KAD -->|No| AGWMP --> GC --> CLAS

    CLAS --> GRAM --> CR --> VQ
    VQ --> DEC
    DEC --> APR --> AA
    DEC --> REV --> CLAS
    DEC --> REJ
    DEC --> ESC

    AA --> VC
    AA --> RD
    AA --> RT
    AA --> GR
    AA --> DB

Governance value added per stage

Stage Governance contribution
Intake Privacy protection; legal basis declaration; source artifact traceability
Classification Semantic typing; structured assertion grammar; confidence-gated review flags
Validation Human approval by authorized role; decision audit trail; OPEN_ISSUE escalation
Registry Immutability; version chain; provenance enforcement (evidence links, owner, review-due)
Surfaces Scope isolation (readouts); semantic traceability (JSON-LD exports); contradiction visibility (graph)

Note: No stage is optional. An artifact that fails privacy assessment cannot proceed. An answer that fails provenance checks cannot enter the registry. The pipeline is designed so that governance is not a review step at the end — it is embedded at every stage.