Question and Answer Lifecycle

This page traces the complete journey of a question and its answer from the moment a source artifact arrives at the API, through every governance gate, to its eventual state in the registry and downstream surfaces. Trend analysis integration points are called out explicitly at each stage where topic signals are extracted or enriched. Transcripts remain the main operational example.

---

1. End-to-End Journey

flowchart TD
    START([Source artifact arrives\nPOST /v1/artifacts])

    subgraph intake["① INTAKE"]
        direction TB
        SEG[TranscriptSegmenter\nSplit into segments\nVTT · SRT · TXT · MD · HTML · PDF · JSON]
        PII[PrivacyTagger\nNLTK NER + Regex\nPII_DETECTED · REDACTED · CLEAR]
        LB{Legal basis\nNOT_ASSESSED?}
        BLOCK([Blocked — artifact held])
        EXT[CandidateExtractor\nDetect questions and answers\nFlow A or Flow B]
        TREND1[🔵 TREND POINT 1\nTranscriptLineNormalizer\nSplit segments → utterance lines\nInherit speaker · offset · privacy]
        TREND2[🔵 TREND POINT 2\nMentionExtractor\nNLTK NP chunking\nPerson masking applied]
        TREND3[🔵 TREND POINT 3\nLineSentimentScorer\nVADER compound · pos · neu · neg\nper utterance line]
        TREND4[🔵 TREND POINT 4\nQuestionMentionLinker\nLink questions to nearby mentions\nProximity window ± N lines]
        TREND5[🔵 TREND POINT 5\nCanonicalizationService\nFrequency gate → similarity match\nAuto-resolve or admin queue]
    end

    subgraph classif["② CLASSIFICATION"]
        direction TB
        HORN[HornTypeClassifier\n9 information types]
        GRAM[GrammarExtractor\nSubject · Predicate · Object · Scope]
        SUB[SubtypeClassifier\nFACT subtypes × 6]
        CR[ClassificationRecord\nconfidence · requires_review]
    end

    subgraph flowb["② FLOW B ONLY — Generation"]
        direction TB
        KAD{Known answer\nWeaviate sim ≥ 0.92?}
        BYPASS([Return existing\napproved answer])
        RAG[RAGContextBuilder\nConfluence · Jira · Registry\n4-tier ranking · token limit]
        CB[CircuitBreaker\nLLM failure guard]
        GEN[OpenAI LLM\nGenerate candidate answer]
        F2{F2 score\n≥ 0.78?}
        ESC1([ESCALATED\nInsufficient RAG sources])
        GC[GeneratedCandidate\ntext · citations · tokens]
    end

    subgraph valid["③ VALIDATION"]
        direction TB
        VQ[Validation Queue\nPENDING]
        CLAIM[Validator claims\nIN_REVIEW]
        AUTH{Authority\ncheck passes?}
        DEC{Validator decision}
        APR[APPROVED]
        REV[REVISED\nCorrected text]
        REJ([REJECTED\nAudit record only])
        ESC2([ESCALATED\nSpecialist queue])
        RECLASSIFY[Re-classify revised text]
    end

    subgraph registry["④ REGISTRY"]
        direction TB
        PROV{Provenance\ncheck passes?}
        PE([ProvenanceError\nBlocked])
        AA[(ApprovedAnswer\nImmutable record)]
        VER[Version chain\nSupersedes prior versions]
        CONF[ConflictDetector\nGrammar overlap check]
        CFLAG[ConflictFlag\nif subject+predicate+scope conflict]
    end

    subgraph surfaces["⑤ DOWNSTREAM SURFACES"]
        direction LR
        RD[Readouts\nCustomer-scoped PDF · MD]
        RT[Retrieval\nWeaviate semantic search]
        GR[Graph\nNeo4j CONFLICTS_WITH\nSUPERSEDES · ABOUT]
        DB[Dashboard\nMetrics · Prometheus]
        TREND6[🔵 TREND POINT 6\nTrendReportService\nJSON API · CSV export\nDashboard trend panels]
        TGP[🔵 TREND POINT 7\nTrendGraphProjector\nLine · Mention · Entity\nTaxonomy · Speaker nodes]
    end

    START --> SEG --> PII --> LB
    LB -->|NOT_ASSESSED| BLOCK
    LB -->|Legal basis OK| EXT
    EXT --> TREND1 --> TREND2 --> TREND3
    TREND2 --> TREND4
    TREND3 --> TREND4
    TREND4 --> TREND5

    EXT -->|Flow A| HORN
    EXT -->|Flow B| KAD
    KAD -->|Yes| BYPASS
    KAD -->|No| RAG --> CB
    CB -->|< 2 sources| ESC1
    CB -->|OK| GEN --> F2
    F2 -->|fail| ESC1
    F2 -->|pass| GC --> HORN

    HORN --> GRAM --> SUB --> CR --> VQ
    VQ --> CLAIM --> AUTH
    AUTH -->|No authority| ESC2
    AUTH -->|OK| DEC
    DEC -->|approved| APR --> PROV
    DEC -->|revised| REV --> RECLASSIFY --> HORN
    DEC -->|rejected| REJ
    DEC -->|escalated| ESC2

    PROV -->|missing fields| PE
    PROV -->|OK| AA
    AA --> VER
    AA --> CONF --> CFLAG
    AA --> RD & RT & GR & DB
    AA --> TREND6
    TREND5 --> TREND6
    TREND5 --> TGP
    AA --> TGP

---

2. Validation State Machine

stateDiagram-v2
    [*] --> PENDING : Assignment created

    PENDING --> IN_REVIEW : claim (validator)\nPENDING → IN_REVIEW

    IN_REVIEW --> APPROVED : decide(approved)\nAuthority check passes
    IN_REVIEW --> REVISED : decide(revised)\nCorrected text supplied
    IN_REVIEW --> REJECTED : decide(rejected)
    IN_REVIEW --> ESCALATED : decide(escalated)\nor authority check fails\nor OPEN_ISSUE type

    REVISED --> IN_REVIEW : Re-classified, re-queued
    ESCALATED --> IN_REVIEW : Governance lead re-assigns

    APPROVED --> SUPERSEDED : New version approved
    APPROVED --> DISPUTED : ConflictFlag raised + reviewed
    APPROVED --> RETIRED : Manually retired

    DISPUTED --> IN_REVIEW : Dispute triggers re-review

    REJECTED --> [*] : Terminal — audit only
    SUPERSEDED --> [*] : Terminal — version chain preserved
    RETIRED --> [*] : Terminal

    note right of APPROVED
        Registry entry created
        Downstream surfaces updated
        Trend graph synced
    end note

    note right of ESCALATED
        OPEN_ISSUE cannot be approved
        Governance lead must resolve
    end note

---

3. Flow B Work Item State Machine

stateDiagram-v2
    [*] --> NEW : POST /v1/generation/work-items

    NEW --> GENERATING : AGWMP task dispatched to Celery
    NEW --> ESCALATED : Known-answer bypass OR < 2 RAG sources

    GENERATING --> PENDING_VALIDATION : F2 score ≥ 0.78
    GENERATING --> ESCALATED : F2 score < 0.78\nor LLM failure\nor circuit open

    PENDING_VALIDATION --> IN_CLASSIFICATION : ClassificationService picks up\nGenerated candidate classified

    IN_CLASSIFICATION --> READY_FOR_REVIEW : ClassificationRecord created

    READY_FOR_REVIEW --> VALIDATED : Reviewer approves generated answer
    READY_FOR_REVIEW --> NEEDS_REWORK : Reviewer rejects quality
    READY_FOR_REVIEW --> ESCALATED : Authority gap

    VALIDATED --> CLOSED : Answer enters registry

    NEEDS_REWORK --> GENERATING : Rework triggers new generation attempt

    ESCALATED --> CLOSED : Governance lead resolves

---

4. Intake API Sequence

sequenceDiagram
    participant C as API Client
    participant API as FastAPI /v1/artifacts
    participant IS as IntakeService
    participant SEG as TranscriptSegmenter
    participant TAG as PrivacyTagger
    participant EXT as CandidateExtractor
    participant LNZ as LineNormalizer
    participant MX as MentionExtractor
    participant SS as SentimentScorer
    participant QML as QuestionMentionLinker
    participant CAN as CanonicalizationService
    participant DB as PostgreSQL

    C->>API: POST /v1/artifacts\n{source_type, legal_basis, raw_content}
    API->>IS: ingest(artifact)
    IS->>DB: artifact_exists_by_ref() — idempotency check
    DB-->>IS: not found
    IS->>SEG: segment(artifact)
    SEG-->>IS: [TranscriptSegment × N]
    IS->>TAG: tag_segments(segments)
    TAG-->>IS: [PrivacyTag × N]
    IS->>EXT: extract(segments)
    EXT-->>IS: [CandidateQuestion × Q, CandidateAnswer × A]
    IS->>LNZ: normalize(segments)
    LNZ-->>IS: [TranscriptLine × L]
    IS->>MX: extract(lines, pii_spans)
    MX-->>IS: [RawMention × M]
    IS->>SS: score(lines)
    SS-->>IS: [LineSentiment × L]
    IS->>QML: link(questions, mentions, lines)
    QML-->>IS: [QuestionMentionLink × K]
    IS->>CAN: process(mentions)
    CAN-->>IS: clusters, resolutions, new_entities
    IS->>DB: save artifact, segments, questions,\nlines, mentions, sentiments,\nlinks, clusters, resolutions
    DB-->>IS: OK
    IS-->>API: IntakeResult {counts}
    API-->>C: 201 {artifact_id, segment_count,\nquestion_count, line_count,\nmention_count, pending_cluster_count, …}

---

5. Validation Decision Sequence

sequenceDiagram
    participant V as Validator
    participant API as FastAPI /v1/validation
    participant VS as ValidationService
    participant SM as StateMachine
    participant AUTH as AuthorityChecker
    participant CD as ConflictDetector
    participant RS as RegistryService
    participant DB as PostgreSQL
    participant N4 as Neo4j (async)

    V->>API: POST /assignments/{id}/claim
    API->>VS: claim_assignment(id, actor)
    VS->>SM: transition(PENDING → IN_REVIEW)
    SM-->>VS: OK
    VS->>DB: update assignment state
    API-->>V: 200 {assignment, current_state: IN_REVIEW}

    V->>API: POST /assignments/{id}/decide\n{decision: approved, rationale}
    API->>VS: decide(id, APPROVED, actor)
    VS->>AUTH: check_authority(actor_role, horn_type)
    AUTH-->>VS: authorized
    VS->>SM: transition(IN_REVIEW → APPROVED)
    VS->>CD: detect_conflicts(candidate_id)
    CD-->>VS: no conflicts
    VS->>RS: publish_to_registry(candidate_id)
    RS->>DB: INSERT approved_answers\nINSERT evidence_links\nINSERT version_chains
    DB-->>RS: OK
    RS-->>VS: ApprovedAnswer
    VS->>N4: (async) upsert_answer()\nlink_question_to_answer()\nlink_answer_to_concept()
    API-->>V: 200 {answer_id, status: approved}

---

6. Trend Analysis Integration Points — Summary

#	Stage	Component	What is captured
1	Intake — after segmentation	TranscriptLineNormalizer	Utterance-level lines with speaker, offset, and source provenance
2	Intake — after privacy tagging	MentionExtractor	NLTK noun-phrase spans; person mentions masked to [PERSON]
3	Intake — parallel with extraction	LineSentimentScorer	VADER compound, pos, neu, neg per utterance line
4	Intake — after extraction	QuestionMentionLinker	Question-to-mention proximity links within ± N lines
5	Intake — after mention scoring	CanonicalizationService	Frequency-gated clusters; auto-resolved entities; admin queue
6	Post-intake — on demand	TrendReportService	JSON API and CSV aggregations; dashboard trend panels
7	Post-intake — async	TrendGraphProjector	Neo4j nodes: TranscriptLine, RawMention, CanonicalEntity, TaxonomyTerm, Speaker, TimeBucket

Trend data is available immediately after intake — no validation or registry publication is required. The trend graph and trend report both operate on raw mention + sentiment data that is persisted during ingestion.