Architecture Overview

Raw to Knowledge is a domain-driven, layered FastAPI service that converts raw conversation transcripts into structured, validated, governed knowledge objects. It enforces privacy protection, semantic typing, human review authority, and full provenance at every stage of the pipeline.

---

1. Four-Layer Architecture

Every request in Raw to Knowledge travels through four layers. Dependencies point inward only — the domain layer has zero knowledge of infrastructure.

flowchart TD
    subgraph API["① API Layer  ·  FastAPI routers"]
        R1["/v1/artifacts\nIntake"]
        R1C["/v1/connectors\nSource Connectors"]
        R2["/v1/classification\nClassification"]
        R3["/v1/validation\nValidation"]
        R4["/v1/registry\nRegistry"]
        R5["/v1/generation\nGeneration"]
        R6["/v1/trends\nTrend Path"]
        R7["/v1/graph · /v1/retrieval\n/v1/readouts · /v1/dashboard"]
    end

    subgraph APP["② Application Layer  ·  Use-case orchestrators"]
        IS[IntakeService]
        CNS[ConnectorService\nConnectorRegistry]
        CS[ClassificationService]
        VS[ValidationService]
        RS[RegistryService]
        GS[GenerationService / AGWMPService]
        TS[TrendReportService\nCanonicalizationService]
        GRS[GraphService]
        RTS[RetrievalService]
    end

    subgraph DOM["③ Domain Layer  ·  Pure business logic"]
        subgraph models["Domain Models (frozen Pydantic)"]
            M1[SourceArtifact\nTranscriptSegment\nCandidateQuestion]
            M2[ClassificationRecord\nAssertionGrammar]
            M3[ValidationRecord\nApprovedAnswer]
            M4[TranscriptLine\nRawMention\nLineSentiment\nCanonicalEntity]
        end
        subgraph services["Domain Services (no I/O)"]
            DS1[TranscriptSegmenter\nPrivacyTagger\nCandidateExtractor]
            DS2[Classifier\nSubtypeClassifier\nGrammarExtractor]
            DS3[StateMachine\nConflictDetector\nContradictionDetector]
            DS4[LineNormalizer\nMentionExtractor\nSentimentScorer\nCanonicalizationSvc]
        end
        PROTO[Repository Protocols\nGraph Protocol\nLLM Protocol\nSourceConnector Protocol]
    end

    subgraph INFRA["④ Infrastructure Layer  ·  Adapters"]
        PG[(PostgreSQL\nAlembic ORM)]
        REDIS[(Redis\nCelery)]
        WV[(Weaviate\nVector index)]
        N4[(Neo4j\nProperty graph)]
        LLM[OpenAI Adapter]
        CF[Confluence Adapter]
        JR[Jira Adapter]
        TGP[TrendGraphProjector]
        FCN[FellowConnector\nFellow API v2]
        GCN[GongConnector\nGong API v2]
    end

    API --> APP --> DOM
    DOM --> PROTO --> INFRA
    R1C --> CNS
    CNS --> IS

Rule: Application services call domain services and repository protocols. Domain services call nothing — they receive data, return data. Infrastructure implementations are injected at startup via api/dependencies.py.

---

2. Component Map

graph LR
    subgraph connector_grp["Source Connectors"]
        CREG[ConnectorRegistry\nsource_system → connector]
        FCNC[FellowConnector\nmeeting notes]
        GCNC[GongConnector\ncall transcripts]
    end

    subgraph intake_grp["Intake"]
        SEG[TranscriptSegmenter\nVTT · SRT · TXT · JSON]
        TAG[PrivacyTagger\nNLTK NER + Regex]
        EXT[CandidateExtractor\nFlow A / Flow B]
        LNZ[TranscriptLineNormalizer\nUtterance-level lines]
    end

    subgraph trend_grp["Trend Path"]
        MX[MentionExtractor\nNLTK NP chunker]
        SS[LineSentimentScorer\nVADER]
        QML[QuestionMentionLinker\nProximity window]
        CAN[CanonicalizationService\nFrequency + Similarity]
        TRS[TrendReportService\nJSON · CSV aggregation]
        TGP2[TrendGraphProjector\nNeo4j write]
    end

    subgraph classif_grp["Classification"]
        CLS[HornTypeClassifier\n9 types]
        SUB[SubtypeClassifier\n6 FACT subtypes]
        GRM[GrammarExtractor\nS·P·O·Scope]
    end

    subgraph valid_grp["Validation"]
        SM[StateMachine\n9 states]
        AUTH[AuthorityChecker\nRole × HornType]
        CD[ConflictDetector\nGrammar overlap]
    end

    subgraph reg_grp["Registry"]
        RT[RegistryTransform\nProvenance enforcement]
        SC[SupersessionChain]
        RCO[RegistryConflictRecord]
    end

    subgraph gen_grp["Generation (Flow B)"]
        RAG[RAGContextBuilder\nToken-limited · 4-tier ranking]
        CB[CircuitBreaker\nLLM failure guard]
        F2[F2Scorer\nβ=2 recall-weighted]
        AGWMP[AGWMPService\nKnown-answer detection]
    end

    subgraph out_grp["Output Surfaces"]
        RDB[ReadoutBuilder]
        ANE[MarkdownExporter\nJSONLDExporter]
        RET[RetrievalService\nWeaviate + PG FTS]
        GRS2[GraphService\nContradiction · Supersession]
        DSH[DashboardService\nPrometheus metrics]
    end

    FCNC --> CREG
    GCNC --> CREG
    CREG --> SEG
    SEG --> TAG --> EXT
    EXT --> LNZ
    LNZ --> MX --> SS
    MX --> QML
    CAN --> TRS --> TGP2
    EXT --> CLS --> SUB --> GRM --> SM --> AUTH --> RT
    RT --> RDB & ANE & RET & GRS2 & DSH
    AGWMP --> RAG --> CB --> F2 --> CLS

---

3. Deployment Topology

graph TB
    subgraph client_grp["Clients"]
        UC([User / Browser])
        IC([API Integration\nCI/CD · Scripts])
    end

    subgraph service_grp["Raw to Knowledge Service Pod"]
        API2[FastAPI\nUvicorn\n:8000]
        CE1[Celery Worker\nraw_to_knowledge_generation\nconcurrency 4]
        CE2[Celery Worker\nraw_to_knowledge_graph\nconcurrency 2]
    end

    subgraph store_grp["Persistent Stores"]
        PG2[(PostgreSQL\n:5432\nSource of truth)]
        RD2[(Redis\n:6379\nBroker + Queue)]
        WV2[(Weaviate\n:8080\nVector index)]
        N42[(Neo4j\n:7687 Bolt\nProperty graph)]
    end

    subgraph ext_grp["External Services"]
        OAI[OpenAI API\nLLM generation]
        CON[Confluence\nRAG source]
        JRA[Jira\nWork items]
        FEL[Fellow API v2\nMeeting notes]
        GON[Gong API v2\nCall transcripts]
    end

    UC & IC -->|HTTPS| API2
    API2 -->|asyncpg| PG2
    API2 -->|async Redis| RD2
    API2 -->|HTTP| WV2
    API2 -->|Bolt async| N42
    RD2 -->|dispatch| CE1 & CE2
    CE1 -->|write| PG2
    CE1 -->|HTTPS| OAI & CON & JRA
    CE2 -->|sync projection| WV2 & N42
    API2 -->|Bearer HTTPS| FEL
    API2 -->|Basic HTTPS| GON

Scaling notes: - The FastAPI pod and Celery workers are independently scalable horizontally. - raw_to_knowledge_graph workers must run with concurrency ≤ 2 per deployment to avoid Neo4j write contention. - Redis must be reachable by both the API pod and all Celery workers.

---

4. Core Domain Class Diagram

classDiagram
    class SourceArtifact {
        +str artifact_id
        +str source_system
        +SourceType source_type
        +str source_ref
        +datetime received_at
        +LegalBasis legal_basis
        +str customer_scope
        +str processing_run_id
        +str raw_content
    }

    class TranscriptSegment {
        +str segment_id
        +str artifact_id
        +int sequence
        +str text
        +str start_offset
        +str end_offset
        +str speaker
        +PrivacyClass privacy_class
    }

    class CandidateQuestion {
        +str candidate_id
        +str question_text
        +float confidence
        +FlowType flow_type
        +str source_segment_id
        +str artifact_id
        +str processing_run_id
    }

    class ClassificationRecord {
        +str record_id
        +str candidate_id
        +str answer_text
        +HornType primary_horn_type
        +float primary_confidence
        +str assertion_subtype
        +AssertionGrammar assertion_grammar
        +bool requires_review
        +str model_version
    }

    class AssertionGrammar {
        +str subject
        +str predicate
        +str object_value
        +str scope
        +str condition
        +str temporal_boundary
        +float completeness_score
    }

    class ApprovedAnswer {
        +str answer_id
        +str candidate_id
        +HornType horn_type
        +str answer_text
        +str status
        +int version
        +str owner_role
        +str validation_record_id
        +datetime review_due
        +str supersedes_id
        +str customer_scope
    }

    class TranscriptLine {
        +str line_id
        +str artifact_id
        +str segment_id
        +int sequence
        +int source_index
        +str speaker
        +str text
    }

    class RawMention {
        +str mention_id
        +str line_id
        +str normalized_text
        +MentionType mention_type
        +int start_char
        +int end_char
        +str privacy_class
    }

    class LineSentiment {
        +str line_id
        +float compound
        +float pos
        +float neu
        +float neg
        +SentimentLabel sentiment_label
    }

    class CanonicalEntity {
        +str canonical_entity_id
        +str label
        +str normalized_label
        +CanonicalEntityStatus status
        +ResolutionMode resolution_mode
    }

    SourceArtifact "1" --> "many" TranscriptSegment : segments
    TranscriptSegment "1" --> "many" CandidateQuestion : extracts
    CandidateQuestion "1" --> "1" ClassificationRecord : classifies
    ClassificationRecord "1" --> "1" AssertionGrammar : grammar
    ClassificationRecord "1" --> "1" ApprovedAnswer : approves
    ApprovedAnswer --> ApprovedAnswer : supersedes
    TranscriptSegment "1" --> "many" TranscriptLine : lines
    TranscriptLine "1" --> "many" RawMention : mentions
    TranscriptLine "1" --> "1" LineSentiment : sentiment
    RawMention --> CanonicalEntity : resolves_to

---

5. Key Design Principles

Principle	Application
Immutability	All Pydantic domain models are frozen=True. ApprovedAnswer is write-once in the database.
Dependency Inversion	Application services depend on protocol interfaces. Adapters are injected at startup.
Privacy by Default	PrivacyTagger runs on every segment before anything leaves the CII boundary. NOT_ASSESSED blocks ingestion.
Provenance Enforcement	Registry insert requires validation_record_id, owner_role, evidence_links, and review_due.
No Opaque Automation	All auto-resolution decisions (canonicalization, known-answer detection) store their score, rule, and mode.
Deterministic Similarity	Canonicalization uses Jaccard token overlap, not opaque embeddings, for auditability.
Open/Closed Segmentation	Adding a transcript format requires only a new SegmentationStrategy — no changes to TranscriptSegmenter.
Open/Closed Connectors	Adding a new transcript source requires only a new SourceConnector implementor, settings, and registration — no changes to IntakeService, domain models, or existing routes.