System Surfaces and Data Stores

Raw to Knowledge is built on three distinct data stores, one async worker layer, and a synchronous API layer. Understanding where data lives and which store is authoritative matters when you are troubleshooting an inconsistency, planning an integration, or debugging a slow query.

The four data stores

PostgreSQL — primary source of truth

PostgreSQL holds the canonical records for every entity in the pipeline. If you need to know the authoritative state of anything in Raw to Knowledge, query PostgreSQL.

Table / Entity	What it holds
`SourceArtifact`	Ingested source artifacts: format, privacy class, legal basis, ingest timestamp
`TranscriptSegment`	Individual segments: speaker, timestamp range, text, artifact reference
`CandidateAnswer`	Extracted Q/A pairs: question, answer text (if Flow A), Horn type assignment, flow type, status
`ClassificationRecord`	Classification output: Horn type, confidence, assertion grammar fields, requires_review flag
`ValidationRecord`	Validator decisions: decision type, validator role, decision timestamp, revision text
`ApprovedAnswer`	Registry entries: answer text, Horn type, version, supersedes_id, evidence_links, review_due, customer_scope
`WorkItem`	Flow B generation requests: question reference, status, assigned worker
`GeneratedCandidate`	AGWMP output: generated answer text, RAG sources, F2 score, model parameters
`ConflictFlag`	Grammar conflicts: subject, predicate, scope, conflicting answer IDs, resolution status

PostgreSQL is write-once for ApprovedAnswer records (immutable after insert). All other tables support updates through their defined state transitions.

Redis — session state and task broker

Redis serves two distinct roles in Raw to Knowledge:

Celery broker and backend. All async task messages pass through Redis. Celery workers poll Redis for new tasks in the raw_to_knowledge_generation and raw_to_knowledge_graph queues. Task results and state (PENDING, STARTED, SUCCESS, FAILURE) are stored in Redis until consumed.

Validation queue. Active validation assignments — which validator has claimed which candidate — are tracked in Redis as session-scoped locks. This prevents two validators from simultaneously reviewing the same candidate. The validation queue is ordered FIFO per Horn type.

Redis does not hold durable business data. If Redis is cleared or restarted, in-flight task state is lost, but all committed records in PostgreSQL are intact. Active validation claims may need to be re-established.

Weaviate — semantic vector index

Weaviate holds vector embeddings of approved answers, indexed for semantic similarity search. It is a derived store: its contents are populated by a sync process that reads from PostgreSQL. Weaviate is not written to by the core pipeline — it is updated when approved answers are added, superseded, or retired.

Weaviate serves two functions:

Known-answer detection. When a Flow B work item is submitted, the AGWMP pipeline queries Weaviate for existing answers that match the question at similarity ≥ 0.92. If a match is found, the existing approved answer is returned and generation is bypassed.
Semantic retrieval. The GET /v1/retrieval/search endpoint queries Weaviate to find approved answers that are semantically relevant to a query string. Results fall back to the PostgreSQL FTS index if Weaviate returns no results above threshold.

Important: Weaviate is a derived projection. If the sync job has fallen behind, Weaviate may not reflect recently approved or superseded answers. If you receive a retrieval result that appears inconsistent with a known recent approval or supersession, query the PostgreSQL registry directly via GET /v1/registry/answers to confirm current state.

Neo4j — property graph

Neo4j holds a property graph projection of approved answers and their relationships. Like Weaviate, it is a derived store updated by the raw_to_knowledge_graph Celery queue.

The graph encodes six relationship types:

Relationship	Meaning
`CONFLICTS_WITH`	Two answers assert incompatible claims (same subject+predicate+scope, different object)
`SUPERSEDES`	A newer answer version replaces an older one
`ANSWERED_BY`	A candidate question is answered by a specific approved answer
`SUPPORTED_BY`	An approved answer is supported by a piece of evidence (Confluence page, Jira issue)
`ABOUT`	An approved answer is about a concept or entity node

Neo4j is used for:

Contradiction detection. The ContradictionDetector traverses CONFLICTS_WITH edges to surface grammar-level conflicts during validation and for the governance lead's quality review.
Supersession chain traversal. Walking SUPERSEDES edges shows the full version history of an answer.
Concept-based retrieval. Querying answers ABOUT a particular concept or entity, enabling graph-structured knowledge navigation.

Important: Neo4j is a derived projection. If the raw_to_knowledge_graph worker queue is backed up, the graph may lag PostgreSQL by minutes to hours. Contradiction flags seen in Neo4j are authoritative only as of the last sync. Use PostgreSQL ConflictFlag records for authoritative conflict state.

The Celery worker layer

Celery workers are separate processes from the API service. The API does not perform long-running work — it enqueues tasks and returns task IDs. Workers consume from two queues:

Queue	Tasks
`raw_to_knowledge_generation`	AGWMP generation tasks: RAG context assembly, LLM generation, F2 scoring, GeneratedCandidate creation
`raw_to_knowledge_graph`	Neo4j sync tasks: answer projection, relationship creation, ContradictionDetector runs

Workers are independently scalable. High-volume deployments can run multiple raw_to_knowledge_generation workers in parallel. The raw_to_knowledge_graph worker should typically be a single worker per deployment to avoid concurrent graph write conflicts (configurable).

The CircuitBreaker on the generation queue protects against downstream LLM or RAG source failures. If the LLM service returns errors above a threshold rate, the circuit opens and generation tasks fail fast with a clear error rather than queuing indefinitely.

The API layer

The FastAPI service exposes 10 route prefixes. All routes are synchronous from the client's perspective — long-running operations return a task ID for polling.

Prefix	Purpose
`/v1/intake`	Ingest source artifacts; initiate segmentation and extraction
`/v1/classification`	Query classification records; re-trigger classification
`/v1/validation`	Manage the validation queue; submit decisions
`/v1/registry`	Query approved answers; manage lifecycle (retire, dispute)
`/v1/retrieval`	Semantic and FTS search; Markdown and JSON-LD export
`/v1/generation`	Submit and query Flow B work items
`/v1/graph`	Query contradictions; traverse version chains
`/v1/readouts`	Assemble customer-scoped pre-meeting briefings
`/v1/dashboard`	Aggregate pipeline metrics
`/v1/health`	Liveness and readiness checks for all services

Prometheus metrics for all services are available at GET /metrics.

Architecture diagram

flowchart TD
    CLIENT([API Client\nUser / Integration])
    API[FastAPI Service\n10 route prefixes\nv0.8.0]

    subgraph stores["Data Stores"]
        PG[(PostgreSQL\nPrimary source of truth\nAll canonical records)]
        RD[(Redis\nCelery broker\nValidation queue)]
        WV[(Weaviate\nVector index\nDerived from PostgreSQL)]
        N4[(Neo4j\nProperty graph\nDerived from PostgreSQL)]
    end

    subgraph workers["Celery Workers"]
        WG[raw_to_knowledge_generation\nAGWMP · RAG · F2 scorer]
        WGR[raw_to_knowledge_graph\nGraph sync · Contradiction]
    end

    EXT[External Sources\nConfluence · Jira\nLLM service]

    CLIENT -->|HTTP| API
    API -->|read/write| PG
    API -->|enqueue tasks| RD
    API -->|search| WV
    API -->|graph query| N4

    RD -->|task dispatch| WG
    RD -->|task dispatch| WGR

    WG -->|write candidates| PG
    WG -->|RAG queries| EXT
    WGR -->|sync approved answers| WV
    WGR -->|sync graph| N4

    PG -.->|authoritative| WV
    PG -.->|authoritative| N4

Source of truth resolution

When stores disagree, use this precedence:

PostgreSQL is always the authoritative source for answer status, validation state, and provenance.
Weaviate reflects PostgreSQL as of the last sync. Query /v1/registry/answers if you suspect a stale retrieval result.
Neo4j reflects PostgreSQL as of the last graph sync. Query /v1/registry/answers/{id} and /v1/validation/records if you suspect a stale contradiction or supersession in the graph.
Redis holds transient state only. No business data should be recovered from Redis after a restart.