Skip to content

Raw Information to Knowledge System

North star: The unit of value is not the transcript, the note, the page, the PDF, the question, the answer, the ticket, or the document. The unit of value is the validated, typed, provenance-bearing knowledge object.

What This System Does

Raw to Knowledge transforms raw information sources — especially transcripts, but also notes, tickets, Markdown, HTML, and PDF-derived text — into governed, reusable knowledge objects. Every answer is classified by type, validated by a human authority, and stored with full provenance: where it came from, who confirmed it, and when it expires.

This is not a summarizer for any single source type. It is an answer governance system.

Milestone 1 — Intake (Current)

The first milestone proves one thing: the system can extract reusable question/answer candidates from raw source artifacts with source traceability.

Deliverables:

  • Raw source ingestion (VTT, SRT, plain text, Markdown, HTML, PDF-derived text, JSON)
  • Segmentation into addressable TranscriptSegment objects
  • Flow A/B detection (answer present vs. question only)
  • Candidate extraction using interrogative pattern heuristics
  • Privacy tagging for PII before content leaves the CII boundary
  • PostgreSQL persistence with Alembic migrations
  • Review list API for human inspection

Quick Start

# Copy and configure environment
cp .env.example .env

# Start Postgres
make docker-up

# Apply migrations
make migrate

# Run the intake service locally
uvicorn raw_to_knowledge.main:app --reload

# Run unit tests
make test-unit

Architecture

The system is organized into four layers:

Layer Package Responsibility
Domain raw_to_knowledge.domain Models, enums, protocols, pure domain logic
Application raw_to_knowledge.application Use case orchestration (IntakeService)
Infrastructure raw_to_knowledge.infrastructure PostgreSQL, Neo4j, Weaviate, LLM adapters
API raw_to_knowledge.api FastAPI routers and request/response schemas

All dependencies point inward. The domain layer has zero infrastructure dependencies.