Milestone 1 — Extract Trends and Questions from Transcripts

Status: Complete
Completed: 2026-06-07

Goal

Prove that meeting material contains reusable question/answer candidates that can be extracted with source traceability.

Stakeholder story: "We can identify the questions customers are actually asking."

Deliverables

Deliverable	Status
`intake-service` container with Docker Compose	✓
Transcript import (VTT, SRT, plain text, JSON)	✓
`TranscriptSegmenter` with Strategy pattern	✓
Flow A/B detector	✓
`CandidateExtractor` (NLTK interrogative patterns)	✓
`PrivacyTagger` (regex + NLTK NER)	✓
PostgreSQL schema + Alembic migrations	✓
Review list API (`GET /v1/candidates`)	✓
Unit tests (segmenter, extractor, privacy tagger)	✓
Integration tests (artifact → candidates roundtrip)	✓
mkdocs documentation	✓
`SourceConnector` protocol + `ConnectorRegistry`	✓
`FellowConnector` (Fellow API v2 meeting notes)	✓
`GongConnector` (Gong API v2 call transcripts)	✓
`ConnectorService` application orchestrator	✓
Connector REST API (`/v1/connectors`)	✓
Persisted connector sync state + cadence API	✓
Scheduled incremental connector sync task	✓
Connector unit tests (77 tests pass)	✓

Quality Gates

Gate	Criterion	Status
Unit test coverage	≥ 90% for domain layer	Enforced via pytest-cov
Integration test	Artifact → segments → candidates round-trip with Postgres	✓
Privacy tagging	Known PII patterns tagged in test fixtures	✓
Flow detection	20 labeled test segments correctly classified as Flow A or B	✓ (test_extractor.py)
Legal basis block	`NOT_ASSESSED` artifact blocked	✓
Idempotency	Re-submitting same source_ref produces no duplicate candidates	✓

Core Intake Pipeline

POST /v1/artifacts
      │
      ▼
SourceArtifact created
      │
      ├─ Idempotency check (source_ref)
      ├─ Legal basis check (NOT_ASSESSED blocks)
      │
      ▼
TranscriptSegmenter.segment()
  └─ Selects strategy: VTT | SRT | PlainText | JSON
      │
      ▼
PrivacyTagger.tag_segments()
  └─ Regex patterns + NLTK NER
  └─ Sets privacy_class on each segment
      │
      ▼
CandidateExtractor.extract()
  └─ Interrogative scoring per sentence
  └─ Lookahead window for answer detection
  └─ Flow A = question + answer found
  └─ Flow B = question only
      │
      ▼
Persist: source_artifacts, transcript_segments,
         candidate_questions, candidate_answers

Connector Architecture

The connector framework adds a pull-based ingestion layer above the standard POST /v1/artifacts endpoint. Each registered connector fetches transcripts from an external platform and feeds them through the same CII pipeline.

External Platform          Connector Layer              CII Pipeline
─────────────────          ───────────────              ────────────
Fellow API v2      ──►  FellowConnector          ──►
Gong API v2        ──►  GongConnector            ──►  IntakeService.ingest()
(future sources)   ──►  SourceConnector (any)    ──►

Connector configuration (environment variables):

Variable	Description
`CONNECTOR_CONFIGS_JSON`	Generic JSON object keyed by source system. Preferred path for new connector packages.
`FELLOW_API_TOKEN`	Fellow Personal Access Token or OAuth 2.0 bearer token
`FELLOW_BASE_URL`	Override Fellow API base URL (default: `https://api.fellow.ai/v2`)
`GONG_ACCESS_KEY`	Gong API access key
`GONG_ACCESS_SECRET`	Gong API access secret
`GONG_BASE_URL`	Override Gong API base URL (default: `https://api.gong.io/v2`)

A connector is only registered when its required config is present. If no connector config is set the connector list is empty; no error is thrown. FELLOW_* and GONG_* remain supported for backward compatibility, but new connector packages should be configured through CONNECTOR_CONFIGS_JSON.

Connector sync state:

PUT /v1/connectors/{source_system}/sync persists cadence, legal_basis, customer_scope, and initial_since
default pulls use last_successful_until .. now once a source has synced successfully
if no successful pull exists yet, the first default window is initial_since .. now
duplicates are still skipped by source_ref

Running Locally

# Start Postgres + Redis
docker compose up -d postgres redis

# Apply schema migrations
alembic upgrade head

# Start API server
uvicorn raw_to_knowledge.main:app --reload --port 8000

# Ingest a transcript directly
curl -X POST http://localhost:8000/v1/artifacts \
  -H "Content-Type: application/json" \
  -d '{
    "source_system": "zoom",
    "source_type": "plain_text",
    "legal_basis": "legitimate_interest",
    "raw_content": "Can we do localized builds?\n\nYes, with locale-specific branches configured."
  }'

# Pull from Fellow (requires FELLOW_API_TOKEN in environment)
curl -X POST http://localhost:8000/v1/connectors/fellow/pull \
  -H "Content-Type: application/json" \
  -d '{
    "since": "2026-06-01T00:00:00Z",
    "legal_basis": "legitimate_interest",
    "customer_scope": "acme-corp"
  }'

# Pull a single Gong call (requires GONG_ACCESS_KEY + GONG_ACCESS_SECRET)
curl -X POST http://localhost:8000/v1/connectors/gong/fetch/call_abc123 \
  -H "Content-Type: application/json" \
  -d '{"legal_basis": "legitimate_interest"}'

# Configure daily scheduled sync for Gong
curl -X PUT http://localhost:8000/v1/connectors/gong/sync \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "cadence": "daily",
    "legal_basis": "legitimate_interest",
    "initial_since": "2026-06-01T00:00:00Z"
  }'

# List registered connectors
curl http://localhost:8000/v1/connectors

# Review extracted candidates
curl http://localhost:8000/v1/candidates