Skip to content

Backup, Retention, and Audit

Data store roles

Raw to Knowledge uses three persistent stores with distinct roles. Understanding which store is authoritative determines what must be backed up and what can be recovered from scratch.

Store Role Authoritative?
PostgreSQL 16 Primary store for all domain objects Yes — this is the source of truth
Weaviate Vector index of approved answers for semantic retrieval No — derived projection of PostgreSQL
Neo4j 5 Graph projection of answers for contradiction detection No — derived projection of PostgreSQL
Redis 7 Celery task broker and validation queue No — transient; only in-flight tasks at risk

What this means for backup: PostgreSQL is the only store that must be backed up to preserve the system state. Weaviate and Neo4j can be rebuilt from PostgreSQL by re-running their respective sync and indexing operations. Redis data loss causes at most the loss of tasks that were enqueued but not yet completed at the time of failure — completed task results are persisted to PostgreSQL.

What to back up

PostgreSQL (required)

Back up the entire database, including all schemas. The migration state (the alembic_version table) must be included so that Alembic knows the schema version on restore.

pg_dump -Fc -U raw_to_knowledge raw_to_knowledge > raw_to_knowledge_$(date +%Y%m%d_%H%M%S).dump

Restore:

pg_restore -U raw_to_knowledge -d raw_to_knowledge raw_to_knowledge_<timestamp>.dump

After restoring to a new instance, verify the migration state:

alembic current

Redis (optional)

Back up Redis only if in-flight task recovery is required. If tasks are lost at failover, they can be re-triggered manually via the API endpoints. This is acceptable in most deployments.

If Redis persistence is enabled (appendonly yes in redis.conf), standard AOF or RDB snapshot approaches apply.

Alembic migration scripts

Include the alembic/versions/ directory in source control. The migration scripts are the schema definition; they are not stored in the database beyond the current revision pointer.

Retention guidance

ValidationRecord

ValidationRecord entries are immutable once created. Every validation decision — approved, revised, rejected, escalated — produces a permanent record with the actor, rationale, and timestamp. These records must be retained for the operational lifetime of the associated knowledge asset. Do not delete them.

ApprovedAnswer

ApprovedAnswer entries represent published knowledge. They are immutable in the sense that their content is never overwritten in place. Lifecycle changes (supersession, dispute, retirement) are modeled as state transitions on existing records, not replacements.

Note: There is no DELETE endpoint for ApprovedAnswer. Retirement is the lifecycle end-state. This is intentional — the audit trail must survive the answer's operational lifetime. If regulatory requirements mandate physical deletion, this must be handled at the database layer by a designated DBA, outside the application API.

SUPERSEDED answers must be retained indefinitely. The supersession chain (via supersedes_id) is the version history of a knowledge asset. Breaking the chain by deleting a superseded answer makes the provenance of the current answer unverifiable.

CandidateAnswer and ValidationRecord (rejected)

REJECTED candidates and their associated ValidationRecord entries should be retained for a minimum audit window appropriate to your compliance requirements. The recommended minimum is 12 months. They represent decisions made about knowledge claims and may be needed to demonstrate due diligence.

TranscriptSegment and SourceArtifact

Transcript data may contain personal information. Retain in accordance with your data retention policy and the legal basis declared at ingest.

Audit trail by design

Raw to Knowledge is designed around an append-only audit trail. The key properties:

Immutable records: ValidationRecord is written once. The revised_text field captures any text change made during revision. The prior_record_id field links a revised decision to its predecessor, creating a decision chain.

Version chain: ApprovedAnswer.supersedes_id links each answer to the answer it replaced. Following the chain backward reveals the full version history.

No destructive API operations: The API exposes no DELETE endpoints for domain objects. All lifecycle changes are state transitions (approve → supersede, supersede → retire). This means the database grows monotonically and the full history is always queryable.

Provenance on publish: Every ApprovedAnswer carries validation_record_id, owner_role, evidence_links, and created_at. These fields cannot be omitted; the ProvenanceError domain exception prevents publication without them.

SourceArtifact.legal_basis records the GDPR Article 6 legal basis declared at ingest time. Artifacts with legal_basis = NOT_ASSESSED are blocked from processing. If your deployment handles personal data in transcripts, ensure:

  1. The correct legal basis is declared at ingest.
  2. Your data retention schedule applies to TranscriptSegment and SourceArtifact rows, not only the derived ApprovedAnswer rows.
  3. Right-to-erasure requests are handled at the database layer by a designated DBA, following your organization's documented procedure.

Rebuilding derived stores

If Weaviate or Neo4j is lost and must be rebuilt:

Weaviate: The ensure_schema() call on API startup re-creates the collection schema. Re-index approved answers by querying PostgreSQL for all answers with status = APPROVED and re-submitting them through the indexing path. There is no dedicated re-index API endpoint — this requires a one-time administrative script against the API or direct use of the WeaviateAdapter.

Neo4j: Run POST /v1/graph/answers/{id}/sync for each approved answer. For large registries, submit these calls in batch using the Celery raw_to_knowledge_graph queue. The ensure_constraints() call on sync task startup re-creates schema constraints.