Skip to content

Privacy, Scope, and Provenance

Raw to Knowledge answers are reused across teams, time periods, and customer engagements. For that reuse to be safe and trustworthy, every answer in the registry must carry three forms of trust infrastructure: privacy tagging, customer scope, and provenance. This topic explains each pillar and how the system enforces them.

Privacy tagging

Before any transcript segment is analyzed for knowledge content, it is scanned for personal information.

The intake pipeline applies NLTK named-entity recognition and regular-expression pattern matching to identify five categories of personal data:

Tag What is detected
PERSON Named individuals identified by NER
EMAIL Email addresses
PHONE_US US-format phone numbers
SSN Social Security numbers
CREDIT_CARD Credit card numbers

Detected personal information is flagged at the segment level. The SourceArtifact record receives a privacy_class based on the highest-sensitivity content found. This class determines how the artifact can be processed and stored.

Every SourceArtifact must have a declared legal_basis before it can advance to extraction. Legal basis values are drawn from GDPR Article 6:

Legal basis value Meaning
CONSENT The individuals whose data appears in the transcript have consented to this processing
LEGITIMATE_INTEREST Processing is necessary for legitimate business interests, with no overriding individual rights
CONTRACT Processing is necessary for performance of a contract with the data subjects
LEGAL_OBLIGATION Processing is required by law
NOT_ASSESSED Legal basis has not been determined

Important: If legal_basis is NOT_ASSESSED, the artifact is held at the intake stage and does not advance to extraction or classification. The pipeline enforces this at the repository layer. Operators must supply a valid legal basis before processing can continue. This is not a warning — it is a hard block.

Customer scope

Every approved answer may carry a customer_scope field that identifies the customer engagement in which the knowledge claim was made. Customer scope is enforced at the repository layer, not only in the user interface.

This enforcement has a concrete consequence: a readout assembled for Customer A will never include answers scoped exclusively to Customer B. This applies even if the underlying factual claim would be equally relevant — the scope boundary respects information obligations and competitive sensitivity.

Some answers are not customer-scoped. Product capability facts, architectural principles, and reference data that apply universally are stored without a customer_scope value and are available to all readout queries.

Answers with a customer_scope are available only in queries that explicitly include that scope value, or to users with cross-scope access (typically knowledge architects).

Note: Scope isolation is designed to reflect real-world information obligations. A customer's state assertions — what they have deployed, what they have agreed to, what problems they have experienced — belong to that customer relationship. Cross-contamination of this information across accounts is a governance failure that Raw to Knowledge prevents by design.

Provenance

Provenance is the record of where an answer came from, who approved it, and what evidence supports it. In Raw to Knowledge, provenance is not metadata — it is a registry insertion requirement. An answer that lacks complete provenance cannot enter the registry.

Required provenance fields

Field Meaning Enforcement
validation_record_id The ID of the human validation decision that approved this answer Required; ProvenanceError if missing
owner_role The role accountable for the answer's accuracy (the approver's role) Required; ProvenanceError if missing
evidence_links A list of URIs to source artifacts, Confluence pages, Jira issues, or other verifiable sources Required (minimum one); ProvenanceError if empty

Evidence links appear in every export format. In Markdown exports, they are rendered as a references section below the answer text. In JSON-LD exports, they are represented as schema:citation triples, making the provenance machine-readable by downstream systems.

Evidence links should point to durable, accessible sources: the source transcript artifact, a Confluence page with supporting documentation, or a Jira issue that records the original decision. Short-lived or access-restricted links should be avoided — an evidence link that a future reviewer cannot open undermines the provenance guarantee.

Review due dates

Every approved answer has a review_due date. This date represents the governance commitment that the answer will be re-examined before it expires.

Default intervals by Horn type

Review-due intervals are configurable per deployment. Typical defaults:

Horn type Default review interval
FACT — STATE 90 days (states change frequently)
FACT — CONFIGURATION 180 days
FACT — CAPABILITY 180 days
FACT — CONSTRAINT 180 days
FACT — EXISTENCE 365 days
FACT — TEMPORAL Set to the asserted date + 30 days
CONCEPT, PROCEDURE, PROCESS 365 days
PRINCIPLE, REFERENCE 365 days
TROUBLESHOOTING, RECOMMENDATION 180 days

Expiry alerts

Answers within 30 days of their review_due date are flagged in readouts with a visual indicator. The dashboard aggregates expiring answers by Horn type and by customer scope, giving governance leads a prioritized list of review work.

An answer past its review_due date is not automatically retired. It remains in active retrieval but is flagged as overdue. The governance lead is responsible for initiating review — re-approving the answer (which resets the clock), superseding it with a newer version, or retiring it.

Note: Allowing answers to accumulate past their review-due dates without action degrades the trust value of the registry. Consumers cannot confidently cite an answer that is overdue for review. Regular review-due triage is a core governance lead responsibility.

Versioning and the immutability guarantee

Answer text cannot be edited in place. This is an architectural decision, not a UI limitation. The ApprovedAnswer table is write-once after insertion.

When new information supersedes an approved answer, the process is:

  1. A new candidate is created with the updated answer text.
  2. The candidate passes through classification and the full validation workflow.
  3. When the new candidate is approved, a new ApprovedAnswer record is created with version = previous_version + 1 and supersedes_id = previous_answer_id.
  4. The previous answer's status is automatically set to SUPERSEDED.

This design preserves the complete history of what the organization believed to be true at any point in time. The full version chain is queryable via the API and traversable in the Neo4j graph. A consumer who receives an answer can always determine whether it is the current version or a superseded one.

flowchart LR
    A1["ApprovedAnswer v1\nstatus: SUPERSEDED\nsupersedes_id: null"]
    A2["ApprovedAnswer v2\nstatus: APPROVED\nsupersedes_id: A1.id"]
    A3["ApprovedAnswer v3\nstatus: APPROVED\nsupersedes_id: A2.id"]

    A1 -->|superseded by| A2 -->|superseded by| A3

Important: Because answers are immutable, there is no "edit" operation in Raw to Knowledge. Any change to an approved answer — even a correction to a typo in a governance-critical fact — requires a new candidate, full validation, and a formal supersession. This process takes time. Plan for it when operating in high-velocity environments.