Skip to content

Raw to Knowledge

Trend Analysis

Trend Analysis Architecture

The trend path runs in parallel with the main ingestion pipeline. It extracts topic signals from every transcript immediately — no validation, no registry step, no admin approval required. Trend data is available the moment intake completes.

1. Trend Extraction Pipeline

flowchart TD
    SEG([TranscriptSegments\nfrom intake pipeline])

    subgraph normalize["Utterance Normalization"]
        LNZ[TranscriptLineNormalizer\nSplit segment text on newlines\nCollapse whitespace · Strip edges\nInherit speaker · offset · privacy_class]
    end

    subgraph extract["Mention Extraction"]
        MX[MentionExtractor\nNLTK word_tokenize\npos_tag\nRegexpParser NP grammar\nDT? JJ* NN+]
        MASK{Person span\noverlap?}
        MOUT[RawMention\nnoun_phrase]
        POUT[RawMention\nperson_masked\ntext = PERSON]
    end

    subgraph sentiment["Sentiment Scoring"]
        SS[LineSentimentScorer\nVADER SentimentIntensityAnalyzer\ncompound · pos · neu · neg]
        SLBL{compound value}
        SPOS[positive\n≥ +0.05]
        SNEU[neutral\nbetween thresholds]
        SNEG[negative\n≤ -0.05]
    end

    subgraph link["Question Linkage"]
        QML[QuestionMentionLinker\nFor each question:\nFind anchor line in segment\nSearch ± window_lines\nCreate QuestionMentionLink\ndistance_lines · link_basis]
    end

    subgraph canon["Canonicalization"]
        FREQ{frequency\n≥ promotion_threshold?}
        IGN[Mark\nignored_low_frequency]
        SIM{Similarity score\nvs existing entities}
        AUTO[Auto-resolve\nAUTO_EXISTING\nmatch_score = similarity]
        PROMO[Auto-promote\nAUTO_PROMOTED\nCreate new CanonicalEntity]
        ADMIN[Admin review queue\nPENDING_ADMIN_REVIEW\nsuggested_entity_id]
    end

    subgraph persist["Persist + Project"]
        DB[(PostgreSQL\ntranscript_lines\nraw_mentions\nline_sentiments\nquestion_mention_links\ncanonical_entities\nmention_resolutions\nmention_clusters)]
        N4[(Neo4j\nTrendGraphProjector\nLine · Mention · Entity\nTaxonomy · Speaker\nTimeBucket nodes)]
    end

    SEG --> LNZ --> MX
    LNZ --> SS
    MX --> MASK
    MASK -->|yes| POUT
    MASK -->|no| MOUT
    SS --> SLBL
    SLBL --> SPOS & SNEU & SNEG

    MOUT & POUT --> QML
    MOUT & POUT --> FREQ
    FREQ -->|below| IGN
    FREQ -->|above| SIM
    SIM -->|≥ auto_resolution_threshold| AUTO
    SIM -->|cohesive cluster\nno match| PROMO
    SIM -->|unresolvable| ADMIN

    AUTO & PROMO & IGN & ADMIN --> DB
    DB --> N4

2. Canonicalization Workflow

flowchart LR
    subgraph input["Input"]
        M[RawMentions\nnormalized_text]
    end

    subgraph cluster["Build Clusters"]
        CL[Group by\nnormalized_text key]
        CF[Count frequency\nper cluster]
    end

    subgraph gate["Frequency Gate"]
        FG{frequency ≥\npromotion_threshold?}
        SKIP[MentionCluster\nignored_low_frequency\nno resolution]
    end

    subgraph match["Similarity Matching"]
        EM{Exact match\nvs active entities?}
        JC[Jaccard similarity\ntoken overlap ÷ union]
        BM{best_score ≥\nauto_resolution_threshold?}
    end

    subgraph resolve["Resolution"]
        RES_E[MentionResolution\nauto_existing\nlink to matched entity]
        RES_P[New CanonicalEntity\n+ MentionResolution\nauto_promoted\nadded to entity pool]
        RES_A[MentionCluster\npending_admin_review\nsuggested_entity_id if any]
    end

    M --> CL --> CF --> FG
    FG -->|no| SKIP
    FG -->|yes| EM
    EM -->|exact| RES_E
    EM -->|no exact| JC --> BM
    BM -->|yes| RES_E
    BM -->|no, cohesive| RES_P
    BM -->|no, incoherent| RES_A

Similarity scoring rules (in priority order):

Exact normalized match → score 1.0
Jaccard token overlap: |A ∩ B| / |A ∪ B| using whitespace-split token sets
Default auto_resolution_threshold: 0.80
Default promotion_threshold: 2 occurrences

3. Trend Report Data Flow

flowchart LR
    subgraph sources["Data Sources"]
        PG_M[(raw_mentions)]
        PG_R[(mention_resolutions)]
        PG_E[(canonical_entities)]
        PG_T[(canonical_entity_taxonomies)]
        PG_S[(line_sentiments)]
        PG_L[(transcript_lines)]
        PG_Q[(question_mention_links)]
    end

    subgraph svc["TrendReportService"]
        AGG[Aggregate\nby entity · week · scope · speaker]
        JOIN[Join with\ntaxonomies and sentiment]
        ROWS[TrendReportRow × N]
    end

    subgraph api["API Surfaces"]
        JSON[GET /v1/trends/report\nJSON response]
        CSV[GET /v1/trends/report.csv\nCSV download]
        DASH[GET /v1/dashboard/trends\nTop-N panel]
    end

    subgraph filters["Active Filters"]
        F1[from_date · to_date]
        F2[customer_scope]
        F3[speaker]
        F4[taxonomy_type · taxonomy_value]
        F5[canonical_entity_id]
    end

    PG_M & PG_R & PG_E & PG_T & PG_S & PG_L & PG_Q --> AGG
    filters --> AGG
    AGG --> JOIN --> ROWS
    ROWS --> JSON & CSV & DASH

4. Trend Neo4j Graph — Cypher Analytics Library

The following Cypher queries are available in infrastructure/graph/cypher_queries.py and are used by the TrendGraphProjector.

Top Topics by Period

MATCH (m:RawMention)-[:RESOLVES_TO]->(e:CanonicalEntity)
MATCH (l:TranscriptLine)-[:HAS_MENTION]->(m)
MATCH (l)-[:IN_WEEK]->(tb:TimeBucket)
WHERE tb.week >= $from_week AND tb.week <= $to_week
WITH e, tb.week AS week, count(m) AS mention_count
ORDER BY mention_count DESC
RETURN e.canonical_entity_id AS entity_id, e.label AS label,
       week, mention_count
LIMIT $limit

Question Count by Canonical Entity

MATCH (q:Question)-[:MENTIONS]->(e:CanonicalEntity)
RETURN e.canonical_entity_id AS entity_id,
       e.label AS label,
       count(q) AS question_count
ORDER BY question_count DESC

Sentiment Trend by Entity and Week

MATCH (m:RawMention)-[:RESOLVES_TO]->(e:CanonicalEntity)
MATCH (l:TranscriptLine)-[:HAS_MENTION]->(m)
MATCH (l)-[:IN_WEEK]->(tb:TimeBucket)
WHERE tb.week >= $from_week AND tb.week <= $to_week
WITH e, tb.week AS week,
     count(CASE WHEN l.sentiment_label = 'positive' THEN 1 END) AS pos_count,
     count(CASE WHEN l.sentiment_label = 'neutral'  THEN 1 END) AS neu_count,
     count(CASE WHEN l.sentiment_label = 'negative' THEN 1 END) AS neg_count
RETURN e.canonical_entity_id AS entity_id, e.label AS label, week,
       pos_count, neu_count, neg_count
ORDER BY week, e.label

Speaker–Topic Heat Map

MATCH (l:TranscriptLine)-[:SPOKEN_BY]->(s:Speaker)
MATCH (l)-[:HAS_MENTION]->(m:RawMention)-[:RESOLVES_TO]->(e:CanonicalEntity)
RETURN s.name AS speaker,
       e.canonical_entity_id AS entity_id,
       e.label AS label,
       count(m) AS mention_count
ORDER BY mention_count DESC

Taxonomy Rollup (any type)

MATCH (e:CanonicalEntity)-[:IN_TAXONOMY]->(t:TaxonomyTerm {type: $taxonomy_type})
MATCH (m:RawMention)-[:RESOLVES_TO]->(e)
RETURN t.value AS taxonomy_value,
       count(DISTINCT e) AS entity_count,
       count(m) AS mention_count
ORDER BY mention_count DESC

Customer-Scope Comparison

MATCH (a:Artifact)-[:IN_SCOPE]->(cs:CustomerScope)
MATCH (l:TranscriptLine {artifact_id: a.artifact_id})
      -[:HAS_MENTION]->(m:RawMention)-[:RESOLVES_TO]->(e:CanonicalEntity)
WHERE cs.name IN $scope_names
RETURN cs.name AS customer_scope,
       e.canonical_entity_id AS entity_id,
       e.label AS label,
       count(m) AS mention_count
ORDER BY customer_scope, mention_count DESC

5. Admin Cluster Review Workflow

stateDiagram-v2
    [*] --> pending_admin_review : High-frequency\ncluster unresolved

    pending_admin_review --> admin_merged : POST .../merge\nActor merges into\nexisting entity
    pending_admin_review --> admin_created : POST .../create-entity\nActor creates\nnew canonical entity
    pending_admin_review --> suppressed : POST .../suppress\nActor marks noisy

    admin_merged --> [*] : MentionResolution created\nresolution_mode = admin_merge
    admin_created --> [*] : CanonicalEntity created\nMentionResolution created\nresolution_mode = admin_create
    suppressed --> [*] : No resolution created\nCluster excluded from reports

    note right of pending_admin_review
        Listed at GET /v1/trends/admin/clusters
        Shows frequency, cluster_key,
        and suggested_entity_id
    end note

6. Privacy Controls in the Trend Path

Data	Behaviour
Raw transcript text	Stays inside the CII boundary; never appears in trend API responses
Person-named mentions	Detected via `PrivacyTagger` PERSON spans; `mention_text` stored as `[PERSON]`; `privacy_class = pii_detected`
Trend JSON API	Returns only `normalized_text` and entity labels — no raw person strings
CSV export	Same masking as JSON — no person clear-text
Neo4j trend graph	`RawMention.mention_text` = `[PERSON]` for masked mentions
`mention_resolutions`	Person-masked mentions never resolve to named-person canonical entities
Admin cluster queue	Clusters with `cluster_key = [person]` are suppressed by convention

7. Configuration Reference

All trend policy parameters are explicit settings in src/raw_to_knowledge/config.py — no hidden constants.

Setting	Default	Description
`trend_promotion_threshold`	`2`	Minimum frequency for a cluster to leave `ignored_low_frequency`
`trend_auto_resolution_threshold`	`0.80`	Minimum Jaccard similarity for automatic entity resolution
`trend_positive_sentiment_threshold`	`0.05`	Minimum VADER compound for positive label
`trend_negative_sentiment_threshold`	`-0.05`	Maximum VADER compound for negative label
`trend_question_mention_window`	`5`	Lines ± around a question anchor to search for mentions
`trend_person_masking_strategy`	`token`	`token` = `[PERSON]`, `label` = `<PERSON>`, `redact` = blanked
`trend_taxonomy_assignment_policy`	`manual`	`manual` requires admin assignment; `auto` applies heuristics