Skip to content

Review a Generated Answer and Its Citations

Before you begin

Audience: Validator, SME, support engineer

The work item must be in PENDING_VALIDATION state before you begin this task. If the work item is still in NEW or GENERATING, wait for the Celery worker to complete processing and poll again. If it is in ESCALATED, see Submit a Generation Work Item for escalation handling guidance.

Generated answers (Flow B) require a higher level of scrutiny than Flow A answers that were extracted directly from human conversation. The LLM synthesizes content from internal source documents; it can produce plausible-sounding text that misrepresents the cited sources. You are the authoritative check on this.

Steps

1. Retrieve the generated candidate

curl -s http://localhost:8000/v1/generation/work-items/{work_item_id}/candidate

The response returns a GeneratedCandidate object. Key fields are described in the steps below.

Example response structure:

{
  "candidate_id": "cand-a1b2c3d4",
  "work_item_id": "wi-9f0a1b2c",
  "generated_text": "The maximum payload size for a webhook delivery in the v3 API is 5 MB...",
  "origin": "AUTOMATED",
  "model_version": "claude-sonnet-4-6",
  "answer_candidate_id": null,
  "citations": [
    {
      "source_url": "https://confluence.example.com/wiki/spaces/APIDOCS/pages/123456",
      "title": "Webhook Configuration Guide — v3",
      "last_modified": "2026-04-15T09:30:00Z",
      "excerpt": "Payloads exceeding 5 MB are rejected with HTTP 413...",
      "authority_weight": 0.9
    },
    {
      "source_url": "https://jira.example.com/browse/API-8812",
      "title": "Increase webhook payload limit to 5 MB",
      "last_modified": "2025-11-02T16:45:00Z",
      "excerpt": "Resolved: Updated limit enforced in v3.2 release.",
      "authority_weight": 0.8
    }
  ]
}

2. Evaluate the generated text

Read the generated_text against the question it was created to answer (available in the parent work item). Ask:

  • Does the answer directly address the question?
  • Is every factual claim in the generated text traceable to a cited source?
  • Are there claims that appear in the generated text but are not supported by any citation?
  • Is the answer complete, or does it hedge excessively without providing actionable information?

If the generated text introduces facts not present in the citations, treat it as unreliable and proceed to step 7 (reject or escalate).

3. Evaluate each citation

For every entry in the citations array, verify:

  1. The excerpt supports the generated text. Open the source_url and confirm the excerpt appears in context and is not taken out of meaning.
  2. The last_modified date is recent enough to be authoritative. An answer generated from a Confluence page last modified two years ago may be outdated.
  3. The authority_weight reflects your confidence in the source.

Authority weight reference:

Source type Typical authority_weight
Confluence (official documentation) 0.9
Jira (resolved tickets, release notes) 0.8
Other connected sources Varies

Higher-weight sources should carry more influence in your judgment. A generated text supported only by low-weight sources warrants additional verification.

Note: Citations are frozen at generation time. If a Confluence page or Jira ticket has been updated since the work item was created, the excerpt in the citation may no longer reflect the current source content. Always check the last_modified field and open the source_url to confirm the content is current.

4. Check the origin field

The origin field determines how the candidate was produced:

origin value Meaning
AUTOMATED The text was generated by the LLM using RAG context from Confluence and Jira
KNOWN_ANSWER A Weaviate similarity match at ≥ 0.92 was found; LLM generation was bypassed; the generated_text is drawn from the existing approved answer

Note: When origin is KNOWN_ANSWER, the answer_candidate_id field points to the existing registry entry that was matched. The system surfaced an already-approved answer rather than generating a new one. Review the referenced entry to confirm it fully addresses the current question before proceeding to validation.

5. Check the model version

The model_version field records which LLM produced the answer. This is relevant when comparing candidates across time periods or when assessing whether a model upgrade has changed generation quality. Note the value in any escalation rationale if quality is a concern.

6. Accept the candidate for validation

If the generated text is accurate, complete, and supported by its citations, proceed to formal validation:

  1. Create a validation assignment for the candidate_id using the classification service.
  2. Claim the assignment in the validation queue.
  3. Apply your approve, revise, or reject decision normally, following the same process used for Flow A candidates.

The generated candidate enters the standard 9-state validation lifecycle from this point forward.

Note: F2 scoring (β = 2, recall-weighted) means the generation pipeline is tuned to prioritize not missing valid answers over avoiding false positives. As a result, some generated candidates will be borderline in quality. Validator judgment is essential — the pipeline intentionally errs toward inclusion, and human review is the corrective step.

7. Reject or escalate a poor candidate

If the generated text is factually incorrect, unsupported, incomplete, or misleading:

  • Reject at validation — Use the standard reject decision and document the specific problem in the rationale field. Be precise: state which claim is wrong and which citation failed to support it.
  • Escalate — If the question is important and the generated content is insufficient, escalate with a note that manual authoring is required.

Documenting the problem in rationale feeds quality improvement cycles. Vague rationale ("answer was bad") does not provide enough signal for pipeline refinement.

Result

You have evaluated the generated candidate against its source citations and determined whether it is fit for publication. If accepted, the candidate proceeds through standard validation to become an approved registry answer. If rejected or escalated, the knowledge gap is documented with a specific rationale that supports future improvement.