Skip to content

Troubleshooting: Generation and Retrieval

This topic covers problems with the LLM generation pipeline, RAG retrieval, and semantic search.

Work item stuck in GENERATING state

Symptom: A work item shows state: GENERATING for an extended period and never transitions to PENDING_VALIDATION or ESCALATED.

Cause: The Celery worker processing the raw_to_knowledge_generation queue has stopped, crashed, or was never started.

Resolution: 1. Check whether the generation worker is running: 

celery -A raw_to_knowledge.infrastructure.celery_app inspect active
2. If no workers respond, start the generation worker: 
celery -A raw_to_knowledge.infrastructure.celery_app worker \
  --queues raw_to_knowledge_generation \
  --concurrency 4 \
  --loglevel info
3. Once the worker is running, the task will be picked up automatically. The work item will transition out of GENERATING. 4. If the task was lost (e.g., Redis was restarted without persistence), re-trigger generation: POST /v1/generation/work-items/{id}/generate.

Work item escalated with reason "no_content"

Symptom: Work item reaches state: ESCALATED with escalation_reason: no_content.

Cause: The RAG retrieval step found fewer than 2 source documents to ground the generation. Raw to Knowledge requires at least 2 sources as a quality gate. This indicates that the knowledge needed to answer the question does not exist in the configured Confluence spaces or Jira project.

Resolution: - Verify that Confluence and Jira connectors are configured correctly. Check CONFLUENCE_BASE_URL, CONFLUENCE_SPACE_KEY, JIRA_BASE_URL, and JIRA_PROJECT_KEY in your environment. - Confirm that the Confluence spaces listed in CONFLUENCE_SPACE_KEY contain content relevant to the question. Try a manual search in Confluence for keywords from the question text. - If the knowledge genuinely does not exist in any configured source, this is a true content gap. The work item must be answered manually by a subject matter expert. - For manually answered escalations, an administrator can close the work item and submit the answer as a new candidate through the intake path.

Work item escalated with an "LLM error" reason

Symptom: Work item reaches state: ESCALATED with escalation_reason starting with LLM error:.

Cause: The LLM API call failed. The error detail is appended to the escalation reason. Common causes: - Invalid or expired OPENAI_API_KEY. - LLM provider rate limit or quota exceeded. - Network connectivity issue between the worker and the LLM API endpoint. - Model identifier in LLM_MODEL is not available to the configured API key.

Resolution: 1. Check OPENAI_API_KEY is set and valid in the worker environment. 2. Verify that LLM_MODEL is a model your API key has access to. 3. Check the structlog output from the generation worker for the full exception message. 4. If the cause was transient (rate limit, network blip), re-trigger generation: POST /v1/generation/work-items/{id}/generate. 5. If errors persist, check your LLM provider's status page and API quota dashboard.

Work item escalated with reason "LLM circuit open"

Symptom: Work item reaches state: ESCALATED with escalation_reason: LLM circuit open.

Cause: The CircuitBreaker protecting LLM calls has tripped. This happens after 5 consecutive LLM failures within a 60-second window. When the circuit is open, all subsequent generation attempts fail immediately without calling the LLM.

Resolution: - Wait 60 seconds. The circuit breaker recovers automatically after the recovery window. - After recovery, re-trigger generation for the affected work items: POST /v1/generation/work-items/{id}/generate. - If you cannot wait 60 seconds, restart the generation worker. The circuit breaker initializes in the closed state on each worker startup. - In a multi-worker deployment, each worker maintains its own independent circuit breaker. Other workers may still be functioning if only one has a tripped circuit. - Investigate the underlying cause of the 5 consecutive failures before re-triggering (see "Work item escalated with an LLM error reason" above).

known_answer_matched returned when you expected LLM generation

Symptom: A GeneratedCandidate has origin: KNOWN_ANSWER and returns an existing answer instead of generating a new one.

Cause: Weaviate found an existing approved answer with cosine similarity ≥ 0.92 to the question. Raw to Knowledge treats this as a sufficient match and returns the existing answer rather than calling the LLM. This is correct behavior — it prevents duplicate answers and reduces LLM costs.

Resolution: - If the matched answer is correct, no action is needed. The existing answer handles this question. - If the matched answer is wrong or outdated, update it: supersede it via POST /v1/registry/answers/{id}/supersede with a corrected version. The corrected answer will be indexed in Weaviate and will become the new match for similar questions. - If you believe the similarity match is a false positive (the matched answer is on a different topic), review the matched answer's answer_text. If it is genuinely irrelevant, file this as a retrieval quality issue — the question phrasing may be too close to an unrelated answer.

Retrieval search returns no results or only registry fallback

Symptom: POST /v1/retrieval/search returns an empty array, or returns only results from the PostgreSQL registry with no Weaviate results.

Cause: One or more of the following: - Weaviate is unavailable or the URL is incorrect. - The Weaviate schema has not been initialized. - No approved answers have been indexed yet (the Weaviate index is empty).

Resolution: 1. Confirm WEAVIATE_URL is correct and the Weaviate service is running: 

curl http://localhost:8080/v1/meta
2. The Weaviate schema is initialized automatically on first API use (ensure_schema() is idempotent). If the schema was not created, check the API startup logs for Weaviate connection errors. 3. If the Weaviate index is empty, approved answers are indexed when they are published. Confirm that at least one answer exists in the registry with status: APPROVED. 4. If recent approved answers are not appearing in search, check whether the vector indexing step is completing successfully in the API logs.

Retrieval results seem irrelevant to the query

Symptom: Semantic search returns results, but they do not seem topically related to the query text.

Cause: The Weaviate index may be stale — approved answers created recently may not have been indexed, or an index corruption may have occurred. Alternatively, the query may be phrased in a way that matches answer embeddings on surface-level word similarity rather than semantic meaning.

Resolution: - Verify that the returned answers are actually in the registry and currently APPROVED: GET /v1/registry/answers/{id}. - Check API logs for errors during the Weaviate indexing step that occurs on answer publication. - If the index seems corrupted or significantly out of sync with PostgreSQL, the Weaviate index can be rebuilt by re-indexing all approved answers. There is no dedicated re-index API endpoint — use the WeaviateAdapter directly or contact your platform administrator. - For query phrasing issues, try rephrasing the search query to use more specific domain terminology.

Export returns 404

Symptom: GET /v1/readout/answers/{answer_id}/export returns 404 Not Found.

Cause: Either the answer_id does not exist, or the answer exists but is not in APPROVED status. The readout endpoint only exports answers that are currently active and approved.

Resolution: 1. Confirm the answer exists: GET /v1/registry/answers/{answer_id}. 2. Check the status field. If the answer is SUPERSEDED, DISPUTED, or RETIRED, it is excluded from readout by design. 3. If the answer should be active, check whether it was incorrectly retired or superseded. Contact an administrator to review the answer lifecycle. 4. If the answer_id does not exist at all, verify you have the correct ID. Use GET /v1/registry/answers with filters to locate the intended answer.