Skip to content

Troubleshooting: Graph and Operations

This topic covers problems with Neo4j graph sync, contradiction detection, Prometheus metrics, and general operational health.

Contradictions not appearing for an answer that should conflict

Symptom: An answer has a conflicting counterpart in the registry (same subject+predicate+scope, different object_value), but GET /v1/graph/answers/{id}/contradictions returns an empty array.

Cause: The Neo4j graph is a derived projection of PostgreSQL. It does not update in real time. If graph sync has not been run for this answer since the conflicting answer was published, the graph will not reflect the conflict.

Resolution: 1. Trigger a graph sync for the answer: 

curl -X POST http://localhost:8000/v1/graph/answers/{answer_id}/sync
2. Then trigger contradiction detection: 
curl -X POST http://localhost:8000/v1/graph/answers/{answer_id}/detect-contradictions
3. Ensure the raw_to_knowledge_graph worker is running to process these tasks: 
celery -A raw_to_knowledge.infrastructure.celery_app inspect active
4. After the tasks complete, re-check GET /v1/graph/answers/{id}/contradictions.

If contradictions still do not appear, confirm that both conflicting answers are in APPROVED status and that Neo4j contains nodes for both. Check the graph worker logs for sync errors.

"Neo4j is unavailable" errors on graph endpoints

Symptom: Graph endpoints return 500 Internal Server Error or an error message indicating Neo4j is unavailable. All other endpoints continue to function normally.

Cause: The Raw to Knowledge service cannot connect to Neo4j. This may be because: - Neo4j is not running. - NEO4J_URI, NEO4J_USERNAME, or NEO4J_PASSWORD are incorrect. - A firewall or network configuration is blocking the Bolt port (default: 7687).

Resolution: 1. Confirm Neo4j is running: 

curl http://localhost:7474/browser/
2. Test the Bolt connection directly using the cypher-shell tool: 
cypher-shell -a bolt://localhost:7687 -u neo4j -p password "RETURN 1;"
3. Verify environment variables in the API service and graph worker environments. 4. Graph features degrade gracefully — the intake, validation, and registry paths are unaffected. Only graph-specific endpoints (/v1/graph/...) will return errors.

Graph worker not processing tasks

Symptom: Tasks are enqueued (202 Accepted from the API), but work items or sync tasks never complete. celery inspect active shows no active tasks for the graph queue.

Cause: The graph worker is not running, or was started without the raw_to_knowledge_graph queue specified.

Resolution: 1. Verify the worker is running and listening on the correct queue: 

celery -A raw_to_knowledge.infrastructure.celery_app inspect active
2. Start the graph worker if it is not running: 
celery -A raw_to_knowledge.infrastructure.celery_app worker \
  --queues raw_to_knowledge_graph \
  --concurrency 2 \
  --loglevel info
3. If a worker is running but not on raw_to_knowledge_graph, restart it with the correct --queues argument. A worker started without --queues raw_to_knowledge_graph will not consume graph tasks. 4. Check Redis connectivity from the worker host: 
redis-cli -u $REDIS_URL ping
If Redis is unreachable, the worker cannot dequeue tasks.

Prometheus /metrics returns empty values or all zeros for gauges

Symptom: GET /metrics returns the 13 Raw to Knowledge metrics, but all gauge values are 0 or the gauges are missing entirely.

Cause: Raw to Knowledge's Prometheus gauges are not push-based. They are only updated when GET /v1/dashboard/metrics is called. If no one has called the dashboard endpoint since the service started (or since the last restart), all gauges will report their initial value of 0.

Resolution: - Call the dashboard endpoint to trigger a metrics refresh: 

curl http://localhost:8000/v1/dashboard/metrics
- Then re-scrape /metrics. The gauges will now reflect current state. - For continuous Prometheus monitoring, configure your scrape job to also call /v1/dashboard/metrics on a schedule, or set up a lightweight polling script that calls the dashboard endpoint periodically.

Note: This architecture means the /metrics endpoint reflects the state at the time of the last /v1/dashboard/metrics call, not real time. If freshness is important, call the dashboard endpoint more frequently.

Dashboard metrics look stale or show incorrect counts

Symptom: GET /v1/dashboard/metrics returns counts that do not match what is visible in the registry or validation queue.

Cause: The dashboard endpoint queries PostgreSQL directly on each call. If counts appear wrong, possible causes include: - A long-running database transaction that has not committed yet, making recently changed rows invisible to the query. - A Celery task that completed but whose result (a state transition on a work item or answer) has not yet been persisted. - A bug in a specific metric query.

Resolution: - Refresh the dashboard endpoint again after a short wait. Uncommitted transactions will resolve within seconds under normal operation. - If specific counts are consistently wrong, check the Celery task logs for errors in state transition persistence. - Compare the raw counts using direct API queries (e.g., GET /v1/registry/answers?status=APPROVED) to determine which metric is inaccurate.

High work_items_pending count in dashboard

Symptom: GET /v1/dashboard/metrics shows a high number of work items in PENDING_VALIDATION, IN_CLASSIFICATION, or READY_FOR_REVIEW state. The count is growing.

Cause: The generation pipeline is backed up. This typically means: - The raw_to_knowledge_generation worker is not running or is processing tasks slowly. - LLM API calls are slow (high latency) or failing (circuit breaker may be tripping repeatedly). - More generation work items are being created than the worker can process.

Resolution: 1. Check the generation worker status: 

celery -A raw_to_knowledge.infrastructure.celery_app inspect active
2. Check the task queue depth: 
celery -A raw_to_knowledge.infrastructure.celery_app inspect reserved
3. If the worker is slow due to LLM latency, consider increasing worker concurrency (--concurrency) to process more tasks in parallel. LLM calls are I/O-bound, so higher concurrency is appropriate. 4. If the circuit breaker is tripping, resolve the underlying LLM connectivity issue first (see Troubleshooting: Generation and Retrieval). 5. If the backlog is large and growing, consider running additional generation worker instances.