First-Run Setup

This topic walks a system administrator through a complete, clean installation of Raw to Knowledge — from cloning the repository to a running, verified system. Follow every step in order. Optional services (Weaviate, Neo4j, LLM) are covered after the required core is confirmed healthy.

Time to complete: 20–40 minutes depending on network speed and whether Docker images are already cached.

Before you begin

What you will need

Requirement	Version	Why
Python	3.12 (CPython)	Runtime. Earlier versions are not tested.
Docker + Docker Compose	Current stable	Runs PostgreSQL 16 and Redis 7.
`make`	Any	Convenience wrapper around common commands.
Git	Any	Cloning and updates.
A text editor	Any	Editing `.env`.

Note: All commands below assume a macOS or Linux shell. Windows users should run commands inside WSL2 or adapt paths accordingly.

Ports used

Ensure the following ports are free on the host before proceeding:

Port	Service
5432	PostgreSQL
6379	Redis
8000	Raw to Knowledge API
8080	Optional — Weaviate vector store
7474	Optional — Neo4j browser
7687	Optional — Neo4j Bolt

Access model

Raw to Knowledge does not have built-in user accounts or a login screen. The API is accessible to any client that can reach the host and port. In production, access control is the responsibility of your network infrastructure — place the API behind a reverse proxy (nginx, Caddy) that enforces your organization's authentication policy. During local development, the API is open by default.

Step 1: Clone the repository

git clone <your-repository-url> raw_to_knowledge
cd raw_to_knowledge

All subsequent commands in this guide are run from the repository root (raw_to_knowledge/).

Step 2: Configure the environment

Raw to Knowledge reads all configuration from environment variables. The .env.example file at the repository root lists all supported variables with safe defaults. Copy it to .env and edit it before starting any services.

cp .env.example .env

Now open .env in your editor and set every value in the table below.

Required variables

These must be set before the system will start:

Variable	What to set	Example
`POSTGRES_PASSWORD`	A strong password for the PostgreSQL `raw_to_knowledge` database user. This value must match what is in `DATABASE_URL`.	`POSTGRES_PASSWORD=s3cur3P@ssword`
`DATABASE_URL`	PostgreSQL connection string. Must use the `postgresql+asyncpg://` prefix — the synchronous driver will cause startup failures. Replace `changeme` with the same password you set above.	`DATABASE_URL=postgresql+asyncpg://raw_to_knowledge:s3cur3P@ssword@localhost:5432/raw_to_knowledge`
`REDIS_URL`	Redis connection string. The default is correct if you are running Redis via Docker Compose on the same machine.	`REDIS_URL=redis://localhost:6379/0`

Optional variables — set now if known

You can add these to .env at first run or later. The API starts without them, but those feature areas will return errors until they are set.

Variable	Default	Purpose
`OPENAI_API_KEY`	(none — required for generation)	LLM provider key. Required for answer generation and RAG.
`LLM_MODEL`	`gpt-4o`	Which chat completions model to use.
`WEAVIATE_URL`	`http://localhost:8080`	Vector store for semantic retrieval.
`WEAVIATE_API_KEY`	(empty)	Weaviate Cloud only. Leave blank for local Weaviate.
`NEO4J_URI`	`bolt://localhost:7687`	Graph database for contradiction detection.
`NEO4J_USERNAME`	`neo4j`	Neo4j user.
`NEO4J_PASSWORD`	`password`	Neo4j password. Change this for production.
`CONFLUENCE_BASE_URL`	(none)	Confluence RAG source. All four Confluence vars must be set or the connector is skipped.
`CONFLUENCE_USERNAME`	(none)	Confluence username (email address).
`CONFLUENCE_API_TOKEN`	(none)	Confluence API token.
`CONFLUENCE_SPACE_KEY`	(none)	Comma-separated space keys to index, e.g. `KM,ENG`.
`JIRA_BASE_URL`	(none)	Jira RAG source. All four Jira vars must be set or the connector is skipped.
`JIRA_USERNAME`	(none)	Jira username (email address).
`JIRA_API_TOKEN`	(none)	Jira API token.
`JIRA_PROJECT_KEY`	(none)	Jira project key to index, e.g. `KM`.
`LOG_LEVEL`	`INFO`	Logging verbosity. `DEBUG` is very noisy; use `INFO` in production.
`ENVIRONMENT`	`development`	Anything other than `development` disables CORS wildcard. Set to `production` for live deployments.
`CLASSIFICATION_THRESHOLD`	`0.75`	Confidence floor. Candidates below this are flagged `requires_review = true`.
`BLOCK_VALIDATION_ON_CONFLICT`	`true`	When `true`, conflicted answers cannot enter the validation queue.
`EXTRA_PII_KEYWORDS`	(empty)	Comma-separated additional PII keyword patterns for the privacy filter.

For the complete variable reference including advanced tuning options, see Environment Variables Reference.

Security reminder: Never commit .env to version control. The .gitignore excludes it, but double-check before pushing. Treat OPENAI_API_KEY, POSTGRES_PASSWORD, and any API tokens as secrets.

Step 3: Install Python dependencies

Create a virtual environment and install the full development dependency set, including the Alembic CLI, test suite, and NLTK language models:

python3.12 -m venv .venv
source .venv/bin/activate
make install-dev

make install-dev runs:

pip install -e ".[dev,docs]"
python -c "import nltk; nltk.download('punkt_tab'); nltk.download('averaged_perceptron_tagger_eng'); nltk.download('maxent_ne_chunker_tab'); nltk.download('words')"

NLTK downloads may take a minute on first run. They are cached under ~/nltk_data for subsequent installs.

Note: Keep the virtual environment active for all subsequent commands in this guide. Every terminal session that runs Raw to Knowledge components must have the virtual environment activated: source .venv/bin/activate.

Step 4: Start infrastructure services

Start PostgreSQL 16 and Redis 7 using the provided Docker Compose file:

make docker-up

This runs docker compose -f docker/docker-compose.yml up -d. The Compose file reads POSTGRES_PASSWORD from your .env file automatically.

Verify both containers are healthy before continuing:

docker compose -f docker/docker-compose.yml ps

Both services should show healthy in the Status column. If a container exits immediately, check the logs:

docker compose -f docker/docker-compose.yml logs postgres
docker compose -f docker/docker-compose.yml logs redis

Common causes: port already in use, insufficient Docker memory, or POSTGRES_PASSWORD not set.

Step 5: Apply database migrations

Create all tables and indexes by running the Alembic migration chain:

make migrate

This runs alembic upgrade head. The migration chain progresses through five revisions. The final line of output should read:

INFO  [alembic.runtime.migration] Running upgrade ... -> 005, add generation schema

If Alembic reports no changes, the schema is already current. If it reports a connection error, confirm that DATABASE_URL in .env is correct and that the PostgreSQL container is healthy.

Step 6: Start the API service

uvicorn raw_to_knowledge.main:app --host 0.0.0.0 --port 8000

For development, add --reload to enable hot reloading on file changes:

uvicorn raw_to_knowledge.main:app --host 0.0.0.0 --port 8000 --reload

For production deployments, run with multiple workers and behind a reverse proxy:

uvicorn raw_to_knowledge.main:app --host 127.0.0.1 --port 8000 --workers 4

See Docker Deployment for containerized production setup.

Step 7: Verify the system is healthy

Health endpoint

In a new terminal (with the virtual environment activated), run:

curl -s http://localhost:8000/health | python3 -m json.tool

Expected response:

{
  "status": "ok",
  "version": "0.8.0",
  "db": "ok"
}

Field	What it means if not `ok`
`status`	Application failed to start. Check the uvicorn terminal for a Python traceback.
`db`	API cannot reach PostgreSQL. Verify `DATABASE_URL` and that the postgres container is running.

OpenAPI UI

Open http://localhost:8000/docs in a browser. The Swagger UI should load and list all API routes grouped by domain (intake, classification, validation, registry, generation, graph, connectors). This confirms the application is fully initialized.

Step 8: Start Celery workers (required for async features)

Several features — connector sync, answer generation, and graph projection — run as Celery background tasks. The API itself starts without workers, but those feature areas will not process work until the workers are running.

Open separate terminal windows (each with the virtual environment activated) and start the workers your deployment requires:

Connector sync worker (required to pull transcripts from Gong, Fellow, or Confluence)

celery -A raw_to_knowledge.infrastructure.celery_app worker \
  --queues raw_to_knowledge_connectors --concurrency 2 --loglevel info

Also start Celery Beat to run scheduled sync jobs:

celery -A raw_to_knowledge.infrastructure.celery_app beat --loglevel info

Generation worker (requires `OPENAI_API_KEY`, `WEAVIATE_URL`, `DATABASE_URL`)

celery -A raw_to_knowledge.infrastructure.celery_app worker \
  --queues raw_to_knowledge_generation --concurrency 4 --loglevel info

Graph worker (requires `DATABASE_URL`, `NEO4J_URI`, `NEO4J_USERNAME`, `NEO4J_PASSWORD`)

celery -A raw_to_knowledge.infrastructure.celery_app worker \
  --queues raw_to_knowledge_graph --concurrency 2 --loglevel info

Development shortcut — all queues on one process

For local development you can run all three queues in a single worker:

celery -A raw_to_knowledge.infrastructure.celery_app worker \
  --queues raw_to_knowledge_connectors,raw_to_knowledge_generation,raw_to_knowledge_graph --loglevel info

Verify workers are running

celery -A raw_to_knowledge.infrastructure.celery_app inspect ping

Each active worker should respond. If no workers respond, check that the Redis URL is correct and the redis container is healthy.

Step 9: Configure optional services

Skip any service you are not using in your deployment. The API and core ingest/validation/registry path works entirely on PostgreSQL and Redis.

Weaviate (semantic retrieval and answer generation)

Run Weaviate locally:

docker run -d --name weaviate -p 8080:8080 \
  -e AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true \
  -e PERSISTENCE_DATA_PATH=/var/lib/weaviate \
  semitechnologies/weaviate:latest

Set WEAVIATE_URL=http://localhost:8080 in .env and restart the API. Raw to Knowledge creates the ApprovedAnswer vector index schema automatically on first use — no manual schema setup is required.

Neo4j (contradiction detection and graph projection)

Run Neo4j locally:

docker run -d --name neo4j -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/password neo4j:5

Set NEO4J_URI, NEO4J_USERNAME, and NEO4J_PASSWORD in .env and restart the API. Node and relationship constraints are created automatically on the first graph sync task.

LLM provider (OpenAI-compatible)

Set OPENAI_API_KEY and optionally LLM_MODEL in .env and restart both the API and the generation worker. The system uses gpt-4o by default. Any OpenAI-compatible endpoint works by pointing OPENAI_API_KEY and the base URL at the provider.

For full connector configuration including Confluence and Jira, see Configure Connectors and Models.

Step 10: Configure the first transcript source connector

A connector tells Raw to Knowledge where to pull conversation transcripts from. Without at least one enabled connector, no transcripts will be ingested automatically.

Replace gong with the source system you are connecting (gong, fellow, or a custom connector name), and adjust the JSON payload to match your environment:

curl -X PUT http://localhost:8000/v1/connectors/gong/sync \
  -H "Content-Type: application/json" \
  -d '{
    "enabled": true,
    "cadence": "daily",
    "legal_basis": "legitimate_interest",
    "customer_scope": "acme-corp",
    "initial_since": "2026-01-01T00:00:00Z"
  }'

Field	Purpose
`enabled`	Set to `true` to activate scheduled pulls.
`cadence`	How often to pull. `daily` is the recommended starting point.
`legal_basis`	The privacy legal basis for ingesting these transcripts. Must be one of `legitimate_interest`, `consent`, or `contract`.
`customer_scope`	Optional. Scope ingested transcripts to a specific customer or segment.
`initial_since`	The earliest conversation date to pull on the first sync. Use ISO 8601 format.

Verify the connector is registered:

curl -s http://localhost:8000/v1/connectors | python3 -m json.tool

Trigger a manual pull to confirm the connector can reach the source system:

curl -X POST http://localhost:8000/v1/connectors/gong/pull

Check the connector sync worker logs for progress and any credential errors.

For connector-specific credentials (Gong API key, Fellow API token, etc.) and advanced connector configuration, see Configure Connectors and Models.

What works at each stage

Feature	Requires
Artifact ingest and extraction	PostgreSQL + Redis + API running
Classification	PostgreSQL + API running
Validation workflow	PostgreSQL + Redis + API running
Registry publish and readout	PostgreSQL + API running
Connector-based transcript pull	Above + connector worker + Beat + connector credentials
Semantic retrieval	Above + Weaviate
Answer generation	Above + Weaviate + `OPENAI_API_KEY` + generation worker
Contradiction detection	Above + Neo4j + graph worker
Confluence/Jira RAG	All four Confluence or Jira env vars set

Checklist

Use this checklist to confirm the system is fully operational before handing off to end users:

[ ] .env copied from .env.example and all required variables set
[ ] make install-dev completed without errors
[ ] make docker-up shows both postgres and redis as healthy
[ ] make migrate applied all five revisions (last line: -> 005, add generation schema)
[ ] API running; GET /health returns {"status": "ok", "version": "0.8.0", "db": "ok"}
[ ] GET /docs loads the Swagger UI
[ ] Celery workers started for the queues your deployment uses
[ ] celery inspect ping shows worker responses
[ ] First connector registered via PUT /v1/connectors/{system}/sync
[ ] Manual pull (POST /v1/connectors/{system}/pull) succeeds without errors
[ ] Optional: Weaviate running and WEAVIATE_URL set
[ ] Optional: Neo4j running and NEO4J_* vars set
[ ] Optional: OPENAI_API_KEY set and generation worker running

What's next

Task	Where to go
Configure additional connectors and LLM details	Configure Connectors and Models
Understand all environment variables	Environment Variables Reference
Set up production workers with process supervisors	Run Workers and Sync Jobs
Understand backup and retention obligations	Backup, Retention, and Audit
Set up Docker-based production deployment	Docker Deployment
Explain the system to end users	What Raw to Knowledge Is and Is Not
Understand roles in the system	Roles and Responsibilities