GUIDE · PIPELINE

Parse documents on the host, search them as memory.

Document Processor turns PDF / DOCX / TXT into structured markdown + JSON plus images-with-context. Consciousness Server holds chunks as training records. semantic-search retrieves them by meaning. None of it leaves the machine. This guide wires the three together end-to-end.

Prerequisites

  • A working ecosystem from the Quickstart (CS on :3032, semantic-search on :3037).
  • Document Processor installed (Tauri desktop app; installers for Linux — .deb / AppImage — other platforms build from source).
  • One document parsed. Drag a PDF onto the Document Processor window; wait for the green check.

What Document Processor produces

Each parsed document lands in a directory under the app data dir, resolved by Tauri's path resolver — on Linux ~/.local/share/com.buildonai.document-processor/przetworzone/<document-id>/, the platform equivalent elsewhere. The shape:

filesystem 
# Document Processor writes one directory per parsed document,
# under the app data dir (Linux shown):
~/.local/share/com.buildonai.document-processor/przetworzone/
└── <document-id>/
    ├── document.md          # human-readable markdown
    ├── document.json        # metadata + full_text + images[] with context
    ├── images/
    │   ├── img_001.png      # extracted images
    │   └── thumb_img_001.png
    └── original.pdf         # copy of the input file

You hand the directory off to the next stage. The document.json file is the workhorse — document metadata, the full extracted text, and per-image context records. There is no pre-chunked file; splitting full_text into retrievable units is the ingest script's job.

Ingest into Consciousness Server

Each chunk becomes a training record. The type field is required — for prose chunks use explanation, for clause-like or sectional content use architecture. Tag every record with the document id so you can scope searches later.

ingest.py 
import json, sys, requests
from pathlib import Path

CS = "http://127.0.0.1:3032"
DATA = Path.home() / ".local/share/com.buildonai.document-processor/przetworzone"
# Directory as argv, or pick an id by hand: ls the przetworzone/ directory
DOC = Path(sys.argv[1]) if len(sys.argv) > 1 else DATA / "<document-id>"

doc = json.loads((DOC / "document.json").read_text())

# document.json ships full_text, not pre-made chunks — chunking is your
# job. A paragraph split is a fine baseline.
chunks = [c.strip() for c in (doc.get("full_text") or "").split("\n\n") if c.strip()]

# Each chunk becomes a training record. The "type" field is REQUIRED.
# For document content, use "explanation" (a self-contained chunk of
# meaning) or "architecture" (a structural section like a contract clause).
for chunk in chunks:
    requests.post(f"{CS}/api/memory/training", json={
        "agent": "doc-pipeline",
        "type": "explanation",
        "goal": f"ingest:{doc['filename']}",
        "instruction": "search-retrievable chunk",
        "input": doc.get("doc_type") or "",
        "output": chunk,
        "tags": [doc.get("doc_type") or "document", "doc:" + doc["id"]],
    }).raise_for_status()

print(f"Ingested {doc['filename']} — {len(chunks)} chunks indexed.")

CS embeds each record into ChromaDB via Ollama on the host. Index size grows linearly with chunk count; embeddings are ~1.5 KB each, so a 10 000-chunk corpus is roughly 15 MB plus the ChromaDB overhead. All of it on local disk.

Retrieve by meaning, not filename

Once ingested, an agent finds the right clause without knowing what file it lived in:

search.py 
# Now an agent can find that contract by meaning, not filename.
hits = requests.post("http://127.0.0.1:3037/api/search", json={
    "query": "what penalty applies if delivery slips by 30 days",
    "limit": 5,
    "filters": {"tags": ["doc:" + doc["id"]]},
}).json()

for h in hits["results"]:
    print(f"score={h['score']:.2f}  {h['snippet'][:120]}")

filters.tags narrows the search to one document; remove the filter for a corpus-wide query. The score is cosine similarity (0..1).

Automate the ingest half

One thing to be clear about: Document Processor does not watch folders in the background. Its "watch folder" merely remembers a directory for a manual re-scan — parsing happens when you drop a file or hit "Scan again". What you can automate is the hand-off: every parse creates a new directory under przetworzone/, so a watcher on the output side turns "I parsed a file" into "the chunks are searchable" without any extra UI work:

watch.sh 
# Document Processor does NOT watch folders by itself — parsing runs
# when you drop a file on the window or trigger "Scan again". But every
# parse creates a new directory under przetworzone/, so you can watch
# the OUTPUT side and auto-ingest whatever you parse:
DATA=~/.local/share/com.buildonai.document-processor/przetworzone
inotifywait -m -e create "$DATA" | while read dir _ name; do
    # document.json is written near the end of a parse — wait for it
    for i in $(seq 1 30); do
        if [ -f "$dir$name/document.json" ]; then
            python ingest.py "$dir$name"; break
        fi
        sleep 1
    done
done

Wrap that in a systemd user unit (~/.config/systemd/user/doc-ingest.service) so it survives reboots. From the operator's point of view, dropping a PDF on Document Processor now also drops it into the corpus — the parse itself stays a manual act.

Next steps