Thoth is my private answer machine for business questions. I fed it thousands of hours of business teaching. When I ask it something hard, it does not guess. It finds the real words the experts said, and it shows me where they said them.
It runs all day on its own server, and my other AI agents use it as their shared memory. It has been torn down and rebuilt until it could win a blind test. The version running now beat the first version 63 to 11 with the labels hidden.
What it is
A knowledge graph over 4,639 transcripts (~30M tokens) built on a fork of Microsoft GraphRAG I maintain, with custom commits adding a direct query-to-text-unit retrieval channel and content-hash incremental updates. The graph holds 101,556 entities, 73,187 relationships, and 19,962 communities, extracted against a purpose-built domain ontology instead of the stock news-style entity types, with community reports generated lazily at query time rather than prepaid at index time.
How it runs
Served over an OpenAI-compatible API with SSE streaming, plus a raw-evidence endpoint that returns retrieval context with no answer-model call. The runtime is memory-bounded. A gateway supervises one worker with an RSS-ceiling recycle, quiet-hours unload, and timed prewarm. Steady-state retrieval runs about 2.5 seconds. A full answered question costs about ten cents; an incremental corpus update costs about nine cents.
How it was proven
It has been built three times, and the third build had to win its place. The gate was 40 stratified questions, two arms, judged blind by two independent judge families with the labels sealed. The third version beat the first 63-11-6, winning every stratum. Separately, the graph's numeric entities, all 27,469 metric and warning records, are deterministically audited against their source transcripts, with 4.54% flagged for review.
The rebuild that lost
The second version was the disciplined one: validated extraction, typed edges, strict prompts. It lost a blind test to the messy first version 8-2. The response was forensics instead of guessing. The corpus had quietly never downloaded most of three source catalogs. The extraction prompt literally ordered sparsity, a few entities per chunk and no more. On 132 matched chunk pairs from identical source text, the old version extracted 6.5× the entities and over 20× the relationships, and its supposed noise measured at 2.7% junk. The big graph was almost all real.
The cheap fixes came first: reallocating the context budget and porting the answer prompt flipped the result from losing 7-5 to winning 9-5 in one commit. The expensive fix, a full rebuild, was validated on a 50-document slice before committing, with knowledge survival rising from 23.6% to 40.4% and zero fabricated numbers in the sampled entities.
Every model seat earned blind
Each production model seat was decided by a sealed-key bake-off: answers written to disk, verdicts recorded, then the key unsealed. The answer seat took 63 calls across 8 candidates; the winner later defended it 12-0 in a rematch. The community-report seat ended in a 6-6 blind tie at 1.67× the cost and latency, so the cheaper setting shipped. One retrieval mode's reasoning setting was breaking a strict-JSON primer 12 times out of 12, so reasoning was turned off for that role. Three separate more-compute-is-not-better results landed in a single day, and none shipped.
The operator battery told the sharpest story. Local source-grounded retrieval beat summary-first retrieval 12-0, and on the hardest question both summary arms reported that no process existed while local retrieval found the actual conversation the summaries had abstracted away. One judge family is an opinion; two is a measurement.
Design rules that survived
Loose nodes, strict edges. The index wants recall, because entities are handles into source text. The structure wants precision, because relationships become communities. The graph is guarded by grounding rules and typed, evidence-prefixed edges, never by count caps.
Community reports generate lazily at query time instead of being prepaid at index time. Spoken-word transcripts mangle names, so aliases normalize at both index and query time. Builds run into a candidate directory and promote atomically only after validation, which is why a runaway mid-rebuild once died without touching the live graph. And the audit that checks the graph needed auditing itself: its first flag rate ran three times too high until the normalizer learned spoken number forms. Validators need validators.
What a question costs
A fully answered question costs about ten cents, with steady-state retrieval near 2.5 seconds. First-query wait after a reboot dropped from about a minute to about seven seconds once boot warmup started paying the cold-storage cost in advance. Adding new transcripts costs about nine cents per incremental update, and a full thousand-document build reconciled to under fifteen dollars in real, credit-verified spend.
The cheap answers sit on a heavy foundation. Creating Thoth consumed over 680 million tokens across more than 365,000 model requests, against a corpus that is itself about 30 million tokens. Between extraction, embeddings, rebuilds, and evals, the source material was effectively processed more than twenty times over, so that every question after it costs a dime.
Questions about this system, or the problem it could solve for you?
Discuss your project