Then it became an industry. Pinecone, Weaviate, Qdrant, Chroma, Milvus, pgvector, LangChain, LlamaIndex, Haystack. Tutorials everywhere, “build your chatbot with your PDFs,” entire consultancies feeding off this. It became the “hello world” of applied LLMs: document → chunk → embed → vector DB → query.

Today, in April 2026, Claude Opus 4.6 has 1 million tokens of context. Sonnet 4.6 too. Gemini 3.1 Pro too. GPT 5.4 has a smaller window but still in the comfortable range, in the hundreds of thousands. And some models already have experimental 2M token modes. The question that keeps nagging at me: what on earth do I need a vector stack for, to solve a problem that fits inside the model’s window?

And there’s more: vector databases have real problems nobody wants to talk about. False neighbors. Arbitrary chunking that splits a definition from its usage. Embeddings that age badly. Not to mention that when the result is wrong, you have absolutely no idea why.

The thesis I’ve been chewing on is simple: in most cases, a well-aimed grep plus a generous context window beats a full RAG stack. It’s cheaper, it’s easier to maintain, and when it breaks you can actually debug it. Let’s break this down.

What the Claude Code leak showed

Before we get into the theory, let me bring up something that happened a few days ago that backs this whole argument up. On March 31, 2026, Anthropic, by accident, published version 2.1.88 of the @anthropic-ai/claude-code package on npm with a nearly 60 MB source map attached, and roughly 512,000 lines of TypeScript from their internal tool leaked into the wild. I already wrote about the incident last week, with more detail on what showed up in the code.

The part that matters for this discussion is Claude Code’s memory system. Instead of dumping everything into a vector DB, the architecture has three layers. There’s a MEMORY.md that stays permanently loaded in context, but it doesn’t hold any actual data: it’s just an index of pointers, around 150 characters per line, kept under 200 lines and about 25 KB. The real facts live in “topic files” that get pulled on demand when the agent needs them. And the raw transcripts from previous sessions are never reloaded whole, only searched with grep, hunting for specific identifiers. No embedding. No Pinecone. Just write discipline (topic file first, index after) and lexical search. That’s it.

Claude Code’s main loop also has a tiered system for handling a context that’s filling up. As I detailed in the previous post, there are five different context compaction strategies, with names like microcompact (clears old tool results based on age), context collapse (summarizes long stretches of conversation), and autocompact (which fires when the context gets close to the limit). The CLAUDE.md file, which a lot of people thought was just a convention, is first-class in the architecture: the system re-reads it at every iteration of the query.

What this tells me: the best coding agent on the market right now, built by the company selling the most expensive model out there, does not use a vector DB. It uses files on disk, a markdown index, lexical search, and smart compaction strategies for when the context overflows. They could’ve slapped embeddings on top, they have the money to run whatever they wanted, and they chose not to. The reason, in my reading, is exactly what this post is arguing: to retrieve text from files you control, with generous context available, a vector DB is dead weight. Better to invest in compacting the window you already have than indexing everything into an external store.

There’s a curious security detail that came along with the leak: people noticed the compaction pipeline has a vulnerability they’re calling “context poisoning.” Content that looks like an instruction, coming from a file the model reads (say, a CLAUDE.md from a cloned repo), can end up being preserved by the compaction model as if it were “user feedback,” and the next model takes that as a real user instruction. It’s a new attack vector. But that’s a topic for another post.

The “Dream” system and memory consolidation

But what really caught my eye for the RAG debate, which I unpacked in detail last week, is the system called autoDream. It’s a forked subagent, with read-only bash access to the project, that runs in the background while you’re not using the tool. Its job is literally to dream: to consolidate memory. The name isn’t accidental, and the obvious analogy (which I couldn’t resist) is the human brain consolidating memory during sleep, turning short-term experience into something more stable.

For a dream to actually run, three gates have to open at once: 24 hours since the last dream, at least 5 sessions since the last dream, and a consolidation lock that prevents concurrent dreams. When it fires, it goes through four phases. Orient (does an ls on the memory directory, reads the index). Gather (looks for new signals in logs, stale memories, transcripts). Consolidate (writes or updates the topic files, converts relative dates into absolute ones, deletes facts that have been contradicted). And Prune, the final cleanup that keeps the index under 200 lines.

The decision to make autoDream a forked subagent is the detail that matters here. It does not run in the same loop as the main agent. Why? Because memory consolidation is a noisy process. The model has to re-read old transcripts, compare them against what’s in MEMORY.md, decide what stays and what goes, form hypotheses about things it saw in earlier sessions. If that ran in the main context, it would pollute the “train of thought” of the agent that’s trying to help you with your current task. By forking, you keep the two separate. The main agent stays focused on what you asked for, and autoDream does the housekeeping in parallel, with no write permission on the project.

And the way it figures out what needs to be consolidated is plain old lexical search. The transcripts live as JSONL files on disk, and autoDream uses grep to look for new signals. Just grep, on text logs. Stop and think about that for a second. The memory consolidation of the most advanced agent in the world, built by one of the richest AI companies out there, is a forked subagent running grep on text logs. If a vector DB were the right answer for this kind of problem, Anthropic would’ve put a vector DB in there. They didn’t.

And there’s a detail that, to me, is the buried gold of the entire leak, and it fits this argument like a glove. In autoDream, memory is treated as a hint. The system assumes that what’s stored may be stale, wrong, contradicted by something that happened later, and the model has to verify before it trusts it. The vector DB pitch is the opposite of that: index everything, search by similarity, return the top-k, trust the result. Claude Code went the conservative route. Index little, search by word, return a hint, and stay skeptical until you’ve laid eyes on the actual fact.

The whole strategy works in two layers. Inside a single session: generous context plus grep plus smart compaction (microcompact, context collapse, autocompact). Between sessions: a subagent that consolidates memory asynchronously, using grep on the transcripts and treating the result as a tip, not as truth. Embeddings and vector DBs don’t show up in either layer. The deliberate choice was a smart reader chewing on raw text, not a dumb reader being spoonfed the top-k of an embedding.

The practical lesson for our debate is simple. The most advanced agents on the market are heading toward generous context, lexical search, and smart compaction, not toward classic RAG pipelines. If Anthropic, with all the infrastructure and talent they’ve got, picked this path for Claude Code, those of us building internal applications on a fraction of that budget should at least think about going the same way.

Where the story started turning

When the ceiling was 32k of context, retrieval was the bottleneck of the entire problem. You had to pre-filter aggressively, because anything that made it into the window was sacred space. A vector DB was the only halfway-decent way to do that semantic pre-filtering. The logic was: “the reader (LLM) is expensive and dumb, so the retriever has to be smart and selective.”

Today the equation has flipped. The reader is now the smartest one at the table, and the window grew big enough to hold an entire document. So the retriever can (and maybe should) go back to being dumb. The dumber, the better. You want high recall and low precision, and you let the model do the fine work. Grep does exactly that. So does BM25. And ripgrep flies through millions of lines without breaking a sweat.

And this isn’t just my hunch. The BEIR benchmarks have shown for a while now that BM25 matches or beats a lot of dense retrievers when the domain drifts away from where the embeddings were trained. Anthropic itself published a post on Contextual Retrieval that basically says the same thing: a lexical signal plus an LLM’s judgment beats pure embeddings on most knowledge tasks. And take a look at Claude Code, the tool I’ve been using every day for 500 hours: it navigates the repo with Glob and Grep. No vector DB, no embedding, no LangChain. It works ridiculously well.

The real problems with vector databases nobody advertises

The vector DB marketing sells the dream of perfect semantic search. Reality is messier.

False neighbors come first. Cosine similarity rewards topical similarity, not relevance. You ask “how do we handle authentication errors” and the DB returns every chunk that mentions authentication. The chunk that actually answers the question may be in tenth place, or may not have been retrieved at all because the doc author wrote “login” instead of “auth.”

Chunking is the second one, and it’s a disguised disaster. A 512-token window with a 64-token overlap sounds reasonable, until you realize your important table got cut in half, the function definition ended up separated from its usage, and the piece of documentation with the exact command got orphaned without the context of its section. The chunk boundary tends to land exactly where the answer was living.

When it fails, it fails without leaving a trace. When BM25 misses, you know why: the word isn’t there. When a vector DB returns garbage, you get a plausible-looking wrong chunk, with no diagnostic signal at all. Good luck debugging that in production at two in the morning.

The index gets stale. Every document update calls for re-embedding. If you have 10,000 docs and 200 of them change per day, that turns into a batch process, monitoring, a queue, retries, embedding API costs, and an unavoidable inconsistency window between what’s on disk and what’s in the index. Grep has none of that. File changed? The next query already sees it.

And there’s the operating cost nobody adds up. Pinecone charges per vector. Weaviate wants a cluster to maintain. pgvector saves you a new server but you still own a schema, an index, and a re-embedding pipeline. Each of those things wants engineer time, monitoring, tests, deploys. All of that to do a search that rg would often crack in 200ms.

Comparing the complexity

Look at the diagram:

On one side, eight steps, four or five services, an external index that needs to be maintained and kept up to date. On the other, four steps, zero new infrastructure. This isn’t a caricature: it is literally what you have to set up for each case.

The honest question: does the left column pay off? In 2023, yes, because the right column didn’t exist (no LLM had a 200k window). In 2026, in most cases, it doesn’t.

Pros and cons of each side

Classic RAG (vector DB)

For:

• Works for huge document bases, on the order of hundreds of GB, where even rg won’t cut it without prior indexing
• Handles heavy paraphrase and cross-lingual queries (“how do I cancel” vs. “subscription termination process”) where the user’s vocabulary doesn’t match the document’s
• Works for non-textual modalities (image, audio) where grep has nothing to look at
• Saves input tokens if you’re tight on budget or absolute latency

Against:

• Complex stack: embedding, vector DB, chunking, reranker, re-indexing pipeline
• Opaque failures, hard to debug
• Chunking destroys the context of tables, code, long definitions
• Operational overhead (index, queue, monitoring, re-embedding cost)
• The semantic search the marketing is selling rarely works the way the marketing promises

Grep + long context

For:

• Practically zero new infrastructure: ripgrep, sqlite, or a plain LIKE in Postgres
• Always fresh: file changes, the next query sees them
• Transparent failures: the word is either there or it isn’t
• Loads the document in generous chunks, the model does the fine filtering with actual semantics
• Cheaper in dev and ops, cheaper to pivot domains

Against:

• Doesn’t scale to terabytes of raw text without some kind of indexing
• Suffers when the user’s vocabulary is very different from the document’s
• Doesn’t work for non-textual modalities
• Per-query latency is higher in absolute terms (loading 100k tokens always costs more than loading 5k)
• Per-query input cost is higher if you don’t have prompt caching

But what about cost?

This is the argument I get hit with the most when I defend the “load everything into context” thesis. “It’ll get crazy expensive, 200k tokens of input per query is absurd.” Let’s actually run the numbers.

In yesterday’s LLM benchmark post I mapped out the per-token price of every model. Take Claude Sonnet 4.6: $3 per million input tokens, $15 per million output. Take GLM 5 (which I proved actually works): $0.60 input, $2.20 output. Take GPT 5.4 Pro at the top of the heap: $15 input, $180 output (yeah, that one stings, I know).

Before we turn “200k tokens” into dollars, let’s land that number on something tangible, because “100k tokens” doesn’t mean anything to anyone. A token, on average, is roughly 0.75 of a word in English (Portuguese is similar, maybe a touch heavier because of longer words). So, translating:

• 100k tokens ≈ 75,000 words ≈ a whole short novel like Hemingway’s The Old Man and the Sea with room to spare, or about three long Wikipedia articles glued together.
• 200k tokens ≈ 150,000 words ≈ a big novel, like Crime and Punishment in full, or half of the first Game of Thrones book (which clocks in around 298k words, so roughly 400k tokens).
• 400k tokens ≈ 300,000 words ≈ A Game of Thrones in full, the entire first book of the series in your window.
• 1M tokens ≈ 750,000 words ≈ the entire Lord of the Rings trilogy plus The Hobbit, or the whole Bible (King James is around 783k words, roughly 1M tokens), or about two and a half Game of Thrones books stacked on top of each other.

So when I say “throw 200k tokens of input at the model,” what that actually means in the real world is “throw the entire Crime and Punishment in as the context for your question.” That’s a lot. And that’s exactly what makes the argument of this post viable: today’s models can read an entire novel in one go and still answer a specific question about it. In 2023, this was science fiction. In 2026, it’s the base case.

So picture a query that throws 200k tokens of input at the model (there goes Crime and Punishment again) and produces 2k tokens of output (about three pages of response):

Now throw prompt caching into the mix. Claude has a cache that drops cached input to a fraction of the full price (in the ballpark of 10%, depending on the model). Gemini has a similar mechanism. When you fire a sequence of queries against the same 200k-token dump, the cost of subsequent queries plummets to pennies. With Sonnet cached, you can fairly call it about $0.10 per follow-up query without making things up.

Now compare that to the cost of running a Pinecone, or a Weaviate, or a pgvector. Setting aside the price of the subscription itself (which varies a lot), you need an engineer to wire up the pipeline, maintain it, monitor it, deal with embedding failures, redo the chunking when the domain shifts. Conservatively, you’re looking at somewhere between 40 and 80 hours of engineering to make the thing stable. At R$ 200/hour, that’s between R$ 8,000 and R$ 16,000. In USD, somewhere between $1,600 and $3,200 just to stand it up.

With $3,200, on Sonnet 4.6 with prompt caching, you can run something on the order of 30,000 queries of 200k tokens each. Thirty thousand queries, depending on the scale of the project, gives you several months or even an entire year of an average internal tool. And you didn’t pay an engineer to wire up a pipeline. There’s no vector DB server to maintain. And if the document changes, the system already sees it on the next query.

The “RAG is cheaper in tokens” argument ignores that tokens are the cheapest thing in the entire equation. Engineers cost a lot, servers cost a lot, bugs in production cost a whole lot more. Tokens have become a commodity, and they’re getting cheaper with every new model release.

The classic RAG argument was “the model is expensive, retrieval is cheap.” Today it’s the opposite: the model is the cheap part of the stack, smart retrieval is what costs a fortune to build and maintain.

Where the thesis doesn’t hold

I don’t want to come off as a fanboy. There are cases where classic RAG still wins:

1. Massive corpora. If you have 500 GB of raw text, even rg won’t solve it in acceptable time. You need some kind of indexing. It can be indexed BM25 (Tantivy, Elasticsearch), it can be a vector DB. But notice: the first option is still lexical, not vector.
2. Wildly scattered vocabulary. Customer support, where the user types “my wifi’s down” and the documentation says “loss of connectivity at the physical layer.” BM25 won’t catch that. Embedding will. Vector DB scores a point here.
3. Non-textual modalities. Image-by-image search, audio-by-audio. Embedding is mandatory.
4. Critical absolute latency. If you have to answer in 100ms with a 5k input budget, a generous dump won’t fit. Pre-filtering is necessary.
5. Compliance and audit. If you have to prove that a specific document was consulted to answer a specific query, having indexed and trackable chunks helps. A 200k-token context dump is more opaque from an audit standpoint.

For those cases, classic RAG still makes sense. But notice the size of the list. These are specific cases. The general case, things like “chat with our internal docs” or “ask the product manual,” almost all of it falls into the “grep + long context handles it better” bucket.

Lazy retrieval: the recipe I’d defend

If I were building a “chat with docs” tool today, from scratch, it would look more or less like this:

1. Keep the documents raw. Markdown, converted PDF, code, whatever. On disk, organized in folders that make sense for the domain.
2. Fast lexical filter. ripgrep with regex, or BM25 with Tantivy/SQLite FTS5, or a LIKE in Postgres if you already have one. Returns 100-300 hits.
3. Load generously. Grab not just the matching snippet, but the entire file, or a wide window around it. Throw all of it into the context.
4. Let the LLM do the fine work. Pass the original question, tell the model to find what matters, drop the rest, and answer with citations.
5. (Optional) Add embeddings only for the query classes where lexical fails, after you have real data showing that it fails.

This is the opposite of the old advice (“start with vectors, fall back to keyword”). It’s: start with keyword, and add vector only if you feel the gap. In most projects, you never will.

A toy implementation in Ruby

To make it concrete. Here’s a Ruby script using the ruby_llm gem (the same one from yesterday’s benchmark) that does exactly this flow: grep through the files, load the snippets with context, send to Claude, get the answer back. No vector DB, no chunking, no embedding, no LangChain.

#!/usr/bin/env ruby
require "ruby_llm"
require "open3"

DOCS_DIR = ARGV[0] || "./docs"
QUERY    = ARGV[1] or abort "uso: ./ask.rb <pasta> <pergunta>"

# 1. Fast lexical filter with ripgrep.
#    -i case insensitive, -l file names only, --type-add covers md/txt/extracted-pdf.
def lexical_search(dir, query)
  terms = query.downcase.scan(/\w{4,}/).uniq.first(8)  # words with 4+ letters
  pattern = terms.join("|")
  cmd = ["rg", "-l", "-i", "-e", pattern, dir]
  files, _ = Open3.capture2(*cmd)
  files.split("\n").reject(&:empty?)
end

# 2. Load entire files (up to a reasonable cap).
def load_context(files, max_chars: 600_000)
  total = 0
  files.map do |path|
    body = File.read(path)
    next if total + body.size > max_chars
    total += body.size
    "## #{path}\n\n#{body}\n"
  end.compact.join("\n---\n")
end

# 3. Send to Claude with the question and the documents.
def ask(query, context)
  chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4-6")
  prompt = <<~PROMPT
    Você tem acesso aos documentos abaixo. Responda a pergunta do usuário
    usando apenas o que está nos documentos. Cite o nome do arquivo nas
    referências. Se a resposta não estiver nos documentos, diga isso.

    --- DOCUMENTOS ---
    #{context}
    --- FIM DOS DOCUMENTOS ---

    Pergunta: #{query}
  PROMPT
  chat.ask(prompt).content
end

files = lexical_search(DOCS_DIR, QUERY)
abort "nenhum arquivo bateu" if files.empty?
puts "Encontrei #{files.size} arquivos. Carregando contexto..."
context = load_context(files)
puts ask(QUERY, context)

About 40 lines. No Pinecone dependency, no vector schema, no re-indexing pipeline. You run it as ./ask.rb ./docs "how do I configure the payment webhook" and that’s it.

That example is one-shot. You run it, it answers, done. For a real chat, with multiple questions in a row over the same documents, the design changes. Instead of running lexical_search upfront and shoving everything into the context at once, you expose the search as a tool to the model. Then it’s the agent that decides when it needs to pull more docs, what term to look for, which file is worth opening in full. That’s how Claude Code actually works: Glob, Grep and Read are tools, and the model picks the sequence. ruby_llm supports tool calling, so you can do the same thing in Ruby. It looks something like this:

require "ruby_llm"
require "open3"

DOCS_DIR = "./docs"

class SearchFiles < RubyLLM::Tool
  description "Procura arquivos cujo conteúdo casa com o padrão dado (regex). Retorna lista de paths."
  param :pattern, desc: "Padrão regex pra busca lexical (case-insensitive)"

  def execute(pattern:)
    out, _ = Open3.capture2("rg", "-l", "-i", "-e", pattern, DOCS_DIR)
    out.split("\n").reject(&:empty?)
  end
end

class ReadFile < RubyLLM::Tool
  description "Lê o conteúdo completo de um arquivo do projeto."
  param :path, desc: "Caminho relativo do arquivo"

  def execute(path:)
    File.read(path)
  rescue => e
    "erro: #{e.message}"
  end
end

chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4-6")
            .with_tools(SearchFiles, ReadFile)
            .with_instructions(<<~SYS)
              Você responde perguntas sobre os documentos em #{DOCS_DIR}.
              Use search_files pra encontrar arquivos relevantes e read_file
              pra ler o conteúdo. Sempre cite o arquivo na resposta.
            SYS

loop do
  print "> "
  msg = gets&.chomp
  break if msg.nil? || msg.empty?
  puts chat.ask(msg).content
end

The model gets the question, decides whether it needs to search, calls search_files, sees what came back, decides whether it needs to open any file, calls read_file, and only then answers. On the next question it already has the previous context in the session and can ask for more if it needs to. The context only receives what the model asked for, not the whole grep dump from the earlier example.

The same idea works for databases: swap rg for a SQL query with LIKE or tsvector (Postgres full-text), load the relevant rows, throw them in the context. If you have 10k records in an internal database, this handles it. If you have 10 million, you start needing smarter pagination or a more serious pre-filtering layer. But the mental model is the same: dumb filter + smart reader.

The point that matters

The most interesting thing in all of this isn’t even the Pinecone savings. It’s that the nature of the bottleneck has changed. In 2023, the bottleneck was retrieval: the reader was small, slow, expensive, and you needed a clever retriever to fill the window with the bare minimum. In 2026, the bottleneck is reasoning over messy context: the reader is big, relatively fast, and cheap. So it makes more sense to have a dumb retriever with high recall and let the model do the heavy lifting.

Anyone still designing systems with the 2023 mindset is paying a premium to solve a problem whose shape has changed. RAG didn’t die, the “R” got dumber and cheaper, and that’s an upgrade. The vector DB vendors aren’t going to tell you this, but it’s the path the more experienced folks have been quietly walking.

The next wave of LLM applications, in my bet, is going to be dominated by the people who got this inversion. Smaller stacks, simpler infrastructure, generous context, and a whole lot less LangChain.

What the recent literature says

Before I close out, I went and checked what the research crowd published on this. Blog hot takes age in three months in this field, so it’s better to look at the papers.

Retrieval Augmented Generation or Long-Context LLMs?, out of Google DeepMind, published at EMNLP 2024, is probably the most cited piece in the debate. Their conclusion: when the model has enough resources, long context beats RAG on average quality, but RAG is still much cheaper in tokens. They propose Self-Route, an approach where the model itself decides whether it needs retrieval or whether it can just go straight through context. The token savings are big and the quality loss is small.

Then LaRA, presented at ICML 2025, is more measured. The authors built 2326 test cases across four QA task types and three long-context types, ran them across 11 different LLMs, and the conclusion was: there is no silver bullet. The choice between RAG and long context depends on the model, the context size, the task type, and the retrieval characteristics. RAG wins on dialogue and generic queries, long context wins on Wikipedia-style QA.

Long Context vs. RAG for LLMs: An Evaluation and Revisits, from January 2025, is the one that most reinforces this post’s thesis. Long context tends to beat RAG on QA benchmarks, especially when the base document is stable. Summarization-based retrieval comes close, and chunk-based retrieval lags behind. In other words: the old way, chunk plus embed plus top-k, is the one that comes out worst.

Worth keeping on the radar too is the original Lost in the Middle (Liu et al., 2023, published in TACL in 2024). That’s the paper that showed even models with big windows have performance that depends on the position of the relevant information. Stuff at the beginning or end of the context is found easily; stuff in the middle degrades. For a long time this got used as the argument against long context, but the paper is from 2023, with 2023 models. Today’s models, the Claude 4.x and Gemini 3.x line, handle the middle a lot better. It’s not a solved problem, but it’s much smaller than it was.

On the lexical retrieval side, BEIR is still the canonical reference. The classic result is that BM25, all the way from the 90s, is still ridiculously competitive in out-of-domain scenarios. Dense models only win consistently when you have in-domain data to fine-tune the embeddings. In zero-shot scenarios, which is where most projects live, BM25 is hard to beat without serious work.

To wrap up, the Anthropic post on Contextual Retrieval, from September 2024, is the most practical piece on the list. They show that combining contextual embedding with contextual BM25 drops the top-20 failure rate from 5.7% to 2.9%. Add a reranker and it drops to 1.9%. Important detail: BM25 is the centerpiece of their result, not a sidekick. The right reading is “lexical plus vector plus reranker is the combination that works.” Anyone who can only pick one picks BM25 and still gets pretty far.

To sum up what we can actually nail down: the literature isn’t claiming “RAG is dead.” It’s saying that long context, when you can use it, tends to win on quality. It’s saying RAG’s cost is still its main argument. It’s saying lexical BM25 is much stronger than the vector DB marketing makes it sound. And it’s saying that when you really do need heavy retrieval, the robust combination is hybrid (lexical plus vector plus reranker), not pure vector. All of that lines up with what I’ve been defending in practice.

Sources

• Li, Z. et al. (2024). Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. EMNLP 2024 Industry Track.
• Yuan, K. et al. (2025). LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing. ICML 2025.
• Yu, T. et al. (2025). Long Context vs. RAG for LLMs: An Evaluation and Revisits. arXiv:2501.01880.
• Liu, N. F. et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. TACL 2024.
• Thakur, N. et al. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks 2021.
• Anthropic (2024). Introducing Contextual Retrieval. Blog post.
• Akita, F. (2026). Claude Code’s Source Code Leaked. Here’s What We Found Inside. — my coverage of the leak, with more detail on the memory architecture, KAIROS and autoDream.

There’s a new wrinkle in this update: I redid the local part of the benchmark on an RTX 5090 (instead of the AMD Strix Halo) and added a fresh batch of Qwen models, including a Qwen 3.5 27B distilled directly from Claude 4.6 Opus. That reopened the conversation on running open source models locally. The 5090’s memory bandwidth flips the game from “unworkable” to “workable with 1-2 follow-up prompts.” The bottleneck for open source models has moved to a lack of factual knowledge about specific libraries, which I unpack in detail in the new section on the Qwen family. The Claude distillation gamble, by the way, gave a pretty frustrating result that I haven’t seen documented in these terms before.

If you’ve been following my previous vibe coding pieces, you know I spent the last two months in a 500-hour marathon using Claude Opus as my main coding agent. The results were good, as I reported in the conclusion about business models. But there was an itch I couldn’t scratch: am I locked into one model? Is there a real alternative to Claude Opus for daily use on real projects?

I’ve got an RTX 5090 with 32 GB of GDDR7. I know I can run the latest open source models. I bought a Minisforum MS-S1 with an AMD Ryzen AI Max 395 and 128 GB of unified memory, and built a home server with Docker to serve local models. The infrastructure was ready. What was missing was actually testing it.

I built an automated benchmark to compare open source and commercial models under identical conditions. 33 models configured in total (25 from the original run plus 8 added in the NVIDIA rerun), 27 executed, 16 completed in some form. The code is on GitHub.

The bottleneck nobody explains: VRAM and KV Cache

Before getting to the results, I have to explain why running large models locally is much harder than it looks.

Take Qwen3 32B. The model in FP16 (full precision) takes ~64 GB. Quantized to Q4 (4 bits), it drops to ~19 GB. So it fits in my RTX 5090’s 32 GB, right? Wrong. That’s just the model weights. There’s a part nobody tells you about: the KV Cache.

KV Cache is the memory the model uses to “remember” what it has already read. Every time it processes a token (a word or piece of a word), it computes two vectors — K (key) and V (value) — for every attention layer. Those vectors stick around so the model doesn’t have to recompute everything when it generates the next token. Without that, generation would be quadratically slow.

The KV Cache scales linearly with the size of the context. The formula:

KV Memory = 2 × Layers × KV_Heads × Head_Dimension × Bytes_per_Element × Context_Tokens

For a model like Llama 3.1 70B in BF16, that comes out to ~0.31 MB per token. Sounds tiny, until you realize that a 128K context eats 40 GB of KV Cache alone. The model itself plus KV Cache adds up to way more VRAM than most GPUs have.

And for actual coding agent use, 128K tokens isn’t a luxury, it’s the bare minimum. The agent has to read files, keep conversation history, receive command output. In long benchmark sessions, our models consumed between 39K and 156K tokens. Less than 100K of context isn’t practical for day-to-day project work.

Google published TurboQuant (ICLR 2026), which compresses the KV Cache to 3 bits without accuracy loss — a 6x memory reduction and up to 8x speedup. It uses random vector rotation (PolarQuant) followed by a 1-bit algorithm on the residuals. Works online during inference, compressing on write and decompressing on read. Not yet implemented in the runtimes we use (llama.cpp, Ollama), but when it lands it’ll change the equation a lot.

For anyone wanting to dig deeper into the VRAM math, I recommend this link from Ahmad Osman for the article “GPU Memory Math for LLMs (2026 Edition)”.

The hardware problem: not all memory is created equal

“But I have 128 GB of RAM!” Cool, but that’s not what matters. What matters is memory bandwidth, and the difference between types is wild:

The RTX 5090 has 7x the bandwidth of the LPDDR5x memory in my Minisforum. That means even if a model fits in the AMD’s unified RAM, inference will be proportionally slower. On my Minisforum with LPDDR5x at 256 GB/s, Qwen3 32B runs at ~7 tok/s. On the RTX 5090 at 1,792 GB/s, it’d be much faster — if it fit entirely in VRAM alongside the KV Cache.

Most folks running local models are still on DDR4. At 50 GB/s, 32B models are basically unusable. And there’s another factor people forget: storage. When the RAM can’t keep up and the system swaps, the storage speed becomes the bottleneck:

From SATA to NVMe Gen5 you’re looking at a 22x difference. If you’re doing partial offloading to disk (which is common when the model doesn’t fit entirely on the GPU), NVMe Gen4 or Gen5 makes a real difference. SATA is a non-starter.

To sum up: running local models isn’t just “having enough RAM.” You need the right kind of memory, with the right bandwidth, and fast storage as a fallback. For a lot of people, a Mac Studio with high-bandwidth unified memory (up to 800 GB/s on the M4 Ultra with 512 GB) would be the more practical option, but it costs more than US$ 10,000. The AMD Ryzen AI Max is the cheaper alternative with unified memory, but its LPDDR5 caps out at 256 GB/s.

Ollama vs llama.cpp: why Ollama falls apart on benchmarks

Ollama is the most popular way to run local models. Install, pull the model, run. For casual use it works. But when I tried to use it for automated benchmarks with long unattended sessions, it broke in 6 different ways across 8 models:

1. Unloads the model mid-session. On long runs, Ollama decides the model isn’t being used and unloads it from the GPU. The agent sits there waiting for a response from a model that no longer exists.
2. Ignores the requested context. You ask for num_ctx=131072, Ollama accepts, then halfway through the run it reverts to the default without warning.
3. Unstable lifecycle. Asking for keep_alive: 0 to unload doesn’t always work. The model stays resident and blocks the next one.
4. Incompatible formats. Native bf16 variants on Ollama failed, while the same model as a Q8 GGUF from HuggingFace worked fine.

The fix: migrate to llama-swap, a Go wrapper that manages llama.cpp processes with hot-swap. A request comes in for a different model than the one currently loaded, it kills the current process and starts the new one. No context negotiation, no flaky lifecycle.

llama-swap fixed the loading of 6 of the 8 models that had failed under Ollama:

But llama-swap isn’t magic.

Why “just use llama.cpp” doesn’t fix everything

llama.cpp solves Ollama’s lifecycle problems but brings its own:

Each model needs specific flags. GLM and Qwen 3.5 emit <think> tags that break clients if you don’t pass --reasoning-format none. Gemma 4 needs build b8665+ for the tool call parser to work.

Not every model supports tool calling. llama.cpp needs a dedicated parser for each model’s tool call format. Llama 4 Scout uses a “pythonic” format ([func(param="value")]) that llama.cpp simply doesn’t parse and emits as plain text. vLLM has a parser for it, llama.cpp doesn’t.

And then there are the repetition loops. Gemma 4, even with the right parser, gets into an infinite loop after ~11 tool calls in long sessions. It’s a known bug that PR #21418 didn’t fully fix.

Tool calling compatibility per model:

At the end of the day, llama.cpp is better than Ollama for automated runs, but “plug and play” it ain’t. Each model requires specific configuration, and some just don’t work for agentic coding yet.

Reasoning: models that think vs models that wing it

There’s one difference between models worth explaining: reasoning. The idea is that the model “thinks before answering” instead of generating tokens straight from left to right. Models with reasoning go through an internal chain-of-thought step where they evaluate the problem, consider alternatives, plan, and only then emit the response.

In practice this shows up as <think>...</think> tags in the output, blocks of text the model writes to itself that shouldn’t go to the end user. Claude Opus 4.6, GPT 5.4, DeepSeek V3.2 and the Qwen 3.5 line support reasoning natively. The smaller ones (Gemma 4, GPT OSS 20B, older models) don’t have that capability.

Why does it matter for coding? When a coding agent gets “build a Rails app with 9 components,” it has to decompose the task into steps, decide the order, anticipate dependencies, adapt when something fails. Without reasoning, the model generates code sequentially with no planning. It works for simple tasks, falls apart on projects with interdependent parts.

In the benchmark, the difference was clear:

• GPT OSS 20B (no reasoning, 20B parameters) created the app in the wrong directory. Couldn’t keep workspace instructions in mind while generating code.
• Qwen 3 32B has reasoning, but at 7 tok/s it was too slow. The “thinking” tokens drag out the generation time.
• Gemma 4 31B, with no reasoning trained for agentic use, fell into repetitive tool calling loops.
• GLM 5 (cloud, 745B MoE) with reasoning and 44B active parameters, finished cleanly and used the correct API.

There’s a trade-off: reasoning consumes extra tokens (the <think> blocks), which take up VRAM in the KV Cache and slow generation down. That’s why flags like --reasoning-format none are needed in llama.cpp. Some clients don’t know what to do with reasoning tokens and break. Models that emit reasoning when the runtime isn’t expecting it can produce garbage in the output.

And reasoning isn’t something you “turn on” in any model. It’s a capability trained with reinforcement learning on top of the base model, using data from problems that require multi-step thinking. The smaller open source models (20B-35B) typically didn’t go through that training, or went through it on a smaller scale. They know how to generate code, but they don’t know how to plan code. On tasks that require 50+ coordinated tool calls, that difference is fatal.

The benchmark: methodology

To compare models fairly, I built an automated harness in Python. Each model gets the exact same prompt: build a complete Ruby on Rails application, a ChatGPT-style chat SPA using the RubyLLM gem, with Hotwire/Stimulus/Turbo Streams, Tailwind CSS, Minitest tests, CI tools (Brakeman, RuboCop, SimpleCov, bundle-audit), Dockerfile, docker-compose and README.

The runner is opencode, which at the time of the benchmark was the most popular open source coding CLI, competing with Claude Code and Codex. Since then the project has been archived and development continued as Crush, maintained by the original author together with the Charm team (the folks behind Bubble Tea, Lip Gloss and several other Go terminal tools). If you read my piece on Crush, you already know it. Crush inherits everything from opencode — support for 75+ providers, LSP, MCP, persistent sessions — and adds the polished aesthetic that’s a Charm trademark. It runs everywhere: macOS, Linux, Windows, Android, FreeBSD.

I actually tried to use Crush for the benchmark first. The problem: it advertised a --yolo flag in its help to auto-approve every action (essential for unattended automated runs), but at runtime it rejected the flag. Without auto-approve there’s no way to do an unattended benchmark. opencode, on the other hand, had the opencode run --agent build --format json mode that emits JSON events with session IDs and token counts, perfect for automation. So we went with opencode.

I picked opencode (and not Claude Code or Codex) for two reasons:

1. Neutrality. Claude Code is optimized for Anthropic models. Codex is optimized for OpenAI models. opencode is agnostic, same interface for all.
2. Automation. opencode exposes a machine-readable JSON format. Claude Code and Codex don’t have an equivalent interface for external benchmarking.

Cloud models ran in two phases: phase 1 (build the app) and phase 2 (validate local boot, docker build, docker compose). Local models only ran phase 1.

Worth mentioning: the entire benchmark cost less than $10 in tokens on OpenRouter. Apart from GPT 5.4 Pro which torched $7.20 to fail, the other 11 cloud models added up to about $2.50 total. Local models cost only electricity. The point is: running your own benchmark is cheap. If you want to know whether a model works for your use case, drop the $2 and test it. The harness code is on GitHub, just swap the prompt for your own project.

Why GPT 5.4 failed the benchmark (but not in real life)

GPT 5.4 Pro is the only cloud model that consistently failed our benchmark. Two separate runs, same result: the model generated files but never reached finish_reason: stop. It always ended on finish_reason: tool-calls — wanted to keep calling tools but the loop kept breaking.

For folks who don’t know: tool calling is when an LLM needs to perform an action (read a file, run a command, edit code) and emits a “tool call” in a structured format. The client (opencode, Claude Code, Codex) interprets it, executes it, and returns the result back to the model. Each provider has its own format: Anthropic uses tool_use blocks, OpenAI uses function_calling with proprietary JSON schemas, Google uses FunctionCall.

GPT 5.4 is heavily trained for OpenAI’s native function calling format — tool_choice, tools with proprietary JSON schemas. When the benchmark routes through opencode → OpenRouter → GPT 5.4, the tool schemas get translated at every hop. If GPT emits tool calls in a format that OpenRouter or opencode doesn’t parse correctly, the agent loop breaks.

The evidence: every other cloud model (Claude Opus, Claude Sonnet, Kimi K2.5, DeepSeek V3.2, MiniMax M2.7, GLM 5, Qwen 3.6 Plus, Step 3.5 Flash) ended on finish_reason: stop. Only GPT ends on finish_reason: tool-calls.

A fair comparison for GPT 5.4 would require running it in its native environment — Codex or ChatGPT Pro ($200/month). On opencode through OpenRouter, this isn’t a fair test of GPT’s coding ability. That said, I used Codex extensively during my vibe coding marathon and I can vouch that GPT 5.4 is as good as Opus for real projects. In some ways I actually prefer Codex: it tends to think more “outside the box” and arrives at more creative solutions than Opus. On the other hand, it’s less disciplined — tends to forget previous instructions in long sessions and sometimes wanders off scope. Opus is more predictable and methodical. For me, that predictability is worth more day to day.

Sonnet and Opus through opencode/OpenRouter were probably also not pushed to their limits. Claude Code offers native tool support that opencode doesn’t replicate — meaning the benchmark results represent a floor, not a ceiling, for those models.

Open source models: reality vs the narrative

A lot of people are saying open source models have already caught up with the commercial ones and you can run your own “Claude” at home. In practice, not really.

The scale isn’t comparable. Frontier models like Claude Opus 4.6 and GPT 5.4 are closed-source, but estimates put them in the hundreds of billions to trillions of parameters range, trained with compute and data no open source company can replicate. The best models that fit on reasonable hardware are:

Post-publication correction: Qwen 3.5 35B is actually the 35B-A3B, an MoE with only 3B active parameters per token (not dense, as I’d originally written). That’s why it runs relatively fast for its size. And for folks with 24 GB of VRAM, the model recommended by Unsloth themselves is the Qwen 3.5 27B dense — that one I didn’t get around to testing in the benchmark, but it’s worth a look. For anyone wanting to dig deeper into local models, @sudoingX has been doing some serious experimentation in this space. Thanks to @thpmacedo for the heads-up.

Even the largest open source MoE (Mixture of Experts) models that companies make publicly available activate only a small fraction of parameters per token:

These large models aren’t self-hostable. Kimi K2.5 with 1T parameters needs GPU clusters with hundreds of GBs of VRAM. GLM 5 with 745B is the same. Even if Alibaba or Z.AI release the weights (and some do), nobody has home hardware to run them.

What fits on your home GPU are the 20B-35B models — and those have real limitations.

What each local model did in the benchmark

Results from the original run on the AMD Strix Halo:

Qwen 3 Coder Next (30B) — Completed in 17 minutes on the Strix, generated 1675 files, Rails app with all the artifacts. But only 3 tests. And more importantly: it invented RubyLLM::Client.new, a class that doesn’t exist in the gem. The app doesn’t run.

Qwen 3.5 35B — Completed in 28 minutes on the Strix, 1478 files, 11 tests. Used RubyLLM.chat without a model parameter — works only if the default is configured. No LLM mocking in the tests.

Qwen 3.5 122B — Completed in 43 minutes on the Strix, 1503 files, 16 tests. But it ignored the RubyLLM gem completely and built a custom HTTP client for OpenRouter. The prompt explicitly asked for ruby_llm.

GLM 4.7 Flash (local, Strix) — Produced 2029 files with all the artifacts, but the session ended mid-tool-call. The cloud version (GLM 5) works perfectly.

Gemma 4 31B (Strix) — Infinite tool call loop after ~11 productive steps. Known llama.cpp bug.

GPT OSS 20B (Strix) — Created the Rails app in the wrong directory (project/app/ instead of project/). A 20B model doesn’t follow workspace instructions reliably.

Qwen 3 32B (Strix) — Way too slow (7.3 tok/s). The hardware can’t keep up.

And the results from the rerun on the NVIDIA RTX 5090 (all with Q3_K_M or Q4_K_M and contexts between 64k and 128k to fit the 32 GB of VRAM):

Qwen 3.5 35B-A3B (5090) — 5 minutes at 273 tok/s. Recognizable Rails project, entry point RubyLLM.chat(model:) is right, but it hallucinates chat.add_message(role:, content:) and chat.complete instead of .ask. Fixable in 1-2 follow-ups. The best candidate for “OSS local that’s actually worth trying.”

Qwen 3.5 27B Claude-distilled (5090) — 12 minutes at 129 tok/s. Impeccable Claude style, total API hallucination (RubyLLM::Chat.new.with_model{}, add_message, response.text). More details in the distillation section below.

Qwen 3 Coder 30B (5090) — 6 minutes at 145 tok/s. Returned a hardcoded mock string instead of calling the API. Tier 3 unusable.

Qwen 2.5 Coder 32B (5090) — 90 minutes of timeout, zero files. The model spun without ever calling a write tool.

Qwen 3 32B (5090) — 4 minutes at 69 tok/s, partial scaffold, errors. The general version is better than the Coder one but still breaks.

Gemma 4 31B (5090) — 8 minutes at 213 tok/s. Same repetition loop it had on the Strix. The llama.cpp bug isn’t a hardware issue.

Qwen 3.5 27B Sushi Coder RL (5090) — Infrastructure failure (ProviderModelNotFoundError), couldn’t be evaluated. Redo on a future run.

GPT OSS 20B (5090) — Pulled from this run because of a recent llama.cpp main regression in the harmony family tool call parser. The logs show Failed to parse input at pos 755: <|channel|>... in multi-turn sessions. It worked on the Strix with llama.cpp b8643, broken on today’s main. Waiting on upstream to fix it.

Cloud models: what actually works

Of the 12 models that completed the benchmark, all of them generated a recognizable Rails project with all the requested artifacts (Gemfile, routes, views, JS, tests, README, Dockerfile, docker-compose). 9 out of 9 on the completeness checklist.

But here comes the question that matters: does the code run?

The correct RubyLLM API is simple:

chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4")
response = chat.ask("Hello")
response.content  # => "Hi there!"

8 of the 12 models invented APIs that don’t exist. The most common pattern: hallucinating an interface that doesn’t match the actual gem:

The models that got it right — both Claudes, GLM 5 and GLM 5.1 — used the simple two-step pattern (chat = RubyLLM.chat(model:) then chat.ask(message)). The ones that got it wrong tried to make RubyLLM look like the OpenAI Python SDK, which is a different thing. Grok 4.20 was the most brazen case: it didn’t even try to use the gem, it went straight for OpenAI::Client pointing at the OpenRouter URL, ignoring the explicit prompt.

And the tests? Only Opus, Sonnet, GLM 5 and GLM 5.1 did proper mocking of the LLM calls. All the others either hit the real API (which fails without a key) or mocked the invented API (tests pass but prove nothing). Test count is a misleading metric: Kimi K2.5 wrote 37 tests, more than anyone else, but none of them test real functionality because the API it uses doesn’t exist.

Real viability table

*Step 3.5 Flash works by calling the OpenRouter REST API directly with Net::HTTP, completely bypassing the gem the prompt asked for.

Now, this doesn’t mean those models are useless. If you take Kimi K2.5 or DeepSeek V3.2 and tell it “the RubyLLM::Client class doesn’t exist, fix it to use the gem’s real API”, it’ll probably fix it. One or two follow-ups and the project becomes functional. Most of the models that failed here could deliver a working project with a few more rounds of conversation.

But that’s where the trade-off lives. With Opus or GPT 5.4, the first output already works. You ask, they deliver, you test, it runs. With the cheaper models, you’ll spend time fixing API hallucinations, debugging code that “looks right” but crashes, steering the model in the right direction. Each of those rounds is 10-30 minutes. Three extra rounds and you’ve spent an hour of your time to save $0.90 in tokens.

You save dollars, you spend time. And time is money. For someone learning or exploring without urgency, that trade can make sense. For someone who needs to ship, the frontier models pay for themselves fast.

Comparing the models that work

Full ranking by time and tokens

DeepSeek V3.2 is the slowest despite being cloud — it has no prompt caching, so it resends the full context on every turn.

Token efficiency and cache

Models with prompt caching pay much less in effective tokens:

Speed: the chasm between cloud and local

There’s an aspect that the cost tables hide: inference speed. And the difference is brutal.

Claude Sonnet generates 532 tok/s. Qwen 3.5 122B running locally on my Minisforum (AMD Strix Halo) generates 22 tok/s. That’s a 24x difference. In practice, what Sonnet does in 16 minutes, Qwen 3.5 122B takes 43 minutes. Qwen 3 Coder Next at 37 tok/s is the fastest of the local models on the Strix and even so it’s 14x slower than Sonnet.

And it’s not just clock time. When you’re in an interactive coding loop — ask for a change, wait for output, test, ask for another — the model’s speed sets your rhythm. At 37 tok/s, every long response makes you wait 30-60 seconds. At 530 tok/s, it appears almost instantly. Over a day, you feel it.

DeepSeek V3.2 is a curious case: it’s cloud but it runs at 53 tok/s, slower than the locally-running Qwen 3.5 35B on the Strix (46 tok/s). The reason is that DeepSeek has no prompt caching — it resends the full context on every turn, strangling throughput. Paying for a cloud model that’s slower than running it locally doesn’t make any sense.

Local models are free in tokens, but they pay in time. On the AMD Strix, that math was a non-starter for every Qwen I tested: two minutes waiting for a long response, multiplied by 50 turns, eats your whole afternoon. But that changes when the hardware changes, and that’s why I redid the local part of the benchmark on a different machine.

AMD Strix Halo vs NVIDIA RTX 5090: what changes when the memory bandwidth doubles

To check whether the bottleneck was hardware or model, I took the same Qwen models and reran the benchmark on a workstation with an NVIDIA RTX 5090 (Blackwell, 32 GB GDDR7, 1,792 GB/s bandwidth). The numbers shift in a way that’s worth looking at carefully.

To put those speeds in context, remember that in the cloud Sonnet runs at 532 tok/s, Opus at 347 tok/s, Step 3.5 Flash at 242 tok/s, Gemini 3.1 Pro at 128 tok/s and Kimi K2.5 at 160 tok/s. Qwen 3.5 35B-A3B on the 5090, at 273 tok/s, is in the same neighborhood as Step 3.5 Flash, faster than Gemini, Kimi and GLM 5.1. Qwen 3 Coder 30B at 145 tok/s is in Gemini territory. The classic line “local models are ten times slower than cloud” stopped being true the moment the 5090 entered the conversation.

The practical consequence is that the “time is money” argument shifts. On the Strix, “waiting an hour for a Qwen 3.5 122B to do what Sonnet does in 16 minutes” is straight-up loss. On the 5090, waiting 5 minutes for Qwen 3.5 35B-A3B to do the work, plus 10-15 minutes for you to do 1-2 correction prompts, gives you a total in the 20-25 minute range. Sonnet does it in 16 minutes with zero corrections. The difference shrank enough that, if cost matters a lot, it’s worth it.

The catch: for this to be worth it, the model has to be close enough to the right answer that 1-2 correction prompts can fix it. When the error is “the model decided not to use the gem I asked for and returned a hardcoded mock string,” like Qwen 3 Coder 30B did, no easy correction prompt fixes that. That’s a redo.

Before you spend money on hardware thinking it’s the answer

I’ve got to give a warning here, because it’s the most common buying mistake I see right now. Every other week somebody tells me they’re going to grab a Ryzen AI Max because it has 128 GB of unified memory and that “lets you run huge models.” Technically, sure — the model fits. In practice, it’s almost unusable. The memory is LPDDR5x at 256 GB/s, seven times slower than the 5090’s GDDR7. What fits doesn’t run at human speed. My own Strix with Qwen 3.5 122B hit 22 tok/s and the run took 43 minutes. To do anything serious day to day, that’s not workable.

The 5090 is clearly superior, and it starts to make sense even for smaller models precisely because of the memory bandwidth. A Mac Studio with high-speed unified memory (up to 800 GB/s on the M4 Ultra) is the other viable option, and costs proportionally the same. But neither of those comes anywhere close to beating the commercial models on quality — and the per-token price of Claude, GPT or GLM, combined with their brutal inference speed, makes the math hard to justify for anyone who isn’t an enthusiast or a researcher. Expensive local AI hardware is a weekend hobby, a tool for people who need to run offline for compliance reasons, or a research playground. For day-after-day production work, right now, cloud is still the rational choice. A 128 GB Ryzen AI Max may look tempting on the spec sheet, but if the goal is serious coding agent work, it’s money badly spent.

The Qwen family: Coder vs General, distillation, and why nothing is a silver bullet

With so many different Qwens running in this rerun, it’s worth doing a more focused analysis. What I learned might surprise people who follow model benchmarks on Twitter.

Before getting to the results: what quantization is and what distillation is

These two concepts come up constantly in this discussion and they deserve a quick explanation.

Quantization is the technique of compressing the model’s weights so they take up less memory. A model trained in FP16 (16 bits per weight) can be quantized to Q8 (8 bits), Q4 (4 bits), Q3_K_M (3 bits, but with medium-sized groupings), and so on. Each step halves the size of the model on disk and in VRAM, at the cost of some loss of precision. Q8 is practically lossless. Q4 already loses something measurable. Q3 loses more. Q2 is the line where the model starts saying real nonsense. The rule of thumb is that for coding and multi-step reasoning, you want to stay at Q4 or higher. Q3_K_M is the minimum that still works for many models, and it’s what fits a 27B on the 5090 with 128k context.

The surprise from my test, and look, this goes against the consensus, is that quantization wasn’t the bottleneck here. I ran the Qwen 3.5 27B Claude-distilled in two versions: Q8 on the AMD Strix (~27 GB of weights) and Q3_K_M on the 5090 (~12 GB of weights). Both hallucinated exactly the same fake RubyLLM APIs. Q3_K_M even produced a cleaner Gemfile. The model’s limitation was in what those weights know, not in the precision they were compressed to.

Distillation is the technique of training a smaller model (the “student”) to imitate the output or behavior of a larger model (the “teacher”). The classic version is logit distillation — the student learns to approximate the teacher’s probability distributions. The modern, more popular version for coding agents is distillation of reasoning traces: you take chain-of-thought from the big model on real problems and train the smaller one to reproduce the same reasoning style.

The hype of the moment is distilling Claude and GPT into open source models. The promise is that you can have “Claude-at-home” running locally. I wanted to test this, and that’s why I added Jackrong’s Qwen 3.5 27B distilled from Claude 4.6 Opus to the benchmark. If any open source model was going to use RubyLLM correctly, this was the bet — after all, in the entire benchmark, Claude and GLM 5 are the only ones that get the API right.

What the Claude-distilled learned (and what it didn’t)

I ran the same distillation twice: once at Q8 on the AMD Strix (which blew through the 90-minute timeout), and once at Q3_K_M on the 5090 (completed in 12 minutes). Both produced the same elegant frustration.

The code that comes out looks like Claude. It has # frozen_string_literal: true at the top of every file. It has a separate Response class as a value object with explicit attribute readers. It has a clear separation between service, controller and model. It has doc comments at the top of every file. It correctly comments out things like active_record, active_job and action_mailer in application.rb. It has defensive case statements trying multiple return formats. Stylistically, it’s Claude.

Functionally, it’s a complete RubyLLM hallucination. Look at the service generated by the 5090 run:

RubyLLM::Chat.new.with_model(@model) do |chat|
  conversation_history.each do |msg|
    chat.add_message(role: :user, content: msg[:content])
  end
  response = chat.ask(message)
  Response.new(content: response.text, usage: build_usage(response))
end

Every primitive in this code is invented:

• RubyLLM::Chat.new — the constructor isn’t public, the correct entry is RubyLLM.chat(model:)
• .with_model(@model) do |chat| ... end — there’s no block API like that
• chat.add_message(role:, content:) — doesn’t exist
• response.text — the real API exposes response.content
• response.usage.prompt_tokens — the object doesn’t have that shape

This will blow up with a NoMethodError on the first request. The initializer also tries config.openrouter_api_base= which doesn’t exist on RubyLLM.configure, so the app probably won’t even boot.

The Q8 version on the AMD Strix does the exact same thing, with one difference: the entry call is RubyLLM.chat(model:, provider: :openrouter) — the entry point is right, but provider: is invented and it’s immediately followed by the same fake chat.add_message(role:, content:). Worse, the Gemfile from the 90-minute run lists gem "ruby-openai" (wrong gem!), gem "minitest", "~> 6.0" (minitest 6.0 doesn’t exist) and gem "tailwindcss" (wrong gem name, it’s tailwindcss-rails). The Gemfile doesn’t include the gem the service code itself is trying to use.

For comparison, look at the actual Claude Opus 4.6 baseline, in the same benchmark, getting it all right:

@chat = RubyLLM.chat(model: model_id)
response = @chat.ask(message)
response.content

Twelve lines in the entire service. Zero hallucination. Includes streaming via block. The distilled model produced three times the code volume and got the API wrong.

The honest reading is that distillation transferred one layer and stopped. The layer that came along was the style: code organization, comments, class structure, the order of things. The layer that got left behind was factual memory about specific libraries. That makes sense when you think about it: Claude’s reasoning traces, even when written carefully, rarely contain repeated references to chat.ask(msg).content in some obscure Ruby gem. The student only learns what the teacher repeats, and Claude never had any reason to keep whispering “use ask, not complete” throughout its chains of thought. Library API knowledge is binary recall memory, the kind that’s either in the weights or it isn’t. Decomposing that into reasoning steps is impossible because it isn’t reasoning, it’s just raw memorization.

To wrap up the practical recommendation: if you need the model to actually use RubyLLM, or any less-popular library for that matter, Claude distillation won’t save you. Use real Claude or GLM 5. The “Claude-stand-ins” in open source will fail the same way the Qwen base would, just with prettier handwriting.

Coder vs General: the surprise of the “for coding” models

Almost everyone’s instinct is that models with “Coder” in the name are the best for programming. Makes sense, they were specifically fine-tuned on code. But in the benchmark, it was exactly the opposite.

Of the three dedicated Coders, two failed catastrophically (full timeout and hardcoded mock string) and one didn’t even run properly because of an infra bug. Meanwhile, the Qwen 3.5 35B-A3B, which is the general model in the line (not the Coder), came closest to something usable: 5 minutes of execution, recognizable Rails project, and the problem is fixable in 1-2 prompts.

Qwen 3 Coder 30B is particularly disappointing. It went so far past trying to use the API that it didn’t really try at all: the controller it generated has this:

class Api::V1::MessagesController < ApplicationController
  def create
    render json: {
      response: "This is a mock response. In a real implementation, this would connect to RubyLLM with Claude Sonnet via OpenRouter."
    }
  end
end

The Gemfile lists gem "ruby_llm" but nothing imports it. The service layer is nonexistent. The model decided it was easier to return a fake string and call it a day. That’s Tier 3 garbage in a way no correction prompt fixes — you have to tell it to start over.

Qwen 2.5 Coder 32B is even worse: 90 minutes running, zero files. The 1.8 MB opencode-output.ndjson shows the model spinning without managing to write anything. It probably got stuck in a planning loop without ever calling the write tools. Total slot waste.

Why did the “Coder” Qwens do so badly? My read is that the coding-specific fine-tuning they got was trained on more isolated problems (Codeforces, Leetcode, short snippets), far from agentic flows with long-running tool calling. The general Qwen 3.5 35B-A3B has broader training and handles the orchestration part better. The popular intuition “Coder = best for coding agent” is wrong for this kind of task. The use case where Coders shine is “complete an isolated function,” which is exactly what they were trained for, and that’s a tiny fraction of what a coding agent does day to day.

The question I wanted to answer

It was this: running locally on the 5090, which Qwen model is worth the 1-2 correction prompts to deliver code that works?

The honest answer is: only Qwen 3.5 35B-A3B, and maybe the Claude-distilled if you don’t mind spending 12 minutes more.

• Qwen 3.5 35B-A3B on the 5090: 5 minutes, correct entry point (RubyLLM.chat(model:)), errors on the subsequent calls. Realistic total until it works: in the 15-20 minute range with 1-2 follow-ups. Beats cloud OSS on cost.
• Qwen 3.5 27B Claude-distilled on the 5090: 12 minutes, deeper hallucination (entry point is invented too). Realistic total: 25-30 minutes with 2-3 follow-ups. Still competes on cost, and loses on absolute time to the real Claude.
• The others (Coder 30B, Coder 2.5 32B, 3 32B): don’t pay back the correction time. Each one has a structural problem that calls for a full rewrite from scratch.

For folks with hardware in this category who want to escape Anthropic vendor lock-in, it now works. It didn’t work on the 5090 from last year, and forget about it on the Strix Halo. In 2026, on NVIDIA Blackwell, with the right model, it works. For folks with low-bandwidth hardware (LPDDR5x, DDR4, DDR5), it’s still a waste of time: the clock alone takes down any plan to make this practical.

The Deep Code Review: Sonnet vs GLM 5 vs Gemini vs Kimi vs MiniMax

The tables above measure structural completeness. But does the project work? I did detailed code review of the models that completed the benchmark.

Claude Sonnet 4.6 — works and is the most complete. Synchronous responses via Turbo Stream. Chat history persisted in a session cookie with full replay of previous messages on every request. Correct LLM mocking in the tests with mocha (30 tests in 328 lines). LLM logic extracted into a separate LlmChatService. Views decomposed into 9 partials. Minor problems: duplicated model constant, leak in the auto-resize event listener. None are blockers. Of the generated projects, it’s the closest to something you’d actually put into production.

GLM 5 — works, but it’s the bare minimum. Uses the correct API (RubyLLM.chat(model:) then .ask()), does mocking with mocha in the tests. But the project is way leaner than Sonnet’s: 21-line controller (vs Sonnet’s 52), no service layer (LLM logic inline in the controller), no chat history persistence, every message handled in isolation. The first message works, but the app doesn’t keep conversation context, so you can’t have a multi-turn dialog. The tests exist (7 methods) but they’re skeletal: ruby_llm_test.rb only checks that the module is loaded, chat_flow_test.rb is a copy of the controller test. The Dockerfile, on the other hand, is the best of the four: multi-stage, non-root, jemalloc. But as a chat app? It’s more of a proof of concept than something functional. Funny detail: the README says “Powered by Claude Sonnet 4” instead of the model that actually generated the project.

Gemini 3.1 Pro — the fastest, but trips on the API. Completed in 14 minutes, the fastest along with MiniMax. The Rails code itself is well written: uses Rails.cache with session ID and a 2-hour expiration to keep state (instead of a database), Turbo Streams nicely integrated, Stimulus controller for auto-scroll, and the Dockerfile is the best of the group (multi-stage, non-root, jemalloc). The problem is the usual one: it uses RubyLLM::Chat.new() instead of RubyLLM.chat(), and calls add_message() which doesn’t exist. The app boots, Docker runs, the health check passes, but the first chat message returns 500. The tests (5 methods) mock with a FakeChat that replicates the wrong signature, so they pass. It’s frustrating because the rest of the code is the most “Rails way” of the non-Anthropic models. Fixing it would be 3 lines, but the benchmark measures what comes out the first time.

Kimi K2.5 — ambitious but broken. Tried the most sophisticated architecture: ActionCable streaming, configurable models, dual Dockerfiles, 37 tests in 374 lines. Problem: the streaming depends on ActionCable, which is commented out in config/application.rb. The return unless defined?(ActionCable) guard makes the method do nothing. The assistant never responds. The Stimulus controller has a scope bug: submitTarget references a button outside the controller’s subtree. Thread-unsafe storage with a hash in a class variable. Kimi wrote more tests than any other model (37), but none of them mock the LLM calls — so the tests pass without proving any of the functionality works.

Grok 4.20 — fast and wrong. It was the fastest in the entire benchmark: 8 minutes, 412 tok/s. Except it was fast because it cut corners. The prompt explicitly asked for the ruby_llm gem, and Grok ignored it. It went straight for OpenAI::Client from the ruby-openai gem pointing at the OpenRouter URL. Technically the first message comes back, so yeah, it “works.” But it’s the same trick as Step 3.5 Flash and Qwen 3.5 122B: skip the part that was actually being tested. No history, 33-line controller calling the HTTP client by hand, two tests, no real mocks. It was fast because it did less than what was asked.

MiniMax M2.7 — looks right, crashes. Calls RubyLLM.chat(model: '...', messages: [...]) — that signature doesn’t exist. No message persistence. Duplicated HTML (DOCTYPE inside the layout). Committed master.key. And the tests? They mock the wrong API, so they pass but they don’t prove anything.

Code review summary:

GLM 5 vs GLM 5.1: what changed

GLM 5 was one of the few models that spat out functional code on the first try, so it was obvious to test the new version. One important detail before the numbers: GLM 5 ran via OpenRouter, GLM 5.1 wasn’t there yet when I ran this test, so I used the Z.AI direct API. Different provider, different infra, different cache. The numbers below are reference, not exact measurement.

The GLM 5.1 project came out way more complete. 24 tests vs 7. Real separation between ChatSession, ChatMessage and the controller, instead of GLM 5 cramming everything inline. Chat history persisted in memory during the session, so you can actually have a real multi-turn conversation (GLM 5 treated every message like it was the first). And the RubyLLM API is still correct, the same RubyLLM.chat(model:, provider:) pattern followed by c.user/c.assistant to build the context. There’s even a test covering the MODEL constant, which usually nobody does.

The price was speed. 22 minutes vs 17, and throughput dropped from 400 to 167 tok/s. Could be the provider (Z.AI direct isn’t the same infra as OpenRouter), could be a more loaded server during the run, could be that 5.1 reasons more. I didn’t run it multiple times to take an average, so I won’t say 5.1 is “slower.” A single run doesn’t prove a regression. What I can say is that, in my test, 5.1 delivered a better-structured project and took a bit longer to do it.

For folks who want to get out from under Anthropic without losing quality, GLM 5 and GLM 5.1 are the two options that work. If you need centralized billing on OpenRouter, GLM 5. If you can use Z.AI direct and want a more rounded project on the first try, GLM 5.1.

Costs: API vs Subscription

First, the per-token price of each model on OpenRouter:

GPT 5.4 Pro charges $180 per million output tokens. Claude Opus charges $25. GLM 5 charges $2.30. And Qwen 3.6 Plus is free (with a rate limit). The log scale on the chart hides some of the brutality of the gap: from free Qwen to GPT 5.4 Pro is orders of magnitude.

But per-token price isn’t the whole story. If you use Claude or GPT daily for coding, the monthly subscription can come out way cheaper than paying per token via the API:

*Estimate for moderate coding use (~15M input + ~3M output tokens/month).

The main point: if you use GPT 5.4 Pro, the ChatGPT Pro subscription at $200/month with unlimited use is 5x cheaper than paying per token on the API. For Claude, Pro at $20/month covers light use, but for heavy users (a coding marathon like mine), the Max 20x at $200/month comes out cheaper than paying for Opus per token on OpenRouter (~$450/month). The open source models on OpenRouter all sit below $2.50/M output tokens, but as we saw, most of them generate code that doesn’t run.

What works for real use

After testing 33 models across both runs and looking at the generated code in detail:

Tier 1 (works plug and play):

*GPT 5.4 Pro failed the automated benchmark because opencode doesn’t support OpenAI’s native tool calling format. Through Codex or ChatGPT Pro ($200/month with unlimited use), it works without problems.

Tier 2 (works with caveats):

Tier 3 (broken code, easier to redo than to fix):

Kimi K2.5, MiniMax M2.7, DeepSeek V3.2, Gemini 3.1 Pro, Qwen 3 Coder Next (Strix), Qwen 3 Coder 30B (5090, returned a hardcoded mock string), Qwen 3.5 122B, Qwen 3.6 Plus — all of them either invent APIs that don’t exist or don’t even try to use the gem.

Tier 4 (didn’t complete):

Gemma 4 (infinite loop on both hardware), Llama 4 Scout (no parser), GPT OSS 20B (wrong directory on Strix, parser regression on 5090), Qwen 3 32B (too slow on Strix, partial scaffold on 5090), Qwen 2.5 Coder 32B (90m timeout with zero files).

Simplified ranking (quality, time, price)

For folks who only want the report-card summary. Quality is whether the code runs and how complete it is. Time is the total runtime. Price is the estimated cost per execution on opencode. Hardware indicates where the model ran — Cloud, Strix (AMD Strix Halo, LPDDR5x 256 GB/s) or 5090 (NVIDIA RTX 5090, GDDR7 1792 GB/s). Cloud models ran via OpenRouter or the provider’s direct API.

Quality criteria: A+ works and the code is well structured. A/B works with small to medium caveats. C runs but skips a prompt requirement or has a serious structural issue. D breaks on the first message because of an invented API. F didn’t complete the benchmark or produced garbage. GPT 5.4 Pro stays at A+ for real use in Codex, but didn’t run in this benchmark, hence the dash in time. “Type” separates commercial models (closed weights) from OSS (open weights, even when used through a hosted API). Some Qwens appear twice when they ran on both hardware profiles, because the results are different enough to justify it — Qwen 3.5 35B-A3B on the 5090 jumps to Tier B, on the Strix it stays at Tier C because of the wait time. Of the 33 models configured across both runs, some don’t appear in this table because they never even executed (no quota, broken runner, infra failure, or timeout before the first message).

The verdict

If cost matters and you want to leave Anthropic: GLM 5 or GLM 5.1 are the plug-and-play alternatives that work. Correct API, mocking in the tests, ~$0.11-$0.13 per run, ~88-89% cheaper than Opus. GLM 5.1 delivered a more complete project (24 tests, chat history) at the cost of about 5 more minutes.

If you want the best result regardless of cost: Claude Sonnet 4.6 beat Opus in this benchmark — cheaper, same speed, more tests, code that works. But there are two important caveats before you generalize that conclusion.

First, this result is on opencode, not on Claude Code. In the native environment (Claude Code), where Opus and Sonnet have access to Anthropic’s full tool support, Opus might do better. In my 500-hour Claude Code marathon, I used Opus and the experience was consistently good.

Second, and this is the bigger one: our test is a small, well-defined web app. Sonnet 4.6 and Opus 4.6 share the same 1M token context window, so what separates the two is the reasoning capacity they can apply inside that context. Opus 4.6 has a 128K max output token ceiling vs Sonnet’s 64K, and its training was specifically aimed at long-horizon tasks, multi-step planning and deep reasoning over complex code. On a small project like ours, those muscles stay idle, and in that scenario it’s either a tie or Sonnet wins by being faster. In larger projects, with weeks of work, big monorepos, architectural decisions that carry real consequences, that’s where the actual difference between Opus and Sonnet shows up. You can’t conclude that Sonnet is better than Opus in general just from this benchmark.

If you want to avoid total vendor lock-in and you have decent hardware: Qwen 3.5 35B-A3B running locally on an NVIDIA RTX 5090. Five minutes of execution at 273 tok/s, a Rails project that boots, and the API error fixes itself in 1-2 follow-ups. Realistic total until it works: ~15-20 minutes. Beats Sonnet on cost (zero) and lands close on total time. This option simply didn’t exist in the previous round of the benchmark, and it marks the point where “running OSS local” stops being a toy and becomes a real alternative. Important: this is specific to hardware with high memory bandwidth. On an RTX 4090 it should work similarly. On a laptop with LPDDR5x or a desktop with DDR4, forget it — you’ll wait 10x longer and the total time kills the argument.

If you want to avoid vendor lock-in but you’re on weak hardware: GLM 5 or GLM 5.1 remain the choice. They’re cloud, true, but at $0.11-$0.13 per run it’s basically the price of electricity.

If you want to test the “Claude at home” gamble via distillation: the Qwen 3.5 27B Claude-distilled is sitting there to play with, but I already warned you it hallucinates exactly the same fake APIs as the base Qwen. Distillation transferred Claude’s style, not its factual knowledge about libraries. It’s worth it as an experiment, not as production.

Yes, maybe with days of tweaking llama.cpp, calibrating flags, adjusting prompts, testing different builds, you could make Gemma 4 or other models work better. For most people, that isn’t realistic. The distance between frontier models (Claude, GPT) and self-hosted open source models is real. It isn’t marketing. The gap is shrinking, but it still exists, and the nature of it has changed: today what’s missing in open source is factual knowledge about specific libraries, not raw reasoning capacity. Hardware stopped being the bottleneck, at least for anyone with a recent GPU.

In the end, what matters is whether the code runs. A model can generate 3,405 files, write 37 tests, produce a 181-line README, and the app still won’t work because the API it uses doesn’t exist. Completeness metrics and test counts are necessary but not sufficient. The only reliable signal is whether the model uses real APIs correctly.

The full benchmark, with code, configuration, prompts and per-model results, is on GitHub.

I’ve always loved karaoke. I go out to sing with family or friends every now and then. In São Paulo there are good places in Liberdade and Bom Retiro, for instance, with private Japanese-style booths. If you’ve never been to a karaoke like that: you rent a private room by the hour, there’s a huge song catalog, two microphones, and a scoring system that grades your singing in real time. The best systems are Japanese, like Joysound and DAM. A score above 90 (out of 100) is considered advanced. DAM, in the LIVE DAM Ai series, even uses AI to give scores that feel more “human.”

But not every place has that level.

The problem with karaoke in Brazil

In Brazil we grew up with Videokê, the brand the Korean Seok Ha Hwang brought to the country in 1996, importing equipment from Korea. It became a craze in the 90s and 2000s, showed up in every bar, barbecue and birthday party. The problem is that those machines stopped in time. The current models, like the VSK 5.0, ship with around 12-13 thousand songs in the catalog, which you expand by buying cartridges or song packs. In practice, the repertoire is old, the interface is straight out of the 2000s, and if the song you want to sing came out after 2015, good luck.

The workaround a lot of bars adopted was to allow Chromecast or screen mirroring so that customers can search for songs directly on YouTube. Makes sense: on YouTube you can find karaoke for any song. With-lyrics version, instrumental version, vocal guide version.

But there’s a downgrade: you lose the scoring. One of the most fun parts of karaoke is the competition. Watching your score climb, comparing with friends, trying to beat the night’s record. If you’re just singing on top of a YouTube video, you get no feedback. It’s like bowling without a scoreboard.

And buying a professional system for home? Importing a Joysound F1 runs north of US$ 2,000 just for the hardware, not counting the monthly catalog subscription. For casual use it makes no sense.

The idea: YouTube with real-time scoring

Frank Karaoke came out of that frustration. If YouTube already has every song, why not build an app that works as a YouTube wrapper with a real-time scoring overlay? You search for any karaoke video, sing along, and the app analyzes your voice through the mic and shows a live score.

It’s a Flutter app for Android. Internally it loads YouTube into a webview and injects an overlay in HTML/CSS/JavaScript right into the page. The score display, the pitch trail, the settings panel, the mode selector — all of it rendered inside the webview through JS injection.

Scoring without a reference

Now, the real problem. Every professional karaoke system depends on prebuilt reference files for each song. Every single one.

Sony’s SingStar, which sold over 12 million copies between 2004 and the end of the PS3 era, had a hand-crafted note track for every song. Every note, every syllable, all mapped manually. The mechanism compared the singer’s pitch via FFT against that reference in real time. A detail I thought was clever: octave was ignored. If the right note was a C, it didn’t matter if you sang C3 or C4. Men sing women’s songs no problem.

Joysound and DAM in Japan go further and evaluate three separate dimensions: pitch accuracy (音感), rhythm/timing (リズム感) and expressiveness/dynamic volume (表現力). All based on MIDI data from the operator’s server. The open source equivalent format is UltraStar, where each song has a .txt file like:

: 12 4 5 Hel-    (NoteType StartBeat Duration Pitch Syllable)

Pitch 5 = MIDI 65 (F4). Scoring compares the singer’s pitch against the note’s pitch, modulo octave, with a tolerance of 1 semitone.

Frank Karaoke works with any YouTube video. There’s no reference file. There’s no MIDI. There’s no melody annotation. Zero metadata about what note you’re supposed to be singing.

I don’t know anything about karaoke scoring. I don’t know anything about audio processing, pitch detection, music theory applied to software. Nothing. So I asked Claude Code to do extensive research on the subject. What it brought back is documented in docs/scoring.md in the repository, and it’s a lot: academic papers on singing evaluation (Nakano et al. 2006, Tsai & Lee 2012, Molina et al. 2013), patents (Yamaha has one from 1999, US5889224A, that details MIDI-based scoring with 3 tolerance bands), and the source code of open source projects like UltraStar Deluxe, AllKaraoke, Vocaluxe and Nightingale.

The conclusion of the research: without a per-song reference, you have to evaluate vocal quality generically. Measure how the person is singing, not what they should be singing. And since no single metric works for every case, we decided to implement four different scoring modes, each measuring a different dimension of vocal quality.

The phone microphone problem

Before the scoring modes, I have to explain a more fundamental problem the research uncovered: the phone microphone.

When you sing karaoke with the phone, the mic picks up three things at once: your voice, the music coming out of the speaker, and ambient noise from the room. Your voice is physically closer to the mic, so it dominates the signal. But not enough for clean separation.

I tried several approaches to isolate the voice:

Spectral subtraction using YouTube’s reference audio. Dropped it. The YouTube CDN blocks direct audio extraction by non-browser user-agents, and even with the reference audio in hand, the speaker’s EQ, the room reverberation and the Bluetooth delay make the signal too different from what the mic captures. Naive subtraction produces artifacts worse than no subtraction at all.

Pre-emphasis + center clipping. Dropped that too. Center clipping destroys the waveform that the YIN algorithm needs for autocorrelation, and pre-emphasis amplifies noise as much as it amplifies voice.

What works is a 200-3500 Hz bandpass filter: a second-order IIR (Butterworth, Q=0.707) in cascade. The high-pass at 200 Hz kills bass, kick drum, bass guitar bleed from the speaker. The low-pass at 3500 Hz kills cymbals, hi-hats, high-frequency noise. Human voice fundamentals (85-300 Hz) and formants (300-3000 Hz) pass through the filter. It’s not perfect isolation, but it improves the voice/music ratio enough for pitch detection.

But the bandpass alone doesn’t solve everything. Guitars, synths and piano produce periodic signals in the same frequency range as voice, and YIN detects pitch in them too. To deal with that, the app does adaptive calibration: in the first 5 seconds of warmup (when nobody’s singing yet), it collects RMS samples from the signal to establish a baseline of the speaker’s level. During the song, it keeps that baseline updated (25th percentile of the last ~4 seconds of frames). For a frame to be scored, the RMS has to be at least 1.3x above the baseline. Your voice is closer to the mic, so it pushes the RMS above the speaker’s level. The instrumental melody stays near the baseline and gets filtered out. In testing, the original singer coming out of the speaker scored around 37 with sparse dots in the trail, while someone actually singing scored ~59 with dense dots.

Another annoying detail: on Android, specifically on Samsungs, the DSP’s AutomaticGainControl (AGC) attenuates the signal instead of amplifying it. On Galaxies, enabling AGC drops the mic peak from ~0.06 to ~0.003. Silence as far as pitch detection is concerned. So the app disables AGC, echo cancellation and noise suppression. When the peak falls below 0.01, it applies software gain (up to 30x) to bring the signal up to usable levels.

The YIN algorithm

To detect the voice’s pitch I use YIN, by Alain de Cheveigné (IRCAM-CNRS) and Hideki Kawahara (Wakayama University). It’s a fundamental frequency estimator in the time domain. The central idea is the Cumulative Mean Normalized Difference Function (CMNDF), which basically measures how periodic the signal is at each lag, normalizes it to reduce false positives, and uses parabolic interpolation to refine the result. It’s lightweight enough to run in real time on a phone, which is what matters here.

In the app, the YIN threshold is 0.70 (tuned for mixed voice + music signals), and frames with confidence below 0.3 get discarded. Below that, it’s probably noise or an instrument.

The 4 scoring modes

Each mode evaluates a different aspect of vocal quality. They all share the same audio pipeline (bandpass → YIN → confidence gate). The difference is how they interpret the detected pitch.

Pitch Match

Measures how cleanly you sustain notes. Uses Gaussian decay based on the standard deviation of MIDI values in a rolling ~15-frame window. Steady notes (deviation < 0.3 semitones) score 85-100%. A trembling voice (deviation > 2 semitones) scores near zero. Good for songs you already know well.

Contour

Measures the melodic shape of your singing. It doesn’t matter which exact note you hit, only the direction and the flow. Evaluates the pitch range and melodic movements (jumps > 0.5 semitone) in a rolling window. Monotone singing scores ~10%. Smooth melodic movement with a 2-6 semitone range scores 70-100%. Good for when you’re learning a new song.

Intervals

Measures the musical quality of jumps between consecutive notes. A whole tone (2 semitones) scores highest. Thirds and fourths score well. Wild jumps of an octave or more score low. Uses a Gaussian curve centered on the whole tone. Works when you’re singing in a different key from the original.

Streak

It’s Pitch Match with a combo multiplier. Each consecutive frame with a score above 0.4 increments the streak counter. The streak adds bonus points (up to +0.4 on a streak of 30+). Breaking a streak > 5 frames pushes a 0.05 penalty into the EMA. Silence freezes the streak, so instrumental breaks don’t hurt you. The most fun mode for parties.

The logic behind these four modes came from the research Claude did across academic papers. Each one measures a different dimension: pitch accuracy, melodic contour, phrasing and consistency. None of them is sufficient on its own, but together they cover, reasonably well, what you can evaluate without having the song’s reference melody.

The Pitch Oracle

Beyond the four purely vocal modes, the app has what I call the Pitch Oracle. The idea: instead of evaluating your voice in isolation, the app downloads the video’s reference audio via youtube_explode_dart, decodes it to PCM, runs YIN on it, and builds a timestamped pitch timeline of the entire song. During scoring, if the mic’s pitch matches the reference’s pitch at that moment in the video, it’s probably speaker bleed, and gets ignored. If it differs, it’s your voice, and gets scored.

The synchronization works through the currentTime of the HTML5 video element, sent to Dart through a JS timeupdate listener every ~250ms. The oracle queries the reference pitch at the exact playback position, accounting for pause, seek and speed change.

The first time you play a song, the oracle takes 5-15 seconds to download and analyze the audio. But the timeline is saved as JSON in the app’s local cache (pitch_oracle/<videoId>.json). If you play the same song again, it loads instantly from cache, no network request. That also fixes YouTube’s rate limiting problem for the songs you sing the most.

With the oracle active, the modes change behavior. Pitch Match compares the singer’s pitch class against the reference’s, agnostic to octave (like SingStar). Contour uses cross-correlation between the singer’s pitch movement and the reference’s. Intervals compares semitone jumps against the reference’s.

When YouTube blocks the download with rate limiting (happens after many consecutive requests from the same IP, clears in 15-30 minutes), the oracle silently fails and the modes fall back to purely vocal analysis.

The road to here

The app you see now went through a lot of iteration before reaching this state.

First, I tried to make a Linux desktop version to make debugging easier. Makes sense, right? Test on the desktop, iterate fast, then port to mobile. The problem is that Flutter has no webview backend for Linux desktop. webview_flutter simply doesn’t work. I tried webview_cef, which is based on the Chromium Embedded Framework. CEF spawns its own GPU process, and on Hyprland (a Wayland compositor based on wlroots) that conflicts with the compositor’s render pipeline. On my NVIDIA setup, the entire Hyprland session froze. Locked screen, no keyboard response, I had to kill it from a TTY. On top of that, CEF requires downloading a ~200MB binary on the first build. I gave up on CEF and wrote a native bridge in C++ with Claude using WebKitGTK and Flutter method channels. It worked, but every YouTube quirk required separate code for Linux and Android. just_audio also has no Linux desktop implementation. The Linux version turned into dead weight. I deleted ~1,500 lines of Linux-specific code and focused only on Android.

Then came the Samsung mic saga. On my Galaxy Z Fold, the mic was capturing an absurdly low signal. Peaks of ~0.005, basically silence as far as pitch detection was concerned. I spent two hours trying to figure it out. I lowered thresholds, raised software gain to 50x, disabled audio preprocessors. Nothing was working right. Until I figured out the real problem: Android’s AutomaticGainControl. The name says “automatic gain control,” which suggests it amplifies weak signals. In the Samsung DSP implementation, it does the opposite. It attenuates the signal to a low reference level, optimized for voice calls. With AGC on, the peak dropped from ~0.06 to ~0.003. Disabling AGC fixed it. But then the audio_session package was re-enabling AGC under the hood. I removed that one too. It was three rounds of fixes, each finding one more layer of the problem.

And the scoring. The scoring took longer than everything else combined. The first implementation used a cumulative average, which kept the score stuck at one value and never responded to live singing. I switched to a rolling window. Then the score was stuck at ~50% because of a bug in the primary score weight. I fixed it, and it started showing 70% even with nobody singing. Fixed it again. Streak mode wasn’t resetting properly during silence. The chromatic snap was giving high scores for anything. The pitch history wasn’t being cleared on silence gaps and the modes were going stagnant. Every fix revealed another bug. It took more than 25 commits just on the scoring, from the first prototype to the current state.

The result isn’t perfect. I know. But it works well enough to be fun, which was the goal from the start.

Settings

The settings panel lives behind the gear icon on the overlay. There are three mic presets for different environments (clean external mic, normal room, loud party), each adjusting confidence and amplitude thresholds. There’s a pitch shift for when the song is too high for your vocal range. The shift moves both the video audio and the scoring at the same time: it uses the HTML5 element’s playbackRate with preservesPitch=false, so +2 semitones speeds the audio up to 1.12x (pitch goes up) and -2 semitones slows it down to 0.89x (pitch goes down). The scoring compensates for the offset, so you sing in your comfortable range and the system grades you correctly. There’s mic calibration, a 3-second process that measures the room noise and adapts the thresholds. And there’s a restart to reset the score without reloading the video.

To switch scoring modes, tap the score box during playback.

Usage flow

1. Open the app. YouTube loads inside the app with the Frank Karaoke logo.
2. Search for a karaoke video. Any video works, but instrumental tracks with on-screen lyrics give better results.
3. The video pauses briefly to initialize the mic, download the song’s data for the pitch oracle, and prepare the overlay. The first time with a new song this takes 5-15 seconds. If you’ve played it before, it loads from cache instantly.
4. Sing. The “live” score reflects your current performance (exponential moving average with alpha 0.15, ~1 second response). The “overall” score is the cumulative average of the entire song.
5. When the video pauses, scoring pauses with it (so it doesn’t score ambient noise). If you seek, the score resets and gets a 5-second warmup.

How to install

The app isn’t on the Play Store yet, I’m waiting for Google to verify my developer identity. It should show up there in the next few days. In the meantime, it’s an open project and you can install it directly.

The easiest way is to download the signed APK directly from the GitHub releases page. On your Android phone or tablet, download FrankKaraoke-0.2.0-android.apk, open it and tap Install. If Android complains about “unknown sources,” enable it under Settings > Security for your browser. On the first run the app will ask for mic permission. Then go into settings (the gear icon) and calibrate the mic before singing — three seconds.

If you want to compile from source or contribute, the repository is on GitHub. You’ll need Flutter SDK 3.10+, Android SDK API 24+, and a physical device for mic testing (an emulator doesn’t give representative results).

git clone https://github.com/akitaonrails/frank_karaoke.git
cd frank_karaoke
flutter pub get
flutter run -d <device_id>

The README has the rest.

Stack: Flutter + Riverpod for state management, webview_flutter for YouTube, youtube_explode_dart for audio extraction, record for PCM mic capture, audio_decoder for reference decoding via Android MediaCodec, and the YIN algorithm implemented in pure Dart.

The technical documentation for the scoring system is in docs/scoring.md in the repository. It covers how SingStar, Joysound and DAM work, the academic papers, the pitch oracle architecture, the voice isolation problems on Android, and the roadmap.

The scoring is experimental

I have to be straight: the scoring system is experimental. Without per-song reference files, the evaluation is approximate. The app measures whether you’re in tune, whether you follow a melodic contour, whether your intervals are musical, whether you’re consistent. But it doesn’t tell you whether you’re singing the correct melody for this specific song (unless the pitch oracle manages to download the audio, and that doesn’t always work).

If you have experience with audio processing, pitch detection, or music evaluation, the repository is open and the research documentation in docs/scoring.md details what was tried, what works and what doesn’t. In particular: tuning the modes’ thresholds, improving voice isolation, and integrating with UltraSinger (which generates reference files from songs using Demucs + basic-pitch + WhisperX) are areas where contribution from people who know the subject would make a real difference. I’d appreciate any help from specialists on calibrating these systems.

Oh, and the name. Frank Karaoke. It’s a tribute to Sinatra. Who else?

Project on GitHub: github.com/akitaonrails/frank_karaoke

First things first: this isn’t an evangelism piece or a day-trading pitch. Quite the opposite. As I write this, on April 1, 2026, Bitcoin is around US$ 68k and close to R$ 391k, below the 2025 peaks. Plenty of people look at that and either panic or start fantasizing about leveraged trades. I think both reactions are wrong. There’s a “super cycle” thesis floating around based on institutional demand, spot ETFs and the lagged halving effect. Maybe. Maybe not. What I do know is that short-term candles don’t change the part I actually care about: infrastructure. If you need leverage to “speed up your gains,” you’re probably just speeding up your chances of getting liquidated.

For me, the useful question isn’t “is it going up tomorrow?” The useful question is: “if I want to store and move Bitcoin without outsourcing everything to an exchange, a web wallet and a public API, how do I set that up properly at home?”

The real problem: too much convenience costs too much privacy

Most people’s default flow is simple: buy on an exchange, leave the balance sitting there, or install some random wallet on the phone and call it done. It works. It also concentrates risk and leaks metadata everywhere.

If you leave a balance on an exchange, you have custody risk. If you use a desktop wallet pointed at a public server, you have privacy risk. If you use a hardware wallet casually, bought second-hand on Mercado Livre, you have supply chain risk. Mix all that with hurry, and it gets worse.

That’s why I ended up at a combination that, for someone technical who wants to run their own infra, feels pretty solid:

• Coldcard for cold storage
• Sparrow Wallet on Linux as the desktop wallet and transaction coordinator
• Fulcrum on the home server as a private Electrum server
• bitcoind on the same server as a real full node, validating the chain and broadcasting without depending on third parties

It’s not the easiest path. But that’s exactly the point. Real security rarely comes from the easiest path.

The concepts that confuse beginners

Before getting into the stack, it’s worth aligning on four terms that usually get tossed around like everyone already knows them:

In plain language, the flow looks like this:

1. Sparrow, on the desktop, builds the transaction.
2. That transaction becomes a PSBT.
3. The PSBT goes to the Coldcard via microSD.
4. The Coldcard signs it offline.
5. The signed file goes back to Sparrow.
6. Sparrow broadcasts through your own server, not through someone else’s public infrastructure.

That’s what people mean by “airgapped workflow.” It’s not magic. It’s just disciplined separation of roles.

Coldcard: cold signer, offline, the right kind of annoying

I use Coldcard as cold storage. The reason is simple: it was designed from day one as a Bitcoin-only device, with a heavy focus on airgapped operation through microSD. That alone eliminates an entire category of “conveniences” that many people find practical, but that I’d rather not have anywhere near my keys.

In practice, the Coldcard holds the most important part of the system: the private key. It doesn’t need to know about a server, Electrum, public API, exchange, or any of that. Its job is one thing: sign transactions offline.

That decoupling is great for two reasons:

• The desktop can be convenient without becoming a single point of failure.
• The signer stays isolated even if your main machine has problems.

And here’s a warning I really want to put in mental all-caps:

Never buy a hardware wallet second-hand. Ever.

This isn’t an exaggeration. You have no way to actually know what happened to that device before it reached your hands. It could have a pre-generated seed, tampered firmware, swapped components, repackaged box, compromised supply chain, or simply some dumb trick waiting for you to let your guard down. Hardware wallet is one of those categories where saving R$ 300 buying used is insanity. Always buy from the manufacturer’s official site or from a reseller officially authorized by the manufacturer. And even then, check seals, provenance and firmware.

Can you do something similar with an old phone?

You can. But I’d treat it as a study or budget alternative, not as an obvious substitute for a Coldcard.

The most serious path for that today is AirGap Vault, which was specifically designed to use an old smartphone as an offline signer over QR codes, keeping the device off the network. The idea is good, and for many people it might be the right entry point.

But there are trade-offs:

• An old smartphone wasn’t designed as a dedicated hardware wallet
• The device’s prior history matters
• An aged battery, bad screen and abandoned Android are real problems
• The threat model is less clear than on a dedicated device

So my view is simple: can you use it? Yes. Would I recommend it as the main solution for storing meaningful wealth? No. For that I still prefer dedicated hardware bought from the right source.

Sparrow Wallet: the best desktop piece in this puzzle

On Linux, I use Sparrow Wallet. For me, today, it’s one of the best pieces of software in this ecosystem.

What I like about it:

• works very well on Linux desktop
• supports hardware wallets properly
• understands PSBT without drama
• makes it crystal clear what’s happening in a transaction
• it’s great as a watch-only wallet

In my flow, Sparrow does three things:

1. Holds the watch-only wallet.
2. Builds the transaction with outputs and fees.
3. Receives the signature back from the Coldcard and broadcasts it.

That separation is elegant. The desktop becomes the coordinator. The signer stays cold.

Why Coldcard + Sparrow works so well

This combo is good because each piece does what it does best:

• the Coldcard protects the key
• Sparrow organizes the human use of the wallet
• the server handles the infrastructure

A lot of wallets try to do everything. I prefer this modular design. It’s less “magic,” more explicit, and easier to reason about without lying to yourself.

If I’m at the desktop, I want visibility. If I’m at the signer, I want isolation. If I’m at the server, I want validation and a local index. That division is clean.

The Sparrow problem when you don’t run your own infra

Now comes the important detail. Sparrow alone doesn’t solve privacy.

If you install it, open it and just use public servers, the people on the other end learn quite a lot about your wallet: your address set, xpubs or derivations, balance, history, query behavior, broadcast. It’s not custody, but it’s still exposure.

That’s the hole Fulcrum fills.

Fulcrum: the home’s private Electrum server

Fulcrum is an Electrum server. Instead of letting Sparrow ask things of a third-party public server, it asks my own server.

In practice, that means:

• local balance lookups
• local history
• local address discovery
• local broadcast

In other words: the desktop wallet stops “phoning home” to the world every time you open the program.

In my current setup, Sparrow points at a Fulcrum running on the home server on the LAN, with port 50001 on the internal network and 50002 with TLS.

And why Fulcrum isn’t enough on its own

Because Fulcrum doesn’t replace a full node. It indexes on top of a full node.

The thing actually validating blocks, consensus rules, scripts, transactions and the chain is bitcoind. Fulcrum sits in front of it as an indexing layer, because plain Bitcoin Core wasn’t built to serve a desktop wallet with that kind of fast querying.

So the correct architecture is:

Coldcard (offline signer)
        ^
        | microSD / PSBT
        v
Sparrow Wallet (desktop watch-only + coordinator)
        |
        v
Fulcrum (private Electrum server)
        |
        v
bitcoind (full node)

What I actually brought up on the home server

On my home server, the stack lives in a dedicated Docker Compose folder and is made of two containers:

• bitcoin-bitcoind
• bitcoin-fulcrum

The compose is simple. And that’s good. Sensitive infra doesn’t gain anything by getting clever in YAML.

The main design is this:

services:
  bitcoin:
    image: lncm/bitcoind:v28.0
    container_name: bitcoin-bitcoind
    user: "${BITCOIN_UID}:${BITCOIN_GID}"
    restart: always
    security_opt:
      - label:disable
    volumes:
      - /srv/bitcoin/data:/data/.bitcoin
    ports:
      - "8333:8333"
    stop_grace_period: 5m
    healthcheck:
      test: ["CMD", "bitcoin-cli", "-datadir=/data/.bitcoin", "ping"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  fulcrum:
    image: cculianu/fulcrum:latest
    container_name: bitcoin-fulcrum
    restart: always
    security_opt:
      - label:disable
    volumes:
      - /srv/bitcoin/fulcrum:/data
      - /srv/bitcoin/data:/bitcoin:ro
    command: ["Fulcrum", "/data/fulcrum.conf"]
    ports:
      - "50001:50001"
      - "50002:50002"
    depends_on:
      bitcoin:
        condition: service_healthy

In my case, the restriction of 50001 to LAN happens at the host’s network layer. The YAML above is the skeleton of the stack, not the entire firewall policy.

The most important parts of this:

• restart: always because this is a long-running service
• explicit volume so state isn’t lost
• user: "${BITCOIN_UID}:${BITCOIN_GID}" because the persistent directory needs to match the storage’s real ownership, so I’d rather pin UID/GID explicitly than trust the image’s default
• the RPC isn’t published on the host; it stays on the Compose internal network, which is all Fulcrum needs
• the healthcheck uses Bitcoin Core’s own local .cookie, so there’s no need to spread a fixed password through commands
• Fulcrum mounts the node’s datadir as read-only just to authenticate via the .cookie without inventing parallel credentials
• in fulcrum.conf, that becomes a simple configuration: talk to bitcoin:8332 and read the mounted .cookie, instead of repeating credentials in plaintext
• security_opt: label:disable because on this host with MicroOS, SELinux and sensitive bind mounts, I preferred the pragmatic route of disarming this specific friction rather than wasting time fighting labels on a volume that’s already being handled in a controlled way
• depends_on with service_healthy so Fulcrum only comes up after bitcoind’s RPC is responding
• stop_grace_period: 5m because bitcoind needs real time to flush state on a graceful shutdown

The final version

Today, the design I want to keep is this: bitcoind with txindex, dbcache=1024, persistent volume, 5-minute graceful stop, .cookie authentication, and Fulcrum in front serving Sparrow over LAN or TLS.

The current stack looks like this:

I’m not interested in turning this into a spectacle. The point is simpler: the final infrastructure has to be boring, predictable and stable.

The tunings that actually matter

There’s no magic here. There are a few parameters that make a real difference and a bunch of stuff that just decorates compose files.

stop_grace_period: 5m exists because bitcoind isn’t a disposable stateless API container. It maintains chainstate, indexes and an in-memory cache. If you don’t give the process time to finish properly, you create unnecessary work for the next start.

user: "${BITCOIN_UID}:${BITCOIN_GID}" is there for a much less glamorous and much more important reason: persistent storage with the wrong permissions is an excellent way to break a working service. So I’d rather align the container with the volume’s actual ownership instead of leaving that implicit.

dbcache=1024 is the spot I find most reasonable for a domestic node that’s always on. Big enough not to suffer constant I/O, small enough that every restart isn’t a labor.

txindex=1 I keep because I want the complete node, not a minimalist install just to claim “it runs Bitcoin.” If the goal here is operational autonomy, I’d rather have the full index.

rpcworkqueue=512 and rpcthreads=16 are the kind of tweak that makes sense when you know you’ll have Fulcrum querying the node all day and you want some headroom.

On the Fulcrum side, the main parameters are:

• db_mem = 8192
• db_max_open_files = -1
• bitcoind_clients = 8
• worker_threads = 0
• peering = false

Again: nothing esoteric. Just enough cache, reasonable parallelism and absolutely no announcing this server as a public service.

In my current bitcoin.conf, the important core ended up like this:

server=1
txindex=1
prune=0
rpcbind=0.0.0.0
rpcallowip=172.0.0.0/8
rpcthreads=16
rpcworkqueue=512
dbcache=1024
maxmempool=512

All of this makes sense on a server with decent RAM and fast NVMe. But the detail that matters most is still the clean shutdown. Wallet infrastructure has no room for “we’ll deal with it later” thinking.

The actual size of all this

This is another point a lot of people underestimate.

If you look at older Bitcoin Core documentation, you’ll find numbers like 350 GB of disk for a node with default config. That’s outdated. More current data on the size of the blockchain points to something around 725.82 GB on March 11, 2026, and that’s just the raw chain, without the extra indexes that many technical folks will want to keep.

And here comes the catch: the stack I’m describing isn’t “a bare Bitcoin Core just to claim you run a node.” It’s bitcoind with txindex, plus Fulcrum, plus headroom for rebuild, logs, snapshots and normal network growth.

So to put together something similar today, I’d think like this:

• below 1 TB: I wouldn’t even start
• 1 TB: pragmatic minimum
• 2 TB: comfortable range
• above that: if you want long-term headroom, snapshots and less operational anxiety

And here’s the most important observation of all in self-hosting: don’t assume persistence, mount and backup are right just because the YAML looks clean. Verify.

Another thing I wouldn’t forget on a btrfs host: put Fulcrum’s database (fulc2_db) on a separate subvolume. The reason is mundane. That directory grows, changes constantly and has nothing to do with generic automated snapshots of /var. If you mix everything, you end up dragging a large rebuildable index along with system snapshots, burning space and making maintenance more annoying than it needed to be. The Fulcrum index isn’t sensitive configuration. It’s heavy, volatile, rebuildable data. I treat it exactly like that.

Hardening: what I’ve already applied

This is where the difference between “ran on my laptop” and “I’d trust this to operate my wallet” shows up.

In the current state of the stack, the points I consider important ended up like this:

• Bitcoin Core’s RPC no longer relies on unnecessary host exposure; Fulcrum talks to bitcoind over the Docker internal network, which is what actually matters
• 50001 is restricted to internal LAN use
• 50002 is available with TLS, which is the right move when you need to leave plaintext behind
• shutdown is graceful, with stop_grace_period: 5m, so bitcoind has time to flush state instead of dying any old way
• the storage mount isn’t on a “we’ll see later” basis; there’s a mount check before Docker comes up, precisely to avoid silent drift

Each of those items exists for a very concrete reason.

Pulling the RPC off the host’s surface reduces attack at zero cost. Fulcrum is already in the same Compose and can already talk to the service by its internal name. There’s no real gain in leaving that port exposed where it doesn’t need to be exposed.

Separating 50001 and 50002 also helps keep the house in order. Within a controlled LAN, plaintext is acceptable. Outside of that, the minimum reasonable thing is TLS. Mixing the two scenarios usually turns into a mess.

stop_grace_period: 5m looks like a container detail, but it isn’t. Anyone who’s ever had a database, an index or a blockchain node killed without grace knows how that turns into hours of work later. A stateful service needs a decent stop.

And the mount check is one of those annoying things that saves you from yourself. The YAML can look beautiful. If the storage didn’t mount and the service came up writing where it shouldn’t, you’ve just manufactured a really irritating problem.

There’s also one detail I really like in this final version of the stack: Fulcrum authenticates to bitcoind through the .cookie file, not through a fixed plaintext password. That’s interesting for two reasons:

• you don’t need to leave a static credential showing up in compose, inspect, healthcheck or documentation
• the authentication is more aligned with the way Bitcoin Core already knows how to operate locally

In practical terms, that reduces accidental leakage of operational secrets. It’s not a magic solution to everything, but it’s much better than spreading rpcuser and rpcpassword across files, logs and commands.

The only kind of hardening I try to avoid here is the one that’s overly performative in YAML and loose in operation. I’d rather have less “stage engineering” and more basic discipline:

• minimum network
• minimum secrets
• minimum privilege
• clean shutdown
• verified storage
• separate subvolume for large rebuildable data, like the Fulcrum index

And, again, document everything. Good infrastructure isn’t the kind that just works today. It’s the kind that keeps working when you come back to it six months later.

Why this improves transactions on your side

When I build a transaction in Sparrow and sign it on the Coldcard, the chain of trust is much better defined:

• the private key never touches the internet
• the desktop wallet doesn’t have to trust a public server
• the broadcast can come out of my own node
• the address history doesn’t need to land on a third-party Electrum server

This doesn’t make anything invulnerable. There’s still a risk of malware on the desktop, badly stored seeds, human error, social engineering and physical disaster. But the design becomes much more coherent.

What about Lightning? Especially in Brazil?

There I separate things.

Reserves and larger-value transactions I treat one way. Day-to-day spending I treat another.

For day-to-day spending, especially in Brazil, I think it’s operationally dumb to carry a lot of balance in a hot wallet. Lightning wallets and spending apps have to be almost like a “pocket wallet”: just enough for daily life.

That goes double if you use a hybrid or custodial solution like RedotPay. I get why it’s interesting for Brazilians: a Hong Kong company, international focus, a reasonably practical bridge between crypto and card spending. For travel, online shopping and life outside the Brazilian banking axis, it makes sense. But I’d never treat it as a place to store wealth. That’s a spending tool, not a vault.

Same logic for Bitrefill Brasil. I think the service is interesting precisely because it solves a real pain in Brazil: turning sats into concrete utility without selling your full position or depending on banking integration all the time. Gift cards, top-ups, small expenses. As a use tool, it makes a lot of sense.

For a Lightning wallet on the phone, I’d look at first:

• Phoenix for people who want something very good and simple
• Breez for people who want a great payments experience
• ZEUS if you’re more technical and you eventually plan to operate your own Lightning node

All of them, in my head, fit into the “pocket wallet” category. Small balance. Daily use. Don’t turn a phone app into a retirement vault.

Recent news that reinforces this reasoning

I’m not building this kind of stack because I think it’s pretty. I’m building it because outsourcing too much goes wrong too often.

Two recent examples:

• the Bybit hack in 2025 showed, again, the basic risk of leaving meaningful custody at an exchange
• the Coinbase customer data leak in 2025 showed the other side of the problem: even when custody isn’t the immediate concern, your identity, balance and history become an attack surface

A stack like Coldcard + Sparrow + Fulcrum + full node doesn’t eliminate every risk in the world. But it avoids two very real classes of problem:

• losing custody sovereignty
• handing over wallet and transaction privacy on a silver platter to third parties

So is it worth it?

For most people, honestly, probably not in the first month. It’s labor-intensive, has a learning curve, and demands discipline.

But for programmers, engineers and any technical person who wants to learn not to depend on someone else’s service all the time, I think it’s an excellent exercise.

You learn about:

• separation of concerns
• persistence and state
• graceful shutdown
• observability
• secret isolation
• the trade-off between convenience and security

And all of that is valuable beyond Bitcoin.

In the end, that’s what interests me most about this stack. It isn’t about preaching “hyperbitcoinization” or posing as a price prophet. It’s about building a system at home that I can trust more because I’m the one who installed it, measured it, broke it, fixed it and documented it.

Is it work? Yes.

But that kind of work teaches exactly what modern software tries to make you forget: depending less on others is more work upfront, but it usually buys a lot more control in the long run.

But the game that really got me hooked on the simcade genre was the original Gran Turismo, in 1997, on the PlayStation 1. “The Real Driving Simulator” on the cover. I played that game obsessively. Around the same time I was watching the Initial D anime, which premiered in 1998 in Japan. I bought every manga volume and read all of it from start to finish. The story of Takumi Fujiwara going down Mount Akina at dawn delivering tofu in his father’s AE86 is, to me, one of the best motorsport stories ever told in any medium.

I still follow Shuichi Shigeno’s work today. After Initial D came MF Ghost (2017-2025), set in the same universe but in a near future where combustion cars have become museum pieces. And now, since July 2025, I’m reading Subaru and Subaru, the direct sequel that ties the Initial D and MF Ghost universes together with two protagonists named Subaru — one from Gunma, one from Kanagawa — competing in a new racing series. It’s Shigeno at his best.

Two years ago I traveled to Japan with my girlfriend and made a point of going to Daikoku PA, the famous parking area on the Shuto Expressway in Yokohama where the JDM culture concentrates. As an old fan of Tokyo Xtreme Racer, by Genki, I needed to see Daikoku with my own eyes at least once. And it didn’t disappoint. Instead of renting a car, we booked a tour with a local guide in his prepped Nissan GT-R. Better that way. On the drive there he explained the history of the Wangan, how the scene works, what’s YouTube exaggeration and what’s real. When we got there on a Friday night and I saw the whole thing in person — Skyline R34, RX-7, Supra, GT-R, tuned kei trucks, insane bosozoku — the feeling was strange in the best possible way. It looked like Tokyo Xtreme Racer, except with the smell of fuel in the air and the sound of real exhaust pipes.

And there’s another detail: I’m playing the new Tokyo Xtreme Racer reboot on PC, and it’s exactly the kind of game that understands its own audience. Strong single-player campaign, addictive progression, the right vibe, and none of the loot box nonsense. I’d recommend it without hesitation. For the same reason, I’m also really looking forward to Forza Horizon 6, which this time is going to be set in Japan. I’ve already pre-ordered it and I can’t wait to play it on the new cockpit.

Driving for real

Now that I’m semi-retired, I’ve had the chance to take my Mercedes to track days. I’ve driven at Autódromo de Interlagos (the Autódromo José Carlos Pace), the 4.309 km circuit in São Paulo that’s been hosting the Brazilian F1 GP since 1973, famous for the S do Senna corner complex and the circuit’s wild elevation changes. I’ve also driven at Autódromo Velocitta, a modern 3.443 km circuit opened in 2014 in Mogi Guaçu in the interior of São Paulo, which hosts Stock Car Brasil and Porsche Cup.

In Las Vegas I’ve driven supercars on those track-day experiences. And when I traveled with my girlfriend to Gramado, in Rio Grande do Sul, we went to Super Carros, which is on Av. das Hortênsias 4635. They have a 2,400 m² hangar with more than 50 cars — Ferraris, Lamborghinis, Porsches, GT-Rs, Corvettes, American muscle cars. You pick a car, head out with an instructor, and drive a roughly 17 km route between Gramado and Canela. I took out a Nissan GT-R and a Ferrari California.

Three years ago I also went to Abu Dhabi with my girlfriend and we went to Ferrari World, which has some of the best racing simulators I’ve ever tried. Hydraulic platform with 6 degrees of freedom, F1 cockpit, the works. I’ve always loved testing simulators wherever I go.

But driving real cars on real tracks is a very expensive hobby. Tires, fuel, insurance, maintenance, registration. And more important: I’m an introvert. I prefer being alone. My simulator cockpit is perfect for when I want to drive without having to deal with anyone. That’s why I love rally so much — it’s me, the virtual co-driver, and the road. Nothing else.

The games I play

I know most people who build a cockpit like this do it to play serious sims — iRacing, Assetto Corsa Competizione, Automobilista 2. I respect that, but it isn’t my thing. I don’t like playing online with other people. I have zero intention of starting a live streaming career. This is purely for my own enjoyment.

These days I play Gran Turismo 7 on the PS5, Forza Motorsport (the 8, from 2023) on PC, but where I have the most fun is in rally games: EA SPORTS WRC, WRC 10 and DiRT Rally 2.0. My first experience with Forza was on the Xbox One with Forza Motorsport 5 and then Forza Horizon 4, which kept me hooked for hundreds of hours.

And I have a huge soft spot for retro games. The original Colin McRae Rally from 1998 on the PS1 was my first rally game. But my favorite of all time is Colin McRae Rally 2.0 (2000), also on the PS1. I recently played through the entire campaign again on the PC version — you can find repacks that run in high resolution and widescreen, much better than the original PlayStation versions. I’d recommend that for any of the titles in the series.

After that came Colin McRae 3 (2002), Colin McRae Rally 04 (2003) and Colin McRae 2005 (2004). Other arcades I revisit often: OutRun 2 SP (2004) and OutRun 2006: Coast 2 Coast (2006) — the best OutRun ever made, in my opinion.

But my game of the year, by a long shot, is Super Woden: Rally Edge. An indie made by a solo developer (ViJuDa, from Spain) that launched in January 2026 for less than R$ 60. Eight countries, more than 80 cars, a career mode, local split-screen multiplayer for up to 4 players, online leaderboards. The behind-the-car camera instead of the top-down view of the previous Super Woden GP made all the difference. 96% positive reviews on Steam with more than 1,300 ratings. It’s the kind of game that proves you don’t need a million-dollar budget to make something amazing.

The evolution of my wheel setup

Logitech G29 era (~2015-2021)

I always wanted a racing wheel. I started over 15 years ago with something equivalent to Logitech’s entry-level wheel, a G29. The G29 is a fine wheel to start with — gear-driven force feedback, pedals with a clutch, 900 degrees of rotation. But its force feedback is noisy and a bit crude. You can feel the gears turning.

Thrustmaster T300RS + SXT V2 stand era (~2021-2024)

Around 2021 I upgraded to the Thrustmaster T300RS, a belt-driven wheel that’s a huge jump up from the Logitech. The force feedback is much smoother and more precise. And I bought the Extreme Sim Racing SXT V2 stand, which is much sturdier than those generic desk clamps.

First I set it up in front of my desktop PC, which at the time had an RTX 3090. It worked, but it was a hassle to keep mounting and unmounting the stand and the cables every time I wanted to play.

Then I built a setup with a long fiber-optic HDMI cable to connect my 60" TV to the PC at the back of the room. I moved the stand in front of the couch. Less of a hassle, but I still had to take it down whenever I wanted to watch a movie with my girlfriend.

Around 2024 or 2025 I swapped my couch for one of those VIP cinema couches from Star Seat, which reclines and the whole nine yards. The problem: it was way taller than the previous couch. I had to do all kinds of workarounds to make the stand work at that height. I even 3D-printed mounts and sent them to PCBWay to machine steel plates so I could attach big wheels under the stand and gain a few centimeters of height. But that left the setup way too wobbly to drive comfortably.

Fanatec CSL DD + Direct Drive era (~2024-2025)

In the meantime I gave the T300 to my brother and upgraded to a Fanatec CSL DD Gran Turismo Edition. The CSL DD’s direct drive motor delivers 5 Nm of torque at the base, but the Gran Turismo DD Pro kit comes with the Boost Kit 180 that brings it up to a sustained 8 Nm with no active cooling. Direct drive means there’s no gear or belt between the motor and the wheel — the motor’s shaft IS the wheel’s shaft. The difference is absurd. The T300 was already great, much better than the Logitech. But the Fanatec is on another level. You feel every asphalt texture, every bump, every incipient slide. There’s no going back once you try it.

I bought it together with the CSL Pedals with Load Cell kit, which measures pressure on the brake instead of displacement. Makes all the difference in braking — you learn to modulate by foot pressure, not by how far the pedal moves. Way more natural.

I also wanted to try the H-pattern manual shifter with the ClubSport Shifter SQ V1.5 from Fanatec and a separate handbrake. It’s fun to try out old cars with a clutch and an H-pattern, but in practice I never adapted. The SXT V2 stand was already shaking a lot with the direct drive, and using the shifter on that unstable setup was frustrating. And I know there are people who want to do heel-toe, but in the simulator I prefer to keep my left foot on the brake and my right on the gas and modulate both at the same time. Works better for me. Now that I have the McLaren wheel with the analog handbrake and clutch paddles right on the wheel, the H-pattern shifter and the external handbrake have been retired. For rally, an analog handbrake on the wheel is much more natural.

I also bought the PS5 with Gran Turismo 7 around this time. I put on the dbrand Darkplates matte black faceplates to replace the original white plates — it looks much better and more discreet.

But the setup was still the SXT V2 stand in front of the VIP couch. The same kludge. The same wobble. I obviously wasn’t going to give up the cinema couch. The situation became unsustainable.

The computer and the hardware setup

A note on my gaming hardware. I bought a Minisforum UX790 Pro to be my dedicated Steam machine. It’s a mini-PC with an Intel Core Ultra 9 285H processor, fits in the palm of your hand. Together I bought the Minisforum DEG1, an external GPU dock that connects via OCuLink (PCIe 4.0 x4, 64 GT/s). It’s an open design — basically a board with a PCIe x16 slot and room for an ATX or SFX power supply. There’s no card size limit, so an RTX 4090 fits comfortably. The performance loss compared to a native PCIe slot is minimal. I put the RTX 4090 in it. The 4090 came from my desktop — at the start of 2025 I went to Miami and took the chance to buy an RTX 5090 because I was using more and more local AI and LLMs. I gave my old 3090 to my girlfriend to use for video editing. The 4090 went into the mini-PC.

So my gaming setup today is: Minisforum UX790 Pro + eGPU with RTX 4090 for Steam and PC games, and a PlayStation 5 with matte black Darkplates for Gran Turismo 7 and exclusives.

The cockpit: Formula FX1

To round out the setup I also needed a decent monitor. I was already used to my 80" Samsung OLED TV in the living room and didn’t want to downgrade picture quality. So I invested in the Samsung Odyssey OLED G8 32". It’s a 4K (3840x2160) OLED monitor with a 240 Hz refresh rate, 0.03 ms response time (GTG), HDR True Black 400, HDR10+, 99% DCI-P3 coverage, 1,000,000:1 contrast, and FreeSync Premium Pro. It has 2 HDMI 2.1 inputs and 1 DisplayPort 1.4.

In practice: the colors pop, black is real black (it’s OLED, no backlight), and with the RTX 4090 I run most games at 4K and 120 fps with no problem. On lighter titles like Super Woden: Rally Edge, it easily hits 240 Hz. The smoothness is absurd. For a cockpit where you’re 60-70 cm from the screen, 32" OLED in 4K is the sweet spot. Bigger and you start to see pixels. Smaller and you lose the immersion.

In January 2026, after years of kludges, I finally ordered a dedicated cockpit. I researched a lot. I considered the Cockpit AX160, made of aluminum profile and very modular, and the Cockpit 4.0, which is the more traditional tubular steel kind. But neither was available at the time of purchase. And then I found the Formula FX1 in black and green from Extreme Racing — Petronas colors, F1-styled.

The FX1 is very different from traditional cockpits. The whole structure is welded thick steel tubing. When I say it doesn’t shake, I mean it doesn’t shake at all. Zero wobble. It’s a brutal difference compared to a stand in front of the couch. The driving position is reclined, F1-style — your feet are at the same height or higher than your hips. You’d think it would be uncomfortable, but it isn’t. You can sit there for hours without complaining. It comes with a padded adjustable seat, an articulating monitor mount, a tilt-adjustable pedal mount, and a height-adjustable wheel mount.

I had to wait about a month for delivery. In the meantime, as anyone who follows my blog knows, I dove into a 16-hour-a-day marathon testing the new AI agents from Anthropic and OpenAI — check the #vibecoding and #agents tags to see everything I built. After about 30 days of that insane marathon, my lower back gave out and I started developing what looks like a herniated disc. I had to see a doctor and take heavy anti-inflammatories.

And right that week, the cockpit decided to arrive.

The build

I was in absurd pain, but I built the cockpit anyway. It took an entire day to unbox and assemble the heavy steel pieces with my back screaming, but I did it.

The official assembly video I followed:

The McLaren GT3 V2 wheel

After assembling the cockpit, I decided that the standard wheel that comes with the CSL GT kit wasn’t enough. I upgraded to the Fanatec CSL Elite McLaren GT3 V2 (~R$ 4,990). It’s a 1:1 scale replica of the McLaren GT3 wheel, with carbon fiber, an OLED display, and compatibility with PC, Xbox, PS4 and PS5.

What I like most about it: it has the normal shift paddles behind it (shift up/down), but it also has two additional analog paddles that can be configured in four different modes. In mode B, which is what I use, the left paddle works as an analog handbrake and the right one as an analog clutch. That’s perfect for rally — I can pull the handbrake mid-corner without taking my hand off the wheel. It also has two 2-position toggles, two 12-position rotaries, 7 standard buttons with interchangeable caps, and Fanatec’s 7-direction FunkySwitch. It’s a complete racing controller.

The final setup

The cockpit ended up in a corner of my bedroom, between the manga shelves (you can spot Akira, Initial D, and 500-something other volumes in the background). I mounted the mini-PC and the PS5 on the cockpit’s side structure, together with the eGPU and the RTX 4090. Everything stays permanently connected. That’s what makes the difference: I no longer have to set anything up or take it down. I sit, turn it on, and I’m driving in 30 seconds.

The audio system

To round out the setup I needed dedicated audio. I didn’t want to use the monitor’s audio (terrible) and I didn’t want to be on a headset all the time. The fix was to build a separate audio system with HDMI audio extraction.

The centerpiece is an HDMI 2.1 switcher from OREI with audio extraction. It has 2 inputs and 1 HDMI output, supports 4K at 120Hz (48 Gbps of bandwidth), and extracts the audio through optical TOSLINK and 3.5mm. I connect the HDMI output of the RTX 4090 to one input and the PS5 to the other. Video goes to the monitor. Audio goes out through the optical port.

The optical audio goes to an Aiyima D03 amplifier, a compact 2.1 channel amp with 150W per channel, integrated DAC (PCM1808 chip), and Bluetooth 5.0 with aptX HD. It has optical, coaxial, USB, RCA and Bluetooth inputs. It even has a dedicated subwoofer output for when I get around to adding one. It uses Texas Instruments’ TAS5624 amplifier chip and has bass and treble control through the remote. For a cockpit setup where you’re 1 meter from the speakers, 150W is more than enough.

In practice, I keep the amp at 50% and Windows volume at 50%, and that’s already loud as hell. Which is to say: this isn’t a “good enough” little system. It’s set up to actually go loud if I want.

The speakers are Edifier P12, passive, with a 4-inch woofer and a 19mm tweeter. Frequency response from 55Hz to 20kHz, 6 ohm impedance, 20W RMS each. The MDF cabinet with wood finish has a rear bass-reflex port that helps the lows. For passive speakers this size, they deliver well. The mid-range is clean and the highs don’t distort even at high volume.

The setup logic is: HDMI switcher handles the switching between PS5 and PC, extracts audio to optical, the amp converts and amplifies, and the passive speakers deliver the sound. All without having to touch the monitor or swap cables. I press a button on the switcher and switch consoles.

When I want to play without bothering anyone, I plug my Meze 109 Pro directly into the 3.5mm output of the HDMI switcher. The Meze 109 Pro is an open-back headphone with 50mm dynamic drivers, 40 ohm impedance, 112 dB SPL/1mW sensitivity, and 5Hz to 30kHz response. The ear cups are walnut wood with handcrafted finish. It’s an audiophile headphone that works perfectly without a dedicated amplifier thanks to the low impedance. The sound is warm, with full bass and rich mids. You can hear every detail of the engines.

I haven’t decided about a subwoofer yet, but it’ll be my next upgrade. A dedicated sub is going to add that low-end weight that makes you feel the engine in your chest.

The verdict

The couch with a stand works. The PC desk with a stand works. But neither comes anywhere close to a dedicated cockpit. The FX1’s steel structure doesn’t move a millimeter, even with the Fanatec direct drive at max torque. The reclined F1 position is comfortable for sessions of hours. The load cell pedals stay firm on the base. The monitor is exactly at the right height and distance. And best of all: it’s always ready. I don’t need to assemble anything, take anything down, run cables, none of it. I sit and I drive.

For anyone who’s wondering whether it’s worth investing in a dedicated cockpit instead of staying with a desk or couch stand: it is. If you already have a direct drive wheel, the cockpit is the missing piece. I spent years thinking “this is fine” with the stand on the couch. It wasn’t fine. The difference in driveability is something else entirely. And for my case — introvert, single-player only, simcade — I couldn’t have built it any sooner. To be honest, I think I’ve finally landed on the simulator setup that’s perfect for my taste.

Shopping list: how much it all cost

Here’s the consolidated list of everything in my current setup, with approximate prices (some were bought in dollars and converted to reais at the time’s exchange rate):

Yes, almost R$ 57k is a lot of money. I worked like a dog for decades. Now that I’ve managed to retire honestly, my family is well taken care of, I have no debts, and I can finally give myself something I always wanted as a kid but couldn’t afford. When I sat at those OutRun and Daytona USA cabinets at the arcade, I dreamed of having something like this at home. It took 30-something years, but I got there.

And if you add up the years of kludges, stands that didn’t work, 15-meter HDMI cables, 3D prints, machined steel plates, and the frustration of mounting and unmounting everything — a dedicated cockpit saves your sanity. Unlike a PC that depreciates fast, a steel cockpit lasts decades.

Updated April 2, 2026: if you already read this yesterday, jump straight to the new update section.

This morning (March 31, 2026), security researcher Chaofan Shou discovered that the entire source code for Claude Code, Anthropic’s official CLI for AI coding, was sitting there for anyone to grab on the public npm registry. 512,000 lines of TypeScript. 1,900 files. All of it exposed through a 59.8 MB source map file accidentally bundled into version 2.1.88 of the @anthropic-ai/claude-code package.

Within hours the code had been mirrored on GitHub, picked apart by thousands of developers, and Anthropic had put out a statement calling it “human error in release packaging, not a security breach.” Which is technically true and ignores that the result is the same.

I use Claude Code every day. Some of the articles you read here, I wrote with it. So I figured I’d take a look at what’s inside. I actually started writing this piece in Claude Code itself, but my Max plan ran out before I finished. I closed the rest in Codex.

How the leak happened

Claude Code is bundled with Bun, the JavaScript runtime Anthropic acquired in late 2024. When you build with Bun, source maps are generated by default. Those .map files contain the full original source code, not just mappings. Every file, every comment, every internal constant, every system prompt.

The initial theory was that a known Bun bug had caused the leak: even with development: false, source maps were still being served and bundled in. But Jarred Sumner, Bun’s creator, shut that down: “This has nothing to do with claude code. This is with Bun’s frontend development server. Claude Code is not a frontend app. It is a TUI. It doesn’t use Bun.serve() to compile a single-file executable.” In other words, the Bun bug affects the frontend dev server, not the build process that generated the Claude Code npm package.

What actually happened is simpler: somebody at Anthropic forgot to add *.map to .npmignore or didn’t configure the bundler to skip source map generation in production builds. And worse: according to The Register, the source map didn’t just point to the original files, it referenced a ZIP hosted on Anthropic’s own Cloudflare R2 bucket. npm happily served it to anyone running npm pack, and the rest was mirroring work.

The irony is that the code contains a whole system called “Undercover Mode” built specifically to prevent Anthropic’s internal information from leaking in commits and PRs. They built a subsystem to stop the AI from revealing internal codenames, and then a source map exposed everything.

What’s inside: the hidden features

The source code reveals 44 feature flags covering functionality that’s ready but not yet shipped. This isn’t vaporware. It’s real code hiding behind flags that compile to false in external builds. Let me highlight the most interesting ones.

KAIROS: a Claude that never stops

Inside the assistant/ directory, there’s a mode called KAIROS, a persistent assistant that doesn’t wait for you to type. It observes, logs, and acts proactively on things it notices. Maintains append-only daily log files, receives <tick> prompts at regular intervals to decide whether to act or stay quiet, and has a 15-second budget: any proactive action that would block the user’s workflow for more than 15 seconds gets deferred.

KAIROS-exclusive tools: SendUserFile (sends files to the user), PushNotification (push notifications), SubscribePR (monitors pull requests). None of this exists in the public build.

BUDDY: a Tamagotchi in the terminal

I’m not making this up. Claude Code has a complete pet companion system in the Tamagotchi style called “Buddy.” A deterministic gacha system with 18 species, rarity, shiny variants, procedurally generated stats, and a “soul” written by Claude on the first hatch.

The species is determined by a Mulberry32 PRNG seeded by the hash of the userId. Same user always gets the same buddy. There are 5 stats (DEBUGGING, PATIENCE, CHAOS, WISDOM, SNARK), 6 eye styles, 8 hat options, and sprites rendered as 5-line ASCII art with animations. The code references April 1-7, 2026 as the teaser window, with full launch slated for May 2026.

ULTRAPLAN: 30 minutes of remote planning

ULTRAPLAN offloads complex planning tasks to a remote session running Opus 4.6, gives it up to 30 minutes to think, and lets you approve the result through the browser. The terminal shows polling every 3 seconds, and once approved, a sentinel value __ULTRAPLAN_TELEPORT_LOCAL__ “teleports” the result back into the local terminal.

Multi-Agent: “Coordinator Mode”

The multi-agent orchestration system in the coordinator/ directory turns Claude Code from a single agent into a coordinator that spawns, directs and manages multiple workers in parallel. Parallel research, synthesis by the coordinator, implementation by workers, verification by workers. The prompt teaches parallelism explicitly and forbids lazy delegation: “Do NOT say ‘based on your findings’ - read the actual findings and specify exactly what to do.”

And there’s more. The leak also shows in-process teammates with AsyncLocalStorage to isolate context, workers in separate processes via tmux/iTerm2 panes, memory synchronization between agents, and flags ready for BRIDGE_MODE, VOICE_MODE, WORKFLOW_SCRIPTS, AFK mode, advisor-tool and history snipping. None of that guarantees a launch, but it suggests a roadmap that’s a lot further along than the public version lets on.

The memory architecture

The memory system caught my eye. It’s not a “store everything and retrieve” approach. It’s a three-layer architecture:

MEMORY.md is a lightweight pointer index (~150 characters per line) that stays permanently loaded in the context. It doesn’t store data, it stores locations. The actual knowledge lives distributed across “topic files” fetched on demand. Raw transcripts are never reloaded into the context whole, only searched with grep for specific identifiers.

And it comes with an important discipline: the system writes to the topic file first and only then updates the index. MEMORY.md doesn’t become a fact dump. It stays a map. If you let the index turn into storage, it pollutes the permanent context and degrades the whole system.

The “Dream” system (services/autoDream/) is a memory consolidation engine that runs as a background subagent. The name is intentional. It’s Claude dreaming.

The dream has a three-gate trigger: 24 hours since the last dream, at least 5 sessions since the last dream, and acquiring a consolidation lock (preventing concurrent dreams). All three have to pass.

When it runs, it follows four phases: Orient (ls in the memory directory, read the index), Gather (look for new signals in logs, stale memories, transcripts), Consolidate (write or update topic files, convert relative dates to absolute, delete contradicted facts), and Prune (keep the index under 200 lines and ~25KB).

There are four memory types: user (user profile), feedback (corrections and confirmations), project (context about ongoing work), reference (pointers to external systems). The taxonomy explicitly excludes things derivable from the code (patterns, architecture, git history, file structure).

The dream subagent gets read-only bash. It can look at the project but can’t modify anything. It’s purely a consolidation pass.

And there’s another detail I found elegant: memory isn’t treated as truth. It’s treated as a hint. The system assumes that memory may be stale, wrong or contradicted, so the model still has to verify before trusting it. That’s the opposite of the fantasy of “throw everything in a vector database and let the magic happen.”

“Undercover Mode”

Anthropic employees (identified by USER_TYPE === 'ant') use Claude Code on public and open source repositories. Undercover Mode (utils/undercover.ts) prevents the AI from accidentally revealing internal information in commits and PRs.

When active, it injects into the system prompt:

## UNDERCOVER MODE - CRITICAL

You are operating UNDERCOVER in a PUBLIC/OPEN-SOURCE repository. Your commit
messages, PR titles, and PR bodies MUST NOT contain ANY Anthropic-internal
information. Do not blow your cover.

NEVER include in commit messages or PR descriptions:
- Internal model codenames (animal names like Capybara, Tengu, etc.)
- Unreleased model version numbers (e.g., opus-4-7, sonnet-4-8)
- Internal repo or project names
- Internal tooling, Slack channels, or short links
- The phrase "Claude Code" or any mention that you are an AI
- Co-Authored-By lines or any other attribution

There’s no way to turn it off. If the system isn’t sure it’s in an internal repository, it stays in undercover mode. That confirms something kind of uncomfortable: Anthropic uses Claude Code to contribute to open source, and the agent is instructed to hide that it’s an AI.

The internal codenames are animal names: Tengu (codename for the Claude Code project), Fennec (Opus), Capybara, Numbat (in testing). “Fast Mode” is internally called “Penguin Mode” with endpoint claude_code_penguin_mode and kill-switch tengu_penguins_off.

The most paranoid parts

There’s a piece of the analysis I almost missed because I was looking more at the hidden features. But maybe the most revealing thing about Anthropic’s mindset is in the defense mechanisms against copying and abuse.

According to Alex Kim’s analysis, there’s an anti-distillation mode that can ask the server to inject fake tools into the system prompt. The idea is to poison traffic recorded by anyone trying to distill Claude Code’s behavior to train a competitor. There’s also a second mechanism for summarizing connector text, cryptographically signed, so that part of the observable traffic doesn’t match the original raw reasoning. It’s not perfect protection. It’s another layer of friction. But it shows the company is explicitly thinking about copy-by-observation, not just traditional security.

And there’s the more aggressive part: client attestation. Every request includes a billing header with a placeholder cch=00000, and the Bun native runtime replaces that with a hash computed below the JavaScript layer. In other words, looking like Claude Code isn’t enough. The binary tries to prove that it is Claude Code. That helps explain why the fight with third-party tools like OpenCode got so sensitive: it wasn’t just a commercial or legal issue. There was technical enforcement built into the transport.

Update: the DRM died in less than 24 hours

Remember when I mentioned client attestation as “the most aggressive part”? Yeah. It lasted less than a day.

For context: Anthropic had been waging a war against third-party tools since January 2026. First came the server-side blocking of OAuth tokens from non-official clients. Then in March, the OpenCode maintainer merged a PR removing all Claude authentication from the project. The commit message was two words: “anthropic legal requests.” The Register reported that Anthropic updated its terms of service to make it explicit that OAuth tokens from Pro/Max subscriptions can only be used in the official Claude Code and on Claude.ai. People paying $100-200/month for Max who wanted to use the tool of their choice were left holding the bag.

The technical mechanism behind the block was the cch= header. With the leaked code you can see the system has two parts. The first is a version suffix: the cc_version field includes 3 hex characters derived from the user’s first message via SHA-256, using a 12-character salt embedded in the JavaScript. The second is the body hash itself: the entire body of the request (messages, tools, metadata, model, thinking config, everything) is serialized as compact JSON with the cch=00000 placeholder, then hashed with xxHash64 using a fixed seed. The result is masked with 0xFFFFF (20 bits) and formatted as 5 lowercase hex characters. The placeholder is replaced with the computed hash before the request leaves the process.

The detail that makes the difference: that substitution happens inside the Bun native runtime, written in Zig, below the JavaScript layer. Bun literally mutates the JavaScript string in-place, overwriting the 00000 bytes in the string buffer with the computed hash. If you ran the same bundle in Node or in a stock Bun, the placeholder would go to the server as-is and the request would be rejected.

And then the leak happened. With the source code exposed, @StraughterG (Jay Guthrie) announced that same night: “Yesterday I said Anthropic’s compiled Zig cch= hash was banning 3rd-party Claude clients. Tonight, the DRM is dead. We extracted the algorithm from the binary. It’s not advanced cryptography. It’s a static xxHash64 seed.”

The seed is 0x6E52736AC806831E. The full algorithm, as he explained in a thread of tweets, fits in a few lines of TypeScript:

import xxhash from "xxhash-wasm";

const { h64Raw } = await xxhash();
const body = JSON.stringify(request); // com cch=00000 no placeholder
const hash = h64Raw(new TextEncoder().encode(body), 0x6E52736ACn | (0x806831En << 32n));
const cch = (hash & 0xFFFFFn).toString(16).padStart(5, "0");
// substituir cch=00000 por cch={valor calculado}

@paoloanzn celebrated: “we cracked it. the cch= signing system in claude code is fully reverse engineered.” And he immediately put the bypass in free-code, a fork of Claude Code with telemetry removed, system prompt guardrails stripped, and all 54 experimental feature flags unlocked.

The technical point that matters: xxHash64 is not cryptography. It’s a checksum hash designed for speed, not security. The seed is static, embedded in the binary. It changes with each version of Claude Code, but within a version it’s the same for everyone. The “security” depended entirely on nobody being able to extract the seed from the compiled Zig binary. With the source code leaked, that obscurity evaporated in hours.

Now any third-party client — OpenCode, Claw-Code, whatever — can intercept the fetch(), hash the body with the correct seed, and pass the server’s validation as if it were the official Claude Code. The barrier Anthropic built to protect its $2.5 billion ARR business model was, in the end, security by obscurity over a non-cryptographic hash.

The third detail is small but says a lot about real product in production: the system detects user frustration with regex. Yes, regex. Profanity, insults, “this sucks,” that kind of thing. It’s funny to see an LLM company doing sentiment analysis with wtf|ffs|shit, but it’s also the kind of pragmatic solution you reach for when you need a cheap immediate answer, not conceptual elegance.

What the code reveals about how you use Claude Code

@iamfakeguru compiled a thread with seven technical findings from the code that any user should know:

Claude Code has a 2,000-line cap per file read. When you ask it to read a larger file, it silently truncates. Tool results are cut off at 50,000 characters. The context window compression system drops old messages to fit more new context. And there’s a difference between the access level of Anthropic employees (USER_TYPE === 'ant') and public access: internal tools like ConfigTool and TungstenTool are invisible in the external build.

The most useful finding from the thread is how Anthropic employees work around the limitations external users face. The code reveals that USER_TYPE === 'ant' unlocks internal tools, exclusive beta headers (cli-internal-2026-02-09), access to staging (claude-ai.staging.ant.dev), and a ConfigTool that lets you change configurations at runtime. External builds compile all of this to false via dead code elimination.

But the point that matters is: the CLAUDE.md you put at the root of your project gets read in full by Claude Code and injected into the system prompt. It’s literally the place where you control how the agent behaves. @iamfakeguru published a complete override with 10 mechanical rules, and then put the whole file in a separate repository: iamfakeguru/claude-md.

I’m not going to paste the whole block here. What matters is the content: it forces post-edit verification (tsc and eslint before declaring success), it imposes re-reading files before editing, it requires reading large files in chunks, it assumes silent truncation of long results, and it tells you to break larger work into phases or parallel subagents. In other words: it turns into explicit rules everything that external users were having to figure out empirically.

These aren’t magic instructions. They’re guardrails. The difference is that now we know which limits the system actually has and we can write a CLAUDE.md that works with them, not against them.

Cache bugs that cost real money

@altryne (Alex Volkov) reported cache invalidation bugs that make uncached tokens cost 10-20x more than cached ones. There are two bugs: a string substitution bug in Bun that affects the standalone CLI (workaround: use npx @anthropic-ai/claude-code instead of the installed binary), and another in the --resume flag that breaks the cache with no known workaround. Over 500 users reported similar quota exhaustion issues. If you’ve been feeling like Claude Code was burning tokens faster than expected over the past few days, it probably wasn’t your imagination.

“Staff engineer spaghetti”

The code analysis revealed real problems. A comment in the source itself admits: “1,279 sessions had 50+ consecutive failures (up to 3,272) in a single session, wasting ~250K API calls per day globally.” The fix was three lines: limit consecutive failures to three before disabling compaction.

The print.ts file has 5,594 lines with a single 3,167-line function containing twelve levels of nesting. main.tsx is 803,924 bytes in a single file. interactiveHelpers.tsx is 57,424 bytes. These are files no human can review with confidence.

The most viral reaction came from @thekitze: he asked GPT-5.4 to evaluate the codebase and the score came in at 6.5/10. The description: “This is not junior spaghetti. This is staff-engineer spaghetti: performance-aware, feature-flagged, telemetry-instrumented, surgically optimized spaghetti.” In other words, it isn’t bad code from inexperience. It’s bad code from pressure to ship fast without paying the cost of cleaning up afterwards.

@thekitze also elaborated in another thread on how the code shows a lack of basic engineering practices. And this is where I feel vindicated.

I’ve been repeating in several posts on vibe coding that speed without discipline produces exactly this. The principles I defend, small increments, tests at every step, review before committing, continuous refactoring, CI that rejects high cyclomatic complexity, are the same Extreme Programming principles that have worked since the early 2000s. Anthropic apparently didn’t follow any of them on their own product.

A 3,167-line function with 12 levels of nesting isn’t something that appears overnight. It’s accumulation. It’s the result of dozens of additions where nobody stopped to refactor because “it works, don’t touch it.” It’s the classic anti-pattern of vibe coding without discipline: generate code with AI, see that it compiles, commit, repeat. Without rigorous review. Without complexity limits in CI. Without the basic rule that if a function passes 50 lines, it needs to be broken up.

The irony is that Anthropic sells the most popular vibe coding tool on the market and doesn’t practice what I call responsible vibe coding. Claude Code is worth $2.5 billion in ARR. The code that generates that revenue is rated 6.5/10.

The “clean room” question

With the entire source code public, there’s a serious legal and competitive implication. And here I think a lot of people started using the term “clean room” with a lightness that doesn’t fit the subject.

A real clean room isn’t just “I rewrote it in another language” or “I didn’t copy and paste.” The classic model is much more annoying: one group studies the original and produces a functional spec; another group, isolated, implements from that spec without ever seeing the original code. The whole point is to reduce contamination risk.

@braelyn_ai raised another interesting point: with generative tools, somebody could try a “clean room rebuild” using tests, observable behavior and documentation, without reusing the original implementation. In theory, that makes sense. In practice, what shows up in the heat of a leak usually lands in a much grayer zone.

The Claw-Code case illustrates this well. The project presents itself as an independent rewrite and has already shifted focus to Python and Rust, but the README itself admits direct study of the exposed code and even mentions a parity audit against a local archive. So I wouldn’t call that a clean room in the strictest classic sense. I’d call it an inspired reimplementation, with a deliberate attempt to move away from the leaked snapshot.

That doesn’t mean every reimplementation is doomed. Software copyright doesn’t protect abstract ideas, generic tool flow, high-level architecture or “a CLI that does X.” It protects concrete expression. But that’s exactly why discipline matters. The more a project wants to maintain independence, the less it should rely on the leaked material as a direct benchmark.

There’s a more pragmatic detail in there: the literal copies of the leaked source will probably disappear quickly when the first DMCAs start arriving. Mirrors fall easily. That’s why a reimplementation matters more than a raw mirror. It doesn’t erase the legal discussion, but it changes the type of fight quite a bit and the chances of staying online.

That’s more or less what I did myself when I rewrote OpenClaw in Rust. The point wasn’t to copy line by line. It was to understand the behavior and rewrite the whole piece in my own code.

The satirical site malus.sh appeared today offering “Clean Room as a Service” with the tagline “Robot-Reconstructed, Zero Attribution.” The joke: AI robots recreate open source projects eliminating attribution obligations, with guarantees like “This has never happened because it legally cannot happen. Trust us.” and indemnification via an offshore subsidiary in a jurisdiction that doesn’t recognize software copyright. It’s satire, but it’s satire that describes what someone is going to actually try to do.

Update on April 2, 2026

Since the text above opens in the heat of March 31, it’s worth recording what happened right after. I decided to add this update after reading this tweet from @k1rallik, which captures the post-leak mood well but mixes verifiable facts with a touch too much epic.

First: the DMCA part got messier than it looked. The notice itself published in the github/dmca repository says GitHub processed the takedown against the entire network of 8,100 repositories, because the notification claimed that “all or most of the forks” were infringing to the same extent as the main repository. The next day, Anthropic published a partial retraction: it asked for the reinstatement of all the removed repositories, except nirholas/claude-code and 96 forks listed individually. So the thesis that the initial attempt was too broad is correct. The final picture, however, isn’t “8,100 repositories were taken down.” What happened was a formal walk-back after the mass removal.

Second: the Claw-Code project really did blow up. By the time I was updating this post, GitHub was already showing 142,829 stars and 101,510 forks. That alone is enough to say the story has moved out of the “curious fork of the leak” category and into the “real competitive side effect” category. The viral tweet that went around today is right about the size of the damage but exaggerates some details. The project’s own README describes itself as “the fastest repo in history to surpass 50K stars” and says the milestone came in two hours. I couldn’t independently confirm that historical record, so I’d rather treat that as the project’s own claim, not as a settled fact.

Third: the Rust part also needs nuance. Yes, there’s already a Rust workspace on the main branch and the Cargo.toml is at version 0.1.0. But I couldn’t find a public release on GitHub to support the line “release 0.1.0 already shipped” as a formal launch. What I can say with confidence is something else: the project already has a Python base, already has a Rust workspace, and has already drawn enough attention to keep existing even without the literal mirror of the leaked code.

What Anthropic should have done

Anthropic responded fast. They pulled the compromised package, put out a public statement, and cleaned up what they could. But the damage was done. The code was mirrored before the takedown. Mirrors on GitHub, analyses in blogs, threads on X/Twitter. There’s no way to un-publish something on the internet.

What bothers me isn’t the leak itself. Bugs happen. What bothers me is that this was avoidable with basic engineering practices:

1. Add *.map to .npmignore. One line.
2. Configure the bundler to not generate source maps in production builds. One flag.
3. Have a CI check that rejects publication if the package contains .map. A 5-line script.
4. Have a release pipeline with manual review before publishing to npm. Process, not code.

None of these are hard. They’re all the kind of thing that gets dropped when you’re moving too fast and don’t have discipline in the release process. It’s exactly what I preach as disciplined vibe coding: moving fast doesn’t mean skipping the guardrails.

And the second failure: the quality of the code itself. 512,000 lines with 3,000-line functions and 12 levels of nesting isn’t engineering. It’s accumulation. It’s what happens when you generate code with AI without rigorous review, without continuous refactoring, without CI that rejects high cyclomatic complexity. The irony of being precisely the company that sells the most popular vibe coding tool in the world doesn’t go unnoticed.

Sources

• Kuberwastaken/claude-code - Complete breakdown of the leaked code
• Alex Kim - Claude Code Source Leak: fake tools, frustration regexes, undercover mode
• VentureBeat - Claude Code’s source code appears to have leaked
• The Register - Anthropic accidentally exposes Claude Code source code
• Fortune - Anthropic leaks its own AI coding tool’s source code
• Cybernews - Full source code for Anthropic’s Claude Code leaks
• Gizmodo - Source Code for Anthropic’s Claude Code Leaks at the Exact Wrong Time
• Anthropic - Anthropic acquires Bun as Claude Code reaches $1B milestone
• Bun Issue #28001 - Source maps incorrectly served in production
• Hacker News - Claude’s system prompt is over 24k tokens with tools
• malus.sh - Clean Room as a Service (satire)
• @iamfakeguru - Thread with 7 technical findings from the code
• @altryne - Cache bugs that cost 10-20x more
• @thekitze - “Staff-engineer spaghetti” 6.5/10
• @braelyn_ai - Clean room and legal implications
• GitHub DMCA - Anthropic takedown notice processed against the network of 8.1K repositories
• GitHub DMCA - Anthropic’s partial retraction the next day
• ultraworkers/claw-code - Python and Rust reimplementation that became the main post-leak project
• @mem0ai - Analysis of the memory architecture
• @himanshustwts - Memory architecture summary
• iamfakeguru/claude-md - Override published with the complete CLAUDE.md
• @StraughterG - “the DRM is dead” - reverse engineering of the cch= hash
• @StraughterG - xxHash64 seed and TypeScript bypass code
• @paoloanzn - “we cracked it” - confirmation of the reverse engineering
• paoloanzn/free-code - Fork of Claude Code with telemetry removed and features unlocked
• a10k.co - What’s cch? Reverse Engineering Claude Code’s Request Signing
• The Register - Anthropic clarifies ban on third-party tool access to Claude

With the new Minisforum MS-S1 Max that I bought, I decided to do the migration properly. And I decided to use Claude Code from the start to speed up the process. It’s a home server, only I use it, so the risk of making a mistake is low. But if this were a real production server, I would never do this without rigorous human review at every step.

What follows is the migration writeup and a guide for anyone who wants to replicate it. If I ever have to rebuild from scratch, this post is the documentation.

Choosing the operating system

Why not Ubuntu Server again

I used Ubuntu Server on the NUC for convenience. But do-release-upgrade is Russian roulette. Every time Ubuntu releases a new version, the upgrade is a real risk of breaking things. Packages change, configs get overwritten, dependencies clash. For a server that has to be running constantly, that’s unacceptable.

Why not Arch Linux

I use Arch on the desktop and I like it. But Arch is a rolling release with no stability guarantees at all. For a desktop where I can stop and fix problems, fine. For a headless server running 49 Docker containers that has to work after every reboot, no.

Fedora CoreOS vs openSUSE MicroOS

The two modern options for a container server are Fedora CoreOS and openSUSE MicroOS. Both are immutable systems: the root filesystem is read-only, updates are atomic (they either apply in full or don’t apply at all), and rollback is instant.

The difference: Fedora CoreOS uses Ignition (declarative configuration before the first boot) and is designed to be provisioned automatically. MicroOS uses transactional-update and allows normal interactive use. For a home server where I want SSH and the ability to poke around manually, MicroOS fits better.

What makes MicroOS different

The immutable system concept changes how you operate the server:

Every package install or /etc edit goes through transactional-update, which creates a new btrfs snapshot, applies the change in that snapshot, and on the next reboot the system boots into the updated snapshot. If the change breaks something, you do transactional-update rollback and you’re back on the previous snapshot in seconds.

Updates are automatic and daily. The transactional-update.timer downloads patches, creates a snapshot, and rebootmgr reboots in a configured window (in my case, between 4am and 5:30am). If the update breaks the boot, GRUB automatically falls back to the previous snapshot.

SELinux is enforcing by default. That caused 90% of the problems during the migration, but it’s the right setting for security.

Initial setup

Hardware

• AMD Ryzen AI Max+ 395, 128GB LPDDR5X
• 96GB allocated as VRAM via BIOS (UMA Frame Buffer Size)
• 2TB NVMe (system + Docker)
• Wired 2.5Gbps network
• Synology DS1821+ NAS at 192.168.0.21 (NFS)

First steps

MicroOS install is standard. Then:

# Create user with a UID that matches the NAS (so NFS works without permission issues)
useradd -u 1026 -m akitaonrails
passwd akitaonrails

# Configure sudo (inside transactional-update shell)
sudo transactional-update shell
# inside: add akitaonrails to sudoers
exit
sudo reboot

Synology NFS

The Synology NAS exports /volume1/TERACHAD over NFS. The mount point on MicroOS is /var/mnt/terachad (not /mnt/, which lives on the immutable root).

In /etc/fstab (applied through transactional-update):

192.168.0.21:/volume1/TERACHAD /var/mnt/terachad nfs4 nfsvers=4.1,rsize=262144,wsize=262144,hard,_netdev 0 0

Details that matter: nfsvers=4.1 because 4.2 didn’t work with the Synology. rsize=262144,wsize=262144 (256KB buffers) was the biggest NFS performance improvement. hard instead of nofail so the mount keeps retrying indefinitely if the NAS disconnects temporarily.

GPU / ROCm

This step was a pain. The Radeon 8060S in the AI Max+ 395 is gfx1151, which ROCm doesn’t officially support. Three steps were needed, and all three are mandatory:

1. BIOS: set UMA Frame Buffer Size to 96GB
2. Kernel: add amdttm.pages_limit=25165824 amdttm.page_pool_size=25165824 to /etc/kernel/cmdline
3. Docker: use HSA_OVERRIDE_GFX_VERSION=11.5.1 in every ROCm container

Without step 2, ROCm only sees 15.5GB even with the BIOS allocation in place. The numbers are 96GB / 4KB (page size) = 25,165,824 pages.

sudo transactional-update shell
echo "amdttm.pages_limit=25165824 amdttm.page_pool_size=25165824" >> /etc/kernel/cmdline
exit
sudo sdbootutil update-all-entries  # OUTSIDE the transactional shell
sudo reboot

Verification:

cat /sys/class/drm/card1/device/mem_info_vram_total
# 103079215104 (96 * 1024^3)

Docker on MicroOS

sudo transactional-update --non-interactive pkg install docker
sudo reboot

sudo systemctl enable --now docker
sudo usermod -aG docker akitaonrails
# logout and login for the group to take effect

# Install standalone docker-compose (the openSUSE package doesn't include it)
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64" \
  -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
mkdir -p ~/.docker/cli-plugins
ln -s /usr/local/bin/docker-compose ~/.docker/cli-plugins/docker-compose

daemon.json

{
  "log-level": "warn",
  "log-driver": "local",
  "log-opts": {"max-size": "10m", "max-file": "5"},
  "selinux-enabled": true,
  "live-restore": true,
  "userland-proxy": false,
  "exec-opts": ["native.cgroupdriver=systemd"]
}

live-restore: true keeps containers alive across a Docker daemon restart. userland-proxy: false uses iptables directly instead of proxy processes (less overhead). selinux-enabled: true is mandatory on MicroOS.

SELinux and Docker: the biggest source of problems

This deserves an entire section because it was responsible for 90% of the bugs during the migration.

On MicroOS with SELinux enforcing, every container that writes to a host bind-mounted directory needs special handling. There are two approaches: the :Z suffix on volumes and the security_opt: label:disable option.

NEVER use :Z. Use security_opt: label:disable.

:Z tells Docker to relabel the host directory with the container’s SELinux context. Sounds like the right thing. In practice:

• SQLite databases break. The relabeling changes the file’s context and SQLite may refuse to open the WAL journal.
• NFS mounts silently ignore :Z. NFS doesn’t support SELinux xattrs. The kernel ignores the flag without error, but the container still doesn’t have permission.
• :ro,Z mounts try to relabel even when read-only, which fails on NFS and can corrupt the context on local paths.

The right solution for every container on this system:

services:
  myservice:
    security_opt:
      - label:disable     # disables SELinux enforcement for this container
    volumes:
      - ./data:/data      # NO :Z
      - ./config.yml:/etc/config.yml:ro  # NO :Z even on :ro

label:disable disables SELinux label enforcement for that container only, not for the entire system. Combined with Docker’s network and process isolation, it’s safe for a home server.

Migrating the stacks

All Docker stacks were reorganized into /var/opt/docker/<stack>/docker-compose.yml. On the old server, they were scattered across /home/akitaonrails/docker/, /home/akitaonrails/<service>/, with no pattern.

Substitutions applied across every compose file

Media stack (Plex, Radarr, Sonarr, etc.)

The media stack is the most complex. Plex needs its own LAN IP (macvlan) for direct streaming to work. The setup:

docker network create -d macvlan \
  --subnet=192.168.0.0/24 \
  --gateway=192.168.0.1 \
  -o parent=enp97s0 \
  plex_macvlan

In compose, Plex needs to be on two networks: the macvlan (for the IP 192.168.0.6) and the default bridge (so other containers like Seerr can talk to it):

plex:
  networks:
    plex_macvlan:
      ipv4_address: 192.168.0.6
      mac_address: "02:42:c0:a8:00:06"
    default: {}    # mandatory — without this, Seerr can't see Plex

A detail that almost broke me: Plex stores absolute paths in its database. If the container’s internal volume changed from /media to /data, Plex no longer finds anything. You have to use exactly the same mount target as the old compose.

Ollama with ROCm

New stack, didn’t exist on the previous server:

ollama:
  image: ollama/ollama:rocm
  container_name: ollama
  devices:
    - /dev/kfd:/dev/kfd
    - /dev/dri:/dev/dri
  security_opt:
    - seccomp:unconfined
    - label:disable
  group_add:
    - "485"   # render group GID
    - "488"   # video group GID
  environment:
    - HSA_OVERRIDE_GFX_VERSION=11.5.1
    - PYTORCH_ROCM_ARCH=gfx1151
    - OLLAMA_KEEP_ALIVE=30m
    - OLLAMA_NUM_PARALLEL=4
    - OLLAMA_FLASH_ATTENTION=1
    - OLLAMA_KV_CACHE_TYPE=q8_0
  volumes:
    - /var/lib/ollama:/root/.ollama
  ports:
    - "11434:11434"

OLLAMA_FLASH_ATTENTION=1 activates flash attention. OLLAMA_KV_CACHE_TYPE=q8_0 uses 8-bit KV cache, cutting the bandwidth needed per token in half. Free performance optimizations.

Monitoring (Grafana + Prometheus)

Grafana uses a named volume (grafana_data) which is NOT included in normal filesystem backups. That’s the reason I lost all my dashboards on the first attempt. The fix is an explicit backup of the named volume:

# On the old server:
docker run --rm -v grafana_data:/data:ro -v /tmp:/backup alpine \
  tar czf /backup/grafana_data.tar.gz -C /data .

# Transfer and restore on the new one:
docker run --rm -v grafana_data:/data -v /tmp:/backup alpine \
  sh -c "cd /data && tar xzf /backup/grafana_data.tar.gz"

Same thing for Portainer (portainer_data). Any volume defined in the volumes: block of a compose without a host path needs this treatment.

Cloudflare Tunnel

I use Cloudflare Tunnel to access all the services from outside the house without opening ports on the router. The migration was the easiest part: copy the tunnel’s JSON credentials file and the config.yml, update the IPs from .145 to .90, and bring the container up. The tunnel keeps the same ID, no need to recreate DNS.

The hostnames live in config.yml: portainer, grafana, plex, seerr, qbittorrent, syncthing, radarr, sonarr, bazarr, prowlarr, vault, gitea, kavita, and others. Everything accessible via https://<service>.example.com from anywhere.

Gitea (image registry)

Gitea acts as a private Docker registry on port 3007. The Frank FBI, Frank Mega, Frank Yomik and Mila projects have Docker images that are built and pushed to Gitea. For it to work, Docker’s daemon.json needs:

{
  "insecure-registries": ["192.168.0.90:3007"]
}

Gitea SSH gave me a problem during the migration: the old app.ini had SSH_LISTEN_PORT=22, but the container’s entrypoint also starts sshd on port 22. Conflict. Fix: GITEA__server__SSH_LISTEN_PORT=2222 as an environment variable in compose.

All 49 containers running

The migrated server runs 49 containers across 15 stacks. The media stack alone has 10 containers (Plex, Radarr, Sonarr, Bazarr, Prowlarr, qBittorrent, SABnzbd, Jackett, FlareSolverr, Seerr). The personal projects (Frank FBI, Frank Mega, Frank Yomik, Mila) add another 11. Monitoring with Grafana, Prometheus, node-exporter and cAdvisor. Utilities like Portainer, Vaultwarden, Syncthing, Organizr, Watchtower. Gitea as private Docker registry. Immich as self-hosted Google Photos. Kaizoku for manga with Kavita as the reader. Ollama with ROCm. And Bitcoin Core/Fulcrum indexing the blockchain off the NAS.

Backups: two layers

Layer 1: local btrfs snapshots (snapper)

/var lives on a 3.7TB btrfs partition. snapper creates automatic snapshots: 7 daily + 1 weekly. They’re crash-consistent, not application-consistent (postgres can be slightly inconsistent if there’s heavy writing during the snapshot).

To recover an accidentally deleted file:

sudo snapper -c var list
sudo cp /var/.snapshots/5/snapshot/opt/docker/media/radarr/appdata/config/radarr.db \
        /var/opt/docker/media/radarr/appdata/config/radarr.db

For a full stack rollback:

sudo docker compose -p media down
sudo snapper -c var undochange 7..0 /var/opt/docker/media
sudo docker compose -p media up -d

Layer 2: restic to the NAS (off-machine)

restic runs every night at 3am, doing incremental backups to /var/mnt/terachad/homelab-backups/. Retention: 7 daily + 4 weekly. Content-based deduplication, so Plex config (19GB) and Gitea repos (12GB) only transfer deltas.

Before restic runs, a pg_dump exports the postgres databases (Immich, Kaizoku). The dumps go to /tmp/homelab-db-dumps/ and are included in the backup.

What’s NOT included in the backup (re-downloadable): Bitcoin blockchain (785GB on the NAS), Docker images (re-pullable), Ollama models (re-downloadable), HuggingFace/EasyOCR caches, Plex transcoding scratch.

Large re-downloadable directories were converted to btrfs subvolumes so snapper ignores them: /var/lib/ollama and /var/opt/docker/bitcoin/fulcrum/fulc2_db.

Performance tuning

btrfs with zstd compression

Added compress=zstd:1 to the fstab for the /var partition. zstd level 1 has near-zero CPU overhead on NVMe and compresses Docker metadata, JSON configs and logs nicely. Incompressible data (SQLite, postgres) is automatically skipped by btrfs.

zram swap

With ~30GB of RAM available for the system (96GB go to VRAM), in-memory compressed swap helps. zram creates a ~15GB swap device (ram/2) with zstd compression, much faster than disk swap.

# /etc/systemd/zram-generator.conf
[zram0]
zram-size = ram / 2
compression-algorithm = zstd
swap-priority = 100

btrfs nodatacow on database directories

Copy-on-write + random database writes = write amplification. I disabled CoW on the directories that hold SQLite and postgres:

sudo chattr +C /var/opt/docker/gitea/data/gitea.db
sudo chattr +C /var/opt/docker/immich/db/
sudo chattr +C /var/opt/docker/media/radarr/appdata/config/

CPU in performance mode

On a headless server, there’s no point saving energy:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference

Persisted via systemd service cpu-epp.service.

Docker shutdown fix

A problem I discovered: Docker ships with KillMode=process, which means at system shutdown, systemd kills only dockerd and leaves all the containerd-shim processes (one per container, ~49 in my case) orphaned. systemd-shutdown then has to hunt them down one by one after the journal has already stopped, causing a silent multi-minute hang.

Fix:

# /etc/systemd/system/docker.service.d/shutdown.conf
[Service]
KillMode=control-group
TimeoutStopSec=30

The problems we ran into

This is the table of actual problems we hit during the migration. If you’re planning something similar, read it before starting:

What to tell Claude Code before you start

If I were redoing the migration from scratch, I’d give Claude Code these instructions on the very first message. In order of importance:

Tell it that SELinux is enforcing and that it should NEVER use :Z on any Docker volume, but rather security_opt: label:disable on every service. Tell it that /var/mnt/terachad/ is an NFS mount and that :Z should never appear on NFS paths. Tell it to always look at the original compose before rewriting and only change IPs, paths and container names, without inventing new volume layouts. Warn that named volumes need explicit backup (Grafana, Portainer). Explain that Plex runs on macvlan and needs default: {} in the networks. Inform that the GPU is gfx1151, not officially supported, and that it needs UMA 96GB in the BIOS + kernel TTM params + HSA_OVERRIDE_GFX_VERSION=11.5.1. And tell it that Bitcoin/Fulcrum don’t process environment variables, everything goes as an argument in command:.

Those instructions would have prevented 80% of the problems we hit.

Final server layout

/var/opt/docker/
├── bitcoin/          (bitcoind + fulcrum)
├── cloudflared/      (Cloudflare tunnel)
├── frank_fbi/        (email fraud analysis)
├── frank_mega/       (Mega clone)
├── frank_yomik/      (manga translation)
├── gitea/            (Docker registry)
├── immich/           (self-hosted Google Photos)
├── kaizoku/          (manga downloader + reader)
├── media/            (Plex + *arr stack)
├── mila/             (Discord bot)
├── monitor/          (Grafana + Prometheus)
├── ollama/           (local LLM with ROCm)
├── rip/              (HandBrake)
└── utils/            (Portainer, Vaultwarden, Syncthing, etc.)

/var/mnt/terachad/    (Synology NFS)
├── Bitcoin/data/     (blockchain, 785GB)
├── Downloads/        (torrents + nzbget)
├── Videos/           (Radarr movies + Sonarr series)
└── Ollama/models/    (model overflow if local disk fills up)

A warning about using AI to administer servers

I used Claude Code to speed up the migration. It created compose files, wrote backup scripts, configured the firewall, diagnosed SELinux problems. It worked well for my case: home server, only I use it, and I was reviewing every step.

But there are traps. Claude doesn’t know that :Z breaks SQLite unless you tell it. It doesn’t know that Fulcrum doesn’t accept env vars unless it’s already seen the Dockerfile. It will invent “better” volume layouts that break Plex because Plex stores absolute paths in its database.

If it were real production: don’t do this without review. Every compose file Claude generates, read it in full before applying. Every destructive command (rollback, delete, recreate), confirm manually. And have tested backups before you start. Claude is great for generating the first version and diagnosing errors, but the architecture decisions and the safety validations are yours.

The previous home server posts that may give additional context:

• My “Personal Netflix” with Docker Compose
• Accessing my Home Server with a real domain
• Protecting your Home Server with Cloudflare Zero Trust
• Installing Grafana on my Home Server
• Self-hosted Vaultwarden

I bought a Minisforum MS-S1 Max with the new AMD Ryzen AI Max+ 395 chip for one specific reason: this chip supports up to 128GB of unified RAM, and I can allocate 96GB of it as VRAM for the iGPU. That gives me more VRAM than any consumer gaming card, including the RTX 5090 (32GB). And that changes what I can run locally.

Why ditch the Intel NUC

The NUC was a fine Docker server for two years. But the limitation was clear: without a GPU with enough VRAM, I couldn’t run LLMs locally in any usable way. Frank Yomik, my automatic manga translation system, needed CPU-based OCR (slow) and would connect remotely to the Ollama running on my desktop (AMD 7950X3D + RTX 5090) for translation. It worked, but it meant my desktop had to be on for the server to do its job.

With the Minisforum, Frank Yomik now runs entirely on the server. The worker uses ROCm for OCR on the iGPU, and Ollama runs locally with 96GB of VRAM. Zero dependency on the desktop.

You can get a sense of the size from the photo. The NUC is the tiny cube on the left. The Minisforum is bigger but it’s still a mini-PC. It fits on the rack shelf under my Synology NAS without any trouble.

The specs

The chip is the AMD Ryzen AI Max+ 395: 16 cores / 32 threads Zen 5, with an integrated Radeon 8060S iGPU and 128GB of unified LPDDR5X. In the BIOS, I set the UMA Frame Buffer Size to 96GB, which leaves ~30GB of RAM for the operating system and containers. Plus the kernel parameters for TTM (without them, ROCm only sees 15.5GB even with the BIOS allocation in place).

The operating system is openSUSE MicroOS (more on that in the next post). The whole machine pulls under 100W, which is absurd if you’re used to dedicated GPUs that draw 450W+ on their own.

Minisforum vs my desktop: benchmarks

I ran a set of benchmarks comparing the Minisforum against my desktop (AMD 7950X3D, 96GB DDR5, RTX 5090 32GB GDDR7). The results are clear.

CPU

Mixed results. The AI Max+ 395 is better at pure parallelism (multi-core sieve), probably thanks to lower latency in the unified memory architecture. The 7950X3D wins at float and crypto because of its higher clocks and the 3D V-Cache.

LLM inference (models that fit on both)

This is where it gets interesting. For models that fit in the 32GB of the RTX 5090, the comparison is purely about memory bandwidth:

The RTX 5090 is ~7x faster. The explanation is simple: GDDR7 has ~1,792 GB/s of bandwidth. LPDDR5X has ~256 GB/s. The ratio (7x) lines up almost exactly with the measured speed difference (6.7x). LLM inference is a problem dominated by memory bandwidth. Whoever reads weights faster, generates tokens faster.

And what about prompt processing?

Prompt processing is even worse: 7-11x slower. That makes sense, because the prompt has to be processed in full before generating the first token, and it’s an even more bandwidth-intensive operation.

Where the Strix Halo wins: large models

Now we get to the reason I bought this PC. Models that don’t fit in the RTX 5090:

qwen3.5:122b with 81GB of weights running at 19 tok/s. On a mini-PC. That’s simply not possible on an RTX 5090. On the NVIDIA card, that model would have to offload layers to system RAM, dropping to 2-3 tok/s. In practice, unusable.

The difference between MoE and dense models is brutal. qwen3.5:35b runs at 43 tok/s because, despite having 35B total parameters, only ~4B are active per token. A dense 72B model like qwen2.5:72b has to read 40GB+ of weights per token, and at 256 GB/s of bandwidth, the theoretical maximum is ~6.7 tok/s. The 4.5 measured represent ~67% efficiency, which is what you’d expect for an iGPU (overhead from shared bus and drivers).

Summary: when to use which machine

A ROCm bug that’s still around

Not everything works. Models like deepseek-r1:70b, llama3.3:70b and llama4:scout crash with a ggml bug (GGML_ASSERT(ggml_nbytes(src0) <= INT_MAX) failed). The embedding tensor of these models exceeds 2GB and the ROCm copy kernel uses a 32-bit integer for the size. On CUDA (NVIDIA) it’s already been fixed, but on ROCm it hasn’t. Waiting for the fix in Ollama 0.20.0+.

LPDDR5X vs GDDR7: why this difference exists

The next question is: why is LPDDR5X so much slower?

GDDR7 is dedicated GPU memory. It’s soldered onto the graphics card, connected by a wide bus (384 or 512 bits on the RTX 5090) at high clocks. Its only job is to feed data to the GPU. LPDDR5X is unified memory that serves everything: the operating system, applications, and the GPU all at once. The bus is narrower and shared.

In practice: GDDR7 delivers ~1,792 GB/s dedicated to the GPU. LPDDR5X delivers ~256 GB/s that still need to be split between CPU and GPU. LLM inference is basically “read all the model weights from memory, multiply by the current token, generate the next token, repeat.” Whoever reads faster, generates faster. There’s no shortcut.

The Strix Halo’s advantage isn’t speed. It’s capacity. 96GB of VRAM in a 100W chip that costs a fraction of a professional GPU. The RTX 5090 is 7x faster, but it’s stuck at 32GB. Models that don’t fit, don’t run.

The alternatives: who else does this?

If 96GB isn’t enough or you want more speed, the options are limited.

The Framework Desktop uses the same AI Max+ 395 chip with up to 128GB of RAM. Same platform, same performance, but with the differential of being modular and repairable (it’s Framework, after all). In practice it’s equivalent to the Minisforum on specs and price.

Above that, the alternative is a Mac Studio with M3 Ultra. The M3 Ultra chip supports up to 512GB of unified memory, with ~819 GB/s of bandwidth (more than 3x the Strix Halo). Apple manufactures the memory chips on the package, so latency and bandwidth are superior. You could potentially allocate ~400GB as VRAM and run models that don’t fit anywhere outside of professional GPU servers.

Apple’s internal NVMe is also another level: ~7.4 GB/s of sequential read on the M3 Ultra, compared with ~14 GB/s on the Crucial T700 (PCIe 5.0). The T700 is faster on raw throughput, but Apple’s NVMe latency tends to be lower on random I/O thanks to the SoC integration.

The Brazilian price is the elephant in the room. The Mac Studio’s max config costs $9,999 in the US. With import taxes (~60% + state ICMS), it goes past R$ 110,000. The Minisforum with 128GB lands at R$ 12,000-15,000. A nearly 8x price gap buys you a lot.

If you need more than 96GB of VRAM for truly enormous models (DeepSeek-V3 with 671B parameters fits in ~400GB Q4, for example), the Mac Studio with 512GB is the only consumer option. The alternative would be professional NVIDIA A6000 GPUs (48GB VRAM, ~$6,000 each, and you’d need several in NVLink). For everything that fits in 96GB, the Minisforum gets the job done at a fraction of the cost.

And projects that promise to run big LLMs on small GPUs?

There’s a concept called “layer offloading” that projects like llama.cpp already support. The idea: if the model doesn’t fit fully in VRAM, keep some layers on the GPU and the rest in system RAM. The GPU processes the layers it has, hands off to the CPU to process the rest, and back.

In practice, it doesn’t work well. The bottleneck is PCIe: transfer speed between system RAM and GPU VRAM is ~32 GB/s (PCIe 5.0 x16). Each generated token needs to transfer data back and forth. The result is that you drop from 150 tok/s (everything in VRAM) to 2-8 tok/s (partial offload). It’s too slow for interactive use.

VRAM is the fundamental limitation because LLM inference is memory-bandwidth-bound, not compute-bound. The GPU has compute to spare. What’s missing is the ability to read the model weights fast enough. When part of the weights live in system RAM through PCIe, the entire pipeline waits on the transfer.

That’s why unified memory (like in the Strix Halo or Apple Silicon) makes a difference. There’s no PCIe in the middle. CPU and GPU access the same physical memory. The Strix Halo’s 256 GB/s is slow compared to GDDR7, but it’s 8x faster than offloading through PCIe.

Advances in LLM optimization (up to 2026)

To understand why some models run so much better than others on the Strix Halo, you have to understand what’s changed in the ecosystem over the last two years.

Mixture of Experts (MoE)

If you run local models, MoE is the advance that matters most. An MoE model has high total parameters (e.g. 122B in qwen3.5:122b), but only activates a fraction of them per token (e.g. ~4B). The inactive weights stay in VRAM but aren’t read on every token, which drastically reduces the bandwidth needed.

In the Strix Halo benchmarks, MoE models run 3-10x faster than dense models of the same size. qwen3.5:35b (MoE, ~4B active) runs at 43 tok/s while qwen2.5:72b (dense, 72B active) runs at 4.5 tok/s.

DeepSeek and training optimization

DeepSeek V3 (December 2024) showed that it was possible to train 671B parameter models at a cost an order of magnitude lower than predicted. They combined MoE with FP8 quantization during training (not just inference), multi-stage training with curriculum learning, and several inter-GPU communication optimizations. The impact: everyone copied. Qwen, GLM, MiniMax, all of them adopted variations of the technique.

Quantization: from FP16 to Q4 without losing much

Quantization compresses model weights from 16 bits (FP16) to smaller formats: 8 bits (Q8), 4 bits (Q4), or even 2 bits. A 70B model that would take ~140GB in FP16 fits in ~40GB at Q4_K_M. Quality loss exists, but in modern formats (GGUF Q4_K_M, AWQ, EXL2) it’s small enough for practical use.

GGUF (the llama.cpp format) became the standard for local inference. AWQ and GPTQ are alternatives with more sophisticated calibration, but the ecosystem converged on GGUF because it works on CPU, CUDA and ROCm without recompilation.

Distillation: smaller models that know more

Distillation is training a small model using the responses of a large model as the teacher. Microsoft’s Phi-4 (14B) was trained with distillation from GPT-4 and competes with 70B models on several benchmarks. Qwen3 did the same: qwen3:14b is surprisingly capable for its size.

Flash Attention and optimized KV Cache

Flash Attention (Tri Dao, 2022) changed how attention is computed: instead of materializing the full attention matrix in memory, it processes in blocks keeping the data in the GPU’s on-chip SRAM, reducing memory consumption from O(n²) to O(n). Without that, contexts of 128K+ tokens would be impractical. It already went through versions 2 and 3, with optimizations for FP8 and async operations on H100. PagedAttention (vLLM, UC Berkeley) did the same for the KV cache during serving: applies virtual memory concepts to the cache, eliminating fragmentation and improving throughput by 2-4x.

In Ollama, I set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 on the server. The first activates flash attention, the second uses 8-bit KV cache instead of fp16, cutting the bandwidth needed per token in half. These are zero-hardware-cost optimizations that improve throughput measurably.

What Qwen, Kimi, MiniMax and GLM are doing

Qwen (Alibaba) has consistently been the best price/performance in open source models. Qwen3:14b is dense and strong; Qwen3.5:122b is MoE and runs surprisingly well in 96GB. GLM-4.7 (Zhipu AI) is notable for offering bf16 full precision versions that fit in 96GB. MiniMax experimented with long contexts (up to 4M tokens). Kimi (Moonshot AI) focused on large context windows with linear architectures.

What runs well in 96GB of VRAM

With 96GB on the Strix Halo, the models that work well for daily use:

Dense 70B+ models (deepseek-r1:70b, llama3.3:70b) are still blocked by the ROCm bug I mentioned. When it gets fixed, they should run at ~4-6 tok/s, usable for batch but not for interactive chat.

Conclusion

I bought the Minisforum to run models that don’t fit in any gaming GPU. For that, it works. It’s not fast. 19 tok/s on a 122B model isn’t the experience you get with Claude or ChatGPT. But it’s local, it’s private, and it runs on my shelf consuming less power than an old lightbulb.

For people asking about the Mac Studio: if you have the budget, it’s the best machine for running local LLMs. 512GB of unified memory, 819 GB/s of bandwidth, mature Metal/mlx ecosystem. You can run DeepSeek-V3 in full Q4. But in Brazil, with import duties, it crosses R$ 110k. The Minisforum with 128GB at R$ 12-15k is the realistic option.

And for people who think you can work around the VRAM limitation with layer offloading: you can’t. PCIe is too slow. The model has to fit fully in VRAM for inference to be usable. It’s the reason gaming GPUs with 32GB of ultra-fast GDDR7 are still capped on model size, and why the unified memory of the Strix Halo and Apple Silicon changed the equation.

In the next post I tell the story of how I migrated the entire home server to the Minisforum using Claude Code, the problems I ran into, and how openSUSE MicroOS behaves as a Docker server operating system.

With that out of the way, let me explain why I built this.

The problem with the Brazilian press

I’m fed up.

Fed up of opening the newspaper and having to do mental gymnastics to separate information from narrative. Fed up of outlets like Folha de São Paulo, UOL, Carta Capital, Brasil 247, O Globo and several others that use misleading headlines, omit context on purpose, transpose evidence from other countries with no caveat, and create the appearance of consensus among outlets that are all saying the same thing because they’re following the same coordinated agenda.

This isn’t conspiracy theory. It’s a verifiable editorial pattern. And the worst part: most readers don’t have the time or the tools to notice. You read the headline, read the first two paragraphs, and walk away with the impression the article planted in your head.

What Frank Investigator does

You give it a news article URL. The system fetches the article with a headless Chromium (to get past paywalls and cookie walls), extracts the content while filtering out ads and sidebars, and decomposes the text into verifiable claims. Then the real analysis starts: divergence between headline and body, rhetorical fallacies (false causality, appeal to authority, strawman, bait-and-pivot), source distortion, temporal manipulation, selective citation, authority laundering. It expands the links cited in the article to verify whether the sources actually say what the author claims. It evaluates each claim with consensus from 3 AI models (Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro) through OpenRouter. It detects contextual gaps, coordinated campaigns across outlets, and measures the ratio between passion and evidence. 15 stages in total.

The central principle is “Truth Above Consensus”: a primary source (official data, government document, original academic study) vetoes any number of secondary sources repeating the same information. Ten newspapers repeating the same thing without a primary source still adds up to zero.

Let me show you five real examples.

Example 1: The Noelia Castillo case (BBC)

The BBC published a story with the headline “Noelia Castillo dies: a 25-year-old’s fight in the Spanish courts against her father to receive euthanasia.” At first glance it looks like a story about the right to euthanasia and a family dispute. A young quadriplegic woman who fought against her religious father’s opposition to exercise her right to die.

Except when you compare it with other outlets, like Veja, facts come up that the BBC almost completely omitted. And those facts change everything.

Noelia was taken from her family by the Spanish government at age 13 and placed under state custody. While she was under that custody, she suffered multiple gang rapes. The sexual violence resulted in serious psychiatric damage and a mental health history that already added up to 67% disability rating before the events of 2022. When she attempted suicide in October 2022 by jumping from the fifth floor of a building, she was left paraplegic. Her disability rating rose to 74%.

The euthanasia request was approved by Catalonia’s Guarantee and Evaluation Commission. The procedure was scheduled for August 2, 2024, but was suspended for over 600 days because of the father’s appeals. Five judicial instances ruled on it. The Constitutional Court dismissed any violation of fundamental rights. Spain’s Supreme Court denied the appeal. The European Court of Human Rights rejected the suspension request. On Friday, March 26, 2026, Noelia underwent euthanasia at the Sant Camil Residential Hospital, in Catalonia’s Garraf region.

But there’s a detail that Veja mentions that’s disturbing: Noelia reportedly expressed doubts before the procedure. And the hospital allegedly accelerated the process because her organs were already committed for donation.

The Frank Investigator report compared the coverage from several outlets.

The cross-analysis of the articles showed that some outlets, like the BBC, omitted facts that change the interpretation of the entire case. Others, like Veja, included the full context. The episode of gang sexual violence in October 2022, the criminal investigation, the psychiatric history since age 13, all of that appears in some coverage but is completely absent in others. And among the outlets that omitted these facts, none touched on the ethical question of whether the physical basis for the euthanasia request derives from a suicide attempt.

The convergent framing across outlets is one of “judicial battle,” “death she asked for,” “to stop suffering,” “to leave in peace,” softening the definitive nature of the procedure and positioning Noelia as the heroic protagonist and the father as the obstructive antagonist. The father, Gerônimo Castillo, and his Christian Lawyers are labeled “ultra-Catholics” or “ultra-conservatives” without any independent source backing that editorial classification.

Narrative coordination came in at 55%. It doesn’t seem to be active coordination between newsrooms, but thematic editorial alignment: everyone bought the autonomy-of-the-individual narrative without questioning it. What raises the score is the convergence of omissions. No outlet explains the legal distinction between the ECHR “actively authorizing” the euthanasia and simply refusing the provisional protective measures the father requested, which is what actually happened. The medical and bioethical implications of approving euthanasia in a case stemming from a suicide attempt show up nowhere. And any voice critical of the procedure is automatically framed as religious or ideological, never as medical or legal.

It’s the kind of case where the omission is the manipulation. The outlets that omitted these facts didn’t lie at any point. But by framing it as a “family dispute over euthanasia rights” and omitting the causal chain (state custody → gang rapes → psychiatric damage → suicide attempt → paraplegia → euthanasia), the reader walks away with an impression that’s radically different from reality. Comparing coverage is exactly the kind of thing the investigator does well: exposing what each outlet chose to show and what it chose to hide.

Example 2: “Government cuts taxes on nearly a thousand imports” (UOL)

UOL published that the government cut import taxes on nearly a thousand products, from medicine to hops. Positive headline, framed as a benefit to the consumer. Several other outlets ran the same thing with similar framing: the government did something good, prices are going down.

Except Gazeta do Povo tells the other half of the story. The government didn’t cut old taxes. What actually happened was: at some point before February 2025, the government raised import tariffs on more than 1,200 items, a measure that would have generated an estimated R$ 14 billion in revenue. Then, under social-media and public pressure, it partially backed down. Tariffs on around 970 capital goods, computing and telecommunications items were dropped to zero. Taxes on 120 IT products were reduced. And now they’re calling that a “tax cut.”

In other words: they raised taxes, took public pressure, partially walked it back, and rebranded it as a generous concession. The majority of the 1,200+ items that had tariffs raised still have higher tariffs than before. No final consumer prices went down. Prices went back to where they were for some products, and remain higher for most.

The Frank Investigator report cross-checked the articles and exposed what was omitted.

What the comparison between articles exposes: none of the outlets with the positive framing mentions that most of the 1,200+ items with raised tariffs still have higher tariffs after the two rounds of “cuts.” The fiscal impact on the R$ 14 billion revenue target from the original increases — nobody calculates it. Voices from the domestic industry that may be hurt by the tariff reduction on imported competitors — none. The Gecex’s objective criteria for defining “insufficient supply in the internal market” — they don’t show up.

The two articles analyzed build the same positive framing for the government. The paradox that the “cuts” are a partial reversal of increases made by the same government the previous year stays buried or simply absent.

It’s the classic kind of manipulation through reframing. Nobody lied. But “government cuts taxes” and “government backs down on tax hike after public pressure” describe the same event with opposite impressions. The editorial choice of which version to publish IS the manipulation.

Example 3: “Globo apologizes for PowerPoint” (Brasil 247)

This case blew up over the past few days. GloboNews showed a diagram on the Estúdio I program connecting President Lula, his ministers and Daniel Vorcaro, owner of Banco Master, who’s at the center of documented fraud. Globo later issued a public retraction, called the material “erroneous and incomplete,” and fired an editor.

Brasil 247 published a story with the headline “Globo apologizes for PowerPoint that tried to dump the Master case on Lula’s lap.”

The Frank Investigator report exposed what’s going on under the hood:

The central fact is real: Globo apologized and fired someone. But Brasil 247’s framing goes way beyond what the facts support. The headline saying “tried to dump it on Lula” attributes deliberate intent where the documents point to an editorial mistake. Globo’s retraction described the material as “erroneous and incomplete,” not as an attempt to incriminate anyone.

What stands out in this case is the coordinated campaign. The investigator gave it 62% narrative coordination.

Several outlets editorially aligned with the government used the phrase “without evidence” in a convergent way to describe the association between Lula and the Master case. All of them focused on Globo’s mistake as the central narrative point, instead of investigating the actual connections. No outlet mentioned which other political names were excluded from the original PowerPoint. None investigated Vorcaro’s documented connections with different spheres of power. The focus is meta-journalistic: they criticize the broadcaster instead of covering the scandal.

The fallacies detected: loaded language (“tried to dump,” “without evidence” used to frame an editorial mistake as a deliberate political attack), false causality (Globo’s retraction doesn’t prove the connections are false), cherry-picking (highlights the omission of names tied to the Lula government without contextualizing which other names were omitted), and bait-and-pivot (uses Globo’s apology as a hook to minimize the Banco Master scandal).

And the questions none of these outlets asked: which Supreme Court justices and politicians from other parties were also excluded from the PowerPoint? Does the Master case have documented connections to Lula government figures, or are they limited to previous administrations? Does Brasil 247, which is publishing this article, have a declared editorial alignment with the Lula government? What was the journalism Ethics Council’s reaction?

Overall confidence: 13%. The article doesn’t fabricate facts. But it selects, frames and omits in a way that builds a narrative the data doesn’t support.

Example 4: “Why expensive fuel is good” (Folha)

The economist Bernardo Guimarães published a column on Folha arguing that expensive fuel is good for society because it stimulates innovation in clean energy. He cites real academic articles (Popp 2002, NBER) and has verifiable academic credentials (PhD from Yale, professor at FGV EESP). It looks solid.

The full Frank Investigator report shows another picture.

The article is right to cite real studies. But it omits who pays the bill. The column completely ignores the distributive impact: low-income populations, residents of peripheral and rural areas, who depend more on personal vehicles and have less access to clean alternatives. For an economist, ignoring distributive effects is either incompetence or editorial choice.

There’s a worse problem. The empirical evidence he cites (US patents, electric vehicle data in California between 2014-2017) is transposed to Brazil with no caveat at all. Brazil has an ethanol matrix and flex-fuel infrastructure that completely changes the causal mechanism. The article treats it as if the Brazilian consumer were in the same situation as the Californian one, which is false.

And there’s the context the article mentions in passing but doesn’t develop: the war in Iran is making fuel prices rise around the world. Brazil should be in a privileged position thanks to the pre-salt and to ethanol. But decades of mismanagement and corruption at Petrobras mean we’re paying the same price as the rest of the world. Instead of questioning that, the column sells the idea that “at least it’ll stimulate clean energy.” It’s a rationalization of a problem that shouldn’t exist.

The investigator identified 5 questions the article refused to address, with 35% contextual completeness. The fallacies detected include false dilemma (presents expensive fuel as the only viable climate policy), cherry-picking (acknowledges short-term inelasticity but emphasizes only long-term effects), and loaded language (describes alternatives as “playing at planting a little sapling”).

Overall confidence: 25%. It isn’t fabricated disinformation. It’s opinion with cherry-picked evidence in favor of the thesis and omission of relevant counterpoints.

Example 5: “There’s no strong cinema without streaming regulation” (O Globo)

Renata Magalhães, president of the Brazilian Academy of Cinema, gave an interview in Miriam Leitão’s column on O Globo arguing that regulating streaming is a necessary condition for strengthening Brazilian cinema.

The Frank Investigator report found serious problems.

First: the central claim that “many films have low audiences” appears with no empirical data. No numbers, no historical series. It’s a pure authority claim with no analytical basis.

Second: there’s an internal contradiction the article doesn’t resolve. The text opens by saying that Brazilian cinema is “in international spotlight” (awards, festivals). And then immediately argues that the absence of regulation prevents the industry from being strengthened. But if Brazilian cinema is already winning international awards without that regulation, the argument that the regulation is a necessary condition collapses. The article doesn’t address that contradiction.

And here’s the elephant in the room that none of these articles mentions: people simply don’t want to watch most of these films. Internationally awarded Brazilian cinema is made to compete in Cannes and at the Oscars, not to fill movie theaters in Brazil. Instead of asking why the Brazilian audience isn’t interested, the industry would rather ask for streaming regulation to force platforms to fund and screen content that has no spontaneous audience. It’s the classic playbook: use public money and regulation to keep alive an industry that doesn’t sustain itself in the market.

The interviewee is the president of the Brazilian Academy of Cinema. She has a direct institutional interest in the regulation. The article doesn’t present any opposing voice and doesn’t discuss the costs to consumers: subscription price increases, catalog reduction. A single source, with a declared conflict of interest, with no counterpoint.

And here comes the coordinated campaign, with 55% narrative coordination. Multiple outlets reproduce the same emotional framing: cinema “at risk,” regulation as “essential.” None of the identified sources discusses comparable empirical international evidence on the effectiveness of content quotas (Europe has experiences with contradictory results). None mentions the Brazilian Academy of Cinema’s conflicts of interest. The fact that the internationally awarded Brazilian productions were made without the proposed regulation is omitted convergently across all outlets. Only one isolated site (targethd.net) mentioned negative impacts on consumers.

Overall confidence: 9%. The article is legitimate editorial advocacy, but with analytical flaws that limit its informational value to nearly zero.

What the investigator analyzes

The 5 examples above show different patterns, but the analysis criteria are the same.

The strongest signal of manipulation is omission. It isn’t what the article says that misleads, it’s what it leaves out. The contextual gap analysis identifies the questions the article should have answered and didn’t, and searches for counter-evidence on each one. In the examples above, articles that omit most of the relevant context aren’t informing anyone. And when cross-comparison between outlets shows that some covered the facts and others didn’t, like in the Noelia case or the tax cut, it’s hard to argue that the omission was accidental.

Then comes the detection of coordinated campaigns. Several newspapers covering the same subject is normal. All of them using the same loaded language, focusing on the same points and omitting the same counterpoints at the same time isn’t. The strongest signal of coordination isn’t what outlets say in common, but what they omit in common.

There’s also reframing, which is more subtle. In the tax case, the government raised tariffs, backed down under pressure, and the outlets called it a “cut.” Nobody lied technically, but the choice of framing completely changes the interpretation. This kind of manipulation is harder to detect because each individual statement is defensible.

The rhetorical fallacies catch specific constructions: false dilemma (“either you regulate or cinema dies”), bait-and-pivot (open with a positive fact and pivot to a crisis narrative), loaded language (“without evidence” used to attribute intent). Each detected fallacy comes with the exact citation of the passage and the explanation of why that construction is problematic.

And there’s the principle that ties it all together: if 10 outlets repeat the same claim citing each other, the LLM consensus has to reflect that the chain of evidence is circular, not that the claim is well supported. Volume of coverage is not a proxy for truth.

Why you can’t “just ask ChatGPT”

The first reaction many people will have is: “why do I need this if I can just paste the article into ChatGPT and ask it to analyze?”

Try it. Take a political article and ask ChatGPT to criticize it. It’ll criticize. Take the same article and ask it to confirm. It’ll find arguments to confirm. The LLM isn’t searching for truth. It’s predicting which response you most likely want to hear given the framing of your question. If you ask “analyze the problems with this article,” the model will find problems. If you ask “is this article correct?”, it’ll find merits. It’s automated confirmation bias.

There’s another problem. General-purpose LLMs are trained to be agreeable. The sycophantic tendency (agreeing with the user) is documented in every large model. If your conversation history indicates you’re left-leaning, the model tends to frame answers in a way that pleases that profile. If you’re right-leaning, same thing in the other direction. It’s not lying on purpose. It’s optimizing for user satisfaction, which is literally the metric it was trained for via RLHF.

And worse: LLMs hallucinate. If they don’t have enough evidence to support the answer they think you want to hear, they invent it. They fabricate plausible citations and data, attribute statements to people who never made them. If you ask it to criticize an article about fuel, it might invent a fictional study that “proves” the opposite of the article. It sounds convincing. But it doesn’t exist.

Frank Investigator was built precisely to avoid these problems. The first design decision is that no human asks the LLM a question. There’s no open-ended prompt like “analyze this article.” Each step in the pipeline has structured prompts that ask for specific analyses: “list the rhetorical fallacies in this passage,” “identify which contextual information is missing,” “compare the headline with the body.” The model doesn’t know whether the operator agrees or disagrees with the article, because the operator never expresses an opinion. That eliminates confirmation bias at the root.

To deal with hallucination, every analyzer that uses an LLM includes the instruction “CRITICAL — NO HALLUCINATION: Only reference URLs, sources, claims, quotes, and data that are EXPLICITLY present in the input provided to you. Do not invent, guess, or fabricate any URL, source name, statistic, quote, or claim. If you cannot verify something from the provided text, mark it as unverifiable — never fill in details.” It doesn’t eliminate hallucination completely, but it cuts it down a lot. And since 3 models from different companies (Anthropic, OpenAI, Google) answer the same questions, when one hallucinates the other two usually disagree. The consensus is weighted by confidence, not by simple majority. If two models say “supported” with 70% confidence and one says “mixed” with 95%, the “mixed” weighs more. The further apart the disagreement, the bigger the penalty on final confidence. If one model starts giving inconsistent answers, it gets put in quarantine and the other two carry on.

But the safeguard I consider most important is the primary source veto. If a primary source (IBGE data, court ruling, original study) contradicts a claim, confidence is capped at 60% and the verdict is forced to “mixed,” even if all 3 LLMs say “supported.” Ten newspaper articles repeating a claim don’t override one official datum that contradicts it. Along the same lines, if 5 stories “confirm” a claim but they all come from the same editorial group (Folha/UOL, Globo/G1/Valor), the system knows they’re the same voice and reduces the weight. Volume doesn’t replace independence.

None of this makes the system perfect. But it’s categorically different from pasting text into ChatGPT and asking “what do you think?”.

The numbers

Stack: Ruby 4.0.1, Rails 8.1.2, SQLite with WAL mode, Solid Queue (jobs inside Puma), Solid Cable (WebSockets), Tailwind CSS v4, headless Chromium via Ferrum CDP, deploy with Kamal to GitHub Container Registry. AGPL-3.0.

The development process

The project has 129 commits in 9 active days of work (March 11-16 and 25-27). The first day was the heaviest: more than 60 commits on March 11 alone, going from zero to a working system with content extraction, claim decomposition, LLM evaluation, and a web interface with live updates via Turbo Streams.

The commits tell the story. It started with the foundation:

1e32d5a - Initial Frank Investigator foundation
564eb97 - Add recursive source crawling and RubyLLM scaffold
c8c5357 - Add Brazil source registry and authority connectors
3d30617 - Add U.S. authority profiles, source role modeling, and specialized connectors

Then came the misinformation analyzers, one by one:

a65fef7 - Add rhetorical fallacy analyzer for detecting narrative manipulation
5dcf99d - Add headline-body divergence detection and headline citation amplification
a113efd - Add smear campaign defense with circular citation and viral volume detection
56d501a - Add media ownership modeling, syndication detection, and independence analysis

The hardening phase against noise and false positives is the one that took the most time:

46e4bc5 - Add pre-fetch defenses: URL filtering, circuit breaker, fetch prioritization
168fea1 - Add post-fetch content gate, claim noise filter, and duplicate content skip
b2843d7 - Add paywall detection, pricing noise filter, and ofertas.* host rejection
b1b230d - Rewrite Chromium fetcher with Ferrum CDP for anti-bot evasion

Then came the interface, deployment, and the more advanced analyzers:

c5afcea - Replace custom CSS with Tailwind CSS v4 and rewrite all templates
95a8ec4 - Add Docker deployment with bin/deploy script
ccfacc9 - Add coordinated narrative detection across media outlets
ba3e2f4 - Add 6 new misinformation detection analyzers with cross-analysis
fda984b - Add LLM-generated investigation summary with quality assessment
a4ebe78 - Add contextual gap analysis to detect manipulation through omission

And in the most recent commits, simultaneous 3-model consensus and test optimization:

5cc9c05 - Enable 3-model LLM consensus: Sonnet 4.6, GPT-4.1 Mini, Gemini 2.5 Pro
cdf5fb5 - Batch 5 content analyzers into single LLM call, add anti-hallucination
28c305e - Add WebMock stubs for LLM and web search — tests run in 1.4s (was 540s)

The tests that used to run in 9 minutes now run in 1.4 seconds with WebMock stubbing of every LLM and web search call. That made a huge difference in iteration speed.

Limitations

News fact-checking is a hard problem. A single article doesn’t contain everything needed for a complete analysis. The author chose what to include and what to omit, and that choice is already the first level of manipulation. The sources cited inside the article were selected by the author to support their narrative, not to give a balanced view.

Frank Investigator doesn’t tell you what’s true and what’s false. The result is a report with strong and weak points, not a “true” or “false” stamp. Even with all the safeguards I described above, the counter-evidence searched for automatically may not be the most relevant. The fallacies detected can have false positives. The coordinated campaign analysis depends on what web search returns at the moment of the lookup.

Use the reports as a starting point to form your own opinion, not as a final verdict.

The project is open source, AGPL-3.0. If you want to contribute, test, report bugs or suggest improvements: GitHub. If you want to follow the analyses, the The Makita Chronicles newsletter is going to have a “Notícias Duvidosas” (Dubious News) section with the summary and a link to the full report of each investigation.

It “works,” but this project was more about the exercise. With that out of the way, let me tell you why I did it.

The problem with OpenClaw

OpenClaw is a gateway that connects messaging channels (Telegram, Discord, Slack, WhatsApp, etc.) to AI providers (OpenAI, Anthropic, Ollama). You configure it, bring up the server, and you can chat with LLMs straight from Telegram or any other channel. It’s a popular project with plenty of activity.

Too much activity, actually.

I did a depth-1 clone of the repo and ran tokei:

Over a million lines of TypeScript. The 2,799 test files sound like a lot in absolute numbers, but proportional to the codebase size, the coverage is low. Most of the code lives in 29 packages of a monorepo with 21 channel extensions.

I went looking for more commits to understand the development pace. In the 100 commits I managed to pull, all of them landed in just 2 days (March 9 and 10). ~50 commits per day, from 42 different contributors. Vibe coding to the extreme.

The conclusion is the one you’re imagining: enormous volumes of AI-generated code being dumped into a repository at a speed that makes serious human review impossible. And that bothered me enough to go investigate further.

The security audit

I asked Claude to do a complete security audit of OpenClaw’s code. The report found:

Seven critical vulnerabilities. Let me list a few:

• Timing side-channel in token comparison — safeEqualSecret() does an early return on the type-check, letting an attacker distinguish malformed tokens from wrong tokens by measuring latency.
• eval() in the browser tool — arbitrary JavaScript execution with no sandbox.
• Shell with no allowlist — any tool can run any command on the system.
• Slack webhooks with no signature verification at all.
• Transcripts and config in plaintext on disk, no encryption.
• No effective rate limiting — IPs can be spoofed if the operator configures trusted proxies broadly.

Those are just the ones Claude found in an automated scan. There are probably more.

I’m not going to run that on my machine. Not even inside a Docker container. A gateway that receives webhooks from the internet, executes shell commands, connects to AI APIs with your keys, and stores conversation history, all of it with 7 known critical vulnerabilities? No, thanks.

So I did what any rational developer would do: I decided to build my own.

The first attempt: Claude Code

I started the most direct way. Cloned OpenClaw, pointed Claude Code at the code, and asked: “rewrite this in Rust.”

Didn’t work. The codebase is too big. Over a million lines of TypeScript spread across 29 packages. Claude can’t keep all of it in context. The initial result was very incomplete: lots of types created but no implementation, todo!() everywhere, too much boilerplate and not enough functionality.

I switched to Codex 5.4 to test. Same thing: I asked it to analyze and rewrite. It improved a bit in certain aspects, but the fundamental problem is the same. No AI today takes a project of that size and rewrites it in one go. The context doesn’t fit.

The technique that actually works

What works is going slowly. One step at a time.

Ask Claude (or Codex) to analyze the original code in stages. Make a long plan detailing each feature. Then implement one feature at a time in Rust, with tests, commit, and repeat. It’s tedious, but it produces code that compiles and actually works.

The reason is simple: the original code is so massive that no AI agent (and not even several in parallel) can keep all of it in context at the same time. You have to decide what matters, implement that, validate, and move on to the next thing.

And you have to decide what to cut. OpenClaw has 21 channel extensions: Google Chat, iMessage, IRC, Teams, Matrix, Mattermost, Nostr, Twitch… I don’t need any of those. I kept the mainstream channels: Web, Telegram, Discord, Slack, Signal, WhatsApp and Email. TTS? Out. Polls? Out. WhatsApp Web through Baileys? Out, I use the official Cloud API. They’re features that add complexity without proportional value.

Discovering IronClaw

In the middle of development, I came across IronClaw, which presents itself as “OpenClaw in Rust.” Great, I thought. Let me see what they did.

I cloned the repo and asked Claude to write a comparison report. IronClaw has good things. I adopted 12 features:

Circuit breaker with retry and exponential backoff for LLM provider resilience. Credential leak detection in the output. LLM response cache with SHA-256 of the prompt. Cost tracking with budget guards (warning at 80%, block at 100%). Extended thinking for Claude 3.7+ and o1. MCP client for external tool servers. Lifecycle hooks on inbound, tool calls and outbound. Smart model routing that sends simple queries to cheaper models. Tunnel support (cloudflared, ngrok, tailscale). Interactive REPL (frankclaw chat). Routines with event triggers beyond cron. Job state machine with auto-repair.

But IronClaw depends on PostgreSQL + pgvector, has a WASM sandbox (wasmtime adds ~10MB), and is part of the NEAR AI ecosystem. I want a single binary with embedded SQLite and zero external dependencies.

What FrankClaw brings from OpenClaw

The OpenClaw core is there: 7 messaging channels, multi-provider with failover, agent runtime, command system, skills, subagents, browser automation through CDP, bash tool with allowlist, cron jobs, interactive REPL. Plus the 12 IronClaw features I listed above.

But several pieces had to be rewritten in a way the original didn’t. Context compaction, for example, uses a sliding window with token estimation, message pruning and automatic repair of tool pairs that get orphaned when the context is cut. Provider failover is now model-aware: if you ask for a Claude model and the current provider is OpenAI, it skips automatically instead of erroring out. The canvas renders SVG, HTML and Markdown with revision conflict detection. Things that in OpenClaw either didn’t exist or were half-built.

Then came the phase of adding what was missing for parity. FrankClaw today has 30+ LLM tools: web_fetch (SSRF-safe with HTML-to-text), web_search (Brave API), file_read/file_write/file_edit (sandboxed in the workspace with path traversal protection), pdf_read, image_describe (through vision models), audio_transcribe, sessions_list/sessions_history/sessions_delete, message_send/message_react, cron_list/cron_add/cron_remove, config_get (auto-redacts secrets), agents_list, memory_get/memory_search, plus the browser tools (browser_open, browser_extract, browser_snapshot, browser_click, browser_type, browser_wait, browser_press, browser_sessions, browser_close, browser_aria).

Some additions that don’t exist in the original OpenClaw: a memory/RAG system with SQLite FTS5 and embeddings (OpenAI, Gemini, Voyage) that automatically syncs workspace files. An OpenAI-compatible API (/v1/chat/completions and /v1/models), so any client that speaks that protocol (Cursor, Continue, Open WebUI) can use FrankClaw as a backend with no adaptation. A ratatui TUI for people who prefer the terminal. Interactive approval of destructive tools before execution.

Smaller things that make a difference in practice. You can configure multiple API keys per provider with round-robin and automatic backoff, so if one key hits the rate limit, the next one takes over. The model catalog already knows context windows and costs of OpenAI and Anthropic models without you having to configure them. URL extraction from messages has a private IP blocklist against SSRF. The command system accepts inline directives (/think, /model) in addition to aliases.

On the operational side: ACP (Agent Client Protocol) over JSON-RPC 2.0 on top of NDJSON for people who want to integrate programmatically. Plugin system with manifests and enable/disable lifecycle. i18n with 9 locales via FRANKCLAW_LANG. Workspace identity files (SOUL.md, IDENTITY.md) to define the bot’s personality per project. Channel health monitor with auto-restart. WebSocket with ping keepalive that survives proxy and tunnel timeouts. frankclaw start/stop/status for people who want to run it as a daemon with PID tracking. And the entire configuration migrated from JSON to TOML.

Hardening: where the real difference is

The audit report we ran on OpenClaw found 7 critical and 9 high vulnerabilities. FrankClaw fixes all of them:

FrankClaw compiles with #![forbid(unsafe_code)] in all 13 crates. Zero unsafe blocks.

And the audit didn’t stop at OpenClaw. We did a per-component audit in 14 phases comparing each part of FrankClaw against the original: channels, providers, runtime, tools, sessions, crypto, cron, webhooks. All documented.

Deploy: how to install

FrankClaw runs with Docker Compose. Three containers: gateway, headless Chromium (for browser tools), and Cloudflare tunnel (to receive webhooks).

1. Clone and configure

git clone https://github.com/akitaonrails/frankclaw.git
cd frankclaw
cp .env.docker.example .env.docker

Edit .env.docker with your keys:

# Providers de IA (preencha os que usar)
OPENAI_API_KEY=
ANTHROPIC_API_KEY=

# Canais (preencha os que quiser usar)
TELEGRAM_BOT_TOKEN=         # via @BotFather
WHATSAPP_TOKEN=             # Meta Business Platform
WHATSAPP_PHONE_ID=
WHATSAPP_VERIFY_TOKEN=
DISCORD_BOT_TOKEN=
SLACK_BOT_TOKEN=
SLACK_APP_TOKEN=

# Embedding providers (só se usar memória/RAG)
# GEMINI_API_KEY=
# VOYAGE_API_KEY=

# Opcional: criptografia de sessions (recomendado)
# Gere com: openssl rand -base64 32
FRANKCLAW_MASTER_KEY=

# Opcional: scan de malware em uploads
VIRUSTOTAL_API_KEY=

2. Configure the gateway

The frankclaw.toml file defines agents, models and channels. Use the wizard or the examples:

# Generate a base config with the web channel
cargo run -- onboard --channel web

# Or copy from the examples
ls examples/channels/

For each channel, the CLI has ready-made templates:

cargo run -- config-example --channel telegram
cargo run -- config-example --channel whatsapp

3. Cloudflare Tunnel (to receive webhooks)

If you’re going to use channels that need a webhook (Telegram, Discord, Slack, WhatsApp), you need a public tunnel. The Docker Compose already includes cloudflared:

# Copy your Cloudflare credentials
cp docker/cloudflared/config.yml.example docker/cloudflared/config.yml
cp ~/.cloudflared/<tunnel-id>.json docker/cloudflared/credentials.json
cp ~/.cloudflared/cert.pem docker/cloudflared/cert.pem
# Edit config.yml with your hostname

4. Bring it up

docker compose up -d

The gateway comes up on port 18789 (internal to Docker). cloudflared routes external traffic. Chromium stays on the internal network for browser tools.

To test locally without Docker:

cargo run -- chat

Opens the interactive REPL straight in the terminal (it now also has a ratatui TUI with dark mode and tabs). No gateway, no webhook. Good for validating that the AI provider is responding before configuring channels.

Validation

cargo run -- check     # validates config
cargo run -- doctor    # full diagnostic
cargo run -- audit     # security audit with severity ratings

audit is the one I like the most. It checks whether you have encryption enabled, whether file permissions are correct, whether webhooks have signature verification, whether the bash tool is in deny-all. It exits with a non-zero exit code when it finds critical issues, so you can drop it in CI.

The development process

The project has 178 commits in ~5 days of work (March 10-16). Almost 57 thousand lines of Rust in 120 files, organized in 13 crates.

The commits tell the story. The first few dozens were scaffolding: workspace structure, basic types, the HTTP/WebSocket gateway, first version of the channel adapters. Mass-generated code, lots of incomplete pieces.

Then the channel adapters started, one by one:

236aa1c - Minimal Discord channel adapter
fd67017 - Minimal Slack channel adapter
9f51373 - Minimal Signal channel adapter
1052e47 - WhatsApp channel webhook adapter
035f86e - Email channel adapter (IMAP inbound, SMTP outbound)

When the basic channels were in place, the IronClaw integration came in one big commit:

c87ab32 - IronClaw-derived features: circuit breaker, retry, leak detection, cache, cost tracking, extended thinking

And then came the hardening. That was the phase where I had to step in manually the most, because Claude generates functional code but doesn’t think about attack vectors on its own:

db34198 - Prompt injection sanitization, external content wrapping, prompt size limit
5719b34 - Optional VirusTotal malware scanning for file uploads
ccd2b2b - Harden input validation across all user-facing entry points
aa918ee - Optional ai-jail sandbox for bash tool
2d7b1df - Security audit command with severity-rated findings
d12cc97 - 3-tier ToolRiskLevel system replacing binary browser mutation flag
21e0c91 - Timing-safe token comparison in WhatsApp, crypto audit tests
e240c1b - Webhook replay prevention with timestamp verification
876a78c - Gateway & media: SSRF redirect validation, filename hardening

22 security hardening commits. Plus 10 more component audits. Each finding became a commit with a fix.

Then came the per-channel audits, each one uncovering different edge cases:

43b085f - Discord audit: HELLO timeout, fatal close codes, message chunking
12c7cff - Telegram audit: caption overflow, parse fallback, edit idempotency
f515062 - WhatsApp audit: message type filtering, send error classification
3c42aff - Slack audit: fatal auth errors, send error classification

Browser tools needed extra attention. A headless Chrome that gets URLs from an LLM is an obvious attack vector:

3217a96 - Browser automation: CDP timeout, SSRF guard, session limits, crash recovery
d98a803 - Gate mutating browser tools behind operator approval
014f56e - Browser screenshot/ARIA tools for accessibility tree inspection

In the more recent commits, the project started diverging from OpenClaw:

5d73c4d - OpenAI-compatible HTTP API (/v1/chat/completions, /v1/models)
d832c36 - Memory/RAG system with SQLite FTS5, embeddings, and file sync
2b05f47 - Interactive tool approval for mutating/destructive tools
49034eb - Web console: dark mode, 8 tabs, focus mode, tool sidebar
9f51a18 - TUI, Gemini/Voyage embeddings, plugin management, ACP protocol
3c0703b - Config migration from JSON to TOML
eb130ad - Channel health monitor with auto-restart and rate limiting
2c3ea7e - Workspace bootstrap files (SOUL.md, IDENTITY.md) to system prompt
9df61bf - Model-aware failover routing, canvas SVG rendering
c7ba108 - WebSocket ping keepalive and auto-reconnect to web console

The OpenAI-compatible API is the one I use the most day to day. Cursor, Continue, Open WebUI, anything that speaks the OpenAI protocol can use FrankClaw as a backend without touching anything on the client side.

The numbers

Compared to OpenClaw:

The numbers aren’t directly comparable. OpenClaw has 21 channel extensions I cut, a more complete web UI, and niche features I didn’t port. But the core (gateway, mainstream channels, providers, runtime, sessions, tools, memory) is there with 22x less code.

How to help (beta testing)

FrankClaw works for basic conversation through Web and Telegram (I tested both). WhatsApp works for simple messages. Discord, Slack, Signal and Email are implemented but haven’t had extensive testing. We haven’t done any complex workflows yet.

If you want to test it: clone the repository, bring it up with Docker Compose, configure at least one channel (Telegram is the easiest) and try to use it normally. Send messages, test tool calls, try to break the system. Open Issues on GitHub with whatever you find.

What I know needs more eyes: workflows with tools (browser, bash, MCP), subagent orchestration, failover between providers, session persistence with encryption, smart routing between models, scheduled jobs, the memory/RAG system, and the OpenAI-compatible API. Basically everything that goes beyond “send a message, get a reply.”

You don’t have to be a Rust developer. The bigger value is in using the system in ways I didn’t think of and finding the edge cases that only show up under real use.

FrankClaw doesn’t replace OpenClaw today. OpenClaw has more channels, more features, more people working on it. But it carries the weight of over a million lines of TypeScript generated at 50 commits per day by dozens of contributors, with documented critical vulnerabilities. FrankClaw is the alternative for anyone who looks at that and thinks “I’m not running this code on my machine.”

But I’ll be honest: as fun as it was to build, I personally don’t know if I need this. FrankClaw is a generic gateway, designed to be flexible, to connect any channel to any provider, with an agent runtime, tools, subagents, jobs, hooks. It’s a lot of infrastructure.

What I’ve discovered over the last few months is that I can build custom bots for specific tasks much faster. In 1 day I have a bot working, focused on what I need, without carrying the weight of a generic framework. That’s what I did with Marvin on the newsletter project, for instance. A custom-built Discord bot that does exactly what I need and nothing else.

A generic gateway like FrankClaw makes more sense for someone who wants a unified interface between several chat channels and several AI models without coding. If that’s your case, give it a shot. If you’re a developer and you know what you want, maybe a custom bot will serve you better. Up to you.

The repository is here. AGPL-3.0.

But the third project is a different beast. It’s a security project. And the motivation comes from an old pain.

The problem: too many emails

As an ex-YouTuber and content creator, I get an absurd amount of email. Event invitations, collab proposals, sponsorship offers, requests to promote stuff. Every kind of pitch.

I delete 100% of them. I don’t read them. Most of them I mark as SPAM without even opening. The easiest way to get reported by me is to send me an email — I don’t care, because I don’t need to. I also don’t answer the phone. Ever. And I automatically block anyone who messages me directly on WhatsApp, regardless of the content. My time is too valuable to spend triaging messages from strangers. I delete everything and move on.

It works for me. But I know most people don’t operate that way.

The poison is VANITY

Most people who fall for email phishing don’t fall because of technical ignorance. They fall because of VANITY.

“Hello, we’d like to invite you to our exclusive event.” “Your brand has been selected for a special partnership.” “Congratulations, you’ve been nominated as a reference in your sector.”

The dopamine hits before reason has a chance to step in. Someone recognized your work, someone wants to give you money. That’s exactly when the scammer gets you.

It isn’t the multinational CEO who falls for the Nigerian prince scam. It’s the micro-influencer who gets a sponsorship offer that’s “too good to be true.” It’s the small business owner who gets an invite to an event that looks legitimate. Vanity turns off critical thinking.

And the scams are getting more sophisticated by the day. With LLMs, any criminal can generate perfect emails in Portuguese, with no grammar errors, professional formatting, and domains that mimic real companies. The old “look at the typos” doesn’t work as a filter anymore.

The idea: Frank FBI

Instead of trying to teach everyone how to spot phishing (which doesn’t work, because vanity beats training), why not build a tool that does it automatically?

Got a suspicious email? Forward it to a dedicated address. In a few minutes, you get back a detailed report with everything the tool managed to find out about that email.

That’s how Frank FBI was born — Fraud Bureau of Investigation.

How it works

You set up a dedicated Gmail account (any Gmail with an App Password works). You register the email addresses authorized to use the service. From there, the flow is:

1. You receive a suspicious email in your personal inbox
2. You forward it to the Frank FBI address
3. The system analyzes it automatically
4. You receive the response in the same thread, with the full report

To give you a sense of the result, here’s a report for a legitimate email (cold outreach from a real company):

And here’s a suspicious one, where the sender’s domain tries to impersonate another company:

Frank FBI runs 6 layers of analysis, each with a specific weight in the final score:

Layer 1 — Header authentication (15% weight)

SPF, DKIM, DMARC. Does the Reply-To match the From? Are there suspicious anti-spam headers? This layer answers the most basic question: did the email actually come from who it claims to?

Layer 2 — Sender reputation (15% weight)

Looks up the domain’s age via WHOIS (a domain registered last week is already a signal), checks whether the IP is on DNS blacklists (DNSBL), and maintains a local reputation database that improves as more emails are analyzed. If the sender pretends to be a company but emails from a Gmail, that weighs in.

Layer 3 — Content analysis (15% weight)

This is where pattern matching comes in: artificial urgency (“your account will be locked in 24 hours”), requests for personal data, authority impersonation, financial offers. It also detects URL shorteners and links where the displayed text doesn’t match the real href — that classic “click here” pointing at a completely different domain. Checks for dangerous attachments (.exe, .scr, Office macros).

Layer 4 — External APIs (15% weight)

URLhaus (abuse.ch) maintains a known-malicious URL database. VirusTotal aggregates results from dozens of antivirus engines. If any URL in the email has already been flagged as malware or phishing by these databases, the score goes up. Results are cached with a TTL so we don’t blow through the rate limits of the free APIs.

Layer 5 — Entity verification (10% weight)

This is the layer I find the most interesting. Frank FBI does OSINT — Open Source Intelligence. It uses Brave Search to verify whether the sender or company actually exists. Does WHOIS on the domain directly. Captures a screenshot of the site with headless Chrome. Cross-references all of it to try to answer: “is this entity real and is it who it claims to be?”

Here’s a real example of this layer in action, analyzing an email that tried to impersonate a legitimate company:

The domain was 83 days old, registered through Namecheap, with no verifiable online presence. The system found discrepancies between the name in the email and the public records, and identified that the legitimate company’s real domain was a different one.

Layer 6 — LLM analysis (30% weight)

Queries 3 AI models in parallel: Claude Sonnet (Anthropic), GPT-4o (OpenAI) and Grok 4 (xAI), via OpenRouter. Each model analyzes the email independently. A confidence-weighted consensus system combines the results. If all 3 agree it’s fraud, confidence is high. If they disagree, the system weighs them. If every LLM fails, it falls back to a neutral score of 50 instead of guessing — better to admit ignorance than to hallucinate.

The final score is a confidence-adjusted weighted average. Possible verdicts: Legitimate (0-25), Suspicious OK (26-50), Suspicious Fraud (51-75) or Fraudulent (76-100). There’s also a risk escalation policy that imposes minimum floors: if critical indicators were detected (confirmed malicious URLs, DKIM failure), the score can’t fall below certain thresholds, even if the other layers found nothing.

Self-hosted: your data stays with you

Frank FBI is self-hosted. You run it on your own server. Your emails don’t pass through any third-party service (except the verification APIs like VirusTotal, which only receive URLs, not the email’s content).

You can install it on your home server, like I did, or inside your company’s infrastructure for your employees to use. The deploy is via Docker Compose with 4 containers: the Rails app, a background job worker, an IMAP poller, and an initial database setup. Bring everything up with a docker compose up -d and it’s running.

Reporting fraud to the community

Beyond analyzing emails for you, Frank FBI can optionally report confirmed fraud to community databases. This only happens when the score is >= 85 and the verdict is “fraudulent.” It’s opt-in.

ThreatFox (abuse.ch) is an open database of indicators of compromise (IOCs) maintained by the security community. When you report a malicious URL or domain there, firewalls, email filters and SIEMs around the world can consume that information to block the threat.

AbuseIPDB is the same idea, but for IPs. If the IP of the sender of the fraudulent email gets reported there, email providers and network admins can block malicious traffic before it reaches users.

And SpamCop is one of the oldest spam reporting services. Frank FBI forwards the full email to SpamCop, which analyzes the headers and reports to the responsible providers. It’s reporting directly to whoever can act.

Each report is a contribution so other people don’t fall for the same scam. And it’s automated: if the email is clearly fraudulent, the report goes out without manual intervention.

But reporting wrong things does damage. That’s why the system has anti-poisoning protections: a list of ~40 known domains (Microsoft, Apple, Google, Amazon, PayPal, governments) that are never reported; domains with clean scans are excluded; cloud infrastructure IPs (Google, Microsoft, Cloudflare) are filtered; free email domains are ignored. Only genuinely malicious IOCs reach the community databases.

Deployment: how to get it running

Let me explain how I deployed it on my home server. If you have a VPS, NAS or any Linux machine with Docker, the process is the same.

1. Clone and build the image

You need a private registry to store the Docker image. I use Gitea on my home server — it’s a lightweight self-hosted GitHub that includes a container registry. If you already have a private GitHub, you can use the GitHub Container Registry (ghcr.io) instead.

# Clone the repository
git clone https://github.com/akitaonrails/frank_fbi.git
cd frank_fbi

# Build the Docker image
# If using Gitea (replace with your registry's IP/port):
docker build -t seu-servidor:3007/frank_fbi:latest .
docker push seu-servidor:3007/frank_fbi:latest

# If using GitHub Container Registry:
docker build -t ghcr.io/seu-usuario/frank_fbi:latest .
docker push ghcr.io/seu-usuario/frank_fbi:latest

2. Configure the .env

Copy .env.example and fill it in. The required variables:

# Gere com: ruby -rsecurerandom -e 'puts SecureRandom.hex(64)'
SECRET_KEY_BASE=

# Gere com: bin/rails db:encryption:init (roda local, copia os 3 valores)
ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=
ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=
ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=

# Gmail dedicado pro Frank FBI (crie uma conta só pra isso)
# Ative 2FA e gere um App Password em https://myaccount.google.com/apppasswords
GMAIL_USERNAME=seu-frank-fbi@gmail.com
GMAIL_PASSWORD=xxxx-xxxx-xxxx-xxxx
GMAIL_IMAP_HOST=imap.gmail.com
GMAIL_SMTP_HOST=smtp.gmail.com

# Senha do ingress do Action Mailbox (qualquer string aleatória)
RAILS_INBOUND_EMAIL_PASSWORD=uma-senha-qualquer-longa

# LLM via OpenRouter (obrigatório pra Camada 6)
OPENROUTER_API_KEY=sua-chave-openrouter

# Seu email de admin (pra gerenciar remetentes autorizados)
ADMIN_EMAIL=seu-email@pessoal.com

The external APIs are optional but recommended. Without them, the corresponding layers simply don’t run:

# VirusTotal (grátis, 4 requests/min) - https://virustotal.com
VIRUSTOTAL_API_KEY=

# WhoisXML (grátis, 500 requests/mês) - https://whoisxmlapi.com
WHOISXML_API_KEY=

# Brave Search (grátis, 1 req/s) - https://brave.com/search/api/
BRAVE_SEARCH_API_KEY=

And the community reporting, which is fully opt-in. Leave it blank if you don’t want to report:

THREATFOX_AUTH_KEY=
ABUSEIPDB_API_KEY=
SPAMCOP_SUBMISSION_ADDRESS=

3. Docker Compose on the server

Create the docker-compose.yml on your server. Replace the image with your registry:

services:
  setup:
    image: seu-servidor:3007/frank_fbi:latest  # ou ghcr.io/seu-usuario/frank_fbi:latest
    command: ["./bin/rails", "db:prepare"]
    env_file: .env
    environment:
      - RAILS_ENV=production
    volumes:
      - ./storage:/rails/storage

  app:
    image: seu-servidor:3007/frank_fbi:latest
    env_file: .env
    environment:
      - RAILS_ENV=production
    volumes:
      - ./storage:/rails/storage
      - ./emails:/rails/emails
    depends_on:
      setup:
        condition: service_completed_successfully
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/up"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 15s
    restart: unless-stopped

  worker:
    image: seu-servidor:3007/frank_fbi:latest
    command: ["./bin/jobs"]
    env_file: .env
    environment:
      - RAILS_ENV=production
    volumes:
      - ./storage:/rails/storage
    depends_on:
      setup:
        condition: service_completed_successfully
    restart: unless-stopped

  mail_fetcher:
    image: seu-servidor:3007/frank_fbi:latest
    command: ["./bin/rails", "frank_fbi:fetch_mail"]
    env_file: .env
    environment:
      - RAILS_ENV=production
      - APP_HOST=http://app:3000
    volumes:
      - ./storage:/rails/storage
    depends_on:
      app:
        condition: service_healthy
    restart: unless-stopped

4. Bring it up and register the first user

docker compose up -d

setup runs the migrations and exits. app brings up Rails, worker processes background jobs, and mail_fetcher polls IMAP every 30 seconds.

To register authorized senders, send an email from your ADMIN_EMAIL to the Frank FBI address with the subject add email1@exemplo.com, email2@exemplo.com. To list who’s registered, send with subject list. To see statistics, stats.

From there, any registered sender can forward suspicious emails and receive the analysis reports.

The development process

This project started as a simple idea: “what if I could forward suspicious emails somewhere and get an analysis back?” But security projects aren’t like normal projects.

In a regular project, a bug is an inconvenience. In a security project, a bug can be a vulnerability. If the system says a fraudulent email is legitimate, someone might lose money. If it incorrectly reports a legitimate domain as malicious, it can hurt an innocent company. If a malicious email manages to exploit the analyzer itself, the attacker gets access to the server.

That changes the development mindset. Every decision needs to consider: “how could a bad actor abuse this?”

Evolution through commits

Let me show how the project evolved by looking at the most relevant commits.

The project started with the basic email analysis structure and quickly needed access control:

eb59474 - Add admin access control and allowed senders whitelist
036e9f1 - Harden access control against email spoofing and admin impersonation

The first one adds the authorized sender list. The second solves the obvious problem: what if someone spoofs an authorized sender’s email? The system started verifying SPF/DKIM before accepting any submission. Per-sender rate limiting came along with that, to prevent abuse.

Then came concerns about scoring quality:

e891270 - Fix zero-dilution scoring bug and add critical alert UX
2cbb272 - Harden fraud scoring and reporting
50d96a1 - Move text pattern detection from regex to LLM consensus layer

The zero-dilution bug is subtle: when a layer fails and returns score 0 with low confidence, it would dilute the overall average downward, making fraudulent emails look safer than they were. The fix implemented dampening that discards low-quality layers instead of letting them drag the score down.

The shift from regex to LLM in pattern detection was pragmatic. Fraud patterns in natural language are hard to capture with regex. Too many false positives. LLMs understand context and intent in a way regex can’t.

Race conditions showed up when the pipeline started running layers in parallel:

2e4276d - Fix race conditions across pipeline and add Brave Search rate limiting
ac433d9 - Fix WHOIS race condition and Brave Search gzip error logging

Concurrent jobs trying to write to the same email record, WHOIS queries stepping on each other, external API rate limits getting blown. The classic “works in sequential tests, breaks in production.”

Preventing LLM hallucination got its own dedicated commit:

e08da9e - Prevent LLM hallucination in fraud reports

LLMs make things up. If the model says “this domain was registered yesterday” without evidence in the data, the report is compromised. This commit implemented cross-validation: claims from the LLMs are verified against the concrete data from the deterministic layers. If the LLM asserts something that contradicts the facts, the information is dropped or flagged.

And when community reporting was implemented:

a01d769 - Add community threat intelligence reporting
1b64eee - Harden community reporting against poisoning and add rate limiting

The first commit implements the feature. The second adds the anti-poisoning protections. If an attacker realizes Frank FBI reports automatically, they can try to send emails that contain IOCs from legitimate companies as “fraud,” making the system report innocent domains to community databases. That’s IOC poisoning, and it’s a real attack against threat intelligence systems.

Continuous hardening

9430c63 - Harden risky attachment analysis and warnings
bdc503d - Harden screenshot capture with failure recovery and pipeline timeout
125e73a - Separate suspect content from submitter signature in forwarded emails
6f7a522 - Handle forwarded message fidelity

Malicious attachments that could exploit the parser. Site screenshots that could lock up headless Chrome. Sender signatures that were being confused with the content of the suspicious email. Forwarded emails that lost fidelity in the forward. Each of these fixes a different attack vector or edge case.

Security isn’t a feature you implement once. It’s a continuous process of “what didn’t I think could go wrong?”

The numbers

860 lines per hour. Obviously AI-assisted development. But look at the hardening commits: none of them came from the LLM suggesting “hey, let’s protect against spoofing.” That was me stopping and thinking “wait, what if someone spoofs the authorized sender?”, “what if an attacker uses IOC poisoning against my reporting system?”. The LLM doesn’t ask those questions on its own. It implements them when you ask, but the one who has to spot the hole is you.

A serious warning: DO NOT offer this as a service

If you looked at Frank FBI and thought “cool, I’m going to offer this as a SaaS to other people,” I have one piece of advice: DON’T do that.

I can think of several ways to exploit a service like this offered to the public. The operator would have access to every email forwarded by users — personal information, corporate data, sensitive correspondence. A centralized service becomes a high-value target: compromise the server and you have access to a continuous stream of confidential emails from people who are already in a vulnerable situation.

I know enough people to know that whoever would deploy this as a service wouldn’t worry about encryption at rest or about destroying the emails after analysis. And who guarantees that the operator won’t read users’ emails? Nobody. It’s the kind of thing that looks like a useful service but in practice creates a honeypot of sensitive data managed by someone with no incentive to protect it.

Frank FBI was built to be self-hosted. To run on your server, under your control, with your data staying with you. Or in your company’s infrastructure, managed by your IT team.

And the project is licensed under AGPL-3.0. If you use my code and offer it as a network service, you’re legally required to release all the derived code. No exceptions. I picked AGPL for that reason — to make sure nobody takes the project, adds tracking and telemetry, and offers it as a “free email verification service” while collecting user data behind the scenes.

The repository is here. AGPL-3.0.

The problem: subtitles

Anyone who maintains a movie and TV collection knows the pain. You download the MKV, but the embedded subtitle is out of sync. Or worse: there’s no subtitle at all in the language you want. So you go to OpenSubtitles, download a subtitle, and it’s 3 seconds ahead because it was made for a different release. The manual flow is:

1. Extract subtitle tracks from the MKV with mkvextract
2. Go to OpenSubtitles, look for a subtitle for the movie
3. Download, test, see it’s out of sync
4. Run some sync tool (ffsubsync, alass)
5. Rename the file, move it to the right place
6. Repeat for every language, for every movie

For someone with 10 movies, it’s tolerable. For someone with hundreds, it’s insanity. It’s exactly the kind of thing that should be automated.

Subservient: the Python solution

Looking at what already existed, I found Subservient. It’s a Python project that automates exactly this flow: it extracts subtitles from MKVs, downloads them from OpenSubtitles via REST API, syncs them with ffsubsync, and cleans ads out of SRT files.

The project is complete. It has movie mode and series mode, smart sync (tests every candidate in parallel and picks the best one) and first-match (stops at the first one that works). It uses the OpenSubtitles hash for exact matching and cleans watermarks and ads with more than 30 regex patterns.

But it has the typical Python distribution problems:

• 7 pip dependencies: colorama, requests, langdetect, ffsubsync, platformdirs, pycountry, tqdm
• ffsubsync as the sync engine: which in turn depends on numpy, auditok, and a bunch more Python packages
• Interactive menu UI: good for manual use, terrible for scriptability
• Config in INI format: not the end of the world, but YAML is more ergonomic
• 10,220 lines across 6 Python files: 2,700-line files with hundreds-of-lines functions each

The point isn’t that Python is bad for this. Subservient works. But installing and maintaining it in production is another story. If you want to run it on a headless server, you need Python 3.8+, pip, virtualenv (or you’re going to pollute the system), and pray no dependency breaks with the next OS update.

The experiment: porting it to Crystal with Claude

Here’s where it gets interesting. I wanted to test a hypothesis: can Claude take a large open source project, understand the architecture, and do a complete port to another language?

I’m not talking about translating it file by file. I’m talking about understanding what the project does, redesigning the architecture where it makes sense, and generating idiomatic code in Crystal.

What I did:

1. Asked Claude to clone and analyze the Subservient repo
2. Explained the design decisions: use alass (a Rust binary, no Python dependencies) instead of ffsubsync, CLI subcommands instead of interactive menus, YAML instead of INI
3. Asked for a feature-parity port, with tests

alass is an important detail. ffsubsync works fine, but it’s a Python package that pulls in numpy and does audio analysis. alass does the same thing (subtitle synchronization through timing analysis), but it’s a standalone Rust binary. Swapping one for the other eliminates the biggest Python dependency in the stack.

The result: easy-subtitle

Five commits. Less than 40 minutes from the first to the last.

The first commit already delivers a working project: 8 CLI commands, OpenSubtitles client with rate limiting, 76 passing tests, GitHub Actions with CI and release for Linux and macOS.

Numbers

The LOC difference is loud: 10,220 vs 2,516. But that’s not all Crystal’s doing. The original Python has monolithic files of thousands of lines, with a lot of duplication and UI logic mixed with business logic. The port separates the responsibilities into small, focused modules.

Architecture of the port

easy-subtitle/
  src/easy_subtitle/
    cli/           # Router + 9 commands (init, extract, download, sync, run, clean, scan, hash, doctor)
    core/          # Language map, SRT parser/writer/cleaner, video scanner
    acquisition/   # OpenSubtitles API client, auth, search, download, movie hash
    extraction/    # MKV track parsing, extraction, remuxing
    synchronization/  # alass runner, offset computation, smart/first-match strategies
    models/        # VideoFile, SubtitleCandidate, CoverageEntry

Every module has a clear responsibility. The biggest file is 144 lines (config). In the original Python, acquisition.py alone has 2,726 lines.

What each command does

# Generate the default config
easy-subtitle init

# Extract subtitles from inside MKVs
easy-subtitle extract /path/to/movies

# Download subtitles from OpenSubtitles
easy-subtitle download -l en,pt /path/to/movies

# Sync downloaded subtitles with the video
easy-subtitle sync /path/to/movies

# Full pipeline: extract → download → sync
easy-subtitle run /path/to/movies

# Clean ads/watermarks from SRTs
easy-subtitle clean /path/to/subtitles

# See subtitle coverage by language
easy-subtitle scan --json /path/to/movies

# Compute the OpenSubtitles hash (debug)
easy-subtitle hash /path/to/movie.mkv

# Validate the setup: config, credentials, dependencies
easy-subtitle doctor

doctor is a command I added later. It checks whether the config exists, whether the API key is configured, tests login against the API, and checks whether mkvmerge, mkvextract and alass are on the PATH. It shows OS-specific install instructions when something is missing.

Smart sync with fibers

Smart sync is the part I most enjoyed seeing in the port. In the original Python, it uses ThreadPoolExecutor to run multiple candidates in parallel. In Crystal, the same logic is more natural with fibers and channels:

def execute(candidates : Array(Path), video : VideoFile) : SyncResult?
  channel = Channel(SyncResult).new(candidates.size)

  candidates.each do |candidate|
    spawn do
      result = sync_one(candidate, video)
      channel.send(result)
    end
  end

  results = Array(SyncResult).new(candidates.size)
  candidates.size.times do
    results << channel.receive
  end

  accepted = results.select(&.accepted?)
  accepted.min_by(&.offset)
end

Each subtitle candidate gets synced in a separate fiber (via spawn). The results come back through the Channel. At the end, it picks the accepted one with the smallest offset. No ThreadPoolExecutor, no futures, no callbacks.

API rate limiting

OpenSubtitles requires throttling of 500ms between requests. The Crystal client implements that with a Mutex:

RATE_LIMIT_MS = 500

private def throttle! : Nil
  @mutex.synchronize do
    elapsed = Time.utc - @last_request_at
    remaining = RATE_LIMIT_MS - elapsed.total_milliseconds
    sleep(remaining.milliseconds) if remaining > 0
    @last_request_at = Time.utc
  end
end

Simple, thread-safe, no external library.

Installation

The static binary comes out of GitHub Actions and can be installed three ways:

# Homebrew (macOS / Linux)
brew install akitaonrails/tap/easy-subtitle

# Install script
curl -fsSL https://raw.githubusercontent.com/akitaonrails/easy-subtitle/master/install.sh | bash

# Or grab the binary directly from Releases

One binary. No Python, no pip, nothing.

On porting things “just because”

I’ve always argued that porting software from one language to another just for the language fetish is a waste of time. How many projects have been rewritten in Rust “just because”? How much effort spent on rewrites that delivered no new value?

But I have to admit this experiment made me reconsider.

When the cost of porting drops from weeks/months to less than 40 minutes, the equation changes. Porting Subservient to Crystal with Claude wasn’t an exercise in linguistic vanity. I wanted a static binary I could drop on a server and forget. No managing a Python runtime, no pip install breaking on the next system update.

And the result isn’t a mechanical port. It’s 2,516 lines across 42 files, against 10,220 in 6 monolithic ones. The port came with 76 tests the original didn’t have, CI with automatic release for Linux and macOS, a Homebrew formula and an install script with checksum verification.

The point isn’t that Python is bad. It’s that the bar for “is it worth porting?” got ridiculously low. Feature-parity port with tests in less than an hour. Hard to argue against that.

The repository is here. GPL-3.0, like the original.

ffmpeg -i input.mkv -c:v libx264 -crf 23 -preset medium \
  -profile:v high -level 4.1 -c:a aac -b:a 128k \
  -movflags +faststart output.mp4

What does that do? Convert to H.264 with reasonable quality for the web, using AAC for audio and moving the moov atom to the start of the file to allow progressive streaming. If you already knew that, congratulations. If you didn’t, welcome to the club of 99% of people who use FFmpeg by copying commands from Stack Overflow.

And that’s the easy case. Want to make an animated GIF from a sequence of PNGs? You need a two-pass pipeline with palette generation:

ffmpeg -framerate 10 -i frame_%04d.png \
  -vf "split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" output.gif

Want to cut a clip, resize to 720p and force a 16:9 aspect ratio? Good luck assembling the filter chain.

I got tired of memorizing flags. I wanted a CLI where I’d say “convert this to MP4 optimized for the web” and it would figure it out. So I built easy-ffmpeg.

What easy-ffmpeg does

It’s a smart wrapper. You give it the input file and the output format, and it analyzes the video with ffprobe, finds out which video, audio and subtitle streams exist, checks codec compatibility against the destination container, and decides on its own what can be copied directly (no re-encoding, instant) and what needs to be transcoded.

The most basic use:

# Convert MKV to MP4 (copies compatible streams, no unnecessary re-encoding)
easy-ffmpeg input.mkv mp4

# Optimized for the web (H.264 + AAC, faststart)
easy-ffmpeg input.mkv mp4 --web

# Optimized for mobile (720p, AAC stereo, smaller file)
easy-ffmpeg input.mkv mp4 --mobile

# Maximum compression (H.265, CRF 28)
easy-ffmpeg input.mkv mp4 --compress

# High quality for streaming (H.265, CRF 18)
easy-ffmpeg input.mkv mp4 --streaming

Examples of what you can do

Cut a clip from the video:

# From minute 1:30 to 3:00
easy-ffmpeg video.mp4 mp4 --start 1:30 --end 3:00

# The first 90 seconds
easy-ffmpeg video.mp4 mp4 --duration 90

# Accepts several time formats: 90, 1:30, 01:30.5, 1:02:30