Dex (Human Layer) and Aaron (CTO of Boundary/BAML) walk through a pipeline that mines a team's coding-agent sessions and turns them into proposed CLAUDE.md entries, rules, and a report on what agents keep getting stuck on. The naive version (strip tool calls, dump user/assistant messages into one structured-output call) was directionally right but bad and expensive; the rebuilt BAML pipeline splits into three sub-flows — facts agents discovered, corrections humans made, and questions agents keep re-researching — with cheap classifier pre-screening and stronger models for judging, landing at roughly two cents per session. Most of the episode is process: build throwaway dashboards, cache and replay steps, read the actual prompts, and compare everything against a golden run made by Claude Code. They end on the unsolved part: how to tell when a memory has gone stale and how to feed human accept/reject decisions back into the pipeline.
- The problem being solved is that learning inside a session — what the agent figured out after 500 tool calls, what a human kept correcting — is lost when the next fresh context window starts; the pipeline extracts those as 'context shards' with reasoning, citations, session IDs and user IDs so they can be traced back and, because Human Layer is multiplayer, correlated across teammates.
- Shards fall into distinct classes: where to find things, behavioral rules (a human told the agent how to act), and facts that are true but hard to discover from the repo itself; the naive single-call Haiku version produced ~947 undifferentiated memories before they split it up.
- The recommended build order is to start with the smartest, most open-ended setup — here, Claude Code reading the session data — see what's possible, watch which scripts and filters it writes, then distill that into a cheaper opinionated workflow; the Claude Code version cost about $100 for a report the final pipeline produced for roughly 50 cents.
- A cheap classifier model (transcribed as 'Jeff') released mid-project handled 97% of the calls for 5% of the bill, doing scoring and enum classification; BAML emits a float that is treated as an actual probability, which is used to prune messages before expensive models see them.
- Iteration speed came from tooling: about ten versions of a local dashboard for browsing sessions, filtering tool calls, and Sankey diagrams, plus on-disk caching of hashed requests so any step could be replayed without rerunning the whole pipeline.
- Dex argues anything in a model-written prompt or skill that you didn't write yourself is already in the weights and adds no information, so read the prompts — doing exactly that is how Aaron caught a bug where long messages were truncated from the end instead of the middle, losing valuable content.
- Quality was measured against a golden set from the Claude Code run; one filtering prompt kept only 60% of the text while retaining 100% of the valuable messages, and memories get judged for whether they are tribal knowledge versus something an agent could just read in the code.
- Open problems discussed: detecting stale memories (ideas floated include tying each memory to a code region via the code map, asking the agent in-context whether a memory is outdated, or back-testing on old traces), and making the pipeline incremental by feeding already-approved and user-rejected memories back in as a fourth input so reports also suggest what to remove.
A concrete, surprising result: the top recurring agent question was already documented, and the real fix was a pointer, not more memorized context.
Agents aren't reading a file we have called architecture.md... they don't know what's going on because you're not telling them to read that file. So the answer may not necessarily be to dump a crap ton of facts into the CLAUDE.md.
A sharp, reusable argument against auto-generated skills and prompts, immediately followed by the truncation bug that reading prompts uncovered.
Anything that is in the skill or the prompt that you yourself did not write is by definition something that is already in the model weights, because the model put it there out of the weights. And so you are by definition not adding any useful content by letting the model write the prompts.
Anything in the skill or the prompt that you yourself did not write is by definition something that is already in the model weights.
Start with the smartest model and the most open-ended pipeline-shaped thing and see what is possible with maximum intelligence — then distill that down.
97% of the calls, 5% of the bill.
You can literally just give the agent keys to the database and be like, oh yeah, you can just run this.