Cole Medin walks through how he's evolving his personal AI second brain into a shared "team brain." The key architectural decision: keep the personal agent (personality, custom memory, interface) on each individual's machine, and centralize only the knowledge base in a database that every agent queries through a single MCP server. The rest of the video covers how to normalize every source into one documents table, label documents by domain at ingestion, enforce row-level permissions in the database using tokens, and use hybrid keyword + semantic search for retrieval. The demo is built on the Oracle AI database (a disclosed partnership) with LangChain, but he stresses the strategies apply regardless of tooling.
- Don't scrap your personal second brain — the team brain is just the shared knowledge base; the personal agent stays distributed so each person keeps their own customization and memory, while policy and knowledge get centralized.
- An Obsidian vault of Markdown files no longer works at team scale; you need a central database as the single source of truth for retrieval and permissions, accessed by each personal agent via an MCP server.
- Everything from every source — Slack threads, GitHub repos, documentation — is normalized into one 'documents' table with a shared schema (title, text, URL, author, domains, metadata) so querying stays simple.
- Each connector's only job is to fetch from its source and mold the data into that standard document format, giving all personal brains a single entry point.
- Permissioning starts at ingestion: sources are mapped to domain labels (marketing, ops, etc.), and documents can carry multiple labels for shared spaces; an MCP tokens table and a principals table map each user's token to the domains they can see.
- The security gate must live in the database, not the personal agent — otherwise a user can get around it with prompt injection or by editing their own agent's code.
- At scale, keyword search alone isn't enough: combining keyword search (exact wording, IDs) with semantic search (concepts, meaning) and weighting the results narrows tens of thousands of records down to the few snippets the agent needs, with source citations for verification.
- A 'who knows' MCP tool uses the author field on records to tell you which colleague to ask when the knowledge base doesn't cover something you can access.
- He claims the personal version of this system saved him roughly 20 hours per week when he last measured it a few months ago.
The single clearest design rule in the video, and the one most people building local-first agent systems get wrong.
The most important takeaway here, no matter how you build this system, is you need the gate to sit in the database. You cannot have this second brain, the personal part of the system, responsible for the security in any way cuz then it's going to be way too easy for the individual to get around it with prompt injection to their own agent or just changing their own agent's implementation.
A concrete live demo showing row-level permissions actually working end to end from a Claude Code session.
If I ask that question with Jeff, it says the knowledge base doesn't cover anything around the enterprise discount ceiling. So the information is completely invisible as it should be cuz we can't access it. But if I ask the same question as Sam, this is where we get that up to 20% off list price.
The last time I actually measured it, it saved me about 20 hours per week.
You need the gate to sit in the database.
We can't just do a Karpathy LLM wiki with just keyword search anymore.