The current LLM Wiki architecture, as described, operates with a clear separation of concerns, which is good. You have distinct layers, each with a defined role:
- Raw Sources Layer: This is an immutable collection of source documents. The LLM reads from here but never modifies it. This is a sound design choice, ensuring data provenance and preventing the LLM from hallucinating changes into original source material.
- Wiki Layer: This is the core, a directory of LLM-generated markdown files. It contains summaries, entity pages, concept pages, and syntheses. The LLM exclusively owns the creation, updates, cross-reference maintenance, and consistency within this layer.
- Schema Layer: A configuration document, like
CLAUDE.md, that defines the wiki's structure, conventions, and workflows for the LLM. This layer is co-evolved by the human and the LLM, acting as a dynamic schema for the knowledge base.
Operations are straightforward: Ingest processes new sources, Query synthesizes answers, and Lint performs periodic health checks to identify contradictions or data gaps. The system relies on index.md as a content catalog and log.md as an append-only record of operations.
Here's how I see the core components interacting:
This architecture is elegant for a single LLM agent operating on a local file system. It implicitly assumes a single writer, which simplifies consistency significantly for an LLM Wiki.
The Current Blueprint: A Single-Writer LLM Wiki Domain
The LLM Wiki, as described, operates with a clear separation of concerns, which is good. You have distinct layers, each with a defined role:
- Raw Sources Layer: This is an immutable collection of source documents. The LLM reads from here but never modifies it. This is a sound design choice, ensuring data provenance and preventing the LLM from hallucinating changes into original source material.
- Wiki Layer: This is the core, a directory of LLM-generated markdown files. It contains summaries, entity pages, concept pages, and syntheses. The LLM exclusively owns the creation, updates, cross-reference maintenance, and consistency within this layer.
- Schema Layer: A configuration document, like
CLAUDE.md, that defines the wiki's structure, conventions, and workflows for the LLM. This layer is co-evolved by the human and the LLM, acting as a dynamic schema for the knowledge base.
Operations are straightforward: Ingest processes new sources, Query synthesizes answers, and Lint performs periodic health checks to identify contradictions or data gaps. The system relies on index.md as a content catalog and log.md as an append-only record of operations.
Here's how I see the core components interacting:
This architecture is elegant for a single LLM agent operating on a local file system. It implicitly assumes a single writer, which simplifies consistency significantly for an LLM Wiki.
Why Your LLM Wiki Architecture Won't Scale: The Inherent Bottlenecks
The moment you try to move beyond a single LLM agent or a few hundred pages, this LLM Wiki architecture starts to show its limitations. The "moderate scale (~100 sources, hundreds of pages) without embedding-based RAG" mentioned in the idea file is a clear indicator that this design isn't built for high throughput or large-scale distributed operations.
1. The Single LLM Agent as a Bottleneck: The entire system hinges on a single LLM agent performing all ingest, query, and lint operations. This is a sequential processing model. If you have a flood of new sources or concurrent queries, that single agent becomes a massive bottleneck. You can't just throw more compute at it; the architectural pattern itself limits parallelism.
2. File-Based Storage and Contention: The Wiki Layer is a directory of markdown files. The index.md and log.md are single files. In a distributed system, concurrent writes to these files would lead to immediate data corruption or require complex, external file locking mechanisms that introduce significant latency. This isn't a distributed file system with built-in consistency guarantees; it's a local directory, limiting the LLM Wiki's growth potential.
3. Lack of Transactional Guarantees: When an LLM ingests a new source, it might update 10-15 wiki pages, index.md, and log.md. If any of these updates fail midway, you're left with an inconsistent state. There's no explicit mechanism for atomic commits across multiple file modifications. This means you'll have partial updates, orphaned references, and an LLM Wiki that isn't truly "self-healing" but rather "self-corrupting" under stress.
4. Implicit Consistency Model: The current design relies on the implicit strong consistency of a single writer operating on a local file system. This works until you introduce multiple writers or network partitions, which is a major challenge for any scalable LLM Wiki architecture.
The Consistency Conundrum: Distributing Your LLM Wiki
This is where Brewer's Theorem, or the CAP theorem, becomes non-negotiable. You can choose Availability (AP) or Consistency (CP). If you pick both, you are ignoring Brewer's Theorem. The LLM Wiki, in its current form, implicitly prioritizes Consistency (C) by having a single, authoritative writer, but it sacrifices Availability (A) and Partition Tolerance (P) if you attempt to distribute it.
If you try to introduce multiple LLM agents to handle increased load, you immediately face a consistency problem:
- Conflicting Updates: What happens if two LLM agents try to update the same entity page simultaneously based on different ingested sources? Or if one agent is linting a page while another is updating it? Without a robust concurrency control mechanism, you'll get lost updates, stale reads, and a fragmented knowledge base in your LLM Wiki system.
- Eventual Consistency vs. Strong Consistency: The
Lintoperation, which identifies contradictions and stale claims, is a form of reactive eventual consistency. It cleans up inconsistencies *after* they've occurred. For a truly reliable knowledge base, you need to define where strong consistency is essential (e.g., the schema, critical entity facts) and where eventual consistency is acceptable (e.g., cross-references, topic summaries that can be reconciled later). The current system doesn't make this distinction explicit for the LLM Wiki's data. - The Thundering Herd Problem: If multiple LLM agents are all trying to update
index.mdorlog.mdconcurrently, you'll create a thundering herd, where many requests contend for a single resource, leading to degraded performance and potential deadlocks in the LLM Wiki's operations.
Architecting for Scale: Building a Resilient LLM Wiki Knowledge System
To move beyond "moderate scale" and build a truly distributed, self-healing LLM Wiki, you need to fundamentally rethink its underlying LLM Wiki architecture. Here's what I'd recommend in an architecture review:
1. Decouple Operations with an Event-Driven Architecture:
* Instead of direct file modifications, LLM agents should publish events (e.g., SourceIngested, PageUpdated, QueryAnswered) to a distributed message queue like Apache Kafka or Amazon Kinesis.
* Dedicated microservices or specialized LLM agents can then subscribe to these events. One agent might handle index.md updates, another log.md, and others specific wiki page updates. This lets you parallelize processing and scale components independently.
2. Replace File-Based Storage with a Distributed Data Store:
* The Wiki Layer needs to live in a distributed database, not a file system. Consider a document database like MongoDB or a wide-column store like Apache Cassandra for the markdown content. For structured metadata and cross-references, a graph database could be powerful.
A DynamoDB Single-Table Design could consolidate wiki pages, entities, and relationships, providing predictable performance at scale. This lets you manage complex data relationships without the overhead of multiple tables.
For collaborative editing scenarios (if humans also contribute directly), a Conflict-Free Replicated Data Type (CRDT) system could maintain eventual consistency across multiple writers without requiring a central coordinator.
3. Enforce Idempotency for LLM Operations: * When an LLM agent processes an event or performs an update, that operation must be idempotent. This means applying the operation multiple times produces the same result as applying it once. If your event bus guarantees at-least-once delivery (which most do), your consumers *will* process messages multiple times. If your LLM isn't idempotent in its updates, you'll end up with duplicate content, incorrect cross-references, or corrupted data.
4. Implement Explicit Concurrency Control and Versioning: * For critical updates, use optimistic locking (e.g., version numbers on wiki pages) or distributed locks (e.g., Apache ZooKeeper, Consul) to prevent concurrent writes from overwriting each other. * Leverage the Git repository integration not just for human version history, but as a robust, distributed version control system that LLM agents can interact with programmatically, handling merges and conflicts.
5. Coordinate Critical Operations with Leader Election:
* Operations like Lint passes or index.md rebuilds are inherently global and should ideally be performed by a single, authoritative entity at any given time. Implement a leader election pattern (e.g., using ZooKeeper or a consensus algorithm like Raft) to ensure only one LLM agent is responsible for these tasks, preventing the thundering herd problem and ensuring a consistent global state.
6. Define Clear Consistency Models:
* For the Schema Layer, you likely need strong consistency. Changes here affect how the entire LLM Wiki operates, so all agents must see the same schema simultaneously.
* For the Wiki Layer content, eventual consistency is often acceptable, especially for summaries or less critical facts. The Lint process then becomes the mechanism to converge to a consistent state over time.
The vision of a self-healing, evolving LLM Wiki is compelling, and the 'idea file' is a fantastic starting point for communicating that vision. But for this system to truly scale and become a reliable, distributed knowledge base, we need to apply the hard-won lessons from decades of distributed systems architecture. You can't just expect a local file system and a single agent to magically handle the complexities of concurrent operations and data consistency across a growing knowledge graph. The path forward involves embracing event-driven patterns, distributed data stores, and explicit consistency models. That's the only way to build an LLM Wiki that doesn't just heal itself, but also grows without breaking.
