From 0b973b73ab93469339218b09516c29c45f110158 Mon Sep 17 00:00:00 2001 From: architect-bot Date: Mon, 17 Aug 2026 18:29:14 +0000 Subject: [PATCH] WIP checkpoint: Address review feedback: Design world chunk data format and persistence schema --- docs/architecture/world.md | 737 +++++++++++++++++++++++++++++++++++++ 1 file changed, 737 insertions(+) create mode 100644 docs/architecture/world.md diff --git a/docs/architecture/world.md b/docs/architecture/world.md new file mode 100644 index 0000000..0674154 --- /dev/null +++ b/docs/architecture/world.md @@ -0,0 +1,737 @@ +# World Chunk Data Format & Persistence Architecture + +> **Status:** Proposed +> **Owner:** Platform team +> **Related:** Server authority + client prediction, avatar component system, content moderation & age verification, backend auth (JWT) + +## 1. Overview + +This document specifies the **world chunk data format** and **persistence +schema** for **agp-game**, a Godot-based multiplayer game built by autonomous +agents. It defines: + +- The **chunk data format** — both a human-readable **JSON** form (for + authoring, tooling, and debugging) and a compact **binary** form (for + runtime loading and network transfer). +- The **SQLite schema** that persists world state on the server, including + chunk metadata, chunk payloads, and the world/room hierarchy. +- The **read/write API** that the server and client use to load, save, and + stream chunks. + +The goal is a **single, versioned chunk contract** that is shared across the +Godot client, the Python backend, and the persistence layer. A chunk is the +atomic unit of world geometry, gameplay objects, and persistence — the server +loads and saves whole chunks, and clients stream the chunks around them. + +### 1.1 Design Goals + +- **Versioned & forward-compatible:** Every chunk carries a format version so + the loader can migrate old data and reject unknown future data gracefully. +- **Dual representation:** JSON for authoring/debugging, binary for + runtime/network. Both encode the *same* logical model so there is exactly one + source of truth for the schema. +- **Server-authoritative persistence:** The server is the only writer to the + SQLite store. Clients submit *edits* (deltas) and the server applies and + persists them, mirroring the server-authoritative networking model. +- **Chunk-granular persistence:** Persistence happens at chunk granularity so + the server can save a dirty chunk without rewriting the whole world, and can + stream only the chunks a client needs. +- **Deterministic & testable:** The binary format is byte-stable for a given + logical payload, enabling checksums, caching, and reproducible tests. +- **Moderation-aware:** Chunks reference content (assets, scripts) that carry + content flags (see `content_moderation.md`); the persistence layer stores the + moderation state alongside chunk data so enforcement can happen at load time. + +--- + +## 2. World Model & Chunking + +### 2.1 World Hierarchy + +The world is organized as a hierarchy that matches the moderation content graph: + +``` +World + └── Room + └── Chunk (the atomic persistence & streaming unit) + ├── Tiles / Terrain + ├── Static objects (props, geometry) + ├── Dynamic objects (spawnable entities, interactables) + └── Metadata (flags, version, timestamps) +``` + +- A **World** is the top-level container (a named, versioned space). +- A **Room** is a bounded region within a world (a level, a building, a zone). +- A **Chunk** is a fixed-size, axis-aligned region of a room. It is the unit of + load, save, and stream. + +### 2.2 Chunk Dimensions + +Chunks are **fixed-size, axis-aligned boxes** in world space. The default +dimensions are a configuration constant so they can be tuned per world: + +| Constant | Default | Description | +|---------------------|---------|----------------------------------------------------| +| `CHUNK_SIZE_X` | 32 | Chunk width in world units (tiles). | +| `CHUNK_SIZE_Y` | 32 | Chunk height in world units (tiles). | +| `CHUNK_SIZE_Z` | 1 | Chunk depth (1 = flat 2D world; >1 = volumetric). | +| `CHUNK_ORIGIN` | `(0,0)` | World-space origin of chunk coordinate `(0,0)`. | + +A chunk is identified by its **integer chunk coordinate** `(cx, cy, cz)`, +computed from a world-space position `p` as: + +``` +cx = floor((p.x - CHUNK_ORIGIN.x) / CHUNK_SIZE_X) +cy = floor((p.y - CHUNK_ORIGIN.y) / CHUNK_SIZE_Y) +cz = floor((p.z - CHUNK_ORIGIN.z) / CHUNK_SIZE_Z) +``` + +The chunk's world-space bounding box is therefore: + +``` +min = (CHUNK_ORIGIN.x + cx*CHUNK_SIZE_X, ...) +max = min + (CHUNK_SIZE_X, CHUNK_SIZE_Y, CHUNK_SIZE_Z) +``` + +### 2.3 Chunk Coordinate Key + +For storage and lookup, a chunk is addressed by a **canonical string key**: + +``` +"{world_id}:{room_id}:{cx}:{cy}:{cz}" +``` + +Example: `"w_0001:r_0001:12:-4:0"`. This key is the primary lookup key in the +SQLite `chunks` table and is used in the read/write API. + +--- + +## 3. Chunk Data Format + +### 3.1 Logical Model + +Both the JSON and binary encodings serialize the same logical chunk model. The +logical model is defined here once; the two encodings are projections of it. + +``` +Chunk + ├── version : uint (format version, currently 1) + ├── key : string (canonical chunk key, see §2.3) + ├── world_id : string + ├── room_id : string + ├── coord : { cx, cy, cz } (int) + ├── bounds : { min_x, min_y, min_z, max_x, max_y, max_z } (float) + ├── tiles : TileLayer[] (terrain / tile data) + ├── objects : WorldObject[] (static + dynamic objects) + ├── flags : ContentFlags (moderation flags, see §3.5) + ├── meta : { created_at, updated_at, revision, author } + └── checksum : string (SHA-256 of canonical payload) +``` + +### 3.2 Tile Layer + +A chunk's terrain is stored as one or more **tile layers**. Each layer is a +dense grid of tile IDs of size `CHUNK_SIZE_X × CHUNK_SIZE_Y` (× `CHUNK_SIZE_Z` +for volumetric worlds). + +``` +TileLayer + ├── name : string (e.g. "ground", "walls", "decor") + ├── layer_id : uint + ├── width : uint (= CHUNK_SIZE_X) + ├── height : uint (= CHUNK_SIZE_Y) + ├── depth : uint (= CHUNK_SIZE_Z) + ├── tiles : uint[] (flattened tile IDs, row-major, length = w*h*d) + └── palette : TileDef[] (tile ID → tile definition, shared per layer) +``` + +**Tile encoding:** A tile ID is a `uint16` index into the layer's `palette`. +`0` is reserved for "empty/air". The palette maps each tile ID to a `TileDef`: + +``` +TileDef + ├── id : uint16 + ├── asset_id : string (reference to a tile/terrain asset) + ├── solid : bool (collision) + ├── flags : uint32 (bitmask: walkable, climbable, liquid, ...) + └── variants : uint8 (number of visual variants for this tile) +``` + +**Rationale for palette + indices:** Storing dense `uint16` indices rather than +full asset strings per cell keeps the binary payload small (2 bytes/cell) while +the palette (shared across the layer) carries the rich per-tile metadata. + +### 3.3 World Objects + +Objects are the non-tile entities in a chunk: static props, interactables, +spawn points, and dynamic-object templates. + +``` +WorldObject + ├── id : string (stable object id, unique within the chunk) + ├── type : string (e.g. "prop", "interactable", "spawn_point", "door") + ├── asset_id : string (reference to the object's mesh/glTF asset) + ├── transform : { pos: [x,y,z], rot: [x,y,z,w], scale: [x,y,z] } + ├── properties : object (type-specific key/value map, JSON-typed) + ├── dynamic : bool (true if this is a spawnable dynamic entity) + └── flags : ContentFlags (per-object moderation flags) +``` + +**Dynamic objects** are stored as *templates* in the chunk. Their *runtime +state* (position, health, inventory) is **not** stored in the chunk; it lives +in the runtime entity store and is persisted separately (see §4.4). This keeps +chunks stable and diffable — a chunk only changes when its *static* content +changes. + +### 3.4 Content Flags + +Each chunk (and each object) carries a **content flag set** that is the +aggregate of the moderation flags of the assets and scripts it references. This +mirrors the aggregation rules in `content_moderation.md`: + +``` +ContentFlags + ├── categories : { violence, sexual, language, substance, gambling, horror, hate, custom } + │ (each → severity: none|mild|moderate|severe) + ├── mature : bool (true if any contributing item is mature) + └── sources : string[] (flag ids that contributed, for audit) +``` + +The server computes and persists these flags at write time so that enforcement +(age gating, content filtering) can happen at load time without re-scanning +assets. + +### 3.5 JSON Encoding + +The JSON encoding is the **authoring and debugging** representation. It is +human-readable, diffable, and used by tools and tests. It is the canonical +logical model serialized directly. + +```json +{ + "version": 1, + "key": "w_0001:r_0001:12:-4:0", + "world_id": "w_0001", + "room_id": "r_0001", + "coord": { "cx": 12, "cy": -4, "cz": 0 }, + "bounds": { "min_x": 384.0, "min_y": -128.0, "min_z": 0.0, + "max_x": 416.0, "max_y": -96.0, "max_z": 1.0 }, + "tiles": [ + { + "name": "ground", + "layer_id": 0, + "width": 32, + "height": 32, + "depth": 1, + "palette": [ + { "id": 0, "asset_id": "", "solid": false, "flags": 0, "variants": 1 }, + { "id": 1, "asset_id": "tiles/grass", "solid": false, "flags": 1, "variants": 4 } + ], + "tiles": [0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1] + } + ], + "objects": [ + { + "id": "obj_0001", + "type": "interactable", + "asset_id": "props/chest", + "transform": { "pos": [400.0, -110.0, 0.0], + "rot": [0.0, 0.0, 0.0, 1.0], + "scale": [1.0, 1.0, 1.0] }, + "properties": { "loot_table": "chest_basic", "locked": false }, + "dynamic": false, + "flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] } + } + ], + "flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] }, + "meta": { "created_at": "2026-08-17T00:00:00Z", + "updated_at": "2026-08-17T00:00:00Z", + "revision": 3, + "author": "agent-bot" }, + "checksum": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" +} +``` + +> **Note on the example checksum:** The value shown above is illustrative only. +> In practice the `checksum` field is the SHA-256 of the canonical binary +> payload body (see §3.6.3), not of the JSON text. The value here is a +> placeholder for documentation purposes. + +### 3.6 Binary Encoding + +The binary encoding is the **runtime and network** representation. It is compact +and byte-stable. It uses a simple, explicitly-specified layout (no external +serialization library) so it can be produced and consumed identically by the +Godot client (GDScript) and the Python backend. + +#### 3.6.1 Header + +| Offset | Size | Field | Type / Encoding | +|--------|------|--------------|------------------------------------------| +| 0 | 4 | magic | ASCII `"AGPC"` (0x41 0x47 0x50 0x43) | +| 4 | 2 | version | `uint16` LE (currently `1`) | +| 6 | 2 | header_len | `uint16` LE (bytes of header after this) | +| 8 | 4 | flags | `uint32` LE (bitmask, reserved) | +| 12 | 4 | payload_len | `uint32` LE (bytes of payload body) | +| 16 | 32 | checksum | SHA-256 of payload body (hex, 32 bytes) | + +Total header size: **48 bytes**. + +#### 3.6.2 Payload Body + +The payload body is a sequence of typed sections. Each section has a small +header: + +| Offset | Size | Field | Type / Encoding | +|--------|------|------------|----------------------------| +| 0 | 1 | section_id | `uint8` (see table below) | +| 1 | 4 | section_len| `uint32` LE (bytes of section data) | +| 5 | ... | data | section-specific | + +Section IDs: + +| section_id | Section | Data layout | +|------------|----------------|-------------------------------------------------------------------| +| `0x01` | metadata | world_id, room_id (length-prefixed strings), coord (3×int32), bounds (6×float32) | +| `0x02` | tile layer | name, layer_id, dims (3×uint16), palette, dense tile indices | +| `0x03` | object | one `WorldObject` (repeated; one section per object) | +| `0x04` | content flags | category severities (8×uint8), mature (uint8), sources (string list) | +| `0x05` | meta | created_at, updated_at (int64 unix), revision (uint32), author (string) | + +**String encoding:** length-prefixed UTF-8 — `uint16` LE byte length followed by +the UTF-8 bytes. + +**Tile layer data layout (section `0x02`):** + +``` +name : string +layer_id : uint16 LE +width : uint16 LE +height : uint16 LE +depth : uint16 LE +palette_count : uint16 LE +palette : palette_count × TileDef +tile_count : uint32 LE +tiles : tile_count × uint16 LE (dense, row-major) +``` + +``` +TileDef: + id : uint16 LE + asset_id : string + solid : uint8 + flags : uint32 LE + variants : uint8 +``` + +**Object data layout (section `0x03`):** + +``` +id : string +type : string +asset_id : string +pos : 3 × float32 +rot : 4 × float32 +scale : 3 × float32 +dynamic : uint8 +properties : JSON string (length-prefixed) +flags : ContentFlags (see 0x04 layout) +``` + +**Content flags layout (section `0x04`):** + +``` +violence, sexual, language, substance, gambling, horror, hate, custom + : 8 × uint8 (severity enum: 0=none,1=mild,2=moderate,3=severe) +mature : uint8 +source_count: uint16 LE +sources : source_count × string +``` + +#### 3.6.3 Byte-Stability + +For a given logical payload, the binary encoding is **deterministic**: + +- All integers are little-endian. +- All strings are UTF-8, length-prefixed, and never null-terminated. +- Maps (e.g. `properties`) are serialized as a **canonical JSON string** with + keys sorted lexicographically. +- The `checksum` is the SHA-256 of the payload body bytes. + +This guarantees that two encoders produce identical bytes, enabling content +addressing, caching, and reproducible tests. + +--- + +## 4. SQLite Persistence Schema + +The server persists world state in a **SQLite** database. SQLite is chosen for +its zero-configuration, single-file, transactional, and embedded nature — ideal +for a server that owns the world store and needs ACID guarantees without a +separate database process. + +### 4.1 Database Layout + +The database file is `world.db` (path configurable). It contains the following +tables: + +``` +worlds +rooms +chunks +chunk_objects +content_flags +revisions +``` + +### 4.2 `worlds` + +Top-level world containers. + +```sql +CREATE TABLE worlds ( + id TEXT PRIMARY KEY, -- e.g. 'w_0001' + name TEXT NOT NULL, + description TEXT, + chunk_size_x INTEGER NOT NULL DEFAULT 32, + chunk_size_y INTEGER NOT NULL DEFAULT 32, + chunk_size_z INTEGER NOT NULL DEFAULT 1, + origin_x REAL NOT NULL DEFAULT 0, + origin_y REAL NOT NULL DEFAULT 0, + origin_z REAL NOT NULL DEFAULT 0, + created_at TEXT NOT NULL, -- ISO-8601 UTC + updated_at TEXT NOT NULL +); +``` + +### 4.3 `rooms` + +Rooms within a world. + +```sql +CREATE TABLE rooms ( + id TEXT PRIMARY KEY, -- e.g. 'r_0001' + world_id TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE, + name TEXT NOT NULL, + bounds_min_x REAL NOT NULL, + bounds_min_y REAL NOT NULL, + bounds_min_z REAL NOT NULL DEFAULT 0, + bounds_max_x REAL NOT NULL, + bounds_max_y REAL NOT NULL, + bounds_max_z REAL NOT NULL DEFAULT 1, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + UNIQUE (world_id, name) +); +CREATE INDEX idx_rooms_world ON rooms(world_id); +``` + +### 4.4 `chunks` + +The core persistence table. One row per chunk. The **payload** is stored as a +BLOB in the **binary format** (§3.6); the JSON form is derived on demand for +authoring/debugging and is not stored (to avoid dual-write drift). + +```sql +CREATE TABLE chunks ( + key TEXT PRIMARY KEY, -- canonical key 'world:room:cx:cy:cz' + world_id TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE, + room_id TEXT NOT NULL REFERENCES rooms(id) ON DELETE CASCADE, + cx INTEGER NOT NULL, + cy INTEGER NOT NULL, + cz INTEGER NOT NULL DEFAULT 0, + format_version INTEGER NOT NULL DEFAULT 1, + payload BLOB NOT NULL, -- binary chunk encoding (§3.6) + payload_len INTEGER NOT NULL, + checksum TEXT NOT NULL, -- SHA-256 of payload + revision INTEGER NOT NULL DEFAULT 0, + dirty INTEGER NOT NULL DEFAULT 0, -- 1 = needs flush to disk + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + UNIQUE (world_id, room_id, cx, cy, cz) +); +CREATE INDEX idx_chunks_world_room ON chunks(world_id, room_id); +CREATE INDEX idx_chunks_coord ON chunks(cx, cy, cz); +``` + +**Design notes:** + +- The `payload` BLOB is the single source of truth for chunk content. The + `checksum` guards against corruption and enables content addressing. +- `revision` increments on every write; it is used for optimistic concurrency + and change detection (see §5.4). +- `dirty` marks chunks that have been modified in memory but not yet flushed, + enabling a write-behind persistence strategy (see §5.5). + +### 4.5 `chunk_objects` + +A denormalized, queryable index of the objects inside each chunk. This is +**derived** from the chunk payload and kept in sync by the write path. It exists +so the server can answer queries like "which chunks contain a door?" without +parsing every payload BLOB. + +```sql +CREATE TABLE chunk_objects ( + id TEXT NOT NULL, -- object id (unique within chunk) + chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE, + type TEXT NOT NULL, -- 'prop', 'interactable', ... + asset_id TEXT, + dynamic INTEGER NOT NULL DEFAULT 0, + pos_x REAL NOT NULL, + pos_y REAL NOT NULL, + pos_z REAL NOT NULL DEFAULT 0, + PRIMARY KEY (chunk_key, id) +); +CREATE INDEX idx_chunk_objects_type ON chunk_objects(type); +CREATE INDEX idx_chunk_objects_asset ON chunk_objects(asset_id); +``` + +**Sync mechanism:** `chunk_objects` is maintained **explicitly by the write +path** — it is not a SQL trigger. Whenever `put_chunk` (or `delete_chunk`) +commits a chunk, the service layer performs a **delete-and-reinsert** of that +chunk's object rows in the same transaction: it deletes all rows for +`chunk_key`, then inserts fresh rows parsed from the new payload. This keeps the +index transactionally consistent with the payload and avoids the complexity and +fragility of SQL triggers over a binary BLOB. Because the index is derived, it +can always be rebuilt from `chunks.payload` if it ever drifts (a `rebuild_index` +utility is provided for this). + +### 4.6 `content_flags` + +Persists the aggregated moderation flags per chunk (and optionally per object). +This lets the server enforce age/content gates at load time without parsing +payloads. + +```sql +CREATE TABLE content_flags ( + chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE, + object_id TEXT NOT NULL DEFAULT '', -- '' = chunk-level flag (sentinel) + violence INTEGER NOT NULL DEFAULT 0, + sexual INTEGER NOT NULL DEFAULT 0, + language INTEGER NOT NULL DEFAULT 0, + substance INTEGER NOT NULL DEFAULT 0, + gambling INTEGER NOT NULL DEFAULT 0, + horror INTEGER NOT NULL DEFAULT 0, + hate INTEGER NOT NULL DEFAULT 0, + custom INTEGER NOT NULL DEFAULT 0, + mature INTEGER NOT NULL DEFAULT 0, + sources TEXT NOT NULL DEFAULT '[]', -- JSON array of flag ids + PRIMARY KEY (chunk_key, object_id) +); +CREATE INDEX idx_content_flags_mature ON content_flags(mature); +``` + +Severity is stored as `0=none, 1=mild, 2=moderate, 3=severe`. + +**Chunk-level vs. object-level flags:** Chunk-level flags use the **sentinel +value `''` (empty string)** in `object_id` rather than `NULL`. This is +deliberate: in SQLite, `NULL` values in a composite primary key are **not** +unique-constrained, so two rows with the same `chunk_key` and `object_id = NULL` +could be inserted accidentally. Using `''` as the sentinel means the composite +`PRIMARY KEY (chunk_key, object_id)` enforces that there is **exactly one** +chunk-level flag row per chunk. Object-level flags use the object's actual `id` +(which is never empty). The application layer treats `object_id = ''` as +"chunk-level" and any non-empty value as "object-level". + +### 4.7 `revisions` + +An append-only audit log of chunk writes. Enables rollback, auditing, and +replication of changes. + +```sql +CREATE TABLE revisions ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + chunk_key TEXT NOT NULL, -- NOT a FK: survives chunk deletion + revision INTEGER NOT NULL, + actor TEXT NOT NULL, -- user / agent id + action TEXT NOT NULL, -- 'create' | 'update' | 'delete' + checksum TEXT NOT NULL, -- SHA-256 of payload + payload BLOB, -- full binary payload at this revision + created_at TEXT NOT NULL +); +CREATE INDEX idx_revisions_chunk ON revisions(chunk_key); +``` + +**Rollback support:** The `payload` BLOB stores the **full binary chunk payload** +at each revision. This is what makes rollback possible: to restore a chunk to a +previous revision, the server reads the `payload` of the target revision and +writes it back to `chunks.payload` (incrementing `revision` and appending a new +`revisions` row). Without the payload, only the checksum would be available, +which is insufficient to reconstruct the chunk. + +**Audit trail preservation:** `chunk_key` is **not** a foreign key to `chunks`. +This means deleting a chunk does **not** cascade-delete its revision history — +the audit trail is preserved even after a chunk is removed. (The `chunk_key` +string is still stored so the history can be correlated to a chunk that existed +at that time.) If a chunk is later recreated with the same key, its history is +still intact. + +### 4.8 Schema Versioning & Migrations + +The database carries a schema version in a `meta` table: + +```sql +CREATE TABLE meta ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +); +-- seed: INSERT INTO meta(key, value) VALUES ('schema_version', '1'); +``` + +Migrations are applied in order at startup. Each migration is a numbered SQL +script; the server tracks the applied version in `meta` and applies any pending +migrations transactionally. This keeps the schema forward-compatible as the +chunk format evolves. + +--- + +## 5. Read/Write API + +The API is exposed as a **Python service layer** on the backend (the same +process that owns the SQLite store) and consumed by the Godot client over the +network transport (see `networking.md`). The service layer is the **only** +writer to the database; clients never touch SQLite directly. + +### 5.1 Service Interface (Python) + +```python +class WorldStore: + """Owns the SQLite connection and chunk read/write operations.""" + + def get_chunk(self, key: str) -> Chunk | None: ... + def get_chunk_binary(self, key: str) -> bytes | None: ... + def put_chunk(self, chunk: Chunk, actor: str) -> Chunk: ... + def delete_chunk(self, key: str, actor: str) -> None: ... + def list_chunks(self, world_id: str, room_id: str | None = None, + bounds: Bounds | None = None) -> list[str]: ... + def get_chunk_flags(self, key: str) -> ContentFlags: ... + def flush_dirty(self) -> int: ... + def rollback_chunk(self, key: str, revision: int, actor: str) -> Chunk: ... + def rebuild_index(self, chunk_key: str | None = None) -> int: ... + def migrate(self) -> None: ... +``` + +### 5.2 Encoding Helpers + +The service layer provides symmetric encode/decode functions shared by the +client and server: + +```python +def encode_binary(chunk: Chunk) -> bytes: ... +def decode_binary(data: bytes) -> Chunk: ... +def encode_json(chunk: Chunk) -> str: ... +def decode_json(text: str) -> Chunk: ... +def checksum(payload: bytes) -> str: ... +``` + +These are the single implementation of the format spec (§3), so the Godot +client and Python backend never drift. + +### 5.3 Network RPCs + +The chunk API is exposed to the client via RPCs (matching the networking +architecture). All chunk traffic is **reliable** (chunks are discrete, +must-arrive data). + +| RPC | Direction | Payload | Notes | +|----------------------------|-----------|-------------------------------------------|-------| +| `request_chunk` | C → S | `{ key }` | Client asks for a chunk | +| `chunk_data` | S → C | `{ key, binary_payload, checksum }` | Server streams a chunk | +| `request_chunk_edit` | C → S | `{ key, base_revision, delta }` | Client submits an edit | +| `chunk_edit_result` | S → C | `{ key, success, new_revision, error? }` | Acknowledges an edit | +| `chunk_invalidated` | S → C | `{ key, new_revision }` | Server pushes a change | +| `request_chunk_flags` | C → S | `{ key }` | Age/content gate check | + +### 5.4 Edit & Concurrency Model + +- **Read:** Client requests a chunk; server returns the binary payload and its + `revision`. +- **Edit:** Client submits a **delta** against a known `base_revision`. The + server applies the delta, increments `revision`, persists, and returns the new + revision. +- **Optimistic concurrency:** If the chunk's current revision differs from the + client's `base_revision`, the server rejects the edit with a conflict error + and the client re-fetches and re-applies. This prevents lost updates when two + clients edit the same chunk. +- **Deltas:** A delta is a JSON patch (RFC 6902 style) applied to the chunk's + logical model. The server validates the patched chunk (schema, bounds, + moderation) before persisting. + +### 5.5 Write-Behind Persistence + +To amortize disk I/O, the server keeps a **dirty set** of chunks modified in +memory and flushes them to SQLite in batches: + +1. `put_chunk` updates the in-memory chunk and marks it `dirty`. +2. A background flusher periodically calls `flush_dirty()`, which writes all + dirty chunks in a single transaction and clears the `dirty` flag. +3. On graceful shutdown, `flush_dirty()` is called to ensure durability. + +This keeps the hot path (in-memory chunk access) fast while guaranteeing +durability at shutdown and on a configurable interval. + +**Recovery after unclean shutdown:** The `dirty` column in the `chunks` table +exists specifically to support recovery if the process crashes before a flush +completes. On startup, the server scans for rows where `dirty = 1` and treats +them as **unflushed in-memory state that was lost** — it re-reads the last +persisted payload from `chunks.payload` (the last durable state) and clears the +`dirty` flag. In other words, `dirty = 1` at startup means "the last write was +not durable; discard the in-memory delta and fall back to the persisted +payload." This guarantees that the on-disk state is always a consistent, +previously-committed snapshot, never a partially-written chunk. (The in-memory +dirty set is authoritative during normal operation; the `dirty` column is only a +recovery marker.) + +### 5.6 Streaming & Caching + +- The server streams only the chunks a client needs (those within the client's + view radius), using `list_chunks` with a bounds query. +- Chunks are cached in memory (LRU) keyed by `key`; the cache is invalidated on + write. +- The binary payload is content-addressed by `checksum`, so unchanged chunks can + be served from cache without re-reading SQLite. + +--- + +## 6. Format Versioning & Migration + +- The chunk format `version` field (currently `1`) is checked on decode. A + decoder that encounters a higher version than it supports **rejects** the + chunk with a clear error rather than misreading it. +- The SQLite `schema_version` in `meta` drives database migrations (§4.8). +- Chunk payloads are migrated **lazily**: when a chunk is read and its + `format_version` is older than the current one, the server migrates it to the + current version in memory and marks it dirty so it is rewritten on the next + flush. This avoids a costly full-world migration at startup. + +--- + +## 7. Security & Integrity + +- **Server-authoritative writes:** Only the `WorldStore` writes to SQLite; + clients submit deltas that are validated before application. +- **Checksum verification:** Every chunk payload is checksummed (SHA-256) on + write and verified on read. Corruption is detected and the chunk is re-fetched + or flagged. +- **Bounds validation:** Chunk coordinates and object positions are validated + against the world's chunk dimensions and room bounds before persistence. +- **Moderation enforcement:** `content_flags` are checked at load time; mature + chunks are gated behind the age-verification flow (see `content_moderation.md`). +- **Input validation:** All string fields are length-limited and validated + against allowed character sets to prevent injection and oversized payloads. + +--- + +## 8. Open Questions / Future Work + +- **Volumetric worlds:** The schema and format already support `CHUNK_SIZE_Z > + 1`, but the tooling and streaming for 3D chunks are not yet specified. +- **Chunk compression:** For very large worlds, the binary payload could be + compressed (e.g. zstd) before storage/transfer. The header reserves a `flags` + bitmask for a compression flag. +- **Multi-server sharding:** If a single SQLite file becomes a bottleneck, the + `chunks` table could be sharded by `world_id` across multiple databases. The + key design already supports this (world_id is a leading key component). +- **Delta compression for streaming:** Instead of sending full chunks, the + server could send only the delta between revisions to reduce bandwidth for + frequently-updated chunks.