# World Chunk Data Format & Persistence Architecture > **Status:** Proposed > **Owner:** Platform team > **Related:** Server authority + client prediction, avatar component system, content moderation & age verification, backend auth (JWT) ## 1. Overview This document specifies the **world chunk data format** and **persistence schema** for **agp-game**, a Godot-based multiplayer game built by autonomous agents. It defines: - The **chunk data format** — both a human-readable **JSON** form (for authoring, tooling, and debugging) and a compact **binary** form (for runtime loading and network transfer). - The **SQLite schema** that persists world state on the server, including chunk metadata, chunk payloads, and the world/room hierarchy. - The **read/write API** that the server and client use to load, save, and stream chunks. The goal is a **single, versioned chunk contract** that is shared across the Godot client, the Python backend, and the persistence layer. A chunk is the atomic unit of world geometry, gameplay objects, and persistence — the server loads and saves whole chunks, and clients stream the chunks around them. ### 1.1 Design Goals - **Versioned & forward-compatible:** Every chunk carries a format version so the loader can migrate old data and reject unknown future data gracefully. - **Dual representation:** JSON for authoring/debugging, binary for runtime/network. Both encode the *same* logical model so there is exactly one source of truth for the schema. - **Server-authoritative persistence:** The server is the only writer to the SQLite store. Clients submit *edits* (deltas) and the server applies and persists them, mirroring the server-authoritative networking model. - **Chunk-granular persistence:** Persistence happens at chunk granularity so the server can save a dirty chunk without rewriting the whole world, and can stream only the chunks a client needs. - **Deterministic & testable:** The binary format is byte-stable for a given logical payload, enabling checksums, caching, and reproducible tests. - **Moderation-aware:** Chunks reference content (assets, scripts) that carry content flags (see `content_moderation.md`); the persistence layer stores the moderation state alongside chunk data so enforcement can happen at load time. --- ## 2. World Model & Chunking ### 2.1 World Hierarchy The world is organized as a hierarchy that matches the moderation content graph: ``` World └── Room └── Chunk (the atomic persistence & streaming unit) ├── Tiles / Terrain ├── Static objects (props, geometry) ├── Dynamic objects (spawnable entities, interactables) └── Metadata (flags, version, timestamps) ``` - A **World** is the top-level container (a named, versioned space). - A **Room** is a bounded region within a world (a level, a building, a zone). - A **Chunk** is a fixed-size, axis-aligned region of a room. It is the unit of load, save, and stream. ### 2.2 Chunk Dimensions Chunks are **fixed-size, axis-aligned boxes** in world space. The default dimensions are a configuration constant so they can be tuned per world: | Constant | Default | Description | |---------------------|---------|----------------------------------------------------| | `CHUNK_SIZE_X` | 32 | Chunk width in world units (tiles). | | `CHUNK_SIZE_Y` | 32 | Chunk height in world units (tiles). | | `CHUNK_SIZE_Z` | 1 | Chunk depth (1 = flat 2D world; >1 = volumetric). | | `CHUNK_ORIGIN` | `(0,0)` | World-space origin of chunk coordinate `(0,0)`. | A chunk is identified by its **integer chunk coordinate** `(cx, cy, cz)`, computed from a world-space position `p` as: ``` cx = floor((p.x - CHUNK_ORIGIN.x) / CHUNK_SIZE_X) cy = floor((p.y - CHUNK_ORIGIN.y) / CHUNK_SIZE_Y) cz = floor((p.z - CHUNK_ORIGIN.z) / CHUNK_SIZE_Z) ``` The chunk's world-space bounding box is therefore: ``` min = (CHUNK_ORIGIN.x + cx*CHUNK_SIZE_X, ...) max = min + (CHUNK_SIZE_X, CHUNK_SIZE_Y, CHUNK_SIZE_Z) ``` ### 2.3 Chunk Coordinate Key For storage and lookup, a chunk is addressed by a **canonical string key**: ``` "{world_id}:{room_id}:{cx}:{cy}:{cz}" ``` Example: `"w_0001:r_0001:12:-4:0"`. This key is the primary lookup key in the SQLite `chunks` table and is used in the read/write API. --- ## 3. Chunk Data Format ### 3.1 Logical Model Both the JSON and binary encodings serialize the same logical chunk model. The logical model is defined here once; the two encodings are projections of it. ``` Chunk ├── version : uint (format version, currently 1) ├── key : string (canonical chunk key, see §2.3) ├── world_id : string ├── room_id : string ├── coord : { cx, cy, cz } (int) ├── bounds : { min_x, min_y, min_z, max_x, max_y, max_z } (float) ├── tiles : TileLayer[] (terrain / tile data) ├── objects : WorldObject[] (static + dynamic objects) ├── flags : ContentFlags (moderation flags, see §3.5) ├── meta : { created_at, updated_at, revision, author } └── checksum : string (SHA-256 of canonical payload) ``` ### 3.2 Tile Layer A chunk's terrain is stored as one or more **tile layers**. Each layer is a dense grid of tile IDs of size `CHUNK_SIZE_X × CHUNK_SIZE_Y` (× `CHUNK_SIZE_Z` for volumetric worlds). ``` TileLayer ├── name : string (e.g. "ground", "walls", "decor") ├── layer_id : uint ├── width : uint (= CHUNK_SIZE_X) ├── height : uint (= CHUNK_SIZE_Y) ├── depth : uint (= CHUNK_SIZE_Z) ├── tiles : uint[] (flattened tile IDs, row-major, length = w*h*d) └── palette : TileDef[] (tile ID → tile definition, shared per layer) ``` **Tile encoding:** A tile ID is a `uint16` index into the layer's `palette`. `0` is reserved for "empty/air". The palette maps each tile ID to a `TileDef`: ``` TileDef ├── id : uint16 ├── asset_id : string (reference to a tile/terrain asset) ├── solid : bool (collision) ├── flags : uint32 (bitmask: walkable, climbable, liquid, ...) └── variants : uint8 (number of visual variants for this tile) ``` **Rationale for palette + indices:** Storing dense `uint16` indices rather than full asset strings per cell keeps the binary payload small (2 bytes/cell) while the palette (shared across the layer) carries the rich per-tile metadata. ### 3.3 World Objects Objects are the non-tile entities in a chunk: static props, interactables, spawn points, and dynamic-object templates. ``` WorldObject ├── id : string (stable object id, unique within the chunk) ├── type : string (e.g. "prop", "interactable", "spawn_point", "door") ├── asset_id : string (reference to the object's mesh/glTF asset) ├── transform : { pos: [x,y,z], rot: [x,y,z,w], scale: [x,y,z] } ├── properties : object (type-specific key/value map, JSON-typed) ├── dynamic : bool (true if this is a spawnable dynamic entity) └── flags : ContentFlags (per-object moderation flags) ``` **Dynamic objects** are stored as *templates* in the chunk. Their *runtime state* (position, health, inventory) is **not** stored in the chunk; it lives in the runtime entity store and is persisted separately (see §4.4). This keeps chunks stable and diffable — a chunk only changes when its *static* content changes. ### 3.4 Content Flags Each chunk (and each object) carries a **content flag set** that is the aggregate of the moderation flags of the assets and scripts it references. This mirrors the aggregation rules in `content_moderation.md`: ``` ContentFlags ├── categories : { violence, sexual, language, substance, gambling, horror, hate, custom } │ (each → severity: none|mild|moderate|severe) ├── mature : bool (true if any contributing item is mature) └── sources : string[] (flag ids that contributed, for audit) ``` The server computes and persists these flags at write time so that enforcement (age gating, content filtering) can happen at load time without re-scanning assets. ### 3.5 JSON Encoding The JSON encoding is the **authoring and debugging** representation. It is human-readable, diffable, and used by tools and tests. It is the canonical logical model serialized directly. ```json { "version": 1, "key": "w_0001:r_0001:12:-4:0", "world_id": "w_0001", "room_id": "r_0001", "coord": { "cx": 12, "cy": -4, "cz": 0 }, "bounds": { "min_x": 384.0, "min_y": -128.0, "min_z": 0.0, "max_x": 416.0, "max_y": -96.0, "max_z": 1.0 }, "tiles": [ { "name": "ground", "layer_id": 0, "width": 32, "height": 32, "depth": 1, "palette": [ { "id": 0, "asset_id": "", "solid": false, "flags": 0, "variants": 1 }, { "id": 1, "asset_id": "tiles/grass", "solid": false, "flags": 1, "variants": 4 } ], "tiles": [0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1] } ], "objects": [ { "id": "obj_0001", "type": "interactable", "asset_id": "props/chest", "transform": { "pos": [400.0, -110.0, 0.0], "rot": [0.0, 0.0, 0.0, 1.0], "scale": [1.0, 1.0, 1.0] }, "properties": { "loot_table": "chest_basic", "locked": false }, "dynamic": false, "flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] } } ], "flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] }, "meta": { "created_at": "2026-08-17T00:00:00Z", "updated_at": "2026-08-17T00:00:00Z", "revision": 3, "author": "agent-bot" }, "checksum": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08" } ``` > **Note on the example checksum:** The value shown above is illustrative only. > In practice the `checksum` field is the SHA-256 of the canonical binary > payload body (see §3.6.3), not of the JSON text. The value here is a > placeholder for documentation purposes. ### 3.6 Binary Encoding The binary encoding is the **runtime and network** representation. It is compact and byte-stable. It uses a simple, explicitly-specified layout (no external serialization library) so it can be produced and consumed identically by the Godot client (GDScript) and the Python backend. #### 3.6.1 Header | Offset | Size | Field | Type / Encoding | |--------|------|--------------|------------------------------------------| | 0 | 4 | magic | ASCII `"AGPC"` (0x41 0x47 0x50 0x43) | | 4 | 2 | version | `uint16` LE (currently `1`) | | 6 | 2 | header_len | `uint16` LE (bytes of header after this) | | 8 | 4 | flags | `uint32` LE (bitmask, reserved) | | 12 | 4 | payload_len | `uint32` LE (bytes of payload body) | | 16 | 32 | checksum | SHA-256 of payload body (hex, 32 bytes) | Total header size: **48 bytes**. #### 3.6.2 Payload Body The payload body is a sequence of typed sections. Each section has a small header: | Offset | Size | Field | Type / Encoding | |--------|------|------------|----------------------------| | 0 | 1 | section_id | `uint8` (see table below) | | 1 | 4 | section_len| `uint32` LE (bytes of section data) | | 5 | ... | data | section-specific | Section IDs: | section_id | Section | Data layout | |------------|----------------|-------------------------------------------------------------------| | `0x01` | metadata | world_id, room_id (length-prefixed strings), coord (3×int32), bounds (6×float32) | | `0x02` | tile layer | name, layer_id, dims (3×uint16), palette, dense tile indices | | `0x03` | object | one `WorldObject` (repeated; one section per object) | | `0x04` | content flags | category severities (8×uint8), mature (uint8), sources (string list) | | `0x05` | meta | created_at, updated_at (int64 unix), revision (uint32), author (string) | **String encoding:** length-prefixed UTF-8 — `uint16` LE byte length followed by the UTF-8 bytes. **Tile layer data layout (section `0x02`):** ``` name : string layer_id : uint16 LE width : uint16 LE height : uint16 LE depth : uint16 LE palette_count : uint16 LE palette : palette_count × TileDef tile_count : uint32 LE tiles : tile_count × uint16 LE (dense, row-major) ``` ``` TileDef: id : uint16 LE asset_id : string solid : uint8 flags : uint32 LE variants : uint8 ``` **Object data layout (section `0x03`):** ``` id : string type : string asset_id : string pos : 3 × float32 rot : 4 × float32 scale : 3 × float32 dynamic : uint8 properties : JSON string (length-prefixed) flags : ContentFlags (see 0x04 layout) ``` **Content flags layout (section `0x04`):** ``` violence, sexual, language, substance, gambling, horror, hate, custom : 8 × uint8 (severity enum: 0=none,1=mild,2=moderate,3=severe) mature : uint8 source_count: uint16 LE sources : source_count × string ``` #### 3.6.3 Byte-Stability For a given logical payload, the binary encoding is **deterministic**: - All integers are little-endian. - All strings are UTF-8, length-prefixed, and never null-terminated. - Maps (e.g. `properties`) are serialized as a **canonical JSON string** with keys sorted lexicographically. - The `checksum` is the SHA-256 of the payload body bytes. This guarantees that two encoders produce identical bytes, enabling content addressing, caching, and reproducible tests. --- ## 4. SQLite Persistence Schema The server persists world state in a **SQLite** database. SQLite is chosen for its zero-configuration, single-file, transactional, and embedded nature — ideal for a server that owns the world store and needs ACID guarantees without a separate database process. ### 4.1 Database Layout The database file is `world.db` (path configurable). It contains the following tables: ``` worlds rooms chunks chunk_objects content_flags revisions ``` ### 4.2 `worlds` Top-level world containers. ```sql CREATE TABLE worlds ( id TEXT PRIMARY KEY, -- e.g. 'w_0001' name TEXT NOT NULL, description TEXT, chunk_size_x INTEGER NOT NULL DEFAULT 32, chunk_size_y INTEGER NOT NULL DEFAULT 32, chunk_size_z INTEGER NOT NULL DEFAULT 1, origin_x REAL NOT NULL DEFAULT 0, origin_y REAL NOT NULL DEFAULT 0, origin_z REAL NOT NULL DEFAULT 0, created_at TEXT NOT NULL, -- ISO-8601 UTC updated_at TEXT NOT NULL ); ``` ### 4.3 `rooms` Rooms within a world. ```sql CREATE TABLE rooms ( id TEXT PRIMARY KEY, -- e.g. 'r_0001' world_id TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE, name TEXT NOT NULL, bounds_min_x REAL NOT NULL, bounds_min_y REAL NOT NULL, bounds_min_z REAL NOT NULL DEFAULT 0, bounds_max_x REAL NOT NULL, bounds_max_y REAL NOT NULL, bounds_max_z REAL NOT NULL DEFAULT 1, created_at TEXT NOT NULL, updated_at TEXT NOT NULL, UNIQUE (world_id, name) ); CREATE INDEX idx_rooms_world ON rooms(world_id); ``` ### 4.4 `chunks` The core persistence table. One row per chunk. The **payload** is stored as a BLOB in the **binary format** (§3.6); the JSON form is derived on demand for authoring/debugging and is not stored (to avoid dual-write drift). ```sql CREATE TABLE chunks ( key TEXT PRIMARY KEY, -- canonical key 'world:room:cx:cy:cz' world_id TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE, room_id TEXT NOT NULL REFERENCES rooms(id) ON DELETE CASCADE, cx INTEGER NOT NULL, cy INTEGER NOT NULL, cz INTEGER NOT NULL DEFAULT 0, format_version INTEGER NOT NULL DEFAULT 1, payload BLOB NOT NULL, -- binary chunk encoding (§3.6) payload_len INTEGER NOT NULL, checksum TEXT NOT NULL, -- SHA-256 of payload revision INTEGER NOT NULL DEFAULT 0, dirty INTEGER NOT NULL DEFAULT 0, -- 1 = needs flush to disk created_at TEXT NOT NULL, updated_at TEXT NOT NULL, UNIQUE (world_id, room_id, cx, cy, cz) ); CREATE INDEX idx_chunks_world_room ON chunks(world_id, room_id); CREATE INDEX idx_chunks_coord ON chunks(cx, cy, cz); ``` **Design notes:** - The `payload` BLOB is the single source of truth for chunk content. The `checksum` guards against corruption and enables content addressing. - `revision` increments on every write; it is used for optimistic concurrency and change detection (see §5.4). - `dirty` marks chunks that have been modified in memory but not yet flushed, enabling a write-behind persistence strategy (see §5.5). ### 4.5 `chunk_objects` A denormalized, queryable index of the objects inside each chunk. This is **derived** from the chunk payload and kept in sync by the write path. It exists so the server can answer queries like "which chunks contain a door?" without parsing every payload BLOB. ```sql CREATE TABLE chunk_objects ( id TEXT NOT NULL, -- object id (unique within chunk) chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE, type TEXT NOT NULL, -- 'prop', 'interactable', ... asset_id TEXT, dynamic INTEGER NOT NULL DEFAULT 0, pos_x REAL NOT NULL, pos_y REAL NOT NULL, pos_z REAL NOT NULL DEFAULT 0, PRIMARY KEY (chunk_key, id) ); CREATE INDEX idx_chunk_objects_type ON chunk_objects(type); CREATE INDEX idx_chunk_objects_asset ON chunk_objects(asset_id); ``` **Sync mechanism:** `chunk_objects` is maintained **explicitly by the write path** — it is not a SQL trigger. Whenever `put_chunk` (or `delete_chunk`) commits a chunk, the service layer performs a **delete-and-reinsert** of that chunk's object rows in the same transaction: it deletes all rows for `chunk_key`, then inserts fresh rows parsed from the new payload. This keeps the index transactionally consistent with the payload and avoids the complexity and fragility of SQL triggers over a binary BLOB. Because the index is derived, it can always be rebuilt from `chunks.payload` if it ever drifts (a `rebuild_index` utility is provided for this). ### 4.6 `content_flags` Persists the aggregated moderation flags per chunk (and optionally per object). This lets the server enforce age/content gates at load time without parsing payloads. ```sql CREATE TABLE content_flags ( chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE, object_id TEXT NOT NULL DEFAULT '', -- '' = chunk-level flag (sentinel) violence INTEGER NOT NULL DEFAULT 0, sexual INTEGER NOT NULL DEFAULT 0, language INTEGER NOT NULL DEFAULT 0, substance INTEGER NOT NULL DEFAULT 0, gambling INTEGER NOT NULL DEFAULT 0, horror INTEGER NOT NULL DEFAULT 0, hate INTEGER NOT NULL DEFAULT 0, custom INTEGER NOT NULL DEFAULT 0, mature INTEGER NOT NULL DEFAULT 0, sources TEXT NOT NULL DEFAULT '[]', -- JSON array of flag ids PRIMARY KEY (chunk_key, object_id) ); CREATE INDEX idx_content_flags_mature ON content_flags(mature); ``` Severity is stored as `0=none, 1=mild, 2=moderate, 3=severe`. **Chunk-level vs. object-level flags:** Chunk-level flags use the **sentinel value `''` (empty string)** in `object_id` rather than `NULL`. This is deliberate: in SQLite, `NULL` values in a composite primary key are **not** unique-constrained, so two rows with the same `chunk_key` and `object_id = NULL` could be inserted accidentally. Using `''` as the sentinel means the composite `PRIMARY KEY (chunk_key, object_id)` enforces that there is **exactly one** chunk-level flag row per chunk. Object-level flags use the object's actual `id` (which is never empty). The application layer treats `object_id = ''` as "chunk-level" and any non-empty value as "object-level". ### 4.7 `revisions` An append-only audit log of chunk writes. Enables rollback, auditing, and replication of changes. ```sql CREATE TABLE revisions ( id INTEGER PRIMARY KEY AUTOINCREMENT, chunk_key TEXT NOT NULL, -- NOT a FK: survives chunk deletion revision INTEGER NOT NULL, actor TEXT NOT NULL, -- user / agent id action TEXT NOT NULL, -- 'create' | 'update' | 'delete' checksum TEXT NOT NULL, -- SHA-256 of payload payload BLOB, -- full binary payload at this revision created_at TEXT NOT NULL ); CREATE INDEX idx_revisions_chunk ON revisions(chunk_key); ``` **Rollback support:** The `payload` BLOB stores the **full binary chunk payload** at each revision. This is what makes rollback possible: to restore a chunk to a previous revision, the server reads the `payload` of the target revision and writes it back to `chunks.payload` (incrementing `revision` and appending a new `revisions` row). Without the payload, only the checksum would be available, which is insufficient to reconstruct the chunk. **Audit trail preservation:** `chunk_key` is **not** a foreign key to `chunks`. This means deleting a chunk does **not** cascade-delete its revision history — the audit trail is preserved even after a chunk is removed. (The `chunk_key` string is still stored so the history can be correlated to a chunk that existed at that time.) If a chunk is later recreated with the same key, its history is still intact. ### 4.8 Schema Versioning & Migrations The database carries a schema version in a `meta` table: ```sql CREATE TABLE meta ( key TEXT PRIMARY KEY, value TEXT NOT NULL ); -- seed: INSERT INTO meta(key, value) VALUES ('schema_version', '1'); ``` Migrations are applied in order at startup. Each migration is a numbered SQL script; the server tracks the applied version in `meta` and applies any pending migrations transactionally. This keeps the schema forward-compatible as the chunk format evolves. --- ## 5. Read/Write API The API is exposed as a **Python service layer** on the backend (the same process that owns the SQLite store) and consumed by the Godot client over the network transport (see `networking.md`). The service layer is the **only** writer to the database; clients never touch SQLite directly. ### 5.1 Service Interface (Python) ```python class WorldStore: """Owns the SQLite connection and chunk read/write operations.""" def get_chunk(self, key: str) -> Chunk | None: ... def get_chunk_binary(self, key: str) -> bytes | None: ... def put_chunk(self, chunk: Chunk, actor: str) -> Chunk: ... def delete_chunk(self, key: str, actor: str) -> None: ... def list_chunks(self, world_id: str, room_id: str | None = None, bounds: Bounds | None = None) -> list[str]: ... def get_chunk_flags(self, key: str) -> ContentFlags: ... def flush_dirty(self) -> int: ... def rollback_chunk(self, key: str, revision: int, actor: str) -> Chunk: ... def rebuild_index(self, chunk_key: str | None = None) -> int: ... def migrate(self) -> None: ... ``` ### 5.2 Encoding Helpers The service layer provides symmetric encode/decode functions shared by the client and server: ```python def encode_binary(chunk: Chunk) -> bytes: ... def decode_binary(data: bytes) -> Chunk: ... def encode_json(chunk: Chunk) -> str: ... def decode_json(text: str) -> Chunk: ... def checksum(payload: bytes) -> str: ... ``` These are the single implementation of the format spec (§3), so the Godot client and Python backend never drift. ### 5.3 Network RPCs The chunk API is exposed to the client via RPCs (matching the networking architecture). All chunk traffic is **reliable** (chunks are discrete, must-arrive data). | RPC | Direction | Payload | Notes | |----------------------------|-----------|-------------------------------------------|-------| | `request_chunk` | C → S | `{ key }` | Client asks for a chunk | | `chunk_data` | S → C | `{ key, binary_payload, checksum }` | Server streams a chunk | | `request_chunk_edit` | C → S | `{ key, base_revision, delta }` | Client submits an edit | | `chunk_edit_result` | S → C | `{ key, success, new_revision, error? }` | Acknowledges an edit | | `chunk_invalidated` | S → C | `{ key, new_revision }` | Server pushes a change | | `request_chunk_flags` | C → S | `{ key }` | Age/content gate check | ### 5.4 Edit & Concurrency Model - **Read:** Client requests a chunk; server returns the binary payload and its `revision`. - **Edit:** Client submits a **delta** against a known `base_revision`. The server applies the delta, increments `revision`, persists, and returns the new revision. - **Optimistic concurrency:** If the chunk's current revision differs from the client's `base_revision`, the server rejects the edit with a conflict error and the client re-fetches and re-applies. This prevents lost updates when two clients edit the same chunk. - **Deltas:** A delta is a JSON patch (RFC 6902 style) applied to the chunk's logical model. The server validates the patched chunk (schema, bounds, moderation) before persisting. ### 5.5 Write-Behind Persistence To amortize disk I/O, the server keeps a **dirty set** of chunks modified in memory and flushes them to SQLite in batches: 1. `put_chunk` updates the in-memory chunk and marks it `dirty`. 2. A background flusher periodically calls `flush_dirty()`, which writes all dirty chunks in a single transaction and clears the `dirty` flag. 3. On graceful shutdown, `flush_dirty()` is called to ensure durability. This keeps the hot path (in-memory chunk access) fast while guaranteeing durability at shutdown and on a configurable interval. **Recovery after unclean shutdown:** The `dirty` column in the `chunks` table exists specifically to support recovery if the process crashes before a flush completes. On startup, the server scans for rows where `dirty = 1` and treats them as **unflushed in-memory state that was lost** — it re-reads the last persisted payload from `chunks.payload` (the last durable state) and clears the `dirty` flag. In other words, `dirty = 1` at startup means "the last write was not durable; discard the in-memory delta and fall back to the persisted payload." This guarantees that the on-disk state is always a consistent, previously-committed snapshot, never a partially-written chunk. (The in-memory dirty set is authoritative during normal operation; the `dirty` column is only a recovery marker.) ### 5.6 Streaming & Caching - The server streams only the chunks a client needs (those within the client's view radius), using `list_chunks` with a bounds query. - Chunks are cached in memory (LRU) keyed by `key`; the cache is invalidated on write. - The binary payload is content-addressed by `checksum`, so unchanged chunks can be served from cache without re-reading SQLite. --- ## 6. Format Versioning & Migration - The chunk format `version` field (currently `1`) is checked on decode. A decoder that encounters a higher version than it supports **rejects** the chunk with a clear error rather than misreading it. - The SQLite `schema_version` in `meta` drives database migrations (§4.8). - Chunk payloads are migrated **lazily**: when a chunk is read and its `format_version` is older than the current one, the server migrates it to the current version in memory and marks it dirty so it is rewritten on the next flush. This avoids a costly full-world migration at startup. --- ## 7. Security & Integrity - **Server-authoritative writes:** Only the `WorldStore` writes to SQLite; clients submit deltas that are validated before application. - **Checksum verification:** Every chunk payload is checksummed (SHA-256) on write and verified on read. Corruption is detected and the chunk is re-fetched or flagged. - **Bounds validation:** Chunk coordinates and object positions are validated against the world's chunk dimensions and room bounds before persistence. - **Moderation enforcement:** `content_flags` are checked at load time; mature chunks are gated behind the age-verification flow (see `content_moderation.md`). - **Input validation:** All string fields are length-limited and validated against allowed character sets to prevent injection and oversized payloads. --- ## 8. Open Questions / Future Work - **Volumetric worlds:** The schema and format already support `CHUNK_SIZE_Z > 1`, but the tooling and streaming for 3D chunks are not yet specified. - **Chunk compression:** For very large worlds, the binary payload could be compressed (e.g. zstd) before storage/transfer. The header reserves a `flags` bitmask for a compression flag. - **Multi-server sharding:** If a single SQLite file becomes a bottleneck, the `chunks` table could be sharded by `world_id` across multiple databases. The key design already supports this (world_id is a leading key component). - **Delta compression for streaming:** Instead of sending full chunks, the server could send only the delta between revisions to reduce bandwidth for frequently-updated chunks.