@@ -1,737 +0,0 @@
# World Chunk Data Format & Persistence Architecture
> **Status:** Proposed
> **Owner:** Platform team
> **Related:** Server authority + client prediction, avatar component system, content moderation & age verification, backend auth (JWT)
## 1. Overview
This document specifies the **world chunk data format ** and **persistence
schema** for **agp-game ** , a Godot-based multiplayer game built by autonomous
agents. It defines:
- The **chunk data format ** — both a human-readable **JSON ** form (for
authoring, tooling, and debugging) and a compact **binary ** form (for
runtime loading and network transfer).
- The **SQLite schema ** that persists world state on the server, including
chunk metadata, chunk payloads, and the world/room hierarchy.
- The **read/write API ** that the server and client use to load, save, and
stream chunks.
The goal is a **single, versioned chunk contract ** that is shared across the
Godot client, the Python backend, and the persistence layer. A chunk is the
atomic unit of world geometry, gameplay objects, and persistence — the server
loads and saves whole chunks, and clients stream the chunks around them.
### 1.1 Design Goals
- **Versioned & forward-compatible:** Every chunk carries a format version so
the loader can migrate old data and reject unknown future data gracefully.
- **Dual representation:** JSON for authoring/debugging, binary for
runtime/network. Both encode the * same * logical model so there is exactly one
source of truth for the schema.
- **Server-authoritative persistence:** The server is the only writer to the
SQLite store. Clients submit * edits * (deltas) and the server applies and
persists them, mirroring the server-authoritative networking model.
- **Chunk-granular persistence:** Persistence happens at chunk granularity so
the server can save a dirty chunk without rewriting the whole world, and can
stream only the chunks a client needs.
- **Deterministic & testable:** The binary format is byte-stable for a given
logical payload, enabling checksums, caching, and reproducible tests.
- **Moderation-aware:** Chunks reference content (assets, scripts) that carry
content flags (see `content_moderation.md` ); the persistence layer stores the
moderation state alongside chunk data so enforcement can happen at load time.
---
## 2. World Model & Chunking
### 2.1 World Hierarchy
The world is organized as a hierarchy that matches the moderation content graph:
```
World
└── Room
└── Chunk (the atomic persistence & streaming unit)
├── Tiles / Terrain
├── Static objects (props, geometry)
├── Dynamic objects (spawnable entities, interactables)
└── Metadata (flags, version, timestamps)
```
- A **World ** is the top-level container (a named, versioned space).
- A **Room ** is a bounded region within a world (a level, a building, a zone).
- A **Chunk ** is a fixed-size, axis-aligned region of a room. It is the unit of
load, save, and stream.
### 2.2 Chunk Dimensions
Chunks are **fixed-size, axis-aligned boxes ** in world space. The default
dimensions are a configuration constant so they can be tuned per world:
| Constant | Default | Description |
|---------------------|---------|----------------------------------------------------|
| `CHUNK_SIZE_X` | 32 | Chunk width in world units (tiles). |
| `CHUNK_SIZE_Y` | 32 | Chunk height in world units (tiles). |
| `CHUNK_SIZE_Z` | 1 | Chunk depth (1 = flat 2D world; >1 = volumetric). |
| `CHUNK_ORIGIN` | `(0,0)` | World-space origin of chunk coordinate `(0,0)` . |
A chunk is identified by its **integer chunk coordinate ** `(cx, cy, cz)` ,
computed from a world-space position `p` as:
```
cx = floor((p.x - CHUNK_ORIGIN.x) / CHUNK_SIZE_X)
cy = floor((p.y - CHUNK_ORIGIN.y) / CHUNK_SIZE_Y)
cz = floor((p.z - CHUNK_ORIGIN.z) / CHUNK_SIZE_Z)
```
The chunk's world-space bounding box is therefore:
```
min = (CHUNK_ORIGIN.x + cx*CHUNK_SIZE_X, ...)
max = min + (CHUNK_SIZE_X, CHUNK_SIZE_Y, CHUNK_SIZE_Z)
```
### 2.3 Chunk Coordinate Key
For storage and lookup, a chunk is addressed by a **canonical string key ** :
```
"{world_id}:{room_id}:{cx}:{cy}:{cz}"
```
Example: `"w_0001:r_0001:12:-4:0"` . This key is the primary lookup key in the
SQLite `chunks` table and is used in the read/write API.
---
## 3. Chunk Data Format
### 3.1 Logical Model
Both the JSON and binary encodings serialize the same logical chunk model. The
logical model is defined here once; the two encodings are projections of it.
```
Chunk
├── version : uint (format version, currently 1)
├── key : string (canonical chunk key, see §2.3)
├── world_id : string
├── room_id : string
├── coord : { cx, cy, cz } (int)
├── bounds : { min_x, min_y, min_z, max_x, max_y, max_z } (float)
├── tiles : TileLayer[] (terrain / tile data)
├── objects : WorldObject[] (static + dynamic objects)
├── flags : ContentFlags (moderation flags, see §3.5)
├── meta : { created_at, updated_at, revision, author }
└── checksum : string (SHA-256 of canonical payload)
```
### 3.2 Tile Layer
A chunk's terrain is stored as one or more **tile layers ** . Each layer is a
dense grid of tile IDs of size `CHUNK_SIZE_X × (× `CHUNK_SIZE_Z`
for volumetric worlds).
```
TileLayer
├── name : string (e.g. "ground", "walls", "decor")
├── layer_id : uint
├── width : uint (= CHUNK_SIZE_X)
├── height : uint (= CHUNK_SIZE_Y)
├── depth : uint (= CHUNK_SIZE_Z)
├── tiles : uint[] (flattened tile IDs, row-major, length = w*h*d)
└── palette : TileDef[] (tile ID → tile definition, shared per layer)
```
**Tile encoding: ** A tile ID is a `uint16` index into the layer's `palette` .
`0` is reserved for "empty/air". The palette maps each tile ID to a `TileDef` :
```
TileDef
├── id : uint16
├── asset_id : string (reference to a tile/terrain asset)
├── solid : bool (collision)
├── flags : uint32 (bitmask: walkable, climbable, liquid, ...)
└── variants : uint8 (number of visual variants for this tile)
```
**Rationale for palette + indices: ** Storing dense `uint16` indices rather than
full asset strings per cell keeps the binary payload small (2 bytes/cell) while
the palette (shared across the layer) carries the rich per-tile metadata.
### 3.3 World Objects
Objects are the non-tile entities in a chunk: static props, interactables,
spawn points, and dynamic-object templates.
```
WorldObject
├── id : string (stable object id, unique within the chunk)
├── type : string (e.g. "prop", "interactable", "spawn_point", "door")
├── asset_id : string (reference to the object's mesh/glTF asset)
├── transform : { pos: [x,y,z], rot: [x,y,z,w], scale: [x,y,z] }
├── properties : object (type-specific key/value map, JSON-typed)
├── dynamic : bool (true if this is a spawnable dynamic entity)
└── flags : ContentFlags (per-object moderation flags)
```
**Dynamic objects ** are stored as * templates * in the chunk. Their *runtime
state* (position, health, inventory) is **not ** stored in the chunk; it lives
in the runtime entity store and is persisted separately (see §4.4). This keeps
chunks stable and diffable — a chunk only changes when its * static * content
changes.
### 3.4 Content Flags
Each chunk (and each object) carries a **content flag set ** that is the
aggregate of the moderation flags of the assets and scripts it references. This
mirrors the aggregation rules in `content_moderation.md` :
```
ContentFlags
├── categories : { violence, sexual, language, substance, gambling, horror, hate, custom }
│ (each → severity: none|mild|moderate|severe)
├── mature : bool (true if any contributing item is mature)
└── sources : string[] (flag ids that contributed, for audit)
```
The server computes and persists these flags at write time so that enforcement
(age gating, content filtering) can happen at load time without re-scanning
assets.
### 3.5 JSON Encoding
The JSON encoding is the **authoring and debugging ** representation. It is
human-readable, diffable, and used by tools and tests. It is the canonical
logical model serialized directly.
``` json
{
"version" : 1 ,
"key" : "w_0001:r_0001:12:-4:0" ,
"world_id" : "w_0001" ,
"room_id" : "r_0001" ,
"coord" : { "cx" : 12 , "cy" : -4 , "cz" : 0 } ,
"bounds" : { "min_x" : 384.0 , "min_y" : -128.0 , "min_z" : 0.0 ,
"max_x" : 416.0 , "max_y" : -96.0 , "max_z" : 1.0 } ,
"tiles" : [
{
"name" : "ground" ,
"layer_id" : 0 ,
"width" : 32 ,
"height" : 32 ,
"depth" : 1 ,
"palette" : [
{ "id" : 0 , "asset_id" : "" , "solid" : false , "flags" : 0 , "variants" : 1 } ,
{ "id" : 1 , "asset_id" : "tiles/grass" , "solid" : false , "flags" : 1 , "variants" : 4 }
] ,
"tiles" : [ 0 , 1 , 1 , 1 , 0 , 1 , 1 , 0 , 0 , 1 , 1 , 1 , 1 , 0 , 1 , 1 ]
}
] ,
"objects" : [
{
"id" : "obj_0001" ,
"type" : "interactable" ,
"asset_id" : "props/chest" ,
"transform" : { "pos" : [ 400.0 , -110.0 , 0.0 ] ,
"rot" : [ 0.0 , 0.0 , 0.0 , 1.0 ] ,
"scale" : [ 1.0 , 1.0 , 1.0 ] } ,
"properties" : { "loot_table" : "chest_basic" , "locked" : false } ,
"dynamic" : false ,
"flags" : { "categories" : { "violence" : "none" } , "mature" : false , "sources" : [ ] }
}
] ,
"flags" : { "categories" : { "violence" : "none" } , "mature" : false , "sources" : [ ] } ,
"meta" : { "created_at" : "2026-08-17T00:00:00Z" ,
"updated_at" : "2026-08-17T00:00:00Z" ,
"revision" : 3 ,
"author" : "agent-bot" } ,
"checksum" : "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
}
```
> **Note on the example checksum:** The value shown above is illustrative only.
> In practice the `checksum` field is the SHA-256 of the canonical binary
> payload body (see §3.6.3), not of the JSON text. The value here is a
> placeholder for documentation purposes.
### 3.6 Binary Encoding
The binary encoding is the **runtime and network ** representation. It is compact
and byte-stable. It uses a simple, explicitly-specified layout (no external
serialization library) so it can be produced and consumed identically by the
Godot client (GDScript) and the Python backend.
#### 3.6.1 Header
| Offset | Size | Field | Type / Encoding |
|--------|------|--------------|------------------------------------------|
| 0 | 4 | magic | ASCII `"AGPC"` (0x41 0x47 0x50 0x43) |
| 4 | 2 | version | `uint16` LE (currently `1` ) |
| 6 | 2 | header_len | `uint16` LE (bytes of header after this) |
| 8 | 4 | flags | `uint32` LE (bitmask, reserved) |
| 12 | 4 | payload_len | `uint32` LE (bytes of payload body) |
| 16 | 32 | checksum | SHA-256 of payload body (hex, 32 bytes) |
Total header size: **48 bytes ** .
#### 3.6.2 Payload Body
The payload body is a sequence of typed sections. Each section has a small
header:
| Offset | Size | Field | Type / Encoding |
|--------|------|------------|----------------------------|
| 0 | 1 | section_id | `uint8` (see table below) |
| 1 | 4 | section_len| `uint32` LE (bytes of section data) |
| 5 | ... | data | section-specific |
Section IDs:
| section_id | Section | Data layout |
|------------|----------------|-------------------------------------------------------------------|
| `0x01` | metadata | world_id, room_id (length-prefixed strings), coord (3× ×
| `0x02` | tile layer | name, layer_id, dims (3×
| `0x03` | object | one `WorldObject` (repeated; one section per object) |
| `0x04` | content flags | category severities (8×
| `0x05` | meta | created_at, updated_at (int64 unix), revision (uint32), author (string) |
**String encoding: ** length-prefixed UTF-8 — `uint16` LE byte length followed by
the UTF-8 bytes.
**Tile layer data layout (section `0x02`): **
```
name : string
layer_id : uint16 LE
width : uint16 LE
height : uint16 LE
depth : uint16 LE
palette_count : uint16 LE
palette : palette_count ×
tile_count : uint32 LE
tiles : tile_count ×
```
```
TileDef:
id : uint16 LE
asset_id : string
solid : uint8
flags : uint32 LE
variants : uint8
```
**Object data layout (section `0x03`): **
```
id : string
type : string
asset_id : string
pos : 3 ×
rot : 4 ×
scale : 3 ×
dynamic : uint8
properties : JSON string (length-prefixed)
flags : ContentFlags (see 0x04 layout)
```
**Content flags layout (section `0x04`): **
```
violence, sexual, language, substance, gambling, horror, hate, custom
: 8 ×
mature : uint8
source_count: uint16 LE
sources : source_count ×
```
#### 3.6.3 Byte-Stability
For a given logical payload, the binary encoding is **deterministic ** :
- All integers are little-endian.
- All strings are UTF-8, length-prefixed, and never null-terminated.
- Maps (e.g. `properties` ) are serialized as a **canonical JSON string ** with
keys sorted lexicographically.
- The `checksum` is the SHA-256 of the payload body bytes.
This guarantees that two encoders produce identical bytes, enabling content
addressing, caching, and reproducible tests.
---
## 4. SQLite Persistence Schema
The server persists world state in a **SQLite ** database. SQLite is chosen for
its zero-configuration, single-file, transactional, and embedded nature — ideal
for a server that owns the world store and needs ACID guarantees without a
separate database process.
### 4.1 Database Layout
The database file is `world.db` (path configurable). It contains the following
tables:
```
worlds
rooms
chunks
chunk_objects
content_flags
revisions
```
### 4.2 `worlds`
Top-level world containers.
``` sql
CREATE TABLE worlds (
id TEXT PRIMARY KEY , -- e.g. 'w_0001'
name TEXT NOT NULL ,
description TEXT ,
chunk_size_x INTEGER NOT NULL DEFAULT 32 ,
chunk_size_y INTEGER NOT NULL DEFAULT 32 ,
chunk_size_z INTEGER NOT NULL DEFAULT 1 ,
origin_x REAL NOT NULL DEFAULT 0 ,
origin_y REAL NOT NULL DEFAULT 0 ,
origin_z REAL NOT NULL DEFAULT 0 ,
created_at TEXT NOT NULL , -- ISO-8601 UTC
updated_at TEXT NOT NULL
) ;
```
### 4.3 `rooms`
Rooms within a world.
``` sql
CREATE TABLE rooms (
id TEXT PRIMARY KEY , -- e.g. 'r_0001'
world_id TEXT NOT NULL REFERENCES worlds ( id ) ON DELETE CASCADE ,
name TEXT NOT NULL ,
bounds_min_x REAL NOT NULL ,
bounds_min_y REAL NOT NULL ,
bounds_min_z REAL NOT NULL DEFAULT 0 ,
bounds_max_x REAL NOT NULL ,
bounds_max_y REAL NOT NULL ,
bounds_max_z REAL NOT NULL DEFAULT 1 ,
created_at TEXT NOT NULL ,
updated_at TEXT NOT NULL ,
UNIQUE ( world_id , name )
) ;
CREATE INDEX idx_rooms_world ON rooms ( world_id ) ;
```
### 4.4 `chunks`
The core persistence table. One row per chunk. The **payload ** is stored as a
BLOB in the **binary format ** (§3.6); the JSON form is derived on demand for
authoring/debugging and is not stored (to avoid dual-write drift).
``` sql
CREATE TABLE chunks (
key TEXT PRIMARY KEY , -- canonical key 'world:room:cx:cy:cz'
world_id TEXT NOT NULL REFERENCES worlds ( id ) ON DELETE CASCADE ,
room_id TEXT NOT NULL REFERENCES rooms ( id ) ON DELETE CASCADE ,
cx INTEGER NOT NULL ,
cy INTEGER NOT NULL ,
cz INTEGER NOT NULL DEFAULT 0 ,
format_version INTEGER NOT NULL DEFAULT 1 ,
payload BLOB NOT NULL , -- binary chunk encoding (§3.6)
payload_len INTEGER NOT NULL ,
checksum TEXT NOT NULL , -- SHA-256 of payload
revision INTEGER NOT NULL DEFAULT 0 ,
dirty INTEGER NOT NULL DEFAULT 0 , -- 1 = needs flush to disk
created_at TEXT NOT NULL ,
updated_at TEXT NOT NULL ,
UNIQUE ( world_id , room_id , cx , cy , cz )
) ;
CREATE INDEX idx_chunks_world_room ON chunks ( world_id , room_id ) ;
CREATE INDEX idx_chunks_coord ON chunks ( cx , cy , cz ) ;
```
**Design notes: **
- The `payload` BLOB is the single source of truth for chunk content. The
`checksum` guards against corruption and enables content addressing.
- `revision` increments on every write; it is used for optimistic concurrency
and change detection (see §5.4).
- `dirty` marks chunks that have been modified in memory but not yet flushed,
enabling a write-behind persistence strategy (see §5.5).
### 4.5 `chunk_objects`
A denormalized, queryable index of the objects inside each chunk. This is
**derived ** from the chunk payload and kept in sync by the write path. It exists
so the server can answer queries like "which chunks contain a door?" without
parsing every payload BLOB.
``` sql
CREATE TABLE chunk_objects (
id TEXT NOT NULL , -- object id (unique within chunk)
chunk_key TEXT NOT NULL REFERENCES chunks ( key ) ON DELETE CASCADE ,
type TEXT NOT NULL , -- 'prop', 'interactable', ...
asset_id TEXT ,
dynamic INTEGER NOT NULL DEFAULT 0 ,
pos_x REAL NOT NULL ,
pos_y REAL NOT NULL ,
pos_z REAL NOT NULL DEFAULT 0 ,
PRIMARY KEY ( chunk_key , id )
) ;
CREATE INDEX idx_chunk_objects_type ON chunk_objects ( type ) ;
CREATE INDEX idx_chunk_objects_asset ON chunk_objects ( asset_id ) ;
```
**Sync mechanism: ** `chunk_objects` is maintained **explicitly by the write
path** — it is not a SQL trigger. Whenever `put_chunk` (or `delete_chunk` )
commits a chunk, the service layer performs a **delete-and-reinsert ** of that
chunk's object rows in the same transaction: it deletes all rows for
`chunk_key` , then inserts fresh rows parsed from the new payload. This keeps the
index transactionally consistent with the payload and avoids the complexity and
fragility of SQL triggers over a binary BLOB. Because the index is derived, it
can always be rebuilt from `chunks.payload` if it ever drifts (a `rebuild_index`
utility is provided for this).
### 4.6 `content_flags`
Persists the aggregated moderation flags per chunk (and optionally per object).
This lets the server enforce age/content gates at load time without parsing
payloads.
``` sql
CREATE TABLE content_flags (
chunk_key TEXT NOT NULL REFERENCES chunks ( key ) ON DELETE CASCADE ,
object_id TEXT NOT NULL DEFAULT ' ' , -- '' = chunk-level flag (sentinel)
violence INTEGER NOT NULL DEFAULT 0 ,
sexual INTEGER NOT NULL DEFAULT 0 ,
language INTEGER NOT NULL DEFAULT 0 ,
substance INTEGER NOT NULL DEFAULT 0 ,
gambling INTEGER NOT NULL DEFAULT 0 ,
horror INTEGER NOT NULL DEFAULT 0 ,
hate INTEGER NOT NULL DEFAULT 0 ,
custom INTEGER NOT NULL DEFAULT 0 ,
mature INTEGER NOT NULL DEFAULT 0 ,
sources TEXT NOT NULL DEFAULT ' [] ' , -- JSON array of flag ids
PRIMARY KEY ( chunk_key , object_id )
) ;
CREATE INDEX idx_content_flags_mature ON content_flags ( mature ) ;
```
Severity is stored as `0=none, 1=mild, 2=moderate, 3=severe` .
**Chunk-level vs. object-level flags: ** Chunk-level flags use the **sentinel
value `''` (empty string)** in `object_id` rather than `NULL` . This is
deliberate: in SQLite, `NULL` values in a composite primary key are **not **
unique-constrained, so two rows with the same `chunk_key` and `object_id = NULL`
could be inserted accidentally. Using `''` as the sentinel means the composite
`PRIMARY KEY (chunk_key, object_id)` enforces that there is **exactly one **
chunk-level flag row per chunk. Object-level flags use the object's actual `id`
(which is never empty). The application layer treats `object_id = ''` as
"chunk-level" and any non-empty value as "object-level".
### 4.7 `revisions`
An append-only audit log of chunk writes. Enables rollback, auditing, and
replication of changes.
``` sql
CREATE TABLE revisions (
id INTEGER PRIMARY KEY AUTOINCREMENT ,
chunk_key TEXT NOT NULL , -- NOT a FK: survives chunk deletion
revision INTEGER NOT NULL ,
actor TEXT NOT NULL , -- user / agent id
action TEXT NOT NULL , -- 'create' | 'update' | 'delete'
checksum TEXT NOT NULL , -- SHA-256 of payload
payload BLOB , -- full binary payload at this revision
created_at TEXT NOT NULL
) ;
CREATE INDEX idx_revisions_chunk ON revisions ( chunk_key ) ;
```
**Rollback support: ** The `payload` BLOB stores the **full binary chunk payload **
at each revision. This is what makes rollback possible: to restore a chunk to a
previous revision, the server reads the `payload` of the target revision and
writes it back to `chunks.payload` (incrementing `revision` and appending a new
`revisions` row). Without the payload, only the checksum would be available,
which is insufficient to reconstruct the chunk.
**Audit trail preservation: ** `chunk_key` is **not ** a foreign key to `chunks` .
This means deleting a chunk does **not ** cascade-delete its revision history —
the audit trail is preserved even after a chunk is removed. (The `chunk_key`
string is still stored so the history can be correlated to a chunk that existed
at that time.) If a chunk is later recreated with the same key, its history is
still intact.
### 4.8 Schema Versioning & Migrations
The database carries a schema version in a `meta` table:
``` sql
CREATE TABLE meta (
key TEXT PRIMARY KEY ,
value TEXT NOT NULL
) ;
-- seed: INSERT INTO meta(key, value) VALUES ('schema_version', '1');
```
Migrations are applied in order at startup. Each migration is a numbered SQL
script; the server tracks the applied version in `meta` and applies any pending
migrations transactionally. This keeps the schema forward-compatible as the
chunk format evolves.
---
## 5. Read/Write API
The API is exposed as a **Python service layer ** on the backend (the same
process that owns the SQLite store) and consumed by the Godot client over the
network transport (see `networking.md` ). The service layer is the **only **
writer to the database; clients never touch SQLite directly.
### 5.1 Service Interface (Python)
``` python
class WorldStore :
""" Owns the SQLite connection and chunk read/write operations. """
def get_chunk ( self , key : str ) - > Chunk | None : . . .
def get_chunk_binary ( self , key : str ) - > bytes | None : . . .
def put_chunk ( self , chunk : Chunk , actor : str ) - > Chunk : . . .
def delete_chunk ( self , key : str , actor : str ) - > None : . . .
def list_chunks ( self , world_id : str , room_id : str | None = None ,
bounds : Bounds | None = None ) - > list [ str ] : . . .
def get_chunk_flags ( self , key : str ) - > ContentFlags : . . .
def flush_dirty ( self ) - > int : . . .
def rollback_chunk ( self , key : str , revision : int , actor : str ) - > Chunk : . . .
def rebuild_index ( self , chunk_key : str | None = None ) - > int : . . .
def migrate ( self ) - > None : . . .
```
### 5.2 Encoding Helpers
The service layer provides symmetric encode/decode functions shared by the
client and server:
``` python
def encode_binary ( chunk : Chunk ) - > bytes : . . .
def decode_binary ( data : bytes ) - > Chunk : . . .
def encode_json ( chunk : Chunk ) - > str : . . .
def decode_json ( text : str ) - > Chunk : . . .
def checksum ( payload : bytes ) - > str : . . .
```
These are the single implementation of the format spec (§3), so the Godot
client and Python backend never drift.
### 5.3 Network RPCs
The chunk API is exposed to the client via RPCs (matching the networking
architecture). All chunk traffic is **reliable ** (chunks are discrete,
must-arrive data).
| RPC | Direction | Payload | Notes |
|----------------------------|-----------|-------------------------------------------|-------|
| `request_chunk` | C → S | `{ key }` | Client asks for a chunk |
| `chunk_data` | S → C | `{ key, binary_payload, checksum }` | Server streams a chunk |
| `request_chunk_edit` | C → S | `{ key, base_revision, delta }` | Client submits an edit |
| `chunk_edit_result` | S → C | `{ key, success, new_revision, error? }` | Acknowledges an edit |
| `chunk_invalidated` | S → C | `{ key, new_revision }` | Server pushes a change |
| `request_chunk_flags` | C → S | `{ key }` | Age/content gate check |
### 5.4 Edit & Concurrency Model
- **Read:** Client requests a chunk; server returns the binary payload and its
`revision` .
- **Edit:** Client submits a **delta ** against a known `base_revision` . The
server applies the delta, increments `revision` , persists, and returns the new
revision.
- **Optimistic concurrency:** If the chunk's current revision differs from the
client's `base_revision` , the server rejects the edit with a conflict error
and the client re-fetches and re-applies. This prevents lost updates when two
clients edit the same chunk.
- **Deltas:** A delta is a JSON patch (RFC 6902 style) applied to the chunk's
logical model. The server validates the patched chunk (schema, bounds,
moderation) before persisting.
### 5.5 Write-Behind Persistence
To amortize disk I/O, the server keeps a **dirty set ** of chunks modified in
memory and flushes them to SQLite in batches:
1. `put_chunk` updates the in-memory chunk and marks it `dirty` .
2. A background flusher periodically calls `flush_dirty()` , which writes all
dirty chunks in a single transaction and clears the `dirty` flag.
3. On graceful shutdown, `flush_dirty()` is called to ensure durability.
This keeps the hot path (in-memory chunk access) fast while guaranteeing
durability at shutdown and on a configurable interval.
**Recovery after unclean shutdown: ** The `dirty` column in the `chunks` table
exists specifically to support recovery if the process crashes before a flush
completes. On startup, the server scans for rows where `dirty = 1` and treats
them as **unflushed in-memory state that was lost ** — it re-reads the last
persisted payload from `chunks.payload` (the last durable state) and clears the
`dirty` flag. In other words, `dirty = 1` at startup means "the last write was
not durable; discard the in-memory delta and fall back to the persisted
payload." This guarantees that the on-disk state is always a consistent,
previously-committed snapshot, never a partially-written chunk. (The in-memory
dirty set is authoritative during normal operation; the `dirty` column is only a
recovery marker.)
### 5.6 Streaming & Caching
- The server streams only the chunks a client needs (those within the client's
view radius), using `list_chunks` with a bounds query.
- Chunks are cached in memory (LRU) keyed by `key` ; the cache is invalidated on
write.
- The binary payload is content-addressed by `checksum` , so unchanged chunks can
be served from cache without re-reading SQLite.
---
## 6. Format Versioning & Migration
- The chunk format `version` field (currently `1` ) is checked on decode. A
decoder that encounters a higher version than it supports **rejects ** the
chunk with a clear error rather than misreading it.
- The SQLite `schema_version` in `meta` drives database migrations (§4.8).
- Chunk payloads are migrated **lazily ** : when a chunk is read and its
`format_version` is older than the current one, the server migrates it to the
current version in memory and marks it dirty so it is rewritten on the next
flush. This avoids a costly full-world migration at startup.
---
## 7. Security & Integrity
- **Server-authoritative writes:** Only the `WorldStore` writes to SQLite;
clients submit deltas that are validated before application.
- **Checksum verification:** Every chunk payload is checksummed (SHA-256) on
write and verified on read. Corruption is detected and the chunk is re-fetched
or flagged.
- **Bounds validation:** Chunk coordinates and object positions are validated
against the world's chunk dimensions and room bounds before persistence.
- **Moderation enforcement:** `content_flags` are checked at load time; mature
chunks are gated behind the age-verification flow (see `content_moderation.md` ).
- **Input validation:** All string fields are length-limited and validated
against allowed character sets to prevent injection and oversized payloads.
---
## 8. Open Questions / Future Work
- **Volumetric worlds:** The schema and format already support `CHUNK_SIZE_Z >
1`, but the tooling and streaming for 3D chunks are not yet specified.
- **Chunk compression:** For very large worlds, the binary payload could be
compressed (e.g. zstd) before storage/transfer. The header reserves a ` flags`
bitmask for a compression flag.
- **Multi-server sharding:** If a single SQLite file becomes a bottleneck, the
` chunks` table could be sharded by ` world_id` across multiple databases. The
key design already supports this (world_id is a leading key component).
- **Delta compression for streaming:** Instead of sending full chunks, the
server could send only the delta between revisions to reduce bandwidth for
frequently-updated chunks.
coder-bot
architect-bot
agent-bot