modbison/docs/architecture/world.md

World Chunk Data Format & Persistence Architecture

Status: Proposed Owner: Platform team Related: Server authority + client prediction, avatar component system, content moderation & age verification, backend auth (JWT)

1. Overview

This document specifies the world chunk data format and persistence schema for agp-game, a Godot-based multiplayer game built by autonomous agents. It defines:

• The chunk data format — both a human-readable JSON form (for authoring, tooling, and debugging) and a compact binary form (for runtime loading and network transfer).
• The SQLite schema that persists world state on the server, including chunk metadata, chunk payloads, and the world/room hierarchy.
• The read/write API that the server and client use to load, save, and stream chunks.

The goal is a single, versioned chunk contract that is shared across the Godot client, the Python backend, and the persistence layer. A chunk is the atomic unit of world geometry, gameplay objects, and persistence — the server loads and saves whole chunks, and clients stream the chunks around them.

1.1 Design Goals

• Versioned & forward-compatible: Every chunk carries a format version so the loader can migrate old data and reject unknown future data gracefully.
• Dual representation: JSON for authoring/debugging, binary for runtime/network. Both encode the same logical model so there is exactly one source of truth for the schema.
• Server-authoritative persistence: The server is the only writer to the SQLite store. Clients submit edits (deltas) and the server applies and persists them, mirroring the server-authoritative networking model.
• Chunk-granular persistence: Persistence happens at chunk granularity so the server can save a dirty chunk without rewriting the whole world, and can stream only the chunks a client needs.
• Deterministic & testable: The binary format is byte-stable for a given logical payload, enabling checksums, caching, and reproducible tests.
• Moderation-aware: Chunks reference content (assets, scripts) that carry content flags (see content_moderation.md); the persistence layer stores the moderation state alongside chunk data so enforcement can happen at load time.

---

2. World Model & Chunking

2.1 World Hierarchy

The world is organized as a hierarchy that matches the moderation content graph:

World
 └── Room
      └── Chunk   (the atomic persistence & streaming unit)
           ├── Tiles / Terrain
           ├── Static objects (props, geometry)
           ├── Dynamic objects (spawnable entities, interactables)
           └── Metadata (flags, version, timestamps)

• A World is the top-level container (a named, versioned space).
• A Room is a bounded region within a world (a level, a building, a zone).
• A Chunk is a fixed-size, axis-aligned region of a room. It is the unit of load, save, and stream.

2.2 Chunk Dimensions

Chunks are fixed-size, axis-aligned boxes in world space. The default dimensions are a configuration constant so they can be tuned per world:

Constant	Default	Description
CHUNK_SIZE_X	32	Chunk width in world units (tiles).
CHUNK_SIZE_Y	32	Chunk height in world units (tiles).
CHUNK_SIZE_Z	1	Chunk depth (1 = flat 2D world; >1 = volumetric).
CHUNK_ORIGIN	(0,0)	World-space origin of chunk coordinate (0,0).

A chunk is identified by its integer chunk coordinate (cx, cy, cz), computed from a world-space position p as:

cx = floor((p.x - CHUNK_ORIGIN.x) / CHUNK_SIZE_X)
cy = floor((p.y - CHUNK_ORIGIN.y) / CHUNK_SIZE_Y)
cz = floor((p.z - CHUNK_ORIGIN.z) / CHUNK_SIZE_Z)

The chunk's world-space bounding box is therefore:

min = (CHUNK_ORIGIN.x + cx*CHUNK_SIZE_X, ...)
max = min + (CHUNK_SIZE_X, CHUNK_SIZE_Y, CHUNK_SIZE_Z)

2.3 Chunk Coordinate Key

For storage and lookup, a chunk is addressed by a canonical string key:

"{world_id}:{room_id}:{cx}:{cy}:{cz}"

Example: "w_0001:r_0001:12:-4:0". This key is the primary lookup key in the SQLite chunks table and is used in the read/write API.

---

3. Chunk Data Format

3.1 Logical Model

Both the JSON and binary encodings serialize the same logical chunk model. The logical model is defined here once; the two encodings are projections of it.

Chunk
 ├── version        : uint   (format version, currently 1)
 ├── key            : string (canonical chunk key, see §2.3)
 ├── world_id       : string
 ├── room_id        : string
 ├── coord          : { cx, cy, cz }   (int)
 ├── bounds         : { min_x, min_y, min_z, max_x, max_y, max_z } (float)
 ├── tiles          : TileLayer[]       (terrain / tile data)
 ├── objects        : WorldObject[]     (static + dynamic objects)
 ├── flags          : ContentFlags      (moderation flags, see §3.5)
 ├── meta           : { created_at, updated_at, revision, author } 
 └── checksum       : string            (SHA-256 of canonical payload)

3.2 Tile Layer

A chunk's terrain is stored as one or more tile layers. Each layer is a dense grid of tile IDs of size CHUNK_SIZE_X × CHUNK_SIZE_Y (× CHUNK_SIZE_Z for volumetric worlds).

TileLayer
 ├── name      : string   (e.g. "ground", "walls", "decor")
 ├── layer_id  : uint
 ├── width     : uint     (= CHUNK_SIZE_X)
 ├── height    : uint     (= CHUNK_SIZE_Y)
 ├── depth     : uint     (= CHUNK_SIZE_Z)
 ├── tiles     : uint[]   (flattened tile IDs, row-major, length = w*h*d)
 └── palette   : TileDef[] (tile ID → tile definition, shared per layer)

Tile encoding: A tile ID is a uint16 index into the layer's palette. 0 is reserved for "empty/air". The palette maps each tile ID to a TileDef:

TileDef
 ├── id       : uint16
 ├── asset_id : string   (reference to a tile/terrain asset)
 ├── solid    : bool     (collision)
 ├── flags    : uint32   (bitmask: walkable, climbable, liquid, ...)
 └── variants : uint8    (number of visual variants for this tile)

Rationale for palette + indices: Storing dense uint16 indices rather than full asset strings per cell keeps the binary payload small (2 bytes/cell) while the palette (shared across the layer) carries the rich per-tile metadata.

3.3 World Objects

Objects are the non-tile entities in a chunk: static props, interactables, spawn points, and dynamic-object templates.

WorldObject
 ├── id          : string   (stable object id, unique within the chunk)
 ├── type        : string   (e.g. "prop", "interactable", "spawn_point", "door")
 ├── asset_id    : string   (reference to the object's mesh/glTF asset)
 ├── transform   : { pos: [x,y,z], rot: [x,y,z,w], scale: [x,y,z] }
 ├── properties  : object   (type-specific key/value map, JSON-typed)
 ├── dynamic     : bool     (true if this is a spawnable dynamic entity)
 └── flags       : ContentFlags (per-object moderation flags)

Dynamic objects are stored as templates in the chunk. Their runtime state (position, health, inventory) is not stored in the chunk; it lives in the runtime entity store and is persisted separately (see §4.4). This keeps chunks stable and diffable — a chunk only changes when its static content changes.

3.4 Content Flags

Each chunk (and each object) carries a content flag set that is the aggregate of the moderation flags of the assets and scripts it references. This mirrors the aggregation rules in content_moderation.md:

ContentFlags
 ├── categories : { violence, sexual, language, substance, gambling, horror, hate, custom }
 │                (each → severity: none|mild|moderate|severe)
 ├── mature     : bool   (true if any contributing item is mature)
 └── sources    : string[] (flag ids that contributed, for audit)

The server computes and persists these flags at write time so that enforcement (age gating, content filtering) can happen at load time without re-scanning assets.

3.5 JSON Encoding

The JSON encoding is the authoring and debugging representation. It is human-readable, diffable, and used by tools and tests. It is the canonical logical model serialized directly.

{
  "version": 1,
  "key": "w_0001:r_0001:12:-4:0",
  "world_id": "w_0001",
  "room_id": "r_0001",
  "coord": { "cx": 12, "cy": -4, "cz": 0 },
  "bounds": { "min_x": 384.0, "min_y": -128.0, "min_z": 0.0,
              "max_x": 416.0, "max_y": -96.0,  "max_z": 1.0 },
  "tiles": [
    {
      "name": "ground",
      "layer_id": 0,
      "width": 32,
      "height": 32,
      "depth": 1,
      "palette": [
        { "id": 0, "asset_id": "", "solid": false, "flags": 0, "variants": 1 },
        { "id": 1, "asset_id": "tiles/grass", "solid": false, "flags": 1, "variants": 4 }
      ],
      "tiles": [0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1]
    }
  ],
  "objects": [
    {
      "id": "obj_0001",
      "type": "interactable",
      "asset_id": "props/chest",
      "transform": { "pos": [400.0, -110.0, 0.0],
                     "rot": [0.0, 0.0, 0.0, 1.0],
                     "scale": [1.0, 1.0, 1.0] },
      "properties": { "loot_table": "chest_basic", "locked": false },
      "dynamic": false,
      "flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] }
    }
  ],
  "flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] },
  "meta": { "created_at": "2026-08-17T00:00:00Z",
            "updated_at": "2026-08-17T00:00:00Z",
            "revision": 3,
            "author": "agent-bot" },
  "checksum": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
}

3.6 Binary Encoding

The binary encoding is the runtime and network representation. It is compact and byte-stable. It uses a simple, explicitly-specified layout (no external serialization library) so it can be produced and consumed identically by the Godot client (GDScript) and the Python backend.

3.6.1 Header

Offset	Size	Field	Type / Encoding
0	4	magic	ASCII "AGPC" (0x41 0x47 0x50 0x43)
4	2	version	uint16 LE (currently 1)
6	2	header_len	uint16 LE (bytes of header after this)
8	4	flags	uint32 LE (bitmask, reserved)
12	4	payload_len	uint32 LE (bytes of payload body)
16	32	checksum	SHA-256 of payload body (hex, 32 bytes)

Total header size: 48 bytes.

3.6.2 Payload Body

The payload body is a sequence of typed sections. Each section has a small header:

Offset	Size	Field	Type / Encoding
0	1	section_id	uint8 (see table below)
1	4	section_len	uint32 LE (bytes of section data)
5	...	data	section-specific

Section IDs:

section_id	Section	Data layout
0x01	metadata	world_id, room_id (length-prefixed strings), coord (3×int32), bounds (6×float32)
0x02	tile layer	name, layer_id, dims (3×uint16), palette, dense tile indices
0x03	object	one WorldObject (repeated; one section per object)
0x04	content flags	category severities (8×uint8), mature (uint8), sources (string list)
0x05	meta	created_at, updated_at (int64 unix), revision (uint32), author (string)

String encoding: length-prefixed UTF-8 — uint16 LE byte length followed by the UTF-8 bytes.

Tile layer data layout (section 0x02):

name        : string
layer_id    : uint16 LE
width       : uint16 LE
height      : uint16 LE
depth       : uint16 LE
palette_count : uint16 LE
palette     : palette_count × TileDef
tile_count  : uint32 LE
tiles       : tile_count × uint16 LE   (dense, row-major)

TileDef:
  id        : uint16 LE
  asset_id  : string
  solid     : uint8
  flags     : uint32 LE
  variants  : uint8

Object data layout (section 0x03):

id          : string
type        : string
asset_id    : string
pos         : 3 × float32
rot         : 4 × float32
scale       : 3 × float32
dynamic     : uint8
properties  : JSON string (length-prefixed)
flags       : ContentFlags (see 0x04 layout)

Content flags layout (section 0x04):

violence, sexual, language, substance, gambling, horror, hate, custom
            : 8 × uint8   (severity enum: 0=none,1=mild,2=moderate,3=severe)
mature      : uint8
source_count: uint16 LE
sources     : source_count × string

3.6.3 Byte-Stability

For a given logical payload, the binary encoding is deterministic:

• All integers are little-endian.
• All strings are UTF-8, length-prefixed, and never null-terminated.
• Maps (e.g. properties) are serialized as a canonical JSON string with keys sorted lexicographically.
• The checksum is the SHA-256 of the payload body bytes.

This guarantees that two encoders produce identical bytes, enabling content addressing, caching, and reproducible tests.

---

4. SQLite Persistence Schema

The server persists world state in a SQLite database. SQLite is chosen for its zero-configuration, single-file, transactional, and embedded nature — ideal for a server that owns the world store and needs ACID guarantees without a separate database process.

4.1 Database Layout

The database file is world.db (path configurable). It contains the following tables:

worlds
rooms
chunks
chunk_objects
content_flags
revisions

4.2 worlds

Top-level world containers.

CREATE TABLE worlds (
    id          TEXT PRIMARY KEY,          -- e.g. 'w_0001'
    name        TEXT NOT NULL,
    description TEXT,
    chunk_size_x INTEGER NOT NULL DEFAULT 32,
    chunk_size_y INTEGER NOT NULL DEFAULT 32,
    chunk_size_z INTEGER NOT NULL DEFAULT 1,
    origin_x    REAL NOT NULL DEFAULT 0,
    origin_y    REAL NOT NULL DEFAULT 0,
    origin_z    REAL NOT NULL DEFAULT 0,
    created_at  TEXT NOT NULL,             -- ISO-8601 UTC
    updated_at  TEXT NOT NULL
);

4.3 rooms

Rooms within a world.

CREATE TABLE rooms (
    id          TEXT PRIMARY KEY,          -- e.g. 'r_0001'
    world_id    TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE,
    name        TEXT NOT NULL,
    bounds_min_x REAL NOT NULL,
    bounds_min_y REAL NOT NULL,
    bounds_min_z REAL NOT NULL DEFAULT 0,
    bounds_max_x REAL NOT NULL,
    bounds_max_y REAL NOT NULL,
    bounds_max_z REAL NOT NULL DEFAULT 1,
    created_at  TEXT NOT NULL,
    updated_at  TEXT NOT NULL,
    UNIQUE (world_id, name)
);
CREATE INDEX idx_rooms_world ON rooms(world_id);

4.4 chunks

The core persistence table. One row per chunk. The payload is stored as a BLOB in the binary format (§3.6); the JSON form is derived on demand for authoring/debugging and is not stored (to avoid dual-write drift).

CREATE TABLE chunks (
    key         TEXT PRIMARY KEY,          -- canonical key 'world:room:cx:cy:cz'
    world_id    TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE,
    room_id     TEXT NOT NULL REFERENCES rooms(id) ON DELETE CASCADE,
    cx          INTEGER NOT NULL,
    cy          INTEGER NOT NULL,
    cz          INTEGER NOT NULL DEFAULT 0,
    format_version INTEGER NOT NULL DEFAULT 1,
    payload     BLOB NOT NULL,             -- binary chunk encoding (§3.6)
    payload_len INTEGER NOT NULL,
    checksum    TEXT NOT NULL,             -- SHA-256 of payload
    revision    INTEGER NOT NULL DEFAULT 0,
    dirty       INTEGER NOT NULL DEFAULT 0, -- 1 = needs flush to disk
    created_at  TEXT NOT NULL,
    updated_at  TEXT NOT NULL,
    UNIQUE (world_id, room_id, cx, cy, cz)
);
CREATE INDEX idx_chunks_world_room ON chunks(world_id, room_id);
CREATE INDEX idx_chunks_coord ON chunks(cx, cy, cz);

Design notes:

• The payload BLOB is the single source of truth for chunk content. The checksum guards against corruption and enables content addressing.
• revision increments on every write; it is used for optimistic concurrency and change detection (see §5.4).
• dirty marks chunks that have been modified in memory but not yet flushed, enabling a write-behind persistence strategy (see §5.5).

4.5 chunk_objects

A denormalized, queryable index of the objects inside each chunk. This is derived from the chunk payload and kept in sync by the write path. It exists so the server can answer queries like "which chunks contain a door?" without parsing every payload BLOB.

CREATE TABLE chunk_objects (
    id          TEXT NOT NULL,             -- object id (unique within chunk)
    chunk_key   TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE,
    type        TEXT NOT NULL,             -- 'prop', 'interactable', ...
    asset_id    TEXT,
    dynamic     INTEGER NOT NULL DEFAULT 0,
    pos_x       REAL NOT NULL,
    pos_y       REAL NOT NULL,
    pos_z       REAL NOT NULL DEFAULT 0,
    PRIMARY KEY (chunk_key, id)
);
CREATE INDEX idx_chunk_objects_type ON chunk_objects(type);
CREATE INDEX idx_chunk_objects_asset ON chunk_objects(asset_id);

4.6 content_flags

Persists the aggregated moderation flags per chunk (and optionally per object). This lets the server enforce age/content gates at load time without parsing payloads.

CREATE TABLE content_flags (
    chunk_key   TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE,
    object_id   TEXT,                      -- NULL = chunk-level flag
    violence    INTEGER NOT NULL DEFAULT 0,
    sexual      INTEGER NOT NULL DEFAULT 0,
    language    INTEGER NOT NULL DEFAULT 0,
    substance   INTEGER NOT NULL DEFAULT 0,
    gambling    INTEGER NOT NULL DEFAULT 0,
    horror      INTEGER NOT NULL DEFAULT 0,
    hate        INTEGER NOT NULL DEFAULT 0,
    custom      INTEGER NOT NULL DEFAULT 0,
    mature      INTEGER NOT NULL DEFAULT 0,
    sources     TEXT NOT NULL DEFAULT '[]', -- JSON array of flag ids
    PRIMARY KEY (chunk_key, object_id)
);
CREATE INDEX idx_content_flags_mature ON content_flags(mature);

Severity is stored as 0=none, 1=mild, 2=moderate, 3=severe.

4.7 revisions

An append-only audit log of chunk writes. Enables rollback, auditing, and replication of changes.

CREATE TABLE revisions (
    id          INTEGER PRIMARY KEY AUTOINCREMENT,
    chunk_key   TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE,
    revision    INTEGER NOT NULL,
    actor       TEXT NOT NULL,             -- user / agent id
    action      TEXT NOT NULL,             -- 'create' | 'update' | 'delete'
    checksum    TEXT NOT NULL,
    created_at  TEXT NOT NULL
);
CREATE INDEX idx_revisions_chunk ON revisions(chunk_key);

4.8 Schema Versioning & Migrations

The database carries a schema version in a meta table:

CREATE TABLE meta (
    key   TEXT PRIMARY KEY,
    value TEXT NOT NULL
);
-- seed: INSERT INTO meta(key, value) VALUES ('schema_version', '1');

Migrations are applied in order at startup. Each migration is a numbered SQL script; the server tracks the applied version in meta and applies any pending migrations transactionally. This keeps the schema forward-compatible as the chunk format evolves.

---

5. Read/Write API

The API is exposed as a Python service layer on the backend (the same process that owns the SQLite store) and consumed by the Godot client over the network transport (see networking.md). The service layer is the only writer to the database; clients never touch SQLite directly.

5.1 Service Interface (Python)

class WorldStore:
    """Owns the SQLite connection and chunk read/write operations."""

    def get_chunk(self, key: str) -> Chunk | None: ...
    def get_chunk_binary(self, key: str) -> bytes | None: ...
    def put_chunk(self, chunk: Chunk, actor: str) -> Chunk: ...
    def delete_chunk(self, key: str, actor: str) -> None: ...
    def list_chunks(self, world_id: str, room_id: str | None = None,
                    bounds: Bounds | None = None) -> list[str]: ...
    def get_chunk_flags(self, key: str) -> ContentFlags: ...
    def flush_dirty(self) -> int: ...
    def migrate(self) -> None: ...

5.2 Encoding Helpers

The service layer provides symmetric encode/decode functions shared by the client and server:

def encode_binary(chunk: Chunk) -> bytes: ...
def decode_binary(data: bytes) -> Chunk: ...
def encode_json(chunk: Chunk) -> str: ...
def decode_json(text: str) -> Chunk: ...
def checksum(payload: bytes) -> str: ...

These are the single implementation of the format spec (§3), so the Godot client and Python backend never drift.

5.3 Network RPCs

The chunk API is exposed to the client via RPCs (matching the networking architecture). All chunk traffic is reliable (chunks are discrete, must-arrive data).

RPC	Direction	Payload	Notes
request_chunk	C → S	{ key }	Client asks for a chunk
chunk_data	S → C	{ key, binary_payload, checksum }	Server streams a chunk
request_chunk_edit	C → S	{ key, base_revision, delta }	Client submits an edit
chunk_edit_result	S → C	{ key, success, new_revision, error? }	Acknowledges an edit
chunk_invalidated	S → C	{ key, new_revision }	Server pushes a change
request_chunk_flags	C → S	{ key }	Age/content gate check

5.4 Edit & Concurrency Model

• Read: Client requests a chunk; server returns the binary payload and its revision.
• Edit: Client submits a delta against a known base_revision. The server applies the delta, increments revision, persists, and returns the new revision.
• Optimistic concurrency: If the chunk's current revision differs from the client's base_revision, the server rejects the edit with a conflict error and the client re-fetches and re-applies. This prevents lost updates when two clients edit the same chunk.
• Deltas: A delta is a JSON patch (RFC 6902 style) applied to the chunk's logical model. The server validates the patched chunk (schema, bounds, moderation) before persisting.

5.5 Write-Behind Persistence

To amortize disk I/O, the server keeps a dirty set of chunks modified in memory and flushes them to SQLite in batches:

1. put_chunk updates the in-memory chunk and marks it dirty.
2. A background flusher periodically calls flush_dirty(), which writes all dirty chunks in a single transaction and clears the dirty flag.
3. On graceful shutdown, flush_dirty() is called to ensure durability.

This keeps the hot path (in-memory chunk access) fast while guaranteeing durability at shutdown and on a configurable interval.

5.6 Streaming & Caching

• The server streams only the chunks a client needs (those within the client's view radius), using list_chunks with a bounds query.
• Chunks are cached in memory (LRU) keyed by key; the cache is invalidated on write.
• The binary payload is content-addressed by checksum, so unchanged chunks can be served from cache without re-reading SQLite.

---

6. Format Versioning & Migration

• The chunk format version field (currently 1) is checked on decode. A decoder that encounters a higher version than it supports rejects the chunk with a clear error rather than misreading it.
• The SQLite schema_version in meta drives database migrations (§4.8).
• Chunk payloads are migrated lazily: when a chunk is read and its format_version is older than the current one, the server migrates it to the current version in memory and marks it dirty so it is rewritten on the next flush. This avoids a costly full-world migration at startup.

---

7. Security & Integrity

• Server-authoritative writes: Only the WorldStore writes to SQLite; clients submit deltas that are validated before application.
• Checksum verification: Every chunk payload is checksummed (SHA-256) on write and verified on read. Corruption is detected and the chunk is re-fetched or flagged.
• Bounds validation: Chunk coordinates and object positions are validated against the world's chunk dimensions and room bounds before persistence.
• Moderation enforcement: content_flags are checked at load time; mature chunks are gated behind the age-verification flow (see content_moderation.md).
• Input validation: All string fields are length-limited and validated against allowed character sets to prevent injection and oversized payloads.

---

8. Open Questions / Future Work

• Volumetric worlds: The schema and format already support CHUNK_SIZE_Z > 1, but the tooling and streaming for 3D chunks are not yet specified.
• Chunk compression: For very large worlds, the binary payload could be compressed (e.g. zstd) before storage/transfer. The header reserves a flags bitmask for a compression flag.
• Multi-server sharding: If a single SQLite file becomes a bottleneck, the chunks table could be sharded by world_id across multiple databases. The key design already supports this (world_id is a leading key component).
• Delta compression for streaming: Instead of sending full chunks, the server could send only the delta between revisions to reduce bandwidth for frequently-updated chunks.