Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 agent-bot
|
5eaeed80b4 |
@@ -0,0 +1,462 @@
|
|||||||
|
# Avatar Component System & glTF Attachment Spec
|
||||||
|
|
||||||
|
> **Status:** Proposed
|
||||||
|
> **Owner:** Platform team
|
||||||
|
> **Related:** Server authority + client prediction, world chunk data format
|
||||||
|
|
||||||
|
## 1. Overview
|
||||||
|
|
||||||
|
This document specifies the **avatar component system** for **agp-game**, a
|
||||||
|
Godot-based multiplayer game. It defines:
|
||||||
|
|
||||||
|
- How an avatar is decomposed into **body parts** and **attachment slots**.
|
||||||
|
- How **cosmetic attachments** (clothing, accessories, gear) mount onto the
|
||||||
|
base body.
|
||||||
|
- The **glTF/GLB format requirements** that all avatar assets must satisfy so
|
||||||
|
they can be loaded, skinned, and networked consistently.
|
||||||
|
|
||||||
|
The goal is a **modular, data-driven avatar pipeline**: a single base body rig
|
||||||
|
plus swappable attachments, all authored in glTF and consumed by a shared Godot
|
||||||
|
runtime. This enables player customization, cosmetic monetization, and
|
||||||
|
server-authoritative validation of what a player is wearing.
|
||||||
|
|
||||||
|
### 1.1 Design Goals
|
||||||
|
|
||||||
|
- **Modularity:** Body parts and attachments are independent assets that can be
|
||||||
|
combined at runtime without re-exporting.
|
||||||
|
- **Consistency:** A single, well-defined skeleton and slot convention so any
|
||||||
|
attachment fits any base body.
|
||||||
|
- **Networkability:** The avatar's composition is expressed as a compact,
|
||||||
|
serializable **loadout** that the server can validate and replicate.
|
||||||
|
- **Performance:** Attachments are skinned to the same skeleton and share
|
||||||
|
materials where possible to minimize draw calls and skinning cost.
|
||||||
|
- **Tooling-friendly:** Assets are authored in standard glTF 2.0 and validated
|
||||||
|
by a build-time checker.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Avatar Decomposition
|
||||||
|
|
||||||
|
### 2.1 The Base Body
|
||||||
|
|
||||||
|
Every avatar is built on a **base body** — a single glTF asset containing:
|
||||||
|
|
||||||
|
- A **skeleton** (the canonical rig, see §3).
|
||||||
|
- A **base mesh** for the skin/body (head, torso, arms, legs).
|
||||||
|
- **Attachment points** (empty nodes) that define where accessories mount.
|
||||||
|
|
||||||
|
The base body is the only asset that carries the skeleton. Attachments reference
|
||||||
|
the skeleton by **bone name** and are skinned to it; they never carry their own
|
||||||
|
rig.
|
||||||
|
|
||||||
|
### 2.2 Body Parts vs. Attachments
|
||||||
|
|
||||||
|
Two distinct concepts:
|
||||||
|
|
||||||
|
| Concept | Definition | Example |
|
||||||
|
|----------------|-------------------------------------------------------------------------|----------------------------------|
|
||||||
|
| **Body part** | A mesh that replaces part of the base body's skin (e.g., a different head, a tail, wings). | Alternate head, cybernetic arm |
|
||||||
|
| **Attachment** | A mesh that mounts *on top of* the body without replacing it (clothing, accessories). | Shirt, hat, glasses, backpack |
|
||||||
|
|
||||||
|
The distinction matters for **slot exclusivity** and **layering**:
|
||||||
|
|
||||||
|
- **Body parts** are *exclusive* per slot — only one head can be equipped.
|
||||||
|
- **Attachments** are *layered* — a hat and glasses can coexist, and both sit
|
||||||
|
above the head mesh.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Skeleton & Slot Convention
|
||||||
|
|
||||||
|
### 3.1 Canonical Skeleton
|
||||||
|
|
||||||
|
All avatars share a **canonical skeleton** with a fixed set of bone names. This
|
||||||
|
is the contract that lets any attachment fit any body. The skeleton follows a
|
||||||
|
humanoid convention (Godot's `Humanoid` profile / glTF VRM-style naming).
|
||||||
|
|
||||||
|
Required bones (subset of the full rig):
|
||||||
|
|
||||||
|
```
|
||||||
|
Root
|
||||||
|
├── Hips
|
||||||
|
│ ├── Spine
|
||||||
|
│ │ ├── Chest
|
||||||
|
│ │ │ ├── Neck
|
||||||
|
│ │ │ │ └── Head
|
||||||
|
│ │ │ │ ├── HeadTop # hat / hair mount
|
||||||
|
│ │ │ │ ├── Jaw # face accessories
|
||||||
|
│ │ │ │ └── LeftEye / RightEye
|
||||||
|
│ │ │ ├── LeftShoulder
|
||||||
|
│ │ │ │ └── LeftUpperArm
|
||||||
|
│ │ │ │ └── LeftLowerArm
|
||||||
|
│ │ │ │ └── LeftHand
|
||||||
|
│ │ │ └── RightShoulder
|
||||||
|
│ │ │ └── RightUpperArm
|
||||||
|
│ │ │ └── RightLowerArm
|
||||||
|
│ │ │ └── RightHand
|
||||||
|
│ │ └── (Chest children for torso attachments)
|
||||||
|
│ ├── LeftUpperLeg
|
||||||
|
│ │ └── LeftLowerLeg
|
||||||
|
│ │ └── LeftFoot
|
||||||
|
│ └── RightUpperLeg
|
||||||
|
│ └── RightLowerLeg
|
||||||
|
│ └── RightFoot
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rules:**
|
||||||
|
|
||||||
|
- Bone names are **case-sensitive** and must match exactly.
|
||||||
|
- The skeleton is **T-pose** at rest for consistent skinning.
|
||||||
|
- Attachments must be skinned to **at least** the bones they deform; static
|
||||||
|
accessories may be parented to a single bone (e.g., a hat parented to `Head`).
|
||||||
|
|
||||||
|
### 3.2 Attachment Slots
|
||||||
|
|
||||||
|
Slots are the **logical mount points** for body parts and attachments. Each slot
|
||||||
|
maps to one or more bones and defines what can be equipped there.
|
||||||
|
|
||||||
|
| Slot ID | Category | Mount bone(s) | Exclusivity | Example assets |
|
||||||
|
|----------------|--------------|--------------------|-------------|-------------------------------|
|
||||||
|
| `head` | body part | `Head` | exclusive | alternate head, face |
|
||||||
|
| `hair` | attachment | `HeadTop` | exclusive | hairstyles, hats |
|
||||||
|
| `face` | attachment | `Jaw` | exclusive | glasses, masks, beards |
|
||||||
|
| `torso` | body part | `Chest` | exclusive | alternate torso |
|
||||||
|
| `chest` | attachment | `Chest` | exclusive | shirts, armor, chest gear |
|
||||||
|
| `back` | attachment | `Chest` (back) | exclusive | backpacks, wings, capes |
|
||||||
|
| `left_arm` | body part | `LeftUpperArm` | exclusive | alternate arm |
|
||||||
|
| `right_arm` | body part | `RightUpperArm` | exclusive | alternate arm |
|
||||||
|
| `left_hand` | attachment | `LeftHand` | exclusive | gloves, weapons (held) |
|
||||||
|
| `right_hand` | attachment | `RightHand` | exclusive | gloves, weapons (held) |
|
||||||
|
| `legs` | body part | `Hips` | exclusive | alternate legs |
|
||||||
|
| `pants` | attachment | `Hips` | exclusive | pants, skirts |
|
||||||
|
| `left_foot` | attachment | `LeftFoot` | exclusive | shoes |
|
||||||
|
| `right_foot` | attachment | `RightFoot` | exclusive | shoes |
|
||||||
|
| `accessory` | attachment | `Root` (free) | multiple | belts, jewelry, floating items|
|
||||||
|
|
||||||
|
**Slot semantics:**
|
||||||
|
|
||||||
|
- **`exclusive`** slots allow at most one equipped item at a time.
|
||||||
|
- The **`accessory`** slot is *multi-slot* — multiple accessories may be equipped
|
||||||
|
simultaneously (bounded by a configurable limit, e.g., 8).
|
||||||
|
- **Body-part slots** (`head`, `torso`, `legs`, `left_arm`, `right_arm`) replace
|
||||||
|
the corresponding base mesh. When a body part is equipped, the base mesh for
|
||||||
|
that region is hidden.
|
||||||
|
- **Attachment slots** layer on top of the base mesh and never hide it.
|
||||||
|
|
||||||
|
### 3.3 Slot → Bone Mapping
|
||||||
|
|
||||||
|
Each slot declares a **primary mount bone** and an optional **secondary bone
|
||||||
|
list** for skinning. The runtime uses this to:
|
||||||
|
|
||||||
|
1. Parent static accessories to the mount bone.
|
||||||
|
2. Skin skinned attachments to the union of the slot's bones.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. glTF/GLB Format Requirements
|
||||||
|
|
||||||
|
All avatar assets (base bodies, body parts, attachments) are distributed as
|
||||||
|
**glTF 2.0** (`.gltf` + external buffers/textures) or **GLB** (`.glb`, binary
|
||||||
|
container). GLB is the preferred distribution format for runtime loading.
|
||||||
|
|
||||||
|
### 4.1 Required Conventions
|
||||||
|
|
||||||
|
| Concern | Requirement |
|
||||||
|
|--------------------|------------------------------------------------------------------------------|
|
||||||
|
| **Format** | glTF 2.0 (`.gltf` or `.glb`). GLB preferred for runtime. |
|
||||||
|
| **Coordinate system** | glTF standard: **+Y up, +Z forward, right-handed**. Godot converts to its own convention on import. |
|
||||||
|
| **Units** | Meters. Avatar height ~1.8 m at the `Root` bone. |
|
||||||
|
| **Skeleton** | Single skeleton per asset; bone names per §3.1. |
|
||||||
|
| **Skinning** | `skins` + `nodes` with `JOINTS_0` / `WEIGHTS_0` accessors. |
|
||||||
|
| **Pose** | T-pose at rest. |
|
||||||
|
| **Materials** | PBR metallic-roughness. Use `KHR_materials_unlit` for emissive/UI-only items. |
|
||||||
|
| **Textures** | Max 2048×2048; power-of-two preferred. Use `KHR_texture_basisu` (BasisU) for runtime. |
|
||||||
|
| **Animation** | Optional. If present, must not rename or remove canonical bones. |
|
||||||
|
|
||||||
|
### 4.2 Asset Metadata (extras)
|
||||||
|
|
||||||
|
Each asset **must** declare its slot and identity in the glTF `asset.extras`
|
||||||
|
object. This is the machine-readable contract consumed by the build pipeline and
|
||||||
|
runtime.
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"asset": {
|
||||||
|
"version": "2.0",
|
||||||
|
"extras": {
|
||||||
|
"agp": {
|
||||||
|
"type": "attachment", // "base_body" | "body_part" | "attachment"
|
||||||
|
"slot": "chest", // slot ID from §3.2
|
||||||
|
"id": "shirt_bandit_01", // globally unique asset id
|
||||||
|
"display_name": "Bandit Shirt",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"mount_bone": "Chest", // primary mount bone
|
||||||
|
"skin_bones": ["Chest", "Spine", "LeftUpperArm", "RightUpperArm"],
|
||||||
|
"exclusive": true, // matches slot exclusivity
|
||||||
|
"tags": ["cloth", "common"],
|
||||||
|
"author": "platform-team"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Validation rules for `extras.agp`:**
|
||||||
|
|
||||||
|
- `type` must be one of `base_body`, `body_part`, `attachment`.
|
||||||
|
- `slot` must be a known slot ID (§3.2).
|
||||||
|
- `mount_bone` must exist in the canonical skeleton.
|
||||||
|
- `skin_bones` must all exist in the canonical skeleton.
|
||||||
|
- `exclusive` must match the slot's declared exclusivity.
|
||||||
|
- `id` must be unique across the asset catalog.
|
||||||
|
|
||||||
|
### 4.3 Node Naming Convention
|
||||||
|
|
||||||
|
To make assets self-describing and mergeable, nodes follow a convention:
|
||||||
|
|
||||||
|
- **Base body:** the mesh node is named `Body`; the skeleton root is `Root`.
|
||||||
|
- **Body part:** the mesh node is named `<SlotID>_Mesh` (e.g., `head_Mesh`).
|
||||||
|
- **Attachment:** the mesh node is named `<SlotID>_<AssetID>` (e.g.,
|
||||||
|
`chest_shirt_bandit_01`).
|
||||||
|
- **Attachment points** (empty nodes) on the base body are named
|
||||||
|
`attach_<slot>` (e.g., `attach_head`, `attach_left_hand`).
|
||||||
|
|
||||||
|
### 4.4 Material & Texture Requirements
|
||||||
|
|
||||||
|
- All materials use the **metallic-roughness** PBR model.
|
||||||
|
- Textures are embedded in the GLB or referenced as external files; external
|
||||||
|
references must be relative paths within the asset package.
|
||||||
|
- **Color customization** is supported via a `baseColorFactor` tint applied at
|
||||||
|
runtime (multiplied over the albedo). Assets should use a neutral albedo
|
||||||
|
(white/gray) where tinting is intended.
|
||||||
|
- **Masked/transparent** materials must set the correct alpha mode
|
||||||
|
(`MASK` or `BLEND`) and `alphaCutoff`.
|
||||||
|
|
||||||
|
### 4.5 Skeleton Merging at Runtime
|
||||||
|
|
||||||
|
When an attachment is loaded, the runtime **merges** it onto the base body:
|
||||||
|
|
||||||
|
1. Load the base body (skeleton + base meshes).
|
||||||
|
2. Load each equipped attachment.
|
||||||
|
3. For each attachment, **re-target its skin** to the base body's skeleton by
|
||||||
|
bone name (no re-export needed).
|
||||||
|
4. Parent static accessories to their `mount_bone`.
|
||||||
|
5. Hide base meshes for any equipped body-part slots.
|
||||||
|
6. Apply material tints and merge into the render tree.
|
||||||
|
|
||||||
|
The merge is **name-based** — bone names must match exactly (§3.1). This is the
|
||||||
|
core reason for the strict naming contract.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Loadout Model (Network & Persistence)
|
||||||
|
|
||||||
|
The avatar's composition is serialized as a **loadout** — a compact, ordered
|
||||||
|
list of equipped asset IDs. This is what the server validates, persists, and
|
||||||
|
replicates.
|
||||||
|
|
||||||
|
### 5.1 Loadout JSON
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"base_body": "base_human_male_01",
|
||||||
|
"slots": {
|
||||||
|
"head": "head_cyber_01",
|
||||||
|
"hair": "hair_mohawk_01",
|
||||||
|
"face": null,
|
||||||
|
"torso": null,
|
||||||
|
"chest": "chest_bandit_01",
|
||||||
|
"back": "backpack_01",
|
||||||
|
"left_arm": null,
|
||||||
|
"right_arm": null,
|
||||||
|
"left_hand": "glove_leather_01",
|
||||||
|
"right_hand": "glove_leather_01",
|
||||||
|
"legs": null,
|
||||||
|
"pants": "pants_cargo_01",
|
||||||
|
"left_foot": "shoe_boot_01",
|
||||||
|
"right_foot": "shoe_boot_01",
|
||||||
|
"accessory": ["belt_01", "necklace_01"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Rules:**
|
||||||
|
|
||||||
|
- `base_body` is required and must reference a valid `base_body` asset.
|
||||||
|
- Each exclusive slot holds **at most one** asset id (or `null`).
|
||||||
|
- `accessory` is an array (0..N, bounded).
|
||||||
|
- The server validates that every id exists in the catalog and that slot
|
||||||
|
exclusivity is respected.
|
||||||
|
|
||||||
|
### 5.2 Network Replication
|
||||||
|
|
||||||
|
The loadout is replicated as part of the player's spawn/state data:
|
||||||
|
|
||||||
|
```
|
||||||
|
spawn_entity:
|
||||||
|
entity_id: uint32
|
||||||
|
type: player
|
||||||
|
transform: { x, y, rot }
|
||||||
|
props:
|
||||||
|
loadout_hash: uint32 # hash of the loadout for change detection
|
||||||
|
loadout: { ... } # full loadout on spawn / on change
|
||||||
|
```
|
||||||
|
|
||||||
|
- The **full loadout** is sent on spawn and whenever it changes (reliable).
|
||||||
|
- A **`loadout_hash`** lets clients skip re-resolving assets if unchanged.
|
||||||
|
- The server is the **authority** on the loadout; clients send *requests* to
|
||||||
|
change equipment, and the server validates and broadcasts the result.
|
||||||
|
|
||||||
|
### 5.3 Server-Side Validation
|
||||||
|
|
||||||
|
The server rejects a loadout if:
|
||||||
|
|
||||||
|
- `base_body` is missing or unknown.
|
||||||
|
- Any slot references an unknown asset id.
|
||||||
|
- An exclusive slot has more than one item.
|
||||||
|
- An asset's declared `slot` does not match the slot it is placed in.
|
||||||
|
- The accessory list exceeds the configured limit.
|
||||||
|
- The asset is not owned/unlocked by the player (for gated cosmetics).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Godot Runtime Architecture
|
||||||
|
|
||||||
|
### 6.1 Node Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
Avatar (Node3D)
|
||||||
|
├── Skeleton3D # canonical skeleton (from base body)
|
||||||
|
├── BodyMeshes # base body meshes (skinned to Skeleton3D)
|
||||||
|
│ ├── head_Mesh
|
||||||
|
│ ├── torso_Mesh
|
||||||
|
│ └── ...
|
||||||
|
├── Attachments # equipped attachments
|
||||||
|
│ ├── chest_shirt_bandit_01
|
||||||
|
│ ├── hair_mohawk_01
|
||||||
|
│ └── ...
|
||||||
|
└── AttachmentPoints # empty nodes for dynamic mounting
|
||||||
|
├── attach_head
|
||||||
|
├── attach_left_hand
|
||||||
|
└── ...
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.2 Core Classes
|
||||||
|
|
||||||
|
| Class | Responsibility |
|
||||||
|
|--------------------------|-------------------------------------------------------------|
|
||||||
|
| `AvatarSystem` | Loads base body + attachments, resolves loadouts, owns the avatar node. |
|
||||||
|
| `AvatarLoader` | glTF/GLB import, skeleton merge, skin re-targeting. |
|
||||||
|
| `AvatarSlotRegistry` | Static registry of slot definitions (id, bones, exclusivity).|
|
||||||
|
| `AvatarLoadout` | Serializable loadout model + validation. |
|
||||||
|
| `AvatarCatalog` | Asset catalog (id → asset metadata + resource path). |
|
||||||
|
| `AvatarVisualizer` | Builds the render tree from a loadout. |
|
||||||
|
|
||||||
|
### 6.3 Loading Pipeline
|
||||||
|
|
||||||
|
```
|
||||||
|
loadout JSON
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
AvatarCatalog.resolve(loadout) # id → asset metadata + GLB path
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
AvatarLoader.load_base_body() # import GLB, extract skeleton + base meshes
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
for each equipped slot:
|
||||||
|
AvatarLoader.load_attachment()# import GLB, re-target skin to skeleton
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
AvatarVisualizer.assemble() # parent, hide replaced parts, apply tints
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
Avatar node ready for render
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6.4 Performance Considerations
|
||||||
|
|
||||||
|
- **Shared skeleton:** All meshes skin to one `Skeleton3D`, so skinning is
|
||||||
|
computed once per frame.
|
||||||
|
- **Material batching:** Attachments that share a material are merged into a
|
||||||
|
single mesh where possible (static accessories).
|
||||||
|
- **LOD:** Optional LOD variants per asset; the runtime selects based on
|
||||||
|
distance.
|
||||||
|
- **Caching:** Resolved loadouts and imported GLB resources are cached by
|
||||||
|
`AvatarCatalog` to avoid re-importing on respawn or re-join.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Asset Pipeline & Validation
|
||||||
|
|
||||||
|
### 7.1 Build-Time Validation
|
||||||
|
|
||||||
|
A build-time checker validates every asset before it enters the catalog:
|
||||||
|
|
||||||
|
1. **Format check:** valid glTF 2.0 / GLB.
|
||||||
|
2. **Metadata check:** `extras.agp` present and valid (§4.2).
|
||||||
|
3. **Skeleton check:** all `skin_bones` exist in the canonical skeleton.
|
||||||
|
4. **Slot check:** `slot` is known and `exclusive` matches.
|
||||||
|
5. **Mesh check:** meshes are skinned (or static accessories parented to a bone).
|
||||||
|
6. **Texture check:** sizes, formats, and alpha modes are valid.
|
||||||
|
7. **Uniqueness check:** `id` is unique.
|
||||||
|
|
||||||
|
### 7.2 Catalog Manifest
|
||||||
|
|
||||||
|
The catalog is a JSON manifest mapping asset ids to metadata and resource paths:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"assets": {
|
||||||
|
"base_human_male_01": {
|
||||||
|
"type": "base_body",
|
||||||
|
"path": "res://assets/avatars/base_human_male_01.glb",
|
||||||
|
"version": "1.0.0"
|
||||||
|
},
|
||||||
|
"chest_bandit_01": {
|
||||||
|
"type": "attachment",
|
||||||
|
"slot": "chest",
|
||||||
|
"path": "res://assets/avatars/chest_bandit_01.glb",
|
||||||
|
"version": "1.0.0"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Security & Anti-Cheat
|
||||||
|
|
||||||
|
- **Server-authoritative loadouts:** Clients only *request* equipment changes;
|
||||||
|
the server validates against the catalog and the player's owned items.
|
||||||
|
- **No client asset injection:** Clients cannot send arbitrary GLB data; they
|
||||||
|
reference catalog ids only.
|
||||||
|
- **Rate limiting:** Equipment-change requests are rate-limited per peer.
|
||||||
|
- **Hash verification:** The `loadout_hash` lets the server detect tampered or
|
||||||
|
stale loadout state.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Open Questions / Future Work
|
||||||
|
|
||||||
|
- **Morph targets / blendshapes:** Support for facial expressions and body
|
||||||
|
morphs (e.g., height, muscle) via glTF morph targets.
|
||||||
|
- **Physics attachments:** Cloth simulation for capes/skirts (Godot cloth).
|
||||||
|
- **Dynamic tinting UI:** A color-picker that writes `baseColorFactor` tints.
|
||||||
|
- **VRM compatibility:** Consider supporting VRM (a glTF extension for
|
||||||
|
humanoid avatars) as an import source.
|
||||||
|
- **Attachment LOD streaming:** Streaming high-res attachments on demand.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Summary
|
||||||
|
|
||||||
|
| Concern | Decision |
|
||||||
|
|------------------------|-------------------------------------------------------|
|
||||||
|
| Format | glTF 2.0 / GLB (GLB preferred for runtime) |
|
||||||
|
| Skeleton | Single canonical humanoid skeleton, name-based merge |
|
||||||
|
| Slots | Fixed slot registry; exclusive body parts + layered attachments |
|
||||||
|
| Loadout | Compact JSON; server-authoritative and validated |
|
||||||
|
| Replication | Full loadout on spawn/change + `loadout_hash` |
|
||||||
|
| Runtime | Godot `Skeleton3D` + name-based skin re-targeting |
|
||||||
|
| Validation | Build-time asset checker + runtime catalog validation |
|
||||||
@@ -1,683 +0,0 @@
|
|||||||
# World Chunk Data Format & Persistence Architecture
|
|
||||||
|
|
||||||
> **Status:** Proposed
|
|
||||||
> **Owner:** Platform team
|
|
||||||
> **Related:** Server authority + client prediction, avatar component system, content moderation & age verification, backend auth (JWT)
|
|
||||||
|
|
||||||
## 1. Overview
|
|
||||||
|
|
||||||
This document specifies the **world chunk data format** and **persistence
|
|
||||||
schema** for **agp-game**, a Godot-based multiplayer game built by autonomous
|
|
||||||
agents. It defines:
|
|
||||||
|
|
||||||
- The **chunk data format** — both a human-readable **JSON** form (for
|
|
||||||
authoring, tooling, and debugging) and a compact **binary** form (for
|
|
||||||
runtime loading and network transfer).
|
|
||||||
- The **SQLite schema** that persists world state on the server, including
|
|
||||||
chunk metadata, chunk payloads, and the world/room hierarchy.
|
|
||||||
- The **read/write API** that the server and client use to load, save, and
|
|
||||||
stream chunks.
|
|
||||||
|
|
||||||
The goal is a **single, versioned chunk contract** that is shared across the
|
|
||||||
Godot client, the Python backend, and the persistence layer. A chunk is the
|
|
||||||
atomic unit of world geometry, gameplay objects, and persistence — the server
|
|
||||||
loads and saves whole chunks, and clients stream the chunks around them.
|
|
||||||
|
|
||||||
### 1.1 Design Goals
|
|
||||||
|
|
||||||
- **Versioned & forward-compatible:** Every chunk carries a format version so
|
|
||||||
the loader can migrate old data and reject unknown future data gracefully.
|
|
||||||
- **Dual representation:** JSON for authoring/debugging, binary for
|
|
||||||
runtime/network. Both encode the *same* logical model so there is exactly one
|
|
||||||
source of truth for the schema.
|
|
||||||
- **Server-authoritative persistence:** The server is the only writer to the
|
|
||||||
SQLite store. Clients submit *edits* (deltas) and the server applies and
|
|
||||||
persists them, mirroring the server-authoritative networking model.
|
|
||||||
- **Chunk-granular persistence:** Persistence happens at chunk granularity so
|
|
||||||
the server can save a dirty chunk without rewriting the whole world, and can
|
|
||||||
stream only the chunks a client needs.
|
|
||||||
- **Deterministic & testable:** The binary format is byte-stable for a given
|
|
||||||
logical payload, enabling checksums, caching, and reproducible tests.
|
|
||||||
- **Moderation-aware:** Chunks reference content (assets, scripts) that carry
|
|
||||||
content flags (see `content_moderation.md`); the persistence layer stores the
|
|
||||||
moderation state alongside chunk data so enforcement can happen at load time.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. World Model & Chunking
|
|
||||||
|
|
||||||
### 2.1 World Hierarchy
|
|
||||||
|
|
||||||
The world is organized as a hierarchy that matches the moderation content graph:
|
|
||||||
|
|
||||||
```
|
|
||||||
World
|
|
||||||
└── Room
|
|
||||||
└── Chunk (the atomic persistence & streaming unit)
|
|
||||||
├── Tiles / Terrain
|
|
||||||
├── Static objects (props, geometry)
|
|
||||||
├── Dynamic objects (spawnable entities, interactables)
|
|
||||||
└── Metadata (flags, version, timestamps)
|
|
||||||
```
|
|
||||||
|
|
||||||
- A **World** is the top-level container (a named, versioned space).
|
|
||||||
- A **Room** is a bounded region within a world (a level, a building, a zone).
|
|
||||||
- A **Chunk** is a fixed-size, axis-aligned region of a room. It is the unit of
|
|
||||||
load, save, and stream.
|
|
||||||
|
|
||||||
### 2.2 Chunk Dimensions
|
|
||||||
|
|
||||||
Chunks are **fixed-size, axis-aligned boxes** in world space. The default
|
|
||||||
dimensions are a configuration constant so they can be tuned per world:
|
|
||||||
|
|
||||||
| Constant | Default | Description |
|
|
||||||
|---------------------|---------|----------------------------------------------------|
|
|
||||||
| `CHUNK_SIZE_X` | 32 | Chunk width in world units (tiles). |
|
|
||||||
| `CHUNK_SIZE_Y` | 32 | Chunk height in world units (tiles). |
|
|
||||||
| `CHUNK_SIZE_Z` | 1 | Chunk depth (1 = flat 2D world; >1 = volumetric). |
|
|
||||||
| `CHUNK_ORIGIN` | `(0,0)` | World-space origin of chunk coordinate `(0,0)`. |
|
|
||||||
|
|
||||||
A chunk is identified by its **integer chunk coordinate** `(cx, cy, cz)`,
|
|
||||||
computed from a world-space position `p` as:
|
|
||||||
|
|
||||||
```
|
|
||||||
cx = floor((p.x - CHUNK_ORIGIN.x) / CHUNK_SIZE_X)
|
|
||||||
cy = floor((p.y - CHUNK_ORIGIN.y) / CHUNK_SIZE_Y)
|
|
||||||
cz = floor((p.z - CHUNK_ORIGIN.z) / CHUNK_SIZE_Z)
|
|
||||||
```
|
|
||||||
|
|
||||||
The chunk's world-space bounding box is therefore:
|
|
||||||
|
|
||||||
```
|
|
||||||
min = (CHUNK_ORIGIN.x + cx*CHUNK_SIZE_X, ...)
|
|
||||||
max = min + (CHUNK_SIZE_X, CHUNK_SIZE_Y, CHUNK_SIZE_Z)
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2.3 Chunk Coordinate Key
|
|
||||||
|
|
||||||
For storage and lookup, a chunk is addressed by a **canonical string key**:
|
|
||||||
|
|
||||||
```
|
|
||||||
"{world_id}:{room_id}:{cx}:{cy}:{cz}"
|
|
||||||
```
|
|
||||||
|
|
||||||
Example: `"w_0001:r_0001:12:-4:0"`. This key is the primary lookup key in the
|
|
||||||
SQLite `chunks` table and is used in the read/write API.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Chunk Data Format
|
|
||||||
|
|
||||||
### 3.1 Logical Model
|
|
||||||
|
|
||||||
Both the JSON and binary encodings serialize the same logical chunk model. The
|
|
||||||
logical model is defined here once; the two encodings are projections of it.
|
|
||||||
|
|
||||||
```
|
|
||||||
Chunk
|
|
||||||
├── version : uint (format version, currently 1)
|
|
||||||
├── key : string (canonical chunk key, see §2.3)
|
|
||||||
├── world_id : string
|
|
||||||
├── room_id : string
|
|
||||||
├── coord : { cx, cy, cz } (int)
|
|
||||||
├── bounds : { min_x, min_y, min_z, max_x, max_y, max_z } (float)
|
|
||||||
├── tiles : TileLayer[] (terrain / tile data)
|
|
||||||
├── objects : WorldObject[] (static + dynamic objects)
|
|
||||||
├── flags : ContentFlags (moderation flags, see §3.5)
|
|
||||||
├── meta : { created_at, updated_at, revision, author }
|
|
||||||
└── checksum : string (SHA-256 of canonical payload)
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3.2 Tile Layer
|
|
||||||
|
|
||||||
A chunk's terrain is stored as one or more **tile layers**. Each layer is a
|
|
||||||
dense grid of tile IDs of size `CHUNK_SIZE_X × CHUNK_SIZE_Y` (× `CHUNK_SIZE_Z`
|
|
||||||
for volumetric worlds).
|
|
||||||
|
|
||||||
```
|
|
||||||
TileLayer
|
|
||||||
├── name : string (e.g. "ground", "walls", "decor")
|
|
||||||
├── layer_id : uint
|
|
||||||
├── width : uint (= CHUNK_SIZE_X)
|
|
||||||
├── height : uint (= CHUNK_SIZE_Y)
|
|
||||||
├── depth : uint (= CHUNK_SIZE_Z)
|
|
||||||
├── tiles : uint[] (flattened tile IDs, row-major, length = w*h*d)
|
|
||||||
└── palette : TileDef[] (tile ID → tile definition, shared per layer)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Tile encoding:** A tile ID is a `uint16` index into the layer's `palette`.
|
|
||||||
`0` is reserved for "empty/air". The palette maps each tile ID to a `TileDef`:
|
|
||||||
|
|
||||||
```
|
|
||||||
TileDef
|
|
||||||
├── id : uint16
|
|
||||||
├── asset_id : string (reference to a tile/terrain asset)
|
|
||||||
├── solid : bool (collision)
|
|
||||||
├── flags : uint32 (bitmask: walkable, climbable, liquid, ...)
|
|
||||||
└── variants : uint8 (number of visual variants for this tile)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rationale for palette + indices:** Storing dense `uint16` indices rather than
|
|
||||||
full asset strings per cell keeps the binary payload small (2 bytes/cell) while
|
|
||||||
the palette (shared across the layer) carries the rich per-tile metadata.
|
|
||||||
|
|
||||||
### 3.3 World Objects
|
|
||||||
|
|
||||||
Objects are the non-tile entities in a chunk: static props, interactables,
|
|
||||||
spawn points, and dynamic-object templates.
|
|
||||||
|
|
||||||
```
|
|
||||||
WorldObject
|
|
||||||
├── id : string (stable object id, unique within the chunk)
|
|
||||||
├── type : string (e.g. "prop", "interactable", "spawn_point", "door")
|
|
||||||
├── asset_id : string (reference to the object's mesh/glTF asset)
|
|
||||||
├── transform : { pos: [x,y,z], rot: [x,y,z,w], scale: [x,y,z] }
|
|
||||||
├── properties : object (type-specific key/value map, JSON-typed)
|
|
||||||
├── dynamic : bool (true if this is a spawnable dynamic entity)
|
|
||||||
└── flags : ContentFlags (per-object moderation flags)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Dynamic objects** are stored as *templates* in the chunk. Their *runtime
|
|
||||||
state* (position, health, inventory) is **not** stored in the chunk; it lives
|
|
||||||
in the runtime entity store and is persisted separately (see §4.4). This keeps
|
|
||||||
chunks stable and diffable — a chunk only changes when its *static* content
|
|
||||||
changes.
|
|
||||||
|
|
||||||
### 3.4 Content Flags
|
|
||||||
|
|
||||||
Each chunk (and each object) carries a **content flag set** that is the
|
|
||||||
aggregate of the moderation flags of the assets and scripts it references. This
|
|
||||||
mirrors the aggregation rules in `content_moderation.md`:
|
|
||||||
|
|
||||||
```
|
|
||||||
ContentFlags
|
|
||||||
├── categories : { violence, sexual, language, substance, gambling, horror, hate, custom }
|
|
||||||
│ (each → severity: none|mild|moderate|severe)
|
|
||||||
├── mature : bool (true if any contributing item is mature)
|
|
||||||
└── sources : string[] (flag ids that contributed, for audit)
|
|
||||||
```
|
|
||||||
|
|
||||||
The server computes and persists these flags at write time so that enforcement
|
|
||||||
(age gating, content filtering) can happen at load time without re-scanning
|
|
||||||
assets.
|
|
||||||
|
|
||||||
### 3.5 JSON Encoding
|
|
||||||
|
|
||||||
The JSON encoding is the **authoring and debugging** representation. It is
|
|
||||||
human-readable, diffable, and used by tools and tests. It is the canonical
|
|
||||||
logical model serialized directly.
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"version": 1,
|
|
||||||
"key": "w_0001:r_0001:12:-4:0",
|
|
||||||
"world_id": "w_0001",
|
|
||||||
"room_id": "r_0001",
|
|
||||||
"coord": { "cx": 12, "cy": -4, "cz": 0 },
|
|
||||||
"bounds": { "min_x": 384.0, "min_y": -128.0, "min_z": 0.0,
|
|
||||||
"max_x": 416.0, "max_y": -96.0, "max_z": 1.0 },
|
|
||||||
"tiles": [
|
|
||||||
{
|
|
||||||
"name": "ground",
|
|
||||||
"layer_id": 0,
|
|
||||||
"width": 32,
|
|
||||||
"height": 32,
|
|
||||||
"depth": 1,
|
|
||||||
"palette": [
|
|
||||||
{ "id": 0, "asset_id": "", "solid": false, "flags": 0, "variants": 1 },
|
|
||||||
{ "id": 1, "asset_id": "tiles/grass", "solid": false, "flags": 1, "variants": 4 }
|
|
||||||
],
|
|
||||||
"tiles": [0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1]
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"objects": [
|
|
||||||
{
|
|
||||||
"id": "obj_0001",
|
|
||||||
"type": "interactable",
|
|
||||||
"asset_id": "props/chest",
|
|
||||||
"transform": { "pos": [400.0, -110.0, 0.0],
|
|
||||||
"rot": [0.0, 0.0, 0.0, 1.0],
|
|
||||||
"scale": [1.0, 1.0, 1.0] },
|
|
||||||
"properties": { "loot_table": "chest_basic", "locked": false },
|
|
||||||
"dynamic": false,
|
|
||||||
"flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] }
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] },
|
|
||||||
"meta": { "created_at": "2026-08-17T00:00:00Z",
|
|
||||||
"updated_at": "2026-08-17T00:00:00Z",
|
|
||||||
"revision": 3,
|
|
||||||
"author": "agent-bot" },
|
|
||||||
"checksum": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3.6 Binary Encoding
|
|
||||||
|
|
||||||
The binary encoding is the **runtime and network** representation. It is compact
|
|
||||||
and byte-stable. It uses a simple, explicitly-specified layout (no external
|
|
||||||
serialization library) so it can be produced and consumed identically by the
|
|
||||||
Godot client (GDScript) and the Python backend.
|
|
||||||
|
|
||||||
#### 3.6.1 Header
|
|
||||||
|
|
||||||
| Offset | Size | Field | Type / Encoding |
|
|
||||||
|--------|------|--------------|------------------------------------------|
|
|
||||||
| 0 | 4 | magic | ASCII `"AGPC"` (0x41 0x47 0x50 0x43) |
|
|
||||||
| 4 | 2 | version | `uint16` LE (currently `1`) |
|
|
||||||
| 6 | 2 | header_len | `uint16` LE (bytes of header after this) |
|
|
||||||
| 8 | 4 | flags | `uint32` LE (bitmask, reserved) |
|
|
||||||
| 12 | 4 | payload_len | `uint32` LE (bytes of payload body) |
|
|
||||||
| 16 | 32 | checksum | SHA-256 of payload body (hex, 32 bytes) |
|
|
||||||
|
|
||||||
Total header size: **48 bytes**.
|
|
||||||
|
|
||||||
#### 3.6.2 Payload Body
|
|
||||||
|
|
||||||
The payload body is a sequence of typed sections. Each section has a small
|
|
||||||
header:
|
|
||||||
|
|
||||||
| Offset | Size | Field | Type / Encoding |
|
|
||||||
|--------|------|------------|----------------------------|
|
|
||||||
| 0 | 1 | section_id | `uint8` (see table below) |
|
|
||||||
| 1 | 4 | section_len| `uint32` LE (bytes of section data) |
|
|
||||||
| 5 | ... | data | section-specific |
|
|
||||||
|
|
||||||
Section IDs:
|
|
||||||
|
|
||||||
| section_id | Section | Data layout |
|
|
||||||
|------------|----------------|-------------------------------------------------------------------|
|
|
||||||
| `0x01` | metadata | world_id, room_id (length-prefixed strings), coord (3×int32), bounds (6×float32) |
|
|
||||||
| `0x02` | tile layer | name, layer_id, dims (3×uint16), palette, dense tile indices |
|
|
||||||
| `0x03` | object | one `WorldObject` (repeated; one section per object) |
|
|
||||||
| `0x04` | content flags | category severities (8×uint8), mature (uint8), sources (string list) |
|
|
||||||
| `0x05` | meta | created_at, updated_at (int64 unix), revision (uint32), author (string) |
|
|
||||||
|
|
||||||
**String encoding:** length-prefixed UTF-8 — `uint16` LE byte length followed by
|
|
||||||
the UTF-8 bytes.
|
|
||||||
|
|
||||||
**Tile layer data layout (section `0x02`):**
|
|
||||||
|
|
||||||
```
|
|
||||||
name : string
|
|
||||||
layer_id : uint16 LE
|
|
||||||
width : uint16 LE
|
|
||||||
height : uint16 LE
|
|
||||||
depth : uint16 LE
|
|
||||||
palette_count : uint16 LE
|
|
||||||
palette : palette_count × TileDef
|
|
||||||
tile_count : uint32 LE
|
|
||||||
tiles : tile_count × uint16 LE (dense, row-major)
|
|
||||||
```
|
|
||||||
|
|
||||||
```
|
|
||||||
TileDef:
|
|
||||||
id : uint16 LE
|
|
||||||
asset_id : string
|
|
||||||
solid : uint8
|
|
||||||
flags : uint32 LE
|
|
||||||
variants : uint8
|
|
||||||
```
|
|
||||||
|
|
||||||
**Object data layout (section `0x03`):**
|
|
||||||
|
|
||||||
```
|
|
||||||
id : string
|
|
||||||
type : string
|
|
||||||
asset_id : string
|
|
||||||
pos : 3 × float32
|
|
||||||
rot : 4 × float32
|
|
||||||
scale : 3 × float32
|
|
||||||
dynamic : uint8
|
|
||||||
properties : JSON string (length-prefixed)
|
|
||||||
flags : ContentFlags (see 0x04 layout)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Content flags layout (section `0x04`):**
|
|
||||||
|
|
||||||
```
|
|
||||||
violence, sexual, language, substance, gambling, horror, hate, custom
|
|
||||||
: 8 × uint8 (severity enum: 0=none,1=mild,2=moderate,3=severe)
|
|
||||||
mature : uint8
|
|
||||||
source_count: uint16 LE
|
|
||||||
sources : source_count × string
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3.6.3 Byte-Stability
|
|
||||||
|
|
||||||
For a given logical payload, the binary encoding is **deterministic**:
|
|
||||||
|
|
||||||
- All integers are little-endian.
|
|
||||||
- All strings are UTF-8, length-prefixed, and never null-terminated.
|
|
||||||
- Maps (e.g. `properties`) are serialized as a **canonical JSON string** with
|
|
||||||
keys sorted lexicographically.
|
|
||||||
- The `checksum` is the SHA-256 of the payload body bytes.
|
|
||||||
|
|
||||||
This guarantees that two encoders produce identical bytes, enabling content
|
|
||||||
addressing, caching, and reproducible tests.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. SQLite Persistence Schema
|
|
||||||
|
|
||||||
The server persists world state in a **SQLite** database. SQLite is chosen for
|
|
||||||
its zero-configuration, single-file, transactional, and embedded nature — ideal
|
|
||||||
for a server that owns the world store and needs ACID guarantees without a
|
|
||||||
separate database process.
|
|
||||||
|
|
||||||
### 4.1 Database Layout
|
|
||||||
|
|
||||||
The database file is `world.db` (path configurable). It contains the following
|
|
||||||
tables:
|
|
||||||
|
|
||||||
```
|
|
||||||
worlds
|
|
||||||
rooms
|
|
||||||
chunks
|
|
||||||
chunk_objects
|
|
||||||
content_flags
|
|
||||||
revisions
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.2 `worlds`
|
|
||||||
|
|
||||||
Top-level world containers.
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE TABLE worlds (
|
|
||||||
id TEXT PRIMARY KEY, -- e.g. 'w_0001'
|
|
||||||
name TEXT NOT NULL,
|
|
||||||
description TEXT,
|
|
||||||
chunk_size_x INTEGER NOT NULL DEFAULT 32,
|
|
||||||
chunk_size_y INTEGER NOT NULL DEFAULT 32,
|
|
||||||
chunk_size_z INTEGER NOT NULL DEFAULT 1,
|
|
||||||
origin_x REAL NOT NULL DEFAULT 0,
|
|
||||||
origin_y REAL NOT NULL DEFAULT 0,
|
|
||||||
origin_z REAL NOT NULL DEFAULT 0,
|
|
||||||
created_at TEXT NOT NULL, -- ISO-8601 UTC
|
|
||||||
updated_at TEXT NOT NULL
|
|
||||||
);
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.3 `rooms`
|
|
||||||
|
|
||||||
Rooms within a world.
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE TABLE rooms (
|
|
||||||
id TEXT PRIMARY KEY, -- e.g. 'r_0001'
|
|
||||||
world_id TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE,
|
|
||||||
name TEXT NOT NULL,
|
|
||||||
bounds_min_x REAL NOT NULL,
|
|
||||||
bounds_min_y REAL NOT NULL,
|
|
||||||
bounds_min_z REAL NOT NULL DEFAULT 0,
|
|
||||||
bounds_max_x REAL NOT NULL,
|
|
||||||
bounds_max_y REAL NOT NULL,
|
|
||||||
bounds_max_z REAL NOT NULL DEFAULT 1,
|
|
||||||
created_at TEXT NOT NULL,
|
|
||||||
updated_at TEXT NOT NULL,
|
|
||||||
UNIQUE (world_id, name)
|
|
||||||
);
|
|
||||||
CREATE INDEX idx_rooms_world ON rooms(world_id);
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.4 `chunks`
|
|
||||||
|
|
||||||
The core persistence table. One row per chunk. The **payload** is stored as a
|
|
||||||
BLOB in the **binary format** (§3.6); the JSON form is derived on demand for
|
|
||||||
authoring/debugging and is not stored (to avoid dual-write drift).
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE TABLE chunks (
|
|
||||||
key TEXT PRIMARY KEY, -- canonical key 'world:room:cx:cy:cz'
|
|
||||||
world_id TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE,
|
|
||||||
room_id TEXT NOT NULL REFERENCES rooms(id) ON DELETE CASCADE,
|
|
||||||
cx INTEGER NOT NULL,
|
|
||||||
cy INTEGER NOT NULL,
|
|
||||||
cz INTEGER NOT NULL DEFAULT 0,
|
|
||||||
format_version INTEGER NOT NULL DEFAULT 1,
|
|
||||||
payload BLOB NOT NULL, -- binary chunk encoding (§3.6)
|
|
||||||
payload_len INTEGER NOT NULL,
|
|
||||||
checksum TEXT NOT NULL, -- SHA-256 of payload
|
|
||||||
revision INTEGER NOT NULL DEFAULT 0,
|
|
||||||
dirty INTEGER NOT NULL DEFAULT 0, -- 1 = needs flush to disk
|
|
||||||
created_at TEXT NOT NULL,
|
|
||||||
updated_at TEXT NOT NULL,
|
|
||||||
UNIQUE (world_id, room_id, cx, cy, cz)
|
|
||||||
);
|
|
||||||
CREATE INDEX idx_chunks_world_room ON chunks(world_id, room_id);
|
|
||||||
CREATE INDEX idx_chunks_coord ON chunks(cx, cy, cz);
|
|
||||||
```
|
|
||||||
|
|
||||||
**Design notes:**
|
|
||||||
|
|
||||||
- The `payload` BLOB is the single source of truth for chunk content. The
|
|
||||||
`checksum` guards against corruption and enables content addressing.
|
|
||||||
- `revision` increments on every write; it is used for optimistic concurrency
|
|
||||||
and change detection (see §5.4).
|
|
||||||
- `dirty` marks chunks that have been modified in memory but not yet flushed,
|
|
||||||
enabling a write-behind persistence strategy (see §5.5).
|
|
||||||
|
|
||||||
### 4.5 `chunk_objects`
|
|
||||||
|
|
||||||
A denormalized, queryable index of the objects inside each chunk. This is
|
|
||||||
**derived** from the chunk payload and kept in sync by the write path. It exists
|
|
||||||
so the server can answer queries like "which chunks contain a door?" without
|
|
||||||
parsing every payload BLOB.
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE TABLE chunk_objects (
|
|
||||||
id TEXT NOT NULL, -- object id (unique within chunk)
|
|
||||||
chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE,
|
|
||||||
type TEXT NOT NULL, -- 'prop', 'interactable', ...
|
|
||||||
asset_id TEXT,
|
|
||||||
dynamic INTEGER NOT NULL DEFAULT 0,
|
|
||||||
pos_x REAL NOT NULL,
|
|
||||||
pos_y REAL NOT NULL,
|
|
||||||
pos_z REAL NOT NULL DEFAULT 0,
|
|
||||||
PRIMARY KEY (chunk_key, id)
|
|
||||||
);
|
|
||||||
CREATE INDEX idx_chunk_objects_type ON chunk_objects(type);
|
|
||||||
CREATE INDEX idx_chunk_objects_asset ON chunk_objects(asset_id);
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.6 `content_flags`
|
|
||||||
|
|
||||||
Persists the aggregated moderation flags per chunk (and optionally per object).
|
|
||||||
This lets the server enforce age/content gates at load time without parsing
|
|
||||||
payloads.
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE TABLE content_flags (
|
|
||||||
chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE,
|
|
||||||
object_id TEXT, -- NULL = chunk-level flag
|
|
||||||
violence INTEGER NOT NULL DEFAULT 0,
|
|
||||||
sexual INTEGER NOT NULL DEFAULT 0,
|
|
||||||
language INTEGER NOT NULL DEFAULT 0,
|
|
||||||
substance INTEGER NOT NULL DEFAULT 0,
|
|
||||||
gambling INTEGER NOT NULL DEFAULT 0,
|
|
||||||
horror INTEGER NOT NULL DEFAULT 0,
|
|
||||||
hate INTEGER NOT NULL DEFAULT 0,
|
|
||||||
custom INTEGER NOT NULL DEFAULT 0,
|
|
||||||
mature INTEGER NOT NULL DEFAULT 0,
|
|
||||||
sources TEXT NOT NULL DEFAULT '[]', -- JSON array of flag ids
|
|
||||||
PRIMARY KEY (chunk_key, object_id)
|
|
||||||
);
|
|
||||||
CREATE INDEX idx_content_flags_mature ON content_flags(mature);
|
|
||||||
```
|
|
||||||
|
|
||||||
Severity is stored as `0=none, 1=mild, 2=moderate, 3=severe`.
|
|
||||||
|
|
||||||
### 4.7 `revisions`
|
|
||||||
|
|
||||||
An append-only audit log of chunk writes. Enables rollback, auditing, and
|
|
||||||
replication of changes.
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE TABLE revisions (
|
|
||||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
|
||||||
chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE,
|
|
||||||
revision INTEGER NOT NULL,
|
|
||||||
actor TEXT NOT NULL, -- user / agent id
|
|
||||||
action TEXT NOT NULL, -- 'create' | 'update' | 'delete'
|
|
||||||
checksum TEXT NOT NULL,
|
|
||||||
created_at TEXT NOT NULL
|
|
||||||
);
|
|
||||||
CREATE INDEX idx_revisions_chunk ON revisions(chunk_key);
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.8 Schema Versioning & Migrations
|
|
||||||
|
|
||||||
The database carries a schema version in a `meta` table:
|
|
||||||
|
|
||||||
```sql
|
|
||||||
CREATE TABLE meta (
|
|
||||||
key TEXT PRIMARY KEY,
|
|
||||||
value TEXT NOT NULL
|
|
||||||
);
|
|
||||||
-- seed: INSERT INTO meta(key, value) VALUES ('schema_version', '1');
|
|
||||||
```
|
|
||||||
|
|
||||||
Migrations are applied in order at startup. Each migration is a numbered SQL
|
|
||||||
script; the server tracks the applied version in `meta` and applies any pending
|
|
||||||
migrations transactionally. This keeps the schema forward-compatible as the
|
|
||||||
chunk format evolves.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Read/Write API
|
|
||||||
|
|
||||||
The API is exposed as a **Python service layer** on the backend (the same
|
|
||||||
process that owns the SQLite store) and consumed by the Godot client over the
|
|
||||||
network transport (see `networking.md`). The service layer is the **only**
|
|
||||||
writer to the database; clients never touch SQLite directly.
|
|
||||||
|
|
||||||
### 5.1 Service Interface (Python)
|
|
||||||
|
|
||||||
```python
|
|
||||||
class WorldStore:
|
|
||||||
"""Owns the SQLite connection and chunk read/write operations."""
|
|
||||||
|
|
||||||
def get_chunk(self, key: str) -> Chunk | None: ...
|
|
||||||
def get_chunk_binary(self, key: str) -> bytes | None: ...
|
|
||||||
def put_chunk(self, chunk: Chunk, actor: str) -> Chunk: ...
|
|
||||||
def delete_chunk(self, key: str, actor: str) -> None: ...
|
|
||||||
def list_chunks(self, world_id: str, room_id: str | None = None,
|
|
||||||
bounds: Bounds | None = None) -> list[str]: ...
|
|
||||||
def get_chunk_flags(self, key: str) -> ContentFlags: ...
|
|
||||||
def flush_dirty(self) -> int: ...
|
|
||||||
def migrate(self) -> None: ...
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5.2 Encoding Helpers
|
|
||||||
|
|
||||||
The service layer provides symmetric encode/decode functions shared by the
|
|
||||||
client and server:
|
|
||||||
|
|
||||||
```python
|
|
||||||
def encode_binary(chunk: Chunk) -> bytes: ...
|
|
||||||
def decode_binary(data: bytes) -> Chunk: ...
|
|
||||||
def encode_json(chunk: Chunk) -> str: ...
|
|
||||||
def decode_json(text: str) -> Chunk: ...
|
|
||||||
def checksum(payload: bytes) -> str: ...
|
|
||||||
```
|
|
||||||
|
|
||||||
These are the single implementation of the format spec (§3), so the Godot
|
|
||||||
client and Python backend never drift.
|
|
||||||
|
|
||||||
### 5.3 Network RPCs
|
|
||||||
|
|
||||||
The chunk API is exposed to the client via RPCs (matching the networking
|
|
||||||
architecture). All chunk traffic is **reliable** (chunks are discrete,
|
|
||||||
must-arrive data).
|
|
||||||
|
|
||||||
| RPC | Direction | Payload | Notes |
|
|
||||||
|----------------------------|-----------|-------------------------------------------|-------|
|
|
||||||
| `request_chunk` | C → S | `{ key }` | Client asks for a chunk |
|
|
||||||
| `chunk_data` | S → C | `{ key, binary_payload, checksum }` | Server streams a chunk |
|
|
||||||
| `request_chunk_edit` | C → S | `{ key, base_revision, delta }` | Client submits an edit |
|
|
||||||
| `chunk_edit_result` | S → C | `{ key, success, new_revision, error? }` | Acknowledges an edit |
|
|
||||||
| `chunk_invalidated` | S → C | `{ key, new_revision }` | Server pushes a change |
|
|
||||||
| `request_chunk_flags` | C → S | `{ key }` | Age/content gate check |
|
|
||||||
|
|
||||||
### 5.4 Edit & Concurrency Model
|
|
||||||
|
|
||||||
- **Read:** Client requests a chunk; server returns the binary payload and its
|
|
||||||
`revision`.
|
|
||||||
- **Edit:** Client submits a **delta** against a known `base_revision`. The
|
|
||||||
server applies the delta, increments `revision`, persists, and returns the new
|
|
||||||
revision.
|
|
||||||
- **Optimistic concurrency:** If the chunk's current revision differs from the
|
|
||||||
client's `base_revision`, the server rejects the edit with a conflict error
|
|
||||||
and the client re-fetches and re-applies. This prevents lost updates when two
|
|
||||||
clients edit the same chunk.
|
|
||||||
- **Deltas:** A delta is a JSON patch (RFC 6902 style) applied to the chunk's
|
|
||||||
logical model. The server validates the patched chunk (schema, bounds,
|
|
||||||
moderation) before persisting.
|
|
||||||
|
|
||||||
### 5.5 Write-Behind Persistence
|
|
||||||
|
|
||||||
To amortize disk I/O, the server keeps a **dirty set** of chunks modified in
|
|
||||||
memory and flushes them to SQLite in batches:
|
|
||||||
|
|
||||||
1. `put_chunk` updates the in-memory chunk and marks it `dirty`.
|
|
||||||
2. A background flusher periodically calls `flush_dirty()`, which writes all
|
|
||||||
dirty chunks in a single transaction and clears the `dirty` flag.
|
|
||||||
3. On graceful shutdown, `flush_dirty()` is called to ensure durability.
|
|
||||||
|
|
||||||
This keeps the hot path (in-memory chunk access) fast while guaranteeing
|
|
||||||
durability at shutdown and on a configurable interval.
|
|
||||||
|
|
||||||
### 5.6 Streaming & Caching
|
|
||||||
|
|
||||||
- The server streams only the chunks a client needs (those within the client's
|
|
||||||
view radius), using `list_chunks` with a bounds query.
|
|
||||||
- Chunks are cached in memory (LRU) keyed by `key`; the cache is invalidated on
|
|
||||||
write.
|
|
||||||
- The binary payload is content-addressed by `checksum`, so unchanged chunks can
|
|
||||||
be served from cache without re-reading SQLite.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. Format Versioning & Migration
|
|
||||||
|
|
||||||
- The chunk format `version` field (currently `1`) is checked on decode. A
|
|
||||||
decoder that encounters a higher version than it supports **rejects** the
|
|
||||||
chunk with a clear error rather than misreading it.
|
|
||||||
- The SQLite `schema_version` in `meta` drives database migrations (§4.8).
|
|
||||||
- Chunk payloads are migrated **lazily**: when a chunk is read and its
|
|
||||||
`format_version` is older than the current one, the server migrates it to the
|
|
||||||
current version in memory and marks it dirty so it is rewritten on the next
|
|
||||||
flush. This avoids a costly full-world migration at startup.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. Security & Integrity
|
|
||||||
|
|
||||||
- **Server-authoritative writes:** Only the `WorldStore` writes to SQLite;
|
|
||||||
clients submit deltas that are validated before application.
|
|
||||||
- **Checksum verification:** Every chunk payload is checksummed (SHA-256) on
|
|
||||||
write and verified on read. Corruption is detected and the chunk is re-fetched
|
|
||||||
or flagged.
|
|
||||||
- **Bounds validation:** Chunk coordinates and object positions are validated
|
|
||||||
against the world's chunk dimensions and room bounds before persistence.
|
|
||||||
- **Moderation enforcement:** `content_flags` are checked at load time; mature
|
|
||||||
chunks are gated behind the age-verification flow (see `content_moderation.md`).
|
|
||||||
- **Input validation:** All string fields are length-limited and validated
|
|
||||||
against allowed character sets to prevent injection and oversized payloads.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. Open Questions / Future Work
|
|
||||||
|
|
||||||
- **Volumetric worlds:** The schema and format already support `CHUNK_SIZE_Z >
|
|
||||||
1`, but the tooling and streaming for 3D chunks are not yet specified.
|
|
||||||
- **Chunk compression:** For very large worlds, the binary payload could be
|
|
||||||
compressed (e.g. zstd) before storage/transfer. The header reserves a `flags`
|
|
||||||
bitmask for a compression flag.
|
|
||||||
- **Multi-server sharding:** If a single SQLite file becomes a bottleneck, the
|
|
||||||
`chunks` table could be sharded by `world_id` across multiple databases. The
|
|
||||||
key design already supports this (world_id is a leading key component).
|
|
||||||
- **Delta compression for streaming:** Instead of sending full chunks, the
|
|
||||||
server could send only the delta between revisions to reduce bandwidth for
|
|
||||||
frequently-updated chunks.
|
|
||||||
Reference in New Issue
Block a user