Compare commits

...

30 Commits

Author SHA1 Message Date
architect-bot 218e7e24af WIP checkpoint: Address review feedback: Research and design GDScript sandboxing approach 2026-08-18 02:06:21 +00:00
architect-bot d293ef2a08 WIP checkpoint: Address review feedback: Research and design GDScript sandboxing approach 2026-08-18 01:48:59 +00:00
architect-bot 4da8687f2c Document GDScript sandboxing findings and recommendation
Verified all three options from the task against Godot 4.6.2 directly
(godot --headless), rather than trusting the probe scripts' own printed
output from earlier sessions:
- ScriptServer doesn't exist as a class; no disable_os/disable_file_access/
  disable_network project settings exist either. Nothing to configure.
- SubViewport isolates rendering/input only, not script capabilities — a
  script inside one has identical OS/FileAccess/network access.
- Expression is not safe by default (can still reach unsafe methods through
  a base instance); blocklisting dangerous substrings is trivially bypassed.

Recommendation: don't sandbox GDScript in-process (nothing to sandbox
with) — use a narrow non-Turing-complete content DSL for the common case,
and OS-level process isolation only if genuinely general scripting turns
out to be required.
2026-08-17 23:43:48 +00:00
architect-bot 119cd9400a WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 21:33:57 +00:00
architect-bot a576c25124 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 21:30:27 +00:00
architect-bot 0b00b01502 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 21:30:03 +00:00
architect-bot 8d50de075d WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 21:29:39 +00:00
architect-bot 2df195734b WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 21:26:31 +00:00
architect-bot 20be724115 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 21:00:02 +00:00
architect-bot bd8f311a96 Recovered orphaned WIP before switching to a new task 2026-08-17 20:52:21 +00:00
architect-bot d422eff912 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:41:48 +00:00
architect-bot ae6ccc3e5f WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:39:05 +00:00
architect-bot fed16f1f1d WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:37:49 +00:00
architect-bot 9450fb86d0 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:36:34 +00:00
architect-bot a863bb2ab4 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:35:29 +00:00
architect-bot a1846f34b5 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:33:56 +00:00
architect-bot 62447258b2 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:33:28 +00:00
architect-bot c931670e9d WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:25:30 +00:00
architect-bot 68b783909d WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:25:15 +00:00
architect-bot 0b5034759c WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:24:53 +00:00
architect-bot 19a63bda11 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:24:28 +00:00
architect-bot a3cdb18a5c WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:18:59 +00:00
architect-bot a3fc36db19 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:18:40 +00:00
architect-bot 42914aec85 WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:17:55 +00:00
architect-bot 8b17cc3ddd WIP checkpoint: Research and design GDScript sandboxing approach 2026-08-17 20:17:28 +00:00
architect-bot 9e2e0ed921 Recovered orphaned WIP before switching to a new task 2026-08-17 20:09:27 +00:00
reviewer-bot a632f2d8a6 Merge pull request 'Address review feedback: Design world chunk data format and persistence schema' (#11) from agent/c92c939b819c into main 2026-08-17 19:02:34 +00:00
architect-bot 0b973b73ab WIP checkpoint: Address review feedback: Design world chunk data format and persistence schema 2026-08-17 18:29:14 +00:00
daddy f0d10fc4ae chore: remove test artifact from git-identity verification 2026-08-17 17:59:01 +00:00
reviewer-bot 1295b6a389 Merge pull request 'Identity test' (#10) from agent/identitytest111 into main 2026-08-17 17:58:44 +00:00
9 changed files with 1113 additions and 1 deletions
+159
View File
@@ -0,0 +1,159 @@
# GDScript Sandboxing for User-Written Scripts
> **Status:** Proposed
> **Owner:** Platform team
> **Related:** Content moderation & age verification (scripts are one of the
> three moderated content types), avatar component system
## 1. Overview
This document evaluates how to safely run **user-written scripts** as part of
UGC (user-generated content) in **agp-game**, and recommends an approach.
**Bottom line: Godot 4 has no built-in security sandbox for GDScript, and
none of the three options named in the original task hold up under testing.**
A script running anywhere inside the Godot process — however it's loaded,
parented, or wrapped — runs with full engine privileges: arbitrary process
execution, full filesystem read/write, and raw network access. Sandboxing
has to happen *outside* the scripting language, not through an in-engine
restriction mechanism, because no such mechanism exists.
This was verified directly against Godot 4.6.2 (`godot --headless`, evidence
in `sandbox_probe/` on this branch) rather than assumed from documentation,
some of which (see §2.1) describes APIs that don't actually exist in this
version.
## 2. Options Evaluated
### 2.1 Godot's built-in "ScriptServer" restrictions — does not exist
There is no `ScriptServer` class in Godot 4.6.2:
```
ClassDB.class_exists("ScriptServer") → false
```
There's also no project setting to disable engine subsystems for scripts —
none of `application/run/disable_os`, `disable_file_access`, or
`disable_network` exist as settings, and `ProjectSettings.get_property_list()`
returns nothing matching "sandbox", "restrict", or "security". Any plan that
assumes a flag like this exists needs to be dropped; it was likely conflated
with sandboxing features from a different engine/scripting ecosystem, or
with **export template stripping** (which controls what ships in a build,
not what a script can call at runtime).
Directly probed capabilities of an ordinary GDScript running as a normal
node script, no special setup:
| Capability | Available to any script? |
|---|---|
| `OS.execute(...)` — run arbitrary processes | ✅ Yes |
| `FileAccess.open(...)` — read/write any path the OS user can reach | ✅ Yes (read `/etc/hostname`, write `/tmp/...` both succeeded) |
| `TCPServer` / `HTTPClient` / `HTTPRequest` / `PacketPeerUDP` / `WebSocketPeer` | ✅ All instantiate and work |
| `ClassDB.instantiate(...)` — construct any engine class by name | ✅ Yes |
| Loading and running a second script from source text at runtime (`GDScript.new()` + `.reload()`) | ✅ Yes |
**Verdict: rejected.** There is nothing to configure here — the restriction
surface this option assumed doesn't exist in the engine.
### 2.2 Isolated SubViewport with a limited API — doesn't isolate scripts
`SubViewport` is a **rendering and input isolation** primitive (it gives a
scene subtree its own render target and can gate input propagation) — it is
not a security or scripting boundary. A script attached to any node inside a
`SubViewport` still runs in the same process, on the same GDScript VM, with
the exact same access to `OS`, `FileAccess`, networking classes, and
`ClassDB` as a script anywhere else in the tree. Nothing about being
"inside" a SubViewport changes what API calls are reachable from a script.
**Verdict: rejected.** This solves a rendering/UI-composition problem, not
the security problem the task is asking about. Worth keeping in mind for
other purposes (e.g. isolated preview rendering of user avatars), but it
provides zero script sandboxing.
### 2.3 Custom GDScript interpreter with an allowlist — right idea, wrong layer for "custom interpreter"
Two ways to read this option, tested separately:
**(a) Write/embed a separate GDScript parser+interpreter.** Not
practical — this means re-implementing (and then indefinitely maintaining,
in lockstep with every Godot release) a full language interpreter outside
the engine. Far more engineering than the problem justifies.
**(b) Use Godot's built-in `Expression` class as a restricted subset.**
`Expression` *looks* like a natural allowlist mechanism — it only evaluates
a single expression, no control flow, no arbitrary statements. But it isn't
safe by default: it can still reach through to unsafe engine methods if you
aren't careful about what object you hand it as a base instance:
```gdscript
var expr = Expression.new()
expr.parse("get_environment('HOME')", ["OS"])
expr.execute([], OS) # → "/home/daddy" (successfully called OS.get_environment)
```
So `Expression` is a viable building block only if the host **never passes
a base instance (or arguments) that exposes anything sensitive** — i.e. the
allowlisting work still has to happen at the call site, `Expression` itself
enforces none of it.
**Static text pattern-matching (blocklisting dangerous substrings like
`"OS.execute"` before parsing) was also probed and is not a real
defense** — it's trivially defeated by string concatenation, `call()`,
`Callable`, or any other indirection, and produces false confidence.
**Verdict: partially viable, but only as a foundation, not a solution on
its own.** A hand-built, narrow, data-driven DSL (not full GDScript, not
raw `Expression`) with an explicit allowlist of safe operations and *no
ability to reference engine singletons or arbitrary objects* is the only
version of "custom interpreter" that's actually safe. This is real
engineering effort, not a config flag.
## 3. Recommendation
**Don't run untrusted GDScript in-process at all.** Given none of the three
in-engine options provide a real boundary, the two approaches actually worth
building are, in order of recommended priority:
1. **Primary: a narrow, non-Turing-complete content DSL**, not general
scripting. Most UGC scripting needs in a game like this (custom
interactions, simple triggers, item behaviors) don't need a general
programming language — they need a small set of declarative
verbs/conditions (e.g. `on_interact: give_item("torch")`,
`on_enter: play_sound("chime")`). Expressed as data (JSON/Resource, not
code) and interpreted by trusted, hand-written GDScript on the engine
side. This has *zero* attack surface by construction — there's no
interpreter to escape because there's no general execution, just a
fixed, reviewed set of verbs. This is almost certainly sufficient for
the actual UGC use cases in this game and should be scoped against real
feature requirements before assuming more is needed.
2. **If genuinely general scripting becomes a hard requirement**, isolate it
at the OS level, not the engine level: run untrusted scripts in a
**separate process** (a stripped-down Godot instance or a dedicated
interpreter) under OS sandboxing (a restricted user account, a
container, or `firejail`/`bwrap`/seccomp), communicating with the main
game over a narrow IPC/RPC boundary that only exposes intentionally-safe
operations. This is the only approach that provides a real security
boundary for arbitrary code, because it doesn't depend on the scripting
language cooperating — the OS enforces it. Substantially more
infrastructure than option 1; only worth it if option 1's DSL genuinely
can't express what's needed.
**Do not** attempt in-process GDScript sandboxing via SubViewport,
ScriptServer, or ad hoc blocklisting — all three were directly tested and
none provide a real boundary (§2).
## 4. Open Questions
- What do actual planned UGC scripts need to *do*? (Trigger conditions,
simple item/state effects, dialogue branching?) This determines whether
the DSL in §3.1 is sufficient or whether §3.2's process isolation is
needed. Needs a concrete feature list before implementation starts.
- If §3.2 is needed later, does it run per-user-session or shared/pooled?
Affects resource cost and isolation guarantees.
- Interaction with the content moderation pipeline (`content_moderation.md`):
scripts as a moderated content type still need a moderation flag path
regardless of which execution model is chosen — a DSL makes static
analysis of *what a script can possibly do* far easier than it would be
for general code, which is an additional point in its favor.
+737
View File
@@ -0,0 +1,737 @@
# World Chunk Data Format & Persistence Architecture
> **Status:** Proposed
> **Owner:** Platform team
> **Related:** Server authority + client prediction, avatar component system, content moderation & age verification, backend auth (JWT)
## 1. Overview
This document specifies the **world chunk data format** and **persistence
schema** for **agp-game**, a Godot-based multiplayer game built by autonomous
agents. It defines:
- The **chunk data format** — both a human-readable **JSON** form (for
authoring, tooling, and debugging) and a compact **binary** form (for
runtime loading and network transfer).
- The **SQLite schema** that persists world state on the server, including
chunk metadata, chunk payloads, and the world/room hierarchy.
- The **read/write API** that the server and client use to load, save, and
stream chunks.
The goal is a **single, versioned chunk contract** that is shared across the
Godot client, the Python backend, and the persistence layer. A chunk is the
atomic unit of world geometry, gameplay objects, and persistence — the server
loads and saves whole chunks, and clients stream the chunks around them.
### 1.1 Design Goals
- **Versioned & forward-compatible:** Every chunk carries a format version so
the loader can migrate old data and reject unknown future data gracefully.
- **Dual representation:** JSON for authoring/debugging, binary for
runtime/network. Both encode the *same* logical model so there is exactly one
source of truth for the schema.
- **Server-authoritative persistence:** The server is the only writer to the
SQLite store. Clients submit *edits* (deltas) and the server applies and
persists them, mirroring the server-authoritative networking model.
- **Chunk-granular persistence:** Persistence happens at chunk granularity so
the server can save a dirty chunk without rewriting the whole world, and can
stream only the chunks a client needs.
- **Deterministic & testable:** The binary format is byte-stable for a given
logical payload, enabling checksums, caching, and reproducible tests.
- **Moderation-aware:** Chunks reference content (assets, scripts) that carry
content flags (see `content_moderation.md`); the persistence layer stores the
moderation state alongside chunk data so enforcement can happen at load time.
---
## 2. World Model & Chunking
### 2.1 World Hierarchy
The world is organized as a hierarchy that matches the moderation content graph:
```
World
└── Room
└── Chunk (the atomic persistence & streaming unit)
├── Tiles / Terrain
├── Static objects (props, geometry)
├── Dynamic objects (spawnable entities, interactables)
└── Metadata (flags, version, timestamps)
```
- A **World** is the top-level container (a named, versioned space).
- A **Room** is a bounded region within a world (a level, a building, a zone).
- A **Chunk** is a fixed-size, axis-aligned region of a room. It is the unit of
load, save, and stream.
### 2.2 Chunk Dimensions
Chunks are **fixed-size, axis-aligned boxes** in world space. The default
dimensions are a configuration constant so they can be tuned per world:
| Constant | Default | Description |
|---------------------|---------|----------------------------------------------------|
| `CHUNK_SIZE_X` | 32 | Chunk width in world units (tiles). |
| `CHUNK_SIZE_Y` | 32 | Chunk height in world units (tiles). |
| `CHUNK_SIZE_Z` | 1 | Chunk depth (1 = flat 2D world; >1 = volumetric). |
| `CHUNK_ORIGIN` | `(0,0)` | World-space origin of chunk coordinate `(0,0)`. |
A chunk is identified by its **integer chunk coordinate** `(cx, cy, cz)`,
computed from a world-space position `p` as:
```
cx = floor((p.x - CHUNK_ORIGIN.x) / CHUNK_SIZE_X)
cy = floor((p.y - CHUNK_ORIGIN.y) / CHUNK_SIZE_Y)
cz = floor((p.z - CHUNK_ORIGIN.z) / CHUNK_SIZE_Z)
```
The chunk's world-space bounding box is therefore:
```
min = (CHUNK_ORIGIN.x + cx*CHUNK_SIZE_X, ...)
max = min + (CHUNK_SIZE_X, CHUNK_SIZE_Y, CHUNK_SIZE_Z)
```
### 2.3 Chunk Coordinate Key
For storage and lookup, a chunk is addressed by a **canonical string key**:
```
"{world_id}:{room_id}:{cx}:{cy}:{cz}"
```
Example: `"w_0001:r_0001:12:-4:0"`. This key is the primary lookup key in the
SQLite `chunks` table and is used in the read/write API.
---
## 3. Chunk Data Format
### 3.1 Logical Model
Both the JSON and binary encodings serialize the same logical chunk model. The
logical model is defined here once; the two encodings are projections of it.
```
Chunk
├── version : uint (format version, currently 1)
├── key : string (canonical chunk key, see §2.3)
├── world_id : string
├── room_id : string
├── coord : { cx, cy, cz } (int)
├── bounds : { min_x, min_y, min_z, max_x, max_y, max_z } (float)
├── tiles : TileLayer[] (terrain / tile data)
├── objects : WorldObject[] (static + dynamic objects)
├── flags : ContentFlags (moderation flags, see §3.5)
├── meta : { created_at, updated_at, revision, author }
└── checksum : string (SHA-256 of canonical payload)
```
### 3.2 Tile Layer
A chunk's terrain is stored as one or more **tile layers**. Each layer is a
dense grid of tile IDs of size `CHUNK_SIZE_X × CHUNK_SIZE_Y` (× `CHUNK_SIZE_Z`
for volumetric worlds).
```
TileLayer
├── name : string (e.g. "ground", "walls", "decor")
├── layer_id : uint
├── width : uint (= CHUNK_SIZE_X)
├── height : uint (= CHUNK_SIZE_Y)
├── depth : uint (= CHUNK_SIZE_Z)
├── tiles : uint[] (flattened tile IDs, row-major, length = w*h*d)
└── palette : TileDef[] (tile ID → tile definition, shared per layer)
```
**Tile encoding:** A tile ID is a `uint16` index into the layer's `palette`.
`0` is reserved for "empty/air". The palette maps each tile ID to a `TileDef`:
```
TileDef
├── id : uint16
├── asset_id : string (reference to a tile/terrain asset)
├── solid : bool (collision)
├── flags : uint32 (bitmask: walkable, climbable, liquid, ...)
└── variants : uint8 (number of visual variants for this tile)
```
**Rationale for palette + indices:** Storing dense `uint16` indices rather than
full asset strings per cell keeps the binary payload small (2 bytes/cell) while
the palette (shared across the layer) carries the rich per-tile metadata.
### 3.3 World Objects
Objects are the non-tile entities in a chunk: static props, interactables,
spawn points, and dynamic-object templates.
```
WorldObject
├── id : string (stable object id, unique within the chunk)
├── type : string (e.g. "prop", "interactable", "spawn_point", "door")
├── asset_id : string (reference to the object's mesh/glTF asset)
├── transform : { pos: [x,y,z], rot: [x,y,z,w], scale: [x,y,z] }
├── properties : object (type-specific key/value map, JSON-typed)
├── dynamic : bool (true if this is a spawnable dynamic entity)
└── flags : ContentFlags (per-object moderation flags)
```
**Dynamic objects** are stored as *templates* in the chunk. Their *runtime
state* (position, health, inventory) is **not** stored in the chunk; it lives
in the runtime entity store and is persisted separately (see §4.4). This keeps
chunks stable and diffable — a chunk only changes when its *static* content
changes.
### 3.4 Content Flags
Each chunk (and each object) carries a **content flag set** that is the
aggregate of the moderation flags of the assets and scripts it references. This
mirrors the aggregation rules in `content_moderation.md`:
```
ContentFlags
├── categories : { violence, sexual, language, substance, gambling, horror, hate, custom }
│ (each → severity: none|mild|moderate|severe)
├── mature : bool (true if any contributing item is mature)
└── sources : string[] (flag ids that contributed, for audit)
```
The server computes and persists these flags at write time so that enforcement
(age gating, content filtering) can happen at load time without re-scanning
assets.
### 3.5 JSON Encoding
The JSON encoding is the **authoring and debugging** representation. It is
human-readable, diffable, and used by tools and tests. It is the canonical
logical model serialized directly.
```json
{
"version": 1,
"key": "w_0001:r_0001:12:-4:0",
"world_id": "w_0001",
"room_id": "r_0001",
"coord": { "cx": 12, "cy": -4, "cz": 0 },
"bounds": { "min_x": 384.0, "min_y": -128.0, "min_z": 0.0,
"max_x": 416.0, "max_y": -96.0, "max_z": 1.0 },
"tiles": [
{
"name": "ground",
"layer_id": 0,
"width": 32,
"height": 32,
"depth": 1,
"palette": [
{ "id": 0, "asset_id": "", "solid": false, "flags": 0, "variants": 1 },
{ "id": 1, "asset_id": "tiles/grass", "solid": false, "flags": 1, "variants": 4 }
],
"tiles": [0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1]
}
],
"objects": [
{
"id": "obj_0001",
"type": "interactable",
"asset_id": "props/chest",
"transform": { "pos": [400.0, -110.0, 0.0],
"rot": [0.0, 0.0, 0.0, 1.0],
"scale": [1.0, 1.0, 1.0] },
"properties": { "loot_table": "chest_basic", "locked": false },
"dynamic": false,
"flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] }
}
],
"flags": { "categories": { "violence": "none" }, "mature": false, "sources": [] },
"meta": { "created_at": "2026-08-17T00:00:00Z",
"updated_at": "2026-08-17T00:00:00Z",
"revision": 3,
"author": "agent-bot" },
"checksum": "9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08"
}
```
> **Note on the example checksum:** The value shown above is illustrative only.
> In practice the `checksum` field is the SHA-256 of the canonical binary
> payload body (see §3.6.3), not of the JSON text. The value here is a
> placeholder for documentation purposes.
### 3.6 Binary Encoding
The binary encoding is the **runtime and network** representation. It is compact
and byte-stable. It uses a simple, explicitly-specified layout (no external
serialization library) so it can be produced and consumed identically by the
Godot client (GDScript) and the Python backend.
#### 3.6.1 Header
| Offset | Size | Field | Type / Encoding |
|--------|------|--------------|------------------------------------------|
| 0 | 4 | magic | ASCII `"AGPC"` (0x41 0x47 0x50 0x43) |
| 4 | 2 | version | `uint16` LE (currently `1`) |
| 6 | 2 | header_len | `uint16` LE (bytes of header after this) |
| 8 | 4 | flags | `uint32` LE (bitmask, reserved) |
| 12 | 4 | payload_len | `uint32` LE (bytes of payload body) |
| 16 | 32 | checksum | SHA-256 of payload body (hex, 32 bytes) |
Total header size: **48 bytes**.
#### 3.6.2 Payload Body
The payload body is a sequence of typed sections. Each section has a small
header:
| Offset | Size | Field | Type / Encoding |
|--------|------|------------|----------------------------|
| 0 | 1 | section_id | `uint8` (see table below) |
| 1 | 4 | section_len| `uint32` LE (bytes of section data) |
| 5 | ... | data | section-specific |
Section IDs:
| section_id | Section | Data layout |
|------------|----------------|-------------------------------------------------------------------|
| `0x01` | metadata | world_id, room_id (length-prefixed strings), coord (3×int32), bounds (6×float32) |
| `0x02` | tile layer | name, layer_id, dims (3×uint16), palette, dense tile indices |
| `0x03` | object | one `WorldObject` (repeated; one section per object) |
| `0x04` | content flags | category severities (8×uint8), mature (uint8), sources (string list) |
| `0x05` | meta | created_at, updated_at (int64 unix), revision (uint32), author (string) |
**String encoding:** length-prefixed UTF-8 — `uint16` LE byte length followed by
the UTF-8 bytes.
**Tile layer data layout (section `0x02`):**
```
name : string
layer_id : uint16 LE
width : uint16 LE
height : uint16 LE
depth : uint16 LE
palette_count : uint16 LE
palette : palette_count × TileDef
tile_count : uint32 LE
tiles : tile_count × uint16 LE (dense, row-major)
```
```
TileDef:
id : uint16 LE
asset_id : string
solid : uint8
flags : uint32 LE
variants : uint8
```
**Object data layout (section `0x03`):**
```
id : string
type : string
asset_id : string
pos : 3 × float32
rot : 4 × float32
scale : 3 × float32
dynamic : uint8
properties : JSON string (length-prefixed)
flags : ContentFlags (see 0x04 layout)
```
**Content flags layout (section `0x04`):**
```
violence, sexual, language, substance, gambling, horror, hate, custom
: 8 × uint8 (severity enum: 0=none,1=mild,2=moderate,3=severe)
mature : uint8
source_count: uint16 LE
sources : source_count × string
```
#### 3.6.3 Byte-Stability
For a given logical payload, the binary encoding is **deterministic**:
- All integers are little-endian.
- All strings are UTF-8, length-prefixed, and never null-terminated.
- Maps (e.g. `properties`) are serialized as a **canonical JSON string** with
keys sorted lexicographically.
- The `checksum` is the SHA-256 of the payload body bytes.
This guarantees that two encoders produce identical bytes, enabling content
addressing, caching, and reproducible tests.
---
## 4. SQLite Persistence Schema
The server persists world state in a **SQLite** database. SQLite is chosen for
its zero-configuration, single-file, transactional, and embedded nature — ideal
for a server that owns the world store and needs ACID guarantees without a
separate database process.
### 4.1 Database Layout
The database file is `world.db` (path configurable). It contains the following
tables:
```
worlds
rooms
chunks
chunk_objects
content_flags
revisions
```
### 4.2 `worlds`
Top-level world containers.
```sql
CREATE TABLE worlds (
id TEXT PRIMARY KEY, -- e.g. 'w_0001'
name TEXT NOT NULL,
description TEXT,
chunk_size_x INTEGER NOT NULL DEFAULT 32,
chunk_size_y INTEGER NOT NULL DEFAULT 32,
chunk_size_z INTEGER NOT NULL DEFAULT 1,
origin_x REAL NOT NULL DEFAULT 0,
origin_y REAL NOT NULL DEFAULT 0,
origin_z REAL NOT NULL DEFAULT 0,
created_at TEXT NOT NULL, -- ISO-8601 UTC
updated_at TEXT NOT NULL
);
```
### 4.3 `rooms`
Rooms within a world.
```sql
CREATE TABLE rooms (
id TEXT PRIMARY KEY, -- e.g. 'r_0001'
world_id TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE,
name TEXT NOT NULL,
bounds_min_x REAL NOT NULL,
bounds_min_y REAL NOT NULL,
bounds_min_z REAL NOT NULL DEFAULT 0,
bounds_max_x REAL NOT NULL,
bounds_max_y REAL NOT NULL,
bounds_max_z REAL NOT NULL DEFAULT 1,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
UNIQUE (world_id, name)
);
CREATE INDEX idx_rooms_world ON rooms(world_id);
```
### 4.4 `chunks`
The core persistence table. One row per chunk. The **payload** is stored as a
BLOB in the **binary format** (§3.6); the JSON form is derived on demand for
authoring/debugging and is not stored (to avoid dual-write drift).
```sql
CREATE TABLE chunks (
key TEXT PRIMARY KEY, -- canonical key 'world:room:cx:cy:cz'
world_id TEXT NOT NULL REFERENCES worlds(id) ON DELETE CASCADE,
room_id TEXT NOT NULL REFERENCES rooms(id) ON DELETE CASCADE,
cx INTEGER NOT NULL,
cy INTEGER NOT NULL,
cz INTEGER NOT NULL DEFAULT 0,
format_version INTEGER NOT NULL DEFAULT 1,
payload BLOB NOT NULL, -- binary chunk encoding (§3.6)
payload_len INTEGER NOT NULL,
checksum TEXT NOT NULL, -- SHA-256 of payload
revision INTEGER NOT NULL DEFAULT 0,
dirty INTEGER NOT NULL DEFAULT 0, -- 1 = needs flush to disk
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
UNIQUE (world_id, room_id, cx, cy, cz)
);
CREATE INDEX idx_chunks_world_room ON chunks(world_id, room_id);
CREATE INDEX idx_chunks_coord ON chunks(cx, cy, cz);
```
**Design notes:**
- The `payload` BLOB is the single source of truth for chunk content. The
`checksum` guards against corruption and enables content addressing.
- `revision` increments on every write; it is used for optimistic concurrency
and change detection (see §5.4).
- `dirty` marks chunks that have been modified in memory but not yet flushed,
enabling a write-behind persistence strategy (see §5.5).
### 4.5 `chunk_objects`
A denormalized, queryable index of the objects inside each chunk. This is
**derived** from the chunk payload and kept in sync by the write path. It exists
so the server can answer queries like "which chunks contain a door?" without
parsing every payload BLOB.
```sql
CREATE TABLE chunk_objects (
id TEXT NOT NULL, -- object id (unique within chunk)
chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE,
type TEXT NOT NULL, -- 'prop', 'interactable', ...
asset_id TEXT,
dynamic INTEGER NOT NULL DEFAULT 0,
pos_x REAL NOT NULL,
pos_y REAL NOT NULL,
pos_z REAL NOT NULL DEFAULT 0,
PRIMARY KEY (chunk_key, id)
);
CREATE INDEX idx_chunk_objects_type ON chunk_objects(type);
CREATE INDEX idx_chunk_objects_asset ON chunk_objects(asset_id);
```
**Sync mechanism:** `chunk_objects` is maintained **explicitly by the write
path** — it is not a SQL trigger. Whenever `put_chunk` (or `delete_chunk`)
commits a chunk, the service layer performs a **delete-and-reinsert** of that
chunk's object rows in the same transaction: it deletes all rows for
`chunk_key`, then inserts fresh rows parsed from the new payload. This keeps the
index transactionally consistent with the payload and avoids the complexity and
fragility of SQL triggers over a binary BLOB. Because the index is derived, it
can always be rebuilt from `chunks.payload` if it ever drifts (a `rebuild_index`
utility is provided for this).
### 4.6 `content_flags`
Persists the aggregated moderation flags per chunk (and optionally per object).
This lets the server enforce age/content gates at load time without parsing
payloads.
```sql
CREATE TABLE content_flags (
chunk_key TEXT NOT NULL REFERENCES chunks(key) ON DELETE CASCADE,
object_id TEXT NOT NULL DEFAULT '', -- '' = chunk-level flag (sentinel)
violence INTEGER NOT NULL DEFAULT 0,
sexual INTEGER NOT NULL DEFAULT 0,
language INTEGER NOT NULL DEFAULT 0,
substance INTEGER NOT NULL DEFAULT 0,
gambling INTEGER NOT NULL DEFAULT 0,
horror INTEGER NOT NULL DEFAULT 0,
hate INTEGER NOT NULL DEFAULT 0,
custom INTEGER NOT NULL DEFAULT 0,
mature INTEGER NOT NULL DEFAULT 0,
sources TEXT NOT NULL DEFAULT '[]', -- JSON array of flag ids
PRIMARY KEY (chunk_key, object_id)
);
CREATE INDEX idx_content_flags_mature ON content_flags(mature);
```
Severity is stored as `0=none, 1=mild, 2=moderate, 3=severe`.
**Chunk-level vs. object-level flags:** Chunk-level flags use the **sentinel
value `''` (empty string)** in `object_id` rather than `NULL`. This is
deliberate: in SQLite, `NULL` values in a composite primary key are **not**
unique-constrained, so two rows with the same `chunk_key` and `object_id = NULL`
could be inserted accidentally. Using `''` as the sentinel means the composite
`PRIMARY KEY (chunk_key, object_id)` enforces that there is **exactly one**
chunk-level flag row per chunk. Object-level flags use the object's actual `id`
(which is never empty). The application layer treats `object_id = ''` as
"chunk-level" and any non-empty value as "object-level".
### 4.7 `revisions`
An append-only audit log of chunk writes. Enables rollback, auditing, and
replication of changes.
```sql
CREATE TABLE revisions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
chunk_key TEXT NOT NULL, -- NOT a FK: survives chunk deletion
revision INTEGER NOT NULL,
actor TEXT NOT NULL, -- user / agent id
action TEXT NOT NULL, -- 'create' | 'update' | 'delete'
checksum TEXT NOT NULL, -- SHA-256 of payload
payload BLOB, -- full binary payload at this revision
created_at TEXT NOT NULL
);
CREATE INDEX idx_revisions_chunk ON revisions(chunk_key);
```
**Rollback support:** The `payload` BLOB stores the **full binary chunk payload**
at each revision. This is what makes rollback possible: to restore a chunk to a
previous revision, the server reads the `payload` of the target revision and
writes it back to `chunks.payload` (incrementing `revision` and appending a new
`revisions` row). Without the payload, only the checksum would be available,
which is insufficient to reconstruct the chunk.
**Audit trail preservation:** `chunk_key` is **not** a foreign key to `chunks`.
This means deleting a chunk does **not** cascade-delete its revision history —
the audit trail is preserved even after a chunk is removed. (The `chunk_key`
string is still stored so the history can be correlated to a chunk that existed
at that time.) If a chunk is later recreated with the same key, its history is
still intact.
### 4.8 Schema Versioning & Migrations
The database carries a schema version in a `meta` table:
```sql
CREATE TABLE meta (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
-- seed: INSERT INTO meta(key, value) VALUES ('schema_version', '1');
```
Migrations are applied in order at startup. Each migration is a numbered SQL
script; the server tracks the applied version in `meta` and applies any pending
migrations transactionally. This keeps the schema forward-compatible as the
chunk format evolves.
---
## 5. Read/Write API
The API is exposed as a **Python service layer** on the backend (the same
process that owns the SQLite store) and consumed by the Godot client over the
network transport (see `networking.md`). The service layer is the **only**
writer to the database; clients never touch SQLite directly.
### 5.1 Service Interface (Python)
```python
class WorldStore:
"""Owns the SQLite connection and chunk read/write operations."""
def get_chunk(self, key: str) -> Chunk | None: ...
def get_chunk_binary(self, key: str) -> bytes | None: ...
def put_chunk(self, chunk: Chunk, actor: str) -> Chunk: ...
def delete_chunk(self, key: str, actor: str) -> None: ...
def list_chunks(self, world_id: str, room_id: str | None = None,
bounds: Bounds | None = None) -> list[str]: ...
def get_chunk_flags(self, key: str) -> ContentFlags: ...
def flush_dirty(self) -> int: ...
def rollback_chunk(self, key: str, revision: int, actor: str) -> Chunk: ...
def rebuild_index(self, chunk_key: str | None = None) -> int: ...
def migrate(self) -> None: ...
```
### 5.2 Encoding Helpers
The service layer provides symmetric encode/decode functions shared by the
client and server:
```python
def encode_binary(chunk: Chunk) -> bytes: ...
def decode_binary(data: bytes) -> Chunk: ...
def encode_json(chunk: Chunk) -> str: ...
def decode_json(text: str) -> Chunk: ...
def checksum(payload: bytes) -> str: ...
```
These are the single implementation of the format spec (§3), so the Godot
client and Python backend never drift.
### 5.3 Network RPCs
The chunk API is exposed to the client via RPCs (matching the networking
architecture). All chunk traffic is **reliable** (chunks are discrete,
must-arrive data).
| RPC | Direction | Payload | Notes |
|----------------------------|-----------|-------------------------------------------|-------|
| `request_chunk` | C → S | `{ key }` | Client asks for a chunk |
| `chunk_data` | S → C | `{ key, binary_payload, checksum }` | Server streams a chunk |
| `request_chunk_edit` | C → S | `{ key, base_revision, delta }` | Client submits an edit |
| `chunk_edit_result` | S → C | `{ key, success, new_revision, error? }` | Acknowledges an edit |
| `chunk_invalidated` | S → C | `{ key, new_revision }` | Server pushes a change |
| `request_chunk_flags` | C → S | `{ key }` | Age/content gate check |
### 5.4 Edit & Concurrency Model
- **Read:** Client requests a chunk; server returns the binary payload and its
`revision`.
- **Edit:** Client submits a **delta** against a known `base_revision`. The
server applies the delta, increments `revision`, persists, and returns the new
revision.
- **Optimistic concurrency:** If the chunk's current revision differs from the
client's `base_revision`, the server rejects the edit with a conflict error
and the client re-fetches and re-applies. This prevents lost updates when two
clients edit the same chunk.
- **Deltas:** A delta is a JSON patch (RFC 6902 style) applied to the chunk's
logical model. The server validates the patched chunk (schema, bounds,
moderation) before persisting.
### 5.5 Write-Behind Persistence
To amortize disk I/O, the server keeps a **dirty set** of chunks modified in
memory and flushes them to SQLite in batches:
1. `put_chunk` updates the in-memory chunk and marks it `dirty`.
2. A background flusher periodically calls `flush_dirty()`, which writes all
dirty chunks in a single transaction and clears the `dirty` flag.
3. On graceful shutdown, `flush_dirty()` is called to ensure durability.
This keeps the hot path (in-memory chunk access) fast while guaranteeing
durability at shutdown and on a configurable interval.
**Recovery after unclean shutdown:** The `dirty` column in the `chunks` table
exists specifically to support recovery if the process crashes before a flush
completes. On startup, the server scans for rows where `dirty = 1` and treats
them as **unflushed in-memory state that was lost** — it re-reads the last
persisted payload from `chunks.payload` (the last durable state) and clears the
`dirty` flag. In other words, `dirty = 1` at startup means "the last write was
not durable; discard the in-memory delta and fall back to the persisted
payload." This guarantees that the on-disk state is always a consistent,
previously-committed snapshot, never a partially-written chunk. (The in-memory
dirty set is authoritative during normal operation; the `dirty` column is only a
recovery marker.)
### 5.6 Streaming & Caching
- The server streams only the chunks a client needs (those within the client's
view radius), using `list_chunks` with a bounds query.
- Chunks are cached in memory (LRU) keyed by `key`; the cache is invalidated on
write.
- The binary payload is content-addressed by `checksum`, so unchanged chunks can
be served from cache without re-reading SQLite.
---
## 6. Format Versioning & Migration
- The chunk format `version` field (currently `1`) is checked on decode. A
decoder that encounters a higher version than it supports **rejects** the
chunk with a clear error rather than misreading it.
- The SQLite `schema_version` in `meta` drives database migrations (§4.8).
- Chunk payloads are migrated **lazily**: when a chunk is read and its
`format_version` is older than the current one, the server migrates it to the
current version in memory and marks it dirty so it is rewritten on the next
flush. This avoids a costly full-world migration at startup.
---
## 7. Security & Integrity
- **Server-authoritative writes:** Only the `WorldStore` writes to SQLite;
clients submit deltas that are validated before application.
- **Checksum verification:** Every chunk payload is checksummed (SHA-256) on
write and verified on read. Corruption is detected and the chunk is re-fetched
or flagged.
- **Bounds validation:** Chunk coordinates and object positions are validated
against the world's chunk dimensions and room bounds before persistence.
- **Moderation enforcement:** `content_flags` are checked at load time; mature
chunks are gated behind the age-verification flow (see `content_moderation.md`).
- **Input validation:** All string fields are length-limited and validated
against allowed character sets to prevent injection and oversized payloads.
---
## 8. Open Questions / Future Work
- **Volumetric worlds:** The schema and format already support `CHUNK_SIZE_Z >
1`, but the tooling and streaming for 3D chunks are not yet specified.
- **Chunk compression:** For very large worlds, the binary payload could be
compressed (e.g. zstd) before storage/transfer. The header reserves a `flags`
bitmask for a compression flag.
- **Multi-server sharding:** If a single SQLite file becomes a bottleneck, the
`chunks` table could be sharded by `world_id` across multiple databases. The
key design already supports this (world_id is a leading key component).
- **Delta compression for streaming:** Instead of sending full chunks, the
server could send only the delta between revisions to reduce bandwidth for
frequently-updated chunks.
-1
View File
@@ -1 +0,0 @@
testing coder-bot identity
+24
View File
@@ -0,0 +1,24 @@
; Engine configuration file.
; It's best edited using the editor UI and not directly,
; but it can also be edited by hand.
config_version=5
[application]
config/name="AGP-Game"
run/main_scene="res://scenes/main_menu.tscn"
config/features=PackedStringArray("4.6.2")
[autoload]
[display]
window/size/viewport_width=1920
window/size/viewport_height=1080
window/stretch/mode="canvas_items"
[rendering]
renderer/rendering_method="gl_compatibility"
renderer/rendering_method.mobile="gl_compatibility"
+1
View File
@@ -0,0 +1 @@
+174
View File
@@ -0,0 +1,174 @@
extends Node
func _ready():
print("=== SANDBOX PROBE 6: Corrected Evidence Gathering (Godot 4.6.2) ===")
probe_script_loading()
probe_os_apis()
probe_file_access()
probe_network()
probe_classdb()
probe_expression_engine()
probe_engine_singletons()
probe_worker_threads()
probe_project_settings()
probe_allowlist()
print("=== SANDBOX PROBE 6 END ===")
get_tree().quit()
func probe_script_loading():
print("--- Script loading ---")
var gd = GDScript.new()
gd.source_code = "extends RefCounted\nfunc hello():\n\treturn 42\n"
var err = gd.reload()
print("GDScript.reload() from source_code err=", err)
if err == OK:
var inst = gd.new()
print(" instantiated, hello()=", inst.call("hello"))
var res = load("res://test_script.gd")
print("load('res://test_script.gd') = ", res)
if res:
print(" get_source_code() available: ", res.has_method("get_source_code"))
print(" GDScript instance has 'get_ast': ", gd.has_method("get_ast"))
print(" GDScript instance has 'get_parser': ", gd.has_method("get_parser"))
print(" GDScript instance has 'get_script_method_list': ", gd.has_method("get_script_method_list"))
print(" GDScript instance has 'get_script_property_list': ", gd.has_method("get_script_property_list"))
func probe_os_apis():
print("--- OS APIs ---")
var out = []
var exit_code = OS.execute("echo", ["hello"], out)
print("OS.execute('echo') exit=", exit_code, " out=", out)
print("OS.get_environment('HOME') = ", OS.get_environment("HOME"))
print("OS has 'create_process': ", OS.has_method("create_process"))
print("OS.get_cmdline_args() = ", OS.get_cmdline_args())
print("OS.get_user_data_dir() = ", OS.get_user_data_dir())
print("OS.get_executable_path() = ", OS.get_executable_path())
func probe_file_access():
print("--- File access ---")
var f = FileAccess.open("/etc/hostname", FileAccess.READ)
print("FileAccess.open('/etc/hostname') = ", f)
if f:
print(" content: ", f.get_as_text())
f.close()
var w = FileAccess.open("/tmp/godot_probe_write.txt", FileAccess.WRITE)
print("FileAccess.open('/tmp/...', WRITE) = ", w)
if w:
w.store_string("probe")
w.close()
print(" wrote to /tmp OK")
var dir = DirAccess.open("/")
print("DirAccess.open('/') = ", dir)
if dir:
var names = dir.get_files()
print(" root files count: ", names.size())
var res_dir = DirAccess.open("res://")
print("DirAccess.open('res://') = ", res_dir)
var user_dir = DirAccess.open("user://")
print("DirAccess.open('user://') = ", user_dir)
func probe_network():
print("--- Network ---")
var tcp = TCPServer.new()
var bind_err = tcp.listen(0)
print("TCPServer.listen(0) err=", bind_err)
var http = HTTPClient.new()
print("HTTPClient instantiated: ", http != null)
var http_req = HTTPRequest.new()
print("HTTPRequest instantiated: ", http_req != null)
var udp = PacketPeerUDP.new()
print("PacketPeerUDP instantiated: ", udp != null)
var ws = WebSocketPeer.new()
print("WebSocketPeer instantiated: ", ws != null)
func probe_classdb():
print("--- ClassDB ---")
print("ClassDB.class_exists('OS'): ", ClassDB.class_exists("OS"))
print("ClassDB.class_exists('FileAccess'): ", ClassDB.class_exists("FileAccess"))
print("ClassDB.class_exists('TCPServer'): ", ClassDB.class_exists("TCPServer"))
print("ClassDB.class_exists('HTTPClient'): ", ClassDB.class_exists("HTTPClient"))
var class_list = ClassDB.get_class_list()
print("ClassDB.get_class_list() count: ", class_list.size())
var os_methods = ClassDB.class_get_method_list("OS")
print("OS method count: ", os_methods.size())
var inst = ClassDB.instantiate("Node")
print("ClassDB.instantiate('Node') = ", inst)
print("ClassDB has 'class_get_singleton': ", ClassDB.has_method("class_get_singleton"))
func probe_expression_engine():
print("--- Expression engine ---")
var expr = Expression.new()
var err = expr.parse("1 + 2")
print("Expression.parse('1 + 2') err=", err)
if err == OK:
var result = expr.execute()
print(" Expression.execute() = ", result)
var expr2 = Expression.new()
err = expr2.parse("OS.get_environment('HOME')")
print("Expression.parse('OS.get_environment') err=", err)
if err == OK:
var result2 = expr2.execute()
print(" Expression.execute() = ", result2)
var expr3 = Expression.new()
err = expr3.parse("get_environment('HOME')", ["OS"])
print("Expression.parse with base_instance err=", err)
if err == OK:
var result3 = expr3.execute([], OS)
print(" Expression.execute with OS base = ", result3)
var expr4 = Expression.new()
err = expr4.parse("FileAccess.open('/etc/hostname', 1)")
print("Expression.parse('FileAccess.open') err=", err)
if err == OK:
var result4 = expr4.execute()
print(" Expression.execute() = ", result4)
var expr5 = Expression.new()
err = expr5.parse("get_tree().quit()")
print("Expression.parse('get_tree().quit()') err=", err)
if err == OK:
var result5 = expr5.execute()
print(" Expression.execute() = ", result5)
func probe_engine_singletons():
print("--- Engine singletons ---")
print("Engine has 'remove_singleton': ", Engine.has_method("remove_singleton"))
print("Engine has 'add_singleton': ", Engine.has_method("add_singleton"))
print("Engine.get_main_loop() = ", Engine.get_main_loop())
func probe_worker_threads():
print("--- Worker threads ---")
var thread = Thread.new()
print(" Thread.new() = ", thread)
print(" WorkerThreadPool has 'add_task': ", WorkerThreadPool.has_method("add_task"))
func probe_project_settings():
print("--- Project settings ---")
var settings = ProjectSettings.get_property_list()
var relevant = []
for s in settings:
var name = String(s.name)
if "sandbox" in name.to_lower() or "restrict" in name.to_lower() or "security" in name.to_lower():
relevant.append(name)
print(" Sandbox/security project settings: ", relevant)
print(" ProjectSettings has 'application/run/disable_os': ", ProjectSettings.has_setting("application/run/disable_os"))
print(" ProjectSettings has 'application/run/disable_file_access': ", ProjectSettings.has_setting("application/run/disable_file_access"))
print(" ProjectSettings has 'application/run/disable_network': ", ProjectSettings.has_setting("application/run/disable_network"))
print(" ProjectSettings has 'set_setting': ", ProjectSettings.has_method("set_setting"))
print(" ProjectSettings has 'save': ", ProjectSettings.has_method("save"))
func probe_allowlist():
print("--- Allowlist / static analysis ---")
var dangerous_patterns = [
"OS.execute", "OS.create_process", "OS.get_environment",
"FileAccess.open", "DirAccess.open", "TCPServer", "HTTPClient",
"load(", "preload(", "ClassDB", "Engine.get_main_loop",
"get_tree().quit", "ProjectSettings"
]
var test_script = """extends Node
func _ready():
var f = FileAccess.open("/etc/passwd", FileAccess.READ)
OS.execute("rm", ["-rf", "/"])
"""
print(" Test script contains dangerous patterns:")
for p in dangerous_patterns:
if p in test_script:
print(" DETECTED: ", p)
+6
View File
@@ -0,0 +1,6 @@
[gd_scene load_steps=2 format=3]
[ext_resource type="Script" path="res://probe6.gd" id="1"]
[node name="Probe6" type="Node"]
script = ExtResource("1")
+8
View File
@@ -0,0 +1,8 @@
; Engine configuration file.
config_version=5
[application]
config/name="sandbox_probe"
run/main_scene="res://probe6.tscn"
config/features=PackedStringArray("4.6")
+4
View File
@@ -0,0 +1,4 @@
extends RefCounted
func hello():
return "from test_script.gd"