From 41fcbe746eff4bc430f402efed3e8992831b56b6 Mon Sep 17 00:00:00 2001 From: agent-bot Date: Mon, 17 Aug 2026 15:46:04 +0000 Subject: [PATCH] docs(moderation): add content moderation and age verification architecture --- docs/architecture/content_moderation.md | 426 ++++++++++++++++++++++++ 1 file changed, 426 insertions(+) create mode 100644 docs/architecture/content_moderation.md diff --git a/docs/architecture/content_moderation.md b/docs/architecture/content_moderation.md new file mode 100644 index 0000000..66f5ae0 --- /dev/null +++ b/docs/architecture/content_moderation.md @@ -0,0 +1,426 @@ +# Content Moderation & Age Verification Architecture + +> **Status:** Proposed +> **Owner:** Platform team +> **Related:** Server authority + client prediction, avatar component system, backend auth (JWT) + +## 1. Overview + +This document specifies the **content moderation** and **age verification** +architecture for **agp-game**, a Godot-based multiplayer game built by +autonomous agents. It defines: + +- How **content flags** propagate through the system across the three primary + content types: **assets**, **rooms**, and **scripts**. +- How flags are **evaluated, aggregated, and enforced** at load time, at + runtime, and at the network boundary. +- The **age verification flow**, combining a lightweight **self-declaration** + with an optional **ID check hook** for higher-assurance verification. + +The goal is a **single, consistent moderation pipeline** that applies the same +rules to user-generated content (UGC) regardless of whether it is a 3D asset, a +room definition, or a script, and that gates access to mature content behind a +verifiable age gate. + +### 1.1 Design Goals + +- **Uniformity:** One flag vocabulary and one propagation mechanism across + assets, rooms, and scripts. +- **Composability:** Flags on a room aggregate the flags of the assets and + scripts it contains; flags on a world aggregate the flags of its rooms. +- **Enforcement at the boundary:** The server is authoritative for what content + a client may load or receive, mirroring the server-authoritative networking + model. +- **Privacy-preserving age checks:** Self-declaration requires no PII; the ID + check hook is optional, pluggable, and never stores raw identity documents. +- **Auditability:** Every flag decision is logged with a reason, an actor, and + a timestamp so moderation actions are reviewable. + +--- + +## 2. Content Flag Vocabulary + +All content carries a set of **content flags**. A flag is a typed, immutable +label with a **severity** and a **scope**. Flags are the atomic unit of +moderation; everything else (aggregation, enforcement, gating) operates on +flags. + +### 2.1 Flag Definition + +A flag is a structured record: + +| Field | Type | Description | +|------------|-----------|--------------------------------------------------------------------------| +| `id` | `string` | Stable machine-readable identifier, e.g. `violence.gore`. | +| `label` | `string` | Human-readable name, e.g. "Graphic Gore". | +| `category` | `enum` | One of `violence`, `sexual`, `language`, `substance`, `gambling`, `horror`, `hate`, `custom`. | +| `severity` | `enum` | One of `none`, `mild`, `moderate`, `severe`. | +| `scope` | `enum` | One of `visual`, `audio`, `text`, `behavior`, `interaction`. | +| `mature` | `bool` | Whether this flag triggers the age gate (see §5). | +| `reason` | `string` | Free-text justification supplied by the flagger. | +| `source` | `enum` | One of `author`, `moderator`, `automated`, `community`. | + +### 2.2 Canonical Flag Set + +The platform defines a **canonical registry** of flags so that different +content types and different flaggers use the same vocabulary. Examples: + +| id | category | severity | scope | mature | +|-----------------------------|------------|-----------|--------------|--------| +| `violence.none` | violence | none | visual | false | +| `violence.mild` | violence | mild | visual | false | +| `violence.gore` | violence | severe | visual | true | +| `sexual.suggestive` | sexual | mild | visual | true | +| `sexual.explicit` | sexual | severe | visual | true | +| `language.mild` | language | mild | text | false | +| `language.profanity` | language | moderate | text | false | +| `substance.references` | substance | mild | text | false | +| `gambling.simulated` | gambling | moderate | behavior | true | +| `horror.jumpscare` | horror | mild | audio/visual | false | +| `hate.hateful` | hate | severe | text | true | + +The registry is **data-driven** (a JSON/YAML manifest shipped with the server) +so new flags can be added without code changes. + +--- + +## 3. Flag Propagation Model + +Flags propagate through the system along the **content dependency graph**. The +three content types form a hierarchy: + +``` +World + └── Room + ├── Asset (mesh, texture, audio, animation, material) + └── Script (behavior, dialogue, event logic) +``` + +### 3.1 Propagation Rules + +1. **Explicit flags** are set directly on a content item by its author or a + moderator. +2. **Inherited flags** flow upward: a room's effective flag set is the **union** + of its own explicit flags and the flags of every asset and script it + references. +3. **Aggregation** uses the **maximum severity** per category across all + contributing items. A room containing one `violence.gore` asset and one + `violence.none` asset is `violence.severe`. +4. **Maturity is monotonic:** if any contributing item is `mature`, the + containing room and world are `mature`. +5. **Cycles are impossible** by construction: content references form a DAG + (an asset cannot contain a room; a room cannot contain itself). + +### 3.2 Propagation Through the Pipeline + +Flags are attached at **authoring time** and carried through every stage: + +``` +Authoring ──► Upload ──► Validation ──► Storage ──► Distribution ──► Runtime + │ │ │ │ │ │ + │ flag │ flag │ automated │ flag │ flag │ flag + │ declared │ preserved │ scan adds │ indexed │ re-checked │ enforced + │ │ │ flags │ │ │ +``` + +| Stage | Responsibility | +|--------------|-------------------------------------------------------------------------| +| **Authoring**| Author declares flags in the editor/upload UI. | +| **Upload** | Flags are validated against the canonical registry; unknown flags rejected. | +| **Validation**| Automated scanner (hash-based + heuristic) adds flags for detected content. | +| **Storage** | Flags are stored alongside the content record and indexed for queries. | +| **Distribution**| Flags are re-checked against the current registry before serving. | +| **Runtime** | The client enforces flags before loading content (see §4). | + +### 3.3 Data Model + +Flags are stored as a **normalized join table** so they can be queried and +aggregated efficiently: + +``` +content_item (id, type, owner_id, ...) +content_flag (content_id, flag_id, source, reason, created_at, actor_id) +flag_registry (id, category, severity, scope, mature, ...) +``` + +Aggregation is computed by a SQL view / query that joins a content item's +transitive dependencies. Because the graph is a DAG, this is a bounded +recursive query (PostgreSQL `WITH RECURSIVE`). + +--- + +## 4. Enforcement + +Enforcement happens at three layers, in order: + +### 4.1 Load-Time Enforcement (Server) + +Before a client is allowed to **request** content, the server evaluates the +content's effective flag set against the requesting account's **content +preferences** and **age verification status**: + +- If the content is `mature` and the account is **not age-verified**, the + request is **rejected** with a `403 MATURE_CONTENT_REQUIRES_VERIFICATION`. +- If the content's flags exceed the account's **content filter** (a per-account + severity threshold per category), the request is rejected or the offending + content is **stripped** (e.g., audio muted, mesh hidden, script not loaded). + +### 4.2 Runtime Enforcement (Client) + +The client receives a **manifest** of allowed content from the server. It never +loads content that is not in the manifest. This prevents a malicious or buggy +client from bypassing the server's filter by loading content directly. + +- Assets not in the manifest are **not instantiated**. +- Scripts not in the manifest are **not executed**. +- Rooms not in the manifest are **not entered**. + +### 4.3 Network Boundary Enforcement + +The server **filters replication** based on flags: + +- Mature or filtered content is **not replicated** to clients that cannot + receive it. +- A room's flag set is broadcast in the room metadata so clients can show a + warning before entry. + +### 4.4 Enforcement Decision Flow + +``` +Client requests content C for account A + │ + ▼ +Compute effective flags F(C) [union + max-severity aggregation] + │ + ▼ +Is A age-verified? ──no──► Is F(C).mature? ──yes──► REJECT (403) + │ yes │ no + ▼ ▼ +Does F(C) exceed A's filter? ──► yes ──► REJECT or STRIP + │ no + ▼ +Serve content + manifest +``` + +--- + +## 5. Age Verification Flow + +Age verification is a **two-tier** system: + +- **Tier 1 — Self-declaration:** The user asserts their date of birth. This is + fast, requires no PII beyond a birthdate, and is sufficient for most content. +- **Tier 2 — ID check (optional hook):** A pluggable identity-verification + provider confirms the user is an adult. This raises the account's assurance + level and unlocks the highest maturity tier. + +### 5.1 Verification States + +An account has an **age verification status**: + +| Status | Meaning | +|-------------------|----------------------------------------------------------------------| +| `unverified` | No declaration on file. | +| `self_declared` | User declared a birthdate; **not** independently confirmed. | +| `id_verified` | User passed the ID check hook; independently confirmed adult. | +| `rejected` | ID check failed or declaration was inconsistent. | + +### 5.2 Tier 1 — Self-Declaration Flow + +``` +User submits { date_of_birth } + │ + ▼ +Server validates format & plausibility (age 13–120, not in future) + │ + ▼ +Compute age from DOB + │ + ▼ +age >= 18 ──► status = self_declared (adult) +age >= 13 ──► status = self_declared (minor, restricted) +age < 13 ──► status = rejected (below minimum platform age) +``` + +- The DOB is stored **encrypted at rest** and is **never exposed** to other + players. +- A user may **upgrade** from `self_declared` to `id_verified` at any time via + Tier 2. + +### 5.3 Tier 2 — ID Check Hook + +The ID check is an **optional, pluggable hook**. The platform defines an +interface; concrete providers (e.g., a third-party KYC service) implement it. + +``` +User requests ID verification + │ + ▼ +Server creates a verification session (idempotent, expiring) + │ + ▼ +Server calls provider hook: verify(identity_token) -> { adult: bool, ref: string } + │ + ▼ +adult == true ──► status = id_verified, store provider ref (not raw doc) +adult == false ─► status = rejected +provider error ──► status unchanged, retry allowed +``` + +**Privacy guarantees:** + +- The platform **never stores** the raw identity document or image. +- Only the provider's **opaque reference** and the boolean `adult` result are + persisted. +- The hook is **asynchronous and idempotent**; a session can be retried without + double-charging or double-verifying. + +### 5.4 Hook Interface + +The ID check hook is defined as a backend interface (FastAPI dependency / +abstract service): + +```python +class IdentityVerificationProvider(Protocol): + def create_session(self, user_id: str) -> VerificationSession: ... + def verify(self, session_id: str, identity_token: str) -> VerificationResult: ... + +class VerificationResult: + adult: bool + provider_ref: str | None + error: str | None +``` + +A default **no-op provider** returns `adult=False` with a clear error, so the +system runs without any third-party integration until one is configured. + +### 5.5 Age Gate Enforcement + +The age gate is enforced at **load time** (see §4.1): + +| Verification status | Can access `mature` content? | +|---------------------|------------------------------| +| `unverified` | No | +| `self_declared` | Yes (adult) / No (minor) | +| `id_verified` | Yes | +| `rejected` | No | + +--- + +## 6. Moderation Workflow + +### 6.1 Flagging Sources + +Content can be flagged by: + +- **Author** (self-declared at upload). +- **Automated scanner** (hash-based detection, heuristic analysis). +- **Community** (players report content; reports become flags after review). +- **Moderator** (manual review and override). + +### 6.2 Review Queue + +Community and automated flags enter a **review queue**. A moderator either: + +- **Approves** the flag (it becomes authoritative), or +- **Rejects** the flag (it is discarded), or +- **Escalates** to a senior moderator for severe/hate flags. + +### 6.3 Actions + +Once a flag is authoritative, a moderator can take actions: + +| Action | Effect | +|-----------------|---------------------------------------------------------------| +| `warn` | Notify the author; content remains. | +| `restrict` | Content hidden from users below a filter threshold. | +| `age_gate` | Content marked `mature`; requires age verification. | +| `remove` | Content taken down; not served. | +| `ban_author` | Author's upload privileges suspended. | + +### 6.4 Audit Log + +Every flag and action is written to an **audit log**: + +``` +audit_event (id, content_id, actor_id, action, flag_id, reason, created_at) +``` + +This log is append-only and used for transparency and dispute resolution. + +--- + +## 7. Client-Side Integration (Godot) + +### 7.1 Content Manifest + +The server sends each client a **content manifest** — the set of content the +client is permitted to load, derived from the account's verification status and +filter. The Godot client: + +- Loads only manifest-listed assets. +- Executes only manifest-listed scripts. +- Shows a **maturity warning** dialog before entering a room whose effective + flags include `mature` content. + +### 7.2 Flag Metadata on Content + +Each content resource carries a `content_flags` metadata field (a Godot +`Dictionary` or custom `Resource`). The client uses this for local UI decisions +(e.g., showing a warning icon) but **never** as the authority for whether to +load — the server manifest is authoritative. + +### 7.3 Age Verification UI + +The client exposes an **age verification screen**: + +- Tier 1: a date-of-birth picker. +- Tier 2: a button that opens the ID check provider flow (deep link / webview). + +The client calls the backend endpoints (§8) and reflects the resulting status. + +--- + +## 8. Backend API Surface + +| Method | Path | Description | +|--------|-----------------------------------|------------------------------------------------| +| `POST` | `/v1/age/declare` | Submit self-declared DOB (Tier 1). | +| `POST` | `/v1/age/verify/session` | Create an ID-check session (Tier 2). | +| `POST` | `/v1/age/verify/confirm` | Confirm an ID-check session with a token. | +| `GET` | `/v1/age/status` | Return the account's verification status. | +| `GET` | `/v1/content/{id}/flags` | Return a content item's effective flags. | +| `POST` | `/v1/content/{id}/flags` | Add a flag (author/community). | +| `POST` | `/v1/moderation/review/{flag_id}` | Moderator approve/reject/escalate. | +| `POST` | `/v1/moderation/action` | Apply a moderation action. | +| `GET` | `/v1/manifest` | Return the client's permitted content manifest.| + +All endpoints require a valid **JWT** (matching the existing auth stack). The +age endpoints additionally require the account to be in a non-`rejected` state. + +--- + +## 9. Security & Privacy Considerations + +- **DOB encryption:** Self-declared birthdates are encrypted at rest; only the + derived age and verification status are used in authorization decisions. +- **No raw identity storage:** The ID check hook persists only a provider + reference and a boolean result. +- **Server authority:** The client manifest prevents bypass; the client never + trusts its own flag metadata for access decisions. +- **Rate limiting:** Age-declaration and ID-check endpoints are rate-limited to + prevent abuse and enumeration. +- **Auditability:** All flag and verification events are logged append-only. + +--- + +## 10. Open Questions / Future Work + +- **Regional age thresholds:** Different jurisdictions may require different + minimum ages; the threshold should be configurable per region. +- **Parental controls:** A future feature may let a parent set a content filter + on a minor's account. +- **Automated scanner accuracy:** The heuristic scanner may produce false + positives; a confidence score should be attached to automated flags. +- **Offline / local content:** How flags apply to locally-authored content that + is never uploaded remains to be defined.