docs(moderation): add content moderation and age verification architecture

2026-08-17 15:46:04 +00:00
1 changed files with 426 additions and 0 deletions
@@ -0,0 +1,426 @@
 # Content Moderation & Age Verification Architecture
 > **Status:** Proposed
 > **Owner:** Platform team
 > **Related:** Server authority + client prediction, avatar component system, backend auth (JWT)
 ## 1. Overview
 This document specifies the **content moderation** and **age verification**
 architecture for **agp-game**, a Godot-based multiplayer game built by
 autonomous agents. It defines:
 - How **content flags** propagate through the system across the three primary
  content types: **assets**, **rooms**, and **scripts**.
 - How flags are **evaluated, aggregated, and enforced** at load time, at
  runtime, and at the network boundary.
 - The **age verification flow**, combining a lightweight **self-declaration**
  with an optional **ID check hook** for higher-assurance verification.
 The goal is a **single, consistent moderation pipeline** that applies the same
 rules to user-generated content (UGC) regardless of whether it is a 3D asset, a
 room definition, or a script, and that gates access to mature content behind a
 verifiable age gate.
 ### 1.1 Design Goals
 - **Uniformity:** One flag vocabulary and one propagation mechanism across
  assets, rooms, and scripts.
 - **Composability:** Flags on a room aggregate the flags of the assets and
  scripts it contains; flags on a world aggregate the flags of its rooms.
 - **Enforcement at the boundary:** The server is authoritative for what content
  a client may load or receive, mirroring the server-authoritative networking
  model.
 - **Privacy-preserving age checks:** Self-declaration requires no PII; the ID
  check hook is optional, pluggable, and never stores raw identity documents.
 - **Auditability:** Every flag decision is logged with a reason, an actor, and
  a timestamp so moderation actions are reviewable.
 ---
 ## 2. Content Flag Vocabulary
 All content carries a set of **content flags**. A flag is a typed, immutable
 label with a **severity** and a **scope**. Flags are the atomic unit of
 moderation; everything else (aggregation, enforcement, gating) operates on
 flags.
 ### 2.1 Flag Definition
 A flag is a structured record:
 | Field      | Type      | Description                                                              |
 |------------|-----------|--------------------------------------------------------------------------|
 | `id`       | `string`  | Stable machine-readable identifier, e.g. `violence.gore`.                |
 | `label`    | `string`  | Human-readable name, e.g. "Graphic Gore".                                |
 | `category` | `enum`    | One of `violence`, `sexual`, `language`, `substance`, `gambling`, `horror`, `hate`, `custom`. |
 | `severity` | `enum`    | One of `none`, `mild`, `moderate`, `severe`.                             |
 | `scope`    | `enum`    | One of `visual`, `audio`, `text`, `behavior`, `interaction`.             |
 | `mature`   | `bool`    | Whether this flag triggers the age gate (see §5).                        |
 | `reason`   | `string`  | Free-text justification supplied by the flagger.                         |
 | `source`   | `enum`    | One of `author`, `moderator`, `automated`, `community`.                  |
 ### 2.2 Canonical Flag Set
 The platform defines a **canonical registry** of flags so that different
 content types and different flaggers use the same vocabulary. Examples:
 | id                          | category   | severity  | scope        | mature |
 |-----------------------------|------------|-----------|--------------|--------|
 | `violence.none`             | violence   | none      | visual       | false  |
 | `violence.mild`             | violence   | mild      | visual       | false  |
 | `violence.gore`             | violence   | severe    | visual       | true   |
 | `sexual.suggestive`         | sexual     | mild      | visual       | true   |
 | `sexual.explicit`           | sexual     | severe    | visual       | true   |
 | `language.mild`             | language   | mild      | text         | false  |
 | `language.profanity`        | language   | moderate  | text         | false  |
 | `substance.references`      | substance  | mild      | text         | false  |
 | `gambling.simulated`        | gambling   | moderate  | behavior     | true   |
 | `horror.jumpscare`          | horror     | mild      | audio/visual | false  |
 | `hate.hateful`              | hate       | severe    | text         | true   |
 The registry is **data-driven** (a JSON/YAML manifest shipped with the server)
 so new flags can be added without code changes.
 ---
 ## 3. Flag Propagation Model
 Flags propagate through the system along the **content dependency graph**. The
 three content types form a hierarchy:
 ```
 World
 └── Room
      ├── Asset  (mesh, texture, audio, animation, material)
      └── Script (behavior, dialogue, event logic)
 ```
 ### 3.1 Propagation Rules
 1. **Explicit flags** are set directly on a content item by its author or a
   moderator.
 2. **Inherited flags** flow upward: a room's effective flag set is the **union**
   of its own explicit flags and the flags of every asset and script it
   references.
 3. **Aggregation** uses the **maximum severity** per category across all
   contributing items. A room containing one `violence.gore` asset and one
   `violence.none` asset is `violence.severe`.
 4. **Maturity is monotonic:** if any contributing item is `mature`, the
   containing room and world are `mature`.
 5. **Cycles are impossible** by construction: content references form a DAG
   (an asset cannot contain a room; a room cannot contain itself).
 ### 3.2 Propagation Through the Pipeline
 Flags are attached at **authoring time** and carried through every stage:
 ```
 Authoring ──► Upload ──► Validation ──► Storage ──► Distribution ──► Runtime
   │            │            │             │             │              │
   │ flag       │ flag       │ automated   │ flag        │ flag         │ flag
   │ declared   │ preserved  │ scan adds   │ indexed     │ re-checked   │ enforced
   │            │            │ flags       │             │              │
 ```
 | Stage        | Responsibility                                                          |
 |--------------|-------------------------------------------------------------------------|
 | **Authoring**| Author declares flags in the editor/upload UI.                          |
 | **Upload**   | Flags are validated against the canonical registry; unknown flags rejected. |
 | **Validation**| Automated scanner (hash-based + heuristic) adds flags for detected content. |
 | **Storage**  | Flags are stored alongside the content record and indexed for queries.  |
 | **Distribution**| Flags are re-checked against the current registry before serving.     |
 | **Runtime**  | The client enforces flags before loading content (see §4).              |
 ### 3.3 Data Model
 Flags are stored as a **normalized join table** so they can be queried and
 aggregated efficiently:
 ```
 content_item (id, type, owner_id, ...)
 content_flag  (content_id, flag_id, source, reason, created_at, actor_id)
 flag_registry (id, category, severity, scope, mature, ...)
 ```
 Aggregation is computed by a SQL view / query that joins a content item's
 transitive dependencies. Because the graph is a DAG, this is a bounded
 recursive query (PostgreSQL `WITH RECURSIVE`).
 ---
 ## 4. Enforcement
 Enforcement happens at three layers, in order:
 ### 4.1 Load-Time Enforcement (Server)
 Before a client is allowed to **request** content, the server evaluates the
 content's effective flag set against the requesting account's **content
 preferences** and **age verification status**:
 - If the content is `mature` and the account is **not age-verified**, the
  request is **rejected** with a `403 MATURE_CONTENT_REQUIRES_VERIFICATION`.
 - If the content's flags exceed the account's **content filter** (a per-account
  severity threshold per category), the request is rejected or the offending
  content is **stripped** (e.g., audio muted, mesh hidden, script not loaded).
 ### 4.2 Runtime Enforcement (Client)
 The client receives a **manifest** of allowed content from the server. It never
 loads content that is not in the manifest. This prevents a malicious or buggy
 client from bypassing the server's filter by loading content directly.
 - Assets not in the manifest are **not instantiated**.
 - Scripts not in the manifest are **not executed**.
 - Rooms not in the manifest are **not entered**.
 ### 4.3 Network Boundary Enforcement
 The server **filters replication** based on flags:
 - Mature or filtered content is **not replicated** to clients that cannot
  receive it.
 - A room's flag set is broadcast in the room metadata so clients can show a
  warning before entry.
 ### 4.4 Enforcement Decision Flow
 ```
 Client requests content C for account A
        │
        ▼
 Compute effective flags F(C)   [union + max-severity aggregation]
        │
        ▼
 Is A age-verified?  ──no──►  Is F(C).mature? ──yes──► REJECT (403)
        │ yes                     │ no
        ▼                         ▼
 Does F(C) exceed A's filter? ──► yes ──► REJECT or STRIP
        │ no
        ▼
 Serve content + manifest
 ```
 ---
 ## 5. Age Verification Flow
 Age verification is a **two-tier** system:
 - **Tier 1 — Self-declaration:** The user asserts their date of birth. This is
  fast, requires no PII beyond a birthdate, and is sufficient for most content.
 - **Tier 2 — ID check (optional hook):** A pluggable identity-verification
  provider confirms the user is an adult. This raises the account's assurance
  level and unlocks the highest maturity tier.
 ### 5.1 Verification States
 An account has an **age verification status**:
 | Status            | Meaning                                                              |
 |-------------------|----------------------------------------------------------------------|
 | `unverified`      | No declaration on file.                                              |
 | `self_declared`   | User declared a birthdate; **not** independently confirmed.          |
 | `id_verified`     | User passed the ID check hook; independently confirmed adult.        |
 | `rejected`        | ID check failed or declaration was inconsistent.                     |
 ### 5.2 Tier 1 — Self-Declaration Flow
 ```
 User submits { date_of_birth }
        │
        ▼
 Server validates format & plausibility (age 13–120, not in future)
        │
        ▼
 Compute age from DOB
        │
        ▼
 age >= 18 ──► status = self_declared (adult)
 age >= 13 ──► status = self_declared (minor, restricted)
 age <  13 ──► status = rejected (below minimum platform age)
 ```
 - The DOB is stored **encrypted at rest** and is **never exposed** to other
  players.
 - A user may **upgrade** from `self_declared` to `id_verified` at any time via
  Tier 2.
 ### 5.3 Tier 2 — ID Check Hook
 The ID check is an **optional, pluggable hook**. The platform defines an
 interface; concrete providers (e.g., a third-party KYC service) implement it.
 ```
 User requests ID verification
        │
        ▼
 Server creates a verification session (idempotent, expiring)
        │
        ▼
 Server calls provider hook:  verify(identity_token) -> { adult: bool, ref: string }
        │
        ▼
 adult == true ──► status = id_verified, store provider ref (not raw doc)
 adult == false ─► status = rejected
 provider error ──► status unchanged, retry allowed
 ```
 **Privacy guarantees:**
 - The platform **never stores** the raw identity document or image.
 - Only the provider's **opaque reference** and the boolean `adult` result are
  persisted.
 - The hook is **asynchronous and idempotent**; a session can be retried without
  double-charging or double-verifying.
 ### 5.4 Hook Interface
 The ID check hook is defined as a backend interface (FastAPI dependency /
 abstract service):
 ```python
 class IdentityVerificationProvider(Protocol):
    def create_session(self, user_id: str) -> VerificationSession: ...
    def verify(self, session_id: str, identity_token: str) -> VerificationResult: ...
 class VerificationResult:
    adult: bool
    provider_ref: str | None
    error: str | None
 ```
 A default **no-op provider** returns `adult=False` with a clear error, so the
 system runs without any third-party integration until one is configured.
 ### 5.5 Age Gate Enforcement
 The age gate is enforced at **load time** (see §4.1):
 | Verification status | Can access `mature` content? |
 |---------------------|------------------------------|
 | `unverified`        | No                           |
 | `self_declared`     | Yes (adult) / No (minor)     |
 | `id_verified`       | Yes                          |
 | `rejected`          | No                           |
 ---
 ## 6. Moderation Workflow
 ### 6.1 Flagging Sources
 Content can be flagged by:
 - **Author** (self-declared at upload).
 - **Automated scanner** (hash-based detection, heuristic analysis).
 - **Community** (players report content; reports become flags after review).
 - **Moderator** (manual review and override).
 ### 6.2 Review Queue
 Community and automated flags enter a **review queue**. A moderator either:
 - **Approves** the flag (it becomes authoritative), or
 - **Rejects** the flag (it is discarded), or
 - **Escalates** to a senior moderator for severe/hate flags.
 ### 6.3 Actions
 Once a flag is authoritative, a moderator can take actions:
 | Action          | Effect                                                        |
 |-----------------|---------------------------------------------------------------|
 | `warn`          | Notify the author; content remains.                           |
 | `restrict`      | Content hidden from users below a filter threshold.           |
 | `age_gate`      | Content marked `mature`; requires age verification.           |
 | `remove`        | Content taken down; not served.                               |
 | `ban_author`    | Author's upload privileges suspended.                         |
 ### 6.4 Audit Log
 Every flag and action is written to an **audit log**:
 ```
 audit_event (id, content_id, actor_id, action, flag_id, reason, created_at)
 ```
 This log is append-only and used for transparency and dispute resolution.
 ---
 ## 7. Client-Side Integration (Godot)
 ### 7.1 Content Manifest
 The server sends each client a **content manifest** — the set of content the
 client is permitted to load, derived from the account's verification status and
 filter. The Godot client:
 - Loads only manifest-listed assets.
 - Executes only manifest-listed scripts.
 - Shows a **maturity warning** dialog before entering a room whose effective
  flags include `mature` content.
 ### 7.2 Flag Metadata on Content
 Each content resource carries a `content_flags` metadata field (a Godot
 `Dictionary` or custom `Resource`). The client uses this for local UI decisions
 (e.g., showing a warning icon) but **never** as the authority for whether to
 load — the server manifest is authoritative.
 ### 7.3 Age Verification UI
 The client exposes an **age verification screen**:
 - Tier 1: a date-of-birth picker.
 - Tier 2: a button that opens the ID check provider flow (deep link / webview).
 The client calls the backend endpoints (§8) and reflects the resulting status.
 ---
 ## 8. Backend API Surface
 | Method | Path                              | Description                                    |
 |--------|-----------------------------------|------------------------------------------------|
 | `POST` | `/v1/age/declare`                 | Submit self-declared DOB (Tier 1).             |
 | `POST` | `/v1/age/verify/session`          | Create an ID-check session (Tier 2).           |
 | `POST` | `/v1/age/verify/confirm`          | Confirm an ID-check session with a token.      |
 | `GET`  | `/v1/age/status`                  | Return the account's verification status.      |
 | `GET`  | `/v1/content/{id}/flags`          | Return a content item's effective flags.       |
 | `POST` | `/v1/content/{id}/flags`          | Add a flag (author/community).                 |
 | `POST` | `/v1/moderation/review/{flag_id}` | Moderator approve/reject/escalate.             |
 | `POST` | `/v1/moderation/action`           | Apply a moderation action.                     |
 | `GET`  | `/v1/manifest`                    | Return the client's permitted content manifest.|
 All endpoints require a valid **JWT** (matching the existing auth stack). The
 age endpoints additionally require the account to be in a non-`rejected` state.
 ---
 ## 9. Security & Privacy Considerations
 - **DOB encryption:** Self-declared birthdates are encrypted at rest; only the
  derived age and verification status are used in authorization decisions.
 - **No raw identity storage:** The ID check hook persists only a provider
  reference and a boolean result.
 - **Server authority:** The client manifest prevents bypass; the client never
  trusts its own flag metadata for access decisions.
 - **Rate limiting:** Age-declaration and ID-check endpoints are rate-limited to
  prevent abuse and enumeration.
 - **Auditability:** All flag and verification events are logged append-only.
 ---
 ## 10. Open Questions / Future Work
 - **Regional age thresholds:** Different jurisdictions may require different
  minimum ages; the threshold should be configurable per region.
 - **Parental controls:** A future feature may let a parent set a content filter
  on a minor's account.
 - **Automated scanner accuracy:** The heuristic scanner may produce false
  positives; a confidence score should be attached to automated flags.
 - **Offline / local content:** How flags apply to locally-authored content that
  is never uploaded remains to be defined.