427 lines
17 KiB
Markdown
427 lines
17 KiB
Markdown
# Content Moderation & Age Verification Architecture
|
||
|
||
> **Status:** Proposed
|
||
> **Owner:** Platform team
|
||
> **Related:** Server authority + client prediction, avatar component system, backend auth (JWT)
|
||
|
||
## 1. Overview
|
||
|
||
This document specifies the **content moderation** and **age verification**
|
||
architecture for **agp-game**, a Godot-based multiplayer game built by
|
||
autonomous agents. It defines:
|
||
|
||
- How **content flags** propagate through the system across the three primary
|
||
content types: **assets**, **rooms**, and **scripts**.
|
||
- How flags are **evaluated, aggregated, and enforced** at load time, at
|
||
runtime, and at the network boundary.
|
||
- The **age verification flow**, combining a lightweight **self-declaration**
|
||
with an optional **ID check hook** for higher-assurance verification.
|
||
|
||
The goal is a **single, consistent moderation pipeline** that applies the same
|
||
rules to user-generated content (UGC) regardless of whether it is a 3D asset, a
|
||
room definition, or a script, and that gates access to mature content behind a
|
||
verifiable age gate.
|
||
|
||
### 1.1 Design Goals
|
||
|
||
- **Uniformity:** One flag vocabulary and one propagation mechanism across
|
||
assets, rooms, and scripts.
|
||
- **Composability:** Flags on a room aggregate the flags of the assets and
|
||
scripts it contains; flags on a world aggregate the flags of its rooms.
|
||
- **Enforcement at the boundary:** The server is authoritative for what content
|
||
a client may load or receive, mirroring the server-authoritative networking
|
||
model.
|
||
- **Privacy-preserving age checks:** Self-declaration requires no PII; the ID
|
||
check hook is optional, pluggable, and never stores raw identity documents.
|
||
- **Auditability:** Every flag decision is logged with a reason, an actor, and
|
||
a timestamp so moderation actions are reviewable.
|
||
|
||
---
|
||
|
||
## 2. Content Flag Vocabulary
|
||
|
||
All content carries a set of **content flags**. A flag is a typed, immutable
|
||
label with a **severity** and a **scope**. Flags are the atomic unit of
|
||
moderation; everything else (aggregation, enforcement, gating) operates on
|
||
flags.
|
||
|
||
### 2.1 Flag Definition
|
||
|
||
A flag is a structured record:
|
||
|
||
| Field | Type | Description |
|
||
|------------|-----------|--------------------------------------------------------------------------|
|
||
| `id` | `string` | Stable machine-readable identifier, e.g. `violence.gore`. |
|
||
| `label` | `string` | Human-readable name, e.g. "Graphic Gore". |
|
||
| `category` | `enum` | One of `violence`, `sexual`, `language`, `substance`, `gambling`, `horror`, `hate`, `custom`. |
|
||
| `severity` | `enum` | One of `none`, `mild`, `moderate`, `severe`. |
|
||
| `scope` | `enum` | One of `visual`, `audio`, `text`, `behavior`, `interaction`. |
|
||
| `mature` | `bool` | Whether this flag triggers the age gate (see §5). |
|
||
| `reason` | `string` | Free-text justification supplied by the flagger. |
|
||
| `source` | `enum` | One of `author`, `moderator`, `automated`, `community`. |
|
||
|
||
### 2.2 Canonical Flag Set
|
||
|
||
The platform defines a **canonical registry** of flags so that different
|
||
content types and different flaggers use the same vocabulary. Examples:
|
||
|
||
| id | category | severity | scope | mature |
|
||
|-----------------------------|------------|-----------|--------------|--------|
|
||
| `violence.none` | violence | none | visual | false |
|
||
| `violence.mild` | violence | mild | visual | false |
|
||
| `violence.gore` | violence | severe | visual | true |
|
||
| `sexual.suggestive` | sexual | mild | visual | true |
|
||
| `sexual.explicit` | sexual | severe | visual | true |
|
||
| `language.mild` | language | mild | text | false |
|
||
| `language.profanity` | language | moderate | text | false |
|
||
| `substance.references` | substance | mild | text | false |
|
||
| `gambling.simulated` | gambling | moderate | behavior | true |
|
||
| `horror.jumpscare` | horror | mild | audio/visual | false |
|
||
| `hate.hateful` | hate | severe | text | true |
|
||
|
||
The registry is **data-driven** (a JSON/YAML manifest shipped with the server)
|
||
so new flags can be added without code changes.
|
||
|
||
---
|
||
|
||
## 3. Flag Propagation Model
|
||
|
||
Flags propagate through the system along the **content dependency graph**. The
|
||
three content types form a hierarchy:
|
||
|
||
```
|
||
World
|
||
└── Room
|
||
├── Asset (mesh, texture, audio, animation, material)
|
||
└── Script (behavior, dialogue, event logic)
|
||
```
|
||
|
||
### 3.1 Propagation Rules
|
||
|
||
1. **Explicit flags** are set directly on a content item by its author or a
|
||
moderator.
|
||
2. **Inherited flags** flow upward: a room's effective flag set is the **union**
|
||
of its own explicit flags and the flags of every asset and script it
|
||
references.
|
||
3. **Aggregation** uses the **maximum severity** per category across all
|
||
contributing items. A room containing one `violence.gore` asset and one
|
||
`violence.none` asset is `violence.severe`.
|
||
4. **Maturity is monotonic:** if any contributing item is `mature`, the
|
||
containing room and world are `mature`.
|
||
5. **Cycles are impossible** by construction: content references form a DAG
|
||
(an asset cannot contain a room; a room cannot contain itself).
|
||
|
||
### 3.2 Propagation Through the Pipeline
|
||
|
||
Flags are attached at **authoring time** and carried through every stage:
|
||
|
||
```
|
||
Authoring ──► Upload ──► Validation ──► Storage ──► Distribution ──► Runtime
|
||
│ │ │ │ │ │
|
||
│ flag │ flag │ automated │ flag │ flag │ flag
|
||
│ declared │ preserved │ scan adds │ indexed │ re-checked │ enforced
|
||
│ │ │ flags │ │ │
|
||
```
|
||
|
||
| Stage | Responsibility |
|
||
|--------------|-------------------------------------------------------------------------|
|
||
| **Authoring**| Author declares flags in the editor/upload UI. |
|
||
| **Upload** | Flags are validated against the canonical registry; unknown flags rejected. |
|
||
| **Validation**| Automated scanner (hash-based + heuristic) adds flags for detected content. |
|
||
| **Storage** | Flags are stored alongside the content record and indexed for queries. |
|
||
| **Distribution**| Flags are re-checked against the current registry before serving. |
|
||
| **Runtime** | The client enforces flags before loading content (see §4). |
|
||
|
||
### 3.3 Data Model
|
||
|
||
Flags are stored as a **normalized join table** so they can be queried and
|
||
aggregated efficiently:
|
||
|
||
```
|
||
content_item (id, type, owner_id, ...)
|
||
content_flag (content_id, flag_id, source, reason, created_at, actor_id)
|
||
flag_registry (id, category, severity, scope, mature, ...)
|
||
```
|
||
|
||
Aggregation is computed by a SQL view / query that joins a content item's
|
||
transitive dependencies. Because the graph is a DAG, this is a bounded
|
||
recursive query (PostgreSQL `WITH RECURSIVE`).
|
||
|
||
---
|
||
|
||
## 4. Enforcement
|
||
|
||
Enforcement happens at three layers, in order:
|
||
|
||
### 4.1 Load-Time Enforcement (Server)
|
||
|
||
Before a client is allowed to **request** content, the server evaluates the
|
||
content's effective flag set against the requesting account's **content
|
||
preferences** and **age verification status**:
|
||
|
||
- If the content is `mature` and the account is **not age-verified**, the
|
||
request is **rejected** with a `403 MATURE_CONTENT_REQUIRES_VERIFICATION`.
|
||
- If the content's flags exceed the account's **content filter** (a per-account
|
||
severity threshold per category), the request is rejected or the offending
|
||
content is **stripped** (e.g., audio muted, mesh hidden, script not loaded).
|
||
|
||
### 4.2 Runtime Enforcement (Client)
|
||
|
||
The client receives a **manifest** of allowed content from the server. It never
|
||
loads content that is not in the manifest. This prevents a malicious or buggy
|
||
client from bypassing the server's filter by loading content directly.
|
||
|
||
- Assets not in the manifest are **not instantiated**.
|
||
- Scripts not in the manifest are **not executed**.
|
||
- Rooms not in the manifest are **not entered**.
|
||
|
||
### 4.3 Network Boundary Enforcement
|
||
|
||
The server **filters replication** based on flags:
|
||
|
||
- Mature or filtered content is **not replicated** to clients that cannot
|
||
receive it.
|
||
- A room's flag set is broadcast in the room metadata so clients can show a
|
||
warning before entry.
|
||
|
||
### 4.4 Enforcement Decision Flow
|
||
|
||
```
|
||
Client requests content C for account A
|
||
│
|
||
▼
|
||
Compute effective flags F(C) [union + max-severity aggregation]
|
||
│
|
||
▼
|
||
Is A age-verified? ──no──► Is F(C).mature? ──yes──► REJECT (403)
|
||
│ yes │ no
|
||
▼ ▼
|
||
Does F(C) exceed A's filter? ──► yes ──► REJECT or STRIP
|
||
│ no
|
||
▼
|
||
Serve content + manifest
|
||
```
|
||
|
||
---
|
||
|
||
## 5. Age Verification Flow
|
||
|
||
Age verification is a **two-tier** system:
|
||
|
||
- **Tier 1 — Self-declaration:** The user asserts their date of birth. This is
|
||
fast, requires no PII beyond a birthdate, and is sufficient for most content.
|
||
- **Tier 2 — ID check (optional hook):** A pluggable identity-verification
|
||
provider confirms the user is an adult. This raises the account's assurance
|
||
level and unlocks the highest maturity tier.
|
||
|
||
### 5.1 Verification States
|
||
|
||
An account has an **age verification status**:
|
||
|
||
| Status | Meaning |
|
||
|-------------------|----------------------------------------------------------------------|
|
||
| `unverified` | No declaration on file. |
|
||
| `self_declared` | User declared a birthdate; **not** independently confirmed. |
|
||
| `id_verified` | User passed the ID check hook; independently confirmed adult. |
|
||
| `rejected` | ID check failed or declaration was inconsistent. |
|
||
|
||
### 5.2 Tier 1 — Self-Declaration Flow
|
||
|
||
```
|
||
User submits { date_of_birth }
|
||
│
|
||
▼
|
||
Server validates format & plausibility (age 13–120, not in future)
|
||
│
|
||
▼
|
||
Compute age from DOB
|
||
│
|
||
▼
|
||
age >= 18 ──► status = self_declared (adult)
|
||
age >= 13 ──► status = self_declared (minor, restricted)
|
||
age < 13 ──► status = rejected (below minimum platform age)
|
||
```
|
||
|
||
- The DOB is stored **encrypted at rest** and is **never exposed** to other
|
||
players.
|
||
- A user may **upgrade** from `self_declared` to `id_verified` at any time via
|
||
Tier 2.
|
||
|
||
### 5.3 Tier 2 — ID Check Hook
|
||
|
||
The ID check is an **optional, pluggable hook**. The platform defines an
|
||
interface; concrete providers (e.g., a third-party KYC service) implement it.
|
||
|
||
```
|
||
User requests ID verification
|
||
│
|
||
▼
|
||
Server creates a verification session (idempotent, expiring)
|
||
│
|
||
▼
|
||
Server calls provider hook: verify(identity_token) -> { adult: bool, ref: string }
|
||
│
|
||
▼
|
||
adult == true ──► status = id_verified, store provider ref (not raw doc)
|
||
adult == false ─► status = rejected
|
||
provider error ──► status unchanged, retry allowed
|
||
```
|
||
|
||
**Privacy guarantees:**
|
||
|
||
- The platform **never stores** the raw identity document or image.
|
||
- Only the provider's **opaque reference** and the boolean `adult` result are
|
||
persisted.
|
||
- The hook is **asynchronous and idempotent**; a session can be retried without
|
||
double-charging or double-verifying.
|
||
|
||
### 5.4 Hook Interface
|
||
|
||
The ID check hook is defined as a backend interface (FastAPI dependency /
|
||
abstract service):
|
||
|
||
```python
|
||
class IdentityVerificationProvider(Protocol):
|
||
def create_session(self, user_id: str) -> VerificationSession: ...
|
||
def verify(self, session_id: str, identity_token: str) -> VerificationResult: ...
|
||
|
||
class VerificationResult:
|
||
adult: bool
|
||
provider_ref: str | None
|
||
error: str | None
|
||
```
|
||
|
||
A default **no-op provider** returns `adult=False` with a clear error, so the
|
||
system runs without any third-party integration until one is configured.
|
||
|
||
### 5.5 Age Gate Enforcement
|
||
|
||
The age gate is enforced at **load time** (see §4.1):
|
||
|
||
| Verification status | Can access `mature` content? |
|
||
|---------------------|------------------------------|
|
||
| `unverified` | No |
|
||
| `self_declared` | Yes (adult) / No (minor) |
|
||
| `id_verified` | Yes |
|
||
| `rejected` | No |
|
||
|
||
---
|
||
|
||
## 6. Moderation Workflow
|
||
|
||
### 6.1 Flagging Sources
|
||
|
||
Content can be flagged by:
|
||
|
||
- **Author** (self-declared at upload).
|
||
- **Automated scanner** (hash-based detection, heuristic analysis).
|
||
- **Community** (players report content; reports become flags after review).
|
||
- **Moderator** (manual review and override).
|
||
|
||
### 6.2 Review Queue
|
||
|
||
Community and automated flags enter a **review queue**. A moderator either:
|
||
|
||
- **Approves** the flag (it becomes authoritative), or
|
||
- **Rejects** the flag (it is discarded), or
|
||
- **Escalates** to a senior moderator for severe/hate flags.
|
||
|
||
### 6.3 Actions
|
||
|
||
Once a flag is authoritative, a moderator can take actions:
|
||
|
||
| Action | Effect |
|
||
|-----------------|---------------------------------------------------------------|
|
||
| `warn` | Notify the author; content remains. |
|
||
| `restrict` | Content hidden from users below a filter threshold. |
|
||
| `age_gate` | Content marked `mature`; requires age verification. |
|
||
| `remove` | Content taken down; not served. |
|
||
| `ban_author` | Author's upload privileges suspended. |
|
||
|
||
### 6.4 Audit Log
|
||
|
||
Every flag and action is written to an **audit log**:
|
||
|
||
```
|
||
audit_event (id, content_id, actor_id, action, flag_id, reason, created_at)
|
||
```
|
||
|
||
This log is append-only and used for transparency and dispute resolution.
|
||
|
||
---
|
||
|
||
## 7. Client-Side Integration (Godot)
|
||
|
||
### 7.1 Content Manifest
|
||
|
||
The server sends each client a **content manifest** — the set of content the
|
||
client is permitted to load, derived from the account's verification status and
|
||
filter. The Godot client:
|
||
|
||
- Loads only manifest-listed assets.
|
||
- Executes only manifest-listed scripts.
|
||
- Shows a **maturity warning** dialog before entering a room whose effective
|
||
flags include `mature` content.
|
||
|
||
### 7.2 Flag Metadata on Content
|
||
|
||
Each content resource carries a `content_flags` metadata field (a Godot
|
||
`Dictionary` or custom `Resource`). The client uses this for local UI decisions
|
||
(e.g., showing a warning icon) but **never** as the authority for whether to
|
||
load — the server manifest is authoritative.
|
||
|
||
### 7.3 Age Verification UI
|
||
|
||
The client exposes an **age verification screen**:
|
||
|
||
- Tier 1: a date-of-birth picker.
|
||
- Tier 2: a button that opens the ID check provider flow (deep link / webview).
|
||
|
||
The client calls the backend endpoints (§8) and reflects the resulting status.
|
||
|
||
---
|
||
|
||
## 8. Backend API Surface
|
||
|
||
| Method | Path | Description |
|
||
|--------|-----------------------------------|------------------------------------------------|
|
||
| `POST` | `/v1/age/declare` | Submit self-declared DOB (Tier 1). |
|
||
| `POST` | `/v1/age/verify/session` | Create an ID-check session (Tier 2). |
|
||
| `POST` | `/v1/age/verify/confirm` | Confirm an ID-check session with a token. |
|
||
| `GET` | `/v1/age/status` | Return the account's verification status. |
|
||
| `GET` | `/v1/content/{id}/flags` | Return a content item's effective flags. |
|
||
| `POST` | `/v1/content/{id}/flags` | Add a flag (author/community). |
|
||
| `POST` | `/v1/moderation/review/{flag_id}` | Moderator approve/reject/escalate. |
|
||
| `POST` | `/v1/moderation/action` | Apply a moderation action. |
|
||
| `GET` | `/v1/manifest` | Return the client's permitted content manifest.|
|
||
|
||
All endpoints require a valid **JWT** (matching the existing auth stack). The
|
||
age endpoints additionally require the account to be in a non-`rejected` state.
|
||
|
||
---
|
||
|
||
## 9. Security & Privacy Considerations
|
||
|
||
- **DOB encryption:** Self-declared birthdates are encrypted at rest; only the
|
||
derived age and verification status are used in authorization decisions.
|
||
- **No raw identity storage:** The ID check hook persists only a provider
|
||
reference and a boolean result.
|
||
- **Server authority:** The client manifest prevents bypass; the client never
|
||
trusts its own flag metadata for access decisions.
|
||
- **Rate limiting:** Age-declaration and ID-check endpoints are rate-limited to
|
||
prevent abuse and enumeration.
|
||
- **Auditability:** All flag and verification events are logged append-only.
|
||
|
||
---
|
||
|
||
## 10. Open Questions / Future Work
|
||
|
||
- **Regional age thresholds:** Different jurisdictions may require different
|
||
minimum ages; the threshold should be configurable per region.
|
||
- **Parental controls:** A future feature may let a parent set a content filter
|
||
on a minor's account.
|
||
- **Automated scanner accuracy:** The heuristic scanner may produce false
|
||
positives; a confidence score should be attached to automated flags.
|
||
- **Offline / local content:** How flags apply to locally-authored content that
|
||
is never uploaded remains to be defined.
|