modbison/docs/architecture/content_moderation.md

Content Moderation & Age Verification Architecture

Status: Proposed Owner: Platform team Related: Server authority + client prediction, avatar component system, backend auth (JWT)

1. Overview

This document specifies the content moderation and age verification architecture for agp-game, a Godot-based multiplayer game built by autonomous agents. It defines:

• How content flags propagate through the system across the three primary content types: assets, rooms, and scripts.
• How flags are evaluated, aggregated, and enforced at load time, at runtime, and at the network boundary.
• The age verification flow, combining a lightweight self-declaration with an optional ID check hook for higher-assurance verification.

The goal is a single, consistent moderation pipeline that applies the same rules to user-generated content (UGC) regardless of whether it is a 3D asset, a room definition, or a script, and that gates access to mature content behind a verifiable age gate.

1.1 Design Goals

• Uniformity: One flag vocabulary and one propagation mechanism across assets, rooms, and scripts.
• Composability: Flags on a room aggregate the flags of the assets and scripts it contains; flags on a world aggregate the flags of its rooms.
• Enforcement at the boundary: The server is authoritative for what content a client may load or receive, mirroring the server-authoritative networking model.
• Privacy-preserving age checks: Self-declaration requires no PII; the ID check hook is optional, pluggable, and never stores raw identity documents.
• Auditability: Every flag decision is logged with a reason, an actor, and a timestamp so moderation actions are reviewable.

---

2. Content Flag Vocabulary

All content carries a set of content flags. A flag is a typed, immutable label with a severity and a scope. Flags are the atomic unit of moderation; everything else (aggregation, enforcement, gating) operates on flags.

2.1 Flag Definition

A flag is a structured record:

Field	Type	Description
id	string	Stable machine-readable identifier, e.g. violence.gore.
label	string	Human-readable name, e.g. "Graphic Gore".
category	enum	One of violence, sexual, language, substance, gambling, horror, hate, custom.
severity	enum	One of none, mild, moderate, severe.
scope	enum	One of visual, audio, text, behavior, interaction.
mature	bool	Whether this flag triggers the age gate (see §5).
reason	string	Free-text justification supplied by the flagger.
source	enum	One of author, moderator, automated, community.

2.2 Canonical Flag Set

The platform defines a canonical registry of flags so that different content types and different flaggers use the same vocabulary. Examples:

id	category	severity	scope	mature
violence.none	violence	none	visual	false
violence.mild	violence	mild	visual	false
violence.gore	violence	severe	visual	true
sexual.suggestive	sexual	mild	visual	true
sexual.explicit	sexual	severe	visual	true
language.mild	language	mild	text	false
language.profanity	language	moderate	text	false
substance.references	substance	mild	text	false
gambling.simulated	gambling	moderate	behavior	true
horror.jumpscare	horror	mild	audio/visual	false
hate.hateful	hate	severe	text	true

The registry is data-driven (a JSON/YAML manifest shipped with the server) so new flags can be added without code changes.

---

3. Flag Propagation Model

Flags propagate through the system along the content dependency graph. The three content types form a hierarchy:

World
 └── Room
      ├── Asset  (mesh, texture, audio, animation, material)
      └── Script (behavior, dialogue, event logic)

3.1 Propagation Rules

1. Explicit flags are set directly on a content item by its author or a moderator.
2. Inherited flags flow upward: a room's effective flag set is the union of its own explicit flags and the flags of every asset and script it references.
3. Aggregation uses the maximum severity per category across all contributing items. A room containing one violence.gore asset and one violence.none asset is violence.severe.
4. Maturity is monotonic: if any contributing item is mature, the containing room and world are mature.
5. Cycles are impossible by construction: content references form a DAG (an asset cannot contain a room; a room cannot contain itself).

3.2 Propagation Through the Pipeline

Flags are attached at authoring time and carried through every stage:

Authoring ──► Upload ──► Validation ──► Storage ──► Distribution ──► Runtime
   │            │            │             │             │              │
   │ flag       │ flag       │ automated   │ flag        │ flag         │ flag
   │ declared   │ preserved  │ scan adds   │ indexed     │ re-checked   │ enforced
   │            │            │ flags       │             │              │

Stage	Responsibility
Authoring	Author declares flags in the editor/upload UI.
Upload	Flags are validated against the canonical registry; unknown flags rejected.
Validation	Automated scanner (hash-based + heuristic) adds flags for detected content.
Storage	Flags are stored alongside the content record and indexed for queries.
Distribution	Flags are re-checked against the current registry before serving.
Runtime	The client enforces flags before loading content (see §4).

3.3 Data Model

Flags are stored as a normalized join table so they can be queried and aggregated efficiently:

content_item (id, type, owner_id, ...)
content_flag  (content_id, flag_id, source, reason, created_at, actor_id)
flag_registry (id, category, severity, scope, mature, ...)

Aggregation is computed by a SQL view / query that joins a content item's transitive dependencies. Because the graph is a DAG, this is a bounded recursive query (PostgreSQL WITH RECURSIVE).

---

4. Enforcement

Enforcement happens at three layers, in order:

4.1 Load-Time Enforcement (Server)

Before a client is allowed to request content, the server evaluates the content's effective flag set against the requesting account's content preferences and age verification status:

• If the content is mature and the account is not age-verified, the request is rejected with a 403 MATURE_CONTENT_REQUIRES_VERIFICATION.
• If the content's flags exceed the account's content filter (a per-account severity threshold per category), the request is rejected or the offending content is stripped (e.g., audio muted, mesh hidden, script not loaded).

4.2 Runtime Enforcement (Client)

The client receives a manifest of allowed content from the server. It never loads content that is not in the manifest. This prevents a malicious or buggy client from bypassing the server's filter by loading content directly.

• Assets not in the manifest are not instantiated.
• Scripts not in the manifest are not executed.
• Rooms not in the manifest are not entered.

4.3 Network Boundary Enforcement

The server filters replication based on flags:

• Mature or filtered content is not replicated to clients that cannot receive it.
• A room's flag set is broadcast in the room metadata so clients can show a warning before entry.

4.4 Enforcement Decision Flow

Client requests content C for account A
        │
        ▼
Compute effective flags F(C)   [union + max-severity aggregation]
        │
        ▼
Is A age-verified?  ──no──►  Is F(C).mature? ──yes──► REJECT (403)
        │ yes                     │ no
        ▼                         ▼
Does F(C) exceed A's filter? ──► yes ──► REJECT or STRIP
        │ no
        ▼
Serve content + manifest

---

5. Age Verification Flow

Age verification is a two-tier system:

• Tier 1 — Self-declaration: The user asserts their date of birth. This is fast, requires no PII beyond a birthdate, and is sufficient for most content.
• Tier 2 — ID check (optional hook): A pluggable identity-verification provider confirms the user is an adult. This raises the account's assurance level and unlocks the highest maturity tier.

5.1 Verification States

An account has an age verification status:

Status	Meaning
unverified	No declaration on file.
self_declared	User declared a birthdate; not independently confirmed.
id_verified	User passed the ID check hook; independently confirmed adult.
rejected	ID check failed or declaration was inconsistent.

5.2 Tier 1 — Self-Declaration Flow

User submits { date_of_birth }
        │
        ▼
Server validates format & plausibility (age 13–120, not in future)
        │
        ▼
Compute age from DOB
        │
        ▼
age >= 18 ──► status = self_declared (adult)
age >= 13 ──► status = self_declared (minor, restricted)
age <  13 ──► status = rejected (below minimum platform age)

• The DOB is stored encrypted at rest and is never exposed to other players.
• A user may upgrade from self_declared to id_verified at any time via Tier 2.

5.3 Tier 2 — ID Check Hook

The ID check is an optional, pluggable hook. The platform defines an interface; concrete providers (e.g., a third-party KYC service) implement it.

User requests ID verification
        │
        ▼
Server creates a verification session (idempotent, expiring)
        │
        ▼
Server calls provider hook:  verify(identity_token) -> { adult: bool, ref: string }
        │
        ▼
adult == true ──► status = id_verified, store provider ref (not raw doc)
adult == false ─► status = rejected
provider error ──► status unchanged, retry allowed

Privacy guarantees:

• The platform never stores the raw identity document or image.
• Only the provider's opaque reference and the boolean adult result are persisted.
• The hook is asynchronous and idempotent; a session can be retried without double-charging or double-verifying.

5.4 Hook Interface

The ID check hook is defined as a backend interface (FastAPI dependency / abstract service):

class IdentityVerificationProvider(Protocol):
    def create_session(self, user_id: str) -> VerificationSession: ...
    def verify(self, session_id: str, identity_token: str) -> VerificationResult: ...

class VerificationResult:
    adult: bool
    provider_ref: str | None
    error: str | None

A default no-op provider returns adult=False with a clear error, so the system runs without any third-party integration until one is configured.

5.5 Age Gate Enforcement

The age gate is enforced at load time (see §4.1):

Verification status	Can access mature content?
unverified	No
self_declared	Yes (adult) / No (minor)
id_verified	Yes
rejected	No

---

6. Moderation Workflow

6.1 Flagging Sources

Content can be flagged by:

• Author (self-declared at upload).
• Automated scanner (hash-based detection, heuristic analysis).
• Community (players report content; reports become flags after review).
• Moderator (manual review and override).

6.2 Review Queue

Community and automated flags enter a review queue. A moderator either:

• Approves the flag (it becomes authoritative), or
• Rejects the flag (it is discarded), or
• Escalates to a senior moderator for severe/hate flags.

6.3 Actions

Once a flag is authoritative, a moderator can take actions:

Action	Effect
warn	Notify the author; content remains.
restrict	Content hidden from users below a filter threshold.
age_gate	Content marked mature; requires age verification.
remove	Content taken down; not served.
ban_author	Author's upload privileges suspended.

6.4 Audit Log

Every flag and action is written to an audit log:

audit_event (id, content_id, actor_id, action, flag_id, reason, created_at)

This log is append-only and used for transparency and dispute resolution.

---

7. Client-Side Integration (Godot)

7.1 Content Manifest

The server sends each client a content manifest — the set of content the client is permitted to load, derived from the account's verification status and filter. The Godot client:

• Loads only manifest-listed assets.
• Executes only manifest-listed scripts.
• Shows a maturity warning dialog before entering a room whose effective flags include mature content.

7.2 Flag Metadata on Content

Each content resource carries a content_flags metadata field (a Godot Dictionary or custom Resource). The client uses this for local UI decisions (e.g., showing a warning icon) but never as the authority for whether to load — the server manifest is authoritative.

7.3 Age Verification UI

The client exposes an age verification screen:

• Tier 1: a date-of-birth picker.
• Tier 2: a button that opens the ID check provider flow (deep link / webview).

The client calls the backend endpoints (§8) and reflects the resulting status.

---

8. Backend API Surface

Method	Path	Description
POST	/v1/age/declare	Submit self-declared DOB (Tier 1).
POST	/v1/age/verify/session	Create an ID-check session (Tier 2).
POST	/v1/age/verify/confirm	Confirm an ID-check session with a token.
GET	/v1/age/status	Return the account's verification status.
GET	/v1/content/{id}/flags	Return a content item's effective flags.
POST	/v1/content/{id}/flags	Add a flag (author/community).
POST	/v1/moderation/review/{flag_id}	Moderator approve/reject/escalate.
POST	/v1/moderation/action	Apply a moderation action.
GET	/v1/manifest	Return the client's permitted content manifest.

All endpoints require a valid JWT (matching the existing auth stack). The age endpoints additionally require the account to be in a non-rejected state.

---

9. Security & Privacy Considerations

• DOB encryption: Self-declared birthdates are encrypted at rest; only the derived age and verification status are used in authorization decisions.
• No raw identity storage: The ID check hook persists only a provider reference and a boolean result.
• Server authority: The client manifest prevents bypass; the client never trusts its own flag metadata for access decisions.
• Rate limiting: Age-declaration and ID-check endpoints are rate-limited to prevent abuse and enumeration.
• Auditability: All flag and verification events are logged append-only.

---

10. Open Questions / Future Work

• Regional age thresholds: Different jurisdictions may require different minimum ages; the threshold should be configurable per region.
• Parental controls: A future feature may let a parent set a content filter on a minor's account.
• Automated scanner accuracy: The heuristic scanner may produce false positives; a confidence score should be attached to automated flags.
• Offline / local content: How flags apply to locally-authored content that is never uploaded remains to be defined.