From 8a63b10c73566a635538081ce41e7dc0825757e7 Mon Sep 17 00:00:00 2001 From: planner-bot Date: Tue, 18 Aug 2026 17:46:24 +0000 Subject: [PATCH] WIP checkpoint: Plan: Trust & Safety Operations --- docs/planning/trust_safety.md | 160 ++++++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 docs/planning/trust_safety.md diff --git a/docs/planning/trust_safety.md b/docs/planning/trust_safety.md new file mode 100644 index 0000000..91e16c9 --- /dev/null +++ b/docs/planning/trust_safety.md @@ -0,0 +1,160 @@ +# Trust & Safety Operations — Plan + +> **Status:** Planned +> **Owner:** Platform team +> **Feature ID:** `trust_safety` +> **Godot version:** 4.6.2 stable +> **Backend:** FastAPI + SQLAlchemy (async) + PostgreSQL (+ Redis available, unused) +> **Related:** `content_moderation.md` (M5 — content rating/flagging at upload time), +> `social.md` (chat + report-message), `app/dependencies.py` (JWT auth) + +## 1. Overview + +This feature is the **enforcement workflow** for content/behavior that is +**already live** and has been reported by a real user. M5 +(`content_moderation.md`) covers rating/flagging at authoring/upload time; this +feature covers what happens **after** something is live and someone reports it: + +1. **Reporting** — report a user, an experience (world/asset), or a chat + message with a reason and optional detail. +2. **Moderation queue** — staff review incoming reports, see report + target + context, and resolve them. +3. **Ban / suspension** — apply a temporary suspension or permanent ban to a + user, with enforcement at the auth boundary. +4. **Appeals** — a sanctioned user can appeal; staff can uphold or overturn. + +The userbase being 18+ does **not** shrink the moderation need — harassment, +hate speech, spam, griefing, and CSAM/policy violations still require real +enforcement. Safety is a **blocking requirement**, not an add-on. + +## 2. Existing constraints (do not re-build) + +- **Auth:** JWT bearer; `get_current_user` / `CurrentUser` in + `app/dependencies.py`. Every new endpoint reuses it. +- **DB:** async SQLAlchemy + Alembic. Current head is `0002_add_date_of_birth_to_users`. +- **Models:** `User`, `World` (owner_id), `Asset` (owner_id, is_public). There + is no `role` field yet, and no chat/message table yet (social.md is planned, + not built). +- **Redis:** configured (`REDIS_URL`) but unused; not required by this feature. +- **Client:** Godot 4.6.2, minimal `main_menu.gd`; no report/chat UI yet. + +## 3. Key design decisions + +| Concern | Decision | +|---|---| +| Report target addressing | Generic `target_type` + `target_id` (string) so user/world/asset/message all flow through one report path; router resolves target existence via a small registry | +| Report reasons | Fixed enum `reason_category` (harassment, hate_speech, sexual_content, violence, spam, self_harm, other) + free-text `details` | +| Staff roles | Add `role` enum on `User` (`user` / `moderator` / `admin`); `RequireStaff` dependency gates the queue/action/appeal-review endpoints | +| Enforcement | Denormalized `banned_until` / `suspended_until` timestamps on `User` (kept in sync by moderation actions); `get_current_user` rejects sanctioned users with `403` so enforcement applies to **all** existing endpoints | +| Ban vs. suspension | Suspension = temporary (`expires_at`); ban = permanent (`expires_at IS NULL`). Both are rows in `moderation_actions` for auditability | +| Appeals | Appeal references the specific `moderation_action`; approving an appeal revokes the action and clears the user's sanction | +| Audit | Every moderation action is an append-only row with `moderator_id`, `reason`, `created_at`, `revoked_at`; reports record `resolved_by` + `resolution` | + +## 4. Data model (target shape) + +``` +users + + role ENUM(user, moderator, admin) default 'user' + + banned_until TIMESTAMPTZ NULL + + suspended_until TIMESTAMPTZ NULL + +reports + id, reporter_id FK users, target_type ENUM(user,world,asset,chat_message), + target_id VARCHAR(64), reason_category ENUM(...), details TEXT, + status ENUM(open,in_review,resolved,dismissed) default 'open', + resolved_by FK users NULL, resolution ENUM(...) NULL, + created_at, updated_at, resolved_at + +moderation_actions + id, moderator_id FK users, target_user_id FK users, + target_type ENUM(user,world,asset,chat_message), target_id VARCHAR(64), + action ENUM(warn,suspend,ban,restrict,remove,restore), + reason TEXT, report_id FK reports NULL, + expires_at TIMESTAMPTZ NULL, revoked_at TIMESTAMPTZ NULL, created_at + +appeals + id, moderation_action_id FK moderation_actions, + user_id FK users, reason TEXT, + status ENUM(pending,approved,rejected) default 'pending', + reviewed_by FK users NULL, reviewed_at TIMESTAMPTZ NULL, created_at +``` + +## 5. User stories + +### US-1 Reporting +As a player, I can report a user, an experience (world/asset), or a chat +message, choosing a reason and adding optional detail. + +**Acceptance criteria** +- Report requires a valid target of the matching type (404 otherwise) and a + reason from the fixed category set. +- Duplicate reports from the same reporter for the same target are coalesced + (409) rather than creating unbounded duplicates. +- A report cannot be filed by a banned/suspended user (enforced by existing auth). +- Each report stores reporter, target, reason, and created_at; status starts `open`. + +### US-2 Moderation queue & review +As a moderator, I can list open reports (filterable by type/status/reason), +view a report with its target, and resolve it with an action. + +**Acceptance criteria** +- Queue is staff-only (`RequireStaff`); non-staff get 403. +- List supports status/type/reason filters + pagination (offset/limit). +- Resolving sets `status`, `resolution`, `resolved_by`, `resolved_at` and + (where applicable) records a `moderation_actions` row. + +### US-3 Ban / suspension enforcement +As a moderator, I can suspend a user temporarily or ban them permanently, and +the sanction blocks the user from authenticated endpoints immediately. + +**Acceptance criteria** +- Suspension sets `suspended_until`; ban sets `banned_until` (permanent). Warn + records an action without blocking. +- `get_current_user` returns `403` for any user past a live sanction. +- Expired suspensions no longer block (time-based, no cron required). +- Removing a ban/suspension is itself a recorded `restore` action. + +### US-4 Appeals +As a sanctioned user, I can appeal an active ban/suspension; as a moderator I +can approve (overturn) or reject (uphold) it. + +**Acceptance criteria** +- A user can submit exactly one open appeal per sanction. +- Appeal is tied to the specific `moderation_action`. +- Approving revokes the action and clears the user's sanction; rejecting + upholds it. Both record reviewer + timestamp. + +## 6. Task breakdown + +Right-sized to ~1–3 files / one coherent piece of functionality. Task IDs are +dependency order. + +| # | Task | Story | Depends on | +|---|------|-------|-----------| +| 1 | Moderation data model + Alembic migration (reports, moderation_actions, appeals, user `role`/`banned_until`/`suspended_until`) | US-1..4 | — | +| 2 | Staff authorization + sanction enforcement dependency (`RequireStaff`; extend `get_current_user` to reject sanctioned users) | US-2, US-3 | 1 | +| 3 | Report submission endpoints (report user / experience / chat message) | US-1 | 1 | +| 4 | Moderation queue + review endpoints (list, detail, resolve→action) | US-2 | 1, 2 | +| 5 | Ban/suspend/restore action application + user-sanction sync | US-3 | 2, 4 | +| 6 | Appeals endpoints (submit, list, review → uphold/overturn) | US-4 | 1, 5 | +| 7 | Integration tests: report → review → ban → appeal → overturn | US-1..4 | 3, 4, 5, 6 | + +## 7. Phasing + +- **Phase 1 (data + enforcement core):** tasks 1–2. Land the schema and make + sanctioned users actually unable to act — highest value. +- **Phase 2 (report → review → action):** tasks 3–5. The moderation loop. +- **Phase 3 (appeals):** task 6. +- **Phase 4 (verification):** task 7 integration tests. + +## 8. Out of scope for this feature + +- **Client (Godot) report/appeal UI** — no chat/social UI exists yet; report + endpoints are consumed once `social.md` US-6 lands. Client is a follow-up. +- Automated content scanning / hash detection (covered by M5 / + `content_moderation.md`). +- Third-party moderation/CSAM-scanning vendors. +- Rate limiting of report submission (Redis is available; add a token-bucket + dependency later if abuse is observed). +- Voice chat moderation. +- Per-jurisdiction legal-takedown workflows (DMCA etc.).