modbison/docs/planning/trust_safety.md

Trust & Safety Operations — Plan

Status: Planned Owner: Platform team Feature ID: trust_safety Godot version: 4.6.2 stable Backend: FastAPI + SQLAlchemy (async) + PostgreSQL (+ Redis available, unused) Related: content_moderation.md (M5 — content rating/flagging at upload time), social.md (chat + report-message), app/dependencies.py (JWT auth)

1. Overview

This feature is the enforcement workflow for content/behavior that is already live and has been reported by a real user. M5 (content_moderation.md) covers rating/flagging at authoring/upload time; this feature covers what happens after something is live and someone reports it:

1. Reporting — report a user, an experience (world/asset), or a chat message with a reason and optional detail.
2. Moderation queue — staff review incoming reports, see report + target context, and resolve them.
3. Ban / suspension — apply a temporary suspension or permanent ban to a user, with enforcement at the auth boundary.
4. Appeals — a sanctioned user can appeal; staff can uphold or overturn.

The userbase being 18+ does not shrink the moderation need — harassment, hate speech, spam, griefing, and CSAM/policy violations still require real enforcement. Safety is a blocking requirement, not an add-on.

2. Existing constraints (do not re-build)

• Auth: JWT bearer; get_current_user / CurrentUser in app/dependencies.py. Every new endpoint reuses it.
• DB: async SQLAlchemy + Alembic. Current head is 0002_add_date_of_birth_to_users.
• Models: User, World (owner_id), Asset (owner_id, is_public). There is no role field yet, and no chat/message table yet (social.md is planned, not built).
• Redis: configured (REDIS_URL) but unused; not required by this feature.
• Client: Godot 4.6.2, minimal main_menu.gd; no report/chat UI yet.

3. Key design decisions

Concern	Decision
Report target addressing	Generic target_type + target_id (string) so user/world/asset/message all flow through one report path; router resolves target existence via a small registry
Report reasons	Fixed enum reason_category (harassment, hate_speech, sexual_content, violence, spam, self_harm, other) + free-text details
Staff roles	Add role enum on User (user / moderator / admin); RequireStaff dependency gates the queue/action/appeal-review endpoints
Enforcement	Denormalized banned_until / suspended_until timestamps on User (kept in sync by moderation actions); get_current_user rejects sanctioned users with 403 so enforcement applies to all existing endpoints
Ban vs. suspension	Suspension = temporary (expires_at); ban = permanent (expires_at IS NULL). Both are rows in moderation_actions for auditability
Appeals	Appeal references the specific moderation_action; approving an appeal revokes the action and clears the user's sanction
Audit	Every moderation action is an append-only row with moderator_id, reason, created_at, revoked_at; reports record resolved_by + resolution

4. Data model (target shape)

users
  + role             ENUM(user, moderator, admin)  default 'user'
  + banned_until     TIMESTAMPTZ  NULL
  + suspended_until  TIMESTAMPTZ  NULL

reports
  id, reporter_id FK users, target_type ENUM(user,world,asset,chat_message),
  target_id VARCHAR(64), reason_category ENUM(...), details TEXT,
  status ENUM(open,in_review,resolved,dismissed) default 'open',
  resolved_by FK users NULL, resolution ENUM(...) NULL,
  created_at, updated_at, resolved_at

moderation_actions
  id, moderator_id FK users, target_user_id FK users,
  target_type ENUM(user,world,asset,chat_message), target_id VARCHAR(64),
  action ENUM(warn,suspend,ban,restrict,remove,restore),
  reason TEXT, report_id FK reports NULL,
  expires_at TIMESTAMPTZ NULL, revoked_at TIMESTAMPTZ NULL, created_at

appeals
  id, moderation_action_id FK moderation_actions,
  user_id FK users, reason TEXT,
  status ENUM(pending,approved,rejected) default 'pending',
  reviewed_by FK users NULL, reviewed_at TIMESTAMPTZ NULL, created_at

5. User stories

US-1 Reporting

As a player, I can report a user, an experience (world/asset), or a chat message, choosing a reason and adding optional detail.

Acceptance criteria

• Report requires a valid target of the matching type (404 otherwise) and a reason from the fixed category set.
• Duplicate reports from the same reporter for the same target are coalesced (409) rather than creating unbounded duplicates.
• A report cannot be filed by a banned/suspended user (enforced by existing auth).
• Each report stores reporter, target, reason, and created_at; status starts open.

US-2 Moderation queue & review

As a moderator, I can list open reports (filterable by type/status/reason), view a report with its target, and resolve it with an action.

Acceptance criteria

• Queue is staff-only (RequireStaff); non-staff get 403.
• List supports status/type/reason filters + pagination (offset/limit).
• Resolving sets status, resolution, resolved_by, resolved_at and (where applicable) records a moderation_actions row.

US-3 Ban / suspension enforcement

As a moderator, I can suspend a user temporarily or ban them permanently, and the sanction blocks the user from authenticated endpoints immediately.

Acceptance criteria

• Suspension sets suspended_until; ban sets banned_until (permanent). Warn records an action without blocking.
• get_current_user returns 403 for any user past a live sanction.
• Expired suspensions no longer block (time-based, no cron required).
• Removing a ban/suspension is itself a recorded restore action.

US-4 Appeals

As a sanctioned user, I can appeal an active ban/suspension; as a moderator I can approve (overturn) or reject (uphold) it.

Acceptance criteria

• A user can submit exactly one open appeal per sanction.
• Appeal is tied to the specific moderation_action.
• Approving revokes the action and clears the user's sanction; rejecting upholds it. Both record reviewer + timestamp.

6. Task breakdown

Right-sized to ~1–3 files / one coherent piece of functionality. Task IDs are dependency order.

#	Task	Story	Depends on
1	Moderation data model + Alembic migration (reports, moderation_actions, appeals, user role/banned_until/suspended_until)	US-1..4	—
2	Staff authorization + sanction enforcement dependency (RequireStaff; extend get_current_user to reject sanctioned users)	US-2, US-3	1
3	Report submission endpoints (report user / experience / chat message)	US-1	1
4	Moderation queue + review endpoints (list, detail, resolve→action)	US-2	1, 2
5	Ban/suspend/restore action application + user-sanction sync	US-3	2, 4
6	Appeals endpoints (submit, list, review → uphold/overturn)	US-4	1, 5
7	Integration tests: report → review → ban → appeal → overturn	US-1..4	3, 4, 5, 6

7. Phasing

• Phase 1 (data + enforcement core): tasks 1–2. Land the schema and make sanctioned users actually unable to act — highest value.
• Phase 2 (report → review → action): tasks 3–5. The moderation loop.
• Phase 3 (appeals): task 6.
• Phase 4 (verification): task 7 integration tests.

8. Out of scope for this feature

• Client (Godot) report/appeal UI — no chat/social UI exists yet; report endpoints are consumed once social.md US-6 lands. Client is a follow-up.
• Automated content scanning / hash detection (covered by M5 / content_moderation.md).
• Third-party moderation/CSAM-scanning vendors.
• Rate limiting of report submission (Redis is available; add a token-bucket dependency later if abuse is observed).
• Voice chat moderation.
• Per-jurisdiction legal-takedown workflows (DMCA etc.).