Metacritic Games

Authorized Metacritic game discovery, enrichment, and monitoring service.

Overview • Architecture • Getting Started • Configuration • API • Testing

Overview

Metacritic Games is a Python service that discovers games from Metacritic, stores canonical Game data in PostgreSQL, and exposes the catalog through a FastAPI backend with a React frontend.

The project is built as a modular monolith with three long-running processes:

• web serves the API and frontend.
• scheduler enqueues Discovery Runs on a fixed interval.
• worker claims queued work, processes new Games, and retries failed fetches.

Core Game data is stored independently from Enrichments. That means a Game can be saved successfully even when review summaries, description translation, or similarity embeddings are temporarily unavailable.

Important

This project is configured for authorized crawling only. The default user agent and settings explicitly assume that you have permission to fetch Metacritic content.

Features

• Discover new Game pages from Metacritic browse pages.
• Fetch canonical Game details, developer names, platform ratings, cover art, and source video links.
• Fetch user scores from the Metacritic backend API per platform.
• Fall back to Playwright when plain HTTP responses do not contain enough page content.
• Store durable Discovery Runs, job state, and retry state in PostgreSQL.
• Generate critic and user Review Summaries through an optional chat-model adapter.
• Translate Game descriptions to Russian through an optional translation adapter.
• Compute Similar Games with optional embeddings stored in pgvector.
• Serve a searchable catalog, Game detail pages, and a live processing monitor backed by SSE.

Architecture

Metacritic browse pages / Game pages / review APIs
                    |
                    v
    httpx + Beautiful Soup + Playwright fallback
                    |
                    v
             GameApplication
       discovery + retries + enrichments
                    |
        +-----------+-----------+
        |                       |
        v                       v
   PostgreSQL + pgvector   FastAPI + React
  Games, jobs, retries,    catalog, detail,
  runs, summaries,         monitor, SSE
  translations, embeddings

The main boundaries are:

• metacritic_games/metacritic_core.py: browsing and core page parsing.
• metacritic_games/metacritic.py: user scores, review fetching, cover extraction, and richer page handling.
• metacritic_games/application.py: Discovery Run orchestration, retry handling, enrichment, and catalog queries.
• metacritic_games/api/: HTTP API and frontend hosting.
• frontend/: React application for catalog, Game detail, and monitor pages.
• metacritic_games/database/: SQLAlchemy models for catalog, processing, jobs, and embeddings.

Processing Model

The service uses a Processing Day ledger and a durable retry queue.

1. The scheduler enqueues one discovery job per interval.
2. The worker claims the next queued discovery job.
3. The application discovers up to 20 Game candidates from the next browse page for the current Processing Day.
4. Each selected Game is reserved once for that day before fetch work starts.
5. Core Game data is saved atomically.
6. Failed Game fetches are moved to the retry queue with exponential backoff.
7. Review summaries, description translation, and embeddings run independently after core data is stored.

This follows the ADRs in docs/adr/:

• Core Game data is separate from Enrichments.
• Daily selection is separate from processing retries.
• PostgreSQL is the single durable store for both catalog data and background work state.

Getting Started

Prerequisites

• Python 3.11+
• Node.js 22+
• PostgreSQL 16+ with the vector extension if you want similarity embeddings
• Chromium for Playwright

Local Development

1. Install Python dependencies and the local CLI.

python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
pip install -e .
python -m playwright install chromium

2. Install frontend dependencies.

npm install

3. Create your local environment file.

cp .env.example .env

4. Build the frontend bundle served by FastAPI.

npm run build

5. Set MCG_DATABASE_URL in .env to match your local database.

For example, for a local PostgreSQL instance:

MCG_DATABASE_URL=postgresql+psycopg://metacritic:metacritic@127.0.0.1:5432/metacritic

6. Initialize the database schema.

metacritic-games init-db

7. Start the services in separate terminals.

metacritic-games web

metacritic-games scheduler

metacritic-games worker

The web UI is available at http://127.0.0.1:8000.

Note

init-db and normal app startup both create the SQLAlchemy schema automatically. On PostgreSQL, the application also enables CREATE EXTENSION IF NOT EXISTS vector before building the schema.

If you want live frontend development instead of serving the built bundle, run Vite separately and keep the backend on port 8000:

npm run dev

Docker Compose

The repository includes a single-node Compose setup with postgres, web, scheduler, and worker.

Before starting Compose, make sure .env contains the Compose database password:

MCG_DATABASE_URL=postgresql+psycopg://metacritic:12345@postgres:5432/metacritic

docker compose up --build

The Compose stack exposes the app on http://127.0.0.1:8000 and mounts ./logs into /data/logs.

Configuration

Settings are loaded from .env with the MCG_ prefix.

Variable	Default	Purpose
MCG_DATABASE_URL	postgresql+psycopg://metacritic:metacritic@postgres:5432/metacritic	SQLAlchemy connection string
MCG_TIMEZONE	Asia/Jerusalem	Service timezone for the Processing Day ledger
MCG_LOG_LEVEL	INFO	Root log level
MCG_LOG_TO_FILES	false	Enable rotating per-service log files
MCG_LOG_DIR	/data/logs	Directory for file logs
MCG_LOG_FILE_MAX_BYTES	10485760	Max size of one log file before rotation
MCG_LOG_FILE_BACKUP_COUNT	5	Number of rotated log files to keep
MCG_SCHEDULE_INTERVAL_SECONDS	3600	How often the scheduler enqueues a Discovery Run
MCG_WORKER_POLL_SECONDS	5	Worker idle polling interval
MCG_HEARTBEAT_SECONDS	10	Heartbeat interval for running discovery jobs
MCG_STALE_JOB_SECONDS	300	When running jobs or retries are considered stale
MCG_REQUEST_DELAY_SECONDS	1	Minimum delay between Metacritic requests
MCG_REQUEST_TIMEOUT_SECONDS	30	HTTP and Playwright request timeout
MCG_FETCH_WORKERS	3	Parallel fetch workers for selected Games
MCG_LLM_WORKERS	1	Thread-pool workers for LLM enrichments
MCG_METACRITIC_USER_AGENT	MetacriticGamesAgent/1.0 (authorized crawler; contact=owner)	User agent sent to Metacritic
MCG_METACRITIC_PERMISSION_REFERENCE	owner-authorized	Human-readable authorization reference
MCG_LLAMA_CHAT_URL	unset	Base URL for summary and translation requests
MCG_LLAMA_CHAT_MODEL	local-chat-model	Model ID stored with summaries and translations
MCG_LLAMA_REQUEST_TIMEOUT_SECONDS	600	Timeout for chat and embedding LLM requests
MCG_LLAMA_EMBEDDING_URL	unset	Base URL for embedding requests
MCG_LLAMA_EMBEDDING_MODEL	local-embedding-model	Model ID stored with embeddings

Optional LLM Features

Setting	Enables
MCG_LLAMA_CHAT_URL	Critic and user Review Summaries plus Russian description translation
MCG_LLAMA_EMBEDDING_URL	Similar Game ranking

Tip

Without LLM settings, the service still discovers Games, stores core catalog data, exposes the UI, and processes retries. The enrichment features simply stay empty or unavailable.

Current UI Localization

The README is in English, but the current Game detail experience is partly Russian by design:

• translated descriptions are returned in Russian when the translator is configured;
• summary section labels and action text on the Game page are currently Russian;
• the catalog and monitor pages are currently English.

API

Frontend Routes

Route	Purpose
/	Catalog page with search, platform filter, sorting, and pagination
/games/{slug}	Game detail page with ratings, summaries, similar Games, refresh actions, and source links
/monitor	Processing monitor with live queue and recent run state

HTTP Endpoints

Method	Path	Purpose
GET	/health	Liveness check
GET	/api/games	Catalog query with search, platform, sort, direction, and page
GET	/api/games/{slug}	Game detail, summaries, and similar Games
POST	/api/games/{slug}/refresh	Re-fetch and persist one Game
POST	/api/games/{slug}/summaries/regenerate	Rebuild review summaries for one Game
POST	/api/runs	Enqueue a manual Discovery Run
GET	/api/status	Poll current queue and run status
GET	/api/events	Server-sent status stream

Testing

Python Tests

.venv/bin/python -m pytest

The Python test suite covers:

• Metacritic parsing and request behavior
• retry handling and job heartbeats
• scheduler behavior
• settings and logging
• summary, translation, and similarity flows
• API and frontend-hosting behavior

Integration Tests

Integration tests require a live PostgreSQL database exposed through TEST_DATABASE_URL.

.venv/bin/python -m pytest -m integration

End-to-End Tests

The Playwright suite builds the frontend and boots tests.e2e_server:app on http://127.0.0.1:8765.

npm run test:e2e

Project Layout

.
├── frontend/                     # React source
├── metacritic_games/
│   ├── api/                      # FastAPI routes and service layer
│   ├── database/                 # SQLAlchemy models and DB core
│   ├── llm/                      # Summary, translation, and embedding adapters
│   ├── frontend_dist/            # Built frontend bundle served by FastAPI
│   ├── application.py            # Discovery, retries, enrichments, catalog queries
│   ├── browser.py                # Playwright HTML fallback
│   ├── cli.py                    # `web`, `scheduler`, `worker`, `init-db`
│   ├── config.py                 # Environment-backed settings
│   ├── factory.py                # Dependency wiring
│   ├── metacritic.py             # Metacritic integration
│   └── scheduler.py              # Interval-based job enqueueing
├── tests/                        # Unit, integration, and e2e tests
├── docs/adr/                     # Architectural decisions
├── docker-compose.yml            # Local multi-service stack
└── Dockerfile                    # Production-style image build

Development Notes

• The backend package ships the built frontend from metacritic_games/frontend_dist.
• npm run build must run again after frontend changes if you want FastAPI to serve the updated bundle.
• npm run dev serves the frontend from frontend/ and proxies /api and /static to http://127.0.0.1:8000.
• The worker reclaims stale running jobs and stale retries automatically.
• Similarity scoring combines cosine similarity with small developer and platform overlap bonuses.