Skip to content

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

pbr-pixelart

Generate full PBR material maps — normal, height, and packed ORM (AO / Roughness / Metallic) — from tiny pixel-art diffuse textures, automatically and in bulk.

It pairs a vision LLM (for what a material is) with procedural image math (for the geometry maps). Built for and battle-tested on a 6,000+ tile 32×32 Minecraft-style pack, but works on any pixel-art textures.

diffuse.png  ──►  diffuse_n.png   (normal)
                  diffuse_h.png   (height)
                  diffuse_orm.png (R=AO  G=Roughness  B=Metallic)

Examples

PBR maps generated from synthetic pixel-art tiles

Columns: input diffuse → generated normal, height, ORM (R=AO, G=Roughness, B=Metallic). Look at iron_ore: the magenta specks in ORM are per-pixel metallic detected on the embedded ore, while the surrounding stone stays non-metal. metal_panel and gold_block read fully metallic; brick, planks and cobble stay dielectric with material-appropriate roughness; brick/plank/cobble seams show up as grooves in height & normal. These tiles are 100% original (generated by examples/make_examples.py, no third-party packs) — regenerate the whole figure with:

python examples/make_examples.py                          # 1. synthetic diffuse tiles
python pbr_pixelart.py examples/diffuse -o examples/out    # 2. PBR maps (needs a vision endpoint)
python examples/make_examples.py --showcase               # 3. rebuild docs/showcase.png

Why this approach

A vision model can't reliably segment a 32×32 image pixel-by-pixel, and it can't read roughness/metallic from colour alone (a grey pixel could be polished steel or rough concrete). So the work is split:

  • Ground truth (deterministic): each tile is quantized to a small palette; per-pixel material is a palette-index lookup. This is exact and gives true per-pixel roughness/metallic — so iron specks embedded in stone come out metallic while the stone stays dielectric.
  • Semantics (the LLM's only job): it names the material of each palette colour and returns a few structural hints (pattern, grain direction, relief, recessed vs raised edges, notable features).
  • Geometry (pure math): height = luminance + seam-carving from palette boundaries + feature embossing; normal = wrap-around gradient of height (seamless tiling); AO = cavity from height.

Result: mortar lines become real grooves, rivets become bumps, and metals/roughness are correct — without asking the model to do anything it's bad at.

Install

pip install -r requirements.txt
# or as a CLI:
pip install -e .

You also need an OpenAI-compatible chat endpoint that accepts images. Any of these work: LM Studio, Ollama, llama.cpp server, vLLM, OpenRouter, OpenAI… with a vision model loaded (e.g. Qwen2.5-VL / Qwen3-VL, MiniCPM-V, Llava, Pixtral, GPT-4o).

Quick start

python pbr_pixelart.py "path/to/diffuse_tiles" -o out

# explicit endpoint + model:
python pbr_pixelart.py tiles -o out \
    --api-url http://localhost:1234/v1/chat/completions \
    --model qwen2.5-vl-7b

# no GPU / no model? colour-heuristic only (no LLM):
python pbr_pixelart.py tiles -o out --no-ai

# hosted endpoint:
python pbr_pixelart.py tiles -o out \
    --api-url https://api.openai.com/v1/chat/completions \
    --model gpt-4o --api-key $OPENAI_API_KEY

--model auto (the default) queries the endpoint and picks the first served vision-looking model.

Output

Three PNGs per input tile, named <tile>_n.png, <tile>_h.png, <tile>_orm.png, at the source resolution. ORM packing is R=AmbientOcclusion, G=Roughness, B=Metallic (Unreal/glTF-style). Point-sample (nearest) in your engine to keep the pixel-art crispness.

Also written: out/_materials_manifest.json — the per-tile classifications, editable.

Robust by design

  • Resumable / crash-safe: skips any tile whose 3 outputs already exist; the manifest is written atomically. Re-run the same command after any interruption and it picks up where it stopped.
  • Editable results: fix a wrong material in _materials_manifest.json, re-run → only that tile's maps regenerate, no LLM call.
  • Content dedup: identical tiles are classified once.
  • Filename prior: standardized pack names (oreIron, planks_acacia, …) correct the LLM's occasional misreads, while deferring on ore* names so embedded specks are preserved.

Key options

Flag Default Purpose
--api-url http://localhost:1234/v1/chat/completions endpoint (env PBR_API_URL)
--model auto model id or auto (env PBR_MODEL)
--api-key — bearer token for hosted endpoints (env PBR_API_KEY)
--no-ai off skip the LLM, colour heuristics only
--strength 1.5 global normal-relief multiplier
--palette-colors 10 max perceptual clusters per tile
--dx off DirectX normals (flip green)
--invert-height off treat bright pixels as low
--reset-every N 0 reload model every N calls to flush image cache (see below)
--force off recompute everything
--limit N — only first N files (testing)

LM Studio image-cache flush

LM Studio accumulates committed image memory across vision calls (~30 MB ceiling) and can eventually error. Use --reset-every 50 to reload the model periodically via the lms CLI. The command is configurable for other servers: --reset-cmd 'your reload {model} command'.

Tuning the look

Edit the tables at the top of pbr_pixelart.py:

  • MATERIALS — roughness/metallic/relief/AO per material. This is where the look lives.
  • FILENAME_HINTS — map your pack's naming conventions to base materials.

Optional: diffusion normals (StableNormal)

There's an experimental --normals stablenormal backend using StableNormal:

pip install -r requirements-diffusion.txt   # needs a CUDA torch + ~several GB weights
python pbr_pixelart.py tiles -o out --normals stablenormal

Honest finding: at 32×32 the diffusion model flattens seams and per-pixel detail — the default procedural normals are crisper and better for pixel art. The diffusion path only pays off on larger (512px+) textures. It's included for that case. Also: a diffusion normal model and a large local LLM will fight for VRAM — run them sequentially, not at once.

How it works (one diagram)

tile.png
  │  quantize → palette + index map (per-pixel)         [PIL median-cut]
  ├─► vision LLM: palette→material + structure JSON      [your endpoint]
  │      └─ filename prior corrects confused base labels
  ├─ roughness/metallic = MATERIALS[material] per pixel  ──────────────┐
  ├─ height = luminance + seam-carve(edges) + features                 │
  ├─ normal = wrap-around gradient(height)  (seamless)                 │
  └─ AO     = cavity(height)                                           │
                                                ORM = pack(AO, R, M) ◄─┘

License

MIT — see LICENSE. You own the maps you generate. If you use the optional StableNormal backend, check that model's license separately (its weights are not bundled).

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages