Generate full PBR material maps — normal, height, and packed ORM (AO / Roughness / Metallic) — from tiny pixel-art diffuse textures, automatically and in bulk.
It pairs a vision LLM (for what a material is) with procedural image math (for the geometry maps). Built for and battle-tested on a 6,000+ tile 32×32 Minecraft-style pack, but works on any pixel-art textures.
diffuse.png ──► diffuse_n.png (normal)
diffuse_h.png (height)
diffuse_orm.png (R=AO G=Roughness B=Metallic)
Columns: input diffuse → generated normal, height, ORM (R=AO, G=Roughness, B=Metallic).
Look at iron_ore: the magenta specks in ORM are per-pixel metallic detected on the embedded ore,
while the surrounding stone stays non-metal. metal_panel and gold_block read fully metallic; brick,
planks and cobble stay dielectric with material-appropriate roughness; brick/plank/cobble seams show up as
grooves in height & normal. These tiles are 100% original (generated by examples/make_examples.py, no
third-party packs) — regenerate the whole figure with:
python examples/make_examples.py # 1. synthetic diffuse tiles
python pbr_pixelart.py examples/diffuse -o examples/out # 2. PBR maps (needs a vision endpoint)
python examples/make_examples.py --showcase # 3. rebuild docs/showcase.pngA vision model can't reliably segment a 32×32 image pixel-by-pixel, and it can't read roughness/metallic from colour alone (a grey pixel could be polished steel or rough concrete). So the work is split:
- Ground truth (deterministic): each tile is quantized to a small palette; per-pixel material is a palette-index lookup. This is exact and gives true per-pixel roughness/metallic — so iron specks embedded in stone come out metallic while the stone stays dielectric.
- Semantics (the LLM's only job): it names the material of each palette colour and returns a few structural hints (pattern, grain direction, relief, recessed vs raised edges, notable features).
- Geometry (pure math): height = luminance + seam-carving from palette boundaries + feature embossing; normal = wrap-around gradient of height (seamless tiling); AO = cavity from height.
Result: mortar lines become real grooves, rivets become bumps, and metals/roughness are correct — without asking the model to do anything it's bad at.
pip install -r requirements.txt
# or as a CLI:
pip install -e .You also need an OpenAI-compatible chat endpoint that accepts images. Any of these work: LM Studio, Ollama, llama.cpp server, vLLM, OpenRouter, OpenAI… with a vision model loaded (e.g. Qwen2.5-VL / Qwen3-VL, MiniCPM-V, Llava, Pixtral, GPT-4o).
python pbr_pixelart.py "path/to/diffuse_tiles" -o out
# explicit endpoint + model:
python pbr_pixelart.py tiles -o out \
--api-url http://localhost:1234/v1/chat/completions \
--model qwen2.5-vl-7b
# no GPU / no model? colour-heuristic only (no LLM):
python pbr_pixelart.py tiles -o out --no-ai
# hosted endpoint:
python pbr_pixelart.py tiles -o out \
--api-url https://api.openai.com/v1/chat/completions \
--model gpt-4o --api-key $OPENAI_API_KEY--model auto (the default) queries the endpoint and picks the first served vision-looking model.
Three PNGs per input tile, named <tile>_n.png, <tile>_h.png, <tile>_orm.png, at the source
resolution. ORM packing is R=AmbientOcclusion, G=Roughness, B=Metallic (Unreal/glTF-style).
Point-sample (nearest) in your engine to keep the pixel-art crispness.
Also written: out/_materials_manifest.json — the per-tile classifications, editable.
- Resumable / crash-safe: skips any tile whose 3 outputs already exist; the manifest is written atomically. Re-run the same command after any interruption and it picks up where it stopped.
- Editable results: fix a wrong material in
_materials_manifest.json, re-run → only that tile's maps regenerate, no LLM call. - Content dedup: identical tiles are classified once.
- Filename prior: standardized pack names (
oreIron,planks_acacia, …) correct the LLM's occasional misreads, while deferring onore*names so embedded specks are preserved.
| Flag | Default | Purpose |
|---|---|---|
--api-url |
http://localhost:1234/v1/chat/completions |
endpoint (env PBR_API_URL) |
--model |
auto |
model id or auto (env PBR_MODEL) |
--api-key |
— | bearer token for hosted endpoints (env PBR_API_KEY) |
--no-ai |
off | skip the LLM, colour heuristics only |
--strength |
1.5 |
global normal-relief multiplier |
--palette-colors |
10 |
max perceptual clusters per tile |
--dx |
off | DirectX normals (flip green) |
--invert-height |
off | treat bright pixels as low |
--reset-every N |
0 |
reload model every N calls to flush image cache (see below) |
--force |
off | recompute everything |
--limit N |
— | only first N files (testing) |
LM Studio accumulates committed image memory across vision calls (~30 MB ceiling) and can eventually
error. Use --reset-every 50 to reload the model periodically via the lms CLI. The command is
configurable for other servers: --reset-cmd 'your reload {model} command'.
Edit the tables at the top of pbr_pixelart.py:
MATERIALS— roughness/metallic/relief/AO per material. This is where the look lives.FILENAME_HINTS— map your pack's naming conventions to base materials.
There's an experimental --normals stablenormal backend using
StableNormal:
pip install -r requirements-diffusion.txt # needs a CUDA torch + ~several GB weights
python pbr_pixelart.py tiles -o out --normals stablenormalHonest finding: at 32×32 the diffusion model flattens seams and per-pixel detail — the default procedural normals are crisper and better for pixel art. The diffusion path only pays off on larger (512px+) textures. It's included for that case. Also: a diffusion normal model and a large local LLM will fight for VRAM — run them sequentially, not at once.
tile.png
│ quantize → palette + index map (per-pixel) [PIL median-cut]
├─► vision LLM: palette→material + structure JSON [your endpoint]
│ └─ filename prior corrects confused base labels
├─ roughness/metallic = MATERIALS[material] per pixel ──────────────┐
├─ height = luminance + seam-carve(edges) + features │
├─ normal = wrap-around gradient(height) (seamless) │
└─ AO = cavity(height) │
ORM = pack(AO, R, M) ◄─┘
MIT — see LICENSE. You own the maps you generate. If you use the optional StableNormal backend, check that model's license separately (its weights are not bundled).