Skip to content

About

3D editor with AI capabilities

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Scenecraft AI

Scenecraft AI is an AI-powered 3D editor for scene creation workflows. It combines interactive 3D scene editing with AI image generation, AI-generated panorama environments, AI-assisted low-poly 3D sketching, AI-generated PBR texture atlases, AI-recommended Poly Haven asset kits, prompt enhancement, and material application in a single workspace, helping users move from an idea to an editable scene draft much faster.

The current version is built with Next.js, React 19, Three.js, Zustand, Drizzle ORM, and better-auth. It already includes a browser-based 3D editor shell, a protected editor workspace, AI image, AI PBR texture atlas, AI panorama, AI Poly Haven asset recommendation, and AI 3D preview-and-optimization workflows.

It also now includes a first end-to-end persistence loop for authenticated users:

  • manual scene save from the editor
  • project list and project re-open flow
  • project thumbnails captured from the active editor camera
  • persisted model / HDR / texture asset references through Vercel Blob
  • project-scoped AI image history

What This Project Is

This is not just a text-to-image demo, and it is not a full desktop-grade DCC tool in the browser either. It is a creative tool designed around the following loop:

  • Improve prompts with AI
  • Generate reference images or textures inside the editor
  • Generate PBR texture atlases and apply them to mesh or ground materials
  • Generate 360 panorama backgrounds and apply them to the scene environment
  • Ask AI to recommend matching Poly Haven HDRIs, PBR materials, and 3D props as scene kits
  • Apply generated results directly to mesh materials
  • Create previewable, editable low-poly 3D sketches from natural language
  • Continue refining the scene with lights, cameras, environment settings, post-processing, and manual editing

Current Features

1. Browser-Based 3D Editor

  • Real-time 3D viewport powered by Three.js
  • Scene tree management with selection, Shift-click multi-selection, deletion, duplication, locking, visibility toggles, and isolation
  • Basic editor shortcuts for undo/redo, delete/backspace, copy/paste, duplicate, clear selection, save, visibility, and locking
  • Property panel editing for transforms, MeshPhysicalMaterial controls, mesh and CSG mesh multi-selection CSG operations, lights, and scene settings
  • Built-in primitive creation such as box, sphere, cylinder, and torus
  • Model import support for gltf/glb, fbx, obj, and vrm
  • Model animation controls including play, pause, stop, step, and playback speed adjustment
  • Camera mode switching between bird view and first-person view
  • View helpers for grid, transform gizmo, light helpers, and shadows

2. Scene and Visual Controls

  • Panorama and HDR environment import
  • Local RoomEnvironment IBL fallback for neutral PBR lighting before a user HDRI is loaded
  • AI-generated 360 panorama environments applied directly to the selected scene
  • Poly Haven HDRI browsing from the editor top bar for authenticated users
  • Poly Haven texture browsing from the mesh material panel for authenticated users
  • Mesh and ground materials use MeshPhysicalMaterial, with common clearcoat/transmission controls in the property panel and advanced physical parameters in a dedicated material panel
  • AI-recommended Poly Haven asset kits with HDRI, ground material, and 3D prop suggestions
  • Temporary studio preview mode for selected models, meshes, and groups, with session-only editable studio room, plinth, and light entities, cylindrical plinth auto-fit, orbit camera inspection, and temporary model scale/rotation controls that do not persist into saved projects
  • Tone mapping and exposure controls
  • Light creation and editing for ambient, hemisphere, directional, point, spot, and rect area lights
  • Built-in lighting presets for faster scene setup
  • Post-processing controls with entry points for effects such as Bloom, SSR, GTAO, Bokeh, Film, Glitch, and Halftone
  • A neutral real-time preview baseline with ACES tone mapping, light GTAO contact shading, and restrained default effect parameters
  • Path trace render export with optional OpenRouter openai/gpt-5.4-image-2 image optimization; export screenshots are compressed below 700KB before optional model processing, and optimized downloads are normalized back to the original capture aspect ratio

3. AI Image Workflow

  • Supports both text-to-image and reference-image-based generation flows
  • Adjustable generation parameters such as seed, image size, CFG, and inference steps
  • Reference image upload for editing and style continuation workflows
  • Target-aware image prompt enhancement and shared Chinese-to-English prompt translation
  • Generated results can be previewed inside the editor and applied directly to mesh materials
  • Saved projects also retain a project-level AI image library, including prompts, generation params, reference images, and generated outputs

4. AI PBR Texture Atlas Workflow

  • Generates a single 3x2 PBR texture atlas with OpenRouter openai/gpt-5.4-image-2
  • Runs from the AI chat Texture mode and supports PBR atlas-aware prompt enhancement plus shared Chinese-to-English prompt translation
  • Supports selected mesh materials and the ground plane material
  • Applies generated atlases immediately to the selected mesh or ground material
  • Reuses the existing PBR material texture fields:
    • Diffuse Map
    • Metalness Map
    • Roughness Map
    • Normal Map
    • AO Map
    • Emissive Map
  • Stores the same atlas URL in all six map slots and uses per-map UV offset/repeat values to sample the correct atlas region
  • Shows texture thumbnails directly in material texture rows after application
  • Saves generated PBR atlases into the project AI asset library
  • Allows saved PBR atlas assets to be reused as a full material atlas on another mesh or ground plane

5. AI Panorama Workflow

  • Runs from the AI chat Panorama mode with panorama-aware prompt enhancement plus shared Chinese-to-English prompt translation
  • Uses OpenRouter openai/gpt-5.4-image-2 through the authenticated panorama API route
  • Uses an OpenRouter text intent step before image generation to preserve the user's subject while filling in missing atmosphere, weather, lighting, time-of-day, and style detail
  • Builds a seamless equirectangular 360-degree environment prompt on the server
  • Returns an AI-recommended environment patch and applies it automatically after the generated panorama is imported
  • Requests the provider-supported 21:9 panorama ratio, then center-crops and normalizes the applied file to 2048x1024 JPEG in the browser
  • Leaves provider resolution at its default to reduce upstream timeouts; the editor owns the final applied image size
  • Automatically selects the scene when Panorama mode is active
  • Applies the generated panorama through the same scene environment path as top bar panorama import
  • Shows the generated panorama in the Scene property panel Panorama section
  • Adds generated panoramas to the project AI Assets dialog with a Panorama badge
  • Registers the generated JPEG as an environment_image so project save can persist it through Vercel Blob

Prompt enhancement for Image, Texture, and Panorama modes uses one /api/ai/prompts/transform endpoint with target-specific optimization rules. Translation remains a shared prompt transform path across these modes.

All authenticated AI API routes are protected by per-user database-backed rate limits and per-route concurrency limits before they call OpenRouter or SiliconFlow. In local development without DATABASE_URL, the limiter is bypassed so the editor can still boot; in production, missing database configuration blocks paid AI routes instead of silently allowing unlimited provider calls.

6. AI Poly Haven Asset Recommendation

The AI chat Assets mode turns a scene description into a curated Poly Haven kit for authenticated users.

  • Uses /api/ai/assets/recommend to parse the prompt into search intent with OpenRouter
  • Searches Poly Haven HDRIs, texture sets, and models through the existing external asset provider
  • Uses a server-side keyword search provider, lightweight ranking, and route-level memory cache in lib/ai/asset-recommendation/
  • Returns multiple scene kit bundles with:
    • HDRI environment suggestions
    • ground PBR texture suggestions
    • selected mesh material suggestions when a mesh is selected
    • 3D model prop suggestions
  • Lets users apply selected items from a bundle in one action
  • Preserves Poly Haven externalSource attribution for recommended HDRIs, texture maps, and imported models
  • Does not use a database, IndexedDB, localStorage, embeddings, or vector search in v1

7. Project Save / Load and Asset Persistence

  • The editor Save action now creates and updates real user-owned projects
  • First save prompts for base metadata such as:
    • project name
    • description
    • tags
  • Every save captures the current camera view as a project thumbnail
  • Project > Select opens a saved-project list with thumbnail and last-updated metadata
  • Saved projects restore:
    • scene JSON
    • imported model URLs
    • environment image URLs
    • uploaded texture URLs
    • project AI image history
  • Imported model files, texture images, environment images, AI-generated panoramas, and generated thumbnails are stored in Vercel Blob

8. AI 3D Sketch Generation and Optimization

This is the most distinctive part of the project right now.

Instead of asking an AI system to output an opaque high-resolution 3D mesh directly, Scenecraft AI generates a validated low-poly construction plan that the editor can preview and apply as editable geometry. The current workflow includes:

  • Generating stylized low-poly 3D sketch previews from natural language
  • Supplying structured intent hints such as:
    • subject type
    • detail level
    • style bias
    • pose
    • symmetry
    • required parts
    • parts to avoid
  • One-click optimization after preview generation
  • Optimization based on preview captures and diagnostics
  • Applying the final plan back into the editable scene

The current AI 3D plan supports operations such as:

  • Creating primitive geometry: box, sphere, cylinder, capsule, cone, torus, plane
  • Creating stylized shape, extrude, and tube presets
  • Updating node transforms
  • Assigning material properties such as color, metalness, roughness, opacity, and emissive settings

This approach makes the result:

  • more stable
  • easier to validate and debug
  • easier to keep editable after generation

Tech Stack

  • App framework: Next.js 16 + React 19
  • 3D rendering: Three.js
  • UI: MUI + Sass
  • State management: Zustand
  • Authentication: better-auth
  • Database: Postgres + Drizzle ORM
  • Object storage: Vercel Blob
  • AI integration: provider-based image generation, prompt transformation, and AI 3D planning APIs
  • Render export optimization: OpenRouter openai/gpt-5.4-image-2
  • PBR texture generation: OpenRouter openai/gpt-5.4-image-2
  • Panorama generation: OpenRouter openai/gpt-5.4-image-2

Project Structure

app/                    Next.js App Router pages and API routes
components/             Home, auth, and editor UI components
frontend/api/           Browser-side API wrappers for projects, AI, assets, and external assets
render/editor/          Editor core: runtime, session, models, commands
lib/ai/                 AI modules for images, PBR textures, panoramas, 3D planning, prompt transforms, providers
lib/externalAssets/     Poly Haven provider integration, contracts, and source metadata helpers
lib/server/             Auth/session, DB guards, asset config, and project persistence services
stores/                 Zustand state stores
db/                     Drizzle schema, database setup, migrations
public/draco/           GLTF Draco decoder assets

Getting Started

1. Install dependencies

Use Node 22 LTS for local development:

nvm use
npm install

2. Configure environment variables

Copy the example file:

cp .env.example .env.local

Recommended minimum variables:

  • BETTER_AUTH_SECRET
  • BETTER_AUTH_URL
  • OPENROUTER_API_KEY required for OpenRouter-backed AI 3D, AI PBR texture, AI panorama, AI render export optimization, and prompt transformation flows
  • OPENROUTER_API_KEY is also required for AI panorama intent completion and AI Poly Haven asset recommendation intent parsing
  • SILICONFLOW_API_KEY if you want to enable that provider
  • DATABASE_URL required if you want project save/load and persistent auth
  • DATABASE_URL is also required for production AI route rate limiting; production AI routes fail closed when the limiter database is unavailable
  • BLOB_READ_WRITE_TOKEN required if you want model / texture / thumbnail uploads and project save to succeed
  • SMTP_HOST
  • SMTP_PORT
  • SMTP_USER
  • SMTP_PASS
  • SMTP_FROM

Optional external asset settings:

  • ASSET_PROVIDER_POLYHAVEN_ENABLED
  • NEXT_PUBLIC_ASSET_PROVIDER_POLYHAVEN_ENABLED
  • POLYHAVEN_ATTRIBUTION_ENABLED
  • POLYHAVEN_REQUEST_IDENTITY

Studio preview creates session-only editor entities for its studio room, plinth, and base studio lights so they can be selected and edited while studio mode is active. Studio room, plinth, decoration, reflector, and light-panel meshes use physical material parameters tuned for product-style lighting. These transient entities are filtered from project snapshots and removed when studio mode exits. The selected product transform, studio camera framing, and generated studio setup remain temporary; they are restored or discarded on exit and do not write into saved project data.

For social login support:

  • GOOGLE_CLIENT_ID
  • GOOGLE_CLIENT_SECRET
  • GITHUB_CLIENT_ID
  • GITHUB_CLIENT_SECRET

3. Start development server

npm run dev

The default development script uses webpack because the editor can trigger high memory usage in Turbopack dev mode on some local Node setups. To explicitly test Turbopack, run npm run dev:turbo.

Default routes:

  • Home: http://localhost:3000/home
  • Editor: http://localhost:3000/editor

4. Initialize database schema

If you just pulled the repository or changed schema files, push the current schema to your database before testing project save:

npm run db:push

Without this step, save requests will fail if tables such as projects or assets do not exist yet.

Database Commands

The project already includes Drizzle configuration plus auth, project, and asset schema.

Useful commands:

npm run db:generate
npm run db:push
npm run db:studio

Key persisted tables now include:

  • user, session, account, verification
  • projects
  • assets

The editor save flow depends on both projects and assets existing in the target database.

Storage Setup

Project save uses Vercel Blob for binary asset persistence.

Typical setup:

  1. Create a Blob store in your Vercel project
  2. Pull environment variables locally:
vercel env pull
  1. Confirm .env.local includes:
  • BLOB_READ_WRITE_TOKEN

This token is required for:

  • project thumbnails
  • imported models
  • uploaded textures
  • environment images
  • AI-generated panorama environment images
  • saved AI image resources
  • saved AI PBR texture atlases

External Asset Integration

The editor can browse Poly Haven assets through authenticated API routes:

  • HDRIs from the top bar import flow
  • texture sets from the mesh material panel
  • models from the top bar import flow
  • AI-recommended asset kits from the AI chat Assets mode

The provider is enabled by default. You can override that behavior with:

  • ASSET_PROVIDER_POLYHAVEN_ENABLED
  • NEXT_PUBLIC_ASSET_PROVIDER_POLYHAVEN_ENABLED

Relevant code paths:

  • app/api/polyhaven/
  • app/api/ai/assets/recommend/
  • frontend/api/externalAssets.ts
  • frontend/api/ai.ts
  • lib/ai/asset-recommendation/
  • lib/externalAssets/
  • components/editor/externalAssets/
  • components/editor/aiComposer/useAiAssetRecommendationComposer.ts

Current Status

The project now has a usable authenticated persistence path, but it is still evolving. A few important boundaries are worth calling out:

  • Project save/load is implemented for authenticated users, but it is currently manual-save only
  • The current persistence model stores full scene snapshots rather than patch diffs
  • Save depends on both DATABASE_URL and BLOB_READ_WRITE_TOKEN
  • Existing schema changes require npm run db:push or equivalent migration application before testing against a fresh database
  • The current AI 3D flow is aimed at low-poly sketching and structural previews, not production-grade high-resolution mesh generation
  • AI PBR texture generation uses a single atlas image and existing material texture UV transforms; it does not currently split maps into six separate image files
  • AI panorama generation completes the prompt with an LLM before image generation, then applies a generated 2048x1024 JPEG and AI-recommended environment intensity/background/exposure values through the existing scene environment fields; it does not analyze generated image pixels or use post-processing recommendations
  • AI Poly Haven asset recommendation currently uses LLM intent parsing plus keyword search/ranking and route-level memory cache; it does not yet use embeddings or a vector database
  • If DATABASE_URL is missing in local development, auth falls back to in-memory mode and will not persist after restart
  • If BLOB_READ_WRITE_TOKEN is missing, editor save cannot persist binary assets and will fail clearly

Good Fit For

  • AI-assisted 3D creation tool prototypes
  • Fast concept blocking for games, installations, and digital content
  • Low-poly character, prop, and icon sketch generation
  • Experiments that connect AI image generation with interactive material editing
  • Experiments with AI-generated PBR material atlases for editable browser scenes
  • Experiments with AI-generated panorama environments for browser-based scene lighting and backgrounds
  • Experiments with AI-curated external asset kits using Poly Haven HDRIs, PBR textures, and models

License

Released under the MIT License.

About

3D editor with AI capabilities

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages