A local Node.js app that turns a text prompt or image into a rigged,
animated, low-poly .glb with an AI-driven Animation Studio — ready to
drop into three.js.
- Fully local — no remote APIs. All inference on your GPU.
- Strict MIT/Apache only — every component is permissive and safe for commercial use.
- Target hardware: RTX 5070 Ti (16 GB) + 32 GB RAM (Blackwell sm_120).
┌──────────────────────────────────────────────────────────────────┐
│ Node.js Orchestrator (TypeScript) ← the "one program" │
│ • Web UI: two-screen workflow (Generate → Animation Studio) │
│ • Drives Python workers + Blender via HTTP/subprocess │
│ • three.js live preview with SkinnedMesh + AnimationMixer │
└───────┬────────────────────────────────┬─────────────────────────┘
│ HTTP │ subprocess
▼ ▼
┌──────────────────────────┐ ┌─────────────────────────────────────┐
│ Python worker (.venv) │ │ Blender 4.2 (headless bpy) │
│ • FLUX.1-schnell (Apache) │ │ • Auto-rig: armature + skin weights │
│ text → image │ │ • Animation: keyframe clip baking │
│ • Qwen3-4B (Apache) │ │ • Export rigged+animated .glb │
│ prompt improvement │ └─────────────────────────────────────┘
│ animation design (LLM) │
│ • CLIP ViT-base (MIT) │ ┌─────────────────────────────────────┐
│ pose verification │ │ Python worker (rig.venv) │
│ • TRELLIS.2-4B (MIT) │ │ • Puppeteer (Apache) skeleton pred │
│ image → 3D mesh+texture │ │ • Heat diffusion skinning (ours) │
│ (runs in trellis.venv) │ │ • Skeleton naming (ours) │
└──────────────────────────┘ └─────────────────────────────────────┘
- Input — image upload, OR text → FLUX.1-schnell → image.
- Pose check — CLIP verifies the generated image is a good T-pose.
- Generate — TRELLIS.2(image) → low-poly PBR-textured mesh. Full 360° texture coverage, no seams, already UV-unwrapped and decimated.
- Rig — Puppeteer skeleton prediction + heat-diffusion skinning + automatic anatomical bone naming (thigh_L, spine_01, head, etc.).
- Animation Studio — Describe animations in natural language. The Qwen3 LLM designs creature-aware motion plans, Blender bakes them as named clips. Play, delete, and export the final rigged GLB.
After model generation, the UI transitions to the Animation Studio where you:
- Describe an animation (e.g., "slow cautious stalk with head low" or "rearing up to attack")
- The LLM understands what creature it is (deer vs human vs bird) and generates anatomically appropriate keyframes
- Blender bakes the keyframes into a named AnimationClip
- Play the clip in the three.js viewport
- Delete clips you don't want
- Export the final GLB with all your custom clips
The same description produces different motion for different creatures:
- "Attack" + deer → rears up on hind legs and kicks
- "Attack" + human → arm swing / punch
| Stage | Component | License |
|---|---|---|
| Text→image | FLUX.1-schnell | Apache 2.0 |
| Image→3D (mesh + PBR texture) | TRELLIS.2-4B | MIT |
| Prompt improvement + animation design | Qwen3-4B-Instruct | Apache 2.0 |
| Pose verification | CLIP ViT-base | MIT |
| Rigging / animation / export | Blender 4.2 | GPL (output is yours) |
| Auto-rig (skeleton) | Puppeteer | Apache 2.0 |
| Auto-rig (skinning) | Heat diffusion | MIT (ours) |
| Skeleton naming | Bone classifier | MIT (ours) |
| GLB tooling | gltf-transform | MIT |
| 3D preview / target | three.js | MIT |
- NVIDIA GPU with ≥16 GB VRAM (tested on RTX 5070 Ti, Blackwell sm_120)
- Python 3.11
- Node.js v20+
- Git
- Blender 4.2
# From the project root
powershell -ExecutionPolicy Bypass -File scripts\bootstrap.ps1This creates three Python venvs:
.venv/— FLUX, Qwen3, CLIP (generation worker)trellis.venv/— TRELLIS.2 + custom CUDA packages (mesh generation)rig.venv/— Puppeteer + scipy/robust-laplacian (skeleton + skinning)
# Log in to HuggingFace (needed for gated models)
.venv/Scripts/hf.exe auth login
# TRELLIS.2 (~16 GB)
.venv/Scripts/python.exe -c "
from huggingface_hub import snapshot_download
snapshot_download('microsoft/TRELLIS.2-4B', local_dir='models/TRELLIS2-4B')
"
# FLUX, Qwen3, CLIP, DINOv3, RMBG — auto-download on first run.venv/Scripts/python.exe -c "import torch; print(torch.cuda.get_device_capability(0))"
# expect: (12, 0) for BlackwellTwo processes:
# Terminal 1 — generation worker (FLUX + CLIP + TRELLIS.2 orchestration)
.venv/Scripts/python.exe workers/python/gen_service.py
# Terminal 2 — orchestrator + web UI
cd orchestrator && npx tsx src/server.tsOpen http://127.0.0.1:8080.
| Phase | Status |
|---|---|
| 0. Environment + scaffold | ✅ Complete |
| 1. Generation (TRELLIS.2) | ✅ Complete — image→textured PBR mesh, 99.9% texture coverage |
| 2. FLUX text→image + VRAM lifecycle + web UI | ✅ Complete |
| 3. Retopo + bake | ✅ Obsolete — TRELLIS.2 outputs low-poly textured mesh directly |
| 4. Auto-rigging | ✅ Complete — Puppeteer skeleton + heat skinning + anatomical bone naming |
| 5. Animation | ✅ Complete — LLM-driven animation studio with play/delete/export |
| 6. Polish | ⏳ Next — presets, compression, error handling |
- TRELLIS.2 (image → textured mesh): ~30s sampling + ~3 min total (model loading + export)
- FLUX.1-schnell (text → image, NF4 4-bit): ~7s per image, peak ~10GB VRAM
- Puppeteer skeleton: ~12s
- Heat skinning: ~5s
- Full pipeline (text → rigged GLB): ~4-5 min
3d-model-generator/
orchestrator/ # Node.js + TypeScript web app
src/ # server.ts, pipeline.ts, config.ts
web/ # index.html (two-screen UI + three.js viewer)
workers/
python/ # gen_service.py, trellis_runner.py, rig_predict.py,
# skeleton_naming.py, heat_skinning.py, pose_verify.py
blender/ # rig_mesh.py (auto-rig), generate_clip.py (animation),
# retopo_bake.py (fallback), bake_only.py
presets/ # deer.json, hunter.json, prop.json, amorphous.json
models/ # HF weight cache (gitignored)
output/ # generated GLBs (gitignored)
vendor/ # TRELLIS2, Puppeteer (gitignored, cloned separately)
scripts/ # bootstrap.ps1
config.json # paths, ports