Skip to content

About

Local AI 3D model generator: text/image → rigged, animated, textured low-poly GLB. TRELLIS.2 + FLUX + Puppeteer + heat skinning. Fully local, MIT/Apache only.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

Repository files navigation

3D Model Generator

A local Node.js app that turns a text prompt or image into a rigged, animated, low-poly .glb with an AI-driven Animation Studio — ready to drop into three.js.

  • Fully local — no remote APIs. All inference on your GPU.
  • Strict MIT/Apache only — every component is permissive and safe for commercial use.
  • Target hardware: RTX 5070 Ti (16 GB) + 32 GB RAM (Blackwell sm_120).

Architecture

┌──────────────────────────────────────────────────────────────────┐
│  Node.js Orchestrator (TypeScript)  ← the "one program"           │
│  • Web UI: two-screen workflow (Generate → Animation Studio)      │
│  • Drives Python workers + Blender via HTTP/subprocess            │
│  • three.js live preview with SkinnedMesh + AnimationMixer        │
└───────┬────────────────────────────────┬─────────────────────────┘
        │ HTTP                            │ subprocess
        ▼                                 ▼
┌──────────────────────────┐   ┌─────────────────────────────────────┐
│ Python worker (.venv)     │   │ Blender 4.2 (headless bpy)          │
│ • FLUX.1-schnell (Apache) │   │ • Auto-rig: armature + skin weights │
│   text → image            │   │ • Animation: keyframe clip baking   │
│ • Qwen3-4B (Apache)       │   │ • Export rigged+animated .glb       │
│   prompt improvement      │   └─────────────────────────────────────┘
│   animation design (LLM)  │
│ • CLIP ViT-base (MIT)     │   ┌─────────────────────────────────────┐
│   pose verification       │   │ Python worker (rig.venv)             │
│ • TRELLIS.2-4B (MIT)      │   │ • Puppeteer (Apache) skeleton pred   │
│   image → 3D mesh+texture │   │ • Heat diffusion skinning (ours)     │
│   (runs in trellis.venv)  │   │ • Skeleton naming (ours)             │
└──────────────────────────┘   └─────────────────────────────────────┘

Pipeline

  1. Input — image upload, OR text → FLUX.1-schnell → image.
  2. Pose check — CLIP verifies the generated image is a good T-pose.
  3. Generate — TRELLIS.2(image) → low-poly PBR-textured mesh. Full 360° texture coverage, no seams, already UV-unwrapped and decimated.
  4. Rig — Puppeteer skeleton prediction + heat-diffusion skinning + automatic anatomical bone naming (thigh_L, spine_01, head, etc.).
  5. Animation Studio — Describe animations in natural language. The Qwen3 LLM designs creature-aware motion plans, Blender bakes them as named clips. Play, delete, and export the final rigged GLB.

Animation Studio

After model generation, the UI transitions to the Animation Studio where you:

  1. Describe an animation (e.g., "slow cautious stalk with head low" or "rearing up to attack")
  2. The LLM understands what creature it is (deer vs human vs bird) and generates anatomically appropriate keyframes
  3. Blender bakes the keyframes into a named AnimationClip
  4. Play the clip in the three.js viewport
  5. Delete clips you don't want
  6. Export the final GLB with all your custom clips

The same description produces different motion for different creatures:

  • "Attack" + deer → rears up on hind legs and kicks
  • "Attack" + human → arm swing / punch

Component licenses (all verified permissive)

Stage Component License
Text→image FLUX.1-schnell Apache 2.0
Image→3D (mesh + PBR texture) TRELLIS.2-4B MIT
Prompt improvement + animation design Qwen3-4B-Instruct Apache 2.0
Pose verification CLIP ViT-base MIT
Rigging / animation / export Blender 4.2 GPL (output is yours)
Auto-rig (skeleton) Puppeteer Apache 2.0
Auto-rig (skinning) Heat diffusion MIT (ours)
Skeleton naming Bone classifier MIT (ours)
GLB tooling gltf-transform MIT
3D preview / target three.js MIT

Setup

Prerequisites

  • NVIDIA GPU with ≥16 GB VRAM (tested on RTX 5070 Ti, Blackwell sm_120)
  • Python 3.11
  • Node.js v20+
  • Git
  • Blender 4.2

Install

# From the project root
powershell -ExecutionPolicy Bypass -File scripts\bootstrap.ps1

This creates three Python venvs:

  • .venv/ — FLUX, Qwen3, CLIP (generation worker)
  • trellis.venv/ — TRELLIS.2 + custom CUDA packages (mesh generation)
  • rig.venv/ — Puppeteer + scipy/robust-laplacian (skeleton + skinning)

Download model weights

# Log in to HuggingFace (needed for gated models)
.venv/Scripts/hf.exe auth login

# TRELLIS.2 (~16 GB)
.venv/Scripts/python.exe -c "
from huggingface_hub import snapshot_download
snapshot_download('microsoft/TRELLIS.2-4B', local_dir='models/TRELLIS2-4B')
"

# FLUX, Qwen3, CLIP, DINOv3, RMBG — auto-download on first run

Verify GPU detection

.venv/Scripts/python.exe -c "import torch; print(torch.cuda.get_device_capability(0))"
# expect: (12, 0) for Blackwell

Running

Two processes:

# Terminal 1 — generation worker (FLUX + CLIP + TRELLIS.2 orchestration)
.venv/Scripts/python.exe workers/python/gen_service.py

# Terminal 2 — orchestrator + web UI
cd orchestrator && npx tsx src/server.ts

Open http://127.0.0.1:8080.


Status

Phase Status
0. Environment + scaffold ✅ Complete
1. Generation (TRELLIS.2) ✅ Complete — image→textured PBR mesh, 99.9% texture coverage
2. FLUX text→image + VRAM lifecycle + web UI ✅ Complete
3. Retopo + bake ✅ Obsolete — TRELLIS.2 outputs low-poly textured mesh directly
4. Auto-rigging ✅ Complete — Puppeteer skeleton + heat skinning + anatomical bone naming
5. Animation ✅ Complete — LLM-driven animation studio with play/delete/export
6. Polish ⏳ Next — presets, compression, error handling

Verified performance (RTX 5070 Ti 16GB)

  • TRELLIS.2 (image → textured mesh): ~30s sampling + ~3 min total (model loading + export)
  • FLUX.1-schnell (text → image, NF4 4-bit): ~7s per image, peak ~10GB VRAM
  • Puppeteer skeleton: ~12s
  • Heat skinning: ~5s
  • Full pipeline (text → rigged GLB): ~4-5 min

Project layout

3d-model-generator/
  orchestrator/          # Node.js + TypeScript web app
    src/                 # server.ts, pipeline.ts, config.ts
    web/                 # index.html (two-screen UI + three.js viewer)
  workers/
    python/              # gen_service.py, trellis_runner.py, rig_predict.py,
                         # skeleton_naming.py, heat_skinning.py, pose_verify.py
    blender/             # rig_mesh.py (auto-rig), generate_clip.py (animation),
                         # retopo_bake.py (fallback), bake_only.py
  presets/               # deer.json, hunter.json, prop.json, amorphous.json
  models/                # HF weight cache (gitignored)
  output/                # generated GLBs (gitignored)
  vendor/                # TRELLIS2, Puppeteer (gitignored, cloned separately)
  scripts/               # bootstrap.ps1
  config.json            # paths, ports

About

Local AI 3D model generator: text/image → rigged, animated, textured low-poly GLB. TRELLIS.2 + FLUX + Puppeteer + heat skinning. Fully local, MIT/Apache only.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages