Skip to content

fix(art): repair the Synty retarget's real faults, and correct the diagnosis #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ralyodio merged 1 commit into from
Jul 26, 2026
Merged

fix(art): repair the Synty retarget's real faults, and correct the diagnosis #33

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 63 additions & 43 deletions docs/HANDOFF-synty.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,54 +26,74 @@ shipped**, because their animation retarget is broken.
Source pack is at `~/src/nightcell7-assets/SourceFiles/` (406 MB, outside the
repo, not committed).

## The one blocking bug
## The blocking bug

Synty ships SourceFiles without animation — the cycles live in the Unity and
Unreal packages, which are engine-locked. So our CC0 clips have to be retargeted
onto Synty's Unreal-standard 55-bone skeleton.

**First attempt (wrong).** Copy-rotation constraints in world space, then
`nla.bake`. That forces the target bone to adopt the source bone's _absolute_
orientation, which is only correct if both skeletons share rest poses. They do
not — Synty uses Unreal's axis convention, our clips another. Every bone sat at
its rest-pose difference and the error compounded down each limb.

Measured: **1.90 x 2.00 x 2.47 m** for a figure that should be about
**0.6 x 0.4 x 1.8**. Limbs splayed in every direction.

**Second attempt (maths right, export wrong).** `import_synty.py` now solves
each bone directly:

```
target.matrix = source.matrix @ (source.rest⁻¹ @ target.rest)
```

so what transfers is the source's motion _away from its own rest_, not its raw
orientation. Parents are solved before children and the view layer is flushed
between bones, because setting a pose bone's matrix reads its parent's current
transform.

That runs. What fails is the export: the manually keyed actions do not survive
`export_scene.gltf` the way `nla.bake`'s did. **All five clips collapse into a
single animation named `target_rig`** — after the object, not the actions.
Unreal packages, which are engine-locked. So our CC0 clips have to be
retargeted onto Synty's Unreal-standard 55-bone skeleton.

**The previous diagnosis in this document was wrong.** It said the maths was
correct and the glTF export was broken. Three separate faults were actually in
play, and the first two hid the third:

1. **`BONE_MAP`'s source side did not match any real bone.** It listed `Hips`,
`Abdomen`, `Torso`, `LowerArm.L`, `Wrist.L`, `UpperLeg.L`; the rig actually
has `hips`, `spine`, `chest`, `forearm.L`, `hand.L`, `thigh.L`. The lookup
_skipped_ anything it could not find, so zero bones mapped, zero keyframes
were written and all five actions came out empty — with no error. Now fixed,
and a mismatch is fatal rather than skipped.

2. **`export_bake_animation=True` overrode the actions.** With it on, the
exporter emits one baked animation per object, named after the object and
covering all 55 bones in T/R/S. That is precisely the reported symptom, "all
five clips collapse into a single animation named `target_rig`" — it was
never an action-handling bug. Now off, with each action pushed to its own
NLA track.

3. **The "limbs splayed, 1.90 x 2.00 x 2.47 m" measurement was an artefact.**
The scene contains a stray 42-vertex `Icosphere` that dominated the bounding
box. Measuring the character mesh alone gives 2.03 x 0.32 x 1.79 — correct
height, correct depth. Nothing was ever splayed. (That sphere is still in the
committed `fighter_*.glb` and should be tracked down separately.)

With 1 and 2 fixed, all five clips export under their own names and the legs
animate correctly. What remains is a real, measured rest-pose mismatch:

| | across | note |
| --------------- | ------ | ------------------------------------- |
| our source rig | 0.69 m | rests arms-down; no clip exceeds 0.72 |
| Synty bind pose | 2.03 m | T-pose, arms straight out |

Rest-relative retargeting transfers _deviation from rest_. The source barely
deviates from its own arms-down rest, so the target barely deviates from its
T-pose: legs move, arms stay out. Every clip measures 2.02-2.05 m across.

## Next steps, in order

1. **Fix the export.** The maths is believed correct; only the action handling
is wrong. Options worth trying, cheapest first:
- Push each baked action to an NLA track before export, rather than relying
on `export_animation_mode="ACTIONS"` finding loose actions.
- Check `action.id_root` is `"OBJECT"` on the actions created with
`bpy.data.actions.new()`.
- Keep the rest-relative solve but write the result through `nla.bake`
(which demonstrably exported correctly), e.g. by driving the target with
`COPY_TRANSFORMS` against a helper armature already offset by the rest
delta.

2. **Verify objectively, not by eye.** Load the exported GLB, set a mid-`walk`
frame, and measure the mesh bounding box. It must be near
`0.6 x 0.4 x 1.8`. A bind-pose preview looks fine even when the animation is
broken — that is exactly how this shipped in the first place.
1. **Stop retargeting. Skin the Synty mesh to our rig instead.**
This is the recommended route. Our rig already has all five clips working;
the Synty character is, for our purposes, just a 4k-triangle mesh. Parenting
it to our existing armature with automatic weights removes the entire
skeleton-to-skeleton problem rather than solving it. Risk is weighting
quality around the shoulders and hips, which is inspectable in a render.

Two approaches that were tried and measured, so they are not repeated:
- _Copy world orientation outright_ — removes the rest precondition and the
bone-axis conventions with it; the figure lies down (2.14 x 0.52 x 0.39).
- _Re-rest the target into the source's A-pose_ by rotating each bone by the
minimal arc onto its counterpart's rest direction, then applying that as
the rest. The corrections come out incoherent — 169 degrees on the left
upper arm against 11 on the right, where a mirror pair must be symmetric —
so the two rest frames are not being compared in a common basis. Worth
revisiting only with that basis problem understood.

2. **Verify objectively, and measure the right mesh.** Load the exported GLB,
assign each action directly (muting or deleting the NLA, or the assigned
action masks whichever strip you unmute), step every frame, and measure the
_character_ mesh — explicitly excluding the stray `Icosphere`, which is what
produced the bogus splay reading above. A walk frame should be near
`0.6-0.9 x 0.4 x 1.8`. A bind-pose preview looks fine even when the
animation is broken, which is how this shipped in the first place.

3. **Then swap the bots back.** Two lines in `apps/game/src/opponents.ts`,
marked in a comment there:
Expand Down
162 changes: 133 additions & 29 deletions tools/art/blender/import_synty.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,34 @@
figure with its arms through its chest. Constraints resolve in world space and
the bake writes the result back as ordinary keyframes.

**This script does not yet produce a shippable character**, and the output is
deliberately not wired into the game. What it does now is run correctly and
fail visibly, which it did not before. The remaining blocker is measured and
specific:

* source rig rest pose: 0.69 m across (arms down), and no clip exceeds 0.72
* Synty bind pose: 2.03 m across (T-pose, arms straight out)

Rest-relative retargeting transfers the source's *deviation from its own rest*.
With rests that far apart the legs transfer correctly and the arms never leave
the T-pose, which is exactly what the exported clips measure.

Two fixes were tried and rejected, both measured rather than eyeballed:

* **Copy world orientation outright.** Removes the rest precondition, and
discards the bone-axis conventions with it — the figure ends up lying down
(2.14 x 0.52 x 0.39 for something 1.79 m tall).
* **Re-rest the target into the source's A-pose** by rotating each bone by
the minimal arc onto its counterpart's rest direction, then applying that
as the rest pose. The corrections come out incoherent — 169 degrees on the
left upper arm against 11 on the right, where a mirror pair must be
symmetric — so the two rigs' rest frames are not being compared in a common
basis, and the applied result is still 2.05 m across.

The more promising route is to stop retargeting altogether: skin the Synty
*mesh* to our existing rig with automatic weights and keep the clips we already
have working, rather than moving animation between skeletons at all.

Usage:
blender --background --factory-startup --python import_synty.py -- \
--fbx <SK_Chr_...fbx> --anims <source.glb> --atlas <texture.png> \
Expand All @@ -32,31 +60,44 @@
import bpy

# Our animation rig on the left, the Unreal-standard Synty rig on the right.
# Fingers, toes, eyes and the IK helper bones are deliberately absent: nothing
# in our clip set animates them, and constraining them only adds bake cost.
#
# Every name here is checked against both armatures at run time and a mismatch
# is fatal — see `build_pairs`. The first version of this table was written from
# memory and got the *source* side wrong on almost every bone ("Hips",
# "LowerArm.L", "Wrist.L" against a rig that actually calls them "hips",
# "forearm.L", "hand.L"). Because the lookup skipped anything it could not find,
# the result was zero mapped bones, zero keyframes, five empty actions and a
# character exported in its bind pose — with no error anywhere. That silence is
# what the validation below exists to prevent.
#
# Fingers, toes, eyes and the IK helper bones are deliberately absent: our clip
# set does not animate them, and constraining them only adds bake cost.
BONE_MAP = {
"Hips": "Pelvis",
"Abdomen": "spine_01",
"Torso": "spine_02",
"Chest": "spine_03",
"Neck": "neck_01",
"Head": "head",
"Shoulder.L": "clavicle_l",
"UpperArm.L": "UpperArm_L",
"LowerArm.L": "lowerarm_l",
"Wrist.L": "Hand_L",
"Shoulder.R": "clavicle_r",
"UpperArm.R": "UpperArm_R",
"LowerArm.R": "lowerarm_r",
"Wrist.R": "Hand_R",
"UpperLeg.L": "Thigh_L",
"LowerLeg.L": "calf_l",
"Foot.L": "Foot_L",
"UpperLeg.R": "Thigh_R",
"LowerLeg.R": "calf_r",
"Foot.R": "Foot_R",
"hips": "Pelvis",
"spine": "spine_01",
"chest": "spine_02",
"neck": "neck_01",
"head": "head",
"shoulder.L": "clavicle_l",
"upperarm.L": "UpperArm_L",
"forearm.L": "lowerarm_l",
"hand.L": "Hand_L",
"shoulder.R": "clavicle_r",
"upperarm.R": "UpperArm_R",
"forearm.R": "lowerarm_r",
"hand.R": "Hand_R",
"thigh.L": "Thigh_L",
"shin.L": "calf_l",
"foot.L": "Foot_L",
"thigh.R": "Thigh_R",
"shin.R": "calf_r",
"foot.R": "Foot_R",
}

# The Synty rig carries a third spine bone our clips have no counterpart for.
# It is left at rest deliberately: an unmapped bone still follows its parent, so
# the chain stays continuous, and inventing motion for it would be worse than
# not moving it.

def arg(name: str, fallback=None):
return sys.argv[sys.argv.index(name) + 1] if name in sys.argv else fallback
Expand Down Expand Up @@ -126,13 +167,32 @@ def main() -> None:
bpy.ops.object.transform_apply(location=False, rotation=True, scale=True)

pairs = []
missing = []
for our_name, synty_name in BONE_MAP.items():
t_bone = target.pose.bones.get(synty_name)
s_bone = source.pose.bones.get(our_name)
if t_bone is None or s_bone is None:
if t_bone is None:
missing.append(f"target has no {synty_name!r}")
continue
if s_bone is None:
missing.append(f"source has no {our_name!r}")
continue
rest_delta = s_bone.bone.matrix_local.inverted() @ t_bone.bone.matrix_local
pairs.append((s_bone, t_bone, rest_delta))
pairs.append((s_bone, t_bone, None))

# Fatal, not skipped.
#
# Skipping an unmatched bone is what made the first version of this script
# produce a silent, plausible-looking failure: the whole map missed, nothing
# was keyed, and a bind-posed character shipped. A name that does not
# resolve means the table is wrong about a rig, and there is no useful
# retarget to be had from the remainder.
if missing:
raise SystemExit(
"import_synty: BONE_MAP does not match these rigs:\n "
+ "\n ".join(missing)
+ f"\n\nsource bones: {sorted(b.name for b in source.pose.bones)}"
+ f"\ntarget bones: {sorted(b.name for b in target.pose.bones)}"
)

# Parents before children: setting a pose bone's matrix reads its parent's
# current transform, so a child solved first is immediately invalidated.
Expand All @@ -145,6 +205,16 @@ def depth(pose_bone):

pairs.sort(key=lambda entry: depth(entry[1]))

pairs = [
(
s_bone,
t_bone,
(source.matrix_world @ s_bone.bone.matrix_local).inverted()
@ (target.matrix_world @ t_bone.bone.matrix_local),
)
for s_bone, t_bone, _ in pairs
]

hips_pair = next((p for p in pairs if p[1].name == "Pelvis"), None)
baked = []

Expand All @@ -167,8 +237,17 @@ def depth(pose_bone):
bpy.context.view_layer.update()

for s_bone, t_bone, rest_delta in pairs:
world = source.matrix_world @ s_bone.matrix @ rest_delta
t_bone.matrix = target.matrix_world.inverted() @ world
# Rest-relative retarget:
#
# T_pose = S_pose @ S_rest⁻¹ @ T_rest
#
# transfers the source's motion away from its own rest onto the
# target's rest. This is the standard formula and it is correct
# — but only when both skeletons share a reference pose, which
# these two do NOT. See the module docstring: the arms come out
# stuck in Synty's T-pose, and that is the remaining blocker.
s_pose_world = source.matrix_world @ s_bone.matrix
t_bone.matrix = target.matrix_world.inverted() @ (s_pose_world @ rest_delta)
# Each child reads the parent just written, so flush per bone.
bpy.context.view_layer.update()

Expand All @@ -177,10 +256,27 @@ def depth(pose_bone):
if hips_pair:
hips_pair[1].keyframe_insert("location", frame=frame)

# An action with no curves is the signature of a retarget that ran
# over an empty bone list. It exports as nothing at all, or — with
# `export_bake_animation` on — as one anonymous baked clip named after
# the object, which is how this last shipped broken.
if not action.fcurves:
raise SystemExit(
f"import_synty: baking {clip.name!r} produced no curves "
f"({len(pairs)} bone pairs, frames {start}-{end})"
)

source_name = clip.name
clip.name = f"__src_{source_name}"
action.name = source_name
action.use_fake_user = True
# Each action needs its own NLA track to survive the export.
# `export_animation_mode="ACTIONS"` does not reliably find actions that
# are merely present in the file with a fake user; a track makes the
# association to this object explicit.
track = target.animation_data.nla_tracks.new()
track.name = source_name
track.strips.new(source_name, int(action.frame_range[0]), action)
baked.append(action.name)

for obj in [source, *source_meshes]:
Expand Down Expand Up @@ -231,7 +327,12 @@ def depth(pose_bone):
export_skins=True,
export_animations=True,
export_animation_mode="ACTIONS",
export_bake_animation=True,
# OFF. With this on, the exporter emits a single baked animation per
# object — named after the object, covering every bone in T/R/S — and
# ignores the actions entirely. That is what "all five clips collapsed
# into one called target_rig" actually was.
export_bake_animation=False,
export_nla_strips=True,
export_cameras=False,
export_lights=False,
export_extras=False,
Expand All @@ -242,7 +343,10 @@ def depth(pose_bone):
export_image_format="NONE",
)

print(f"SYNTY {os.path.basename(out)} bytes={os.path.getsize(out)} clips={len(baked)}")
print(
f"SYNTY {os.path.basename(out)} bytes={os.path.getsize(out)} "
f"clips={len(baked)} bones={len(pairs)}"
)
print(f" {', '.join(sorted(baked))}")


Expand Down
Loading