CoDex: Learning Compositional Dexterous Functional Manipulation without Demonstrations

Why is Functional Object Manipulation challenging?

"Robots can pick and place, but can they use a spray bottle or a glue‑gun?"

The challenge. CD‑FOM tasks demand coordinated control over internal (buttons, triggers) and external (object pose) DoF so the effect lands on the right target.

Our bridge. We let a VLM provide semantics (what to actuate, where to aim) and translate them into semantic constraints that drive analytic optimization and RL to achieve physical dexterity (functional contacts, stable grasps, collision‑free motion).

CoDex: Learning Compositional Dexterous Functional Manipulation without Demonstrations

Stage 1 — VLM‑Generated Semantic Constraints

We parse (language, RGB‑D) to (i) segment and reconstruct the tool, (ii) locate the Actuation and Function points and directions, and (iii) derive a global goal pose via VLM‑CEM—an iterative, keypoint‑anchored pose search guided by VLM scoring.

Stage 2 — Constraint‑Guided Policy Training

Analytic constrained optimization turns local constraints into function‑aligned, stable grasp candidates; then constraint‑guided RL in ManiSkill3 refines them into a single motion primitive that completes grasp–move–actuate end‑to‑end.

Stage 3 — Real‑world Execution

The learned primitive transfers directly to a FRANKA arm with a LEAP hand, executing the full pipeline on unseen objects without task‑specific demonstrations or tuning.