Kirill Mazur, Nikita Karaev, Matthew Chang, Jitendra Malik, Nur Muhammad "Mahi" Shafiullah
Amazon FAR (Frontier AI and Robotics)
Project page: https://agenticstar.github.io/
This is a harness for agentic shape tracking of articulated objects. A coding
agent (Codex or Claude Code) looks at the frames of a monocular video, writes a
Blender script that builds the object from primitives, poses it per frame in
front of the known cameras and declares its joints, then renders, compares to
the frames and iterates. The output is a GLB with named parts and a pose.json
with per-frame poses and joint states.
The two main components the agent works with are the pose search views
(sweep/apply) and the
temporal tool. The task the agent reads
is AGENT_TASK.md; the scene.py contract is in
conventions/.
The repo was designed and tested with GPT-5.6-Sol and Claude Fable 5. The
results in the paper were obtained with the mechanism module turned on
(--enable mechanism; it is off by default). Newer models tend to do better
with it off. The GPT-5.6-Sol results were obtained with the critic branch,
which adds a second VLM as a critic; newer models should work fine without it.
Linux x86_64 with an NVIDIA GPU. You need micromamba, git, curl, tar,
xz, python3, tmux, bubblewrap, iproute2 and ffmpeg on the host.
Root (or CAP_NET_ADMIN) is needed for the sandbox's network namespace.
./install.sh # artscript env, Blender 4.2.5, pi3x env
export ANTHROPIC_API_KEY=sk-ant-... # for Claude Code
tools/install_claude_bwrap.sh # once; stores the key in a state dir outside the checkout
export OPENAI_API_KEY=sk-... # for Codex
tools/install_codex_bwrap.shclaude and/or codex must be on PATH.
One supervised run on the committed garden-shears example, on GPU 0:
tools/run_kf.sh --agent claude --timeout-hours 4 examples/garden_shears/capture:0
tools/claude-supervisor attach --run-dir runs/capture_kf_<stamp> # watch itUse --agent codex and tools/agent-supervisor for Codex. Add
--enable mechanism to reproduce the paper setting. Outputs land in
runs/<name>/: scene.py, mesh/object.glb, mesh/pose.json, and every
pass's renders and side-by-sides under iterations/. A run takes hours and
many tokens; --timeout-hours bounds it.
The agent runs inside a Bubblewrap sandbox with network access limited to the
model provider's API host. SANDBOX_NET=open disables that for debugging.
./run.sh --help lists the remaining options (GPU pinning, frame selection,
resume, --prompt-only).
A capture is a directory with:
frames/000000.jpg, 000005.jpg, ... video frames
mask_object/<stem>.png binary object mask per frame, same size as the frame
mask_hand/<stem>.png optional; occluder mask, treated as don't-care
tracking/cameras.npz per-frame camera-to-world and intrinsics
tracking/keyframes.json the frames that have a camera
Masks can come from any segmenter. Cameras come from
Pi3X, which tools/make_capture.py runs
jointly over the keyframes:
tools/video_to_frames.sh clip.mov my_input --every 5
# add my_input/mask_object/<stem>.png for those frames
micromamba run -n pi3x python tools/make_capture.py --src my_input --out captures/my_object
tools/run_kf.sh --agent claude captures/my_object:0--depth additionally keeps Pi3X's per-frame pointmaps in depth/ for the
depth scorer. examples/garden_shears/input/ is such an input directory and
examples/garden_shears/capture/ is what the converter produced from it. The
schema, reader and validator live in datasets/common/.
We thank the authors of Pi3 for releasing their model and code.
@misc{mazur2026agentstar,
title = {{AgentSTAR}: Agentic Shape Tracking and Reconstruction from Monocular Videos},
author = {Mazur, Kirill and Karaev, Nikita and Chang, Matthew and Malik, Jitendra and Shafiullah, Nur Muhammad},
year = {2026},
eprint = {2609.24487},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.24487}
}MIT, see LICENSE. harness/views/sweeps/lib/_vendor/ carries two
optimizer routines adapted from SciPy under its BSD-3-Clause license
(SCIPY_LICENSE.txt).