_{by Mia's AI Lab}

Serve GLM-5.3-Flash from two NVIDIA DGX Sparks (GB10, 128 GB each, linked by their ConnectX-7 ports) through an OpenAI-compatible API, with 4 concurrent requests, the model's full 1,048,576-token context and image and video input. It runs TensorFold v0.6.0 on both Sparks (one rank on each) in NVIDIA's PyTorch container, plus 70 patches (65 of v1.4, 3 for 3 Sparks, experimental, 1 for up to 8 requests at once, 1 for stopping serial requests): DFlash2 and copy drafts, 4-bit dense weights, an FP8 KV cache, faster prompt kernels, a one-shot RoCE all-gather between the Sparks, several requests over one shared cache pool, vision, tool calling, /tokenize and /metrics.

• Checkpoint: Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold, Mia's AI Lab's own EXL3 quantization (routed experts at 4 bits a weight, BF16 elsewhere, ~176 GB), calibrated for how TensorFold serves it (model card)
• Drafter: incoai/GLM-5.3-Flash-DFlash2, or the checkpoint's own MTP head (DRAFTER, see Configuration)
• API model id: GLM-5.3-Flash-EXL3
• Context: 1,048,576 tokens a request; the 4 requests share an FP8 KV pool of about 2.9M tokens (2,922,496 at the measured start)
• Tool calling, structured outputs (xgrammar), /tokenize, and reasoning_effort low / high / max
• One command on the first Spark: ./start.sh sets up both Sparks and starts both ranks; ./stop.sh stops them

Two DGX Sparks at the default configuration (4 streams, 1,048,576-token window, FP8 KV cache, 4-bit dense weights, DFlash2 plus copy drafts, vision on), with the GPU clocks capped at 2,200 MHz. Decode and prefill were measured with sparkDash through the OpenAI API, from another machine on the network.

Decode (aggregate across the concurrent requests, per request, and time to first token)

Replies served 4 at a time are identical to the same requests served one at a time (11 of 11 cases staggered, and 11 of 11 sent in a burst).

Up to 8 requests at once (PARALLEL=8, v1.5; its verify window then defaults to 64 rows). Measured with sparkDash on one boot (2026-10-03); one or two requests decode as fast as at PARALLEL=4:

Eight at once: +27% prose and +32% code over four. The window matters for code (8 requests: 141.2 / 160.3 / 167.0 tok/s at 32 / 48 / 64 rows), not prose. Replies stay identical to one at a time (11 of 11 staggered and in a burst, 8 in flight; drafted == serial). The memory reserve grows with PARALLEL (see MEMORY_RESERVE_GIB), so the shared pool is ~1.5M tokens at 8 instead of 2.0-2.6M at 4, with the head's lowest free memory at 12.9 GiB under a 195k-token prompt. Two Sparks therefore stay at 4 by default; three Sparks default to 8 (3 Sparks).

Prefill

Prompt reuse (the server resumes from a kept prompt state instead of prefilling, with the same reply)

Quality (FP8 KV cache and 4-bit dense weights, see Checks)

Details on the model card.

• Two DGX Sparks (or two GB10 systems with 128 GB unified memory), with nothing else large on their GPUs: each needs ~110 GiB free memory when the server starts (start.sh warns below that; stop other GPU work).
• A direct ConnectX-7 link: a QSFP cable between the CX7 ports and an IPv4 address on each end in one private subnet (e.g. 192.0.2.1/24 and 192.0.2.2/24; ping must work), with a RoCE v2 GID (start.sh checks). One QSFP port of a Spark reaches the GB10 over two PCIe Gen5 x4 links, so it appears as two netdevs and two RoCE devices (enp1s0f0np0 / enP2p1s0f0np0, rocep1s0f0 / roceP2p1s0f0). Address both twins on each Spark, in the link's subnet or each in its own (NVIDIA's two-Spark playbook gives them different ones): both are then used, as is a second cabled port in the link's subnet. A prompt chunk's all-gather is ~1.8x faster on two rails; one rail is one x4 (~112 Gb/s of the port's 200).
• Key-based ssh from the first Spark (the head, which runs ./start.sh and the API) to the second (the worker): ssh-copy-id user@<worker> (after ssh-keygen -t ed25519 if you have no key); check with ssh -o BatchMode=yes user@<worker> true.
• Docker with the NVIDIA container runtime, your user in the docker group, and rsync, on both Sparks.
• Disk, on each Spark: ~205 GB: ~176 GB (164 GiB) for the checkpoint and ~2.3 GB for DFlash2 under ~/.cache/huggingface, ~25 GB for the image under Docker's root. prepare.sh asks for what the download still needs (files already in the cache, from an earlier revision say, do not count; 180 GB, MIN_FREE_GB, when it cannot ask the Hub what is missing) and 35 GB under Docker's root (IMAGE_FREE_GB; the sum when they share a filesystem), and checks the worker for what the copy must send. With WORKER_WEIGHTS=nfs the worker needs only the image (Worker weights over NFS).
• Optional: the hf CLI on the head (faster download) and a Hugging Face token (~/.cache/huggingface/token or HF_TOKEN).

On the head:

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh     # set WORKER=user@192.0.2.2 in it (and FABRIC_PEER, see below)
./start.sh

WORKER is the worker's ssh target. The ranks talk over the route to it: if it is on another network than the link, set FABRIC_PEER to the worker's CX7 address. The worker needs no copy of this repository.

The first run sets up both Sparks (see below): the image (~25 GB) on each, the checkpoint (~176 GB) and DFlash2 downloaded on the head and copied to the worker, then the CUDA kernels compile once per image (into ~/.cache/tensorfold-glm53/<image hash>). Later starts take 2 to 6 minutes to load ~80 GiB of weights on each Spark. start.sh shows each step and the server's log, runs a smoke test through both ranks, and prints GLM-5.3-Flash-EXL3 is now LIVE! on port 8888 with the endpoint.

Any OpenAI client works with base_url = "http://<head-address>:8888/v1" and the model GLM-5.3-Flash-EXL3. The model thinks before it answers (reasoning_content), so give replies enough max_tokens.

curl -s http://<head-address>:8888/v1/models
curl -s http://<head-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.3-Flash-EXL3",
  "messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
  "max_tokens": 2000
}'

./start.sh restart                                       # restart both ranks, e.g. after changing a setting
./stop.sh                                                # stop both ranks and free their GPU memory
docker logs -f glm53-flash-tf                            # rank 0's log (here)
ssh <worker> docker logs -f glm53-flash-tf               # rank 1's log (why it exited, if it did)
curl -s http://<head-address>:8888/health                # busy flag, streams, free pool tokens

Logs of earlier runs. docker rm deletes a container's log, so stop.sh (and start.sh, before it removes a stopped container left from an earlier run, a crashed one say) first saves each rank's log, stdout and stderr with timestamps, gzipped, and prints where: ~/.cache/tensorfold-glm53/logs/<date>-<time>-rank0.log.gz on the head (LOG_DIR) and ~/.cache/tensorfold-glm53/logs/<date>-<time>-rank1.log.gz on the worker. The newest 10 of each rank are kept (LOG_KEEP; 0 saves none). Read one with zcat or zless; attach both when you report a crash.

If start.sh stops at a check:

GLM's own vision tower runs on the head (rank 0: 1.05 GiB of bf16 weights and 0.75 GiB of workspace). Send images and videos as OpenAI-style content parts in a user message:

IMG=$(base64 -w0 photo.jpg)
curl -s http://<head-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.3-Flash-EXL3",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'"$IMG"'"}},
    {"type": "text", "text": "What is in this picture?"}]}],
  "max_tokens": 2000
}'

A video is a video_url part ({"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}}). Pictures and clips can also be parts of a tool result ("role": "tool", as agents return screenshots): the model reads them inside that result (with VISION=0, it is told it cannot see them, in the template's own words). A conversation with pictures in its history resumes from its kept prompt states like a text one, as long as the pictures are the same.

A request body can be up to 96 MiB, so data URLs carry about 70 MB of pictures and clips in all. By default only data URLs are accepted; VISION_URLS=1 also lets the server fetch public https:// URLs. VISION=0 serves text only and leaves the tower's ~1.8 GiB on rank 0 to the cache.

./start.sh works in five steps, each shown as it runs:

1. Setup: scripts/prepare.sh, when the setup is not ready on both Sparks (first run, new patches, another model, drafter, revision, image or worker).
2. Checks: the arguments (TensorFold's own parser), the link, the previous server (stopped on restart, or when only one rank is up; a stopped container left from an earlier run is removed, its log saved first), the port, free memory (a warning below ~110 GiB on either Spark).
3. Launch: rank 1 on the worker over ssh, then rank 0 and the API here, with the settings from scripts/config.sh.
4. Loading: rank 0's log as it comes, and every 15 s the elapsed time and how much of the startup estimate is on each Spark's GPU; if a rank stops, both ranks' last log lines and why.
5. Smoke test: one chat completion through both ranks, then the LIVE message and the endpoint.

A running server is left alone; ./start.sh restart stops it only after the setup and argument check pass, so a typo leaves it running (running requests are cut off; stop.sh warns). Extra arguments go to tensorfold serve on both ranks after the defaults, so they win (./start.sh restart --max-tokens 16384); ./start.sh --help lists all. FOREGROUND=1 ./start.sh stays attached to rank 0's log and exits with its exit code (for a systemd unit), without the progress lines, the window retry or the smoke test; when either rank ends, it stops the other one.

scripts/prepare.sh does the one-time setup, and is safe to re-run (each step skips work already done):

1. Preflight on both Sparks: Docker, the GPU, rsync, key-based ssh, the RoCE link, disk space.
2. The image tensorfold-glm53:v0.6.0 on the head: TensorFold v0.6.0 with every patches/*.patch applied, plus PyAV (video decoding) and xgrammar (structured outputs), on NVIDIA's nvcr.io/nvidia/pytorch:26.07-py3. It first pulls the published image ghcr.io/miaai-lab/glm-5.3-flash-exl3-2x-dgx-sparks-tensorfold:v0.6.0-<image hash>, by the digest pinned in scripts/config.sh (IMAGE_TAG / IMAGE_DIGEST) while the patches are this release's (the hash covers the patches and those pip packages); after you change patches/, it pulls that hash's tag if one is published, else (or with PULL=0) it builds.
3. The same image on the worker: pulled, else streamed from the head (docker save | docker load), checked identical.
4. The checkpoint, and DFlash2 with DRAFTER=dflash2, downloaded into ~/.cache/huggingface on the head at their pinned revisions (the checkpoint checked with tensorfold info), then copied to the worker with rsync over ssh and checked file by file. Both resume.

scripts/prepare.sh             # set up both Sparks without starting the server
scripts/prepare.sh --rebuild   # rebuild the image from scratch
PREPARE=1 ./start.sh restart   # force prepare.sh, then restart; PREPARE=0 skips the check

After changing patches/, scripts/publish-image.sh pushes the new image to GitHub Container Registry (latest and v0.6.0-<image hash>, the tag prepare.sh looks for).

GLM-5.3-Flash keeps a compressed latent cache (DSA) and the sparse indexer's pooled keys for every token, plus small recurrent states for its linear-attention (KDA) layers. With PARALLEL above 1, all requests draw their per-token caches from one shared pool:

Any one request can grow to the full window, and the four together share the pool: e.g. one 1M-token conversation next to one more of 1M, or next to three of ~640k. A request the pool cannot place yet waits until others finish (kept prompt states give way first); /health shows pool_tokens, pool_free_tokens and the streams decoding, filling and paused.

TensorFold's budget on each Spark is MemAvailable at start minus a host reserve (MEMORY_RESERVE_GIB, 14.5 GiB here; TensorFold's own default is a tenth of RAM). The server uses about 10 GiB beyond its own estimate at its peak, so the reserve also sets the lowest free memory. On the Spark's unified memory, running out tends to freeze the machine rather than fail an allocation. A setting that does not fit is refused before any weights load, with the largest window that fits; start.sh then restarts once with that window and says so (free memory on both Sparks for the full one). Other settings' windows:

By default the worker keeps its own copy of the checkpoint and DFlash2 (~166 GiB, copied over the link by prepare.sh). With WORKER_WEIGHTS=nfs it keeps none: rank 1 reads the head's Hugging Face cache over NFS, read-only. Loading is as fast as from the worker's own disk (both ranks were live in ~2.2 minutes).

1. On the head, export the cache to the worker once (this needs root; the worker needs nothing installed):

   sudo apt install nfs-kernel-server
   echo "$HOME/.cache/huggingface <worker CX7 address>(ro,no_subtree_check)" | sudo tee -a /etc/exports
   sudo exportfs -ra

2. Put WORKER_WEIGHTS=nfs in scripts/local.sh or .env, and run ./start.sh restart.

prepare.sh then creates a read-only docker NFS volume on the worker (NFS_VOLUME, default glm53-hf, no sudo) and checks that the worker sees every file of both snapshots as the head has them, instead of copying. NFS_PATH is the exported path as the worker mounts it (default: the head's HF_CACHE; / for an NFSv4 export with fsid=0), and NFS_SERVER the head's address (default: its address on the link).

./start-tp3.sh runs the same recipe as tensor parallel over three Sparks (TP=3 with ./start.sh's options; ./stop.sh stops every configured worker). The engine for --tp N comes from this repo's patches 0066-0068 (TensorFold v0.6.0 itself serves two ranks only), in the same published image as two Sparks (prepare.sh pulls it on the head and copies it to every worker). Three Sparks were tested on v1.3.2's patches (exact against two Sparks, drafted == serial, images and tool calls, concurrent requests). On top of v1.4 (2026-10-03, KV_POOL_GIB=27, NCCL, two boots: sliced fill on and off): concurrent requests equal one at a time (22/22 each boot), drafted == serial (6/6), long-prompt replies and every streamed reply the same with the sliced fill on and off, the 195k needle right (prefill 115.0 s), a prompt cancelled mid-fill gone in 0.16 s with the other replies unchanged, tool calls whole. Prefill 12k / 50k / 149k: 6.2 / 25.7 / 84.2 s (two Sparks: 6.5 / 26.8 / 87.1). Streaming, gaps p50 / p90 between a reply's events: one reply 15 / 18 ms, four at once 34 / 46 ms; three replies while a ~25k-token prompt fills 81 / 134 ms with the sliced fill (134 / 400 ms with FILL_BUDGET_MS=0). Rank 0 had 10.6 GiB free idle and 8.4 GiB at its lowest (a 149k prompt); at the default 32 GiB it had 5.8 GiB free idle, below the ~10 GiB this recipe keeps under load, so these runs used 27. With v1.5's defaults at three Sparks (8 requests, a 64-row window, the reserve grown to 19.6 GiB) the pool is ~4.0M tokens and rank 0 had 14.4 GiB free idle and 11.5 GiB at its lowest under a 195k-token prompt (ranks 1 and 2: 27.6 / 21.1 GiB).

Three Sparks, measured with sparkDash (the default configuration otherwise: 4 streams, 1,048,576-token window, FP8 KV cache, 4-bit dense weights, DFlash2 plus copy drafts, vision on)

Against two Sparks (Performance): prose decode 60.4 -> 77.6 tok/s for one request and 108.8 -> 146.2 tok/s for four at once; prefill about the same (1-4% faster).

Three Sparks, 8 requests at once (v1.5's default there: PARALLEL=8, 64-row window, COMM=nccl; one boot, 2026-10-03, so compare its rows with each other rather than with the table above, which another boot measured):

Eight at once: +36% prose and +28% code over four. Replies identical to one at a time (11 of 11 staggered and in a burst), drafted == serial, the 195k needle right.

DRY_RUN=1 ./start-tp3.sh shows what would run (every rank's docker run, the links found, nothing stopped or started). It uses NCCL for every all-gather by default (COMM=nccl); COMM=roce ./start-tp3.sh sends the small ones over RoCE, each peer on the devices that share its subnet (the TP-N engine's RoCE; TF_ROCE_HCA lists a node's devices toward all its peers, and each device's RoCE v2 GID is found on its own).

• Cabling. A triangle, one direct QSFP cable per pair, each cable its own subnet (both CX7 ports of each Spark are used, each toward one peer). Wire it as a directed ring, each Spark's port 0 to the next Spark's port 1; a mirrored ring leaves one pair NCCL cannot connect. start.sh refuses a pair of ranks without a common subnet.
• Settings (in scripts/local.sh): WORKER (rank 1) and WORKER2 (rank 2), each with its own FABRIC_PEER, FABRIC_PEER2 if needed (./start.sh uses WORKER only). Weights per worker: WORKER_WEIGHTS, WORKER_WEIGHTS2 (default: WORKER_WEIGHTS). With nfs, each worker mounts the head's export from the head's address on that worker's own link (NFS_SERVER, NFS_SERVER2 override it), so the export must allow every worker's link address or subnet. prepare.sh says which worker cannot mount it. A worker whose Hugging Face cache is not its HF_HOME: WORKER_HF_CACHE, WORKER_HF_CACHE2.
• Every worker as the one worker at two Sparks: prepare.sh checks each worker's image by content (#8), compares each copy's file list in byte order (#21) and counts only what rsync must send (#24); start.sh and stop.sh save every rank's log (<date>-<time>-rank<N>.log.gz in ~/.cache/tensorfold-glm53/logs on that worker).
• Links. At two Sparks the link and its rails are found as before (both PCIe twins of a cabled port, #30). Past two, the nodes' RoCE devices are paired by subnet: a twin is used when it has an address in a subnet its peer shares; one without an address is not added by name as at two Sparks.
• What it sets. The rendezvous (--master) is the head's LAN address, what its hostname resolves to (MASTER_ADDR overrides it); every worker must route to it. NCCL's bootstrap socket uses each node's default-route netdev (SOCKET_IFNAME overrides it), since no CX7 port reaches both peers of a triangle. Data stays on RoCE: NCCL_IB_HCA lists every CX7 device toward every peer (found from the subnets the nodes share), with NCCL_CROSS_NIC=1, NCCL_IB_SUBNET_AWARE_ROUTING=1 (the image's NCCL 2.30.7 has it), the RoCE v2 GID index when it is the same on all of a node's devices, and no P2P or SHM transport. These come from a vLLM recipe that ran on the same three Sparks.
• prepare.sh on its own takes TP too: TP=3 scripts/prepare.sh.

Every setting lives in scripts/config.sh. Set one for a single run from the environment (PARALLEL=2 ./start.sh restart), or keep it in scripts/local.sh (sourced as bash) or in a .env file next to start.sh (plain KEY=value lines, read, never run); both files are yours, not the repository's. The first that sets a value wins: the environment, then scripts/local.sh, then .env, then the default.

Speed settings' measured effects: What the patches change. Any TENSORFOLD_*, TF_GLM_* or TF_ROCE_* variable is passed to both ranks (export it in scripts/local.sh; .env lines are).

Revision pins. The checkpoint and DFlash2 are pinned to the revisions this recipe was measured with (MODEL_REVISION, DFLASH2_REVISION): both ranks serve exactly those from the local cache, and a new upstream commit changes nothing until the pin does. Set one empty to take the Hub's main when first downloaded.

Less common settings are described in scripts/config.sh and scripts/nodes.sh: MODEL_ID, DFLASH2_ID, TF_VERSION, TF_REPO, BASE_IMAGE (the patches are made for TensorFold v0.6.0; after changing any of these run scripts/prepare.sh --rebuild), IMAGE, CONTAINER_NAME, GHCR_IMAGE, HF_CACHE (default $HF_HOME or ~/.cache/huggingface), KERNEL_CACHE, STATE_DIR, MIN_FREE_GB, IMAGE_FREE_GB, NCCL_RAILS (1: one RoCE port even when the cabled port's two PCIe links, or a second port, are up), NCCL_CHANNELS (4), NCCL_DEBUG, RSYNC_OPTS. start.sh also takes HF_HUB_OFFLINE=0 (let the server reach Hugging Face; by default it serves from the local cache only).

The checkpoint's own sampling defaults apply (temperature 1.0, top_p 0.95). Per request:

• temperature, top_p, top_k, min_p and seed override them (temperature: 0 decodes greedily). Without a seed, the sampler's key comes from the prompt, so the same request gives the same reply.
• reasoning_effort: "low", "high" or "max", at the top level or in chat_template_kwargs ("medium" is "high", "xhigh" is "max"). Without it, GLM's template uses max; "none" (or "chat_template_kwargs": {"enable_thinking": false}) answers without thinking. chat_template_kwargs.thinking (true / false, or {"type": "enabled"} / {"type": "disabled"}, as DeepSeek-V4 clients such as pi send it) is read as enable_thinking when that is absent; other values leave the server's default.
• Earlier turns' reasoning stays in the prompt (as in zai-org's current template), so an agent's previous tool loop is resumed from the cache at a new user message; "chat_template_kwargs": {"clear_thinking": true} or TF_GLM_CLEAR_THINKING=1 drops it, as the checkpoint's template does.
• The reasoning comes back in reasoning_content, the answer in content. A request without max_tokens gets 32,768 tokens for both (MAX_TOKENS; cut to what the window has left, never refused). An empty content means the model thought until max_tokens: give more, or use reasoning_effort: "low".

• Endpoints: /v1/chat/completions, /v1/completions, /v1/responses, /v1/models, /tokenize and /detokenize (also under /v1/), /health, and Prometheus /metrics (TensorFold v0.6.0's request counters, latency and time-to-first-token histograms, plus /health's figures as tensorfold_health: metrics). No Anthropic /v1/messages.
• Context limits: a request whose prompt plus max_tokens does not fit the window is refused with HTTP 400 in OpenAI's wording, "code": "context_length_exceeded" and param naming the field (messages or prompt).
• Tool calling: tools / tool_calls, each call streamed whole once it is written (empty deltas every 2 s meanwhile, for clients with an idle timeout; a reply that ends inside a call, at its token limit, ends with length and never sends that call; a call the model ends without its </tool_call> is closed and sent when it then parses, else returned as text; a missing <arg_key> is put back), with arguments typed by their schema (an array as a JSON array, null and const values as such). Calls written inside the think block count when the reply ends on them, and a tool-calling step's reasoning is put back into the next request when an agent client drops it (TF_GLM_KEEP_REASONING=0 turns that off). A past tool call whose arguments are not a JSON object is left out of the prompt with its result (and logged) instead of failing the request.
• Structured outputs: response_format (json_object or json_schema) and the guided_json / guided_regex / guided_choice / guided_grammar / structured_outputs fields, enforced with xgrammar (after the think block).
• /tokenize / /detokenize: vLLM's fields; a prompt or messages (rendered as the chat route does) to token ids with their count and max_model_len, and back.
• Refused with HTTP 400: logprobs: true / top_logprobs (this engine returns no token probabilities) and n other than 1. Accepted but ignored, with no error: logit_bias, and the presence, frequency and repetition penalties. priority: "background" serves a request after the others.
• Prompt reuse: a new prompt that extends a recent one token for token resumes from its kept state (with pictures and clips, only when they are the same ones), and a shared system prompt's state is reused (SHARED_PREFIX).
• A POST to an unknown route is answered 404 after its body is read, so a kept-alive connection stays usable; a client that leaves is noticed however many connections the server holds.

scripts/prepare.sh bakes every patches/*.patch into the image (diffs against TensorFold v0.6.0's site-packages, applied with patch -p0 in filename order); start.sh rebuilds or re-pulls the image when the patches change.

Concurrent prompt fill and smooth streaming on two Sparks (PARALLEL=4, DFlash2, 1,024-row chunks): three 400-token replies already streaming when a fresh ~25k-token prompt arrives. Gaps are between one reply's events as the client receives them, from that prompt's arrival to its first token; tokens = what the three replies received in that time. One boot a row; the replies' text is the same in every row. The default is the last two rows.

Without a prompt filling, smooth streaming turns each round's burst into evenly spaced tokens: one reply's gaps p50 / p90 50 / 68 -> 16 / 20 ms, four replies at once 107 / 128 -> 40 / 58 ms; the replies end when they did (last token 6.6 s alone, 16.8-17.0 s four at once, either way). A request alone fills as before: 50k and 150k-token prompts within 1%, 12k within the boot-to-boot drift (+0.25% against an unchanged build on the same day). The first boot of a new image compiles a kernel during its first concurrent fill (one ~2 s pause, once).

Outputs. Drafts only propose: every drafted token is checked against the model's own keyed sample, so drafted replies equal TensorFold's serial, one-token-at-a-time decoding (send "draft": false for that reference). Replies served 4 at a time equal the same requests served alone; COMM=roce, SPLIT=1, TF_GLM_L2PF and TF_GLM_EXL3_LOADS move the same bits. Three defaults are not exact against the checkpoint in bf16, for speed and the 1M window: DENSE=q4 and KV=fp8 are lossy (quality and the 1M needle: Performance), and KDA_CHUNKED=1 is close to the serial kernel but not its bits. DENSE=bf16 KV=bf16 KDA_CHUNKED=0 serves the checkpoint as it is, with a shorter window (KV pool and memory).

The checks in tools/ talk to the running server (API_URL, default http://127.0.0.1:8888; or just PORT), from the head or another machine (API_URL=http://<head-address>:8888 tools/needle.py). Performance is measured with sparkDash (Performance).

start.sh      set up (first run) and start both ranks
start-tp3.sh  the same on three Sparks (experimental, patches 0066-0068)
stop.sh       stop them
scripts/      config.sh (all settings), local.sh.example (this setup's WORKER), prepare.sh (image + checkpoint on
              both Sparks), nodes.sh (ssh and the RoCE links), publish-image.sh (push the image to GHCR),
              banner.sh (start.sh's banner)
patches/      patches baked into the image
tools/        checks against the running server (needle, tool calls, end of turn)
CHANGELOG.md  what changed in each release
CREDITS.md    who and what this builds on
LICENSE       Apache License 2.0
NOTICE        third-party notices (TensorFold's MIT and Apache-2.0 notices, b12x, glm53-tensorfold-spark, ShapleyMcg)

Apache License 2.0, see LICENSE. NOTICE carries the third-party notices that go with it: the files in patches/ modify TensorFold v0.6.0, and the TensorFold code they change or quote as context stays under TensorFold's licenses (Apache 2.0 from v0.6.0, and the MIT notice of code written before it, both in NOTICE); parts of patches 0006 (b12x), 0036, 0046 and 0047 (glm53-tensorfold-spark) come from Apache-2.0 projects, credited there and in CREDITS.md. The model files are downloaded from Hugging Face and are not part of this repository:

• The checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold is under the Apache License 2.0; the base model it derives from is MIT-licensed by Z.AI.
• The base model GLM-5.3-Flash is under the license on its model card.
• The DFlash2 drafter is under CC BY-NC-ND 4.0, non-commercial use only (commercial licensing: contact@inco.ai); DRAFTER=mtp serves without it.

Third-party software in the image. The prebuilt image (and the one scripts/prepare.sh builds) is based on NVIDIA's PyTorch container nvcr.io/nvidia/pytorch:26.07-py3, redistributed as a value-added runtime image. The NVIDIA software in it is governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products, which the container prints at every start; by pulling or running the image you accept them. The image also contains PyAV (BSD) with its FFmpeg libraries (LGPL) and xgrammar (Apache 2.0). The Apache License above covers this repository's own work only.

Built on TensorFold by Ash Hart (ashhart), GLM-5.3-Flash by Z.ai, the EXL3 format and converter (exllamav3) by turboderp, the DFlash2 drafter by IncoAI, b12x's RoCE transport by local-inference-lab, and code from glm53-tensorfold-spark by Jay Leaton (tool calling, L2 prefetch, expert loads). The full list, including the runtime stack and licenses, is in CREDITS.md.