StartLuxLabs/StartLux-Decision

Models on Hugging Face · Results · Inference · Evaluation · Fine-tuning · Raw results

New, 2026-10-03: images and 256K-token context. Every size now reads images as part of a request's evidence and prompts up to 262,144 tokens. See Images and long inputs.

StartLux-Decision is a family of typed decision models: five dense sizes from 0.8B to 27B, and a 35B-A3B mixture of experts. You send a state and a set of questions: pick one of several options, yes or no, or a rating on a scale. The state can be text, JSON or images, and as long as 262,144 tokens (256K) at every size. Every question comes back with a probability for each option. Nothing is generated; the answer is read from the option letters after one forward pass, so a short question takes a few milliseconds and the probabilities can be used as confidence. Requests and responses use the TypeSafe /v1/systemone format, so clients written for Jev work unchanged. This repository has the inference code, the evaluation scripts, the results and the raw game logs. The weights are on Hugging Face, in the StartLux-Decision collection.

StartLux-Decision-27B finds the cheapest AA 8-pack with free delivery and orders it to the right address	StartLux-Decision-27B, with White, checkmates Jev 1.13 on move 19 of a Scotch Game; neither side searches
StartLux-Decision-27B clears World 1-1 of Super Mario Bros.	StartLux-Decision-27B makes the strategic calls for a Terran bot and beats the built-in Hard AI
StartLux-Decision-27B plays a Doom deathmatch against four built-in bots	StartLux-Decision-27B makes the decisions for a football team in real time

All six are real runs of StartLux-Decision-27B; click a preview for the video.

Computer use (our own harness). A real Chrome window opens a small mock site, and every step is one request with two typed questions: which of the controls visible on the page to use next, and whether the task is done. The harness carries out the chosen action and nothing else. The model never types text; a text box it clicks only opens its suggestion list. The task was completed, and the harness checked the result against it. The side panel lists the four likeliest controls and the rest in one row, so that block adds up to 100%; below it, set apart, is the probability that the task is done. The harness and the two sites are in demos/computer_use.

Chess (wondertwins/jev-benchmark). StartLux-Decision-27B plays Jev 1.13 through the harness at its default rich level: every turn is one request with one choice question over all legal moves, given the board, the move history and the pieces in text. Both sides get the same request, neither searches, and code never overrides a move; Stockfish only draws the evaluation bar. This is one of the 256 games of the match under Games, from the position after the first four moves of a Scotch Game; over the 256 games StartLux-Decision-27B scores 58.4%. The match tools are in demos/chess.

Super Mario Bros. (4esv/jev-mario). Every step is one request with one choice question over eleven moves. The harness plays each move ahead in the emulator and describes the outcome in text, such as progress and whether Mario survives; the model reads only these descriptions and picks a move. The panel shows the probability it gave each move.

StarCraft II (our own bot on burnysc2). Every 12 seconds of game time, one request asks three questions: what to build next, which units to prioritise and whether to attack, defend or gather. A scripted bot carries out the choices: workers, placement, production and unit control. The game is sped up; the panel shows each decision with its probabilities.

Doom (our own harness on ViZDoom). Every four tics, one request asks which enemy to target, whether to fire and how to move. The state is a text description from the game engine. Aiming, pathing and getting unstuck are done in code. The game waits for each decision.

JevBall (atarikcaliskan/jevball). One request covers the players near the play, one choice each over as many as fourteen actions: pass, shot, dribble, press, run and more. The match never waits for the model: the game handles movement and physics, and its own policy covers players far from the ball and any late answer. The other team is the game's built-in policy.

StartLux-Decision-27B reaches 63.88 on Decision Index 0.2.1, StartLux-Decision-35B-A3B 61.55 and StartLux-Decision-9B 58.63, scored with the board's own kit on the full suite. The highest entry on the public board (2026-09-28) is Jev 1.13 at 57.91; our runs are not on the board. StartLux-Decision-27B scores higher than Jev on 31 of the 38 benchmarks in the index. Our training data includes the public train splits of 14 of them, marked † below; their test items were filtered out of it.

Every cell below is on the index's own scale: the benchmark's metric, corrected for chance, so 0% is random guessing and 100% is perfect. The index is a weighted mean of these cells, which is why the first row matches the chart.

★ benchmarks weigh 1.2 in the index. The rows with bold names are the index's own area scores; a bold value marks the best score in its row. The other systems' values come from the public board. The metric of each benchmark and the raw scores of all six models are in docs/results.md and results/decision_index_benchmarks.csv.

The public JevBench items are the comparison most decision models of this kind report. The table lists the systems in the Intern-Decision bundle next to ours; a blank cell means the number is not published.

JevBench public counts the correct answers on the 231 public items in the Intern-Decision bundle, Intern avg is the average accuracy over its seven suites, and DI is the Decision Index under both editions (the public board's values for the other systems). Our latencies are for one H200, with the three questions answered in one forward pass, as Intern-Decision does. ¹ Intern-Decision's own measurement on an RTX 4090. ² The server time the TypeSafe API gateway reports for the same request, mean of 100, so the network is left out as in ours; Intern-Decision reports 109.7 ms end to end.

The Intern-Decision bundle has seven suites: the three public JevBench tiers, Typed Decisions, ToolACE, AG News and WildJailBreak. Its ToolACE items are drawn from the public ToolACE training set, which has no held-out split, so docs/results.md also gives the average without it (StartLux-Decision-4B: 90.67, Intern-Decision-4B: 88.95). JevBench public counts the correct answers on the 231 public items in the bundle. It is not the official JevBench score, which adds a sealed tier, speed and cost and is measured only by the maintainers.

StartLux-Decision-35B-A3B, a mixture of experts with about 3B of its 35B parameters active for each token, answers the same request in 52.5 ms, half the time of the 27B; its experts run as grouped matrix multiplications, so the CUDA graphs cover them too. It fits one 80 GB GPU (about 71 GiB at its peak).

The fast linear-attention kernels (flash-linear-attention, causal-conv1d) are required; the server refuses to start on a GPU without them. All questions of a request run in one forward pass, and CUDA graphs remove most of the launch overhead for short requests: StartLux-Decision-4B takes 90.3 ms for the same request without them. Bulk evaluation goes through a batched path instead. Details in docs/inference.md.

Every size reads images and long documents. A request can carry images as part of its evidence, and a prompt can run to 262,144 tokens (256K), the models' native context: a long state is read once, in chunks, and every question of the request branches off it.

answers, usage = m.decide("Photo taken at delivery: <image>",
                          {"damaged": {"type": "noul", "instructions": "Is the parcel damaged?"}},
                          images=["parcel.jpg"])

Over HTTP, "images" is a list of base64 strings or data URIs. Details are in docs/inference.md; the MLX and GGUF backends read text only.

Every dense size also comes as GGUF files for llama.cpp: BF16, which keeps the weights unchanged, and llama.cpp's standard Q8_0 and Q4_K_M quantizations of it. Each file has its own Hugging Face repository, startlux-models/StartLux-Decision-<size>-<precision>-GGUF, linked from the table below. The decision procedure is not in the weights; python -m startlux_decision.gguf_server runs it in front of llama-server (see docs/inference.md). Against the original weights on the 231 public JevBench items:

BF16 and Q8_0 give the original answer on 99 to 100% of the items. Q4_K_M keeps 96.5 to 98.3% from 4B up; at 0.8B and 2B it changes more answers, so Q8_0 is the better choice there. The original rows are the same weights run through this repository's package on the same machine; they differ from the tables above by one or two items of bf16 rounding. StartLux-Decision-35B-A3B is not available as GGUF yet.

To run one, download its repository (the GGUF file plus the small files the decision server needs), then start llama.cpp and the server (for another precision, replace Q8_0):

hf download startlux-models/StartLux-Decision-4B-Q8_0-GGUF --local-dir StartLux-Decision-4B-Q8_0-GGUF
cd StartLux-Decision-4B-Q8_0-GGUF && pip install -r requirements.txt
llama-server -m StartLux-Decision-4B-Q8_0.gguf -ngl 99 -c 16384 --parallel 4 --port 8081 &
python -m startlux_decision.gguf_server --model-dir . --llama http://127.0.0.1:8081 --port 8090

Jev's numbers are the ones its harness authors report, except Dino Run and the chess match, which we ran for Jev through its API in the same harness as ours. The chess match is 256 games: 128 level openings (within 0.6 pawns by Stockfish), each played once with each colour, at the harness's default rich level. Games end by the rules or are adjudicated at ±300 centipawns after 160 plies; since neither side searches, most draws are fivefold repetitions. StartLux-Decision-27B scores 58.4%, +59 Elo (95% interval +37 to +82), and wins 64 of the 85 decisive games. With the harness's tactical hints, which give both sides the one-move consequences of every move, the two are even (48.6% and 46.1% over 256 games each); all four conditions are in docs/results.md. The misheard-names variant is the same set of lines as a lower-case speech-to-text transcript in which names are misheard. The 0.8B and 2B models do much worse on these games. Every game, position and line is in results/games: Elo ladder games with PGN, chess positions and mate-in-one answers, and NPC predictions per line in all three transcript variants.

The weights are on Hugging Face, as the original checkpoints for this package and as GGUF files for llama.cpp (see GGUF). Each model folder also carries the inference package from this repository.

hf download startlux-models/StartLux-Decision-4B --local-dir StartLux-Decision-4B
pip install -r requirements.txt
python -m startlux_decision.check StartLux-Decision-4B          # must print "fast kernels: active" (on a Mac: MLX)
python -m startlux_decision.server --model StartLux-Decision-4B --port 8090

On Apple Silicon the same commands run the model with MLX; add --int8 on an M5 or later. See Apple Silicon (MLX).

curl -s localhost:8090/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"ticket": "I was charged twice for order #4411 and the app still shows it as unpaid."},
  "questions": {
    "team":   {"type": "choice", "instructions": "Which team should handle this ticket?",
               "criteria": {"billing": "Payments, refunds and invoices",
                            "shipping": "Delivery and tracking",
                            "technical": "App, login and account problems"}},
    "urgent": {"type": "noul", "instructions": "Should this ticket be answered today?"},
    "severity": {"type": "score", "instructions": "How severe is the impact?",
                 "criteria": ["cosmetic", "annoying", "blocks the customer"]}
  }
}'

Or in Python:

from startlux_decision import StartLuxDecision

m = StartLuxDecision("StartLux-Decision-4B")
answers, usage = m.decide(state, questions)          # one request
many = m.decide_batch([(state, questions), ...])     # many requests, batched together

docs/inference.md covers the request format, the speed-ups and FP8. docs/evaluation.md explains how to reproduce every number above, including the batched Decision Index runner. docs/finetuning.md shows how to adapt a model to your own decisions.

startlux_decision/       inference: prompt rendering, letter readout, CUDA graphs, long inputs, images, MLX for Apple Silicon, HTTP servers (also for GGUF), kernel check
demos/          the computer-use harness and its two mock sites, and the chess match tools
eval/           evaluation: Intern-Decision suites and JevBench public tiers, Typed Decisions, Decision Index, latency
finetune/       LoRA fine-tuning on your own data and temperature calibration
docs/           inference, evaluation, fine-tuning and full results
results/        our numbers as JSON and CSV, per-benchmark Decision Index scores, raw game and computer-use logs
media/          figures and recordings used here

The code in this repository is Apache-2.0 (LICENSE). The model weights on Hugging Face are released under CC BY-NC 4.0: free for research and other non-commercial use, with attribution. Commercial use requires a separate license from StartLux Labs; contact contact@startlux.com. Benchmark data is fetched from its original sources under their own terms.