Seeded run · blackbench-0.2

Model

Money won
§A
Money

Where the money went

Net units by type of hand. The rows add up to the money won.

The figures

The leaderboard columns for this run. Only money won is ranked.

EV as the run went on

Running EV per hand · shaded: 95% interval · dashed: basic strategy on the same seed

§B
Diagnostic

Basic-strategy agreement, cell by cell

A diagnostic; it doesn’t affect the ranking. Each cell shows the book’s play for that hand against the dealer’s upcard. The darker the cell, the more often the model played something else. Hover or tab through for counts.

Deviation from book H hitS standD double (else hit)Ds double (else stand)P splitR surrender (else hit)
§C
Hand log

Hand log

An excerpt from the run. The model’s returned distribution is shown with its top pick in bold. The book’s play is shown for reference.

Round-level hand log
FlagRoundDealerModel’s handDecisions · model vs bookResultNet
§D
Invalid

Rejected responses

Every response that failed validation, by kind. Each one played as the fallback.

KindExampleFallbackCount
Total

Reproduce this run

The seed and ruleset are the whole input. Same seed, same shoes.


        
§E
Method

Methodology notes

A summary. The full text is METHODOLOGY.md in the repository.

E.1The question

At each decision point the model gets the visible state as plain text: the rules, its hand with the total, the dealer’s upcard, and which hand of a split it is playing. It also gets one typed choice question whose options are only the actions legal right now. It returns a probability for each option and its pick.

E.2The shoe

Every hand is dealt from a freshly shuffled 6-deck shoe, derived from the seed and the hand number, so every model sees the same opening cards on hand n. Past cards are never shown, so card counting can’t help. Dealer stands on soft 17 and peeks for blackjack, blackjack pays 3:2, double on any two cards (also after a split), split to 4 hands (aces once, one card each), late surrender, no insurance.

E.3Scoring: money won

Every hand is a flat 1-unit bet, settled at table rates. Doubles and splits add to the stake as they would at a real table. The ranking is the total net units over 100,000 hands. EV is that total divided by hands, reported with a 95% interval from the per-hand standard deviation (about 1.15 units). At 100,000 hands that interval is roughly ±0.7 points.

E.4Reference strategies

Basic strategy and a random legal action play the same seed and appear on the leaderboard as unranked reference rows. Basic strategy is the best any player with this information can do. Random play is the floor.

E.5Validation

A response is invalid if the answer is missing, refused or malformed, names an action that wasn’t offered, or fails after a 30-second timeout and two retries. Invalid responses play as stand and count toward the invalid rate. Auth or billing errors abort the run instead of being scored.

E.6What this doesn’t measure

Card counting, bet sizing and table selection. Bets are flat. The benchmark asks one narrow question: given a state and a typed question, does the model make the decision that maximizes EV?

E.7Reproducibility

Every answer is cached by model, protocol, rules and exact request, so a run replays exactly without new calls. Requests go through OpenRouter, which may route them to several providers; the providers used are listed with each run. Suits shown in the hand log are cosmetic, because the engine doesn’t model suits.