Protocols

Puzzles puzzles v2

The sample is 1,000,000 puzzles of the Lichess puzzle database: every mate in 3, 4 and 5 moves, and the others in proportion by kind and band of 200 rating points, with seed 1. Each puzzle has a weight, the number of puzzles of the database that it stands for.

Each move of the solver is searched from empty tables with one thread and a hash of 16 MB, limited by nodes, by depth, or by time per move (with MoveOverhead 0).

A move is right when it is the move of the solution or when it gives checkmate. A puzzle is solved when every move of the solver is right, and the line stops at the first wrong move.

Shares are weighted, with Wilson intervals at 95% from the effective sample size. The puzzle rating is the rating at which the logistic model of solving a puzzle fits the weighted results best.

Search to a fixed depth search-nodes v1

Each position of a suite of 15 (5 openings, 5 middlegames, 5 endgames) is searched from empty tables to each depth, with one thread and a hash of 16 MB.

The moves, scores and node counts must be equal in every round, and the time is the median over the rounds, with the configurations interleaved.

Move generation rules v1

Six reference positions are counted by perft at every depth whose published count is at most 200 million nodes, and compared with the published counts. The deepest such depth of each position is repeated 7 times for the speed.

Match match v2

Two engines play pairs of games from the openings of the book 8moves_v3 (the two games of an opening swap the colors), with one thread and a hash of 16 MB, at a fixed time per move, depth or node count. A move that takes up to 20 ms longer than its time per move is not lost on time.

The sequential test (SPRT) has Elo bounds 0 and 10 and error probabilities 0.05, on the pentanomial counts of the pairs. The verdicts of meristats and of fastchess are both stored.

Comparisons compare v1

Two runs on the same puzzles are compared. The difference is in the share solved, A minus B, with a 95% interval from the weighted differences, and the exact McNemar test counts the puzzles that only one of them solved. A difference whose interval holds 0 is inside the noise.

Rating against Stockfish ladder v2

The engine plays Stockfish 19 at several levels (UCI_LimitStrength and UCI_Elo), and the rating is fitted from all levels together, with the levels as fixed ratings.

The rating is on the scale of UCI_Elo, which Stockfish calibrates for longer time controls, so it holds for this speed on this machine and is not a rating-list rating.