Tournaments

I run chess engine tournaments using the Myracle GUI.

Structure

• Tournament
  • Divisional
    • 12 Engines per Division
    • Type = Round Robin
    • Cycles = 200
    • 2,200 Games per Engine = 13,200 Games per Tournament
  • Cross-Division
    • 6 Engines
    • 3 from Upper Division
    • 3 from Lower Division
    • Type = Round Robin
    • Cycles = 200
    • 1,000 Games per Engine = 3,000 Games per Tournament
• Time Control
  • Moves = All
  • Lightning = 30 sec / game +  0.3 sec / move
  • Bullet =     2 min / game +  1   sec / move
  • Blitz =      5 min / game +  3   sec / move
  • Rapid =     15 min / game + 10   sec / move
• Opening Book
  • File = Arasan.pgn (20,376 positions)
  • Maximum Plies = All
• Adjudication = None except for rapid time control. See Rapid Ratings for details.
• Ponder = Unchecked
• Repeat Reverse = Checked
• Repeat Openings = Checked

Multithreading

I configure all engines to run using a single CPU. My Chess PC has 64 CPU cores (128 logical processors with hyper-threading). I configure Myracle to use 60 cores via Tools > Settings > Concurrency > Core Budget. This leaves 4 cores (8 logical processors) for the operating system and other applications.

Not related to multithreading but related to the issue of fairness: I configure all engines to use 128 MB for their hash / transposition table.

Quality Control

The Myracle GUI makes it very easy to determine if an engine repeatedly causes game terminations due to illegal moves, UCI disconnects, or other technical issues. I don’t need to search through a tournament’s PGN file for telltale words such as “abandon” or “forfeit”. Myracle displays a count of game anomalies in the tournament crosstable and provides an anomaly filter in the tournament games list. Buggy engines have nowhere to hide.

If any engine repeatedly causes such terminations, I remove the engine from my PC (to prevent accidentally including it in future tournaments), download a replacement engine of similar strength, and restart the tournament. If I don’t notice terminations until the tournament has progressed significantly, I’ll wait for the tournament to complete, remove the buggy engine’s games (using Hiarcs Chess Explorer Pro), then run a gauntlet tournament for a replacement engine matched against the same opponents of the buggy engine.

Myracle provides a tournament editing feature that may simplify the process of replacing a buggy engine. But I haven’t encountered buggy engines recently, so I haven’t experimented with this feature.

My quality control process eliminates invalid games that artificially raise the rating of other engines by crediting them cheap wins. I am willing to tolerate a few terminated games per engine. However, any engine with a failure rate higher than one tenth of one percent reduces the reliability of my rating lists (not really, I just don’t tolerate bad code), so I replace the buggy engine with a reliable engine of similar playing strength.