Emulate Matches

EMULATE is a tool that creates random matches between 2 identical hypotetical chess engines to measure how many games it needs to finally settle on 50%. One would expect that a few hundred games are enough as after all the engines are identical. That logical assumption is far beyond the truth as the below examples will reveal. The whole idea of the tool to get insight in the phenomenon : How many games should one play to get a reliable result from my self-play matches.

Let's begin with a 100 game match.

1. 10 random matches are created.

2. each 100 game match is dived by 4 (so 25 games) to watch its progress.

3. Only 1 of the 10 matches comes close (49.5%) to the real strength of the identical engines.

4. Conclusion, playing only 100 games is a very unreliable way to measure strength.

Playing 10 matches of 1.000 games

1. Only the first match (49.8%) comes close to 50%.

2. The last match (52.9%) shows a difference of 20 elo !

3. Conclusion, DON'T.

Playing 10 matches of 5.000 games

1. Already looking much better.

2. Match scores fluctuating between -5½ elo (match 2) and +1½ elo (match 9).

3. Conclusion, nice but I want more security.

In search of the perfect 50%, how many games ?

. In a nutshell with 10.000 games match scores using 20 rounds all scores fluctuate between -3 and +2 elo.

. With 25.000 games match scores using 20 rounds all scores also fluctuate between -3 and +2 elo.

. With 50.000 games match scores using 20 rounds all scores fluctuate between -1 and +1 elo. We can measure 1 elo !!

REMARKS

. I wrote this tool decades ago as a guide to play reliable self play matches.

. However since nowadays engines play so strong the numbers are outdated because of the high draw rates of engines.

. Self play matches between 3500 elo rated engines easily produce a draw rate of >80% and at longer time control >90%.

. Hence the draw rate input is inserted in the tool, it makes a difference.

. Check it out for yourself.

Emulate.zip