top of page

The NNUE development of REBEL has come to a halt because my available hardware has become insufficient to make further progress in an acceptable time. For that reason I like to document my work during the last 2 years in the hope it will be useful. Much thanks goes to Chris Whittington who introduced me to NNUE programming.

   

The current public version is REBEL-EAS, the current best development version is REBEL-Dev about 25-30 elo stronger not meant for release. Instead I prefer to publish the data it is made off together with most of the tools that I created during the last 2 years.

nnue.jpg

The main download link is found here and is divided into 4 sections.

1. Best-Data - contains the 26.5B data split into parts of 4B the REBEL-Dev version is made off.

2. Good-Data - contains 2 theme based data sets that have increased the Rebel-Dev version each with ~5 elo.

3. New-Data - contains interesting theme based data sets we are going to test using a small dataset of 4B on which you can follow the progress and results in the REBEL-NNUE-BLOG.

4. Utils - contains the EPD based NU utility (described below) and some BIN utilities, see also below.

-----------------------------------------------------------------------------------------------------

        

The GOOD-DATA folder

     

huge-evals - is a collection of 980M midgame positions that have one thing in common, the material on the board does not fit the epd score with a large margin. For instance, [1] if the material on the board is equal and the epd score is 3.00 or higher the position is interesting to extract and [2] if the side to move is behind in material on the board with one or two pawns but the epd score is still positive then the position is also interesting to extract.  Bullet testing shows an ~5 elo progress, no LTC testing has been done.

hp - is a collection of 1.3B tactical positions. The formula - when the side to move has 2 hanging pieces (or more)  the position is interesting to extract because in most cases the hanging piece with the lowest value is lost. Done with the NU utility that has ProDeo HCE inside. Bullet testing shows an ~5 elo progress, no LTC testing has been done.

The NEW-DATA folder

A collection of untested but promising data.

     

ks - a collection of 934M epds created with NU containing only positions with a ProDeo King Safety evaluation of 0.25 or higher.

   

mob - a collection of 389M epds created with NU containing only positions with a ProDeo Mobility evaluation of 0.50 or higher.

   

special - a collection of 481M epds created with NU+[F2] containing dangerous king pins mainly but also positions with an untouchable bishop pair (bishop can't be captured by an opponents knight or bishop), dangerous xrays, rare pins like a white rook on D8 while the black queen side is undeveloped.

  

horizon-effects - a collection of 942M epds that suffer from horizon effects. Created with NU+[F3]

    

queen-imbalances - Imbalance Queen. 72M positions. Created with NU+[F6]

rook-imbalances - Imbalance Rook. 960M positions split into 2 parts. Created with NU+[F6] 

bishop-pair - When one side has the bishop pair and the other side has no bishops. 96M positions. Created with NU+[F6]

basic - some basic 5-men stuff. 104M positions. Created with NU+[F6]

REMARK

All the epd data is quiet, meaning that the data does not contain captures, promotions or king in check situations and if your epd data does run the NU utility - Make EPD Quiet to remove them since they are too material based for the trainer.

------------------------------------------------------------------------------------------

    

The NU utility

nu.png

The NU utility is fully EPD based for compatibility reasons. The main focus is to extract interesting positions that may give elo as variety is important.

The tool is key board driven using the arrow keys. For each menu function a short description is given on the last 2 rows.

Extract Endgame - From an EPD collection extract the late endgame positions. In total 9  late endgame patterns are stored in separate EPD files in the EPD folder.

Extract KS/MOB/HP - From an EPD collection extract interesting King Safety, Mobility and Tactical  positions with ProDeo HCE evaluation value into 3 separate files. King Safety uses a margin of 0.25, Mobility a margin of 0.50, these margins can be changed in the ProDeo.eng file in the personal folder.

Extract 5-men and then Analyze 5-men requires the 5-men Syzygy bases installed in the root of the C-drive, you can download the 5-men from the utils folder.

Function keys

[F1] - Trims an epd based on its score via a margin, for instance if you type 900 all epds with an abs score higher than 9.00 will be culled.

 

[F2] - Extract special positions based on ProDeo HCE evaluation, dangerous king pins mainly but also positions with an untouchable bishop pair (bishop can't be captured by an opponents knight or bishop), dangerous xrays, rare pins like a white rook on D8 while the black queen side is undeveloped. 

[F3] - Extract horizon effects - since we only use low depth material horizon effects are plenty and how strange it might sound are needed, the trainer has to learn the difference between winning and losing after all. The function offers to add more horizon effects to your net. A typical trial on error case. Remark - this function does not work properly with random or shuffled epds but only with epds extracted from complete games.

[F4] - The REBEL-EAS approach - King Safety and Mobility were heavily used, the epd scores of King Safety and Mobility even increased with 25% with this function to get the desired effect, to play more aggressive measured with the EAS tool.

[F5] - EPD shuffler - one phase shuffle. Installation Python required.  EDP's are shuffled and stored in parts of 100,000 and then the stored parts of 100,000 are randomly appended into one file back into the EPD folder as shuffled.epd. For the best results the trainer needs shuffled data.

[F6] - Extract imbalances - 4 new interesting data ideas.

1. Basic 5-men stuff, KPK, lone knight or bishop vs 2 pawns or more, lone rook vs 3 pawns or more, KBNK, KQKR.

2. Imbalance Queen, for instance QvsRR, QvsRB, QvsRN, QvsBBN etc.

3. Imbalance Rook, for instance RvsBN, RvsB, RvsN.

4. Bishop Pair, when one side has the bishop pair and the other side has no bishops.

        

[F7] - A fun function, from the downloadable epds make a PGN and create a Polyglot book. It gives some insight about the openings that are played to create data.

[F8] - EPD sanity check. For the best results NNUE data should be in balance, equal white wins, equal black wins and a draw percentage not higher than 55%. Ideally 30 | 30 | 40 or something close. Furthermore a Material Division and Piece Count Overview statistic are created. Also here balance is important.

[F9] - The Bishop ending is notorious for programmers because there are 2 types, the same colored bishop ending and the unequal colored bishop ending each having its own dynamics. This function extracts 4 types.

1. Unequal bishop endings.

2. Equal bishop endings.

3. Equal and unequal bishop endings [1+2]

4. Also extracted are the so called bad-bishop-ending KBPK cases.

[F10] - Delete suspect Draws, Wins and Losses.

1. An epd that is a draw with a score >=  abs(5.00) is removed. 

2. An epd that is a win  with a score <=  -3.00 is removed. 

3. An epd that is a loss  with a score >=  3.00 is removed. 

A new file is created without the suspect cases.

[F12] - Tuning Piece Division,  this is how our 4B test data looks like

when we make an overview from the 4B data with NU [F8]. As you

see some are not well synchronized, especially the high odd piece

counts. With NU [F12]  we create extra data to fill the gaps in order

that the data is better synchronized.  Click on the picture to view

the data before and after. Edit the file balance-piece-count.txt

and in millions enter the data that should be added for each of the

low piece-counts. And good for ~10 elo, in this particular case.

    

Another use of NU [F12] is to set everything (for instance) on "1" in balance-piece-count.txt witch will create a new epd where every piece count has exactly 1.000.000 positions. In other words you can design any piece-count division data you wish, from midgame to endgame.

Remaining functions

    

Modify EPD evals - Increase / Decrease scores in an epd file.

    

Trim Draws  - Trim draws with 50% when there is not much to play for based on ProDeo HCE evaluation. Ideally a neural net contains a balanced division between won games by white, won games by black and a draw rate between 45-55%. When  the draw rate is too high then with this tool you can manipulate the draw rate. Used criteria to trim draws are, [1] low on King Safety, [2] low on Mobility, [3] low on passed pawn evaluation and [4] is not tactical in nature.

    

Reduce Draw Rate - Like Trim Draws but random by an user defined percentage. The tool will first quickly calculate the draw rate of the first 10 million positions and thereafter you can enter the percentage draws to be culled.

Known issues

The display of the EPD counters on the screen are 32 bit meaning the display has a reach of 2.147.483.647 position before it starts counting down. At the start of the NNUE project I had no idea if I ever would cross that value, if memory serves me well the first net only had a size of 400 or 600M. One day I will convert the counters to 64 bit. It has of course no effect on the functionality of the tools.

-----------------------------------------------------------------------------------------------------

*.BIN utilities

BIN files are compressed edps with a fixed size of 40 bytes using the Stockfish data layout. If your bins have the same data layout you can safely use the below tools.

Split-bins.exe - Split large bin files into smaller parts. Aside for the filename to split you are also prompted to type the split size in millions. If you type 10 the bin file is split into parts of 10 million.

Count-double-bins.exe - Massive double positions in your data can be a problem for the trainer. But how can you possibly know with giant data of xx billion positions since checking for doubles requires sorting the data first which is mission impossible, unless you have 1tb of memory or more. This tool can help  in a limited way by testing only a suitable percentage. The higher the percentage the longer it takes, the more accurate the double check is. For small bins use 100%, for giant bins use 1, 5 or 10%.

Compare-two-bins-for doubles.exe - Same as Count-double-bins.exe but now compare two bin files for doubles.

Merge-bins.cpp - For developing REBEL (as you must have noted from the NU tool) I intensively use extracted specific pattern data from other sources and then add them to my current best data to see if it is an improvement. To avoid a complete shuffle run of the data that for 30B data takes about half a day even with fast SSD's this tool merges the new data into the shuffled 30B in such a way the new data is still well shuffled which decreases the process with a factor 5-6.

Edit the 3 files in the code, best spread over 3 different SSD drives.

char bin_one[256] = "n:/train/best-data.bin";    
char bin_two[256] = "f:/nu/ks.bin";
char bin_out[256] = "e:/temp/new-";
 

Compile and run.

When finished -- and only if you want the merged data into one big file -- go to the "e:/temp" folder and type : copy /b new-*.bin new-data.bin
 

-----------------------------------------------------------------------------------------------------

Piece-Count-Update.jpg

                                  Credits

Chris Whittington for the dataloader utility.

Stockfish 10 for Syzygy access.

Ronald de Man for Syzygy bases.

Leela team for data.

bottom of page