Extract Endgame - From an EPD collection extract the late endgame positions. In total 9  late endgame patterns are stored in separate EPD files in the EPD folder.

​

Extract KS/MOB/HP - From an EPD collection extract interesting King Safety, Mobility and Tactical  positions with ProDeo HCE evaluation value into 3 separate files. King Safety uses a margin of 0.25, Mobility a margin of 0.50, these margins can be changed in the ProDeo.eng file in the personal folder.

​

Extract 5-men and then Analyze 5-men requires the 5-men Syzygy bases installed in the root of the C-drive, you can download the 5-men from the utils folder.

​

Function keys

​

[F1] - Trims an epd based on its score via a margin, for instance if you type 900 all epds with an abs score higher than 9.00 will be culled.

 

[F2] - Extract special positions based on ProDeo HCE evaluation, dangerous king pins mainly but also positions with an untouchable bishop pair (bishop can't be captured by an opponents knight or bishop), dangerous xrays, rare pins like a white rook on D8 while the black queen side is undeveloped. 

​

[F3] - Extract horizon effects - since we only use low depth material horizon effects are plenty and how strange it might sound are needed, the trainer has to learn the difference between winning and losing after all. The function offers to add more horizon effects to your net. A typical trial on error case. Remark - this function does not work properly with random or shuffled epds but only with epds extracted from complete games.

​

[F4] - The REBEL-EAS approach - King Safety and Mobility were heavily used, the epd scores of King Safety and Mobility even increased with 25% with this function to get the desired effect, to play more aggressive measured with the EAS tool.

​

[F5] - EPD shuffler - one phase shuffle. Installation Python required.  EDP's are shuffled and stored in parts of 100,000 and then the stored parts of 100,000 are randomly appended into one file back into the EPD folder as shuffled.epd. For the best results the trainer needs shuffled data.

​

[F6] - Extract imbalances - 4 new interesting data ideas.

1. Basic 5-men stuff, KPK, lone knight or bishop vs 2 pawns or more, lone rook vs 3 pawns or more, KBNK, KQKR.

2. Imbalance Queen, for instance QvsRR, QvsRB, QvsRN, QvsBBN etc.

3. Imbalance Rook, for instance RvsBN, RvsB, RvsN.

4. Bishop Pair, when one side has the bishop pair and the other side has no bishops.

        

[F7] - A fun function, from the downloadable epds make a PGN and create a Polyglot book. It gives some insight about the openings that are played to create data.

​

[F8] - EPD sanity check. For the best results NNUE data should be in balance, equal white wins, equal black wins and a draw percentage not higher than 55%. Ideally 30 | 30 | 40 or something close. Furthermore a Material Division and Piece Count Overview statistic are created. Also here balance is important.

​

[F9] - The Bishop ending is notorious for programmers because there are 2 types, the same colored bishop ending and the unequal colored bishop ending each having its own dynamics. This function extracts 4 types.

1. Unequal bishop endings.

2. Equal bishop endings.

3. Equal and unequal bishop endings [1+2]

4. Also extracted are the so called bad-bishop-ending KBPK cases.

​

[F10] - Delete suspect Draws, Wins and Losses.

1. An epd that is a draw with a score >=  abs(5.00) is removed. 

2. An epd that is a win  with a score <=  -3.00 is removed. 

3. An epd that is a loss  with a score >=  3.00 is removed. 

A new file is created without the suspect cases.

​

[F12] - Tuning Piece Division,  this is how our 4B test data looks like

when we make an overview from the 4B data with NU [F8]. As you

see some are not well synchronized, especially the high odd piece

counts. With NU [F12]  we create extra data to fill the gaps in order

that the data is better synchronized.  Click on the picture to view

the data before and after. Edit the file balance-piece-count.txt

and in millions enter the data that should be added for each of the

low piece-counts. And good for ~10 elo, in this particular case.

    

Another use of NU [F12] is to set everything (for instance) on "1" in balance-piece-count.txt witch will create a new epd where every piece count has exactly 1.000.000 positions. In other words you can design any piece-count division data you wish, from midgame to endgame.

​

Remaining functions

    

Modify EPD evals - Increase / Decrease scores in an epd file.

    

Trim Draws  - Trim draws with 50% when there is not much to play for based on ProDeo HCE evaluation. Ideally a neural net contains a balanced division between won games by white, won games by black and a draw rate between 45-55%. When  the draw rate is too high then with this tool you can manipulate the draw rate. Used criteria to trim draws are, [1] low on King Safety, [2] low on Mobility, [3] low on passed pawn evaluation and [4] is not tactical in nature.

    

Reduce Draw Rate - Like Trim Draws but random by an user defined percentage. The tool will first quickly calculate the draw rate of the first 10 million positions and thereafter you can enter the percentage draws to be culled.

​

Known issues

The display of the EPD counters on the screen are 32 bit meaning the display has a reach of 2.147.483.647 position before it starts counting down. At the start of the NNUE project I had no idea if I ever would cross that value, if memory serves me well the first net only had a size of 400 or 600M. One day I will convert the counters to 64 bit. It has of course no effect on the functionality of the tools.

​

-----------------------------------------------------------------------------------------------------

​

*.BIN utilities

BIN files are compressed edps with a fixed size of 40 bytes using the Stockfish data layout. If your bins have the same data layout you can safely use the below tools.

​

Split-bins.exe - Split large bin files into smaller parts. Aside for the filename to split you are also prompted to type the split size in millions. If you type 10 the bin file is split into parts of 10 million.

​

Count-double-bins.exe - Massive double positions in your data can be a problem for the trainer. But how can you possibly know with giant data of xx billion positions since checking for doubles requires sorting the data first which is mission impossible, unless you have 1tb of memory or more. This tool can help  in a limited way by testing only a suitable percentage. The higher the percentage the longer it takes, the more accurate the double check is. For small bins use 100%, for giant bins use 1, 5 or 10%.

​

Compare-two-bins-for doubles.exe - Same as Count-double-bins.exe but now compare two bin files for doubles.

​

Merge-bins.cpp - For developing REBEL (as you must have noted from the NU tool) I intensively use extracted specific pattern data from other sources and then add them to my current best data to see if it is an improvement. To avoid a complete shuffle run of the data that for 30B data takes about half a day even with fast SSD's this tool merges the new data into the shuffled 30B in such a way the new data is still well shuffled which decreases the process with a factor 5-6.

​

Edit the 3 files in the code, best spread over 3 different SSD drives.

char bin_one[256] = "n:/train/best-data.bin";    
char bin_two[256] = "f:/nu/ks.bin";
char bin_out[256] = "e:/temp/new-";
 

Compile and run.

​

When finished -- and only if you want the merged data into one big file -- go to the "e:/temp" folder and type : copy /b new-*.bin new-data.bin
 

-----------------------------------------------------------------------------------------------------