Tuesday, 09 April 2013

Magics

As mentioned earlier, I wanted to try Magic bitboards for the sliding pieces (rooks, bishop and queens). Magic bitboards are actually pretty easy once you get the gist of it. There are websites that explain the concept quite nicely (such as www.rivalchess.com).  Basically you take the attack board of your sliding piece, minus the edges.  For example, a rook on a1:

(bitboard 1)
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 1 1 1 1 1 1 0

This board represents all the squares that the piece can go to, if the board was completely empty.  These can easily be calculated before-hand and stored in an array (long[64]).  I call these the attack bitboard.

(I did not quite understand why the edge squares should be left out.  It's actually pretty easy... Less bits equal less complexity and the piece will always stop on an edge, so we can simplify the board by leaving them out from the start)

...this values gets bitwise "and"-ed with the board containing all the pieces, or the occupancy board.  The result is a board, which indicate all the piece blockers for a rook stationed at a1 (i.e. blocker bitboards).  The big problem and the reason for the various techniques (rotated bitboards, magics, etc.) is that we only really want the first blocker in a direction.  For example, the following blocker bitboards:

(bitboard 2,3 and 4)
0 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 1 1 1 1 0    0 0 0 1 0 0 0 1    0 0 0 1 0 0 0 0

could all reduce to:

(bitboard 5)

0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 1 0 0 0 0

...and in turn, this board should be used to provide us with:

(bitboard 6)

0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 1 1 1 0 0 0 0

Bitboard 5 (reduced bitboard) indicates a blocker on a6 and d1.  Bitboard 6 represents a rook on a1 that can move to b1,c1 and potentially capture on d1, as well as move to a2, a3, a4, a5, (result bitboard) with a potential capture on a6.  (To determine if we can capture/move on d1 and a6 is trivial as we only need to "and" the result bitboard with the inverse on the friendly pieces.  If d1 or a6 is still set, it means that there the square is either empty or contains an enemy piece that can be captured).

Anyway, back to the problem.  If we had infinite memory, we could create a table long[9,223,372,036,854,775,808] and store the result bitboard for each blocker board (i.e. bitboard 6 for bitboards 2,3 and 4).  However... we don't have that amount of memory, and even if we had, it will be a terrible waste.  This is where Magic bitboards comes in very handy.  Simply take your blocker board, multiply it with some magic number and then simply mod it with a power of 2 (which is essentially a shift).  You end up with some index number which is then used for a lookup in a much smaller database, i.e.:

((attack-bitboards-for-rooks[a1] AND occupancy) * magic-number[a1]) >> (64 - bits-required[a1])

It turns out, that it is easy to find unique locations for a database the size 2 to the power of the number of bits in the original attack board (and thanks to the missing edge-bits, it is 2 bits smaller).  For example, a rook on a1 requires a 12-bit database and we therefore have to shift right with 52 bits, leaving an index in the range 0...4095.

A chess program will typically have a magic number for each piece for each square (i.e. 128 in total), as well as different space requirements for each square, ranging from 5 to 12 bits (and possibly less...).

Finding the magics can be done brute force, using random numbers.  To find a magic number for a specific square, for a specific piece, it boils down to the following:

  1. Pick some random 64-bit number, call it magic.
  2. Get all the variations for the piece on the square (i.e. for example bb 2,3 and 4).
  3. Create a database of size 2^bits (starting of with the number of bits in the original attack-board, i.e. bb 1).
  4. For each attack variation
    1. Calculate the result bitboard for the variation, i.e. bb 6 for bb 2,3 and 4. 
    2. Create an index by multiplying variation with magic and shift it right with 64 - bits.
    3. If database[index] is not 0 and the result board stored there is not compatible with variation, then we have a clash and we have to try a new magic number.
    4. Otherwise, we store the result bitboard for variation at database[index].
  5. If the loop ends with no clashes, then magic is a valid magic number for the piece on a1.

It takes a few seconds to find all 128 magics, if the bits is equal to the number of bits in the original attack bitboard.  However, for a rook on a1, we actually only need a database of size 49. There are only 49 unique configurations with a blocker on the a-rank and first-file.  To find a magic number that magically hashes all 4096 variations to a database of 49 result bitboards is a monumental task and turns out to be ridiculously difficult. In fact, no one really knows whether it is possible or not.

Fortunately, we don't have to go the absolute minimum.  Shaving off a single bit, will half the space requirements for that square already.  So instead of searching for a magic for a 12-bit database, we could try and search for a magic for an 11-bit database for the rook on a1.  This is still difficult, but doable...

I have written a program that searches for magics for all pieces, all squares for all database sizes.  So far, all my efforts have produced, the following:

Piece Square Bits in attack board Bits required for database Magic! (base-16)
Bishop a1 6 5 90a207c5e7ae23ff
Bishop b1 5 4 caad949dbf3ffed4
Bishop g1 5 4 2a0f37664bfef560
Bishop h1 6 5 f174631c5963ffd7
Bishop a2 5 4 b2d764c66b3617f5
Bishop b2 5 4 f97758b06ebb3f25
Bishop g2 5 4 2ba278d2d3287f3f
Bishop h2 5 4 0ca85d99b283fe74
Bishop a3 5 4 8440f94bf539afdb
Bishop b3 5 4 c6e0343a763ab7f6
Bishop g3 5 4 78b4002cccfabedf
Bishop h3 5 4 cd4a03745eacbfd0
Bishop a6 5 4 24ffd9f54b3ec007
Bishop b6 5 4 6ccffd59e5fae014
Bishop g6 5 4 2eff79dda4a23407
Bishop a7 5 4 e3dff263999c90f5
Bishop b7 5 4 f06bfd8ab019c8bd
Bishop g7 5 4 3fffa27235d24ee9
Bishop h7 5 4 cf3fa4b1f1bdb60c
Bishop a8 6 5 83c3fd4a7d787936
Bishop b8 5 4 f317ee03f569a587
Bishop g8 5 4 9ad97f53b94894e3
Bishop h8 6 5 91fff9fd4502052f

If you are interested in the code for the above, drop me a mail.  I will send it to you and put it on here if enough people bug me.  I will continue the search and keep this table updated.

7 comments:

Dany Anulliero said...

hi jacko!
nice to see vicki is still alive !
I am dany (jars and yoda programmer remember?)
Well I stopped the dévelopement of jars (even if it is always stronger than yoda) and I restarted yoda from scratch since some months . I wrote my own move générator with précalculated tables . I know I've may be reinvented the wheel but it's my own ;-) Now the new yoda works not bad , I have also implemented a tappered eval fonction but actualy I have some transposition table problem ( remember somethings to you?;-)) Hope the new vicki is on a good way , chess programming is addictive seems , I never stop CC but the problem is : not much time for that in this life and CC need lot of time lol
All the bests !
Dany

Dany Anulliero said...
This comment has been removed by a blog administrator.
Dany Anulliero said...
This comment has been removed by a blog administrator.
Jaco van Niekerk said...

Hi Dany!

Sure I remember you! I am also glad your are still involved in CC. CC is all about reinventing the wheel - you're own wheel! It's the fun that makes us do it.

Precalculation is a great idea to increase speed - I am very impressed with magics for the sliders. It works very well.

Keep well and thanks for the friendly post!

tttony said...

Hi Jaco

I'm trying to implement the magic bitboard too, but I'm facing problems that should not happen, I have implemented the the magic bitboard from Rival Chess I converted the code from Java to C (is not so difficult) but sometimes with some squares don't give the expected result

Even I've tested with this code and this code and the same thing happen

Here is a test that I made using the code from syzygy1/tb

I've not seen your email thats why I'm writing here, can you give some example code or I'm missing something?

Thanks, Happy New Year and good luck with your Vicki Chess Engine

Jaco van Niekerk said...

Hi tttony... I will have a look at your code sometime this week. It is 23:42 here right now. Thanks for the interest.

andres garcia said...

Hello Jaco, I was a little confused following the algorithm to generate the magics. I would appreciate it if you can send me the code?

Thanks!