Magics

As mentioned earlier, I wanted to try Magic bitboards for the sliding pieces (rooks, bishop and queens). Magic bitboards are actually pretty easy once you get the gist of it. There are websites that explain the concept quite nicely (such as www.rivalchess.com).  Basically you take the attack board of your sliding piece, minus the edges.  For example, a rook on a1:

(bitboard 1)
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 1 1 1 1 1 1 0

This board represents all the squares that the piece can go to, if the board was completely empty.  These can easily be calculated before-hand and stored in an array (long[64]).  I call these the attack bitboard.

(I did not quite understand why the edge squares should be left out.  It's actually pretty easy... Less bits equal less complexity and the piece will always stop on an edge, so we can simplify the board by leaving them out from the start)

...this values gets bitwise "and"-ed with the board containing all the pieces, or the occupancy board.  The result is a board, which indicate all the piece blockers for a rook stationed at a1 (i.e. blocker bitboards).  The big problem and the reason for the various techniques (rotated bitboards, magics, etc.) is that we only really want the first blocker in a direction.  For example, the following blocker bitboards:

(bitboard 2,3 and 4)
0 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 1 1 1 1 0    0 0 0 1 0 0 0 1    0 0 0 1 0 0 0 0

could all reduce to:

(bitboard 5)

0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 1 0 0 0 0

...and in turn, this board should be used to provide us with:

(bitboard 6)

0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 1 1 1 0 0 0 0

Bitboard 5 (reduced bitboard) indicates a blocker on a6 and d1.  Bitboard 6 represents a rook on a1 that can move to b1,c1 and potentially capture on d1, as well as move to a2, a3, a4, a5, (result bitboard) with a potential capture on a6.  (To determine if we can capture/move on d1 and a6 is trivial as we only need to "and" the result bitboard with the inverse on the friendly pieces.  If d1 or a6 is still set, it means that there the square is either empty or contains an enemy piece that can be captured).

Anyway, back to the problem.  If we had infinite memory, we could create a table long[9,223,372,036,854,775,808] and store the result bitboard for each blocker board (i.e. bitboard 6 for bitboards 2,3 and 4).  However... we don't have that amount of memory, and even if we had, it will be a terrible waste.  This is where Magic bitboards comes in very handy.  Simply take your blocker board, multiply it with some magic number and then simply mod it with a power of 2 (which is essentially a shift).  You end up with some index number which is then used for a lookup in a much smaller database, i.e.:

((attack-bitboards-for-rooks[a1] AND occupancy) * magic-number[a1]) >> (64 - bits-required[a1])

It turns out, that it is easy to find unique locations for a database the size 2 to the power of the number of bits in the original attack board (and thanks to the missing edge-bits, it is 2 bits smaller).  For example, a rook on a1 requires a 12-bit database and we therefore have to shift right with 52 bits, leaving an index in the range 0...4095.

A chess program will typically have a magic number for each piece for each square (i.e. 128 in total), as well as different space requirements for each square, ranging from 5 to 12 bits (and possibly less...).

Finding the magics can be done brute force, using random numbers.  To find a magic number for a specific square, for a specific piece, it boils down to the following:

1. Pick some random 64-bit number, call it magic.
2. Get all the variations for the piece on the square (i.e. for example bb 2,3 and 4).
3. Create a database of size 2^bits (starting of with the number of bits in the original attack-board, i.e. bb 1).
4. For each attack variation
1. Calculate the result bitboard for the variation, i.e. bb 6 for bb 2,3 and 4. 
2. Create an index by multiplying variation with magic and shift it right with 64 - bits.
3. If database[index] is not 0 and the result board stored there is not compatible with variation, then we have a clash and we have to try a new magic number.
4. Otherwise, we store the result bitboard for variation at database[index].
5. If the loop ends with no clashes, then magic is a valid magic number for the piece on a1.

It takes a few seconds to find all 128 magics, if the bits is equal to the number of bits in the original attack bitboard.  However, for a rook on a1, we actually only need a database of size 49. There are only 49 unique configurations with a blocker on the a-rank and first-file.  To find a magic number that magically hashes all 4096 variations to a database of 49 result bitboards is a monumental task and turns out to be ridiculously difficult. In fact, no one really knows whether it is possible or not.

Fortunately, we don't have to go the absolute minimum.  Shaving off a single bit, will half the space requirements for that square already.  So instead of searching for a magic for a 12-bit database, we could try and search for a magic for an 11-bit database for the rook on a1.  This is still difficult, but doable...

I have written a program that searches for magics for all pieces, all squares for all database sizes.  So far, all my efforts have produced, the following:

Piece	Square	Bits in attack board	Bits required for database	Magic! (base-16)
Bishop	a1	6	5	90a207c5e7ae23ff
Bishop	b1	5	4	caad949dbf3ffed4
Bishop	g1	5	4	2a0f37664bfef560
Bishop	h1	6	5	f174631c5963ffd7
Bishop	a2	5	4	b2d764c66b3617f5
Bishop	b2	5	4	f97758b06ebb3f25
Bishop	g2	5	4	2ba278d2d3287f3f
Bishop	h2	5	4	0ca85d99b283fe74
Bishop	a3	5	4	8440f94bf539afdb
Bishop	b3	5	4	c6e0343a763ab7f6
Bishop	g3	5	4	78b4002cccfabedf
Bishop	h3	5	4	cd4a03745eacbfd0
Bishop	a6	5	4	24ffd9f54b3ec007
Bishop	b6	5	4	6ccffd59e5fae014
Bishop	g6	5	4	2eff79dda4a23407
Bishop	a7	5	4	e3dff263999c90f5
Bishop	b7	5	4	f06bfd8ab019c8bd
Bishop	g7	5	4	3fffa27235d24ee9
Bishop	h7	5	4	cf3fa4b1f1bdb60c
Bishop	a8	6	5	83c3fd4a7d787936
Bishop	b8	5	4	f317ee03f569a587
Bishop	g8	5	4	9ad97f53b94894e3
Bishop	h8	6	5	91fff9fd4502052f

If you are interested in the code for the above, drop me a mail.  I will send it to you and put it on here if enough people bug me.  I will continue the search and keep this table updated.