Magics

As mentioned earlier, I wanted to try Magic bitboards for the sliding pieces (rooks, bishop and queens). Magic bitboards are actually pretty easy once you get the gist of it. There are websites that explain the concept quite nicely (such as www.rivalchess.com).  Basically you take the attack board of your sliding piece, minus the edges.  For example, a rook on a1:

(bitboard 1)
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 1 1 1 1 1 1 0

This board represents all the squares that the piece can go to, if the board was completely empty.  These can easily be calculated before-hand and stored in an array (long[64]).  I call these the attack bitboard.

(I did not quite understand why the edge squares should be left out.  It's actually pretty easy... Less bits equal less complexity and the piece will always stop on an edge, so we can simplify the board by leaving them out from the start)

...this values gets bitwise "and"-ed with the board containing all the pieces, or the occupancy board.  The result is a board, which indicate all the piece blockers for a rook stationed at a1 (i.e. blocker bitboards).  The big problem and the reason for the various techniques (rotated bitboards, magics, etc.) is that we only really want the first blocker in a direction.  For example, the following blocker bitboards:

(bitboard 2,3 and 4)
0 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0    1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0
0 0 0 1 1 1 1 0    0 0 0 1 0 0 0 1    0 0 0 1 0 0 0 0

could all reduce to:

(bitboard 5)

0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
0 0 0 1 0 0 0 0

...and in turn, this board should be used to provide us with:

(bitboard 6)

0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0
0 1 1 1 0 0 0 0

Bitboard 5 (reduced bitboard) indicates a blocker on a6 and d1.  Bitboard 6 represents a rook on a1 that can move to b1,c1 and potentially capture on d1, as well as move to a2, a3, a4, a5, (result bitboard) with a potential capture on a6.  (To determine if we can capture/move on d1 and a6 is trivial as we only need to "and" the result bitboard with the inverse on the friendly pieces.  If d1 or a6 is still set, it means that there the square is either empty or contains an enemy piece that can be captured).

Anyway, back to the problem.  If we had infinite memory, we could create a table long[9,223,372,036,854,775,808] and store the result bitboard for each blocker board (i.e. bitboard 6 for bitboards 2,3 and 4).  However... we don't have that amount of memory, and even if we had, it will be a terrible waste.  This is where Magic bitboards comes in very handy.  Simply take your blocker board, multiply it with some magic number and then simply mod it with a power of 2 (which is essentially a shift).  You end up with some index number which is then used for a lookup in a much smaller database, i.e.:

((attack-bitboards-for-rooks[a1] AND occupancy) * magic-number[a1]) >> (64 - bits-required[a1])

It turns out, that it is easy to find unique locations for a database the size 2 to the power of the number of bits in the original attack board (and thanks to the missing edge-bits, it is 2 bits smaller).  For example, a rook on a1 requires a 12-bit database and we therefore have to shift right with 52 bits, leaving an index in the range 0...4095.

A chess program will typically have a magic number for each piece for each square (i.e. 128 in total), as well as different space requirements for each square, ranging from 5 to 12 bits (and possibly less...).

Finding the magics can be done brute force, using random numbers.  To find a magic number for a specific square, for a specific piece, it boils down to the following:

1. Pick some random 64-bit number, call it magic.
2. Get all the variations for the piece on the square (i.e. for example bb 2,3 and 4).
3. Create a database of size 2^bits (starting of with the number of bits in the original attack-board, i.e. bb 1).
4. For each attack variation
1. Calculate the result bitboard for the variation, i.e. bb 6 for bb 2,3 and 4. 
2. Create an index by multiplying variation with magic and shift it right with 64 - bits.
3. If database[index] is not 0 and the result board stored there is not compatible with variation, then we have a clash and we have to try a new magic number.
4. Otherwise, we store the result bitboard for variation at database[index].
5. If the loop ends with no clashes, then magic is a valid magic number for the piece on a1.

It takes a few seconds to find all 128 magics, if the bits is equal to the number of bits in the original attack bitboard.  However, for a rook on a1, we actually only need a database of size 49. There are only 49 unique configurations with a blocker on the a-rank and first-file.  To find a magic number that magically hashes all 4096 variations to a database of 49 result bitboards is a monumental task and turns out to be ridiculously difficult. In fact, no one really knows whether it is possible or not.

Fortunately, we don't have to go the absolute minimum.  Shaving off a single bit, will half the space requirements for that square already.  So instead of searching for a magic for a 12-bit database, we could try and search for a magic for an 11-bit database for the rook on a1.  This is still difficult, but doable...

I have written a program that searches for magics for all pieces, all squares for all database sizes.  So far, all my efforts have produced, the following:

Piece	Square	Bits in attack board	Bits required for database	Magic! (base-16)
Bishop	a1	6	5	90a207c5e7ae23ff
Bishop	b1	5	4	caad949dbf3ffed4
Bishop	g1	5	4	2a0f37664bfef560
Bishop	h1	6	5	f174631c5963ffd7
Bishop	a2	5	4	b2d764c66b3617f5
Bishop	b2	5	4	f97758b06ebb3f25
Bishop	g2	5	4	2ba278d2d3287f3f
Bishop	h2	5	4	0ca85d99b283fe74
Bishop	a3	5	4	8440f94bf539afdb
Bishop	b3	5	4	c6e0343a763ab7f6
Bishop	g3	5	4	78b4002cccfabedf
Bishop	h3	5	4	cd4a03745eacbfd0
Bishop	a6	5	4	24ffd9f54b3ec007
Bishop	b6	5	4	6ccffd59e5fae014
Bishop	g6	5	4	2eff79dda4a23407
Bishop	a7	5	4	e3dff263999c90f5
Bishop	b7	5	4	f06bfd8ab019c8bd
Bishop	g7	5	4	3fffa27235d24ee9
Bishop	h7	5	4	cf3fa4b1f1bdb60c
Bishop	a8	6	5	83c3fd4a7d787936
Bishop	b8	5	4	f317ee03f569a587
Bishop	g8	5	4	9ad97f53b94894e3
Bishop	h8	6	5	91fff9fd4502052f

If you are interested in the code for the above, drop me a mail.  I will send it to you and put it on here if enough people bug me.  I will continue the search and keep this table updated.

FEN Strings

I've made some progress yesterday.  I have an elementary bitboard representation working and I can set and get the chess board using a FEN string.  A FEN String essentially represents the current board setup in a convenient notation.  Here is an example after white's initial e4:

rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1

Each string in between the slashes represents a rank in the chessboard, starting from the 8th rank. Lower case is black, upper case is white. Numbers represents empty squares between pieces.  For example '4P3' represents four empty squares, followed by a pawn, followed by three more empty squares, totaling eight.  The '/8/' therefore means (you've guessed it!) a complete empty rank.

The remaining string 'b KQkq e3 0 1' represents the side to move, the castling rights (e.g. 'K' means white can caste king-side, 'q' that black can caste queen-side, etc.), en-passant square and the move counters.  The en-passant square is the square directly behind the pawn that made the two-square move.  In this case, white move e4, which means that e3 is the en-passant square.  Black can capture the pawn if he had a pawn on either d4 or f4.  Regardless, the en-passant square is always indicated, whether it can be taken or not.  The move counts are the half-moves since a last capture or pawn move (for the 50-move draw rule), followed by the number of full moves.

I found myself constantly checking the validity of the FEN string during the parsing stage, which started to look really ugly.  So I turned to one of my best friends... regular expressions!

Regular expressions can get tricky...quickly, but is incredibly powerful to recognise patterns in strings.  Here is a short extract from Vicki that shows how a FEN string can be checked:

 
The 'verifyRank()' method simply verifies that the a rank adds up to eight. Doing that in a regular expression may get very hairy (actually I don't think it can be done). The code also verifies that there are 2 kings at least.  Of course, that can easily be done by the regular expression, but the string will get rather long. 

Move generation

The first step is move generation.  I have committed myself to Java and bitboards for Vicki 2.0.  The advantage of Java is that it is much more platform independent than C... well, mostly.  Bitboards is also a good move, given that it is theoretically faster than the postbox method and besides, it is something interesting to learn.

So the list of tasks for the next few days is as follows:

1.	Set up data structures	Done
2.	Set and get board (FEN-based)	Done
3.	Simple textual display	Done
4.	Move generation for non-sliders (pawns, knights and kings)	In progress...
5.	Move generation for sliders (bishops, rooks and queens)	Pending
6.	Make move	Pending
7.	Unmake move	Pending
8.	Perft testing	Pending

For now, the representation simply contains positions for each of the pieces.  The derived pieces will be added later.

Something stirring in the shadows...

I have only fond memories of Vicki...  The website is still up, but Vicki has not used a clock cycle in years.  The current release is several years old... written in C.

I am currently (as in already busy) rewriting Vicki.  Vicki 2.0 is a complete rewrite of the engine, reusing nothing but the name.

I am not looking at any existing computer chess code and I am avoiding any heavy bias to single papers and single units of theory - trying my own. The research in computer chess is all but exhausted.  So all that is left is to redesign the wheel for the fun of it and looking at existing code will spoil it for me... Vicki is not meant to "extend" the envelope of computer chess.  It's my Everest and South Pole.  

Vicki 2.0 is being written in Java.  Java's speed is comparable to C and C++ (in some people's opinions!) Also, I want something which is easily transferable between Linux and Windows platforms.  C does not really cut it for me any more.

Board representation will the elusive bitboard (or bitmap) technique.  It cannot be that hard... (famous last words!) and I cannot ignore it's advantages any longer.

I hope to post regularly and have something on the main website up and running soon.

Ping...

I've not posted for over a year.  Although I have not had much time to work in Vicki, the engine is definitely not dead.

I will start with a rewrite in the foreseeable future.  I am not yet sure which direction I'd like to take with Vicki - but I am toying with the following:

1. Possibly rewriting the engine in Java.  The speed difference between Java and C is negligible, when the number of objects used are limited.  Java also has better testing frameworks (well, from what I could find) and is platform independent.  I can also throw in my own GUI, if I like.
2. General improvement on absolutely everything from move generation write through to opening/ending databases.
3. Multi-threaded searching for multi-core systems.  I have access to some serious hardware at work (i.e. 64Gb RAM with 8 quad core processors).  Would be interesting to use of of those for a tournament!
4. I would like to finally get learning right!
5. Full-time up server playing on servers.

Watch this space!

Delays

Seems like I'm going to need a few more days. I've manged to split the move generation so that I now incrementally generate moves and search them (as appose to generate all the moves at once). This seems to be paying off. However, I'm having some transposition table problems again!

I'm hoping for end January.

Resurrected? ...or just out of the coma?

I guess life happened while I was busy working on my chess engine. I few months ago, I dumped the entire engine and started fresh. Then after a month or so, I dumped that idea and went back to the original codebase. I started integrating the sections of code that I rewrote and improved. To be frank, large sections of code in current release of Vicki is a terrible mess and these needed to be rewritten. I'm almost done with that.

I have already rewritten the move generation, which resulted in a speed increase of around 10%. I’ve also changed the representation of the move (I use a 32 bit integer for this) so that a full 8 bits are available for move ordering (this use to be 6). This change will greatly enhance the move ordering capabilities of Vicki.

I am now working on the actual search function and the transposition table. Apart from fixing a nasty bug with the transposition table (it returns a mate score, but just chases the king around with trivial end game positions), I also need to rethink all the pruning and speed-up techniques individually. I threw all these together without giving any much thought. I paid the price.

I'm also planning pondering, opening book learning and better interface support.

The next release is scheduled for the 4th of January 2010.

Vicki is back... and smelling blood!

Frustration

Even though I haven't posted in a while - Vicki is not dead. I'm having huge issues with Vicki at the moment which is taking up a great deal of time. The biggest problem at the moment is the transposition tables that are not working as it should. Every time I implement it, the engine plays weaker...substantially weaker. This can only be as the result of a bug. I hope to have this sorted out in the next week or so...

The transposition table is a very important component and I want that working first, before spending any more time with move ordering and evaluation as these elements share some common ground

Refactoing & revisiting

There are few things as annoying as a dormant blog - and here I am doing the exact same thing. I've recently revisit Vicki and I am in the process of refactoring the engine. I have found some bugs with the repetition detection and I am spending some time to get the engine stable. Once this is complete, I'll start working in getting the engine up to scratch.

One element that has been missing from Vicki is decent testing. For the time being I've identified 28 engines, ranging from weak to rather strong. The idea is to put Vicki in a gantlet against them and work with the final score as an indication of strength (2 rounds, from the starting position). Later on I'll try the test suites again.

Let see how it goes...

New Vicki almost ready...

I've been promising a new release for some time now... Fortunately, my thesis was accepted with little change and I can now focus that attention on my chess engine too. Vicki has been a bit a of a disappointment and I am working hard on getting her ready for WBEC Ridderkerk's 15th edition. Hopefully we'll see an engine around 2100 in stength :-)

Time-off...

Hi all

I need to take some time off from Vicki for a few weeks or so. My thesis has just arrived back from the moderators and I have some minor changes to make before I hand it in on the 24th of August... then there is also a research paper that needs to be submitted before the end of this month. Then apart from that, I also have to work to pay for everything...

So, Vicki will have to wait a while...

Sorry guys :-(

Null moves

I finally got null moves to work in Vicki. I had some trouble with the capturing of the king (which caused Vicki to hang), but I eventually got it fixed. I am utterly amazed at the effect of null moves in the engine. With null moves Vicki searches (on average) 1-2 plies deeper and even more later in the game. In late-middle games, depths of 12 plies are reached! That is without any other form of pruning or transposition tables. The new version of Vicki is already playing on FICS and I am conducting tests against the older version to evaluate the improvement.

Why didn't I have this ready for the WRCC?

Test cases and transposition tables!

The test cases was a great idea and it paid off! I detected a problem with the Zobrist keys in that en-passant was not correctly added/removed. I also detected a nasty (and embarrassing) castling bug. I omitted the break-statement after each case-statement when verifying that the castling was valid. This means that if white wants to castle king-side, queen side castling must not move through check and white must not prevent black to castling either side, either! A very silly bug! I only detected it after adding a large series of perft checks to my test cases. By squashing this bug, my engine plays much better chess already!

Transposition tables is turning out to be very annoying! I keep on getting them "working" only to find that in a duel (100 games, 3 minutes per side, different starting positions) with the previous version that is does badly... or at best, just as good. Shouldn't transposition tables increase the ELO with 200+ points? As a result I've put this on ice for awhile.

In the WRCC tournament, my engine lost a particular game by means of a long capturing sequence that ended in a check. As checks are not part of the quiescence search, my engine never saw the pending disaster. I am thinking of ways to incorporate checks into the quiescence search without slowing things down too much. But how?

I'm also going to attempt to incorporate null moves into the engine. After reading up on the concept a few months ago I was most intrigued with the concept and it would be interesting to see what effect it will have on my engine.

Vicki playing online

I've just checked on Vicki playing online at FICS. The engine is doing pretty well...actually too well. Vicki has a rating of over 2000 for standard and lightning play and 1897 for blitz (see my website for more exact figures - or better yet - log onto FICS and pay Vicki a visit. She plays under the "myvicki" handle).

I've shifted my approach when working on the code. Instead of hacking away and trying to squeeze a few ELO points in between work, family and social life (actually work and family only - sadly this blog is pretty much the limits of my social life :-(), I want to set definite milestones for the engine. The first important change is test cases. I've devoted an entire c-source file for test cases, which I'll run periodically to ensure that I did not break something while adding some nifty new feature. My "hacking"-approach is what cost me to play without a transposition table in last weekend's tournament. This needs to end...

Vicki tasted blood!

It is with great pride that I announce that Vicki is now playing on the Free Internet Chess Server (FICS) under the handle of "myvicki". Please pay her a visit! After all the fun of the WCRCC I am very motivated in getting Vicki to play decent chess!

Vicki is also competing in ChessWars XI under the promotion section. So far Vicki is not doing that bad, having drawn the first game. It would be interesting to see how she does later on.

Final standings...

The tournament was very exciting and Vicki actually won one game and drew another. For an engine that's missing serious components, I am quite surprised. However, I can't wait for the next tournament to start so that I can amaze a few people... grrr... :-)

Mediocre (blog here) did particularly well - the name really does not suit it well. It is far from being Mediocre and to think it is all written in Java! Well done to Jonatan Peterson. He's a real inspiration!

Just for fun, here is the game that Vicki won (my little engine was playing black). I hope to have many of these soon!!

Official tournament!

Hi all...

I haver entered Vicki into the "The 2007 Second Annual ACCA President's Tournament & 2007 World Computer Rapid Chess Championships". So far, 40 engines have entered and I fear them all! Although this will be a good exposure for Vicki.

I am having difficulties with the transposition table and it seems that Vicki may be the only engine participating without a transposition table at all. At least the improved move ordering is allowing fairly deep tree searches. Also the evaluation function has been improved somewhat and bug fixes have been made. I'll add this version of Vicki to the website soon.

I know Vicki will be obliterated in this tournament - but if I can only get one draw, I'll be very happy :-)

Hold thumbs!

Hello? Someone there?

Not a single post in June does not mean that I have not been busy! I have the new version of Vicki almost ready. Only a few more tweaks before release. My focus has been on the static evaluation code which was rather poor up till now. I separated the evaluation function into 3 sets, depending on opening, middle or endgame.

The transposition table is working... but I did expect a bit more... at least it is working. A few nasty bugs are still lurking in the code during draws which I still need to resolve.

I am also implementing better time control and rewriting the winboard section so that Vicki claims draws and end the game after losing/winning.

I'm also running tournaments to measure the development of the engine. This can be viewed at http://www.vicki.co.za/test.html and is updated real-time.

If only I had more time!

Transposition tables and tidying up.

I realised this morning that it has been quite a while since my last posting. I am currently working on a number of things in Vicki and my guess is that release 0.04a would be significantly stronger than the previous versions. I am somewhat reluctant to drop the alpha qualifier, but perhaps I will in the next release (version 0.05beta).

I currently have a structure called CHESSENGINE that I pass around (or the pointer to it) between functions. I realised that this parameter-passing may actually degrade performance, cause problems because of its size (I’m not completely sure, but isn’t there a limit of 64k on a "struct"s – even on modern operating systems?) and waste quite a few clock cycles to find the data element within this structure. Regardless, all I gain from this is the ability to have multiple engines in the same process as its state is completely described by this one chunk of data. I’m moving away from this and dumping everything as global variables and structures in a header file. Who wants multiple engines (of the same kind and strength) in the same process anyway?

I am also implementing an elementary transposition table and cleaning up the code as I go. At the moment I am having problems getting it to work, but I’m sure it is just a case of a silly bug lurking somewhere. Once this are done, Vicki will not be such a pushover anymore! ;-)

I’ll keep you all posted...

Testing 1, 2, 3,..., 99282111917764

I am currently experimenting with killer-moves, splitting of the evaluation function (opening, middle and end-game), aspiration windows and all kinds of small tricks. For now, I'm leaving transposition tables for later. The idea is to optimise this part of the engine so that the impact of transposition tables will be maximum.

I'm running tournaments on my computer and I wrote a little Java application to upload the file to my FTP account every now and again. The current tourney is between 9 engines, 15 minutes a side and 2 rounds. For up-to-date, semi-real time updates, visit the test page.