Detecting chess problems on Twitter – fun with computer vision

A couple of weekends ago, I was wondering if I could find a way to use my chess engine, Vice (, to solve problems found on Twitter.

I’ve been involved in the computer chess scene for many years, and this seemed like a really cool way to extend the program!

It turned out to be quite a task, and a nice way to get into some computer vision and neural networks.

The first part was fairly easy, find tweets with text such as:

  • White to play
  • Black to move

That also contain an image. Something like this:

Identifying the board.

The next step was to try to identify a board inside an image. This is not as easy as it sounds!

The simplest way I could find was to filter the original image and make it black and white. I tried different threshold values for white to get different images. I then tried to match two templates and identify 49 corners.

using this method, the AI can identify the board:

And we can extract the squares:

Creating the neural network training data

Now we move on to the meat of the project.

We are able to extract the board pieces, like so:

.. but we need to be able to detect what piece is on each square.

Considering the high variation in piece types and square colours, this is something that is ideal for a neural network.

To train the network, we need data. Which means getting hold of a lot of pieces via web-scraping and then creating all combinations we can think of for colours and pieces:

In total, 750000 images were created!

Building and training the model

There is a really good post that explains how to approach image recognition with neural networks here:

I highly recommend reading that.

Using the above post as a basis, we can use the pieces we created to train, validate and test the network.

Here are the results training the model on just 5000 images:

You can see that we reach over 99% accuracy on the validation data (1000 images)

Running this model on a completely new set of 1000 images yielded 99.9% accuracy:

Now all that remained was to train a model using the whole image set (750k images).

As expected, after 5000 images gave 99%, the 750k image trained model gave 100% accuracy.

Implementing the bot!

All that remained now was to write a script that put everything together.

The result was a bot that can identify chess problems on Twitter and reply..!

Below you can find a video of it being built.


Leave a Comment

Your email address will not be published. Required fields are marked *