This is a PyQt5 and Tensorflow project to take screenshots of chess boards and convert to FEN notation.
This program is designed to work with (nearly) any chessboard by training a neural net, at runtime, to learn the shapes and graphics of the pieces and board.
It exacts the FEN string from an image of a chessboard, but it is not an engine and does not do any game analysis.
Download, install dependencies, and then run the following command:
python main.py
Then..
- open a chess board somewhere on your screen
- ensure all pieces are in their initial starting position
- click the "Capture" button
- use the mouse to draw a rectangle around the target board
- click "Train"
- wait 2-3 minutes while the neural net learns the pieces
- once it says.. "DONE"
- start playing chess normally
You should see the FEN string update as you move pieces around.
This is only verified to work for Macs (see notes below)
Installing with pip from requirements.txt
pip install -r requirements.txt
The command above should just work.
However, there seem to be some problems and incompatibilities with the packaging of python, tensorflow, tensorflow-macos and numpy for OSX. You may have better luck installing the dependencies manually. Below is a configuration that works for Python 3.10. These instructions assume ARM64 architecture, which is the case for M-series Macs.
Installing dependencies with uv:
echo "cpython-3.10.16-macos-aarch64-none" > .python-version
uv add albumentations chess mss pillow pyqt5 scikit-learn
uv pip install tensorflow tensorflow-macos 'numpy<2'
Bug reports are welcome. However, this project is not under active development, so reported bugs may not get fixed.
Before submitting an issue, please see the troubleshooting info below.
This program has only been tested on Macs. It SHOULD be possible to run on Windows, Linux or other operating systems. But that hasn't been tested.
This program can be trained to work with most chess boards. But there may be cases where it simply doesn't work due to the pieces or graphical style of the board.
Below are a few things to try, in order to give it the best chance of working:
- re-do the "capture" rectangle and make sure you are getting just the board itself, with no extra edges or borders
- be sure to "train" from the initial starting position, before either player has moved
- be sure that white is on the bottom, and black is on the top
- be sure the program has permission to capture the screen
- be sure the board is not obscured by another window
- turn off any rank/file labels on the squares, if possible
- turn off any extra icons on the squares, if possible
- choose a simpler set of piece icons, if possible
- choose a plain board without textures, if possible
You can "pause" the training early, if you see that accuracy has reached 99%. If you are still seeing a few random pieces being misclassified, try re-training and wait for it to hit 100%
Running this project will create a temporary file named models/model.keras in the models folder. This file stores the learned model weights and can be safely deleted (but it will be necessary to re-train the neural net again)
The program may take 1-2 minutes the first time that it starts. This is due to Tensorflow initializing and fetching dependencies. Startup should be faster after the first time.
While the UI does change automatically as the pieces move, it doesn't "follow along" with the game by maintaining a history of the moves. The image recognition doesn't understand castling, en passant, or the 50-move draw rule. Since these are important parts of FEN notation, there are buttons to control these flags manually.
Basically, yes, as long as the board is visible on the screen, it can be captured.
Not suiting your needs?
There are LOTS of similar projects on Github! In fact, that there is a whole #chessboard-detection topic for it.
Doing a web search for things like "chess piece tensorflow" or "chess detection pytorch" will also turn up a lot of results.