The first time I used Chat GPT was on January 1st, 2023, shortly after midnight. That was the first and only New Year’s Eve I celebrated alone – I chose to self-quarantine after testing positive for COVID on a home test. I was impressed with Chat GPT’s capabilities, so much so that it quickly and permanently became part of my daily work routine.
Roughly a week later, I challenged it to a game of chess. To my surprise, it simply started the game with “e4” and offered no board in which to visualize the game. I decided to take this opportunity to play the game without a board myself for the first time. I was surprised at my own ability to do so, as I’d never attempted it before and had only three months’ worth of chess experience. After about ten moves, I thought I was losing my grasp of the game, as Chat GPT was beginning to make moves that didn’t seem legal to me. I stopped playing, pulled out a board, and saw that Chat GPT actually was at fault. I played a few more games and found that Chat GPT invariably made illegal moves as the game progresses. Nonetheless, this piqued my curiosity of blindfold chess and led me down the path to eventually creating a blindfold chess app myself, available on playblindfoldchess.com.
Throughout the two years since, I’ve witnessed Chat GPT’s growth firsthand. Compared to its state in 2023, it can now generate cleaner images, reason about more complex topics, and provide more accurate and precise answers to inquiries. Today, I decided to try another chess game.
Just as before, it offered no board, and just as before, I didn’t ask for one once the game started. This playthrough was much better. The first eight moves proceeded flawlessly. However, on the ninth move, I accidentally typed “Ng5” instead of “Bg5”, so ChatGPT thought I was attempting to move my knight to the g5 square instead of my bishop. It readily obliged and made its next move. However, the move “Ng5” was invalid because I had no knights on the board capable of reaching that square in one move. ChatGPT didn’t flag this inaccuracy and made its next move. Had I not caught this, the game would’ve proceeded in an invalid state from then on. It was a typo on my part, but it was also a failure on Chat GPT’s part.
I asked ChatGPT to apply “Bg5” and proceed accordingly, and it was able to make the shift. One move later, I made a second mistake. I wanted to move my knight from the d4 square to the e2 square, so I typed “Ne2,” but I also had a knight on c3, which means both knights could access the e2 square. I should’ve disambiguated that, but ChatGPT should also have challenged me. Fortunately, it understood what piece I wanted to move from context and moved it accordingly.
From there, it proceeded to make three mistakes. None of these were in the form of illegal moves, they were in the form of assessing the board incorrectly, forgetting that certain pieces were captured or moved. In its third mistake, it moved a queen and described the pieces the queen was threatening, but none of those pieces were as described.
At that point, fourteen moves in, I decided to stop the game. It seemed that it would only be a matter of time before ChatGPT made an illegal move, and I didn’t want to deal with that this far in. I asked it to reproduce the list of moves so I could analyze it on chess.com. In 2023, it failed this request, but today, it fulfilled it accurately, the list of moves was exactly as I remembered them.
I was surprised to see that the first eight moves were all part of well-known, commonly studied chess openings, often called ‘book moves.’ I’d never studied this sequence of moves, but I believe that ChatGPT may have it stored in some manner in its large language model, so it naturally replied to the moves I was making accordingly, helping to lead me down the path to coincidentally playing these book moves myself. It wasn’t until its castling at the end of the eighth move that it deviated. I suspect this partly explains why ChatGPT was able to play correctly for so long before encountering problems. However, even when it did begin making mistakes, it held on for much longer than before, and it still continued making only legal moves even if its assessment of these moves was incorrect. In a sense, it felt as if I were playing a person who was gradually losing their grasp of the board as the game proceeded.
This may seem to cast ChatGPT in a negative light, but I find its capabilities impressive. It was not explicitly programmed to play chess – it only knows the game from the chess literature present in the vast ocean of literature it was provided as training material. In a sense, it taught itself how to play by reading chess related material. In its own words: “I don’t rely on a dedicated chess engine by default, but I’ve been educated through exposure to a vast array of chess literature, annotated games, and strategic insights, which equips me to play, analyze, and explain chess effectively.”
I’m curious to see how long it takes ChatGPT to improve to the point where it can play full games without mistake. I look forward to challenging it to a rematch. For now, I’ll share the game we just played below, where I played as white:
