No comments

Create an Opening Repertoire Using Hiarcs Chess Explorer Pro and Pgn-Extract.exe

  By

If you wish to skip the backstory, go directly to the Authoring Procedure and Usage.

Needs

About a year ago, I decided to shift my focus from programming chess to playing chess. Since then, I’ve played many games online and gotten a feel for blitz and rapid time controls.

I’ve realized a valuable technique for improving my play is to review my games. This is especially important for an adult learner, such as myself, in my 50s. Ideally, immediately after finishing a game, I’ll review it and determine where I erred tactically or where I allowed my opponent to erode my position. Understanding what to do and not do (reflexively consult a chess engine) when reviewing one’s games is a complex topic I won’t explore in this blog post. Rather, I intend to focus on one aspect of a subset of games: For games where I arrived at a difficult or outright losing middlegame position, where did I go astray in the opening?

Where did I go astray in the opening?

To answer this question, I’d like to build a repertoire of my preferred openings as White and Black. To maximize the usefulness of this knowledge for a novice player such as myself, the repertoire should be minimalist and focused on positions I actually encounter in my games. Therefore, I don’t view reliance on commercial opening books such as those offered by Hiarcs or ChessBase as a useful solution. Such comprehensive opening books are valuable. But they contain numerous highly theoretical variations only of use to masters- because only two master-level players ever would arrive at such positions. Or possess sufficient skill to navigate razor sharp continuations arising from positions reachable by patzers. Two patzers either never get there, or, by some miracle, having arrived, are incapable of navigating such dangerous waters (“Here be dragons”). Conversely, commercial opening books exclude positions two patzers often stumble into.

Considering these drawbacks, my preference is to create and maintain a custom opening repertoire tailored to my needs.

I intend to consult this repertoire in my post-game review sessions. Both to understand where I misplayed documented positions, and to expand the repertoire as I encounter new and challenging opening positions. So I asked myself, “How do I create such a repertoire? What software do I use? What procedures do I follow to create it and maintain it?”

How do I create an opening repertoire? What software do I use? What procedures do I follow to create it and maintain it?

First, I defined success.

Constraints on Potential Solutions

  1. I would like to both author and use my repertoire in Hiarcs Chess Explorer Pro (HCE Pro), if possible.
    1. As opposed to acquiring specialty software or registering with a new website to manage my repertoire.
    2. I like the simplicity of relying on only two products for chess play and study: HCE Pro on my local PC + chess.com online.
  2. I prefer not to use any ChessBase software.
    1. Even though ChessBase supports creating custom opening books, and HCE Pro supports displaying ChessBase opening books (but not authoring them), I refuse to use ChessBase software.
    2. ChessBase’s instructional article on Creating Your Own Opening Books contains numerous broken screenshot images.
    3. More importantly, their GUI has suffered from persistent, annoying screen flickering issues for more than ten years.
    4. This demonstrates incompetence and / or disregard for customer experience.
  3. If any intermediate steps are required between authoring and using the repertoire, they should be simple and repeatable.
    1. Otherwise, the added complexity will deter me from maintaining it.
    2. I don’t want to build a Rube Goldberg machine.
  4. I would like to easily navigate between recommended moves and explanations of why those moves are recommended.

Next, I explored options.

Exploring Options

The time I spent exploring options is more accurately described as ruling out options. I thought about my preferences, how much of a kludge I’d tolerate, and drew up the above list of constraints.

I quickly focused my efforts on determining how to coerce the Hiarcs Tree Explorer (shown above) into behaving like an opening advisor. The primary difficulty I faced is the Tree Explorer displays moves and statistics aggregated from the main line of games. This aggregation does not include variations.

Tree Explorer displays moves and statistics aggregated from the main line of games, not from variations. I do not consider this a bug.

I do not consider this a bug. In Portable Game Notation (PGN), the main line describes what actually happened in a game. Variations describe what could have happened had the hero played this or the opponent played that. I consider it a perfectly reasonable decision by the Hiarcs development team to provide a Tree Explorer based on what actually happened in the associated games. After all, a common variation added by annotators is to explain why an obvious or intuitive move doesn’t work (often due to tactics). Including such moves would pollute the Tree Explorer’s statistics. It’s impossible to account for how a move that never was played contributes to White’s or Black’s expected chances to win, draw, or lose.

I opened a chess.com lesson on the Italian Opening and snapped it to the left half of my screen. I opened HCE Pro, created a new database, and snapped it to the right half of my screen. I watched the lesson and attempted to add a new game to my HCE database for every interesting variation discussed by the lecturer. I quickly realized this is a tedious, impractical process. I was fighting against HCE Pro, which facilitates quick entry of variations, not games. But, if I was to succeed in coercing the Tree Explorer to behave like an opening advisor, I had to enter one game for every recommend move for every opening position of interest. Or is there a better way?

I decided to use HCE Pro the way it’s intended to be used: I entered numerous variations into a single game. I added comments, colored arrows, and highlighted squares to illustrate the ideas behind recommended moves. Any practical solution must begin with this technique.

Proposal

How to add numerous games to a HCE Pro database (call this the “source database”), each with numerous variations of recommended moves, then automatically transform this into a derivative database (call this the “opening book”) with one game per variation? That would create a database of games with nothing other than a main line. Therefore, all the positions and moves I entered into the source database would appear in a Tree Explorer connected to the derivative database.

How to accomplish this? I propose the following procedure.

Authoring Procedure

Download the PGN Extract command-line program.

All steps performed in HCE Pro unless otherwise noted.

  1. Create a source database via File > New.
    1. Name it “MyRepertoireWhiteSource”.
    2. Save it as a .hce file for optimal performance.
  2. Add a game via Game > New or Ctrl+N.
  3. Save the empty game via Game > Save or Ctrl+S.
    1. I recommend repurposing White, Black, Event and Site columns for Opening, Branch, Source, and Location.
    2. For example: White = Italian Opening, Black = Many Variations, Event = Chess.com Lessons, Site = Learn the Italian Game
    3. For example: White = Caro-Kann Defense, Black = Panov Attack, Event = John Watson Openings Vol 1, Site = Pages 256 – 261
  4. Add moves to main line and variations, saving periodically.
    1. Whether moves are located in the main line or in a variation has no impact on how they’re displayed in Tree Explorer.
    2. Add comments, colored arrows, and square highlights to explain the rationale of recommended moves.
    3. Never enter a weak White move in the White repertoire.
    4. If you feel a need to explain why a White move is weak…
      1. Do so in text comments, not actual variations.
      2. Enter refutation variations in the Black repertoire.
      3. If you encounter this position as Black, you’ll know how to punish White’s weak move.
    5. Same restrictions for Black moves in the Black repertoire.
  5. Add more games and variations based on your preferred authoring structure.
    1. Decide what belongs together in a single game.
    2. Decide what to segregate into separate games.
  6. Export the repertoire source via File > Export.
    1. Format = PGN database
    2. Games = Database
    3. Filename = MyRepertoireWhiteSource.pgn
  7. Transform the repertoire source by running the following command.
    1. .\pgn-extract.exe --splitvariants --output MyRepertoireWhiteOpeningBook.pgn MyRepertoireWhiteSource.pgn
    2. This transforms the variation-dense source database to a one-game-per-variation opening book.
  8. Create an opening database via File > New.
    1. Name it “MyRepertoireWhiteOpeningBook”.
    2. Save it as a .hce file for optimal performance.
  9. Import games via File > Import >MyRepertoireWhiteSource.pgn

Usage

In HCE Pro, open the MyRepertoireWhiteOpeningBook.hce database (assuming you played White). Open a game you’d like to review and navigate to your opponent’s move immediately before you went astray in the opening. Click View > Tree Explorer and select the MyRepertoireWhiteOpeningBook database.

The Tree Explorer displays a list of recommended moves from your repertoire, alongside a count of games for each move. The count actually represents a count of variations.

In the game, I played 4.Nc3.

Let’s assume I’m not happy with my play and would like to explore alternative moves from my repertoire, though I don’t remember ideas behind the moves. Click the 4.c3 move in the Tree Explorer. HCE Pro adds a variation to your game.

Now click the folder icon to open a dialog with a list of games (actually variations) that illustrate possible continuations from 4.c3. Use your mouse or Ctrl-UpArrow and Ctrl-DownArrow to select variations in the list. Notice the chessboard and move list in the dialog display colored arrows, highlighted squares, and comments. Check the box in the leftmost column of the list for interesting variations. Click the Insert Into Current Game button to insert the selected variations into your game.

Notice variation 16 in the above screenshot contains a comment warning a particular White move “isn’t the threat it appears to be.” Notice the warning is expressed in a text comment, not a variation, so the Tree Explorer will not suggest 7.exf6. The comment refers to your Black repertoire if you wish to see the refutation of 7.exf6.

You’ve made it to the end of a long, technical blog post. I hope I’ve clearly described a process for authoring and using a personal opening repertoire. I hope the process I’ve proposed isn’t too cumbersome and benefits your opening preparation. Let me know in the comments.

No comments

MadChess 3.3 Released

  By

I have released version 3.3 of my chess engine.

This release adds static exchange evaluation and integrates it into futility pruning and deferring the search of losing captures. This pushed MadChess’ playing strength above 2800 Elo.

In addition, I adjusted formulas used to calculate internal limit-strength engine parameters to be based on 2800 Elo instead of 2600 Elo. If you adjusted any values in the MadChess.AdvancedConfig.json file, please recalculate per instructions included in comments in the file. The engine’s UCI_Elo option still has a max value of 2600, though, because the non-PVS search (alpha / beta window remains open at the root in order to determine exact scores for more than a single move) used in limit-strength mode significantly increases time-to-depth compared to a PVS search, reducing engine strength.

You may download x64 and x86 versions of MadChess 3.3 from the Downloads page. Install the appropriate version for your computer’s CPU. The x64 binary is the strongest version of the engine.

No comments

Do Tough Things

  By ,

I read an interesting post on a computer chess forum last night. It began:

Dixondeuxyeux’s Post

I thought I would share a few observations that might be of interest to some of you. It was Mark Twain who wrote, “It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” I like this adage because it is a humbling reminder that the game we enjoy so much is a brutally honest and cruel taskmaster. It is believed that after the first nine moves of any game, the possible continuations outnumber all the atoms in the known universe. Chess is a mix of mental logic, mathematical calculation, emotional risk, physical courage and a hint of spiritual enlightenment. When we play chess, we place our egos and our sense of self on the line every game. Have you ever lost a game and wanted to crush the world, lash out at your opponent, question your reason for living? I have. Have you ever seen a fight break out during or after a match? I have. If chess is just a game, why are we so prone to violence? Kasparov said that chess would have a lot more adherents if it were correctly seen as a blood sport.

The entire post, written by a Swede with username dixondeuxyeux, is worth reading here: I Play Computer Chess To Lose.

The Swede’s post got me thinking about chess, games, and life in general. I composed a response. Stayed up two hours later than I had planned to ensure I properly expressed the thoughts swirling in my head. I’ll share my response here, with minor edits for clarity.

My Response

Thank you dixondeuxyeux for sharing your thoughts. Great quote from Twain.

Your post got me thinking late on a Friday night.

Personally, I feel it’s beneficial for your mental health to do things you’re not expert at. (We GenXers+ should say so often to Gen Zers and other victims of social media overexposure.) I am expert in certain areas of my (software development) career, though I’d never describe myself in such terms outside of here to make a point. (Scandinavians will recognize my hesitancy is due to Jantelagen, taught to me by my father.) Should I focus only on those areas in which I’m proficient? No, that would stroke my ego, which is pointless.

I feel it’s beneficial for your mental health to do things you’re not expert at. We GenXers+ should say so often to Gen Zers and other victims of social media overexposure.

Struggle, be uncomfortable, make mistakes, practice, push yourself to improve. It helps you keep an even keel and take things in stride when real shit happens. You know… life.

Michael Jordan, Magnus Carlsen, and Usain Bolt

Did I not play basketball when I was young because I saw Michael Jordan play? Did I think, “What’s the point? Who’s ever going to play at a higher caliber than MJ?” No, I played because it’s a fun, competitive game and great exercise. Even at middling skill levels.

My wife, while we were dating, got me into running. We’ve been happily married nine years now (minus a few days here and there, ha ha).

Am I a fast runner? No. I’m at about the 40th percentile, so I make a lot of other people feel fast. You’re welcome. You just wait- I’ll get relatively faster as I get older but stay in shape. I’m 50 now.

I look fast based on my slim build. But I have pectus excavatum. Also, my wife points out, I don’t train enough. Pfft. What does she know? (She’s a Boston Marathon qualifier and full Ironman finisher… whatever.) 🙂 My pectus excavatum is not severe (1), but bad enough I was self-conscious as a teenager when playing shirts-and-skins basketball games. I’m a pretty good athlete but, well, you feel eyes on you. (Not only women have body image issues.) Did I stop playing basketball… or baseball? No, that would have been stupid. Besides, my sophomore year high school team really needed me as a starting pitcher. I was 4-1, I think, until the varsity coach poached me and then promptly sat me on his varsity bench. What a meathead he was.

Do tough things.

When my wife encouraged me to run with her- not just run for fitness, run in races… 5Ks, 10Ks, 10 milers, eventually a half-marathon, then another half and another half, then… it was inevitable we’d get to the big one. She asked me to run a full marathon with her.

I said no, I don’t think I can do it due to my condition. She told me it’s all in my head and I should get over my hangup. She was right. We’ve run (and completed) a marathon every year since 2016.

So yeah… chess. I need to stop tinkering with my chess engine and play more. Play the game (and follow a study plan), not program the game.

Play the game (and follow a study plan), not program the game.

To that end, I just won a 3m+2s game tonight that lasted 73 moves (rare for two patzers in blitz), surviving serious time pressure. So that’s a start. I would have delivered mate but my opponent realized, in a KP vrs K endgame, that my pawn was going to promote, so he (she?) let his clock run out- only 12 seconds so I didn’t see it as bad sportsmanship. Other than hanging my queen in the late middlegame (32. Qf5+?? I thought I was forcing a trade of queens but overlooked his knight… he missed it too), I’m proud of my effort.

Anyhow, I don’t know what my point is… Perhaps it’s this: Don’t get down because you’re not perfect and can’t play chess precisely like Grandmaster Magnus Carlsen or the Ethereal, Komodo Dragon, or Stockfish chess engines. I seriously doubt Usain Bolt loses any sleep worrying about the F1 race car that’s faster than he is. Or the shitty used car driven by a high school kid in his neighborhood, ha ha.

I seriously doubt Usain Bolt loses any sleep worrying about the F1 race car that’s faster than he is.

Grind of a Blitz Game

Here’s the blitz game I referred to above. As I said earlier, it’s pretty rare for two patzers in blitz to play more than 70 moves and reach a KP vrs K endgame. Especially considering I had only seven seconds on my clock from move 30 onward and basically was living off the increment. After I won, I gave Komodo Dragon four threads and one minute to analyze the game and insert suggested improvements.

(1) I’ve never been medically tested so I don’t know how severe my case is, nor how to officially measure the deformity. My sternum appears to be sunken two inches.

No comments

Missed Blocking Tactic

  By

Lately, I’ve been playing blitz games against MadChess 3.3 (not yet released) to get a feel for how strongly it plays when weakened to my patzer Elo level. Sparring against MadChess helps me calibrate its limit-strength parameters and perhaps glean ideas for classifying more unreasonable inferior moves.

My most recent game begins 1.e4 c5 2.Nf3 d6 3.d4 Nf6 4.Nc3 Nc6 5.Bb5 a5 6.Be3 d5 7.e5 Ng4 8.Qd2 Nxe3 9.Qxe3 cxd4 10.Nxd4 Qd7 11.O-O f5 12.e6 Qd6 13.Nxf5 Qxe6 14.Qxe6 Bxe6 15.Nd4 Bd7 16.Rad1 g6 17.a4 Nb4 18.Rfe1 h5 19.Rd2 b6.

So far I’ve played well enough for a player of my strength in the limited time available in 3m+2s blitz. I’ve developed my pieces, protected my king, and weakened Black’s pawn structure near its king. Material is even. I’ve missed opportunities to advance or attack with my central pawns, but haven’t outright blundered. I spot Black’s weak c7 square and realize my knight on d4 can reach it via e6 because Black’s light squared bishop is pinned to its king by my bishop on b5 and cannot capture on e6. If I can get my knight to c7 it will fork Black’s king and rook on a8.

r3kb1r/3bp3/1p4p1/pB1p3p/Pn1N4/2N5/1PPR1PPP/4R1K1 w kq - 0 20

I play 20.Ne6 and MadChess replies 20…Bxb5. Black trading bishops improves my position as it enables me to place a knight on the b5 outpost, controlling Black territory without fear of harassment from any Black pawns. Black should have developed the dark squared bishop via 20…Bh6 to harass my rook on d2.

r3kb1r/4p3/1p2N1p1/pb1p3p/Pn6/2N5/1PPR1PPP/4R1K1 w kq - 0 21

I should have played 21.Nxb5 to capture Black’s light squared bishop and place a knight on an outpost. But I was fixated on the fork on c7. Play continues 21.Nc7+ Kf7 22.Nxa8 Bh6.

N6r/4pk2/1p4pb/pb1p3p/Pn6/2N5/1PPR1PPP/4R1K1 w - - 1 23

Damn. By moving its dark squared bishop to a square where it attacks my rook on d2, MadChess also reveals an attack on my adventurous knight on a8. Now I’m under double attack. What to do? I can’t capture either of Black’s attacking pieces, so I should retreat the more valuable of my attacked pieces.

23.Rde2?? Doubling rooks on the e file, but blundering. I move my rook out of the crosshairs of Black’s dark squared bishop but into the crosshairs of the light squared bishop. Doh! I need to improve my “whole board” vision. I was focused on the lower half and upper right portion of the board and failed to see a ranged attack originating from the upper left portion of the board.

N6r/4pk2/1p4pb/pb1p3p/Pn6/2N5/1PP1RPPP/4R1K1 b - - 2 23

While reviewing the game after its conclusion, but before peeking at any engine suggestions, I thought I should have simply retreated my threatened rook via 23.Rdd1 Nxc2 24.axb5 Nxe1 25.Nxb6, which creates a passed pawn for White. However, I have a defensive resource in this position. Can you find it?

I missed this blocking tactic, thinking it was nothing more than a delaying move that loses a pawn. However, we see (in the puzzle above and annotated game below) that Black cannot capture the pawn on f4.

My blunder cost me an exchange and a knight, a net loss of five pawns using traditional material values. I played on, as patzer chess players should, attempting to force MadChess to blunder a piece back to me.

23…Bxe2 24.Rxe2 Rxa8 25.f3 Kf8 26.h3 Bf4 27.Kf2 Ke8 28.g3 Bc1 29.b3 Rd8 30.Re6 Rc8 31.Ne2 Rxc2 32.Rxb6 g5 0-1

But I ran out of time. Overall, a competitive game where MadChess played convincingly like a 635 Elo blitz player. (I’m attempting to calibrate “the feel of” MadChess’ UCI_Elo parameter to chess.com’s blitz Elo ratings. Other chess sites may differ by 200 Elo points or so. See Chess.com Versus Lichess Ratings.) The world-class Komodo Dragon chess engine believes MadChess played slightly worse than I. But as Grandmaster Savielly Tartakower said many years ago, “The winner of the game is the player who makes the next-to-last mistake.”

“The winner of the game is the player who makes the next-to-last mistake.” -Savielly Tartakower

I capitalized on MadChess’ 11… f5?? blunder with the correct 12.e6! and held the advantage (unevenly due to numerous inaccuracies) until I overlooked my 23.f4! defensive resource and this “last mistake” proved decisive.

I have annotated the game with a few variations and comments, which you may review below. In addition, I gave Komodo Dragon, a world-class chess engine, one minute and 60 CPU threads (I have a powerful home PC) to analyze the game and insert its evaluations and suggested variations.

No comments

MadChess 3.2.2 Released

  By

I have released version 3.2.2 of my chess engine.

This release fixes a bug that caused truncated principal variations when MultiPV > 1. Also, it improves the efficiency of MultiPV search, so it gets deeper faster.

This release does not add any features nor playing strength. Testers may substitute this version for version 3.2 without any need to restart or re-run tournaments.

You may download x64 and x86 versions of MadChess 3.2.2 from the Downloads page. Install the appropriate version for your computer’s CPU. The x64 binary is the strongest version of the engine.