Finally multithreading!

[Akababa]

@benediamond @Zeta36
After many hours of hopeless debugging I discovered locks which are amazing. The overall speedup on my machine is quite a lot, I would say at least 2x.
That being said, I haven't tested it fully and the code is almost completely rewritten/refactored by now, so please feel free to use it and tell me if I missed anything :)
https://github.com/Akababa/chess-alpha-zero/blob/opts/src/chess_zero/agent/player_chess.py

TODO:

• Testing
• Ctrl+C stop it
• C++ implementation, looks lock-bound

[benediamond]

Hi @Akababa, this looks great, I'm going over it now.

In the mean time, I noticed your flip_policy step. Could you say more about this?

@Zeta36 I'm wondering if this is the crucial bug that appeared since the DeepMind-style board representation. When you flip the board to orient the features to the perspective of the current player... Then the final NN map onto the policy vector must be flipped back as well!? Furthermore it would be necessary to "preemptively" flip visit count information before feeding it into the neural network, e.g. during the convert_to_training_data step.

[benediamond]

Also, why did you remove the manual annealing of the learning rate?

[Akababa]

Hey @benediamond , thanks for commenting!
Btw I'm pushing more optimizations to https://github.com/Akababa/chess-alpha-zero/blob/opts/src/chess_zero/agent/player_chess.py now. It looks like it's working well.

I didn't train for 100,000 steps anyway (the first time lr changes) so it doesn't really matter, I was just experimenting with different optimizers. On the scale of our testing Google's annealing isn't really applicable.

[Akababa]

yes, I think I did the flipping stuff correctly here, but would really appreciate if you could take a quick look to see if it checks out from your point of view.

If you have code you'd like a sanity check on too I'd be happy to help out :)

[Zeta36]

@benediamond:

When you flip the board to orient the features to the perspective of the current player... Then the final NN map onto the policy vector must be flipped back as well!?

Well, this is really a problem we didn't take into account. Certainty it may be the cause of the converge failing.

[benediamond]

Yes. I can't believe we didn't think of this. @Akababa, kudos! I'll be looking through your code making sense of everything. I'll let you know how things work.

[Akababa]

Yeah that's always a worry in the back of my mind (hence the paranoid asserts). I'm a little confused by the conversation though, is there already a bug found in my code? Or is it a previous one before my implementation?

@benediamond Thank you! Please feel free to write some test cases and sanity checks.

[benediamond]

@Akababa The point is that I had developed a "DeepMind-style" feature plane input on my own, but I hadn't realized (as you did) that the policy vector needed to be flipped for black. @Zeta36 and I were wondering why it didn't converge. I'll be updating it accordingly as soon as possible.

[Akababa]

Doesn't the DeepMind input actually use an extra plane to encode the side to move?

The main reason I did this was for fun, and also it might make the network train faster, as I believe it's strictly better than having the color plane and using this transformation to augment the training data.

[benediamond]

Yes, they did. You can see my current approach here.

Here is another quick question. It appears that you clear the move tables at the beginning of each call to action. Yet isn't this contrary to the DeepMind approach, where, as they say, after each move the non-chosen portion of the tree is discarded but the chosen one is kept? Here, we will have to build visit counts from scratch each time a new move is chosen. Previously, memory was released only at the end of the game (when self.white and self.black are reassigned in self_play.py).

Also, what do you mean by "losing MCTS nodes...!"?

[Akababa]

But why is there a need to flip the policy if you are feeding in the side to move?
Yes that was before I read that part of the paper, but even then I'm not sure how move counts from previous transpositions would affect the table. In any case I'm mostly doing this as a "functional" approach to make results and bugs reproducible and make things easier to think about for now.

[benediamond]

Hmm, I see what you're saying. But that'd be much harder for the network, no? The entire mapping from the convolved input stack to the policy vector will have to be re-learned from scratch for black, in a new way that is totally an arbitrary scrambling of the first. At that point, there is no reason to place the side-to-move on top of the stack, orient the board from the player's perspective, etc... Right?

[Akababa]

That might be true especially at the beginning, before the model has the chance to learn the rules of chess.

However I think we are doing something similar with the flattened policy output layer anyway. Google's paper does mention that final result (between flattened policy and huge stack of one-hot layers) was the same but training was slightly slower with a "compressed" format, which for us with our 0 TPUs probably means we won't see significant results from scratch for a while.

One thing I considered doing is having two 64-unit FC outputs for to and from square (and maybe ignore underpromotions for now); it might be a little easier for the network to use. BUT I don't know if this would output a sensible probability distribution with regard to softmax and ranking chess moves.

[benediamond]

By the way, do you know what the "alternative" to the flat approach is? I can't figure out what the "non-flat" approach they're referring to is.