Some eXtremely Interesting Positions From Pittsburgh 2010

Very nice article as usual!

Regarding position 6, my guess is that the problem with making the 5-point is just that you don't want to leave another blot lying around when your opponent is dying to attack. With a stronger board and more checkers back, he is eager to engage in a blot-hitting contest. If you change the position by moving a spare from your 6-point back to your 9-point, for example, then making the 5-point becomes a much more viable option. On the other hand if you instead move your two checkers on the 9-point back to the 11-point and change the roll to 6-1, then making the 5-point is still a whopper, so it's not just the blocking value of the 9-point that's at issue here.

I have a request, by the way. When you roll out positions, it pains me to see fewer than 100 games and an uncertainty of something like plus or minus 0.067. Since eXtreme Gammon is so fast, I would strongly encourage you to roll every position out at least 1296 times, or at the very least 432 times. It's often valuable to know not only that the bot thinks one play is better than another, but also how much better it is. For that, one needs a decent estimate of the absolute equity, which usually requires a longer rollout.

There are two caveats to keep in mind regarding "confidence is confidence." The first is that confidence applies only to the assertion that the top play is better than the next best play, and not how much better it is. In practice it is very difficult to resist the temptation to make pronouncements like "the second-best play is a whopper" or "the second-best play is wrong by .08." But such pronouncements make use of the absolute equity estimates. Just because you're 99.5% confident that Play A is better than Play B does not mean that you can be 99.5% confident that "Play B is a whopper" or that "Play B is wrong by .08" or even that "Plays B and C are wrong but B is better than C." You generally need a longer rollout before you can make such statements (that go beyond the bare fact that Play A is better than Play B) with confidence.

The second caveat is that the confidence intervals that the bots give you are based on the assumptions that the equities can be approximated with a Gaussian probability distribution, and that the random samples are independent. In most cases these are probably reasonable assumptions if you do enough trials. As the number of trials increases, the approximation tends to get better and better. But with less than 100 trials, it is not clear how good the approximation is. Indeed, the experts on opening rollouts, who deal with rollouts with tens of thousands or even hundreds of thousands of trials, have found that the bots tend to be overconfident. Nack Ballard tells me that he doesn't put much stock in an opening rollout between close plays unless he has at least 15552 trials! And I agree with him. A rollout with allegedly 99.5% confidence will probably, in practice, be overturned more than once in 200 times. This is another reason to roll things out a little more than the numbers might suggest is necessary.