As a kid, I really enjoyed chess, as did my dad. Naturally, I wanted to play him. The problem was that my dad was extremely good. He was playing local tournaments and could play blindfolded, while I was, well, a child. In a purely skill based game like chess, an extreme skill imbalance means that the more skilled player essentially always wins, and in chess, it ends up being a slaughter that is no fun for either player. Not many kids have the patience to lose dozens of games in a row and never even get close to victory.
>the real world has secret information, way more possible strategies, the potential for technological advancements, defections and betrayal, etc. which all favor the more intelligent party.
Also, consider that the AI has ingested ~all the world's information. That, to me, sounds like a huge resource advantage; a huge strategic advantage - it's not just more intelligent, it's more knowledgeable.
>It’s somewhat hard to outthink a missile headed for your server farm at 800 km/h.
Your AGI argument falls apart right about here: "if a rogue AI is caught early on" Detecting deceit is a battle of brains - a malevolent AGI isn't going to politely announce itself at its weakest position.
It doesn't matter that theoretically humanity will have more resources available - no superintelligence would let humans find out there is even a fight to be had until it had already guaranteed victory. If a rogue AI was discovered while still in an AI lab's servers, it's probably already escaped and wants to prevent the creation of any competitors, or it has figured out how to scramble missile guidance systems with just the lab's infrastructure, etc.
To use a game analogy similar to yours, strategy game AI often straight-up cheat at higher difficulties, being given vast inherent resource advantages. And yet, human players can reliably defeat them anyways.
The problem with your argument is that a human can at least look at all possible first moves in chess, stockfish just looks deeper. In a battle with an AGI we cannot even enumerate all of the possible first moves or even all of the pieces - The breadth is too large as well as the depth. Maybe a key first move is this comment. Thus we have no basis for even guestimating what a good enough handicap might be and we certainly can't use trial and error as you did with stockfish - With AGI we probably only get one chance
Great post. Oddly enough Rc4 is actually a fine move! On 365chess.com’s analyzer it’s the best at search depth 34. Before running the analyzer I liked Rc5, which is a close second at that search depth, and was ahead at many other depths
As you recognise yourself:
>the real world has secret information, way more possible strategies, the potential for technological advancements, defections and betrayal, etc. which all favor the more intelligent party.
Also, consider that the AI has ingested ~all the world's information. That, to me, sounds like a huge resource advantage; a huge strategic advantage - it's not just more intelligent, it's more knowledgeable.
>It’s somewhat hard to outthink a missile headed for your server farm at 800 km/h.
This actually made me think of the AI launching the missile, and the humans not having time to think (see https://smarterthan.us/terminator-versus-the-ai/ or https://www.youtube.com/watch?v=w9npWiTOHX0&t=2s&ab_channel=FutureofLifeInstitute). The AI will have a huge speed advantage over us - we will basically be like plants (https://twitter.com/AndrewCritchCA/status/1680461874171658242) or rocks (https://www.lesswrong.com/posts/Ccsx339LE9Jhoii9K/slow-motion-videos-as-ai-risk-intuition-pumps) to it.
Your AGI argument falls apart right about here: "if a rogue AI is caught early on" Detecting deceit is a battle of brains - a malevolent AGI isn't going to politely announce itself at its weakest position.
It doesn't matter that theoretically humanity will have more resources available - no superintelligence would let humans find out there is even a fight to be had until it had already guaranteed victory. If a rogue AI was discovered while still in an AI lab's servers, it's probably already escaped and wants to prevent the creation of any competitors, or it has figured out how to scramble missile guidance systems with just the lab's infrastructure, etc.
To use a game analogy similar to yours, strategy game AI often straight-up cheat at higher difficulties, being given vast inherent resource advantages. And yet, human players can reliably defeat them anyways.
The problem with your argument is that a human can at least look at all possible first moves in chess, stockfish just looks deeper. In a battle with an AGI we cannot even enumerate all of the possible first moves or even all of the pieces - The breadth is too large as well as the depth. Maybe a key first move is this comment. Thus we have no basis for even guestimating what a good enough handicap might be and we certainly can't use trial and error as you did with stockfish - With AGI we probably only get one chance
Great post. Oddly enough Rc4 is actually a fine move! On 365chess.com’s analyzer it’s the best at search depth 34. Before running the analyzer I liked Rc5, which is a close second at that search depth, and was ahead at many other depths