Some of my friends have strong views on how long it will be until AGI is created. The best arguments on the subject establish that creating a superintelligent AGI is possible, and that such an AGI would by default be “unfriendly”, which is a lot worse than it sounds. So far as speculative engineering goes, this is on relatively solid ground. It’s quite possible that as research continues, we’ll learn more about what sorts of intelligence are possible and discover some reason that an AGI can’t actually be built—such discoveries have happened before in the history of science and technology—but at this point, a discovery like that would be a surprise.
I think it's worth observing that at the time of Nick Bostrom's 2014 book Superintelligence the success of NN algorithms had not yet broken out in the form of Google's DeepDream. Eliezer Yudkowsky claims this as a success i.e. "progress in GAI has occurred even faster than I have anticipated" but I think it goes to your point that making bets on technology is not reliable. Even after we narrowed it down to NN (or appear to have!), the impact of just scaling up was not appreciated until OpenAI's ChatGPT.
So what it will take to make AI general, what it will take to overcome data limitations, what it will take to make AI agentic, and what it will take to control it I think are all bets that I don't think we can make reliably.
I think it's worth observing that at the time of Nick Bostrom's 2014 book Superintelligence the success of NN algorithms had not yet broken out in the form of Google's DeepDream. Eliezer Yudkowsky claims this as a success i.e. "progress in GAI has occurred even faster than I have anticipated" but I think it goes to your point that making bets on technology is not reliable. Even after we narrowed it down to NN (or appear to have!), the impact of just scaling up was not appreciated until OpenAI's ChatGPT.
So what it will take to make AI general, what it will take to overcome data limitations, what it will take to make AI agentic, and what it will take to control it I think are all bets that I don't think we can make reliably.