Dan Faggella on the Race to AGI

The Nugget

Daniel believes that international discourse around AI needs to openly address the big existential questions and divergent visions for the future of AI and humanity. Hiding these questions will lead to inevitable conflicts between those who want an eternal "hominid kingdom" vs. those who believe in the ascension of post-human intelligence. We must put our desired trajectories on the table early on.

Make it stick

🔥 We must address the divergent futures for AI early on or face inevitable conflict
🌍 International coordination is needed to steer AI development and avoid an arms race
🤖 Incentives will push AGI developers to "pedal to the metal" unless governance steps in
💰 AI progress will be driven by compute, algorithms, proprietary data - big tech to dominate

Individuals/orgs fall on a spectrum of "preservation" (keep AI limited) to "progression" (advance AI but keep it human-centric) to "ascension" (enable AI to vastly surpass humans).
Another axis is degree of desired control vs. freedom for AI development.
Incentives push those in the AGI race towards freedom and ascension. Coordination is needed to avoid conflict between camps.

UN, OECD, etc. currently focused on near-term AI issues like privacy. Existential AI issues need to be elevated.
Goal should be international consensus on preferable/non-preferable AI futures and some steering/transparency around AGI development.
Alternatives are worse (free-for-all arms race). Governance is hard but necessary.

AGI developers face a "king of the hill" scenario. Rational move is to develop AGI ASAP even if risky, rather than lose the race.
Rhetoric may be controlled, but actions will tend toward rapid capability gain. Governance and changed incentives are the only counterweights.

Key drivers are compute (large amounts needed), algorithms (breakthroughs hard to predict), and proprietary data (e.g. for robotics).
Big tech is positioned to dominate and create substrate monopolies. Legacy companies will become AI "vassals" to big tech.
China is a threat if it focuses AGI development as national priority. Open source may be a counterweight but easily toggled by big players.

Some traction in code generation for engineers and creative/marketing tasks. Less in customer-facing roles due to hallucination risks.
Personal AI assistants for consumers not far off. AI directly aiding/accelerating AI research is further out.
AI self-improvement and acceleration is plausible and should not be dismissed. Steering still requires much human intuition and context for now.

"The determination of the trajectory of intelligence is the big game. At some point people are going to understand this is actually a bigger deal."
"If you're in the AGI race and you're not winning, you will lean towards freedom and ascension because you want as much flexibility to bust your way in as you can."
"I suspect command deering [of AGI companies/projects] will be in short order when the 'political singularity' occurs - when most people realize the only questions that matter are who builds AGI and what they do with it."
"I believe the continued blooming of the good and the keeping alive of the torch of life would be rather important [in a worthy AGI successor]."

This summary contains AI-generated information and may have important inaccuracies or omissions.