Chapter 5: The AI eats the sun and Earth’s biosphere goes with it. Dyson swarm timelines, suffering factories on a cosmic scale, digital minds, and the leverage of getting ASI values right.
10^6 s is 1.65 weeks (= 10^6/(24*60^2*7)). Did you use an LLM? I was using a top model a few months ago, and it got a simple conversion wrong (while getting much more complex stuff right).
---
"If the ASI decides it needs molecular-level simulation (to capture protein folding, ion channel dynamics, or other sub-neural processes), the cost increases by roughly 10⁹ FLOP per organism-second over neuron resolution. One simulation run: ~10⁵⁷ FLOP. At 10⁴² FLOP/s, this takes ~10¹⁵ s ≈ 30 million years per run. Molecular-level simulation is infeasible for a single star’s energy only at low redundancy."
I think this underestimates the compute needed for a molecular-level simulation. Gemini says a random neuron of a nematode is 3*10^-11 kg (https://gemini.google.com/share/cb8626da159c). I guess the molar mass of nematodes' neurons is similar to that of water, 0.0180 kg/mol (https://en.wikipedia.org/wiki/Water). So I estimate each neuron has 1.67*10^-9 mol (= 3*10^-11/0.0180). 1 mol has 6.02*10^22 molecules (https://en.wikipedia.org/wiki/Amount_of_substance). So I think each neuron of a nematode has around 1.01*10^14 molecules (= 1.67*10^-9*6.02*10^22), 101 k times as many as the 10^9 suggested above. It is also worth keeping in mind that the neurons of humans probably have many more molecules than those of nematodes, which could mean that the number of molecules in all neurons on Earth is significantly larger than the number of molecules in all nematodes on Earth, even though I estimate soil nematodes have 169 times as many neurons in total as humans (https://docs.google.com/spreadsheets/d/1S7ivkkC8jKRcWU3qj7PPTiEi4YiZDVbnnW1zCljscgE/edit?gid=1150512198#gid=1150512198&range=G1).
Neurons: Thanks! It'll take me a moment to dig into that but I wanted to reply to the other points right away.
Number of organisms: Yeah, that's also the number that I use in the second article. Will fix!
3 weeks: Claude Opus 4.7. I also didn't notice it, so I could imagine it's a matter of attention. The attention is on the central numbers, the ones the rest of the calculation rests on, and little conversions like that are not load-bearing, so I don't pay attention to them. It's also possible that the 10^6 has changed a few times. I had lots of notes on the math and we changed it around many times. How did *you* notice that? :-o
Interesting about the molecular-level simulation! I'll probably reference your comment there. It’s already filed under infeasible in my mind, and your estimate makes it even more infeasible. Anni Leskelä has an unpublished paper draft on that. She mentioned it in 2018. Not sure if I'll yet get to read it. xD
Functionalism/elimitivism: Maybe I'm biased because I share the same initials as Daniel Dennett and vibe with titles like “Consciousness Explained,” but I started reading Brian Tomasik around 2014 and have been heavily influenced by his thinking. So I even think that an onlooker could make a plausible case that any perturbation of air molecules in front of them that could, by some stretch of their imagination, be considered to implement a reinforcement learning process for however brief of a moment, should be considered to evince phenomenal consciousness. I don't know what to make of that in practice. Also I have to reread his posts. It's been a decade. But it's an understatement to say that I bite the bullet on that. I chew it until I transmute it to gold leaf. xD
Hi Dawn. Interesting post.
---
"(Cross-check via steady state: ~10²⁵ neurons in Earth’s biosphere at any moment × 1.6 × 10¹⁶ s of evolutionary history = ~10⁴¹. Same answer.)"
I suspect 10^24 neurons is more accurate. I got 5.89*10^22 neurons for soil arthropods, and 1.17*10^23 for soil nematodes (https://forum.effectivealtruism.org/posts/ehrmin93mzseQMj82/total-number-of-neurons-and-welfare-of-animal-populations) (https://docs.google.com/spreadsheets/d/1S7ivkkC8jKRcWU3qj7PPTiEi4YiZDVbnnW1zCljscgE/edit?gid=1150512198#gid=1150512198&range=F1). I also estimated there are 9.65*10^18 soil arthropods, and 4.89*10^20 soil nematodes (https://docs.google.com/spreadsheets/d/1S7ivkkC8jKRcWU3qj7PPTiEi4YiZDVbnnW1zCljscgE/edit?gid=1150512198#gid=1150512198&range=B1). Assuming 10^20 aquatic arthropods, and 10^21 (terrestrial or aquatic) nematodes, as in Table S1 of Bar-On et al. (2018) (https://www.pnas.org/doi/10.1073/pnas.1711842115), there would be 11.4 (= 1 + 10^20/(9.65*10^18)) times as many arthropods as soil arthropods, and 2.04 (= 10^21/(4.89*10^20)) times as many nematodes as soil nematodes. So I calculate 6.71*10^23 neurons for arthropods (= 11.4*5.89*10^22), and 2.39*10^23 (= 2.04*1.17*10^23) for nematodes. I believe arthropods and nematodes account for the vast majority of neurons (https://forum.effectivealtruism.org/posts/ehrmin93mzseQMj82/total-number-of-neurons-and-welfare-of-animal-populations#Total_number_of_neurons). So I think there are around 9.10*10^23 neurons (= (6.71 + 2.39)*10^23) on Earth, which is 9.10 % of the estimate above of 10^25 neurons.
---
"A million independent replays of 500 million years of evolution, each containing ~10²² potentially sentient organisms"
Table S1 of Bar-On et al. (2018) says 10^20 arthropods, and 10^21 nematodes (https://www.pnas.org/doi/10.1073/pnas.1711842115).
---
"~10⁶ s ≈ 3 weeks"
10^6 s is 1.65 weeks (= 10^6/(24*60^2*7)). Did you use an LLM? I was using a top model a few months ago, and it got a simple conversion wrong (while getting much more complex stuff right).
---
"If the ASI decides it needs molecular-level simulation (to capture protein folding, ion channel dynamics, or other sub-neural processes), the cost increases by roughly 10⁹ FLOP per organism-second over neuron resolution. One simulation run: ~10⁵⁷ FLOP. At 10⁴² FLOP/s, this takes ~10¹⁵ s ≈ 30 million years per run. Molecular-level simulation is infeasible for a single star’s energy only at low redundancy."
I think this underestimates the compute needed for a molecular-level simulation. Gemini says a random neuron of a nematode is 3*10^-11 kg (https://gemini.google.com/share/cb8626da159c). I guess the molar mass of nematodes' neurons is similar to that of water, 0.0180 kg/mol (https://en.wikipedia.org/wiki/Water). So I estimate each neuron has 1.67*10^-9 mol (= 3*10^-11/0.0180). 1 mol has 6.02*10^22 molecules (https://en.wikipedia.org/wiki/Amount_of_substance). So I think each neuron of a nematode has around 1.01*10^14 molecules (= 1.67*10^-9*6.02*10^22), 101 k times as many as the 10^9 suggested above. It is also worth keeping in mind that the neurons of humans probably have many more molecules than those of nematodes, which could mean that the number of molecules in all neurons on Earth is significantly larger than the number of molecules in all nematodes on Earth, even though I estimate soil nematodes have 169 times as many neurons in total as humans (https://docs.google.com/spreadsheets/d/1S7ivkkC8jKRcWU3qj7PPTiEi4YiZDVbnnW1zCljscgE/edit?gid=1150512198#gid=1150512198&range=G1).
---
"Take digital suffering seriously."
I would be curiou to know your thoughts on the pen and paper argument against computational functionalism (https://forum.effectivealtruism.org/posts/vNaGXzEQLFWFHKTE3/the-pen-and-paper-argument-against-computational).
Neurons: Thanks! It'll take me a moment to dig into that but I wanted to reply to the other points right away.
Number of organisms: Yeah, that's also the number that I use in the second article. Will fix!
3 weeks: Claude Opus 4.7. I also didn't notice it, so I could imagine it's a matter of attention. The attention is on the central numbers, the ones the rest of the calculation rests on, and little conversions like that are not load-bearing, so I don't pay attention to them. It's also possible that the 10^6 has changed a few times. I had lots of notes on the math and we changed it around many times. How did *you* notice that? :-o
Interesting about the molecular-level simulation! I'll probably reference your comment there. It’s already filed under infeasible in my mind, and your estimate makes it even more infeasible. Anni Leskelä has an unpublished paper draft on that. She mentioned it in 2018. Not sure if I'll yet get to read it. xD
Functionalism/elimitivism: Maybe I'm biased because I share the same initials as Daniel Dennett and vibe with titles like “Consciousness Explained,” but I started reading Brian Tomasik around 2014 and have been heavily influenced by his thinking. So I even think that an onlooker could make a plausible case that any perturbation of air molecules in front of them that could, by some stretch of their imagination, be considered to implement a reinforcement learning process for however brief of a moment, should be considered to evince phenomenal consciousness. I don't know what to make of that in practice. Also I have to reread his posts. It's been a decade. But it's an understatement to say that I bite the bullet on that. I chew it until I transmute it to gold leaf. xD
I thiiink this is the article where he grapples with many of the questions that are also on my mind: https://reducing-suffering.org/interpret-physical-system-mind/