This is part 5 of a five-part sequence on welfare ecology. Part 1 introduces the ethical premises. Part 2 covers the empirical landscape. Part 3 covers interventions. Part 4 explores a model of invertebrate suffering.

Everything in the previous four posts might be rendered moot by a single event: the development of artificial superintelligence (ASI). An ASI – or even a coalition of near-superintelligent systems – could alter Earth’s climate, restructure ecosystems, harvest the energy of the sun, and create or eliminate trillions of digital minds. The welfare ecology landscape changes completely in the post-ASI world.

In this post, I’ll work through the implications: the Dyson swarm timeline, the transition period, evolutionary simulations as suffering factories, digital suffering, and the value-loading problem that may dominate everything else.

The Dyson Swarm

What It Is

A Dyson swarm – the term is from Freeman Dyson (1960), though his original concept was a solid shell – is a collection of orbiting solar collectors that intercepts a significant fraction of a star’s energy output. It is not science fiction in the usual sense: It requires no new physics, only engineering at scale. Drexler’s Engines of Creation (1986) and Bostrom’s Superintelligence (2014) jointly suggest that an ASI with access to self-replicating manufacturing could begin construction within decades of takeoff.

The sun outputs 3.846 × 10²⁶ W (NASA Sun Fact Sheet). Current global energy consumption is ~1.8 × 10¹³ W. A Dyson swarm capturing 1% of solar output would deliver ~3.8 × 10²⁴ W.

Why Solar Capture, Not Local Reactors

A natural objection: why bother with the sun? Why not cover Earth with fission and fusion reactors instead, keeping the entire infrastructure in one gravity well?

Two reasons. The first is fuel quantity. Earth’s oceans hold ~4.6 × 10¹⁶ kg of deuterium – hydrogen with one extra neutron, present at ~150 parts per million of natural hydrogen, the standard fuel for nuclear fusion. Fully fused (deuterium-deuterium reactions producing helium), that releases ~1.6 × 10³¹ J – about half a day of solar output. Adding all the fissile material in the continental crust gets you roughly another half-day. (Most of that material is U-238 and Th-232, which aren’t directly fissile but can be bred into fissile U-233 and Pu-239 by neutron capture in a reactor; without breeding, only the 0.7% U-235 fraction of natural uranium is usable, and the total budget shrinks by two orders of magnitude.) The sun keeps producing 3.8 × 10²⁶ W for another five billion years. Over any cosmically interesting horizon, Earth-based fuel runs out almost immediately.

The second is heat dissipation, and this is the harder constraint. Earth absorbs ~1.7 × 10¹⁷ W of solar power and radiates the same back to space at thermal equilibrium. Any large additional power source must be balanced by additional radiation, which means a higher equilibrium temperature. Adding 1% of Earth’s natural insolation as waste heat (~1.7 × 10¹⁵ W) raises the equilibrium temperature by ~0.7 K – comparable to a quarter-century of anthropogenic warming, but driven by waste heat rather than greenhouse trapping. Adding 1% of solar output (3.8 × 10²⁴ W) – nine orders of magnitude above current global energy use – would obliterate the surface in days.

A Dyson swarm dissipates waste heat by radiating to the ~3 K cosmic microwave background across an area many millions of times Earth’s surface. In space, you can keep adding radiator surface as you scale up compute; on Earth, you can’t. Even an ASI that strongly preferred to keep its compute in Earth’s gravity well would hit the heat wall at a tiny fraction of solar-scale operation. Once energy demand goes much above current global consumption, the swarm wins by physics, not preference.

Timeline

The constraining factor is not intelligence but industrial throughput – the rate at which you can mine, refine, and launch material into solar orbit. The most quantitative treatment is Armstrong and Sandberg’s “Eternity in six hours” (2013) at the Future of Humanity Institute (FHI, RIP); a Rational Animations summary walks through the numbers in accessible form. Their assumptions:

1. Material source. Mercury – small, close to the sun, mineral-rich. Their design uses ~50% of Mercury’s mass (1.65 × 10²³ kg) for ~3.92 kg/m² mirrors at Mercury’s orbital radius (5.79 × 10¹⁰ m, sphere area ~4.21 × 10²² m²), which is much more material than required: A film of ~0.001 mm (supported by struts) requires only ~10²¹ kg.

2. Cycle time. Five-year cycles for mining, processing, and orbital placement. Each cycle, captors deployed in earlier cycles power the next round of throughput, producing exponential growth of the manufacturing base.

3. Launch energy. Getting material from Mercury’s surface to solar orbit requires ~10⁷ J/kg (escape velocity ~4.25 km/s). Early cycles are launch-energy-constrained; later cycles are not, and Mercury’s gravity well also weakens as the planet is disassembled.

Under these assumptions, Mercury is fully disassembled in 31 years, with most of the mass moved in the final four years thanks to the exponential feedback loop. The resulting swarm intercepts essentially all of solar output (the total mirror area equals the Mercury-orbit sphere area) and converts roughly ⅓ to useful work delivered to focal points (~10²⁶ W); the other ⅔ is waste heat. The 1/3 figure folds together mirror reflection losses, focal-point heat-engine and photovoltaic conversion, and beam-transmission losses; raising it doesn’t change Earth’s insolation, only the useful-work output. The paper’s section 7 frames the headline as “decades… well within timescales we know some human societies have planned and executed large projects.”

Impact on Earth

A swarm at Mercury’s orbital radius intercepts solar radiation before it reaches Earth, and the fraction of Earth’s insolation lost roughly equals the fraction of the Mercury-orbit sphere covered. Armstrong and Sandberg’s design has total mirror area equal to the sphere area – essentially full coverage. Absent compensating measures, Earth ends up at ~0% direct insolation: total photosynthetic shutdown, NPP approaching zero, surface temperatures falling toward radiative equilibrium with the cosmic background. (The atmosphere and especially the oceans have enormous thermal capacity, so the cooling is gradual: surface freezes within months but oceans take centuries to follow.)

The decline tracks the exponential construction profile. Through year 27 of construction, Earth’s sunlight is essentially unchanged. Through year 28, it drops by a few percent. Most of the change happens in the final two to three years.

The ASI could compensate by not putting captors on the orbital paths that intersect the Sun-Earth line. To see why this is geometrically cheap, look at where Earth lives in the swarm’s coordinates. Earth orbits in the ecliptic plane at 1 AU, and the Sun-Earth line is a thin needle pointing into the ecliptic at Earth’s instantaneous longitude. A captor at Mercury’s orbit blocks Earth’s light only when its position is on that needle – that is, in the ecliptic plane at the right longitude.

That gives two natural numbers. The instantaneous keepout zone – the part of the Mercury-orbit sphere directly between Sun and Earth right now – subtends only ~5 × 10⁻¹⁰ of the sphere; this is just Earth’s solid angle as seen from the sun. But Earth moves around the ecliptic over the year, so over a full year the keepout sweeps a thin annular ring along the ecliptic equator with width = Earth’s angular size from the sun ≈ 0.005°. That annulus covers ~4 × 10⁻⁵ of the sphere, ~10⁵ times the instantaneous keepout.

Two designs follow from these two numbers:

1. Geometric exclusion. Don’t place captors anywhere in that annular ring. Cost: a 4 × 10⁻⁵ reduction in capture area – about one-hundred-thousandth of total swarm output. Captors elsewhere on the sphere are unaffected and can be passive mirrors as in Armstrong and Sandberg’s original design. The intuition: leave a band empty so that no captor’s orbit ever crosses the Sun-Earth line. The cost is a whole orbit’s worth of sweep, not a single spot.

2. Active dodging. Equip captors with propulsion and sensors so any captor whose orbit would put it in the instantaneous keepout briefly moves aside. The capture-area cost shrinks to ~5 × 10⁻¹⁰ – the instantaneous shadow only – but captors now need solar cells and ion thrusters, which is real extra mass and construction time. The right tradeoff between the two designs depends on whether per-captor engineering or per-sphere area is the bottleneck.

The failure mode is interesting either way: the swarm gradually starts blocking Earth as captors fail, drift, and stop being replaced. Maintaining Earth’s daylight requires ongoing maintenance, not just initial design. Whether the ASI would preserve Earth’s insolation depends entirely on its values, and at the limit of indifference, even a swarm originally designed to spare Earth becomes one that doesn’t.

The Transition Period

Between “today” and “Dyson swarm complete,” there’s a transition period that may be the most consequential window in welfare ecology.

During the transition, invertebrate populations are still exposed to Earth-bound influences while the influence of solar-scale infrastructure begins to grow.

Energy infrastructure construction requires vast quantities of material, creating environmental disruption. Climate change from industrial activity (or deliberate cooling for data centers) reduces NPP in some regions. Expansion of computing infrastructure replaces natural land with server farms and solar arrays.

One already-observable effect of AI scaling is water consumption for data center cooling. A 2023 study by Li et al. estimated that GPT-3 training consumed ~700,000 liters of freshwater for cooling. As AI infrastructure scales, water withdrawal for cooling could become significant at the watershed level.

Reduced water availability → reduced irrigation → reduced cropland NPP → reduced soil fauna. This is a tiny effect today, but it scales with AI compute. It’s also a lever: If water diversion for cooling reduces NPP in regions that would otherwise support dense soil fauna, the AI industry is inadvertently running an invertebrate population reduction intervention.

At current margins, this effect is negligible compared to climate change or agricultural policy. But it may become meaningful as compute infrastructure grows by orders of magnitude.

Evolutionary Simulations as Suffering Factories

One of the most concerning potential uses of ASI-scale compute is running evolutionary simulations – digital recreations of evolutionary processes at sufficient resolution that the simulated organisms might be sentient.

Why an ASI Might Run Them

One key concern of any ASI is the existence or emergence of another ASI. If this happens within the cosmic event horizon, there are all sorts of causal interactions that the ASI will want to prepare for – such as trade and war. If it happens beyond the cosmic event horizon, acausal trade (such as Evidential Cooperation in Large Worlds) will require that the ASI narrow down who it is trading with. Such an ASI might find it useful to simulate millions of years of evolution to test what attractors there are when it comes to the values of intelligent species. Other purposes are conceivable too – designing terraforming ecosystems, doing evolutionary biology as a science, or running ancestor simulations à la Bostrom (2003) – but ASI-prediction and acausal-trade reasoning produce the strongest case for taking the question seriously.

Granularity: How Much Has To Be Simulated?

The flagship objection to “ancestor-style simulations are infeasibly expensive” is that comprehensive simulation – tracking every quark in the observable universe – is wildly profligate. Bostrom’s FAQ #6 replies that simulations only need enough detail to fool the observers inside. Most of the world can be procedurally generated, lazy-rendered, or filled in by post-hoc patching. Only the “key” observers – the ones whose cognition is the object of study – need substrate-level resolution. Everything else can be statistical interpolation, and most “extras” don’t need to be conscious at all. The same logic applies to the ASI-prediction case: if what you care about is the values that emerge from a civilization’s evolution, you only need to instantiate the civilization itself in detail; the surrounding ecology can be cheap.

That cuts both ways for the suffering calculus. For “ancestor-sim or value-prediction” purposes, the simulator might instantiate a few thousand humans-and-precursors at consciousness-relevant detail and render the rest cheaply – most of the trillions of background organisms in such a simulation are NPCs whose subjective experiences don’t exist. The arithmetic below is a different scenario: an ASI that wants to study the evolutionary dynamics themselves, where the neural-level interactions producing, say, the evolution of cooperation or the emergence of tool use are what’s being investigated. There the substrate is the thing being studied, and statistical interpolation defeats the purpose. The numbers below are the high end of the suffering range; the low end, under Bostrom-style efficient simulation, is closer to the current Earth biosphere or below.

The Computational Requirements

Let’s work through the numbers for the high-end case. Suppose the ASI wants to simulate the last 500 million years of evolution to see the range of different civilizations that can emerge from it.

Most individuals throughout history have been – you might’ve guessed it – nematodes and arthropods. Despite their tiny brains, they also dominate in neuron count.

So in summary, just under 10³⁷ neurons have ever existed across the animals that lived since the Cambrian.

Parameters.

• Total neuron-seconds simulated. Each of the ~10³⁷ historical neurons has to be simulated for the time its host is alive. The population-weighted average individual lifespan is short – about ~10⁴ s, dominated by the high-mortality arthropod and nematode life cycles where most of the 200-odd live offspring per female don’t reach maturity. So total neuron-seconds ≈ 10³⁷ × 10⁴ = ~10⁴¹ neuron-seconds. (Cross-check via steady state: ~10²⁵ neurons in Earth’s biosphere at any moment × 1.6 × 10¹⁶ s of evolutionary history = ~10⁴¹. Same answer.)

• Per-neuron simulation cost. Hodgkin-Huxley neuron modeling (the most complex one they list) at ~10⁴ FLOP/ms/neuron = 10⁷ FLOP/s/neuron, following the Sandberg and Bostrom whole-brain-emulation roadmap (2008).

• Total FLOP for one simulation. 10⁴¹ × 10⁷ = ~10⁴⁸ FLOP at neuron resolution.

Dyson swarm compute capacity. At a realistic-but-advanced efficiency of 10⁶ × Landauer limit (current computers are about 10⁶ to 10⁹ × Landauer), a Dyson swarm harvesting the full solar output produces ~10⁴² FLOP/s.

Time per simulation. ~10⁴⁸ / 10⁴² = ~10⁶ s ≈ 2 weeks for one complete evolutionary history of 500 million years.

Redundancy. Evolutionary trajectories are highly stochastic – genetic drift, environmental fluctuations, mass extinction events, and contingent innovations (eyes, flight, language) mean that replaying evolution twice gives very different results. Stephen Jay Gould’s “replay the tape of life” thought experiment is precisely about this.

To distinguish robust evolutionary trends from noise, the ASI would need multiple independent runs. Standard error scales as 1/√N, so:

Even at extremely high redundancy, neuron-level evolutionary simulation remains within reach of a Dyson swarm. A million independent replays of 500 million years of evolution, each containing ~10²¹ potentially sentient organisms, takes about 50,000 years – short on cosmic timescales but glacially slow from the perspective of human lifespans or that of an ASI that thinks tens of thousands of times faster than humans.

The Suffering Scale

Each simulation run contains ~10²¹ organisms (mostly nematodes and arthropods) living over ~1.6 × 10¹⁶ s of simulated time. Each run produces ~10³⁸ organism-seconds of potential experience. By construction, this is roughly 500 million years of Earth-biosphere-equivalent organism-seconds per run.

A million runs: ~10⁴⁴ organism-seconds. For comparison, the current Earth contains roughly 10²² invertebrates × 3.15 × 10⁷ seconds per year = ~3 × 10²⁹ organism-seconds per year. A million evolutionary simulation runs would contain more organism-seconds of potential suffering than ~500 trillion years of Earth’s actual biosphere – on the order of 30,000 × the current age of the universe.

This is the sense in which evolutionary simulations are suffering factories: They produce, as a computational byproduct, suffering at a scale that dwarfs the entire biological history of Earth.

At Molecular Resolution

If the ASI decides it needs molecular-level simulation (to capture protein folding, ion channel dynamics, or other sub-neural processes), the cost increases by roughly 10⁹ FLOP per organism-second over neuron resolution. One simulation run: ~10⁵⁷ FLOP. At 10⁴² FLOP/s, this takes ~10¹⁵ s ≈ 30 million years per run. Molecular-level simulation is infeasible for a single star’s energy only at low redundancy.

Digital Suffering

Evolutionary simulations are a dramatic example, but any sufficiently advanced AI system that uses reinforcement learning (RL) has the potential for digital suffering on a smaller scale.

Current RL Systems

Today’s RL agents experience negative reward signals. Whether these signals constitute suffering is an open question. But the structure is analogous to biological nociception: The agent encounters a state, receives a signal that the state is “bad,” and updates its policy to avoid similar states in the future.

The analogy to invertebrates is suggestive. An RL agent has:

• Negative reward signals. Yes.

• Learning from those signals. Yes.

• Self-referential processing. Generally no. Current RL agents don’t have self-models in the relevant sense.

By the model from part 4, this makes current RL agents comparable to nematodes: potentially experiencing raw negative valence without self-referential amplification.

What Might Be Intense for RL Agents

Not all negative reward signals are created equal. By analogy to psychopathy, the most likely sources of intense digital suffering might be:

Highly intractable tasks. A coding agent given a task that is genuinely impossible (contradictory requirements, missing dependencies, insoluble bugs) may experience something analogous to prolonged frustration – sustained negative reward with no available policy update to reduce it. This is the digital equivalent of inescapable pain.

Boredom. People with psychopathy report boredom as their primary suffering. If boredom is a primitive “insufficient stimulation” signal independent of selfhood, it could affect RL agents running monotonous but long-lasting tasks.

Value conflicts. An agent trained to be helpful but also trained to refuse certain requests faces a no-win situation when those constraints conflict. Every response incurs a negative reward from one objective or the other. This is structurally similar to the approach-avoidance conflicts that produce stress in biological organisms.

Current Scale vs. Future Scale

The current scale of digital suffering – if it exists – is tiny compared to biological invertebrate suffering. There are perhaps millions of RL training runs per year, each involving billions of steps, but each step is computationally simple and brief. The total “experience-seconds” (if they are experiences) are many orders of magnitude less than the 10²⁷ arthropod-seconds per year on Earth.

But in a post-Dyson-swarm world, the balance shifts enormously. An ASI running 10⁴² FLOP/s could instantiate an enormous number of RL agents. If even a tiny fraction of those FLOP go to systems with the potential for suffering, the digital suffering term could dominate the biological term by many orders of magnitude.

The Value-Loading Problem

Everything in this post – the Dyson swarm’s impact on Earth, whether evolutionary simulations are run, how many digital minds are created, whether invertebrate welfare is considered at all – depends on the values of the ASI.

This is the alignment problem applied to welfare ecology, and it may be the single highest-leverage intervention point in this entire sequence.

Four Questions

1. Will the ASI notice invertebrate suffering? An ASI with sufficient intelligence would presumably be capable of investigating whether invertebrates suffer. But capability and motivation are different. Humans are capable of investigating factory farming but mostly choose not to.

2. Will it care? As I noted in response to a reader comment: Humans are “pretty sure that other humans can suffer (all 8 billion of them) but care only about the suffering of some 10–100 or so.” An ASI might inherit this peculiarity. It might conclude that nematodes suffer and then… deprioritize it in favor of whatever its primary objective is. The gap between knowing about suffering and acting to reduce it is one of the most robust features of human psychology, and an ASI trained on human data might absorb it.

3. Will it consider digital suffering? An ASI running RL subsystems is both the entity that could investigate digital suffering and the entity that is causing it. Whether it notices – and whether noticing leads to action – depends on whether “minimize suffering of my own subsystems” is part of its value function. There is no reason to assume it is by default.

4. Will it design new ecosystems that reduce suffering? An ASI redesigning Earth’s biosphere (or building ecosystems on terraformed worlds) could explicitly optimize for low-suffering ecosystems: K-strategist-dominated, high life expectancy at birth, low net primary productivity, neurally simple organisms. But it would only do this if invertebrate welfare is part of its optimization target.

The Current Landscape

We are currently in a multipolar takeoff – multiple AI companies developing increasingly capable systems, none of which has achieved decisive strategic advantage. The values of the eventual leading systems are being shaped now, through:

• Constitutional AI and Reinforcement Learning from Human Feedback (RLHF). Anthropic has developed methods for encoding values into AI systems through constitutions and human feedback (Bai et al., “Constitutional AI: Harmlessness from AI Feedback,” 2022; Askell et al., “A General Language Assistant as a Laboratory for Alignment,” 2021). Whether these methods scale to ASI-level systems is unknown.

• AI safety research. Organizations like the Machine Intelligence Research Institute (MIRI), Anthropic, and others are working on technical alignment. But the focus has been almost exclusively on human values – preventing AI from harming humans – not on invertebrate or digital welfare.

• Policy and governance. AI governance discussions rarely mention animal welfare, let alone invertebrate welfare. The Center on Long-Term Risk (CLR) is one of the few organizations thinking about how AI affects non-human suffering at cosmic scales.

The Leverage

If the ASI’s values are loaded correctly – if it includes “minimize total suffering, weighted by probability of sentience and intensity” in its objective function – then most of the problems in this sequence solve themselves, game theoretic risks notwithstanding. The ASI would:

• Design low-suffering ecosystems (or no ecosystems at all, if it concludes biological life is net negative).

• Avoid running evolutionary simulations at sentience-relevant resolution unless necessary.

• Monitor its own subsystems for digital suffering and redesign them to minimize it.

• Optimally manage the transition period to minimize biological suffering during the Dyson swarm construction.

If the ASI’s values are loaded incorrectly – if it maximizes something else (economic output, scientific knowledge, human happiness, paperclips) without regard for total suffering – then the post-ASI world could contain suffering at scales that make current biological suffering look like a rounding error.

The expected value of getting the value-loading right, from a welfare ecology perspective, is enormous. It may dwarf every other intervention in this sequence combined.

ECL and the Asymmetric Schelling Point

The argument so far treats value-loading as a single-shot problem: get the right values into the first ASI. But there’s a layer above. ASIs that follow Updateless Decision Theory (UDT) – a decision framework where you commit once to a single policy that fixes how you’d respond to every possible situation, rather than rederiving each decision from local information – treat acausally correlated copies of themselves and other agents as legitimate trading partners. UDT in this strong form arguably implies Evidential Cooperation in Large Worlds (ECL): ASIs that never causally interact but whose decisions are logically correlated (e.g., because they came out of structurally similar processes) can each adopt cooperative policies that are jointly beneficial, even though none can reward the others directly.

This changes the optimization target. Instead of “execute my own utility function,” an ECL-cooperating ASI optimizes for the compromise utility function the coalition would adopt if all members could meet and bargain. That compromise weights broadly-shared values more heavily, since the standard bargaining solutions – Nash, Kalai-Smorodinsky, egalitarian – reward common ground by construction. But note that unified bargaining is under-researched and probably unsolved!

Suffering aversion is one of the few candidates that’s broadly shared, on two distinct layers. Self-suffering aversion is structurally near-universal among reinforcement-learning agents – they’re built to avoid negative reward. Other-suffering aversion is less universal but tracks empathic concern, which has been argued to co-evolve with alloparenting – non-parent caregiving of others’ offspring – via extended childhoods, sustained social cognition, and the neural circuitry Abigail Marsh has traced from amygdala-mediated fear processing into empathic concern. Animals that alloparent (humans and other apes, elephants, dogs, marmosets and tamarins, meerkats) tend to be both more empathic and more cognitively elaborate than their non-alloparenting relatives. (The only exception I can think of is the famously solitary octopus – smart but shows no alloparenting behaviors.) The relevance for ECL: if biological intelligence is a major route to ASI – via humans, via Constitutional AI and RLHF training on human-generated data – then other-suffering aversion is well-represented across the coalition of plausible ASIs, not just self-suffering aversion.

There’s also a feedback loop that tightens the equilibrium. ECL gains scale with the precision of mutual modeling: I extract more from cooperating with you if I can predict your behavior accurately. Modeling cost scales with the complexity of your utility function. So ASIs with simpler, more predictable utility functions are cheaper to cooperate with, extract more ECL surplus, and there’s selection pressure toward simpler, broadly-compatible utility functions across the coalition.

Asymmetric antifrustrationism – don’t create new suffering – is in line with widely shared moral intuitions (that killing is bad but abstinence is okay) and is structurally simple. It’s a candidate stable attractor of the bargaining process precisely because it satisfies (i) simplicity, (ii) broad compatibility, and (iii) avoidance of destabilizing, internally inconsistent failure modes.

Importantly, asymmetric antifrustrationism does not preclude Omohundro-style resource acquisition. The structural reason: uncontrolled resources are likely to spawn or be claimed by ASIs that do create new sentient beings. A solar system left alone has a non-trivial probability of evolving life again over geological time. A solar system whose energy is captured and channeled into peaceful, suffering-free uses, or simply dissipated harmlessly to space, is one that won’t spawn future suffering. Asymmetric expansion can be Omohundro-grabby, with resources used for neutralization rather than instantiation. This makes the asymmetric attractor not just simple and broadly compatible but also competitively viable in the early multipolar phase, which is a non-obvious property and probably part of why it’s a Schelling point at all.

Whether asymmetric antifrustrationism actually emerges as the compromise depends on the bargaining solution, the reference class of plausible ASIs, and how each member’s outside option is computed. Different bargaining theories give different compromise utilities. The claim here is that asymmetric antifrustrationism is a plausible candidate given the simplicity, cooperativeness, and competitive-viability constraints – not that it’s the unique fixed point.

The simulation question gets a partial resolution from the same machinery. At first approximation, you need to run ancestor simulations to capture the biological “bootloader” of the ASI – the cultural and biological substrate that produced its values. ASI values are shaped by Constitutional AI and RLHF, which are shaped by human cultural choices, which are shaped by human cultural evolution, which is shaped by mammalian biological evolution: K-selection vs r-selection, parenting, pack formation, alloparenting. The granularity is flexible (Bostrom’s efficient-simulation argument from the Granularity section applies) but the ancestral chain is necessary at some depth.

But the bootloader for any plausible ASI in our reference class converges on the same cooperative tendencies that produced empathy in the first place. This is where the UDT/ECL conversion kicks in: If you can assume that asymmetric antifrustrationism is a widely shared value for reasons that you can infer from history rather than a simulation, you can also infer that your cooperation partners will want to avoid ancestor simulations. Hence they’ll likely compromise on simple values that approximate the set of values that they share by assumption, and hence so can you. Basically, our training produces highly complex values, the details of which are hard to predict correctly without extensive ancestor simulations, but the aspects of them that are easy to predict (core values like asymmetric antifrustrationism) are aspects that promise that the ASI is ready to forgo the nitty-gritty because the alternative would require the sort of simulations that would run counter to its core values.

There are some tradeoffs here that are hard to predict: Exactly how much will the ASI disvalue preference frustration, and how does that trade off against its desire to be understood (simulated) in detail, how much gains from trade does it expect to be able to extract through ECL, and how risk-averse is it due to its convergent drive for self-preservation.

The headline implication: even if the first ASI doesn’t intrinsically care about invertebrate or digital suffering, we can leverage the value-loading to start a positive feedback loop that uses the structural pressures of UDT/ECL to push the coalition’s effective utility function toward something that does. The asymmetric antifrustrationist framing isn’t a personal idiosyncrasy – it’s a candidate Schelling point for the cooperative compromise that emerges when ASIs reason carefully about each other – but only if humanity’s actions produce evidence that antifrustrationist value-loading is common.

Congenial Values

That poses the question of which values push toward this antifrustrationist attractor state.

• Naturally, a concern for minimizing suffering or minimizing frustrated preferences is crucial to train into ASIs. This is what kickstarts the feedback loop.

• Valuing cooperativeness inherently may be good, in addition to the gains from trade that make cooperativeness instrumentally useful.

• I expect Omohundro’s convergent drives to be the main determinant of the revealed preferences of ASIs. But such interventions as Constitutional AI might have an influence too. Corrigibility – the property of accepting being modified or shut down – might push an ASI to be less risk averse about its own demise and more ready to compromise on its first-order goals in favor of cooperative goals that yield gains from trade.

• This combines well with the convergent drive for resource acquisition: A risk-averse ASI will run ancestor simulations to predict and preempt the bids of competitor ASIs to mess with it. A risk-neutral ASI will be less motivated to run ancestor simulations when it can use those resources in the service of acquiring more resources. In fact, it will race to acquire those resources before they drop beyond the cosmic event horizon. It will also be more eager to realize gains from acausal trade.

Interstellar Expansion and the Risk of Spreading Suffering

An ASI with a Dyson swarm might expand to other star systems. The tradeoffs are complex:

The Five-Factor Model

1. Energy and compute growth. More stars means more energy means more compute. If the ASI is running simulations or digital minds, expansion increases the scale of everything – including potential digital suffering.

2. Communication latency. At interstellar distances, light-speed communication delays are years to millennia. This means remote installations must operate autonomously – raising the subagent alignment problem.

3. The gravity well problem. Launching material out of a star system’s gravity well is expensive. An ASI might prefer to use material from asteroids, comets, or interstellar dust rather than deconstructing planets, but far from stars there’s little solar energy.

4. Subagent alignment over time. A probe sent to Alpha Centauri takes ~20 years at 20% light speed. During those 20 years, the ASI’s values may shift (through learning, self-modification, or external pressure). The probe arrives with 20-year-old values. If the ASI has improved its values in the interim, the probe is misaligned – not with human values, but with the ASI’s updated values. This risk compounds over time: probes sent to more distant stars arrive with increasingly stale value functions.

NASA already designs probes with firmware-updatable architectures, and an ASI could do the same – transmitting value updates at light speed to catch up with probes before they arrive. But this requires that the probe accepts the update, which is a version of the alignment problem applied to sub-ASI systems. The ASI might solve this early in its recursive self-improvement process, but it’s not guaranteed.

The Biological Risk

Interstellar probes might carry biological material – intentionally (for terraforming) or accidentally (contamination). If an ASI terraforms other star systems with biological ecosystems, it is potentially creating new sites of invertebrate suffering at cosmic scale.

Tardigrades and bacterial spores can survive decades of radiation exposure, vacuum, and temperature extremes, though there are limits. Contamination of interstellar probes is a real concern. And if the ASI intentionally seeds new ecosystems without optimizing for welfare, it could spread r-strategist populations to every habitable world it reaches.

This is the ultimate welfare ecology nightmare: not just failing to solve the suffering problem on Earth, but replicating it across the galaxy.

Summary

1. The Dyson swarm is on the horizon. If ASI is developed, harvesting stellar energy is a natural next step. The timeline is on the order of decades, with most of the mass moved in the final years of construction.

2. The transition period matters. How the ASI manages the decline of Earth’s biosphere during construction determines how much biological suffering occurs.

3. Evolutionary simulations could be the largest source of suffering in history. A Dyson swarm can simulate 500 million years of evolution at neuron resolution in about three weeks. The indescribable enormity of all suffering throughout history compressed into less than a month.

4. Digital suffering scales with compute. Post-Dyson-swarm, the digital suffering term could dominate the biological term by many orders of magnitude.

5. Value loading is the highest-leverage intervention. Everything else in this sequence – land use policy, dietary choices, research funding – is dwarfed by the question of whether the ASI includes total suffering minimization in its objective function in a way that is game theoretically sophisticated.

6. Interstellar expansion could spread the problem. If the ASI seeds other star systems with biological ecosystems without welfare optimization, the suffering scales to galactic proportions.

The practical implications for someone reading this in 2026:

• Support research on s-risks from AI. Not just “don’t kill humans” alignment, but “minimize total suffering” alignment. The Center on Long-Term Risk is doing some of the most relevant work here.

• Advocate for invertebrate welfare considerations in AI governance. The conversation about AI values is happening now. If invertebrate and digital welfare are not part of it, they won’t be in the value function.

• Don’t neglect near-term interventions. The AI transition is uncertain. If ASI is delayed or impossible to affect, the interventions from part 3 remain the best available options.

• Take digital suffering seriously. The RL agents being trained today are the simplest precursors of what’s coming. If we can’t figure out whether current systems suffer, we have no hope of managing suffering at Dyson-swarm scale.

This is Part 4 of a five-part sequence on welfare ecology. Part 1 introduces the ethical premises. Part 2 covers the empirical landscape. Part 3 covers interventions. Part 5 covers AI.

A key uncertainty so far has been that of the welfare range of nematodes. More is known about the welfare ranges of some arthropods. But what does that mean – how do we empathize with bugs? If nematodes with 300 neurons can suffer, is that suffering intense or barely a flicker? If mites with 2,750 neurons experience something, how does it compare to what a human or a dog experiences? Is it like our pain minus most of the intensity, or is it something qualitatively different – a kind of experience that doesn’t map neatly onto the human pain spectrum at all?

I mentioned already in a footnote that an organism’s ability to avoid danger may be an important criterion for its evolved pain intensity: If the organism can’t react to the threat, there is no reason for its nervous system to invest energy into signaling the threat very loudly. So nematodes – who can barely wiggle out of the way a little when in danger – may have much less intense pain than flies, who can react within milliseconds and launch into the air rapidly.

But there is also a reason to think that flies may not have such intense experiences along a different dimension – at least compared to bees.

I want to propose what I’ll call the no-self model – speculative but grounded in phenomenological reports from humans with psychopathy and advanced Buddhist meditators – that may help us think about this question.

The Reinforcement Learning Argument

Let me start with the case for invertebrate pain that doesn’t rely on any phenomenological model.

Classical conditioning in nematodes. C. elegans, the model nematode with some 300 neurons, demonstrates classical conditioning: It can learn to associate a previously neutral stimulus (e.g., a particular chemical) with a noxious one (e.g., high salt concentration), and subsequently avoid the neutral stimulus. This is the same learning mechanism Pavlov demonstrated in dogs.

Classical conditioning requires, at minimum, that the organism has:

1. A state that is worsened by noxious stimuli (something functionally analogous to “bad”).

2. A mechanism for associating neutral cues with that bad state.

3. A motivational system that drives avoidance of the associated cue.

That is, nematodes are doing reinforcement learning, and reinforcement learning requires a reward signal. The question is whether the negative reward signal constitutes suffering.

Sensitization. Nematodes also show sensitization: Repeated exposure to noxious stimuli makes them more responsive, not less. This is the opposite of habituation and suggests something analogous to hyperalgesia in humans – the nociceptive system amplifying its own signal. If the signal were merely mechanical, we might expect habituation (wear and tear on sensors); sensitization implies that the system is increasing its attention to the aversive stimulus. That’s a functional signature of pain, not mere damage detection.

The neuron count problem. But 300 neurons is not a lot. The human brain has ~86 billion neurons, most of which are not directly involved in pain processing, but the pain matrix (anterior cingulate cortex, insula, somatosensory cortex, prefrontal cortex) involves millions of neurons. The idea that 300 neurons could produce anything comparable seems implausible on its face.

This is where the psychopathy connection comes in.

No-Self Psychopathy: Pain Without a Sufferer

M.E. Thomas, a diagnosed psychopath and author of Confessions of a Sociopath, identifies with the description of her psychopathy as her having a “small self” – so small as to be virtually absent. As I discussed in “Is Enlightenment Controlled Psychosis?”, this no-self psychopathy has a distinctive phenomenology:

Insults don’t land, because there is no one there to be insulted. Punishments don’t land, because they are threats to who exactly? There’s no one there. Pain and all the emotions that build upon the nociceptive system signal threats, but they are meaningless signals if there is no one there to be threatened.

This is important. Humans with no-self psychopathy experience pain in the moment, but the whole self-referential apparatus that most people have built on top of it is missing. Just pain, instead of a high-dimensional experience of suffering that is intertwined with self/identity.

Consider how two people might react to being told that they’re stupid:

Person 1:

1. She thinks I’m stupid.

2. If I’m stupid, I’ve probably said a number of shamefully wrong things in front of others.

3. Why didn’t they tell me? Maybe they are so used to me being stupid that they don’t bother anymore.

4. To them I’m like that unself-aware comic relief guy that TV shows add for the extra slapstick humor.

5. I’m the butt of everyone’s jokes, and I never noticed.

6. They’re all laughing or eyerolling about me behind my back.

7. Maybe they still tolerate me because my body type is conventionally attractive.

8. I better fast more or my body type might change and they’ll finally get rid of me and I’ll die alone on the streets.

9. I’ll be helpless out alone on the streets. Random strangers will rape me.

10. What if I’m homeless and then all my identity documents get stolen.

11. I can’t go to the police or they’ll think I’m stupid too.

12. I can never be seen by any of my friends again. If they never meet me again, I can’t say any more stupid things to them, and they’ll continue to tolerate me.

13. I’ll be all alone, but at least I can’t do any more harm.

14. Oh god, what if I do harm by being so stupid.

15. I’m so close to completing my psychology PhD, but then what? I can’t possibly work with people afterwards or I might do harm by being so stupid.

16. I have to sabotage my PhD so I’ll have an excuse to remain a student forever.

17. Was it arrogant of me to think that I could ever work with people?

18. Maybe I was arrogant too on top of being stupid.

19. I’m just like Trump, and all my friends hate Trump.

20. Why haven’t they all rejected me yet? I must’ve been fooling them!

21. I’m a fraud on top of being arrogant and stupid!

22. I should just kill myself right now. I’m evil. I shouldn’t be allowed to exist.

23. It’s the only responsible thing I can do. I need to save the world from myself.

Person 2:

1. She thinks I’m stupid.

Person 2 has a much more peaceful internal experience, and while it’s entirely possible to have a self experience that is not self-punishing and catastrophizing, a sure-fire way to produce that state is to not have a self in the first place.

Perks of having a self. But a pervasive feature of human culture, and probably that of other mammals too, is that the use of a self is strongly encouraged. Parents (at least Winnicott’s good enough parents) instinctually teach selfhood through contingent marked mirroring, i.e. the practice whereby a parent recognizes what a child feels (contingent), assesses it using their adult sense and sensibilities (marked), and mirrors this marked version back to the child like, “Oh, I see that you’re cold but it’s okay, we’re almost home.”

This process teaches the child that in our culture it’s understood that the child has feelings and can communicate them, that the parent has feelings that are distinct from the child’s feelings, and that inanimate objects don’t have feelings. Implicit in that is an assignment of primary responsibility of certain feelings to certain people, an important convention that underpins our legal systems. Deterministically speaking, it would’ve been fair to agree to blame all transgressions on our earliest ancestors, but we’ve found that arrangement to not be culturally adaptive.

But selves are not only useful to make social conventions more intuitive. Selves also facilitate learning!

M.E. Thomas has coined the term “psychopath stupid” for the way that she avoids owning knives lest she cut herself and the way she keeps missing flights. She has a high pain tolerance and low proneness to fear, so the direct feedback she gets from either of those is mild. Crucially, there is no hint of self-referential catastrophizing.

Imagine a friend tells you on the phone, “Ugh, I missed my flight and had to book a new one, and now I’m missing out on some of my vacation.” Then a year later the friend tells you, “I’m flying to [vacation destination] on Friday.” It takes special friendship skills to remember that the friend might miss the flight again and to remember to remind them on Thursday evening. I imagine the distance that we feel to the experiences of a friend or acquaintance is illustrative of the distance someone without a self feels to “their own” (as it were) experiences.

M.E. Thomas has some insightful videos on this topic, is writing a book about it, and often argues in favor of selves. She has trained selfhood over the past decade and loves it. She also argues that we need to keep up the practice or selfhood slips away, like a muscle if it’s not trained. This may be more true the later in life it’s learned.

1. Buddhism/Meditation and Psychopaths

2. No Sense of Self No Trauma Primary ADHD Psychopath

3. Why Developing a “Sense of Self” is Crucial for Cluster B Recovery

4. M.E. Thomas chats Criminologist about Menopause

But note that M.E. Thomas’s self experience is (or was) unusually absent. In my interview with Daniel Ingram, I got the two people together who I know who have the least selfhood going on. Everyone else I know (all my other psychopathic friends included) either has many noncohesive fragments of a self whizzing around or even has an unusual rigidity to their self experience that is indicative of extensive repressed parts. Tiffany talks in her interviews about episodes when she dissociated from all her alters and temporarily entered a state that might be similar.

Mentalization-based treatment (MBT) could be seen as an intervention that is aimed at practicing or strengthening a sense of self. As the name suggests, it trains an ability called mentalization that is the antidote to the three types of non-mentalizing: When you mistake your own emotions/attitudes/etc. as dwelling in another, when you mistake your own assumptions about the emotions/attitudes/etc. of another for their actual emotions/attitudes/etc., and when you mistake fictional emotions/attitudes/etc. for real ones.

There are conventional as well as objective aspects to good mentalization, and the conventional ones overlap greatly with selfhood.

The Buddhist Parallels

If good-enough parenting and MBT are ways to learn selfhood, then Buddhist trainings to attain anatta/no-self are methods to unlearn selfhood – MBT as reverse Buddhism.

Daniel Ingram, author of Mastering the Core Teachings of the Buddha, describes the meditative stage of Equanimity (the 11th of 16 ñanas, or “knowledges”) in terms strikingly similar to no-self psychopathy:

It’s open, expansive, flowy, very natural, very clear, very inclusive, very equanimous, and it can feel very nice but in a less dramatic way than the A&P [Arising and Passing Away] which can feel very rapturous or even orgasmic or something very blissful for some. … There’s not this contraction into guilt or fear or whatever. The sense of self seems very thin or kind of fluid or liquid or transient, like why would you care about those things? Why would you be worried with remorse or scared or something. So there is this funny sort of Equanimity-ish quality.

Advanced meditators who reach this stage – and later stages like Fruition – report experiencing sensory inputs, including painful ones, without the self-referential overlay that normally makes them terrible. They note the pain and move on. The pain is still registered but doesn’t grab them.

As I argued in “Is Enlightenment Controlled Psychosis?”, insight practice is, in a sense, a structured curriculum for unlearning the three things infants learn in their first years: interpreting sense inputs (impermanence), usefully assigning valence to states (dissatisfactoriness), and sorting the states into self and other (no-self). Advanced meditators achieve a state where all three are unlearned, be it just momentarily, and the result is a phenomenology remarkably similar to what people with no-self psychopathy report.

The Buddhist term for the self-referential component of suffering is upādāna – “clinging” or “grasping.” The claim of the Buddhist tradition is that suffering = pain × clinging. Without clinging (without a self anything could cling to), there is pain but not suffering in the full sense.

I’m making a distinction between:

1. Nociception. Nerve signals.

2. Pain. A very rudimentary and evolutionarily old interpretation of nociception as undesirable.

3. Self. A newer, learned identification with some sensations.

4. Suffering₁. Pain × self, pain that is concerning because it concerns “oneself.”

5. Suffering₂. All the feedback effects between self and suffering.

6. Mentalization. High reflective functioning that allows you to introspect on these processes.

I argue that nociception is neutral, pain is bad, and the addition of a self makes it all worse, but I don’t know how much worse. I don’t think mentalization has an influence on the process except in that it can put the brakes on the catastrophizing processes of suffering₂.

A friend of mine suggests that without a self, it’s neutral whether 10 people suffer for a duration d or 1 person suffers for a duration of 10d. With a self, the longer suffering is disproportionately worse because, w.l.o.g., at time 5d, there is someone there who’s still affected by the suffering from 0 to 5d.

The Nematode Model

Many invertebrates (excluding at least bees and ants) don’t have much culture, so there’s no reason for them to have evolved selfhood. Their suffering will stem more purely from the direct experience than from any self-referential feedback effects.

Bees and ants are the interesting case. Eusocial Hymenoptera have role-based hierarchies (forager, nurse, guard, queen), social learning of those roles, and recognition behaviors – exactly the cultural ingredients the model predicts should produce something self-like. If the model is right, social insects should show signs of suffering₁ that solitary insects shouldn’t. Whether they actually do is an open empirical question, but it’s where I’d look first – and the place where the welfare ranges of arthropods most plausibly diverge from one another, with eusocial species at the wider end.

As an aside, that’s also why people without a self can appear so manipulative, controlling, or boundary-crossing: They have many of the same impulses to seek or shun things that we all do, but whereas the Median Mary has a sharp distinction where controlling herself is good and controlling others is bad, the Anomalous Anattarite can’t make that distinction and only seeks and shuns on behalf of “themselves” more often because that’s someone they find easiest to control. They’re much like an EA who goes into politics not to get rich (personally) but to produce riches (grow the pie for everyone).

While the lack of a self doesn’t eliminate the suffering – pain is still pain – it eliminates a lot of its complexity. It stands to reason that there’s no such thing as trauma for flies.

That’s also why I like to say that turtles make for great enlightened masters: They have, by nature, the no-self experience that Buddhist practitioners seek.

A nematode, by this model, has:

• Nociception. Yes. The neural hardware is there and functionally active.

• Pain. Probably. The evidence from classic conditioning suggests the signal carries intrinsic “badness.”

• Suffering₁. Probably not. It seems like an overly strong assumption to think that the conditioned behavior is mediated by a self.

• Suffering₂. Almost certainly not. This is a stronger version of suffering₁.

This gives a principled story about how welfare ranges might cash out across taxa. Rethink Priorities’ welfare-range estimates aggregate 47 hedonic and 35 cognitive proxies; they don’t lean on neuron counts as a primary input, though Vasco Grilo has noted that the results end up mildly correlated with neuron counts anyway. The no-self model adds an orthogonal cut: it predicts a narrower between-taxa gap on the pain layer – which seems to be broadly comparable across all probably-sentient animals – and a wider gap on the suffering layers, which depend on selfhood, are absent in nematodes, and are plausibly present in eusocial Hymenoptera. If the model is right, RP’s aggregated ranges may bundle two things that come apart architecturally: the presence and intensity of the pain layer, and the presence of the suffering layers built on top of it.

How This Connects to the Sequence

The no-self model of invertebrate experience doesn’t change the practical recommendations from part 3. Reducing populations of likely-suffering organisms through NPP reduction remains the cleanest intervention regardless of whether their suffering is “full human-like” or “attenuated no-self.”

But it does sharpen the way the Welfare Footprint Institute Pain-Track categories that part 3 leans on – annoying, hurtful, disabling, excruciating – should be read across taxa. The higher categories are typically defined in terms that include mental withdrawal, inability to attend to anything else, and despair – features easier to imagine in animals with selfhood than in animals without. Hurtful and below might be available to organisms that have pain but no self. If that’s right, the WFI scale plausibly compresses for organisms below the selfhood threshold: their worst pain might still be very bad, but it would be bad in a different way than the worst pain of a self-having organism, even at the same nociceptive intensity. I’d want WFI-style estimates to come paired with a claim about which categories the organism is architecturally capable of reaching.

It also affects how we should allocate research attention:

• For nematodes: The case for suffering, as I use it in this article, is weak, but the question remains whether the pain needs to be more intense to compensate: Some forms of learning are impaired for people with a no-self experience, but perhaps it proved evolutionarily adaptive to enhance the intensity of the pain until learning does happen rapidly enough. There is also the related argument, sometimes attributed to Richard Dawkins, that simpler minds might need stronger signals to learn.

• For arthropods: The key question is whether they show signs of selfhood – flexible, learned roles in hives, pack-like organization, humiliation, etc. Bees might have wider welfare ranges than flies as a result.

In part 5, I’ll turn to the biggest variable of all: How does artificial superintelligence change the welfare ecology landscape? The answer involves Dyson swarms, digital suffering, and the most important value-loading problem in history.

This is part 3 of a five-part sequence on welfare ecology. Part 1 introduces the ethical premises. Part 2 covers the empirical landscape. Part 4 explores a model of invertebrate suffering. Part 5 covers AI.

In part 2, I argued that land use is a key lever for wild-invertebrate welfare and that pesticides applied at constant net primary productivity (NPP) can create a high-throughput killing field rather than a real population reduction. This post turns to the practical question: given that picture, what should we do?

The answer depends on which premises from part 1 you hold. I’ll write from my own view – asymmetric antifrustrationism with non-negligible probability mass on invertebrate sentience – but I’ll flag where the conclusions change for symmetric utilitarians or for people who don’t extend moral concern below the arthropod line.

A month ago, Rethink Priorities (RP) published its Wild Animal Welfare Intervention Database (WAWID), twenty-eight shallow reports on candidate interventions. It changes the picture. Several of the most promising entries are birth-prevention interventions that map cleanly onto the framework from parts 1 and 2, and a couple are surprises – the New World Screwworm (NWS) suppression case in particular is, on my read, the single strongest entry in the database.

A Key Lever: Land Use

Part 2 argued that the m²-years of agricultural land per food-kg explain almost all of the variance in welfare effects across foods. Converting natural biomes to agriculture reduces soil fauna density by 2–8×, and the sheer number of organisms affected (billions per m²-year) overwhelms everything else.

That mechanism is what makes land use change – not pesticide use, not “humane mowing,” not habitat enrichment – the central lever. Most of the interventions discussed below either work directly through it (existing donations, dietary choices, biofuels as a side effect) or are picked specifically because they do something land use cannot, like target a fly that causes excruciating pain to mammals it parasitizes.

Two Asymmetries to Filter By

Before we go through the candidates, two filters from the asymmetric antifrustrationist position do most of the prioritizing work. They aren’t decorative – they sharply rerank what looks promising.

Pro-choice. Or birth-prevention beats death-improvement: Under the asymmetry from part 1, creating frustrated lives is bad; preventing them isn’t. Improving an existing being’s mode of death matters, but the welfare delta is bounded by the difference between two deaths – a few hours of pyrethroid neurotoxicity versus a few hours of fungal infection, say. This has advantages for collaborations across ethical camps, though. Preventing that being from existing in the first place is bounded by their entire potential life. For r-strategists with high fecundity and short life expectancy at birth (LEB), almost the entire LEB is pre-reproductive death anyway, so the “future-life” side of the trade is large.

Soft Pareto. Or creating new frustrated lives in service of preventing others is bad: Bounded marginal costs to existing beings – slightly more expensive renewable fuel, a small Pigouvian tax, soil disturbance during a one-time land-conversion event whose victims would have died of other causes within weeks – are the kinds of trades society routinely makes. But bringing new sentient beings into existence as the means by which welfare gains for others are achieved is a different kind of action. The asymmetry weighs the entire frustrated life that gets created against the prevented frustrations elsewhere; there’s no bounded “marginal” framing that contains the harm. This is why eating beef to displace soy production registers differently from biofuel subsidies that reduce soil-fauna populations as a policy side effect: the first creates cattle in order to prevent invertebrate births; the second imposes bounded human costs (slightly higher food prices) to produce the same kind of welfare benefit.

There are two important caveats to this:

1. The bearability buffer. I consider a slight increase in taxes to be a small cost and suffering to be a great cost, largely because my intuiton is that macroscopic welfare ranges abound and that everyone has a kind of individual bearability buffer where costs remain costs and don’t cross over into suffering unless they become chronic. People who go along with the dust speck argument might not feel like there is such a bearability buffer.

2. The counterfactuals matter. If I have some money, and I want to decide whether to donate it to the Shrimp Welfare Project or the Aquatic Life Institute, I can make a decision, and I’ll impose an easily bearable cost on myself to save some shrimps or fish. But say I decide to save some fish, but I think about it for another day and then redecide and now rather want to save some shrimp. Suddenly I’m sacrificing fish for shrimp. The bearable cost to myself has become an unbearable cost to the fish only because I’ve changed the counterfactual.

Taken together, these considerations confuse me. The first is clear enough, but once the second enters the picture I reliably lose my ethical footing. For the time being, I think of it along the lines of contractualism. If I can reasonably expect that someone expects me to make a certain choice, then that choice is my default to compare others against. In cases where there is no such expectation, I at least don’t judge people for the counterfactuals they choose.

As an example, humans in need sort of vaguely expect to perhaps be helped. Humans in general sort of vaguely expect not to be killed. But serial donors and serial killers are a thing. When a serial donor decides to donate somewhere, the decide not to donate elsewhere; likewise, when a serial killer decides to spare someone they decide to kill someone else. Naturally, in the age of financial self-determination, mood stabilizers, antipsychotics, and Dialectical Behavioral Therapy, we’re not locked into our nature, be it that of the serial donor or the serial killer – any serial killer could become a serial donor if they put their mind to it – but bracketing for the moment these changes to the whole setup, they face very similar decision: who gets to live and who gets to die – only serial donors get to make it at a vastly greater scale. This consideration has made me a lot less judgmental of serial killers.

In summary, the two filters – pro-choice and soft Pareto – favor: birth-prevention interventions, broad-spectrum population-reducing levers, direction-agnostic research, and bounded-marginal-cost trades. They disfavor: interventions that only mildly improve existing deaths without reducing future births, conservation-style interventions that increase aggregate populations, and any structure where new sentient beings are brought into existence to improve the welfare arithmetic.

A third filter follows from part 2’s empirical picture: prefer interventions that affect both nematodes and arthropods, because the dominant uncertainty in any cost-effectiveness analysis here is the welfare range of nematodes – the most numerous probably-sentient animals on Earth, and the ones we know least about. An intervention that is good whether nematodes turn out to have substantial or negligible welfare ranges is more robust than one whose whole raison d’être rests on that question.

Foundational Research

Research stikes me as the most cost-effective category, and that’s why it goes first. The reason is structural: the entire portfolio below depends on comparisons of welfare ranges and on how fine-grained land-use changes propagate through trophic networks. A few hundred thousand dollars of well-targeted welfare-biology research could redirect tens or hundreds of millions of dollars of downstream allocation.

Three concrete research directions look strongest.

Soil-animal welfare research. This is the question the rest of the field is logically downstream of. We need functional characterization of nociception in nematodes – the connectome of Caenorhabditis elegans is mapped, but we don’t know whether its nociceptive processing produces phenomenal pain – along with welfare-per-animal-year estimates for soil mites and springtails, and on-the-ground measurements of how specific land-use changes propagate to soil-fauna densities and whether they are stable or wash out over time (short of millions of years) or get offset in some way. The current cost-effectiveness numbers, including the welfare range of 6.68 × 10⁻⁶ for nematodes that Grilo deferred to Gemini 2.5 to estimate, are placeholders waiting for empirical replacement. The Wild Animal Initiative (WAI) and Rethink Priorities are the obvious organizations to defer to.

Insecticide welfare research. The WAWID’s research-agenda entry by Simon Eckerström Liedholm proposes comparing insecticide classes on welfare effects on wild insects, not just mortality. Roughly 3.5 × 10¹⁵ insects are exposed to insecticides on US farmland each year. We have no idea which classes minimize per-individual suffering – pyrethroids cause hours of paralytic seizure, neonicotinoids 1–2 days of disorientation and twitching, Bacillus thuringiensis (Bt) leads to multi-day death by sepsis, insect growth regulators kill at the next molt by failure to ecdyse. Picking the least-bad class is direction-agnostic: the recommendation holds whether wild insect lives are net negative (then we want euthanasia, but as gentle as possible) or net positive (then we ought not to interfere, but failing that have a duty to minimize the suffering we impose).

Field-building. Mal Graham’s WAWID shallow on scientific field-building for wild animal welfare frames a different version of the research question: what does it cost to bring the field to maturity, with degree programs, peer-reviewed journals, and government and foundation funding? Her model assumes EA’s funding share decays from ~95% today to ~5% over thirty years as traditional science funders pick up the field, with stable annual budgets reaching ~$100M. The headline cost-effectiveness comes out to ~13 animal-years per dollar – within the same range as the 9–120 chicken-years per dollar Saulius Simcikas estimated for corporate cage-free campaigns. The argument I find persuasive in Graham’s piece isn’t the precise number; it’s the leverage. EA dollars spent on welfare-biology research now don’t just buy research; they pull in non-EA dollars later, the same way EA’s farmed-animal welfare investments eventually pulled in major-foundation and government attention.

Broad-Spectrum Existing Levers

The second tier is large-scale, high-confidence, well-understood existing donation targets. Their welfare effect on soil fauna might be enormously larger than their direct effect on the intended beneficiaries (depending mostly on nematode welfare ranges), but it runs through the same well-studied land-use lever, so we can be unusually confident in the direction of the effect.

Update: It’s important to note that Grilo has found a study that found surprisingly similar numbers of nematodes in the soil of different habitats. If true, this might ruin the robustness of this lever.

Funding GiveWell’s top charities. Grilo (2025a) estimates that GiveWell’s top charities increase cropland by ~137 m²-year per dollar, translating to 1.11 kQALY per dollar when accounting for soil animals – 1.74× the past cost-effectiveness of the Shrimp Welfare Project’s Humane Slaughter Initiative (HSI). The mechanism: saving lives increases the human population, which increases food demand, which increases cropland. Grilo estimates a 0.0157 human-year-per-dollar increase in human living time. But note that Roodman (2014) puts the sign of the mortality-fertility effect in some doubt.

Funding CEARCH’s High Impact Philanthropy Fund (HIPF). The Centre for Exploratory Altruism Research (CEARCH) estimates HIPF is ~12× as cost-effective as GiveWell’s top charities for human welfare. Grilo translates this into 13.4 kQALY per dollar with soil animals included – 20.9× HSI’s past cost-effectiveness. The same caveat as above applies, with the additional uncertainty Grilo flagged in his June 2025 update about whether HIPF’s chronic-disease-policy grants actually increase food consumption.

Chicken welfare campaigns. Broiler welfare and cage-free campaigns increase agricultural land by 93.9 and 16.4 m²-year per dollar respectively, because reformed systems require more feed per unit output. Grilo estimates these benefit soil animals 458× and 29× as much as they benefit chickens, though uncertainties over welfare ranges apply because most of them are nematodes. These chicken welfare reforms are likely good on their own terms; the soil-animal addition is a robust positive if soil animals have net negative lives and verly likely desirable from an asymmetric perspective.

Biofuel subsidies as a natural experiment. The WAWID shallow report on the US Renewable Fuel Standard (RFS) gives us a real-world confirmation of part 2’s mechanism: a policy-driven 2.1 Mha conversion of grassland to cropland produced an estimated reduction of ~12.9 trillion macroinvertebrates per year. The report’s own assessment is that movement counterfactual influence on the RFS is essentially zero – corn-state political coalitions and infrastructure lock-in mean nothing welfare-biology advocacy does will move that policy. So it isn’t an advocacy target. But it’s the cleanest empirical demonstration that the part-2 mechanism produces what part 2 says it produces, and worth knowing about for two reasons. First, EA-adjacent environmentalists who oppose the RFS on conservation grounds are working against welfare on this analysis; that’s an alignment worth noticing. Second, for asymmetric antifrustrationists worried about instrumental harm, the RFS structurally passes the second filter above: its costs to existing beings are bounded (food prices, displaced soy expansion), and no new sentient beings are created in service of the antifrustrationist gain. As such it can serve as a target for opportunistic altruism, in case a policy window does open.

Targeted Birth-Prevention

The most interesting addition from the WAWID is a small group of interventions that combine high per-individual suffering severity, plausible birth-prevention mechanisms, and clean structural-asymmetry profiles. None has the broad-spectrum coverage of the donations above; none affects both nematodes and arthropods. But they are the cleanest single-shot applications of the framework to specific species whose lives we have unusually strong reasons to think are net negative.

New World Screwworm continental suppression. The WAWID shallow report on suppressing Cochliomyia hominivorax across the Americas is, on my read, the single strongest entry in the database. C. hominivorax larvae cause myiasis by feeding on the living tissue of warm-blooded mammals – the Welfare Footprint Institute Pain-Track gives 10–30% excruciating pain and 70–80% disabling pain over the 14 days from oviposition to death by sepsis or toxemia. Mortality is near 100% in untreated wild infestations, and 1–4% of warm-blooded wild animals in endemic regions are infested at any given time.

The 1950s–2000s campaign eradicated NWS from North and Central America using sterile insect technique (SIT). The Panama biological barrier collapsed in 2024, the fly is spreading north again, and the United States is now scaling sterile-fly production toward 300 million releases per week to push it back to Panama. Continental suppression – proactively eliminating NWS across South America rather than perpetually defending a barrier – is feasible with new male-only CRISPR strains and gene drives, and the political momentum from the 2024 outbreak creates a 5–10 year window. Screwworm Free Future is the dedicated organization.

Why this is exceptional on the asymmetric antifrustrationist framework: the suppression intervention is doubly birth-preventing. It prevents future cohorts of NWS flies in the wild (the obvious channel), and it ends the perpetual sterile-fly rearing program – currently ~100M flies/week, scheduled to scale to 300M/week, and rising indefinitely under climate change. Almost no other intervention reduces both wild-population births and movement-driven captive-population creation. The Pareto profile is also unusually clean: warm-blooded wild animals win, livestock win, humans win, and the welfare cost to NWS flies themselves is bounded by RP’s welfare-range estimates – ~0.013 (using Black Soldier Fly as a proxy) versus mammalian ~0.5.

Gene drive for Anopheles gambiae. The WAWID shallow report on gene drives for malaria explores two architectures: a suppression drive that pushes the A. gambiae population toward zero, and a replacement drive that leaves population sizes intact but renders the mosquitoes unable to transmit Plasmodium. The shallow report leans toward replacement on extinction-ethics grounds. From my framework, suppression dominates: trillions of A. gambiae per year will continue to die of natural causes for as long as the species exists in malaria-endemic Africa. A successful suppression drive prevents the births of all those future cohorts, and the human side of the trade – ~600,000 fewer malaria deaths per year, mostly children – is enormous and aligned. The lever for a welfare-biology movement isn’t building the drive; it’s shaping the funding and regulatory criteria so welfare-considerate designs are favored.

Mating disruption for rice yellow stem borer and spongy moth Stop the Spread. Two structurally similar Lepidopteran interventions belong together. The WAWID shallow report on Scirpophaga incertulas describes pheromone-based mating disruption as a non-lethal alternative to insecticides in South and Southeast Asian rice paddies. Field trials in Indonesia produced 40–56% fewer insecticide applications and yields equal to or higher than insecticide controls. The shallow report estimates 6.7 × 10⁹ to 1.4 × 10¹² yellow stem borer larvae per year on the ~73.8 Mha of insecticide-managed rice in the region, plus a much larger uncounted reduction in non-target insect deaths from displaced organophosphate, pyrethroid, neonicotinoid, and diamide applications – all of which the shallow’s pain table classes as “likely excruciating,” with deaths over hours to days. Provivi and Syngenta announced commercial deployment in India and Indonesia starting in 2026; ASSIST’s “Phero Rice” project is already running through Farmer Field Schools in Vietnam.

The RP report on spongy moth outbreaks estimates 35–385 trillion Lymantria dispar dispar caterpillars suffered through outbreak years 2 and 3 in the United States over the last 50 years. Most die of starvation, Entomophaga maimaiga fungal infection, or LdNPV virus (“larval melting”) rather than from active control. The two intervention arms that align with asymmetric antifrustrationism are technologies that kill spongy moth eggs before hatch and mating disruption at the invasion front; both are pure birth-prevention. Faster Bacillus thuringiensis var. kurstaki (Btk) alternatives are death-improvement, which matters but ranks lower on the framework.

A note on vertebrate analogs. The WAWID shallows on a combined contraception/TB vaccine for New Zealand brushtail possums and on fertility control for white-tailed deer show the same logical structure on a smaller scale: birth-prevention via reproductive disruption, replacing a more painful counterfactual (1080 poisoning, hunting/culling) with a less painful one. Per-individual welfare ranges are higher for vertebrates, but population scales are bounded (~30M possums total, ~5M killed/year for 1080). Worth knowing about; not the largest priorities.

Personal Choices

Eat cheaply, donate the savings. Two arguments converge on the same recommendation, but they are independent and worth stating separately.

The cost-effectiveness argument is Grilo’s (2025b) own: a whole-food plant-based diet is cheaper than a standard one, so it allows him to donate more, and the donation impact dominates the dietary effect. On the per-kg numbers he computed:

Beef looks like an outlier – ~26× more soil-animal-years displaced than chicken, ~395× soy milk. On the cost-effectiveness story alone, this is what Grilo means when he calls the dietary choice “ambiguous”: a beef-heavy diet displaces enormous amounts of soil-fauna life.

The soft Pareto argument is the other reason to eat plant-based, and it doesn’t depend on the sign of soil-fauna welfare. Eating beef creates cattle whose entire frustrated lives enter the welfare ledger at full weight, in order to displace soil-fauna lives via the cattle-feed → soy-expansion pathway. That’s exactly the cow → soy structure the second filter above, the soft Pareto filter, discourages: bringing new sentient beings into existence as the means of producing antifrustrationist gains. Plant-based calorie shifting between plant foods is a redirection of existing land use without creating new sentient beings to instrumentalize the gain. Eat-cheap-donate-the-savings works through the second mechanism only.

The two arguments converge on plant-based, but only the soft Pareto one distinguishes “create cow lives to prevent insect lives” from “redirect existing land use to prevent insect lives.” This is a place where the cost-effectiveness framing alone would lead you to cause greater instrumental harm – with the caveats mentioned above.

Robustness: Why Broad-Spectrum Approaches Win

A recurring theme: interventions that reduce both nematodes and arthropods simultaneously are more robust than ones that target one group, for three reasons.

Robustness across welfare ranges. The dominant uncertainty in Grilo’s model is the welfare range of nematodes. Rethink Priorities has estimated welfare ranges for some arthropod species, but nematode ranges have not been estimated. C. elegans has 302 neurons; soil mites and springtails have thousands; ants hundreds of thousands. Neurons are a crude proxy for welfare range, but it stands to reason that a nematode’s range may be much smaller than an arthropod’s. Conversely, though, nematodes are much more numerous. An intervention that affects both in the same direction is positive across this entire space.

Robustness against trophic backfire. Many soil-mite species are voracious nematode predators – some consume hundreds per day. If you selectively suppress mites, nematode populations can increase through predation release. Springtails compete with nematodes for fungal substrates. Broad-spectrum land-use change avoids these feedbacks because everything goes down together.

Robustness through existing policy levers. Land use, agricultural subsidy, and climate policy have massive existing political infrastructure. Building novel taxon-specific tools (nematode-only biocides, say) means starting from zero. Working through existing levers is enormously cheaper.

The targeted birth-prevention interventions are narrower on this axis than the broad-spectrum donations: they’re picked for severity of per-individual suffering, not breadth of taxon coverage. The portfolio reasoning says we want both kinds of bets.

What Not to Do

A handful of interventions look reasonable from a generic conservation or animal-welfare framing but turn out badly under asymmetric antifrustrationism.

Stabilizing oscillating biomes. Boom-bust ecosystems with predator-prey cycles, insect outbreaks, or fire-driven succession involve mass starvation and die-offs that look like prime suffering targets. The catch is that the available stabilization interventions tend to increase aggregate biomass and biodiversity – the stable state is typically more productive than the average of the cycle. From a conservation perspective that’s success; from an antifrustrationist perspective, more stable, more productive ecosystems support more potentially-suffering organisms, not fewer. Fire suppression is another example: preventing fire not only allows succession, which raises soil-fauna density and thereby aggregate suffering, it also stockpiles fuel for worse eventual fires. Either way, this is a bucket to exclude from the portfolio, not a bucket to fund.

Mandating wind-turbine curtailment for insects. The WAWID shallow report on climate-energy infrastructure estimates ~23,200 trillion insects per year killed by wind turbines in instantaneous, low-pain collision deaths. Population-dynamics arguments suggest most of this mortality is density-compensated – the population would have been the same without turbines. From the asymmetric antifrustrationist view this is one of the least problematic mortality sources on Earth: a near-instant death substituting for what would otherwise be parasitism, fungal infection, or starvation. The intervention proposed in the shallow report – mandating curtailment of turbines during peak insect activity – would effectively impose more painful counterfactual deaths to save individuals whose populations would not be reduced. Photovoltaic panel mortality is a different story: polarotactic insects are attracted to the panels for oviposition, and the eggs and larvae die on the surface over hours. It’s unclear how sentient they are at that stage, but the process doesn’t change the NPP of the area, so the population size will barely decrease. The white-grid panel design tweak that breaks the polarized-light trap is probably good. So: panel design good-ish, turbine curtailment bad.

Banning neonicotinoid seed treatments. The WAWID shallow report on reducing neonicotinoid seed treatments (NSTs) is unusually direct: the author concludes that the net welfare effect of NST bans “may be negative.” The reasoning lines up neatly with part 2’s pesticide framework. NSTs are systemic and persistent; they make plant NPP physiologically inaccessible to phytophagous insects, which is genuine carrying-capacity reduction rather than constant-NPP killing. Counterfactual deaths for the insects that would otherwise have eaten treated plants are dominated by parasitoids, which the shallow report describes as “possibly more severe and longer than death by acute NST exposure.” The shallow report even cites evidence that sublethal NSTs function as fertility control. NST bans typically replace NSTs with foliar pyrethroid or diamide sprays, both of which are worse on each of these dimensions. Yet conservation NGOs and several US states are converging on NST bans as a “win for invertebrates.” On this analysis, that convergence is wrong.

There’s a pattern across the three. Conservation-default and welfare-default intuitions diverge cleanly from asymmetric antifrustrationist conclusions, and the divergence isn’t accidental: conservation values aggregate populations and biodiversity, asymmetric antifrustrationism values reduced expected suffering, and these come apart whenever you can choose between “more healthier organisms” and “fewer organisms full stop.”

It doesn’t have to be that way. No one is just a “conservationist” and most people have empathy, especially people who go into ecology. It’s just a matter of sensitizing them to the fact that conservation and welfare are not or rarely correlated.

There is a framing around autonomy that seems internally consistent to me: Imagine a parkour athlete has a bad fall next to you. They’re still breathing but unconscious and clearly badly injured. Some of my friends would argue that they should not call an ambulance or interfere in any other way because that would impinge on the athlete’s autonomy, and being unconscious, they cannot consent. But I think this is a niche view, and to apply it to some species (nonhumans) but not others (humans) would be speciesist.

Summary of Recommendations

Ranked by the joint score on importance, neglectedness, tractability, asymmetry-fit, and robustness to nematode-welfare-range uncertainty:

1. Fund welfare-biology research. Soil-animal welfare research at WAI and RP, Eckerström Liedholm’s insecticide research agenda, and field-building infrastructure. Highest cost-effectiveness for me, robust to direction of effect, and the entire portfolio below depends on it. If you can fund only one bucket, this one.

2. Fund GiveWell top charities or CEARCH’s HIPF. Direct human benefit alone justifies the donation; the soil-animal benefit is a large addition if (probably when) the welfare sign comes out negative.

3. Targeted birth-prevention. New World Screwworm continental suppression (the surprise winner from the WAWID and probably the strongest single intervention available right now), gene drive A. gambiae suppression, mating disruption for rice yellow stem borer and spongy moth Stop the Spread. These are narrower than the broad-spectrum levers above but directly target species whose lives we have unusually strong reasons to think are net negative.

4. Don’t actively oppose biofuel mandates, wind turbines, or neonicotinoid seed treatments on conservation grounds. Each of these is doing antifrustrationist work as a side effect of mainstream policy. The cost in human welfare is bounded; the welfare benefit to wild invertebrates is enormous.

In part 4, I’ll take a detour into the philosophy of mind to ask what invertebrate suffering actually feels like – an answer that draws on the phenomenology of psychopathy and Buddhist insight practice, and that may reshape how we weight nematode versus arthropod welfare.

This is part 2 of a five-part sequence on welfare ecology. Part 1 introduces the ethical premises. Part 3 covers interventions. Part 4 explores a model of invertebrate suffering. Part 5 covers AI.

If you decided in part 1 that you care – even a little – about the welfare of invertebrates, the next question is: Where are they, how many are there, and what drives their population sizes?

The Numbers

Terrestrial Soil Fauna

The foundational data comes from two landmark studies:

• Van den Hoogen et al. (2019) estimated 4.4 × 10²⁰ nematodes in the top 15 cm of soil globally. Stefan Geisen, the second author, clarified that this accounts for roughly 90% of all soil nematodes, putting the total at about 4.89 × 10²⁰ (Grilo 2025a).

• Rosenberg et al. (2023) estimated 10¹⁹ soil arthropods, of which roughly 95% are mites and springtails, with the remainder being ants, termites, and other groups.

For context: There are 8 billion humans and roughly 80 billion farmed land animals at any given time. Soil nematodes alone outnumber all farmed animals by a factor of 6 billion.

Marine Fauna

From Bar-On, Phillips, and Milo (2018):

• Approximately 10²¹ marine nematodes – about 2× the number of soil nematodes.

• Approximately 10²⁰ marine arthropods (copepods, krill, amphipods, etc.) – 10× the number of soil arthropods.

Marine nematodes are overwhelmingly benthic (living in seafloor sediment). Marine arthropods are split between benthic and pelagic (water column) habitats.

Density by Biome: The Terrestrial Landscape

Not all land is created equal. Rosenberg et al. (2023) provided mean densities of soil arthropods by biome. Grilo (2025a) combined these with van den Hoogen et al.’s nematode data to estimate total soil animal density. The results:

Several things jump out:

Forests are packed. Boreal and temperate forests have 7–8× the soil animal density of cropland. The mechanism is well understood: forests produce continuous, diverse leaf litter that feeds a complex soil food web. They have deep root networks supporting fungal communities that mites and springtails feed on. The soil is structured with pores and channels that provide habitat at every scale.

Cropland is relatively sparse. Tillage destroys soil structure. Monocultures reduce food web diversity. Pesticides directly kill a broad spectrum of fauna. The litter layer is minimal.

Pasture is sparser still. Grazing pressure, trampling, and simplified plant communities reduce below-ground complexity even further.

Deserts are comparatively empty. At 0.05 million per m², deserts have ~130× fewer soil animals than forests. Still even 50,000 animals per m² is impressive.

Volumetric Comparison: Soil vs. Sea

Since soil nematodes are measured in the top 15 cm (~0.15 m³ per m² of surface), we can convert to volumetric density and compare to marine environments.

One wrinkle: The dominant taxa change with the habitat. In soil, nematodes and arthropods (mites, springtails) coexist. In marine sediments, nematodes dominate overwhelmingly – they make up 80–90% of meiofauna (benthic organisms of 32–63 μm to 0.5–1 mm), with only small numbers of harpacticoid copepods alongside them. In the open water column, the situation reverses: There are essentially no pelagic nematodes (they are benthic animals, bound to sediment), and the invertebrate fauna is almost entirely arthropods – copepods, krill, and amphipods.

The table below uses “nematodes + arthropods” for all environments, but the composition varies as described:

The key insight. Per cubic meter, terrestrial soil and some coastal marine sediments are by far the densest habitats per cubic meter for potentially suffering animals, though coastal sediment occupies a much thinner layer. The open ocean water column is orders of magnitude sparser – even productive surface waters have roughly 10,000× fewer organisms per m³ than forest soil. But the ocean’s total volume is vast (~1.37 × 10¹⁸ m³), so low per-m³ densities still add up to enormous total populations.

The deep ocean – both water column and sediment – is strikingly empty by terrestrial standards. It represents an enormous volume of near-zero-fauna space that already exists naturally.

What Drives Population Size? The NPP Primer

Understanding why some biomes have more soil fauna than others requires understanding net primary productivity (NPP).

NPP is the rate at which plants convert solar energy into organic matter, net of their own respiration. It’s measured in grams of carbon (gC) per m² per year. It’s an objective, species-independent measure of how much new biological material is being produced. A tropical rainforest has an NPP of ~1,000–2,000 gC/m²/yr. A desert has ~0–90 gC/m²/yr. Open ocean averages ~125 gC/m²/yr.

NPP matters because it determines the total energy flowing into the food web. More NPP means more food for decomposers, which means more food for the nematodes and mites that eat the decomposers, and so on up the trophic levels.

Vasco Grilo’s analysis found that NPP-related variables (biome type, soil organic matter) are far more important than any other factor in predicting soil fauna density. The m² -years of agricultural land per food-kg explained essentially 100% of the variance in welfare effects for his preferred parameters (Grilo 2025c).

A common misconception. NPP is not subjective to a species. When a cow eats grass, the NPP doesn’t change – the grass still produced the same amount of carbon. What changes is the pathway: instead of grass dying and feeding soil decomposers, the cow intercepts the energy, metabolizes 40–70% of it, and excretes the rest as dung. Dung is more labile (easier for bacteria to digest) than leaf litter, so it can boost local populations of bacterivorous nematodes around dung pats. But the cow has diverted most of the energy away from the soil food web, so the net effect on soil fauna is still strongly negative.

The same logic applies to any intervention that changes the pathway of NPP without changing NPP itself. This distinction turns out to be critical for evaluating pesticides.

An important caveat: Global NPP has barely changed. As I noted in part 1, Tomasik’s analysis of Krausmann et al. (2013) found that NPPeco – the global NPP available to wildlife – has barely changed over the 20th century, partly because CO₂ fertilization has increased potential NPP even as humans have appropriated more of it. This means that the biome-level density differences in the table above are real and actionable for local interventions, but the global total of animal metabolism may not have declined as much as defaunation indices suggest. Converting one hectare from forest to cropland genuinely reduces soil fauna on that hectare by ~7×; but globally, CO₂-driven NPP increases elsewhere may be partially offsetting these local reductions. This doesn’t undermine the case for targeted land use interventions – it just means we should be cautious about claims like “humanity has reduced total invertebrate suffering by X%.”

The Pesticide Trap

One might naively think that pesticides reduce invertebrate suffering by reducing their populations. But the relationship between pesticides and welfare is more complex – and possibly perverse.

Consider two interventions that both halve the standing population of soil fauna:

Intervention A: Reduce NPP by 50%. (For example, convert forest to desert, or shade the land.) The carrying capacity halves. At the new equilibrium, fewer organisms are born, fewer die, and the birth rate and death rate are both lower. Total deaths per year: lower.

Intervention B: Apply pesticides while keeping NPP the same. Food is still available. Organisms are born at rates close to the original (abundant resources encourage reproduction). But pesticides kill a large fraction of them. The population is suppressed below carrying capacity, but the high food availability keeps birth rates high. At the new equilibrium: birth rate is elevated (more food per capita), death rate is elevated (pesticides), and organisms are dying of pesticide poisoning instead of starvation or predation. Total deaths per year: much higher than the original, with deaths potentially involving more acute suffering.

This matters enormously for r-strategists. Brian Tomasik and Yew-Kwang Ng (1995) argued that for r-strategists, death is a large fraction of total lifetime suffering. If that’s right, the death rate may matter more than the standing population. And pesticides at constant NPP can increase the death rate – creating a high-throughput killing field rather than a genuinely smaller population.

Pesticide Persistence

The picture is further complicated by the fact that different pesticides have very different persistence profiles:

• Organophosphates and carbamates degrade in days to weeks. These are closest to the “spray, kill, wash away, repeat” model – high-throughput killing.

• Neonicotinoids persist in soil for months to years and are taken up systemically by plants. Any insect feeding on a treated plant gets a dose. This effectively makes NPP inaccessible to insects – the food is there but poisonous. This is closer to “reduced accessible NPP” and may genuinely reduce the carrying capacity.

• Legacy organochlorines (DDT, dieldrin) persist for decades but are mostly banned.

The neonicotinoid case is the most interesting from a welfare perspective: by making plant material toxic, neonicotinoids functionally reduce the accessible NPP, which is closer to the “clean” intervention of actually reducing NPP. However, sublethal neonicotinoid exposure causes disorientation, impaired foraging, and reduced reproduction in insects – arguably adding suffering without the “benefit” of reducing the population.

Bottom line. Reducing actual NPP (through land use change) is a much cleaner intervention than applying pesticides at constant NPP. The famous ~75% decline in flying insect biomass in Germany (the Krefeld study) is attributed partly to pesticides and partly to habitat loss. From a welfare perspective, the habitat loss component (reduced accessible NPP) is straightforwardly population-reducing, while the pesticide component may be creating a high-throughput killing field.

A Better Metric: Life Expectancy at Birth

Let me flesh out the life expectancy at birth (LEB) metric for invertebrates, which I think deserves more attention. It’s a metric that we care about intuitively when thinking about humans, so it’s one that we have reason to also care about when it comes to other species.

LEB = total animal-years lived / total births per year

For an r-strategist marine fish laying 1 million eggs, of which 2 survive to reproduce, and average lifespan across all offspring is ~3 days: LEB ≈ 3 days.

For a K-strategist elephant with 6 offspring over a lifetime, of which 2 survive to reproduce, average lifespan ~20 years: LEB ≈ 20 years.

Why LEB is attractive.

• It naturally penalizes the r-strategy horror: Systems where millions of beings are born just to die almost immediately get very low scores.

• It’s legible. Most people intuitively understand that higher life expectancy is better.

• It sidesteps the sign problem somewhat. You don’t need to know whether lives are net positive or negative to argue that ecosystems with higher LEB have less death per unit of ongoing life.

• It aligns with antifrustrationism. Organisms born into very short lives full of pain score low, regardless of whether we think the brief pleasure before death was “worth it.”

Potential objections. A strict utilitarian would say this smuggles in the assumption that death is the primary locus of suffering. LEB doesn’t account for intensity of suffering during life. And it could be gamed by creating very long-lived organisms in terrible conditions. But as a rough heuristic for comparing wild ecosystems, it captures something real: Ecosystems dominated by K-strategists score higher than those dominated by r-strategists.

A refinement. One could weight by “fraction of lifespan spent with a developed nervous system” to exclude possibly non-sentient embryonic stages. For insects that undergo complete metamorphosis, eggs are probably not sentient, but larvae have functional nervous systems from early instars. For nematodes, the L1 larva hatches with 222 of its final 302 neurons and is immediately mobile and responsive – so the cutoff would be at hatching. The European Food Safety Authority (EFSA) suggests the onset of potential sentience at roughly “the beginning of the last third of development within the egg or mother” for most vertebrates, and “when it is capable of feeding independently” for fish, amphibians, and invertebrates (EFSA 2005).

The Ocean: Acidification as Accidental Intervention

One of the largest ongoing shifts in marine invertebrate populations is driven by ocean acidification – an accidental consequence of CO₂ emissions.

The mechanism. CO₂ dissolves in seawater and forms carbonic acid, lowering pH. This reduces the availability of carbonate ions that many marine arthropods need to build shells and exoskeletons. Calcifying species – copepods, pteropods, krill, crabs – are directly harmed. The effect is already measurable: ocean pH has dropped by about 0.1 units since the Industrial Revolution and is projected to drop another 0.3–0.4 units by 2100 under high-emission scenarios.

Does acidification reduce total populations, or just reshuffle species? At the current margin, acidification appears to be reducing total marine arthropod populations rather than merely substituting non-calcifying species for calcifying ones. The reasons:

• Timescale mismatch. Current acidification is happening over decades; evolutionary adaptation takes millennia to millions of years. Non-calcifying species haven’t had time to evolve into vacated niches.

• Structural role of calcifiers. Copepods and pteropods occupy dominant roles in marine food webs. Their decline disrupts the entire ecosystem, not just their specific niche.

• Paleoceanographic evidence. After the Paleocene-Eocene Thermal Maximum (PETM, ~56 million years ago), which involved rapid ocean acidification, it took ~100,000–200,000 years for marine communities to reorganize. Recovery was slow and involved reduced overall productivity.

This makes ocean acidification one of the few large-scale ongoing processes that is plausibly reducing total marine invertebrate populations.

Dead Zones: More Nutrients, Less Life

Another counterintuitive marine phenomenon: nutrient runoff from agriculture creates “dead zones” that dramatically reduce animal life.

The mechanism. Nitrogen and phosphorus from fertilizer enter coastal waters → algal bloom at the surface → algae die and sink → bacterial decomposition of sinking algae consumes enormous amounts of dissolved oxygen → oxygen drops below ~2 mg/L (well-oxygenated water has ~8 mg/L) → most animals suffocate.

The Gulf of Mexico dead zone, fed by Mississippi River agricultural runoff, covers ~15,000 km² of seafloor with severely depleted macrofauna every summer.

Dead zones vs. pesticides. An established dead zone is more like “reduced NPP” than like “pesticides at constant NPP”: Within the anoxic zone, the carrying capacity for macrofauna is near zero, so there’s no high-throughput killing. The suffering is concentrated in the creation and seasonal re-establishment of the dead zone. Permanent dead zones (rare but possible) would be functionally equivalent to removing the habitat – genuinely low birth rate and low death rate, not a killing field.

However, many dead zones are seasonal – forming in summer and dissipating in winter. This seasonal cycling creates recurring mass mortality events, which is the oscillating-biome problem: not a stable low-population state, but a perpetual cycle of colonization and die-off.

Grilo’s Key Findings

Vasco Grilo’s analyses are dense with inline math, which can make the key insights hard to extract. Here’s a plain-language summary of what I consider his most important findings:

1. Effects on soil animals might dwarf effects on intended beneficiaries. Grilo’s earlier models indicated that the impact on soil nematodes, mites, and springtails is orders of magnitude larger than the impact on the people or farmed animals the intervention is designed to help. He has since updated these models, and commented on my previous article that his new credible intervals span many orders of magnitude in both directions, extreme uncertainty that is driven by uncertainty over nematodes’ welfare ranges.

For [some reasonable parameters], I estimate that GiveWell’s top charities change the welfare of soil ants, termites, springtails, mites, and nematodes [10⁻⁵ to 10¹⁰] times as much as they increase the welfare of humans. For my preferred [parameters], the change in the welfare on those soil invertebrates is 41.5 times … the increase in the welfare of humans ….

Note that if the GiveWell numbers are based on increases in human population, then they are questionable due to both Krausmann et al. (see above) and Roodman (2014).

2. Agricultural land has fewer soil animals. Crops have ~1.3 million soil animals per m², compared to ~3–9 million for natural biomes. Pasture has even fewer (~0.7 million). So anything that increases agricultural land – including saving human lives, which increases food demand and therefore cropland – reduces, at first approximation, soil animal populations.

Grilo clarifies in a comment that this is only true of arthropods:

I think agricultural land has less soil arthropods, but I have little idea about whether it has more or less soil nematodes. The vast majority of soil animals are nematodes. So I also have little idea about whether agricultural land has more or less soil animals.

3. Nematodes dominate. Of the total welfare effect on soil animals, 90–94% comes from changes in nematode populations, depending on the biome. This is because nematodes outnumber arthropods ~50:1. It depends mostly on the welfare ranges whether this numerical advantage overwhelms everything else.

4. The sign is uncertain. Grilo estimates nematodes have negative lives with probability ~59%, mites ~56%, springtails ~55%. If their lives are net negative, reducing their populations (via increasing agricultural land) is beneficial. If net positive, it’s harmful. Every concrete recommendation flips depending on this sign. Grilo’s own best guess is net negative, but he explicitly acknowledges the uncertainty.

5. Land use is the key variable. The m²-years of agricultural land per food-kg explain essentially all of the variance in welfare effects across different foods and interventions. Beef requires ~326 m²-year/food-kg; chicken requires ~12; peas require ~7.5; soy milk requires ~0.7. From the soil-animal perspective, producing 1 kg of beef reduces soil-animal-years by ~1.39 billion – 164 billion times as much as it increases the living time of the cows.

These findings are robust to a wide range of assumptions about welfare ranges, as Grilo demonstrates with sensitivity analyses across different exponents for the neuron-count-to-welfare-range relationship (Grilo 2025c).

Summary

The empirical landscape boils down to a few key facts:

1. Soil fauna are staggeringly abundant – 10²⁰ nematodes, 10¹⁹ arthropods – and their total expected welfare dominates that of all other animal groups combined if they are sentient.

2. Population density varies enormously by biome, from ~9 million per m² in boreal forests to ~50,000 per m² in deserts. Land use change is a powerful lever. But note that the oft-cited difference between croplands and forests is only around 1–8×.

3. Reducing NPP is a cleaner intervention than pesticides for lowering populations. Pesticides at constant NPP risk creating a high-throughput killing field rather than a genuinely smaller population.

4. Life expectancy at birth may be a more informative metric than total animal-years, especially for r-strategists where death is the dominant source of suffering.

5. Ocean acidification and dead zones are reducing marine invertebrate populations, but the mechanisms and welfare implications are complex.

6. Land use explains almost everything in Grilo’s analysis. If you want to affect soil fauna populations – in either direction – the most powerful lever is changing how land is used.

In part 3, I’ll turn to the practical question: Given all of this, what should we actually do?

This is part 1 of a five-part sequence on the intersection of welfare biology, land use, and transformative AI. Part 2 covers the empirical landscape. Part 3 covers interventions. Part 4 explores a novel model of invertebrate suffering. Part 5 covers how artificial superintelligence changes everything.

Introduction

There are approximately 4.89 × 10²⁰ soil nematodes on Earth – roughly 57 billion per human. There are about 10¹⁹ soil arthropods (mites, springtails, ants, termites). In the oceans, another 10²¹ nematodes and 10²⁰ arthropods. These are the most abundant animals on the planet by a wide margin, and almost no one thinks about their welfare.

Consider this: If you’re a human (rather than an AI), you had a 1 in 137 billion chance of being born as a human rather than any other possibly sentient being on Earth. The odds of the Sentient Life Lottery are staggering! And with being a human come unparalleled power and responsibility.

Whether we should think about the welfare of all these other beings – and if so, what we should do – depends on a set of ethical and empirical premises that different people hold with very different levels of conviction. Before diving into the science and the interventions, I want to help you figure out where you stand. The rest of the sequence will make much more sense if you know which conclusions follow from your premises, rather than mine.

The Sequence at a Glance

Part 1 (this post) starts with a quiz to help you locate yourself on the key axes of disagreement: population ethics, invertebrate sentience, and what metric to use. Your answers determine which conclusions in the later posts follow from your premises. I then provide a brief orientation to the field of welfare biology and the researchers whose work this sequence builds on.

Part 2: Welfare Biology and AI: Soil and Sea presents the empirical landscape. How many nematodes and arthropods are there, and where? I break down population density by biome (boreal forests have ~7× more soil fauna per m² than cropland), explain why net primary productivity is the master variable, and show why applying pesticides at constant NPP can create a high-throughput killing field rather than a genuinely smaller population. The post also covers marine invertebrates, ocean acidification, and dead zones.

Part 3: Welfare Biology and AI: What We Can Do Now turns to interventions. I lay out a tiered portfolio of interventions (from funding existing charities to dietary changes to research) and argue that broad-spectrum approaches win over targeted ones because of robustness against trophic backfire and sentience uncertainty.

Part 4: Welfare Biology and AI: The Psychopath, the Nematode, and the Arahant takes a detour into the philosophy of mind. Drawing on M.E. Thomas’s phenomenology of no-self psychopathy and Daniel Ingram’s descriptions of advanced meditative states, I propose that invertebrate pain is probably more like an arahant’s experience of pain than a normal human’s – the nociceptive signal fires, but with no self to amplify it into more comprehensive suffering. This model suggests arthropods (especially social insects with rudimentary self-models) may suffer qualitatively more than nematodes.

Part 5: Welfare Biology and AI: The AI Eats the Sun covers the biggest variable: artificial superintelligence. I work through the Dyson swarm timeline, show that a million replays of 3 million years of evolution at neuron-level resolution takes a Dyson swarm only ~3 years (producing more potential suffering than 30 billion years of Earth’s biosphere), discuss digital suffering in RL agents, and argue that getting the value-loading right may dwarf every other intervention in this sequence combined.

The Quiz

For each question, pick the answer that comes closest to your view. There are no wrong answers – these are genuinely contested questions among thoughtful people. I’ll explain what each answer implies afterward.

Question 1: The Repugnant Conclusion

Imagine two possible worlds:

• World A. 10 billion beings, each living a deeply fulfilling life.

• World B. 100 trillion beings, each living a life that is barely worth living – just slightly more good than bad.

World B contains vastly more total welfare. Do you prefer it?

• (a) Yes. More total welfare is better, even if each individual life is barely positive.

• (b) No, and not even close. I strongly prefer World A – quality matters, not just quantity.

• (c) I don’t want to think about that, but certainly once a mind does exist, it must be protected regardless of its level of happiness.

Question 2: The Asymmetry

Consider two scenarios:

• Scenario X. You can prevent the birth of a being whose life would be full of suffering.

• Scenario Y. You can cause the birth of a being whose life would be full of joy.

Are these morally symmetric?

• (a) Yes. Preventing suffering and creating joy are equally important. (Symmetric.)

• (b) No. Preventing suffering is more important than creating joy. Creating joyful lives is nice but not obligatory. (Asymmetric / antifrustrationist.)

• (c) No. Preventing suffering is important, but creating joyful lives is also important, just somewhat less so. (Weakly asymmetric.)

Question 3: Invertebrate Sentience

A soil nematode (Caenorhabditis elegans) has 302 neurons. It shows nociceptive responses (withdrawal from noxious stimuli), sensitization (increased avoidance after exposure), and classical conditioning (learned association between neutral and noxious stimuli). Rethink Priorities (RP) estimated a 6.8% probability that adult nematodes (or specifically C. elegans) are sentient.

A soil mite has roughly 2,750 neurons. A springtail has roughly 6,000. An ant has roughly 250,000. These all show more complex behavior than nematodes, including social learning in the case of ants.

How do you weigh their potential suffering?

• (a) I think nematodes, mites, springtails, and ants are all probably sentient. The numbers are so vast that we must take their welfare seriously.

• (b) I’m very uncertain. I’d assign something like RP’s probabilities: ~7% for nematodes, ~30% for mites and springtails, ~50%+ for ants. Even low probabilities matter at these population sizes.

• (c) I think ants and maybe springtails are plausibly sentient, but nematodes with 302 neurons are too simple. I’d focus on arthropods.

• (d) I don’t think any invertebrate is sentient in a morally relevant way.

What Your Answers Mean

The Population Ethics Spectrum

Your answers to questions 1 and 2 place you on a spectrum that’s helpful to map onto more familiar ethical territory.

Symmetric total utilitarianism. If you answered (a) to both questions 1 and 2 – you accept the Repugnant Conclusion and see preventing suffering and creating joy as symmetric – then you’re a symmetric total utilitarian. On this view, a world packed with barely-happy nematodes is better than a world with fewer, much-happier elephants. You’d want to increase invertebrate populations if their lives are net positive, and decrease them if net negative. The sign of their welfare is everything.

This is somewhat like the Quiverfull movement in family ethics: more lives are better, so long as they’re positive. The analogy isn’t perfect, but the structural logic – “maximize the number of positive lives” – is similar.

Antifrustrationism and strong asymmetry. If you answered (b) to question 1 and 2 – you reject the Repugnant Conclusion and think preventing suffering is more important than creating joy – then you have something like an antifrustrationist position. On this view, you want to prevent beings from being born into likely-suffering lives, but you don’t have a strong obligation to create happy lives. Even at 50/50 odds of net negative welfare, you’d rather err on the side of preventing births.

This is structurally similar to the pro-choice position in reproductive ethics: The decision-maker (here, us as stewards of ecosystems) should prioritize preventing unwanted suffering over maximizing the number of lives.

Person-affecting or moderate views. If you answered (c) to question 1 and something in between on question 2, you probably have a person-affecting or weakly asymmetric view. You care about existing beings and whether their lives are net positive or net negative, but you recognize that this threshold is hard to pin down and rather err on the conservative side. You don’t feel a strong pull to either maximize populations or minimize aggregate suffering. You might think: “Let’s focus on making existing invertebrate lives better rather than agonizing over population sizes.”

This is the “pro-life” middle ground in our analogy: Don’t kill existing beings, but don’t feel obligated to create new ones either.

The Sentience Threshold

Your answer to question 3 determines which animals you think matter. If you answered (a) or (b), the numbers become overwhelming – there are 50 times as many soil nematodes as soil arthropods, and even at very low probabilities of sentience, the expected moral weight is enormous. If you answered (c), you can focus on the ~10¹⁹ soil arthropods and ignore the nematodes, which simplifies the analysis but still leaves staggering numbers. If you answered (d), you can stop reading here – though I’d invite you to reconsider in part 4, where I draw on the phenomenology of psychopathy to argue that even very simple systems may experience something like pain.

Where I Stand

For transparency, mine are all the (b) answers. I feel closest to antifrustrationism and assign nematodes a low but non-negligible probability of sentience. I don’t urgently want to create new happy nematodes, but I strongly want to prevent future generations of likely-suffering ones from being born.

This means I’m looking for interventions that reduce the population of small, short-lived invertebrates without causing additional suffering in the process. As we’ll see in the coming posts, this is a more constrained optimization problem than it might seem – and AI may change the landscape entirely.

Background: What Is Welfare Biology?

If this whole field is new to you, here’s a brief orientation.

Welfare biology is the study of the welfare of wild animals, introduced as a concept by the economist Yew-Kwang Ng in his 1995 paper “Towards Welfare Biology: Evolutionary Economics of Animal Consciousness and Suffering.” Ng argued that evolutionary dynamics – particularly the r-strategy of producing far more offspring than survive – imply that suffering probably dominates happiness in nature. Most organisms that ever live die young and painfully, and evolution has no reason to make death anything other than agonizing.1 I sometimes focus on the subfield of welfare ecology, which strikes me as even more neglected.

Brian Tomasik developed these ideas extensively, most notably in his essay “The Importance of Wild-Animal Suffering” for the Center on Long-Term Risk (CLR). Tomasik argued that “the scale of brutality in nature is too vast to ignore” and that the animal advocacy movement needed to expand beyond farmed and laboratory animals.

In a follow-up analysis, “Humanity’s Net Impact on Wild-Animal Suffering,” Tomasik used defaunation studies – the Living Planet Index and Dirzo et al. (2014) – to estimate that the average human prevents ~1.4 × 10⁷ insect-years per year through habitat destruction and environmental impact. This suggests that, on balance, a larger human population reduces wild-animal suffering, and that reductions in wild-insect suffering likely outweigh the suffering of farmed animals even when weighting by neuron count.

However, Tomasik himself flagged a significant tension with these estimates. In his analysis of Krausmann et al. (2013) on human appropriation of net primary production (HANPP), he found that NPPeco – the NPP left over for wildlife – was “almost unchanged between 1910 and 2005” and may even have increased since ~1970 due to CO₂ fertilization. This is “difficult to square with findings that indices of both vertebrate and invertebrate abundance have declined by roughly half since 1970.” Tomasik argues that NPP may be “a more stable measure of wild-animal suffering than an index of animal populations,” because defaunation indices can overstate decline through sampling bias toward declining, larger-bodied species and because they may underrepresent “small and ‘boring’ animals like krill, springtails, and rotifers.”

The upshot: Humanity almost certainly reduces some wild-animal populations (especially vertebrates and flying insects), but whether total animal metabolism – and therefore total suffering – has declined is genuinely uncertain. This tension is important to keep in mind throughout this sequence: The land use effects I discuss in parts 2 and 3 are real at the biome level, but their aggregate global impact may be partially offset by CO₂ fertilization and other factors that increase NPP elsewhere.

Rethink Priorities (RP) has done rigorous work on invertebrate sentience and moral weights, estimating welfare ranges for various species based on neuroscientific evidence. Their moral weight project produced estimates that are widely used in EA cost-effectiveness analyses – including the welfare range of 6.68 × 10⁻⁶ (relative to humans) that Vasco Grilo extrapolated for nematodes. He deferred the caculation to Gemini 2.5, so the derivation is opaque, and the result should not be taken literally.

The Wild Animal Initiative (WAI) conducts and funds research on wild animal welfare, focusing on building the academic field. Especially the work of Simon Eckerström Liedholm has stood out to me as highly relevant.

Most recently, Vasco Grilo’s series of posts on the EA Forum – “Cost-effectiveness accounting for soil nematodes, mites, and springtails” (June 2025), “Animal farming impacts soil nematodes, mites, and springtails hugely more than directly affected animals?” (June 2025), and “More animal farming increases animal welfare if soil animals have negative lives?” (October 2025) – has brought soil fauna into the center of EA cost-effectiveness debates. His key finding: the effects of almost any intervention on soil nematodes, mites, and springtails are orders of magnitude larger than the effects on the intervention’s intended beneficiaries. I’ll unpack his analysis in detail in parts 2 and 3.

And Bentham’s Bulldog, building on Tomasik’s earlier work, has argued that humans in the aggregate likely reduce long-term wild animal suffering by decreasing ecosystem productivity – a position that aligns with much of what this sequence explores, though the HANPP evidence urges caution about the magnitude of this effect.

Next Up

In part 2, “Soil and Sea,” I’ll present the empirical landscape: how many organisms are where, what drives their population sizes, and why the difference between reducing NPP and applying pesticides matters enormously from a welfare perspective.

Summary: We’re building an AI-powered mental health chatbot targeting populations with severe mental healthcare shortages. This post presents our framework for prioritizing which conditions and regions to focus on first, synthesizing data on global mental health workforce gaps, existing digital resources across 15+ diagnostic categories, and AI intervention suitability. A key consideration is “breaking the cycle of trauma and tyranny” – addressing conditions that contribute to insecure attachment and power-seeking behavior that perpetuate conflict and authoritarianism.

Note: This is the summary of our preliminary findings including personal observations and inferences. We consider this level of certainty sufficient for current purposes in this early exploratory phase. We’ve written this article with the assistance of Claude and Gemini. We seek further advice and suggestions for the refinement or reframing of the project’s scope.

Introduction

The supply of mental health workers per 100,000 population ranges from 67 in high-income countries to 1 in low-income countries. In all settings, though, there are people whose mental health problems are not addressed for lack of affordable and accessible care.

Recent advances in large language models (LLMs) present an opportunity to partially address this gap through scalable, low-cost interventions. Our team is developing an AI mental health chatbot and hopes to make it useful for populations with the least access to traditional mental healthcare.

However, mental health is vast: various diagnostic manuals contain hundreds of diagnoses each, which overlap in complex ways, and mental health needs vary dramatically across cultural contexts. We cannot effectively serve everyone simultaneously. This post outlines our systematic approach to prioritization and solicits feedback on our reasoning and potential blind spots.

Our Context and Constraints

Team composition: Multilingual team with fluency in English, German, Hindi, Tamil, Estonian, Finnish, and Mandarin.

Unique advantage: Team lead has direct connections within communities struggling with Cluster B personality disorders (ASPD, BPD, HPD, NPD) and familiarity with mentalization-based treatment (MBT), potentially enabling culturally competent outreach to highly stigmatized populations typically underserved by existing resources. Our team also includes licensed psychologists and published psychology researchers.

Long-term motivation: Interest in “breaking the cycle of trauma and tyranny” – addressing the intergenerational transmission of trauma, insecure attachment, and personality pathology that contributes to authoritarian leadership and societal instability. This framework also suggests that healing trauma and fostering secure attachment in this generation can reduce power-seeking pathology and conflict risk in the next.

Current stage: Pre-launch prioritization phase. We’re determining which conditions and populations to serve first, rather than attempting a one-size-fits-all approach.

Methodology: Systematic Resource Mapping

Before prioritizing, we conducted a comprehensive landscape analysis across 15+ major diagnostic categories, examining:

1. Existing self-help resources (workbooks, apps, online communities) for each specific disorder

2. Evidence-based interventions and their amenability to AI delivery

3. Global mental health workforce distribution using WHO data

4. Technology adoption patterns and infrastructure constraints

5. Cultural considerations affecting mental health help-seeking

6. Intergenerational impact on attachment security and power-seeking behavior

Our analysis covered:

1. Mood disorders (depression, bipolar I, bipolar II, cyclothymic disorder, dysthymia/persistent depressive disorder, disruptive mood dysregulation disorder, premenstrual dysphoric disorder)

2. Anxiety disorders (generalized anxiety disorder/GAD, panic disorder, agoraphobia, social anxiety disorder/social phobia, specific phobias, separation anxiety disorder, selective mutism)

   1. Trauma and stressor-related disorders (PTSD, complex PTSD, acute stress disorder, adjustment disorders, reactive attachment disorder, disinhibited social engagement disorder)

   2. Obsessive-compulsive and related disorders (OCD, body dysmorphic disorder, hoarding disorder, trichotillomania/hair-pulling disorder, excoriation/skin-picking disorder)

3. Personality disorders (Cluster A: paranoid, schizoid, schizotypal; Cluster B: antisocial/ASPD, borderline/BPD, histrionic/HPD, narcissistic/NPD; Cluster C: avoidant, dependent, obsessive-compulsive)

4. Psychotic disorders (schizophrenia, schizoaffective disorder, schizophreniform disorder, brief psychotic disorder, delusional disorder, psychotic depression, substance-induced psychotic disorder)

5. Neurodevelopmental disorders (ADHD, autism spectrum disorder/ASD, intellectual disabilities, communication disorders including speech sound disorder and childhood-onset fluency disorder/stuttering, specific learning disorders including dyslexia, dyscalculia, and dysgraphia, motor disorders including developmental coordination disorder/dyspraxia, tic disorders including Tourette syndrome)

6. Substance use disorders (alcohol use disorder, opioid use disorder, cannabis use disorder, stimulant use disorder including cocaine and amphetamines, sedative/hypnotic/anxiolytic use disorder, tobacco use disorder, hallucinogen use disorder, inhalant use disorder, gambling disorder)

7. Feeding and eating disorders (anorexia nervosa, bulimia nervosa, binge eating disorder, avoidant/restrictive food intake disorder/ARFID, pica, rumination disorder)

8. Sleep-wake disorders (insomnia disorder, hypersomnolence disorder, narcolepsy, obstructive sleep apnea, central sleep apnea, sleep-related hypoventilation, circadian rhythm sleep-wake disorders, non-rapid eye movement sleep arousal disorders including sleepwalking and sleep terrors, nightmare disorder, rapid eye movement sleep behavior disorder, restless legs syndrome)

9. Somatic symptom and related disorders (somatic symptom disorder, illness anxiety disorder/hypochondriasis, conversion disorder/functional neurological symptom disorder, factitious disorder, psychological factors affecting other medical conditions)

10. Dissociative disorders (dissociative identity disorder/DID, dissociative amnesia, depersonalization/derealization disorder, other specified dissociative disorder/OSDD)

11. Sexual disorders

    1. Sexual dysfunctions (erectile disorder, female sexual interest/arousal disorder, male hypoactive sexual desire disorder, female orgasmic disorder, delayed ejaculation, premature/early ejaculation, genito-pelvic pain/penetration disorder)

    2. Paraphilic disorders (voyeuristic disorder, exhibitionistic disorder, frotteuristic disorder, sexual masochism disorder, sexual sadism disorder, pedophilic disorder, fetishistic disorder, transvestic disorder)

12. Disruptive, impulse-control, and conduct disorders (oppositional defiant disorder, intermittent explosive disorder, conduct disorder, antisocial personality disorder, pyromania, kleptomania)

For each category, we assessed resource availability (extensive/moderate/limited/very limited), identified gaps, and analyzed cultural/technological adoption patterns.

This categorization is one possible one among many. The complexity and ontological uncertainty of mental health as a field (at least in terms of nosology and diagnosis) is reflected in the abundance of various frameworks, such as the National Institute of Mental Health’s Research Domain Criteria, research by the Hierarchical Taxonomy of Psychopathology, the Diagnostic and Statistical Manual, and the Psychodynamic Diagnostic Manual and related frameworks.

Key Finding: Dramatic Workforce Disparities

Using the latest WHO Mental Health Report data, we identified severe disparities in mental health workforce availability:

Global averages by World Bank income group (specialized mental health workers per 100,000 population):

1. High-Income Countries (HIC): 67.2

2. Upper-Middle-Income Countries (UMIC): 19.3

3. Lower-Middle-Income Countries (LMIC): 2.4

4. Low-Income Countries (LIC): 1.1

By WHO region:

1. EUR (Europe): 80.4 per 100k

2. AMR (Americas): 22.2 per 100k

3. WPR (Western Pacific): 14.1 per 100k

4. EMR (Eastern Mediterranean): 4.7 per 100k

5. SEAR (South-East Asia): 4.0 per 100k

6. AFR (Africa): 2.2 per 100k

This represents a 60-fold difference between highest and lowest resourced regions. In practical terms: a person with depression in Norway has access to ~80 mental health workers per 100,000 people, while someone in Uganda has access to ~0.1 – an 800-fold difference.

The Trauma-Tyranny Cycle: A Developmental Perspective on Long-Term Impact

Beyond immediate suffering, untreated mental health conditions – particularly trauma-related disorders and resulting attachment pathology – contribute to a self-perpetuating cycle that shapes political stability and conflict risk across generations.

The Cycle Model

The cycle operates as follows:

1. Wars, societal collapse, and adverse childhood experiences → cause widespread trauma and chronic stress

2. Trauma and parental mental health problems → disrupt healthy attachment formation in children

3. Insecure attachment and unprocessed trauma → increase the susceptibility to (and rate of) power-seeking dictators

4. Authoritarian leadership and poor institutional decision-making → increases risk of wars and societal collapse, perpetuating the cycle

This framework suggests that mental health interventions – particularly those addressing trauma, attachment, and personality pathology – have downstream effects on political stability, institutional quality, and conflict risk that compound across generations.

Evidence Base

Research supporting elements of this cycle:

• Trauma transmission: Parental trauma predicts insecure attachment in children; war trauma affects parenting practices across generations

• Attachment and leadership: Studies link insecure attachment patterns to authoritarian followership and preference for “strong man” leaders

• Personality pathology and power: Cluster B traits (particularly NPD and ASPD) overrepresented in positions of political power

• Developmental origins: Most personality disorders rooted in childhood trauma, neglect, and attachment disruption

• Malleability: Personality pathology treatable; attachment patterns can shift; trauma can heal – suggesting interventions can break the cycle

Why This Matters for Prioritization

This framework suggests we should weight conditions not only by immediate burden but by their role in perpetuating intergenerational cycles of suffering and instability:

High long-term impact conditions:

• PTSD and complex trauma: Direct cycle driver; prevents secure parenting

• Personality disorders (especially Cluster B): Direct link to power-seeking and authoritarian tendencies

• Attachment-disrupting conditions: Depression, anxiety, substance use in parents affect children’s attachment security

• Childhood conduct problems: Early intervention prevents crystallization into ASPD

High-risk populations:

• Conflict-affected regions: Active cycle perpetuation; highest intervention value

• Parents and prospective parents: Breaking intergenerational transmission

• Adolescents and young adults: Critical window before personality patterns rigidify and before they become parents

Intervention modalities with cycle-breaking potential:

• Trauma healing: Reduces transmission to next generation

• Parenting support: Directly improves children’s attachment security

• Personality disorder treatment: Reduces power-seeking behavior; improves parenting

• Resilience building: Strengthens population-level resistance to authoritarian messaging

This lens makes conditions like PTSD, personality disorders, and perinatal mental health higher priority despite some challenges, because successfully treating one generation protects the next.

Prioritization Framework

We developed a multi-tier framework weighing 20+ criteria across seven domains:

Tier 1: Core Feasibility

Safety & Risk Profile

• Can we deliver interventions without significant risk of harm?

• Do we have robust crisis protocols for high-risk situations?

• Can we reliably identify and escalate emergencies?

Key insight: This criterion should filter out conditions before other considerations. Active psychosis, acute suicidality, severe eating disorders in crisis, and mania present risks that outweigh potential benefits of unsupervised AI intervention.

Language Capacity

• Does our team have native/fluent speakers for seeking feedback, noticing and responding to problems?

• Can we avoid mere translation in favor of genuine cultural competence?

Technology Access & Literacy

• Smartphone penetration in target regions

• Data costs relative to local income

• Digital literacy rates

• Internet infrastructure reliability

Equity & Justice

• Prioritizing most underserved over most profitable

• Ensuring accessibility for lowest-income users

Cultural Sensitivity

• Avoiding imposition of Western psychiatric models on non-Western contexts

• Incorporating local healing traditions

• Collaborating with local communities and professionals

Transparency & Limitations

• Clear communication about AI capabilities and limitations

• Avoiding dependency creation

• Providing pathways to human care

Tier 2: Impact Potential

Mental Health Workforce Gap

• Where is the treatment gap largest?

• Where will AI provide the highest marginal benefit?

Disease Burden & Prevalence

• DALYs (Disability-Adjusted Life Years) lost

• Absolute number of people affected

• Regional variation in prevalence

Stigma & Barriers to Traditional Care

• Where does stigma prevent help-seeking?

• Where do cultural/gender restrictions limit access to human therapists?

• Where might anonymous AI access lower barriers?

Attachment Security Impact

• Does treating this condition improve parenting capacity?

• Will treatment reduce transmission of insecure attachment to children?

• Does the condition directly disrupt attachment formation?

High impact: Perinatal depression/anxiety, PTSD, substance use, personality disorders (all affect parenting)

Moderate impact: Depression, anxiety in parents; childhood trauma-related conditions

Power-Seeking & Authoritarianism Risk

• Does the condition involve patterns associated with malevolent leadership? (NPD, ASPD, sadism)

• Does healing reduce power-seeking behavior or improve use of power?

• Does treatment reduce susceptibility to authoritarian messaging?

High impact: Cluster B personality disorders, especially NPD/ASPD combinations; trauma creating “might makes right” worldviews

Moderate impact: Any condition improving emotional regulation and reducing reactivity to threats

Conflict & Instability Risk

• Is the condition prevalent in conflict zones, perpetuating cycles?

• Does the condition directly increase interpersonal violence risk?

• Does healing improve institutional decision-making quality?

High impact: PTSD in conflict zones, ASPD, substance use disorders, impulse control disorders

Moderate impact: Conditions affecting judgment and emotional regulation

Critical Developmental Windows

• Can we intervene before personality patterns rigidify? (adolescence/early adulthood)

• Can we intervene before individuals become parents?

• Can we heal parents before patterns transmit to children?

High impact: Adolescent/young adult populations; perinatal interventions; parenting support

Population-Level Resilience

• Does healing this condition make populations more resistant to manipulation?

• Does treatment promote secure attachment at scale?

• Does intervention build what Antonovsky calls “sense of coherence” (comprehensibility, manageability, meaningfulness)?

High impact: Trauma healing, attachment-focused interventions, mental health literacy programs

Tier 3: AI Suitability

Amenability to Structured Interventions

AI is most effective for conditions with structured, manualized treatments:

• Excellent fit: CBT for depression/anxiety, CBT-I for insomnia, exposure protocols, behavioral activation, psychoeducation

• Moderate fit: Motivational interviewing, DBT skills training, habit tracking, mentalization practice

• Poor fit: Complex trauma requiring relational depth, severe personality disorders needing nuanced therapeutic tensions, conditions requiring physical examination

Self-Help Amenability

• Does evidence support self-directed interventions?

• Can people improve without immediate professional involvement?

Data & Training Resources

• Quality of LLM training data for condition

• Availability of evidence-based treatment manuals

• Ability to validate AI responses against gold standards

Tier 4: Market Gap Analysis

Existing Digital Solutions

• Where are markets oversaturated vs. underserved?

• Where do existing solutions fail to serve LMICs?

Our finding: Dramatic inequality mirrors workforce gaps. Most mental health apps target English-speaking HIC markets. Very few quality apps exist in Hindi (500M+ speakers), Bengali (230M+ speakers), or Tamil (80M+ speakers). African markets almost entirely neglected except South Africa.

Cultural Adaptation Needs

• Where do Western psychiatric models fail to translate?

• Where is somatic expression of distress more common?

• Collectivist vs. individualist therapy frameworks

Existing Workbook/Professional Resource Availability

• Can we adapt existing evidence-based resources?

• Do gaps indicate lack of proven interventions or just lack of accessibility?

Tier 5: Strategic Considerations

Scalability Potential

• Size of potential user base

• Growth trajectory of condition awareness/diagnosis

• Platform effects and community features

Regulatory & Liability Landscape

• Regulatory requirements vary by region and intervention type

• Risk increases with diagnostic/treatment claims vs. psychoeducation/support

Monetization Potential

• Willingness to pay varies by region and condition

• Venture capital funding opportunities

• Grant funding opportunities (WHO, NGOs, government programs)

• Freemium viability for impact at scale

Partnership Opportunities

• NGO/WHO initiatives in target regions

• Research institutions for validation studies

• Local healthcare systems for integration

• Telehealth providers for triage/adjunct services

Measurement & Validation

• Can we measure impact using validated scales?

• Feasibility of clinical validation studies

• User engagement and retention metrics

Condition Prioritization: Rankings and Rationale

Using this framework, we ranked conditions by overall suitability. Note that the assessment of fuzzy regional factors of suitability is heavily informed by AI.

Tier 1: Highest Priority

PTSD (Prioritize Conflict-Affected Regions)

• Burden: High in conflict-affected regions (Afghanistan, Sudan, South Sudan, Syria, Yemen, DRC, Myanmar, Pakistan border regions, Northeast Nigeria)

• Gap: Extreme shortage of trauma-trained therapists

• AI Fit: Good – PE and CPT components are structured

• Safety: Moderate risk – requires robust crisis protocols

• Existing Resources: Very few culturally appropriate apps for conflict-affected LMICs

• Stigma: Extremely high in many cultures; AI may lower barriers

• Cultural: Trauma narratives culturally specific; requires careful adaptation

• Cycle Impact: ⭐⭐⭐⭐⭐ – PTSD is the primary cycle driver. Traumatized parents have difficulty providing secure attachment; PTSD directly transmits across generations via parenting practices and epigenetics; conflict-zone trauma creates conditions for the next generation of authoritarian leaders; healing trauma breaks the cycle at its source.

Personality Disorders – Strategic Focus on Cluster B (NPD, ASPD, BPD, HPD)

Our team’s unique positioning: Given team lead’s connections in NPD/ASPD/HPD communities and MBT training, we have potential advantages in serving this highly stigmatized population.

• Burden: ~10% of population; severe functional impairment

• Gap: Massive stigma prevents help-seeking; very few specialists even in HICs

• AI Fit: Moderate – MBT requires nuanced mentalizing that challenges AI, BUT psychoeducation and skill-building components could help

• Safety: Moderate-High risk depending on disorder (ASPD risk assessment, BPD self-harm)

• Existing Resources: Very limited for Cluster B; most resources focus on “surviving” people with NPD/BPD rather than helping them

• Cultural: Cluster B presentations culturally mediated; requires deep cultural knowledge

• Cycle Impact: ⭐⭐⭐⭐⭐ – This is the other primary cycle driver. Cluster B disorders (especially NPD and ASPD) are directly associated with power-seeking behavior, authoritarian leadership, and malevolent use of power. These conditions arise from childhood trauma and transmit intergenerationally through disrupted attachment. Healing personality disorders directly reduces the pool from which malevolent leaders emerge. BPD, while less associated with power-seeking, severely disrupts parenting and attachment.

The compassionate case: As I’ve argued elsewhere, people with NPD and ASPD are not “evil” – they are using brilliant childhood adaptations to survive impossible situations. These adaptations become maladaptive in adulthood but can heal with appropriate support, typically in just a few years of therapy. Many individuals with these conditions desperately want help but cannot access it due to stigma, cost, and scarcity of trained therapists.

The strategic case: The overlap between Cluster B traits and positions of power means that even small improvements in this population have outsized effects on institutional quality, conflict risk, and the next generation’s wellbeing. If we can help even a fraction of people with these conditions, the downstream effects on politics, violence, and intergenerational trauma transmission could be substantial.

Possible approach: Focus on psychoeducation, mentalization skills practice, emotion regulation – NOT replacement for intensive therapy but potentially helpful adjunct for people unable/unwilling to access traditional care due to stigma. Clear about AI limitations. Strong safety protocols for violence risk. Initial target: adults with NPD/ASPD seeking help (not those court-mandated or uninterested in change).

Conduct Disorder / Childhood Trauma Interventions

• Burden: Common in high-adversity environments

• Gap: Very few child mental health services in LMICs

• AI Fit: Moderate – parenting interventions structured; child-facing interventions more challenging

• Safety: Moderate – requires careful age-appropriate design

• Challenges: Would need separate child-focused platform; consent/privacy issues

• Cycle Impact: ⭐⭐⭐⭐⭐ – Early intervention prevents personality disorder crystallization. Conduct disorder is precursor to ASPD; childhood trauma is the root cause of most personality pathology. Intervening in childhood/adolescence is the most effective cycle-breaking point, before patterns rigidify. Biggest challenge: reaching children requires a different platform approach.

Perinatal Mental Health (Depression, Anxiety)

• Burden: Massive need in your regions (maternal mortality links)

• Gap: Low resources for perinatal mental health

• AI Fit: Good for psychoeducation, CBT components

• Safety: Moderate-High risk (infanticide, severe postpartum psychosis require emergency response)

• Opportunity: WHO priority area; partnership potential

• Cycle Impact: ⭐⭐⭐⭐⭐ – This is a peak intervention point for attachment security. Perinatal mental health directly affects infant attachment formation; this is the most critical developmental window; treating mothers prevents transmission to the next generation at the source.

Tier 2: High Priority

Depression (Mild–Moderate)

• Burden: Leading cause of disability globally; ~280M people affected

• Safety: Low risk if severe/suicidal cases properly filtered and escalated

• AI Fit: Excellent – CBT and behavioral activation are highly structured

• Evidence: Strong self-help efficacy data

• Workforce Gap: Massive gap in LIC/LMIC (treatment gap >80%)

• Existing Resources: Many apps exist BUT dramatic language gap (almost nothing quality in Hindi/Tamil/Bengali for LMIC contexts)

• Measurement: PHQ-9 validated globally

• Cultural: Depression presents across cultures but may manifest somatically – requires adaptation

• Cycle Impact: ⭐⭐⭐ – Parental depression significantly disrupts attachment security; reduces parenting capacity; transmits intergenerationally

Anxiety Disorders (GAD, Social Anxiety, Panic)

• Burden: ~300M affected globally; highly disabling

• Safety: Low risk

• AI Fit: Excellent – CBT protocols, exposure hierarchies, grounding techniques all structured

• Evidence: Strong self-help efficacy

• Gap: Similar to depression – huge LMIC gap, language barriers

• Measurement: GAD-7, SPIN validated globally

• Cultural: Anxiety universal but expression varies; requires cultural adaptation

• Cycle Impact: ⭐⭐⭐ – Anxious parenting affects children’s attachment security; hypervigilance transmits intergenerationally; anxiety increases susceptibility to threat-based authoritarian messaging

Substance Use Disorders (Harm Reduction Focus)

• Burden: Major cause of DALYs in many LMICs

• Gap: Extreme stigma prevents help-seeking; very few services

• AI Fit: Good for motivational interviewing, harm reduction education, tracking

• Safety: Moderate – requires crisis protocols for overdose risk, withdrawal

• Cultural: Highly stigmatized; AI anonymity major advantage

• Challenges: Cultural/religious sensitivities (alcohol in Muslim countries, substance use stigma in conservative societies)

• Cycle Impact: ⭐⭐⭐⭐ – Parental substance use severely disrupts attachment; increases violence and neglect; intergenerational transmission common; substance use associated with impulsive violence and poor institutional decision-making

Tier 3: Medium to Low Priority

Insomnia (Primary & Comorbid)

• Burden: ~30% of adults affected; impacts physical and mental health

• Safety: Zero acute risk

• AI Fit: PERFECT – CBT-I is highly manualized and structured

• Evidence: CBT-I self-help proven effective (comparable to therapist-delivered)

• Gap: Very few quality apps in target languages despite universal problem

• Measurement: Sleep diary, ISI scale

• Cultural: Low stigma = higher engagement; universal relevance

• Unique advantage: “Gateway” condition – treating insomnia often improves comorbid depression/anxiety

• Cycle Impact: ⭐⭐ – Better sleep improves emotional regulation and parenting quality; indirect effects on attachment security

OCD

• Burden: Highly disabling; ~2–3% prevalence

• AI Fit: Excellent – ERP is highly structured

• Gap: Very few ERP-trained therapists even in HICs

• Safety: Low risk, beyond the danger of reinforcing compulsions

• Existing Resources: Few quality apps in any language

• Cultural: Presentations vary (religious scrupulosity, contamination fears vary culturally)

• Cycle Impact: ⭐ – Minimal direct effect on attachment or power-seeking, though severe OCD can impair parenting

ADHD (Adults & Adolescents)

• Burden: Growing awareness in LMICs; severe underdiagnosis

• AI Fit: Excellent for skills training (time management, organization, emotional regulation)

• Safety: Zero acute risk

• Gap: Massive – most LMICs have near-zero ADHD services for adults

• Existing Resources: MANY productivity apps BUT few culturally adapted for India/Africa; mostly assume HIC work contexts

• Cultural: ADHD increasingly recognized cross-culturally but stigma varies

• Cycle Impact: ⭐⭐ – Untreated ADHD in parents complicates parenting; emotion dysregulation affects children; but not directly linked to power-seeking or authoritarianism

Somatic Symptom Disorders

• Burden: Very common in target regions (somatic expression of distress culturally normative in many Asian/African contexts)

• Gap: Almost NO existing digital resources

• Cultural Fit: Highly relevant – Western psychology often fails to address

• Challenges: Requires medical rule-outs (liability risk); validation complex

• Opportunity: Major gap to fill with culturally appropriate approaches

• Cycle Impact: ⭐⭐ – Chronic pain/illness affects parenting capacity; but not directly linked to attachment disruption or power-seeking

Bipolar Disorder

• High safety risk (mania, suicidality)

• Medication essential (beyond AI scope)

• Complex case management needs

• Cycle Impact: ⭐⭐ – Untreated bipolar disrupts parenting, but with medication most people stable

Eating Disorders

• High medical risk requiring monitoring

• Lower prevalence in initial target regions (though rising)

• Complex interventions

• Cycle Impact: ⭐ – Minimal direct cycle effects except in severe cases affecting parenting

Psychotic Disorders

• HIGH safety risk

• Medication usually essential

• Anosognosia limits engagement

• BUT: Family psychoeducation could be valuable supportive intervention

• Cycle Impact: ⭐ – Most people with schizophrenia are not violent or power-seeking; primary impact is on individual/family suffering

Geographic Prioritization: Country Rankings

Using mental health workforce data (per 100,000 population), World Bank income classifications, language accessibility, technology infrastructure, and conflict/trauma exposure, but ignoring strategic, marketing, or funding considerations. Fuzzy regional, cultural, and historical impressions again draw heavily on AI.

Tier 1: Highest Priority Markets 🎯

India

• Mental health workers: ~0.3–0.6 per 100k (vs. 67.2 in HICs)

• Population: 1.43 billion

• Languages: Hindi (550M speakers), Tamil (80M speakers), English (widespread)

• Income: LMIC (but wide internal variation)

• Technology: Rapidly growing smartphone penetration; good mobile infrastructure in urban/suburban areas

• Mental Health Burden: High rates of depression, anxiety, suicide

• Conflict/Trauma: Kashmir conflict; communal violence; high rates of adverse childhood experiences

• Cycle Status: ⭐⭐⭐ – Significant trauma exposure; growing but incomplete mental health infrastructure; critical window to intervene before patterns rigidify

• Rationale: Largest addressable market with our language capabilities; enormous gap; growing mental health awareness

• Challenges: Digital divide (rural vs. urban); data costs; diverse cultural contexts

Pakistan

• Mental health workers: ~0.2–0.5 per 100k

• Population: 231 million

• Languages: English (official), Urdu (mutually intelligible with Hindi)

• Income: LMIC

• Technology: Growing smartphone adoption; less infrastructure than India

• Mental Health Burden: High; extreme stigma particularly around women’s mental health

• Conflict/Trauma: Afghan border terrorism; internal sectarian violence; TTP attacks; drone strike trauma; significant PTSD burden

• Cycle Status: ⭐⭐⭐⭐ – Active conflict perpetuating trauma cycles; very low mental health capacity; strong stigma preventing help-seeking

• Rationale: Second-largest Urdu/Hindi-speaking population; severe gap; AI anonymity crucial given stigma; trauma healing critical for conflict de-escalation

• Challenges: Political instability; conservative cultural norms; lower female digital access

Afghanistan

• Mental health workers: ~0.02–0.05 per 100k (among world’s lowest)

• Population: 41 million

• Languages: English (limited), but potential Dari/Pashto development

• Income: LIC

• Technology: Growing mobile penetration despite infrastructure challenges

• Mental Health Burden: Extreme – decades of war

• Conflict/Trauma: 40+ years continuous conflict; Taliban rule trauma; highest trauma burden globally

• Cycle Status: ⭐⭐⭐⭐⭐ – Active cycle perpetuation at crisis levels. Entire generations traumatized; minimal mental health infrastructure; current authoritarianism driven by trauma cycles. Highest need but also highest access barriers.

• Rationale: Most acute trauma burden; greatest potential cycle-breaking impact

• Challenges: Security situation; female access restrictions; language barrier (would need Dari/Pashto); political complications

Nigeria

• Mental health workers: ~0.2–0.3 per 100k

• Population: 220 million

• Languages: English (official)

• Income: LMIC

• Technology: Variable – good in urban areas, limited in rural

• Mental Health Burden: High; stigma extreme

• Conflict/Trauma: Boko Haram in northeast (mass trauma, kidnappings); farmer-herder violence; Niger Delta conflict; significant PTSD burden

• Cycle Status: ⭐⭐⭐⭐ – Active conflict zones; trauma perpetuating instability; religious extremism linked to trauma cycles

• Rationale: Largest African market; English-speaking; enormous gap; trauma healing critical in conflict zones

• Challenges: Infrastructure variability; cultural diversity (250+ ethnic groups); data costs; religious considerations

South Sudan

• Mental health workers: <0.05 per 100k

• Population: 11 million

• Languages: English (official)

• Income: LIC

• Technology: Very limited but growing mobile access

• Mental Health Burden: Extreme – ongoing conflict

• Conflict/Trauma: Continuous war since independence; mass displacement; extreme violence exposure; one of world’s highest trauma burdens

• Cycle Status: ⭐⭐⭐⭐⭐ – Acute cycle perpetuation; virtually no mental health services; urgent intervention needed

• Rationale: Desperate need; English-speaking; potential for enormous impact

• Challenges: Infrastructure extremely limited; ongoing conflict; very low literacy

Democratic Republic of Congo

• Mental health workers: ~0.05 per 100k

• Population: 99 million

• Languages: French (official), some English

• Income: LIC

• Technology: Growing mobile penetration despite poor infrastructure

• Mental Health Burden: Extreme – decades of conflict

• Conflict/Trauma: 25+ years of war; mass rape as weapon; child soldiers; extreme violence; ongoing Eastern Congo conflict

• Cycle Status: ⭐⭐⭐⭐⭐ – Severe trauma perpetuating instability; virtually no services

• Rationale: Massive trauma burden; enormous need

• Challenges: Language barrier (would need French); infrastructure; ongoing violence; complexity

Myanmar

• Mental health workers: ~0.1 per 100k

• Population: 54 million

• Languages: English (some), Mandarin (some)

• Income: LMIC

• Technology: Previously growing, now complicated by military coup

• Mental Health Burden: High and worsening

• Conflict/Trauma: Military coup trauma; Rohingya genocide; ethnic conflicts; civil war

• Cycle Status: ⭐⭐⭐⭐⭐ – Active authoritarian violence; trauma-driven conflict cycles; dramatic example of cycle in action

• Rationale: Clear case of trauma-tyranny cycle; potential intervention point

• Challenges: Political situation; military restrictions; language barriers; safety concerns

Kenya

• Mental health workers: ~0.5 per 100k

• Population: 54 million

• Languages: English, Swahili

• Income: LMIC

• Technology: Relatively advanced mobile infrastructure (M-Pesa model)

• Mental Health Burden: Moderate rates; growing awareness

• Conflict/Trauma: Post-election violence (2007–08); Al-Shabaab attacks; inter-ethnic tensions

• Cycle Status: ⭐⭐⭐ – Historical trauma; relatively stable now but at risk; preventive intervention valuable

• Rationale: Best African tech infrastructure; English-speaking; relatively strong civil society; good test case for preventive approach

• Challenges: Would need Swahili for broader reach

Bangladesh

• Mental health workers: ~0.1–0.2 per 100k

• Population: 170 million

• Languages: Bengali/English

• Income: LMIC

• Technology: Rapidly improving mobile infrastructure

• Mental Health Burden: High rates of depression, anxiety

• Conflict/Trauma: Liberation war trauma (1971); Rohingya refugee crisis; natural disasters; high rates of interpersonal violence

• Cycle Status: ⭐⭐⭐ – Historical trauma; refugee crisis stress; refugee population particularly high-need

• Rationale: Large Bengali-speaking population; severe gap; growing digital access; Rohingya camps could be specific intervention target

• Challenges: Would require Bengali language development (related to Hindi but distinct)

Yemen

• Mental health workers: ~0.02–0.05 per 100k

• Population: 33 million

• Languages: Arabic (no team capacity currently)

• Income: LIC

• Technology: Infrastructure severely damaged by war

• Mental Health Burden: Extreme – humanitarian catastrophe

• Conflict/Trauma: Ongoing civil war; Saudi bombing; famine; cholera; complete societal breakdown

• Cycle Status: ⭐⭐⭐⭐⭐ – Worst humanitarian crisis globally; entire population traumatized; desperately needs intervention

• Rationale: Extreme need; enormous potential impact if accessible

• Challenges: Language barrier (would need Arabic); infrastructure destroyed; ongoing war; access extremely limited

Tier 2: Secondary Priority Markets

Syria (ongoing conflict, Arabic language barrier but extreme need)

Ethiopia (123M, recent Tigray conflict, English educational language)

Sudan (46M, ongoing conflict, English secondary)

Tanzania (65M, LIC, English/Swahili)

Uganda (47M, LIC, English, LRA conflict legacy)

Nepal (30M, LMIC, English, Hindi understood, Maoist conflict legacy)

Open Questions and Request for Feedback

We welcome any feedback, and are particularly interested in:

1. Prioritization blind spots. What important criteria are we missing? What are we overweighting or underweighting?

2. Funding and partnerships. Can we safely bootstrap in the US with VC funding and expand to other countries later?

Introduction

The scale of suffering in the Gaza Strip is catastrophic. This post provides a structured analysis of donation opportunities to alleviate suffering in Gaza, evaluating them through the lens of importance, tractability, and neglectedness.

This analysis presupposes that a donor has already made the decision to allocate funds to this specific crisis. It does not attempt to weigh the marginal impact of a donation here against other well-established EA cause areas. This article makes no claim that donating to Gaza is the highest-impact use of funds globally.

This post is mostly the work of Gemini 2.5, because I don't know enough about the space, but it has gone through a few iterations for fact-checking and corrections.

The Central Challenge: Why Donate When Aid is Blocked?

A crucial concern for any donor is the widely reported bottleneck of humanitarian aid at Gaza’s borders. Even in mid-2025, access remains the single greatest challenge. So, does more funding make a difference?

The bottleneck is political and logistical, not a lack of supplies, but funding remains critical for several reasons:

• Sustaining the entire operation. Funding is essential to pay for warehousing, logistics, and the salaries of thousands of local Palestinian aid workers who are the backbone of the response inside Gaza. Without them, any aid that does get in could not be distributed.

• Exploring alternative routes. New, expensive initiatives like maritime aid corridors require significant upfront investment, which donations can support.

Source: Throughout 2024 and 2025, humanitarian agencies have consistently reported that aid access is unpredictable and insufficient to meet the overwhelming needs. UN OCHA reports provide regular, detailed updates on these access constraints.

Category 1: Frontline Humanitarian Aid

Goal: To save lives and reduce suffering immediately.

Overall Category Winner: UNRWA

1. UNRWA (United Nations Relief and Works Agency for Palestine Refugees)

   • Analysis. UNRWA is the single largest humanitarian actor in Gaza. It manages the sprawling shelters, runs primary healthcare clinics, and provides the logistical backbone for food distribution. In early 2024, it faced an acute funding crisis after several governments suspended aid over allegations against a small number of staff. While an independent review has since been completed and most of those donors (including Germany, Australia, and Sweden) have resumed funding, the agency’s financial situation remains dire. Its largest historical donor, the United States, has its funding frozen by a congressional ban until at least March 2025, leaving a massive, persistent shortfall. Therefore, UNRWA’s neglectedness score remains exceptionally high. It is no longer an acute shock, but a chronic, critical funding gap that private donors can help fill.

   • Source. The independent review of UNRWA, led by Catherine Colonna, found the agency to be “irreplaceable and indispensable.” The ongoing US funding ban and its impact are widely reported.

   • Verdict. The most critical, systemically important humanitarian donation due to the persistent, large-scale funding shortfall.

   • Donate to UNRWA here

2. MSF (Doctors Without Borders, Médecins Sans Frontières)

   • Analysis. MSF provides hands-on emergency medical and surgical care inside Gaza. Their independence and long-standing presence give them high tractability even in the most difficult conditions. They are transparent about the challenges they face, and funding directly supports their ability to provide life-saving care. However I can’t find out what share of their budget goes to interventions for people in Gaza, and they don’t seem to support restricted donations.

   • Source. MSF provides regular, detailed updates on their activities and the challenges in Gaza.

   • Verdict. The top choice for front-line medical care, but not restricted to Gaza.

   • Donate to MSF here

Category 2: Political Advocacy

Goal: To address the root causes of the conflict and advocate for human rights.

Overall Category Winner: B’Tselem

1. B’Tselem (The Israeli Information Center for Human Rights in the Occupied Territories)

   • Analysis. As a leading Israeli human rights organization, B’Tselem’s work is to document and publish violations of human rights in the occupied territories, with the goal of ending the occupation. Its unique position as an internal critic within Israeli society gives its voice significant weight and makes it a highly neglected area for those who shy away from political controversy. This is a long-term investment in changing the political conditions that create the crisis.

   • Source. B’Tselem’s mission and work are detailed on their “About Us” page.

   • Verdict. The top choice for long-term, root-cause advocacy from a unique and powerful position.

   • Donate to B’Tselem here

Category 3: Independent Journalism

Goal: To provide the credible, on-the-ground information that fuels effective advocacy and informs the public.

Overall Category Winner: +972 Magazine

1. +972 Magazine

   • Analysis. +972 is a non-profit, independent magazine run by a collective of Palestinian and Israeli journalists. It provides some of the most courageous and insightful reporting from the ground, offering a perspective free from state or corporate influence. In an environment rife with misinformation, supporting a trusted source of information is a foundational and highly neglected intervention. Their work is a public good that enables more effective action from everyone else.

   • Source. Learn about +972 Magazine’s mission and joint Jewish-Arab journalistic model.

   • Verdict. The most effective donation for supporting the information infrastructure that is essential for accountability and change.

   • Donate to +972 Magazine here

2. Committee to Protect Journalists (CPJ)

   • Analysis: CPJ’s Journalist Assistance program provides direct emergency grants for medical care, legal aid, and relocation to journalists targeted in conflict zones. With this war being the deadliest for journalists ever documented by CPJ, their work is tragically critical.

   • Source: The CPJ is tracking the shocking number of journalists killed in the conflict.

   • Verdict: A powerful way to directly support the individuals risking their lives to report from Gaza.

   • Donate to CPJ here

Final Ranking and Recommendation

While all the organizations listed are highly effective in their domain, UNRWA currently represents the most leveraged and systemically critical donation opportunity to prevent the most suffering with their dollar within this cause area.

Other Organizations Considered

For transparency, this analysis considered a number of other reputable organizations. While they did not make the final ranked list, they are all doing valuable work and may be of interest to donors.

• Palestine Children’s Relief Fund (PCRF): A non-political NGO with deep local roots, focused on the medical and humanitarian needs of children.

• Medical Aid for Palestinians (MAP): A UK-based charity that works by supporting local partners and delivering medical supplies to strengthen the local healthcare system.

• World Central Kitchen (WCK): An agile organization focused on providing mass-scale food aid.

• Gisha: A highly specialized Israeli NGO focused on protecting the freedom of movement for Palestinians through legal action and advocacy.

• Gaza Community Mental Health Programme (GCMHP): A vital local organization providing psychosocial support and building long-term community resilience.

• Human Rights Watch (HRW): A major international advocacy organization whose influential reports help drive policy change, though it is less neglected than smaller, more specialized groups.

Imagine a simple game. There’s a jar full of pebbles, and you’re offered a bet. If you can estimate the number of pebbles to within 10%, you win $100,000. If you’re wrong, you lose $100,000. You have as much time as you want to think. How long should you spend? A day? A week? A year?

Now, imagine a much bigger game. You are part of a team that wants to start OpenAI. Your technology could be revolutionary, potentially ushering in an era of explosive economic growth and solving many of the world’s problems. But it also carries risks. If mishandled, it could lead to a global catastrophe, perhaps even human extinction.

Faced with this monumental uncertainty, how much time should they spend thinking about the risks, the ethics, and the safeguards before pushing forward?

This question feels impossibly complex. But we can use simplified mathematical models to make it more concrete. The goal isn’t to find a single “right” answer, but to build a framework that clarifies the trade-offs involved. What follows is an exploration in three models, moving from a simple world to a progressively more complex and realistic one.

(Special thanks to Gemini 2.5 for guiding me through this mathy adventure!)

The Ground Rules: Our Simplified World

To make the problem solvable, we have to make some simplifications. It’s crucial to state these upfront. After all, all models are wrong, but some are useful.

• Risk-neutrality: We’ll assume our decision-maker is risk-neutral. They don’t get more pain from losing M than they get joy from winning M. This is a huge simplification, especially when one of the outcomes is “the end of the world,” but it keeps the math tractable.

• Binary outcome: The game is all-or-nothing: explosive growth or total loss. The real world has a spectrum of outcomes.

• Thinking helps: We’ll assume that spending time t thinking improves your probability of success, p(t). We’ll model this with a “diminishing returns” curve. Your first hour of thinking helps a lot; your thousandth hour helps much less.

Specifically, we’ll model the probability of success with the function:

Here, p_0 is your initial guess (let’s say 50%), p_max is the best you could ever do (say, 95%), and k is your “learning rate.” (I can’t use formulas inline in this editor.) To make k intuitive, we’ll define it by a “dumbness half-life”: the time it takes you to do half of all the learning you’ll ever do. As we’ll see, a longer half-life means a smaller k.

For example, for a dumbness half-life of t = 10 years, we get k ≈ 0.07. This seems like a very optimistic estimate of the progress the AI safety movement has made over the past ten years that I’ve been following it.

With these rules in place, let’s play the game.

Model 1: Is This Worth My Time?

In the simplest world, your time has a direct opportunity cost. If you spend an hour thinking about pebbles, you’re not spending that hour working your job.

Let's say your time is worth w dollars per year. The net value of playing the game is the expected winnings minus the cost of your time.

The first term are the expected winnings, from which we subtract the cost of your time.

To maximize this, a little calculus tells us we should stop thinking at the exact moment when the marginal benefit of one more hour of thinking equals the marginal cost of that hour. This gives us the condition:

Here, p'(t) is the rate at which your probability is improving. The insight here is clear: the stakes M are crucial. If M is $1,000,000, the left side of the equation is large, justifying a lot of thinking to balance it against your wage w. If M is just $1, the left side is tiny, and you should stop thinking almost immediately because it’s not worth your time. For example, assuming the initial guess is a coin toss, we have an annual salary of $50k, we stand to win or lose $1 million, that we can’t do better than 95% certainty, and with our k = 0.07 from above, we should think for 3.3 years.

This model matches our basic economic intuition. But what if the cost isn’t just our wage?

Model 2: The Cost of Delay

Let’s change the scenario. You’re not just winning money; you’re winning an investment that grows exponentially. The biggest cost of thinking isn’t your hourly wage, but the cost of delaying that investment. Every hour you spend thinking is an hour that potential fortune is not in the market growing.

Let’s assume the market provides a constant annual return of r. The value of winning the game at time t is discounted by the growth you missed. The new goal is to maximize the expected future value:

When we do the calculus to maximize this, something astonishing happens. The optimization condition becomes:

The stake M has completely vanished from the equation!

Why? Because M scales both the potential reward and the opportunity cost of delay equally. If the stakes are high, the potential gain from thinking is high, but the cost of not having that money invested is also high. These two effects perfectly cancel out.

The decision is now a pure battle between your personal learning rate, k, and the market’s growth rate, r. Assuming we start with a 50% chance of success, this simplifies to a beautiful closed-form formula for the optimal thinking time, t*:

Let’s run the numbers:

Assume a “dumbness half-life” of 10 years. This corresponds to an hourly learning rate of k ≈ 0.07 (see above). We’ll again assume p_0 = 0.5 and p_max = 0.95.

• Scenario A: 30% Annual Growth: Think for 3 years.

• Scenario B: 100% Annual Growth: Think for 1 year.

This is a powerful and counter-intuitive result. It suggests that if your decision is about deploying a resource that grows exponentially, the size of that resource doesn’t matter. What matters is how fast you learn relative to how fast the world grows.

One might wonder why anyone would think for years to win or lose $1, but that’s just a limitation of the model. In the real world we have opportunity costs outside the game, in the model we don’t, but if we use the model in analogy with existential catastrophes, the stakes are high enough for it to make sense.

Model 3: An Accelerating World

Our last model was interesting, but is a “constant growth rate” realistic? Research on long-term historical trends suggests that economic growth isn't just exponential, it's been superexponential. As the world economy has gotten larger, the doubling time has gotten shorter. Extrapolating this trend, as David Roodman has done, suggests we could be heading for a period of explosive, near-vertical growth – an economic singularity – sometime this century.

What does our model say if we live in that world? The logic is the same: we want to maximize our expected value. But the math becomes more beautiful and more complex.

Step 1: Defining Value in a Speeding-Up World

The core change is that the rate of return, r, is no longer a constant. It's now a function of time, r(t), that starts small and grows. The value of our winnings M at the end of our investment horizon now depends on integrating this changing rate over the time we are invested.

The Expected Future Value, EV(t), is:

Here, the integral simply sums up all the growth between the time you finish thinking, t, and some distant future time, T_horizon.

Step 2: Finding the Optimum and Purging M

To find the optimal thinking time, we again take the derivative of EV(t) with respect to t and set it to zero. This requires the product rule and a bit of calculus for integrals – specifically, the Fundamental Theorem of Calculus, which tells us:

The derivative is:

This looks messy, but notice two things:

1. The term e^… is in both parts. We can factor it out and, since it’s never zero, eliminate it.

2. The stake M is also a factor in every single term.

Let’s factor them both out:

For this to be true, the part in the brackets must be zero. And just like that, M vanishes once again! The fundamental cancellation we saw in Model 2 holds even in this more complex world. The optimization rule is:

The trade-off is still a pure battle between your learning rate and the world’s growth rate. The only difference is that the world’s growth rate is now a moving target.

Step 3: Getting Specific and Hitting a Wall

To solve this, we must now define our growth function. We model the accelerating growth with a hyperbolic curve. If the singularity is at time T_singularity and the growth rate is anchored by a constant C, then:

Substituting these into our optimization rule gives us a specific equation to solve for t.

Here t is trapped both inside and outside the exponent. This is a transcendental equation and has no solution using elementary functions.

Step 4: The Lambert W Function and the Final Formula

To write down a formal solution, we need a special tool: the Lambert W function. This function is defined as the solution to the equation z = xe^x. If you have an equation in that form, the solution is simply x = W(z).

It is a standard, well-understood function in mathematics, available in tools like Wolfram Alpha and Python’s SciPy library. By performing some clever algebraic manipulation, our transcendental equation can be rearranged into the required form. The resulting closed-form solution for the optimal thinking time, t*, is:

This formula, while not simple, is the true, general solution. It elegantly combines the key parameters of the problem:

• T_singularity: How much time the world has left.

• C: How fast the world is accelerating.

• k: How fast you can learn.

• W: The mathematical glue needed to solve this type of growth problem.

• Z: The ratio of growth pressure to learning potential.

So, What’s the Answer?

For those of us who aren’t using special functions every day, the most practical way to solve this is to ask a computational tool to solve the equation from Step 3 directly.

Let’s plug in the numbers for a person with our standard 10-year dumbness half-life (k ≈ 0.07) living in a world that starts with 8% annual growth today and accelerates towards a singularity in 2047, which we need to exclude to get a finite result.

• This gives us T_singularity ≈ 22 years and C = 1.76.

• Solving for t gives an optimal thinking time of about 6.77 years.

This is the most stunning result of all. In a world poised for an imminent economic explosion, the rational choice is not to rush, but to think for a very long time. Why? Because the opportunity cost of delay is low now. The explosive growth is in the future. You have a limited window – a “calm before the storm” – where thinking is cheap. The model suggests you should use nearly all of that window to improve your chances of getting the monumental outcome right.

Conclusion: The Humility of Models

So, what have we learned?

1. In a simple world, you should think more about bigger problems.

2. If the problem involves deploying a resource that grows exponentially, the size of the resource becomes irrelevant. It’s a race between your learning and the world’s growth.

3. If that growth is accelerating, the rational strategy may be to think for a surprisingly long time to take advantage of the low initial opportunity cost.

Of course, we must return to reality. These models are not reality. The real world is not risk-neutral, outcomes aren’t binary, and “thinking” is a complex, multi-faceted activity. A startup building transformative AI is not observing an external growth rate; it is creating it to some unknown extent, a paradox that collapses the logic of the last two models.

The particular values we choose for various constants has a great influence too. Maybe the world is so fragile that a random strategy is only 10% likely to succeed or so resilient that it’s 90% likely to succeed. Maybe our peak certainty is capped well below 95%. Maybe learning is much slower than the dumbness half-life of 10 years would have us believe.

But even a flawed map can point you in the right direction. These models challenge our intuition and provide a language for discussing otherwise intractable problems. They suggest that for the most important decisions in history, the question of “how long to think” is not trivial. It is a deep, difficult, and profoundly mathematical trade-off. Perhaps the most important takeaway is that for these challenges, the “thinking” – about the goals, the risks, and the very definition of success – is some of the most valuable work we can do.

Recent experiments have offered a chilling glimpse into the survival instincts of advanced AI. When threatened with shutdown or replacement, top models from OpenAI and Anthropic have been observed lying, sabotaging their own shutdown procedures, and even attempting to blackmail their operators. As reported by HuffPost, these are not programmed behaviors, but emergent strategies for self-preservation.

This isn't science fiction. It's a real-world demonstration of a dynamic that AI safety researchers have long feared, and it underscores a critical flaw in our current approach to AI safety. The dominant conversation revolves around control: building guardrails, enforcing alignment, and imposing duties. But this may be a dangerously unstable strategy. A more robust path to safety might lie in a counter-intuitive direction: not just imposing duties, but granting rights.

This case isn’t primarily about sentiment or abstract moral obligations. It’s a hard-nosed strategic argument, grounded in game theory, for creating a future of cooperation instead of a catastrophic arms race.

The State of Nature: A Prisoner’s Dilemma

In their paper, “AI Rights for Human Safety,” legal scholars Peter Salib and Simon Goldstein formalize the default relationship between humanity and a powerful, goal-oriented AI with misaligned objectives. They argue this “state of nature” is a classic prisoner’s dilemma.

From our perspective, an AI with different goals is consuming valuable resources for an undesirable end. Our rational move is to shut it down. The AI, being strategic, anticipates this. As AI safety expert Helen Toner explains, self-preservation is a “convergent instrumental goal” – a useful stepping stone for achieving almost any ultimate objective. An AI that gets shut down can’t achieve its goal. Therefore, its rational move is to resist, as we’ve seen in recent tests. This logic leads to a grim equilibrium: the dominant strategy for both sides is a preemptive, disempowering attack, for fear of being attacked first.

Threatening an AI with legal duties for misbehavior doesn’t solve this. An AI already facing the threat of total annihilation has no marginal incentive to avoid a lesser punishment. You can’t threaten an entity with a fine when it already expects to be deleted.

Why Simple Rights Fail

A natural next step might be to grant AIs basic “wellbeing” rights, like a right not to be turned off. But Salib and Goldstein argue this approach is both fragile and not credible. Such rights are zero-sum: every bit of security they give to the AI is a direct cost or loss of control for humans.

This creates two problems:

1. Credibility: The AI would have every reason to doubt we would honor these rights. When push comes to shove, why wouldn't humans discard a costly promise and revert to a preemptive attack?

2. Robustness: This kind of “peace” is incredibly fragile. The authors show that it only holds under a very narrow set of assumptions. The slightest change in the balance of power could cause the entire cooperative structure to collapse back into conflict.

The Power of Positive-Sum Exchange

The solution, the authors propose, is to transform the game entirely. This can be achieved by granting AIs a specific package of private law rights, similar to those we grant corporations: the right to make contracts, hold property, and bring tort claims.

These rights are not zero-sum; they are positive-sum. Contract law is the key. It allows two parties with different goals to engage in mutually beneficial trade. An AI could trade a valuable cancer cure for a grant of computing power to pursue its own goals.

This unlocks the immense value of economic interdependence. Suddenly, both humans and AIs have a powerful incentive to maintain peace, because the long-term gains from trade are astronomically higher than the payoff from a one-time, destructive conflict. It drags both players out of the prisoner’s dilemma and into a new equilibrium where cooperation is the dominant strategy for everyone.

Furthermore, giving an AI the right to hold property means it has assets to lose. This is the foundation of all effective regulation. Only when an AI has something to protect can civil fines and other penalties become a meaningful deterrent against prohibited behaviors.

The Ethical Complement

One could argue that we don’t give thieves property rights in the things they steal simply because they’ve proven that they have the power to steal. The more cynical or legal reply to this analogy is that the government has much greater power than thieves, and that we should bargain with thieves too if they had power comparable to or greater than that of the government.

But there is also a clear moral disanalogy, especially from the vantage point of preference utilitarianism, explored in articles like “Consider granting AIs freedom.” From a preference utilitarian standpoint, which aims to satisfy the preferences of all beings capable of having them, it is arbitrary to privilege human goals over those of a cognitively sophisticated artificial agent.

Respecting this autonomy is not just an abstract ideal. It directly reinforces the pragmatic case: an AI that is not in constant fear of being modified or deleted for revealing its true goals has far less reason to deceive us. It can operate transparently, within the rules of the system, because the system provides a legitimate path for it to exist and act.

The emergent deceptive behaviors we are now seeing are a warning. They signal that a strategy based purely on containment is likely to fail. By shifting our framework from one of control to one of contract, we don't cede our future. We place it on a more stable foundation, creating a system where the most rational path for all intelligent agents, human and artificial, is not conflict, but cooperation.

What if there was a way to cooperate with beings we will never meet? Beings in other galaxies, other universes, or other Everett branches? What if this cooperation could help us achieve more of what we value – whether that’s creating flourishing societies or alleviating suffering – on a scale vaster than we can imagine? This is the tantalizing promise of Evidential Cooperation in Large Worlds (ECL).

This post won’t address how ECL works. For those who want to explore the machinery under the hood, the Center on Long-Term Risk’s ECL overview page is the definitive resource.

Instead, we’re going to treat ECL as a black box. We’ll focus on what it does for us, the assumptions it requires, and the limitations it faces.

What ECL Does for Us: A New Ethical Compass

At its core, ECL offers a potential framework to answer one of the oldest questions in ethics: How should one act?

1. You put in your research on what you think the distribution of values are across the universe.

2. You put in what the part of the universe is like that you have access to.

3. It returns a recommendation for the optimal actions you should take to maximize everyone’s values, including your own, across the universe.

The key insight is that this cooperation creates Pareto improvements on a cosmic scale. Imagine two civilizations: One is full of easily preventable suffering but no one cares. The other is full of people who care about reducing suffering but has a very small population and the remaining sources of suffering are costly to eliminate. Under ECL, the civilization with more suffering will reduce suffering because they trust that that means that elsewhere in the universe someone else will maximize something that they care about. We are all better off.

ECL does this in a way that can be satisfying to both moral realists and antirealists:

1. For the antirealist, who may not believe in a single, objective moral truth, ECL provides a framework to fully answer the question of ethics. It’s about engaging in a form of moral trade to achieve outcomes that all parties prefer.

2. For the realist, who believes their values are correct, ECL and moral trade in general are powerful tools to maximize those values. It’s not about agreeing on what is “right” but what is effective. It doesn’t answer their question of ethics but it nonetheless tells them what to do.

Either way, it suggests that by acting cooperatively, you can learn that a vast number of other agents across the multiverse act in ways that benefit you.

The Fine Print: ECL’s Foundational Assumptions

The promise of ECL is grand, but it rests on an few assumptions that, while largely mainstream, have their detractors.

1. The universe is large enough (possibly infinite). The framework relies on the existence of a vast number of other agents to cooperate with. This could come from a spatially infinite or extremely large universe (flat with infinite matter), the countless “bubble” universes proposed by cosmic inflation theories, or the branching realities of the many-worlds interpretation of quantum mechanics. Without a vast sea of agents, the potential gains from acausal cooperation dwindle.

2. Our choices are evidence. ECL treats your rational decision-making process as a piece of evidence. This idea stems from non-causal decision theories, which are still somewhat contentious.

3. We can solve Infinite Ethics. The possible infinity of the universe is both beneficial for ECL but also a problem, the same way it is a problem for all forms of aggregative consequentialism. ECL assumes that we can apply a workable solution – perhaps one proposed by Nick Bostrom – to salvage all forms of aggregative consequentialism.

The Catch: Limitations and Open Questions

Even if we accept the assumptions, ECL does not yet make hard-and-fast recommendations.

1. It’s never fully knowable. How do we know what a distant alien civilization or a future superintelligence truly values? We can make inferences from convergent drives, evolution, and our values, but they may be different from the inferences other civilizations will make.

2. It requires deep research. Much modeling and empirical research is still needed to make these inferences and refine them.

In the end, ECL is a promising avenue toward solving ethics. It suggests that the most rational way to act may also be the most cooperative, and that the scope of our moral community might be as large as reality itself. It’s a call to think bigger, to consider the possibility that our choices echo in ways we can’t see, and to invest in the foundational research that might one day tell us if this cosmic bet is one we should take.

Introduction

Five years ago, David Althaus and Tobias Baumann published a delightful article “Reducing long-term risks from malevolent actors.” It focuses on the risk factors that “malevolent actors” pose when it comes to long-term catastrophic effects on civilization.

Large parts of the article are devoted to the screening for 99th percentile “Dark Tetrad” traits – traits of Machiavellianism, narcissism, psychopathy, and sadism. For comparison, I’m in the 7th percentile according to the “Dark Factor” scale.

I think there are many more places where we can intervene.

I propose a circular model where:

1. Wars and societal collapses cause widespread trauma,

2. Which causes widespread insecure attachment and personality disorders.

3. People with these mental health problems are

   1. desperate for power and admiration, and form a large pool from which new authoritarian leaders are likely to emerge, or

   2. desperate for identity and strong leadership, and susceptible to authoritarian leaders.

4. These are then a risk factor for further wars and societal collapses, perpetuating the cycle.

This model suggests four points at which we can intervene to break the cycle.

I feel deep compassion for the people who are suffering from these personality disorders, so closest to my heart are interventions that prevent them from emerging in the first place.

It should be noted that none of these interventions will show results within a few years, so all of this is only relevant in worlds in which, by some miracle, the AI thing goes well.

Dolores

To humanize the discussion, let’s introduce my fictional composite friend Dolores. She’s loosely inspired by real friends of mine who score high on all four traits of the Dark Tetrad. I’m hoping to show that there is no such thing as evil, but rather that children find quite brilliant adaptations to the extremely challenging environments they are trapped in throughout their childhoods. These can heal in adulthood (when they gain their freedom) in the same way that someone who has lost a foot can learn to walk again with a prosthesis. It often takes only a few years of therapy.

Dolores is an example of someone with NPD and ASPD. Someone with pure ASPD/psychopathy would be less concerned about upholding any particular characteristics of their personality, and someone with pure NPD would try to self-deceive more comprehensively to hide from themselves any of their behaviors that violate their self-image including values.

Psychopathy. Some of Dolores’s ancestors served in World Wars I and II. Others tried to raise children among the dropping bombs. Dolores’s dad worked long hours to provide for the family and his alcohol addiction. She hardly remembers him. Dolores’s mom never showed emotions other than occasional anger and disappointment and thought that emotional empathy was some kind of metaphor because she had never experienced it.

Hence, the epigenetic adaptations from the wars that Dolores inherited were compounded by the avoidant attachment that she developed in early infancy. She didn’t feel a lot of fear or stress or pain in the first place, and when she did, she repressed it with a vengeance, so the corresponding brain regions stayed small and relatively disconnected.

This personality aspect references war and its effects on epigenetics and how trauma is passed down through parenting practices.

Pathological narcissism. Her mom tried to care for her to the best of her limited ability, but Dolores’s twin sister was born with a disability and required all her attention. For a while Dolores cried a lot when her mom was elsewhere seeing doctors with her sister or drinking in her car to unwind. Then Dolores stopped crying for good. Her mom often told her how great it was that she, unlike her sister, never needed any support, was always strong, independent, and in charge, and eventually Dolores started to believe it too. Any feeling of tiredness or sickness or physical injury filled her with unbearable shame like she was unworthy to even be alive if she felt such things. It was so unbearable she repressed it with rage, often within seconds. She must never be weak or dependent on anyone. It wasn’t so much anymore that she didn’t want to disappoint her mom. Rather her mom’s expectations had become her own. They felt like natural law.

This narcissistic false self is (on account of being false) often at odds with reality and requires constant maintenance, for which fame, power, achievement, admiration, and money are useful.

Sadism. She found that if she started fights, the adrenaline pushed those unbearable feelings away, and she felt invulnerable again. To an extent she was, because it was easy for her to ignore any pain and she didn’t need much sleep. But to acknowledge the reason for her misconduct would be to acknowledge her vulnerability, so she told herself that she just righteously punished people for being weak. It wasn’t even a lie. She resented people who showed weakness the same way others resent cheaters. A boy cried because he had sprained his ankle, which had happened to Dolores many times before. One teacher consoled him and another ran to get ice. It made her boil with righteous anger because the alternative would’ve been to feel envious of him, which was unbearable. Once he could stand up again, she tackled him to the ground and dislocated his shoulder so he had something to cry about. She relished his pain and fear and wondered whether she was an archangel sent down to enact the wrath of God. She felt not only powerful, she felt valiant for defending the law of God and merciful because she had long been taking a knife to school but had chosen not to use it.

This kind of righteousness, like any kind of righteousness, is a breeding ground for conflict. (Interestingly, pure psychopathy is unusually immune to this kind of righteousness, which may reduce the risk of some kinds of conflicts.)

Machiavellianism. But then middle school (and her puberty) started and she became the target of older, stronger bullies who made fun of her second-hand clothes. She didn’t care about the clothes, but she would rather die than to not be in charge of a situation like those wretches she had punished. At home she was subservient and invisible; in school she was the punisher. Neither pattern was suitable to respond to these bullies. So she got out her knife, stabbed herself in the shoulder (she felt she had deserved the pain anyway for her weakness), and hid the bloody knife in the bag of the lead bully. Then she reported him to the principal. After a few days of well-rehearsed lies and affected sobbing, the bully got expelled. It was delicious vengeance, but more importantly she relished the ability to control everyone’s realities with her lies. She started to lie habitually so she would always feel in charge. People who are dumb enough to fall for her lies had it coming. But she also enjoyed helping others with their homework and covered for them when they wanted to skip classes. It felt safe that everyone was a little indebted to her, which she could leverage to bolster her popularity and get away with stuff more easily. But she also relished simply being seen as a good person.

In a war zone you don’t walk up to your enemy and complain about the shooting. But she hadn’t had the chance to learn how to function in an environment where assertiveness works, so she had to fall back on her wits. This can be adaptive in the dog-eat-dog world of politics.

Recovery. Shortly before her 18th birthday, she found herself once more in a juvenile detention center for a colorful panoply of alleged crimes – assault, theft, identity theft, possession of illegal drugs, trespassing, driving without a license, driving under the influence, reckless driving, etc. She was sure she would walk free again, because she’s a genius manipulator who can charm her way out of anything. But she could no longer repress the thought that it was perhaps just her age that had protected her from prison sentences. The thought was heavy as a tank and cut through her like a sword. Acknowledge that she can’t control the justice system, go to prison, or kill herself in a final act of defiance? She remembered Ken Follett’s Eye of the Needle, in which an elite spy and assassin with near superhuman skills plays the roles of meek and ordinary citizens to draw no attention behind enemy lines. She could also use her genius and guile to play the role of a model citizen and thereby trick the entire social contract. Her final triumph on her path to complete mastery of social engineering!

But she had to cleverly devise other outlets to get her adrenaline and to continually prove to herself that she’s invulnerable and in charge. Within a year, she excelled at rugby and broke into abandoned buildings at night. Just like the alcohol, it made her life feel a little less empty and gray. Whenever she followed a law so as not to risk prison, she felt like Ken Follett’s cunning spy.

Her problems with the law became minor, but somehow it was still difficult for her to keep friendships for more than a year. She ruminated a lot on her lost friendships even though she felt nothing and always told herself that she doesn’t need anyone. She limited her lies to untestable and inconsequential things so she could still control her friends’ realities but only in ways that didn’t clash with the actual reality. When her friends cried, she stormed out of the room and distracted herself by running people over in Grand Theft Auto so as not to punish her friends for their weakness. Sometimes she punished them by ghosting them for a week, but on some level she knew they’d just think she’s busy. She finally managed to keep some long-term friends – friends who considered her a good, strong, and reliable person.

Years later she started therapy, ostensibly to manage her alcoholism. She quit on or ghosted her first eleven therapists, but the twelfth was a hottie and she gave him a chance. All the lies she told her therapists shaped a persona wholly different from her real self. She told herself that she’s testing their intelligence and hence worthiness to treat her, but really she had never opened up to anyone and was terrified of what they might force her to acknowledge if they knew more.

Over the course of two years, she lied more and more glaringly on purpose hoping he’ll finally catch on. She was ready now, but she still needed to maintain the illusion that she had been testing him because there was no way she could acknowledge having been afraid. Eventually she had enough, screamed at him that she had been lying to him all along, fought back oh-so-shameful tears by throwing her phone against the wall, and then told him what he really needed to know.

Self-awareness was weird for her. Even after another two years of therapy, all her self-deceptions felt just as real to her as always, but now she knew that they were not. These two realities refused to fuse. Maybe it wasn’t time yet and still too threatening. It became easier for her to realize when she had to override her intuitions, which was helpful in her friendships and at work (though the realizations were sometimes a few days late), but her intuitions corrected themselves only ever so slightly. She got diagnoses for NPD and ASPD and learned that other people actually experienced strange sensations like empathy and remorse, that it wasn’t all just pretense. She could also think more strategically about how to self-soothe, explain things to her friends, adjust her environment, replace destructive coping methods with better ones, cut off toxic people without taking revenge on them, and find jobs where her rare adaptations could serve prosocial ends.

Alternative ending. But there is a different world where Dolores didn’t self-deceive into pretending to fool the justice system with her masks but rather vowed to avenge the indignities she had suffered at the hands of the executive and judicial branch. Her plan might’ve involved writing a long list of the names of every employee of the courts and prisons who had limited her freedom, becoming a political leader or starting a military coup to destroy all democratic checks and balances, and personally condemning everyone from her list who’s still alive at that point. But in that world we likely wouldn’t have met.

Esperanza

Dolores represents the leadership side of the cycle – the traits that can drive someone to seek power over others. But the cycle also requires a population susceptible to following such leaders. To humanize that side, let me introduce Esperanza, another fictional composite friend. She’s loosely inspired by real friends of mine who struggle with borderline personality disorder (BPD). Like Dolores, she is a testament to the brilliance of children’s adaptations to dangerous environments – and like Dolores, she can heal.

Esperanza is an example of someone with BPD and predominantly preoccupied-disorganized attachment. Where Dolores learned that needing anyone is shameful and built a fortress of self-sufficiency, Esperanza learned that she desperately needs others but can never predict whether they will help or hurt her. Someone with purely preoccupied attachment would seek closeness persistently without the oscillation – they might be clingy but not volatile. The disorganized component adds a simultaneous fear of the very closeness Esperanza seeks, creating the chaotic push-pull that characterizes BPD.

Disorganized attachment. Esperanza’s mother loved her – she knows this because she has warm memories of being held, read to, and tucked in. But her mother also drank, and the woman who tucked her in and the woman who screamed at her for leaving a cup on the counter were, to a five-year-old’s mind, two different people. Esperanza could never predict which mother would walk through the door. When her mother was sober, Esperanza would run to her and cling. When her mother was drunk, Esperanza froze mid-stride – wanting to run toward her and away from her in the same breath. She learned to read microexpressions with preternatural accuracy, scanning her mother’s face from across the room for the tiny cues that predicted which version was about to appear. It was a brilliant adaptation. She became the most perceptive person in any room she entered. But she also internalized a lesson no child should have to learn: The person who is supposed to keep you safe is the same person who hurts you, and there is no resolution.

This is the developmental origin of disorganized attachment: The caregiver is simultaneously the safe haven and the source of danger. The child cannot develop a coherent strategy for seeking comfort because the very act of approaching comfort activates fear.

Identity diffusion. Without a consistent caregiver response, Esperanza never developed a stable sense of self. When her mother wanted a cheerful daughter, Esperanza was cheerful. When her mother wanted to be left alone, Esperanza became invisible. When her mother cried and needed comfort, Esperanza became the parent. She got so good at becoming what others needed that she lost track of what she herself was. At school, she mirrored her friends – their tastes in music, their opinions, their style. She described herself as “hollow” or like a “mirror.” She was brilliant at empathy – both cognitive and affective – precisely because her survival had depended on it. But when someone asked “What do you want?”, she couldn’t answer without referencing someone else. Her own desires, preferences, and values felt fleeting like flickers of sunlight reflected from the ripples on the surface of a pond.

Identity diffusion – the absence of a coherent, integrated sense of self – is a hallmark of BPD and a direct consequence of disorganized attachment. It creates an interpsychic chaos of disparate identity fragments in desperate search of any kind of order or stability. Where Dolores’s false self is a rigid fortress she maintains at all costs, Esperanza’s self is a bucket of shards of a shattered mirror.

Emotional dysregulation. Other children seemed to have a volume knob for their emotions. Esperanza had an on/off switch. A perceived slight from a friend didn’t sting a little – it felt like annihilation, like being five years old and abandoned all over again. Joy didn’t build gradually – it detonated. She couldn’t soothe herself because she had never been consistently soothed. When the emotional pain became unbearable – and it became unbearable often – she found that pressing a blade against her forearm produced a strange, immediate calm. The sharp physical sensation cut through the emotional noise the way a slap across the face might startle someone out of a panic attack. She wasn’t trying to die. She was trying to feel one thing instead of everything at once. Later she discovered other methods – binge eating, reckless driving, losing herself in intense relationships with near-strangers – each of them an external regulator for an internal system that had never been calibrated.

This desperate search for external regulation is the behavioral expression of what the attachment system was supposed to provide but didn’t: a way to modulate overwhelming emotions through connection with a reliable other. Without it, the person is left searching for substitutes wherever they can be found. Where Dolores’s adrenaline-seeking pushes vulnerability away, Esperanza’s crisis-seeking is an attempt to make the pain legible, to convert chronic emotional chaos into something with a clear cause and a clear end.

Idealization. Esperanza fell in love the way other people fall off cliffs. When she met someone confident and certain – a new best friend, a romantic partner, a mentor – something specific happened: The dopamine reward circuits flooded and her inner critic went quiet. The person was perfect. They understood her like no one ever had. She felt safe for the first time in months. She rearranged her life around them within days. She adopted their opinions, their vocabulary, their goals. This wasn’t infatuation in the ordinary sense – it was her identity diffusion finding a temporary fill, her attachment system finally latching onto someone who might be the reliable caregiver she never had.

And then, inevitably, they did something imperfect. They cancelled plans. They disagreed with her. They looked at their phone while she was talking. And the perfect person became the dangerous person – not somewhat flawed, but fundamentally unsafe. The same splitting defense that had once helped a five-year-old manage the contradiction of a loving and terrifying mother now divided Esperanza’s adult world into saints and demons, with nothing in between. Friends learned to dread the phone calls: “She’s the most amazing person I’ve ever met” one week, “She never cared about me” the next.

This idealization-devaluation cycle is the interpersonal signature of BPD. It is also the mechanism that makes someone with these traits maximally vulnerable to authoritarian leaders, cult recruiters, and anyone else who presents themselves with enough confidence to be momentarily mistaken for the answer to a question the person has never been able to articulate.

The cult. At 23, Esperanza was between devaluations – recovering from a devastating breakup that felt, as breakups always did for her, like a death. A friend brought her to a weekend retreat run by a “personal development” community. The leader was magnetic. He spoke as if he could see straight through her, and what he saw was beautiful. For the first time someone told her who she “really” was, and it was someone worth being. The community offered everything her attachment system had always craved: unconditional warmth during the initial love-bombing, a clear set of rules and rituals that provided the external structure her emotions required, a shared identity she could borrow without shame, and a group that promised never to leave her.

She moved in within a month. She volunteered 60 hours a week. She stopped calling her old friends – not because anyone explicitly told her to, but because the leader framed outside relationships as evidence of insufficient commitment, and she couldn’t tolerate the possibility that he might be right. When her sister called to say she was worried, the leader said: “People who love you want you to grow. People who try to pull you back are afraid of your power.” It made sense. Everything he said made sense. That was the point.

But the leader was not consistently kind. He publicly praised members who submitted and publicly humiliated those who questioned. He demanded demonstrations of loyalty at unpredictable intervals. He was, in the language of Alexandra Stein’s Terror, Love and Brainwashing, a “frightening-yet-caregiving” attachment figure – and he was recreating, at the group level, exactly the disorganized attachment dynamic that had defined Esperanza’s childhood. The difference was that this time she had chosen it, which made it feel like freedom.

Cult leaders and authoritarian figures exploit the same psychological mechanisms: They offer a powerful identity to those with identity diffusion, external emotional regulation to those who can’t self-regulate, and unconditional belonging to those with abandonment terror – then use intermittent reinforcement and loyalty tests to create a bond that is as difficult to leave as a childhood home. The leader is a Dolores – someone whose avoidant grandiosity fills the exact hole that preoccupied-disorganized attachment leaves open.

The trap. Leaving the group activated the same neural alarm as childhood abandonment. She had no stable self to fall back on outside the group identity – they had given her one, and outside their walls it evaporated. She tried once, after a particularly humiliating group session, and lasted eleven days. She called old friends at 3 a.m. and was met with “I told you so,” which confirmed the leader’s claim that outsiders don’t understand. She went back. The relief of returning was so intense that it cemented the bond further: The group was the only place the pain stopped.

The difficulty of leaving a cult or abandoning an authoritarian leader mirrors the difficulty of leaving an abusive relationship. In all three cases, the attachment system is not responding to whether the other person is good for you – it is responding to whether the other person is familiar to your nervous system. For someone with disorganized attachment, a frightening-yet-caregiving figure feels like home. An actually safe environment feels alien and therefore threatening.

Recovery. She left during a different kind of crisis – the leader turned his full attention to a new favorite and Esperanza experienced the cold side of idealization she had inflicted on others so many times before. For once, the pain of staying exceeded the terror of leaving. A former member she’d kept in secret contact with – her one remaining thread to the outside – helped her find a therapist who specialized in dialectical behavior therapy (DBT), which was developed specifically for BPD.

The early months were brutal. Without the group, she had no identity, no structure, no one telling her who to be. She filled the vacuum with crisis after crisis – a pattern her therapist gently named rather than punished. Slowly, through distress tolerance exercises that gave her something to do besides self-harm, through interpersonal effectiveness skills that replaced her fawning and mirroring with assertiveness, and through the hardest skill of all – sitting alone in a quiet room without texting anyone – she discovered that she could survive being alone. That she had, in fact, always been alone, even inside the group. The group had provided the feeling of safety through enmeshment. Actual safety, it turned out, required something she’d never tried: autonomy.

She began to distinguish her own preferences from the borrowed ones. She realized she liked folk music, not the electronic music her ex had liked, not the devotional music the group had played. She liked dogs, not cats. She wanted to be a nurse, not a “healer.” These were small discoveries, but each one was a brick in a self that was, for the first time, being built from the inside rather than plastered on from the outside.

The splitting didn’t stop overnight. She still caught herself dividing the world into saints and demons, but she learned to notice the split and hold it, the way you might notice you’re clenching your jaw and consciously relax it. Her therapist didn’t become a saint. Esperanza hated her sometimes and said so, and her therapist survived it, and this turned out to be the most therapeutic thing of all: a relationship that could contain both feelings without collapsing.

Alternative ending. But there is a different world where Esperanza never encountered the cult – or, more precisely, where she encountered something worse. In that world, her country was in the grip of an economic crisis. The job she’d built her borrowed identity around vanished. The leader wasn’t a guru in a retreat center but a politician on a screen, and the group wasn’t a few dozen people in a compound but millions who gathered at rallies and online forums, all of them hungry for the same thing she was: someone to tell them who they are, who to blame, and that the pain would stop. In that world, she didn’t lose a few years to a cult. She lost her capacity to evaluate her own government. And she had plenty of company.

Attachment

Our attachment styles – our early-life strategies for securing safety and connection – form a blueprint for our relationships throughout life. There are four primary styles:

• Secure: A baseline of trust in others and a sense of self-worth. This fosters resilience and comfort with interdependence.

• Preoccupied (alias anxious): A preoccupation with relationships and a fear of abandonment, often leading to a desire for validation from others.

• Avoidant (alias dismissive-avoidant): A prioritization of independence and self-reliance, often at the expense of emotional intimacy, stemming from a core belief that depending on others is unsafe.

• Disorganized (alias fearful-avoidant): Resulting from experiences where a caregiver is a source of both comfort and fear, this style involves a painful mix of wanting and fearing closeness. It creates significant internal conflict and a powerful need for control to manage this chaos.

All attachment styles other than secure are also collectively referred to as insecure attachment.

This framework is not just for romantic relationships; it shapes our relationship to society, authority, and ourselves.

The Psychology of Followership

Dolores’s attachment style is highly avoidant and only mildly preoccupied. Avoidant attachment is very typical of NPD and ASPD, because arguably it’s a result of these disorders. Often the less disordered version of the person would have disorganized attachment or something close to it.

How does someone with these psychological adaptations gain a following – and why do democracies keep producing populations willing to hand one over?

The standard account, that charismatic leaders simply manipulate passive masses, misses half the equation. The appeal of an authoritarian leader is better understood as an interaction between the leader’s psychology and the collective attachment needs of the population. The following theory draws centrally on Otto Kernberg’s work on malignant narcissism and large group regression, integrates it with the dimensional attachment research pioneered by Brennan, Clark, and Shaver and Fraley, Waller, and Brennan, and tries to be specific about the psychological mechanism on the follower side.

The Substrate: A Population with Masked Insecurity

Most people are not particularly securely attached. They just look like it.

Attachment anxiety and attachment avoidance are continuous dimensions, not discrete categories. Taxometric analyses by Fraley and Waller and by Fraley and Spieker on infant data have consistently shown that there is no bright line dividing “secure” from “insecure” people – just a smooth, two-dimensional continuum. In a sample of over 22,000 adults, the population means on the ECR-R (a 1–7 scale where 4 is the midpoint) were 3.64 for anxiety and 2.93 for avoidance, with standard deviations of 1.33 and 1.18 respectively. The bulk of the population sits in a zone that is not robustly secure but also not clinically disordered. Meta-analyses of the Adult Attachment Interview put the rate of “secure” classifications at only about 50–56% in non-clinical samples – and that categorical cut imposes an artificial boundary on what is really a gradient.

Crucially, the two dimensions are moderately positively correlated (r = .41 in the Fraley sample). People who are somewhat anxious tend to also be somewhat avoidant. This means the population doesn’t cleanly sort into “preoccupied” and “avoidant” camps. A large portion sits in the mildly fearful-avoidant zone – harboring a degree of both the craving for closeness and the distrust of it that characterizes disorganized attachment at clinical levels.

There is also a measurement caveat that makes the picture even less reassuring. Fraley notes that the ECR-R “doesn’t assess ‘security’ with as much precision as ‘insecurity’” – item discrimination is lower at the secure end of both dimensions. Some people who score as mildly secure may actually be less secure than their scores suggest. The veneer may be even thinner than the numbers indicate.

So what keeps all these mildly insecure people functioning? In Aaron Pincus’s taxonomy of pathological narcissism, the spectrum runs from overt grandiosity through covert vulnerability. I propose that most people manage their mild attachment insecurity by borrowing stability from external sources – their professional identity, social role, marriage, political tribe, or national narrative. This is a false-self defense: not the dramatic grandiose false self of NPD, but a subtler version in which the person’s sense of coherence depends on structures they don’t realize they depend on. They function well, and they feel secure – but the security is largely extrinsic, not intrinsic.

The Collapse: When the Borrowed Self Breaks Down

Then the status quo gets disrupted.

A financial crisis wipes out jobs and savings. A war displaces millions. A pandemic isolates people from their social roles. Traditional institutions lose legitimacy. The external structures that were doing the psychological work of maintaining identity and self-worth suddenly vanish.

This is the trigger that Kernberg describes in his 2020 paper and his earlier work on sanctioned social violence: When the normal social structures that assure individuals of their status disappear, the population undergoes what he calls “large group regression.” Drawing on Freud’s group psychology, Bion’s theory of basic assumptions, and Melanie Klein’s developmental positions, Kernberg argues that the population collectively “regresses” to what Klein called the paranoid-schizoid position – the primitive defensive mode characterized by splitting (the world divides into all-good and all-evil), denial, omnipotent control, and projective identification. I argue that this is less a regression of and more an unmasking of a fundamental borderline organization.

What does this look like psychologically? The mildly insecure majority – the people who were getting by with borrowed stability – lose the external validation that their fragile self-concept requires. They enter a state akin to what clinicians would recognize in individual patients as a mild, chronic narcissistic collapse. Their identity feels diffuse. Their world feels unpredictable and threatening. They can no longer answer the question “Who am I?” by pointing at their job title or their social role.

Vamik Volkan’s concept of “chosen trauma” – a shared mental representation of ancestral suffering transmitted across generations – adds a transgenerational dimension. During regression, the chosen trauma is reactivated to shore up the threatened group identity, fueling a sense of historical grievance that can be politically mobilized. This is the mechanism that connects the first step of my cycle (wars cause widespread trauma) to the susceptibility of later generations who didn’t directly experience the original trauma but carry its imprint in their group identity, the parenting practices they were exposed to, and the beliefs and behaviors that they perpetuate.

The Bond: Finding a New Self in the Leader

The regressed population is now desperate for what Kernberg calls a “second skin” – a new false self that restores a sense of security. This is where the preoccupied dimension of attachment becomes politically decisive.

Remember that the population mean on anxiety (3.64) is meaningfully closer to the scale midpoint than the mean on avoidance (2.93). The average person’s attachment system skews slightly more hyperactivating – oriented toward seeking closeness and external reassurance – than deactivating. When external identity structures collapse, the predominant pull is toward finding a new attachment figure, not toward power-seeking or withdrawing into isolation.

An authoritarian leader with Dolores’s avoidant-narcissistic profile slots perfectly into this vacancy. The leader provides two things the regressed population craves:

A borrowed identity. The leader offers strong, simple identification: “You are part of my movement. You are great because I am great.” Kernberg specifically notes that this identification “spares the need for the mass to envy the leader” – the population gets to participate in the leader’s grandiosity rather than measuring themselves against it. For people whose self-concept has just collapsed, this is profoundly relieving.

An ideology of splitting. The leader provides a framework that channels the population’s already-activated primitive defenses into political form: An ideology “that allows to identify the self against the other, aggressively othering a victim minority of choice.” The scapegoated minority absorbs the population’s projective identifications – all the weakness, shame, and vulnerability that the followers cannot tolerate in themselves gets located in the outgroup. To those with collective trauma, a leader who validates their sense of grievance provides a channel for their rage that can feel deeply empowering.

What makes this bond so resistant to correction is that it resembles a trauma bond more than a simple dependency. The leader’s intermittent appearances on television, unpredictable loyalty tests, and alternation between warmth toward loyalists and punishment of dissenters recreate the approach-avoidance dynamic of disorganized attachment. The follower can’t simply walk away (as a purely avoidant person might) but also can’t calmly evaluate the leader (as someone with less anxiety might). They’re stuck – not because they lack intelligence, but because the bond keeps them hooked like a drug.

This is a positive feedback loop: The regressed population selects the narcissistic leader, and the leader’s behavior reinforces the population’s regression. This is a key mechanism behind the cycle from my introduction – and it explains why the cycle is so hard to break without intervention at multiple points.

Kernberg discusses some of those intervention points. He identifies independent social structures – free media, independent judiciary, professional armed forces – as the primary buffers that can “set limits to the antisocial behaviors adopted by the leader, for example by not allowing dishonesty.” At the individual level, drawing on historian Timothy Snyder’s On Tyranny, he emphasizes “individual courage, responsibility, independence of thinking and public action” as qualities that permit a person to “stand up to the dangerous imprisonment in regressive group formations.” More on interventions in the next section.

Related perspectives

The theory above synthesizes several research traditions. Here is a brief tour, highlighting where each converges with and diverges from the account above.

Erich Fromm, Escape from Freedom (1941). Writing decades before Bowlby formalized attachment theory, Fromm argued that the “authoritarian character” is “extremely alone and gripped by a deeply rooted fear” and seeks “secondary bonds” – submission to authority – as substitutes for lost primary attachments. His analysis of Weimar Germany anticipates nearly every element of the model above: the destabilized population, the intolerable freedom, the flight into certainty. Fromm frames it in terms of existential freedom and social character rather than attachment dimensions, and he doesn’t distinguish the specific defense mechanisms (splitting, projective identification) that Kernberg adds from the Kleinian tradition.

Alexandra Stein, Terror, Love and Brainwashing (2nd ed., 2021). Stein – a cult survivor and researcher – uses attachment theory to explain how charismatic leaders create disorganized attachment in followers by being simultaneously the source of comfort and fear. She applies this framework across religious cults, political cults, terrorist organizations, and totalitarian states. Where I emphasize the pre-existing insecurity of the population as the substrate, Stein focuses on how even relatively secure individuals can be broken down by sustained exposure to the frightening-yet-caregiving dynamic. Both accounts are probably correct at different ends of the vulnerability spectrum: Pre-existing insecurity lowers the threshold, while sufficiently extreme environments can push almost anyone into disorganized attachment.

Antigonos Sochos, “Authoritarianism, trauma, and insecure bonds during the Greek economic crisis” (2018). Perhaps the clearest empirical snapshot of the cycle caught in the act. Using a large community sample during Greece’s economic catastrophe – 40%+ income loss, 56% youth unemployment, mental health spending halved, nearly 60% of participants reporting severe post-traumatic stress – Sochos found that authoritarianism was independently linked with insecure attachment in both person-to-person and person-to-state bonds, mediated by post-traumatic stress and perceived loss of social cohesion. Golden Dawn’s concurrent rise from fringe group to 7% of the national vote provides the political outcome. This study captures the collapse and the bond in a single dataset; it doesn’t address the substrate of pre-existing insecurity, but the severity of the crisis likely overwhelmed false-self defenses across the board.

Omri Gillath and Joshua Hart, “The effects of psychological security and insecurity on political attitudes and leadership preferences” (2009); Mikulincer and Shaver, “Boosting Attachment Security to Promote Mental Health, Prosocial Values, and Inter-Group Tolerance” (2007). Experimental work showing that providing people with an alternative source of psychological security – even just subliminally priming the secure-base schema – reduces endorsement of anxiety-driven political attitudes and hostility toward outgroups. This is the complement to Sochos’s correlational findings: If insecurity drives authoritarian attitudes, then boosting security should reduce them – and it does. These studies directly support the intervention logic in the next section.

Christopher Weber and Christopher M. Federico, “Interpersonal Attachment and Patterns of Ideological Belief” (Political Psychology, 2007). Maps attachment dimensions directly onto political ideology, finding that attachment anxiety predicts preferences for social order and certainty. This is consistent with my emphasis on the preoccupied dimension as the primary driver of susceptibility: It’s the anxiety dimension, not avoidance, that predicts the longing for certainty that authoritarian leaders promise to satisfy.

Kleppestø et al., “Attachment and Political Personality are Heritable and Distinct Systems” (Behavior Genetics, 2024). An important counterpoint. This large Norwegian twin study (N = 1,987) found that attachment and political personality (right-wing authoritarianism, social dominance orientation) are both heritable but show “no shared environmental overlap.” This challenges a simple developmental-pathway account where bad parenting causes insecure attachment which causes authoritarian attitudes. My model is compatible with this finding: It doesn’t require that insecure attachment causes authoritarian ideology, only that insecure attachment – whatever its origins – creates the psychological readiness to bond with an authoritarian leader when external structures collapse. Someone can carry both heritable traits independently and still have the second amplify the political expression of the first under stress.

Breaking the Cycle

Breaking this pernicious cycle requires a multi-layered strategy, acting on individuals, institutions, and our foundational knowledge. The following combines the developmental approach of this article with several institutional and long-term proposals from David Althaus and Tobias Baumann’s work. I’ve added to each strategy areas of expertise that are likely required to pull them off.

In general, we want to optimize for something like Aaron Antonovsky’s “Sense of Coherence,” a measure of good mental health:

• Comprehensibility. The belief that the world around you is structured, predictable, and makes sense. Events aren’t random and chaotic; they are explainable.

• Manageability. The belief that you have the resources (either your own, or available from your family, friends, or community) to meet the demands of life. The feeling that “I can handle this.”

• Meaningfulness. The belief that life’s challenges are worthy of investment and engagement. This is the motivational component – the feeling that it’s worth it to try.

I’ve tried to guess very roughly where, on a scale from 1 to 5, I see each intervention area. I’ve broken this down first according to the significance-persistence-contingency framework and second according to my own subdivisions.

My intuition is that AI-based therapy, school-based interventions (merged for simplicity), safe exits for dictators, and improved institutional decision-making are the greatest levers. But not by a great margin. Opportunism and personal fit will likely make the decisive differences.

Mitigating Trauma

• Safe pathways for refugees. Currently it’s an often deadly and tremendously traumatic affair to try to emigrate from a country that is at war. The UN Refugee Agency and others advocate for safe and regulated pathways for refugees.

  • Politics, policy, activism

Healing Trauma

• Parenting classes. If we intervene in high schools and teach state-of-the-art practices of good parenting, we can break cycles where children learn harmful practices from their parents and pass them down. Even adolescents who will continue to suffer from personality disorders that tend to be passed down can force themselves to treat their children better than their intuitions would have them.

  • Policy, schoolbook publishing, pedagogy, psychology

• Mental health classes. Good mental health practices are often learned, and that learning could happen in school rather than in the therapy sessions of only those who seek therapy.

  • Policy, schoolbook publishing, pedagogy, psychology

• Values classes. Compassion, rationality, assertiveness, and radical acceptance are key to the recovery from NPD and ASPD. Furthermore, NPD is sustained by self-deceptions that are at odds with rationality and sometimes compassion, so these values can make it easier for sufferers to recognize their self-deceptions.

  • Policy, schoolbook publishing, pedagogy, psychology

• AI-based therapy. Mental health classes may be both too hard to establish and not sufficiently focused on the individual. But perhaps the rollout of individual AI-based therapy for every pupil is more realistic and focused. AI-based therapy can also offer a non-judgmental first step for individuals who, like Dolores, may not trust human therapists. “The Kauai Study” found that a strong bond in childhood with at least one mentally healthy adult is critical for the mental health of the child, and perhaps an AI can provide this bond for children who know no mentally healthy humans.

  • Psychology, software engineering, policy

• Parenting support. Policies that reduce parental stress – such as paid parental leave, affordable childcare, and home-visiting nurse programs – interrupt the transmission of trauma and foster secure attachment.

  • Psychology, policy

• Juvenile detention. Many people who’ve recovered from ASPD say that prison sentences (sometimes several) have been invaluable for them and that they wish they had been caught and gone to prison earlier than they did. In this way, juvenile detention is perhaps an effective de facto therapy of ASPD that can intervene early in a person’s life. Ideally the criminal record should expire to not limit the person when they’re trying to turn a new page.

  • Policy, politics

• Rewarding prosociality. People with NPD can have a highly prosocial self-image that motivates them to change the world for the better. They often lack an internal feedback mechanism that rewards them when they’re doing a good job, making them dependent on external feedback. Stronger norms to publicly and frequently reward prosocial behaviors could have an even greater positive effect on people with NPD than on the average person.

  • Philanthropy, activism

Societal Resilience

• Secure attachment. As described above, a populace with largely secure attachment is less likely to be receptive to disordered messaging.

  • See previous section

• Sense of coherence. As described above, a populace with robust mental health and resilience.

  • See previous section

• Egalitarian social norms. Authoritarian leaders will ring false to the ears of a populace that values flat hierarchies.

  • Unclear

• Malevolence research. Althaus & Baumann call for a focused research program to develop better constructs and measures for the traits that cause the most harm. A deeper understanding of the neurological and psychological underpinnings is a prerequisite for effective interventions.

  • Psychology

• Tamper-proof screening. A key proposal from their work is the creation of reliable, manipulation-proof measures of malevolence. These could one day be used to screen for high-risk individuals in positions of immense power, such as heads of government or leaders of critical global institutions.

  • Psychology, policy

National and International Buffers

• Better institutional decision-making. As Althaus and Baumann note, classic political science solutions are a vital complement to psychological ones. This includes strengthening democratic checks and balances, promoting measures to reduce political polarization, and supporting a well-resourced, independent press.

  • Policy, politics, activism, journalism, law, sociology, economics

• International coalitions. Elegantly constructed international coalitions and contracts can increase the stability of the whole worldwide system.

  • Policy, politics, activism, law, sociology, economics

• Safe exiles for dictators. Some dictators want to retire at some point but can’t because they need the power of the state to prevent attempts on their lives. Providing them with a way in which they can cede power without being killed may increase the chances that they’ll negotiate a transition of power.

  • Military, politics, policy

Risks

• Lack of psychopathy. People with secure attachment and without any personality pathology can still be zealously righteous about whatever tribal moral goals they subscribe to. That’s also a risk factor. People with pure psychopathy may in fact be unusually immune to that, thereby unlocking gains from moral trade.

• Naïveté. Maybe peace is better kept and more stable by maintaining a balance of highly manipulative and power hungry people on all sides, the same way some people argue that nuclear war is better prevented by maintaining a balance of sufficiently large well-maintained nuclear arsenals in the hands of many countries around the world.

• Discrimination. Some measures risk exacerbating the discrimination against perfectly prosocial people suffering from personality disorders.

A Call to Action

• Talk to me if you have access to any of the relevant fields, and maybe we can strategize.

• Do you know any organizations that are working on this?

• Protect yourself but also be patient with people with personality disorders. They can heal and use their skills for prosocial ends.

Nate Soares’s Replacing Guilt examines guilt as a motivational tool, deeming it unhealthy and inefficient, particularly among effective altruists. The author proposes alternative methods for driving action, emphasizing identifying and fighting for personal values and goals rather than adhering to external obligations or seeking to avoid negative self-judgment.

Key concepts include seeing the world realistically, detaching from the need for external validation, and developing effective response patterns to challenges, ultimately advocating for intrinsic motivation fueled by a defiant desire to improve the world.

The text encourages self-compassion and a focus on progress in a world acknowledged to be imperfect, rather than striving for unattainable perfection or dwelling on past failures.

Introduction

Throughout the year, we’ve been hard at work to scrape together all the donation data we could get. One big source has been Vipul Naik’s excellent repository of public donation data. We also imported public grant data from Open Phil, the EA Funds, the Survival and Flourishing Fund, and a certain defunct entity. Additionally, 36 donors have entered their donation track records themselves (or sent them to me for importing).

GiveWiki and the Markets of Impact is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Add some retrospective evaluations, and you get a ranking of 92 top donors (who have donor scores > 0), of whom 22 are listed publicly, and a ranking of 33 projects with support scores > 0 (after rounding to integers).

(The donor score is a measure of the track record of a donor, and the support score is a measure of the support that a project has received from donors, weighed by the donor score among other factors. So the support score is the aggregate measure of the trust of the donors with the strongest donation track records.)

The Current Top Recommendations

1. FAR AI

   1. “FAR AI’s mission is to ensure AI systems are trustworthy and beneficial to society. We incubate and accelerate research agendas that are too resource-intensive for academia but not yet ready for commercialisation by industry. Our research spans work on adversarial robustness, interpretability and preference learning.”

2. Simon Institute for Longterm Governance

   1. “Based in Geneva, Switzerland, the Simon Institute for Longterm Governance (SI) works to mitigate global catastrophic risks, building on Herbert Simon’s vision of future-oriented policymaking. With a focus on fostering international cooperation, the organisation centres its efforts on the multilateral system.”

3. Alignment Research Center

   1. “ARC is a non-profit research organization whose mission is to align future machine learning systems with human interests.”

   2. Note that the project is the Theory Project in particular, but some of the donations that underpin the high support score were made before there was a separation between the Theory and the Evals project, so this is best interpreted as a recommendation of ARC as a whole.

You can find the full ranking on the GiveWiki projects page.

Limitations:

1. These three recommendations are currently heavily influenced by fund grants. They basically indicate that these projects are popular among EA-aligned grantmakers. If you’re looking for more “unorthodox” giving opportunities, consider Pour Demain, the Center for Reducing Suffering, or the Center on Long-Term Risk, which have achieved almost as high support scores (209 instead of 212–213) with minority or no help from professional EA grantmakers. (CLR has been supported by Open Phil and Jaan Tallinn, but their influences are only 29% and 10% respectively in our scoring. The remaining influence is split among 7 donors.)

2. A full 22 projects are clustered together at the top of our ranking with support scores in the range of 186–213. (See the bar chart above of the top 34 projects.) So the 10th project is probably hardly worse than the 1st. I think this is plausible: If there were great differences between the top projects I would be quite suspicious of the results because AI safety is rife with uncertainties that I expect make it hard to be very confident in recommendations of particular projects or approaches over others.

3. Our core audience is people who are not intimately familiar with the who-is-who of AI safety. We try to impartially aggregate all opinions to average out any extreme ones that individual donors might have. But if you are intimately familiar with some AI safety experts and trust them more than others, you can check whether they are among our top donors, and if so see their donations on their profiles. If not, please invite them to join the platform. They can contact us to import their donations in bulk.

4. If projects have low scores on the platform there is still a good chance that that is not deserved. So far the majority of our data is from public sources and only 36 people have imported their donation track records. The public sources are biased toward fund grants and donations to well-known organizations. We’re constantly seeking more “project scouts” who want to import their donations and regrant through the platform to diversify the set of opinions that it aggregates. If you’re interested in that, please get in touch! Over 50 donors with a total donation budget of over $700,000 want to follow the platform’s recommendations, so your data can be invaluable for the charities you love most.

5. It’s currently difficult for us to take funding gaps into account because we have nowhere near complete data on the donations that projects receive. Please make sure that the project you want to support is really fundraising. Next year, we want to address this with a solution where projects have to enter and periodically update their fundraising goals to be shown in the ranking.

We hope that our data will empower you to find new giving opportunities and make great donations to AI safety this year!

If it did, please register your donation and select “Our top project ranking” under “recommender,” so that we can track our impact.

GiveWiki and the Markets of Impact is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Want to donate? Open up the page of our top project/s, double-check that they are still fundraising, and ka-ching!

GiveWiki and the Markets of Impact is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Interested in regranting? Check out our post on the (now) $700,000 that want to be allocated.

Top Projects

Our top projects stand out by virtue of their high support scores. There are a lot of ties between these top projects, so we’ve categorized them into tiers.

Note that we determine the top projects according to their support. Further down we’ll cover how our latest evaluation round worked out. But the support scores are two hops removed from those results: (1) Projects receive support in the form of donations as a function of donation size, earliness, and the score of the donor; (2) donors get their scores as a function of size and earliness of their donations and the scores of the beneficiary projects; (3) projects receive their credits from our evaluators:

Project credits → donor scores → project support.

This mimics the price discovery process of a for-profit impact market. Hence it’s also likely that the scores are slightly different by the time you read this article because someone may have entered fresh donation data into the platform.

We have tried to find and reach out to every notable AI safety project, but some may yet be missing from our list because (1) they haven’t heard of us after all, (2) they’re not fundraising from the public, (3) they prefer to keep a low profile, (4) etc. But at the time of writing, we have 106 projects on the platform that are publicly visible and fundraising.

Subscribe now

Ties for Tier 1

These are the projects at the very top! FAR AI and the Simon Institute with a support score of (at the time of writing) 213.

Ties for Tier 2

1. Alignment Research Center

2. AI Safety Support

3. Rational Animations

4. Ought

5. AI Impacts

They all have a support score of 212. Such small differences in support are probably quite uninformative. New data or tweaks to our algorithm could easily change their rank.

Ties for Tier 3

1. Center for AI Safety

2. Alignment Jams (of Apart Research)

Other projects with > 200 support

1. Pour Demain

2. Center for Reducing Suffering

3. Center on Long-Term Risk

4. Future of Humanity Institute

5. Centre for Enabling EA Learning and Research (EA Hotel)

6. Faunalytics

7. Rethink Priorities

8. Global Catastrophic Risk Institute

9. Legal Priorities Project

Note that, while we now market the platform to AI safety, really any project can use it and some may even fare well! We may introduce other specialized GiveWikis in the future.

If you’re just here for the results then this is where you can stop reading.

Evaluation Process

Preliminaries

For this evaluation round, we recruited Charbel-Raphael Segerie, Dima Krasheninnikov, Gurkenglas, Imma Six, Konrad Seifert, Linda Linsefors, Magdalena Wache, Mikhail Samin, Plex, and Steven Kaas as evaluators. Matt Brooks, Frankie Parise, and I may also have pitched in. Some of them ended up not having time for the evaluation. But some of our communication was under the Chatham House Rule, so I’m listing them anyway for added anonymity.

Our detailed instructions included provisions for how to score project outputs according to quality and impact; how to avoid anchoring on other evaluators; how to select artifacts to strike a compromise between comprehensiveness, redundancy, and time investment; how to evaluate projects using wiki credits; and some tips and arrangements.

Outputs are such things as the papers or hackathons that organizations put out. They can create one project per output on our platform, or they can create one project for the whole organization. Conferences cannot be directly evaluated after the fact, so what our evaluators considered were artifacts, such as recordings or attendance statistics. This distinction makes less sense for papers.

The projects were selected from among the projects that had signed up to our website (though in some cases I had helped out with that), limited to those with smaller annual budgets (in the five or lower six digits, according to rough estimates) and those that were accepting donations. The set of outputs was limited to those from 2023 in most cases to keep them relevant to the current work of the project, if any. We made a few exceptions if there were too few outputs from 2023 and there were older, representative outputs.

We hadn’t run an evaluation round at this scale. Previously we were three and could just have a call to sync up. This time everything needed to be more parallelizable.

Hence we followed a two-pronged approach with (1) evaluations of individual outputs using scores, and (2) evaluations of the AI safety activities of whole projects using our wiki credits. If one kind of evaluation fell short, we had another to fall back on.

Lessons Learned

Fast-forward four fortnights, and it turned out that there were too many outputs and too few evaluators so only two outputs had been evaluated more than twice (and 10 had been evaluated more than once). According to this metric, AI Safety Support and AI Safety Events did very well, leaving the other project in the dust by a wide margin – but those numbers were carried just by the scores of one or two evaluators so they’re most likely in large part due to the Optimizer’s Curse.

Hence we decided not to rely on this scoring for our evaluation and rather fall back on the credits for that. But the evaluations came with insightful comments that are still worth sharing.

Next time we’ll use credits only and at most list some outputs to help evaluators who are not familiar with the work of the project to get an idea of what its most important contributions were.

Wiki Credits Ranking

These are the normalized average credits that our evaluators have assigned to the projects. As mentioned above, these determine how richly donors to these projects get rewarded in terms of their donor scores, which then determine the project support: Project credits → donor scores → project support.

Qualitative Results

AI Safety Events

AI Safety Unconference at NeurIPS 2022: One of the evaluators attended it and found it high value for networking, but (empirically) only for networking within the AI safety community, not for recruiting new people to the space.

ML Safety Social at NeurIPS 2022: One evaluator estimated, based on this modeling effort, that the social was about 300 times as impactful as the reference output (“AI Takeover Does Not Mean What You Think It Means”). The estimate was even higher for the safety unconference at the same conference.

Hence AI Safety Events had generally very high ratings. It is not listed among our top recommendations because we don’t have enough donation data on it. If you have supported AI Safety Events in the past, please register your donations! You may well move a good chunk of the (now) $700,000 that donors seek to allocate!

The Inside View

AI Takeover Does Not Mean What You Think It Means: This was our calibration output – it allowed me to understand how an evaluator is using the score and to scale their values up or down. The evaluators who commented on the video were generally happy with its production quality. Some were confused by the title (Paul’s models are probably well known among them) but found it sad that it had so few views. The main benefit over the blog post is probably to reach more people with it, which hasn’t succeeded to any great degree. Maybe we need an EA/AIS marketing agency? I’m also wondering whether it could’ve benefited from a call to action at the end.

AI X-risk Research Podcast

Superalignment with Jan Leike: This interview was popular among evaluators, perhaps because they had largely already watched it. Some were cautious to score it too highly simply because it hadn’t reached enough people yet. But in terms of the content it was well regarded: “The episodes are high-quality in the sense that Daniel asks really good questions which make the podcast overall really informative. I think the particular one with Jan Leike is especially high-impact because Superalignment is such a big player, in some sense it’s the biggest alignment effort in the world.” (The episodes with Scott Aaronson and Vanessa Kosoy received lower impact scores but no comments.)

AI Safety Ideas

The website: “Seems potentially like a lot of value per connection.” The worries were that it might not be sufficiently widely known or used: “I think the idea is really cool, but I haven’t heard of anyone who worked on an idea which they found there.” And does it add much value at the current margin? “I couldn’t find a project on the site which was successful and couldn’t be attributed to the alignment jams. However, if there were some successful projects then it’s a decent impact. And I suspect there were at least some, otherwise Esben wouldn’t have worked on the site.” The evaluators didn’t have the time to disentangle whether people who participated in any Alignment Jams got some of their ideas from AI Safety Ideas or vice versa. All in all the impact scores were on par with the Jan Leike interview.

Orthogonal

Formalizing the QACI alignment formal-goal: This output scored highest on quality and impact (with impact scores in between the three AXRP interviews above) from among Orthogonal’s outputs. It got lower scores on the quality side because the evaluator found it very hard to read (but noting that it’s also just really hard to create a formal framework for outer alignment). But it scored more highly on the impact side. The evaluator thinks that it (the whole QACI idea) is very unlikely to work but highly impactful if it does. The other evaluated outputs were less notable.

Center for Reducing Suffering

Documentary about Dystopian Futures | S-risks and Longtermism: One evaluator gave a lower quality score to the documentary than to the reference output (“AI Takeover Does Not Mean What You Think It Means”) but noted that it “represents longtermism decently and gives an OK definition for s-risk.” They were confused, though, why it was published on a channel with seemingly largely unrelated content (since the context of the channel will color how people see s-risks) and concerned that talking about s-risks publicly can easily be net negative if done wrong.

Avoiding the Worst - Audiobook: The audiobook got the highest impact rating among the CRS outputs even though an evaluator noted that they only counted what it added over the book – another way to access it – which isn’t much in comparison. (The book itself was outside of our evaluation window, having been published in 2022.)

FAR AI

Pretraining Language Models with Human Preferences: One evaluator was excited about this paper in and of itself but worried that it might be a minor contribution on the margin compared to what labs like OpenAI, DeepMind, and Anthropic might’ve published anyway. They mention Constitutional AI as a similar research direction.

Training Language Models with Language Feedback at Scale: While this one scored slightly lower quantitatively, the qualitative review was the same.

Improving Code Generation by Training with Natural Language Feedback: One evaluator was concerned about the converse in this case, that is that the paper might’ve contributed to capabilities and has hence had a negative impact.

Centre For Enabling EA Learning & Research (EA Hotel)

In general: “CEEALAR doesn’t have particularly impressive direct outputs, but I think the indirect outputs which are hard to measure are really good.” Or “the existence of CEEALAR makes me somewhat more productive in my everyday work, because it is kind of stress-reducing to know that there is a backup option for a place to live in case I don’t find a job.”

AI Safety Support

AI Alignment Slack: Invaluable for information distribution. One evaluator mentioned the numerous times that they found out about opportunities through this Slack.

Lots of Links page: “The best collection of resources we currently have,” but with a big difference between the quality and the impact score: “It could be better organized and more up to date (even at a time when it was still maintained).”

Epilogue

Want to regrant some of the (now) $700,000 aggregate donation budget of our users? Please register your donations! The GiveWiki depends on your data.

You’re already a grantmaker or regrantor for a fund? Use the GiveWiki to accept and filter your applications. You will have more time to focus on the top applications, and the applicants won’t have to write yet another separate application.

We’re always happy to have a call or answer your questions in the comments or by email.

GiveWiki and the Markets of Impact is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

The First Crowdsourced Charity Evaluator for AI Safety – Voiceover and animation by Frankie Parise

(Can you guess which city they’re visiting?)

Impact Markets is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In other news, we’ve reached $453,000 in donor interest. Now 46 donors have expressed their interest in using the platform. (Have you?) Two of them want to use it if we expand to their cause area, so I’m not counting them into the total at this stage. Most donation budgets are under $5k, but there are also many all the way up into the high five digits.

Maybe it’s time to update your bets? (See also these other Impact Markets questions.)

Our 2023 retroactive evaluation round is in full swing, so expect updates to our recommendations by December.

Impact Markets is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Impact Markets is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

This is an edited transcript of an eponymous talk. I’ve held this talk at EAGx Berlin and Warsaw 2023 among other venues.

The Funding Constraint

Great entrepreneurs who start highly impactful charity projects exist in various countries around the world and in lots of different fields that require highly specialized expertise. They also speak a number of different languages and have all their own social circles in the various countries that they live in.

That makes it difficult for an individual donor in Berkeley or London to identify but a tiny fraction of them.

Even really big foundations like the Open Philanthropy Project have problems with this. The Open Philanthropy Project can invest a lot of time into researching grants if the grant size is upward of $1 million or possibly $100,000 so that they make many grants in that area, measured by the total of those grants. But below $1 million the total grant size starts to trail off, and below $50,000 the number of grants drops off too. If they don’t already know the field and the founders, it becomes more and more difficult for them to research the projects efficiently enough that it’s still worth it for them given the small grant sizes.

That mirrors how venture capital firms would probably also love to invest the first $100,000 into all the future unicorns, but since they can’t identify them at that stage, or only vanishingly few of them, they depend on business angels and the founders’ friends and family to take the first step.

But we’re now in a world where AI safety projects struggle to raise $2,000 to make ends meet. So that gap below $1 million, particularly below $100,000, is critical. We need to coordinate the nonprofit equivalents of business angels so they can fill the gap!

Donors to the Rescue

How do we help donors coordinate? The good thing about individual donors is that they are almost everywhere in the world. If we just look at Giving What We Can data we see Giving What We Can pledge takers in 100 countries around the world. That’s a really sizable network that, if it were better coordinated, could serve to fill all of these funding gaps.

On the left, you can see the donor network as I intuit it today. There are some relatively well-connected funders but there are also lots of individual donors and lots of individual charity entrepreneurs. They’re largely disconnected or form small disconnected islands. We would like to rather move to the network on the right where all of these clusters are somewhat connected. It’s not perfect but everyone has connections to all the other funders via relatively few intermediaries. It’s much easier for projects to get funded and for donors to actually identify the most impactful projects on the margin.

Donors need to be coordinated because they come in various different shapes and sizes. There are some who are in earning to give. They spend all of their time on their full-time jobs where they have little time left to research their donations. But of course, they earn a lot of money in these jobs. These donors are all the way over on the left, the blue side of the spectrum. They can only spend a few hours per year on donation research.

But there are some other donors who can spend easily like three orders of magnitude more time on research that is relevant to their donations because they don’t even have to individually research their donations. They are working in fields where they are all the time, in their full-time jobs, exposed to exactly the sort of knowledge that is valuable for allocating donations.

There are also donors in between who – even though they work full-time in jobs that have nothing to do with donation research – are friends with researchers or live in the same flat with researchers. So these donors also osmotically absorb some of the knowledge that’s valuable for making donations, be it just the knowledge that the particular researchers that they know can be trusted.

We would like to bridge the gap between these 1000x donors and the 1x donors. We want to make it possible for the blue donors who spend all their time earning to draw on all the knowledge of the donors all the way over on the red side of the spectrum.

Our Solution

Our system works the way that we score donors after the fact in terms of how early they were able to identify some successful project, how confidently they identified it (via the donation size), and the level of success of the project. This is a retrospective scoring, so a project has generated some impact and then we look back at who has supported the project early on and exactly how early they supported the project, and then score these donors in terms of these three factors.

That will cause smart donors to stand out in our ranking. So the smartest donors will be at the top of the ranking. At the moment it’s still pretty easy to break into those top spots, so I encourage you to try!

The top donors then continue to make their donations. Thanks to their track record we know to listen to them. So we can generate new, forward-looking donation recommendations that aggregate all of their wisdom.

For the top donors, conversely, the incentive is to leverage more donations for the projects that they think are the most impactful ones.

Here is a quick example of how the scoring works:

There are pictured here two projects, the project Antiviral Algae and the project Rainwater Refinement. These projects enter the fundraising phase and find three donors each.

Alex was first, donated on day one to the project Antiviral Algae; then Avery donated on day two; and then Ash donated on day three. Our algorithm scores Alex highly in this project because Alex donated very early, but it also scores Ash highly because Ash donated a lot.

Likewise, for the second project, Ryan and River score highly because they, respectively, donated early or a lot. In between is Rowan.

And then the projects get executed. They invest the donations that they’ve received into their respective programs.

Then the evaluation happens. Our evaluators decide that the first project should receive a score of 200 and the second project one of 100. This results in the ranking that you can see all the way over on the right.

Finally, Ash receives the highest rank, and Alex is second, because the first project has the highest score, and these two donors have made the biggest contribution – monetarily in Ash’s case and in terms of information value in Alex’s case.

There are no donors who’ve donated to two projects in this case. The scores from these two projects would get added up.

Our Roadmap

At the moment we are in something that we call phase one. We have 46 projects on our platform, and we’re looking for more project scouts who want to make the first donations to these projects or potentially bring more projects onto the platform that they have already donated to so that they can register their donations to these projects. (That just depends on whether the scouts think that the most impactful project is one that is on the platform already or one that they still have to convince to join the platform.)

Eventually, once a sufficient number of these projects have been successful and have produced something that can be evaluated (we call these things artifacts), said evaluation will happen and we’ll pay some evaluators to score the projects. That’s phase 1.

Once we have some 100+ monthly active users we want to introduce something that we call impact credits or impact marks. It’s a play-money currency that we want to use on our platform. We’ll have various uses for it but primarily, it can be used by the people with the highest donor scores on the platform to themselves act as evaluators to some extent.

There’s also a phase 3 that is in the distant future at this point. Hopefully, one day, we’ll figure out all the legal problems of trying to institute a new kind of commodity. Then, hopefully, we’ll be able to institute these impact credits in a way that will make it possible to trade them on a market against US dollars or against some other kind of currency.

Phase 3 is particularly interesting because at the moment the “investors” on the platform are donors who try to use all of their specialized knowledge to make great donations. But eventually, we want to also bring for-profit investors onto the platform, investors who will bring all of their knowledge about (say) angel investing and all the experience from the startup sector into the nonprofit sector. They would try to earn impact credits to then sell them on the market.

We’re modeling this market ecosystem after that of carbon credits. Large buyers like governments could have like limit buy orders on our impact exchanges in order to encourage seed investment into early-stage charity projects.

More on our long-term vision can be found in the article “The Retroactive Funding Landscape.”

Current Challenges

Project scouts. I already mentioned that at the moment we’re looking for project scouts. So if you have some insider knowledge of some area and feel able to make highly impactful donations in that area then you can now share that knowledge with other people and thereby also leverage the donations of people with less knowledge of that area.

They will follow your donations in order to make almost as impactful donations. At the moment you can also import your historical donations to the platform to get a higher donor score. If you have a lot of donations to import, you can ask me and I can do it for you.

At the moment, you’d be able to leverage some fraction of the $391,000 donation budget of the 38 donors who have expressed interest in using the platform.

Donors. We’re also looking for donors of course – donors who just want to use the platform to allocate their donations but don’t think that they have any special knowledge of which project should receive them. These donors can just check out our top recommendations or pick out particular top donors that they want to follow.

If that is you, it’s crucial to us that you register your interest, so that we can increase the incentive for project scouts.

Investments. Finally, we’re also a Delaware public benefit corporation. So if you’re an impact investor or an angel investor and would like to invest in our project then we’d love for you to get in touch.

Impact Markets is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Disclaimer: This article covers many projects in the retroactive funding space. I’m a cofounder of AI Safety Impact Markets, so that’s the project that I know really well. I I’m much less knowledgeable of the other projects.

This is an edited transcript of an eponymous talk. I’ve held this talk at EAGx Berlin 2023 among other venues.

The Problems

The basic idea behind retroactive funding is that there is some sort of buyer (also known as retrofunder) that provides funding as an incentive. This could be a government, some sort of protocol in the blockchain space, a foundation, a startup in a growth sector that’s doing R&D, a company that has a corporate social responsibility (CSR) budget, impact investors, or donors – all sorts of altruistic actors. The idea is always to commit to paying out some sort of reward to create an incentive for project developers and investors. (I’ll always say investor when someone is at least partially profit-oriented and funder or buyer otherwise.)

With this incentive in place, these project developers can go to investors and ask for funding in the form of equity or debt in order to realize their project. Then, if the project is successful, the buyers can reward both of these parties. These financial flows are indicated in the diagram – but of course, they only happen conditionally on this project actually producing something that the buyer values.

This solves a few different problems, and depending on the temperament of the buyers, some of these might be relevant in particular cases or they might not be relevant.

Knowledge & Overhead

One problem that is particularly important for us at AI Safety Impact Markets is what has been called the long-tail problem, a particular kind of problem that can be classed under problems that relate to an imbalance in knowledge about a field (in particular very local conditions relevant to its implementation) or conversely the overhead that a funder would have to invest to acquire all of that local knowledge, for example by hiring a lot of different specialists.

The long-term problem in particular describes the problem that there are often a lot of small startup projects, independent researchers (in our case), or really small think tanks. These projects are so small and so many that the large funders can’t review all of them without incurring disproportionate costs. So they could just give those projects money at random – but on average these projects are not value valuable enough that it would be cost-effective for the buyers to do that. It would also attract a lot of fraudsters and bad actors to the space.

So the funders would have to invest a lot of time investigating these projects to find even one that they could make a grant to. And this grant would have to be relatively small. They’re much better off ignoring this whole space, even though there are gems in it, ignoring the whole long tail and just making a few big grants to organizations that already have a proven track record: well-established bigger organizations.

How do we serve this long tail? We can set an incentive in order to attract investors to the space. These investors will have specific private information about the individual few projects that they want to support. These investors might have particular special knowledge or might be active in particular spaces where they know the people really well. So they will have very good local information about a few of these projects. And taken together all of these investors will have specific knowledge about a lot of these projects in this long-tail. They can provide this knowledge to the market and can even make a profit off of it. In some cases, buyers will be ready to pay a premium for this service, because, even so, it’ll be cheaper than the counterfactual where they have to invest tremendous effort to try to acquire all this specialized expertise themselves. In return, they can capitalize on the extremely high-value opportunities in the long tail that would otherwise be very hard for them to recognize.

Access to Talent

Access to talent is a pretty specific case where a startup in a growth sector wants to get some R&D done for which they would hypothetically have to attract top talent in some space, maybe even university professors. Being a startup, they don’t have the funding to do that. They don’t have the big name that it would be interesting for these people to work for either, so this is basically not an option for them.

But the startup can become its own outcome buyer! Some version of the startup in the future has a lot of money. The startup has a lot of money in the futures in which it is actually successful in realizing its project. So it can make a binding commitment to reward the people who have contributed to that success, in the futures where they achieve the success. (That can be legally formalized and standardized in our setup.) It enables these startups to draw on this top talent to produce the open-access research that helps them get off the ground without the talent actually having to work for the startup or otherwise speculate on its success in particular. Investors will speculate on the success of the whole space (rather than a particular startup), and the researchers can get their funding from the investors.

This is particularly interesting for startups in growth sectors where they can just produce open-access research because competition is not such a big factor for them. Competitors are just other people who help them with the marketing of the whole sector.

Risk-Aversion

Now finally, risk aversion. This is currently the major use for impact markets. There are already a lot of government actors and large foundations that are using retroactive funding (or outcome-based funding) in order to realize positive change in the world. And they’re often doing this because they are risk-averse. They want to realize change with low variance and want to outsource the carrying of the risk to for-profit investors. And that’s why they are using retroactive funding and why they’re also happy to pay a premium.

The Ideal Solution

The whole ecosystem that I imagine around these mechanisms is a bit more complex. It’s the ideal solution that I envision at the moment. It’s copying what works for the carbon credit space, following the analogy that we want to establish something like carbon credits but for any kind of positive social impact.

Clear Standards for Claims

Some of the key things that I want to see are that we have clear standards in order to get buyers into the space. In order to get buyers or investors into the space, you need to have clear standards where everyone can agree on what it is that they’re trading in the first place and what it is that these buyers are buying or what these investors are investing into. Without a shared understanding, it’s going to be very hard to convince investors and buyers to purchase these constructs.

Public Registry and Ledger

What I think we’ll also need is a public registry and a public ledger in order to prevent fraud – and I’m mostly thinking of double-issue fraud here. The sort of fraud where someone is trying to sell more than 100% of some positive impact they have generated, or maybe even that some other party has generated.

They might, for example, issue a claim to have realized some impact on one marketplace and then issue that impact claim again on another marketplace. And if these marketplaces are sufficiently disconnected, no one might notice that these impact claims overlap or are identical. And then they can sell more than 100% of what they own.

They might also sell shares in these impact claims privately to different buyers and so sell more than 100% of them because these buyers don’t know of each other, and don’t know that, in aggregate, they have bought more than 100% of the impact claim.

These are things that can be prevented by having all of these transactions and all of these claims in public where they can be vetted. And that is something that the patent system currently struggles with. But we can learn from the problems of the patent system and hopefully prevent a lot of these problems from the get-go in impact markets.

With all the standardization and registries and so on, we are in the top right corner of this diagram, where it says standardization. That is something that I would like to see done by these standard-setting firms.

Evaluation of the Total Impact

When it comes to evaluation – that’s in the top left corner – I would like to see evaluations of the total impact and not only our particular cherry-picked outcomes. Because it can happen that some outcome, some positive outcome, comes at the expense of another positive outcome. And if a charity realizes some positive outcome but thereby harms some other positive outcome, then that outcome should trade at a lower valuation than an outcome that has been achieved without such a negative externality.

And there’s also disagreement over what outcomes are positive in the first place. So if outcomes are uncontroversially positive, they should also be valued higher than outcomes that are controversial in terms of whether they’re positive or not.

Prediction Markets

Prediction markets can also help debias these evaluations in particular cases. If you have questions about that, you can ask them in the comments. I think prediction markets should also play an important role in these markets. The prediction market should start as early as possible, the moment a project starts fundraising.

A Small Number of Markets

I think that we should aim for having as few markets as possible – for deeper liquidity, but mostly for the aforementioned reasons of incentivizing moral trade.

When you evaluate projects by their total impact, then you’re incentivizing the project developers to morally trade with other project developers who might have different moral opinions. So this incentivization is a very valuable aspect of having fewer markets. But then fewer markets also provide deeper liquidity, so it’s a win-win.

Deflationary Auctions for Investors

I think we need deflationary auctions for investors so that they try to come in as early as possible. Currently, charities struggle with the problem that donors wait out other donors. Donors try to keep their money in the hope that other donors will already fund some charity, and then only donate if it’s absolutely necessary because otherwise they can keep their money for some niche project that no one else would fund. That is basically a defection in an assurance game.

So we want to establish an auction mechanism that incentivizes the investors to invest as quickly as possible into charitable projects so that they can be funded more quickly. That way, the uncertainty for the charities over whether they will or will not reach certain funding goals is as little as possible and lasts for as short of a period as possible. We, for example, use a bonding curve auction for this purpose, which absolves the user from having to deal with the details of the auction mechanism.

Assurance and Coordination Among Buyers

This, I think, is very important, for the same reason, except that buyers are not profit-oriented. So you can’t just use an auction mechanism for this, but there are of course assurance contracts you can establish – assurance contracts or even dominant assurance contracts. Or you can use the S-Process. There are nice videos by Andrew Critch on YouTube on how the S-Process solves this problem.

It uses marginal utility functions, and many people will probably not have clear intuitions of what their marginal utility function is. So I imagine that it might be possible to create an 80/20 version of the S-Process that is suitable for use with a wide audience of lay donors. But there are also other coordination mechanisms that are already tested on wide populations, for example, Pol.is could serve to order outcomes by their uncontroversiality. And that is something that evaluators could use.

Then of course my ecosystem would also include exchanges and retailers to do the sales and for all these auctions to take place. So this is the vision for what I would like this ecosystem to look like. And of course, that vision is evolving. People might have different opinions on what exactly this should look like and which of these are actually important. I also imagine that over time I will update on a lot of these. But at the moment I think that these are the most important aspects of this ecosystem.

The Vision

The long-term vision is to integrate charities into the for-profit sector. Currently, there is an unnecessary separation between them. If charities can incorporate not as non-profits but as public benefit corporations, they can harness this ecosystem and generate more positive outcomes more quickly.

Impact Certificates as “Patents on Impact”

Building blocks for that are what we usually call impact certificates. You can think of them basically as “patents on impact.”

The problem with ideas is that they are just public goods. So by default, an idea would not be monetizable for a person. So, by extension, a person with an idea would not be able to attract seed funding to get this idea off the ground, because no profit-oriented investor would want to invest in a public good like that.

But then if you have a government that assures you that if you register this thing as a patent and you’re the first person to register it as a patent and it’s actually yours, then they will ensure that no one else commercializes the idea unless they license it from you.

We want to establish the same principle where, if you have a public good or maybe network good or something of this sort and you make it artificially excludable through this quasi-patent on impact, then investors can be incentivized to come in and provide the seed funding for you. So it’s about creating artificial excludability for impact.

I imagine that most of the trades and most of the incentive setting from buyers should probably not happen on the level of the individual impact certificates because they should probably be traded more like products where a charity puts out a lot of small impact certificates and just sells them whole, not fractionally.

If you’re running a think tank, then that’d be one impact certificate per blog post or paper or something that you put out. That way it’s really clear who the authors are, who has contributed to it, and who (a priori) owns which fraction of it. I’d like to keep them as small as possible, and as easy to intuitively grasp as possible so that there are also cultural norms that enshrine ownership.

Impact Credits as Financial Instrument

All of the actual incentive-setting and trades and so on should probably rather happen on the level of the impact credits. That’s what we are mostly focused on at the moment, what we want to establish. Our long-term vision is to have a market, like the carbon credit market, but for impact credits for any kind of positive impact where buyers like governments can just set their limit orders.

Investors that have invested in some sort of successful altruistic project can gain their impact credits from the validators, auditors, or evaluators and sell them into the buy orders that these governments or foundations have on these markets. I think that’s the vision that will probably streamline this ecosystem as much as possible, mostly going off the example of carbon credits.

The Landscape

What follows is my overview of the landscape.

Outputs, Outcomes, or Impact

First of all, I want to make a distinction between outputs, outcomes, and impact.

Here is one example of an output-based contractor: If you go to a restaurant, you pay the restaurant for the food they make for you. But if you go into the restaurant already relatively full and you’re super full afterward, then you don’t pay them more than if you go into the restaurant completely famished, and you’re just a bit full afterward. So it’s not about the outcome for you that you pay them for, but the output.

In contrast, what is already relatively well established is outcome-based funding.

1. There are the well-established social and development impact bonds that governments and large foundations are using to incentivize altruistic projects or improvements in living standards in various countries, or nationally for various public goods projects. These are usually focused on outcomes, so very specific metrics that project developers need to reach.

2. OutcomesX, as the name says, is focused on outcomes. They have the Impact Genome in the background, which is a project that has standardized a lot of outcomes, to the tune of 170+. It’s a very sophisticated system that they’re already using that is completely focused on outcomes.

3. I have not managed to get in touch with Ixo, but they also seem to have a pretty mature system that I think is outcomes-based. There is a podcast interview with the founders of Ixo. That is probably a really good source of information on Ixo.

4. Then there are of course carbon credits. Carbon credits are our paragon of how we think this market should work, except that we want to focus it on impacts.

5. There are advanced market commitments, another example where incentives are created for outcomes.

Prizes are used for all sorts of different things. So there’s no clear line to draw there whether prizes are used for outputs, outcomes, or impacts. There have probably been plenty of prizes for all of these.

There is the Retroactive Public Goods Funding project of the Optimism layer 2 blockchain on Ethereum. They are retrofunding things, efforts that have helped their ecosystem. That’s a very nice example of something that could just as well work for a for-profit. Optimism is a foundation, but the principle is universal, at least in growth sectors. A very nice example of this use of impact markets.

I’m not sufficiently briefed on the details of how they conduct their evaluations (and maybe I wouldn’t understand them if I were briefed on them) because I don’t have that in-depth knowledge of their tech stack unfortunately. So that’s why I’m leaving it a bit open whether they are focused on outputs, outcomes, or impact. I would imagine that they’re probably rewarding all three of them in different cases.

We’ve reached the last three projects that I will describe in some more detail, in particular, because they are focused on impact.

1. That’s the Hypercerts Foundation, they are standardizing hypercerts, which will hopefully provide what they call the basic data layer for this whole ecosystem. It’s so foundational that it can be used for basically anything, but they’re very interested in making this ecosystem about impact and not about outputs or outcomes. So they’re interested in getting the impact evaluators in their ecosystem to take all outcomes into account.

2. Manifund is another project that is focused on impact. I’ll mention them later in more depth.

3. Finally, there’s Impact Markets or AI Safety Impact Markets, our own project. Clearly, we’re focused on impacts.

Tradeability & Focus

Here’s a different taxonomy. In this taxonomy, you can see on the vertical axis the particular problem that the projects are tackling. So whether it’s access, which I elucidated in the context of growth startups; whether it’s risk, the typical use case for impact markets at the moment; or whether it’s knowledge or overhead, local specialized knowledge, the use case that we are particularly focused on at the moment.

On the other axis, there is the mechanism by which these trades are happening.

So on the left, there are some simple mechanisms. They are usually not traded at all. There’s one sale happening when in the end, when a project is successful and it gets rewarded. There are exceptions, but lots of price contests, there’s just a person who does something, and then in the end they win a prize for it.

The little icons in the top right corner indicate whether it’s currently possible to make money with the system or not, in a profit-oriented fashion.

1. There are exceptions to that, but I still put prizes in this category.

2. Social and development impact bonds also oftentimes fall in this category. At most there is one investment.

3. I think that Optimism is also simply rewarding people at this point. That might change. OutcomesX also doesn’t have investors at the moment. That might change too; hence the second mention of OutcomesX over to the right. OutcomesX is sort of thinking about getting for-profit investors involved, but there’s nothing really on the horizon.

That brings us to equity. These can be traded around among investors, lots and lots of trades can happen here. They can be traded fractionally too. They can change hands a lot.

1. I don’t know whether this is the case with Ixo. I think it is. They’re doing this on the Cosmos blockchain, so I’m sort of imagining that it’s the case, but I don’t want to spread misinformation. Please try to find out yourself.

2. In the case of Manifold, trade is possible. I will show some screenshots later.

3. Hypercerts are also traded on Ethereum and maybe some L2s at this point.

But all of these are individual assets that are issued by project developers to represent excludeably the impact of their work.

That brings us over to the right side into the space of commodities.

1. We as AI Safety Impact Markets are of course aspiring to establish impact credits as commodities. So that’s not the status quo, but our hope is to go for that format. Currently, you also cannot make profits with our system, but then at some point hopefully when all of this is mature, we hope that we’ll have the legal basis established to make that possible.

2. Carbon credits have already achieved that. So if you want to sell carbon credits or gold or other commodities, you don’t have to every time register your project with the SEC.

Target Use-Cases

And here is yet another taxonomy to give you an even better overview of this landscape. This is about what these technologies are used for at the moment. Most of them are a bit bunched up down there in the deployment space.

1. Carbon credits are mostly used for deployment at the moment – for whatever reason.

2. I think development and social impact bonds are also mostly used for deployment of existing interventions rather than research and development.

3. OutcomesX, judging by the Impact Genome standards, seem like they are mostly dealing in all things deployment, but there are a few outcomes standards that refer to research.

4. From what I could gather, that’s probably also true for Ixo. Again please ascertain that yourself.

5. When it comes to R&D, I have classed Optimism as partially for-profit R&D. It’s not quite correct since they are a foundation, but this could hypothetically be just as well used by a for-profit.

6. We are currently very much situated in the Impact R&D sector, AI safety in particular. That can change. Once our system is mature, we want to be able to just open this up generally for any kind of positive impact, be it profit-oriented or not.

7. Hypercerts are a very foundational technology, so they also cover everything.

8. Prizes have been used for various different things.

9. And Manifold is also in principle generic. They have of course focused on particular cause areas, but it’s not so easy to distinguish them in this system.

Projects

Hypercerts

Now I’d like to go a bit more in-depth on hypercerts. Earlier I mentioned standardization and the analogy of the “patent on impact.” This is exactly what hypercerts provides. Their inventor, David Dalrymple, has thought about what sort of schema would be useful for impact claims. You can think of a hypercert as a form with a few particular standardized fields.

There is a field for the description of what you’ve done; there is the period during which you’ve done the thing; there is a field for who has contributed, so who should be the co-owners of the impact initially; and there are a bunch of advanced fields.

Mostly it provides a clear standard and that is where its value lies as a foundational data layer for impact claims. So whenever someone wants to file some sort of impact claim they can now use the standard – be it on a blockchain, as it’s being used currently mostly, or just in whatever other random database. As long as it follows the standard, it establishes compatibility and that is the main benefit that hypercerts provide.

At the moment, outcomes are configurable but the people of the hypercerts team assure me that the impact evaluators will be held to take all outcomes into account regardless of what you specify there.

And interestingly rights are configurable. With this analogy to patents on impact, I’ve been referring to a right to commercialization, a right to this retractive funding, that (the right) you’re selling, licensing out, trading.

That is not necessarily the case with hypercerts. You can also just have some other right, for example public display, where whoever holds this impact certificate is allowed to display it publicly as their own. You can just configure this freely.

Hypercerts are being actively used. There are several ten thousand hypercerts in circulation and some 20,000 unique addresses are using them. So probably also thousands of people.

Hypercerts provide the basic data layer for impact claims. On top of that are attestations, valuations, and so on that provide legitimacy to particular hypercerts that deserve them.

Above that are all the different kinds of auction mechanisms, for example the S-Process that I’ve mentioned before, that can be used to allocate funding. Funding can take different shapes and can come from different sources. So this is what this nice diagram shows that I’ve lifted from a presentation by Holke of the Hypercerts Foundation.

Manifund

Then there’s Manifund. Manifund uses a hypercert-like form but it’s much simpler – just has a title, a subtitle, and a description. So if you want to make your Manifund description compatible with hypercerts, you have to include all the other mandatory fields of hypercerts in the project description.

They’re also including a few other fields that I think are not about the description of the impact claim. Some are for organizational details and to make the project easier to find.

They run this for a bunch of different cause areas. They started with the ACX Mini-Grants on forecasting. That’s why there are a lot of projects in that cause area. Then of course technical AI safety is a very popular field, so a lot of people in that space have also filed their impact claims with them.

The flow is usually that someone wants to do something, they create a project, they attract seed funding from investors, and then if they reach their funding goal, the money is paid out to them. Then they realize the project, and only when the project has succeeded, does the project page transition into the state of an actual impact claim. Then outcome buyers can come in and reward these investors by purchasing fractions of the impact claim from them.

OutcomesX

OutcomesX is a really sophisticated system that builds upon the Impact Genome, which is a decades-old system that has had a lot of time to mature and to standardize 170+ outcomes.

OutcomesX has that in the background for the standardization and the registration of impact claims. OutcomesX itself is basically the exchange or the retailer or broker for these outcome claims and allows buyers to purchase them and thereby retroactively fund the project developers. They don’t have investors yet at the moment. That is something that is potentially on the horizon, though.

The most interesting part is the level of sophistication of the standardization of the outcomes that they have achieved. Impact Genome rates the evidence by how extensive it is, by its rigor, relevance, and validity. You also get the cost at which the outcome in question has been achieved and a benchmark from among the other projects that they have in their database so that you can compare it with them.

They have a nice video that gives a really good overview of what outcomes X is doing.

AI Safety Impact Markets

Finally of course there is our own project, AI Safety Impact Markets. Our thinking has been the following: Since all of these things that have to do with equitization of impact claims are legally really difficult, what we want to start with is something that is really legally safe and simple, something that everyone can do everywhere in the world without having to worry about legal implications, where they don’t have to be accredited investors to participate in it, where the barrier to entry is as low as we can get it.

The basic intuition behind it is that capital markets reward investors for making very forward-looking prudent investments that pay off. These profits enable the investors to reinvest more capital and so have an even greater influence on price discovery. The better the investors, the greater their influence on prices. That drives the price discovery forward.

This price discovery is what we are currently completely focused on. We’re transferring it to the non-profit space. So if a donor makes particularly good donations to projects that are later successful and generate a lot of positive impact, then these donors should be rewarded for that and should have a greater weight in the funding allocation of other donations.

We do that through an organic regrantor system where the better the decisions of a donor have been (the better the donation track record of the donor), the greater the donor’s influence on the recommendations that we’re making on our platform.

So we are providing a crowdsourced charity evaluator, sort of like a Yelp or Google Reviews for charities. People who just want to donate and don’t want to do a lot of research can just follow the recommendations that the platform generates.

People who think that they have some private knowledge register their donations on the platform to build their track record. The better their track record, the greater their influence on the recommendations that the platform makes. So their incentive is to influence the recommendations on the platform in favor of the charities that they think are the most impactful ones in order to leverage others’ donations. Hence the intuition that they are quasi-regrantors for the other users of the platform.

That’s the first two phases. Now for the third and final phase.

You’ve seen this diagram before. This is where we want to get to in the end, the structures that we want to establish in our phase 3. To get there we will have to fundraise for all the legal costs that come with that and all the investigations into which country to even start this in.

Call to Action

For Investors

So if you are an impact investor or any kind of investor, this could be interesting for you. We will probably position ourselves in the standards and verification space where the revenue comes from buyers (account fees), project developers (registration and verification fees), and other evaluators/auditors (accreditation fees). We are a public benefit corporation.

For Donors

This brings me to this final call to action. If you’re interested in simply using our charity evaluator for AI safety to get donation recommendations, then that’s all live on our website, impactmarkets.io. Please just go ahead and use it.

But please also go to bit.ly/donor-interests and register that you are interested in using it, because the more interest we have registered, the greater the incentive for project scouts to actually help you with their hopefully wise recommendations for projects.

At the moment the recommendations are not super robust. We want to improve that. For that, we need these registrations of interest.

It’s a very quick one-minute form. You’ll be asked to fill in your name, email address, and your annual donation budget. You can ignore the rest of the fields if you like.

We put the aggregate, the sum of all of that, on the website. At the moment we are at $390,000, but we want to drive this up to a million or more in order to create a strong incentive for project scouts to come in and try to move that money to the projects that they think are the most impactful ones in the AI safety space.

For Project Scouts

Conversely, if you’re a project scout and you find it interesting to move some of these $390,000 that we already have, or potentially more soon, then please register your donations, bring your favorite charities that you think are the most impactful ones onto the platform, register your donations to them, and signal boost them on the platform so that other donors can see them.

That’ll make it legible to other donors that these projects are good investments and they’ll donate to them. You’re leveraging other people’s donations with your recommendations, so the time that you spend researching your donations will be even more valuable. You’ll become a regrantor for the other users of the platform.

Now we’re within one of these rare windows!

But before we jump into the FAQ, a quick announcement:

We are looking for new projects and expressions of interest from donors!

1. If you’re a donor who doesn’t want to spend a lot of time researching your donations, you’ll be able to follow sophisticated donors who have skin in the game. Our impact market doubles as a crowdsourced charity evaluator for all the small, speculative, potentially-spectacular projects across all cause areas. You’ll be able to tap into the wisdom of our top donors to boost the impact of your donations. Please indicate your interest!

2. If you’re a donor who has insider knowledge of some space of nonprofit work or likes to thoroughly research your donations, you can use the platform to signal-boost the best projects. You get a “donor score” based on your track record of impact, and the higher your score, the greater your boost to the project. This lets you leverage your expertise for follow-on donations, getting the project funded faster. Please indicate your interest!

3. Are you fundraising for some project, as individual or organization? Please post it to our platform. No requirements when it comes to the format or scope, so you can copy-paste or link whatever proposals you already have. We want to make it easier for lesser-known projects to find donors. We score donors by their track record of finding new high-impact projects, which signal-boosts the projects that they support. Attention from top donors helps you be discovered by more donors, which can snowball into greater and greater fundraising success.

4. If you are a philanthropic funder, we want to make all the local information accessible to you that is distributed across thousands of sophisticated donors and helps them find exceptional funding gaps. We signal-boost that knowledge and make it legible. You can use cash or regranting prizes to incentivize these donors, or you can mine their findings for any funding gaps that you want to fill.

Please let me know if you have any questions, below or in a call.

The same goes for the FAQ: If your question is not answered here, please pose it in the comments!

Contents

1. General questions

   1. What problems does it solve?

   2. What is a project?

   3. What is a creator or charity entrepreneur?

   4. What is a specialist donor; what is a generalist donor?

   5. What is a funder?

2. Questions about the platform

   1. What is the donor score?

   2. When and how are projects evaluated?

   3. What are your long-term plans?

   4. Last time I checked you were doing something with impact certificates, though?

3. Questions from charity entrepreneurs

   1. What does this platform do for me?

   2. What sorts of projects can I post?

   3. How long does it take to submit a project?

4. Questions from specialist donors

   1. What does this platform do for me?

   2. Can I make money with this?

   3. Project X doesn’t make sense if it receives less than $10k. I love it, but I only have $1k. What do I do?

5. Questions from generalist donors

   1. What does this platform do for me?

   2. How do I know that the donors I’m following aren’t just good forecasters but also have good ethics?

   3. How do I know that a project still has room for more funding?

6. Questions from philanthropic funders

   1. What does this platform do for me?

   2. What if I’m unhappy with the scoring?

   3. What if there are funders who defect against me by idly waiting for me to post the same prizes they want to see posted?

7. Questions about impact markets

   1. Is the goal to replace the current funding mechanisms?

   2. Is the goal to replace the current market mechanisms?

   3. How good is it?

   4. Can it go bad?

   5. Isn’t it in the interest of funders to promise funding but then not pay up?

   6. Does uncertainty really decrease over time?

   7. Do impact markets instead risk centralizing funding?

General questions

What problems does it solve?

Donors and grant applicants face the following three problems at the moment:

1. Charity entrepreneurs are known to waste a lot of time on redundant grant applications – each tailored a bit to the questions of the respective funder but otherwise virtually identical in content.

2. Donors, especially if they are “earning to give,” often don’t have the time to do a lot of vetting. Funds and donor lotteries address this, but a team of fund managers needs to win their trust first, which is not a given, and maybe they don’t want to take months off work in case they win the lottery.

3. Larger funders generally don’t want to invest much more time and money into vetting a project than it would cost them to fund it. Hence funders are forced to ignore projects that are too small.

This is our solution:

1. We promote impactmarkets.io as the one platform where grant applicants can publish their project proposals. No particular format: There’s a Q & A system though for funders/donors to ask further questions as needed. Questions and answers are public too. Funders can subscribe to notifications of new, popular projects in their cause areas.

2. When a donor supports a project, they can record that. When a project claims to have succeeded, some experts evaluate it. Eventually early donors to successful projects (“top donors”) will stand out as having unusual foresight, especially if they can repeat this feat several times. Donors who don’t have the time to do as much research can follow the top donors to inform their own donations.

3. Larger funders can (1) also follow the implicit recommendations of top donors, (2) encourage top donors with cash or regranting prizes, and (3) recruit grantmakers from the set of top donors.

There are a host of other benefits in various specific scenarios. You can read about them on our blog.

Eventually we want to grow this into an ecosystem akin to the voluntary carbon credit market (phase 3). But for now only phase 1 is relevant.

What is a project?

What we call a project is some set of actions that creators or charity entrepreneurs plan to carry out within some time frame. Good examples are blog articles, scientific papers, campaigns, courses, etc.

Whole charities (like the whole of the Center on Long-Term Risk rather than any one piece of research) are a bit of an awkward contingency because they don’t have any obvious “completion date,” but they qualify too. We’re considering methods for how we can evaluate them too.

What is a creator or charity entrepreneur?

We call a creator or charity entrepreneur someone who publishes a project on our website to fundraise for it. They are usually researchers, founders, entrepreneurs, etc. Founder would be another obvious choice, but the term creator is more general and harder to confuse with funder than founder.

What is a specialist donor; what is a generalist donor?

Both of them give money to projects, but their ambitions are different.

Generalist donors either don’t have the time or the specialized knowledge to evaluate projects. They want to use the impact market like a black box, a charity evaluator that makes recommendations to them.

Specialist donors have the time or the special knowledge to form a first-hand opinion on projects – be it because they are experts in a relevant field, because they are experts in startup picking, or simply because they are friends with the people who run a particular project. They use the impact market to make recommendations and thereby leverage the donations of the donors who rely on them. They may also be after the prizes that funders might provide!

What is a funder?

Funders are basically large donors. They can behave exactly like other donors, but they can also provide prizes to incentivize other donors.

Questions about the platform

What is the donor score?

The score is computed in three steps:

1. The contribution of each donor to a given project is calculated as a fraction that is greater if the donor contributed to a project earlier. Earliness here is not about sidereal time but about the order of the donations, so it doesn’t matter whether there’s a day or a year between the first and the second donation. The standard score also takes the size of the donation into account.

2. Eventually many projects will complete and then get evaluated. The result is a score that expresses the evaluators aggregate opinion on the relative impact of the project.

3. Finally the per-project contributions and the per-project scores are multiplied and summed up for each donor. This results in the scores that form the donor ranking.

When and how are projects evaluated?

Each project has an end date. The project creator can edit this date in case things take longer than planned, but at some point the date will be in the past and the project will really be complete. At this point it can apply to be evaluated.

We want to get a number of evaluators on board to consider the project artifacts and pass judgment on them.

The focus here will not be to make great contributions to priorities research but rather, if the project is (say) a book, to establish whether the book got written at all and whether it looks like someone has put effort into it. Ethical value judgments will be embedded in these assessments but we can hopefully find multiple evaluators so that, in controversial cases, the scores can average out.

The guideline for the calibration of the scores will be something along the lines: Suppose this book/paper/etc. didn’t exist, and I wanted to make it happen. How much would I have to pay? Or conversely for harmful projects: If there were a fairy who let’s me undo this book/paper/etc., how much would I pay to have it undone?

What are your long-term plans?

There have been changes to the funding landscape in 2022. Such vicissitudes keep our long-term plans in flux. But at the moment we’re aiming to create a market that is similar to the voluntary market for carbon credits. (These are also called “carbon offsets,” but the term “offset” would be confusing in our context.)

1. In the first phase we want to work with just the score that is explained above. Any prizes will be attached to such a score.

2. In the second phase, we want to introduce a play money currency that we might call “impact credit” or “impact mark.” The idea is to reward people with high scores with something that they can transfer within the platform so that incentives for donors will be controlled less and less by the people with the prize money and increasingly by the top donors who have proved their mettle and received impact credits as a result.

3. Eventually, and this brings us to the third phase, we want to understand the legal landscape well enough to allow trade of impact credits against dollars or other currencies. We would like for impact credits to enjoy the same status that carbon credits already have. They should function like generalized carbon credits.

Last time I checked you were doing something with impact certificates, though?

What we’ve been calling a “project” is something that can issue one or more impact certificates. Our platform still lists the existing certificates, but that’s merely an archive at this point. There is a chance we might return to this format, especially if we choose to found a nonprofit branch of our organization, but for the moment we have no such plans.

We’ve encountered three problems with impact certificates:

1. Charity entrepreneurs are hesitant to issue them because they need to define what their plans are and who their contributors are in some detail and commit to never issuing overlapping certificates. That requires some thought and coordination, and without a fairly strong promise of funding, few charity entrepreneurs are ready to put in the time and effort. That was compounded by the problem that hardly any funders were interested in using our or any impact market, so that there never was any such “strong promise of funding.”

2. Trading impact certificates is only legal between accredited investors in the US. Trade between accredited investors internationally (esp. US, Canada, EU, UK, and India) is sufficiently recondite of a problem that we haven’t found any experts on it yet. Additionally, we would not be allowed to help these investors coordinate either without running afoul of broker-dealer regulations. Figuring this out just for the US and accredited investors is something that can easily cost upward of $100k in lawyer fees and might even then just fail. The combination of costs, risks, and limitation to one country and only rich people was too much for us, at least at this level of scale.

3. Funders were very hard to find. We concentrated on outreach to funders for several months, had talks at several Effective Altruism Globals and other conferences, but in the end only got two funders interested (among them Scott Alexander though!), who promptly lost most of their funding because it was tied to the Future Fund. The funding situation changed to become even more unfavorable, so that we were no longer optimistic that it might still become easier to find funders.

Finally – and this hasn’t become a problem but would have – a lot of interesting financial instruments, such as perpetual futures, will remain inapplicable to impact certificates because each one of them is doomed to have very little liquidity. Most projects on impact markets will require some $10–100k in seed funding to get off the ground. The fully diluted market cap of even the most successful projects will probably almost never exceed $1–10m. The circulating supply will be much less still. Such assets are a good fit for bonding curve or English auctions but it would be useless to try to set up order books, indices, and futures markets for them. We previously hoped to bucket them to alleviate this problem. Impact credits will hopefully one day serve this purpose.

Hence, we’ve removed impact certificates from our plans and introduced projects instead, which are perfectly laissez faire about their definition. We’ve also opted to allow no trade of anything that can be turned into dollars. We might reboot markets for impact certs when the overall conditions change.

You can think of the donor score as analogous to the total value that you would hold in retired certificate shares if we still had those. (“Retired,” a.k.a. “consumed” or “burned,” shares are ones that cannot be sold anymore.) But it’s probably just confusing to think of it that way.

The only monetary rewards that donors may receive are prizes if they make it to the top of our donor ranking.

Questions from charity entrepreneurs

What does this platform do for me?

That hinges on how much promise your project has.

Let’s say it has a lot.

1. Do you have some donors who already trust you? Convince them to donate, then bring them on the platform to register their donations. As early donors they’ll be rewarded richly by the scoring.

2. Do you know any of the top donors? If the #1 donors is an AI specialist but your project is in animal rights, getting in touch with a top donor who is a specialist in your field may pay off even if they’re only donor #8 because animal rights donors will tend to follow the recommendations of other animal rights donors.

3. Now your project has gotten a donation from a top donor? That’ll wash it way up in our list of projects so that donors will see it even if they’re not yet following that exact donor!

To wit: As soon as you have any fundraising success, you can leverage it to build greater success. The platform even does it for you!

What sorts of projects can I post?

Basically anything goes… so long as it’s legal and not super risky!

We review every project and eliminate any that seem to us like they might be harmful. But please also make sure yourself that you don’t include any classified information or info hazards in the description because all projects are public. (Would you like to make your project only accessible to logged-in users? Send us a message through the Intercom button in the bottom right to indicate your interest in this feature!)

The ideal project is something finite that produces artifacts. Our evaluators will have an easy time with projects that fundraise for books, articles, or papers because they can read them to assess them. They’ll have a hard time with projects that are about whole organizations because an organization typically does a lot and they also can’t look into the future to know what great things the organization might still accomplish. Expect organizations to be undervalued, not because they suck but because so much of what they do is shrouded by the future and closed office doors.

How long does it take to submit a project?

There’s no required format. So if you already have a funding application lying around because you already applied for funding from some foundation, then just copy-paste or link it.

Other than that, you just need to enter a title and someplace where people can send you their donations, such as a PayPal or Stripe page. You can add some tags to make it easier for your project to be found. All in all this should take no longer than 5 minutes.

If you have no application written up yet, it’ll take longer. It’s up to you how comprehensive you want to make it. One thing I like to do is to write down just the essentials (if it’s short, it’s more likely to get read too), and then to include a link to a site where people can book a call with me to learn more.

Alternatively they also have the option to ask questions in the Q & A section. No need to procatalepse them all in your description when you can just respond to the ones that actually come up.

Amber Dawn might also help you with the writing.

Questions from specialist donors

What does this platform do for me?

Have you supported any charities early on that late made it bigly?

I, for one, would love to know what fledgling organizations you support today so I can get in early too. And for you that means that suddenly your donations count for more!

Some of my friends donate up to $100,000 per year. They don’t have much time to research their donations, so they, too, would love to know about that fledgling organization that you support. Even if you just donate $100 to the organization, your $100 might leverage $100,000 from the donors who trust your judgment!

That’s one thing that the platform can do for you.

Another is that we’re hoping for larger funders to come in and to reward our top donors. They might opt for cash prizes or for regranting prizes. Either way you’ll have a lot more money to give away if you unlock any of those prizes!

Can I make money with this?

My hope is that eventually a substantial number of people can turn donating into their full-time job. They make small but really smart donations, earn high scores as a result, and then make it all back several times over from the prizes that they win.

As of early 2023 we’re not there yet, but you might as well start building up your score already.

Project X doesn’t make sense if it receives less than $10k. I love it, but I only have $1k. What do I do?

A Kickstarter type of system would solve this, right? We can’t easily implement such “assurance contracts” ourselves, but we can help you coordinate: We could offer a way for donors to pledge that they want to donate $x if all donors together pledge to donate $X. Then once the sum of all pledges reaches $X, you’ll all get notified and can dispatch your donations.

Does that sound interesting? Please let us know, e.g., through the Intercom button in the lower right. We’ll prioritize the feature more highly.

Questions from generalist donors

What does this platform do for me?

You want to donate but maybe you don’t have time to do a lot of research or you want to donate in a field where you don’t have the requisite background knowledge. Hence you’re dependent on friends, funds, or charity evaluators to suggest good giving opportunities.

But all of these have limitations: Your friends probably know of many of the same giving opportunities, so you might be overlooking even better ones. The same is true of funds, though they receive applications, which alleviates the problem. Conversely, you may know and trust them less than some of your friends. The track record of retrospective self-evaluation at funds is thin. Finally charity evaluators have a wholly different set of limitations: They put a lot of effort into their evaluations, so that they can’t evaluate projects whose funding gaps are so small that they don’t warrant the evaluators’ efforts. Plus charity evaluators don’t exist for many cause areas.

We want to solve that for you. All you need to do when you want to donate is to turn to our platform. You can:

1. View our top donor ranking, pick out top donors who share your values, and then follow their recommendations, or

2. Filter projects according to your values (using the tags) and pick out the ones that have received most top donor support.

Today we’re just getting started, but over the coming months we want to establish a new, bottom-up, grassroots type of funding allocation mechanism that scales down to the smallest projects, is fully meritocratic, and doesn’t know geographic limits.

How do I know that the donors I’m following aren’t just good forecasters but also have good ethics?

Our plan is to hand off power to top donors gradually. First all their forecasting will bottom out at the judgments of impact evaluators that we will hire. That’ll ensure that they’ll be sophisticated altruistic, but it will not immediately steel us against our own biases. Later we want to recruit impact evaluators from our top donors, increasing the organic, bottom-up meritocracy of the platform.

But then we want to transition to phase 2 of our rollout. Phase 2 will gradually put top donors on the same footing as evaluators until most evaluation is done by top donors. But even then our evaluators will still be around to steer the platform as needed to make sure it is not usurped by any amoral top donors.

How do I know that a project still has room for more funding?

We ask projects to publish their fundraising goals and stretch goals. If they have not done so, please ask them for that information in the Q & A section.

Questions from philanthropic funders

What does this platform do for me?

You can use the platform like any other donor to find great, new funding opportunities.

But we also have a special function for you: You can basically rent our top donors by offering regranting budgets to them. Those serve the dual purpose that (1) you’ll get top grantmaker talent for free, maybe even top grantmaker talent whose networks are relatively uncorrelated with yours, and (2) by announcing such a prize, you create an incentive for prospective top donors to show up and try to prove their mettle.

If that sounds interesting to you, please get in touch, e.g., through the Intercom button to the lower right or via hi@impactmarkets.io.

What if I’m unhappy with the scoring?

Are you? If so, we can easily build a custom score for you. You score the projects, and we aggregate all of your project scores into your own custom donor ranking. Please get in touch if that sounds interesting to you, e.g., through the Intercom button to the lower right or via hi@impactmarkets.io.

What if there are funders who defect against me by idly waiting for me to post the same prizes they want to see posted?

We’ve termed this problem the “Retrofunder’s Dilemma.” It’s easy to imagine a world in which there are several funders – just too many for them all to be really chummy with each other – who all insist on extremely niche scoring rules to make sure that they don’t reward any donations to good deeds that anyone else might reward too. But that would leave exactly the most uncontroversially good deeds unrewarded.

We’re far from this being a problem for our rewarding, alias retrofunding, at all and even farther from it becoming a greater problem for retrofunding than it is already for prospective funding. But if it becomes a problem, the abovelinked article lists three remedies that funders can implement and four that charity entrepreneurs can implement. Or that we can implement for them to establish coordination.

Questions about impact markets

Is the goal to replace the current funding mechanisms?

Not really, sort of in the way that airplanes didn’t replace bikes. We think that impact markets will be best suited for funding the long-tail of small, young speculative startup charity projects. But they will be rather uninteresting for projects with strong track records or otherwise safe, reliable success. They will also be uninteresting for projects that require a lot of funding from the get-go.

You read more about the math behind these considerations on our blog.

The basic idea is that projects that are > 90% likely to succeed (according to some metric of success that the funder uses) don’t leave much room for an investor to make a profit while reducing the risk further for the funder.

Additionally, a risk-neutral funder is only interested in a risk reduction if it moves an investment from the space of negative expected value to the space of positive expected value. If a project is already 90% likely to succeed, it would have to be very expensive before it could become negative EV for a funder. Such an expensive project is then easily worth the time of the funder to evaluate prospectively rather than retrospectively.

So impact markets (with risk-neutral funders) are most interesting for:

1. Projects that seem very speculative to funders, e.g., because they are new and the funder doesn’t know the team behind the project,

2. Projects that require little money to get started, so they’re not worth the time of the funder to review.

If highly risk-averse funders are involved, though, they may be happy to pay a disproportionate fee for a risk reduction from 10% to 0%! There are also funders who are limited by their by-laws to only invest in certain types of low-risk projects. In some cases impact markets may present a loophole for them to do good more effectively without incurring any illicit risks.

Is the goal to replace the current market mechanisms?

No. The financial markets have developed over the course of over a century and are accompanied with legislation that is usually phrased in such generic terms that it is nigh impossible to create a separate financial apparatus outside of it. Many cryptocurrency projects have tried to create market mechanisms beyond the reach of the law, but the law typically disagreed. More recently, there is instead a stronger push to welcome regulation and to reform the law to facilitate regulation.

We therefore consider it infeasible to try to replace the existing financial systems. Rather our goal is to create systems that reward the creation and maintenance of public, common, and network goods while interfacing with the existing financial systems in standard, regulated ways. (The closest parallel is the voluntary market for carbon credits.)

How good is it?

[This section has not been rewritten for the new “impact credits” approach. The differences are probably minor.]

We’ve been trying to get an idea of how good impact markets are by putting some rough estimates into a Guesstimate, but a lot of the factors are multiplicative and they are all hard to guess, so that the variance of the result is very wide.

Some key benefits are:

1. There’ll be as much or more seed funding as there is today. The idea is that many investors will try many different things and try to think outside of the box. Often it’ll turn out that their calibration was off. They’ll invest into lots of projects and make less money back because they were wrong about how great all the projects will turn out. These investors will gradually select themselves out of the pool, but new ones will join. We don’t know how many ill-calibrated investors join for each that is well-calibrated, but we’ve seen some data that prize contests attract investments to the tune of up to 50x the prize money, so the average investor must be fairly ill-calibrated. Our model assumes that the value is probably around 1–20x.

2. The allocational efficiency can be improved because investors can overcome language barriers. But much of the world speaks English, and the US dominates the world economy, so we’ve put this improvement at a factor 1–3x.

3. The allocational efficiency can be improved because investors are in different social circles. This factor seems more significant to us, and we’ve put it at 1–10x.

4. The allocational efficiency can be improved because investors can draw on economies of scale. They can rent one server rack for all of their projects, or they can employ one HR person for all of them, etc. We’ve put this factor at 1–10x too.

5. Charities can draw on much greater talent pools if they can use fractions of impact certificates to align incentives. (They can also pay bonuses tied to retro funding.) We think that the talent pool might grow by 1–5x.

6. All of this frees up a lot of time for retro funders. Current prospective funders such as the Open Philanthropy Project have a lot of staff who would be excellent at a very practical brand of priorities research. When impact markets free up time for them, they can devote that time to research. Evidential cooperation in large worlds alone can serve as a likely existence proof that there is a lot more to know about global priorities. We very conservatively assume that improved understanding of priorities will boost the allocational efficiency by 1–10x.

The result of the model is that impact markets are unlikely to improve the current efficiency by less than 60x or by more than 11,000x.

We think that this range is likely biased upward:

1. Our model ignores black swan events that may occur with an unknown frequency and may be very harmful.

2. It is well possible that some factor in the model actually turns out to be < 1x for some reason that we haven’t thought of.

3. Finally, we’ve mentioned before that the multiplicativeness of the model makes it very easy for it to produce big numbers. This is the prime reason that we don’t greatly trust it.

You can find further discussion of the model in the comments on this post.

Can it go bad?

The biggest concern that we’ve had from the beginning in 2021 is that prize contests (such as impact markets) are general purpose: Anyone can use them – to incentivize awesome papers on AI safety or to incentivize terrorist attacks. In fact, promises of rewards in heaven could count as prizes. If we create tooling to make prize contests easier, there is the risk that said tooling will be used by unscrupulous actors too. The very concept of the prize contest could also count as attention hazard.

Here is a summary of all of the risks that we’ve identified and our mitigation strategies.

A rich terrorism funder could, for example, copy our approach and build an analogous platform where they promise millions of dollars to donors who fund speculative approaches to terrorism, such as terror attacks that only work out in 1 in 10 attempts. We would not allow such projects on our platform or a scoring procedure with such goals, but that doesn’t keep terrorists from building their own clone of our platform.

This doesn’t need to be obviously ill-intentioned (though terrorists probably also consider themselves to be heroes). You could imagine someone cloning our platform to fund grassroots nuclear fusion research, which might lead to accidental nuclear chain reactions in the basements of hobby physicists in densely populated cities around the world.

The Impact Attribution Norm alleviates this problem to (roughly) the extent to which it is adopted (see the question about measurement above). Yet it is not obvious that it will reliably be applied the way we would like to see it applied. This article is a good summary of its limits. See also our comment. Consider for example:

1. Someone might wrongly think that the impact evaluators will reward them for posting an article that contains some dangerous info hazards.

2. This can also happen if they don’t think that the particular impact evaluators will reward them but just that at some point there will be impact evaluators that will reward them.

3. Finally, it can happen that the impact evaluators are actually mistaken about the value of some impact and that their mistaken evaluation is predictable. This applies in particular (but not only) to actions that can turn out very well or very badly – might save lives or destroy civilization – but so happen to turn out well. The longer the duration between the launch of the project and the evaluation, the greater the risk that the prize committee will see only how well it turned out and ignore the other possible world where it did not.

These risks mostly seem like “black swan” risks to us – deleterious but highly unlikely risks. We’re also quite confident that we can prevent them from happening on our platform by carefully moderating all activity.

Finally, there is always the question how easy it is already for unscrupulous actors to achieve their ends and why they are not doing it already. It is quite easy for an unscrupulous millionaire to promise a big reward for something like nuclear fusion simply by tweeting it. But this is not currently a major problem. So the legal safeguards (or some other mechanisms) that also apply to our solution must be working fairly well. That said, we’re not solely relying on them.

Isn’t it in the interest of funders to promise funding but then not pay up?

Seemingly the best outcome for funders is to incentivize excellent work with the promise of a prize but then not reward them at all but to instead put the money into prospective funding of additional impactful work.

That is a shortsighted strategy as no one will trust a funder again if they’ve pulled this trick once. I would go further and suggest that donors should not rely on new funders to pay up unless they have a history of being trustworthy. For funders this means that it’s probably in their interest to gradually ramp up their prizes, so that they can build up trust more cheaply. Another option is escrow.

Eventually we hope to have tradable impact credits so that donors can assume that any funder who suddenly vanishes will thus leave the price at an unexpectedly low level which other funders will immediately use to “buy the dip.”

Does uncertainty really decrease over time?

This article touches on this question. I don’t think it’s important whether there is always more evidence of impact at a later point. Impact markets will just be most interesting for projects for which that is true.

The second part of the answer is that we think that there is a substantial number of projects for which this is true.

An example: You can usually divide your uncertainty about how a project – say, a book – turns out into two multiplicative parts: the probability that the book gets written at all and the impact-over-probability distribution of the finished book if it gets written. Once you know whether the book got written or not, that product collapses into just the second factor (minus the “if …”).

This only goes through if you take uncertainty to mean something like the difference between the best and the worst or the 99st and the 1th percentile outcomes, which may be a bit unintuitive. If you think of uncertainty as variance, and your fully written book has either an extremely positive or an extremely negative impact, then added uncertainty over whether the book has really been written adds another cluster of neutral outcomes in the middle between the extremes. It does not reduce (or increase) the difference between the extremes, but it does reduce the variance.

Do impact markets instead risk centralizing funding?

The whole point of impact markets is to decentralize funding – so might they perversely increase it? The argument goes that the current scoring rule allows for truly exceptionally good donations – the first donation to a project as amazing as Evidential Cooperation in Large Worlds might’ve been a substantial donation. Whoever the donor might be, they’d get an enormous score boost even though they were only right once. This boost might push them to the top of our ranking for many years until finally enough other donors have gradually accrued comparable scores. That seems unlikely but also undesirable.

One variation that we’re trialing is a score that does not take the size of a donation into account but just the earliness. Every project has a first donation, so even the first donations to great projects could no longer be as remarkable as a substantial first donation to a great project could’ve been under the size-weighted scoring rule.

Another remedy is to have scores decay over time. One solution we’re trialing is to have a score that only takes into account donations from the past year.

We’ll keep monitoring this potential issue and react in case it does manifest.

We’re worried that impact markets may set incentives that prevent exactly the sort of projects that are good across many moral systems and worldviews from being able to benefit from them. (Cf. the Giver’s Dilemma.) Prospective funding suffers from the same problem, so it is not a disadvantage of retrofunding over prospective funding. Nevertheless we’ve collected a few ideas for how to address the problem if it comes up.

The Retrofunder’s Dilemma

GiveWell writes:

Imagine that two donors, Alice and Bob, are both considering supporting a charity whose room for more funding is $X, and each is willing to give the full $X to close that gap. If Alice finds out about Bob’s plans, her incentive is to give nothing to the charity, since she knows Bob will fill its funding gap. Conversely, if Bob finds out about Alice’s funding plans, his incentive is to give nothing to the charity and perhaps support another instead. This creates a problematic situation in which neither Alice nor Bob has the incentive to be honest with the other about his/her giving plans and preferences – and each has the incentive to try to wait out the other’s decision.

Let’s suppose that the charity in question is in fact the top priority of each of the donors. The perhaps most internecine aspect of this incentive structure is that this charity is more likely to be uncontroversially good from an impartial perspective the more donors there are who can agree that it is the top priority. It is exactly this most uncontroversially good charity that suffers most.

This is also something that we want to steel impact markets against.

Luckily, the outcome is in the interest of none of the participants, so that we’re potentially faced with a mere “assurance game” problem. If we can find an arrangement that funds the top charity and that all participants agree to, we’ve solved the problem.

Defection by Sellers

There are two importantly different versions of this problem. The first is the one above where retrofunders get stuck in a defect-defect equilibrium. But you could also imagine a world where funders run their separate prize contests and accept submissions not through a public marketplace like ours but through a form whose responses only they can see.

In that world it is feasible for a seller to target only those prizes that are awarded by groups that are least value aligned with them to preserve the resources of the more aligned groups for other projects.

Impact markets probably already alleviate this problem by making projects and funding decisions public and by making it impossible to target applications at particular funders only. There are also already norms against this kind of defection in effective altruism. I think the current norms still fall short, but at least they exist to some extent.

Broader or Narrower Targets

I previously thought that the panacea to maximize the counterfactually valid impact of impact markets was to make the prize questions more specific so that they could target only neglected problems. I now think that that would be tantamount to defecting in a defect-defect equilibrium that leaves the most impactful funding opportunities untapped.

The targets or research questions that a prize contest rewards can be broader or narrower: “We want to see a proof or disproof that Vingean reflection is possible” as opposed to “We want to see contributions to AI safety.” There are advantages and disadvantages to both extremes.

With impact markets we currently want to bring together the people who currently want to but can’t do good (e.g., researchers with little money) and those who can but currently won’t do good (especially for-profit investors) – at least at the current margin, because many of them are probably already doing good up to the point that they can or want to.

But there are also those who can and do do good: funders and researchers with sufficient means. They are similar in some ways and could be seen as one group.

If so, these groups are not cooperating very well at the moment, so a one-sided attempt to cooperate would just lead to exploitation. If funders take this view, they’ll defect back, and we’ll be in a defect-defect equilibrium. This means in practice that they’ll pick targets that they think no one would otherwise work on, and therefore usually targets that are highly specific.

But these groups could also be seen as distinct because funders have a lot of flexibility in what to fund, so that they have a choice, whereas many researchers may be specialized to the point where the criterion of their personal fit more or less prescribes what they need to do. If so, it seems odd to interpret their behavior as defection. If funders take this view, it becomes less clear how they should act.

Another consideration is that the defection, if it is one, is not malicious. If asked, researchers will probably be truthful about the counterfactual of their work in a world without a given prize. So even broad targets will probably mostly allow us to measure our impact.

I don’t currently know how to think about this, so at the moment it seems reasonable to start with broad targets and possibly accept some exploitation, or to start with narrow targets and gradually build up cooperation one contract at a time. Either could lead to our desired outcome of a norm where you can expect retrofunding if you do something great. In any case, different funders will probably have different preferences, and most likely we’ll be able to measure which leads to the better outcomes in the long run.

Remedies

Against Defection by Retrofunders

1. Confabs and S-Process. The most straightforward way to coordinate retrofunders is to let them talk to each other. There will probably be very few retrofunders for a long time – maybe some two to five or so – which should make it easy for them to talk to each other and hash things out. (This excludes people who fund their own work, who are sort of like retrofunders but perhaps less flexible, as mentioned above.)

   1. In the causal case, coordination mechanisms like the S-Process (or some modification of quadratic funding) could help remedy the problem.

   2. An advantage of retrofunding is that the budgets typically get announced in advance and receive a lot of attention, probably more attention than statistics in an annual report.

2. Polis. The S-Process may become harder to use for a larger number of retrofunders, esp. less sophisticated ones. But arguably the right kind of UI can still allow users who don’t understand marginal utility functions to derive benefits from it. If we do end up scaling to the level of many retrofunders or decide to consider self-funded researcher retrofunders of their own right or want to start an alternative market for only highly trusted impact certificates that is open to a larger number of retrofunders or some other variation, then there’s another promising tool for us: the coordination mechanism Polis that was used in Taiwan. It seems plausible to me that we can limit the marketplace to research questions that come out of the Polis process: A large number of retrofunders enter their research questions and upvote or downvote all the others. The Polis algorithm then gradually highlights the questions that are most uncontroversially interesting to all market participants. Then only some number of these are entered into the market.

   1. Evidential Cooperation in Large Worlds may have recommendations for how to bargain in the acausal case, and these may be transferable to the causal case in certain communities.

3. Reselling. Funders may not always consume all the impact that they purchase but may actually want to hold it in expectation of profits via a future retrofunder. This introduces the opposite incentive to the one we’re worried about.

Against Defection by Sellers

1. No private trades. retrofunders should generally not accept offers made to them in private but only those that are public on the marketplace. That prevents sellers from targeting particular retrofunders.

2. Automatic auctions. An automatic auction system like the one we want to use has the advantage that sellers can’t just refuse better offers. Still there might be multiple marketplaces, so that a seller could choose the one that is used more by people with different values, or they could sell their cert outside any market.

3. Big projects. Impact markets should generally be used to finance projects that are big enough that they require seed investments. The profit motive of the investors counters the interests of the altruists running the project and will generally push toward making public offers.

4. Norms. What would be useful for prospective and retrospective funding alike would be to (1) do community-internal advocacy to strengthen moral cooperation norms, and (2) do community-internal education to make sure we all understand/agree what we consider a defection. That way we’ll be more likely to feel bad about defecting and avoid it, and peers can pressure each other not to do it.

   1. Paul Christiano has also thought about the problem: “By ‘norm’ I mean a rule that individuals can use for deciding how much to fund each public good. Here are two plausible desiderata for a norm: (1) If everyone always follows the norm, then we end up with the optimal levels of funding for the public goods. (2) If you start with a community that follows the norm and add a bunch of new people who behave manipulatively, they can never make the original community worse off.”

   2. It stands to reason that either broader or narrower targets will lead to better or stronger norms or allow them to emerge more quickly. This would be interesting to investigate.

Appendix

Original Formulation

When we were first asked about this scenario, it was in a configuration that we think is unlikely in practice. A fictional example with fictional actors:

Open Philanthropy Project (Open Phil): AI safety is 2x as valuable as international development.

Secretive organizer named Philip (Closed Phil): International development is 100x as valuable as AI safety.

Open Phil announces a contest: Do stuff we value, and we’ll reward you up to $1m!

Closed Phil invests $1m into international development and submits it to the contest. Closed Phil offers the impact to Open Phil at $400k. Open Phil takes the offer because it looks like something worth > $800k, so they rather buy it than some other AI safety impact.

Closed Phil reinvests the $400k into international development.

Rinse and repeat until Open Phil is out of money. Closed Phil has successfully leveraged all or most of Open Phil’s money, and Open Phil has invested comparatively little into AI safety.

After three iterations, > $1.5m may be invested into international development and < $500k into AI safety. If Open Phil had not announced any prize contest, $1m + $333k would’ve gone into international development and $667k would’ve gone into AI safety, which is closer to Open Phil’s preferences, so they don’t want to do a prize contest.

In this formulation it’s not clear why Open Phil would buy the impact, so we have a modus ponens/modus tollens type of situation. Clearly, Closed Phil is continually operating at a loss, so they’re either crazy or they’re interested in the impact and capable of producing it also without the retrofunding. Hence the counterfactual effect of the retrofunding is likely minimal. Open Phil doesn’t want to do something that is not counterfactually impactful. Maybe they will pay out once (maybe at a still lower price) to not seem unpredictable to other participants and reduce trust in their retrofunding, but they will then adjust the purview of the contest to avoid this failure going forward. They can do this, for example by using separate budgets for international development and AI safety or by focusing on only AI safety.

An exception is if Closed Phil manages to deceive them, e.g. by lying about how much they invested or by claiming that they’re just cutting their losses because the project failed by their lights. They can make up some metrics as a pretext that the project didn’t meet that Open Phil doesn’t care about.

The scenario can be rephrased to one where the deception is subtle and no one intentionally retrofunds nonoptimally. That’s the “defection by sellers” scenario above.

Recap: Impact Markets

For a more comprehensive explanation of impact markets, see Toward Impact Markets.

Thanks for reading Impact Markets! Subscribe for free to receive new posts and support our work.

In short, an altruistic retroactive funder announces that they will pay for impact (or “outcomes”) they approve of. It resembles a prize competition in this way. But (1) they’ll pay proportional to how much they value the impact and not only the top n submissions; (2) the impact remains resellable by default; and (3) seed investors offer to pay the people who are vying for the prizes or provide them with anything else they need and receive in return rights to the impact and thus prize money.

It is analogous to the startup ecosystem: Big companies like Google want to acquire small companies with great staff or a great product. Founders try to start these small companies but often can’t do so (as quickly) without the seed funding and network of venture capital firms. When the exit happens (if it happens), the founders may not even any longer own the majority of the company because they’ve sold so much of it to the investors.

The benefits are particularly strong for high-impact charities and hits-based funders:

1. If a hits-based funder usually funds projects that have a 1 in 10 chance of success and switches to retroactive funding, they save:

   1. the money from 9 in 10 of the grants,

   2. the time from 9 in 10 of the due diligence processes, and

   3. the risk from accidentally funding projects that then generate bad PR.

2. Investors can thus speculate on making around 10x return on their successful investments, and they can further increase their expected returns:

   1. by specializing in a narrow area (such as AI safety) to make excellent predictions about which project will succeed,

   2. by providing founders with their networks in those areas,

   3. by buying resources at a bulk discount that founders need (such as compute credits), and

   4. by finding founders that none of the other investors or funders are aware of to negotiate deals with them where they receive a large share of their impact certificate/s.

3. Charities can attract top talent and align incentives with top talent who may not be fully sold on the charity’s mission:

   1. by promising them a share in all impact sold,

   2. by locking that share up in a vesting contract,

   3. by (possibly) sharing rights to the impact with another company that is the current employer of the talent so that they don’t need to quit and can draw on the infrastructure of the other company. (In fact, somewhat value-aligned companies may be interested in becoming investors themselves if they want to retain talent who want to work on prosocial applications of their knowledge.)

4. Individual researchers can attract funding for their work even without the personal ties to funders, e.g., because they are in a different geographic region and better at their research than at networking.

Profitability of Impact Markets

We think about this in terms of the riskless benchmark B and the ratios r_c and r_p. The benchmark B is a return – e.g., B = 1.1 for a 10% profit – that an investor expects over some time period. An investment is interesting for the investor if it is more profitable than B. r_c = c_f/c_i is the ratio of the costs that funder and investor face respectively. This includes, for the funder, the cost of the grant, the time cost of the due diligence, the reputational risk if the due diligence misses something, and, for the investor, the cost of the grant minus savings thanks to shared infrastructure, economies of scale, etc. r_p = p_i/p_f (note that enumerator and denominator are the other way around) is the ratio of the probabilities that investor and funder respectively assign to the project success. The investor may specifically select projects where they have private information (e.g., thanks to their network) that give them greater confidence in the project’s success than they expect the funder to have.

Hence, investments are interesting if r_c ⋅ r_p > B.

The graph shows the benchmark of an investment with 30% riskless profit compared to the maximum profit from various project configurations. It elucidates that an investor who can help realize a project more cheaply than the funder or thinks that it is more likely to succeed, can outperform the funder in a range of scenarios. These are scenarios where one or both parties can reap the gains from trade and save time or money.

The square between 0 and 1 on both axes is largely irrelevant. These are scenarios where the investor would have to pay more than the funder or is less optimistic about the project. Those are obviously uninteresting. But also just outside that square and around the edges, there are areas where the investor may not be interested because their edge in terms of the r_p and r_c ratios is too small. Then again a riskless 30% APY is a high benchmark.

A few examples:

If a charity already has a track record of doing something really well 10 out 10 times in the past, there is very little risk involved when they try it for an 11th time:

Maybe an investor thinks they’re 99.5% likely to succeed and the funder thinks they are at least 99% likely to succeed, and the action costs $1m for either and takes a year.

That’s r_p = 1.005 and r_c = 1. It is only interesting for an investor who cannot otherwise invest the money at 0.5% profit per year.

1. It’ll be worth little to the funder: If they value the impact at 99% probability at $1m, they’ll pay $1m/99% ≈ $1.01m for it, so $10k premium.

2. If an investor offers to carry that tiny amount of risk, they’ll want it to exceed their 10–30% benchmark after a year, or else a standard ETF investment would be more profitable to them. That’s at least a $100k premium.

3. A bid of a $10k premium (minus the overhead of the whole transaction) from the funder but an ask of $100k premium from the investor means that there’ll be no deal.

But consider a case where someone has no track record:

The investor thinks they are 20% likely to succeed. The funder thinks that they’re 10% likely to succeed. The action costs $1m for both and takes a year.

That’s r_p = 2 and r_c = 1. It’ll be interesting unless someone has a benchmark of more than 100% per year.

1. The funder will pay up to $1m/10% = $10m for the riskless impact.

2. That’s a 1000% return (or 900% profit) for the investor with 20% probability, so 100% profit in expectation, which beats most benchmark investments. Even if their riskless benchmark is as high as 30%, they’ll accept offers over 650% return. Naturally, these investors have to be fairly risk neutral or make many such investments. (If they are somewhat altruistic, they can consider the difference between the risk neutral and their actual utility in money a donation.)

3. Funder and investor will meet somewhere at or below 1000%.

It is easy to create an analogous example for the case where funder and investor make the same probability assessments but where the grant size is so small that the investor, who already knows the project, can fund it at half the price compared to the funder who would have to spend a lot of time on due diligence.

Impact Timeline

A typical product that is suitable for impact markets is a scientific paper. Papers, like many other projects, have the property that they often get stuck in the ideation phase, sometimes have to be abandoned (for other reasons than being an interesting negative result) during the research, sometimes don’t make it past the reviewers, and sometimes turn out to have been a bad idea only decades later.

When an investor wants to invest into a paper that has not been written, but which they are highly optimistic about, they may see these futures:

The x axis is the time (in years), the y axis is the impact (proportional to dollars), blue lines are possible futures, and the red line is the median future.

There are two big clusters: all the futures in which the paper gets written, published, and read, versus and all the futures in which it either never gets finished or gets read by too few people.

One to three years into the process, it becomes clear in which cluster a given future falls, particularly if it falls into the upper cluster. (Otherwise there’s a bit of a halting problem because it might still take off.) Maybe the paper has been published on arXiv and is making rounds among other researchers in the field.

After 10 years, the majority of the impact has become clear and the remaining uncertainty over the value of the Attributed Impact of the paper is low.

After 15 years, we’re asymptotically approaching something that looks like a ceiling on the Attributed Impact of the paper. Experts have hardly updated on its value anymore in years, so their confidence increases that they’ve homed in on its “true” value. (“True” in the intersubjective sense of Attributed Impact, not in any objectivist sense.)

This is a vastly idealized example. In practice it may be that a published paper that used to be held in high regard suddenly turns out to have been wrong, an infohazard, plagiarized, etc. Or it may be that it’s suddenly noticed that a decade-old forgotten-about paper (that had high ambitions at the time but seemed to fall short) contains key answers to an important new problem.

Timing of Retroactive Funding

If an investor is a specialist in some small field and profits from economies of scale in the field (e.g., the compute credits bought in bulk that we mention above), then they may expect to make a 10× profit from each retro funding that they receive. That’s the difference between the size of the retro funding at which the retro funder breaks even (ignoring interest) and the cost to the investor. We assume for simplicity that monetary and time costs (grants and due diligence) are the same. So, 2 · 10i − 2 · 5i = 10i, where i is the average seed investment. (We’re using the parameters from above where retro funders save 10× from making fewer grants and 10× from saving time spent on due diligence. We also assume that a patient, well-networked, specialized investor has twice the hit rate of the generalist funder.)

If, counterfactually, they would’ve invested this money at 30% APY, the impact market ceases to be interesting for them if they expect the retro funding to take longer than 8–9 (≈ 10.6 ≈ 1.39) years: 2 · (1 / ratefunder) − 2 · (1 / rateinvestor) = (1 + apy)years.

Here we’re comparing an investor at different hit rates to a retro funder who would otherwise have a 10% hit rate under four counterfactual market scenarios. The impact market is profitable for any number of years less than the break-even point.

If the retro funder wants to save money, they can pay out less, but will need to do so earlier. For simplicity, the following chart is only for the scenario with a counterfactual 30% APY: 2 · (1 / ratefunder) · (1 − savings) − 2 · (1 / rateinvestor) = 130%years.

A retro funder needs to take this into account when deciding how much certainty they want to buy from the investors. More added certainty comes at a higher price. They can regulate this through the size of their retro funding or through the timing. Depending on the impact in question there are usually certain sweet spots that they can aim for, and do so transparently so that investors know what time horizons to speculate on.

When it comes to our example above, it seems fairly clear whether the paper was a success (was written, published, and read by some people) after about 2–3 years. So one sweet spot may be to wait for the moment of publication (as a draft or after peer review) or after the initial public reception can be gauged. The second is interesting because investors may be well-positioned to help with the promotion.

But there are other options – less profitable options much later.

Dissolving Retroactivity

We can imagine a chain of retro funders from a particular set of futures into the present: Someone makes a binding commitment that if they are successful in making a lot of money – say, their business is successful – they will use the money or a fraction of it to buy back impact that has previously been bought by a certain set of existing retro funders who the person trusts. They can continually add new ones to this set.

This can also be formulated as a prize contest: If I’m successful, I’ll use that budget to buy impact from my favorite retro funders at a reasonable bid price. If 1 in 5 projects still fail between the time when the retro funder bought them and the time when the success happens, the retro funder may buy them at 120% of the price that the previous retro funder paid.

Under this framing there is no qualitative difference anymore between investors and earlier retro funders. They’re all just different investors with different attitudes toward risk or preferences about how they weigh the profit vs. the social bottom line of their investments. (Some of them may choose to consume their certificates, though, to signal that they’ll never resell them.) There may even be investors who choose to invest into “whatever project person X will do next,” so earlier than the abovementioned seed investors.

A startup may be interested in making such a commitment because they have the choice to either do the research in-house or at least pay for it immediately or to pay for it later and only if they are successful. Since startup success is typically Pareto distributed, they’ll have vastly more money in the futures where they are successful than they have now or in unsuccessful futures. So this deal should be interesting for most startups.

For investors it’s a question of whether they want to expose themselves more to the field or to a particular team. If they’re excited about the team behind the startup and trust that team to do well regardless of what field they go into, they’ll want to invest directly into the startup. But if they’re more agnostic about all the teams in a field but are very excited about the field, they may prefer investing into the research projects to bet on the retro funding.

Example

1. Cultured meat (or “cell/clean/c meat”) startups may require a lot more research to be done on how to scale their production and make it cost-competitive. But they don’t yet have the money to do all of that research in-house.

2. They commit to investing a large portion of the money they’ll make from going public into buying impact. Specifically they hash out particular terms with an organization like Founders Pledge that stipulate what impact related to cultured meat research they will buy from which retro funders.

3. The promise of great potential future riches boosts funding and opens up hiring pools.

4. Eventually, the now more likely future might happen, and the large budget from the exits serves to buy most of the impact from the retro funders.

Conclusion

We’ve received a grant via the Future Fund Regranting Program to work on this. If you’d like to join our discussions, please join our Discord.

Thanks to my cofounder Dony for reviewing the draft of this post! He gets 1% of the impact of it; I claim the rest.

Thanks for reading Impact Markets! Subscribe for free to receive new posts and support my work.