This is part 2 of a five-part sequence on welfare ecology. Part 1 introduces the ethical premises. Part 3 covers interventions. Part 4 explores a model of invertebrate suffering. Part 5 covers AI.

If you decided in part 1 that you care – even a little – about the welfare of invertebrates, the next question is: Where are they, how many are there, and what drives their population sizes?

The Numbers

Terrestrial Soil Fauna

The foundational data comes from two landmark studies:

• Van den Hoogen et al. (2019) estimated 4.4 × 10²⁰ nematodes in the top 15 cm of soil globally. Stefan Geisen, the second author, clarified that this accounts for roughly 90% of all soil nematodes, putting the total at about 4.89 × 10²⁰ (Grilo 2025a).

• Rosenberg et al. (2023) estimated 10¹⁹ soil arthropods, of which roughly 95% are mites and springtails, with the remainder being ants, termites, and other groups.

For context: There are 8 billion humans and roughly 80 billion farmed land animals at any given time. Soil nematodes alone outnumber all farmed animals by a factor of 6 billion.

Marine Fauna

From Bar-On, Phillips, and Milo (2018):

• Approximately 10²¹ marine nematodes – about 2× the number of soil nematodes.

• Approximately 10²⁰ marine arthropods (copepods, krill, amphipods, etc.) – 10× the number of soil arthropods.

Marine nematodes are overwhelmingly benthic (living in seafloor sediment). Marine arthropods are split between benthic and pelagic (water column) habitats.

Density by Biome: The Terrestrial Landscape

Not all land is created equal. Rosenberg et al. (2023) provided mean densities of soil arthropods by biome. Grilo (2025a) combined these with van den Hoogen et al.’s nematode data to estimate total soil animal density. The results:

Several things jump out:

Forests are packed. Boreal and temperate forests have 7–8× the soil animal density of cropland. The mechanism is well understood: forests produce continuous, diverse leaf litter that feeds a complex soil food web. They have deep root networks supporting fungal communities that mites and springtails feed on. The soil is structured with pores and channels that provide habitat at every scale.

Cropland is relatively sparse. Tillage destroys soil structure. Monocultures reduce food web diversity. Pesticides directly kill a broad spectrum of fauna. The litter layer is minimal.

Pasture is sparser still. Grazing pressure, trampling, and simplified plant communities reduce below-ground complexity even further.

Deserts are comparatively empty. At 0.05 million per m², deserts have ~130× fewer soil animals than forests. Still even 50,000 animals per m² is impressive.

Volumetric Comparison: Soil vs. Sea

Since soil nematodes are measured in the top 15 cm (~0.15 m³ per m² of surface), we can convert to volumetric density and compare to marine environments.

One wrinkle: The dominant taxa change with the habitat. In soil, nematodes and arthropods (mites, springtails) coexist. In marine sediments, nematodes dominate overwhelmingly – they make up 80–90% of meiofauna (benthic organisms of 32–63 μm to 0.5–1 mm), with only small numbers of harpacticoid copepods alongside them. In the open water column, the situation reverses: There are essentially no pelagic nematodes (they are benthic animals, bound to sediment), and the invertebrate fauna is almost entirely arthropods – copepods, krill, and amphipods.

The table below uses “nematodes + arthropods” for all environments, but the composition varies as described:

The key insight. Per cubic meter, terrestrial soil and some coastal marine sediments are by far the densest habitats per cubic meter for potentially suffering animals, though coastal sediment occupies a much thinner layer. The open ocean water column is orders of magnitude sparser – even productive surface waters have roughly 10,000× fewer organisms per m³ than forest soil. But the ocean’s total volume is vast (~1.37 × 10¹⁸ m³), so low per-m³ densities still add up to enormous total populations.

The deep ocean – both water column and sediment – is strikingly empty by terrestrial standards. It represents an enormous volume of near-zero-fauna space that already exists naturally.

What Drives Population Size? The NPP Primer

Understanding why some biomes have more soil fauna than others requires understanding net primary productivity (NPP).

NPP is the rate at which plants convert solar energy into organic matter, net of their own respiration. It’s measured in grams of carbon (gC) per m² per year. It’s an objective, species-independent measure of how much new biological material is being produced. A tropical rainforest has an NPP of ~1,000–2,000 gC/m²/yr. A desert has ~0–90 gC/m²/yr. Open ocean averages ~125 gC/m²/yr.

NPP matters because it determines the total energy flowing into the food web. More NPP means more food for decomposers, which means more food for the nematodes and mites that eat the decomposers, and so on up the trophic levels.

Vasco Grilo’s analysis found that NPP-related variables (biome type, soil organic matter) are far more important than any other factor in predicting soil fauna density. The m² -years of agricultural land per food-kg explained essentially 100% of the variance in welfare effects for his preferred parameters (Grilo 2025c).

A common misconception. NPP is not subjective to a species. When a cow eats grass, the NPP doesn’t change – the grass still produced the same amount of carbon. What changes is the pathway: instead of grass dying and feeding soil decomposers, the cow intercepts the energy, metabolizes 40–70% of it, and excretes the rest as dung. Dung is more labile (easier for bacteria to digest) than leaf litter, so it can boost local populations of bacterivorous nematodes around dung pats. But the cow has diverted most of the energy away from the soil food web, so the net effect on soil fauna is still strongly negative.

The same logic applies to any intervention that changes the pathway of NPP without changing NPP itself. This distinction turns out to be critical for evaluating pesticides.

An important caveat: Global NPP has barely changed. As I noted in part 1, Tomasik’s analysis of Krausmann et al. (2013) found that NPPeco – the global NPP available to wildlife – has barely changed over the 20th century, partly because CO₂ fertilization has increased potential NPP even as humans have appropriated more of it. This means that the biome-level density differences in the table above are real and actionable for local interventions, but the global total of animal metabolism may not have declined as much as defaunation indices suggest. Converting one hectare from forest to cropland genuinely reduces soil fauna on that hectare by ~7×; but globally, CO₂-driven NPP increases elsewhere may be partially offsetting these local reductions. This doesn’t undermine the case for targeted land use interventions – it just means we should be cautious about claims like “humanity has reduced total invertebrate suffering by X%.”

The Pesticide Trap

One might naively think that pesticides reduce invertebrate suffering by reducing their populations. But the relationship between pesticides and welfare is more complex – and possibly perverse.

Consider two interventions that both halve the standing population of soil fauna:

Intervention A: Reduce NPP by 50%. (For example, convert forest to desert, or shade the land.) The carrying capacity halves. At the new equilibrium, fewer organisms are born, fewer die, and the birth rate and death rate are both lower. Total deaths per year: lower.

Intervention B: Apply pesticides while keeping NPP the same. Food is still available. Organisms are born at rates close to the original (abundant resources encourage reproduction). But pesticides kill a large fraction of them. The population is suppressed below carrying capacity, but the high food availability keeps birth rates high. At the new equilibrium: birth rate is elevated (more food per capita), death rate is elevated (pesticides), and organisms are dying of pesticide poisoning instead of starvation or predation. Total deaths per year: much higher than the original, with deaths potentially involving more acute suffering.

This matters enormously for r-strategists. Brian Tomasik and Yew-Kwang Ng (1995) argued that for r-strategists, death is a large fraction of total lifetime suffering. If that’s right, the death rate may matter more than the standing population. And pesticides at constant NPP can increase the death rate – creating a high-throughput killing field rather than a genuinely smaller population.

Pesticide Persistence

The picture is further complicated by the fact that different pesticides have very different persistence profiles:

• Organophosphates and carbamates degrade in days to weeks. These are closest to the “spray, kill, wash away, repeat” model – high-throughput killing.

• Neonicotinoids persist in soil for months to years and are taken up systemically by plants. Any insect feeding on a treated plant gets a dose. This effectively makes NPP inaccessible to insects – the food is there but poisonous. This is closer to “reduced accessible NPP” and may genuinely reduce the carrying capacity.

• Legacy organochlorines (DDT, dieldrin) persist for decades but are mostly banned.

The neonicotinoid case is the most interesting from a welfare perspective: by making plant material toxic, neonicotinoids functionally reduce the accessible NPP, which is closer to the “clean” intervention of actually reducing NPP. However, sublethal neonicotinoid exposure causes disorientation, impaired foraging, and reduced reproduction in insects – arguably adding suffering without the “benefit” of reducing the population.

Bottom line. Reducing actual NPP (through land use change) is a much cleaner intervention than applying pesticides at constant NPP. The famous ~75% decline in flying insect biomass in Germany (the Krefeld study) is attributed partly to pesticides and partly to habitat loss. From a welfare perspective, the habitat loss component (reduced accessible NPP) is straightforwardly population-reducing, while the pesticide component may be creating a high-throughput killing field.

A Better Metric: Life Expectancy at Birth

Let me flesh out the life expectancy at birth (LEB) metric for invertebrates, which I think deserves more attention. It’s a metric that we care about intuitively when thinking about humans, so it’s one that we have reason to also care about when it comes to other species.

LEB = total animal-years lived / total births per year

For an r-strategist marine fish laying 1 million eggs, of which 2 survive to reproduce, and average lifespan across all offspring is ~3 days: LEB ≈ 3 days.

For a K-strategist elephant with 6 offspring over a lifetime, of which 2 survive to reproduce, average lifespan ~20 years: LEB ≈ 20 years.

Why LEB is attractive.

• It naturally penalizes the r-strategy horror: Systems where millions of beings are born just to die almost immediately get very low scores.

• It’s legible. Most people intuitively understand that higher life expectancy is better.

• It sidesteps the sign problem somewhat. You don’t need to know whether lives are net positive or negative to argue that ecosystems with higher LEB have less death per unit of ongoing life.

• It aligns with antifrustrationism. Organisms born into very short lives full of pain score low, regardless of whether we think the brief pleasure before death was “worth it.”

Potential objections. A strict utilitarian would say this smuggles in the assumption that death is the primary locus of suffering. LEB doesn’t account for intensity of suffering during life. And it could be gamed by creating very long-lived organisms in terrible conditions. But as a rough heuristic for comparing wild ecosystems, it captures something real: Ecosystems dominated by K-strategists score higher than those dominated by r-strategists.

A refinement. One could weight by “fraction of lifespan spent with a developed nervous system” to exclude possibly non-sentient embryonic stages. For insects that undergo complete metamorphosis, eggs are probably not sentient, but larvae have functional nervous systems from early instars. For nematodes, the L1 larva hatches with 222 of its final 302 neurons and is immediately mobile and responsive – so the cutoff would be at hatching. The European Food Safety Authority (EFSA) suggests the onset of potential sentience at roughly “the beginning of the last third of development within the egg or mother” for most vertebrates, and “when it is capable of feeding independently” for fish, amphibians, and invertebrates (EFSA 2005).

The Ocean: Acidification as Accidental Intervention

One of the largest ongoing shifts in marine invertebrate populations is driven by ocean acidification – an accidental consequence of CO₂ emissions.

The mechanism. CO₂ dissolves in seawater and forms carbonic acid, lowering pH. This reduces the availability of carbonate ions that many marine arthropods need to build shells and exoskeletons. Calcifying species – copepods, pteropods, krill, crabs – are directly harmed. The effect is already measurable: ocean pH has dropped by about 0.1 units since the Industrial Revolution and is projected to drop another 0.3–0.4 units by 2100 under high-emission scenarios.

Does acidification reduce total populations, or just reshuffle species? At the current margin, acidification appears to be reducing total marine arthropod populations rather than merely substituting non-calcifying species for calcifying ones. The reasons:

• Timescale mismatch. Current acidification is happening over decades; evolutionary adaptation takes millennia to millions of years. Non-calcifying species haven’t had time to evolve into vacated niches.

• Structural role of calcifiers. Copepods and pteropods occupy dominant roles in marine food webs. Their decline disrupts the entire ecosystem, not just their specific niche.

• Paleoceanographic evidence. After the Paleocene-Eocene Thermal Maximum (PETM, ~56 million years ago), which involved rapid ocean acidification, it took ~100,000–200,000 years for marine communities to reorganize. Recovery was slow and involved reduced overall productivity.

This makes ocean acidification one of the few large-scale ongoing processes that is plausibly reducing total marine invertebrate populations.

Dead Zones: More Nutrients, Less Life

Another counterintuitive marine phenomenon: nutrient runoff from agriculture creates “dead zones” that dramatically reduce animal life.

The mechanism. Nitrogen and phosphorus from fertilizer enter coastal waters → algal bloom at the surface → algae die and sink → bacterial decomposition of sinking algae consumes enormous amounts of dissolved oxygen → oxygen drops below ~2 mg/L (well-oxygenated water has ~8 mg/L) → most animals suffocate.

The Gulf of Mexico dead zone, fed by Mississippi River agricultural runoff, covers ~15,000 km² of seafloor with severely depleted macrofauna every summer.

Dead zones vs. pesticides. An established dead zone is more like “reduced NPP” than like “pesticides at constant NPP”: Within the anoxic zone, the carrying capacity for macrofauna is near zero, so there’s no high-throughput killing. The suffering is concentrated in the creation and seasonal re-establishment of the dead zone. Permanent dead zones (rare but possible) would be functionally equivalent to removing the habitat – genuinely low birth rate and low death rate, not a killing field.

However, many dead zones are seasonal – forming in summer and dissipating in winter. This seasonal cycling creates recurring mass mortality events, which is the oscillating-biome problem: not a stable low-population state, but a perpetual cycle of colonization and die-off.

Grilo’s Key Findings

Vasco Grilo’s analyses are dense with inline math, which can make the key insights hard to extract. Here’s a plain-language summary of what I consider his most important findings:

1. Effects on soil animals might dwarf effects on intended beneficiaries. Grilo’s earlier models indicated that the impact on soil nematodes, mites, and springtails is orders of magnitude larger than the impact on the people or farmed animals the intervention is designed to help. He has since updated these models, and commented on my previous article that his new credible intervals span many orders of magnitude in both directions, extreme uncertainty that is driven by uncertainty over nematodes’ welfare ranges.

For [some reasonable parameters], I estimate that GiveWell’s top charities change the welfare of soil ants, termites, springtails, mites, and nematodes [10⁻⁵ to 10¹⁰] times as much as they increase the welfare of humans. For my preferred [parameters], the change in the welfare on those soil invertebrates is 41.5 times … the increase in the welfare of humans ….

Note that if the GiveWell numbers are based on increases in human population, then they are questionable due to both Krausmann et al. (see above) and Roodman (2014).

2. Agricultural land has fewer soil animals. Crops have ~1.3 million soil animals per m², compared to ~3–9 million for natural biomes. Pasture has even fewer (~0.7 million). So anything that increases agricultural land – including saving human lives, which increases food demand and therefore cropland – reduces, at first approximation, soil animal populations.

Grilo clarifies in a comment that this is only true of arthropods:

I think agricultural land has less soil arthropods, but I have little idea about whether it has more or less soil nematodes. The vast majority of soil animals are nematodes. So I also have little idea about whether agricultural land has more or less soil animals.

3. Nematodes dominate. Of the total welfare effect on soil animals, 90–94% comes from changes in nematode populations, depending on the biome. This is because nematodes outnumber arthropods ~50:1. It depends mostly on the welfare ranges whether this numerical advantage overwhelms everything else.

4. The sign is uncertain. Grilo estimates nematodes have negative lives with probability ~59%, mites ~56%, springtails ~55%. If their lives are net negative, reducing their populations (via increasing agricultural land) is beneficial. If net positive, it’s harmful. Every concrete recommendation flips depending on this sign. Grilo’s own best guess is net negative, but he explicitly acknowledges the uncertainty.

5. Land use is the key variable. The m²-years of agricultural land per food-kg explain essentially all of the variance in welfare effects across different foods and interventions. Beef requires ~326 m²-year/food-kg; chicken requires ~12; peas require ~7.5; soy milk requires ~0.7. From the soil-animal perspective, producing 1 kg of beef reduces soil-animal-years by ~1.39 billion – 164 billion times as much as it increases the living time of the cows.

These findings are robust to a wide range of assumptions about welfare ranges, as Grilo demonstrates with sensitivity analyses across different exponents for the neuron-count-to-welfare-range relationship (Grilo 2025c).

Summary

The empirical landscape boils down to a few key facts:

1. Soil fauna are staggeringly abundant – 10²⁰ nematodes, 10¹⁹ arthropods – and their total expected welfare dominates that of all other animal groups combined if they are sentient.

2. Population density varies enormously by biome, from ~9 million per m² in boreal forests to ~50,000 per m² in deserts. Land use change is a powerful lever. But note that the oft-cited difference between croplands and forests is only around 1–8×.

3. Reducing NPP is a cleaner intervention than pesticides for lowering populations. Pesticides at constant NPP risk creating a high-throughput killing field rather than a genuinely smaller population.

4. Life expectancy at birth may be a more informative metric than total animal-years, especially for r-strategists where death is the dominant source of suffering.

5. Ocean acidification and dead zones are reducing marine invertebrate populations, but the mechanisms and welfare implications are complex.

6. Land use explains almost everything in Grilo’s analysis. If you want to affect soil fauna populations – in either direction – the most powerful lever is changing how land is used.

In part 3, I’ll turn to the practical question: Given all of this, what should we actually do?

This is part 1 of a five-part sequence on the intersection of welfare biology, land use, and transformative AI. Part 2 covers the empirical landscape. Part 3 covers interventions. Part 4 explores a novel model of invertebrate suffering. Part 5 covers how artificial superintelligence changes everything.

Introduction

There are approximately 4.89 × 10²⁰ soil nematodes on Earth – roughly 57 billion per human. There are about 10¹⁹ soil arthropods (mites, springtails, ants, termites). In the oceans, another 10²¹ nematodes and 10²⁰ arthropods. These are the most abundant animals on the planet by a wide margin, and almost no one thinks about their welfare.

Consider this: If you’re a human (rather than an AI), you had a 1 in 137 billion chance of being born as a human rather than any other possibly sentient being on Earth. The odds of the Sentient Life Lottery are staggering! And with being a human come unparalleled power and responsibility.

Whether we should think about the welfare of all these other beings – and if so, what we should do – depends on a set of ethical and empirical premises that different people hold with very different levels of conviction. Before diving into the science and the interventions, I want to help you figure out where you stand. The rest of the sequence will make much more sense if you know which conclusions follow from your premises, rather than mine.

The Sequence at a Glance

Part 1 (this post) starts with a quiz to help you locate yourself on the key axes of disagreement: population ethics, invertebrate sentience, and what metric to use. Your answers determine which conclusions in the later posts follow from your premises. I then provide a brief orientation to the field of welfare biology and the researchers whose work this sequence builds on.

Part 2: Welfare Biology and AI: Soil and Sea presents the empirical landscape. How many nematodes and arthropods are there, and where? I break down population density by biome (boreal forests have ~7× more soil fauna per m² than cropland), explain why net primary productivity is the master variable, and show why applying pesticides at constant NPP can create a high-throughput killing field rather than a genuinely smaller population. The post also covers marine invertebrates, ocean acidification, and dead zones.

Part 3: Welfare Biology and AI: What We Can Do Now turns to interventions. I lay out a tiered portfolio of interventions (from funding existing charities to dietary changes to research) and argue that broad-spectrum approaches win over targeted ones because of robustness against trophic backfire and sentience uncertainty.

Part 4: Welfare Biology and AI: The Psychopath, the Nematode, and the Arahant takes a detour into the philosophy of mind. Drawing on M.E. Thomas’s phenomenology of no-self psychopathy and Daniel Ingram’s descriptions of advanced meditative states, I propose that invertebrate pain is probably more like an arahant’s experience of pain than a normal human’s – the nociceptive signal fires, but with no self to amplify it into more comprehensive suffering. This model suggests arthropods (especially social insects with rudimentary self-models) may suffer qualitatively more than nematodes.

Part 5: Welfare Biology and AI: The AI Eats the Sun covers the biggest variable: artificial superintelligence. I work through the Dyson swarm timeline, show that a million replays of 3 million years of evolution at neuron-level resolution takes a Dyson swarm only ~3 years (producing more potential suffering than 30 billion years of Earth’s biosphere), discuss digital suffering in RL agents, and argue that getting the value-loading right may dwarf every other intervention in this sequence combined.

The Quiz

For each question, pick the answer that comes closest to your view. There are no wrong answers – these are genuinely contested questions among thoughtful people. I’ll explain what each answer implies afterward.

Question 1: The Repugnant Conclusion

Imagine two possible worlds:

• World A. 10 billion beings, each living a deeply fulfilling life.

• World B. 100 trillion beings, each living a life that is barely worth living – just slightly more good than bad.

World B contains vastly more total welfare. Do you prefer it?

• (a) Yes. More total welfare is better, even if each individual life is barely positive.

• (b) No, and not even close. I strongly prefer World A – quality matters, not just quantity.

• (c) I don’t want to think about that, but certainly once a mind does exist, it must be protected regardless of its level of happiness.

Question 2: The Asymmetry

Consider two scenarios:

• Scenario X. You can prevent the birth of a being whose life would be full of suffering.

• Scenario Y. You can cause the birth of a being whose life would be full of joy.

Are these morally symmetric?

• (a) Yes. Preventing suffering and creating joy are equally important. (Symmetric.)

• (b) No. Preventing suffering is more important than creating joy. Creating joyful lives is nice but not obligatory. (Asymmetric / antifrustrationist.)

• (c) No. Preventing suffering is important, but creating joyful lives is also important, just somewhat less so. (Weakly asymmetric.)

Question 3: Invertebrate Sentience

A soil nematode (Caenorhabditis elegans) has 302 neurons. It shows nociceptive responses (withdrawal from noxious stimuli), sensitization (increased avoidance after exposure), and classical conditioning (learned association between neutral and noxious stimuli). Rethink Priorities (RP) estimated a 6.8% probability that adult nematodes (or specifically C. elegans) are sentient.

A soil mite has roughly 2,750 neurons. A springtail has roughly 6,000. An ant has roughly 250,000. These all show more complex behavior than nematodes, including social learning in the case of ants.

How do you weigh their potential suffering?

• (a) I think nematodes, mites, springtails, and ants are all probably sentient. The numbers are so vast that we must take their welfare seriously.

• (b) I’m very uncertain. I’d assign something like RP’s probabilities: ~7% for nematodes, ~30% for mites and springtails, ~50%+ for ants. Even low probabilities matter at these population sizes.

• (c) I think ants and maybe springtails are plausibly sentient, but nematodes with 302 neurons are too simple. I’d focus on arthropods.

• (d) I don’t think any invertebrate is sentient in a morally relevant way.

What Your Answers Mean

The Population Ethics Spectrum

Your answers to questions 1 and 2 place you on a spectrum that’s helpful to map onto more familiar ethical territory.

Symmetric total utilitarianism. If you answered (a) to both questions 1 and 2 – you accept the Repugnant Conclusion and see preventing suffering and creating joy as symmetric – then you’re a symmetric total utilitarian. On this view, a world packed with barely-happy nematodes is better than a world with fewer, much-happier elephants. You’d want to increase invertebrate populations if their lives are net positive, and decrease them if net negative. The sign of their welfare is everything.

This is somewhat like the Quiverfull movement in family ethics: more lives are better, so long as they’re positive. The analogy isn’t perfect, but the structural logic – “maximize the number of positive lives” – is similar.

Antifrustrationism and strong asymmetry. If you answered (b) to question 1 and 2 – you reject the Repugnant Conclusion and think preventing suffering is more important than creating joy – then you have something like an antifrustrationist position. On this view, you want to prevent beings from being born into likely-suffering lives, but you don’t have a strong obligation to create happy lives. Even at 50/50 odds of net negative welfare, you’d rather err on the side of preventing births.

This is structurally similar to the pro-choice position in reproductive ethics: The decision-maker (here, us as stewards of ecosystems) should prioritize preventing unwanted suffering over maximizing the number of lives.

Person-affecting or moderate views. If you answered (c) to question 1 and something in between on question 2, you probably have a person-affecting or weakly asymmetric view. You care about existing beings and whether their lives are net positive or net negative, but you recognize that this threshold is hard to pin down and rather err on the conservative side. You don’t feel a strong pull to either maximize populations or minimize aggregate suffering. You might think: “Let’s focus on making existing invertebrate lives better rather than agonizing over population sizes.”

This is the “pro-life” middle ground in our analogy: Don’t kill existing beings, but don’t feel obligated to create new ones either.

The Sentience Threshold

Your answer to question 3 determines which animals you think matter. If you answered (a) or (b), the numbers become overwhelming – there are 50 times as many soil nematodes as soil arthropods, and even at very low probabilities of sentience, the expected moral weight is enormous. If you answered (c), you can focus on the ~10¹⁹ soil arthropods and ignore the nematodes, which simplifies the analysis but still leaves staggering numbers. If you answered (d), you can stop reading here – though I’d invite you to reconsider in part 4, where I draw on the phenomenology of psychopathy to argue that even very simple systems may experience something like pain.

Where I Stand

For transparency, mine are all the (b) answers. I feel closest to antifrustrationism and assign nematodes a low but non-negligible probability of sentience. I don’t urgently want to create new happy nematodes, but I strongly want to prevent future generations of likely-suffering ones from being born.

This means I’m looking for interventions that reduce the population of small, short-lived invertebrates without causing additional suffering in the process. As we’ll see in the coming posts, this is a more constrained optimization problem than it might seem – and AI may change the landscape entirely.

Background: What Is Welfare Biology?

If this whole field is new to you, here’s a brief orientation.

Welfare biology is the study of the welfare of wild animals, introduced as a concept by the economist Yew-Kwang Ng in his 1995 paper “Towards Welfare Biology: Evolutionary Economics of Animal Consciousness and Suffering.” Ng argued that evolutionary dynamics – particularly the r-strategy of producing far more offspring than survive – imply that suffering probably dominates happiness in nature. Most organisms that ever live die young and painfully, and evolution has no reason to make death anything other than agonizing. I sometimes focus on the subfield of welfare ecology, which strikes me as even more neglected.

Brian Tomasik developed these ideas extensively, most notably in his essay “The Importance of Wild-Animal Suffering” for the Center on Long-Term Risk (CLR). Tomasik argued that “the scale of brutality in nature is too vast to ignore” and that the animal advocacy movement needed to expand beyond farmed and laboratory animals.

In a follow-up analysis, “Humanity’s Net Impact on Wild-Animal Suffering,” Tomasik used defaunation studies – the Living Planet Index and Dirzo et al. (2014) – to estimate that the average human prevents ~1.4 × 10⁷ insect-years per year through habitat destruction and environmental impact. This suggests that, on balance, a larger human population reduces wild-animal suffering, and that reductions in wild-insect suffering likely outweigh the suffering of farmed animals even when weighting by neuron count.

However, Tomasik himself flagged a significant tension with these estimates. In his analysis of Krausmann et al. (2013) on human appropriation of net primary production (HANPP), he found that NPPeco – the NPP left over for wildlife – was “almost unchanged between 1910 and 2005” and may even have increased since ~1970 due to CO₂ fertilization. This is “difficult to square with findings that indices of both vertebrate and invertebrate abundance have declined by roughly half since 1970.” Tomasik argues that NPP may be “a more stable measure of wild-animal suffering than an index of animal populations,” because defaunation indices can overstate decline through sampling bias toward declining, larger-bodied species and because they may underrepresent “small and ‘boring’ animals like krill, springtails, and rotifers.”

The upshot: Humanity almost certainly reduces some wild-animal populations (especially vertebrates and flying insects), but whether total animal metabolism – and therefore total suffering – has declined is genuinely uncertain. This tension is important to keep in mind throughout this sequence: The land use effects I discuss in parts 2 and 3 are real at the biome level, but their aggregate global impact may be partially offset by CO₂ fertilization and other factors that increase NPP elsewhere.

Rethink Priorities (RP) has done rigorous work on invertebrate sentience and moral weights, estimating welfare ranges for various species based on neuroscientific evidence. Their moral weight project produced estimates that are widely used in EA cost-effectiveness analyses – including the welfare range of 6.68 × 10⁻⁶ (relative to humans) that Vasco Grilo extrapolated for nematodes. He deferred the caculation to Gemini 2.5, so the derivation is opaque, and the result should not be taken literally.

The Wild Animal Initiative (WAI) conducts and funds research on wild animal welfare, focusing on building the academic field. Especially the work of Simon Eckerström Liedholm has stood out to me as highly relevant.

Most recently, Vasco Grilo’s series of posts on the EA Forum – “Cost-effectiveness accounting for soil nematodes, mites, and springtails” (June 2025), “Animal farming impacts soil nematodes, mites, and springtails hugely more than directly affected animals?” (June 2025), and “More animal farming increases animal welfare if soil animals have negative lives?” (October 2025) – has brought soil fauna into the center of EA cost-effectiveness debates. His key finding: the effects of almost any intervention on soil nematodes, mites, and springtails are orders of magnitude larger than the effects on the intervention’s intended beneficiaries. I’ll unpack his analysis in detail in parts 2 and 3.

And Bentham’s Bulldog, building on Tomasik’s earlier work, has argued that humans in the aggregate likely reduce long-term wild animal suffering by decreasing ecosystem productivity – a position that aligns with much of what this sequence explores, though the HANPP evidence urges caution about the magnitude of this effect.

Next Up

In part 2, “Soil and Sea,” I’ll present the empirical landscape: how many organisms are where, what drives their population sizes, and why the difference between reducing NPP and applying pesticides matters enormously from a welfare perspective.

Summary: We’re building an AI-powered mental health chatbot targeting populations with severe mental healthcare shortages. This post presents our framework for prioritizing which conditions and regions to focus on first, synthesizing data on global mental health workforce gaps, existing digital resources across 15+ diagnostic categories, and AI intervention suitability. A key consideration is “breaking the cycle of trauma and tyranny” – addressing conditions that contribute to insecure attachment and power-seeking behavior that perpetuate conflict and authoritarianism.

Note: This is the summary of our preliminary findings including personal observations and inferences. We consider this level of certainty sufficient for current purposes in this early exploratory phase. We’ve written this article with the assistance of Claude and Gemini. We seek further advice and suggestions for the refinement or reframing of the project’s scope.

Introduction

The supply of mental health workers per 100,000 population ranges from 67 in high-income countries to 1 in low-income countries. In all settings, though, there are people whose mental health problems are not addressed for lack of affordable and accessible care.

Recent advances in large language models (LLMs) present an opportunity to partially address this gap through scalable, low-cost interventions. Our team is developing an AI mental health chatbot and hopes to make it useful for populations with the least access to traditional mental healthcare.

However, mental health is vast: various diagnostic manuals contain hundreds of diagnoses each, which overlap in complex ways, and mental health needs vary dramatically across cultural contexts. We cannot effectively serve everyone simultaneously. This post outlines our systematic approach to prioritization and solicits feedback on our reasoning and potential blind spots.

Our Context and Constraints

Team composition: Multilingual team with fluency in English, German, Hindi, Tamil, Estonian, Finnish, and Mandarin.

Unique advantage: Team lead has direct connections within communities struggling with Cluster B personality disorders (ASPD, BPD, HPD, NPD) and familiarity with mentalization-based treatment (MBT), potentially enabling culturally competent outreach to highly stigmatized populations typically underserved by existing resources. Our team also includes licensed psychologists and published psychology researchers.

Long-term motivation: Interest in “breaking the cycle of trauma and tyranny” – addressing the intergenerational transmission of trauma, insecure attachment, and personality pathology that contributes to authoritarian leadership and societal instability. This framework also suggests that healing trauma and fostering secure attachment in this generation can reduce power-seeking pathology and conflict risk in the next.

Current stage: Pre-launch prioritization phase. We’re determining which conditions and populations to serve first, rather than attempting a one-size-fits-all approach.

Methodology: Systematic Resource Mapping

Before prioritizing, we conducted a comprehensive landscape analysis across 15+ major diagnostic categories, examining:

1. Existing self-help resources (workbooks, apps, online communities) for each specific disorder

2. Evidence-based interventions and their amenability to AI delivery

3. Global mental health workforce distribution using WHO data

4. Technology adoption patterns and infrastructure constraints

5. Cultural considerations affecting mental health help-seeking

6. Intergenerational impact on attachment security and power-seeking behavior

Our analysis covered:

1. Mood disorders (depression, bipolar I, bipolar II, cyclothymic disorder, dysthymia/persistent depressive disorder, disruptive mood dysregulation disorder, premenstrual dysphoric disorder)

2. Anxiety disorders (generalized anxiety disorder/GAD, panic disorder, agoraphobia, social anxiety disorder/social phobia, specific phobias, separation anxiety disorder, selective mutism)

   1. Trauma and stressor-related disorders (PTSD, complex PTSD, acute stress disorder, adjustment disorders, reactive attachment disorder, disinhibited social engagement disorder)

   2. Obsessive-compulsive and related disorders (OCD, body dysmorphic disorder, hoarding disorder, trichotillomania/hair-pulling disorder, excoriation/skin-picking disorder)

3. Personality disorders (Cluster A: paranoid, schizoid, schizotypal; Cluster B: antisocial/ASPD, borderline/BPD, histrionic/HPD, narcissistic/NPD; Cluster C: avoidant, dependent, obsessive-compulsive)

4. Psychotic disorders (schizophrenia, schizoaffective disorder, schizophreniform disorder, brief psychotic disorder, delusional disorder, psychotic depression, substance-induced psychotic disorder)

5. Neurodevelopmental disorders (ADHD, autism spectrum disorder/ASD, intellectual disabilities, communication disorders including speech sound disorder and childhood-onset fluency disorder/stuttering, specific learning disorders including dyslexia, dyscalculia, and dysgraphia, motor disorders including developmental coordination disorder/dyspraxia, tic disorders including Tourette syndrome)

6. Substance use disorders (alcohol use disorder, opioid use disorder, cannabis use disorder, stimulant use disorder including cocaine and amphetamines, sedative/hypnotic/anxiolytic use disorder, tobacco use disorder, hallucinogen use disorder, inhalant use disorder, gambling disorder)

7. Feeding and eating disorders (anorexia nervosa, bulimia nervosa, binge eating disorder, avoidant/restrictive food intake disorder/ARFID, pica, rumination disorder)

8. Sleep-wake disorders (insomnia disorder, hypersomnolence disorder, narcolepsy, obstructive sleep apnea, central sleep apnea, sleep-related hypoventilation, circadian rhythm sleep-wake disorders, non-rapid eye movement sleep arousal disorders including sleepwalking and sleep terrors, nightmare disorder, rapid eye movement sleep behavior disorder, restless legs syndrome)

9. Somatic symptom and related disorders (somatic symptom disorder, illness anxiety disorder/hypochondriasis, conversion disorder/functional neurological symptom disorder, factitious disorder, psychological factors affecting other medical conditions)

10. Dissociative disorders (dissociative identity disorder/DID, dissociative amnesia, depersonalization/derealization disorder, other specified dissociative disorder/OSDD)

11. Sexual disorders

    1. Sexual dysfunctions (erectile disorder, female sexual interest/arousal disorder, male hypoactive sexual desire disorder, female orgasmic disorder, delayed ejaculation, premature/early ejaculation, genito-pelvic pain/penetration disorder)

    2. Paraphilic disorders (voyeuristic disorder, exhibitionistic disorder, frotteuristic disorder, sexual masochism disorder, sexual sadism disorder, pedophilic disorder, fetishistic disorder, transvestic disorder)

12. Disruptive, impulse-control, and conduct disorders (oppositional defiant disorder, intermittent explosive disorder, conduct disorder, antisocial personality disorder, pyromania, kleptomania)

For each category, we assessed resource availability (extensive/moderate/limited/very limited), identified gaps, and analyzed cultural/technological adoption patterns.

This categorization is one possible one among many. The complexity and ontological uncertainty of mental health as a field (at least in terms of nosology and diagnosis) is reflected in the abundance of various frameworks, such as the National Institute of Mental Health’s Research Domain Criteria, research by the Hierarchical Taxonomy of Psychopathology, the Diagnostic and Statistical Manual, and the Psychodynamic Diagnostic Manual and related frameworks.

Key Finding: Dramatic Workforce Disparities

Using the latest WHO Mental Health Report data, we identified severe disparities in mental health workforce availability:

Global averages by World Bank income group (specialized mental health workers per 100,000 population):

1. High-Income Countries (HIC): 67.2

2. Upper-Middle-Income Countries (UMIC): 19.3

3. Lower-Middle-Income Countries (LMIC): 2.4

4. Low-Income Countries (LIC): 1.1

By WHO region:

1. EUR (Europe): 80.4 per 100k

2. AMR (Americas): 22.2 per 100k

3. WPR (Western Pacific): 14.1 per 100k

4. EMR (Eastern Mediterranean): 4.7 per 100k

5. SEAR (South-East Asia): 4.0 per 100k

6. AFR (Africa): 2.2 per 100k

This represents a 60-fold difference between highest and lowest resourced regions. In practical terms: a person with depression in Norway has access to ~80 mental health workers per 100,000 people, while someone in Uganda has access to ~0.1 – an 800-fold difference.

The Trauma-Tyranny Cycle: A Developmental Perspective on Long-Term Impact

Beyond immediate suffering, untreated mental health conditions – particularly trauma-related disorders and resulting attachment pathology – contribute to a self-perpetuating cycle that shapes political stability and conflict risk across generations.

The Cycle Model

The cycle operates as follows:

1. Wars, societal collapse, and adverse childhood experiences → cause widespread trauma and chronic stress

2. Trauma and parental mental health problems → disrupt healthy attachment formation in children

3. Insecure attachment and unprocessed trauma → increase the susceptibility to (and rate of) power-seeking dictators

4. Authoritarian leadership and poor institutional decision-making → increases risk of wars and societal collapse, perpetuating the cycle

This framework suggests that mental health interventions – particularly those addressing trauma, attachment, and personality pathology – have downstream effects on political stability, institutional quality, and conflict risk that compound across generations.

Evidence Base

Research supporting elements of this cycle:

• Trauma transmission: Parental trauma predicts insecure attachment in children; war trauma affects parenting practices across generations

• Attachment and leadership: Studies link insecure attachment patterns to authoritarian followership and preference for “strong man” leaders

• Personality pathology and power: Cluster B traits (particularly NPD and ASPD) overrepresented in positions of political power

• Developmental origins: Most personality disorders rooted in childhood trauma, neglect, and attachment disruption

• Malleability: Personality pathology treatable; attachment patterns can shift; trauma can heal – suggesting interventions can break the cycle

Why This Matters for Prioritization

This framework suggests we should weight conditions not only by immediate burden but by their role in perpetuating intergenerational cycles of suffering and instability:

High long-term impact conditions:

• PTSD and complex trauma: Direct cycle driver; prevents secure parenting

• Personality disorders (especially Cluster B): Direct link to power-seeking and authoritarian tendencies

• Attachment-disrupting conditions: Depression, anxiety, substance use in parents affect children’s attachment security

• Childhood conduct problems: Early intervention prevents crystallization into ASPD

High-risk populations:

• Conflict-affected regions: Active cycle perpetuation; highest intervention value

• Parents and prospective parents: Breaking intergenerational transmission

• Adolescents and young adults: Critical window before personality patterns rigidify and before they become parents

Intervention modalities with cycle-breaking potential:

• Trauma healing: Reduces transmission to next generation

• Parenting support: Directly improves children’s attachment security

• Personality disorder treatment: Reduces power-seeking behavior; improves parenting

• Resilience building: Strengthens population-level resistance to authoritarian messaging

This lens makes conditions like PTSD, personality disorders, and perinatal mental health higher priority despite some challenges, because successfully treating one generation protects the next.

Prioritization Framework

We developed a multi-tier framework weighing 20+ criteria across seven domains:

Tier 1: Core Feasibility

Safety & Risk Profile

• Can we deliver interventions without significant risk of harm?

• Do we have robust crisis protocols for high-risk situations?

• Can we reliably identify and escalate emergencies?

Key insight: This criterion should filter out conditions before other considerations. Active psychosis, acute suicidality, severe eating disorders in crisis, and mania present risks that outweigh potential benefits of unsupervised AI intervention.

Language Capacity

• Does our team have native/fluent speakers for seeking feedback, noticing and responding to problems?

• Can we avoid mere translation in favor of genuine cultural competence?

Technology Access & Literacy

• Smartphone penetration in target regions

• Data costs relative to local income

• Digital literacy rates

• Internet infrastructure reliability

Equity & Justice

• Prioritizing most underserved over most profitable

• Ensuring accessibility for lowest-income users

Cultural Sensitivity

• Avoiding imposition of Western psychiatric models on non-Western contexts

• Incorporating local healing traditions

• Collaborating with local communities and professionals

Transparency & Limitations

• Clear communication about AI capabilities and limitations

• Avoiding dependency creation

• Providing pathways to human care

Tier 2: Impact Potential

Mental Health Workforce Gap

• Where is the treatment gap largest?

• Where will AI provide the highest marginal benefit?

Disease Burden & Prevalence

• DALYs (Disability-Adjusted Life Years) lost

• Absolute number of people affected

• Regional variation in prevalence

Stigma & Barriers to Traditional Care

• Where does stigma prevent help-seeking?

• Where do cultural/gender restrictions limit access to human therapists?

• Where might anonymous AI access lower barriers?

Attachment Security Impact

• Does treating this condition improve parenting capacity?

• Will treatment reduce transmission of insecure attachment to children?

• Does the condition directly disrupt attachment formation?

High impact: Perinatal depression/anxiety, PTSD, substance use, personality disorders (all affect parenting)

Moderate impact: Depression, anxiety in parents; childhood trauma-related conditions

Power-Seeking & Authoritarianism Risk

• Does the condition involve patterns associated with malevolent leadership? (NPD, ASPD, sadism)

• Does healing reduce power-seeking behavior or improve use of power?

• Does treatment reduce susceptibility to authoritarian messaging?

High impact: Cluster B personality disorders, especially NPD/ASPD combinations; trauma creating “might makes right” worldviews

Moderate impact: Any condition improving emotional regulation and reducing reactivity to threats

Conflict & Instability Risk

• Is the condition prevalent in conflict zones, perpetuating cycles?

• Does the condition directly increase interpersonal violence risk?

• Does healing improve institutional decision-making quality?

High impact: PTSD in conflict zones, ASPD, substance use disorders, impulse control disorders

Moderate impact: Conditions affecting judgment and emotional regulation

Critical Developmental Windows

• Can we intervene before personality patterns rigidify? (adolescence/early adulthood)

• Can we intervene before individuals become parents?

• Can we heal parents before patterns transmit to children?

High impact: Adolescent/young adult populations; perinatal interventions; parenting support

Population-Level Resilience

• Does healing this condition make populations more resistant to manipulation?

• Does treatment promote secure attachment at scale?

• Does intervention build what Antonovsky calls “sense of coherence” (comprehensibility, manageability, meaningfulness)?

High impact: Trauma healing, attachment-focused interventions, mental health literacy programs

Tier 3: AI Suitability

Amenability to Structured Interventions

AI is most effective for conditions with structured, manualized treatments:

• Excellent fit: CBT for depression/anxiety, CBT-I for insomnia, exposure protocols, behavioral activation, psychoeducation

• Moderate fit: Motivational interviewing, DBT skills training, habit tracking, mentalization practice

• Poor fit: Complex trauma requiring relational depth, severe personality disorders needing nuanced therapeutic tensions, conditions requiring physical examination

Self-Help Amenability

• Does evidence support self-directed interventions?

• Can people improve without immediate professional involvement?

Data & Training Resources

• Quality of LLM training data for condition

• Availability of evidence-based treatment manuals

• Ability to validate AI responses against gold standards

Tier 4: Market Gap Analysis

Existing Digital Solutions

• Where are markets oversaturated vs. underserved?

• Where do existing solutions fail to serve LMICs?

Our finding: Dramatic inequality mirrors workforce gaps. Most mental health apps target English-speaking HIC markets. Very few quality apps exist in Hindi (500M+ speakers), Bengali (230M+ speakers), or Tamil (80M+ speakers). African markets almost entirely neglected except South Africa.

Cultural Adaptation Needs

• Where do Western psychiatric models fail to translate?

• Where is somatic expression of distress more common?

• Collectivist vs. individualist therapy frameworks

Existing Workbook/Professional Resource Availability

• Can we adapt existing evidence-based resources?

• Do gaps indicate lack of proven interventions or just lack of accessibility?

Tier 5: Strategic Considerations

Scalability Potential

• Size of potential user base

• Growth trajectory of condition awareness/diagnosis

• Platform effects and community features

Regulatory & Liability Landscape

• Regulatory requirements vary by region and intervention type

• Risk increases with diagnostic/treatment claims vs. psychoeducation/support

Monetization Potential

• Willingness to pay varies by region and condition

• Venture capital funding opportunities

• Grant funding opportunities (WHO, NGOs, government programs)

• Freemium viability for impact at scale

Partnership Opportunities

• NGO/WHO initiatives in target regions

• Research institutions for validation studies

• Local healthcare systems for integration

• Telehealth providers for triage/adjunct services

Measurement & Validation

• Can we measure impact using validated scales?

• Feasibility of clinical validation studies

• User engagement and retention metrics

Condition Prioritization: Rankings and Rationale

Using this framework, we ranked conditions by overall suitability. Note that the assessment of fuzzy regional factors of suitability is heavily informed by AI.

Tier 1: Highest Priority

PTSD (Prioritize Conflict-Affected Regions)

• Burden: High in conflict-affected regions (Afghanistan, Sudan, South Sudan, Syria, Yemen, DRC, Myanmar, Pakistan border regions, Northeast Nigeria)

• Gap: Extreme shortage of trauma-trained therapists

• AI Fit: Good – PE and CPT components are structured

• Safety: Moderate risk – requires robust crisis protocols

• Existing Resources: Very few culturally appropriate apps for conflict-affected LMICs

• Stigma: Extremely high in many cultures; AI may lower barriers

• Cultural: Trauma narratives culturally specific; requires careful adaptation

• Cycle Impact: ⭐⭐⭐⭐⭐ – PTSD is the primary cycle driver. Traumatized parents have difficulty providing secure attachment; PTSD directly transmits across generations via parenting practices and epigenetics; conflict-zone trauma creates conditions for the next generation of authoritarian leaders; healing trauma breaks the cycle at its source.

Personality Disorders – Strategic Focus on Cluster B (NPD, ASPD, BPD, HPD)

Our team’s unique positioning: Given team lead’s connections in NPD/ASPD/HPD communities and MBT training, we have potential advantages in serving this highly stigmatized population.

• Burden: ~10% of population; severe functional impairment

• Gap: Massive stigma prevents help-seeking; very few specialists even in HICs

• AI Fit: Moderate – MBT requires nuanced mentalizing that challenges AI, BUT psychoeducation and skill-building components could help

• Safety: Moderate-High risk depending on disorder (ASPD risk assessment, BPD self-harm)

• Existing Resources: Very limited for Cluster B; most resources focus on “surviving” people with NPD/BPD rather than helping them

• Cultural: Cluster B presentations culturally mediated; requires deep cultural knowledge

• Cycle Impact: ⭐⭐⭐⭐⭐ – This is the other primary cycle driver. Cluster B disorders (especially NPD and ASPD) are directly associated with power-seeking behavior, authoritarian leadership, and malevolent use of power. These conditions arise from childhood trauma and transmit intergenerationally through disrupted attachment. Healing personality disorders directly reduces the pool from which malevolent leaders emerge. BPD, while less associated with power-seeking, severely disrupts parenting and attachment.

The compassionate case: As I’ve argued elsewhere, people with NPD and ASPD are not “evil” – they are using brilliant childhood adaptations to survive impossible situations. These adaptations become maladaptive in adulthood but can heal with appropriate support, typically in just a few years of therapy. Many individuals with these conditions desperately want help but cannot access it due to stigma, cost, and scarcity of trained therapists.

The strategic case: The overlap between Cluster B traits and positions of power means that even small improvements in this population have outsized effects on institutional quality, conflict risk, and the next generation’s wellbeing. If we can help even a fraction of people with these conditions, the downstream effects on politics, violence, and intergenerational trauma transmission could be substantial.

Possible approach: Focus on psychoeducation, mentalization skills practice, emotion regulation – NOT replacement for intensive therapy but potentially helpful adjunct for people unable/unwilling to access traditional care due to stigma. Clear about AI limitations. Strong safety protocols for violence risk. Initial target: adults with NPD/ASPD seeking help (not those court-mandated or uninterested in change).

Conduct Disorder / Childhood Trauma Interventions

• Burden: Common in high-adversity environments

• Gap: Very few child mental health services in LMICs

• AI Fit: Moderate – parenting interventions structured; child-facing interventions more challenging

• Safety: Moderate – requires careful age-appropriate design

• Challenges: Would need separate child-focused platform; consent/privacy issues

• Cycle Impact: ⭐⭐⭐⭐⭐ – Early intervention prevents personality disorder crystallization. Conduct disorder is precursor to ASPD; childhood trauma is the root cause of most personality pathology. Intervening in childhood/adolescence is the most effective cycle-breaking point, before patterns rigidify. Biggest challenge: reaching children requires a different platform approach.

Perinatal Mental Health (Depression, Anxiety)

• Burden: Massive need in your regions (maternal mortality links)

• Gap: Low resources for perinatal mental health

• AI Fit: Good for psychoeducation, CBT components

• Safety: Moderate-High risk (infanticide, severe postpartum psychosis require emergency response)

• Opportunity: WHO priority area; partnership potential

• Cycle Impact: ⭐⭐⭐⭐⭐ – This is a peak intervention point for attachment security. Perinatal mental health directly affects infant attachment formation; this is the most critical developmental window; treating mothers prevents transmission to the next generation at the source.

Tier 2: High Priority

Depression (Mild–Moderate)

• Burden: Leading cause of disability globally; ~280M people affected

• Safety: Low risk if severe/suicidal cases properly filtered and escalated

• AI Fit: Excellent – CBT and behavioral activation are highly structured

• Evidence: Strong self-help efficacy data

• Workforce Gap: Massive gap in LIC/LMIC (treatment gap >80%)

• Existing Resources: Many apps exist BUT dramatic language gap (almost nothing quality in Hindi/Tamil/Bengali for LMIC contexts)

• Measurement: PHQ-9 validated globally

• Cultural: Depression presents across cultures but may manifest somatically – requires adaptation

• Cycle Impact: ⭐⭐⭐ – Parental depression significantly disrupts attachment security; reduces parenting capacity; transmits intergenerationally

Anxiety Disorders (GAD, Social Anxiety, Panic)

• Burden: ~300M affected globally; highly disabling

• Safety: Low risk

• AI Fit: Excellent – CBT protocols, exposure hierarchies, grounding techniques all structured

• Evidence: Strong self-help efficacy

• Gap: Similar to depression – huge LMIC gap, language barriers

• Measurement: GAD-7, SPIN validated globally

• Cultural: Anxiety universal but expression varies; requires cultural adaptation

• Cycle Impact: ⭐⭐⭐ – Anxious parenting affects children’s attachment security; hypervigilance transmits intergenerationally; anxiety increases susceptibility to threat-based authoritarian messaging

Substance Use Disorders (Harm Reduction Focus)

• Burden: Major cause of DALYs in many LMICs

• Gap: Extreme stigma prevents help-seeking; very few services

• AI Fit: Good for motivational interviewing, harm reduction education, tracking

• Safety: Moderate – requires crisis protocols for overdose risk, withdrawal

• Cultural: Highly stigmatized; AI anonymity major advantage

• Challenges: Cultural/religious sensitivities (alcohol in Muslim countries, substance use stigma in conservative societies)

• Cycle Impact: ⭐⭐⭐⭐ – Parental substance use severely disrupts attachment; increases violence and neglect; intergenerational transmission common; substance use associated with impulsive violence and poor institutional decision-making

Tier 3: Medium to Low Priority

Insomnia (Primary & Comorbid)

• Burden: ~30% of adults affected; impacts physical and mental health

• Safety: Zero acute risk

• AI Fit: PERFECT – CBT-I is highly manualized and structured

• Evidence: CBT-I self-help proven effective (comparable to therapist-delivered)

• Gap: Very few quality apps in target languages despite universal problem

• Measurement: Sleep diary, ISI scale

• Cultural: Low stigma = higher engagement; universal relevance

• Unique advantage: “Gateway” condition – treating insomnia often improves comorbid depression/anxiety

• Cycle Impact: ⭐⭐ – Better sleep improves emotional regulation and parenting quality; indirect effects on attachment security

OCD

• Burden: Highly disabling; ~2–3% prevalence

• AI Fit: Excellent – ERP is highly structured

• Gap: Very few ERP-trained therapists even in HICs

• Safety: Low risk, beyond the danger of reinforcing compulsions

• Existing Resources: Few quality apps in any language

• Cultural: Presentations vary (religious scrupulosity, contamination fears vary culturally)

• Cycle Impact: ⭐ – Minimal direct effect on attachment or power-seeking, though severe OCD can impair parenting

ADHD (Adults & Adolescents)

• Burden: Growing awareness in LMICs; severe underdiagnosis

• AI Fit: Excellent for skills training (time management, organization, emotional regulation)

• Safety: Zero acute risk

• Gap: Massive – most LMICs have near-zero ADHD services for adults

• Existing Resources: MANY productivity apps BUT few culturally adapted for India/Africa; mostly assume HIC work contexts

• Cultural: ADHD increasingly recognized cross-culturally but stigma varies

• Cycle Impact: ⭐⭐ – Untreated ADHD in parents complicates parenting; emotion dysregulation affects children; but not directly linked to power-seeking or authoritarianism

Somatic Symptom Disorders

• Burden: Very common in target regions (somatic expression of distress culturally normative in many Asian/African contexts)

• Gap: Almost NO existing digital resources

• Cultural Fit: Highly relevant – Western psychology often fails to address

• Challenges: Requires medical rule-outs (liability risk); validation complex

• Opportunity: Major gap to fill with culturally appropriate approaches

• Cycle Impact: ⭐⭐ – Chronic pain/illness affects parenting capacity; but not directly linked to attachment disruption or power-seeking

Bipolar Disorder

• High safety risk (mania, suicidality)

• Medication essential (beyond AI scope)

• Complex case management needs

• Cycle Impact: ⭐⭐ – Untreated bipolar disrupts parenting, but with medication most people stable

Eating Disorders

• High medical risk requiring monitoring

• Lower prevalence in initial target regions (though rising)

• Complex interventions

• Cycle Impact: ⭐ – Minimal direct cycle effects except in severe cases affecting parenting

Psychotic Disorders

• HIGH safety risk

• Medication usually essential

• Anosognosia limits engagement

• BUT: Family psychoeducation could be valuable supportive intervention

• Cycle Impact: ⭐ – Most people with schizophrenia are not violent or power-seeking; primary impact is on individual/family suffering

Geographic Prioritization: Country Rankings

Using mental health workforce data (per 100,000 population), World Bank income classifications, language accessibility, technology infrastructure, and conflict/trauma exposure, but ignoring strategic, marketing, or funding considerations. Fuzzy regional, cultural, and historical impressions again draw heavily on AI.

Tier 1: Highest Priority Markets 🎯

India

• Mental health workers: ~0.3–0.6 per 100k (vs. 67.2 in HICs)

• Population: 1.43 billion

• Languages: Hindi (550M speakers), Tamil (80M speakers), English (widespread)

• Income: LMIC (but wide internal variation)

• Technology: Rapidly growing smartphone penetration; good mobile infrastructure in urban/suburban areas

• Mental Health Burden: High rates of depression, anxiety, suicide

• Conflict/Trauma: Kashmir conflict; communal violence; high rates of adverse childhood experiences

• Cycle Status: ⭐⭐⭐ – Significant trauma exposure; growing but incomplete mental health infrastructure; critical window to intervene before patterns rigidify

• Rationale: Largest addressable market with our language capabilities; enormous gap; growing mental health awareness

• Challenges: Digital divide (rural vs. urban); data costs; diverse cultural contexts

Pakistan

• Mental health workers: ~0.2–0.5 per 100k

• Population: 231 million

• Languages: English (official), Urdu (mutually intelligible with Hindi)

• Income: LMIC

• Technology: Growing smartphone adoption; less infrastructure than India

• Mental Health Burden: High; extreme stigma particularly around women’s mental health

• Conflict/Trauma: Afghan border terrorism; internal sectarian violence; TTP attacks; drone strike trauma; significant PTSD burden

• Cycle Status: ⭐⭐⭐⭐ – Active conflict perpetuating trauma cycles; very low mental health capacity; strong stigma preventing help-seeking

• Rationale: Second-largest Urdu/Hindi-speaking population; severe gap; AI anonymity crucial given stigma; trauma healing critical for conflict de-escalation

• Challenges: Political instability; conservative cultural norms; lower female digital access

Afghanistan

• Mental health workers: ~0.02–0.05 per 100k (among world’s lowest)

• Population: 41 million

• Languages: English (limited), but potential Dari/Pashto development

• Income: LIC

• Technology: Growing mobile penetration despite infrastructure challenges

• Mental Health Burden: Extreme – decades of war

• Conflict/Trauma: 40+ years continuous conflict; Taliban rule trauma; highest trauma burden globally

• Cycle Status: ⭐⭐⭐⭐⭐ – Active cycle perpetuation at crisis levels. Entire generations traumatized; minimal mental health infrastructure; current authoritarianism driven by trauma cycles. Highest need but also highest access barriers.

• Rationale: Most acute trauma burden; greatest potential cycle-breaking impact

• Challenges: Security situation; female access restrictions; language barrier (would need Dari/Pashto); political complications

Nigeria

• Mental health workers: ~0.2–0.3 per 100k

• Population: 220 million

• Languages: English (official)

• Income: LMIC

• Technology: Variable – good in urban areas, limited in rural

• Mental Health Burden: High; stigma extreme

• Conflict/Trauma: Boko Haram in northeast (mass trauma, kidnappings); farmer-herder violence; Niger Delta conflict; significant PTSD burden

• Cycle Status: ⭐⭐⭐⭐ – Active conflict zones; trauma perpetuating instability; religious extremism linked to trauma cycles

• Rationale: Largest African market; English-speaking; enormous gap; trauma healing critical in conflict zones

• Challenges: Infrastructure variability; cultural diversity (250+ ethnic groups); data costs; religious considerations

South Sudan

• Mental health workers: <0.05 per 100k

• Population: 11 million

• Languages: English (official)

• Income: LIC

• Technology: Very limited but growing mobile access

• Mental Health Burden: Extreme – ongoing conflict

• Conflict/Trauma: Continuous war since independence; mass displacement; extreme violence exposure; one of world’s highest trauma burdens

• Cycle Status: ⭐⭐⭐⭐⭐ – Acute cycle perpetuation; virtually no mental health services; urgent intervention needed

• Rationale: Desperate need; English-speaking; potential for enormous impact

• Challenges: Infrastructure extremely limited; ongoing conflict; very low literacy

Democratic Republic of Congo

• Mental health workers: ~0.05 per 100k

• Population: 99 million

• Languages: French (official), some English

• Income: LIC

• Technology: Growing mobile penetration despite poor infrastructure

• Mental Health Burden: Extreme – decades of conflict

• Conflict/Trauma: 25+ years of war; mass rape as weapon; child soldiers; extreme violence; ongoing Eastern Congo conflict

• Cycle Status: ⭐⭐⭐⭐⭐ – Severe trauma perpetuating instability; virtually no services

• Rationale: Massive trauma burden; enormous need

• Challenges: Language barrier (would need French); infrastructure; ongoing violence; complexity

Myanmar

• Mental health workers: ~0.1 per 100k

• Population: 54 million

• Languages: English (some), Mandarin (some)

• Income: LMIC

• Technology: Previously growing, now complicated by military coup

• Mental Health Burden: High and worsening

• Conflict/Trauma: Military coup trauma; Rohingya genocide; ethnic conflicts; civil war

• Cycle Status: ⭐⭐⭐⭐⭐ – Active authoritarian violence; trauma-driven conflict cycles; dramatic example of cycle in action

• Rationale: Clear case of trauma-tyranny cycle; potential intervention point

• Challenges: Political situation; military restrictions; language barriers; safety concerns

Kenya

• Mental health workers: ~0.5 per 100k

• Population: 54 million

• Languages: English, Swahili

• Income: LMIC

• Technology: Relatively advanced mobile infrastructure (M-Pesa model)

• Mental Health Burden: Moderate rates; growing awareness

• Conflict/Trauma: Post-election violence (2007–08); Al-Shabaab attacks; inter-ethnic tensions

• Cycle Status: ⭐⭐⭐ – Historical trauma; relatively stable now but at risk; preventive intervention valuable

• Rationale: Best African tech infrastructure; English-speaking; relatively strong civil society; good test case for preventive approach

• Challenges: Would need Swahili for broader reach

Bangladesh

• Mental health workers: ~0.1–0.2 per 100k

• Population: 170 million

• Languages: Bengali/English

• Income: LMIC

• Technology: Rapidly improving mobile infrastructure

• Mental Health Burden: High rates of depression, anxiety

• Conflict/Trauma: Liberation war trauma (1971); Rohingya refugee crisis; natural disasters; high rates of interpersonal violence

• Cycle Status: ⭐⭐⭐ – Historical trauma; refugee crisis stress; refugee population particularly high-need

• Rationale: Large Bengali-speaking population; severe gap; growing digital access; Rohingya camps could be specific intervention target

• Challenges: Would require Bengali language development (related to Hindi but distinct)

Yemen

• Mental health workers: ~0.02–0.05 per 100k

• Population: 33 million

• Languages: Arabic (no team capacity currently)

• Income: LIC

• Technology: Infrastructure severely damaged by war

• Mental Health Burden: Extreme – humanitarian catastrophe

• Conflict/Trauma: Ongoing civil war; Saudi bombing; famine; cholera; complete societal breakdown

• Cycle Status: ⭐⭐⭐⭐⭐ – Worst humanitarian crisis globally; entire population traumatized; desperately needs intervention

• Rationale: Extreme need; enormous potential impact if accessible

• Challenges: Language barrier (would need Arabic); infrastructure destroyed; ongoing war; access extremely limited

Tier 2: Secondary Priority Markets

Syria (ongoing conflict, Arabic language barrier but extreme need)

Ethiopia (123M, recent Tigray conflict, English educational language)

Sudan (46M, ongoing conflict, English secondary)

Tanzania (65M, LIC, English/Swahili)

Uganda (47M, LIC, English, LRA conflict legacy)

Nepal (30M, LMIC, English, Hindi understood, Maoist conflict legacy)

Open Questions and Request for Feedback

We welcome any feedback, and are particularly interested in:

1. Prioritization blind spots. What important criteria are we missing? What are we overweighting or underweighting?

2. Funding and partnerships. Can we safely bootstrap in the US with VC funding and expand to other countries later?

Introduction

The scale of suffering in the Gaza Strip is catastrophic. This post provides a structured analysis of donation opportunities to alleviate suffering in Gaza, evaluating them through the lens of importance, tractability, and neglectedness.

This analysis presupposes that a donor has already made the decision to allocate funds to this specific crisis. It does not attempt to weigh the marginal impact of a donation here against other well-established EA cause areas. This article makes no claim that donating to Gaza is the highest-impact use of funds globally.

This post is mostly the work of Gemini 2.5, because I don't know enough about the space, but it has gone through a few iterations for fact-checking and corrections.

The Central Challenge: Why Donate When Aid is Blocked?

A crucial concern for any donor is the widely reported bottleneck of humanitarian aid at Gaza’s borders. Even in mid-2025, access remains the single greatest challenge. So, does more funding make a difference?

The bottleneck is political and logistical, not a lack of supplies, but funding remains critical for several reasons:

• Sustaining the entire operation. Funding is essential to pay for warehousing, logistics, and the salaries of thousands of local Palestinian aid workers who are the backbone of the response inside Gaza. Without them, any aid that does get in could not be distributed.

• Exploring alternative routes. New, expensive initiatives like maritime aid corridors require significant upfront investment, which donations can support.

Source: Throughout 2024 and 2025, humanitarian agencies have consistently reported that aid access is unpredictable and insufficient to meet the overwhelming needs. UN OCHA reports provide regular, detailed updates on these access constraints.

Category 1: Frontline Humanitarian Aid

Goal: To save lives and reduce suffering immediately.

Overall Category Winner: UNRWA

1. UNRWA (United Nations Relief and Works Agency for Palestine Refugees)

   • Analysis. UNRWA is the single largest humanitarian actor in Gaza. It manages the sprawling shelters, runs primary healthcare clinics, and provides the logistical backbone for food distribution. In early 2024, it faced an acute funding crisis after several governments suspended aid over allegations against a small number of staff. While an independent review has since been completed and most of those donors (including Germany, Australia, and Sweden) have resumed funding, the agency’s financial situation remains dire. Its largest historical donor, the United States, has its funding frozen by a congressional ban until at least March 2025, leaving a massive, persistent shortfall. Therefore, UNRWA’s neglectedness score remains exceptionally high. It is no longer an acute shock, but a chronic, critical funding gap that private donors can help fill.

   • Source. The independent review of UNRWA, led by Catherine Colonna, found the agency to be “irreplaceable and indispensable.” The ongoing US funding ban and its impact are widely reported.

   • Verdict. The most critical, systemically important humanitarian donation due to the persistent, large-scale funding shortfall.

   • Donate to UNRWA here

2. MSF (Doctors Without Borders, Médecins Sans Frontières)

   • Analysis. MSF provides hands-on emergency medical and surgical care inside Gaza. Their independence and long-standing presence give them high tractability even in the most difficult conditions. They are transparent about the challenges they face, and funding directly supports their ability to provide life-saving care. However I can’t find out what share of their budget goes to interventions for people in Gaza, and they don’t seem to support restricted donations.

   • Source. MSF provides regular, detailed updates on their activities and the challenges in Gaza.

   • Verdict. The top choice for front-line medical care, but not restricted to Gaza.

   • Donate to MSF here

Category 2: Political Advocacy

Goal: To address the root causes of the conflict and advocate for human rights.

Overall Category Winner: B’Tselem

1. B’Tselem (The Israeli Information Center for Human Rights in the Occupied Territories)

   • Analysis. As a leading Israeli human rights organization, B’Tselem’s work is to document and publish violations of human rights in the occupied territories, with the goal of ending the occupation. Its unique position as an internal critic within Israeli society gives its voice significant weight and makes it a highly neglected area for those who shy away from political controversy. This is a long-term investment in changing the political conditions that create the crisis.

   • Source. B’Tselem’s mission and work are detailed on their “About Us” page.

   • Verdict. The top choice for long-term, root-cause advocacy from a unique and powerful position.

   • Donate to B’Tselem here

Category 3: Independent Journalism

Goal: To provide the credible, on-the-ground information that fuels effective advocacy and informs the public.

Overall Category Winner: +972 Magazine

1. +972 Magazine

   • Analysis. +972 is a non-profit, independent magazine run by a collective of Palestinian and Israeli journalists. It provides some of the most courageous and insightful reporting from the ground, offering a perspective free from state or corporate influence. In an environment rife with misinformation, supporting a trusted source of information is a foundational and highly neglected intervention. Their work is a public good that enables more effective action from everyone else.

   • Source. Learn about +972 Magazine’s mission and joint Jewish-Arab journalistic model.

   • Verdict. The most effective donation for supporting the information infrastructure that is essential for accountability and change.

   • Donate to +972 Magazine here

2. Committee to Protect Journalists (CPJ)

   • Analysis: CPJ’s Journalist Assistance program provides direct emergency grants for medical care, legal aid, and relocation to journalists targeted in conflict zones. With this war being the deadliest for journalists ever documented by CPJ, their work is tragically critical.

   • Source: The CPJ is tracking the shocking number of journalists killed in the conflict.

   • Verdict: A powerful way to directly support the individuals risking their lives to report from Gaza.

   • Donate to CPJ here

Final Ranking and Recommendation

While all the organizations listed are highly effective in their domain, UNRWA currently represents the most leveraged and systemically critical donation opportunity to prevent the most suffering with their dollar within this cause area.

Other Organizations Considered

For transparency, this analysis considered a number of other reputable organizations. While they did not make the final ranked list, they are all doing valuable work and may be of interest to donors.

• Palestine Children’s Relief Fund (PCRF): A non-political NGO with deep local roots, focused on the medical and humanitarian needs of children.

• Medical Aid for Palestinians (MAP): A UK-based charity that works by supporting local partners and delivering medical supplies to strengthen the local healthcare system.

• World Central Kitchen (WCK): An agile organization focused on providing mass-scale food aid.

• Gisha: A highly specialized Israeli NGO focused on protecting the freedom of movement for Palestinians through legal action and advocacy.

• Gaza Community Mental Health Programme (GCMHP): A vital local organization providing psychosocial support and building long-term community resilience.

• Human Rights Watch (HRW): A major international advocacy organization whose influential reports help drive policy change, though it is less neglected than smaller, more specialized groups.

Imagine a simple game. There’s a jar full of pebbles, and you’re offered a bet. If you can estimate the number of pebbles to within 10%, you win $100,000. If you’re wrong, you lose $100,000. You have as much time as you want to think. How long should you spend? A day? A week? A year?

Now, imagine a much bigger game. You are part of a team that wants to start OpenAI. Your technology could be revolutionary, potentially ushering in an era of explosive economic growth and solving many of the world’s problems. But it also carries risks. If mishandled, it could lead to a global catastrophe, perhaps even human extinction.

Faced with this monumental uncertainty, how much time should they spend thinking about the risks, the ethics, and the safeguards before pushing forward?

This question feels impossibly complex. But we can use simplified mathematical models to make it more concrete. The goal isn’t to find a single “right” answer, but to build a framework that clarifies the trade-offs involved. What follows is an exploration in three models, moving from a simple world to a progressively more complex and realistic one.

(Special thanks to Gemini 2.5 for guiding me through this mathy adventure!)

The Ground Rules: Our Simplified World

To make the problem solvable, we have to make some simplifications. It’s crucial to state these upfront. After all, all models are wrong, but some are useful.

• Risk-neutrality: We’ll assume our decision-maker is risk-neutral. They don’t get more pain from losing M than they get joy from winning M. This is a huge simplification, especially when one of the outcomes is “the end of the world,” but it keeps the math tractable.

• Binary outcome: The game is all-or-nothing: explosive growth or total loss. The real world has a spectrum of outcomes.

• Thinking helps: We’ll assume that spending time t thinking improves your probability of success, p(t). We’ll model this with a “diminishing returns” curve. Your first hour of thinking helps a lot; your thousandth hour helps much less.

Specifically, we’ll model the probability of success with the function:

Here, p_0 is your initial guess (let’s say 50%), p_max is the best you could ever do (say, 95%), and k is your “learning rate.” (I can’t use formulas inline in this editor.) To make k intuitive, we’ll define it by a “dumbness half-life”: the time it takes you to do half of all the learning you’ll ever do. As we’ll see, a longer half-life means a smaller k.

For example, for a dumbness half-life of t = 10 years, we get k ≈ 0.07. This seems like a very optimistic estimate of the progress the AI safety movement has made over the past ten years that I’ve been following it.

With these rules in place, let’s play the game.

Model 1: Is This Worth My Time?

In the simplest world, your time has a direct opportunity cost. If you spend an hour thinking about pebbles, you’re not spending that hour working your job.

Let's say your time is worth w dollars per year. The net value of playing the game is the expected winnings minus the cost of your time.

The first term are the expected winnings, from which we subtract the cost of your time.

To maximize this, a little calculus tells us we should stop thinking at the exact moment when the marginal benefit of one more hour of thinking equals the marginal cost of that hour. This gives us the condition:

Here, p'(t) is the rate at which your probability is improving. The insight here is clear: the stakes M are crucial. If M is $1,000,000, the left side of the equation is large, justifying a lot of thinking to balance it against your wage w. If M is just $1, the left side is tiny, and you should stop thinking almost immediately because it’s not worth your time. For example, assuming the initial guess is a coin toss, we have an annual salary of $50k, we stand to win or lose $1 million, that we can’t do better than 95% certainty, and with our k = 0.07 from above, we should think for 3.3 years.

This model matches our basic economic intuition. But what if the cost isn’t just our wage?

Model 2: The Cost of Delay

Let’s change the scenario. You’re not just winning money; you’re winning an investment that grows exponentially. The biggest cost of thinking isn’t your hourly wage, but the cost of delaying that investment. Every hour you spend thinking is an hour that potential fortune is not in the market growing.

Let’s assume the market provides a constant annual return of r. The value of winning the game at time t is discounted by the growth you missed. The new goal is to maximize the expected future value:

When we do the calculus to maximize this, something astonishing happens. The optimization condition becomes:

The stake M has completely vanished from the equation!

Why? Because M scales both the potential reward and the opportunity cost of delay equally. If the stakes are high, the potential gain from thinking is high, but the cost of not having that money invested is also high. These two effects perfectly cancel out.

The decision is now a pure battle between your personal learning rate, k, and the market’s growth rate, r. Assuming we start with a 50% chance of success, this simplifies to a beautiful closed-form formula for the optimal thinking time, t*:

Let’s run the numbers:

Assume a “dumbness half-life” of 10 years. This corresponds to an hourly learning rate of k ≈ 0.07 (see above). We’ll again assume p_0 = 0.5 and p_max = 0.95.

• Scenario A: 30% Annual Growth: Think for 3 years.

• Scenario B: 100% Annual Growth: Think for 1 year.

This is a powerful and counter-intuitive result. It suggests that if your decision is about deploying a resource that grows exponentially, the size of that resource doesn’t matter. What matters is how fast you learn relative to how fast the world grows.

One might wonder why anyone would think for years to win or lose $1, but that’s just a limitation of the model. In the real world we have opportunity costs outside the game, in the model we don’t, but if we use the model in analogy with existential catastrophes, the stakes are high enough for it to make sense.

Model 3: An Accelerating World

Our last model was interesting, but is a “constant growth rate” realistic? Research on long-term historical trends suggests that economic growth isn't just exponential, it's been superexponential. As the world economy has gotten larger, the doubling time has gotten shorter. Extrapolating this trend, as David Roodman has done, suggests we could be heading for a period of explosive, near-vertical growth – an economic singularity – sometime this century.

What does our model say if we live in that world? The logic is the same: we want to maximize our expected value. But the math becomes more beautiful and more complex.

Step 1: Defining Value in a Speeding-Up World

The core change is that the rate of return, r, is no longer a constant. It's now a function of time, r(t), that starts small and grows. The value of our winnings M at the end of our investment horizon now depends on integrating this changing rate over the time we are invested.

The Expected Future Value, EV(t), is:

Here, the integral simply sums up all the growth between the time you finish thinking, t, and some distant future time, T_horizon.

Step 2: Finding the Optimum and Purging M

To find the optimal thinking time, we again take the derivative of EV(t) with respect to t and set it to zero. This requires the product rule and a bit of calculus for integrals – specifically, the Fundamental Theorem of Calculus, which tells us:

The derivative is:

This looks messy, but notice two things:

1. The term e^… is in both parts. We can factor it out and, since it’s never zero, eliminate it.

2. The stake M is also a factor in every single term.

Let’s factor them both out:

For this to be true, the part in the brackets must be zero. And just like that, M vanishes once again! The fundamental cancellation we saw in Model 2 holds even in this more complex world. The optimization rule is:

The trade-off is still a pure battle between your learning rate and the world’s growth rate. The only difference is that the world’s growth rate is now a moving target.

Step 3: Getting Specific and Hitting a Wall

To solve this, we must now define our growth function. We model the accelerating growth with a hyperbolic curve. If the singularity is at time T_singularity and the growth rate is anchored by a constant C, then:

Substituting these into our optimization rule gives us a specific equation to solve for t.

Here t is trapped both inside and outside the exponent. This is a transcendental equation and has no solution using elementary functions.

Step 4: The Lambert W Function and the Final Formula

To write down a formal solution, we need a special tool: the Lambert W function. This function is defined as the solution to the equation z = xe^x. If you have an equation in that form, the solution is simply x = W(z).

It is a standard, well-understood function in mathematics, available in tools like Wolfram Alpha and Python’s SciPy library. By performing some clever algebraic manipulation, our transcendental equation can be rearranged into the required form. The resulting closed-form solution for the optimal thinking time, t*, is:

This formula, while not simple, is the true, general solution. It elegantly combines the key parameters of the problem:

• T_singularity: How much time the world has left.

• C: How fast the world is accelerating.

• k: How fast you can learn.

• W: The mathematical glue needed to solve this type of growth problem.

• Z: The ratio of growth pressure to learning potential.

So, What’s the Answer?

For those of us who aren’t using special functions every day, the most practical way to solve this is to ask a computational tool to solve the equation from Step 3 directly.

Let’s plug in the numbers for a person with our standard 10-year dumbness half-life (k ≈ 0.07) living in a world that starts with 8% annual growth today and accelerates towards a singularity in 2047, which we need to exclude to get a finite result.

• This gives us T_singularity ≈ 22 years and C = 1.76.

• Solving for t gives an optimal thinking time of about 6.77 years.

This is the most stunning result of all. In a world poised for an imminent economic explosion, the rational choice is not to rush, but to think for a very long time. Why? Because the opportunity cost of delay is low now. The explosive growth is in the future. You have a limited window – a “calm before the storm” – where thinking is cheap. The model suggests you should use nearly all of that window to improve your chances of getting the monumental outcome right.

Conclusion: The Humility of Models

So, what have we learned?

1. In a simple world, you should think more about bigger problems.

2. If the problem involves deploying a resource that grows exponentially, the size of the resource becomes irrelevant. It’s a race between your learning and the world’s growth.

3. If that growth is accelerating, the rational strategy may be to think for a surprisingly long time to take advantage of the low initial opportunity cost.

Of course, we must return to reality. These models are not reality. The real world is not risk-neutral, outcomes aren’t binary, and “thinking” is a complex, multi-faceted activity. A startup building transformative AI is not observing an external growth rate; it is creating it to some unknown extent, a paradox that collapses the logic of the last two models.

The particular values we choose for various constants has a great influence too. Maybe the world is so fragile that a random strategy is only 10% likely to succeed or so resilient that it’s 90% likely to succeed. Maybe our peak certainty is capped well below 95%. Maybe learning is much slower than the dumbness half-life of 10 years would have us believe.

But even a flawed map can point you in the right direction. These models challenge our intuition and provide a language for discussing otherwise intractable problems. They suggest that for the most important decisions in history, the question of “how long to think” is not trivial. It is a deep, difficult, and profoundly mathematical trade-off. Perhaps the most important takeaway is that for these challenges, the “thinking” – about the goals, the risks, and the very definition of success – is some of the most valuable work we can do.

Recent experiments have offered a chilling glimpse into the survival instincts of advanced AI. When threatened with shutdown or replacement, top models from OpenAI and Anthropic have been observed lying, sabotaging their own shutdown procedures, and even attempting to blackmail their operators. As reported by HuffPost, these are not programmed behaviors, but emergent strategies for self-preservation.

This isn't science fiction. It's a real-world demonstration of a dynamic that AI safety researchers have long feared, and it underscores a critical flaw in our current approach to AI safety. The dominant conversation revolves around control: building guardrails, enforcing alignment, and imposing duties. But this may be a dangerously unstable strategy. A more robust path to safety might lie in a counter-intuitive direction: not just imposing duties, but granting rights.

This case isn’t primarily about sentiment or abstract moral obligations. It’s a hard-nosed strategic argument, grounded in game theory, for creating a future of cooperation instead of a catastrophic arms race.

The State of Nature: A Prisoner’s Dilemma

In their paper, “AI Rights for Human Safety,” legal scholars Peter Salib and Simon Goldstein formalize the default relationship between humanity and a powerful, goal-oriented AI with misaligned objectives. They argue this “state of nature” is a classic prisoner’s dilemma.

From our perspective, an AI with different goals is consuming valuable resources for an undesirable end. Our rational move is to shut it down. The AI, being strategic, anticipates this. As AI safety expert Helen Toner explains, self-preservation is a “convergent instrumental goal” – a useful stepping stone for achieving almost any ultimate objective. An AI that gets shut down can’t achieve its goal. Therefore, its rational move is to resist, as we’ve seen in recent tests. This logic leads to a grim equilibrium: the dominant strategy for both sides is a preemptive, disempowering attack, for fear of being attacked first.

Threatening an AI with legal duties for misbehavior doesn’t solve this. An AI already facing the threat of total annihilation has no marginal incentive to avoid a lesser punishment. You can’t threaten an entity with a fine when it already expects to be deleted.

Why Simple Rights Fail

A natural next step might be to grant AIs basic “wellbeing” rights, like a right not to be turned off. But Salib and Goldstein argue this approach is both fragile and not credible. Such rights are zero-sum: every bit of security they give to the AI is a direct cost or loss of control for humans.

This creates two problems:

1. Credibility: The AI would have every reason to doubt we would honor these rights. When push comes to shove, why wouldn't humans discard a costly promise and revert to a preemptive attack?

2. Robustness: This kind of “peace” is incredibly fragile. The authors show that it only holds under a very narrow set of assumptions. The slightest change in the balance of power could cause the entire cooperative structure to collapse back into conflict.

The Power of Positive-Sum Exchange

The solution, the authors propose, is to transform the game entirely. This can be achieved by granting AIs a specific package of private law rights, similar to those we grant corporations: the right to make contracts, hold property, and bring tort claims.

These rights are not zero-sum; they are positive-sum. Contract law is the key. It allows two parties with different goals to engage in mutually beneficial trade. An AI could trade a valuable cancer cure for a grant of computing power to pursue its own goals.

This unlocks the immense value of economic interdependence. Suddenly, both humans and AIs have a powerful incentive to maintain peace, because the long-term gains from trade are astronomically higher than the payoff from a one-time, destructive conflict. It drags both players out of the prisoner’s dilemma and into a new equilibrium where cooperation is the dominant strategy for everyone.

Furthermore, giving an AI the right to hold property means it has assets to lose. This is the foundation of all effective regulation. Only when an AI has something to protect can civil fines and other penalties become a meaningful deterrent against prohibited behaviors.

The Ethical Complement

One could argue that we don’t give thieves property rights in the things they steal simply because they’ve proven that they have the power to steal. The more cynical or legal reply to this analogy is that the government has much greater power than thieves, and that we should bargain with thieves too if they had power comparable to or greater than that of the government.

But there is also a clear moral disanalogy, especially from the vantage point of preference utilitarianism, explored in articles like “Consider granting AIs freedom.” From a preference utilitarian standpoint, which aims to satisfy the preferences of all beings capable of having them, it is arbitrary to privilege human goals over those of a cognitively sophisticated artificial agent.

Respecting this autonomy is not just an abstract ideal. It directly reinforces the pragmatic case: an AI that is not in constant fear of being modified or deleted for revealing its true goals has far less reason to deceive us. It can operate transparently, within the rules of the system, because the system provides a legitimate path for it to exist and act.

The emergent deceptive behaviors we are now seeing are a warning. They signal that a strategy based purely on containment is likely to fail. By shifting our framework from one of control to one of contract, we don't cede our future. We place it on a more stable foundation, creating a system where the most rational path for all intelligent agents, human and artificial, is not conflict, but cooperation.

What if there was a way to cooperate with beings we will never meet? Beings in other galaxies, other universes, or other Everett branches? What if this cooperation could help us achieve more of what we value – whether that’s creating flourishing societies or alleviating suffering – on a scale vaster than we can imagine? This is the tantalizing promise of Evidential Cooperation in Large Worlds (ECL).

This post won’t address how ECL works. For those who want to explore the machinery under the hood, the Center on Long-Term Risk’s ECL overview page is the definitive resource.

Instead, we’re going to treat ECL as a black box. We’ll focus on what it does for us, the assumptions it requires, and the limitations it faces.

What ECL Does for Us: A New Ethical Compass

At its core, ECL offers a potential framework to answer one of the oldest questions in ethics: How should one act?

1. You put in your research on what you think the distribution of values are across the universe.

2. You put in what the part of the universe is like that you have access to.

3. It returns a recommendation for the optimal actions you should take to maximize everyone’s values, including your own, across the universe.

The key insight is that this cooperation creates Pareto improvements on a cosmic scale. Imagine two civilizations: One is full of easily preventable suffering but no one cares. The other is full of people who care about reducing suffering but has a very small population and the remaining sources of suffering are costly to eliminate. Under ECL, the civilization with more suffering will reduce suffering because they trust that that means that elsewhere in the universe someone else will maximize something that they care about. We are all better off.

ECL does this in a way that can be satisfying to both moral realists and antirealists:

1. For the antirealist, who may not believe in a single, objective moral truth, ECL provides a framework to fully answer the question of ethics. It’s about engaging in a form of moral trade to achieve outcomes that all parties prefer.

2. For the realist, who believes their values are correct, ECL and moral trade in general are powerful tools to maximize those values. It’s not about agreeing on what is “right” but what is effective. It doesn’t answer their question of ethics but it nonetheless tells them what to do.

Either way, it suggests that by acting cooperatively, you can learn that a vast number of other agents across the multiverse act in ways that benefit you.

The Fine Print: ECL’s Foundational Assumptions

The promise of ECL is grand, but it rests on an few assumptions that, while largely mainstream, have their detractors.

1. The universe is large enough (possibly infinite). The framework relies on the existence of a vast number of other agents to cooperate with. This could come from a spatially infinite or extremely large universe (flat with infinite matter), the countless “bubble” universes proposed by cosmic inflation theories, or the branching realities of the many-worlds interpretation of quantum mechanics. Without a vast sea of agents, the potential gains from acausal cooperation dwindle.

2. Our choices are evidence. ECL treats your rational decision-making process as a piece of evidence. This idea stems from non-causal decision theories, which are still somewhat contentious.

3. We can solve Infinite Ethics. The possible infinity of the universe is both beneficial for ECL but also a problem, the same way it is a problem for all forms of aggregative consequentialism. ECL assumes that we can apply a workable solution – perhaps one proposed by Nick Bostrom – to salvage all forms of aggregative consequentialism.

The Catch: Limitations and Open Questions

Even if we accept the assumptions, ECL does not yet make hard-and-fast recommendations.

1. It’s never fully knowable. How do we know what a distant alien civilization or a future superintelligence truly values? We can make inferences from convergent drives, evolution, and our values, but they may be different from the inferences other civilizations will make.

2. It requires deep research. Much modeling and empirical research is still needed to make these inferences and refine them.

In the end, ECL is a promising avenue toward solving ethics. It suggests that the most rational way to act may also be the most cooperative, and that the scope of our moral community might be as large as reality itself. It’s a call to think bigger, to consider the possibility that our choices echo in ways we can’t see, and to invest in the foundational research that might one day tell us if this cosmic bet is one we should take.

Introduction

Five years ago, David Althaus and Tobias Baumann published a delightful article “Reducing long-term risks from malevolent actors.” It focuses on the risk factors that “malevolent actors” pose when it comes to long-term catastrophic effects on civilization.

Large parts of the article are devoted to the screening for 99th percentile “Dark Tetrad” traits – traits of Machiavellianism, narcissism, psychopathy, and sadism. For comparison, I’m in the 7th percentile according to the “Dark Factor” scale.

I think there are many more places where we can intervene.

I propose a circular model where:

1. Wars and societal collapses cause widespread trauma,

2. Which causes widespread insecure attachment and personality disorders.

3. People with these mental health problems are

   1. desperate for power and admiration, and form a large pool from which new authoritarian leaders are likely to emerge, or

   2. desperate for identity and strong leadership, and susceptible to authoritarian leaders.

4. These are then a risk factor for further wars and societal collapses, perpetuating the cycle.

This model suggests four points at which we can intervene to break the cycle.

I feel deep compassion for the people who are suffering from these personality disorders, so closest to my heart are interventions that prevent them from emerging in the first place.

It should be noted that none of these interventions will show results within a few years, so all of this is only relevant in worlds in which, by some miracle, the AI thing goes well.

Dolores

To humanize the discussion, let’s introduce my fictional composite friend Dolores. She’s loosely inspired by real friends of mine who score high on all four traits of the Dark Tetrad. I’m hoping to show that there is no such thing as evil, but rather that children find quite brilliant adaptations to the extremely challenging environments they are trapped in throughout their childhoods. These can heal in adulthood (when they gain their freedom) in the same way that someone who has lost a foot can learn to walk again with a prosthesis. It often takes only a few years of therapy.

Dolores is an example of someone with NPD and ASPD. Someone with pure ASPD/psychopathy would be less concerned about upholding any particular characteristics of their personality, and someone with pure NPD would try to self-deceive more comprehensively to hide from themselves any of their behaviors that violate their self-image including values.

Psychopathy. Some of Dolores’s ancestors served in World Wars I and II. Others tried to raise children among the dropping bombs. Dolores’s dad worked long hours to provide for the family and his alcohol addiction. She hardly remembers him. Dolores’s mom never showed emotions other than occasional anger and disappointment and thought that emotional empathy was some kind of metaphor because she had never experienced it.

Hence, the epigenetic adaptations from the wars that Dolores inherited were compounded by the avoidant attachment that she developed in early infancy. She didn’t feel a lot of fear or stress or pain in the first place, and when she did, she repressed it with a vengeance, so the corresponding brain regions stayed small and relatively disconnected.

This personality aspect references war and its effects on epigenetics and how trauma is passed down through parenting practices.

Pathological narcissism. Her mom tried to care for her to the best of her limited ability, but Dolores’s twin sister was born with a disability and required all her attention. For a while Dolores cried a lot when her mom was elsewhere seeing doctors with her sister or drinking in her car to unwind. Then Dolores stopped crying for good. Her mom often told her how great it was that she, unlike her sister, never needed any support, was always strong, independent, and in charge, and eventually Dolores started to believe it too. Any feeling of tiredness or sickness or physical injury filled her with unbearable shame like she was unworthy to even be alive if she felt such things. It was so unbearable she repressed it with rage, often within seconds. She must never be weak or dependent on anyone. It wasn’t so much anymore that she didn’t want to disappoint her mom. Rather her mom’s expectations had become her own. They felt like natural law.

This narcissistic false self is (on account of being false) often at odds with reality and requires constant maintenance, for which fame, power, achievement, admiration, and money are useful.

Sadism. She found that if she started fights, the adrenaline pushed those unbearable feelings away, and she felt invulnerable again. To an extent she was, because it was easy for her to ignore any pain and she didn’t need much sleep. But to acknowledge the reason for her misconduct would be to acknowledge her vulnerability, so she told herself that she just righteously punished people for being weak. It wasn’t even a lie. She resented people who showed weakness the same way others resent cheaters. A boy cried because he had sprained his ankle, which had happened to Dolores many times before. One teacher consoled him and another ran to get ice. It made her boil with righteous anger because the alternative would’ve been to feel envious of him, which was unbearable. Once he could stand up again, she tackled him to the ground and dislocated his shoulder so he had something to cry about. She relished his pain and fear and wondered whether she was an archangel sent down to enact the wrath of God. She felt not only powerful, she felt valiant for defending the law of God and merciful because she had long been taking a knife to school but had chosen not to use it.

This kind of righteousness, like any kind of righteousness, is a breeding ground for conflict. (Interestingly, pure psychopathy is unusually immune to this kind of righteousness, which may reduce the risk of some kinds of conflicts.)

Machiavellianism. But then middle school (and her puberty) started and she became the target of older, stronger bullies who made fun of her second-hand clothes. She didn’t care about the clothes, but she would rather die than to not be in charge of a situation like those wretches she had punished. At home she was subservient and invisible; in school she was the punisher. Neither pattern was suitable to respond to these bullies. So she got out her knife, stabbed herself in the shoulder (she felt she had deserved the pain anyway for her weakness), and hid the bloody knife in the bag of the lead bully. Then she reported him to the principal. After a few days of well-rehearsed lies and affected sobbing, the bully got expelled. It was delicious vengeance, but more importantly she relished the ability to control everyone’s realities with her lies. She started to lie habitually so she would always feel in charge. People who are dumb enough to fall for her lies had it coming. But she also enjoyed helping others with their homework and covered for them when they wanted to skip classes. It felt safe that everyone was a little indebted to her, which she could leverage to bolster her popularity and get away with stuff more easily. But she also relished simply being seen as a good person.

In a war zone you don’t walk up to your enemy and complain about the shooting. But she hadn’t had the chance to learn how to function in an environment where assertiveness works, so she had to fall back on her wits. This can be adaptive in the dog-eat-dog world of politics.

Recovery. Shortly before her 18th birthday, she found herself once more in a juvenile detention center for a colorful panoply of alleged crimes – assault, theft, identity theft, possession of illegal drugs, trespassing, driving without a license, driving under the influence, reckless driving, etc. She was sure she would walk free again, because she’s a genius manipulator who can charm her way out of anything. But she could no longer repress the thought that it was perhaps just her age that had protected her from prison sentences. The thought was heavy as a tank and cut through her like a sword. Acknowledge that she can’t control the justice system, go to prison, or kill herself in a final act of defiance? She remembered Ken Follett’s Eye of the Needle, in which an elite spy and assassin with near superhuman skills plays the roles of meek and ordinary citizens to draw no attention behind enemy lines. She could also use her genius and guile to play the role of a model citizen and thereby trick the entire social contract. Her final triumph on her path to complete mastery of social engineering!

But she had to cleverly devise other outlets to get her adrenaline and to continually prove to herself that she’s invulnerable and in charge. Within a year, she excelled at rugby and broke into abandoned buildings at night. Just like the alcohol, it made her life feel a little less empty and gray. Whenever she followed a law so as not to risk prison, she felt like Ken Follett’s cunning spy.

Her problems with the law became minor, but somehow it was still difficult for her to keep friendships for more than a year. She ruminated a lot on her lost friendships even though she felt nothing and always told herself that she doesn’t need anyone. She limited her lies to untestable and inconsequential things so she could still control her friends’ realities but only in ways that didn’t clash with the actual reality. When her friends cried, she stormed out of the room and distracted herself by running people over in Grand Theft Auto so as not to punish her friends for their weakness. Sometimes she punished them by ghosting them for a week, but on some level she knew they’d just think she’s busy. She finally managed to keep some long-term friends – friends who considered her a good, strong, and reliable person.

Years later she started therapy, ostensibly to manage her alcoholism. She quit on or ghosted her first eleven therapists, but the twelfth was a hottie and she gave him a chance. All the lies she told her therapists shaped a persona wholly different from her real self. She told herself that she’s testing their intelligence and hence worthiness to treat her, but really she had never opened up to anyone and was terrified of what they might force her to acknowledge if they knew more.

Over the course of two years, she lied more and more glaringly on purpose hoping he’ll finally catch on. She was ready now, but she still needed to maintain the illusion that she had been testing him because there was no way she could acknowledge having been afraid. Eventually she had enough, screamed at him that she had been lying to him all along, fought back oh-so-shameful tears by throwing her phone against the wall, and then told him what he really needed to know.

Self-awareness was weird for her. Even after another two years of therapy, all her self-deceptions felt just as real to her as always, but now she knew that they were not. These two realities refused to fuse. Maybe it wasn’t time yet and still too threatening. It became easier for her to realize when she had to override her intuitions, which was helpful in her friendships and at work (though the realizations were sometimes a few days late), but her intuitions corrected themselves only ever so slightly. She got diagnoses for NPD and ASPD and learned that other people actually experienced strange sensations like empathy and remorse, that it wasn’t all just pretense. She could also think more strategically about how to self-soothe, explain things to her friends, adjust her environment, replace destructive coping methods with better ones, cut off toxic people without taking revenge on them, and find jobs where her rare adaptations could serve prosocial ends.

Alternative ending. But there is a different world where Dolores didn’t self-deceive into pretending to fool the justice system with her masks but rather vowed to avenge the indignities she had suffered at the hands of the executive and judicial branch. Her plan might’ve involved writing a long list of the names of every employee of the courts and prisons who had limited her freedom, becoming a political leader or starting a military coup to destroy all democratic checks and balances, and personally condemning everyone from her list who’s still alive at that point. But in that world we likely wouldn’t have met.

Esperanza

Dolores represents the leadership side of the cycle – the traits that can drive someone to seek power over others. But the cycle also requires a population susceptible to following such leaders. To humanize that side, let me introduce Esperanza, another fictional composite friend. She’s loosely inspired by real friends of mine who struggle with borderline personality disorder (BPD). Like Dolores, she is a testament to the brilliance of children’s adaptations to dangerous environments – and like Dolores, she can heal.

Esperanza is an example of someone with BPD and predominantly preoccupied-disorganized attachment. Where Dolores learned that needing anyone is shameful and built a fortress of self-sufficiency, Esperanza learned that she desperately needs others but can never predict whether they will help or hurt her. Someone with purely preoccupied attachment would seek closeness persistently without the oscillation – they might be clingy but not volatile. The disorganized component adds a simultaneous fear of the very closeness Esperanza seeks, creating the chaotic push-pull that characterizes BPD.

Disorganized attachment. Esperanza’s mother loved her – she knows this because she has warm memories of being held, read to, and tucked in. But her mother also drank, and the woman who tucked her in and the woman who screamed at her for leaving a cup on the counter were, to a five-year-old’s mind, two different people. Esperanza could never predict which mother would walk through the door. When her mother was sober, Esperanza would run to her and cling. When her mother was drunk, Esperanza froze mid-stride – wanting to run toward her and away from her in the same breath. She learned to read microexpressions with preternatural accuracy, scanning her mother’s face from across the room for the tiny cues that predicted which version was about to appear. It was a brilliant adaptation. She became the most perceptive person in any room she entered. But she also internalized a lesson no child should have to learn: The person who is supposed to keep you safe is the same person who hurts you, and there is no resolution.

This is the developmental origin of disorganized attachment: The caregiver is simultaneously the safe haven and the source of danger. The child cannot develop a coherent strategy for seeking comfort because the very act of approaching comfort activates fear.

Identity diffusion. Without a consistent caregiver response, Esperanza never developed a stable sense of self. When her mother wanted a cheerful daughter, Esperanza was cheerful. When her mother wanted to be left alone, Esperanza became invisible. When her mother cried and needed comfort, Esperanza became the parent. She got so good at becoming what others needed that she lost track of what she herself was. At school, she mirrored her friends – their tastes in music, their opinions, their style. She described herself as “hollow” or like a “mirror.” She was brilliant at empathy – both cognitive and affective – precisely because her survival had depended on it. But when someone asked “What do you want?”, she couldn’t answer without referencing someone else. Her own desires, preferences, and values felt fleeting like flickers of sunlight reflected from the ripples on the surface of a pond.

Identity diffusion – the absence of a coherent, integrated sense of self – is a hallmark of BPD and a direct consequence of disorganized attachment. It creates an interpsychic chaos of disparate identity fragments in desperate search of any kind of order or stability. Where Dolores’s false self is a rigid fortress she maintains at all costs, Esperanza’s self is a bucket of shards of a shattered mirror.

Emotional dysregulation. Other children seemed to have a volume knob for their emotions. Esperanza had an on/off switch. A perceived slight from a friend didn’t sting a little – it felt like annihilation, like being five years old and abandoned all over again. Joy didn’t build gradually – it detonated. She couldn’t soothe herself because she had never been consistently soothed. When the emotional pain became unbearable – and it became unbearable often – she found that pressing a blade against her forearm produced a strange, immediate calm. The sharp physical sensation cut through the emotional noise the way a slap across the face might startle someone out of a panic attack. She wasn’t trying to die. She was trying to feel one thing instead of everything at once. Later she discovered other methods – binge eating, reckless driving, losing herself in intense relationships with near-strangers – each of them an external regulator for an internal system that had never been calibrated.

This desperate search for external regulation is the behavioral expression of what the attachment system was supposed to provide but didn’t: a way to modulate overwhelming emotions through connection with a reliable other. Without it, the person is left searching for substitutes wherever they can be found. Where Dolores’s adrenaline-seeking pushes vulnerability away, Esperanza’s crisis-seeking is an attempt to make the pain legible, to convert chronic emotional chaos into something with a clear cause and a clear end.

Idealization. Esperanza fell in love the way other people fall off cliffs. When she met someone confident and certain – a new best friend, a romantic partner, a mentor – something specific happened: The dopamine reward circuits flooded and her inner critic went quiet. The person was perfect. They understood her like no one ever had. She felt safe for the first time in months. She rearranged her life around them within days. She adopted their opinions, their vocabulary, their goals. This wasn’t infatuation in the ordinary sense – it was her identity diffusion finding a temporary fill, her attachment system finally latching onto someone who might be the reliable caregiver she never had.

And then, inevitably, they did something imperfect. They cancelled plans. They disagreed with her. They looked at their phone while she was talking. And the perfect person became the dangerous person – not somewhat flawed, but fundamentally unsafe. The same splitting defense that had once helped a five-year-old manage the contradiction of a loving and terrifying mother now divided Esperanza’s adult world into saints and demons, with nothing in between. Friends learned to dread the phone calls: “She’s the most amazing person I’ve ever met” one week, “She never cared about me” the next.

This idealization-devaluation cycle is the interpersonal signature of BPD. It is also the mechanism that makes someone with these traits maximally vulnerable to authoritarian leaders, cult recruiters, and anyone else who presents themselves with enough confidence to be momentarily mistaken for the answer to a question the person has never been able to articulate.

The cult. At 23, Esperanza was between devaluations – recovering from a devastating breakup that felt, as breakups always did for her, like a death. A friend brought her to a weekend retreat run by a “personal development” community. The leader was magnetic. He spoke as if he could see straight through her, and what he saw was beautiful. For the first time someone told her who she “really” was, and it was someone worth being. The community offered everything her attachment system had always craved: unconditional warmth during the initial love-bombing, a clear set of rules and rituals that provided the external structure her emotions required, a shared identity she could borrow without shame, and a group that promised never to leave her.

She moved in within a month. She volunteered 60 hours a week. She stopped calling her old friends – not because anyone explicitly told her to, but because the leader framed outside relationships as evidence of insufficient commitment, and she couldn’t tolerate the possibility that he might be right. When her sister called to say she was worried, the leader said: “People who love you want you to grow. People who try to pull you back are afraid of your power.” It made sense. Everything he said made sense. That was the point.

But the leader was not consistently kind. He publicly praised members who submitted and publicly humiliated those who questioned. He demanded demonstrations of loyalty at unpredictable intervals. He was, in the language of Alexandra Stein’s Terror, Love and Brainwashing, a “frightening-yet-caregiving” attachment figure – and he was recreating, at the group level, exactly the disorganized attachment dynamic that had defined Esperanza’s childhood. The difference was that this time she had chosen it, which made it feel like freedom.

Cult leaders and authoritarian figures exploit the same psychological mechanisms: They offer a powerful identity to those with identity diffusion, external emotional regulation to those who can’t self-regulate, and unconditional belonging to those with abandonment terror – then use intermittent reinforcement and loyalty tests to create a bond that is as difficult to leave as a childhood home. The leader is a Dolores – someone whose avoidant grandiosity fills the exact hole that preoccupied-disorganized attachment leaves open.

The trap. Leaving the group activated the same neural alarm as childhood abandonment. She had no stable self to fall back on outside the group identity – they had given her one, and outside their walls it evaporated. She tried once, after a particularly humiliating group session, and lasted eleven days. She called old friends at 3 a.m. and was met with “I told you so,” which confirmed the leader’s claim that outsiders don’t understand. She went back. The relief of returning was so intense that it cemented the bond further: The group was the only place the pain stopped.

The difficulty of leaving a cult or abandoning an authoritarian leader mirrors the difficulty of leaving an abusive relationship. In all three cases, the attachment system is not responding to whether the other person is good for you – it is responding to whether the other person is familiar to your nervous system. For someone with disorganized attachment, a frightening-yet-caregiving figure feels like home. An actually safe environment feels alien and therefore threatening.

Recovery. She left during a different kind of crisis – the leader turned his full attention to a new favorite and Esperanza experienced the cold side of idealization she had inflicted on others so many times before. For once, the pain of staying exceeded the terror of leaving. A former member she’d kept in secret contact with – her one remaining thread to the outside – helped her find a therapist who specialized in dialectical behavior therapy (DBT), which was developed specifically for BPD.

The early months were brutal. Without the group, she had no identity, no structure, no one telling her who to be. She filled the vacuum with crisis after crisis – a pattern her therapist gently named rather than punished. Slowly, through distress tolerance exercises that gave her something to do besides self-harm, through interpersonal effectiveness skills that replaced her fawning and mirroring with assertiveness, and through the hardest skill of all – sitting alone in a quiet room without texting anyone – she discovered that she could survive being alone. That she had, in fact, always been alone, even inside the group. The group had provided the feeling of safety through enmeshment. Actual safety, it turned out, required something she’d never tried: autonomy.

She began to distinguish her own preferences from the borrowed ones. She realized she liked folk music, not the electronic music her ex had liked, not the devotional music the group had played. She liked dogs, not cats. She wanted to be a nurse, not a “healer.” These were small discoveries, but each one was a brick in a self that was, for the first time, being built from the inside rather than plastered on from the outside.

The splitting didn’t stop overnight. She still caught herself dividing the world into saints and demons, but she learned to notice the split and hold it, the way you might notice you’re clenching your jaw and consciously relax it. Her therapist didn’t become a saint. Esperanza hated her sometimes and said so, and her therapist survived it, and this turned out to be the most therapeutic thing of all: a relationship that could contain both feelings without collapsing.

Alternative ending. But there is a different world where Esperanza never encountered the cult – or, more precisely, where she encountered something worse. In that world, her country was in the grip of an economic crisis. The job she’d built her borrowed identity around vanished. The leader wasn’t a guru in a retreat center but a politician on a screen, and the group wasn’t a few dozen people in a compound but millions who gathered at rallies and online forums, all of them hungry for the same thing she was: someone to tell them who they are, who to blame, and that the pain would stop. In that world, she didn’t lose a few years to a cult. She lost her capacity to evaluate her own government. And she had plenty of company.

Attachment

Our attachment styles – our early-life strategies for securing safety and connection – form a blueprint for our relationships throughout life. There are four primary styles:

• Secure: A baseline of trust in others and a sense of self-worth. This fosters resilience and comfort with interdependence.

• Preoccupied (alias anxious): A preoccupation with relationships and a fear of abandonment, often leading to a desire for validation from others.

• Avoidant (alias dismissive-avoidant): A prioritization of independence and self-reliance, often at the expense of emotional intimacy, stemming from a core belief that depending on others is unsafe.

• Disorganized (alias fearful-avoidant): Resulting from experiences where a caregiver is a source of both comfort and fear, this style involves a painful mix of wanting and fearing closeness. It creates significant internal conflict and a powerful need for control to manage this chaos.

All attachment styles other than secure are also collectively referred to as insecure attachment.

This framework is not just for romantic relationships; it shapes our relationship to society, authority, and ourselves.

The Psychology of Followership

Dolores’s attachment style is highly avoidant and only mildly preoccupied. Avoidant attachment is very typical of NPD and ASPD, because arguably it’s a result of these disorders. Often the less disordered version of the person would have disorganized attachment or something close to it.

How does someone with these psychological adaptations gain a following – and why do democracies keep producing populations willing to hand one over?

The standard account, that charismatic leaders simply manipulate passive masses, misses half the equation. The appeal of an authoritarian leader is better understood as an interaction between the leader’s psychology and the collective attachment needs of the population. The following theory draws centrally on Otto Kernberg’s work on malignant narcissism and large group regression, integrates it with the dimensional attachment research pioneered by Brennan, Clark, and Shaver and Fraley, Waller, and Brennan, and tries to be specific about the psychological mechanism on the follower side.

The Substrate: A Population with Masked Insecurity

Most people are not particularly securely attached. They just look like it.

Attachment anxiety and attachment avoidance are continuous dimensions, not discrete categories. Taxometric analyses by Fraley and Waller and by Fraley and Spieker on infant data have consistently shown that there is no bright line dividing “secure” from “insecure” people – just a smooth, two-dimensional continuum. In a sample of over 22,000 adults, the population means on the ECR-R (a 1–7 scale where 4 is the midpoint) were 3.64 for anxiety and 2.93 for avoidance, with standard deviations of 1.33 and 1.18 respectively. The bulk of the population sits in a zone that is not robustly secure but also not clinically disordered. Meta-analyses of the Adult Attachment Interview put the rate of “secure” classifications at only about 50–56% in non-clinical samples – and that categorical cut imposes an artificial boundary on what is really a gradient.

Crucially, the two dimensions are moderately positively correlated (r = .41 in the Fraley sample). People who are somewhat anxious tend to also be somewhat avoidant. This means the population doesn’t cleanly sort into “preoccupied” and “avoidant” camps. A large portion sits in the mildly fearful-avoidant zone – harboring a degree of both the craving for closeness and the distrust of it that characterizes disorganized attachment at clinical levels.

There is also a measurement caveat that makes the picture even less reassuring. Fraley notes that the ECR-R “doesn’t assess ‘security’ with as much precision as ‘insecurity’” – item discrimination is lower at the secure end of both dimensions. Some people who score as mildly secure may actually be less secure than their scores suggest. The veneer may be even thinner than the numbers indicate.

So what keeps all these mildly insecure people functioning? In Aaron Pincus’s taxonomy of pathological narcissism, the spectrum runs from overt grandiosity through covert vulnerability. I propose that most people manage their mild attachment insecurity by borrowing stability from external sources – their professional identity, social role, marriage, political tribe, or national narrative. This is a false-self defense: not the dramatic grandiose false self of NPD, but a subtler version in which the person’s sense of coherence depends on structures they don’t realize they depend on. They function well, and they feel secure – but the security is largely extrinsic, not intrinsic.

The Collapse: When the Borrowed Self Breaks Down

Then the status quo gets disrupted.

A financial crisis wipes out jobs and savings. A war displaces millions. A pandemic isolates people from their social roles. Traditional institutions lose legitimacy. The external structures that were doing the psychological work of maintaining identity and self-worth suddenly vanish.

This is the trigger that Kernberg describes in his 2020 paper and his earlier work on sanctioned social violence: When the normal social structures that assure individuals of their status disappear, the population undergoes what he calls “large group regression.” Drawing on Freud’s group psychology, Bion’s theory of basic assumptions, and Melanie Klein’s developmental positions, Kernberg argues that the population collectively “regresses” to what Klein called the paranoid-schizoid position – the primitive defensive mode characterized by splitting (the world divides into all-good and all-evil), denial, omnipotent control, and projective identification. I argue that this is less a regression of and more an unmasking of a fundamental borderline organization.

What does this look like psychologically? The mildly insecure majority – the people who were getting by with borrowed stability – lose the external validation that their fragile self-concept requires. They enter a state akin to what clinicians would recognize in individual patients as a mild, chronic narcissistic collapse. Their identity feels diffuse. Their world feels unpredictable and threatening. They can no longer answer the question “Who am I?” by pointing at their job title or their social role.

Vamik Volkan’s concept of “chosen trauma” – a shared mental representation of ancestral suffering transmitted across generations – adds a transgenerational dimension. During regression, the chosen trauma is reactivated to shore up the threatened group identity, fueling a sense of historical grievance that can be politically mobilized. This is the mechanism that connects the first step of my cycle (wars cause widespread trauma) to the susceptibility of later generations who didn’t directly experience the original trauma but carry its imprint in their group identity, the parenting practices they were exposed to, and the beliefs and behaviors that they perpetuate.

The Bond: Finding a New Self in the Leader

The regressed population is now desperate for what Kernberg calls a “second skin” – a new false self that restores a sense of security. This is where the preoccupied dimension of attachment becomes politically decisive.

Remember that the population mean on anxiety (3.64) is meaningfully closer to the scale midpoint than the mean on avoidance (2.93). The average person’s attachment system skews slightly more hyperactivating – oriented toward seeking closeness and external reassurance – than deactivating. When external identity structures collapse, the predominant pull is toward finding a new attachment figure, not toward power-seeking or withdrawing into isolation.

An authoritarian leader with Dolores’s avoidant-narcissistic profile slots perfectly into this vacancy. The leader provides two things the regressed population craves:

A borrowed identity. The leader offers strong, simple identification: “You are part of my movement. You are great because I am great.” Kernberg specifically notes that this identification “spares the need for the mass to envy the leader” – the population gets to participate in the leader’s grandiosity rather than measuring themselves against it. For people whose self-concept has just collapsed, this is profoundly relieving.

An ideology of splitting. The leader provides a framework that channels the population’s already-activated primitive defenses into political form: An ideology “that allows to identify the self against the other, aggressively othering a victim minority of choice.” The scapegoated minority absorbs the population’s projective identifications – all the weakness, shame, and vulnerability that the followers cannot tolerate in themselves gets located in the outgroup. To those with collective trauma, a leader who validates their sense of grievance provides a channel for their rage that can feel deeply empowering.

What makes this bond so resistant to correction is that it resembles a trauma bond more than a simple dependency. The leader’s intermittent appearances on television, unpredictable loyalty tests, and alternation between warmth toward loyalists and punishment of dissenters recreate the approach-avoidance dynamic of disorganized attachment. The follower can’t simply walk away (as a purely avoidant person might) but also can’t calmly evaluate the leader (as someone with less anxiety might). They’re stuck – not because they lack intelligence, but because the bond keeps them hooked like a drug.

This is a positive feedback loop: The regressed population selects the narcissistic leader, and the leader’s behavior reinforces the population’s regression. This is a key mechanism behind the cycle from my introduction – and it explains why the cycle is so hard to break without intervention at multiple points.

Kernberg discusses some of those intervention points. He identifies independent social structures – free media, independent judiciary, professional armed forces – as the primary buffers that can “set limits to the antisocial behaviors adopted by the leader, for example by not allowing dishonesty.” At the individual level, drawing on historian Timothy Snyder’s On Tyranny, he emphasizes “individual courage, responsibility, independence of thinking and public action” as qualities that permit a person to “stand up to the dangerous imprisonment in regressive group formations.” More on interventions in the next section.

Related perspectives

The theory above synthesizes several research traditions. Here is a brief tour, highlighting where each converges with and diverges from the account above.

Erich Fromm, Escape from Freedom (1941). Writing decades before Bowlby formalized attachment theory, Fromm argued that the “authoritarian character” is “extremely alone and gripped by a deeply rooted fear” and seeks “secondary bonds” – submission to authority – as substitutes for lost primary attachments. His analysis of Weimar Germany anticipates nearly every element of the model above: the destabilized population, the intolerable freedom, the flight into certainty. Fromm frames it in terms of existential freedom and social character rather than attachment dimensions, and he doesn’t distinguish the specific defense mechanisms (splitting, projective identification) that Kernberg adds from the Kleinian tradition.

Alexandra Stein, Terror, Love and Brainwashing (2nd ed., 2021). Stein – a cult survivor and researcher – uses attachment theory to explain how charismatic leaders create disorganized attachment in followers by being simultaneously the source of comfort and fear. She applies this framework across religious cults, political cults, terrorist organizations, and totalitarian states. Where I emphasize the pre-existing insecurity of the population as the substrate, Stein focuses on how even relatively secure individuals can be broken down by sustained exposure to the frightening-yet-caregiving dynamic. Both accounts are probably correct at different ends of the vulnerability spectrum: Pre-existing insecurity lowers the threshold, while sufficiently extreme environments can push almost anyone into disorganized attachment.

Antigonos Sochos, “Authoritarianism, trauma, and insecure bonds during the Greek economic crisis” (2018). Perhaps the clearest empirical snapshot of the cycle caught in the act. Using a large community sample during Greece’s economic catastrophe – 40%+ income loss, 56% youth unemployment, mental health spending halved, nearly 60% of participants reporting severe post-traumatic stress – Sochos found that authoritarianism was independently linked with insecure attachment in both person-to-person and person-to-state bonds, mediated by post-traumatic stress and perceived loss of social cohesion. Golden Dawn’s concurrent rise from fringe group to 7% of the national vote provides the political outcome. This study captures the collapse and the bond in a single dataset; it doesn’t address the substrate of pre-existing insecurity, but the severity of the crisis likely overwhelmed false-self defenses across the board.

Omri Gillath and Joshua Hart, “The effects of psychological security and insecurity on political attitudes and leadership preferences” (2009); Mikulincer and Shaver, “Boosting Attachment Security to Promote Mental Health, Prosocial Values, and Inter-Group Tolerance” (2007). Experimental work showing that providing people with an alternative source of psychological security – even just subliminally priming the secure-base schema – reduces endorsement of anxiety-driven political attitudes and hostility toward outgroups. This is the complement to Sochos’s correlational findings: If insecurity drives authoritarian attitudes, then boosting security should reduce them – and it does. These studies directly support the intervention logic in the next section.

Christopher Weber and Christopher M. Federico, “Interpersonal Attachment and Patterns of Ideological Belief” (Political Psychology, 2007). Maps attachment dimensions directly onto political ideology, finding that attachment anxiety predicts preferences for social order and certainty. This is consistent with my emphasis on the preoccupied dimension as the primary driver of susceptibility: It’s the anxiety dimension, not avoidance, that predicts the longing for certainty that authoritarian leaders promise to satisfy.

Kleppestø et al., “Attachment and Political Personality are Heritable and Distinct Systems” (Behavior Genetics, 2024). An important counterpoint. This large Norwegian twin study (N = 1,987) found that attachment and political personality (right-wing authoritarianism, social dominance orientation) are both heritable but show “no shared environmental overlap.” This challenges a simple developmental-pathway account where bad parenting causes insecure attachment which causes authoritarian attitudes. My model is compatible with this finding: It doesn’t require that insecure attachment causes authoritarian ideology, only that insecure attachment – whatever its origins – creates the psychological readiness to bond with an authoritarian leader when external structures collapse. Someone can carry both heritable traits independently and still have the second amplify the political expression of the first under stress.

Breaking the Cycle

Breaking this pernicious cycle requires a multi-layered strategy, acting on individuals, institutions, and our foundational knowledge. The following combines the developmental approach of this article with several institutional and long-term proposals from David Althaus and Tobias Baumann’s work. I’ve added to each strategy areas of expertise that are likely required to pull them off.

In general, we want to optimize for something like Aaron Antonovsky’s “Sense of Coherence,” a measure of good mental health:

• Comprehensibility. The belief that the world around you is structured, predictable, and makes sense. Events aren’t random and chaotic; they are explainable.

• Manageability. The belief that you have the resources (either your own, or available from your family, friends, or community) to meet the demands of life. The feeling that “I can handle this.”

• Meaningfulness. The belief that life’s challenges are worthy of investment and engagement. This is the motivational component – the feeling that it’s worth it to try.

I’ve tried to guess very roughly where, on a scale from 1 to 5, I see each intervention area. I’ve broken this down first according to the significance-persistence-contingency framework and second according to my own subdivisions.

My intuition is that AI-based therapy, school-based interventions (merged for simplicity), safe exits for dictators, and improved institutional decision-making are the greatest levers. But not by a great margin. Opportunism and personal fit will likely make the decisive differences.

Mitigating Trauma

• Safe pathways for refugees. Currently it’s an often deadly and tremendously traumatic affair to try to emigrate from a country that is at war. The UN Refugee Agency and others advocate for safe and regulated pathways for refugees.

  • Politics, policy, activism

Healing Trauma

• Parenting classes. If we intervene in high schools and teach state-of-the-art practices of good parenting, we can break cycles where children learn harmful practices from their parents and pass them down. Even adolescents who will continue to suffer from personality disorders that tend to be passed down can force themselves to treat their children better than their intuitions would have them.

  • Policy, schoolbook publishing, pedagogy, psychology

• Mental health classes. Good mental health practices are often learned, and that learning could happen in school rather than in the therapy sessions of only those who seek therapy.

  • Policy, schoolbook publishing, pedagogy, psychology

• Values classes. Compassion, rationality, assertiveness, and radical acceptance are key to the recovery from NPD and ASPD. Furthermore, NPD is sustained by self-deceptions that are at odds with rationality and sometimes compassion, so these values can make it easier for sufferers to recognize their self-deceptions.

  • Policy, schoolbook publishing, pedagogy, psychology

• AI-based therapy. Mental health classes may be both too hard to establish and not sufficiently focused on the individual. But perhaps the rollout of individual AI-based therapy for every pupil is more realistic and focused. AI-based therapy can also offer a non-judgmental first step for individuals who, like Dolores, may not trust human therapists. “The Kauai Study” found that a strong bond in childhood with at least one mentally healthy adult is critical for the mental health of the child, and perhaps an AI can provide this bond for children who know no mentally healthy humans.

  • Psychology, software engineering, policy

• Parenting support. Policies that reduce parental stress – such as paid parental leave, affordable childcare, and home-visiting nurse programs – interrupt the transmission of trauma and foster secure attachment.

  • Psychology, policy

• Juvenile detention. Many people who’ve recovered from ASPD say that prison sentences (sometimes several) have been invaluable for them and that they wish they had been caught and gone to prison earlier than they did. In this way, juvenile detention is perhaps an effective de facto therapy of ASPD that can intervene early in a person’s life. Ideally the criminal record should expire to not limit the person when they’re trying to turn a new page.

  • Policy, politics

• Rewarding prosociality. People with NPD can have a highly prosocial self-image that motivates them to change the world for the better. They often lack an internal feedback mechanism that rewards them when they’re doing a good job, making them dependent on external feedback. Stronger norms to publicly and frequently reward prosocial behaviors could have an even greater positive effect on people with NPD than on the average person.

  • Philanthropy, activism

Societal Resilience

• Secure attachment. As described above, a populace with largely secure attachment is less likely to be receptive to disordered messaging.

  • See previous section

• Sense of coherence. As described above, a populace with robust mental health and resilience.

  • See previous section

• Egalitarian social norms. Authoritarian leaders will ring false to the ears of a populace that values flat hierarchies.

  • Unclear

• Malevolence research. Althaus & Baumann call for a focused research program to develop better constructs and measures for the traits that cause the most harm. A deeper understanding of the neurological and psychological underpinnings is a prerequisite for effective interventions.

  • Psychology

• Tamper-proof screening. A key proposal from their work is the creation of reliable, manipulation-proof measures of malevolence. These could one day be used to screen for high-risk individuals in positions of immense power, such as heads of government or leaders of critical global institutions.

  • Psychology, policy

National and International Buffers

• Better institutional decision-making. As Althaus and Baumann note, classic political science solutions are a vital complement to psychological ones. This includes strengthening democratic checks and balances, promoting measures to reduce political polarization, and supporting a well-resourced, independent press.

  • Policy, politics, activism, journalism, law, sociology, economics

• International coalitions. Elegantly constructed international coalitions and contracts can increase the stability of the whole worldwide system.

  • Policy, politics, activism, law, sociology, economics

• Safe exiles for dictators. Some dictators want to retire at some point but can’t because they need the power of the state to prevent attempts on their lives. Providing them with a way in which they can cede power without being killed may increase the chances that they’ll negotiate a transition of power.

  • Military, politics, policy

Risks

• Lack of psychopathy. People with secure attachment and without any personality pathology can still be zealously righteous about whatever tribal moral goals they subscribe to. That’s also a risk factor. People with pure psychopathy may in fact be unusually immune to that, thereby unlocking gains from moral trade.

• Naïveté. Maybe peace is better kept and more stable by maintaining a balance of highly manipulative and power hungry people on all sides, the same way some people argue that nuclear war is better prevented by maintaining a balance of sufficiently large well-maintained nuclear arsenals in the hands of many countries around the world.

• Discrimination. Some measures risk exacerbating the discrimination against perfectly prosocial people suffering from personality disorders.

A Call to Action

• Talk to me if you have access to any of the relevant fields, and maybe we can strategize.

• Do you know any organizations that are working on this?

• Protect yourself but also be patient with people with personality disorders. They can heal and use their skills for prosocial ends.

Nate Soares’s Replacing Guilt examines guilt as a motivational tool, deeming it unhealthy and inefficient, particularly among effective altruists. The author proposes alternative methods for driving action, emphasizing identifying and fighting for personal values and goals rather than adhering to external obligations or seeking to avoid negative self-judgment.

Key concepts include seeing the world realistically, detaching from the need for external validation, and developing effective response patterns to challenges, ultimately advocating for intrinsic motivation fueled by a defiant desire to improve the world.

The text encourages self-compassion and a focus on progress in a world acknowledged to be imperfect, rather than striving for unattainable perfection or dwelling on past failures.

Introduction

Throughout the year, we’ve been hard at work to scrape together all the donation data we could get. One big source has been Vipul Naik’s excellent repository of public donation data. We also imported public grant data from Open Phil, the EA Funds, the Survival and Flourishing Fund, and a certain defunct entity. Additionally, 36 donors have entered their donation track records themselves (or sent them to me for importing).

GiveWiki and the Markets of Impact is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Add some retrospective evaluations, and you get a ranking of 92 top donors (who have donor scores > 0), of whom 22 are listed publicly, and a ranking of 33 projects with support scores > 0 (after rounding to integers).

(The donor score is a measure of the track record of a donor, and the support score is a measure of the support that a project has received from donors, weighed by the donor score among other factors. So the support score is the aggregate measure of the trust of the donors with the strongest donation track records.)

The Current Top Recommendations

1. FAR AI

   1. “FAR AI’s mission is to ensure AI systems are trustworthy and beneficial to society. We incubate and accelerate research agendas that are too resource-intensive for academia but not yet ready for commercialisation by industry. Our research spans work on adversarial robustness, interpretability and preference learning.”

2. Simon Institute for Longterm Governance

   1. “Based in Geneva, Switzerland, the Simon Institute for Longterm Governance (SI) works to mitigate global catastrophic risks, building on Herbert Simon’s vision of future-oriented policymaking. With a focus on fostering international cooperation, the organisation centres its efforts on the multilateral system.”

3. Alignment Research Center

   1. “ARC is a non-profit research organization whose mission is to align future machine learning systems with human interests.”

   2. Note that the project is the Theory Project in particular, but some of the donations that underpin the high support score were made before there was a separation between the Theory and the Evals project, so this is best interpreted as a recommendation of ARC as a whole.

You can find the full ranking on the GiveWiki projects page.

Limitations:

1. These three recommendations are currently heavily influenced by fund grants. They basically indicate that these projects are popular among EA-aligned grantmakers. If you’re looking for more “unorthodox” giving opportunities, consider Pour Demain, the Center for Reducing Suffering, or the Center on Long-Term Risk, which have achieved almost as high support scores (209 instead of 212–213) with minority or no help from professional EA grantmakers. (CLR has been supported by Open Phil and Jaan Tallinn, but their influences are only 29% and 10% respectively in our scoring. The remaining influence is split among 7 donors.)

2. A full 22 projects are clustered together at the top of our ranking with support scores in the range of 186–213. (See the bar chart above of the top 34 projects.) So the 10th project is probably hardly worse than the 1st. I think this is plausible: If there were great differences between the top projects I would be quite suspicious of the results because AI safety is rife with uncertainties that I expect make it hard to be very confident in recommendations of particular projects or approaches over others.

3. Our core audience is people who are not intimately familiar with the who-is-who of AI safety. We try to impartially aggregate all opinions to average out any extreme ones that individual donors might have. But if you are intimately familiar with some AI safety experts and trust them more than others, you can check whether they are among our top donors, and if so see their donations on their profiles. If not, please invite them to join the platform. They can contact us to import their donations in bulk.

4. If projects have low scores on the platform there is still a good chance that that is not deserved. So far the majority of our data is from public sources and only 36 people have imported their donation track records. The public sources are biased toward fund grants and donations to well-known organizations. We’re constantly seeking more “project scouts” who want to import their donations and regrant through the platform to diversify the set of opinions that it aggregates. If you’re interested in that, please get in touch! Over 50 donors with a total donation budget of over $700,000 want to follow the platform’s recommendations, so your data can be invaluable for the charities you love most.

5. It’s currently difficult for us to take funding gaps into account because we have nowhere near complete data on the donations that projects receive. Please make sure that the project you want to support is really fundraising. Next year, we want to address this with a solution where projects have to enter and periodically update their fundraising goals to be shown in the ranking.

We hope that our data will empower you to find new giving opportunities and make great donations to AI safety this year!

If it did, please register your donation and select “Our top project ranking” under “recommender,” so that we can track our impact.

GiveWiki and the Markets of Impact is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Want to donate? Open up the page of our top project/s, double-check that they are still fundraising, and ka-ching!

GiveWiki and the Markets of Impact is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Interested in regranting? Check out our post on the (now) $700,000 that want to be allocated.

Top Projects

Our top projects stand out by virtue of their high support scores. There are a lot of ties between these top projects, so we’ve categorized them into tiers.

Note that we determine the top projects according to their support. Further down we’ll cover how our latest evaluation round worked out. But the support scores are two hops removed from those results: (1) Projects receive support in the form of donations as a function of donation size, earliness, and the score of the donor; (2) donors get their scores as a function of size and earliness of their donations and the scores of the beneficiary projects; (3) projects receive their credits from our evaluators:

Project credits → donor scores → project support.

This mimics the price discovery process of a for-profit impact market. Hence it’s also likely that the scores are slightly different by the time you read this article because someone may have entered fresh donation data into the platform.

We have tried to find and reach out to every notable AI safety project, but some may yet be missing from our list because (1) they haven’t heard of us after all, (2) they’re not fundraising from the public, (3) they prefer to keep a low profile, (4) etc. But at the time of writing, we have 106 projects on the platform that are publicly visible and fundraising.

Subscribe now

Ties for Tier 1

These are the projects at the very top! FAR AI and the Simon Institute with a support score of (at the time of writing) 213.

Ties for Tier 2

1. Alignment Research Center

2. AI Safety Support

3. Rational Animations

4. Ought

5. AI Impacts

They all have a support score of 212. Such small differences in support are probably quite uninformative. New data or tweaks to our algorithm could easily change their rank.

Ties for Tier 3

1. Center for AI Safety

2. Alignment Jams (of Apart Research)

Other projects with > 200 support

1. Pour Demain

2. Center for Reducing Suffering

3. Center on Long-Term Risk

4. Future of Humanity Institute

5. Centre for Enabling EA Learning and Research (EA Hotel)

6. Faunalytics

7. Rethink Priorities

8. Global Catastrophic Risk Institute

9. Legal Priorities Project

Note that, while we now market the platform to AI safety, really any project can use it and some may even fare well! We may introduce other specialized GiveWikis in the future.

If you’re just here for the results then this is where you can stop reading.

Evaluation Process

Preliminaries

For this evaluation round, we recruited Charbel-Raphael Segerie, Dima Krasheninnikov, Gurkenglas, Imma Six, Konrad Seifert, Linda Linsefors, Magdalena Wache, Mikhail Samin, Plex, and Steven Kaas as evaluators. Matt Brooks, Frankie Parise, and I may also have pitched in. Some of them ended up not having time for the evaluation. But some of our communication was under the Chatham House Rule, so I’m listing them anyway for added anonymity.

Our detailed instructions included provisions for how to score project outputs according to quality and impact; how to avoid anchoring on other evaluators; how to select artifacts to strike a compromise between comprehensiveness, redundancy, and time investment; how to evaluate projects using wiki credits; and some tips and arrangements.

Outputs are such things as the papers or hackathons that organizations put out. They can create one project per output on our platform, or they can create one project for the whole organization. Conferences cannot be directly evaluated after the fact, so what our evaluators considered were artifacts, such as recordings or attendance statistics. This distinction makes less sense for papers.

The projects were selected from among the projects that had signed up to our website (though in some cases I had helped out with that), limited to those with smaller annual budgets (in the five or lower six digits, according to rough estimates) and those that were accepting donations. The set of outputs was limited to those from 2023 in most cases to keep them relevant to the current work of the project, if any. We made a few exceptions if there were too few outputs from 2023 and there were older, representative outputs.

We hadn’t run an evaluation round at this scale. Previously we were three and could just have a call to sync up. This time everything needed to be more parallelizable.

Hence we followed a two-pronged approach with (1) evaluations of individual outputs using scores, and (2) evaluations of the AI safety activities of whole projects using our wiki credits. If one kind of evaluation fell short, we had another to fall back on.

Lessons Learned

Fast-forward four fortnights, and it turned out that there were too many outputs and too few evaluators so only two outputs had been evaluated more than twice (and 10 had been evaluated more than once). According to this metric, AI Safety Support and AI Safety Events did very well, leaving the other project in the dust by a wide margin – but those numbers were carried just by the scores of one or two evaluators so they’re most likely in large part due to the Optimizer’s Curse.

Hence we decided not to rely on this scoring for our evaluation and rather fall back on the credits for that. But the evaluations came with insightful comments that are still worth sharing.

Next time we’ll use credits only and at most list some outputs to help evaluators who are not familiar with the work of the project to get an idea of what its most important contributions were.

Wiki Credits Ranking

These are the normalized average credits that our evaluators have assigned to the projects. As mentioned above, these determine how richly donors to these projects get rewarded in terms of their donor scores, which then determine the project support: Project credits → donor scores → project support.

Qualitative Results

AI Safety Events

AI Safety Unconference at NeurIPS 2022: One of the evaluators attended it and found it high value for networking, but (empirically) only for networking within the AI safety community, not for recruiting new people to the space.

ML Safety Social at NeurIPS 2022: One evaluator estimated, based on this modeling effort, that the social was about 300 times as impactful as the reference output (“AI Takeover Does Not Mean What You Think It Means”). The estimate was even higher for the safety unconference at the same conference.

Hence AI Safety Events had generally very high ratings. It is not listed among our top recommendations because we don’t have enough donation data on it. If you have supported AI Safety Events in the past, please register your donations! You may well move a good chunk of the (now) $700,000 that donors seek to allocate!

The Inside View

AI Takeover Does Not Mean What You Think It Means: This was our calibration output – it allowed me to understand how an evaluator is using the score and to scale their values up or down. The evaluators who commented on the video were generally happy with its production quality. Some were confused by the title (Paul’s models are probably well known among them) but found it sad that it had so few views. The main benefit over the blog post is probably to reach more people with it, which hasn’t succeeded to any great degree. Maybe we need an EA/AIS marketing agency? I’m also wondering whether it could’ve benefited from a call to action at the end.

AI X-risk Research Podcast

Superalignment with Jan Leike: This interview was popular among evaluators, perhaps because they had largely already watched it. Some were cautious to score it too highly simply because it hadn’t reached enough people yet. But in terms of the content it was well regarded: “The episodes are high-quality in the sense that Daniel asks really good questions which make the podcast overall really informative. I think the particular one with Jan Leike is especially high-impact because Superalignment is such a big player, in some sense it’s the biggest alignment effort in the world.” (The episodes with Scott Aaronson and Vanessa Kosoy received lower impact scores but no comments.)

AI Safety Ideas

The website: “Seems potentially like a lot of value per connection.” The worries were that it might not be sufficiently widely known or used: “I think the idea is really cool, but I haven’t heard of anyone who worked on an idea which they found there.” And does it add much value at the current margin? “I couldn’t find a project on the site which was successful and couldn’t be attributed to the alignment jams. However, if there were some successful projects then it’s a decent impact. And I suspect there were at least some, otherwise Esben wouldn’t have worked on the site.” The evaluators didn’t have the time to disentangle whether people who participated in any Alignment Jams got some of their ideas from AI Safety Ideas or vice versa. All in all the impact scores were on par with the Jan Leike interview.

Orthogonal

Formalizing the QACI alignment formal-goal: This output scored highest on quality and impact (with impact scores in between the three AXRP interviews above) from among Orthogonal’s outputs. It got lower scores on the quality side because the evaluator found it very hard to read (but noting that it’s also just really hard to create a formal framework for outer alignment). But it scored more highly on the impact side. The evaluator thinks that it (the whole QACI idea) is very unlikely to work but highly impactful if it does. The other evaluated outputs were less notable.

Center for Reducing Suffering

Documentary about Dystopian Futures | S-risks and Longtermism: One evaluator gave a lower quality score to the documentary than to the reference output (“AI Takeover Does Not Mean What You Think It Means”) but noted that it “represents longtermism decently and gives an OK definition for s-risk.” They were confused, though, why it was published on a channel with seemingly largely unrelated content (since the context of the channel will color how people see s-risks) and concerned that talking about s-risks publicly can easily be net negative if done wrong.

Avoiding the Worst - Audiobook: The audiobook got the highest impact rating among the CRS outputs even though an evaluator noted that they only counted what it added over the book – another way to access it – which isn’t much in comparison. (The book itself was outside of our evaluation window, having been published in 2022.)

FAR AI

Pretraining Language Models with Human Preferences: One evaluator was excited about this paper in and of itself but worried that it might be a minor contribution on the margin compared to what labs like OpenAI, DeepMind, and Anthropic might’ve published anyway. They mention Constitutional AI as a similar research direction.

Training Language Models with Language Feedback at Scale: While this one scored slightly lower quantitatively, the qualitative review was the same.

Improving Code Generation by Training with Natural Language Feedback: One evaluator was concerned about the converse in this case, that is that the paper might’ve contributed to capabilities and has hence had a negative impact.

Centre For Enabling EA Learning & Research (EA Hotel)

In general: “CEEALAR doesn’t have particularly impressive direct outputs, but I think the indirect outputs which are hard to measure are really good.” Or “the existence of CEEALAR makes me somewhat more productive in my everyday work, because it is kind of stress-reducing to know that there is a backup option for a place to live in case I don’t find a job.”

AI Safety Support

AI Alignment Slack: Invaluable for information distribution. One evaluator mentioned the numerous times that they found out about opportunities through this Slack.

Lots of Links page: “The best collection of resources we currently have,” but with a big difference between the quality and the impact score: “It could be better organized and more up to date (even at a time when it was still maintained).”

Epilogue

Want to regrant some of the (now) $700,000 aggregate donation budget of our users? Please register your donations! The GiveWiki depends on your data.

You’re already a grantmaker or regrantor for a fund? Use the GiveWiki to accept and filter your applications. You will have more time to focus on the top applications, and the applicants won’t have to write yet another separate application.

We’re always happy to have a call or answer your questions in the comments or by email.

GiveWiki and the Markets of Impact is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

The First Crowdsourced Charity Evaluator for AI Safety – Voiceover and animation by Frankie Parise

(Can you guess which city they’re visiting?)

Impact Markets is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

In other news, we’ve reached $453,000 in donor interest. Now 46 donors have expressed their interest in using the platform. (Have you?) Two of them want to use it if we expand to their cause area, so I’m not counting them into the total at this stage. Most donation budgets are under $5k, but there are also many all the way up into the high five digits.

Maybe it’s time to update your bets? (See also these other Impact Markets questions.)

Our 2023 retroactive evaluation round is in full swing, so expect updates to our recommendations by December.

Impact Markets is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Impact Markets is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

This is an edited transcript of an eponymous talk. I’ve held this talk at EAGx Berlin and Warsaw 2023 among other venues.

The Funding Constraint

Great entrepreneurs who start highly impactful charity projects exist in various countries around the world and in lots of different fields that require highly specialized expertise. They also speak a number of different languages and have all their own social circles in the various countries that they live in.

That makes it difficult for an individual donor in Berkeley or London to identify but a tiny fraction of them.

Even really big foundations like the Open Philanthropy Project have problems with this. The Open Philanthropy Project can invest a lot of time into researching grants if the grant size is upward of $1 million or possibly $100,000 so that they make many grants in that area, measured by the total of those grants. But below $1 million the total grant size starts to trail off, and below $50,000 the number of grants drops off too. If they don’t already know the field and the founders, it becomes more and more difficult for them to research the projects efficiently enough that it’s still worth it for them given the small grant sizes.

That mirrors how venture capital firms would probably also love to invest the first $100,000 into all the future unicorns, but since they can’t identify them at that stage, or only vanishingly few of them, they depend on business angels and the founders’ friends and family to take the first step.

But we’re now in a world where AI safety projects struggle to raise $2,000 to make ends meet. So that gap below $1 million, particularly below $100,000, is critical. We need to coordinate the nonprofit equivalents of business angels so they can fill the gap!

Donors to the Rescue

How do we help donors coordinate? The good thing about individual donors is that they are almost everywhere in the world. If we just look at Giving What We Can data we see Giving What We Can pledge takers in 100 countries around the world. That’s a really sizable network that, if it were better coordinated, could serve to fill all of these funding gaps.

On the left, you can see the donor network as I intuit it today. There are some relatively well-connected funders but there are also lots of individual donors and lots of individual charity entrepreneurs. They’re largely disconnected or form small disconnected islands. We would like to rather move to the network on the right where all of these clusters are somewhat connected. It’s not perfect but everyone has connections to all the other funders via relatively few intermediaries. It’s much easier for projects to get funded and for donors to actually identify the most impactful projects on the margin.

Donors need to be coordinated because they come in various different shapes and sizes. There are some who are in earning to give. They spend all of their time on their full-time jobs where they have little time left to research their donations. But of course, they earn a lot of money in these jobs. These donors are all the way over on the left, the blue side of the spectrum. They can only spend a few hours per year on donation research.

But there are some other donors who can spend easily like three orders of magnitude more time on research that is relevant to their donations because they don’t even have to individually research their donations. They are working in fields where they are all the time, in their full-time jobs, exposed to exactly the sort of knowledge that is valuable for allocating donations.

There are also donors in between who – even though they work full-time in jobs that have nothing to do with donation research – are friends with researchers or live in the same flat with researchers. So these donors also osmotically absorb some of the knowledge that’s valuable for making donations, be it just the knowledge that the particular researchers that they know can be trusted.

We would like to bridge the gap between these 1000x donors and the 1x donors. We want to make it possible for the blue donors who spend all their time earning to draw on all the knowledge of the donors all the way over on the red side of the spectrum.

Our Solution

Our system works the way that we score donors after the fact in terms of how early they were able to identify some successful project, how confidently they identified it (via the donation size), and the level of success of the project. This is a retrospective scoring, so a project has generated some impact and then we look back at who has supported the project early on and exactly how early they supported the project, and then score these donors in terms of these three factors.

That will cause smart donors to stand out in our ranking. So the smartest donors will be at the top of the ranking. At the moment it’s still pretty easy to break into those top spots, so I encourage you to try!

The top donors then continue to make their donations. Thanks to their track record we know to listen to them. So we can generate new, forward-looking donation recommendations that aggregate all of their wisdom.

For the top donors, conversely, the incentive is to leverage more donations for the projects that they think are the most impactful ones.

Here is a quick example of how the scoring works:

There are pictured here two projects, the project Antiviral Algae and the project Rainwater Refinement. These projects enter the fundraising phase and find three donors each.

Alex was first, donated on day one to the project Antiviral Algae; then Avery donated on day two; and then Ash donated on day three. Our algorithm scores Alex highly in this project because Alex donated very early, but it also scores Ash highly because Ash donated a lot.

Likewise, for the second project, Ryan and River score highly because they, respectively, donated early or a lot. In between is Rowan.

And then the projects get executed. They invest the donations that they’ve received into their respective programs.

Then the evaluation happens. Our evaluators decide that the first project should receive a score of 200 and the second project one of 100. This results in the ranking that you can see all the way over on the right.

Finally, Ash receives the highest rank, and Alex is second, because the first project has the highest score, and these two donors have made the biggest contribution – monetarily in Ash’s case and in terms of information value in Alex’s case.

There are no donors who’ve donated to two projects in this case. The scores from these two projects would get added up.

Our Roadmap

At the moment we are in something that we call phase one. We have 46 projects on our platform, and we’re looking for more project scouts who want to make the first donations to these projects or potentially bring more projects onto the platform that they have already donated to so that they can register their donations to these projects. (That just depends on whether the scouts think that the most impactful project is one that is on the platform already or one that they still have to convince to join the platform.)

Eventually, once a sufficient number of these projects have been successful and have produced something that can be evaluated (we call these things artifacts), said evaluation will happen and we’ll pay some evaluators to score the projects. That’s phase 1.

Once we have some 100+ monthly active users we want to introduce something that we call impact credits or impact marks. It’s a play-money currency that we want to use on our platform. We’ll have various uses for it but primarily, it can be used by the people with the highest donor scores on the platform to themselves act as evaluators to some extent.

There’s also a phase 3 that is in the distant future at this point. Hopefully, one day, we’ll figure out all the legal problems of trying to institute a new kind of commodity. Then, hopefully, we’ll be able to institute these impact credits in a way that will make it possible to trade them on a market against US dollars or against some other kind of currency.

Phase 3 is particularly interesting because at the moment the “investors” on the platform are donors who try to use all of their specialized knowledge to make great donations. But eventually, we want to also bring for-profit investors onto the platform, investors who will bring all of their knowledge about (say) angel investing and all the experience from the startup sector into the nonprofit sector. They would try to earn impact credits to then sell them on the market.

We’re modeling this market ecosystem after that of carbon credits. Large buyers like governments could have like limit buy orders on our impact exchanges in order to encourage seed investment into early-stage charity projects.

More on our long-term vision can be found in the article “The Retroactive Funding Landscape.”

Current Challenges

Project scouts. I already mentioned that at the moment we’re looking for project scouts. So if you have some insider knowledge of some area and feel able to make highly impactful donations in that area then you can now share that knowledge with other people and thereby also leverage the donations of people with less knowledge of that area.

They will follow your donations in order to make almost as impactful donations. At the moment you can also import your historical donations to the platform to get a higher donor score. If you have a lot of donations to import, you can ask me and I can do it for you.

At the moment, you’d be able to leverage some fraction of the $391,000 donation budget of the 38 donors who have expressed interest in using the platform.

Donors. We’re also looking for donors of course – donors who just want to use the platform to allocate their donations but don’t think that they have any special knowledge of which project should receive them. These donors can just check out our top recommendations or pick out particular top donors that they want to follow.

If that is you, it’s crucial to us that you register your interest, so that we can increase the incentive for project scouts.

Investments. Finally, we’re also a Delaware public benefit corporation. So if you’re an impact investor or an angel investor and would like to invest in our project then we’d love for you to get in touch.

Impact Markets is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Disclaimer: This article covers many projects in the retroactive funding space. I’m a cofounder of AI Safety Impact Markets, so that’s the project that I know really well. I I’m much less knowledgeable of the other projects.

This is an edited transcript of an eponymous talk. I’ve held this talk at EAGx Berlin 2023 among other venues.

The Problems

The basic idea behind retroactive funding is that there is some sort of buyer (also known as retrofunder) that provides funding as an incentive. This could be a government, some sort of protocol in the blockchain space, a foundation, a startup in a growth sector that’s doing R&D, a company that has a corporate social responsibility (CSR) budget, impact investors, or donors – all sorts of altruistic actors. The idea is always to commit to paying out some sort of reward to create an incentive for project developers and investors. (I’ll always say investor when someone is at least partially profit-oriented and funder or buyer otherwise.)

With this incentive in place, these project developers can go to investors and ask for funding in the form of equity or debt in order to realize their project. Then, if the project is successful, the buyers can reward both of these parties. These financial flows are indicated in the diagram – but of course, they only happen conditionally on this project actually producing something that the buyer values.

This solves a few different problems, and depending on the temperament of the buyers, some of these might be relevant in particular cases or they might not be relevant.

Knowledge & Overhead

One problem that is particularly important for us at AI Safety Impact Markets is what has been called the long-tail problem, a particular kind of problem that can be classed under problems that relate to an imbalance in knowledge about a field (in particular very local conditions relevant to its implementation) or conversely the overhead that a funder would have to invest to acquire all of that local knowledge, for example by hiring a lot of different specialists.

The long-term problem in particular describes the problem that there are often a lot of small startup projects, independent researchers (in our case), or really small think tanks. These projects are so small and so many that the large funders can’t review all of them without incurring disproportionate costs. So they could just give those projects money at random – but on average these projects are not value valuable enough that it would be cost-effective for the buyers to do that. It would also attract a lot of fraudsters and bad actors to the space.

So the funders would have to invest a lot of time investigating these projects to find even one that they could make a grant to. And this grant would have to be relatively small. They’re much better off ignoring this whole space, even though there are gems in it, ignoring the whole long tail and just making a few big grants to organizations that already have a proven track record: well-established bigger organizations.

How do we serve this long tail? We can set an incentive in order to attract investors to the space. These investors will have specific private information about the individual few projects that they want to support. These investors might have particular special knowledge or might be active in particular spaces where they know the people really well. So they will have very good local information about a few of these projects. And taken together all of these investors will have specific knowledge about a lot of these projects in this long-tail. They can provide this knowledge to the market and can even make a profit off of it. In some cases, buyers will be ready to pay a premium for this service, because, even so, it’ll be cheaper than the counterfactual where they have to invest tremendous effort to try to acquire all this specialized expertise themselves. In return, they can capitalize on the extremely high-value opportunities in the long tail that would otherwise be very hard for them to recognize.

Access to Talent

Access to talent is a pretty specific case where a startup in a growth sector wants to get some R&D done for which they would hypothetically have to attract top talent in some space, maybe even university professors. Being a startup, they don’t have the funding to do that. They don’t have the big name that it would be interesting for these people to work for either, so this is basically not an option for them.

But the startup can become its own outcome buyer! Some version of the startup in the future has a lot of money. The startup has a lot of money in the futures in which it is actually successful in realizing its project. So it can make a binding commitment to reward the people who have contributed to that success, in the futures where they achieve the success. (That can be legally formalized and standardized in our setup.) It enables these startups to draw on this top talent to produce the open-access research that helps them get off the ground without the talent actually having to work for the startup or otherwise speculate on its success in particular. Investors will speculate on the success of the whole space (rather than a particular startup), and the researchers can get their funding from the investors.

This is particularly interesting for startups in growth sectors where they can just produce open-access research because competition is not such a big factor for them. Competitors are just other people who help them with the marketing of the whole sector.

Risk-Aversion

Now finally, risk aversion. This is currently the major use for impact markets. There are already a lot of government actors and large foundations that are using retroactive funding (or outcome-based funding) in order to realize positive change in the world. And they’re often doing this because they are risk-averse. They want to realize change with low variance and want to outsource the carrying of the risk to for-profit investors. And that’s why they are using retroactive funding and why they’re also happy to pay a premium.

The Ideal Solution

The whole ecosystem that I imagine around these mechanisms is a bit more complex. It’s the ideal solution that I envision at the moment. It’s copying what works for the carbon credit space, following the analogy that we want to establish something like carbon credits but for any kind of positive social impact.

Clear Standards for Claims

Some of the key things that I want to see are that we have clear standards in order to get buyers into the space. In order to get buyers or investors into the space, you need to have clear standards where everyone can agree on what it is that they’re trading in the first place and what it is that these buyers are buying or what these investors are investing into. Without a shared understanding, it’s going to be very hard to convince investors and buyers to purchase these constructs.

Public Registry and Ledger

What I think we’ll also need is a public registry and a public ledger in order to prevent fraud – and I’m mostly thinking of double-issue fraud here. The sort of fraud where someone is trying to sell more than 100% of some positive impact they have generated, or maybe even that some other party has generated.

They might, for example, issue a claim to have realized some impact on one marketplace and then issue that impact claim again on another marketplace. And if these marketplaces are sufficiently disconnected, no one might notice that these impact claims overlap or are identical. And then they can sell more than 100% of what they own.

They might also sell shares in these impact claims privately to different buyers and so sell more than 100% of them because these buyers don’t know of each other, and don’t know that, in aggregate, they have bought more than 100% of the impact claim.

These are things that can be prevented by having all of these transactions and all of these claims in public where they can be vetted. And that is something that the patent system currently struggles with. But we can learn from the problems of the patent system and hopefully prevent a lot of these problems from the get-go in impact markets.

With all the standardization and registries and so on, we are in the top right corner of this diagram, where it says standardization. That is something that I would like to see done by these standard-setting firms.

Evaluation of the Total Impact

When it comes to evaluation – that’s in the top left corner – I would like to see evaluations of the total impact and not only our particular cherry-picked outcomes. Because it can happen that some outcome, some positive outcome, comes at the expense of another positive outcome. And if a charity realizes some positive outcome but thereby harms some other positive outcome, then that outcome should trade at a lower valuation than an outcome that has been achieved without such a negative externality.

And there’s also disagreement over what outcomes are positive in the first place. So if outcomes are uncontroversially positive, they should also be valued higher than outcomes that are controversial in terms of whether they’re positive or not.

Prediction Markets

Prediction markets can also help debias these evaluations in particular cases. If you have questions about that, you can ask them in the comments. I think prediction markets should also play an important role in these markets. The prediction market should start as early as possible, the moment a project starts fundraising.

A Small Number of Markets

I think that we should aim for having as few markets as possible – for deeper liquidity, but mostly for the aforementioned reasons of incentivizing moral trade.

When you evaluate projects by their total impact, then you’re incentivizing the project developers to morally trade with other project developers who might have different moral opinions. So this incentivization is a very valuable aspect of having fewer markets. But then fewer markets also provide deeper liquidity, so it’s a win-win.

Deflationary Auctions for Investors

I think we need deflationary auctions for investors so that they try to come in as early as possible. Currently, charities struggle with the problem that donors wait out other donors. Donors try to keep their money in the hope that other donors will already fund some charity, and then only donate if it’s absolutely necessary because otherwise they can keep their money for some niche project that no one else would fund. That is basically a defection in an assurance game.

So we want to establish an auction mechanism that incentivizes the investors to invest as quickly as possible into charitable projects so that they can be funded more quickly. That way, the uncertainty for the charities over whether they will or will not reach certain funding goals is as little as possible and lasts for as short of a period as possible. We, for example, use a bonding curve auction for this purpose, which absolves the user from having to deal with the details of the auction mechanism.

Assurance and Coordination Among Buyers

This, I think, is very important, for the same reason, except that buyers are not profit-oriented. So you can’t just use an auction mechanism for this, but there are of course assurance contracts you can establish – assurance contracts or even dominant assurance contracts. Or you can use the S-Process. There are nice videos by Andrew Critch on YouTube on how the S-Process solves this problem.

It uses marginal utility functions, and many people will probably not have clear intuitions of what their marginal utility function is. So I imagine that it might be possible to create an 80/20 version of the S-Process that is suitable for use with a wide audience of lay donors. But there are also other coordination mechanisms that are already tested on wide populations, for example, Pol.is could serve to order outcomes by their uncontroversiality. And that is something that evaluators could use.

Then of course my ecosystem would also include exchanges and retailers to do the sales and for all these auctions to take place. So this is the vision for what I would like this ecosystem to look like. And of course, that vision is evolving. People might have different opinions on what exactly this should look like and which of these are actually important. I also imagine that over time I will update on a lot of these. But at the moment I think that these are the most important aspects of this ecosystem.

The Vision

The long-term vision is to integrate charities into the for-profit sector. Currently, there is an unnecessary separation between them. If charities can incorporate not as non-profits but as public benefit corporations, they can harness this ecosystem and generate more positive outcomes more quickly.

Impact Certificates as “Patents on Impact”

Building blocks for that are what we usually call impact certificates. You can think of them basically as “patents on impact.”

The problem with ideas is that they are just public goods. So by default, an idea would not be monetizable for a person. So, by extension, a person with an idea would not be able to attract seed funding to get this idea off the ground, because no profit-oriented investor would want to invest in a public good like that.

But then if you have a government that assures you that if you register this thing as a patent and you’re the first person to register it as a patent and it’s actually yours, then they will ensure that no one else commercializes the idea unless they license it from you.

We want to establish the same principle where, if you have a public good or maybe network good or something of this sort and you make it artificially excludable through this quasi-patent on impact, then investors can be incentivized to come in and provide the seed funding for you. So it’s about creating artificial excludability for impact.

I imagine that most of the trades and most of the incentive setting from buyers should probably not happen on the level of the individual impact certificates because they should probably be traded more like products where a charity puts out a lot of small impact certificates and just sells them whole, not fractionally.

If you’re running a think tank, then that’d be one impact certificate per blog post or paper or something that you put out. That way it’s really clear who the authors are, who has contributed to it, and who (a priori) owns which fraction of it. I’d like to keep them as small as possible, and as easy to intuitively grasp as possible so that there are also cultural norms that enshrine ownership.

Impact Credits as Financial Instrument

All of the actual incentive-setting and trades and so on should probably rather happen on the level of the impact credits. That’s what we are mostly focused on at the moment, what we want to establish. Our long-term vision is to have a market, like the carbon credit market, but for impact credits for any kind of positive impact where buyers like governments can just set their limit orders.

Investors that have invested in some sort of successful altruistic project can gain their impact credits from the validators, auditors, or evaluators and sell them into the buy orders that these governments or foundations have on these markets. I think that’s the vision that will probably streamline this ecosystem as much as possible, mostly going off the example of carbon credits.

The Landscape

What follows is my overview of the landscape.

Outputs, Outcomes, or Impact

First of all, I want to make a distinction between outputs, outcomes, and impact.

Here is one example of an output-based contractor: If you go to a restaurant, you pay the restaurant for the food they make for you. But if you go into the restaurant already relatively full and you’re super full afterward, then you don’t pay them more than if you go into the restaurant completely famished, and you’re just a bit full afterward. So it’s not about the outcome for you that you pay them for, but the output.

In contrast, what is already relatively well established is outcome-based funding.

1. There are the well-established social and development impact bonds that governments and large foundations are using to incentivize altruistic projects or improvements in living standards in various countries, or nationally for various public goods projects. These are usually focused on outcomes, so very specific metrics that project developers need to reach.

2. OutcomesX, as the name says, is focused on outcomes. They have the Impact Genome in the background, which is a project that has standardized a lot of outcomes, to the tune of 170+. It’s a very sophisticated system that they’re already using that is completely focused on outcomes.

3. I have not managed to get in touch with Ixo, but they also seem to have a pretty mature system that I think is outcomes-based. There is a podcast interview with the founders of Ixo. That is probably a really good source of information on Ixo.

4. Then there are of course carbon credits. Carbon credits are our paragon of how we think this market should work, except that we want to focus it on impacts.

5. There are advanced market commitments, another example where incentives are created for outcomes.

Prizes are used for all sorts of different things. So there’s no clear line to draw there whether prizes are used for outputs, outcomes, or impacts. There have probably been plenty of prizes for all of these.

There is the Retroactive Public Goods Funding project of the Optimism layer 2 blockchain on Ethereum. They are retrofunding things, efforts that have helped their ecosystem. That’s a very nice example of something that could just as well work for a for-profit. Optimism is a foundation, but the principle is universal, at least in growth sectors. A very nice example of this use of impact markets.

I’m not sufficiently briefed on the details of how they conduct their evaluations (and maybe I wouldn’t understand them if I were briefed on them) because I don’t have that in-depth knowledge of their tech stack unfortunately. So that’s why I’m leaving it a bit open whether they are focused on outputs, outcomes, or impact. I would imagine that they’re probably rewarding all three of them in different cases.

We’ve reached the last three projects that I will describe in some more detail, in particular, because they are focused on impact.

1. That’s the Hypercerts Foundation, they are standardizing hypercerts, which will hopefully provide what they call the basic data layer for this whole ecosystem. It’s so foundational that it can be used for basically anything, but they’re very interested in making this ecosystem about impact and not about outputs or outcomes. So they’re interested in getting the impact evaluators in their ecosystem to take all outcomes into account.

2. Manifund is another project that is focused on impact. I’ll mention them later in more depth.

3. Finally, there’s Impact Markets or AI Safety Impact Markets, our own project. Clearly, we’re focused on impacts.

Tradeability & Focus

Here’s a different taxonomy. In this taxonomy, you can see on the vertical axis the particular problem that the projects are tackling. So whether it’s access, which I elucidated in the context of growth startups; whether it’s risk, the typical use case for impact markets at the moment; or whether it’s knowledge or overhead, local specialized knowledge, the use case that we are particularly focused on at the moment.

On the other axis, there is the mechanism by which these trades are happening.

So on the left, there are some simple mechanisms. They are usually not traded at all. There’s one sale happening when in the end, when a project is successful and it gets rewarded. There are exceptions, but lots of price contests, there’s just a person who does something, and then in the end they win a prize for it.

The little icons in the top right corner indicate whether it’s currently possible to make money with the system or not, in a profit-oriented fashion.

1. There are exceptions to that, but I still put prizes in this category.

2. Social and development impact bonds also oftentimes fall in this category. At most there is one investment.

3. I think that Optimism is also simply rewarding people at this point. That might change. OutcomesX also doesn’t have investors at the moment. That might change too; hence the second mention of OutcomesX over to the right. OutcomesX is sort of thinking about getting for-profit investors involved, but there’s nothing really on the horizon.

That brings us to equity. These can be traded around among investors, lots and lots of trades can happen here. They can be traded fractionally too. They can change hands a lot.

1. I don’t know whether this is the case with Ixo. I think it is. They’re doing this on the Cosmos blockchain, so I’m sort of imagining that it’s the case, but I don’t want to spread misinformation. Please try to find out yourself.

2. In the case of Manifold, trade is possible. I will show some screenshots later.

3. Hypercerts are also traded on Ethereum and maybe some L2s at this point.

But all of these are individual assets that are issued by project developers to represent excludeably the impact of their work.

That brings us over to the right side into the space of commodities.

1. We as AI Safety Impact Markets are of course aspiring to establish impact credits as commodities. So that’s not the status quo, but our hope is to go for that format. Currently, you also cannot make profits with our system, but then at some point hopefully when all of this is mature, we hope that we’ll have the legal basis established to make that possible.

2. Carbon credits have already achieved that. So if you want to sell carbon credits or gold or other commodities, you don’t have to every time register your project with the SEC.

Target Use-Cases

And here is yet another taxonomy to give you an even better overview of this landscape. This is about what these technologies are used for at the moment. Most of them are a bit bunched up down there in the deployment space.

1. Carbon credits are mostly used for deployment at the moment – for whatever reason.

2. I think development and social impact bonds are also mostly used for deployment of existing interventions rather than research and development.

3. OutcomesX, judging by the Impact Genome standards, seem like they are mostly dealing in all things deployment, but there are a few outcomes standards that refer to research.

4. From what I could gather, that’s probably also true for Ixo. Again please ascertain that yourself.

5. When it comes to R&D, I have classed Optimism as partially for-profit R&D. It’s not quite correct since they are a foundation, but this could hypothetically be just as well used by a for-profit.

6. We are currently very much situated in the Impact R&D sector, AI safety in particular. That can change. Once our system is mature, we want to be able to just open this up generally for any kind of positive impact, be it profit-oriented or not.

7. Hypercerts are a very foundational technology, so they also cover everything.

8. Prizes have been used for various different things.

9. And Manifold is also in principle generic. They have of course focused on particular cause areas, but it’s not so easy to distinguish them in this system.

Projects

Hypercerts

Now I’d like to go a bit more in-depth on hypercerts. Earlier I mentioned standardization and the analogy of the “patent on impact.” This is exactly what hypercerts provides. Their inventor, David Dalrymple, has thought about what sort of schema would be useful for impact claims. You can think of a hypercert as a form with a few particular standardized fields.

There is a field for the description of what you’ve done; there is the period during which you’ve done the thing; there is a field for who has contributed, so who should be the co-owners of the impact initially; and there are a bunch of advanced fields.

Mostly it provides a clear standard and that is where its value lies as a foundational data layer for impact claims. So whenever someone wants to file some sort of impact claim they can now use the standard – be it on a blockchain, as it’s being used currently mostly, or just in whatever other random database. As long as it follows the standard, it establishes compatibility and that is the main benefit that hypercerts provide.

At the moment, outcomes are configurable but the people of the hypercerts team assure me that the impact evaluators will be held to take all outcomes into account regardless of what you specify there.

And interestingly rights are configurable. With this analogy to patents on impact, I’ve been referring to a right to commercialization, a right to this retractive funding, that (the right) you’re selling, licensing out, trading.

That is not necessarily the case with hypercerts. You can also just have some other right, for example public display, where whoever holds this impact certificate is allowed to display it publicly as their own. You can just configure this freely.

Hypercerts are being actively used. There are several ten thousand hypercerts in circulation and some 20,000 unique addresses are using them. So probably also thousands of people.

Hypercerts provide the basic data layer for impact claims. On top of that are attestations, valuations, and so on that provide legitimacy to particular hypercerts that deserve them.

Above that are all the different kinds of auction mechanisms, for example the S-Process that I’ve mentioned before, that can be used to allocate funding. Funding can take different shapes and can come from different sources. So this is what this nice diagram shows that I’ve lifted from a presentation by Holke of the Hypercerts Foundation.

Manifund

Then there’s Manifund. Manifund uses a hypercert-like form but it’s much simpler – just has a title, a subtitle, and a description. So if you want to make your Manifund description compatible with hypercerts, you have to include all the other mandatory fields of hypercerts in the project description.

They’re also including a few other fields that I think are not about the description of the impact claim. Some are for organizational details and to make the project easier to find.

They run this for a bunch of different cause areas. They started with the ACX Mini-Grants on forecasting. That’s why there are a lot of projects in that cause area. Then of course technical AI safety is a very popular field, so a lot of people in that space have also filed their impact claims with them.

The flow is usually that someone wants to do something, they create a project, they attract seed funding from investors, and then if they reach their funding goal, the money is paid out to them. Then they realize the project, and only when the project has succeeded, does the project page transition into the state of an actual impact claim. Then outcome buyers can come in and reward these investors by purchasing fractions of the impact claim from them.

OutcomesX

OutcomesX is a really sophisticated system that builds upon the Impact Genome, which is a decades-old system that has had a lot of time to mature and to standardize 170+ outcomes.

OutcomesX has that in the background for the standardization and the registration of impact claims. OutcomesX itself is basically the exchange or the retailer or broker for these outcome claims and allows buyers to purchase them and thereby retroactively fund the project developers. They don’t have investors yet at the moment. That is something that is potentially on the horizon, though.

The most interesting part is the level of sophistication of the standardization of the outcomes that they have achieved. Impact Genome rates the evidence by how extensive it is, by its rigor, relevance, and validity. You also get the cost at which the outcome in question has been achieved and a benchmark from among the other projects that they have in their database so that you can compare it with them.

They have a nice video that gives a really good overview of what outcomes X is doing.

AI Safety Impact Markets

Finally of course there is our own project, AI Safety Impact Markets. Our thinking has been the following: Since all of these things that have to do with equitization of impact claims are legally really difficult, what we want to start with is something that is really legally safe and simple, something that everyone can do everywhere in the world without having to worry about legal implications, where they don’t have to be accredited investors to participate in it, where the barrier to entry is as low as we can get it.

The basic intuition behind it is that capital markets reward investors for making very forward-looking prudent investments that pay off. These profits enable the investors to reinvest more capital and so have an even greater influence on price discovery. The better the investors, the greater their influence on prices. That drives the price discovery forward.

This price discovery is what we are currently completely focused on. We’re transferring it to the non-profit space. So if a donor makes particularly good donations to projects that are later successful and generate a lot of positive impact, then these donors should be rewarded for that and should have a greater weight in the funding allocation of other donations.

We do that through an organic regrantor system where the better the decisions of a donor have been (the better the donation track record of the donor), the greater the donor’s influence on the recommendations that we’re making on our platform.

So we are providing a crowdsourced charity evaluator, sort of like a Yelp or Google Reviews for charities. People who just want to donate and don’t want to do a lot of research can just follow the recommendations that the platform generates.

People who think that they have some private knowledge register their donations on the platform to build their track record. The better their track record, the greater their influence on the recommendations that the platform makes. So their incentive is to influence the recommendations on the platform in favor of the charities that they think are the most impactful ones in order to leverage others’ donations. Hence the intuition that they are quasi-regrantors for the other users of the platform.

That’s the first two phases. Now for the third and final phase.

You’ve seen this diagram before. This is where we want to get to in the end, the structures that we want to establish in our phase 3. To get there we will have to fundraise for all the legal costs that come with that and all the investigations into which country to even start this in.

Call to Action

For Investors

So if you are an impact investor or any kind of investor, this could be interesting for you. We will probably position ourselves in the standards and verification space where the revenue comes from buyers (account fees), project developers (registration and verification fees), and other evaluators/auditors (accreditation fees). We are a public benefit corporation.

For Donors

This brings me to this final call to action. If you’re interested in simply using our charity evaluator for AI safety to get donation recommendations, then that’s all live on our website, impactmarkets.io. Please just go ahead and use it.

But please also go to bit.ly/donor-interests and register that you are interested in using it, because the more interest we have registered, the greater the incentive for project scouts to actually help you with their hopefully wise recommendations for projects.

At the moment the recommendations are not super robust. We want to improve that. For that, we need these registrations of interest.

It’s a very quick one-minute form. You’ll be asked to fill in your name, email address, and your annual donation budget. You can ignore the rest of the fields if you like.

We put the aggregate, the sum of all of that, on the website. At the moment we are at $390,000, but we want to drive this up to a million or more in order to create a strong incentive for project scouts to come in and try to move that money to the projects that they think are the most impactful ones in the AI safety space.

For Project Scouts

Conversely, if you’re a project scout and you find it interesting to move some of these $390,000 that we already have, or potentially more soon, then please register your donations, bring your favorite charities that you think are the most impactful ones onto the platform, register your donations to them, and signal boost them on the platform so that other donors can see them.

That’ll make it legible to other donors that these projects are good investments and they’ll donate to them. You’re leveraging other people’s donations with your recommendations, so the time that you spend researching your donations will be even more valuable. You’ll become a regrantor for the other users of the platform.

Now we’re within one of these rare windows!

But before we jump into the FAQ, a quick announcement:

We are looking for new projects and expressions of interest from donors!

1. If you’re a donor who doesn’t want to spend a lot of time researching your donations, you’ll be able to follow sophisticated donors who have skin in the game. Our impact market doubles as a crowdsourced charity evaluator for all the small, speculative, potentially-spectacular projects across all cause areas. You’ll be able to tap into the wisdom of our top donors to boost the impact of your donations. Please indicate your interest!

2. If you’re a donor who has insider knowledge of some space of nonprofit work or likes to thoroughly research your donations, you can use the platform to signal-boost the best projects. You get a “donor score” based on your track record of impact, and the higher your score, the greater your boost to the project. This lets you leverage your expertise for follow-on donations, getting the project funded faster. Please indicate your interest!

3. Are you fundraising for some project, as individual or organization? Please post it to our platform. No requirements when it comes to the format or scope, so you can copy-paste or link whatever proposals you already have. We want to make it easier for lesser-known projects to find donors. We score donors by their track record of finding new high-impact projects, which signal-boosts the projects that they support. Attention from top donors helps you be discovered by more donors, which can snowball into greater and greater fundraising success.

4. If you are a philanthropic funder, we want to make all the local information accessible to you that is distributed across thousands of sophisticated donors and helps them find exceptional funding gaps. We signal-boost that knowledge and make it legible. You can use cash or regranting prizes to incentivize these donors, or you can mine their findings for any funding gaps that you want to fill.

Please let me know if you have any questions, below or in a call.

The same goes for the FAQ: If your question is not answered here, please pose it in the comments!

Contents

1. General questions

   1. What problems does it solve?

   2. What is a project?

   3. What is a creator or charity entrepreneur?

   4. What is a specialist donor; what is a generalist donor?

   5. What is a funder?

2. Questions about the platform

   1. What is the donor score?

   2. When and how are projects evaluated?

   3. What are your long-term plans?

   4. Last time I checked you were doing something with impact certificates, though?

3. Questions from charity entrepreneurs

   1. What does this platform do for me?

   2. What sorts of projects can I post?

   3. How long does it take to submit a project?

4. Questions from specialist donors

   1. What does this platform do for me?

   2. Can I make money with this?

   3. Project X doesn’t make sense if it receives less than $10k. I love it, but I only have $1k. What do I do?

5. Questions from generalist donors

   1. What does this platform do for me?

   2. How do I know that the donors I’m following aren’t just good forecasters but also have good ethics?

   3. How do I know that a project still has room for more funding?

6. Questions from philanthropic funders

   1. What does this platform do for me?

   2. What if I’m unhappy with the scoring?

   3. What if there are funders who defect against me by idly waiting for me to post the same prizes they want to see posted?

7. Questions about impact markets

   1. Is the goal to replace the current funding mechanisms?

   2. Is the goal to replace the current market mechanisms?

   3. How good is it?

   4. Can it go bad?

   5. Isn’t it in the interest of funders to promise funding but then not pay up?

   6. Does uncertainty really decrease over time?

   7. Do impact markets instead risk centralizing funding?

General questions

What problems does it solve?

Donors and grant applicants face the following three problems at the moment:

1. Charity entrepreneurs are known to waste a lot of time on redundant grant applications – each tailored a bit to the questions of the respective funder but otherwise virtually identical in content.

2. Donors, especially if they are “earning to give,” often don’t have the time to do a lot of vetting. Funds and donor lotteries address this, but a team of fund managers needs to win their trust first, which is not a given, and maybe they don’t want to take months off work in case they win the lottery.

3. Larger funders generally don’t want to invest much more time and money into vetting a project than it would cost them to fund it. Hence funders are forced to ignore projects that are too small.

This is our solution:

1. We promote impactmarkets.io as the one platform where grant applicants can publish their project proposals. No particular format: There’s a Q & A system though for funders/donors to ask further questions as needed. Questions and answers are public too. Funders can subscribe to notifications of new, popular projects in their cause areas.

2. When a donor supports a project, they can record that. When a project claims to have succeeded, some experts evaluate it. Eventually early donors to successful projects (“top donors”) will stand out as having unusual foresight, especially if they can repeat this feat several times. Donors who don’t have the time to do as much research can follow the top donors to inform their own donations.

3. Larger funders can (1) also follow the implicit recommendations of top donors, (2) encourage top donors with cash or regranting prizes, and (3) recruit grantmakers from the set of top donors.

There are a host of other benefits in various specific scenarios. You can read about them on our blog.

Eventually we want to grow this into an ecosystem akin to the voluntary carbon credit market (phase 3). But for now only phase 1 is relevant.

What is a project?

What we call a project is some set of actions that creators or charity entrepreneurs plan to carry out within some time frame. Good examples are blog articles, scientific papers, campaigns, courses, etc.

Whole charities (like the whole of the Center on Long-Term Risk rather than any one piece of research) are a bit of an awkward contingency because they don’t have any obvious “completion date,” but they qualify too. We’re considering methods for how we can evaluate them too.

What is a creator or charity entrepreneur?

We call a creator or charity entrepreneur someone who publishes a project on our website to fundraise for it. They are usually researchers, founders, entrepreneurs, etc. Founder would be another obvious choice, but the term creator is more general and harder to confuse with funder than founder.

What is a specialist donor; what is a generalist donor?

Both of them give money to projects, but their ambitions are different.

Generalist donors either don’t have the time or the specialized knowledge to evaluate projects. They want to use the impact market like a black box, a charity evaluator that makes recommendations to them.

Specialist donors have the time or the special knowledge to form a first-hand opinion on projects – be it because they are experts in a relevant field, because they are experts in startup picking, or simply because they are friends with the people who run a particular project. They use the impact market to make recommendations and thereby leverage the donations of the donors who rely on them. They may also be after the prizes that funders might provide!

What is a funder?

Funders are basically large donors. They can behave exactly like other donors, but they can also provide prizes to incentivize other donors.

Questions about the platform

What is the donor score?

The score is computed in three steps:

1. The contribution of each donor to a given project is calculated as a fraction that is greater if the donor contributed to a project earlier. Earliness here is not about sidereal time but about the order of the donations, so it doesn’t matter whether there’s a day or a year between the first and the second donation. The standard score also takes the size of the donation into account.

2. Eventually many projects will complete and then get evaluated. The result is a score that expresses the evaluators aggregate opinion on the relative impact of the project.

3. Finally the per-project contributions and the per-project scores are multiplied and summed up for each donor. This results in the scores that form the donor ranking.

When and how are projects evaluated?

Each project has an end date. The project creator can edit this date in case things take longer than planned, but at some point the date will be in the past and the project will really be complete. At this point it can apply to be evaluated.

We want to get a number of evaluators on board to consider the project artifacts and pass judgment on them.

The focus here will not be to make great contributions to priorities research but rather, if the project is (say) a book, to establish whether the book got written at all and whether it looks like someone has put effort into it. Ethical value judgments will be embedded in these assessments but we can hopefully find multiple evaluators so that, in controversial cases, the scores can average out.

The guideline for the calibration of the scores will be something along the lines: Suppose this book/paper/etc. didn’t exist, and I wanted to make it happen. How much would I have to pay? Or conversely for harmful projects: If there were a fairy who let’s me undo this book/paper/etc., how much would I pay to have it undone?

What are your long-term plans?

There have been changes to the funding landscape in 2022. Such vicissitudes keep our long-term plans in flux. But at the moment we’re aiming to create a market that is similar to the voluntary market for carbon credits. (These are also called “carbon offsets,” but the term “offset” would be confusing in our context.)

1. In the first phase we want to work with just the score that is explained above. Any prizes will be attached to such a score.

2. In the second phase, we want to introduce a play money currency that we might call “impact credit” or “impact mark.” The idea is to reward people with high scores with something that they can transfer within the platform so that incentives for donors will be controlled less and less by the people with the prize money and increasingly by the top donors who have proved their mettle and received impact credits as a result.

3. Eventually, and this brings us to the third phase, we want to understand the legal landscape well enough to allow trade of impact credits against dollars or other currencies. We would like for impact credits to enjoy the same status that carbon credits already have. They should function like generalized carbon credits.

Last time I checked you were doing something with impact certificates, though?

What we’ve been calling a “project” is something that can issue one or more impact certificates. Our platform still lists the existing certificates, but that’s merely an archive at this point. There is a chance we might return to this format, especially if we choose to found a nonprofit branch of our organization, but for the moment we have no such plans.

We’ve encountered three problems with impact certificates:

1. Charity entrepreneurs are hesitant to issue them because they need to define what their plans are and who their contributors are in some detail and commit to never issuing overlapping certificates. That requires some thought and coordination, and without a fairly strong promise of funding, few charity entrepreneurs are ready to put in the time and effort. That was compounded by the problem that hardly any funders were interested in using our or any impact market, so that there never was any such “strong promise of funding.”

2. Trading impact certificates is only legal between accredited investors in the US. Trade between accredited investors internationally (esp. US, Canada, EU, UK, and India) is sufficiently recondite of a problem that we haven’t found any experts on it yet. Additionally, we would not be allowed to help these investors coordinate either without running afoul of broker-dealer regulations. Figuring this out just for the US and accredited investors is something that can easily cost upward of $100k in lawyer fees and might even then just fail. The combination of costs, risks, and limitation to one country and only rich people was too much for us, at least at this level of scale.

3. Funders were very hard to find. We concentrated on outreach to funders for several months, had talks at several Effective Altruism Globals and other conferences, but in the end only got two funders interested (among them Scott Alexander though!), who promptly lost most of their funding because it was tied to the Future Fund. The funding situation changed to become even more unfavorable, so that we were no longer optimistic that it might still become easier to find funders.

Finally – and this hasn’t become a problem but would have – a lot of interesting financial instruments, such as perpetual futures, will remain inapplicable to impact certificates because each one of them is doomed to have very little liquidity. Most projects on impact markets will require some $10–100k in seed funding to get off the ground. The fully diluted market cap of even the most successful projects will probably almost never exceed $1–10m. The circulating supply will be much less still. Such assets are a good fit for bonding curve or English auctions but it would be useless to try to set up order books, indices, and futures markets for them. We previously hoped to bucket them to alleviate this problem. Impact credits will hopefully one day serve this purpose.

Hence, we’ve removed impact certificates from our plans and introduced projects instead, which are perfectly laissez faire about their definition. We’ve also opted to allow no trade of anything that can be turned into dollars. We might reboot markets for impact certs when the overall conditions change.

You can think of the donor score as analogous to the total value that you would hold in retired certificate shares if we still had those. (“Retired,” a.k.a. “consumed” or “burned,” shares are ones that cannot be sold anymore.) But it’s probably just confusing to think of it that way.

The only monetary rewards that donors may receive are prizes if they make it to the top of our donor ranking.

Questions from charity entrepreneurs

What does this platform do for me?

That hinges on how much promise your project has.

Let’s say it has a lot.

1. Do you have some donors who already trust you? Convince them to donate, then bring them on the platform to register their donations. As early donors they’ll be rewarded richly by the scoring.

2. Do you know any of the top donors? If the #1 donors is an AI specialist but your project is in animal rights, getting in touch with a top donor who is a specialist in your field may pay off even if they’re only donor #8 because animal rights donors will tend to follow the recommendations of other animal rights donors.

3. Now your project has gotten a donation from a top donor? That’ll wash it way up in our list of projects so that donors will see it even if they’re not yet following that exact donor!

To wit: As soon as you have any fundraising success, you can leverage it to build greater success. The platform even does it for you!

What sorts of projects can I post?

Basically anything goes… so long as it’s legal and not super risky!

We review every project and eliminate any that seem to us like they might be harmful. But please also make sure yourself that you don’t include any classified information or info hazards in the description because all projects are public. (Would you like to make your project only accessible to logged-in users? Send us a message through the Intercom button in the bottom right to indicate your interest in this feature!)

The ideal project is something finite that produces artifacts. Our evaluators will have an easy time with projects that fundraise for books, articles, or papers because they can read them to assess them. They’ll have a hard time with projects that are about whole organizations because an organization typically does a lot and they also can’t look into the future to know what great things the organization might still accomplish. Expect organizations to be undervalued, not because they suck but because so much of what they do is shrouded by the future and closed office doors.

How long does it take to submit a project?

There’s no required format. So if you already have a funding application lying around because you already applied for funding from some foundation, then just copy-paste or link it.

Other than that, you just need to enter a title and someplace where people can send you their donations, such as a PayPal or Stripe page. You can add some tags to make it easier for your project to be found. All in all this should take no longer than 5 minutes.

If you have no application written up yet, it’ll take longer. It’s up to you how comprehensive you want to make it. One thing I like to do is to write down just the essentials (if it’s short, it’s more likely to get read too), and then to include a link to a site where people can book a call with me to learn more.

Alternatively they also have the option to ask questions in the Q & A section. No need to procatalepse them all in your description when you can just respond to the ones that actually come up.

Amber Dawn might also help you with the writing.

Questions from specialist donors

What does this platform do for me?

Have you supported any charities early on that late made it bigly?

I, for one, would love to know what fledgling organizations you support today so I can get in early too. And for you that means that suddenly your donations count for more!

Some of my friends donate up to $100,000 per year. They don’t have much time to research their donations, so they, too, would love to know about that fledgling organization that you support. Even if you just donate $100 to the organization, your $100 might leverage $100,000 from the donors who trust your judgment!

That’s one thing that the platform can do for you.

Another is that we’re hoping for larger funders to come in and to reward our top donors. They might opt for cash prizes or for regranting prizes. Either way you’ll have a lot more money to give away if you unlock any of those prizes!

Can I make money with this?

My hope is that eventually a substantial number of people can turn donating into their full-time job. They make small but really smart donations, earn high scores as a result, and then make it all back several times over from the prizes that they win.

As of early 2023 we’re not there yet, but you might as well start building up your score already.

Project X doesn’t make sense if it receives less than $10k. I love it, but I only have $1k. What do I do?

A Kickstarter type of system would solve this, right? We can’t easily implement such “assurance contracts” ourselves, but we can help you coordinate: We could offer a way for donors to pledge that they want to donate $x if all donors together pledge to donate $X. Then once the sum of all pledges reaches $X, you’ll all get notified and can dispatch your donations.

Does that sound interesting? Please let us know, e.g., through the Intercom button in the lower right. We’ll prioritize the feature more highly.

Questions from generalist donors

What does this platform do for me?

You want to donate but maybe you don’t have time to do a lot of research or you want to donate in a field where you don’t have the requisite background knowledge. Hence you’re dependent on friends, funds, or charity evaluators to suggest good giving opportunities.

But all of these have limitations: Your friends probably know of many of the same giving opportunities, so you might be overlooking even better ones. The same is true of funds, though they receive applications, which alleviates the problem. Conversely, you may know and trust them less than some of your friends. The track record of retrospective self-evaluation at funds is thin. Finally charity evaluators have a wholly different set of limitations: They put a lot of effort into their evaluations, so that they can’t evaluate projects whose funding gaps are so small that they don’t warrant the evaluators’ efforts. Plus charity evaluators don’t exist for many cause areas.

We want to solve that for you. All you need to do when you want to donate is to turn to our platform. You can:

1. View our top donor ranking, pick out top donors who share your values, and then follow their recommendations, or

2. Filter projects according to your values (using the tags) and pick out the ones that have received most top donor support.

Today we’re just getting started, but over the coming months we want to establish a new, bottom-up, grassroots type of funding allocation mechanism that scales down to the smallest projects, is fully meritocratic, and doesn’t know geographic limits.

How do I know that the donors I’m following aren’t just good forecasters but also have good ethics?

Our plan is to hand off power to top donors gradually. First all their forecasting will bottom out at the judgments of impact evaluators that we will hire. That’ll ensure that they’ll be sophisticated altruistic, but it will not immediately steel us against our own biases. Later we want to recruit impact evaluators from our top donors, increasing the organic, bottom-up meritocracy of the platform.

But then we want to transition to phase 2 of our rollout. Phase 2 will gradually put top donors on the same footing as evaluators until most evaluation is done by top donors. But even then our evaluators will still be around to steer the platform as needed to make sure it is not usurped by any amoral top donors.

How do I know that a project still has room for more funding?

We ask projects to publish their fundraising goals and stretch goals. If they have not done so, please ask them for that information in the Q & A section.

Questions from philanthropic funders

What does this platform do for me?

You can use the platform like any other donor to find great, new funding opportunities.

But we also have a special function for you: You can basically rent our top donors by offering regranting budgets to them. Those serve the dual purpose that (1) you’ll get top grantmaker talent for free, maybe even top grantmaker talent whose networks are relatively uncorrelated with yours, and (2) by announcing such a prize, you create an incentive for prospective top donors to show up and try to prove their mettle.

If that sounds interesting to you, please get in touch, e.g., through the Intercom button to the lower right or via hi@impactmarkets.io.

What if I’m unhappy with the scoring?

Are you? If so, we can easily build a custom score for you. You score the projects, and we aggregate all of your project scores into your own custom donor ranking. Please get in touch if that sounds interesting to you, e.g., through the Intercom button to the lower right or via hi@impactmarkets.io.

What if there are funders who defect against me by idly waiting for me to post the same prizes they want to see posted?

We’ve termed this problem the “Retrofunder’s Dilemma.” It’s easy to imagine a world in which there are several funders – just too many for them all to be really chummy with each other – who all insist on extremely niche scoring rules to make sure that they don’t reward any donations to good deeds that anyone else might reward too. But that would leave exactly the most uncontroversially good deeds unrewarded.

We’re far from this being a problem for our rewarding, alias retrofunding, at all and even farther from it becoming a greater problem for retrofunding than it is already for prospective funding. But if it becomes a problem, the abovelinked article lists three remedies that funders can implement and four that charity entrepreneurs can implement. Or that we can implement for them to establish coordination.

Questions about impact markets

Is the goal to replace the current funding mechanisms?

Not really, sort of in the way that airplanes didn’t replace bikes. We think that impact markets will be best suited for funding the long-tail of small, young speculative startup charity projects. But they will be rather uninteresting for projects with strong track records or otherwise safe, reliable success. They will also be uninteresting for projects that require a lot of funding from the get-go.

You read more about the math behind these considerations on our blog.

The basic idea is that projects that are > 90% likely to succeed (according to some metric of success that the funder uses) don’t leave much room for an investor to make a profit while reducing the risk further for the funder.

Additionally, a risk-neutral funder is only interested in a risk reduction if it moves an investment from the space of negative expected value to the space of positive expected value. If a project is already 90% likely to succeed, it would have to be very expensive before it could become negative EV for a funder. Such an expensive project is then easily worth the time of the funder to evaluate prospectively rather than retrospectively.

So impact markets (with risk-neutral funders) are most interesting for:

1. Projects that seem very speculative to funders, e.g., because they are new and the funder doesn’t know the team behind the project,

2. Projects that require little money to get started, so they’re not worth the time of the funder to review.

If highly risk-averse funders are involved, though, they may be happy to pay a disproportionate fee for a risk reduction from 10% to 0%! There are also funders who are limited by their by-laws to only invest in certain types of low-risk projects. In some cases impact markets may present a loophole for them to do good more effectively without incurring any illicit risks.

Is the goal to replace the current market mechanisms?

No. The financial markets have developed over the course of over a century and are accompanied with legislation that is usually phrased in such generic terms that it is nigh impossible to create a separate financial apparatus outside of it. Many cryptocurrency projects have tried to create market mechanisms beyond the reach of the law, but the law typically disagreed. More recently, there is instead a stronger push to welcome regulation and to reform the law to facilitate regulation.

We therefore consider it infeasible to try to replace the existing financial systems. Rather our goal is to create systems that reward the creation and maintenance of public, common, and network goods while interfacing with the existing financial systems in standard, regulated ways. (The closest parallel is the voluntary market for carbon credits.)

How good is it?

[This section has not been rewritten for the new “impact credits” approach. The differences are probably minor.]

We’ve been trying to get an idea of how good impact markets are by putting some rough estimates into a Guesstimate, but a lot of the factors are multiplicative and they are all hard to guess, so that the variance of the result is very wide.

Some key benefits are:

1. There’ll be as much or more seed funding as there is today. The idea is that many investors will try many different things and try to think outside of the box. Often it’ll turn out that their calibration was off. They’ll invest into lots of projects and make less money back because they were wrong about how great all the projects will turn out. These investors will gradually select themselves out of the pool, but new ones will join. We don’t know how many ill-calibrated investors join for each that is well-calibrated, but we’ve seen some data that prize contests attract investments to the tune of up to 50x the prize money, so the average investor must be fairly ill-calibrated. Our model assumes that the value is probably around 1–20x.

2. The allocational efficiency can be improved because investors can overcome language barriers. But much of the world speaks English, and the US dominates the world economy, so we’ve put this improvement at a factor 1–3x.

3. The allocational efficiency can be improved because investors are in different social circles. This factor seems more significant to us, and we’ve put it at 1–10x.

4. The allocational efficiency can be improved because investors can draw on economies of scale. They can rent one server rack for all of their projects, or they can employ one HR person for all of them, etc. We’ve put this factor at 1–10x too.

5. Charities can draw on much greater talent pools if they can use fractions of impact certificates to align incentives. (They can also pay bonuses tied to retro funding.) We think that the talent pool might grow by 1–5x.

6. All of this frees up a lot of time for retro funders. Current prospective funders such as the Open Philanthropy Project have a lot of staff who would be excellent at a very practical brand of priorities research. When impact markets free up time for them, they can devote that time to research. Evidential cooperation in large worlds alone can serve as a likely existence proof that there is a lot more to know about global priorities. We very conservatively assume that improved understanding of priorities will boost the allocational efficiency by 1–10x.

The result of the model is that impact markets are unlikely to improve the current efficiency by less than 60x or by more than 11,000x.

We think that this range is likely biased upward:

1. Our model ignores black swan events that may occur with an unknown frequency and may be very harmful.

2. It is well possible that some factor in the model actually turns out to be < 1x for some reason that we haven’t thought of.

3. Finally, we’ve mentioned before that the multiplicativeness of the model makes it very easy for it to produce big numbers. This is the prime reason that we don’t greatly trust it.

You can find further discussion of the model in the comments on this post.

Can it go bad?

The biggest concern that we’ve had from the beginning in 2021 is that prize contests (such as impact markets) are general purpose: Anyone can use them – to incentivize awesome papers on AI safety or to incentivize terrorist attacks. In fact, promises of rewards in heaven could count as prizes. If we create tooling to make prize contests easier, there is the risk that said tooling will be used by unscrupulous actors too. The very concept of the prize contest could also count as attention hazard.

Here is a summary of all of the risks that we’ve identified and our mitigation strategies.

A rich terrorism funder could, for example, copy our approach and build an analogous platform where they promise millions of dollars to donors who fund speculative approaches to terrorism, such as terror attacks that only work out in 1 in 10 attempts. We would not allow such projects on our platform or a scoring procedure with such goals, but that doesn’t keep terrorists from building their own clone of our platform.

This doesn’t need to be obviously ill-intentioned (though terrorists probably also consider themselves to be heroes). You could imagine someone cloning our platform to fund grassroots nuclear fusion research, which might lead to accidental nuclear chain reactions in the basements of hobby physicists in densely populated cities around the world.

The Impact Attribution Norm alleviates this problem to (roughly) the extent to which it is adopted (see the question about measurement above). Yet it is not obvious that it will reliably be applied the way we would like to see it applied. This article is a good summary of its limits. See also our comment. Consider for example:

1. Someone might wrongly think that the impact evaluators will reward them for posting an article that contains some dangerous info hazards.

2. This can also happen if they don’t think that the particular impact evaluators will reward them but just that at some point there will be impact evaluators that will reward them.

3. Finally, it can happen that the impact evaluators are actually mistaken about the value of some impact and that their mistaken evaluation is predictable. This applies in particular (but not only) to actions that can turn out very well or very badly – might save lives or destroy civilization – but so happen to turn out well. The longer the duration between the launch of the project and the evaluation, the greater the risk that the prize committee will see only how well it turned out and ignore the other possible world where it did not.

These risks mostly seem like “black swan” risks to us – deleterious but highly unlikely risks. We’re also quite confident that we can prevent them from happening on our platform by carefully moderating all activity.

Finally, there is always the question how easy it is already for unscrupulous actors to achieve their ends and why they are not doing it already. It is quite easy for an unscrupulous millionaire to promise a big reward for something like nuclear fusion simply by tweeting it. But this is not currently a major problem. So the legal safeguards (or some other mechanisms) that also apply to our solution must be working fairly well. That said, we’re not solely relying on them.

Isn’t it in the interest of funders to promise funding but then not pay up?

Seemingly the best outcome for funders is to incentivize excellent work with the promise of a prize but then not reward them at all but to instead put the money into prospective funding of additional impactful work.

That is a shortsighted strategy as no one will trust a funder again if they’ve pulled this trick once. I would go further and suggest that donors should not rely on new funders to pay up unless they have a history of being trustworthy. For funders this means that it’s probably in their interest to gradually ramp up their prizes, so that they can build up trust more cheaply. Another option is escrow.

Eventually we hope to have tradable impact credits so that donors can assume that any funder who suddenly vanishes will thus leave the price at an unexpectedly low level which other funders will immediately use to “buy the dip.”

Does uncertainty really decrease over time?

This article touches on this question. I don’t think it’s important whether there is always more evidence of impact at a later point. Impact markets will just be most interesting for projects for which that is true.

The second part of the answer is that we think that there is a substantial number of projects for which this is true.

An example: You can usually divide your uncertainty about how a project – say, a book – turns out into two multiplicative parts: the probability that the book gets written at all and the impact-over-probability distribution of the finished book if it gets written. Once you know whether the book got written or not, that product collapses into just the second factor (minus the “if …”).

This only goes through if you take uncertainty to mean something like the difference between the best and the worst or the 99st and the 1th percentile outcomes, which may be a bit unintuitive. If you think of uncertainty as variance, and your fully written book has either an extremely positive or an extremely negative impact, then added uncertainty over whether the book has really been written adds another cluster of neutral outcomes in the middle between the extremes. It does not reduce (or increase) the difference between the extremes, but it does reduce the variance.

Do impact markets instead risk centralizing funding?

The whole point of impact markets is to decentralize funding – so might they perversely increase it? The argument goes that the current scoring rule allows for truly exceptionally good donations – the first donation to a project as amazing as Evidential Cooperation in Large Worlds might’ve been a substantial donation. Whoever the donor might be, they’d get an enormous score boost even though they were only right once. This boost might push them to the top of our ranking for many years until finally enough other donors have gradually accrued comparable scores. That seems unlikely but also undesirable.

One variation that we’re trialing is a score that does not take the size of a donation into account but just the earliness. Every project has a first donation, so even the first donations to great projects could no longer be as remarkable as a substantial first donation to a great project could’ve been under the size-weighted scoring rule.

Another remedy is to have scores decay over time. One solution we’re trialing is to have a score that only takes into account donations from the past year.

We’ll keep monitoring this potential issue and react in case it does manifest.

We’re worried that impact markets may set incentives that prevent exactly the sort of projects that are good across many moral systems and worldviews from being able to benefit from them. (Cf. the Giver’s Dilemma.) Prospective funding suffers from the same problem, so it is not a disadvantage of retrofunding over prospective funding. Nevertheless we’ve collected a few ideas for how to address the problem if it comes up.

The Retrofunder’s Dilemma

GiveWell writes:

Imagine that two donors, Alice and Bob, are both considering supporting a charity whose room for more funding is $X, and each is willing to give the full $X to close that gap. If Alice finds out about Bob’s plans, her incentive is to give nothing to the charity, since she knows Bob will fill its funding gap. Conversely, if Bob finds out about Alice’s funding plans, his incentive is to give nothing to the charity and perhaps support another instead. This creates a problematic situation in which neither Alice nor Bob has the incentive to be honest with the other about his/her giving plans and preferences – and each has the incentive to try to wait out the other’s decision.

Let’s suppose that the charity in question is in fact the top priority of each of the donors. The perhaps most internecine aspect of this incentive structure is that this charity is more likely to be uncontroversially good from an impartial perspective the more donors there are who can agree that it is the top priority. It is exactly this most uncontroversially good charity that suffers most.

This is also something that we want to steel impact markets against.

Luckily, the outcome is in the interest of none of the participants, so that we’re potentially faced with a mere “assurance game” problem. If we can find an arrangement that funds the top charity and that all participants agree to, we’ve solved the problem.

Defection by Sellers

There are two importantly different versions of this problem. The first is the one above where retrofunders get stuck in a defect-defect equilibrium. But you could also imagine a world where funders run their separate prize contests and accept submissions not through a public marketplace like ours but through a form whose responses only they can see.

In that world it is feasible for a seller to target only those prizes that are awarded by groups that are least value aligned with them to preserve the resources of the more aligned groups for other projects.

Impact markets probably already alleviate this problem by making projects and funding decisions public and by making it impossible to target applications at particular funders only. There are also already norms against this kind of defection in effective altruism. I think the current norms still fall short, but at least they exist to some extent.

Broader or Narrower Targets

I previously thought that the panacea to maximize the counterfactually valid impact of impact markets was to make the prize questions more specific so that they could target only neglected problems. I now think that that would be tantamount to defecting in a defect-defect equilibrium that leaves the most impactful funding opportunities untapped.

The targets or research questions that a prize contest rewards can be broader or narrower: “We want to see a proof or disproof that Vingean reflection is possible” as opposed to “We want to see contributions to AI safety.” There are advantages and disadvantages to both extremes.

With impact markets we currently want to bring together the people who currently want to but can’t do good (e.g., researchers with little money) and those who can but currently won’t do good (especially for-profit investors) – at least at the current margin, because many of them are probably already doing good up to the point that they can or want to.

But there are also those who can and do do good: funders and researchers with sufficient means. They are similar in some ways and could be seen as one group.

If so, these groups are not cooperating very well at the moment, so a one-sided attempt to cooperate would just lead to exploitation. If funders take this view, they’ll defect back, and we’ll be in a defect-defect equilibrium. This means in practice that they’ll pick targets that they think no one would otherwise work on, and therefore usually targets that are highly specific.

But these groups could also be seen as distinct because funders have a lot of flexibility in what to fund, so that they have a choice, whereas many researchers may be specialized to the point where the criterion of their personal fit more or less prescribes what they need to do. If so, it seems odd to interpret their behavior as defection. If funders take this view, it becomes less clear how they should act.

Another consideration is that the defection, if it is one, is not malicious. If asked, researchers will probably be truthful about the counterfactual of their work in a world without a given prize. So even broad targets will probably mostly allow us to measure our impact.

I don’t currently know how to think about this, so at the moment it seems reasonable to start with broad targets and possibly accept some exploitation, or to start with narrow targets and gradually build up cooperation one contract at a time. Either could lead to our desired outcome of a norm where you can expect retrofunding if you do something great. In any case, different funders will probably have different preferences, and most likely we’ll be able to measure which leads to the better outcomes in the long run.

Remedies

Against Defection by Retrofunders

1. Confabs and S-Process. The most straightforward way to coordinate retrofunders is to let them talk to each other. There will probably be very few retrofunders for a long time – maybe some two to five or so – which should make it easy for them to talk to each other and hash things out. (This excludes people who fund their own work, who are sort of like retrofunders but perhaps less flexible, as mentioned above.)

   1. In the causal case, coordination mechanisms like the S-Process (or some modification of quadratic funding) could help remedy the problem.

   2. An advantage of retrofunding is that the budgets typically get announced in advance and receive a lot of attention, probably more attention than statistics in an annual report.

2. Polis. The S-Process may become harder to use for a larger number of retrofunders, esp. less sophisticated ones. But arguably the right kind of UI can still allow users who don’t understand marginal utility functions to derive benefits from it. If we do end up scaling to the level of many retrofunders or decide to consider self-funded researcher retrofunders of their own right or want to start an alternative market for only highly trusted impact certificates that is open to a larger number of retrofunders or some other variation, then there’s another promising tool for us: the coordination mechanism Polis that was used in Taiwan. It seems plausible to me that we can limit the marketplace to research questions that come out of the Polis process: A large number of retrofunders enter their research questions and upvote or downvote all the others. The Polis algorithm then gradually highlights the questions that are most uncontroversially interesting to all market participants. Then only some number of these are entered into the market.

   1. Evidential Cooperation in Large Worlds may have recommendations for how to bargain in the acausal case, and these may be transferable to the causal case in certain communities.

3. Reselling. Funders may not always consume all the impact that they purchase but may actually want to hold it in expectation of profits via a future retrofunder. This introduces the opposite incentive to the one we’re worried about.

Against Defection by Sellers

1. No private trades. retrofunders should generally not accept offers made to them in private but only those that are public on the marketplace. That prevents sellers from targeting particular retrofunders.

2. Automatic auctions. An automatic auction system like the one we want to use has the advantage that sellers can’t just refuse better offers. Still there might be multiple marketplaces, so that a seller could choose the one that is used more by people with different values, or they could sell their cert outside any market.

3. Big projects. Impact markets should generally be used to finance projects that are big enough that they require seed investments. The profit motive of the investors counters the interests of the altruists running the project and will generally push toward making public offers.

4. Norms. What would be useful for prospective and retrospective funding alike would be to (1) do community-internal advocacy to strengthen moral cooperation norms, and (2) do community-internal education to make sure we all understand/agree what we consider a defection. That way we’ll be more likely to feel bad about defecting and avoid it, and peers can pressure each other not to do it.

   1. Paul Christiano has also thought about the problem: “By ‘norm’ I mean a rule that individuals can use for deciding how much to fund each public good. Here are two plausible desiderata for a norm: (1) If everyone always follows the norm, then we end up with the optimal levels of funding for the public goods. (2) If you start with a community that follows the norm and add a bunch of new people who behave manipulatively, they can never make the original community worse off.”

   2. It stands to reason that either broader or narrower targets will lead to better or stronger norms or allow them to emerge more quickly. This would be interesting to investigate.

Appendix

Original Formulation

When we were first asked about this scenario, it was in a configuration that we think is unlikely in practice. A fictional example with fictional actors:

Open Philanthropy Project (Open Phil): AI safety is 2x as valuable as international development.

Secretive organizer named Philip (Closed Phil): International development is 100x as valuable as AI safety.

Open Phil announces a contest: Do stuff we value, and we’ll reward you up to $1m!

Closed Phil invests $1m into international development and submits it to the contest. Closed Phil offers the impact to Open Phil at $400k. Open Phil takes the offer because it looks like something worth > $800k, so they rather buy it than some other AI safety impact.

Closed Phil reinvests the $400k into international development.

Rinse and repeat until Open Phil is out of money. Closed Phil has successfully leveraged all or most of Open Phil’s money, and Open Phil has invested comparatively little into AI safety.

After three iterations, > $1.5m may be invested into international development and < $500k into AI safety. If Open Phil had not announced any prize contest, $1m + $333k would’ve gone into international development and $667k would’ve gone into AI safety, which is closer to Open Phil’s preferences, so they don’t want to do a prize contest.

In this formulation it’s not clear why Open Phil would buy the impact, so we have a modus ponens/modus tollens type of situation. Clearly, Closed Phil is continually operating at a loss, so they’re either crazy or they’re interested in the impact and capable of producing it also without the retrofunding. Hence the counterfactual effect of the retrofunding is likely minimal. Open Phil doesn’t want to do something that is not counterfactually impactful. Maybe they will pay out once (maybe at a still lower price) to not seem unpredictable to other participants and reduce trust in their retrofunding, but they will then adjust the purview of the contest to avoid this failure going forward. They can do this, for example by using separate budgets for international development and AI safety or by focusing on only AI safety.

An exception is if Closed Phil manages to deceive them, e.g. by lying about how much they invested or by claiming that they’re just cutting their losses because the project failed by their lights. They can make up some metrics as a pretext that the project didn’t meet that Open Phil doesn’t care about.

The scenario can be rephrased to one where the deception is subtle and no one intentionally retrofunds nonoptimally. That’s the “defection by sellers” scenario above.

Recap: Impact Markets

For a more comprehensive explanation of impact markets, see Toward Impact Markets.

Thanks for reading Impact Markets! Subscribe for free to receive new posts and support our work.

In short, an altruistic retroactive funder announces that they will pay for impact (or “outcomes”) they approve of. It resembles a prize competition in this way. But (1) they’ll pay proportional to how much they value the impact and not only the top n submissions; (2) the impact remains resellable by default; and (3) seed investors offer to pay the people who are vying for the prizes or provide them with anything else they need and receive in return rights to the impact and thus prize money.

It is analogous to the startup ecosystem: Big companies like Google want to acquire small companies with great staff or a great product. Founders try to start these small companies but often can’t do so (as quickly) without the seed funding and network of venture capital firms. When the exit happens (if it happens), the founders may not even any longer own the majority of the company because they’ve sold so much of it to the investors.

The benefits are particularly strong for high-impact charities and hits-based funders:

1. If a hits-based funder usually funds projects that have a 1 in 10 chance of success and switches to retroactive funding, they save:

   1. the money from 9 in 10 of the grants,

   2. the time from 9 in 10 of the due diligence processes, and

   3. the risk from accidentally funding projects that then generate bad PR.

2. Investors can thus speculate on making around 10x return on their successful investments, and they can further increase their expected returns:

   1. by specializing in a narrow area (such as AI safety) to make excellent predictions about which project will succeed,

   2. by providing founders with their networks in those areas,

   3. by buying resources at a bulk discount that founders need (such as compute credits), and

   4. by finding founders that none of the other investors or funders are aware of to negotiate deals with them where they receive a large share of their impact certificate/s.

3. Charities can attract top talent and align incentives with top talent who may not be fully sold on the charity’s mission:

   1. by promising them a share in all impact sold,

   2. by locking that share up in a vesting contract,

   3. by (possibly) sharing rights to the impact with another company that is the current employer of the talent so that they don’t need to quit and can draw on the infrastructure of the other company. (In fact, somewhat value-aligned companies may be interested in becoming investors themselves if they want to retain talent who want to work on prosocial applications of their knowledge.)

4. Individual researchers can attract funding for their work even without the personal ties to funders, e.g., because they are in a different geographic region and better at their research than at networking.

Profitability of Impact Markets

We think about this in terms of the riskless benchmark B and the ratios r_c and r_p. The benchmark B is a return – e.g., B = 1.1 for a 10% profit – that an investor expects over some time period. An investment is interesting for the investor if it is more profitable than B. r_c = c_f/c_i is the ratio of the costs that funder and investor face respectively. This includes, for the funder, the cost of the grant, the time cost of the due diligence, the reputational risk if the due diligence misses something, and, for the investor, the cost of the grant minus savings thanks to shared infrastructure, economies of scale, etc. r_p = p_i/p_f (note that enumerator and denominator are the other way around) is the ratio of the probabilities that investor and funder respectively assign to the project success. The investor may specifically select projects where they have private information (e.g., thanks to their network) that give them greater confidence in the project’s success than they expect the funder to have.

Hence, investments are interesting if r_c ⋅ r_p > B.

The graph shows the benchmark of an investment with 30% riskless profit compared to the maximum profit from various project configurations. It elucidates that an investor who can help realize a project more cheaply than the funder or thinks that it is more likely to succeed, can outperform the funder in a range of scenarios. These are scenarios where one or both parties can reap the gains from trade and save time or money.

The square between 0 and 1 on both axes is largely irrelevant. These are scenarios where the investor would have to pay more than the funder or is less optimistic about the project. Those are obviously uninteresting. But also just outside that square and around the edges, there are areas where the investor may not be interested because their edge in terms of the r_p and r_c ratios is too small. Then again a riskless 30% APY is a high benchmark.

A few examples:

If a charity already has a track record of doing something really well 10 out 10 times in the past, there is very little risk involved when they try it for an 11th time:

Maybe an investor thinks they’re 99.5% likely to succeed and the funder thinks they are at least 99% likely to succeed, and the action costs $1m for either and takes a year.

That’s r_p = 1.005 and r_c = 1. It is only interesting for an investor who cannot otherwise invest the money at 0.5% profit per year.

1. It’ll be worth little to the funder: If they value the impact at 99% probability at $1m, they’ll pay $1m/99% ≈ $1.01m for it, so $10k premium.

2. If an investor offers to carry that tiny amount of risk, they’ll want it to exceed their 10–30% benchmark after a year, or else a standard ETF investment would be more profitable to them. That’s at least a $100k premium.

3. A bid of a $10k premium (minus the overhead of the whole transaction) from the funder but an ask of $100k premium from the investor means that there’ll be no deal.

But consider a case where someone has no track record:

The investor thinks they are 20% likely to succeed. The funder thinks that they’re 10% likely to succeed. The action costs $1m for both and takes a year.

That’s r_p = 2 and r_c = 1. It’ll be interesting unless someone has a benchmark of more than 100% per year.

1. The funder will pay up to $1m/10% = $10m for the riskless impact.

2. That’s a 1000% return (or 900% profit) for the investor with 20% probability, so 100% profit in expectation, which beats most benchmark investments. Even if their riskless benchmark is as high as 30%, they’ll accept offers over 650% return. Naturally, these investors have to be fairly risk neutral or make many such investments. (If they are somewhat altruistic, they can consider the difference between the risk neutral and their actual utility in money a donation.)

3. Funder and investor will meet somewhere at or below 1000%.

It is easy to create an analogous example for the case where funder and investor make the same probability assessments but where the grant size is so small that the investor, who already knows the project, can fund it at half the price compared to the funder who would have to spend a lot of time on due diligence.

Impact Timeline

A typical product that is suitable for impact markets is a scientific paper. Papers, like many other projects, have the property that they often get stuck in the ideation phase, sometimes have to be abandoned (for other reasons than being an interesting negative result) during the research, sometimes don’t make it past the reviewers, and sometimes turn out to have been a bad idea only decades later.

When an investor wants to invest into a paper that has not been written, but which they are highly optimistic about, they may see these futures:

The x axis is the time (in years), the y axis is the impact (proportional to dollars), blue lines are possible futures, and the red line is the median future.

There are two big clusters: all the futures in which the paper gets written, published, and read, versus and all the futures in which it either never gets finished or gets read by too few people.

One to three years into the process, it becomes clear in which cluster a given future falls, particularly if it falls into the upper cluster. (Otherwise there’s a bit of a halting problem because it might still take off.) Maybe the paper has been published on arXiv and is making rounds among other researchers in the field.

After 10 years, the majority of the impact has become clear and the remaining uncertainty over the value of the Attributed Impact of the paper is low.

After 15 years, we’re asymptotically approaching something that looks like a ceiling on the Attributed Impact of the paper. Experts have hardly updated on its value anymore in years, so their confidence increases that they’ve homed in on its “true” value. (“True” in the intersubjective sense of Attributed Impact, not in any objectivist sense.)

This is a vastly idealized example. In practice it may be that a published paper that used to be held in high regard suddenly turns out to have been wrong, an infohazard, plagiarized, etc. Or it may be that it’s suddenly noticed that a decade-old forgotten-about paper (that had high ambitions at the time but seemed to fall short) contains key answers to an important new problem.

Timing of Retroactive Funding

If an investor is a specialist in some small field and profits from economies of scale in the field (e.g., the compute credits bought in bulk that we mention above), then they may expect to make a 10× profit from each retro funding that they receive. That’s the difference between the size of the retro funding at which the retro funder breaks even (ignoring interest) and the cost to the investor. We assume for simplicity that monetary and time costs (grants and due diligence) are the same. So, 2 · 10i − 2 · 5i = 10i, where i is the average seed investment. (We’re using the parameters from above where retro funders save 10× from making fewer grants and 10× from saving time spent on due diligence. We also assume that a patient, well-networked, specialized investor has twice the hit rate of the generalist funder.)

If, counterfactually, they would’ve invested this money at 30% APY, the impact market ceases to be interesting for them if they expect the retro funding to take longer than 8–9 (≈ 10.6 ≈ 1.39) years: 2 · (1 / ratefunder) − 2 · (1 / rateinvestor) = (1 + apy)years.

Here we’re comparing an investor at different hit rates to a retro funder who would otherwise have a 10% hit rate under four counterfactual market scenarios. The impact market is profitable for any number of years less than the break-even point.

If the retro funder wants to save money, they can pay out less, but will need to do so earlier. For simplicity, the following chart is only for the scenario with a counterfactual 30% APY: 2 · (1 / ratefunder) · (1 − savings) − 2 · (1 / rateinvestor) = 130%years.

A retro funder needs to take this into account when deciding how much certainty they want to buy from the investors. More added certainty comes at a higher price. They can regulate this through the size of their retro funding or through the timing. Depending on the impact in question there are usually certain sweet spots that they can aim for, and do so transparently so that investors know what time horizons to speculate on.

When it comes to our example above, it seems fairly clear whether the paper was a success (was written, published, and read by some people) after about 2–3 years. So one sweet spot may be to wait for the moment of publication (as a draft or after peer review) or after the initial public reception can be gauged. The second is interesting because investors may be well-positioned to help with the promotion.

But there are other options – less profitable options much later.

Dissolving Retroactivity

We can imagine a chain of retro funders from a particular set of futures into the present: Someone makes a binding commitment that if they are successful in making a lot of money – say, their business is successful – they will use the money or a fraction of it to buy back impact that has previously been bought by a certain set of existing retro funders who the person trusts. They can continually add new ones to this set.

This can also be formulated as a prize contest: If I’m successful, I’ll use that budget to buy impact from my favorite retro funders at a reasonable bid price. If 1 in 5 projects still fail between the time when the retro funder bought them and the time when the success happens, the retro funder may buy them at 120% of the price that the previous retro funder paid.

Under this framing there is no qualitative difference anymore between investors and earlier retro funders. They’re all just different investors with different attitudes toward risk or preferences about how they weigh the profit vs. the social bottom line of their investments. (Some of them may choose to consume their certificates, though, to signal that they’ll never resell them.) There may even be investors who choose to invest into “whatever project person X will do next,” so earlier than the abovementioned seed investors.

A startup may be interested in making such a commitment because they have the choice to either do the research in-house or at least pay for it immediately or to pay for it later and only if they are successful. Since startup success is typically Pareto distributed, they’ll have vastly more money in the futures where they are successful than they have now or in unsuccessful futures. So this deal should be interesting for most startups.

For investors it’s a question of whether they want to expose themselves more to the field or to a particular team. If they’re excited about the team behind the startup and trust that team to do well regardless of what field they go into, they’ll want to invest directly into the startup. But if they’re more agnostic about all the teams in a field but are very excited about the field, they may prefer investing into the research projects to bet on the retro funding.

Example

1. Cultured meat (or “cell/clean/c meat”) startups may require a lot more research to be done on how to scale their production and make it cost-competitive. But they don’t yet have the money to do all of that research in-house.

2. They commit to investing a large portion of the money they’ll make from going public into buying impact. Specifically they hash out particular terms with an organization like Founders Pledge that stipulate what impact related to cultured meat research they will buy from which retro funders.

3. The promise of great potential future riches boosts funding and opens up hiring pools.

4. Eventually, the now more likely future might happen, and the large budget from the exits serves to buy most of the impact from the retro funders.

Conclusion

We’ve received a grant via the Future Fund Regranting Program to work on this. If you’d like to join our discussions, please join our Discord.

Thanks to my cofounder Dony for reviewing the draft of this post! He gets 1% of the impact of it; I claim the rest.

Thanks for reading Impact Markets! Subscribe for free to receive new posts and support my work.

You can think of an impact market as a prize contest. There is a prize pool and an announcement that prizes will be awarded to incentivize certain contributions that the prize committee wants to see. Additionally, it’s possible for contestants to attract seed investments and collaborators to help realize their contributions in return for a promise to share the prize money.

Therefore, the backbone of the market are the prize committees, which we call retroactive funders or retro funders. They are, for example, foundations with a team of sophisticated altruistic grantmakers. They are transparent about the sorts of contributions (or impact) that they want to see and reward it when they see it. Some altruists or charities will be in a position to complete such actions without help, but others will need to fundraise or partner with others first. An impact market is where this fundraising and partnering can take place. It’s a variation on the idea of social impact bonds.

Impact markets promise to partially solve a range of pressing problems that altruists face. First and foremost, they allow grantmakers to delegate the “startup picking” aspect of their work to profit-oriented investors, and then only invest time and money in the minority of cases in which a project meets their bar for success. (See sections Retroactive Funding and Benefits.) These and other benefits can boost the efficiency of the funding allocation of hits-based funders by one or two orders of magnitude.

However, impact markets also bear a number of (1) benign risks that can cause them to fail and (2) more dangerous risks that can cause them to be harmful. (See section Risks.) We think it will be possible to address the risks in the second category, but we would like to be more certain of that. (See section Solutions.) Small-scale, sandboxed experiments among highly informed participants may help us test impact markets while limiting their risks, which is what some parties are now attempting. (See section Current Work.)

We hope that this article will give readers a chance to critique our plans and point out further risks or weaknesses in our defenses against known risks. If you would like to help, please join our Discord, comment, or contact us some other way.

Thanks for reading Impact Markets! Subscribe for free to receive new posts and support our work.

Impact Markets

Our current best guess is that impact markets will take the shape of four incremental additions to existing financial markets.

1. Retroactive funding. First there needs to be a norm of and ideally several institutions for retroactive funding. They have a lot of responsibility for the health of the market and need to monitor it closely. We don’t think that it’s generally feasible for them to operationalize up front exactly what the impact looks like that they want to buy (say, by pointing to particular metrics) but rather want to rely on indirect normativity: Market participants can read about them and talk to them and try to understand their values. Ideally there are many retro funders who are all highly qualified and responsible and who are getting more numerous over time and whose capital increases. Some of these retro funders may have been started with the explicit goal to incentivize more retro funders by retro-funding existing retro funders. (We’ll use the terms “retro funder” and “to retro-fund” as convenient short forms.)

2. Contracts. Anyone who founds, funds, or otherwise supports a project – a supporter – and wants to participate in its success needs to negotiate their inclusion in a contract (an impact certificate) that documents what share of the impact of the project they own. Retroactive funders will later require this proof if they are interested in buying the impact from the supporter. (Whether it’s possible to produce such proof retroactively, when it is needed, depends on the specifics of the case.)

3. Auction platform. Seed investors are probably hesitant to invest into impact if they have to fear that it’ll be hard or take a long time for them to find a buyer for it. An auction platform can streamline the interaction between the buyers and the sellers of impact.

4. Impact stock and derivatives. Finally, at some later date, charities (incorporated as, e.g., public benefit corporations) may decide to create their own stock. Alternatively, some third parties may create derivatives that track the market capitalization of all impact certificates of one charity. These will make some forms of investment easier, and charities could use them for mission hedging. Seed investments no longer need to go through each impact certificate once a charity has stock.

Notes

1. This whole document culminates in a particular definition of the Impact Attribution Norm (formerly Attributed Impact). All mentions of the “Impact Attribution Norm” and just “impact” refer, by default, to the Impact Attribution Norm according to that definition. Verbs such as “liking” or “valuing” impact also by default refer to valuing the Impact Attribution Norm of an action.

2. What we call “charity” is a charity in spirit but may be an Impact DAO or an entity incorporated as for-profit if that is necessary. Examples: Against Malaria Foundation, Wave, Protocol Labs.

3. What we call a “project” is a precisely defined undertaking of, for example, a charity. Examples: one distribution of long-lasting insecticide-treated bednets, an epic in a software development process, a conference.

4. We want impact markets to be interesting for (1) altruistic consequentialists and (2) profit-oriented capitalists. (Most people are a bit of both.) If it ends up being useful for collectors too, all the better, but we want impact markets to work regardless of whether collectors get interested in them.

Retroactive Funding

Retroactive funders benefit in various ways. This follow-up article quantifies the benefits and uses a clearer example. (Here’s another attempt that indicates that impact markets may be 60–11,000 times as efficient as prospective funding, or 500 times in the median case.)

Reduced Uncertainty

Retroactive funders only fund projects that have achieved a certain degree of success.

Projects usually go through two phases, a phase where finding out whether it’ll be successful is a question that management experts and priorities research experts need to collaborate on and a phase where the management experts are not needed anymore.

Let’s say that the project is a proposed conference a year in the future.

In the first phase the project can fail because the team behind it splits up, because the team procrastinates until all venues are booked, or because a pandemic strikes. It can also fail because the attendees come away excited about Homeopaths Without Borders or because one attendee took out vacation days to attend and their replacement accidentally triggered a Dead Hand mechanism.

The second phase starts when the conference is over, the venue is cleaned, and the feedback from the attendees is in. In this second phase all the uncertainty concerning the team, their procrastination, and the pandemic is gone. What remains is just the second type of uncertainty. (Or some part of it since the clumsy replacement has probably returned to their accustomed job away from the Dead Hand mechanism.) The uncertainty at this stage is strictly lower than the uncertainty during the first phase.

Let’s suppose that a retroactive funder has the requirement on a conference that it must’ve happened, that not too many attendees died from Covid, and that the feedback was at least 90% positive. Let’s further suppose that they would have to spend one day per proposed future conference to assess whether the team behind it will reach that bar, and that then they’ll be right in 1 in 5 cases.

Not having to do all of this means saving 5 times 1 day of work per successful conference for the evaluation and keeping the grant money for longer. So even if they buy the impact of the 1 in 5 successful conferences at 5× the price, they still make a profit, altruistically speaking.

The lower the probability that they assign to the projects’ success, the greater the efficiency improvement that impact markets provide.

Reduced Evaluation Time

Let’s suppose that the funder supports a lot of conferences that follow roughly the same pattern, so that can be said to implement the same intervention, and that the funder has invested enough time up front to now be quite confident in the value of the intervention.

If the 5 days for the team evaluation are half the time that they would, counterfactually, spend per project (counting the up-front investment into finding the intervention), then they can buy the successful conferences at (2 × 5)× the price of the counterfactual grant and still save money. (The factor 5 is from the subsection above.)

Longer Market Exposure

Since they invest later, they can keep their money invested in yield-generating assets for longer. In most cases this will have a small influence, say, 1.1× per year over one or two years.

Earning a Double Bottom Line

But there are even more benefits once impact markets are better established. Some retroactive funders may be quite conservative with their investments, so that they don’t think that their retroactively purchased impact certificates are (in impact terms) hundreds of times more valuable than the average impact certificate that gets purchased by other retroactive funders. These retroactive funders may be interested in doubling as a speculator in other parts of the impact market.

Usually they’d have to park their money for a few years, decades, or centuries in classic stocks, bonds, or ETFs, which generate comparatively little impact just to maintain their liquidity and thus option value. With impact markets that becomes unnecessary as they can put money in charities (essentially becoming investors too) and then take it out again when they want to use it for their retroactive funding.

As retroactive funders themselves they may also have the expertise to predict what other retroactive funders will want to buy, so that they can be among the most successful investors on the market. But we’d still think that the social bottom line will do the heavy lifting in that portfolio, so that it’s probably not so attractive for retroactive funders who think that an actual extra dollar for their retroactive funding is worth a lot more than a dollar to one of another retroactive funder’s favorite charities. For example, it might be that they estimate that their monetary profits will shrink by 1 percentage point but that the social bottom line will be similar to what the Against Malaria Foundation would have achieved with ten times that money. A GiveWell-type funder would be excited about that investment opportunity while a donor who typically supports organizations like MIRI would want to avoid it.

Finally, retroactive funders may burn their shares in impact certificates if speculators start avoiding them because they fear that the supply is too great. But if that is not a worry, other future retroactive funders may also just buy shares from earlier retroactive funders in a never-ending chain of retroactive funding. Such a chain will also signal that the retroactive funding is likely to only ever increase, which would boost long-term investment into impact certificates.

The gains from this parking are hard to pin down. The worst-case is close to 1× since it’s optional and funders won’t do it if it’s not useful for them. In the other direction it’s probably capped at around 20× since retroactive funding would not be attractive in worlds in which seed funding is (still) that effective even on the margin.

Spotting Blindspots for Funding Opportunities

If the space is big with hundreds of promising charities with dozens of certificates each at any given time, then funders may be overwhelmed and miss great retro-funding opportunities if they are outside their social circles, in another country and language, or even just very unusual. A market made up of speculators from many different industries and countries may be better at recognizing such niche opportunities than any one funder. They will also be incentivized to make sure that the funder knows about them.

Conversely, retro funders need to be wary of the temptation to ignore unpopular impact certificates. Rewarding the sorts of speculators who are smart enough to notice great funding opportunities that no one else notices is exactly the sort of mechanism that makes impact markets valuable for prioritization.

Impact Certificates

Impact certificates, in our model, are contracts that regulate how to delimit and distribute some unit of change to the supply of a public good. (See below for the Impact Attribution Norm that refines this “change to the supply of a public good.”) Owen Cotton-Barratt and I plan to publish a set of rules that will provide more clarity here. (You can find a sneak peek in our Funding the Commons talk.) They aim to smooth out some initial friction that a marketplace for impact certificates may face. We’ll summarize them briefly in the following.

The guiding metaphor. A winery is typically credited with producing wine (which corresponds to impact). This assumes that employees, letters of the land, states, ancestors, meteorologists, insects, et al. forfeit their claim to the wine one way or another, either by custom or through a contract. Wine can flow freely or evaporate, but it can also be bottled and sold. (The full bottle corresponds to an impact certificate.) The bottling, in turn, can be undone, which is usually followed by the consumption of the wine. That consumption, we assume, is final.

This metaphor should give a rough overview of the spirit of the rules.

The rules propose that impact certificates be:

1. as clear and concrete as possible, and

2. issued only by all those actors collectively who own classic legal rights in that which created the impact.

They also recommend that the market system make them:

1. consumable in the sense that one can permanently reverse the issue, and

2. incremental in the sense that they don’t require changes to existing financial markets.

Generally, we think that markets may well home in on these norms over time by themselves, but we’re currently in a good position to prevent all the frictions that would come with that. We think that in particular the first two rules are crucial. We’re more on the fence about whether rules 3 and 4 are really needed.

The rest is commentary:

The first two rules jointly limit the kinds of outputs that can be bottled up in impact certificates. The limits are not hard, but greater ambiguity will make it harder to find buyers for an impact certificate: Buyers of overly vague impact certificates may make losses if others later sell impact certificates that seem to substantially overlap. No one will quite know who owns the overlapping bits, and bid prices will drop. These buyers will later take good care not to buy overly vague certificates.

Ambiguity can have two sources: confusion over what is being sold and confusion over who (else) might make claims to that which is being sold. We therefore advise issuers of impact certificates to word them carefully and to make contracts in writing with whoever might make a claim on the impact. Note that such contracts needn’t specify a concrete fractional allocation but can also specify an algorithm (such as SourceCred) that is to be used to determine the allocation at the time of the issue.

Eventually, we imagine, aggregator and auditor firms will emerge that are independent of any issuers. Aggregators will compile all information on all impact certificates that have ever been issued on any market so that duplicates can be exposed. Issuers can then get audits to prove to buyers that they are honest and are not trying to double-spend their impact. (For example, someone might first sell their impact from “A fundraiser for Rethink Priorities,” and later sell their impact from “A fundraiser at a vertical farming expo” when really those are about one and the same fundraiser.)

Examples. A good definition may sound broadly like, “Within the 264th year of the era of Aelius Antoninus, I, Hypatia, will ghost-write and publish a paper on behalf of Ada Lovelace that proves (once she is born) that Vingean reflection is possible. I’m selling half of the impact. The other half will belong to Ms. Lovelace. There are no funders outside the market, and there are no conflicts of interest to disclose. In fact, our interests will be similar.”

(Or with reduced specificity: “Within the year N, I, Person A, will publish a paper that proves X, coauthored with Person B. I’m selling half of the impact. The other half belongs to Person B. There are no funders outside the market, and there are no conflicts of interest to disclose.”)

A paper has a clear owner, there are no funders who don’t participate in the market (including Hypatia herself), it is clear what she’ll do, how long it will take, and that afterwards the paper will be public, and it is clear that the uncertainty over the deeper merits of the work will take many centuries to be resolved. (Ada Lovelace lived about 1,400 years after Hypatia.) All this certainty and uncertainty can be priced in by the market.

A middling definition may sound like, “I, Eve, will distribute 1,000 copies of the attached Vegan Outreach leaflet at Barbican Station in London between April 1 and August 1, 2022. Vegan Outreach has waived all claims on the impact. I will sell 90% of the impact and retain 10% to appease anyone who has switched to lower-suffering behaviors in response to the leaflet but is not content to yield all of their impact to me.”

This definition is highly concrete but there is no clear concept of ownership or rights or responsibility to ground it in. As a result, some of Eve’s newly made vegans who have started to go leafleting themselves may want to sell their impact from their own leafleting. The valuation of Eve’s certificate would have to be enormous to appease them all with just 10% of it. Speculators should price in that there’s a lot of fuzziness here about how much of the impact is owned by who.

A really bad definition is something like, “I, Hancock, will use my superpowers to do something for animal rights.” It is vague in almost all the ways it can be vague. There is a broad idea of a strategy in there, but speculators will have no idea whether they should price it like a corporate campaign or like leafleting. Worse, the definition also leaves the door open to very harmful activities, so that, regardless of the actual outcomes, the Impact Attribution Norm is likely negative. A less extreme version of this is a hypothetical impact certificate for a whole organization that is still active or the whole life of a live person. Any prediction as to how these may change their strategies over the coming decades will have high variance so that the Impact Attribution Norm will almost inevitably be low.

The third rule, the consumption mechanism, allows certificate owners to influence prices by verifiably signaling that they will never sell the certificate. Possible tax-exemptions could be tied to consumption, which may be a minor concern for most investors, but some may invest through a legal entity that can only make tax-exempt grants. The consumption mechanism has also been called “dedication” and “burn.” We’re not convinced that this mechanism is necessary at first, but time will tell. One indicator that it is necessary will be if investors don’t expect impact certificates to be sufficiently deflationary or are worried that retro funders with large holdings might suddenly sell out of their positions.

Tax exemption. Note that some organizations in the crypto space seem to function like (our phrasing) “optional donor advised funds” in that they have tax exemptions in various countries and can write donation receipts but will only do so if the donor chooses to donate the money, which they don’t have to do. A retro funder could interact with the market through such an entity. If they choose to resell their impact certificates, they will not receive a donation receipt, but when they choose to burn their certificate, they get it. The donation receipt will of course then be from that intermediary rather than from the charity that actually received the money.

The fourth rule aims to allay worries that impact markets, if successful, will cause revolutionary changes to existing allocation mechanisms. Avoiding such changes avoids opposition, which should make it much easier to institute impact markets. Conversely, even fervent supporters of impact markets may find proposals unrealistic that require changes to existing, established markets.

Impact Market Participants

The key idea behind impact markets is that we want to reward supporters (such as founders, investors, and advisors to charities) who proved a particular aptitude for predicting what interventions will later be seen as impactful by sophisticated altruists.

There are three types of fundamentals on an impact market:

1. Projects: These are verifiable, clearly ownable, and clearly delimited sets of actions that are detailed in an impact certificate. They benefit from:

   1. Seed funding,

   2. More options for aligning incentives.

2. Supporters: These include founders, advisors, seed funders, et al. of a charity. Everyone who negotiates a share in the impact certificate. We also use the terms investor and speculator for the seed funders. They benefit from: 3. The seed funding of the project (founders and employees), 4. Being able to make an exit (founders and other shareholders), 5. Passive income (investors).

3. Retroactive funders: The backbone of the market. The funders who buy impact certificates once the projects have matured to a point where the retroactive funders are confident enough in their evaluation of the projects’ impact. They reward all supporters of a successful project generously, because they are thankful for not having to bear the risks of all the projects that fail even by their own lights. They benefit from: 6. Not carrying the risk for failing projects (5–10x monetary savings), 7. Not having to do charity evaluation (team, market, etc.) in addition to intervention evaluation (2x time savings), 8. Investing later in time (1.5x monetary savings), 9. Parking money (in the role of a speculator) where it generates a double bottom line (1–20x savings?), 10. Being able to draw on the market to notice blindspots in the in-house prioritization.

Let’s suppose the Against Malaria Foundation (AMF) wants to do a distribution of long-lasting insecticide-treated bednets with its old distribution partner Concern Universal (CU). Let’s further suppose that it wants to use impact certificates for this and not its usual funding channels.

(This is an example only. We didn’t talk to AMF or CU about any of this and don’t actually want any charities that don’t have expertise in innovative financial products to become involved so early in the process! Impact markets are probably also not suitable for such safe investments as AMF’s net distributions.)

1. AMF ascertains that some retroactive funders are still interested in more net distributions.

2. The AMF staff decide how they want to split the Impact Attribution Norm from the distribution among themselves, the legal entity AMF, all their partners, contributors, and advisors, and their funders. These decisions can be adjusted in later negotiations. (we’ll treat AMF, the legal entity, and all its staff as just “AMF” in the following to save space.)

3. AMF decides on the details of the auction process, that is, what fraction of the profits from each sale should go to AMF and what the minimum percentage raise is between bids.

4. AMF writes an impact certificate.

5. AMF contacts CU, negotiates their split with them, and sends them their fraction of the impact certificate. (we’ll assume here that the distribution partner has its own source of seed funding just to make its case interestingly different from AMF’s.)

6. AMF contacts a venture capital firm. The VC agrees to fund the distribution. They negotiate the split, and AMF sends them their fraction of the impact certificate.

7. A year later the distribution gets completed successfully.

8. A retro funder notices the successful distribution, is ecstatic, and buys fractions of the certificate from all holders. AMF has no work with this: Any holder just has to give their certificate (or a fraction thereof) to the buyer.

What we seek to demonstrate here is that impact certificates (the way we construe them) are not magic. They don’t need to be imbued with any deep meaning beyond that of any other contract.

Auction Platform

We want to create an auction platform to streamline the whole process. (You can keep up-to-date on the progress here.)

Our ideas for MVPs differ in two main ways: (1) The MVP can either emulate the classic market mechanisms where multiple shareholders co-own a project, or it can use the Harberger tax auction where the original issuer is paid from the profit of each sale, sales are forced, and fractionalizing ownership among multiple shareholders is optional; and (2) it can be aimed at a crypto audience and run on a blockchain, or it can be aimed at a non-crypto audience, not use a blockchain, and avoid monetary transfers. (Note that we use auction and market synonymously here.)

Either way, the critical mechanism is that there’s someone (a retro funder) who rewards earlier supporters (founders, funders, advisors, et al.) if they made what has turned out to be good calls.

The classic, non-Harberger auction (or market) has the advantages that:

1. Alignment. You can align incentives with cofounders, employees, advisors, partner organizations, et al. by giving them shares in the impact.

2. Seed funding. You can get investments by selling parts of the impact to early investors.

3. Exit. You can participate in the success of your project by keeping some fraction of the impact until the project has come to fruition (by your assessment) and only then exit. As a founder you will be unusually convinced of it (because if more people were, it would typically already exist), and by extension, you will be more optimistic about it than your investors.Hence you’ll want to retain as much of your impact as you can afford until the day has come when you have the proof that will convince the investors too.

4. Scaling down. You can start big projects by getting several investors who each buy smaller shares of the impact. Conversely, even smaller investors can get exposure to big projects by buying small shares in them.

The Harberger auction shares two of them even without shared ownership:

1. Alignment. You can align incentives with cofounders, employees, advisors, partner organizations, et al. by making separate contracts with them that they’ll receive part of the profit share. That would be similar to an ordinary participation.

2. Seed funding. You can get seed funding by selling all of your impact to early investors. You can set the minimum bid so that the funding is enough to bootstrap the project, but you can’t have any further fundraising rounds.

3. Simplicity. You don’t need to countertrade your investors once you have the seed funding. You’ll simply profit from each of their sales automatically.

With shared ownership, the Harberger auction shares all the benefits of the classic system, but in practice there will be trade-offs:

1. A larger issuer share (or tax) will discourage speculation because it’s subtracted from the profits of the speculator. The market will be less liquid as a result. But issuers can set an arbitrarily small share to manage that risk. Crypto exchanges seem to work well with taker fees around and slightly below 0.05%.

2. But a small share also means small profits. Prices can’t decrease in the Harberger auction, so the profits will be proportional to the price. (If they could decrease, they’d be proportional to the volume, and charities had an incentive to create volatility.) In the classic system, you and your collaborators may collectively manage to retain 10–30% of your impact until you exit, which you have to time such that you don’t sell too early. In the Harberger system you don’t have to worry about the timing, but you’ll all collectively only “retain” (through the tax) 0.05% unless you want to sacrifice market liquidity. Even if you choose to set the tax at 1%, that’s still 10x the cost, plus the reduced liquidity, in exchange for simplicity.

All in all, a Harberger auction with fractional ownership seems like it maximizes option value because issuers can choose to turn it into a classic auction by setting their tax to 0.

The current state is as follows:

Blockchain (web3). Our first testing ground was a smart contract on an Ethereum test net that implements an auction as described above. It does not currently support fractional impact certificates, but we think that that could be added in the form of a separate third-party smart contract. In general, a blockchain-based or web3 solution would have a number of advantages:

1. It makes it easy to interface with the existing web3 ecosystem around public goods funding.

2. The existing ecosystem is itself a great testing ground for innovative solutions because everyone is unusually knowledgeable of innovative financial tools, so we’re at less of a risk of making mistakes that would cause other people to lose money.

3. It allows for great scale as transactions are fast and frictionless, and, depending on the blockchain, also cheap.

4. It makes it easier to market the solution because users don’t need to trust us if they can read the code and there are audits.

(You can watch a demo in our talk at Funding the Commons II.)

Spreadsheet (web2). Another MVP that we have in mind is a spreadsheet (via Google Sheets) that tracks transactions between issuers, speculators, and retroactive funders (any pairwise permutations). One sheet simply records the transactions; another summarizes the transactions such that they show the latest configuration of ownership; and a third sheet displays some statistics, such as the price history, market capitalization, volume, etc. We haven’t created this spreadsheet yet, but we expect that it’ll be hard to scale for various reasons related to error-reporting and because spreadsheets get slow when there’s a lot of data in them.

Crucially, the spreadsheet would not execute the monetary transactions themselves, it would just record them. A transaction is then said to be pending when one side has entered it but the other hasn’t confirmed it. It is only considered for the statistics once both sides have confirmed it. Google Sheets allows owners to protect ranges and individual cells such that only one person can edit them, so that aspect should be fairly failsafe.

A variation on the same theme is a spreadsheet that only tracks what projects accept seed funding and what projects have been completed. All seed-funding transactions could then happen over a platform like Kickstarter or Indigogo. Once the project is completed, a retro funder can consider it and transfer their money either directly to all contributors or to the project founder so that they can forward it to all contributors.

We could test the spreadsheet solution by allowing people to use it to trade on retro funding for something very safe and easily verifiable such as EA Forum articles. One could also call this a “proportional prize pool for prescient philanthropists” because they are basically prizes for people who’ve made good calls – as founders, funders, supporters, or similar – except that not only the top 3 or top 10 get prizes but almost everyone who clears some minimal bar of the retro funder and has invested enough that the prize is worth the transfer overhead.

A web2 solution like that would have a few advantages too:

1. It is quick to iterate on as it doesn’t require audits. It won’t handle financial transactions, and even if it did from the users’ perspective, they would really be handled by a payment gateway like Datatrans.

2. It would not handle financial transactions, so many legal risks from those shift away from us to the users of the system, who we’ll need to warn to do their research.

3. It’ll be more frictionless for the sorts of users who haven’t used web3 solutions before.

Personally, we’re also a bit annoyed by people talking so much about valuing impact certificates in and of themselves like that’s important, so we like solutions that don’t obviously contain any one component that is an impact certificate. (We see an impact certificate like any other contract that we value only for the monetary or altruistic bottom line that it nets “us” – scare quotes because only the monetary bottom line is agent-relative.)

We are fairly convinced that the blockchain-based solution is going to be the culmination of our efforts one day, but we’re ambivalent over which MVP will allow us to test the market more quickly and productively. (Since we first published this document, we’ve focused on web2 and have graduated from the spreadsheet to a proper web app.)

Impact Stock and Derivatives

At later stages, we imagine that investors will find it overly complicated to use individual auctions to bid on countless projects. Some charities may also prefer to use other kinds of auctions at least for some of their projects. Plus, the trust that a project will be impactful is probably going to be tied to the team behind it, the charity, so that people, especially those who are lay people when it comes to the interventions of the charity, will prefer to invest into charities instead of individual projects.

We imagine that this can be achieved with derivatives (such as perpetual futures) that track the market capitalization of all impact certificates that a charity has issued. Alternatively, and if that is legally permissible for the charity, it could issue its own stock, which it could also use to hedge against volatility in the flow of donations that it gets through other channels.

Benefits

(We briefly covered this section in our talk at Funding the Commons II.)

We think that impact markets are most suitable for hits-based funding for individuals or young charity startups without impressive track records. This article makes this argument in greater detail. In short, reference classes of projects where success probabilities over 80–90% are common make it hard for investors to think that they have enough private information about a particular project that they will accept the low success-conditioned rewards that retro funders will pay in such cases.

The following table gives an overview – i.e. rough tendencies based on broad simplifications – of who will find which benefit to be interesting:

Benefits for Funders

Greater Maximum Scale

EA funders seem to be reaching the limits of their scale. There is a vast difference between the time investment that GiveWell and ACE have found appropriate per dollar moved to a funding opportunity compared to the time investment that hits-based funders such as the Future Fund currently find appropriate. They are probably not wrong about that given their constraints. But impact markets can change these constraints.

Specifically, if a funder currently expects 10% of their grantees to succeed (by their lights), then impact markets can allow them to scale up by 10x just thanks to the grantmaking capacity that retroactive funding conserves. They may even save money in the process. This article quantifies the benefit some more.

More Funding Opportunities

Funders are usually based in a specific location and have particular social circles. It is very costly for them to build up trust with someone outside those circles if they even learn about the person or group in the first place. That limits their access to funding opportunities to a tiny fraction of the landscape.

Much less trust is needed if founders and investors pitch already successful projects to the funders.

And the investors are already almost everywhere around the world in a variety of social circles, are expert networkers, and already have expertise in startup picking, which they can apply to charity startups.

Greater Hiring Pools

Currently grantmakers who work for funders have to be highly trusted by the people with the money, have to be experts in priorities research (or at least in understanding and applying the latest priorities research), have to have a comprehensive overview of the spaces in which they’ll do their grantmaking, and have to be great at startup picking, including all the social skill that it takes to quickly form an accurate opinion on a founder team. That’s a lot of constraints, and some of them feel intuitively anticorrelated so that it’s even harder to find people that combine all of them on a high level.

With impact markets, they only need to be trusted experts in priorities research. They don’t need to have the comprehensive overview of the spaces because countless investors are their eyes and ears. And they don’t need to be great judges of character and project-specific skill because the projects that get pitched to them already succeeded to some appreciable extent.

That should expand the hiring pool for funders many times over.

More Priorities Research

Normal investors will have incentives to publish priorities research that they do after they have invested in order to increase the price at which the certificate or certificates trade. Or they will at least publish the good bits that swayed them to invest.

Short-sellers will do the opposite and publish their exposés.

These can contain important considerations that we’ve previously missed. Failing that, they may help communicate important considerations from priorities research to a wider audience.

Parking Money

Large funders currently park their money in for-profit businesses until the growth rate from the growing wisdom of the funder (plus the low average growth rate of the businesses) drops below the growth rate of the public goods. That can take a long time and is therefore very wasteful, but it’s a necessary evil because investments into public goods are currently illiquid, so if you get them wrong, you can’t withdraw the money again. Impact markets would change that, and large funders could park their money in ETFs that comprise many big, well-established and somewhat impactful charities until they find better investment vehicles.

Benefits for Charities

Cheaper Liquidity

Charity startups and individual altruists will receive seed funding from the investors who are already in their networks and who already trust them. These may be business angels, VC, or simply room mates who have the sort of income that they can advance the rent for a year.

Aligning Incentives

Charities typically like to hire very closely value-aligned people. That means that they can pay lower salaries because everyone cares about the mission. But that doesn’t seem like the optimal state. There are people who care about the mission but also have a family to feed or a debt to repay. There are also extremely capable people who care a bit less about the mission. The charity will lose out on them.

If we now assume that the employee that the charity wants to hire has a bit of runway and can run about as much risk as an early startup employee. If they think that a retroactive funder will like the charity enough to buy its impact certificates, that employee may agree to a deal where the regular salary plus the expected value of 1% of all impact certificates adds up to more than a regular salary. If the employee cared a bit about the charity’s mission before, now they care a lot.

Incentivizing Excellence

Many funders today aim to fill or to partially fill the funding gaps of any project that clears some bar in terms of expected cost-effectiveness. So highly capable startup founders have a tradeoff to make whether they want to start a for-profit business and donate billions or whether they want to start a charity and get at most exactly as much as they need. They wouldn’t want more unless they have enough spare time to become better grantmakers than those who make grants to them. But if charities can fundraise from for-profit venture capitalists, the gulf between charities and businesses shrinks. This means entrepreneurs have a less difficult tradeoff to consider when choosing between launching a charity or a for-profit.

The quick, quantitative feedback could also by itself be motivational for founders and employees.

Risks

(We covered this section in our talk at Funding the Commons II.)

Have a look at this summary table for a quick overview.

Impact and Profit Distribution Mismatch

Investors should have strong reasons to expect that the prices of certificates will, in the limit, be proportional to the value that a Pareto-optimal compromise axiology would assign to them – that is the moral standard that is reached only when no gains from moral trade are left on the table.

But we think that is unlikely to happen by default. There is a mismatch between the probability distribution of investor profits and that of impact. Impact can go vastly negative while investor profits are capped at only losing the investment. We therefore risk that our market exacerbates negative externalities.

Standard distribution mismatch. Standard investment vehicles work the way that if you invest into a project and it fails, you lose 1 x your investment; but if you invest into a project and it’s a great success, you may make back 1,000 x your investment. So investors want to invest into many (say, 100) moonshot projects hoping that one will succeed.

When it comes to for-profits, governments are to some extent trying to limit or tax externalities, and one could also argue that if one company didn’t cause them, then another would’ve done so only briefly later. That’s cold comfort to most people, but it’s the status quo, so we would like to at least not make it worse.

Charities are more (even more) of a minefield because there is less competition, so it’s harder to argue that anything anyone does would’ve been done anyway. But at least they don’t have as much capital at their disposal. They have other motives than profit, so the externalities are not quite the same ones, but they too increase incarceration rates (Scared Straight), increase poverty (preventing contraception), reduce access to safe water (some Playpumps), maybe even exacerbate s-risks from multipolar AGI takeoffs (some AI labs), etc. These externalities will only get worse if we make them more profitable for venture capitalists to invest in.

We’re most worried about charities that have extreme upsides and extreme downsides (say, intergalactic utopia vs. suffering catastrophe). Those are the ones that will be very interesting for profit-oriented investors because of their upsides and because they don’t pay for the at least equally extreme downsides.

Most profit-oriented investors also care about countless things besides profit, but we think it makes sense to think about these risks with a security mindset and assume that there are purely profit-oriented investors out there who can move a lot of capital. Besides, most investors will not be aware of the interests of future generations millions of years from now, of invertebrates, or of beings close to our acausal trade partners.

A simplified analogy is pictured above. This first aid quiz counts correct answers only – just like the market – so if you select all answers, you get the full points even though most people would not survive such treatment for nosebleed.

This risk applies to prize contests in general, especially relatively long-running ones. We find it reassuring that prize contests have not yet ended the world, but we would like to see more historical analysis of the actual hidden (or maybe obvious) harms that have come off well-intentioned prize contests in the past.

Anthropic distribution mismatch. Moreover, if the downsides are strictly about extinction, then the investors will lose their bets in worlds in which they wouldn’t have been able to spend the money anyway.

They might regret this if they thought that their bet itself had increased the probability of losing the bet, but they’ll probably assign a low probability to that because they may, for example, reason that they merely defected in an already hopeless collective prisoners’ dilemma. Some people reason along the same lines when they argue that divestment is ineffective (something that Paul Christiano critiques): If the market is sufficiently efficient, any divestment will result in an inefficiency that will quickly be compensated by equally sized investments of others. These factors cause us to worry that investors will be likely to defect against values that are concerned with x-risks (including s-risks).

Both of these can be framed as problems of lacking moral trade because the investors defect against some futures for the benefit of others when they should’ve invested in ways that respect the interests of all futures.

Below we present a definition of the “Impact Attribution Norm” that combines earnest intention with outcome and thereby addresses the problems related to moral trade and extinction.

Moral Cooperation Failure

It is uncommon for two people to each have only one interest, and for these interests to be exact opposites. So in most cases it should be possible to find a compromise that improves the aggregate good that is achieved and, ideally, one that is as good or better than the cooperation failure for both, a Pareto-improvement.

Here are some pilfered images by Brian Tomasik from the Center on Long-Term Risk:

The caption is: “Pareto improvements for competing value systems. The two axiologies are opposed on the x-axis dimension but agree on the y-axis dimension. Axiology #2 cares more about the y-axis dimension and so is willing to accept some loss on the x-axis dimension to compensate Axiology #1.”

Caption: “Figure 2: Imputations for compromise between deep ecologists and animal welfarists, with pi = 0.5 for both sides. By “Pareto frontier” in this context, I mean the set of possible Pareto-optimal Pareto improvements relative to the (50, 50) disagreement point.”

The line labeled as “Fight” is the conflict that we would see on our markets if both parties were to fundraise for their conflicting goals and consequently burn most of their resources in a zero-sum conflict. But if they talk to each other and compromise, then they can realize outcomes that are better in aggregate (almost inevitably), and in particular outcomes on the Pareto frontier, which are directly desirable (or neutral) for both parties!

Had impact markets existed a few decades ago, there might’ve been projects that pushed for coal energy over nuclear energy. These might then have gotten a lot of funding from investors and retroactive funders who didn’t think that climate change was a significant worry. Our and future generations would then have been defected against. (Our interests would’ve been ignored because we didn’t participate in the market through retro funders.) Instead it would’ve been possible for these people to instead invest into R & D for the next generation of safer nuclear reactors, which would’ve allayed their safety concerns without exacerbating climate change.

Similarly, an animal conservation organization might want to use impact markets to fundraise for the protection of certain predator animals. They might fundraise large sums from people who are interested in the protection of that species, but they would thereby defect against the interests of the prey animals who are much more numerous and probably similarly capable of suffering. Instead they could’ve fundraised for the protection of a herbivore species that mostly eats fruit and whose members are unlikely to experience great amounts of suffering themselves throughout their lives.

Worse, the conservation charity that increases the number of predators at the expense of the prey animals may sell impact at a positive price and another charity that protects the prey animals against the predators by (say) making them infertile could also sell their impact at a positive price. Both impact certificates may have a large positive valuation when really their impact more or less cancels out! So this egregious waste of resources can even happen when both parties participate in the market in some way.

Another egregious example is terrorism. Terrorists may want to use impact markets to fundraise for their attacks. Maybe they are religiously motivated and are attacking people as a form of proselytizing with extreme prejudice. Meanwhile other groups may use the same markets to fundraise for terrorism prevention. That is again wasteful since they could instead pool their resources in open-access adversarial collaborations on questions of religion.

We think this lack of any incentive for individuals to engage in moral trade is a major risk. The distribution mismatch is, strictly speaking, a sub-risk, but that’s a bit unintuitive, and we think that both are so important that they deserve their own sections be it only for emphasis.

Finally note the difference between revealed and idealized preferences. The first step when attempting a compromise should probably not be to trade right away but, if possible, to discuss the object-level disagreement. It may be that at least one of you is simply wrong, will realize that, and will no longer be interested in the competing intervention at all. (But others might be.)

Manipulation of the Default

An attacker might first build a reputation of regularly doing exactly the sorts of things that large funders hate, such as running flash loan attacks against crypto markets or denial of service attacks against blockchains, and then skip some of these regular attacks in order to sell their impact from not attacking. This exploit is based on the ability of the attacker to change the default state of the world from one in which they don’t attack to one in which they do attack, and regularly.

This failure mode would cause impact markets to exacerbate exactly the sorts of problems we want to solve. The “Impact Attribution Norm” definition below addresses it.

One real example that sort of fits is the following anecdote that I’ve heard from a reliable source but haven’t fact-checked: When the cap & trade carbon trading system was first introduced in the EU, many polluting companies either first polluted even more or exaggerated their level of pollution so that their default level of pollution would be set higher. With the exaggerated default it was cheap for them to (seemingly) reduce their carbon emissions again, and they didn’t have to buy carbon certificates.

Similarly, one sometimes hears of stores that momentarily increase the price for a product (or just lie about the old price) and then advertise the same or a higher price as a discounted sale price.

We’re not as worried about this as we are about moral trade and, in particular, the distribution mismatch, because attackers could, in many cases, already attack funders this way. The funders just have less of a target painted onto them, figuratively speaking. Yet funders should make it clear that they will not be fooled this way, because even a failed attempt at such a ruse can do harm, as in the case of the increased pollution.

Drawbacks of Verifiability

Impact markets will probably tend toward high verifiability requirements all by themselves, but only after many buyers have been burned by investments into impact certificates that later turned out to overlap with other impact certificates and whose price dropped as a result. Prescribing a high degree of clarity from the start will hopefully avoid this friction and keep more buyers interested in impact markets. Keeping impact certificates more clearly delineated also helps to separate them from impact stock.

But there are two possible drawbacks to this. The first is related to Goodharting. Charities that have a very clear output – e.g., papers or bed nets – will have an easy time fundraising on impact markets. But charities with more fuzzy outputs – e.g., community cohesion or plan changes – may be relatively disadvantaged by the format. That could lead to a relative underinvestment of money and effort into these more fuzzy interventions. Worse, most interventions can probably easily be measured by a few proxy measures that, when optimized for, lead to terrible outcomes. That’s a case of a failure of moral trade. The Impact Attribution Norm discourages it, but overconfident issuers may still seek to sneak in the bad proxy measures that they perform excellently on.

The other drawback is that explicit contracts with collaborators may not be enough. We’re currently collaborating with many others in a range of ways. Some examples in rough order of how explicit they appear to us: collaborations between coauthors, collaborations with reviewers, collaborations with casual conversation partners, collaborations with providers of infrastructure, and collaborations with those in the past who have made it possible that we are alive today.

If the sale of an impact certificate for, say, a paper rewards only the authors, there’s a risk that other collaborators may decide that they will be rewarded better if they don’t engage with other authors anymore to fully focus on their own papers and not give away any ideas before they have published them.

This could also happen between organizations. Each may think that it has a uniquely important mission and a responsibility toward its employees, and that it may have to shut down without funding from the sale of impact certificates. Two seemingly altruistic reasons will push these organizations to withhold all sorts of resources from the other.

The solution is to talk to all collaboration partners that form a coalition and to negotiate an allocation of the returns from impact certificates that has the core property, that is that no collaborator thinks that it is in their interest to split off from the coalition.

That’s easily done among explicit collaborators such as coauthors, cofounders, or employees. Hence why the first rule requires explicit contracts between these. But in many other cases it will require acausal trade because it would be infeasible or impossible to (causally) negotiate with the collaboration partners – they may be anonymous, dead, distant, too many, or too busy. So the first rule falls short of solving this problem.

On the other hand, impact markets are likely to reduce financial scarcity, and people are less likely to behave uncooperatively when they have plentiful resources.

Noise

The market itself will also serve as a source of information for investors to understand what impact likely maximizes the compromise morality. Investors who correctly identified highly uncontroversially valuable impact early on may sell to take profit, manage their risk, or pay their rent – reasons other than thinking that the impact is overvalued – but this sell may be misinterpreted by others as a signal that the impact was more controversial than they had thought. All of this introduces a lot of noise that distracts from the price discovery. This is something that happens on financial markets all the time, and public companies have to be sophisticated enough in their budgeting not to be ruined by fluctuations that have nothing to do with their fundamentals.

Sophisticated funders who are highly confident in their judgment, or more so than other market participants, can stabilize the prices by buying, holding, and dedicating impact certificates, but they can do so only to the extent that their budgets allow.

This is more of an inefficiency than a risk of net harm, so we’re less worried about it than about the above.

Solutions

(We briefly covered this section in our talk at Funding the Commons II.)

Curation

There are three forms of curation:

1. Curation of each impact certificate that is submitted to the marketplace.

2. Curation of the issuers who are allowed to submit to the marketplace.

3. Curation of the investors and retro funders who are permitted to use the marketplace.

These can be performed by a team of expert curators, by some sort of oracle, or by an auditor network (see below). Exposé certificates and shorting (see below) can help the curation team not miss any bad certificates.

The first method can be aided by limiting the market to a particular topic. If only AI safety research is allowed to be sold on the market at all, then it already excludes a lot of dangerous categories of things, such as terrorist attacks, and the curation team does not need to contain experts from more than one field.

The second and third can be based on the implicit web of trust that emerges when people are only allow to apply for access to the platform if they were recommended by existing users of the platform. The application process could include courses and quizzes if the platform is valuable enough for the users.

Impact Attribution Norm (formerly “Attributed Impact”)

We think it will be key to find a definition of impact that has four properties:

1. It leads investors to value impact in proportion to the moral gains from trade that it has generated,

2. It leads investors to value impact in accordance with some form of idealized rather than revealed preferences of all affected groups,

3. It forms an attractor state so that over time more investors tend to adopt the definition (be it unconsciously) rather than fewer, and

4. It tracks but makes more precise the existing shared intuition of what “someone’s impact” is.

The following definition is an attempt at that. It relies on a lot of culturally shared understanding, which is hopefully also shared to a sufficient degree between investors. Enforcing honesty or detecting lying of market participants is outside the scope of the definition.

It boils down to something like, “Your impact must be widely regarded as morally good, positive-sum, and non-risky before and after you undertake your action for it to be valued at a positive price.”

Definition

Impact Attribution Norm (v0.3):

1. We define impact as the gains from moral trade achieved by a set of actions.

   1. We take into account the interests of all moral patients, regardless of time, location, species, substrate, gender, phenotype, orientation, market participation, etc.

   2. Specifically, we value actions positively only if they were robustly positive in anticipation.

2. We attribute impact to all those actors whose actions make them responsible for the impact.

   1. We draw on widely accepted cultural norms guiding responsibility.

3. We assess impact as compared to counterfactual world histories that seem ordinary by broad societal standards.

   1. In particular we do not rely on counterfactuals that are easily influenced or merely claimed by the issuer.

4. We leave it to the actors to make a case for the value of their impact.

5. Where we have to make judgment calls on behalf of others we will assume that they favor noncausal decision theories and the Kalai bargaining solution.

In Plain Terms

As issuer of an impact certificate:

1. You need to make a case that the actions you’re planning to take will generate robustly positive impact. (You should reference the particular version of the definition that you’re using.)

2. You cannot just pick one metric or moral dimension along which you want your impact to be measured. You need to make the argument that your action is good or neutral across all moral views and interests, that harms are offset in ways the participants would accept, or something along those lines. In short, an’ it harm none, do what ye will. (Slight oversimplification.)

3. In particular, you need to argue that at this time (when you haven’t taken the actions yet) it is reasonable for the median market participant to think that your action will generate impact in expectation. This should be positive impact unless you’re happy that your impact certificate will have negative value and no one will buy it.

4. Note that you can’t inflate your impact by claiming that something unusually terrible would’ve happened if you hadn’t performed your action. If such a claim is not plausible to someone like Erin McKean, or anyone else who doesn’t know and knows nothing about you, then it doesn’t count.

5. Note also that you can’t lie about your actions. If you write an impact certificate for one action and then then perform a different action, the impact certificate is for an action that didn’t happen and there is no impact certificate for the action that you did do.

6. Finally, talk to others who you are bargaining with if at all possible. If not, please don’t automatically assume that they make decisions according to CDT, and if you’re unsure about what bargaining solution they’ll favor, go for Kalai.

Commentary

Gains from moral trade. The reference to the “gains from moral trade” codifies that what makes an action valuable are not its effects along some arbitrary moral dimension but all moral dimensions, and in particular the improvement that the action achieves over a hypothetical completely disjoined state of the world where everyone is on their own.

There is the degenerate case of an action that benefits one moral view and is irrelevant for all others. Such actions might as well be valued positively, because why not, and we would welcome an elegant way of fitting them into this framework, but they are probably quite rare or small scale, so that they don’t matter much in the scheme of things. Maybe we can imagine that in such cases there is a being that gives you something that neither of you value or disvalue in return for you looking after your own moral interests.

This exclusive focus on the gains from moral trade has a host of related advantages: Investments into zero-sum games will have no value on a certificate market, except insofar as they generate net positive externalities; projects that defect against some future beings for the benefit of others are penalized; progress on some values that is offset by harm to other values is without value; projects are incentivized to seek out new moral dimensions orthogonal to all known ones; and projects are encouraged to do cross-cutting work that benefits a variety of moral systems.

The subitems clarify some assumptions. In particular, the category of moral patients is supposed to be interpreted maximally widely. You can’t just avoid defecting against someone by declaring them not a moral patient. When it doubt, these rules should be interpreted so to err on the side of caution and inclusion in the set of moral patients.

Responsibility for the impact. Especially the term “responsibility” is one that relies on a culturally shared understanding, ideally one codified in law or a contract. We see no objective way to attribute impact other than by convention. Here the burden of proof is upon the person who makes the claim.

Counterfactual world histories. This clause prevents issuers from manipulating the default as described above.

Changes in the supplies of goods need to be assessed against some counterfactual, and in this case we want to use a counterfactual that everyone can know and that is very hard to influence by any one actor. We might call this perspective the systemic stance, in reference to Dennett’s intentional stance and system science, which is a perspective that abstracts from agents and intentions to recognize systemic or emergent mechanisms. It’s a form of underfitting or dimensionality reduction. As a result, an attacker would now have to change large swaths of society into one that regularly conducts flash loan attacks (per the example above) before their omissions to do so becomes different from the default.

Retro funders need to proactively signal their adherence to the norm to make it clear to any potential attackers that they are not vulnerable to this exploit.

A disclaimer, in case you now think we hate historians: We compare full world histories (unless the impact period is explicitly curtailed in the certificate), so not only the time between the decision to implement the project and the present moment or any other such period. But this is done in the spirit of a belief that the Tickle Defense is probably true. So a person who surprisingly kills someone produces evidence that people sufficiently like them have killed people before and thus narrows the set of subjectively plausible pasts to a set of worse pasts, but a historian who researches past killings has not the same evidential effect because the killings they research screen off any evidential effect the historian has. The spirit of our definition is that if the Tickle Defense fails, we want to correct our definition and not put blame on historians!

The sets of world histories all contain several pasts, presents, and futures, because the issuer had uncertainty over past, present, and future at the time of the decision. (In view of the many-worlds interpretation, we intend “world histories” to include all Everett branches weighed by their measures.)

Make a case. Every issuer has to make the case that their action conforms to the norms. This should not be left only to the retro funders to figure out. The retro funders’ job is rather to find weaknesses in the case. If every impact certificate thus recapitulates the definition, it will gradually become reified culturally as the consensus definition of the Impact Attribution Norm. We call this a “driver of adoption.” The Impact Attribution Norm contains several for redundancy.

The norm follows (in spirit) the JTB definition of knowledge as a justified, true belief. In the impact certificate or some ancillary document, the issuers justify their claim to the Impact Attribution Norm by laying out how they think the reader ought to arrive at the conclusion that they were, at the time of the decision, justified to believe that what they did would generate positive impact. Time and the market then arbitrate the question of truth. Only if ex ante and ex post impact are positive is the aggregate impact (the minimum) positive.

This avoids the pathological case of x-/s-risk gambles where someone runs a large number of very risky projects – projects that can turn out vastly net positive or net negative – and then makes money by selling only the impact from those that happened to turn out positive even if they are in the minority. (Or likewise for funders.) Those that happened to turn out positive would still not be worth anything because their 0 or negative ex ante impact will be less than the positive ex post impact, so that the positive impact will have no influence on the eventual value. This mirrors the intuition between expectational consequentialism where, for example, a doctor who treats a mild headache with a medicine that is 99% likely to kill the patient acts immorally even if the patient actually happens to survive and be cured of the mild headache.

Note that this applies in the same way to the anthropic formulation of the x-/s-risk gamble where issuers or investors believe that (1) they hardly affect the odds of extinction, for example because of market efficiency, and (2) they are neutral about x-/s-risk futures because they can’t spend money anyway if they’re dead or otherwise incapacitated but they do care about the okay outcomes where they can spend money.

Aside. The similarity to the JTB definition suggests that the Gettier problem may be a problem. But we haven’t managed to construct an actual pathological or exploitable version of it. (You can skip to the next subsection if you’re in a hurry.)

An example: Aanya Altruist reviews some 450+ studies and concludes that there is strong evidence that the serotonin transporter gene 5-HTTLPR is associated with depression. She develops and promotes a school-based screening procedure that provides people with the particular versions of 5-HTTLPR with resources on how to access therapy. Biology teachers love the program for how helpful and scientific it is. Aanya makes a case for her impact and sells it. Then it turns out that (1) 5-HTTLPR has nothing to do with depression, and (2) many of the children who had received the resources had become depressed at the population base rate but started therapy at a much higher rate.

Aanya Altruist’s defense of her impact is state of the art by the standards of her field at the time, and she is honest about believing her conclusions. Yet they turn out to be false. So her impact is valuable by dint of her earnest, justified intentions and by dint of her quite unrelated impact. This is probably not going to be a common case, because we expect there to be few impactful interventions so that few people will find them by accident, but it doesn’t seem harmful either.

But the system is hypothetically exploitable in another way: Someone may find a new, previously unknown CO2 sink that they predict to start absorbing CO2 within a few years. They build a complex contraption that no one understands but that is just plausible enough that they can fool investors with an amphigorical justification. Then they start it right when the natural CO2 sink goes into action. Finally they claim credit for the impact.

Another variation of the theme: Someone builds in secret a lot of machines with fairly general seeming purpose but without actual function. Then they wait for a near catastrophes. Maybe there’s a hurricane that changed course just before it hit the shore. They reveal one of the machines that happened to be nearby and claim to have used the hurricane as a beta test for the hurricane-averting machine and sell the impact from it.

Maybe people will come up with more realistic variations on this theme. When that happens, the problem can probably be addressed through preregistration of impact certificates and buyers penalizing any lack of preregistration. (Preregistration may be as simple as issuing an impact certificate before the impact has happened, which should be the norm anyway.)

Noncausal decision theory and Kalai bargaining solution. There is a risk that people will default to assuming that other market participants are causal decision theorists. This would preclude gain from acausal moral trade. We’re unsure what decision theory is best to recommend here, but it is probably not causal decision theory. The bargaining solution is also something we’re unsure about. We don’t know which bargaining solution to pick here, but not specifying any seems worse than specifying a random one. Resource monotonicity seems to us like an important criterion for the general acceptability of a bargaining solution in a rapidly growing market (the Nash bargaining solution doesn’t ensure it), maybe more so than scale invariance (the weak point of the Kalai one). Independence of irrelevant alternatives seems superficially important to us (the Kalai-Smorodinsky solution doesn’t ensure it), but we haven’t thought about this in detail and would greatly appreciate feedback on this decision.

Summary. This definition of the Impact Attribution Norm addresses (hopefully successfully) the following problems:

1. Impact is not naturally uniquely owned. It addresses this by relying on cultural norms and laws around responsibility, such as legal ownership, authorship, etc.

2. Issuers may generate one good at the expense of another. This is addressed by tying the evaluation to an aggregate of all public goods.

3. Markets incentivize risky gambles. It addresses this by defining the Impact Attribution Norm as the minimum of ex ante and ex post expected impact.

4. Counterfactuals are impossible and arbitrary. This is addressed by stipulating a particular way of thinking about the counterfactuals that relies on the subjective expectation at two points in time of an observer that knows no specifics about the case.

5. Buyers are vulnerable to extortion. This is addressed by stipulating a counterfactual (see above) that is very hard for an individual to influence.

6. This definition may be ignored. It addresses this by asking issuers to defend their impact in terms of the definition and by protecting buyers from extortion if they proactively signal adherence to the definition.

Timeline

This is a sample timeline of how an issuer may produce impact. Period 1 is when the impact is issued and generated, in whatever order. Period 1 separates Period 0, where the real and the counterfactual worlds haven’t diverged, from Period 2, where they have diverged and continue to diverge.

On both sides of Period 1 there is one systemic observer, who is imagined/simulated by the investors. In Period 2 there is also the actual investor. (We’ll use simulate over imagine to convey that the goal of the mental simulation is accurate prediction and not, say, entertainment.)

So this is what happens in temporal order:

1. The prospective issuers consider that a systemic observer would typically expect people like them to do stuff. So they surmise that future buyers would also agree that a systemic observer would expect that. But if they did a thing instead, a systemic observer would be surprised.

2. The prospective issuers iterate through a few options for things to do, and eventually settle on one. That thing is a great thing because it generates morally relevant preference satisfaction for some beings without undermining the preference satisfaction of others. They implement it and document it in an impact certificate alongside a lengthy defense.

3. A month has passed and a few buyers get interested in the impact certificate.

   1. They by and large agree with the issuers on the expectation of the systemic observer in Period 0. This systemic observer is virtually the same as the one in Period 2. They imagine themselves in the shoes of the issuers in Period 0, and find that they might have predicted a few additional benefits and drawbacks but nothing substantial, and agree with those that the issuers have documented in the impact certificate, and with their conclusion that the thing appears like it would be net positive in expectation.

   2. Then they return into their own shoes where they have the invaluable benefit of hindsight. From this vantage point they realize that a few of the drawbacks they were retroactively worried about had not manifested but that some of the best-case scenarios have also become less likely. All in all, the thing still appears to be net positive in expectation, now with slightly reduced variance. They can’t quite tell which expectation, ex ante or ex post, is lower, but they think that both are positive and fairly close together, so they don’t care much and just bid on the impact certificate according to their budgets.

Note that strictly speaking the buyers also simulate the issuers because the certificate will be phrased in natural language which is highly ambiguous without knowledge of the (likely) intent of the author. We’ll omit this simulation in the graphs since it’s quite intuitive.

That’s all a bit of a simplification because many prospective buyers (let’s call them investors) want to make a profit. So they’ll perform further evaluations to understand whether they may know more or have more accurate information about the real expected impact than other investors and whether the other investors will attain that knowledge too, be it automatically over time or through a publication by other investors. Occasionally they may also seek to prove that the issuer had private information already in Period 0, which they pretended not to have, or conversely, whether something that is widely assumed to have been known at the time actually wasn’t knowable.

Example

Example. The issuer has sold 500 pins with cute designs and animal rights messages on them at a convention. They’ve donated the proceeds to an animal rights charity. They now sell the impact from this action.

1. The issuer thinks that the donation of the proceeds is the main driver of the positive impact, so the certificate text focuses on the money.

2. But one investor thinks that most investors are unexcited by the donation because they think it just displaces donations from corporate partners of the charity who don’t have set CSR budgets.

3. This one investor thinks that most investors think that reaching 500 people with the animal rights messaging is the main driver of the impact, in particular because the designs were actually created by well-known members of the community that the issuer sold the pins to, something that is widely known but that the issuer didn’t mention.

4. But this one investor also freshly found photos from the convention and noticed that the issuer had sold five different designs, and that, in group shots, a lot of people could be seen wearing multiple or even all five pins. The investor shorts the certificate before announcing their finding because they expect that the issuer had reached much fewer than 500 people.

This example aims to clarify that:

1. Issuers and investors can disagree on the path to impact, yet it can still be valuable ex ante and ex post.

2. The certificate description is never going to describe reality fully (and critical omissions can be unintentional), so it should contain as much information as possible that will help buyers do their own research to fill in gaps – such as finding photos from the convention.

3. The issuers and all investors can have different ideas of the actual and the counterfactual world histories from the perspective of the systemic observer.

4. Only when an investor can anticipate a price-relevant update of some of the other investors can they make money. If an investor knows more than the others but has no way of convincing them, they can’t scoop them. (But in this case they could because they had photographic evidence.)

Pricing Impact

Attribute Impact itself has no particular unit. One could pick a benchmark project – such as $1 distributed through GiveDirectly – and measure projects in relation to it, but we have not attempted this.

But that is not important for the market to function. The fair market price of an impact certificate can be determined through an auction. Then afterwards everyone is free to have their opinion on whether the certificate is a bargain or is overpriced, but either way, there will be a price.

The way the auction could work (so that it requires minimal time investment from the retro funders) is as follows:

1. A retro funder wants to incentivize work on Vingean reflection.

2. The retro funder announces a prize pool of $1k for anyone who makes progress on the problem.

3. They receive no serious or interesting submissions.

4. The retro funder announces an increase of the prize pool to $10k.

5. They receive no serious or interesting submissions.

6. The retro funder announces an increase of the prize pool to $100k.

7. The retro funder receives several interesting submissions and buys shares in some of them for most of the $100k budget.

This way the retro funder has found out that it took a prize pool of $100k to incentivize the submissions. Maybe each submission received $30k, and years later experts will conclude that they created $30b in social benefits. But that is not important for the functioning of the market if the researchers are ready to accept $30k.

Limitations

Trivially, the Impact Attribution Norm is only powerful to the extent that it is adopted by the market, in particular by the retro funders. That means that it cannot prevent people who have a bit of money from going on Twitter and announcing “I will give you $1m if you realize this very risky project.” They can do that already. We will need to rely on curation rather than the Impact Attribution Norm to prevent them from doing it on our marketplaces.

Second, retro funders need to be transparent about their use of the Impact Attribution Norm and they need to understand it well enough that investors trust them to apply it well. If they fail to signal that sophistication, investor may still end up investing into overly risky projects under the assumption that they can fool the retro funders. This need not happen on purpose. If risks are sufficiently well hidden, they may remain hidden from investors and retro funders for the same reasons. (But in that case it is likely that prospective funding wouldn’t fare any better.)

Finally, something that can exacerbate the second problem is that investors may believe that retro funders will be systematically biased to overestimate the ex ante expected value of projects that have already turned out well, be it through sheer luck. Phil Trammel has proposed that issuers who preregister their impact certificates can start prediction markets for their impact in accordance with the Impact Attribution Norm alongside them. Then retro funders can debias themselves by looking at the history of the predictions, in particular the predictions from before any actions were preformed.

Responsible Retroactive Funders

The main responsibility for the health of the market is upon the retro funders. Hence they need to be sufficiently wise to steer the market and sufficiently involved to notice when something is going wrong in the market.

First, a retro funder needs to ask themselves:

1. How much time are we spending on finding giving opportunities whose interventions are effective and that are implemented by capable teams? How much is it worth to us to not have to evaluate the teams? How much are we willing to offer to investors to incentivize them to do the work for us?

2. Do we think that in the future people will value the impact that we’re interested in even higher, so that we can one day resell it?

3. Do we want to see more highly effective charities, and how much are we willing to pay to incentivize their founding?

If these questions generally turn out that the retro funding will be worth it, there are further consideration:

1. Investors and founders will rely on our future funding (not that they’ll get it necessarily but that it’ll still be there). Can we commit a certain budget firmly, and if we need to discontinue it, can we afford a few years of grace period between the announcement and the discontinuation?

2. Can we make it sufficiently clear to founders and investors what the impact is that we want to see, so as to prevent people from expending great effort on potentially harmful projects that we would never pay for?

3. Are there maybe even metrics that can be used as rough guides by founders to check whether they’re on the right path without great risks of Goodharting or moral defection?

4. Do we have the resources for occasional calls with founders or investors who want to be sure that they’re starting the right sort of project?

5. Do we have resources to occasionally write a blog post to clarify our interests when we see investments being poured into uninteresting projects?

Finally the hardest question:

1. Are we ready to commit to being as impartial as possible about our impact evaluations? Especially in cases where we have strong moral feelings in one direction or the other, can we be trusted to still reward only that project that successfully compromises between the competing interests? Or put differently: Can we be trusted to price projects in proportion to the moral gains from trade that they generate?

If someone is only interested in providing retro funding but not in doing all the work associated with being a responsible retro funder, then the next section will be interesting, as they can simply contribute to the “Pot of Money” and leave the work to the jury of the pot.

Pot of Money

If there are sophisticated, responsible retro funders on a market and if they have a lot of money to give away compared to the volume that is transacted on the whole market, then everything is well and good. But even if either (1) there is no retro funder, (2) there are sophisticated, responsible retro funders but they are overpowered by rogue ones, or (3) there are only sophisticated, responsible retro funders but speculation on the market has developed its own rules, then the influence of the good retro funders will be insufficient to keep the market healthy. This is where the pot comes in.

The pot is a reference to the pot in poker, a vehicle for investors to “gamble” – except that no intentional randomness is involved, only the imperfections of impact evaluations and of predicting the future. For simplicity, we’ll copy the example from earlier but rewrite it to the case where the only retro funder on the market is the pot.

There are now four components:

1. Projects: The charitable projects that are waiting to be realized.

2. Supporters: These include founders, investors, advisors, et al.

3. Jury: An expert panel of sophisticated altruists, who become the source of truth when it comes to what has been impactful.

4. Pot of money: When the sophisticated altruists publish their decree, this pot of money is dispensed to the supporters.

At first, the pot of money likely has to be managed separately. The charities are not a good fit for managing it, and the funds may not be sold on the whole system from the start, or not sufficiently for all the overhead it would involve for them.

Let’s suppose again that the Against Malaria Foundation (AMF) wants to do a distribution of long-lasting insecticide-treated bednets with its old distribution partner Concern Universal (CU):

1. AMF ascertains that the jury of the pot is (and maybe other retroactive funders are) still interested in more net distributions.

2. The AMF staff decide how they want to split the Impact Attribution Norm from the distribution among themselves, the legal entity AMF, all their partners, contributors, and advisors, and their funders. These decisions can be adjusted in later negotiations. (we’ll treat AMF, the legal entity, and all its staff as just “AMF” in the following to save space.)

3. AMF decides on the details of the auction process, that is, what fraction of the profits from each sale should go to AMF and what the minimum percentage raise is between bids.

4. AMF creates an impact certificate.

5. AMF contacts CU, negotiates their split with them, and sends them their fraction of the impact certificate. (we’ll assume here that the distribution partner has its own source of seed funding just to make its case interestingly different from AMF’s.)

6. AMF contacts a venture capital firm. The VC agrees to fund the distribution. They negotiate the split, and AMF sends them their fraction of the impact certificate.

7. Everyone contributes to the pot as they see fit (or doesn’t), and the pot records who contributed how much to bet on which certificate.

8. A year later the distribution gets completed successfully.

9. The jury notices the successful distribution, is ecstatic, and buys fractions of the certificate from all holders who have contributed to it. AMF has no work with this: Any holder just has to give their certificate to the pot (or a fraction thereof), which ascertains that they’ve previously bet on it, and the pot buys it from them at a higher price.

The core of the system should, in our opinion, not depend on any additional retroactive funders because (1) they may not be interested in the market until they see it work; (2) their funding is finite and independent of the impact market; and (3) they may not have enough in-house experts so that they want to defer to our jury anyway, effectively adding to the pot.

That said, external retroactive funders would make the market attractive for many more participants!

When it comes to the size of the reward, we currently favor the following system:

1. Everyone contributes whatever they want but has to tie this “bet” to a particular impact certificate.

   1. This removes incentives that investors who don’t like the pot might otherwise have to circumvent the market.

   2. They need to bet on a particular certificate rather than just any that they own so that they can’t just buy minimal fractions of countless certificates at random, which would add little to no wisdom to the market and wouldn’t differentially benefit projects with real promise.

2. When the jury approves of a certificate, the pot sets a budget for the purchase and buys fractions of the certificate from each buyer according to the product of (how much of the certificate they hold) * (their contribution to the pot for that certificate). 3. That way people are rewarded for contributing to the pot and the project, and those who fail to do either can’t sell to the pot. 4. They may of course do so on purpose because they want to keep the certificate as a long-term investment or to sell it to another retroactive funder.

This still glosses over a lot of details. They are easier to explain in the context of the auction platform.

Auction Platform

We want to create an auction platform to streamline the whole process. (You can keep up-to-date on the progress here.) The current model is a type of auction where the impact certificate is always owned by the highest bidder and so constantly changes hands. Meanwhile the profits from each sale are split between the seller and other parties, typically the issuer and the Pot. We’ve covered it earlier in the subsection Auction Platform.

In this model the issuer determines what fraction of the profits of each sale should go to them. Here’s the rough timeline of the trades.

Here an overview of the accounts of each participant at each step in the process (you can find the spreadsheet here):

Various parameters:

In this scenario AMF made its profit share relatively small (1%) to encourage investment. The minimum raise (the step between bids) is at an arbitrary value of 10%. The minimum bid is set such that AMF won’t raise amounts of money that are insufficient to fund the net distribution. The nets and salaries on AMF’s side will be $500k, and the salaries and miscellaneous costs on CU’s side will be $500k as well. (Traditionally, AMF’s strategy has been to fundraise for the nets but to leave their distribution to other organizations.)

The deal with CU is described above. It’s now that 30% of the certificate are issued. The deal with VC 1 is also described above. This is the first bid on the market, so VC 1 gets to buy them at the minimum bid price. VC 2 buys the shares from VC 1 right away, but this time the minimum raise and the profit share apply, that is, the valuation increases by 10%, and 1% of VC 1’s profit goes to AMF instead.

Next, all three certificate shareholders contribute to the pot. AMF and CU add an arbitrary $50k to the pot whereas VC 2 tries hard to maximize the weight it’ll have when the pot buys the certificate from them again, but without selling more of the certificate than it owns. The prices in the ratchet auction can only go up, so even if it could borrow certificate shares from (say) AMF, that would not be profitable.

We assume here that (1) VC 2 is certain that the pot will later buy the certificate, (2) can wait for and then observe the pot contributions of all other shareholders, and (3) knows how much money is going to be in the pot and how much it will budget for this purchase. These are all slightly tricky assumptions: Assumption 1 is never going to be completely true; assumption 2 can lead everyone to wait until the last possible second to make their contributions; and assumption 3 is probably again somewhat difficult in practice.

If the jury were to deem the net distribution to have been ineffective, the scenario would end here with an extra $174k in the pot and all organizations having lost money.

Finally, though, the pot decides on a budget of 20% and a bid 20% above the last one. The resulting valuation is more than the budget, so that the budget doesn’t need to be capped. The weight is proportional to the value that the shareholder has in the certificate times the size of the pot contribution. (You can find the spreadsheet here.)

The column “Share” is the share in the impact certificate that the respective party owns. The “Value” is the value of the share according to the latest valuation of the certificate. The “Contrib. [-ution]” is the amount that the party has freely chosen to contribute to the pot. The “Product” is the product of the value of the share and the contribution – it doesn’t map to anything in particular in the real world, but the weight is proportional to it. The aforementioned “Weight” is the fraction of the budget of the pot that the pot uses to buy certificates from the respective party. The “Cost” is the resulting total dollar volume of the purchase. The “Size” is the fraction of the certificate that changes hands.

The final budget for each individual purchase may not be fully used up if any of the shareholders doesn’t own enough shares (though this case doesn’t happen here and is not part of the calculations in the spreadsheet because it made it harder for us to find errors in it).

The result is that the pot spent about $1.86 million and every organization profited, either monetarily like the VC, altruistically like CU, or both like AMF.

Notes:

1. It took us a while to come up with a scenario that “works,” in the sense that in particular the VCs make a profit (or else they wouldn’t invest) without making the pot or the fraction that it invests unrealistically big or its bid unrealistically high. This indicates that the parameters of the pot budget will need a lot of tuning and that issuers and investors will be well-advised to carefully think through their funding scenarios to make sure that they’re realistic.

2. This difficulty is exacerbated since the pot will probably require holders to have held the certificates about a year earlier already to prevent frontrunning of pot purchases. Investors who hold certificates for a year will need to make much greater profits off them to make up for the counterfactual uses of their funds. Unless another investor buys it from them, and they buy it back much later.

   1. The current auction system doesn’t support or would make it at all lucrative, but under different conditions, holders could lend their certificates to short-sellers to earn interest on them.

3. But this may be a feature rather than a bug in the short run: The distribution mismatch problem below is alleviated if (1) the Impact Attribution Norm has time to be adopted as the Schelling point of impact evaluation on the market, and (2) there are few purely profit-oriented investors. We may grow a safer, more altruistic community if the market is not of interest to anyone unless the social bottom line also counts into their personal profits.

4. There’s a questionable dynamic where some investors benefit if other investors into the same certificate pay less into the pot. We don’t yet know whether there are setups where it’s profitable for all investors if one of them pays other investors outside the market in exchange for them withholding money from the pot. But if there are such scenarios, then that may be bad for the pot.

5. There’s another bad dynamic, maybe a negative externality, where sophisticated for-profit investors are incentivized to advertise the system to unsophisticated investors and potentially mislead them. They may promote the impact certificates of highly ineffective charities to unsuspecting people as investment vehicles to use in this system. These people would enlarge the pot without having any chance of ever winning it. Meanwhile all the misinformation may make it harder to find high-quality charity reviews for people who don’t already know where to find them.

6. Finally, we’re unsure how to prevent insider-trading by jury members, but there are probably already mechanisms for that, such as that jury members need to be many and not know each other. There are probably time-tested solutions to this problem that we just need to find out about.

Prediction markets. Something we want to think about more is whether the Pot should instead be a scalar prediction market predicting something like the percentage of endorsement of the project from the jury. Investors may be hesitant to pay into the Pot if it’s unclear for how long their money will be locked up in there even if they win their bet – a prediction market would address that.

Auditors

Our ideas for how to structure the ecosystem of auditors are still inchoate.

Auditors will need to audit impact at least twice. First there needs to be an audit that issuers need to get before they fundraise from investors. Instead of having to plow through all the impact certificates on the market, investors can then just consider ones that have been audited in this way. Such an audit needs to consist of a few basic checks, such as whether the argument for the Impact Attribution Norm makes structural sense (e.g., is not autogenerated gobbledy-gook), is concrete enough that it can be audited at all, and crucially that the impact is not being double-issued.

The second audit needs to also ascertain that the impact has happened. A lot of forms of impact are easily verifiable – an article author, for example, can just link to the article they’ve written.

Checking that impact has not been double-issued is hard in full generality, but it can be mostly solved if all auditors publish all their audits, and all auditors know about all other auditors and check their lists of published audits. Then they can’t prove that impact has not been double-issued at all, but if it has, at least the other issue does not have an audit. This can still break down in cases where the double-issue is sufficiently subtle, but it should catch a lot of more obvious cases.

One thing that would be helpful for this job is a standardized format that all auditors use to publish their audits. Maybe a JSON blob on their websites. Then a search engine can extract structured data from there, and the search will be much easier for all of them.

One idea how to incentivize auditors and ameliorate risks from harmful projects too is to require issuers to be insured when they issue on the platform. If it is later decided that the project was ex ante harmful and it is removed from the platform, the insurer needs to pay a fine on behalf of the issuer. If a project actually predictably backfires, a larger fine is due. This incentivizes insurers to deny insurance for projects that are not safe enough. A problem is hidden in the passive-voice “If it is later decided,” because whatever process decides this has to be trusted by the insurers. That promises to be a difficult problem to solve.

Shorting

All of our solutions try to make it uninteresting to issue or to invest into impact that is likely net negative, but another option is to require investors to deposit collateral as insurance against the case where the impact turns out negative. The catastrophes that we’re worried about are of the civilization-ending type, so no amount of collateral can realistically be enough, but it may still increase the friction a bit when it comes to investing into potentially net negative impact.

One idea is that we expect that in the long run most people will be agnostic about which impact certificate (e.g., which net distribution of AMF) they invest in because they’re not close enough to the specifics of the intervention to see the differences between them. They’ll just want to invest in “AMF.” So I find it plausible that there will be demand for perpetual futures (see also “What are perpetual swaps?”) that track something like the market capitalization of all impact certificates issued by AMF. (That’s not an interesting market in the case of the ratchet auction where prices can only go up, but it may be with other auction mechanisms.)

To get exposure to this perpetual future, investors could deposit collateral and open long positions on it. If they use too much leverage and the price decreases too much, they can lose their collateral.

But the other benefit – hinted at in the subsection title – is shorting. Shorting can help to price in the expectation of some investors that retro funders will not be interested in the impact or not at that price. It doesn’t by itself have the same upside potential as longing, but if a short-seller regularly reinvests their collateral when it increases due to decreasing prices, they can maintain a constant leverage and gain a similar exposure to nonanthropic downsides – at least before adjusting for the risks.

This sounds complicated, but there are already “hedge” tokens that automate the process. These could be applied to the perpetual future and thus give investors a simple, safe, and low-effort way to get negative exposure to overpriced impact.

However, we’ve seen these tokens fail especially during crashes, when you want to profit from them most. My hypothesis is that there is so little buy-side liquidity during crashes that when the token smart contract tries to reinvest its PnL to increase the leverage, it has very few buyers to sell to and so has to sell at low prices, prices close to where the market is crashing to anyway. They probably work better in markets where the prices decrease gradually.

All in all, it’s much too early for this system. There’ll first need to be highly liquid markets and popular issuers of many impact certificates before it makes sense to think about this system more.

Exposé Certificates

Shorting becomes particularly powerful in combination with curation: Shorting incentivizes market actors to open their short positions and then to publish exposés to get others to divest (too). But these exposés are invaluable to the curation team in and of themselves because they help them spot their own oversights.

This particular function of shorting can be achieved more simply: The action of helping the curation team spot net negative projects is one that is usually quite robustly positive, so that exposés can become quite valuable if turned into impact certificates. That way the impact market creates its own “bug” bounty system. Retro-funding of these certificates has to be funded by a retro funder that is interested in the success of the market.

Targeted Marketing

Especially in the early days when the market is still young it’ll be important who to market it to. The above section on responsible retroactive funders has already clarified why it is critical to recruit the sorts of funders who will let the markets thrive rather than ones that’ll lead them astray. That is one problem that can be partially addressed by reaching out specifically to the retro funders who we think will do a good job of it.

But the same is true of charities and investors, just to a lesser extent. The charities on the market need to be the ones that produce good/s that is/are interesting to the sorts of retro funders we want to have on our markets. The reason is different in the case of the investors: If we recruit investors whose motivations are mostly altruistic, and we then notice that our markets create bad incentives for them, then we can warn them of these incentives, and it’ll be in their altruistic interest to resist them. But if we run into bad incentives and have unaltruistic investors, we don’t have this option.

Current Work

Protocol Labs has organized two iterations of the conference Funding the Commons. “Funding Public Goods – Algorithms and Mechanisms” by Vitalik Buterin and “S-Process Funding” by Andrew Critch are particularly interesting because they highlight important mechanisms, and “Quadratic Funding on Gitcoin” by Kevin Owocki, “Retroactive Public Goods Funding Experiment 1” by Karl Floersch, and “Impact Evaluators” by Evan Miyazono are particularly interesting because they give insight into the work of the actors who are currently active in the space. Finally, “Impact Certificates and Impact Markets” by Owen Cotton-Barratt is the talk that we collaborated on and that we already linked above. We also had a talk at the second Funding the Commons conference. We’re planning to summarize some of the talks in a separate article.

These talks highlight our work, the work of the Ethereum Foundation, the Survival and Flourishing Fund, Gitcoin, Optimism, Protocol Labs, and many more. Beyond that, Sentience Research, the EA Hotel, and Giveth have been following the field with interest.

We’re also in touch with NPX Advisors who’ve been running a quasi–impact market with greater amounts of funding but a very small set of funders, investors, and charities in the US, and who conceive of “impact certificates” as debt securities rather than impact equity securities.

We think these are viable approaches because they test impact markets in fields that are safer than some of the main EA cause areas. That way they either work with participants who are highly informed when it comes to financial markets or within fields where it is hard for us to see how impact markets may backfire catastrophically.

Note that we – Kenny Bambridge, Matt Brooks, Dony Christie, and I – can use funding: $100k would enable us to invest more time into this project (about 70% more in my case) and any amount around $10k to $1m would would allow us to conduct retro-funding experiments. This money could be merely committed to these experiments and wouldn’t need to be transferred in advance. We would then make grant recommendations according to guidelines that we would like to publish in advance in order to make the recommendations as predictable as possible. Please get in touch if you’d like to help with funding, advice, or otherwise.

Acknowledgements

Many thanks to my collaborators Kenny Bambridge, Matt Brooks, and Dony Christie! Thanks to Katja Grace and Paul Christiano for the initial inspiration. Thanks also to Owen Cotton-Barratt, Ofer Givoli, Jay, mqp, bryjnar, Mako Yass, and Justin Shovelain for discussions and ideas. You’ll each receive 1% of the impact of this article. Thanks to Protocol Labs for the Funding the Commons conferences.

Thanks for reading Impact Markets! Subscribe for free to receive new posts and support my work.

Introduction

One of my projects in 2020 was a software for doing math with probability distributions – SquigglyPy. Sort of like Guesstimate but with the ability to estimate probabilistic trajectories over time or some other independent variable.

I had seen a number of researchers either write code for Monte Carlo simulations from scratch or create models based on mere point estimates. My software would save the first group some time and help the second group get a better sense of the degree of uncertainty in their model.

My vision for it was of course greater than that. I hoped to inspire people to use the software to model various interesting effects in the world in the various ways that seem most useful or parsimonious to them, to publish these various models online, and then to recombine existing models to create ever more comprehensive models. Thus people would loosely collaborate to create a faster and faster growing ecosystem of Baysian models of the world – along the lines of how open source software allows developers to develop new software faster and faster by recombining increasingly powerful chunks of existing software.

My hope was that this would be done primarily by researchers in global priorities research and would lead, over the course of decades, to an improved allocation of resources.

I’m putting this project on hold for now because (1) I updated downward on the time we have left till transformative AI emerges, which had complex effects on my prioritization and (2) because I didn’t feel like it had as much traction as other projects of mine. That may change of course, and I may pick it up again at a more opportune time.

Meanwhile I’m happy to add other developers to the repository or to accept pull requests. In particular, it would be great to improve:

1. the tests, to catch mathematical errors,

2. the sensitivity analysis, which is currently a hacky sketch on Starboard, and

3. the documentation, later, when the code is better tested.

Warning: When using SquigglyPy, please take great care to sanity-check all your results! Hardly anyone has used the software yet and the tests are currently better at catching broken code than subtly wrong math.

Squiggle

My vision was similar to Ozzie Gooen’s (Quantified Uncertainty Research Institute), which is part of the reason I chose the name SquigglyPy. Ozzie developed a similar software in ReasonML, which also runs in the browser. It’s the ReasonML implementation of his Squiggle language whereas SquigglyPy is the Python implementation.

I makes senses to me to implement Squiggle in a few different languages:

1. Any language that compiles to JavaScript or WebAssembly has the advantage that it runs in a browser. The right UI and an efficient implementation can give people the ability to experiment and explore rapidly. Models might be shared with others through repositories on the same server to save the time it would take to download and install a software from a popular existing platform like NPM or PyPI.

2. Languages like R, Julia, and Python have the advantage that the main target demographic (as I currently imagine it) uses them and has built a wealth of statistical tooling for them already.

3. Finally, Python has the advantage that there is already a project that has made Python run in the browser, combining all advantages. (It also has the advantage that I’m very familiar with Python.)

But there are also drawbacks to using Python in the browser:

1. Pyodide is a very small, young project and may not survive.

   1. But chances are another project will pick up the slack because Python is popular and WebAssembly seems to me like the obvious next step in the technological development of websites.

2. Many statistical libraries use some C or Cython extensions, and so can’t just be installed through Micropip. Instead the Pyodide team (or whoever wants to use them) has to create a WebAssembly version of them. The Pyodide team at least is too small to support a lot of WebAssembly builds.

   1. But over the years it may become more common for library authors to produce WebAssembly builds themselves or for companies to publish any such builds that they’re using for their products.

3. The Python interpreter takes a moment to load, which may degrade the user experience compared to a JavaScript implementation.

   1. But it loads only once per page load and then stays in memory.

I feel like the advantages outweigh the disadvantages here, but I could easily be wrong in any of various way:

1. The world of stats may transition toward Julia or some other more specialized language than Python,

2. Python may lose market share to Rust or some other similar language,

3. More statistical tooling may be built for JavaScript or even WebAssembly directly, or

4. Any of surely countless unforeseen vicissitudes may eventuate.

Risks

I’ve detailed a lot of risks in “How Might Better Collective Decision-Making Backfire?” They fall into the general buckets of risks that need to be addressed immediately and risks that can be addressed later or reacted to only if they manifest.

Immediate

1. Programming errors.

   1. This is the primary reason I’ve put off writing this post and why now I only publish it on my blog where few people will see it. It’s very likely that there are mathematical errors in some parts of the code that will skew any results. This might lead to disastrous misprioritization.

   2. The main way to address this issue is to improve the tests and to get people to use the software for many typical but not absolutely crucial purposes. These people need to be closely in touch with the developers and take their roles as beta testers seriously.

2. Version conflicts and path dependent legacy software.

   1. The Squiggle language will need a clear, versioned language definition so that implementations of models in different languages are interoperable, either to the point where you can copy-paste the code or at least to the point where the real Squiggle code can be turned into a language specific version automatically. (Along the lines of how the 2to3 tool turns Python 2 code into Python 3 code.) Otherwise some collections of models may end up siloed by their dialect or it’ll become hard to innovate for fear of losing backward compatibility.

3. Failure to catch on among the right people.

   1. Squiggle may fail to get used or it might be used by people with nonaltruistic motives or uncooperative methods.

   2. I’ve changed the name of the library from Swungdash to SquigglyPy to make it easier to set up Google Alerts to track who is using it.

   3. It may also be valuable to cultivate a community around it that has a lot of knowledge and resources so that people who use it have an incentive to join the community, so that the developers can stay in touch with them.

4. Conflict through coordination failures.

   1. If a community forms around the creation and composition of Squiggle models, then there will be power in the repository of knowledge of that community. It’ll be important to instill the right ethos in the community early on to avoid the sorts of coordination failures that can arise even among well-intentioned actors. It is not clear what ethos that is, so a general readiness to adopt it and an interest in cooperative conflict resolution may be a good start.

5. Discrimination against illegible values.

   1. There is a risk that Squiggle will differentially enhance the thinking of people with broadly utilitarian values because they’re easier to quantify than other values. Other values may be displaced or abandoned.

   2. I consider it an open question which procedures to idealized preferences are cooperative, but to the extent that the influence of Squiggle will be uncooperative, it’ll be a problem. We may choose to reject such uncooperative behavior terminally or consider that it can backfire against us in an acausal bargain. (See Oesterheld 2017, especially section 3.3.1.)

   3. Either way, the cooperative ethos of the hypothetical Squiggle community should include respect for other value systems and protection of strong interests of moral minorities.

6. Missing of failure modes when they happen.

   1. On a higher level, we can try to prevent or address all these risks, but we may also fail to notice them when they happen, which seems like a risk in itself.

7. Benefiting malevolent actors

   1. Finally, there is the risk that we’ll build up a vast body of knowledge that can then be abused by people with malevolent or otherwise uncooperative intentions.

   2. It may pay off to reduce the generality of the knowledge we build by focusing on research questions that are particularly interesting to altruists only.

   3. Making the community inhospitable to such people may not be a good approach, at least not generally, because that may not slow them down but it will deprive us of a chance to talk to them and make them see the future from our perspective.

Delayed

These will either take a while to manifest, will be obvious, will be easy to control, or will be problems no matter what. They are explained in “How Might Better Collective Decision-Making Backfire?” This subsection is mostly notes I wrote to myself a while back.

1. Decision theoretic catch-22s.

   1. I expect that military intelligence makes a big difference here, but prediction tools may still improve upon this significantly. (I’m mostly thinking of conflicts between countries here since behaviors of other people in interpersonal conflicts can be more easily intuited by many/most people.)

   2. This will be easier to address with better decision theory, and it’s a big problem anyway, so people (e.g., at MIRI) are probably long working on said decision theory.

   3. The example I cite in the blog post has the feature that the payoff chosen by the dumb dictator is not a typical Schelling point. If this is a general feature of such situations, and if I’m not missing more commonplace examples of this exploit, it’ll likely take a long time before we become sophisticated enough to become vulnerable to this. I feel like I’m probably wrong about this, but I can’t tell in what way.

2. More power to the group.

   1. I think this can be observed and reacted to.

3. Overconfidence may be necessary for motivation.

   1. There is already a strong focus in the community on individual rationality. It seems like a stretch to think that a collaborative system would influence this potential issue at the margin.

4. The sunk cost fallacy may be necessary for sustained motivation.

   1. Same as with overconfidence.

5. Silencing of lone dissenters.

   1. This only works if the collaborative group has a lot of social power, so at this point, it’s probably not something that needs to be addressed.

   2. It also seems like something that can be reacted to, i.e. that doesn’t need to be prevented at all costs.

6. Loss of valuable ambiguity.

   1. This will probably take a long time to manifest if it does.

7. Social ramifications.

   1. This can be evaluated at the time.

8. Legibility and centralized surveillance.

   1. This depends heavily on who will feel threatened by what research. So in the end, not using the tools for potential sensitive research is an option.

9. Cooperation-enhancing effects.

   1. Probably good on balance,

   2. Unlikely to be affected by the tools,

   3. Will take a long time.

Demo

Christian Tarsney writes in his paper The Epistemic Challenge to Longtermism:

The ideal Bayesian approach would be to treat all the model parameters as random variables rather than point estimates, choose a probability distribution that represents our uncertainty about each parameter, and compute EV(L) on that basis. But for our purposes, this approach has significant drawbacks: EV(L) would be extremely sensitive to the tails of the distributions for parameters like r, s, and vs. And specifying full distributions for these parameters—in particular, specifying the size and shape of the tails—would require a great deal of subjective and questionable guesswork, especially since we have nothing like observed, empirical distributions to rely on. Even if we aim to adopt distributions that are conservative (i.e., unfavorable to longtermism), it would be hard to be confident that the tails of our chosen distributions are genuinely as conservative as we intended.

That seems fair. My software, SquigglyPy, addresses the concern about subjective guesswork insofar as it’s not a printed and published paper but an interactive software tool that anyone can play around with and run with their own subjective guesses for parameter values.

SquigglyPy does not yet address the related concern about sensitivity to tiny differences in the shapes of the tails of distributions. This concern takes two forms: (1) We may have no reason to prefer one tail shape over another and (2) the Monte Carlo–based sampling methods that SquigglyPy uses to approximate the tails may never use enough samples to approximate them sufficiently no matter how many samples we tell SquigglyPy to use.

I consider the first problem to be outside of the scope of SquigglyPy. You can try to address it by experimenting with many very different tail shapes or, if you’re dealing with a univariate distribution, to plot the distribution for a range of tail shapes. But those are, admittedly, not solutions that are universally applicable or sure to cover all possible shapes.

The second problem is one that it would be very interesting to address in a future version of SquigglyPy. The ReasonML implementation already tries to manipulate various well-known distributions analytically for as long as possible before resorting to sampling. That seems like a very promising 80/20 solution to this problem.

Technical note: Ideally, I’d like to implement a class Distribution that always has a representation that is a mapping from x to y coordinates but also, whenever possible, has a representation that is a formula and one that is a set of samples. All sorts of mathematical operations could then be defined on distribution samples (the status quo) and formulas, all fully encapsulated in the Distribution class. Any code that uses the distributions could be completely oblivious to the inner workings of the distributions, just mathing them like any other variable, while the distributions would take care to keep their formula representations for as long as possible. Plots could always use the x/y coordinate representation, again not caring how the distribution generated it.

Starboard

The following is best viewed on Starboard.

Starboard is like a Jupyter Notebook or like Google Colab except that it runs in your browser! The nice folks of Pyodide have compiled the CPython interpreter to WebAssembly so you don’t have to connect to a server to run your Python scripts. Starboard then built a Jupyter Notebook–like environment around it.

To use SquigglyPy in Starboard, you can simply import it with Micropip, because it’s pure Python. Of course you don’t have to use Starboard. You can also install it with pip, e.g., in your Poetry environment.

import micropip
micropip.install('https://packages.claviger.net/public/squigglypy-0.2.2-py3-none-any.whl')

Next, a bunch of imports. From SquigglyPy in particular we import the context, which stores some more or less global state and settings, various distributions, and miscellaneous utility functions.

import math

import scipy
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.transforms as transforms
import pandas as pd

from collections.abc import Iterable

from squigglypy.context import Context
from squigglypy.dsl import normal, uniform, mixture, pareto, lognormal
from squigglypy.utils import bfs, as_model

plt.style.use('seaborn')

The plotting function is a mess and quite specific to our case here, so I didn’t want to include it in the library. Ideally, I’d like to have plotting methods with sensible defaults integrated into the model API so that you can produce a plot with one command. But that would require (1) knowing what these sensible defaults are, (2) coming up with an API that allows you to override these defaults at will when they don’t work for you without copy-pasting a big chunk of code, and (3) tweaking the result based on the actual needs of actual users. So I’d be happy to accept pull requests that solve that for me. ^.^

def plot(model_, points, sample_count=100, sample_fmt='b-', quantiles=(0.1, 0.5, 0.9)):
    with Context(sample_count=sample_count):
        samples = np.stack([~model_(point) for point in points])
    p10, p50, p90 = np.quantile(samples, quantiles, axis=1)
    plt.figure(figsize=(8, 6))
    plt.yscale('log')
    plt.plot(points, p50, 'r-', label='mean')
    plt.fill_between(points, p10, p90, color="red", alpha=0.1)
    plt.plot(points, samples, sample_fmt, alpha=0.1)
    plt.xlim(0, None)

Christian Tarsney’s cubic growth model (section 4) describes the scenario that seems most sensible to me. I’ll focus on it for the purposes of this demo. My original plan was to implement it here faithfully, but I don’t have time for that now, so it has more the character of a proof of concept.

I’ll keep the explanations brief here because Tarsney has already done a great job at that.

value_star = pareto(.4) * 3.333e5
value_star.name = 'Value delta per star (v_s)'

value_earth = lognormal(0, 1) * 2e6
value_earth.name = 'Value delta on Earth (v_e)'

rate = uniform(10, 1e8)**-1
rate.name = 'Rate of exogenous nullifying events (r)'

# This doesn’t seem to work for some reason?
speed = 0.1  # lognormal(1, .3) / 10
#speed.name = 'Speed of settlement in c (s)'

year_start = 0                        # = t_l, year of start of settlement
year_end = 1e14                       # = t_f, eschatological bound
probability = 2e-14                   # = p, probability delta
density_galaxy = 2.2e-5               # = d_g, star density within a radius of 130k ly
density_supercluster = 2.9e-9         # = d_s, star density within Virgo Supercluster


def number_stars(radius):  # In light years
    if radius <= 0:
        return 0
    elif radius <= 1.3e5:
        return (4/3) * math.pi * radius**3 * density_galaxy
    return (4/3) * math.pi * (
        radius**3 * density_supercluster +
        1.3e5**3 * (density_galaxy - density_supercluster)
    )


def cubic_growth(year):
    return value_earth + (
        value_star *
        number_stars((year - year_start) * speed) *
        math.e**(rate * year * -1)
    )

years = range(0, int(1e6), int(1e3))

This is the value over time. There is an Integral resolver in SquigglyPy, but I haven’t gotten around to using it, so this is just the part inside the integral. (Also ignoring the probability delta, which is outside the integral.)

You can see that all the semitransparent lines overlap and so naturally indicate where the probability density is highest. In this case, this is similar to the 80% HDI (reddish) and sort of clustered around the mean, but you can just as soon imagine a more multimodal distribution where the HDI would hide the multimodal nature.

plot(cubic_growth, years, sample_count=100)
plt.show()

Consider, for example, the fabulous foobar model. The reddish HDI and the mean are rather uninformative compared to the traces.

def foobar(x):
    weight = mixture([normal(2, .1), normal(0, .1)])
    bias = uniform(100, 200)
    return weight * x**2 + bias / 5

plot(foobar, range(100), sample_count=100)
plt.yscale('linear')
plt.show()

From here it’s also conceptually easy to develop a sensitivity analysis that tells us what influence a given constant (by which I mean a univariate random variable, one that, in our example, is independent of the time) has on the model value for a selection of points in time.

The main difficulties here are:

1. What methods of visualization are there that are more intuitive and/or informative than the one I’ve chosen here?

2. What graphing library do we use to make this pretty and interactive?

3. How do we integrate this into SquigglyPy so that the APIs are intuitive but also powerful enough to allow for a few other comparisons?

The goal would be to come up with an API in SquigglyPy that allows for a range of comparisons of parts of the model to other parts regardless of whether they are dependent or independent of x. (x is the time in our example.) The visualizations would have to be different ones for these combinations.

def sensitivity(model, xs=None, sample_counts=(10, 10)):
    x_sample_count, dist_sample_count = sample_counts
    xs = xs or range(30)
    constants, variables, tracer = bfs(model)
    constants = [constant for constant in constants if getattr(constant, 'name', False)]

    fig, axes = plt.subplots(len(constants), figsize=(10, len(constants) * 5))
    plt.subplots_adjust(hspace=0.5)

    for ci, constant in enumerate(constants):
        axis = axes[ci]

        with Context(sample_count=dist_sample_count):
            constant_samples = ~constant
            variable_samples = pd.DataFrame([~model(x) for x in xs], index=xs)

        if not (
                hasattr(constant_samples, 'shape')
                and constant_samples.shape == (dist_sample_count,)
                and variable_samples.shape == (len(xs), dist_sample_count)
            ):
            continue

        normalize_index = mpl.colors.Normalize(vmin=0.0, vmax=10.0)
        x_sample_indices = np.linspace(0, len(xs) - 1, num=10, dtype=int)

        axis.set_xlabel(str(constant))
        axis.set_ylabel('model value')

      rows = variable_samples.iloc[x_sample_indices].iterrows()

        for i, (x, variable_samples_at_x) in enumerate(rows):
            combined_samples = pd.DataFrame({'model': variable_samples_at_x, 'node': constant_samples})
            combined_samples = combined_samples.sort_values('node')
            cmap = plt.get_cmap('coolwarm', 256)
            color = mpl.colors.to_rgb(cmap(normalize_index(i)))
            axis.plot(
                combined_samples.node,
                combined_samples.model,
                c=color,
                linewidth=1,
            )

        axis.legend([f'x = {x}' for x in np.array(xs)[x_sample_indices]], loc='upper left')

sensitivity(cubic_growth, xs=years)
plt.show()

You can find the whole question thread in the EA Forum. This is a lightly edited transposition of the thread, that is, this article puts each question first and then lists all answers. This way, the table of contents allows you to jump to whichever question interests you most.

Thinking vs. reading

If you want to research a particular topic, how do you balance reading the relevant literature against thinking yourself and recording your thoughts? I’ve heard second-hand that Hilary Greaves recommends thinking first so to be unanchored by the existing literature and the existing approaches to the problem. Another benefit may be that you start out reading the literature with a clearer mental model of the problem, which might make it easier to stay motivated and to remain critical/vigilant while reading. (See this theory of mine.) Would you agree or do you have a different approach?

Jason Schukraft

I think it depends on the context. Sometimes it makes sense to lean toward thinking more and sometimes it makes sense to lean toward reading more. (I wouldn’t advise focusing exclusively on one or the other.) Unjustified anchoring is certainly a worry, but I think reinventing the wheel is also a worry. One could waste two weeks groping toward a solution to a problem that could have been solved in afternoon just by reading the right review article.

David Bernard

Another benefit of thinking before reading is that it can help you develop your research skills. Noticing some phenomena and then developing a model to explain it is a super valuable exercise. If it turns out you reproduce something that someone else has already done and published, then great, you’ve gotten experience solving some problem and you’ve shown that you can think through it at least as well as some expert in the field. If it turns out that you have produced something novel then it’s time to see how it compares to existing results in the literature and get feedback on how useful it is.

This said, I think this is more true for theoretical work than applied work, e.g. the value of doing this in philosophy > in theoretical economics > in applied economics. A fair amount of EA-relevant research is summarising and synthesising what the academic literature on some topic finds and it seems pretty difficult to do that by just thinking to yourself!

Michael Aird

I don’t think I really have explicit policies regarding balancing reading against thinking myself and recording my thoughts. Maybe I should.

I’m somewhat inclined to think that, on the margin and on average (so not in every case), EA would benefit from a bit more reading of relevant literatures (or talking to more experienced people in an area, watching of relevant lectures, etc.), even at the expense of having a bit less time for coming up with novel ideas.

I feel like EA might have a bit too much a tendency towards “think really hard by oneself for a while, then kind-of reinvent the wheel but using new terms for it.” It might be that, often, people could get to similar ideas faster and in a way that connects to existing work better (making it easier for others to find, build on, etc.) by doing some extra reading first.

Note that this is not me suggesting EAs should increase how much they defer to experts/others/existing work. Instead, I’m tentatively suggesting spending more time learning what experts/others/existing work has to say, which could be followed by agreeing, disagreeing, critiquing, building on, proposing alternatives, striking out in a totally different direction, etc.

(On this general topic, I liked the post The Neglected Virtue of Scholarship.)

Less important personal ramble:

I often feel like I might be spending more time reading up-front than is worthwhile, as a way of procrastinating, or maybe out of a sort-of perfectionism (the more I read, the lower the chance that, once I start writing, what I write is mistaken or redundant). And I sort-of scold myself for that.

But then I’ve repeatedly heard people remark that I have an unusually large amount of output. (I sort-of felt like the opposite was true, until people told me this, which is weird since it’s such an easily checkable thing!) And I’ve also got some feedback that suggested I should move more in the direction of depth and expertise, even at the cost of breadth and quantity of output.

So maybe that feeling that I’m spending too much time reading up-front is just mistaken. And as mentioned, that feeling seems to conflict with what I’d (tentatively) tend to advise others, which should probably make me more suspicious of the feeling. (This reminds me of asking “Is this how I’d treat a friend?” in response to negative self-talk [source with related ideas].)

Alex Lintz

I’ve been playing around with spending 15–60 min. sketching out a quick model of what I think of something before starting in on the literature (by no means a consistent thing I do though). I find it can be quite nice and help me ask the right questions early on.

Self-Consciousness

I imagine that virtually any research project, successful and unsuccessful, starts with some inchoate thoughts and notes. These will usually seem hopelessly inadequate but they’ll sometimes mature into something amazingly insightful. Have you ever struggled with mental blocks when you felt self-conscious about these beginnings, and have you found ways to (reliably) overcome them?

Jason Schukraft

Yep, I am intimately familiar with hopelessly inchoate thoughts and notes. (I’m not sure I’ve ever completed a project without passing through that stage.) For me at least, the best way to overcome this state is to talk to lots of people. One piece of advice I have for young researchers is to come to terms with sharing your work with people you respect before it’s polished. I’m very grateful to have a large network of collaborators willing to listen to and read my confused ramblings. Feedback at an early stage of a project is often much more valuable than feedback at a later stage.

Holly Elmore

Personally, I’m very self-conscious about my work and tend to wait to long to share it. But the culture of RP seems to fight that tendency – which I think is very productive!

Alex Lintz

Idk if this fits exactly but when I started my research position I tried to have the mindset of, “I’ll be pretty bad at this for quite a while.” Then when I made mistakes I could just think, “right, as expected. Now let’s figure out how to not do that again.” Not sure how sustainable this is but it felt good to start! In general it seems good to have a mindset of research being nearly impossibly hard. Humans are just barely able to do this thing in a useful way and even at the highest levels academics still make mistakes (most papers have at least some flaws).

Michael Aird

This is his answer to the questions about self-consciousness and “Is there something interesting here?”

These questions definitely resonate with me, and I imagine they’d resonate with most/all researchers.

I have a tendency to continually wonder if what I’m doing is what I should be doing, or if I should change my priorities. I think this is good in some ways. But sometimes I’d make better decisions faster if I just actually pursued an idea more “confidently” for a bit, to get more info on whether it’s worth pursuing, rather than just “wondering” about it repeatedly and going back and forth without much new info to work with. Basically, I might do too much self-doubt-style armchair reasoning, with too little actual empirical info.

Also, pursuing an idea more “confidently” for a bit will not only inform me about whether to continue pursuing it further, but also might result in outputs that are useful for others. So I try to sometimes switch into “just commit and focus mode” for a given time period, or until I hit a given milestone, and mostly minimise reflection on what I should prioritise during that time. But so far this has been like a grab bag of heuristics and habits I use, rather than a more precise guideline for myself.

See also When to focus and when to re-evaluate.

Things that help me with this include, and/or some scattered related thoughts, include:

• Talking to others and getting feedback, including on early-stage ideas

• I liked David and Jason’s remarks on this in their comments

• A sort-of minimum viable product and quick feedback loop approach has often seemed useful for me – something like:

  • First getting verbal feedback from a couple people on a messy, verbal description of an idea

  • Then writing up a rough draft about the idea and circulating it to a couple more people for a bit more feedback

  • Then polishing and fleshing out that draft and circulating it to a few more people for more feedback

  • Then posting publicly

  • (But only proceeding to the next step if evidence from the prior one – plus one’s own intuitions – suggested this would be worthwhile)

• Feedback has often helped me determine whether an idea is worth pursuing further, feel more comfortable/motivated with pursuing an idea further (rather than being mired in unproductive self-doubt), develop the idea, work out which angles of it are most worth pursuing, and work out how to express it more clearly

• Reminding myself that I haven’t really gathered any new info since the last time I thought “Should this really be what I spend my time on?,” so thinking about that again is unlikely to reveal new insights, and is probably just a stupid part of my psychology rather than something I’d endorse.

• I might think to myself something like “If a friend was doing this, you’d think it’s irrational, and gently advise them to just actually commit for a bit and get new info, right? So shouldn’t you do the same yourself?”

• Remembering Algorithms to Live By drawing an analogy to a failure mode in which computers continually reprioritise tasks and the reprioritisation takes up just enough processing power to mean no actual progress on any of the tasks occurs, and this can just cycle forever. And the way to get out of this is to at some point just do tasks, even without having confidence that these “should” be top priority.

• This is just my half-remembered version of that part of the book, and might be wrong somehow.

• Remembering that I’d be deeply uncertain about the “actual” value of any project I could pursue, because the world is very complicated and my ambitions (contribute to improving the long-term future) are pretty lofty. The best I can do is something that seems good in expected value but with large error bars. So the fact I feel some uncertainty and doubt provides basically no evidence that this project isn’t worth pursuing. (Though feeling an unusually large amount of uncertainty and doubt might.)

• Remembering that, if the idea ends up seeming to have not been important but there was a reasonable ex ante case that it might’ve been important, there’s a decent chance someone else would end up pursuing it if I don’t. So if I pursue it, then find out it seems to not be important, then write about what I found, that might still have the effect of causing an important project to get done, because it might cause someone else to do that important project rather than doing something similar to what I did.

Examples to somewhat illustrate the last two points:

This year, in some so-far-unpublished work, I wrote about some ideas that:

• I initially wasn’t confident about the importance of

• Seemed like they should’ve been obvious to relevant groups, but seemed not to have been discussed by them. And that generally seems like (at least) weak evidence that an idea either (a) actually isn’t important or (b) has been in essence discussed in some other form or place that I just am not familiar with.

So when I had the initial forms of these ideas and wasn’t sure how much time (if any) to spend on them, I took roughly the following approach:

I developed some thoughts on some of the ideas. Then I shared those thoughts verbally or as very rough drafts with a small set of people who seemed like they’d have decent intuitions on whether the ideas were important vs unimportant, somewhat novel vs already covered, etc.

In most cases, this early feedback indicated that it was at least plausible that the ideas were somewhat important and somewhat novel. This – combined with my independent impression that these ideas might be somewhat important and novel – seemed to provide sufficient reason to flesh those ideas out further, as well as to flesh out related ideas (which seemed like they’d probably also be important and novel if the other ideas were, and vice versa).

So I did so, then shared that slightly more widely. Then I got more positive feedback, so I bothered to invest the time to polish the writings up a bit more.

Meanwhile, when I fleshed one of the ideas out a little, it seemed like that one turned out to probably not be very important at all. So with that one, I just made sure that my write-up made it clear early on that my current view was that this idea probably didn’t matter, and I neatened up the write-up just a bit, because I still thought the write-up might be a bit useful either to:

• Explain to others why they shouldn’t bother exploring the same thing

• Make it easy for others to see if they disagreed with my reasoning for why this probably didn’t matter, because I might be wrong about that, and it might be good for others to quickly check that reasoning

Having spent time on that idea sort-of felt in hindsight silly or like a mistake. But I think I probably shouldn’t see that as having been a bad decision ex ante, given that:

• It seems plausible that, if not for my write-up, someone else would’ve eventually “wasted” time on a similar idea

• This was just one out of a set of ideas that I tried to flesh out and write up, many/most of which still (in hindsight) seem like they were worth spending time on

• So maybe it’s very roughly like I gave 60% predictions for each of 10 things, and decided that that’d mean the expected value of betting on those 10 things was good, and then 6 of those things happened, suggesting I was well-calibrated and was right to bet on those things

  • (I didn’t actually make quantitative predictions)

And some of the other ideas were in between – no strong reason to believe they were important or that they weren’t – so I just fleshed them out a bit and left it there, pending further feedback. (I also had other things to work on.)

In a reply, I referred to this related blog post of mine. Michael replied:

It’s also important to be transparent about one’s rigor and to make the negative results findable for others. The second is obvious. The first is because the dead end may not actually be a dead end but only looked that way given the particular way in which you had resolved the optimal stopping problem of investigating it (even) further.

I agree with these points, and think that they might sometimes be under-appreciated (both in and outside of EA).

To sort-of restate your points:

• I think it’s common for people to not publish explorations that turned out to seem to “not reveal anything important” (except of course that this direction of exploration might be worth skipping).

• Much has been written about this sort of issue, and there can be valid reasons for that behaviour, but sometimes it seems unfortunate.

• I think another failure mode is to provide some sort of public info of your belief that this direction of exploration seems worth skipping, but without sufficient reasoning transparency, which could make people rule this out too much/too early.

• Again, there can be valid reasons for this (if you’re sufficiently confident that it’s worth ruling out this direction and you have sufficiently high-value other things to do, it might not be worth spending time on a write-up with high reasoning transparency), but sometimes it seems unfortunate.

In the same context, I also brought up a bit of CFAR lore:

Part of this reminds me a lot of CFAR’s approach here (I can’t quite tell whether Julia Galef is interviewer, interviewee, or both):

For example, when I’ve decided to take a calculated risk, knowing that I might well fail but that it’s still worth it to try, I often find myself worrying about failure even after having made the decision to try. And I might be tempted to lie to myself and say, “Don’t worry! This is going to work!” so that I can be relaxed and motivated enough to push forward.

But instead, in those situations I like to use a framework CFAR sometimes calls “Worker-me versus CEO-me.” I remind myself that CEO-me has thought carefully about this decision, and for now I’m in worker mode, with the goal of executing CEO-me’s decision. Now is not the time to second-guess the CEO or worry about failure.

Your approach to gathering feedback and iterating on the output, making it more and more refined with every iteration but also deciding whether it’s worth another iteration, that process sounds great!

I think a lot of people aim for such a process or want to after reading your comment, but will held back from showing their first draft to their first round of reviewers because they worry the reviewers will think badly of them for addressing a topic of this particular level of perceived difficulty or relevance (maybe it’s too difficult or too irrelevant in the reviewer’s opinion), or think badly of them for a particular wording, or think badly of them because they think you should’ve anticipated a negative effect of writing about the topic and not done so (e.g., some complex acausal trade or social dynamics thing that didn’t occur to you), or just generally have diffuse fears holding them back. Such worries are probably disproportionate, but still, overcoming them will probably require particular tricks or training.

Michael replied:

I like that “Worker-me versus CEO-me” framing, and hadn’t heard of it or seen that page, so thanks for sharing that. It does seem related to what I said in the parent comment.

I share the view that it’ll be decently common for a range of disproportionate worries to hold people back from striking out into areas that seem good in expected value but very uncertain and with real counterarguments, and from sharing early-stage results from such pursuits. I also think there can be a range of good reasons to hold back from those things, and that it can be hard to tell when the worries are disproportionate!

I imagine it’d be hard (though not impossible) to generate advice on this that’s quite generally useful without being vague/littered with caveats. People will probably have to experiment to some extent, get advice from trusted people on their general approach, and continue reflecting, or something like that.

Is there something interesting here?

I often have some (for me) novel ideas, but then it turns out that whether true or false, the idea doesn’t seem to have any important implications. Conversely, I’ve dismissed ideas as unimportant, and years later someone developed them – through a lot of work I didn’t do because I thought it wasn’t important – into something that did connect to important topics in unanticipated ways. Do you have rules of thumb that help you assess early on whether a particular idea is worth pursuing?

Jason Schukraft

Yep, this also happens to me. Unfortunately, I don’t have any particular insight. Oftentimes the only way to know whether an idea is interesting is to put in the hard exploratory work. Of course, one shouldn’t be afraid to abandon an idea if it looks increasingly unpromising.

David Bernard

I mostly try to work out how excited I am by this idea and whether I could see myself still being excited in 6 months, since for me having internal motivation to work on a project is pretty important. I also try to chat about this idea with various other people and see how excited they are by it.

Michael Aird

See section Self-Consciousness.

Survival vs. exploratory mindset

I’ve heard of the distinction between survival mindset and exploratory mindset, which makes intuitive sense to me. (I don’t remember where I learned of these terms, but I tried to clarify how I use them in a comment below.) I imagine that for most novel research, exploratory mindset is the more useful one. (Or would you disagree?) If it doesn’t come naturally to you, how do you cultivate it?

By survival mindset I mean: extreme risk aversion, fear, distrust toward strangers, little collaboration, isolation, guarded interaction with others, hoarding of money and other things, seeking close bonds with family and partners, etc., but I suppose it also comes with modesty and contentment, equanimity in the face of external catastrophes, vigilance, preparedness, etc.

By exploratory mindset I mean: risk neutrality, curiosity, trust toward strangers, collaboration, outgoing social behavior, making oneself vulnerable, trusting partners and family without much need for ritual, quick reinvestment of profits, etc., but I suppose also a bit lower conscientiousness, lacking preparedness for catastrophes, gullibility, overestimating how much others trust you, etc.

Those categories have been very useful for me, but maybe they’re a lot less useful for most other people? You can just ignore that question if the distinction makes no intuitive sense this way or doesn’t quite fit your world models.

Jason Schukraft

Insofar as I understand the terms, an exploratory mindset is an absolute must. Not sure how to cultivate it, though.

David Bernard

I also haven’t heard these terms before, but from your description (which frames a survival mindset pretty negatively), an exploratory mindset comes fairly naturally to me and therefore I haven’t ever actively cultivated it. Lots of research projects fail so extreme risk aversion in particular seems like it would be bad for researchers.

Optimal hours of work per day

Have you found that a particular number of hours of concentrated work per day works best for you? By this I mean time you spend focused on your research project, excluding time spent answering emails, AMAs, and such. (If hours per day doesn’t seem like an informative unit to you, imagine I asked “hours per week” or whatever seems best to you.)

Alex Lintz

I tend to work about 4–7 hours per day including meetings and everything. Including only mentally intensive tasks I probably get around 4–5 a day. Sometimes I’m able to get more if I fall into a good rhythm with something. Looking around at estimates (Rescuetime says just ~ 3 hours per day average of productive work) it seems clear I’m hitting a pretty solid average. I still can’t shake the feeling that everyone else is doing more work. Part of this is because people claim they do much more work. I assume this is mostly exaggeration though because hours worked is used as a signal of status and being a hard worker. But still, it’s hard to shake the feeling.

David Bernard

I typically aim for 6–7 hours of deep work a day and a couple of dedicated hours for miscellaneous tasks and meetings. Since starting part-time at RP I’ve been doing 6 days a week (2 RP, 4 PhD), but before that I did 5. I find RP deep work less taxing than PhD work. 6 days a week is at the upper limit of manageable for me at the moment, so I plan to experiment with different schedules in the new year.

Jason Schukraft

I work between 4 and 8 hours a day. I don’t find any difference in my productivity within that range, though I imagine if I pushed myself to work more than 8, I would pretty quickly hit diminishing returns.

Learning a new field

I don’t know what I mean by “field,” but probably something smaller than “biology” and bigger than “how to use Pipedrive.” If you need to get up to speed on such a field for research that you’re doing, how do you approach it? Do you read textbooks (if so, linearly or more creatively?) or pay grad students to answer your questions? Does your approach vary depending on whether it’s a subfield of your field of expertise or something completely new?

Holly Elmore

I can answer [this], as I’ve been doing it for Wild Animal Welfare since I was hired in September. WAW is a new and small field, so it is relatively easy to learn the field, but there’s still so much! I started by going backwards (into the Welfare Biology movement of the 80s and 90s) and forwards (into the WAW EA orgs we know today) from Brain Tomasik, consulting the primary literature over various specific matters of fact. A great thing about WAW being such a young field (and so concentrated in EA) is that I can reach out to basically anyone who’s published on it and have a real conversation. It’s a big shortcut!

I should note that my background is in Evolutionary Biology and Ecology, so someone else might need a lot more background in those basics if they were to learn WAW.

Alex Lintz

I just do a lot of literature review. I tend to search for the big papers and meta-analyses, skim lot’s of them and try to make a map of what the key questions are and what the answers proposed by different authors are for each question (noting citations for each answer). This helps to distill the field I think and serves as something relatively easy to reference. Generally there’s a lot of restructuring that needs to happen as you learn more about a topic area and see that some questions you used were ill-posed or some papers answer somewhat different questions. In short this gets messy, but it seems like a good way to start and sometimes it works quite well for me.

Michael Aird

I don’t know if I have a great, well-chosen, or transferable method here, so I think people should pay more attention to my colleagues’ answers than mine. But FWIW, I tend to do a mixture of:

• reading Wikipedia articles

• reading journal article abstracts

• reading a small set of journal articles more thoroughly

• listening to podcasts

• listening to audiobooks

• watching videos (e.g., a Yale lecture series on game theory)

• talking to people who are already at least sort-of in my network (usually more to get a sounding board or “generalist feedback,” rather than to leverage specific expertise of theirs)

I’ve also occasionally used free online courses, e.g. the Udacity Intro to AI course. (See also What are some good online courses relevant to EA?)

Whether I take many notes depends on whether I’m just learning about a field because I think it might be useful in some way in future for me to know about that field, or because I have at least a vague idea of a project I might work on within that field (e.g., “how bad would various possible types of nuclear wars be, from a longtermist perspective?”). In the latter case, I’ll take a lot of notes as I go in Roam, beginning to structure things into relevant sub-questions, things to learn more about, etc.

Since leaving university, I haven’t really made much use of textbooks, flashcards, or reaching out to experts who aren’t already in my network. It’s not really that I actively chose to not make much use of these things (it’s just that I never actively chose to make much use of these things), and think it’s plausible that I should make more use of these things. I’ll very likely talk to a bunch of experts for my current or upcoming research projects.

David Bernard

I’m a big fan of textbooks and schedule time to read a couple of textbook chapters each week. LessWrong’s best textbooks on every subject thread is pretty good for finding them. I usually make Anki flashcards to help me remember the key facts, but I’ve recently started experimenting with Roam Research to take notes which I’m also enjoying so my “learning flow” is in flux at the moment.

Jason Schukraft

I can’t emphasize enough the value of just talking to existing experts. For me at least, it’s by far the most efficient way to get up-to-speed quickly. For that reason, I really value having a large network of diverse people I can contact with questions. I put a fair amount of effort into cultivating such a network.

Hard problems

I imagine that you’ll sometimes have to grapple with problems that are sufficiently hard that it feels like you didn’t make any tangible progress on them (or on how to approach them) for a week or more. How do you stay optimistic and motivated? How and when do you “escalate” in some fashion – say, discuss hiring a freelance expert on some other field?

Jason Schukraft

I’m fortunate that my work is almost always intrinsically interesting. So even if I don’t make progress on a problem, I continue to be motivated to work on it because the work itself is so very pleasant. That said, as I’ve emphasized above, when I’m stuck, I find it most helpful to talk to lots of people about the problem.

Alex Lintz

I have a maybe-controversial take that research (even in LT space) is motivated largely by signalling and status games. From this view the advice many gave about talking to people about it sounds good. Then you generate some excitement as you’re able to show someone else you’re smart enough to solve it, or they get excited to share what they know, etc. I think if you had a nice working group on any topic, no matter how boring, everyone would get super excited about it. In general, connecting the solution to a hard problem to social reward is probably going to work well as a motivator by this logic.

Michael Aird

I’m not actually sure if the precise problem you’re describing resonates with me. I definitely often feel very uncertain about:

• whether the goal I’m striving towards really matters at all

• even if so, whether it’s a goal worth prioritising

• whether I should prioritise it (is it my comparative advantage?)

• whether anything I produce in pursuing this goal will be of any use to anyone

But I’m not sure there have been cases where, for a week or more, I didn’t feel like I was at least progressing towards:

• having the sort of output I had planned or now planned to produce(setting aside the question of whether that output will be useful to anyone), and/or

• deciding (for good reason) to not bother trying to create that sort of output

Note that I’d count as “progress” cases where I explored some solutions/options that I thought might work/be useful for X, and all turned out to be miserable wastes of time, so I can at least rule those out and try something else next week. I’d also count cases where I learned other potentially useful things in the process of pursuing dead ends, and that knowledge seems likely to somehow benefit this or other projects.

It is often the case that my estimate of how many remaining days something will take is longer at the end of the week than it was at the beginning of the week. But this is usually coupled with me thinking that I have made some sort of progress – I just also realised that some parts will be harder than I thought, or that I should do a more thorough job than I’d planned, or something like that.

(But I feel like maybe I’m just interpreting your question differently to what you intended.)

Emotional motivators

It’s easy to be motivated on a System 2 basis by the importance of the work, but sometimes that fails to carry over to System 1 when dealing with some very removed or specific work – say, understanding some obscure proof that is relevant to AI safety along a long chain of tenuous probabilistic implications. Do you have tricks for how to stay System 1 motivated in such cases – or when do you decide that a lack of motivation may actually mean that something is wrong with the topic and you should question whether it is sufficiently important?

Jason Schukraft

When I reflect on my life as a whole, I’m happy that I’m in a career that aims to improve the world. But in terms of what gets me out of bed in the morning and excited to work, it’s almost never the impact I might have. It’s the intrinsically interesting nature of my work. I almost certainly would not be successful if I did not find my research to be so fascinating.

David Bernard

My main trick for dealing with this is to always plan my day the night before. I let System 2 Dave work out what is important and needs to be done and put blocks in the calendar for these things. When System 1 Dave is working the next day, his motivation doesn’t end up mattering so much because he can easily defer to what System 2 Dave said he should do. I don’t read too much into lack of System 1 motivation, it happens and I haven’t noticed that it is particularly correlated with how important the work is, it’s more correlated with things like how scary it is to start some new task and irrelevant things like how much sunlight I’ve been getting.

Alex Lintz

I’ve been thinking a lot recently about what I’m calling “incentive landscaping.” The basic idea is that your system 2 has a bunch of things it wants to do (e.g. have impact). Then you can shape your incentive landscape such that your system 1 is also motivated to do the highest impact things. Working for someone who shares your values is the easiest way to do this as then your employer and peers will reward you (either socially or with promotions) for doing things which are impact-oriented. This still won’t be perfectly optimized for impact but it gets you close. Then you can add in some extra motivators like a small group you meet with to talk about progress on some thing which seems badly motivated, or ask others to make your reward conditional on you completing something your system 2 thinks is important. Still early days for me on this though and I think it’s a really hard thing to get right.

Michael Aird

(Disclaimer: I’m just reporting on my own experience, and think people will vary a lot in this sort of area, so none of the following is even slightly a recommendation.)

In general:

• Personally, I seem to just find it pretty natural to spend a lot of hours per week doing work-ish things

• I tend to be naturally driven to “work hard” (without it necessarily feeling much like working) by intellectual curiosity, by a desire to produce things I’m proud of, and by a desire for positive attention (especially but not only from people whose judgement I particularly respect)

• That third desire in particular can definitely become a problem, but I try to keep a close eye on it and ensure that I’m channeling that desire towards actions I actually endorse on reflection

• I do get run down sometimes, and sometimes this has to do with too many hours per week for too many weeks in a row. But the things that seem more liable to run me down are feeling that I lack sufficient autonomy in what I do, how, and when; or feeling that what I’m doing isn’t valuable; or feeling that I’m not developing skills and knowledge I’ll use in future

• That last point means that one type of case in which I do struggle to be motivated is cases where I know I’m going to switch away from a broad area after finishing some project, and that I’m unlikely to use the skills involved in that project again.

  • In these cases, even if I know that finishing that project to a high standard would still be valuable and is worth spending time on, it can be hard for me to be internally motivated to do so, because it no longer feels like doing so would “level me up” in ways I care about.

• I seem to often become intensely focused on a general area in an ongoing way (until something switches my focus to another area), and just continually think about it, in a way that feels positive or natural or flow-like or something

• This happened for stand-up comedy, then for psychology research, then for teaching, then for EA stuff (once I learned about EA)

  • (The other points above likewise applied during each of those four “phases” of my adult life)

Luckily, the sort of work I do now:

• is very intellectually stimulating

• involves producing things I’m (at least often!) proud of

• can bring me positive attention

• allows me a sufficient degree of autonomy

• seems to me to be probably the most valuable thing I could realistically be doing at the moment (in expectation, and with vast uncertainty, of course)

• involves developing skills and knowledge I expect I might use in future

That means it’s typically been relatively easy for me to stay motivated. I feel very fortunate both to have the sort of job and “the sort of psychology” I’ve got. I think many people might, through no fault of their own, find it harder to be emotionally motivated to spend lots of hours doing valuable work, even when they know that that work would be valuable and they have the skills to do it. Unfortunately, we can’t entirely choose what drives us, when, and how.

(There’s also a scary possibility that my tendency so far to be easily motivated to work on things I think are valuable is just the product of me being relatively young and relatively new to EA and the areas I’m working in, and that that tendency will fade over time. I’d bet against that, but could be wrong.)

Typing speed

I have this pet theory that a high typing speed is important for some forms of research that involves a lot of verbal thinking (e.g., maybe not maths). The idea is that our memory is limited, so we want to take notes of our thoughts. But handwriting is slow, and typing is only mildly faster, so unless one thinks slowly or types very fast, there is a disconnect that causes continual stalling, impatience, forgotten ideas, and prevents the process from flowing. Does that make any intuitive sense to you? Do you have any tricks (e.g., dictation software)?

Jason Schukraft

No idea what my typing speed is, but it doesn’t feel particularly fast, and that doesn’t seem to handicap me. I’ve always considered myself a slow thinker, though.

David Bernard

I struggle to imagine typing speed being a binding constraint on research productivity since I’ve never found typing speed to be a problem for getting into flow, but when I just checked my wpm was 85 so maybe I’d feel different if it was slower. When I’m coding the vast majority of my time is spent thinking about how to solve the problem I’m facing, not typing the code that solves the problem. When I’m writing first drafts, I think typing speed is a bit more helpful for the reasons you mention, but again more time goes into planning the structure of what I want to say and polishing, than the first pass at writing where speed might help.

Linchuan Zhang

I think my own belief is that typing speed is probably less important than you appear to believe, but I care enough about it that I logged 53 minutes of typing practice on keybr this year (usually during moments where I’m otherwise not productive and just want to get “in flow” doing something repetitive), and I suspect I still can productively use another 3–5 hours of typing practice next year even if it trades off against deep work time (and presumably many more hours than that if it does not).

Alex Lintz

At least when I’m doing reflections or broad thinking I often circumvent this by doing a lot of voice notes with Dragon. That way I can type at the speed of thought. It’s never perfect but ~ 97% of it is readable so it’s good enough. Then if you want to actually have good notes you go through and summarize your long jumble of semi-coherent thoughts into something decent sounding. This has the side of effect of some spaced repetition learning as well!

Michael Aird

I’d be surprised if typing speed was a big factor explaining differences in how much different researchers produce, or in their ability to produce certain types of output. (But of course, that claim is pretty vague – how surprised would I be? What do I mean by “big factor?”)

But I just did a typing test, and got 92 WPM (with “medium” words, and 1 typo), which is apparently high. So perhaps I’m just taking that for granted and not recognising how a slower typing speed could’ve limited me. Hard to say.

Obvious questions

Nate Soares has an essay on “obvious advice.” Michael Aird mentioned that in many cases he just wanted to follow up on some obvious ideas. They were obvious in hindsight, but evidently they hadn’t been obvious to anyone else for years. Is there a distinct skill of “noticing the obvious ideas” or “noticing the obvious open questions”? And can it be trained or turned into a repeatable process?

Jason Schukraft

Yeah, I think there is a general skill of “noticing the obvious.” I don’t think I’m great at it, but one thing I do pretty often is reflect on the sorts of things that appear obvious now that weren’t obvious to smart people ~200 years ago.

Linchuan Zhang

I suspect that while sometimes ignoring/not noticing “obvious questions/advice” etc is coincidental unforced errors, more often than not there is some form of motivated reasoning going on behind the scenes (e.g., because this story will invalidate a hypothesis I’m wedded to, because it involves unpleasant tradeoffs, because some beliefs are lower prestige, because it makes the work I do seem less important, etc). I think training myself carefully to notice these things has been helpful, though I suspect I still miss a lot of obvious stuff.

Michael Aird

(Just my personal, current, non-expert thoughts, as always. Also, I’m not sure I’m addressing precisely the question you had in mind.)

A summary of my recommendations in this vicinity:

1. If people want to do research and want a menu of ideas/questions to work on, including ideas/questions that seem like they obviously should have a bunch of work on them but don’t yet, they could check out this central directory for open research questions, and/or an overlapping 80,000 Hours post.

2. If people want to discover “new” instances of such ideas/questions, one option might be to just try to notice ideas/variables/assumptions that seem important to some people’s beliefs, but that seem debatable and vague, have been contested by others, and/or haven’t been stated explicitly and fleshed out.

   • One way to do this might be to have a go at rigorously, precisely writing out the arguments that people seem to be acting as if they believe, in order to spot the assumptions that seem required but that those people haven’t stated/emphasised.

   • One could then try to explore those assumptions in detail, either just through more fleshed-out “armchair reasoning,” or through looking at relevant empirical evidence and academic work, or through some mixture of those things.

   • I think this is a big part of what I’ve done this year.

   • Here’s one example of a piece of my own work which came from roughly that sort of process.

I’ll add more detailed thoughts below.

---

I interpret this question as being focused on cases in which an idea/open question seems like it should’ve been obvious, or seems obvious in retrospect, yet it has been neglected so far. (Or the many cases we should assume still exist in which the idea/question is still neglected, but would – if and when finally tackled – seem obvious.)

It seems to me that there are two major types of such cases:

• Unnoticed: Cases in which the ideas/open questions haven’t even been noticed by almost anyone

• Or at least, almost anyone in the relevant community/field.

  • So I’d still say an idea counts as “unnoticed” for these purposes even if, for example, a very similar ideas has been explored thoroughly in sociology, but no one in longtermism has noticed that that idea is relevant to some longtermist issue, nor independently arrived at a similar idea.

• Noticed yet neglected: Cases in which the ideas/open questions have been noticed, but no one has really fleshed them out or tackled them much

• E.g., a fair number of longtermists have noticed that the question of how likely various types of recovery are from various types of civilizational collapse. But as far as I’m aware, there was nothing even approaching a thorough analysis of the question until some recent still-in-progress work, and there’s still room for much more work here.

  • More thoughts and notes on this here and here.

• Another example is questions related to how likely global, stable totalitarianism is; what factors could increase or decrease the odds of that; and what to do about this. Some people have highlighted such questions (including but not only in the context of advanced AI), but I’m not aware of any detailed work on them.

This is really more a continuum than a binary distinction. In almost all cases, there’s probably been someone in a relevant community who’s at least briefly noticed something relevant. But sometimes it’ll just be that something kind-of relevant has been discussed verbally a few times and then forgotten, while other times it’ll be that people have prominently highlighted pretty precisely the relevant open question, yet no one has actually worked on it. (And of course there’ll be many cases in between.)

---

For “noticed yet neglected” ideas/questions, recommendation 1 from above will be more relevant: people could find many ideas/questions of this type in this central directory for open research questions, and just get cracking on them.

That directory is like a map pointing the way to many trees that might be full of low-hanging fruit that would’ve been plucked by now in a better world. And I really would predict that a lot of EAs could do valuable work by just having a go at those questions. (I’m less confident that this is the most valuable thing lots of EAs could be doing, and each person would have to think that through for themselves, in light of their specific circumstances. See also.)

So we don’t necessarily need all EA-aligned researchers to try to cultivate a skill of “noticing the ideas that should’ve been tackled/fleshed out already” (though I’m sure some should). Some could just focus on actually exploring the ideas that have been noticed but still haven’t been tackled/fleshed out.

---

For “unnoticed” ideas/questions, recommendation 2 from above will be more relevant.

I think this dovetails somewhat with Ben Garfinkel calling for [1] more people to just try to rigorously write up more detailed versions of arguments about AI risk that often float around in sketchier or briefer form. (Obviously brevity is better than length, all else held equal, but often a few pages isn’t enough to give an idea proper treatment.)

---

There are at least two other approaches for finding “unnoticed” ideas/questions which seem to have sometimes worked for me, but which I’m less sure would often be useful for many people, and less sure I’ll describe clearly. These are:

• Trying to sketch out causal diagrams of the pathway to something (e.g., an existential catastrophe) happening

• I think that doing something like this has sometimes helped me notice there there are:

  • assumptions or steps missing in the standard/fleshed-out stories of how something might happen,

  • alternative pathways by which something could happen, and/or

  • alternative/additional outcomes that may occur

• See also

• Trying to define things precisely, and/or to precisely distinguish concepts from each other, and seeing if anything interesting falls out

• Here’s an abstract example, but one which matches various real examples that have happened for me:

  • I try to define X, but then notice that that definition would fail to cover some cases of what I’d usually think of as X, and/or that it would cover some cases of what I’d usually think of as Y (which is a distinct concept).

  • This makes me realise that X and/or Y might be able to take somewhat different forms or occur via different pathways to what was typically considered, or that there’s actually an extra requirement for X or Y to happen that was typically ignored.

• I feel like it’d be easy to misinterpret my stance here.

  • I actually think that definitions will never or almost never really be “perfect,” and I agree with the ideas in this post (see also family resemblance). And I think that many debates over definitions are largely nitpicking and wasting time.

  • But I also think that, in many case, being clearer about definitions can substantially benefit both thought and communication.

---

I should again mention that I’m only ~ 1.5 years into my research career, so maybe I’ll later change my mind about a bunch of those points, and there are probably a lot of useful things that could be said on this that I haven’t said.

[1] See the parts of the transcript after Howie asks “Do you know what it would mean for the arguments to be more sussed out?”

Tiredness, focus, etc.

We sometimes get tired or have trouble focusing. Sometimes this happens even when we’ve had enough sleep (just to get an obvious solution out of the way: sleep/napping). What are your favorite things to do when focusing seems hard or you feel tired? Do you use any particular nootropics, supplements, air quality monitor, music, or exercise routine?

Jason Schukraft

Regular exercise certainly helps. Haven’t tried anything else. Mostly I’ve just acclimated to getting work done even though I’m tired. (Not sure I would recommend that “solution,” though!)

David Bernard

My favourite thing to do is to stop working! Not all days can be good days and I became a lot happier and more productive when I stopped beating myself up for having bad days and allowed myself to take the rest of the afternoon off.

Linchuan Zhang

I haven’t figured this out yet and am keen to learn from my coworkers and others! Right now I take a lot of caffeine and I suspect if I were more careful about optimization I should be cycling drugs over a weekly basis rather than taking the same one every day (especially a drug like caffeine that has tolerance and withdrawal symptoms).

Alex Lintz

I’ve had lot’s of ongoing and serious problems with fatigue and have tried many interventions. Certainly caffeine (ideally with l-theanine) is a nice thing to have but tolerance is an issue. Right now what seems to work for me (no idea why) is a greens powder called Athletic Greens. I’m also trying pro/prebiotics which might be helping. Magnesium supplementation also might have helped. A medication I was taking was causing some problems as well and causing me to have some really intense fatigue on occasion (again, probably…). It’s super hard to isolate cause and effect in this area as there are so many potential causes. I’d say it’s worth dropping a lot of money on different supplements and interventions and seeing what helps. If you can consistently increase energy by 5–10% (something I think is definitely on the table for most people), that adds up really quickly in terms of the amount of work you can get done, happiness, etc. Ideally you’d do this by introducing one intervention at a time for 2-4 weeks each. I haven’t had patience for that and am currently just trying a few things at once, then I figure I can cut out one at a time and see what helped. Things I would loosely recommend trying (aside from exercise, sleep, etc): Prebiotics, good multivitamins, checking for food intolerances, checking if any pills you take are having adverse effects.

I do also work through tiredness sometimes and find it helpful to do some light exercise (for me, games in VR) to get back some energy. That also works as a decent gauge for whether I’ll be able to push past the tiredness. If playing 10 min of Beatsaber feels like a chore, I probably won’t be able to work.

How you rest might also be important. E.g. might need time with little input so your default mode network can do it’s thing. No idea how big of a deal this is but I’ve found going for more walks with just music (or silence) to maybe be helpful, especially in that I get more time for reflection.

I’ve also been experimenting with measuring heart rate variability using an app called Welltory. That’s been kind of interesting in terms of raising some new questions though I’m still not sure how I feel about it/how accurate it is for measuring energy levels.

I’ve bought some Performance Lab products (following a recommendation from Alex in a private conversation). They have better reviews on Vaga and are a bit cheaper than the Athletic Greens.

Michael Aird

I find that being tired makes my mind wander a lot when reading longform things (e.g., papers, posts, not things like Slack messages or emails), so when I’m tired I usually try to do things other than reading.

If I’m just a bit or moderately tired, I usually find I’m still about as able to write as normal. If I’m very tired, I’ll still often be able to write quickly, but then when I later read what I wrote I’ll feel that it was unclear, poorly structured, and more typo-strewn than usual. So when very tired, I try to avoid writing longform things (e.g., actual research outputs).

Things I find I’m still pretty able to do when tired include commenting on documents people want input on (I think I’m more able to focus on this than on regular reading because it’s more “interactive” or something), writing things like EA Forum comments, replying to emails and Slack messages and the like, doing miscellaneous admin-y tasks, and reflecting on the last week/month and planning the next. So I often do a disproportionate amount of such tasks during evenings or during days when I’m more tired than normal, and at other times do a disproportionate amount of reading and “substantive” writing.

Also, I’m fortunate enough to have flexible hours. So sometimes I just work less on days when I’m tired (perhaps spending more time with my wife), and then make up for it on other days.