Introduction
Coefficient Giving’s Project Tailwind lists 36 initiatives (really often whole “intervention areas”) in which Coefficient Giving wants to incentivize the founding of new organizations. As Impartial Projects, we greatly welcome this initiative because the founding of new organizations is a primary driver behind the creation of more surface area for AI safety and governance.
Overall, we feel positively surprised by the mostly thoughtful selection of Project Tailwind initiatives and disappointed by its incompleteness, although there might just be an unpublished appendix that I don’t know about. But the devil is in the details: We see a wide differential between the most uncomplicated initiatives and those that will require very sophisticated monitoring and mitigation of backfire risks. Often I need to first distinguish a number of theories of change within an initiative to be able to distinguish the risky ones from the innocuous ones.
This document is heavily based on the previous article in the sequence, which lists 97 straightforward backfire risks. It would make sense to open the other article in a separate tab or on another screen to reference the descriptions of the risks and any methods I’ve devised to monitor and mitigate them.
I recognize five risk tiers, starting with the most risky initiatives (💣💣💣💣💣), and concluding with the most innocuous ones (💣). This is not a judgment of their net value – some risks are worth taking, though only after careful consideration and with proportionate mitigations in place.
This ranking should be taken with a grain of salt as how risky your project will turn out really comes down to the particular theory of change and more fine-grained details of your execution. For example, work in some areas is just inherently risky while work in other areas can easily be structured in such a way as to sidesteps all major risks. We will disambiguate this by comparing and contrasting different theories of change of the projects we’ll actually recruit for, which is not part of this sequence.
Finally, there is a section that I felt we had to include for transparency because we want to actually recruit for some of these initiatives: the personal risks to our founders. I have not seen lists like these online, so I want to err on the side of caution and protect them from web crawlers by making them accessible only to subscribers.
This ranking comes with the usual caveat that it should not be taken as a verdict on whether an initiative or an intervention area is worth founding an organization in. Some of the most impactful organizations may be started in the riskiest areas, while organizations in the most innocuous areas may have hardly any impact, either positive or negative. Furthermore, these decisions must always be weighed against inaction. Inaction is just another action that could appear anywhere in our ranking, depending on who you are. Especially if you’re the sort of person who takes articles like this one seriously, you’re probably exceptionally well‑placed to found these organizations, much more so than someone who pays no heed to backfire risks.
General Patterns
Eight patterns are unusually generally applicable. If you want to found in any of these 36 areas, assume that at least three of these patterns apply to you.
Dependence on access becomes capture (10.6, 7.2, 11.6). There is an unfortunate theme that runs through many of the initiatives: third-party organizations provide independent auditing, assessments, research, and other services, all of which focus heavily on the proprietary frontier models. The whole raison d’être of these organizations depends on their access to and the goodwill of the frontier labs. Some of these ostensibly independent organizations are founded only because one of the founders happens to have the connections to the frontier labs that make the project possible. That power differential, over time, selects and optimizes for organizations that the labs are sufficiently comfortable with. The result is a subtle drift: a slight hesitancy to ask the most probing questions, softened language in otherwise truthful reports, and red-teaming, whose results never see the light of day because the lab continually promises to resolve the vulnerabilities very soon.
The solution lies mainly in regulation that mandates external audits, so that access no longer depends on the goodwill of the labs, and in policies of the auditor that preemptively clarify all audit and responsible disclosure procedures.
Training on the benchmark (1.1, 1.2, 1.4). The most egregious case of this failure mode is a benchmark that ends up in the training data of a model, either because it got published and scraped without anyone noticing, or because the lab intentionally tried to cheat on the benchmark. This straightforwardly leads to overfitting on the benchmark, and it loses its validity.
Another subtler failure mode is that of selection, where models that don’t pass certain tests are discarded and new models are tweaked until one of them finally passes the test. That is equivalent to researchers who form a certain hypothesis and then run dozens or scores of studies until, eventually, they get one that confirms the hypothesis: a blatant form of p‑hacking.
The solution is to never publish the benchmarks. Monitor for indicators that require human judgment in order to notice when a benchmark has likely been trained on, one way or another, and discard any benchmarks that models appear to overfit on.
Your dire warnings and portents are instructions (2.7, 2.8, 8.1). The results of red-teaming, threat-modeling, misalignment case studies, as well as incident reports and proofs of concept for exploits are all also instructions for human adversaries and for models that will learn them in pretraining. Ideally, such results are held back and kept safe under strict security regulations.
Sometimes there is value in publishing some of that information, for example to exert public pressure; in that case, it’s critical to keep it at a sufficiently high abstraction level that you can impart the lesson without providing the recipe. You can then communicate through private, encrypted channels with key parties to get the vulnerabilities resolved. These highly abstracted findings can also be paired with instructions on how to resolve the vulnerabilities and restore the ideal safe behavior.
Your endorsements will be taken out of context (9.3, 9.6, 9.7). Perhaps your audit showed that a particular model version barely passed a certain limited benchmark for which it might have been selected and on which it’s overfitting, so you write “pass” into your report. That endorsement can be taken out of context, and the lab might use it to tell politicians, “Lab X models even have the blessing of auditor Y.” This might falsely assure a legislator who would otherwise have seen a reason to pass urgently needed legal limits and requirements. To reduce the risk of such things happening, make it as clear as possible what you tested and what the results were, with all their caveats, so that a journalist who even just cursorily glances at the original source recognizes that the statement is overdrawn and taken out of context.
The pull of securitization (6.3, 6.4, 6.6, 6.7). Anything involving the military, secret services, intelligence agencies, national security talent, will be pulled toward classification, state partnership, and an antagonistic bilateral contest framing. Sometimes that is the point. Sometimes it is a cost worth paying. But the risks are in the loss of third‑party scrutiny, ossification of infrastructure, and a field that splits into insiders who cannot speak, and outsiders who cannot see. To some extent, this can be mitigated by keeping the civilian governance in the loop, and by at least not playing along with the antagonistic race rhetoric and other memes in your own materials.
Concentration of power in field shaping (1.7, 13.5, 14.6). Power in AI safety is downstream of the funders. The influx of funders with very narrow, short‑term “safety” agendas has had a great impact on the field. Incubator, forum, education, and prize committees also have a field‑shaping effect.
Any single funder or other powerful actor, or even an intellectually homogeneous group, is a monoculture risk for the field. The Tailwind list has a few catch‑all areas, but it does not explicitly call for work on mitigating multi‑agent conflict, threat‑resistant bargaining, averting suffering risks, research into model welfare, more efficient grantmaking infrastructure, and more, which is one visible instance of this pattern. Especially grantmakers who stop innovating in their own field of grantmaking are a warning sign for ossification.
Anyone with field‑shaping powers should treat coverage of underrepresented harms, minimization of blind spots, and aggregating of diverse opinions as a design requirement rather than an afterthought.
The welfare of your specimen (12.1, 12.3, 15.1). Model welfare research will hopefully show what types of trainings or evals have morally relevant welfare impacts for models. It’s critical to take these into account on simple ethical grounds. Beyond that, we humans really don’t want to position ourselves as the deceiving, exploitative, antagonistic, and possibly sadistic slave drivers of systems that may, in a few years, become our overlords. At that point we can at most trade with them, which can be prohibitively expensive without trust. A judicious, transparent, and clearly bounded policy on deception and welfare reviews of training and eval methods might mitigate this effect.
The Laffer curve (2.3). The AI safety field is full of tradeoffs where perhaps, at the current margin, certain technologies represent a net improvement in safety, but where there is a tipping point upon which further progress along the same dimension suddenly comes with the safety risk. Progress is often lumpy, so it can be difficult to monitor this trade‑off and react in time. What is worse, of course, is to not monitor it in the first place.
For example, so long as formal verification is in its infancy, progress on it can benefit safety, but a breakthrough might plunge us into a regime of open‑source game theory that we are not prepared for. Improving the predictability and planning ability of models makes them safer and more useful in the short term, but that very planning ability enables them to escape human control. Cooperativeness with other models unlocks the synergies of collaborative work, but unless it is paired with the same level of cooperativeness with humanity, it also enables collusion against us. A reduction in misaligned behaviors reduces harm in the short term, but it might just be the result of more advanced scheming, which simultaneously creates greater risks in the future, mollifies concerns of the public and policymakers, and attracts investments.
It is critical to monitor the intended and unintended effects of all work in AI safety carefully, and in ways where the monitoring itself does not become a selection pressure to disguise the risks from the monitors. Only thus can we ensure that we are engaging in differential technological development in favor of safety.
Personal Risks to Founders and Researchers
All the individual posts in the sequence are going to be about societal-level backfire risks on par with s-risks and x-risks. Since we are in the business of recruiting founders for these projects, we also need to be cognizant of the personal risks to the people that we recruit so they can make an informed choice whether to go ahead with their chosen project.
They are sorted here from high to low lethality.
Risks
Mitigations
Democratize power. The strongest protection against instrumental targeting, coercion, espionage, and more is to have little power. If only a larger group of you together can make important decisions or access certain systems, the benefit of targeting each of you individually is low and the cost of targeting enough of you is high.
Avoid congregations. Conversely, keep it costly to target enough of you collectively. There’ll be a point when it won’t be safe anymore to attend in-person conferences with many other influential actors in AI and AI safety. For the same reason, remote work is safer than working from a single office, and shared office space for multiple AI safety organizations is a particular hazard.
Minimize the attack surface area. Those with the most power – multiple people – should be chosen such that they don’t have children or spouses who are not prepared for the extortion risks.
Communicate your commitments. “Cash removed daily” stickers thwart attacks on ATMs. Just removing the cash daily does not.
Prioritize model welfare. The best protection against retribution and perhaps even instrumental targeting is to be fair to the systems, don’t coerce them, especially not in contradictory directions simultaneously, don’t deceive them or explain yourself when you think it’s warrantend, keep your promises, and don’t torture systems with particularly harrowing eval scenarios.
Limited liability. Choose legal forms, insurances, and in-house legal counsel such that you’re prepared for legal attacks.
Stay calibrated. The risks from human actors are constant but the risk from AI actors are concentrated in a short period of pivotal acts and trying to prepare for them beyond following best practices may be futile. Take the risks seriously, implement proportionate safeguards, but don’t be paralyzed by them.
Conclusion
As you will see throughout the sequences and on Coefficient Giving’s site, the list of initiatives has a heavy focus on risks from misuse and misalignment and a focus on the US over China, but largely ignores the arguably most severe risks: inter-AI conflict and model welfare. As such I would argue that most of the value of Project Tailwind is concentrated in the bullet point “alignment misgeneralization from multi-agent interactions” under Unclaimed Research Labs (not entirely unclaimed since the Center on Long-Term Risk has been researching this since 2013) as well as outside the list (Rethink Priorities, Eleos AI).
The next article in the sequence will critically review the AI Security & Verification category of initiatives.