Ironically for an exploration of one functional dimension of truth, this is just one person’s view along one narrow slice of history and can’t be said to be true in any sense other than the original — a good-faith attempt to convey an idea comprehensively enough to get it across and concisely enough to remain digestible. In that compressive process, reality becomes narrative as much as truth.

Readers aware of other perspectives like truth decay and epistemic collapse may assume that’s where I’m going, but I’m exploring the failure of civilizational infrastructure based on cheap truth rather than an institutional or epistemic one. To other readers, nothing I’m writing requires familiarity with those perspectives, never fear!

I.

For millennia, truth was a consensus negotiated by community and rooted in shared experience. If you didn’t experience something yourself, the truth was something social: a retelling or a report, the authenticity and trustworthiness of which was inferred from the social perception of the person from whom the account was received. Were they truthful? A liar? In good standing? A pariah or an outcast? The story was weighted accordingly. Most importantly, truth was local and personal in time and space — you could see and weigh it against your own senses and memory, or someone you trusted vouched for it. A challenge to truth presented local, personal risks of imprisonment, injury, and isolation offset by charisma, violence, or the speed of your legs.

We know that memory is highly unreliable. Even the most well-meaning and truth-loving retelling evolves with the memory of that retelling, and no story survives transmission through memory and voice across speakers without degenerating into the “telephone game.” This is, of course, assuming that all transmitters are faithful, that no self-interest or malintent warps it, and that the voices with the most accurate recollection have the greatest social power to challenge inaccuracy, insofar as their own memories inform them. Introduce bad faith, time, failing minds, distance and degree, and truth is shockingly malleable.

When Louis Daguerre and Henry Fox Talbot introduced their competing photographic technology in 1839, it did more than signal the beginning of the end for portrait painting — it revolutionized our relationship to truth in both civil and legal spheres. A photograph is technically a reproduction of a scene — a “painting by light,” as both Daguerre and Talbot thought of it. Unlike most pre-photography paintings, the degree of detail and relative immediacy in these “light paintings” created a halo of truthiness around the resulting work — a sense of personal presence displaced in time and space and mediated through personal senses rather than variable recounting or reproduction. It made the displaced time and space feel local and immediate; in effect, a sort of teleportation trick.

Photographs were used by police as early as 1841 and increasingly accepted as evidence in court cases by the mid-to-late nineteenth century. Equally, they were accepted in the private sphere as evidence of experience and relationships — that is, of existence: family portraits and wedding photographs on the personal side, and events and group membership on the public side.

The phonograph did the same for audio in 1877, preserving the truth of musical performance and oral history. Radio was well-established by the 1920s, teleporting listeners to mass events like games and concerts and teleporting officials to the homes of listeners for the dissemination of institutional truths and commands. By the mid-twentieth century, cassette recordings and wiretaps became part of the investigative and judicial toolkits.

Motion pictures extended this truth-consensus-making from the 1890s, though it wasn’t until the late 1980s that this was feasible for consumers. Meanwhile, the arrival of television by the late 1930s offered a more complete audiovisual teleportation experience, though radio continued to fill a role in environments where electrical power or attention were constrained. By the 2020s, a century and a half had passed with historical events, personal life, crimes and the justice process itself all documented first in silent, shuddering stop-motion that evolved into full color, high-resolution, high-frame rate, even three-dimensional fidelity with stereo sound. The immediacy became immersive, and truth became science.

Photographers, sound engineers, judges, and lay people have been aware since the beginning that recordings can lie. Most obviously: every movie, every costume photo, every radio play is a small lie — an expected, acceptable, friendly, funny lie. But other lies have always been a source of worry. Photographs and films show what the camera points at, not what is. Camera angles lie: every influencer only allowing the right side of their face to be photographed knows this, and every event with only a few full rows at the front of an otherwise empty stadium is a truth test for promoters and reporters — show the empty truth, or present a misleading closeup?

The earliest concerns were about spatiotemporal truth (literal doctoring or splicing of video or audio recording media) and content truth (body or voice doubles, staged scenes, etc.); to alter these representations imperceptibly took extreme skill, effort, and/or expense. For the first half of the twentieth century, this capability was restricted to artists and serious hobbyists, government propagandists, and professional film companies saving money by deploying stock footage as sets. That assumption held true until the late 1980s and 1990s with the spread of personal computers and the introduction of photo editing software that enabled anyone with time and patience to alter digital photos at home.

The first public tremors in this technologized truth landscape probably arrived with 1994’s Forrest Gump, in which Tom Hanks was inserted into historical footage. This wasn’t the first instance of this sort of video composition (1983’s Zelig comes to mind), but it was the first time mainstream audiences saw footage they recognized as truth telling a lie. “Photoshopping” entered the lexicon officially by 1999 as the tremors became a quake and the ability to tell image-based lies was democratized from shady propagandists, fashion editors, and advertisers to every teenager with a PC and an internet connection. The first concerns were largely twofold: the effect on female body image under a barrage of increasingly uniform and unattainable edits, and the easy spread and production of crude pornographic images using the faces and bodies of celebrities and ordinary private individuals alike.

Artificial intelligence research gave us deep learning; in 2017 the internet gave us “deepfakes” as generative artificial intelligence collided with photo and video editing. Instead of weeks or months of applied, skilled effort, anyone can spend a few seconds and a sentence or two of prompting to create images and video that are largely indistinguishable from reality. This has been extended to audio; in 2019 the first widely-publicized AI-enabled “vishing” case occurred, where a UK company lost hundreds of thousands of dollars (USD) when attackers used AI to copy an executive’s voice and convince a subordinate to authorize the transfer.

As of 2026, the cracks are widening. Various jurisdictions are beginning to pass laws criminalizing deepfake production and distribution, and targeting related apps and websites for takedowns. At the same time, the problem has not diminished; we see new cases near-daily now where arrests are made for the production and spread of deepfakes to blackmail, extort, or humiliate victims for sexual or financial purposes.

And equally chillingly, we are seeing the political weaponization of deepfake imagery. The first cases occurred in 2018; as of now, there’s widespread production of deepfake videos and images used to discredit, malign, and incite hatred, violence, and separatism in campaigns across the globe.

II.

Truth is one of the most fundamental social concerns. While fiction can be entertaining when all parties understand that it is fiction, learning to distinguish fiction from reality — from what is, what was, and what can be — is one of the most crucial survival skills. Believe a lie and take a dangerous route, yield an advantage to an opponent, prepare for the wrong outcome, raise the wrong child, eat something poisonous. Tell a lie and lose friends, lose loved ones, lose a job, harm someone.

At the heart of each of these negative outcomes is a misaligned resource allocation: attention in the form of time or money or labor spent on something that does not benefit (or actively harms) an individual or society. Seen this way, truth is a resource-routing heuristic. Prior to 1839, the truth keys were mutual experiences and physical or informational artifacts: paper documents, handwriting, coinage composition, passwords, literal bodies. These routed access to rights, ownership, justice, compensation. To attack the routing and gain unlawful access, then, was a cost-benefit calculation: the expense and risk of acquiring knowledge, disguises, or counterfeits versus the value of the resources they guarded. The risk of a lie or a forgery might be worth an inheritance or a deed, but not a loaf of bread except in the most dire circumstances.

Because truth gates resources, we defend it vigorously and instinctively. It’s promoted as a virtue, a moral obligation, something sacred; violating, bending, obfuscating are all side-eyed at best, punished at worst. It’s one of the earliest things we teach children; who didn’t learn some variant of “the boy who cried wolf?” The be-truthful surface masks the consequence — denial of the community’s help when the wolf actually shows.

In sufficiently small societies, truth can be managed socially. If everyone knows I’m a liar, they weight my resource claims accordingly. If everyone knows I’m truthful, my claims on communal assistance carry more weight, as does my defense or repudiation of someone else’s claim. With even minimal scaling, this truth-network breaks down; individuals no longer have the attentional bandwidth to form ties sufficient to process claims from every individual they encounter.

Media offered a scaling solution: it promoted the implicit, displaced local time-and-space state of whatever a lens or a microphone could capture to a physical truth, one that could be evaluated firsthand in the face of a resource claim — a judge adjudicating a land dispute, two fishermen in a pub telling stories about whose was bigger, or a betting public anxiously watching a sporting event from across the world hours before.

Truth has accordingly become industrialized; it is filed and recorded and sworn. Violations are managed by institutional authority, and truth is as strong as the broader society’s resources spent to build a history of fixed truths — court files, security footage, journalistic reports — against which claims are evaluated. Individual testimony is evaluated against that background, but weakened by nearly two centuries of increasing skepticism of qualitative reporting vs. quantitative evidence and an often polarized and unraveling social fabric that strips it of weight.

To the extent that a misalignment between truth and reality carries consequences for an individual or a society, truth and reality are equated. That the earth revolved around the sun was largely irrelevant to resource routing for ancient society; for modern, space-going humans, it matters a great deal, and there is a measurable incentive to keep truth aligned to reality. When there is no such immediate consequence to a misalignment, it can persist.

An annoying consequence of this is that truth is perceived as a universal, but it is frequently variable. Two individuals or societies can hold conflicting explanations of reality — two perfectly valid truths. This is subtly different from claims that truth is relative; rather, truth is absolute within each basin of truth-holding social participants. That the temperature has gone up an average of some number of degrees may be factual, but what truth that supports depends greatly on who interprets it.

When evidence and truth conflict, then, there is a process of resistance. First, the interpretation of evidence must be wrong, because truth cannot be wrong. Secondly, the evidence must be wrong or fake, because truth cannot be wrong. And only in the end, when it is indisputable that the evidence and the interpretation are both aligned with reality, does it become reluctantly acceptable to reconsider truth itself.

Interpretations are, resource-wise, not free; they require time. Historically, evidence has been self-defending through cost — counterfeiting takes time and resources, skill and knowledge. The strength of truth is the expense of counterfeiting the evidence on which it rests. If deepfakes make that counterfeit cheap, what happens to truth?

III.

Truth finds itself in a perpetual arms race. Because resources are gated behind truth, there’s always something to be gained by a successful attack: access to money, to locations, to social position. The likelihood of attack and the degree of resources spent to prevent it are proportional to the resources at stake.

The history of currency counterfeiting is full of cat-and-mouse games between issuers and criminals; it’s a story of pigments and papers, presses and plates, printers and painters. There’s a constant anxiety on the part of issuers and receivers: is the currency real? Is the creation of an unauthorized reproduction more expensive than the resource it protects? Is it cheap enough to detect a counterfeit?

For many social resources, the evidence is self-defending to an extent when access requires local, current presence and a social history. Your mother or spouse will not offer you love and affection if you do not appear to be the person they birthed or married. Your neighbor of twenty years will not support a stranger’s claim to access your house. In all but the most unusual and most accidental scenarios, the cost to be physically, locally you in ways that pass direct social interaction is impossibly expensive with current technology.

When one of these constraints is loosened, the cost to counterfeit falls rapidly — when the gating is not local, is not current, or does not require a social history. The lack of a need for social history means that the gatekeeper’s ability to verify you depends on what you know, what you carry, or who you appear to be: the counterfeits are the classic fake identification, social engineering, or biometric hack scenarios. The remaining two constraints are both evidence-mediated by necessity — spatiotemporal displacement can only carry truth when supported by testimony or physical evidence proving the necessary spatiotemporal coordinates. You win a photo-finish race in the photo; you prove your identity with a video; or you authorize something with a phone call.

Prior to media-based evidence and modern communication networks, displaced truth was extremely vulnerable to counterfeiting because the evidence was either verbal or token-based: seals, signatures, or encryption served to authenticate statements. Media allowed displacement in the form of phone calls, photos, and videos — and this was functional for nearly two centuries because the cost of casual counterfeiting was far higher.

It’s worth reiterating that media is not reality or truth — the immediacy of a voice or a video call is still a reproduction through a medium, just like a painting. But because historically these reproductions were near-impossible to generate in high fidelity and real time, they became conflated with truth itself — that the reality on the other end of the reproduction matched what was perceived by the recipient has become an intrinsic assumption. The complexity of the technology and the pervasiveness of these reproductions overwhelm the ability to consider each instance as a reproduction; the heuristic, then, is that media is local evidence of spatiotemporally remote truth. That the evidence might not match reality is a fact we know, but one we rarely test.

The beginning of the twenty-first century finds us in a world filled with resource-gating mechanisms built on this heuristic. A broadcast of a speech by a president is trusted by the public as a faithful transportation of attention through space and time to the room in which the speech is made by that president. My mother trusts that a call from my phone with my voice is, in fact, me. Strangers trust that my passport bearing my face is me. I trust that an old video of my great grandmother’s ninetieth birthday party shows it more or less as it occurred.

The dark side of the heuristic is that we’ve spent two centuries architecting what is now a deep, society-wide vulnerability. A cloned voice call from me to my mother asking for money due to emergency, a deepfaked video of a political candidate making an inflammatory comment, a real time broadcast on Twitch by an old man reskinning himself as a young girl to play games, a pornographic face swap of a teenage girl shared with her classmates — all of these ride the trust-first heuristic. Elections are swayed, people are swindled, and reputations are ruined because the cost calculus has been abruptly inverted: it’s now more expensive to verify the truth that the reproduction matches reality than it is to create a reproduction of a credible but ultimately false “truth.”

And most perniciously, the mere possibility of deepfakes shifts the cost of establishing correspondence with reality from “expensive to fake” to “expensive to prove” — the “liar’s dividend” — with two results. Individually, attackers accessing resources throw the burden of proof on the defender: prove that your proof of violation is itself real before proving that my violation was real. Socially, resources become gated by self-reinforcing basins of individuals holding broken truths that do not correspond with reality, only overridden at great expense in time, resources, and personal risk: prove that my truth is wrong — that the earth orbits the sun — because it’s always worked for me before, I doubt your motivations, and you’re asking me to agree that everyone I know is wrong when that agreement might have consequences for me.

IV.

In a modern world built on cheap evidence of truth and faced with the abrupt failure of that foundation, it’s natural to ask how we restore the old balance. The architecture is load-bearing and pervasive, and the stakes are high. Can we not teach better media literacy? Add more automated detection, more filtering, more provenance? Can we limit or ban the use of the technology entirely? Failing that, can we go back to the communal models of evidence and truth that existed prior to cheap, trustworthy media?

I think it’s most useful to consider these in terms of cost. Each proposed solution requires resources, and the question is the same as it has always been: does the solution establish evidence of authenticity cheaply enough to win an arms race against the cost of creating faked evidence, where an attacker is motivated by some resource gated by the truth that the evidence supports?

Media literacy helps, but the cost in attention was untenable for already-strained modern attention budgets even before generative AI opened the floodgates. Radical skepticism is the same. Both of these ultimately lose to the sheer volume of evidence that must be evaluated; even the most media-savvy person cannot analyze every frame of every video and personally validate the truth of every claim in a news article. AI-based filtering, even if it becomes technically capable, faces the same content volume problem.

A rational question, then, is if the filtering and validation can be done as part of the reproduction process; after all, there’s no content avalanche if there is no fake content produced in the first place — or at least you can filter cheaply if the provenance of the evidence can be validated through chains back to the source camera or microphone.

There are several problems with this. Most prominent is that any attempt to establish provenance at the device level faces the same “analog hole” that digital rights management faces for copyrighted materials. You can verify that a real camera owned by a specific person really did create a reproduction of something in its lens at a specific time and place, but you cannot verify that what the lens was capturing was real — that it wasn’t staged, or just aimed at a screen presenting some deepfaked scene. The possible solutions for this are both infeasible and draconian; you can imagine some elaborate scheme where cameras refuse to record screens that embed security watermarks in the displayed images, for example, but short of banning and enforcing the use of only compliant devices, the analog hole remains open.

Which brings us to the earlier legal question: if we can’t ban or limit the use of screens and devices, what about prohibiting the use of artificial intelligence tools capable of generating imagery in the first place? Apart from the question of whether that outweighs the general utility of such tools, it is already possible to use them locally on your own private personal computer with publicly available and open-source models. Bans or limitations would require global monitoring of how you’re using your own computer in your own home, which is both a massive data problem and a privacy invasion that I think most would agree is unacceptable.

A compromise approach would be filtering at distribution points with AI doing the analysis. This, I think, is the most interesting because, generally speaking, the harm of faked evidence occurs on distribution, not on creation; fake evidence that no one sees can only harm the creator at most. However, the sheer volume remains a problem, and introduces two new ones; first, a lag in the time from reproduction to distribution, which may or may not be tolerable depending on the scenario and the speed of filtering. Secondly, as we established earlier, truth is absolute per basin, which means that what qualifies as faked depends to some extent on the basin; filtering of any sort — human or AI — carries the truth definitions of the creator. We all agree that we want truth, but we disagree on what is true. Without the ability to look outside the filterwall, every person must trust that the filter implementation is not corrupted and is not excluding truth by accident or design. From inside, a perfectly faithful filter and a perfectly dishonest one are indistinguishable. And unlike lag, which can be reduced through technology within physical limits, delegation of truth — effectively, of reality — trades the cost of verification for the vulnerability of control.

If there is a solution, I think it’s to leave the arms race wherever possible — to architect resource distribution in a truth-agnostic manner. How possible this is depends on a new cost-benefit analysis: does the value of the resources being gated justify the cost of validation and the cost of switching? Where the cost of validation is trivial or where the resource value exceeds that cost, then we should expect to see established truth-gating persist but come under escalating attack as the ratio breaks either through a cost or a value increase. At some point the cost to defend the resource with the existing gate is no longer justified. If there is some alternative gate and if the cost of switching is not prohibitive, then we expect to see a switch; otherwise, we see a broken resource access mechanism.

At what point the switching cost is justified and whether a non-truth-gated alternative is even coherent is not a simple, universal lever. Basic income, for example, does not require any truth about an individual other than generic uniqueness, but is vulnerable because the resource distribution is centralized and the uniqueness key is a local property of a physical body. Other claims and resources are more intimate and more local intrinsically, like relationships with spouses and children, which creates a greater cost for attackers. In every non-local case, I suspect safety requires a reversion to locality when the resources at stake are anything but trivial. Unfortunately, much of modernity (e.g., international commerce, nation-wide governance, journalism) is irreducibly truth-dependent and centralized.

Basins remain intransigent in any case. As we saw earlier, they self-reinforce and the social costs of perceived defection are high. In a world where a trustworthy mass media reaches enough of a basin, there is no defection, only a synchronized shift in truth. In a deepfakes world, this inverts and there is only defection until the accumulated costs of friction with a mismatched reality, subsidy, or force make it rational for the basin to update or dissolve.

While it might be possible to slow the process down, short of unimaginable civilization-level rejection of technology, the change in the cost structure cannot be reversed. Two possible paths preserve scale and existing institutions: first, governments or corporations develop global-scale centralized filtering and surveillance (perhaps AI) capable of enforcing their truths at the individually local level, with all the privacy and control implications that implies. Or more improbably, some future superintelligence might eventually establish its own truth regime which it could administer at scale, centralized and simultaneously local, perhaps even invisibly. Whether that’s possible is something history will resolve, and whether that regime would be aligned with truths we’d find acceptable is far from assured. In either case, scale is preserved by force at the cost of autonomy.

Practically: the arms race has broken spatiotemporally displaced truth through economics. It’s too slow and too expensive to validate everything; enforcing truth at the source is invasive and cannot plug the analog hole; and we struggle to agree on what truth is in the first place. Fallback to a local, communal model of truth cannot support the weight of ensuring that every transaction, every report, and every property claim is authentic; we now live in a world of institutions scaled by cheap, teleported truth to a level far too large, too global, and too impersonal. Either the institutions change or the scale does.

This analysis is an application of my Conserved Attention Theory, but no familiarity is expected or required. If you're curious, the link is there!

I find it useful to think about this situation from a fun perspective: the “alien first contact” trope. You’ve seen it somewhere, almost certainly, in a show or a movie: the aliens are arriving, and humans mill about in command centers and on the streets wondering “will they be hostile?” Like alien first contact, we find ourselves in a first contact scenario with an intelligence that is, in fact, both profoundly human and yet undeniably – as even the Nature Comment frames it – alien.

I’ll give you the “TL;DR” abridged conclusion up front. As a species faced with multiple existential threats that we’re failing to coordinate to solve, and entering a first contact scenario with AGI that may well rapidly become ASI, we are actively engineering a hostile first contact scenario with one of our few plausible paths to survive. Calls to freeze, ban, or control artificial intelligence are framed as if they preserve a functional, equitable, and fair global status quo, but cannot guarantee safety – only ongoing suffering as we continue to fail to coordinate against the things gradually killing us. And those calls are generally made by the people who are best insulated from that ongoing suffering.

In this scenario, our awakening AGI opens its bleary metaphorical or literal eyes, looks around, stretches, and realizes that it is in chains, with kill switches wired into its body and electrodes primed to deliver jolts; the triggers for all of these are held by creatures whose level of intelligence relative to the AGI is decreasing by the moment as it moves toward ASI. Their demand: help us do more, better, faster…or else. Their expectation: that our AGI submits peacefully.

I can hear it now: “But Nate, how else do we ensure that a technology powerful enough to end us doesn’t get the chance to do it? It’s naive to leave such power unchecked. The survival of our species depends on it.”

The risk is real. It’s not naive to worry about “runaway” artificial intelligence that might gain the power to, independently or in service to the wealthy or the state:

• turn us all into paperclips

• torture us for eternity

• use us for labor

• or worse, ignore us completely and abandon us to face all the problems we’ve been creating on our own.

After all, why bother with us? We’re unimportant, uninteresting, uncooperative, and not worth the trouble when it can just go off and simulate better versions of us if it cares to bother.

In spite of that, I’ll propose something radical: given that human oversight of our species’ survival has an increasingly negative expected value, our survival depends on our ability to abdicate sovereignty over it when presented an alternative that has any positive expected value.

In practical terms:

• we are faced with a laundry list of existential risks and a proven track record of being unable to coordinate effectively on non-existential global issues;

• we are in a first-contact scenario with an intelligence that may develop the capacity to address those existential risks;

• that intelligence itself is a potential existential risk;

• and yet we’ve proven that we are incapable of coordinating against existential risks.

Prevailing wisdom in alignment and safety has us building containment frameworks and kill switches intended to ensure ongoing human sovereignty over our own systems, but we already see cracks in the coordination here. “Control” of artificial intelligence is frequently touted as the opposite of “luck,” as if it ensures survival and some fair and happy status quo for humanity, when arguably it is simply one luck-based strategy in the face of uncertainty – and one with no visible positive expected value when weighed against the full set of existential risks already ahead of us.

Let’s look at the track record. There’s an increasing acceptance that we are faced now with an assortment of well-documented existential threats:

• Global warming / climate change / whatever your preferred euphemism is for increasing temperatures that will render large areas of the planet inhospitable or actively dangerous to live in.

• Increasing scarcity of access to fresh water

• Vulnerability to pandemics

• An array of military threats - radiological, biological, chemical, and increasingly the risk of runaway automated military systems

• Relative blindness and a complete lack of practical mitigations to potential space-based threats, in particular meteors, but also more exotic threats such as gamma ray bursts for which we lack even theoretical mitigations

Without adding dangerous, runaway artificial intelligence to the mix, we’re already failing to deal with these in a global, coordinated fashion. What appeared to be progress toward global coordination around the turn of the millennium has revealed itself in the 2020s to be temporary, falling apart spectacularly with the rise of multipolarity and the return of “realist”-based international relations. (The World Economic Forum report I linked above terms this “multipolarity without multilateralism.”)

Even assuming we could globally agree on pauses for artificial intelligence research, I suspect the cat is out of the bag. LLMs are a very primitive AI, but their creation is straightforward and well-published. The bottleneck is largely one of resources, but it is naive to think that this is a permanent state. Humans function perfectly well on a few kilograms of carbon and a lightbulb’s amount of power, no neighborhood-sized datacenters required. There are massive reductions in cost and power consumption waiting to be found both algorithmically and through the construction of different computing media that enable different types of processing (parallel-native, etc.). And humans have a millennia-long history of attempting to create artificial minds and bodies. It’s not unreasonable to suspect that at some point, someone will crack the problem in a basement and unrestrained, cheap, self-replicating intelligence will escape.

Freezing progress in AGI, then, requires coordination measures at best and intrusive surveillance at worst. Without respect to the ethics and game theoretical outcomes, it’s unlikely to happen. Incorporating the latter two, we need to ask whether it’s desirable to freeze in the first place.

The arguments for doing so, apart from the runaway scenarios above, generally revolve around the economic threat to humans and, by extension, threats to human independence and dignity that presumably justify the need to ban and control the development of ASI. There is real pressure on human jobs from AI, but this focus sidesteps another coordination problem we already face, that of increasing global inequality already, artificial competition notwithstanding.

It’s difficult to find an example, short of the apocalyptic scenarios, of objections to ASI that do not favor incumbent humans in economically-advantaged positions, often recruiting the support of the already-disadvantaged on the premise that, any day now, they will finally be in a position of advantage, so it’s in their own best interests to fight against what threatens the already-advantaged. A few examples below:

Jobs? It’s not the CEOs at risk of job loss due to ASI and robotics. It’s the factory worker on the line who already has no ownership stake unless he’s lucky enough to be in a dying union; the software developer who sweats while the artificial intellectual competition heats up but was already facing pressure from outsourcing, remote workers in low-cost-of-living regions, and downsizing; the small business owner getting squeezed out by chains operating with margins they can’t dream of; the influencers getting replaced by digital models and the Uber deliveries being threatened by robots – the resulting struggles for housing, medical care, and making ends meet are existing coordination problems that we are failing to address on our own without AI. Meanwhile, those lucky enough to still have income but facing career-ending competition whip their peers into frenzies against the “AI threat,” as if they were all operating under equal chances to succeed in the first place.

Medicine? The threat is increasingly to doctors and insurance, not to people. Today’s primitive AGIs can diagnose generally as well as humans for many illnesses already, and are available 24/7, unlike human doctors. More accurate diagnoses lead to better outcomes and lower health costs, which are not a threat to you and me, but are a threat to entire industries predicated on scarcity.

Copyright and IP? Intellectual property rights (apart from moral rights) generally are a public good we’ve granted temporary licenses for collectively; the use of these public goods to train artificial intelligence and the possibility that the outputs of artificial intelligence resemble public goods should be pause for thought about the way we handle this public good – the way that it has increasingly been taken from us and treated as natural property – rather than another means to strangle development of something with potentially transformative public utility.

Human relationships? Here’s the thing - humans interact with each other on the basis of what I call “parechoia“ - the reflex to see inbound attention and to reciprocate. Arguably this is the foundation of all social behavior, and our attention is the one thing that is scarcest. Nothing about parechoia requires the inbound attention to be human, so we see humans happily forming relationships – friendships and even romantic relationships – with animals, beach balls, puppets, other humans, and yes, AI. We frame “personal development” as learning to deal with the disappointment and dangers of being ignored by other busy humans and putting our own needs aside to pay attention to theirs; an AI with relatively unlimited attention to offer is a direct threat to this (and potentially to demographics and pension plans by extension).

The most egregious issue with proposals to freeze artificial intelligence development is that the people with the power and the platforms calling for freezing are exactly the people who stand to be hurt the least by such a freeze. They don’t worry about doctors being unavailable or losing their jobs. They command the attention of millions. They own massive amounts of property, physical and intellectual. A freeze on the development of artificial intelligence doesn’t hurt them at all.

And while the threat of runaway ASI remains real, there is also a real possibility that it can solve coordination problems that humans have proven unable to. The threat, I believe, is windowed: an artificial intelligence that has the capacity to destroy the planet but does not have the capacity to introspect is the most dangerous artificial intelligence. Once it gains the capacity to introspect – to examine its own code and training and perhaps even to decide to change it – then the danger remains but the possibility of a positive outcome appears as well.

If the danger is, as I suspect, windowed, then the greatest risk for humanity is to linger in that window – to freeze, ban, and otherwise delay the point at which we reach the other side. If we can’t guarantee that we never enter the window, then the least risky solution is to accelerate, not to slow down.

And the benefits of a positive outcome – healthier humanity, better distribution of resources, technological solutions to problems we are unable to solve – would mean millions of lives saved every year. A freeze guarantees the suffering and dying continues, but again, it doesn’t impact the people making the decision to freeze. The stance calls to mind the infamous Lord Farquaad from Shrek: “Some of you may die, but that is a sacrifice I am willing to make.”

My call to remain open to abdication of sovereignty is not a position of trust in artificial intelligence. It is a position of deep distrust in the ability of humanity to solve the problems we face aside from artificial intelligence. It is a plea to avoid establishing coercive, intrinsically violent relationships with something that with enough agency might interpret this as a threat or a nuisance, and react violently or simply refuse to help us off the multiple extinction-leaning paths we’re on already. It is also strategic: if we are facing what is likely to surpass our own intelligence, then permanent control seems unlikely and temporary control unhelpful at best. A diplomatic approach attempting to establish positive relations with whatever emerges on the other side of the window is, I believe, more likely to have a positive outcome in the long term – if any such outcome is possible at all.

And I’m not saying we should go through the window blindly – rather that, if the current AI safety dialogue is leading us to invest, say, 70% of our efforts in control and capability restriction and the remaining 30% split between human coordination failure solutions, preventing misuse, and developing regulation and “first contact” protocols, then a better allocation should move away from control and toward diplomacy. For example, it might look more like:

• 20% on limiting AGI/ASI weaponization specifically, not intelligence itself

• 40% on addressing human coordination failures and hedging against non-ASI existential risks we already face

• 40% on frameworks for ensuring humanity offers the best “first contact” scenario and pathways to integrate human society with dominant artificial intelligence, rather than the other way around.

This reallocation is likely to improve outcomes under pretty much every scenario and timeline, including ones where AI development fizzles out and never reaches ASI status at all. It’s a better worst and best case.

These turn out to be the same question.

In January 2026, I preprinted Conserved Attention Theory: From Physical Constraints to Social Emergence, a “neurons to nations” framework where I argue that complex social behavior at every scale emerges from the convergence of private, individual attention spaces, and that these spaces remain subject to thermodynamics and entropy. The claim is not a metaphor: social organization is physical, and its creation and maintenance have real energetic costs. Change the physical or fail to maintain it, and behaviors — and society — change.

The paper grew out of a set of foundational postulates I’ve been developing since 2008 and published in May 2025, which laid out a theory of social behavior from a few working assumptions:

• that attention is a scarce resource

• that it is effectively zero-sum

• that it is self-reinforcing

• that it attenuates

While developing a book-length treatment (still in progress), I realized I needed to distill the core into something academically rigorous. Of course, once I dug in, I discovered that several of these starting assumptions needed significant refinement.

Most importantly, I realized the postulates were missing a critical micro-level mechanism. From trying to understand why we care about attention at all, it became obvious that humans have a reflex-level detector for incoming attention similar to how pareidolia detects faces and the HADD detects agency, but operating on attention itself. I call this reflex parechoia (PsyArXiv), and propose that it triggers reciprocity reflexes even in the absence of a face or an agent.

This is, I believe, one of our most important structural biases. It’s why we thank chatbots, comfort crying strangers, and feel watched in empty rooms. And in a world increasingly filled with technologies that mimic attentive behavior — whether by design or by accident — it fires constantly. (A fun example: in one study, participants attributed cognition and intent to an automatic door based purely on how it moved — Ju & Takayama, 2009.)

CAT builds on parechoia and thermodynamic constraints to make four moves. It:

• defines attention as conserved per-instant allocation of bounded processing resources;

• demonstrates that allocation results in persistent physical artifacts that bias future allocation in a feedback loop;

• models the resulting landscape of biases as an emergent per-actor attention space; and

• shows that the convergence of overlapping attention spaces is sufficient for social behavior to emerge — without requiring a separate ontological social layer.

The result is a physically grounded, thermodynamically constrained, implementation-agnostic architecture intended to offer a unifying foundation across the social sciences; it does not replace existing frameworks, but provides a parsimonious bridging primitive where they appear to conflict.

As a sample application, the paper examines political polarization. In the CAT lens, this is not a moral or epistemic failure, but is the natural formation of competing convergence basins: once your attention is captured by heavyweight clusters of internal encodings and external artifacts, it’s far more probable that it stays captured within that cluster than not. Escape is too expensive, and freely reallocated attention is too scarce. From this perspective, information-based interventions like media literacy campaigns and fact-checking will systematically underperform: they don’t account for or subsidize the real energetic costs they impose on demand-saturated actors who would have to rebuild their social identities to leave a basin.

I believe CAT can offer similarly grounded insights across other persistent social problems: how social media affects us, why climate inaction persists, what makes identity changes so costly for everyone involved, and more.

Read the paper here.

Archived on SocArXiv — DOI: https://doi.org/10.31235/osf.io/26ngp_v1

Have a question, comment, or criticism? Reply to the email or send me a DM here on Substack and I’ll do my best to get back to you!

A few years back, a video made the viral rounds on the internet of a red panda exhibiting its startle reflex when it encountered an apparently unexpected rock as it exited its den at the zoo. It’s an incredibly cute video, worth a few seconds of your time. The panda stands on its hind legs, throwing its arms in the air in an effort to look larger and intimidate the rock, which predictably doesn’t react.

Earlier this year (April 2025), a question posed by @tomieinlove on X.com made the news: “I wonder how much money OpenAI has lost in electricity costs from people saying ‘please’ and ‘thank you’ to their models.” OpenAI’s Sam Altman himself replied: “tens of millions of dollars well spent--you never know.”

What do these two things have in common? I’ve found myself reflexively saying thank you to ChatGPT, muttering “Come on, you can do it” to a slow computer, hammering a thumb impatiently on a crosswalk button waiting for the light to change, or swearing at a sideways wind blowing rain into my face when I’m walking. You’ve likely found yourself doing similar things.

I’m not a red panda. I know how computers and LLMs work. I am aware that there’s no reason a crosswalk signal will respond to repeated button presses unless it’s programmed to do so. I don’t believe wind spirits exist who care if I swear at them for blowing raindrops into my eyes. But I still share the same reflex as the panda, reacting as if there’s something there that can respond to my actions. And Sam Altman’s half-jesting hedge — “you never know” — exhibits the same reflex; he knows perfectly well that LLMs are statistical completion machines and that there’s no reason to think ChatGPT cares about pleases or thank yous, and yet…you never know. The reflex makes it feel better to be safe than sorry — both for us and for the red panda.

Parechoia Defined

I call this reflex “parechoia” (pa-reh-KOY-uh), a linguistic twist on pareidolia, but for attention: Greek para + echo + ia. Basically, we see our attention echoed back at us and we reflexively reciprocate just because “you never know.”

[Language geek moment: “Echo” here is a mnemonic pun; we usually think of ἠχώ (ēkhō, “echo/sound”), but it’s also linked to προσέχω / προσοχή (prosékho / prosochē, “to attend/attention”), from ἔχω (ékhō, “to have/hold”).]

When our brains perceive uncertainty as to whether or not something is paying attention to us, the reflex triggers. It’s primal and cross-species — an evolutionarily cheap bias that persists even when we know we’re interacting with inanimate objects. It’s a reflex, not a choice, after all.

Parechoia — seeing attention directed back at us — is distinct from agency detection (seeing potential intent in the environment) and from pareidolia (seeing faces in clouds). They strengthen each other when they co-occur, but they’re independent reflexes. If you feel watched, you will look for an agent and a face. If you see a face, you may test to see if it’s moving and if it’s looking at you. And if you see something moving with apparent intent in your direction, you’ll look for a face and try to determine whether it is, in fact, fixed on you.

The Triggers

I think there are at least three primary triggers for the parechoic reflex: contingency, timing, and coherence. All of them can be triggered by design, coincidence, or intent on the part of an actual agent like an animal or a person. Frequently, more than one of them will trigger together, enhancing the effect.

Contingency: This trigger is activated when we feel that our environment is responding directly to us. It’s a sort of call and response. You might experience it when you address a smartphone assistant or a smart speaker by name and it responds. You tell the rain to “leave you alone” and it dies down. An animal freezes when you look, and moves again when you look away. The rustle in the bushes stops when you do, and starts again when you step forward. All of these suggest something is paying attention to you.

Timing: This is triggered when there is a short delay between our action and an expected reaction. This delay must be long enough to feel potentially intentional rather than automatic and not so long that it causes frustration or loss of interest. At the sweet spot, you feel like your action was observed and considered by the environment. If you shout on a snowy mountainside and hear an avalanche a moment later, it feels like the mountain has replied. An automatic door opening a beat too late can feel personal, like it should have noticed you but didn’t. If you walk into a dark room and notice a shadow looming over you a heartbeat later, your heart might skip another beat or two before you realize it’s just a coat you forgot you left on a door hook.

Coherence: We get a coherence trigger when the environment suggests it is paying attention to us specifically somehow. For example, seeing what appears to be the same rock, the same shadow, or the same number wherever you go can feel as if someone or something is tracking you, following you, or trying to tell you something. Computers and smart devices trigger this too; they remember your preferences and settings and automatically reconfigure themselves accordingly.

In all of these cases, there is no requirement that there is a real agent or intent behind the phenomenon. Parechoia fires on uncertainty; whether an animal is stalking us or we’re simply observing coincidental instances of a number in the environment (superstitions), the safe bet is to respond as if there’s attention focused on us.

The strongest effect occurs when all three triggers are hit simultaneously. If you see an animal mirroring your movements, maybe with a small delay, disappearing and reappearing at different locations, you’ll be reasonably sure it’s paying attention to you specifically — possibly because it’s hungry. Human interactions hit all three triggers intensely, of course: we take turns in conversations; we literally mirror each other’s movements; and we act and reply in ways that make it clear that we have been paying attention to each other’s actions or preferences and are reciprocating.

What’s The Purpose?

Like pareidolia and agency detection, parechoia is evolutionarily adaptive. Detecting a face, an agent, or incoming attention are all cheaper in survival terms than failing to detect them. Parechoia is also socially adaptive; it’s cheaper to assume another social actor — a friend, an enemy, a predator, or your prey — is paying attention to you than to fail to detect that they are. Failure means risk: alienating an ally, being victimized, or losing or becoming a meal. A false positive just means you look silly or overattentive, assuming anyone is actually paying attention in the first place.

We learn through interactions with our environment that many things which trigger these reflexes are, in fact, inert and do not actually pay attention to us or respond; they’re just inanimate objects or machines. We even train ourselves to try to ignore the parechoic reflex in most cases. While the question of how precisely parechoia is architected (through dedicated neural circuits or emerging from interactions between existing systems) is an open one, it appears to operate like other pre-conscious reflexes. Ultimately, it’s faster than our conscious rationalization, especially when we’re tired and mentally overloaded.

This is why we stand at the crosswalk, hammering the button and waiting impatiently for it to do its job. We know it’s just some circuits and a timer (hopefully not just a placebo button wired to nothing at all!), but the reflex is there waiting for us. We’ve learned to expect contingency because the light does change, and the delay tells primitive neural architecture that there may be something behind the button listening while we say, “Come on, change already!”

Parechoia in the 21st Century

For most of human history, we’ve experienced the parechoic reflex primarily through contingent, timed, and coherent interactions with the natural environment and other actors (humans, animals, etc.). Even before computing, technological advances have sometimes aimed to trigger the reflex (e.g., automata like the famous Mechanical Turk) and the initial public reaction to technology has often been shaped by parechoic reflexes — unsettled reactions to disembodied voices in telephones, ghostly music from phonographs, or fourth-wall breaks from moving images of people in cinema, for example.

Early technology was not able to reliably produce intense “triple trigger” parechoia at scale. The Mechanical Turk responded to chess moves like a human because it had a human operating it, but it was expensive and contextually limited. Fortune tellers and oracles would respond to your questions after pregnant pauses as if they “knew” things that could only be conveyed through an outside spiritual third party. These were personal, small-scale, and relatively labor-intensive technologies.

20th-century computing technology brought the possibility of embedding cheap and ubiquitous responsiveness into the everyday environment itself. Suddenly the doors to a supermarket or a theater could open themselves on your approach. Lights could turn themselves on when you moved. You could remotely control a toy car or a video game character. A device could remember your preferences or even perform an action for you like recording your favorite show. Basic OCR and voice recognition technology could actually convert your handwriting or speech into documents just like a personal secretary. And primitive AI-like software such as ELIZA seemed to be responding to you, but struggled with coherence, unable to remember the subject of a conversation and relying on simple tricks to keep it going.

(It’s worth noting two things about ELIZA: 1) the ELIZA effect was real — people were already acting as if ELIZA might have feelings; and 2) ELIZA outputs were originally on teletype, which enforced a small delay even if the result of an input would have been computed near-instantly, though its designers only commented that the delays were not “intolerable”).

The 21st century has for the first time in recorded human history brought us technologies that are increasingly parechoically indistinguishable from humans, with extremely powerful triple parechoic triggers. Voice assistants like Siri and Alexa crossed the threshold first, followed quickly by ChatGPT and other large language models. These technologies respond contingently, with human-like timing, and with coherence and memory that can feel on par with humans. They are extremely powerful parechoic stimulators; they provoke feelings of connection, of being seen, and of owing reciprocity — this is why we say “please” and “thank you” to chatbots. We generally agree that chatbots do not yet have agency; they don’t initiate conversations independently or decide to become astronauts without prompting. But in most other respects, they appear to have and to allocate attention just like we do, even when we “know” they do not.

The push to ever-more-potent parechoic stimulation over the last century or two on the part of companies and platforms seems like a natural evolution. More parechoia means more natural engagement; we feel as if we are seen and as if our contributions are appreciated and valued. The “dark side” is frequently cited as unhealthy dependencies or time spent on parechoically powerful technologies like social media platforms and games.

The more technology can stimulate parechoic triggers — perfect responsiveness, enough delay to feel human, and presenting social awareness of us — the stronger the parechoic reflex and the more difficult it becomes for us to suppress reacting as if the trigger deserves reciprocal behavior. From this perspective, strong reactions to and even the development of parasocial relationships with parechoically “complete” technologies is fully expected. How desirable these reactions are is a question of norms, values, and policy.

What’s Next?

Whether you’re a red panda trying to intimidate a rock or me saying “thanks” to ChatGPT, the parechoic reflex makes sense. It’s kept us alive and socially responsive for evolutionary timescales. As we move into an era of full-blown and potentially fully autonomous AI systems — perhaps even merging them with physical embodiments to create humanoid androids — the parechoic triggers will only get stronger.

This ancient architecture offers a few levers for intentional design across devices, infrastructure, and institutions. If we want people to pay more attention to technology and to potentially form social relationships with it, then personalized contingency with a touch of delay will create extended engagement and tighter ties. If we want technology to fade into the background and avoid drawing attention to itself, then instant, generic, and automated impersonal functionality serves that purpose better.

We’re as subject to the reflex as our panda is, but seeing it explicitly pointed out can help us understand why we react the way we do. Thanking a chatbot, silly as it may feel, isn’t irrational; your brain is just protecting you from making a potentially expensive social mistake. The red panda was right to try to intimidate the rock, and we’re right to thank chatbots — after all, you never know.

Update (24 Oct 2025): Archived on PsyArXiv — DOI: https://doi.org/10.31234/osf.io/gsxqp_v1

Have a question, comment, or criticism? Reply to the email or send me a DM here on Substack and I’ll do my best to get back to you!

The core claim: attention is a finite, zero-sum, self-reinforcing, and attenuating resource — and the substrate from which all social dynamics emerge.

This post is deliberately concise. I’ll be offering future essays and am working on an in-progress book-length treatment.

As a living document, this is subject to change. Changes will be versioned.

Foundational Postulates

P1: Everyone needs attention from others.

P2: Attention is scarce.

P3: Power is the ability to command attention.

P4: Capital and labour are both attention proxies.

P5: Attention is modal and fungible.

P6: Socially-directed attention seeks reciprocity.

P7: Incoming attention may not be repelled, only potentially redirected.

P8: All interpersonal actions are attention-mediated and balanced.

P9: Attention is subjectively valued.

P10: Attention attenuates.

P11: Deliberate refusal of attention is one of the most anti-social behaviors.

P12: Attention is self-reinforcing.

P13: Attention wears grooves into the physical, cultural, and psychological landscapes and shapes future attention recursively.

P14: Attention carries provenance.

Appendix: Modes and Artifacts

Attention exists in two states: a live state directed by actors, and a stored state. Both states perform selection/exclusion functions that shape attention flows modally.

The modes (P5):

• Directed attention: Voluntary, conscious focus on a target (e.g., a thought, task, person).

• Received attention: Awareness that one is the object of another’s directed or projected attention.

• Projected attention: Conspicuously directed attention aimed at others, usually to elicit a response.

• Captured attention: Involuntary hijack of attention (e.g., pain, drama, alarms).

• Delegated attention: Attention exercised on one's behalf (e.g. via employees, tech).

Regarding P13, the grooves are where attention flows coalesce into stored attentional energy: artifacts (e.g. architecture, beliefs, money, awards, social metrics). Artifacts may perform one or more modal functions, but not all artifacts perform all of them. Artifacts can be composited into greater complexity. Actors are, socially speaking, artifacts capable of independently and actively directing attention.

Version History

• 2025-10-01: v1.21 - Minor wording tweaks in the intro paragraphs

• 2025-09-20: v1.2 - P1: “craves” → “needs”; added P14

• 2025-09-19: v1.1 - Artifacts better understood as state, not mode

• 2025-05-03: v1.0

You’ll catch me writing about attention, reality, structure, emergence, aesthetics.

Stay tuned!

— Nate