Access the PDF version of this essay by clicking the icon to the right.

New information technologies are sometimes described as having a “democratizing” effect. Anyone’s single-sentence post or 10-second video can potentially reach millions over the internet, meaning that technology has broken barriers that long constrained the reach of individual ideas, opinions, or observations. Yet, paradoxically, there is growing anxiety that our system of communication is posing severe, unprecedented, and even fatal threats to democracy.

In the mid-1990s and early 2000s, new media and internet technologies were celebrated for their potential to advance knowledge, unyoke us from long-standing constraints, and democratize access to communication and speech. A dominant theme was the power of these new media to break the chokehold that old media elites had over what was printed and broadcast, determining what people saw and heard. These gatekeepers decided what stories, experiences, opinions, and artworks were worthy of attention—and what were not. The initial wave of optimism embraced the idea that the new digital medium had the capacity to extend the power to participate in the creation, publication, and distribution of material to an ever-broader spectrum of readers, listeners, and choosers of material.

The passing years have borne witness to an evolving media landscape that is more complicated than these visions, one with a darker edge. Although it is true that almost anyone can publish on the internet, there is no guarantee that anyone will read or hear them. The sheer volume of content online means that gatekeeping is no less important now than it was in pre-internet days. This time, however, the concern is less about who and what gets published and more about who and what gets attention. Belying the ideals of the early optimistic vision of a diversified mediascape, we seem to have reverted to one that is concentrated, vesting the powers of selection, recommendation, and distribution in the hands of a few. But the conditions now are markedly different. Instead of being up to human decision-makers (acting in capacities of chief editors, heads of broadcast media, heads of marketing companies, etc.), creation, selection, and distribution are directed through a patchwork of automated systems and human decisions functioning atop social media platforms, which, for the most part, are owned by a dominant few companies. We refer to these systems and decisions as algorithmic when they incorporate formalized, computational procedures, rules, or instructions, which execute some function or purpose.

Framed in a positive light, the use of algorithms to curate, direct, and aggregate content is a way of coping with its massive scale while remaining true to the spirit of democratization. Automating some of these operations through algorithmic systems can enable participation and representation in new and exciting ways, even though bottlenecks and gatekeeping may still be required. Finding quality shows, movies, books, and restaurants, for example, no longer depends on the published reviews of one or a handful of journalists; instead, masses of people can contribute comments and opinions. Social movements have been empowered by algorithmic systems that connect like-minded individuals—sometimes geographically dispersed—to mobilize against long-standing injustices and abuses of centralized powers.

The downsides of algorithmic curation, however, are becoming increasingly evident, too. Savvy users have been able to exploit algorithmic media systems, for example, to embolden vocal support for far-right, racist ideologies and conspiracy theories. Coordinated attacks on algorithmic systems—including by nation-states—have undermined the reliability of information distributed by historically trustworthy networks, sowing dissent and distrust. Biases embedded in algorithmic systems, sometimes too subtle to easily detect, systematically and deterministically exclude and marginalize victims in new ways, even proliferating novel forms of bigotry. Increasingly polarized societies are vulnerable to fragmentation, as insular subgroups take form with differing values and ideological commitments. Horrific acts of violence have been carried out, inspired by algorithmically steered vicious, divisive, and skewed content.

Motivated by mounting concerns over the effects of algorithmically circulating online content on dangerous behaviors and corrosive social formations, scholars and journalists have reached for the concept of “algorithmic amplification” as both an explanation and a warning. Experts in the areas of media and technology have long posed questions about “amplification,” about how and why certain content, information, viewpoints, or names spread both broadly and unevenly. In the face of the further proliferation of hate speech, mis- and disinformation, and the acute rise in bias and polarization, experts are taking novel lines of investigation into the ascent of algorithmic systems as dominant engines for selecting, distributing, and recommending content.

Our paper takes up this line of questioning, with a focus on what we are calling problematic algorithmic amplification. As search, recommendation, and other information and media services are formalized and operationalized algorithmically, the threats they pose do not stop at the proliferation of problematic content, including hate speech, polarization, mis- and disinformation, etc. Though content of this sort is worrying, we seek to identify underlying reasons why algorithmic systems have increased our collective social vulnerability to their proliferation. The pages that follow will lay out our argument, that algorithmic systems, including those found on dominant social media platforms, pose existential threats to entrenched processes that perform many of the same functions, namely, selective circulation of information and content.

These processes, which may have evolved over long stretches of time, are sound to the extent that they serve societal and institutional purposes. Sound processes, as we have defined them, provide good reason for trusting what we uncover at certain times and for certain needs, whatever the nature of what it is we seek at those times and for those needs. To illustrate these ideas, we will discuss epistemic processes, aimed at achieving trustworthy knowledge, and democratic processes, aimed at the collective determination of governance. Before returning to our discussion of processes and threats from problematic algorithmic amplification, we provide a few definitions and clarifications.

Algorithmic Amplification: What Is It?

Our research suggests that the term algorithmic amplification was not formally introduced but, instead, emerged organically in public discussion. Although the term has been a useful organizing concept for journalists and academics, many who use it agree it is imprecise. For example, Keller (2021), a legal scholar, admits that “the concept of internet amplification may be inevitably fuzzy at the edges.” Writing on journalistic practices for reporting on extremists and manipulators, Phillips (2018), a journalism and communications scholar, noted “how fraught questions of amplification really are; just how damned if we do, damned if we don’t the landscape can be.” Eckles (2021), approaching the topic from statistics and social science, writes:

Assessing what a ranking algorithm is amplifying is not a trivial task that platforms have simply neglected. And we may be substantially mislead [sic] by assessments of algorithmic amplification that simply compare two rankings of the same content.

The imprecision surrounding algorithmic amplification should not be too surprising, considering its origins as a metaphor, drawn from signal processing and dynamical systems, predating the internet and other computational technologies by decades. And before the 20th century, its prosaic meaning was “to make ample,” an expansion or growth of something. With the advent of electrical signals and acoustics, an “amplifier” was a tool that could increase the amplitude of a signal, with a range of applications from live music to electrical circuits.

In the 1950s, amplification entered conversations around cybernetics and systems science. Applied to early efforts in artificial intelligence (AI), the term “intelligence amplification” (IA), coined by Ashby (1956), referred to the potential for information technologies to augment human intelligence and develop knowledge at a faster rate. The idea of IA is contrasted with AI because IA focuses more on augmenting and advancing human knowledge while AI has historically been about replicating intelligence in a machine. This usage suggests societal-scale, complex systems thinking that connects with algorithmic amplification concerns raised today.

The term’s recent use in describing the distribution of content on social media marks an expansion in meaning. In these contexts, the term is used not to highlight technology’s ability to expand knowledge but to warn about its ability to expand the spread of all sorts of undesirable information and activities. Misinformation, extremism, hate speech, bias, and other corrosive social effects have been bolstered by algorithmic systems, tuned to provide relevant or entertaining content. Critics have sourced these corrosive effects to the internet and other digital systems as a way of expressing and making sense of their worries. However, articulations of the problems surrounding amplification suffer from the term’s lingering ambiguities. On one hand, an overly broad definition runs the risk of lacking “teeth.” If any reproduction of speech is amplification, it becomes difficult to pinpoint why amplification matters. On the other hand, overly narrow definitions, such as “departure from a baseline feed,” may squeeze out amplifying practices that people worry about, by excusing baseline behaviors and choices as neutral.

Is amplification necessarily problematic? At least part of the term’s imprecision arises because those who use it have not converged on an answer. Often, “algorithmic amplification” is used both to describe and appraise. As a description, it characterizes and circumscribes various practices without judging them in ethical terms. Normative uses, by contrast, identify these practices as ones deserving of moral scrutiny and valuation, whether positive or negative. To achieve greater clarity, this paper teases apart these two uses, first elaborating a descriptive concept of algorithmic amplification, and then providing a normative account, further distinguishing the focus of greatest interest to us, here, as cases of problematic algorithmic amplification.

We use the term amplification to cover the systematic procedures that platforms, publishers, and services use in order to expand the reach of content, either in absolute terms (i.e., across-the-board) or selectively for particular audiences and categories of content. It is worth noting that by virtue of some content being boosted, in relative terms, the neglected content might be considered suppressed. Amplification may be further intensified due to the dynamic effects of the initiating steps; for example, people whose beliefs, incentives, and behaviors have been affected by amplified content may further amplify this content in their interactions with one another or with the platform.

We use the term algorithmic amplification to refer to the amplification of content as a result of formalized and operationalized sets of instructions, typically carried out by computer systems. Although algorithmic amplification involves machine automation, the degree of automation can vary across applications, in some instances requiring significant human involvement and oversight. When the logic dictating the reach of content is expressed as an algorithm, whether or not the content gets amplified depends on component procedures and rules. For example, a social media post that uses a certain tag, or contains text limited to a certain length, may be rewarded through the encoded rules and instructions that are collectively referred to as an algorithm. The advantages of algorithmic amplification to platforms and services that utilize it (and—they might argue—to their users), as with other automated systems, are efficiencies of speed and scale. Algorithmic amplification, implicitly and explicitly, is indubitably baked into systems that are integral to contemporary, digital societies, such as automated web-searching, all manner of matching (e.g., dating), and a great variety of recommendation platforms (e.g., Yelp, Reddit).

Problematic Algorithmic Amplification

Armed with this general account of algorithmic amplification, we turn to the normative question at the heart of this article: When is algorithmic amplification ethically problematic, and why care? An obvious place to seek answers is the problematic content itself, which, as noted earlier, is an immediate source of concern. This covers all manner of misinformation and disinformation, including damaging and dangerously misleading information that can result in poor decisions and distrust. It also covers pernicious, vicious, and vile content that can lead to paranoia, radicalization, hate, bigotry, sexism, and an increasing polarization of views, opinions, and political stances. This includes, of course, aggressive, threatening, and harassing content that is not only frightening but also coercive, carrying the potential to incite hate-based violence.

Tech companies and other industry incumbents have addressed the problems of algorithmically generated feeds, reels, recommendations, and matches by taking action against problematic content. They have collectively invested billions of dollars in moderating and selectively removing content they deem unacceptable. According to a 2020 report from NYU Stern School of Business, each day, over 3 million posts are flagged for potential removal by Facebook; among them, an estimated 300,000 posts are erroneously marked, either as a violation of standards or as benign. Ethical norms dictating what content is acceptable are hardly ever clear-cut; they differ depending on culture, community, and context. Even accepting (reasonably) that some content moderation is inevitable and necessary, the practice has been controversial in some cases. For example, photos documenting war crimes and other atrocities have been censored by Facebook and other internet platforms, leading to moral outrage. Beyond persistent rifts between free speech fundamentalists, on the one hand, and supporters of free speech balanced by other rights, on the other, content moderation through deletion and censorship continues to be controversial despite significant resource expenditures on them.

Content moderation and deletion, like the arcade game of “whack-a-mole,” is Sisyphusian: Companies hire armies of workers to detect, identify, and scrub problematic content, and yet it quickly pops up again. Why? The answer, in our view, is rooted in the underlying algorithmic processes which, like an invisible hand, ensure its resurgence. The irony is not lost on critics that the companies paying to scrub, frequently, are the very companies responsible for the algorithmic systems they have designed, which generate clickbait to hold users’ attention and presence. The streams and cycles of problematic posts—as it were, the moles—will continue to appear for as long as the processes themselves, surrounding social malaise, or both are not radically addressed.

Problematic content, in our view, while important, is not the defining characteristic of problematic algorithmic amplification; instead, it stands in relation to algorithmic amplification as symptoms to an underlying disease. Though it can be helpful to suppress painful symptoms, it is more lasting and decisive to address underlying causes and other sources of vulnerability. Casting instances of problematic content as one-off errors—letting in bad language through the cracks, allowing bad people to tell lies online, leading search and recommender systems to “get it wrong”—that can be handled by directly moderating it misses the point. When bad content is viewed, instead, as a symptom, an inevitable upshot of problematic algorithmic amplification, it draws attention to these underlying causes and, as in the case of underlying disease, a sounder, more comprehensive approach to treatment. Problematic algorithmic systems of amplification, in short, are systems that are unsuitable for the valuable purposes or tasks to which they have been put. When these systems, furthermore, nudge aside preexisting processes for performing equivalent amplifying functions, the harm is more jarring. Reducing vulnerability to individual, social, and political harm, therefore, requires that we ask two questions: First, whether algorithmic systems amplifying content are mismatched for the task their purveyors claim them to perform; and second, whether they have nudged aside historically entrenched social processes for performing these tasks.

What is a “process”?

A supporter claims that Donald Trump, actually, won the 2020 United States presidential election. When you ask why she believes this, she answers that Donald Trump said so on Twitter, confirmed by many others whose Tweets appeared in her feed. Questioning this evidence, you look elsewhere for confirmation. This may take you to processes leading up to the declaration that a particular candidate has won an election. In the United States, as in other democracies, achieving the ideal of democratic elections in concrete, real-world settings means affording each individual the right and opportunity to freely cast votes in accordance with a rigorous set of procedures. A myriad rules and conventions cover each stage of an election cycle, including eligibility criteria, registration requirements, criteria for fair geographic representation, and protection against fraud, to name a mere handful. A myriad details define local conditions at polling stations to ensure safe and accurate voting and, later, vote counting. Where certain functions are delegated to machines, care is taken to ensure that hardware designs, systems specifications, and software code embody the required, traditional protocols for ensuring a trustworthy election. Adjustments to entrenched processes may take hold over time, as existing elements are shown to be ineffective in achieving the aims and values of democratic election. For example, laws ensuring distance between pollsters, protesters, and voters might be introduced to prevent intimidation or other forms of undue influence. Finally, beyond processes for ensuring trustworthy democratic elections, we also rely on processes whereby the results of an election are announced, including the qualification required for trusted sources.

Now we introduce the idea of a social process for the purposes of this article. Briefly, a process refers to an ordered assemblage of conventions, rules, steps, institutions, norms, etc., guiding action, activity, and practice toward the attainment of specific ends. In the case of voting, as sketched above, the end could be described as the formation and recognition of a duly elected governing body. A process is considered successful, or sound, if it facilitates the fulfillment of these purported ends. It is flawed to the extent it fails to do so. In the case of voting in democratic elections, polling procedures are flawed if, for example, they fail to secure confidentiality in societies where individuals are at risk of intimidation, coercion, or harm based on voting preferences. There is complexity in the ends or aims too: They may not be reducible to a simple slogan; they may or may not serve general societal welfare; they may also be parochial, serving the narrow interests of some over others; and so on. Certain behaviors and conventions in the realm of lobbying might offer a case of a political process whose ends are misaligned with broader societal goals. This paper focuses predominantly on socially valuable and just ends and identifies departures from these as problematic.

A caveat: Our aim, thus far, is not to define, in all necessary fullness, what we mean by a social process—impossible within the scope of this paper. For such a deep and complicated idea, we take comfort from the famous quote about pornography, knowing the concept of process will be recognized through illustration, if not through formal definition. Accordingly, the aim here is to render an idea of a social process that is deeply familiar and commonly experienced, even if not explicitly recognized or named as such. Before we sketch a few more instances, we emphasize three key aspects of what we mean by social processes: 1) They are teleological, meaning they are designed to achieve an end, or set of ends; 2) like living creatures, they may change, mutate, and evolve, typically, in order to maintain efficacy against a backdrop of societal or even material (technological) change; and 3) they are prescriptive, governing behaviors implicitly, through norms and conventions, explicitly, through codes and rules, partially, or fully, through constraints embedded in technical or other material systems.

Process, aim, alternatives

It is difficult to enumerate cases of social processes not because they are hard to find but because social life is rife with them, ranging from the trivial to the exceedingly complex, from those with little at stake to those with the highest stakes. In the workplace, processes abound, from hiring to those surrounding promotion, firing, and more. In academic life, there is a dizzying array of processes governing the review of student applicants, aspiring faculty, and journal and conference submissions. Peer review alone covers a range of differing options—single-blind, double-blind, open, secret, etc. Those of us who have experienced any of these know that they are prescriptive, sometimes highly so: They are designed around goals, such as identifying qualified applicants and accepting excellent articles. Background or contextual factors also affect the shape of a process—time, financial resources, and geographic constraints merely scratch the surface of the vast set of factors guiding behavior. We have also witnessed the complaints that ensue when processes are seen to have failed at achieving their goals. When societal or cultural prejudices, or sloppy evaluations, for example, lead to unqualified candidates being hired, when reviewers favor work from prestigious institutions instead of the most excellent, or when errors in submissions are overlooked—these failures may call for reassessment and calibration of the processes underlying these outcomes. More complex processes, such as approving new drug treatments for medical use, involve hierarchies of processes prescribing a host of component parts, including those addressing sound research practices both in the laboratory, in clinical settings, and “in the wild,” whether managed by academic researchers or drug companies. These can be granular, as, for example, prescribing the random assignment of subjects to control and test groups in a trial aimed at assessing the causal effect of an intervention. A myriad components and procedures constitute necessary steps toward regulatory approval, including disclosures of efficacy, risks, and ethical treatment of research subjects. Why these steps, methods, practices? At the risk of stating the obvious, in the ideal scenario, they would be perceived as optimally efficacious for achieving desired ends—always open to challenge and adjustment due to contextual shifts, novel methods and technologies, and even, adjusted purposes.

Faulty component practices, tools, or conventions lead to unreliable conclusions, for example, they may yield outcomes that cannot be reproduced, potentially leading to unsafe drugs on the market and, ultimately, poor outcomes for health. Such issues may be attributed to poor execution of methods by individual researchers or, if persistent and pervasive, may reveal a flaw in norms, rules, conventions, methods, or tools. If that is the case, the spotlight shines on what we have called process. When recognized as such, problematic processes may result in targeted revisions of particular components, for example, requiring preregistration of experiments to overcome publication bias—a key cause of nonreplicable, unsound results. Or, they may yield far-reaching overhauls of community practices. Whether change is achieved through systematic analysis, rhetorical calls for action, trial and error efforts, or combinations of these, for proposed adjustments to entrenched processes, the ultimate test is trustworthiness—in this instance, replicable study outcomes. Adjusted processes are justified if they are shown to be at least more successful in achieving the aims set out by respective institutional practices.

Details aside, the takeaway from this section is the connection drawn between a process, that is, a structured arrangement of rules, norms, established methods and practices, standards, and guides, and the aims, or set of aims, around which it is oriented. We refer to a process as sound or trustworthy to the extent it successfully meets or promotes these aims. Although there is no reason to exclude automated algorithmic systems from this account of processes, there is also no reason to exempt them from undergoing the necessary scrutiny to establish that they meet the conditions of trust and soundness. A fair evaluation would require, further, that the automated systems in question are comparable to alternative processes for performing equivalent tasks.

Together the different strands of our discussion converge on a conception of problematic algorithmic amplification as formal systems that fail to achieve their purported socially valuable functions. Algorithmic systems are particularly problematic when they subsume, undermine, disrupt, erode, or replace existing processes (procedures, protocols), which may include automated systems, honed over time to achieve socially valued ends more effectively and efficiently than their algorithmic counterparts.

To signal the qualities of efficacy and efficiency, we apply the terms sound and trustworthy. We do so without prejudice against automated algorithmic systems, which may earn these qualities when they are shown to meet certain criteria, even when these systems replace historically entrenched processes. Accordingly, with the selective publication of ideas and content—the case that opened this article—our conception does not support prior practices where the full power falls to a privileged few to decide, without account, what is published or earns airtime, etc. Trustworthy processes require reflective assessment against a society’s valued ends and, likely, will continue being refined and improved in competition with reasonable alternatives, algorithmic or otherwise.

Returning to the supporter who is affirmed in her belief that Trump won the 2020 election based on an overwhelming proportion of Tweets in her Twitter feed. One might ask why Twitter’s selective amplification does not qualify as a social process as we have described it, alongside the more analog and long-standing processes that, traditionally, have established and proclaimed the winners of political elections. This question is legitimate and sits at the crux of our substantive thesis. Yes, the sociotechnical systems of algorithmic amplification can be conceived of as types of processes. In this instance, the reasons for favoring entrenched traditional processes over Twitter’s algorithmic amplification, however, are rooted in their respective aims (and their performance vis-a-vis their aims). In the entrenched process, these aims are to achieve and proclaim democratically elected leaders and they stand or fall on their capacities to do so. When they fail, or are believed to perform suboptimally, they become targets of adjustment or even replacement. Approaches to evaluation, whether empirical, historical, analytical, statistical, or other, are informed by fundamental commitments to democracy and legitimate approaches to governance, given popular expression in maxims, such as, “of the people, by the people, for the people.” This is a far cry from the litmus test of Twitter’s algorithmic systems for constructing individual feeds (to the extent we are able to know these). Twitter is a privately controlled corporate platform whose primary commitment is to building profit for its shareholders by directing user attention toward paying advertisers’ messages on the platform. To serve these aims, Twitter’s “algorithm” displays content likely to capture user attention and promote engagement.

Different algorithmic systems may be more or less appropriate when appraised as part of different processes (and in light of different aims). For example, Reddit’s algorithms rank posts based on the numbers of “up-votes” and “down-votes” they receive from users. Members of subcommunities (“subreddits”), relying on an algorithm that elevates posts based on numbers of up-votes, might extol these as “democratic” and might find them useful for locating memes, stories, art, and other cultural content that are generally resonant, inspiring, or uplifting. These algorithmic ranking mechanisms determined by up-votes, however, are not necessarily appropriate for all ends; for example, scientific findings about the risks of vaccines, the casualty count in a distant war, or the threat level of an impending natural disaster. A voting-based mechanism that tracks popularity does not guarantee that true, reliable findings will make their way to the top of a Reddit feed. If the purpose is to convey trustworthy information, a popularity-based algorithmic system for elevating and demoting certain posts may not, systematically, align with the purpose of highlighting the most trustworthy information.

The Case of Epistemic Processes

For centuries, the process for disseminating and amplifying information has co-evolved with the process for vetting claims as truthful and trustworthy. People have collectively invested in institutions and processes aimed at verifying and underwriting claims as factually truthful. They have designed and vetted institutions and processes as trustworthy, in order to address and resolve disagreements over factual claims. But these arrangements, and our faith in them, are fraying. Curatorial practices on the web are driving a wedge between truth claims that are algorithmically amplified and those that have been vetted through sound, trustworthy processes. This section focuses on the family of threats that plague epistemic processes as a test case for our account of problematic algorithmic amplification, not because they are the only threats or processes we should be concerned about, but because they are particularly potent and wide-reaching.

Digital platforms, including the web itself, have raised the stakes for epistemology, the branch of philosophical inquiry into the nature of knowledge—how we come to know, and the validity of knowledge claims. Since ancient times, and through the present day, debates have raged over whether knowledge requires direct experience with one’s senses, rational analysis from premises to conclusion, deep and deliberative reflection about the conceptual world, exposure of belief claims to counterattack by adversaries, and more. It takes only a moment’s reflection to understand that if “seeing is believing,” then believing only what one directly experiences with one’s senses would shrink the universe of belief to almost nothing. A moment longer and we grasp that almost all of what we know, or claim to know, comes not from direct experience but from the testimony of others, including what we learn from family, community, school, books, newspapers, TV, and more. The degree to which we depend on others, on sources outside of ourselves, for almost all that we know, explains why the stakes are so high. Digital platforms and websites, increasingly, have joined and taken over as dominant sources and curators of what people believe and claim as knowledge. Herein lies one of the serious risks of problematic algorithmic amplification: Inundated by information vying, algorithmically, for our attention and credence and calling, algorithmically, for our commitment, action, and engagement, we are frequently at a loss as to what to trust, whom to trust, and why.

These thorny issues have been pondered for centuries. In the Socratic dialogues, Plato relentlessly attacks his archenemies, the sophists, arguing that their clever rhetoric amounts to a “sort of conjectural or apparent knowledge only of all things, which is not the truth.” Their performances, paid for by wealthy patrons, misled, even corrupted their audiences. To this day, sophists and sophistry are associated with false and deceptive rhetoric, unreliable means toward untrustworthy knowledge claims. By contrast, Plato’s Socrates is depicted as a philosopher, humble in acknowledging his ignorance. A philosopher is not an oracle but a general seeker of knowledge, attained through careful, systematic argumentation and deliberately exposing claims to rebuttal, challenge, and test. Known as the Socratic method, the approach comprises a process of back-and-forth assertions among interlocutors who ask and challenge one another in a sequence of questions aimed at flushing out contradiction and, ultimately, the refinement of an idea. For Plato, argument through deductive, conceptual, and theoretical reasoning was the most reliable process for acquiring knowledge, superior even to direct perception. Though not recommending a return to Plato’s epistemology, we see these ancient writings as a reminder that truth is not all there is to knowledge. As important are the processes we rely on for achieving it.

We use the term epistemic process (or knowledge process) to capture, in broad strokes, the structured practices aimed at achieving knowledge, understanding, expertise, or, generally, at justifying truth claims. Analogous to the processes for voting and those for identifying safe medical treatments, epistemic processes are rich assemblages of procedures, methods, actions, rules, norms, conventions, institutions, etc., whose aim, in this case, is knowledge. Different knowledge-seeking activities may be relied on for a vast array of knowledge types, ranging from questions about a train schedule to highly specialized questions, such as the causes of brain cancer, the origins of the universe, the state of a faraway war, or the costs and benefits of childhood vaccinations.

A child in a library completing a history project about, say, the Middle East, might pull a textbook or encyclopedia from the library stacks or might turn to Wikipedia or Google Search. Although these sources may seem roughly interchangeable, the epistemic processes they rely on are not. Reputable textbooks or hard-bound encyclopedias conform to an assortment of norms governing the selection of topics and sources, expert authorship, primary source engagement, credible citations, editorial scrutiny, etc. That they sit on library shelves, furthermore, implies that they have weathered systematic vetting by librarians. Wikipedia, which may seem equivalent to hardcopy encyclopedias in a different medium, conforms with markedly different norms, relying on networks of volunteers who contribute, edit, and moderate pages. Whether Wikipedia earns people’s trust depends on a number of factors, such as whether independent processes and trusted sources corroborate its entries, and whether people approve of the norms and rules (in our terms, the processes) that Wikipedia supports for evaluating the merits of a given entry (including familiar warnings when these processes have not been followed).

When students turn to Google Search, they are relying on yet another process—a complex set of Google’s famous and secret indexing and ranking algorithms, which place on their screen a list of sources, including some that have paid for placement. Google Scholar, which might, arguably, stand in for the vetting processes of librarians, similarly ranks and recommends sources of information according to an opaque algorithmic system, leaving to Google a host of evaluative decisions about the relative importance of citation counts, download counts, journal, title, and text for establishing credibility and for determining relevance.

What makes an epistemic process successful? What supports the conclusion that a particular assemblage of norms, practices, methods, procedures, etc. is a reliable pathway to knowledge? Philosopher John Hardwig pointed to trust as an essential component. For Hardwig, people are epistemically tied to one another through trust and testimony, and thus, “the trustworthiness of members of epistemic communities is the ultimate foundation for much of our knowledge.” Epistemic processes organize and structure epistemic communities and, depending on their characteristics and qualities, may give us more or less reason to place our trust in them. Trusting a process means placing faith in its conclusions; without trust, the process loses its capacity to instill new beliefs. Trust alone, however, makes us vulnerable and exposes us to risk; it does not ensure that our beliefs are sound. Accordingly, a process must also be trustworthy. As we have described in this paper, processes exhibit trustworthiness, according to our definition, if they follow norms, conventions, and rules that, generally, are successful in serving their aims. In the case of epistemic processes, this means that they generate reliable knowledge claims. Notice that trustworthiness, alone, without trust, absent communal or societal buy-in, may also be insufficient for knowledge.

Returning to our hypothetical scenario with a further twist, we find that our wayward student, having procrastinated too long on his research project, resorts to his favorite social media application (interchangeably, Instagram, Facebook, Twitter, Snapchat, TikTok, or YouTube). Scrolling through endless reels and feeds of content, he sees posts conveying not only information about his friends and family but also news about an ongoing war in the Middle East. Seeking more information, he expands a post and reads updates about the state of things. In doing so, he is relying, perhaps inadvertently, on a social media company’s algorithmic recommendation as a component in his epistemic discovery. But the communicative process detailing the production and direction of information about war in the Middle East to the student’s phone is not geared, solely, to a sound and trustworthy account of events. The secret algorithmic brews that shape what stories are served to which readers are optimized for user attention and engagement more than the reliability and quality of the producers of news stories or the knowledge and expertise of sources. The same risk holds for the recommendations of search engines, which may also be distorted by the exploits of those wishing to be found as well as the presumed interests of searchers and seekers.

If information is recommended because it attracts attention—measured via clicks, taps, scrolls, and time spent on a site—no one should be surprised that these systems disserve the aims of knowledge and understanding; soundness does not, or does not fully, figure into the process. One should also not be surprised that contradictory or inconsistent claims are amplified simultaneously, by the same platform, because both capture attention. The posts that engagement-driven algorithms tend to elevate and amplify—short, simple, controversial, vitriolic, humorous—may well be true, but if true, only accidentally. In short, if knowledge is the aim, these algorithmic systems of amplification are not trustworthy as epistemic processes.

Often contested and controversial, the domain of news reporting has experienced significant upheaval from the growing dominance of digital media and, even more so, incursions by online platforms. Problematic algorithmic amplification poses a particularly acute challenge, undermining processes that have held sway and have supported the epistemic standing of traditional news outlets (e.g., The Wall Street Journal, The New York Times, Le Monde, The Hindu, Financial Times, The Asahi Shimbun, etc.). Trustworthy news reporting demands adherence to sound epistemic processes, including an array of norms, conventions, and assumptions around key practices of news publication. These include selecting and prioritizing stories, reliable and impartial reporting of facts, clarity of language, and more. News outlets explicitly endorse their commitments through public statements of editorial policies, and in practice, their editorial leadership may implement oversight and enforcement mechanisms. As an illustration, we highlight attribution. Not limited, of course, to the reporting of news, attribution is a centuries-old practice of linking assertions of fact or opinion to the testimonials of others—to sources. How effectively attribution validates these assertions depends on the authority, credibility, expertise, or perspective of sources and the viability of access by the writer, among other factors. In academic research publication, for example, citation is one of the key modes of attribution and is further governed by a host of granular norms, veering to the banal (e.g., whether to list first or last name first, what information to italicize, etc.) Attribution not only serves to validate assertions by citing sources, but it also serves as a means of engendering research integrity, acknowledging priority, and crediting others for innovative ideas.

In news reporting, too, attribution is critical. At the highest layer, disclosing the identity of the reporter not only provides credit but also connects a claim to an individual who is responsible for the article and its account of events. One layer down, good reporting will cite and quote other authors and sources, ideally a plurality of them, identified either by name or by role and frequently, both. The Associated Press, for example, publishes guidelines for reporting on various sources under various circumstances, including cases where it is necessary to keep a source anonymous—these cases require meeting additional standards such as managerial approval, additional corroborating sources, and explanations in the article detailing why the source is credible and their reason for requesting anonymity. These norm-driven practices are not arbitrary. As Hardwig might affirm, they contribute to the integrity of a story, and establish both its soundness, impartiality, and, ultimately, its trustworthiness. Finally, and perhaps most crucially, ideal attribution practices generate transparent lines of accountability from publishers, through authors, to sources.

Analogous practices taking hold on the internet—on webpages, social media sites, blogs, apps, etc.—loosely resemble long-standing attribution practices. Although many webpages are not associated with named authors, hyperlinks may be used to source claims. In fact, the network structure that these links constitute inspired automated internet search algorithms, such as Google Search, to locate useful and authoritative information. But the policies and practices of online platforms have not, uniformly, embodied trustworthy attribution practices, eventually contributing to degraded content. For example, Twitter has apparently penalized posts with external links and blocked links to competing platforms, presumably to reduce user traffic and attention to sources outside of Twitter. This means those who aim to follow nascent norms of attribution (academics, journalists, and others) to signal the trustworthiness of their posts are penalized. Instead, Twitter’s algorithms reward those who “screenshot” catchy visuals or headlines, cut out detailed text or descriptions of method, omit information about the publisher, and remove hyperlinks. When scientific plots and visualizations spread on Twitter and similar platforms now, readers are deprived of the means to locate sources and, ultimately, are less equipped to evaluate them. Platforms that apply algorithmic systems, tuned to user attention and engagement, undermine their own trustworthiness as an epistemic source while simultaneously sowing suspicion of long-trusted and trustworthy sources, including family members, norm-abiding news outlets, or encyclopedias.

Experts have noted that misinformation is endemic to major internet platforms, including YouTube, Facebook, Twitter, and elsewhere. Algorithmic recommender systems, which utilize historical user behavior to profile and draw inferences about user preference, can exacerbate misinformation by personalizing their recommendations. For example, in Hussein et al.’s (2020) study of YouTube’s personalized search and recommendation algorithms, they found that personalization based on watch history increases the prevalence of misinformation appearing in search results and recommended videos. Empirical findings, such as these, demonstrate the ongoing prevalence of fake or false information online but, as we have argued throughout this article, do not address the root causes of the epistemic risks we confront. Our focus should be on problematic algorithmic systems insidiously taking the place of processes that we have relied on (sometimes unconsciously and implicitly) for pointing to knowledge and truth. Experimental hypothesis testing, investigative journalistic standards, peer review, trial proceedings, indexing and references, encyclopedia publishing norms, classroom teaching methods, mathematical proof—these all constitute processes and procedures, typically employed with the aim of generating sound and practical knowledge. Whether explicit and formalized or implicit and taken for granted, the processes we rely upon are fundamental to the proper functioning of legal, industrial, and social systems.

The endpoint of this downward trend is that societies with weakened, fragmented epistemic processes might lose the capacity to distinguish between reliable reporting and disinformation and fail to find common ground among believers of opposing facts—a modern-day Tower of Babel.

Conclusion

The central argument of this paper is that algorithmic amplification is problematic not, primarily, because the content it amplifies is problematic. Instead, algorithmic amplification is problematic because, like an invasive species, it chokes out trustworthy processes that we have relied on for guiding valued societal practices and for selecting, elevating, and amplifying content. Problematic content is a symptom, only partially addressed when disassociated from its underlying causes. Sound processes, continually fine-tuned to illuminate pathways toward valued and valuable societal ends (e.g., from reliable procedures to trustworthy knowledge claims) have inoculated us against processes devised by stakeholders to promote their own or other parochial interests (e.g., cost-effectiveness or profitability). We have argued that in an inexorable progression toward digital life, our societies have become increasingly reliant on computationally encoded algorithmic systems, such as those governing the distribution of content on digital media and platforms, managing voting machines, and automating a host of life-critical decisions, to name a few. We are not advocating for a reversal to an old order that privileges exclusive elites. Nor are we opposed to automated algorithmic systems being embedded in sound processes. Instead, we call attention to what is at stake when processes tuned to the fulfillment of aims and values held in common are nudged aside by algorithmic systems tuned (sometimes stealthily) to different, even incompatible ends.

As the internet increasingly mediates all aspects of our lives, we are likely to have encountered some form or another of problematic content—clickbait, misinformation, extremist ideology, bigotry, among other varieties. This content may have made us feel uncomfortable, distressed, angry, even disgusted. We have argued here that the stakes of problematic algorithmic amplification extend well beyond these experiences. Algorithmic systems have demonstrated astonishing capabilities and have inspired hopes for more informed citizens and more robust democratic institutions. In this optimism, we may not notice that when algorithms nudge aside legacy processes in the name of efficiency and scale, they leave us vulnerable to dogmatic machine oracles or charming present-day sophists. Bad content, we have argued, is the canary in the coal mine, warning of a deeper problem that will not be cured by more diligent moderation or more solicitous recommendation.

A postscript concerning generative AI

While working on this article, generative AI burst onto the public scene and, particularly with the release of ChatGPT, sparked widespread fascination. In print and informal discussions, experts and nonexperts have expressed awe over its capacities to reason, converse, and generate knowledge and information, while pundits and promoters are already predicting revolutionary change in many walks of life, including healthcare, law, finance, and education. At the same time, many commentators have warned of dangers and threats, including threats to the livelihoods of creative artists, libelous and defamatory assertions, privacy violations, and more. We were immediately drawn to the many reports of errors, “hallucinations,” and falsehoods, among other strange and toxic behaviors. Based on our findings, we know that the immediate reaction—to delete or correct falsehood—is to miss the most insidious problem, namely, the gradual extinction of processes that have soundly guided us toward knowledge and sound decision-making.

Our account of problematic algorithmic amplification holds equally urgent lessons for generative AI. In the foreseeable future, as we welcome AI-powered tools across social domains, it will remain necessary to fact-check, corroborate, justify, and moderate the content that these tools produce. This will not be possible unless we maintain sound and strong epistemic processes, robustly sustained by social systems and institutions. Erosion of these processes through diminished support or simple neglect leaves us terribly exposed. When we rely on generative AI at the expense of preexisting processes and allow these sound processes to atrophy, the logical end point of this displacement is a society with unimaginable vulnerability—the loss and weakening of other standards against which to check the veracity or soundness of AI assertions, no counterpoints for students to critically assess ChatGPT’s hallucinations, no independent sources beyond the problematic algorithmic systems themselves, nowhere to turn for answers.

Acknowledgments

This work is supported by a grant from the John D. and Catherine T. MacArthur Foundation and a SaTC NSF grant CNS-1704527. Ben Laufer is additionally supported by a Bowers CIS-LinkedIn PhD Fellowship and a Digital Life Initiative Doctoral Fellowship. We thank the members of the Digital Life Initiative (DLI) at Cornell Tech, the AI, Policy, and Practice (AIPP) group at Cornell University, and the attendees, participants, and organizers of the Symposium on Algorithmic Amplification and Society at the Knight First Amendment Institute at Columbia University. We acknowledge the following people for their comments and suggestions: Robyn Caplan, Sarah Cen, Stephanie May Chan, Amelie Ortiz De Leon, Katy Glenn Bass, Seth Lazar, Smitha Milli, Mor Naaman, Arvind Narayanan, Judith Simon, Mia Speier, Daniel Susser, and Maya Von Ziegesar.

 

© 2023, Benjamin Laufer and Helen Nissenbaum.

 

Cite as: Benjamin Laufer and Helen Nissenbaum, Algorithmic Displacement of Social Trust, 23-12 Knight First Amend. Inst. (Nov. 29, 2023), https://knightcolumbia.org/content/algorithmic-displacement-of-social-trust [https://perma.cc/2TPM-JGWX].