When the Algorithm Accuses You: The Growing Risk of AI-Driven Surveillance and Wrongful Identification

In January 2020, Robert Williams was arrested in his driveway in Farmington Hills, Michigan, in front of his wife and daughters. He was handcuffed, transported to a Detroit detention facility, and held for thirty hours before being shown the evidence against him: a grainy surveillance photo of a man who had allegedly stolen watches from a Shinola store. Williams held the photo next to his own face. "This is not me," he told the detective. The detective reportedly acknowledged the resemblance was imperfect. Williams was released. The case was eventually dismissed. The culprit was a facial recognition system that had matched Williams — a Black man — against the wrong face in a commercial database.

His case is not an outlier. It is a case study in what happens when probabilistic technology is treated as prosecutorial certainty.

The Architecture of Algorithmic Suspicion

Law enforcement agencies across the United States have quietly integrated a suite of AI-driven surveillance tools into their investigative workflows over the past decade. The most publicly discussed is facial recognition, but the ecosystem is considerably broader.

Predictive policing platforms — such as the now-discontinued PredPol (rebranded as Geolitica before its shutdown) and ShotSpotter — use historical crime data and acoustic sensors to generate risk scores for geographic areas or individuals. Metadata analysis tools ingest call records, location pings, financial transactions, and social media activity to construct behavioral timelines. License plate readers accumulate movement histories on millions of vehicles. And increasingly, AI-powered video analytics systems can flag individuals based on gait, clothing color, or behavioral patterns without ever capturing a clear facial image.

The common thread across all of these systems is that they generate outputs expressed in the language of probability — risk scores, confidence percentages, likelihood rankings — which are then interpreted by human investigators who may not fully understand the statistical assumptions embedded in the underlying models.

The Error Rate Problem No One Wants to Discuss

Facial recognition technology has a well-documented accuracy disparity across demographic groups. A landmark 2019 study by the National Institute of Standards and Technology evaluated 189 facial recognition algorithms and found that most produced significantly higher false-positive rates for Black and Asian faces compared to white faces, and that the disparity was most pronounced for Black women.

This is not a fringe finding. It has been replicated across multiple independent evaluations and acknowledged by several technology vendors. Yet deployment has continued to outpace regulatory response.

Beyond facial recognition, predictive algorithms carry their own structural biases. When trained on historical arrest data — itself a product of decades of racially uneven policing — these systems tend to concentrate risk scores in communities that were already over-policed. The algorithm learns from the past and encodes it as the future, creating a feedback loop that is statistically self-reinforcing and practically very difficult to audit.

Michael Oliver, a Detroit resident, was arrested in 2019 after facial recognition software identified him as a suspect in a road rage incident. He was held for ten days before the charges were dropped. Nijeer Parks of New Jersey spent ten days in jail in 2019 after a facial recognition match linked him to a shoplifting and assault case — a match that investigators later acknowledged was incorrect. Parks had to pay $5,000 to secure his release on bail for a crime he did not commit.

Your Digital Footprint as Circumstantial Evidence

Beyond facial recognition, the data trails that Americans generate through ordinary daily life are increasingly being harvested as investigative material — often without a warrant.

Geofence warrants, which compel technology companies such as Google to produce a list of all devices present within a defined geographic area during a specified time window, have been used by law enforcement to generate suspect pools from individuals who simply happened to be near a crime scene. A 2021 case in Florida revealed that a geofence warrant had initially flagged an innocent bicyclist as a suspect in a burglary because his phone had been detected in the area.

Cell-site location data, purchased commercially from data brokers rather than obtained through carriers via legal process, has been used by federal and local agencies to reconstruct the movements of individuals without judicial oversight. Social media monitoring tools scrape public and semi-public posts to build behavioral profiles used in threat assessments. And automated license plate reader networks operated by private companies — including Flock Safety, which operates in thousands of American neighborhoods — create movement records that can be accessed by law enforcement under relatively low evidentiary thresholds.

The cumulative effect is a surveillance environment in which a person's physical movements, online behavior, social connections, and financial patterns can be assembled into a detailed dossier without that person's knowledge, and without the constitutional protections that would apply to a traditional search.

The Legal Framework Is Not Keeping Pace

The Fourth Amendment's protections against unreasonable search and seizure were designed for a world of physical evidence. Courts are still working out how those protections apply to data collected passively by private companies and then purchased or subpoenaed by government agencies.

The Supreme Court's 2018 decision in Carpenter v. United States established that long-term cell-site location data requires a warrant — a meaningful precedent. But the ruling was narrow, and its application to other forms of digital surveillance remains contested. Congress has not passed comprehensive federal legislation governing law enforcement use of facial recognition or predictive policing tools. A small number of cities — San Francisco, Boston, New Orleans — have enacted municipal bans or moratoriums on government facial recognition use, but these are exceptions rather than the rule.

The American Civil Liberties Union and the Electronic Frontier Foundation have litigated several cases challenging warrantless digital surveillance, with mixed results. Meaningful federal reform has repeatedly stalled.

What Individuals Can Do — and What They Cannot

It would be misleading to suggest that individual privacy measures can fully insulate a person from algorithmic misidentification. If a facial recognition system produces a false match, the subject of that match has no advance warning and no mechanism for preemptive correction.

That said, reducing your general digital footprint does meaningfully limit the volume of data available for automated analysis. Disabling location history on mobile devices, using a reputable VPN when browsing, minimizing the use of apps that continuously harvest location data, and auditing the permissions granted to installed applications all reduce the richness of the behavioral profile that could be assembled about you.

Perhaps more importantly, awareness of your rights in the event of a misidentification is essential. If you are questioned or detained based on what investigators describe as a digital or photographic match, you have the right to remain silent, the right to an attorney, and — in many jurisdictions — the right to request disclosure of the investigative techniques used to identify you as a suspect. Organizations such as the Innocence Project and the ACLU's National Security Project have resources for individuals who believe they have been wrongly identified through surveillance technology.

Algorithms do not make arrests. People do. But when those people treat an algorithmic output as a conclusion rather than a hypothesis, the consequences can be devastating — and the legal infrastructure to prevent it remains dangerously incomplete.