I submitted a comment to the California State Bar's Standing Committee on Professional Responsibility and Conduct (COPRAC) on its proposed amendments to the attorney ethics rules related to AI. Given the growing epidemic of attorneys submitting AI-tainted filings in court (which I wrote about here last fall), the proposed amendments would require attorneys (as part of the duty of competence) to stay abreast of the benefits and risks associated with AI, and would make clear (as part of the duty of candor) that attorneys have "the obligation to verify the accuracy and existence of cited authorities, including ensuring no cited authority is fabricated, misstated, or taken out of context, before submission to a tribunal, including any cited authorities generated or assisted by artificial intelligence or other technological tools." In my comments, I say that this is a good start, but that the duties of competence and candor should also address the potential for attorneys to be fooled by deepfakes and unwittingly offer them as evidence (or fail to challenge them when the other side tries to introduce them into evidence), a topic I first started writing about nearly seven years ago.

On March 31, I gave a virtual guest lecture in Mailyn Fidler’s course “The Digital Fourth Amendment” at Harvard Law School. Prof. Fidler invited me to come talk about warrants to AI companies after she read my October 2025 blog post about a warrant issued to OpenAI. As a sort of follow-up to that blog post, I’ve reproduced a portion of my lecture below (with some tweaks to better fit the blog post format).

AI models are used by hundreds of millions of people every day. They have replaced traditional search engines for many users, in addition to becoming companions that some people get very emotionally attached to. The larger the context window for a particular model – the more it can “remember” – the more useful it is to the user… and the more comprehensive the log the AI company is storing about someone’s life, creating a treasure trove for law enforcement. How should we think about AI chatbots and their providers for purposes of digital privacy law?

The SCA Requires a Warrant for AI Users’ Prompts and Models’ Responses

Georgetown Law professor Paul Ohm recently published a paper for Lawfare about the problems posed by reverse searches. In it, he notes that keyword searches and geofence warrants are the two main types of reverse searches right now, but that the list is likely to expand to include AI chatbots, given their popularity and the “gold mine of evidence” they contain about users. Ohm’s argument is that reverse searches probably aren’t authorized under the Stored Communications Act (SCA) and also likely violate the Fourth Amendment – a question on which the Supreme Court will very soon hear oral argument in a case called Chatrie. 

Another Lawfare paper on the SCA, published the same day as Ohm’s, goes a bit further in discussing AI chatbots. Digital surveillance experts Rick Salgado and Stephanie Pell delve into the legislative history of the Electronic Communications Privacy Act (ECPA, of which the SCA is a part) to support their argument that “a search query or prompt to be processed by an AI service for text or image generation” qualifies as “contents of an electronic communication” for SCA purposes, “even if there is no second party or communicant .… The term does not require the involvement of some ‘other’ from whom it is sent or to whom it is conveyed.” Legislative history is also invoked to argue that AI companies should count as, at minimum, RCS providers (as other commentators have argued). Implicitly, I believe they’re arguing that AI companies count as ECS providers too.

To Salgado and Pell’s argument, I will add that there have been numerous ECPA cases involving transmissions between a user and a first-party server – also known as “visiting a website.” Your browser sends info to a server, the server sends data back; it’s a computer, not a human, on the other end of the transmission. For over 20 years, the courts have had zero trouble understanding those transmissions to be “electronic communications.” What counts as “content” has proved more divisive, but things like search queries users type into a search engine, or information users enter into a form provided by a website, have qualified as “contents.” Therefore, if you’re a user interacting with an AI chatbot on your phone or your laptop, those interactions seem very clearly to be “contents of electronic communications.” (Not just the user prompts; the AI’s response to a prompt, which may be text, an image, a video, or a combination thereof, is also straightforwardly “information concerning the substance, purport, or meaning of” an “electronic communication.”) 

So: For SCA purposes, AI model prompts and responses are “contents of electronic communications,” and AI companies count as ECS and/or RCS providers. 

The Federal Government Seems to Think So, Too

Does the federal government agree? Looking at the warrant to OpenAI that I wrote about last fall, I think it does. In the affidavit filed in support of the search warrant application, paragraph 4 says: “This Court has jurisdiction to issue the requested warrant because it is “a court of competent jurisdiction” as defined by 18 U.S.C. § 2711. See 18 U.S.C. §§ 2703(a), (b)(1)(A), & (c)(1)(A).”

Those citations are to the SCA: Section 2711 is the SCA’s definitions section; section 2703 deals with compelled disclosures to law enforcement of customer communications (both content and non-content information). 2703(a) is about “contents of wire or electronic communications in electronic storage”; 2703(b) is about “contents of wire or electronic communications in a remote computing service”; 2703(c) is about “records concerning [an] electronic communications service or remote computing service.”

These citations indicate to us that the government – or at least, the Special Agent with Homeland Security Investigations (HSI) making this affidavit – thinks of OpenAI as being subject to the SCA, and that the government thinks ChatGPT’s prompts and responses are contents of electronic communications in electronic storage. However, the affidavit is kind of cagey about whether the government considers OpenAI to be an ECS provider or an RCS provider; the affidavit simply tosses off a string citation to Section 2703(a), (b), and (c) all in a row. At minimum, the government seems to think OpenAI counts as either one or the other, and possibly both. (The distinction has long been critiqued as unworkable in the modern Internet age.) 

That said, maybe I’m reading way too much into this language. It is entirely possible that the affiant just took his usual warrant affidavit template for stored communications (e.g., emails, cloud storage files) and copy-pasted the language into the OpenAI warrant application, without thinking about it too much. It’s also possible that, sitting in a field office up in Portland, Maine, the HSI agent here didn’t consult with any D.C.-based higher-ups. Agency leadership typically likes to have a say in whether to try out a novel type of legal process for electronic surveillance — such as a reverse prompt warrant to an AI company under the SCA. Perhaps boilerplate language in one paragraph of a warrant affidavit is too thin a reed to bear the weight I’m giving it.

Still, whatever the backstory, on the face of it this OpenAI warrant was sought under the SCA. Absent any other contextual information, we may as well treat that as a concession by the federal government that the SCA applies to AI companies, and that means “get a warrant.” 

The latter conclusion is further bolstered by the fact that another part of the federal government, the Federal Bureau of Investigation (FBI), was recently revealed to have served a warrant to xAI and thereby gotten a suspect’s prompts to xAI’s Grok AI tool. (This was not a reverse warrant; unlike in the OpenAI case, the suspect’s identity was already known, so the FBI could specify whose account they wanted.) The warrant to xAI is sealed, so we don’t know what authority (i.e., the SCA) it invoked. Nevertheless, it is another indication that federal government policy when seeking AI user data is to get a warrant.

AI Companies Do, and Should, Stand Up for Users’ Privacy

Of course, they had little choice, really. OpenAI’s policy for government requests for user data says, “OpenAI US … only discloses requested user content to a law enforcement request in response to a valid warrant or equivalent.” That’s a standard requirement for tech companies to impose on government demands post-Warshak. (xAI’s policy is more ambiguous, requiring “appropriate legal process such as a subpoena, court order, or warrant.”) 

It is important and meaningful that OpenAI has staked out this “get a warrant” position from the get-go, while the number of government demands for user data that it receives still remains shockingly minuscule for a company that counts more than one-tenth of the Earth’s population as weekly active users. As Harvard Law now-3L Jackie O’Neil wrote in her 2025 essay about geofence warrants (before the Supreme Court’s grant of cert in the Chatrie case on that topic):

While courts wrestle with options for sophisticated legal regulation of [reverse warrants], innovation outside of the criminal procedure context may be a stopgap. Technology companies wield ultimate control over [reverse] warrants’ efficacy. … Without means to compel … private companies to retain or organize their data, law enforcement agencies are at the mercy of large private companies with respect to [reverse warrants].

Put simply, AI companies already have a big role to play in protecting users’ digital privacy. AI tools like ChatGPT have become runaway successes with gargantuan user bases, long before the courts have had an opportunity to apply the niceties of ECPA definitions to them and the data they have about their users. Deciding in advance to demand a warrant for the troves of personal data they hold, and then communicating that policy to law enforcement and the public, is a way of setting norms and expectations, ensuring internal practice consistency, gaining public trust, and (hopefully) preempting government attempts at shenanigans. 

Protecting User Privacy Means Saying “No” to Overbroad Reverse Prompt Warrants

What remains to be seen is whether OpenAI (and its brethren) will push back if and when shenanigans do happen. The warrant to OpenAI sought only one specific user’s account, but if OpenAI could locate one account that entered a particular prompt and got a particular response, that implies the ability to return a list of multiple accounts that entered a particular prompt (though this would violate its requirement that legal demands “unambiguously identify the user account(s) at issue”). Like I said in my original post, we don’t know OpenAI’s precise capabilities with regard to its gargantuan data stores, but they seem to be quite considerable.

AI companies should keep in mind that it’s in their interest, not just that of their users, not to let overbroad reverse prompt warrants become the new geofence warrants, whose constitutionality will soon be decided in Chatrie. Once Google started complying with them, geofence warrants ballooned to constitute over a quarter of all warrants Google received in the U.S. Eventually, Google changed how it stores user location data so that it couldn’t comply with those warrants anymore. There’s a lesson there for AI companies’ in-house counsel (many of whom came from Google, as it happens). 

I am skeptical that the result in Chatrie, whatever it may be, will definitively settle the constitutionality of reverse warrants in the AI context to the satisfaction of all involved (including AI companies, their users, law enforcement, and judges). After all, following the Court’s last big digital Fourth Amendment decision in 2018, many lower courts elected to read the decision narrowly when considering other flavors of digital surveillance. And anyone to whom the Chatrie outcome proves unfavorable will have an incentive to split hairs. 

We can anticipate litigation in the years to come over how Chatrie applies to AI. But that only makes it all the more important for AI companies to proactively adopt a robustly privacy-protective stance in the here and now. The major AI companies will rapidly become very powerful evidence intermediaries. We, the general public and the users of those companies, also have power. We can influence how the big AI companies respond to novel law enforcement demands. Consider how the number of downloads of Claude surged after Anthropic stood up to the Department of Defense over issues including the mass surveillance of Americans. That’s the kind of signal from users that’s hard for these companies to ignore. It wasn’t that long ago that it was in vogue for tech companies to loudly stand up for their users’ rights. It could become cool again. We can help make that happen.

[Watch on YouTube]

Thank you for your invitation. I’ll offer seven points.

First: American driving is dangerous.

Automated driving could help, if we’re careful about it. But people are dying today not because we’re careful about automated driving but, rather, because we’re careless about road safety generally.

Other countries do care. Driving in the US is twice as deadly as in Canada and Australia. As a South Carolinian, I’m ten times more likely to die in a crash than my friends in the UK. Ten times.

These countries aren’t hiding some vast secret fleet of AVs. I can’t yet hail a robotaxi in London. But I can cross the street.

Second: Arrogance is careless.

AVs have tremendous potential. But believing they will be a panacea virtually guarantees they won’t, because that confidence blinds us to risks.

Many engineers working on AVs show humility. They talk with me about what’s hard, what went wrong, and what’s uncertain. They want to learn from local officials. I wish AV companies would show more of this humility in their PR.

Third: The best proxy for the safety of AVs is the trustworthiness of AV companies.

There are no “self-driving” or “driverless” cars. The companies that develop and deploy AVs are the drivers. This means that an AV is only as safe as the companies responsible for it. We can and should proactively assess their trustworthiness.

AVs won’t be perfect, but a company can still do right after its technology fails. It can explain what went wrong, how it’s addressing the actual harm, how it’s reducing future risks, and—critically—what it’s learned more broadly. We need more of this.

Doing right does not mean forcing victims into arbitrations, and it does not mean buying their silence and thereby misleading the public. These are betrayals of trust.

Fourth: Safety is a marriage, not a wedding.

Safety is a lifelong commitment that continues as long as an AV is on the road. It’s not just a one-time test or certification or checklist. A credible safety case must be a living document that is clearly supported, robustly interrogated, and routinely updated.

Vehicles placed on our roads stay there for decades and therefore need oversight for decades. NHTSA provides some of this oversight, and AVs will dramatically expand the scope of it. Yet both NHTSA and FMCSA are tiny, underresourced agencies with huge mandates.

Fifth: AVs are an especially visible part of a much broader discussion of AI.

As a society we’re likely to place many of our hopes and fears about AI generally on AVs specifically. The Transforming Transportation Advisory Committee, which I vice-chaired, addressed many of these issues, including employment, accessibility, sustainability in the face of climate change, privacy vis-à-vis both companies and governments, and fundamental questions of power. For each, we need clear policy goals and an iterative approach to achieving them.

Sixth: Local government has essential expertise.

Few appreciate how much local governments are subsidizing automated driving. First responders, for example, solve all kinds of problems, from waking up people in robotaxis to literally moving AVs that are stuck.

These local officials deserve our respect. They want the people who remotely assist AVs to be in the US. They need to know AVs will help rather than hurt their response to disasters. They want to be able to ticket AV companies for moving violations just as they would any other driver. They emphasize that every city is unique and AVs must operate accordingly.

Finally: We must empower, not disempower, our communities.

Preempting state and local authority would be profoundly short-sighted—and I say this as someone who believes strongly in the potential of AVs.

Many states want the federal government to lead on AV policy. But great leaders lead. Telling US DOT what to do (and providing the resources needed to do it) would help much more than telling states what not to do.

Preemption could create litigation rather than certainty. It could bar states from getting unsafe vehicles and unsafe drivers—human or otherwise—off the road.

Our AV industry started through federal research decades ago and then grew through our system of federalism. Brand America does have a serious credibility problem abroad, but preemption does not solve it.

In a scary time of technological change, we need to make sure that communities, and the people in them, have control and feel in control. We can deploy both technology and policy in a way that protects and empowers them.

Thank you.

[See also answers to senators' written questions]

On December 11, 2025, President Trump signed an executive order (EO) that purports to deprive states of the ability to regulate artificial intelligence (AI) – to the modest extent possible given the limited power of EOs, which cannot require or forbid states to do anything. (Whether specific state AI laws will founder under other authorities, such as Section 230 or the First Amendment, is a separate question beyond the scope of this blog post.)

There is plenty to say about this EO, much of it unflattering (since it is very bad). However, one positive aspect is that the EO appears to tell the executive branch not to hassle states for cracking down on AI-generated child sex abuse material (AI-CSAM), the topic of dozens of state laws enacted in recent years. And because that “hands off” direction extends to online child safety more broadly, it looks like states will also be able to turn their attention to the red-hot topic of AI chatbots’ safety risks for children without drawing White House ire. (Again, a caveat that other authorities might still doom those state laws; also, laws purporting to protect children online are often terrible ideas, whether this administration approves of them or not.) 

Section 8(a) of the EO tasks two senior administration officials with proposing a legislative recommendation for how to preempt state-level AI laws. There are significant changes to this section since the draft of the EO that leaked last month. Newly added is Section 8(b), which provides a list of carve-outs: 

“(b) The legislative recommendation called for in subsection (a) of this section shall not propose preempting otherwise lawful State AI laws relating to:

(i) child safety protections;

(ii) AI compute and data center infrastructure, other than generally applicable permitting reforms;

(iii) State government procurement and use of AI; and

(iv) other topics as shall be determined.”

Nominally, Section 8(b) only modifies Section 8(a) – the legislative proposal for preempting state AI laws. Elsewhere in the EO, Section 3 requires the Attorney General to create an “AI Litigation Task Force (Task Force) whose sole responsibility shall be to challenge State AI laws inconsistent with” White House AI policy (namely, “to sustain and enhance the United States’ global AI dominance through a minimally burdensome national policy framework for AI”). Section 4 calls for the creation of a list of litigation targets, i.e., state AI laws that conflict with that policy; Section 5 directs the withholding of federal broadband funding from states unless they regulate AI the way the White House wants them to. 

None of those sections contains the same limiting language that Section 8(b) imposes on 8(a). Nevertheless, I read that list of carve-outs as implicitly a statement of policy that carries over to the previous sections. That is, I find it highly unlikely that states’ AI-CSAM laws will be deemed to conflict with the EO’s stated policy (Section 4) or the federal FTC Act (Section 7), that the AI Litigation Task Force will sue all those states for having passed laws prohibiting AI-CSAM (Section 3), or that those laws will be implicated in the carrot/stick game of federal broadband funds (Section 5). This is for several reasons.

One, AI-CSAM prohibitions are perfectly consonant with the EO’s stated AI policy anyway. As said, that policy is “to sustain and enhance the United States’ global AI dominance through a minimally burdensome national policy framework for AI.” By and large, state AI-CSAM laws and bills target AI models’ outputs, not the models themselves. They will be enforced against the end user who creates and/or shares AI-CSAM using a generative AI tool, not the entities that provided the tool (though we are starting to see laws and bills targeting nudifier app services). Criminalizing AI-CSAM doesn’t hamper the business of OpenAI, Meta, Google, et al., nor the development of open-source models by nonprofits or academia. 

In fact, federal government policy on AI-CSAM is already very clear. Congress passed, and the President signed, the TAKE IT DOWN Act earlier this year. The law outlaws nonconsensual deepfake pornography whether it depicts adults or minors, with both criminal and civil liability. Plus, using a computer to create CSAM of real kids has been a federal crime for almost three decades already, as I cover in this paper.

Second, for this administration to sue states to try to invalidate their child-protection laws is what seasoned political experts, in their specialized jargon, would call “a completely dumbfuck idea.” “Child safety” is already a sacrosanct topic, as I’ve learned from a decade working on tech policy. Invoking “child safety” tends to shut down nuanced discussion and rational thought, and it makes opposing even bad bills very difficult politically, as voicing concerns gets you called a pedophile. As one of the state legislative staffers interviewed for my recent AI-CSAM paper remarked, “Nobody objects to trying to protect children.” 

There would be terrible optics to the Department of Justice (DOJ) going after states for CSAM- or other child safety-related laws. That would be true at any time, but it is particularly ill-advised now. The Epstein files (of which another batch are in the news today) have proven remarkably resilient as a topic preoccupying the American public, even among Trump’s own base. Even Trump loyalists in the GOP voted to release the files. The President’s approval rating is in the toilet. He is not going to tell Pam Bondi to give Gavin Newsom more ammo against him. 

Third, executing the EO will require resource management. There are only so many federal government employees available to carry out the EO – especially since this administration has illegally fired so many federal workers and redirected so many others to helping kidnap nannies and gardeners, with a measurable impact on child safety investigations. Whoever works on implementing the EO will have only so many hours in the day, and with six different sections of the EO creating various workstreams, it’s possible some employees will end up on more than one. I assume the number of personnel carrying out the EO will be relatively modest, at least compared with the number tasked with immigration enforcement. Or redacting the Epstein files. 

As said, there are dozens of states with AI-CSAM laws. And CSAM is just one topic; states are proposing and passing way more bills on AI besides that. For the federal government to challenge every state AI law simply does not scale. That means the carve-outs listed in Section 8(b) – which allows for “other topics as shall be determined” – are an operational necessity. It’s also why I think Section 8(b)’s list will, in practice, be read to apply to the previous sections too. The EO tasks various agencies and individuals with time-sensitive projects; as they plan their work, they can now deem the Section 8(b) topics out-of-scope. Whereas many states have “State AI laws relating to child safety protections,” far fewer have the kinds of big-picture AI governance laws, like those passed by Colorado and California, that are likely to be the highest-priority targets under this EO. The 8(b) carve-outs free up employees’ limited time and resources to focus on those priorities. (To be clear, I’m not saying I agree with the EO – only that triage is unavoidable for those tasked with implementing it.)

Finally, it bears noting that the wording of Section 8(b) exempts state AI laws relating to “child safety protections,” not AI-CSAM in particular. I suspect that this wording is intentionally broad so that it encompasses not just AI-CSAM legislation, but also other online child safety bills (in past, present, or future sessions) – of which there are a lot – that incidentally or explicitly cover AI, not just social media, gaming, and the like. 

Child safety has been a predominant issue in the federal and state legislatures alike for several years running, and even if state laws addressing that topic might be “burdensome” on AI companies, it is a hard sell politically to let only AI companies get a hall pass from compliance. That’s why child safety was also a carve-out from some versions of a proposed federal legislative moratorium on state AI laws earlier this year. The House passed a moratorium (sans child safety exemption) in its version of the One Big Beautiful Bill Act, but the Senate ultimately removed it, prompting this small-beer simulacrum from the executive branch.

What’s more, AI chatbots are the latest, hottest topic in online child safety right now, following multiple high-profile cases of teen suicides allegedly related to chatbot interactions. States are already starting to enact legislation, and I believe AI chatbots will be as popular a legislative topic in the next few state legislative sessions as AI-CSAM was in the last few. After all, “Nobody objects to trying to protect children.” 

To that end, the EO’s child-safety carve-out may operate as a limit on EO execution strategy – even though, again, it’s found only in Section 8, not other sections. In practice, I predict that the “child safety protections” language in Section 8(b) will be read by the AI Litigation Task Force, the Federal Trade Commission, et al. as an instruction not to get in states’ way as they start to regulate AI chatbots with respect to child safety, even if those regulations would otherwise be squarely in the EO’s sights. For example, Section 7 of the EO (evoking July’s “Woke AI” EO) frowns on “State laws that require AI models to alter their truthful outputs.” Technically, that would include, say, a bill that would prohibit chatbots from providing true information to under-18 users about how to make a noose, like ChatGPT allegedly gave to a teenager before his suicide. Under the EO’s child safety carve-out, the administration is not likely to challenge a bill like that (or a part of a bill, or the enforcement thereof by the state). 

Lastly, to the extent that some states do meekly go along with this EO instead of standing up for themselves (I thought “states’ rights” were sacrosanct?), the child-safety carve-out also serves as a green light to the states. Even though the EO is plainly meant to scare them out of passing and enforcing new laws regulating AI, it’s a sign that they can pass AI laws governing CSAM, chatbots, and so on, and the administration won’t punish them for it. 

In conclusion, attacking states’ right to regulate AI and protect their own residents is abhorrent. But there are more than 50 states and territories, and their legislatures seem determined to ensure that their constituents are benefited by AI, not harmed by it. Even GOP-led state governments are standing up against the new EO. And no wonder: not only do they have to protect their own right to govern, but also, they’re listening to their constituents’ desires. More AI regulation is what Americans want. This EO is not going to stop that from happening.

Earlier this week, the indefatigable Thomas Brewster at Forbes, a journalist who’s been covering the digital surveillance beat for years, reported on a search warrant to OpenAI seeking to unmask a particular ChatGPT user. Brewster says it’s “the first known federal search warrant asking OpenAI for user data.” (Note the “known”: similar warrants might still be under seal like this one was until recently.) The warrant’s supporting affidavit describes a long-running child exploitation investigation into multiple dark web sites hosting child sex abuse material (CSAM), whose administrator the government sought to identify. (It has now arrested a suspect.)

So what’s the link between various dark web sites and OpenAI? Well, like 800 million other people, the sites’ administrator is allegedly a frequent ChatGPT user. As the affidavit describes, an agent from Homeland Security Investigations (HSI) had been chatting undercover with the admin, who described to the agent his ChatGPT usage, including specific prompts and partial or full responses. Based on that information, the government sought and obtained a warrant to OpenAI for (per Brewster) “various kinds of information on the person who entered the prompts, including details of other conversations they’d had with ChatGPT, names and addresses associated with the relevant accounts, as well as any payment data.” The affidavit includes two “unique, specific” prompts and the “unique responses” that ChatGPT generated. OpenAI apparently complied with the warrant in the form of an Excel spreadsheet of data. (We don’t know what’s in the spreadsheet.)

This warrant may be a first, but it’s not a surprise. User data is “if you keep it, they will come”: If a tech company stores data about its users, in a form the company can legibly access (i.e., not end-to-end encrypted, or E2EE), it will receive a government request for some user’s data eventually… and then for more users after that. And the more data the company stores, the richer the trove for investigators to tap into. (The day after his ChatGPT story, Brewster reported on how HSI used WhatsApp user data in an immigration-related investigation.) That’s why, as digital rights attorney Jen Lynch commented in the ChatGPT article, “it’s more important than ever for OpenAI and other AI companies to think about how to limit the amount of data they collect on their users.”

This warrant may be the first, but it won’t be the last. OpenAI’s transparency reports (each of which is a whopping one page long) say it got a total of 71 government requests for user data in the second half of 2024 (the latest reporting period available), after receiving only half that many in the first half of that year. That number is still small, but it will keep increasing, and this news is surely going to add to the load. Once word gets out that a company can provide various information about its users and will comply with valid legal process, other investigative agencies will follow suit, and the volume of requests goes way up. 

That’s what happened with so-called “reverse warrants” to Google for keywords users entered into Google Search and for users’ location history in Google Maps. It got so bad that Google stopped storing Android users’ location history. The company also pushes back on at least some reverse keyword warrants. 

Reverse warrants pose major constitutional problems because they’re inherently overbroad: they seek information on everybody who was in a particular place during a particular timeframe, or everybody who searched for a particular query. That flies in the face of the Fourth Amendment, as former federal magistrate judge Brian Owsley explains in a recent article in the Stanford Technology Law Review. As his opening says: “Traditional law enforcement warrants begin with a suspect and, supported by a finding of probable cause, seek additional evidence about that person. Keyword search warrants reverse this process.” The ChatGPT warrant news immediately raised concerns that a new chapter in the tawdry saga of reverse warrants has just begun, with OpenAI replacing Google as the main character. 

True, as Lynch noted, this particular warrant looks properly scoped to seek information about one specific ChatGPT user, for whom the affidavit provided ample information to support probable cause linking that user to the CSAM sites. The warrant was not casting a dragnet for every unknown user who entered a query without any reason to suspect each of them individually, the way reverse keyword warrants do. This warrant also makes false positives highly unlikely by limiting the request to whichever account entered one specific prompt and received two specific and lengthy responses to prompts. That’s different from, say, a reverse keyword warrant for information on every person who googled a particular street address, which could cover a suspected arsonist but also anyone from party guests to a taxi driver to a pizza delivery guy. 

Nevertheless, we should take this warrant to OpenAI as a harbinger of more to come. That this warrant application was properly scoped doesn’t mean the next one will be. And even if they’re all properly scoped, there are going to be more of them, and it’ll take some amount of work for OpenAI to deal with them. Therefore, here are some questions we should be asking about user data requests to OpenAI (and its ilk) going forward:

1. Judges are the gatekeepers who are supposed to reject unconstitutional or otherwise faulty warrant applications. Will judges prove to be as willing to authorize reverse warrants over ChatGPT prompts – let’s call them “reverse prompt warrants” – as so many of them have been for reverse keyword warrants and reverse location warrants?
2. Given the novelty of reverse prompt warrants, will prosecutors judge-shop until they get a friendly judge who’ll sign off on one that (unlike this warrant) isn’t narrowed down to one specific user? Which courts are most susceptible to judge-shopping?
3. OpenAI says warrant responses include “content data, such as text or files input or output in connection with the use of OpenAI services.” That sounds like OpenAI can produce both users’ prompts and ChatGPT’s responses – but what if the warrant doesn’t specify an account? What capabilities does OpenAI have to search for a specific prompt and/or response that surfaces results tied to specific accounts’ chat histories? How far back can it do that, and how technically burdensome is it? Is there any variance for free vs. paid plans? 
4. OpenAI’s Law Enforcement User Data Request Policy currently states that a request must “include sufficient information to unambiguously identify the user account(s) at issue.” How will OpenAI apply this policy to reverse prompt warrants that correspond to multiple users’ accounts (rather than just one user, as with this warrant)?
5. At a time when the federal government and various state governments are cracking down on people and activities they disfavor, will OpenAI move to quash a warrant for a user’s account if the warrant specifies prompts relating to First Amendment-protected activity (such as protesting)? How about “crimes” like abortion, gender-affirming care, or lack of legal immigration status?
6. When responding to legal process from the federal government, do political considerations influence OpenAI’s response, given CEO Sam Altman’s close ties to the Trump administration (which now involve the phrase “weapons-grade plutonium”)?
7. Will OpenAI move to quash legal process if there’s a gag order that keeps the target user(s) in the dark so they can’t move to quash on their own? Has it ever fought to get a gag order lifted?
8. What technical changes could OpenAI make so that they would be unable to comply with a reverse prompt warrant, like Google did with location history?

If OpenAI’s numbers are to be believed, one out of every ten people on Earth already uses ChatGPT at least once a week. In that sense, it’s remarkable that the volume of user data requests the company receives is as low as it is – and that it took this long for the first known reverse prompt warrant to arrive. As said, this one seems OK, but OpenAI needs to be ready for future ones to try to push the envelope. Navigating an increasing volume of requests from governments is a normal part of a startup tech company’s growth. But the larger its user base, the more important it is for the company to stand up for its users’ rights – and to do so vigorously straight out of the gate. Frankly, AI as a replacement for traditional web search has been disappointing enough already. We don’t need the AI version of superabundant unconstitutional reverse keyword search warrants too.

It seems like every day brings another news story about a lawyer caught unwittingly submitting a court filing that cites nonexistent cases hallucinated by AI. The problem persists despite courts’ standing orders on the use of AI, formal opinions and continuing legal education (CLE) courses on ethical use of AI in law practice, and revelations that AI-powered legal research tools are more fallible than they purport to be. 

Who are the attorneys submitting AI-tainted briefs? A recent 404 Media article about lawyers’ use of AI drew my attention to a database of AI Hallucination Cases compiled and maintained by Damien Charlotin, a French lawyer and scholar. Charlotin classifies the nature of the incident by various types of inaccuracies: fabricated cases, false quotes from or misrepresentations of real cases, or outdated invocations of cases that have been overturned. Besides helping the public understand how lawyers are getting tripped up by AI, Charlotin’s database also enables a better view of who is getting tripped up by AI.

Using the database, I analyzed 114 cases from U.S. courts where, according to either opposing counsel and/or the court’s own investigation, an attorney’s filing included inaccuracies that were suspected or shown to have been caused by the use of AI. I find that the vast majority of the law firms involved – 90% – are either solo practices or small firms. What’s more, in 56% of the cases, the AI hallucinations were attributed to the plaintiff’s counsel, compared with 31% to the defense. And, while most cases in the sample did not specify the AI tool used, of those that did, fully half involved some version of ChatGPT. 

Methodology

I based my analysis on cases I downloaded in a .csv file from Charlotin’s database on October 9, 2025. The time period covers court orders issued from June 2023 (the month of the landmark order in Mata v. Avianca) through October 7, 2025. 

[Note: October 9 was a Thursday; by the following Monday, when I began drafting this write-up, Charlotin had added three new matters involving pro se litigants (which I exclude from my analysis) as well as two updates on cases that were already in the database (and thus already in my sample), plus there was news coverage of an oral argument where an attorney was grilled about hallucinations in his briefing. I did not add that last matter, which had not yet yielded a written opinion at the time I wrote this, to my sample. This is all to give the reader some idea of just how frequently these incidents are happening and consequently to highlight that my sample should not be considered comprehensive – the data became outdated almost immediately.]

Charlotin has helpfully coded the data by a number of selectors including country. I restricted my download to the USA only. After importing the .csv file into Google Sheets, I then filtered by Party to include all cases involving a Federal Defender, Lawyer, and/or Paralegal. (Prosecutor is also an option in the database, but there were zero such cases in the USA for that time period.) This resulted in 117 cases, which fell to 114 after I excluded three cases from the sample (two cases that actually involved pro se litigants rather than lawyers and one duplicate case). (EDIT 10/17/25: Note that my choice to exclude cases tagged as Expert meant that my sample excluded some high-profile matters such as Concord v. Anthropic.)

I reviewed the court orders that Charlotin included for each database entry in order to determine which party was accused of submitting hallucinated citations and the name and law firm affiliation of the attorney(s) for that party. If that information was unavailable in the court order, I looked up the case docket in federal or state court records. 

Most of the cases in the sample are typical adversarial matters where the parties can be classified as either plaintiff or defendant. For matters that fall outside that usual structure (such as bankruptcy cases), I created an “other” category. Where the court order came from an appellate case, I tried to classify the party as plaintiff or defendant as per the parties’ trial-court posture.

My data for the number of attorneys at each firm came from either the firm’s website or some other authoritative source (such as the NALP, Vault, or Law.com). I sometimes had to guess that an attorney was solo, typically where the attorney does not have a dedicated website and the firm name listed in court records, if any, is indicative of a solo practice (e.g., “Law Office of Jane Smith”). 

For firm size, I have used the following bands: solo; 2-25; 26-100; 101-200; 201-500; 501-700; 701-1000; 1001+. These are the bands the NALP uses for its Directory of Legal Employers, except that it uses “1-25” as a band. I chose to split out solo attorneys as a separate category because I believe solo attorneys deserve recognition as a standalone group with unique characteristics that differentiate their practices from firms of 10 or 20 lawyers. I added a “government” category for the rare cases involving government attorneys (two: a public defender and attorneys for a county), but did not attempt to count how many attorneys were part of that particular government unit.

There may be errors in my data, thanks to having to guess about some things (such as whether someone is a solo practitioner) or relying on inaccurate or outdated sources (for example, third-party reporting on firm size). If you find an error, please email me (riana at stanford dot edu) and I’ll fix it and update this post.

The Party Submitting Hallucination-Tainted Filings Is Usually the Plaintiff

The plaintiff is more commonly the party allegedly responsible for submitting filings containing AI hallucinations. Out of 114 cases, 64 were attributed to the plaintiff (56.1%), compared with 35 to the defendant (30.7%). There were 15 “other” cases (13.2%): bankruptcy, family, probate, and tax court matters, agency matters, a habeas petition, and an attorney disciplinary proceeding. (The lawyer allegedly submitted filings with AI hallucinations during that disciplinary proceeding, not in an underlying case involving that lawyer like other disciplinary proceedings in the sample. Where the attorney was facing discipline for misusing AI while representing a client, I classified the lawyer according to the party they were representing in the underlying case.)

AI Hallucination Cases Overwhelmingly Involve Solo or Small Firms

Some of the 114 cases in the sample involved attorneys from more than one firm – for example, local counsel filing briefs drafted by a different firm. I counted each firm separately, except that if the court’s order faulted only one firm’s attorney, I did not count the other firm(s). The total number of firms (including government entities) was 129. 

Solo practices and small firms represent the overwhelming majority of that number. Solos account for half (50.4%) and small firms of 2-25 lawyers for another 39.5%. Of the remaining 10% of firms, 3.1% are firms of 26-100 lawyers; 2.3% are firms of 201-500 lawyers; 1.6% are firms of 1001+ attorneys; 1.6% are government entities; and firms of either 101-200 or 501-700 lawyers each represent less than 1%. There were no cases involving firms of 701-1000 lawyers.

The number of firms in the sample with more than 25 lawyers is small enough to count on two AI-generated hands. Four have up to 100 lawyers: Ellis George, Hagens Berman Sobol Shapiro, Merlin Law Group, and Williams Kastner. Five have 101-700 lawyers: Butler Snow, Goldberg Segalla, Morrison Mahoney, Quintairos Prieto Wood & Boyer, and Spencer Fane. Two have more than 1000 attorneys: K&L Gates and Morgan & Morgan.

Five lawyers are implicated in more than one case in the sample. All are either solo practitioners or small-firm lawyers: solo Maren Miller Bam of Salus Law; Jane Watson of Watson & Norris (who was only admitted to the bar in 2024); Chris Kachouroff of McSweeney Cynkar & Kachouroff (who gained notoriety for appearing pantsless at a Zoom court hearing); solo Tyrone Blackburn (who got arrested for assault in June in connection with a different case of his); and William Panichi, a family-court attorney. While the first four allegedly misused AI in two separate cases, Panichi was called out in an astonishing four cases in one 30-day period; he has supposedly begun winding down his law practice and surrendering his license.

ChatGPT Was the Most Commonly Used AI Tool

Of the 114 cases in the sample, only 34 (30%) identified the specific AI tool(s) used by the attorneys. Some cases involved the use of more than one AI tool. OpenAI’s ChatGPT (any version, including in-house versions and the ChatGPT-powered app Ghostwriter Legal) was far and away the most common: it was implicated in fully half (18) of the 34 cases that specified a tool. Coming in a distant second were AI tools offered by Westlaw, followed by Anthropic’s Claude (any version), Microsoft Copilot, Google Gemini, and LexisNexis’s AI tools. 

Discussion 

This analysis confirms what many lawyers and judges may have suspected: that the archetype of misplaced reliance on AI in drafting court filings is a small or solo law practice using ChatGPT in a plaintiff’s-side representation. 

Ultimately, the buck stops with the attorney to make sure that she can stand behind every word of every brief filed over her signature. But the 404 Media article that led me to Charlotin’s database paints a picture of how hard it is to live up to that obligation, particularly for solo or small-firm attorneys. Lawyers struggle with busy caseloads, the trustworthiness of their co-counsel, junior attorneys, and support staff, and personal issues (health problems, caregiving obligations, etc.) that compete with work for their time and attention. Of course, that was already true long before AI. Lawyers, even very good ones, have always made the occasional mistake or oversight in their work. AI tools have merely provided a new way to make those errors – while also promising a way out of the underlying issues that contribute to them, like time crunches and insufficient support. As the 404 Media article observed, “the legal industry is under great pressure to use AI.” To overworked attorneys at small law offices, these tools must seem like a godsend.

However, as the lawyers in this analysis learned the hard way, these tools are not reliable for their core purpose of accurate, comprehensive legal research results. Several of my Stanford colleagues are coauthors on a recent paper that investigated AI legal tools’ claims to be “hallucination-free” or to “eliminate” or “avoid” hallucinations. To the contrary, they found disturbingly high levels of hallucinations in all the tools they studied: OpenAI’s GPT-4, Lexis+ AI (offered by LexisNexis), Westlaw’s AI-Assisted Research, and Ask Practical Law AI (which, like Westlaw, is owned by Thomson Reuters). All of those companies are represented in the 34 cases analyzed above.

The incidents in Charlotin’s database illustrate the real-world impact of AI legal tools’ shortcomings – and not just on the lawyers, who end up humiliated and sanctioned for relying on tools they thought were reliable. AI-tainted legal briefs negatively affect those lawyers’ clients, who depend on them for high-quality representation, including in incredibly high-stakes matters such as criminal prosecutions or the termination of parental rights. They affect opposing counsel, who must waste their time tracking down nonexistent case citations. And they affect the courts, which are busy enough already without also having to police this new form of attorney ethics violations and take care not to let nonexistent cases cited by counsel creep into court opinions. 

What Is To Be Done?

These cases keep happening at an alarming pace. Dozens of cases have been added to Charlotin’s database since the American Bar Association (ABA) issued its formal opinion warning about generative AI tools in July 2024. For all the news stories about lawyers caught flat-footed by these tools, clearly there are lawyers who never read them and subsequently become the headline of the next one. It may be that nothing will sufficiently penetrate lawyers’ consciousness about the pitfalls of relying on AI tools until every practicing lawyer is personally confronted with that knowledge through some combination of (1) every single type of court – federal, state, tribal, agency; civil, criminal, bankruptcy, family, probate, you name it – requiring every lawyer who appears in every case to file a declaration attesting that they understand and acknowledge the fallibility of AI tools and have educated all their staff as well, and (2) every single state bar (including D.C. and U.S. territories) imposing CLE requirements specifically about AI tools for legal research, like they now do for topics like substance abuse and elimination of bias. 

Even then, there will be failures. Inevitably, some lawyers will dutifully certify that they understand that AI tools are unreliable, then file an AI-tainted brief anyway. But perhaps the incidence of lawyers sanctioned for unwittingly misusing AI will slacken with time and more pervasive awareness of AI’s perils. And hopefully AI legal research tools themselves will improve over time (as their paying customers surely expect them to) – though it is as unreasonable to expect perfection from them as from humans. “Trust, but verify” must remain the watchword.

With all that said, no amount of CLE courses and state bar ethics opinions will fix the problem I haven’t discussed until now: use of AI by pro se litigants. I wanted to figure out which lawyers were getting tripped up by AI, so I only analyzed U.S. cases involving lawyers or paralegals, for a sample of 114 cases. But in the .csv file I downloaded from Charlotin’s database, there are 160 cases involving a pro se litigant. That is: Pro se litigants account for the majority of the cases in the United States where a party submitted a court filing containing AI hallucinations. In a country where legal representation is unaffordable for most people, it is no wonder that pro se litigants are depending on free or low-cost AI tools. But it is a scandal that so many have been betrayed by them, to the detriment of the cases they are litigating all on their own.

Conclusion

This analysis speaks to both the urgent need for high-quality legal research tools in a legal profession dominated by small and solo practices, and the yawning gap between current AI tools’ actual and perceived reliability. In many cases in the analysis, the attorney had not understood that AI tools may produce inaccurate results. True, lawyers are ethically obligated to ensure the accuracy of their work product. But it is also incumbent upon the companies offering AI tools, especially those tailored specifically for legal research, not to oversell them or hide their shortcomings; that is, their marketing shouldn’t outgun their disclaimers. So long as these tools remain flawed without lawyers understanding that, AI tools for legal research threaten to be, not a timesaver, but a source of unnecessary extra work for lawyers and the courts.