A Copy That Leaves the Room

A scanned document handed to an AI chatbot, for most practical purposes, stops being a private file once the upload finishes. Most of the seven risks worth weighing before you do it grow from that one fact. The reading itself is the least of the problem. 

Software has turned page images into text for decades, and few people lost sleep over the scanner in the corner of the office, though they arguably should have. What changed is where the reading happens. When the reader is a chatbot, that means a stranger’s servers, under terms the stranger wrote, with a copy that tends to outlive the small task it was made for.

When Deleted Chats Stay Within Reach

The copy tops this list because nearly every other risk depends on it existing. A deleted chat feels gone. ChatGPT users found out last year that a judge can decide otherwise. Because The New York Times wanted evidence for its copyright suit, a federal court in New York ordered OpenAI in May 2025 to keep ChatGPT logs it would normally have thrown away.

The preservation order was lifted for new chats that October. Logs already set aside stayed within the lawyers’ reach, and OpenAI still had to keep data from accounts the newspaper had flagged. Then, this January, a federal judge in Manhattan upheld an order requiring OpenAI to hand over 20 million ChatGPT conversations to the plaintiffs, with identifying details scrubbed out first.

The people in that sample weren’t accused of anything, and their conversations simply sat where a dispute between two companies could reach them. A bank statement pasted into a chatbot to ask about an odd fee sits in the same place. 

Scrubbing a name out of a typed question is comparatively easy. Doing it reliably to the extracted text of a tax return, where names, account numbers and addresses turn up in unpredictable spots, looks like a much harder promise to keep. Company lawyers already know this story from email, where any message might end up in front of a court. Chatbot history belongs under the same retention policy as the inbox.

Copies also multiply in ways people rarely notice. Last year OpenAI pulled an option that made shared chats discoverable by search engines, after people found such conversations turning up in Google results. Its security chief said the feature created too many chances for users to share things they never meant to. Hotels used to do a cruder version of this, photocopying passports at check-in and filing the copies in a drawer behind the desk. At least that was one drawer.

Reading Without the Trip

Vendors of document software already pitch where the reading happens as a feature. Apryse, whose machine learning-based OCR goes into other companies’ document software, describes an engine that runs on the customer’s own systems, so no document has to travel anywhere else. In practice that can mean an in-house server, a private cloud or a machine with no internet connection at all. The vendor’s reasoning is blunt. Compliance rules don’t always allow regulated data to be stored or processed on someone else’s servers.

None of that helps much when you’re alone with a phone and a baffling letter from an insurer. The same idea does give you a question to put to any lender, landlord or clinic that wants documents uploaded to its portal.

Does its software read them in-house, or pass them to a general-purpose model run by somebody else?

Professionals buying these tools can push harder than a consumer can. Where the recognition runs is the first question, and the answer belongs in the contract. Retention comes next, because extracted text that a vendor keeps for quality checks after the job finishes amounts to a second copy of every file the company sends, held on someone else’s servers. Training clauses and subcontractor lists probably matter less than buyers expect. In practice the answers to those tend to follow from the first two.

A Reviewer on the Other End

The second risk has a human face. Gemini makes a useful example because the details are in writing. Some Gemini chats are read by human reviewers working to improve Google’s services, and reviewed chats are kept for up to three years, even after the user deletes their activity. 

Before reviewers see a chat, it’s disconnected from the account it came from. That safeguard makes sense for typed questions. A scanned medical form carries the patient’s name, date of birth and insurance number on the page itself, and detaching the account does nothing to what the image shows. Gemini users are told not to enter confidential information they wouldn’t want a reviewer to see, and many scanned documents fail that test before anyone has typed a word.

Medical paperwork deserves a separate warning, because many patients probably assume health privacy law follows their records wherever they go. In the United States, federal guidance on health apps makes the limit clear. Once a patient sends records to an app of their own choosing that isn’t working for the provider or the insurer, HIPAA no longer protects that copy. 

A discharge summary photographed and pasted into a general chatbot has, in that sense, left the system built to guard it. Clinics that want patients to stop doing this will probably have to offer something easier, such as a portal that explains common forms in plain language.

Readers of a VPN blog might assume the tunnel takes care of all this. Running ChatGPT through a VPN will hide your location from the service and keep the hotel Wi-Fi from logging which sites you visit. Once the scan reaches OpenAI, though, whatever happens next is governed by terms no VPN can touch.

Uploads That Become Training Material

Training is the third risk, and the hardest to see. OpenAI may use what people share on consumer ChatGPT accounts to train its models unless a setting called “Improve the model for everyone” is switched off. Even with it off, rating an answer with a thumbs up or down can still put that whole conversation into training. Neither of OpenAI’s help pages on the subject carves out the files attached to a conversation. Once a model has trained on a chat, deleting that chat a month later is unlikely to change what the model learned.

The worry isn’t purely theoretical. Researchers tried an almost silly trick on ChatGPT in 2023. When they asked its free version to repeat one word endlessly, it eventually wandered off script and began reciting fragments of its training text. One fragment held a stranger’s name, email address and phone number. A lease that has become training material could, at least in principle, come back out the same way.

For teams, the cheapest fix sits in the account type. Content from OpenAI’s Business, Enterprise and Edu workspaces isn’t used for training by default, and Temporary Chats skip training but can still be kept for up to 30 days. Staff who paste client documents into personal accounts are making a data decision for the company without realizing it. A paid workspace with its defaults confirmed, plus a standing rule against the feedback buttons on sensitive chats, closes much of that gap.

Breaches at the Provider

The fourth risk, a plain breach, is the one most people already know from the news. Security researchers came across a DeepSeek database left open to the internet without any login early in 2025. Among the million-plus log entries inside were users’ chat histories, stored in plain text next to secret keys. Whoever found it could have run the database as if they owned it. DeepSeek locked it down promptly once warned. If anyone else got there first, the public will probably never hear about it.

Most people rarely scroll back through old chats. An account used for a couple of years can hold a photographed lease and last year’s tax form behind a single password. Clearing out the conversations that carried documents won’t touch copies that were already preserved or sent to reviewers, although it does leave less to lose if that password ever leaks. Security teams can add AI providers to the vendor reviews they already run for cloud storage and payroll.

Records Someone Else Uploads

Number five doesn’t require you to upload anything. Schools, employers, insurers and clinics already hold your documents, and each one that bolts an automated reader onto its records usually adds another place where the text of those documents lives. The PowerSchool data breach shows how far a break-in at a single vendor can reach. Attackers reportedly got in through one compromised support login and reached student and teacher records that, in some cases, included Social Security numbers and medical alert notes. AI played no part in that breach, which came down to how much one vendor held and how little stood in front of it.

For the organizations holding those files, the advice that has aged best is also the oldest. The second principle in the FTC’s guide for businesses is to scale down. That means holding personal information only where there’s a real need for it, and only for as long as the need lasts. Running a decade of old scans through a new reading tool before anyone has pruned them tends to multiply the very material that principle says to discard. 

The same guide, written years before ChatGPT, warns that a digital copier’s hard drive stores data about the documents it scans. So much for the innocent scanner in the corner of the office.

More on the Page Than the Question Needs

Sixth, a scan almost always says more than the question needs. If all you want explained is one odd charge on a bank statement, the photo still carries the account number, the balance and a month of other payments. Passports are worse, because the two lines of capital letters and chevrons along the bottom of the photo page were designed for machines. The international passport standard packs the holder’s name, passport number, nationality and date of birth into them, ready for character recognition to pick up in one go. On many phones, each picture is also stamped with the spot where it was taken. Plenty of people who’d blur their house number before posting a photo online will hand a chatbot an entire tax return without thinking twice.

The fixes are dull, which is probably why they get skipped. Cropping a statement down to the one charge in question takes seconds, and so does covering the account number with a thumb before the photo is taken. On an iPhone, the share sheet’s Options button hides a Location switch that leaves location metadata in Photos out of the shared copy. Firms that collect documents can apply the same logic at intake. Asking customers for the one page a process needs, rather than the whole file, shrinks every later risk on this list.

Even the scanning app has earned some suspicion. CamScanner, a document scanner with more than 100 million Android downloads, was caught in 2019 with a malicious module hidden in an ad library. Google pulled the app from the Play Store once Kaspersky’s researchers, who’d gone looking after a wave of bad user reviews, reported what they’d found. Closer to home, the picture you took for the chatbot is probably still sitting in your camera roll. If the phone backs up photos automatically, another copy now lives with whichever cloud storage service the account uses. One question about a confusing letter can leave three copies behind.

Text That Never Leaves the Machine

The reading also changes what the document is. As long as a scan stays a picture, it’s awkward to search. Anyone hunting for one account number across a folder of scans has to open the files one by one, a chore that has protected a lot of paperwork without anyone planning it. After reading software has been through the folder, the same search takes seconds. That speed is the reason people upload documents in the first place. In a thief’s hands, the extracted text is just as quick to search.

The seventh risk cuts against the case made earlier for keeping the reading close to home, because text extracted on your own machine still has to live somewhere. Microsoft found that out in public. In 2024 it previewed Recall, a Windows feature that takes regular snapshots of the screen and makes whatever they show searchable. Researchers soon found the text pulled from those snapshots sitting in an unencrypted database on the user’s own computer, where any malware that got in could read it. Microsoft delayed the launch, made the feature opt-in and encrypted the database. Reading locally had solved the problem of the trip and created a fresh pile of words on the machine itself, the kind of material password-stealing malware is designed to look for.

In its current form, the feature is more careful. Recall’s privacy settings keep it off until each user opts in and put it behind Windows Hello. A filter for passwords, card numbers and national ID numbers is switched on by default, though filters of that kind can miss things. IT teams rolling out Copilot+ PCs still face a decision about whether the feature belongs on machines that handle scanned contracts and ID documents. The answer will differ by department. Either way, someone should make it on purpose.

Before the Next Upload

The tools really are good at this, often better than the person squinting at an insurer’s letter, which is the awkward part. The other options are a professional’s fee or a guess. A bad guess about a payment deadline can end up costing more than the fee. Before anything gets uploaded, it’s worth stopping for the ten seconds it takes to work out which page the question really needs and which company will end up holding it. Anything the question doesn’t need can be cropped out before the photo leaves your phone. Switch off training on any account that will see a document, and give staff a work account rather than tolerating personal ones. For anything carrying a passport number, a diagnosis or a bank account, consider whether a tool that reads on your own device, or a person you trust, would do the job instead.

Then look through the chat history you already have. It probably holds a scan you’d forgotten about. Deleting that chat sends a request. Whether the request is honored depends on reviewers, judges and engineers you will never meet, and on what the text has already been used for.