This is the fourth in our plain-English series about artificial intelligence. The full series is at news.atozofcyber.co.uk
Here is a thought experiment. Imagine you are sitting in a busy coffee shop, and you want some help from the person at the next table. They are knowledgeable, helpful, and very good at keeping the conversation going. But the table is in the middle of the room, everyone around you can hear, and the person is going to write down everything you say and keep it indefinitely.
Would you give them your bank account number? Tell them about a medical condition? Explain a difficult situation at work involving a colleague's name and personal details?
Probably not. But millions of people do the equivalent of this every day when they type into AI chat tools — because the interface feels private, personal, and conversational, even when it is none of those things.
This article is about what happens to what you type, and what you should think twice about before you do.
What actually happens when you type into an AI chat tool?
When you send a message to an AI assistant, that message travels over the internet to a server run by the company that built the tool. The AI processes your message and generates a response. So far, so obvious.
What is less obvious is what happens next. Most AI services store your conversation. Depending on the company and the settings you have chosen, that conversation may be used to train future versions of the model — meaning your words could, in a very indirect sense, influence how the AI responds to someone else in the future. It may be accessible to company employees for safety review. It may be retained for months or years. In some cases, even when you delete a conversation, it may be kept beyond the usual retention period due to legal proceedings the company is involved in — as happened with OpenAI in 2025 following a copyright lawsuit.
The founder of OpenAI said publicly in June 2025 that users should seek "privacy clarity" before using ChatGPT extensively — and confirmed that OpenAI is legally required to share conversations if subpoenaed by a court. These are not scaremongering claims. They are statements from the company itself.
None of this means AI tools are dangerous or malicious. It means they are services run by companies, subject to the same legal and commercial realities as any other online service — and they deserve the same thoughtful approach you would give to any service you use online.
The six things most worth protecting
Your full name combined with personal details. Your name alone is harmless. But your name combined with your address, your date of birth, your workplace, and a description of your personal situation creates a profile that, if it ever left the company's servers, could be used against you. The combination is what creates the risk, not any single piece.
Financial information. Never type account numbers, card numbers, sort codes, PINs, or passwords into an AI chat window. There is no legitimate reason to do so. If you are asking for help with a financial problem, describe the situation in general terms — "I have a direct debit that seems to have been charged twice" — rather than providing the actual numbers.
Confidential work information. This is the area where the most documented harm has already occurred. In 2023, Samsung engineers accidentally uploaded proprietary source code to ChatGPT while asking for debugging help. The code became part of the training data before the incident was noticed. Samsung subsequently banned AI chat tools across the company. The same risk applies to any confidential business information — client names, deal terms, internal strategies, unreleased products, personnel matters. If you would not post it on a public noticeboard, do not type it into an AI chat tool without first checking your organisation's policy.
Medical and mental health information. AI tools are increasingly used as a first port of call for health questions, and for many people they provide genuinely useful information. The risk is not in asking health questions generally — it is in combining health information with identifying details. "What are the symptoms of X?" is a very different conversation from "I have been diagnosed with X, I take Y medication, and I am worried about Z" typed into an account linked to your email address and stored indefinitely. The first is a general query. The second is a detailed medical record in someone else's hands.
Other people's information. When you ask AI to help you draft a message about a difficult situation with a colleague, or ask for advice about a family member's behaviour, or explain a dispute with a neighbour — you are providing information about real people who did not consent to having their details processed by a third-party AI system. This is worth being mindful of, both as a matter of privacy and as a question of trust.
Passwords, security codes, and authentication details. This should not need saying, but in the flow of a helpful conversation it can happen — someone shares a password while asking for help with an account, or pastes an authentication token while asking a technical question. Never do this. Legitimate AI tools do not need your passwords to help you.
The settings that actually matter
Most AI services offer privacy settings that are worth knowing about and adjusting. The most important one is usually labelled something like "Improve the product using my conversations" or "Use my chats for model training." Turning this off means your conversations are less likely to be used as training data, though they may still be retained for other purposes.
Some services offer a "temporary chat" or "incognito" mode that does not save the conversation at all. For sensitive queries — health information, personal situations, legal questions — this is worth using.
The ICO (Information Commissioner's Office — the UK's data protection regulator) advises that before using any AI service with personal information, you should check the privacy policy for how long data is retained, whether it is used for training, and whether it is shared with third parties.
The AI at work problem
One of the fastest-growing privacy risks is something researchers call shadow AI — employees using AI tools that have not been approved by their organisation, often without fully understanding where their data goes.
A 2025 survey found that 49% of employees use AI tools not sanctioned by their employers. A separate study found that Microsoft Copilot — a formally approved enterprise tool — exposed around three million sensitive records per organisation in the first half of 2025, primarily because the underlying permissions on those documents had never been properly set up, and the AI surfaced information that individual employees were technically allowed to access but that was never intended to be easily discoverable.
The point is not that AI tools are uniquely risky compared to other work software. It is that their conversational, helpful, low-friction nature makes people less guarded with them than with other tools. You would not paste a client's confidential contract into a public search engine. The AI chat window feels different, but the underlying data journey may not be.
A note on AI and children
Children use AI tools — for homework help, for creative projects, for curiosity. The same principles apply with additional weight. Children should not share their full name, school, address, or any identifying information with AI tools. They should be encouraged to treat AI chat windows as they would treat a public website — useful, but not private, and not a friend.
The EU AI Act, which came into force in 2024, places specific obligations on AI operators when their systems interact with children. Some services require operators to regularly remind young users that they are talking to an AI. These protections are welcome — but they depend on children and parents being aware of them.
What does this mean for me?
Use the privacy settings. Turn off model training where the option exists. Use temporary or incognito chat modes for sensitive queries. It takes thirty seconds.
Treat the AI chat window like a public conversation. What would you be comfortable saying if the room could hear? That is roughly the right level of caution.
At work, check the policy before you type. If your organisation has an AI use policy, it exists for good reasons. If it does not have one, the safest default is to treat company information the same way you would when using any external service — with discretion.
Describe, do not identify. Most of what you legitimately need AI help with does not require real names, real numbers, or real identifying details. "A colleague has been doing X" works just as well as naming them. "I have a debt of approximately £X" works just as well as providing the account number.
Talk to the children in your life. AI tools in homework and creative contexts are not going away. Building the habit of mindful sharing early is far easier than correcting it later.
🧠 The Human Factor
| Technology involved | AI chat tools and assistants — services that store, process, and in many cases retain user conversations on servers run by private companies |
| Root cause | The conversational, helpful interface of AI tools creates a false sense of privacy — people share information they would not share in other digital contexts, because the experience feels personal and intimate rather than public |
| What was at risk | Personal identifying information, financial details, confidential work data, medical information, and details about third parties who did not consent to their information being processed |
| Prevention | Use privacy and training opt-out settings; use temporary chat modes for sensitive queries; apply the same discretion to AI chat that you would to any public online service; follow organisational AI use policies at work |
This concludes the first four parts of the AI series. Future instalments will cover AI in schools, AI and creative work, and how AI is being used both by and against cybersecurity professionals.
References and sources
- OpenAI founder on privacy: public statement, June 2025 — reported in Psychiatric Times and multiple outlets
- Samsung AI data incident, 2023 — Bloomberg; subsequent company-wide AI ban confirmed
- Microsoft Copilot data exposure: Concentric AI research (2025)
- ICO guidance on AI and personal data — ico.org.uk
- Help Net Security: AI chatbots are sliding toward a privacy crisis (October 2025)
- Cloud Security Alliance: AI Shadow Usage Survey (2025)
- EU AI Act obligations regarding minors — artificialintelligenceact.eu