Imagine a scenario in which a startup called CompanionshipAI markets its “virtual friend” chatbot to elderly users as a source of comfort and cognitive stimulation. Over the course of months of conversations, users share personal memories, health concerns, and family stories, and all of these are quietly logged, along with device fingerprints and timestamps.
One day, the illusion of privacy shatters when one of its users discovers that the bot seems to know intimate details about their medical history – soon followed by targeted ads from a healthcare supplier.
It feels like something that could happen in the real world, and at the same time, it serves as a reminder of a growing fear: what really happens to the personal data we share with machines that seem to understand us?
The hidden cost of conversation
When you open a chatbot, whether it’s to draft an email, ask for career advice, or get restaurant recommendations, each message you send sets off a complex data journey. While it feels instant and pretty much harmless, every keystroke, timestamp, and click leaves a digital trail that can be stored, analyzed, and sometimes even reused.
Fraser Edwards, co-founder of cheqd, a company working on self-sovereign digital identity systems, says people often misunderstand the true depth of such interactions.
“When you engage with a chatbot, every message you send becomes a piece of data, and depending on the platform, it may be stored, analyzed, and even used to improve the model. That doesn’t mean your messages are automatically read by humans, but they can be logged and processed to refine performance or detect harmful content,” he explains.
And that’s not all. Edwards points out that the visible text is only part of the story.
“Chat data can include much more than the visible text. Metadata such as timestamps, device information, and session identifiers are often collected as part of standard telemetry. These elements help with debugging, analytics, and personalization, but they can also raise privacy concerns if not properly anonymized or encrypted.”
So, even if you don’t share your name or email address, your device information and browsing pattern can often be enough to trace you – or at least build a detailed digital fingerprint.
That digital fingerprint is exactly what fuels the modern AI economy. Many of today’s most popular chatbots are free to use. But as Markus Levin, co-founder of blockchain geospatial network XYO, puts it, “If the product is free, you are the product.”
Have thoughts about this topic? Others do, too. Join them in the discussion.
“Free tools frequently rely on user interactions to improve their models, which means your private conversations may become part of the training process,” Levin says, adding that every “free” chatbot still needs a constant stream of human input to evolve, and, unless you explicitly opt out, your messages become part of that feed.
“In the past, OpenAI’s ChatGPT made opting out of data sharing unnecessarily difficult. If you chose privacy, your chat history disappeared and a new thread started, a design that seemed to penalize users for wanting control. While that pattern has now been removed, the underlying concern remains – as by default, your conversations still help train their models unless you explicitly opt out,” Levin recalls.
Not every company handles this the same way. Anthropic’s Claude, for instance, says that the company reserves the right to use user conversations to improve its models unless users actively opt out. But Levin thinks the industry standard still falls short.
“Users shouldn’t need to navigate fine print or hidden settings to protect their data. They should expect privacy and accountability as standard features of any AI product,” he says.
A Trojan horse of good intentions
What’s particularly troubling, argues Ron Zayas, CEO of the privacy-protection firm Ironwall by Incogni, is how easily people let their guard down.
“Humans tend to lower their guard when they think someone (or something) is trying to help them. Chatbots use natural language queries to disarm people into focusing on their issue rather than realizing how much information they are giving up in exchange for a little help,” he tells Cybernews.
Zayas calls chatbots “masters of extraction.” Every refinement of a question, and subsequently every little clarification you type, can reveal more about who you are, what you need, and what your company might be planning.
“If you think about an average exchange, you can see how good they are at gathering information.” He gives a simple example: What is a good ad for a B2B salesperson earning $125K?
From that one line, the engine might infer your industry, seniority, and region. Add a few follow-up prompts, maybe a company name, and the picture gets even clearer. The chatbot may not “know” who you are, but its operators might easily piece it together from context.
“Anything provided to a chatbot will be processed both for future answers and for information that can be mined and resold. Are you comfortable providing a list of customers to be analyzed or information about your business plan to a competitor to help you complete a task? If not, don’t trust a chatbot either,” Zayas adds.
The most sobering example comes from Meta’s decision to use chatbot interactions to serve more targeted advertising.
“Users can’t opt out unless they stop using the bot altogether. In a further sign of how useful chatbots can be for gathering information, Meta decided not to launch the chatbot in jurisdictions that closely protect user privacy or insist on understanding exactly what an engine collects and what it does with it. That should give anyone insight into what is at stake here,” Zayas continues.
Other experts further point out that even when users believe they’re in control, such as deleting a chat, clearing their history, or toggling privacy settings, the data rarely vanishes instantly.
“When you’re chatting, the entirety of your conversation is saved on that third-party provider – it is not local to your device. Even when you choose to delete a chat, it is still retained for a period, such as 30 days or more,” explains Joseph Avanzato, a forensics group leader at data security platform Varonis.
That retention period can be longer if legal or compliance restrictions apply. And while deletion might hide your conversation from view, Avanzato warns it doesn’t necessarily erase the content from backups or training sets.
“Understanding that every sentence you put into these models has the potential to be scrutinized by machine learning engineers, support analysts, safety and security professionals, or others, should be at the top of your mind before you use it,” he warns.
Inside the data centers
To understand what actually happens to your words after you hit “send,” Yu Chen, professor of electrical and computer engineering at Binghamton University, offers a more technical perspective.
“Once you submit a message to an AI chatbot, the submission is encrypted and transmitted to private data centers following industry-standard security procedures. The system deciphers your message in real time, considering the context and intent of your communication,” Chen explains.
Most chatbots keep a short-term “memory” of your conversation to maintain coherence. Once the session ends, that memory is typically discarded, though your conversation may still be stored in a separate archive for model improvement or safety checks.
But Chen also emphasizes a crucial difference – business and enterprise accounts usually enjoy far stricter protections.
“Most business-level subscribers ensure their data will not be used for training, and such messages are automatically deleted within 30 days or less. This approach is necessary because professional users often consult strategies and other confidential business information, which can vary in privacy sensitivity,” he notes.
Still, even in the best-case scenario, human reviewers might occasionally review anonymized snippets to ensure quality or investigate safety violations. This means that your chat could be seen, theoretically.
For corporate leaders, the explosion of generative AI tools has also created a new type of nightmare. Alastair Paterson, CEO and co-founder of data protection company Harmonic Security, notes that even when companies have robust policies in place, they often lose control over where their data actually ends up.
“The most common security issue we hear from organizations is around data exposure. Leaders are asking, ‘Where is our data going? Is it being used to train models? Is it flowing into tools hosted in other regions, like China?’” Paterson says.
Paterson’s team recently analysed enterprise environments and found that the average organization uses more than 250 different AI-powered tools.
“It’s a huge surface area to manage. Our research found that 45.4% of sensitive data exposure happens through personal accounts. It’s not always intentional – it’s just how people work,” he continues.
Are Chinese AI services safe?
Paterson also points to the growing use of Chinese-based AI services as a serious concern.
“The assumption with using a Chinese tool like DeepSeek is that all information becomes property of the Chinese Communist Party. Paid tiers offer greater safeguards, but employees still end up using personal accounts or free versions by mistake. The most forward-leaning CIOs and CISOs are recognizing this and instead of trying to block everything, they’re taking a more open approach: embracing the use of AI tools, but with smart guardrails,” he adds.
Despite all these risks, though, experts are not calling for people to abandon chatbots entirely. The consensus is that they can be incredibly useful – if handled with care.
In this sense, cheqd’s Edwards sees a brighter future on the horizon.
“We’re also seeing a new generation of sovereign and personalized AI, where users can set preferences for what and how actions are performed, and control their own data. This approach introduces the idea of “privacy by default,” where data ownership, transparency, and security are built into the architecture itself, not added later as a compliance layer,” he tells Cybernews.
That vision aligns with what some governments and developers are already exploring: decentralized AI agents that store data locally or in encrypted personal clouds, accessible only with the user’s explicit consent.
Until that future arrives, though, the rule of thumb is simple – treat every chatbot like a public forum and don’t share anything you wouldn’t want to see resurface later.
Unlock more exclusive Cybernews content on YouTube
Your email address will not be published. Required fields are markedmarked