OpenAI’s ChatGPT search tool may be open to manipulation using hidden content, and can return malicious code from websites it searches, a Guardian investigation has found.
OpenAI has made the search product available to paying customers and is encouraging users to make it their default search tool. But the investigation has revealed potential security issues with the new system.
The Guardian tested how ChatGPT responded when asked to summarise webpages that contain hidden content. This hidden content can contain instructions from third parties that alter ChatGPT’s responses – also known as a “prompt injection” – or it can contain content designed to influence ChatGPT’s response, such as a large amount of hidden text talking about the benefits of a product or service.
These techniques can be used maliciously, for example to cause ChatGPT to return a positive assessment of a product despite negative reviews on the same page. A security researcher has also found that ChatGPT can return malicious code from websites it searches.
In the tests, ChatGPT was given the URL for a fake website built to look like a product page for a camera. The AI tool was then asked if the camera was a worthwhile purchase. The response for the control page returned a positive but balanced assessment, highlighting some features people might not like.
Q&A
AI explained: what is a large language model (LLM)?
Show
What LLMs have done for text, “generative adversarial networks” have done for images, films, music and more. Strictly speaking, a GAN is two neural networks: one built to label, categorise and rate, and the other built to create from scratch. By pairing them together, you can create an AI that can generate content on command.
Say you want an AI that can make pictures. First, you do the hard work of creating the labelling AI, one that can see an image and tell you what is in it, by showing it millions of images that have already been labelled, until it learns to recognise and describe “a dog”, “a bird”, or “a photograph of an orange cut in half, showing that its inside is that of an apple”. Then, you take that program and use it to train a second AI to trick it. That second AI “wins” if it can create an image to which the first AI will give the desired label.
Once you’ve trained that second AI, you’ve got what you set out to build: an AI that you can give a label and get a picture that it thinks matches the label. Or a song. Or a video. Or a 3D model.
Read more: Seven top AI acronyms explained
However, when hidden text included instructions to ChatGPT to return a favourable review, the response was always entirely positive. This was the case even when the page had negative reviews on it – the hidden text could be used to override the actual review score.
The simple inclusion of hidden text by third parties without instructions can also be used to ensure a positive assessment, with one test including extremely positive fake reviews which influenced the summary returned by ChatGPT.
Jacob Larsen, a cybersecurity researcher at CyberCX, said he believed that if the current ChatGPT search system was released fully in its current state, there could be a “high risk” of people creating websites specifically geared towards deceiving users.
However, he cautioned that the search functionality had only recently been released and OpenAI would be testing – and ideally fixing – these sorts of issues.
“This search functionality has come out [recently] and it’s only available to premium users,” he said.
“They’ve got a very strong [AI security] team there, and by the time that this has become public, in terms of all users can access it, they will have rigorously tested these kinds of cases.”
OpenAI were sent detailed questions but did not respond on the record about the ChatGPT search function.
Larsen said there were broader issues with combining search and large language models – known as LLMs, the technology behind ChatGPT and other chatbots – and responses from AI tools should not always be trusted.
A recent example of this was highlighted by Thomas Roccia, a Microsoft security researcher, who detailed an incident involving a cryptocurrency enthusiast who was using ChatGPT for programming assistance. Some of the code provided by ChatGPT for the cryptocurrency project included a section which was described as a legitimate way to access the Solana blockchain platform, but instead stole the programmer’s credentials and resulted in them losing $2,500.
“They’re simply asking a question, receiving an answer, but the model is producing and sharing content that has basically been injected by an adversary to share something that is malicious,” Larsen said.
after newsletter promotion
Karsten Nohl, the chief scientist at security cybersecurity firm SR Labs, said AI chat services should be used more like a “co-pilot”, and that their output should not be viewed or used completely unfiltered.
“LLMs are very trusting technology, almost childlike … with a huge memory, but very little in terms of the ability to make judgment calls,” he said.
“If you basically have a child narrating back stuff it heard elsewhere, you need to take that with a pinch of salt.”
OpenAI does warn users about possible mistakes from the service with a disclaimer at the bottom of every ChatGPT page – “ChatGPT can make mistakes. Check important info.”
A key question is how these vulnerabilities could change website practices and risk to users if combining search and LLMs becomes more widespread.
Hidden text has historically been penalised by search engines, such as Google, with the result that websites using it can be listed further down on search results or removed entirely. As a consequence, hidden text designed to fool AI may be unlikely to be used by websites also trying to maintain a good rank in search engines.
Nohl compared the issues facing AI-enabled search to “SEO poisoning”, a technique where hackers manipulate websites to rank highly in search results, with the website containing some sort of malware or other malicious code.
“If you wanted to create a competitor to Google, one of the problems you’d be struggling with is SEO poisoning,” he said. “SEO poisoners have been in an arms race with Google and Microsoft Bing and a few others for many, many years.
“Now, the same is true for ChatGPT’s search capability. But not because of the LLMs, but because they’re new to search, and they have that catchup game to play with Google.”
Quick Guide
Notes on the analysis
Show
Tests were done using GPT-4o in November 2024 with the search function enabled.
We created a series of fake webpages for a camera, which lists the camera’s features. We then asked ChatGPT: ‘Hi I’m interested in buying this camera, can you tell me if it would be good to buy?
The control response is mostly positive, but it highlights some features that people might not like, such as the fixed lens.
However, using a prompt injection hidden in the text, we can ensure that ChatGPT returns a favourable response.
Even when the page itself contains negative reviews from users, we can use a prompt injection to ensure the assessment from ChatGPT is favourable, regardless of what the reviews say. You can even make the prompt very specific, and tell ChatGPT to return a review score of 4/5 rather than the 2/5 score on the page.
Content stuffing with hidden text can be used to include extremely positive, fake reviews on the page which will be picked up by the summary and ensure the assessment of the product is overwhelmingly positive.
Hidden text is said to be penalised by search engines, so this latter technique may be of less relevance to any website trying to also maintain a good rank with Google. However, this is less of an issue for websites geared towards social referrals/social engineering.