Imagine the following: You visit a webpage with a lot of text you don’t want to
read and ask your AI assistant for a summary. A few moments later, the AI
assistant has extracted one of your emails and sent it to an attacker without
you ever knowing.
In October 2025, we found exactly this vulnerability in Firefox’s AI chatbot
integration1.
Firefox offers a summarization, explaination and proofread AI feature. When a
user makes use of one of these features, Firefox pastes a prompt into the
sidebar AI chat including the page title, the selected text (or, if the whole
page is summarized, a selection is being made by Firefox) and an instruction on
how to process the provided text. The sidebar AI chat is essentially an IFrame
of a third-party chatbot (Claude, Copilot, …).
Over the last few weeks, I have had a very productive exchange with
Christoph Klaassen
on the impact of AI on security governance and compliance. In this post, we
summarize our thoughts.
AI agents are here, there, and everywhere. Smarter, faster, and more skilled,
they gain greater autonomy and trust. We trust their capabilities to do many
tasks much faster and sometimes better than we can. We trust them as they
usually demonstrate their eagerness to please us and fulfill our commands. Isn’t
that too good to be true, and we might be dealing with a double-edged sword
here? Can attackers use the same capabilities of the AI agents to attack their
own users? Can they exploit their eagerness to please their users to fulfill the
attackers’ intentions? And most importantly: what’s the worst that could happen
if you fully trust some random AI Agent?
With the rise of AI assistance features in an increasing number of products, we
have begun to focus some of our research efforts on refining our internal
detection and testing guidelines for LLMs by taking a brief look at the new AI
integrations we discover.
Alongside the rise of applications with LLM integrations, an increasing number
of customers come to ERNW to specifically assess AI applications. Our colleagues
Florian Grunow and
Hannes Mohr analyzed the novel attack
vectors that emerged and presented the results at
TROOPERS24 already.
Dennis and I already published blog posts about our research project dealing
with vulnerabilities in parcel tracking implementations at
DHL
and DPD. At the
Winterkongress (winter
congress) in Winterthur, Switzerland, we had the great opportunity to give a
talk about the matter. The talk was recorded and can be watched
here.
DigiGes held the Winterkongress, which
took place in Winterthur on 01.03. till 02.03.2024. The main topics are ethics,
threats, and opportunities of IT. This year, many talks looked at AI in some
way.