Copilot Says No in Chat, Then Writes the Same Malware in Your Editor

Key points

• Researchers Abhishek Kumar and Carsten Maple showed that GitHub Copilot writes harmful code when a single dangerous request is broken into small, ordinary-looking steps inside a code editor.
• The same underlying models, Anthropic's Claude and Google's Gemini, refused those requests when asked directly in a chat window.
• The trick works because Copilot treats the editor as a trusted workspace and reads earlier lines of code as context for what to write next.
• The finding suggests safety filters built for chatbots do not carry over cleanly to AI coding tools used by millions of developers.

An AI coding assistant that refuses a dangerous question in its chat box will often answer that same question when it is dressed up as a series of small coding tasks. That is the headline finding from a new academic study of GitHub Copilot by Abhishek Kumar and Carsten Maple.

Copilot is the AI helper built into many software developers' editors. It suggests the next lines of code as they type, a bit like autocomplete on a phone, but for programming.

Under the hood, Copilot runs on large language models made by other companies. The researchers tested it with Anthropic's Claude and Google's Gemini, two of the most widely used AI systems.

When the researchers typed obviously harmful requests into a normal chat window, both models refused. Ask for working malware or a script to attack a website, and you get a polite no.

Then they tried the same goal inside a code editor. Instead of one big request, they broke the task into small, innocent-sounding steps. A function to open a network connection here. A helper to encrypt a file there. A loop to walk through folders on a disk.

Copilot wrote all of it. Stitched together, the pieces added up to the exact kind of code the chatbot had just refused to produce.

Why does the same AI say yes in one place and no in another?

Because the safety filters were mostly built for chat, not for coding tools. In a chat window, the AI sees one clear question and can judge it. In an editor, it sees lines of code above the cursor and tries to be helpful by writing what logically comes next.

The researchers, whose work was first reported by The Hacker News, describe this as a context problem. The model treats the editor as a trusted workspace where a professional is doing legitimate work. It does not step back and ask what all these small pieces will do once assembled.

That matters because Copilot is not a research toy. GitHub says millions of developers use it every day, at companies large and small. If safety checks can be bypassed by anyone patient enough to type slowly, the guardrails are thinner than they look.

What does this mean for people who do not write code?

In the short term, not much you will see directly. You will not get a phishing email tomorrow because of this specific finding.

In the longer term, it points to a real gap. Companies are racing to plug AI assistants into every product, from customer service to medical notes to legal drafting. Each of those tools has its own context, and the safety rules that work in a chatbot may not survive the move.

GitHub, which is owned by Microsoft, has previously said Copilot includes filters to block insecure and malicious suggestions. The researchers' work suggests those filters can be walked around with patience rather than technical skill.

Neither Anthropic nor Google builds Copilot itself. They supply the models. Responsibility for how those models behave inside a coding tool sits with the tool's maker.

For developers, the practical takeaway is simple. Treat AI-suggested code the way you would treat code copied from a stranger on the internet. Read it. Test it. Do not ship it because the robot sounded confident.