May 27, 2025 08:00:00

How OpenAI’s o3 model uncovered a zero-day vulnerability in the Linux kernel

Technology expert Sean Heelan explained how he discovered a zero-day vulnerability in Linux using OpenAI's AI model '

o3 ', which dramatically improves its ability to reason about code.

How I used o3 to find CVE-2025-37899, a remote zeroday vulnerability in the Linux kernel's SMB implementation – Sean Heelan's Blog
https://sean.heelan.io/2025/05/22/how-i-used-o3-to-find-cve-2025-37899-a-remote-zeroday-vulnerability-in-the-linux-kernels-smb-implementation/

While investigating vulnerabilities in the Linux kernel server 'ksmbd,' which implements the SMB 3 protocol in kernel space to share files over a network, Heelan ran benchmark tests to measure o3's capabilities with bugs he found in ksmbd.

The vulnerability that Heelan used as an indicator was the vulnerability '

CVE-2025-37778 ' that he discovered himself. This is a type of use-after-free vulnerability that can lead to data corruption or malicious code execution by referencing memory that has already been freed.

When verifying this vulnerability, all that is required is to check whether a process such as 'if a certain code is enabled, perform a certain process. If not, perform a different process' is working properly, but since it requires reading a minimum amount of code and inferring what process will be performed in a certain situation, Mr. Heelan thought that this would test the AI's reasoning capabilities.

So Heelan tasked o3 with looking for use-after-free vulnerabilities and asked them to provide a clear, high-level overview of what ksmbd is, what its architecture is, and what the threat poses.

As a result, o3 found vulnerabilities in 8 out of 100 runs. In a different model, Claude Sonnet 3.7 found vulnerabilities in 3 out of 100 runs, and Claude Sonnet 3.5 found vulnerabilities in 0 out of 100 runs.

Having tested o3's capabilities, they gave it more detailed prompts and tried again, and in one out of 100 tries it identified a previously unknown zero-day vulnerability and provided a detailed explanation of how to deal with it.

Heelan verified this and officially reported it as '

CVE-2025-37899 '.

'O3's report shows that AI tools can be useful in vulnerability research. O3 has made great strides in its ability to reason about code. If you're an expert-level vulnerability researcher or exploit developer, O3 won't replace you, but it will make you a lot more efficient. There's a good chance that it will frustrate you with meaningless results, but it's also a good chance that it will get it right, so it's well worth using,' said Heelan.