Is AI pentesting as good as a human pentester?

With the rise of AI pentests, many people are wondering whether they are as good as a human. I started as a pentester twelve years ago and have gone through similar roles that I have worked on Intruder AI pentester,

I understood the pros and cons of both. This is how they compare.

The strict methodology was never a good thing

You can expect AI pentesters to follow strict methods, as is the case with many human pentests.

At Intruder, we experimented with this approach early on – structured systems that required each agent to search for a specific type of vulnerability. We found that agents missed things that were outside of their jurisdiction or misclassified what they found as the only thing they should be looking for, resulting in confusing reports.

A complete lack of structure also did not produce good results, as completely free agents could not properly break down and manage large applications without any guidance. They might find interesting things, but at the cost of missing a lot of other vulnerabilities in our targets.

We conducted a lot of testing to find the balance between these approaches and agreed to give agents freedom with broadly defined roles to help them tackle and manage large applications, as well as stricter guardrails to prevent them from causing harm.

However, this means that our AI pentester will not check off the parts of a methods document that you would normally see in a human pentest. This can make some people uncomfortable – a strict methodology gives people peace of mind that someone has gone through your application and tested everything on the list. An unauthenticated SQL injection must be found and there is a checklist to ensure this is the case.

The methodology gives many people the impression that a human pentest is a perfectly repeatable product because any tester can follow the methodology and everything important will be found. However, the reality is much more nuanced, as pen testing is a demanding task and many of the most powerful results from a pen test come from the tester having experience noticing strange behaviors and tracking them.

For our AI pentester, we ended up with much more interesting and meaningful results because we weren’t trying to walk our agents through specific vulnerability types. A good example is a complex authentication bypass where agents were found sitting in a codebase performing pentests for five years without a human tester recognizing them.

The mechanics of this problem mean that one would never think to write a testing guide that could be strictly followed to identify it, and if we had tried to limit our AI pentester too much, he would never have found it either.

You could say that just finding the usual things would still be pretty good, but there are so many vulnerabilities that you don’t often find in a pentest. And just because they aren’t common doesn’t mean an attacker can’t find them, especially now AI is being used to enable more attacks.

A strict methodology conducted by a competent tester only ever provided a low level of baseline reliability, and that wasn’t even perfect. What makes a pentest good is the part that an experienced tester does outside of the checklist, and this is where AI thrives: it follows its nose through your application and finds the things that would never make it onto a list.

What they know and what they can’t ask

Our AI pentests take one White box approachwhich means the agents receive the source code. You work through the entire codebase and build a picture of not only what each page is, but also how the application should work and what requests can be made. Doing this at the level of detail that AI can achieve is not feasible for a human pentester during a typical pentest window.

The AI ​​pentester also develops a sense of what the application is intended for, and a good example of this is access control – there is no need to say that low-privilege users should not have access to admin functionality or that a particular bit of data is confidential.

You can also give them additional information, just like a human tester. For example, people often say they are worried about their customers accessing each other’s data, and the pentester did a great job focusing on these issues and identifying many of them.

What agents can’t do is ask you for anything you haven’t already given or told them. During a pentest, if I don’t understand what part of an application does, I can ask my contact person. These conversations can be very useful to properly test the application and assess the risk in the final report.

If necessary, I can also clarify the scope with my contact person or ask permission to check something if I am not sure whether there is a risk of disruption.

We had to consciously urge our AI pentesting agents to be cautious in this regard so as not to go outside the box.

Neither finds everything

The freedom that AI pentesting agents allow to find something like a complex authentication bypass also makes them less predictable. You decide where to dig, and it won’t be the same every time.

However, the results of pentests always varied depending on the person conducting them. This is why many consulting firms change their tester every year – different people find different things on the same application.

This depends on people’s experience, expertise and additional knowledge.

I encountered issues based on a blog post or spec I happened to read years ago that stuck in the back of my mind. And even the things on a checklist are sometimes overlooked. Humans are fallible and it can be difficult to methodically cover an entire application over several days.

We are actively working to improve consistent coverage, but no one has ever offered it perfectly. This is why every human pentest report contains a disclaimer stating this, even if most people understand this as a formality.

I couldn’t have written the remedy

Every pentester has a clear opinion about what makes a good report. I have built many of my own over the years, as have the rest of the current and former pentesters at Intruder. We based our report on these opinions.

The executive summary is the most difficult page of a report to get right. For many people, it is their entire experience with your work, and good advice puts a lot of effort into it. We’ve worked to get to the point where agents now write really good reports, and I often read them first in our own reports to get a good overview before going into detail.

What they still lack is the goal risk. They rate things higher than I do, partly because they can’t have the conversations with you that would lead to a downgrade in rating, and partly because LLMs tend to be cautious when you start talking about cybersecurity.

That’s the right direction to go wrong, but it also means things are labeled as more serious than they are, and that’s what we’re working on.

Restructuring advice is the agents’ strength. After a long week on a pentest, it’s tempting for a person to write general advice and you find yourself trying to do better. Unfortunately, they are usually limited in time and usually do not have the source code.

The agents have neither problem, so for each problem they tell you which line of code you need to change and what you need to change it to. They also write about mitigating controls that can help reduce the impact of the problem or provide a temporary solution.

We see that this is accelerating the speed at which people actually repair things. We had a customer with 78 issues who fixed them all within a few days, something I’ve never experienced as a human pentester. I could not have written the level of detail that would have made these corrections possible in the time I had.

So is AI pentesting as good as a human?

AI pentesting and human pentesting are different, and when you stop ranking them, the comparison becomes much more useful.

The agents find serious problems in applications that human testers have already run through multiple times. They read an entire code base, work through every path within it, and write down every instance of what they find with the fix down to the line of code.

And you can run one whenever you want. For the cost of a single human engagement, you can run four AI pentests, meaning you can run comprehensive tests on a regular basis rather than just once a year.

This is an important thing that is overlooked in human pentests, and often applications have undergone many untested changes between pentests.

See what Intruder’s AI pentester finds in your application. Learn more.

author

Written by Daniel Thatcher, Security Research Engineer, Intruder

Daniel is a security researcher at Intruder with experience in penetration testing and a focus on exploiting web applications in black box scenarios.

Sponsored and written by intruder.