Is penetration testing still effective?

This article was inspired by a mix of recent events and a discussion with a client that asked the question ‘has testing now become so artificial that it is no longer real?’ Having recently started a new business and witnessed the evolution of the security market in Sydney, I find myself as a seasoned tester questioning the value of penetration testing in its current form.

I had to reluctantly agree that the artificiality of a lot of testing has come from the evolution (or devolution) of the testing process. What was once a bespoke field has now become a necessity for a lot of organisations. However, its success and growth has also led a fall in quality.

I see six common faults on the ground:

1. Insufficient scope or context

Hackers without a cause no longer exist, and security vulnerabilities nowadays demand much more context. If the scope of an attack is not understood, if a reason for targeting an organisation is not obvious, or if the broader landscape is unknown, then all you’re doing is spitting out an automated test or vulnerability scan.

Businesses should understand that they, not their technologies or applications, are the targets and testers should understand the importance of reconnaissance in planning its role in facilitating this fact.

Taking the time to identify flaws in business logic will be more rewarding for an organisation and help drive future security planning. In one case with a customer a half hour scoping activity identified IP address ranges that they were unaware of, as well as a misunderstanding of patch management based on what was observed within these ranges. If nothing else, scope and context will identify the most rewarding opportunities for testing in an already constricted time period. I will be writing a future article on how to scope properly and what the benefits are (watch this space).

2. Poor time/space analysis

Anyone who says they can complete testing in less than eight hours at a premium rate may be wasting a business’s time. A lot of penetration testing firms will bank on short, high intensity work with full utilisation periods in order to generate revenue. While all this approach typically needs is a single flaw or a caveat that testing was time-boxed, the reality is that by taking their time a tester will be more conversant with the environment and the organisation, and subsequently provide value for money.

Unfortunately, cost benefit analysis all too often falls back on cost. Businesses should evaluate testing relative to what they want to achieve and how. Testers should have a plan that provides confidence in the assurance process, confirms that the time taken is appropriate, and achieves its purpose with an economy of effort.

Sitting down for a brief planning session, will save time and help shape both time and targets which will make for an effective plan.

3. Understanding qualification, skill and aptitude

This has long been a contentious item within the security community; how do we assure we’re providing appropriate talent and vetting ourselves without providing a false sense of security? Academic qualifications have always represented a baseline, as have some industry certifications. Having said this, customers may not fully understand a qualification or its limitations. This is a greater concern for the security community than the need for its members to achieve qualifications.

Skill and aptitude can be very difficult to measure, however having a standard does help. The Certified Registered Ethical Security Tester (CREST) has helped evolve the measurement of evaluating skills and I think that this will play more in our future evaluation of individuals as we start to move away from quantifiable measurement. Hackers qualify their skills with little more than results and I can only hope such regimes bring about in this approach. In the interim, I personally am looking at people at a grass roots level; are they getting involved and do they love what they do? I would encourage providers & consumers to seek this in their testers; whilst it may not be quantifiable the end result is a more well rounded tester who understands their role in the security assurance process.

4. Effective conduct of testing

A sense of structure goes a long way when testing is performed. Up to this point, good strong lead-in tasks should have set the testers up for success. The following activities indicate effective testing:

• Keeping the customer informed of start/stop times and updates as to what you’re doing. It may sound counterintuitive to testing and ‘keeping it real’, but it’s surprising how the flow of information keeps it effective.
• Well documented notes. While I still use pen and paper I had, in previous organisations, used note templates to provide hints and reminders of what I should be doing. In order to have the ‘offensive’ mind ticking over, notes taken against categories or attacks will help provide a clear picture.
• Cool, calm and measured. Hitting something with a tool or automation triggers alerts and can get people upset. At best you’ve demonstrated that defensive technologies will detect and react against common platforms. A  tester who really wants to be effective should be able to get through with little or no detection or be able to identify where a defender needs to ‘plug a gap’.
• Ongoing appreciation ‘on the ground’. All plans work until the first shot is fired. What happens if i trigger a denial of service condition? Who do you go to if there’s evidence of prior compromise? How do you act if what you’ve found is inconsistent with what’s been detailed by the customer? What if the environment suddenly blows out into something larger than anticipated? These are things that need to be thought through before and during testing and testers on the ground need to understand this. 

5. Effective outputs of testing

A 400 page report may pass the weight test for a lot of bureaucrats, but will this achieve the result that’s intended? Emphasis on reporting has been a distraction − where an engagement demands an extended report, that’s less time spent on identifying issues and providing practical assistance. I’m a firm believer that shortened reports, video and working alongside stakeholders will ensure that testing is effective.

Procurers of testing need to understand what they want. If it is a report to meet ‘XYZ’ compliance, go for it but sometimes you can save on reporting costs if it’s not what is needed.

Testing firms should have a diverse set of report offerings; if you’ve got a generic template that hardly changes (and regularly features other customers’ names) then you’re doing it wrong.

Human interaction − sitting down with other technically oriented individuals and talking through − will resolve more current future vulnerabilities than a one way paper exchange.

6. The self fulfilling prophecy

My final point and perhaps the most frustrating one is the use of penetration testing as a gateway to more services or specific products. Nothing will undermine the effectiveness of testing (at the individual level and in the community) than the absence of independence.

Before engaging an organisation for testing, customers should understand who the organisation is. Should it happen to be a systems integrator, they should be prepared for the fact that they might be paying for a means to an end.

Validation of independence can only come with time. If you offer a good, objective service that is not used to generate sales, people will keep coming back.

Penetration testing is falling into the same hole as the antivirus industry. Vulnerability assessment services attempt to identify a known attack vector using a known signature which will equate to exploitation. This has become a substitute for penetration testing for a lot of organisations because of its price and ease of use. While penetration testing is at a crossroads I can only foresee that astute consumers will hold it to account, and that providers will adapt or disappear.

About the author

Edward Farrell is a seasoned penetration tester and information security consultant with nearly 10 years’ experience. In 2015 Edward sought to go out on his own and created Mercury Information Security Services. Edward’s new organisation provides customised information security services and advice for Australian businesses.

This article is brought to you by Enex TestLab, content directors for CSO Australia.