Con: Can't See Everything
Among the areas pen testing falls short, Bellis said the craft can't be used to get a panoramic, 360-degree fix on the organizations entire security state.
"You won't find more than 2 percent of all your weaknesses," Bellis said. "You have to prioritize what you want that 2 percent to include, and that can be difficult."
Orbitz's priority is to protect customers from those who would use the company's websites to infect the customer -- a tall order in itself, Bellis said.
Con: Doesn't Always Work
Bellis also noted that like any security tool, sometimes the pen test won't work completely. Sometime a test will fail to find a serious weakness, he said. But then that's why it should only be seen as one tool in a larger security arsenal.
"The key is to know what you're expecting to find with a pen test and set expectations accordingly," he said. "In the end, though, no security tool is 100 percent effective on its own."
Turning back to Chess' prediction that pen testing would die out, Bellis noted that certain security technologies are always being marked for death. There was Gartner's prediction in 2003 that IDS was dead (intrusion detection and prevention systems live on today), for example.
"None of these work completely, but none of these are completely worthless," he said.