Indie Security Product Testing Is Really Hard — and More Important than Ever

Product testing
Lorna Garey

Lorna Garey

Customers are doing more of their own purchasing research, so it’s likely you’ve had a call asking about lofty promises by security vendors. A little hyperbole is to be expected. This is a crowded field — the 2017 version of Gartner’s Magic Quadrant for endpoint protection alone includes 22 providers. Customers are rightly worried about ransomware and other threats, and are looking for a magic bullet. How do you as trusted adviser vet vendor claims?

“I talk to a lot of partners who say, ‘Oh, we tested it ourselves,'” Dan Schiappa, senior vice president and general manager of Sophos’ end-user and network security groups, told me at the company’s recent partner summit. When asked how: “We went to VirusTotal and downloaded a hundred pieces of malware.”

The problem with partners simply throwing a few hundred known exploits at a firewall or AV engine is twofold. First, you may end up with a false sense of how well the product will perform in the real world, and not just because of zero-day attacks. These tests often lack the proper context, including where the malware came from.

“Did it come from a browser? Did someone click on it? Did someone download it?” said Schiappa. “If you just throw it in a folder and run it, you’ll lose a lot of that context … it’s a terrible way to test real-world scenarios.”

And, next-gen products that depend on behavioral heuristics and post-execution detection could underperform. Schiappa points out that simply asking, “Did [product] catch an executable before it ran?” isn’t going to evaluate the efficacy of software from companies such as CrowdStrike, Cylance or SentinelOne, whose primary value proposition is behavior-based detection. That is, the systems watch an executable run, evaluate its actions and then decide whether to block it. As security vendors use more machine learning to spot malware trying to exploit something, rather than just shutting down known attacks or carpet bombing all suspect processes, the need for sophisticated, independent testing gets more urgent.

I spent several years as a reviews editor, so I get how complex any product evaluation is. Vendors will argue endlessly over every tiny detail of a testbed setup — as they should, because the stakes are high. Few partners, or even master agents or distributors, are equipped or, frankly, willing to spend the money and time to do in-depth testing of technology, especially a tech as complex and fast-changing as security. Yet customers depend on you for informed advice.

What about depending on tests the vendor did internally? That’s sort of like a take-home midterm exam, if you let the student also write the questions. Instead, when evaluating a new security provider, ask whether it has results from reputable third-party testers such as MRG Effitas, NSS Labs, AV-Test and SE Labs.

“MRG doesn’t have a standard published test, but man, when they test your stuff, they go to the dark web, they do some real testing,” said Schiappa. “We hired them to do the first Intercept X test and they’re like ‘We’re cashing your check and whatever comes out, comes out. If it comes out terrible, we’re going to …

Pages:  1 2 3 Next

Leave a comment

Your email address will not be published. Required fields are marked *

The ID is: 64775