Vetting your vendors: money isn’t everything

Over the past year, we’ve seen a handful of cyber threat intelligence vendors sputter out. Some with an effluence of defensive lawsuits, some charged with not really doing much of anything, and one that ended with a dramatic FBI raid and numerous charges of hacking and extortion. What’s concerning from a CSO’s perspective is the veneer of legitimacy all these companies had; scammy cybersecurity companies generally have slick, professional websites, convincing sales engineers, legions of on shore support executives, and almost invariably, one or more executives with ties to a government intelligence agency, whether in the US or abroad. So given that almost all cybersecurity companies on the market strive to project an image of the quiet professional, how can a CSO sort out companies that are a value add from those with a less than legitimate business model? And what about the companies that are above board, but just not very good? Let’s take a look.

The Ugly

Most harmful to a business in the long run are the security vendors who either don’t do much of anything, or have a business model that skirts the edge of the law. The simplest and most cost effective way of avoiding these companies is doing a community temperature check. Bad vendors tend to acquire a collective disapproval in the infosec community long before their business model fails. A quick Twitter or Google search of the vendor name can often reveal detailed accounts by analysts who have used them and can provide candid assessments.

But the gold standard for a temperature check is to ask your own team. Cross-pollination of infosec personnel is at an all time high – as such, your team most likely has a broad range of experience with multiple vendors on a host of platforms. Your team can provide invaluable data, like added operations costs over the long term, company billing practices, and interoperability with existing systems. They can also tip you off on issues with vaporware; generally defined as giving the appearance of having a product which is in reality much more limited or even non-existent.

Like most vendors of higher quality, the ugly will also have former intelligence agency personnel to give themselves a veneer of authority and competence. A question that rarely gets asked, though, is “Which agency?” Is it an agency with a formal mandate for addressing cyber threats, with an established university pipeline and well regarded reputation? Is it an agency whose cyber division was stood up relatively recently, with repurposed employees from other departments? Further, how relevant is that experience to your business needs? If the majority of your security losses are coming from phishing and malvertising, is having access to analysts experienced in state sponsored intrusions really relevant?

The Bad

Some infosec vendors really do try their best to provide a valuable product to the end user…but fall awfully short of the mark most of the time. The problem here isn’t that they’re not trying to deliver a good product – it’s that they don’t necessarily understand what ‘good’ is to you. In the public sector, intelligence is often defined as information that is timely, accurate, and relevant. This applies to cyber threat intelligence as well. If you kick out any one of the legs on the threat intelligence tripod, you’re left with a platform too unstable to make any reliable judgement on cyber risk. An organizational threat delivered to SOC personnel in a timely manner that hasn’t been vetted (i.e. inaccurate), is not intelligence. Threat data that is timely and accurate, but not adapted to your business vertical (i.e irrelevant) is not intelligence. What these things amount to tends to be a drag on organizational resources as in house security personnel get tasked with vetting ever increasing quantities of data that doesn’t address business needs. Don’t those tier two SOC techs have better things to do than retrace vague, un-targeted analysis?

Bad cyber threat intel vendors often correctly identify the desired end goal of intelligence, but lack an understanding of appropriate methodology. Again, these companies often out themselves as undesirable with a quick community check. A poorly sourced, unreviewed report using inflated claims will quickly reveal itself as such as the infosec community reviews the content. Timely, accurate, and relevant threat data will be shared, retweeted, and commented upon much more frequently then less useful sources. Pausing for a moment to see how other organizations have integrated threat data being offered to you can provide a valuable check against letting a bad vendor slip through the cracks.

Some questions to ask the sales engineer:

• How will this data be tailored to my organization?
• How is the data delivered to us, and if it’s a portal, what is your upgrade release schedule?
•  And most importantly: How do you vet your sources?

Note: do not accept “We have to protect our sources and methods.” This is a phrase borrowed from government intelligence, who generally uses it in situations involving threats to human lives. More commonly, it’s used to express sentiments akin to “I’m not going to tell you because I don’t want to, don’t know, or it would embarrass me.”

The Good

Here’s the most difficult category and the holy grail for augmenting your security team: a company that delivers well-targeted services to your organization in a manner that is timely, accurate, and relevant. The catch here is that to properly spot the good company, your own organization has to have timely, accurate, and relevant defined down to a T. This brings us to the last and most important aspect of vetting: metrics.

Certain companies can provide an awfully impressive “real time demonstration” of the product, sometimes offering you a head to head with competing products. They might reference number of threats detected, speed of detections, or analysis, or number of endpoints providing data. There is a barrage of cybersecurity metrics available to benchmark performance, so how do you know which are valuable? The answer is none of them. The only metric relevant to evaluate security performance is that which has been generated by your own team against a mature risk tolerance posture. Vendor metrics can’t possibly address the various risk tolerances of all their customers and therefore can’t be relevant to how they would perform for you. Once you know your own metrics, evaluating vendors can be a piece of cake. (And requires much fewer meetings.)

Some questions to ask the relationship manager for a great vendor:

• How can I share feedback from my security team?
• When can we revisit my business needs?
• What improvements do you have planned for next quarter.

To sum up, vetting vendors doesn’t have to be painful – if you know your risk tolerance posture, and have a mature communication channel with your own security team.

https://blog.malwarebytes.com/feed/