The Intelligence Pipeline
Threat hunting is more than comparing provided indicators of compromise (IOCs) to collected data and finding a "known bad." Threat hunting relies on the application and analysis of data into information and then into intelligence – this is known as the Intelligence Pipeline. To process data through the pipeline, there are several proven analytical models that can be used to understand where an adversary is in their campaign, where they'll need to go next, and how to prioritize threat hunting resources (mainly, time) to disrupt or degrade an intrusion.
The Intelligence Pipeline isn't my invention. I first read about it in an extremely nerdy traditional intelligence-doctrine publication from the United States Joint Chiefs of Staff, JP 2-0 (https://www.jcs.mil/Portals/36/Documents/Doctrine/pubs/jp2_0.pdf). In this document, this process is referred to as the Relationship of Data, Information, and Intelligence process. However, as I've taken it out of that document and made some adjustments to fit my experiences and the cyber domain, I feel that the Intelligence Pipeline is more apt. It is the pipeline and process that you use to inform data-driven decisions:
Figure 1.2 – The Intelligence Pipeline
The idea of the pipeline is to introduce the theory that intelligence is made, and generally not provided. This is an anathema to vendors selling the product of actionable intelligence. I should note that selling data or information isn't wrong (in fact, it's really required in one form or another), but you should know precisely what you're getting – that is, data or information, not intelligence.
As illustrated, the operating environment is everything – your environment, the environment of your trust relationships, the environment of your MSSP, and so on. From here, events go through the following processes:
- Events are collected and processed to turn them into data.
- Context and enrichment are added to turn the data into information.
- Internal analysis and production are applied to the information to create intelligence.
- Data-driven decisions can be created (as necessary).
As an example, you might be informed that "this IP address was observed scanning for exposed unencrypted ports across the internet." This is data, but that's all it is. It isn't really even interesting. It's just the "winds of the internet." Ideally, this data would have context applied, such as "this IP address is scanning for exposed unencrypted ports across the internet for ASNs owned by banks"; additionally, the enrichment added could be that this IP address is associated with the command and control entities of a previously observed malicious campaign.
So now we know that a previously identified malicious IP address is scanning financial services organizations for unencrypted ports. This is potentially interesting as it has some context and enrichment and is perhaps very interesting if you're in the financial services vertical, meaning that this is information and is on its way to becoming intelligence. This is where most vendors lose their ability to provide any additional value. That's not to say that this isn't necessarily valuable, but an answer to "did this IP address scan my public environment and do I have any unencrypted exposed ports?" is a level of analysis and production that an external party cannot provide (generally). This is where you, the analyst or the operator, come in to create intelligence. To do this, you need to have a few things, most notably, your own endpoint and network observations so that you can help inform a data-driven decision about what your threat, risk, and exposure could be – and no less importantly, some recommendations on how to reduce those things. The skills that we'll teach later on in this book will discuss how we can do this.
As an internal organization, rarely do you have the resources at your disposal to collect the large swaths of data needed to (eventually) generate intelligence. Additionally, adding context and enrichment at that scale is monumentally expensive in terms of personnel, technology, and capital. So acquiring those services from industry partnerships, generic or vertical-specific Information Sharing and Analysis Centers (ISACs), government entities, and vendors is paramount to having a solid intelligence and threat hunting program. To restate what I mentioned previously, buying or selling "threat intelligence" isn't bad – it's necessary, you just need to know that what you're receiving isn't a magic bullet and almost certainly isn't "actionable intelligence" until it is analyzed into an intelligence product by internal resources so that decision-makers are properly informed in formulating their response.