Choosing data that matters
Quality data management is critical to the success of big data analytics and forms the basis of how an SIEM solution works. Gathering large volumes of data for analysis is required to find security threats and unusual behavior across a vast array of infrastructure and applications. However, there needs to be a balance between capturing too little and too much data. Too little data will mean not having enough to find correlating activities, but too much data will increase the signal noise associated with alert fatigue and will increase the cost of the security solution to store and analyze the information. In this case, the security solution is Azure Log Analytics and Microsoft Sentinel, but this principle also applies to other SIEM solutions.
A recent shift in the data security landscape is the introduction of multiple platforms that carry out log analysis locally and only forward relevant events to the SIEM solution. Instead of duplicating the logs, hoping...