Three Typical Challenges with EDR Analysis and Our SOC Team’s Approach

This article is English version of “EDRを使った分析の課題とSOCでの取り組みについて” translated by Hisayo Enomoto, NTTSH SOC analyst.

---

Our SOC has provided real-time analysis service of several Endpoint Detection and Response (EDR) solutions for about five years. We are analyzing alerts of several hundred thousand EDR endpoints around-the-clock and reporting incidents as needed.

The number of organizations implementing EDR solutions has grown rapidly over the last few years, and the same applies to our SOC’s clients. Some of our clients consider analyzing EDR solutions in their own organization’s security team; however, EDR analysis has some difficulties.

This article focuses on three typical challenges when analyzing EDR in your own organization. In addition, it explains our SOC team’s approach to these challenges.

Three Typical Challenges:

1. Although analyzing a huge number of EDR alerts requires time and effort, it is unlikely that an incident has actually occurred.
2. There is often a discrepancy between the severity of an EDR alert and the severity of an actual incident. Therefore, you should not blindly believe the Severity of an EDR alert.
3. Some cyberattacks are difficult to detect even by EDR solutions. EDR solutions alone can overlook incidents.

Challenge １

EDR solutions record the behavior of processes, files, registries and networks, and so on. For example, the activities such as "process started" and "file created" are logged, and the number of logs is approximately more than 10,000 per host per day as shown in Figure 1. The simple calculation shows that an environment with 1,000 EDR endpoints generates approximately ten million (10,000 logs×1,000 endpoints) logs per day.

In general, EDR logs contain more evidence of malware infections than network logs. Thus, EDR has an advantage that it is easy to investigate the extent of the impact of malware infection. However, one of the disadvantages of EDR is that the large number of logs make analysis time consuming.

Figure 1. Number of logs output from one EDR endpoint

In addition to simple logging capabilities, the EDR solutions leverages pattern matching and machine learning to generate threat activity alerts. Table 1 shows the number of incidents, alerts, EDR endpoints, and alerts per EDR endpoint for one month. As you can see from the table, the number of incidents is very few compared to the number of alerts. This indicates that most of the EDR alerts are False Positive. Some examples of False Positive alerts observed in our SOC include behavior of system monitoring tools and in-house software, and besides, installation behavior of general-purpose software.

Another point of view is that the number of alerts per endpoint is a big difference for each organization. This indicates that it can vary greatly depending on the specifications of EDR solutions or the organization’s system environment. The number of alerts tends to be particularly high in organizations that give each user permission to install software.

Table 1. Number of incidents, alerts and endpoints per organization

From the above, one of the challenges associated with EDR analysis is that it takes a lot of effort to analyze numerous logs, including False Positive.

Challenge 2

Our SOC analyzes all alerts generated by EDR solutions, regardless of their severity. This is because our many years of analysis experience have identified a discrepancy in severity between EDR alerts and actual incidents. For example, the alert severity was Low, but in fact it was a detection of malware infection, while the alert severity was High, but it actually detected legitimate internal operations.

Figure 2 shows the percentage distribution of severity of EDR alerts that detected malware infection or successful remote attacks over the past three months. This graph shows 50% of the alert severity was High, 21% was Medium and 29% was Low.

Therefore, filtering Low level alerts to reduce the volume of alerts can lead to overlook incidents.

Figure 2. Percentage of EDR alert Severity at the time of incident detection

The following is an example of an incident where the alert was Medium, but it was actually a malware infection caused by a targeted attack.

An alert detected generation of unsigned DLL with suspicious filename. This was a very generic alert. The hash value of the DLL had not been present in any threat intelligence services, but this situation was relatively common. Most of the cases were installation behavior of general-purpose software. However, our detailed analysis discovered that this case was malware infection caused by a targeted attack. Figure 3 shows the malware infection chain revealed by our EDR analysis.

Figure 3. Malware infection chain

① A downloader creates a signed legitimate executable file (legitimate.exe) and a malicious DLL file (malicious.dll) in the ProgramData folder (EDR detected the creation of the DLL file).

② The downloader executes “legitimate.exe” and the legitimate executable file loads “malicious.dll” in the same folder (DLL Side-Loading).

③ The “malicious.dll” communicate with C&C server through the legitimate executable file process.

④ Bypass the User Account Control (UAC) by using the “HKCU\Software\Classes\Folder\shell\open\command” registry key.

⑤ Register a scheduled task to run the suspicious DLL every 10 minutes by using the “schtasks” command.

As noted above, judgment based on EDR alert severity can lead oversight of incidents or unnecessary incident responses.

Challenge 3

EDR solutions primarily monitor and log the behavior of processes, files, registries and networks. However, due to the tremendous amount of these logs, monitoring all these logs would lead to high-load conditions of EDR solutions or computers. Therefore, some behaviors are excluded from monitoring and logging. The behaviors are slightly different for each EDR solution, but for example, logging of some registry keys and file types is suppressed as well as file access behaviors. In addition to these, behaviors of executing in-memory or commands of interactive shells are often excluded from monitoring and logging.

Therefore, it is difficult to detect cyberattacks exploiting gaps in the specifications of these EDR solutions. The following are typical attacks that are difficult to detect in EDR solutions.

・Info Stealer Malware

The prominent behaviors of Info stealer malware are the process start of the malware itself, file access and network communication of data exfiltration. For example, it collects password information stored in an infected computer’s browser and sends the sensitive data from the compromised system to a C&C server.

One of the characteristic behaviors of info stealer malware is to access files where passwords are stored. However, as explained above, some EDR products suppress logging of these behaviors, making them difficult to detect. For more information, visit our blog.

・Remote Access Trojan (RAT)

RAT has various features:

• Keylogging
• Capturing screen and audio
• Searching, accessing or downloading files
• Executing Windows commands

Before using these features, the only infection behavior of the RAT is to launch RAT process on the victim host and communicate with a C&C server. These small behaviors make it difficult to identify and detect the RAT.

In addition, Some RATs are modularized, and a RAT operator downloads the required features from the C&C server as needed and runs them in memory. In this case, the only visible behaviors are process initiation and network communication, making detection difficult.

・Attacks Using Interactive Shell

There are many attack tools such as Metasploit and Mimikatz that have the ability to execute malicious commands interactively.

In most cases, EDR solutions are unable to log interactively executed commands by these attack tools. Therefore, similar to the RAT malware described above, the only visible activities are process initiation and C&C network communication.

・Vulnerability Exploitation

A product vulnerability could allow an attacker to execute arbitrary code using shellcode. Because the shellcode runs in memory, its activity is not logged. This makes it difficult for many EDR solutions to detect.

To be clear, not all of the above attacks are undetectable with EDR solutions. Some EDR solutions have the ability to detect these attack methods.

Next, we look at cyberattacks that use EDR bypass techniques. This is a method of avoiding detection, for example, by disguising attacks that are inevitably logged by EDR solutions as legitimate product behaviors. EDR bypass techniques are now used not only in targeted attacks but also in indiscriminate attacks. Common techniques for EDR evasion are shown below:

EDR Bypass Techniques

• Using standard Windows commands frequently
• Using renamed standard Windows commands
• Using commonly used product filename for malware
• Obfuscating a command line
• Using a code-signed malware
• Injecting into a legitimate process
• DLL Sideload (DLL Hijack)
• Process Herpaderping

Figure 4 shows one of the examples of attacks to bypass EDR detection.

Figure 4. Attack chain attempted to bypass EDR detection

① Open a doc file and enable macros lead to download two encoded files from a malware distribution site.

② The macro executes “copy” command to rename certutil.exe, the standard Microsoft Windows program, to an alias file name.

③ The macro concatenates the two downloaded files into one file.

④ Decode concatenated data using renamed certutil.exe. This data is malware.

⑤ A standard Microsoft Windows program mavinject.exe injects malware into a legitimate svchost.exe that is already running.

⑥ The malware uses the injected svchost.exe to communicate with a C&C server.

The following EDR bypass techniques are utilized in this attack chain:

• Use macros to download encoded data to victim host
• Copy the standard Microsoft Windows program certutil.exe and start it with a different file name
• Use an asterisk (*) to obscure the original filename when copying certutil.exe to another filename.
• Inject into the legitimate process using the standard Microsoft Windows program mavinject.exe.

As shown in the example above, when conducting EDR analysis, if you do not understand the product specifications and characteristics, you may overlook attacks that attempt to avoid detection.

Our SOC team’s Approach

We have introduced three major challenges in terms of EDR analysis, and the following section describes how our SOC is addressing these difficulties.

・Developing Automatic Analysis Tools

In order to determine whether an alert indicates the actual incident, behavior of user’s manipulation or an installation of legitimate software, we need to search and analyze information related to the alert from a huge number of logs.

For example, if the alert indicates suspicious file creation, we check the file information (filename, path information, hash values, etc.) and obtain relevant information such as the process information that created the file or the parent process information that called the process. Based on that information, we make a comprehensive decision whether there is anything suspicious.

Some EDR solutions have a function that automatically extracts that information and displays it graphically. However, as they are not sufficient for analysis, our SOC has developed our own tool to display rich information to improve the efficiency of analysis.

・Filtering Unnecessary Alerts

Our SOC obtains alert information through APIs, but a single API query does not provide enough information of an alert. Therefore, we execute multiple API queries to get as much detailed information as possible about the alert. We use this information to sort out which alerts should be analyzed, and which ones are not, regardless of the Severity. Specifically, if an alert of a suspicious process launch is triggered, we filter it by process name, path, command line, parent process, child process, code signature.

・Creating Custom Signatures

Our SOC collects and analyzes massive logs from various security devices of our domestic and international clients. In addition, we also use our original Sand boxes to gather information about cyberattacks. These collected indicators are used to create custom signatures to detect attacks that are undetectable by EDR vendor signatures. For more information, visit our blog.

・Utilizing Network Logs

Our SOC utilizes network logs as well as EDR endpoint logs for analysis. Proxy logs and IPS/IDS alerts will contribute to detecting attacks that are difficult to detect by EDR solutions.

・Creating EDR SIEM rules

EDR essentially generates alerts by matching a single action, such as creating a file or launching a process, with the IOC. However, such a single action may fail to detect cyberattacks, which are becoming more sophisticated day by day. Therefore, our SOC collects audit logs and creates SIEM rules that combine alerts generated by EDR and the audit logs to detect sophisticated cyberattacks.

In addition, our SOC is making various efforts regarding EDR analysis. For example, we observe the behavior of malware samples in a Sandbox environment where EDR solutions are deployed. We then use that EDR logs to conduct EDR analysis training, accumulate knowledge about malware, and understand the unique analysis techniques of each EDR solution.

Our SOC implements the initiatives described in this article and makes improvements as needed to achieve highly accurate threat detection in order to protect our clients' environments from increasingly sophisticated cyberattacks.