312-50 Question 180
Single answer▪ Anti-Malware SoftwareDuring an internal security assessment, you execute a custom PowerShell-based payload on a Windows 11 test workstation to validate the organization’s endpoint defenses. The payload is newly compiled, is not detected by signature-based scans at rest, and uses only built-in Windows APIs. However, as soon as it starts decrypting an embedded shellcode buffer and attempting in-memory execution, Microsoft Defender terminates the process and logs a behavior-based detection. The client asks which anti-malware capability most likely stopped the payload even though no known file signature matched it. Which feature best explains this detection?
- A
Signature-based scanning of the file hash against a local malware definition database
- B
Heuristic/behavior-based analysis that identified suspicious actions such as shellcode decryption and memory execution
- C
Disk quota enforcement that prevented PowerShell from allocating enough memory for the payload
- D
Full-disk encryption detecting unauthorized code running from user space
Show answer and explanation
Correct answer: B
Explanation
The best answer is heuristic/behavior-based analysis. In modern endpoint protection, anti-malware is no longer limited to static signature matching. Products such as Microsoft Defender Antivirus and broader EDR/EPP platforms monitor runtime behavior, API usage, memory allocation patterns, script execution, and other indicators associated with malware techniques. In this scenario, the payload avoided detection at rest but was blocked when it began decrypting shellcode and attempting in-memory execution, which is exactly the type of activity behavior-based engines are designed to catch. This reflects common best practices documented by major vendors and frameworks: use layered defenses that combine signatures, heuristics, cloud-assisted intelligence, and behavioral monitoring to detect novel or obfuscated threats. From a CEH perspective, the practical takeaway is that anti-malware controls can detect malicious technique patterns even when a payload is custom-built and has no known signature.
- A. Incorrect.
Incorrect. Signature-based scanning relies on known patterns such as hashes, byte sequences, or previously cataloged indicators. The scenario explicitly states the payload was newly compiled and not detected at rest by signature scans, which makes a pure signature match unlikely. Candidates may choose this because traditional antivirus is commonly associated with signatures, but the key clue is that detection occurred only when the payload began executing suspicious actions.
- B. Correct.
Correct. Modern anti-malware products, including Microsoft Defender, commonly use heuristic and behavior-based detection to identify malicious techniques even when a sample has no known signature. Actions like decrypting shellcode in memory, allocating executable memory, and attempting in-memory execution are classic indicators of fileless or evasive malware behavior. This aligns with real-world endpoint protection capabilities that monitor process behavior rather than relying only on static signatures.
- C. Incorrect.
Incorrect. Disk quota enforcement is an operating system resource management function, not an anti-malware control. It would not generate a behavior-based malware detection related to shellcode decryption or in-memory execution. This distractor is plausible only if a candidate confuses process termination due to system limits with security-driven blocking.
- D. Incorrect.
Incorrect. Full-disk encryption protects data at rest by encrypting storage media, but it does not analyze process behavior or detect malicious execution in memory. A candidate might choose this if they misunderstand the role of endpoint security controls versus data protection technologies. Encryption would not identify shellcode decryption or stop suspicious runtime actions.