The arms-race between cyber attackers and security researchers is forcing defense mechanisms to move down the software/hardware stack. Compared with software approaches, hardware-supported security systems are harder to compromise and have lower performance overhead. In this thesis, we explore hardware-supported security systems from three perspectives: detection of attacks, diagnosis of vulnerabilities and building efficient defenses.

This thesis first explores the feasibility of using architectural footprints for malware detection. We present a framework for hardware-assisted malware detection based on monitoring and classifying memory access patterns using machine learning. This frame- work is applied to the application-specific malware detection scenario which targets detecting malware-infected runs of known applications and is evaluated for both kernel and user level attacks. An experimental evaluation with practical traces shows a detection rate above 99.0% with less than 5% false positives which outperforms previous proposals for hardware-assisted malware detection.

Next, we introduce a fully automated method for malware analysis that utilizes memory access traces from program execution. While machine learning methods are effective in classifying malware attacks based on hardware features, they do not help diagnose the vulnerabilities that were exploited in a particular attack. This method fills this gap by using a novel memory trace data analysis method to help identify the vulnerabilities. An evaluation using the RIPE memory attack benchmarks demonstrates its capability to accurately perform diagnosis and characterize different attacks.

Third, we present ScopeTag, a RISC-V based architecture prototype designed to stop both control-flow attacks and data-oriented attacks. ScopeTag prevents these attacks by enforcing the data-flow scope of any untrusted source to ensure the integrity of a program’s critical data. It utilizes a static analysis to define the data-flow scope of untrusted sources and uses a tagged memory system to enforce this scope during program execution. Experimental results show that this architecture protects the system against both control-flow attacks and data-oriented attacks for different known vulnerabilities.

In summary, this thesis presents novel secure architectures and methods to utilize hard- ware features for system security. They provide lower overhead and a reduced trusted computing base compared with common software approaches.

Adviser: Sharad Malik