Graduation Year


Document Type




Degree Name

Doctor of Philosophy (Ph.D.)

Degree Granting Department

Computer Science and Engineering

Major Professor

Zhuo Lu, Ph.D.

Co-Major Professor

Yao Liu, Ph.D.

Committee Member

Jay Ligatti, Ph.D.

Committee Member

Attila Altay Yavuz, Ph.D.

Committee Member

Kaiqi Xiong, Ph.D.


Adversarial Machine Learning, MU-MIMO, Network Security, Topology Inference, Topology Obfuscation, User Selection


Machine learning plays a vital role in understanding threats, vulnerabilities, and security policies. In this dissertation, two machine learning empowered approaches on improving the security of critical cyber infrastructures and functionalities will be discussed.

The first work focuses on preventing attacks that use adversarial, active end-to-end topology inference to obtain the topology information of a target network. The topology of a network is fundamental for building network infrastructure functionalities. In many scenarios, enterprise networks may have no desire to disclose their topology information. To this end, we propose a Proactive Topology Obfuscation (ProTO) system that adopts a detect-then-obfuscate framework: (i) a lightweight probing behavior identification mechanism based on machine learning is designed to detect any probing behavior, and then (ii) a topology obfuscation design is developed to proactively delay all identified probe packets in a way such that the attacker will obtain a structurally accurate yet fake network topology based on the measurements of these delayed probe packets, therefore deceiving the attacker and decreasing its appetency for future inference. We show that ProTO is very effective against active topology inference with minimum performance disruption. Experimental results under different evaluation scenarios show that ProTO is able to (i) achieve a detection rate of 99.9% with a false alarm of 3%, (ii) effectively disrupt adversarial topology inference and lead to the topology inferred by the attacker close to a fake topology, and (iii) result in an overall network delay performance degradation of 1.3% - 2.0%.

In the second work, we aim to investigate the potential attacks against CSI-based user selection algorithms, reveal the impacts of such attacks, and derive corresponding counter- measures to improve the security of MU-MIMO networks. WiFi 5/6 relies on a key feature, Multi-User Multiple-In-Multiple-Out (MU-MIMO), to offer high-volume network throughput and spectrum efficiency. MU-MIMO uses a user selection algorithm, based on each user’s channel state information (CSI), to schedule transmission opportunities for a group of users to maximize the service quality and efficiency. In this work, we discover that such algorithm creates a subtle attack surface for attackers to subvert user selection in MU-MIMO, causing severe disruptions in today’s wireless networks. We develop a system, named MU-MIMO user selection strategy inference and subversion (MUSTER), to systematically study the at- tack strategies and further to seek efficient mitigation. MUSTER is designed to include two major modules: (i) strategy inference, which leverages a new neural group-learning strategy named MC-grouping via combining Recurrent Neural Network (RNN) and Monte Carlo Tree Search (MCTS) to reverse-engineer a user selection algorithm, and (ii) user selection sub- version, which proactively fabricates CSI to manipulate user selection results for disruption. Experimental evaluation shows that MUSTER achieves a high accuracy rate around 98.6% in user selection prediction and effectively launches the attacks to disrupt the network performance. Finally, we create a Reciprocal Consistency Checking technique to defend against the proposed attacks to secure MU-MIMO user selection.