2.6 Key Components of Threat Models in ML

Threat actors are entities that exploit ML system vulnerabilities. They can be categorized as:

• External attackers: Hackers or cybercriminals attempting to manipulate or steal ML models.
• Malicious insiders: Employees or researchers with access to ML training data who may leak or misuse information.
• Competitors: Rival organizations trying to extract or replicate proprietary models.

Attack surfaces define the entry points where adversaries can target an ML system:

• Data (Training & Input Data): Adversaries can poison datasets, introduce biased samples, or manipulate input queries.
• Model (Training & Inference Phase): Attackers may conduct evasion, extraction, or backdoor attacks.
• Deployment (API & Infrastructure): Attackers may exploit cloud-based ML services via model inversion or denial-of-service (DoS) attacks.

ML attacks can be categorized based on the following:

Adversary’s Goal

• Integrity Attacks: Seek to modify ML predictions (e.g., fraud detection bypass).
• Availability Attacks: Prevent legitimate users from accessing ML services (e.g., DoS attacks).
• Privacy Attacks: Aim to extract sensitive data from models (e.g., membership inference attacks).

Adversary’s Knowledge

• White-box attacks: The attacker has full access to the model architecture and parameters.
• Black-box attacks: The attacker has no direct access but can query the model to infer information.

Adversary’s Capabilities

• Evasion Attacks: Trick an ML model into misclassifying inputs (e.g., adversarial examples).
• Poisoning Attacks: Manipulate training data to degrade model performance.
• Backdoor Attacks: Inject hidden triggers in training data to force misclassification.
• Model Extraction: Reverse-engineer a proprietary ML model through queries.