MLSEC04-BP03 Protect against data poisoning threats
Protect your machine learning models and data by implementing security measures against data poisoning attacks, which can compromise model performance and accuracy. Data poisoning occurs through data injection (adding corrupt training data) or data manipulation (changing existing data like labels), resulting in inaccurate and weakened predictive capabilities. By identifying and addressing corrupt data using security methods and anomaly detection algorithms, you can maintain data integrity and protect against threats including ransomware and malicious code in third-party packages.
Desired outcome: You have implemented robust protection mechanisms for your machine learning training data and models. These protections include data validation procedures, monitoring for data drift, version control for both data and models, and rollback capabilities. Your ML systems can detect potential poisoning attempts and maintain model performance integrity through security best practices that protect data throughout its lifecycle.
Common anti-patterns:
-
Collecting training data from untrusted or unverified sources without validation.
-
Neglecting to monitor data distributions for unexpected shifts.
-
Deploying updated models without thorough testing against baseline performance.
-
Failing to implement version control for both training data and models.
-
Not having a rollback strategy for compromised models.
Benefits of establishing this best practice:
-
Improved model reliability and accuracy through clean, trusted data.
-
Early detection of potential security breaches targeting training data.
-
Reduced risk of deploying compromised models to production.
-
Ability to quickly recover from poisoning incidents through rollback mechanisms.
-
Enhanced overall ML system security and resilience.
Risk level for not implementing this practice: High
Implementation guidance
Data poisoning represents a security threat to machine learning systems. When malicious actors manipulate training data, they can compromise model integrity and cause downstream impacts on decisions or predictions made by those models. You need to implement comprehensive protections throughout your ML pipeline, from data collection to model deployment and monitoring.
Start by establishing strict controls over data sources and implementing validation procedures to detect anomalies before training. During model development, implement monitoring for data drift that could indicate poisoning attempts. Before deployment, thoroughly compare new models against previous versions to identify unexpected behavior changes. Finally, maintain versioned copies of both training data and models to enable rapid recovery from compromise.
By combining these defensive approaches, you create multiple layers of protection that make your ML systems resilient against data poisoning attempts.
Implementation steps
-
Use only trusted data sources for training data. Verify the provenance of data used for training and implement audit controls that allow you to track changes to training data. This includes recording who made changes, what changes were made, and when they occurred. Before using data for training, validate its quality to identify potential outliers and incorrectly labeled samples that could indicate poisoning attempts.
-
Look for underlying shifts in the patterns and distributions in training data. Implement continuous monitoring for data drift to detect unexpected changes in data distributions. Use tools like Amazon SageMaker AI Model Monitor
to track these changes automatically. Deviations from established patterns can serve as early warning signs of unauthorized access or manipulation targeting training data. -
Identify model updates that negatively impact the results before moving them to production. Compare newly trained models against previous versions using consistent test datasets. Look for unexpected performance changes, especially degradations in specific areas that weren't present in earlier model iterations. Use Amazon SageMaker AI MLflow Model Registry to track model versions and their performance metrics.
-
Have a rollback plan. Implement versioning for both training data and models to enable quick recovery from compromised states. Use Amazon SageMaker AI Feature Store
to maintain secure, versioned features for your ML models. The Feature Store provides a centralized repository for features with built-in security controls. Configure Amazon SageMaker AI MLflow Model Registry to support rollback capabilities so you can quickly revert to a known good model version if issues are detected with a newly deployed model. -
Use low-entropy classification cases. Establish performance thresholds and monitor for unexpected classification patterns. Define boundaries for acceptable model behavior and create alerts when outputs deviate from expected patterns. This can identify subtle poisoning attempts that might otherwise go undetected through conventional testing.
-
Implement end-to-end encryption for ML data. Secure your training data, feature sets, and models using encryption both at rest and in transit. Use AWS Key Management Service (KMS)
to manage encryption keys and apply them consistently across your ML pipeline. Encryption protects against unauthorized access that could lead to data poisoning. -
Regularly scan for vulnerabilities in ML dependencies. Use tools like Amazon Inspector
to detect vulnerabilities in the software packages and dependencies used in your ML environment. Data poisoning can occur through compromised third-party libraries, so regular scanning can identify potential entry points for bad actors. -
Implement input validation for AI systems. For AI models, validate inputs for potential poisoning attempts. Implement filtering and sanitization of inputs to block adversarial inputs that could manipulate model behavior or extract sensitive information.
Resources
Related documents:
Related videos:
Related examples:
