Analyzing the Security of Machine Learning Research Code

jwitsoe · October 4, 2023, 6:00pm

Originally published at: https://developer.nvidia.com/blog/analyzing-the-security-of-machine-learning-research-code/

The NVIDIA AI Red Team is focused on scaling secure development practices across the data, science, and AI ecosystems. We participate in open-source security initiatives, release tools, present at industry conferences, host educational competitions, and provide innovative training. Covering three years and totaling almost 140GB of source code, the recently released Meta Kaggle for Code…

nicholasteaguellc · October 11, 2023, 8:09pm

Since the pickle serialization issue is listed as top concern, worth noting that python native library has a signature option via the hmac module that mitigates risk of unpickling files. see eg The ultimate guide to Python pickle | Snyk

nicholasteaguellc · October 27, 2023, 4:55pm

As an update, I just published a tutorial notebook the the Automunge GitHub demonstrating the integration of a hmac signature into a pickle workflow as well as some alternate conventions using the dill library.

github.com

Automunge/AutoMunge/blob/master/Tutorials/14 - pickle tutorial - security and alternate workflows.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0375e76e",
   "metadata": {},
   "source": [
    "# pickle tutorial - security and alternate workflows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b96cb98e",
   "metadata": {},
   "source": [
    "The purpose of this notebook will be to demonstrate some conventions for saving the \"postprocess_dict\" dictionary, which is the returned dictionary from an automunge(.) call which may be used to productionize a model with preprocessing built through the Automunge library. \n",
    "\n",
    "Note that our tutorials so far have relied on the pickle library which is a native python module used to serialize and download python dictionaries, where the serialization is a non-encrypted form of data compression, and then the same pickle module may then be used to upload and initialize that same dictionary in a new notebook or environment. In some cases the serialization of a postprocessdict may be beneficial from a storage memory standpoint when preprocessing includes forms of ML infill which may store in the dictionary a trained model specific to each tabular feature. We understand that there are other options available to download a python dictionary without serialization (e.g. libraries like JSON, YAML, etc), however the pickle module has support for serializing additional datya types populated in a dictionary that may not be supported by other forms of JSON download. The most relevant data types that pickle may help with include those trained machine learning models returned Automunge library components for e.g. ML infill, PCA, feature selection, etc, as well as any custom data transformation functions that may have been iniatialized by a user for purposes of custom data transformations integrated into the family tree primitives API.\n",
    "\n",
    "The pickle library, although widely used in data science and other circles, has a known security vulnerability associated with cases with an uploaded serialized dictionary had been altered and not matched to the original intended form of distribution. We note in our readme that python's documentation has suggested a mitigation for this vulnerability that relies on a second python module called [hmac](https://docs.python.org/3/library/hmac.html#module-hmac), which serves the purposes of deriving a form of signature for an original downloaded pickle object which can then be compared to an uploaded object received from a potentially unsecure channel in order to validate that the uploaded form matches the original intended basis. In the demonstrations of this notebook the hmac signature will be derived and then affixed to the pickled object for comparison prior to loading. An alternative and even more secure approach may be to share the signature thorugh a seperate more secure channel as an additional means of redundancy. This notebook is our first demonstration of the incorporation of the hmac tool into a workflow in our documentation, which will be one of the agendas of this notebook.\n",

This file has been truncated. show original