<jupyter_start><jupyter_text>Python Object Serialization <NAME> ___Principal Software Engineer____Bentley Systems Inc._ PyData meetup December, 2017  Pickle Pickle is a very powerful library to serialize and deserialize python objects.Use Cases :* Save state of program to disk* Transfer data over network connection* Store python objects in database<jupyter_code>import pickle
import sys<jupyter_output><empty_output><jupyter_text>The pickle module implements a fundamental, but powerful algorithm for serializing and de-serializing a Python object structure. “Pickling” is the process whereby a Python object hierarchy is converted into a byte stream, and “unpickling” is the inverse operation, whereby a byte stream is converted back into an object hierarchy. cpickle is a high performance compatible API. Note - Python 3 automatically will try to use cpickle if availableProtocol versions:* 0 is the original “human-readable” protocol and is backwards compatible with earlier versions of Python.* 1 is an old binary format which is also compatible with earlier versions of Python.* 2 was introduced in Python 2.3. It provides much more efficient pickling of new-style classes.* 3 was added in Python 3.0. It has explicit support for bytes objects and cannot be unpickled by Python 2.x. * 4 was added in Python 3.4. supports for very large objects, pickling more kinds of objects, and some data format optimizations.<jupyter_code># Helper functions
# delete pickle files if any
!rm *.p

def savePickle(object, filename, protocol = pickle.HIGHEST_PROTOCOL):
    pickle.dump(object, open(filename, "wb",),protocol)

def loadPickle(filename):
    return pickle.load(open(filename, "rb"))

def printVariables():
    for var,obj in globals().items():
        if var.startswith("_") == 0 :
            print( var + ":  "+str(sys.getsizeof(obj)) )<jupyter_output><empty_output><jupyter_text>Dictionary Sample<jupyter_code>myDict = {'Mammal': { 'Feline':['Cat','Tiger'],'Canine':['Dog','Wolf']},
          'Reptile':{'Snakes':['Rattler','King Cobra','Mamba'],'Lizard':['Monitor','Gila Monster']  }
         }
print(myDict["Mammal"])

savePickle(myDict,"MyDictionary.p")

%ls -al
#%cat "MyDictionary.p"
# Delete variable from Memory
del(myDict)
printVariables()
#print(myDict)
newDict = loadPickle("MyDictionary.p")

print(newDict["Mammal"])<jupyter_output>{'Feline': ['Cat', 'Tiger'], 'Canine': ['Dog', 'Wolf']}<jupyter_text>Save Pandas to pickle<jupyter_code>import pandas as pd

df = pd.DataFrame([range(1000), range(1,10000,10)])

df.iloc[:,:20]

#Save using pickle API
savePickle(df,"MyDF.p") 
#Save using DF's API
df.to_pickle("MyDFpd.p")
#Save using DF's API and  compression
df.to_pickle("MyDFpdCompressed.p",compression ='gzip')
%ls -al
df1 = loadPickle("MyDF.p")
df2 = pd.read_pickle("MyDFpd.p")
df3 = pd.read_pickle("MyDFpdCompressed.p",compression ='gzip')
df1.iloc[:,:20]
df2.iloc[:,:20]
df3.iloc[:,:20]<jupyter_output><empty_output><jupyter_text>Save model Pickle can be used to save and load a pre trainined scikit model<jupyter_code>from sklearn import datasets
from sklearn import tree
from sklearn.cross_validation import train_test_split
from sklearn.cross_validation import  cross_val_score

diabetes = datasets.load_diabetes()
X = diabetes.data
y = diabetes.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)

print ("* Training sample size : ", len(X_train))
print ("* Validation sample size : ", len(X_test))

dt = tree.DecisionTreeClassifier(criterion='gini',min_samples_split=5, random_state=1024)
dt.fit(X_train, y_train)


print(dt)
savePickle(dt,"DecisionTree.p")

def printScores(amodel, xtrain,ytrain):
    tscores = amodel.score( xtrain, ytrain)
    print ("Training score is %f" % tscores)
    print ("Model depth is %i" % amodel.tree_.max_depth )
    
printScores(dt,X_train,y_train)<jupyter_output>Training score is 0.592068
Model depth is 24<jupyter_text>Load the Model<jupyter_code>newDT = loadPickle("DecisionTree.p")

print(newDT)

printScores(newDT,X_train,y_train)<jupyter_output>DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
            max_features=None, max_leaf_nodes=None,
            min_impurity_decrease=0.0, min_impurity_split=None,
            min_samples_leaf=1, min_samples_split=5,
            min_weight_fraction_leaf=0.0, presort=False, random_state=1024,
            splitter='best')
Training score is 0.592068
Model depth is 24
{'fraction_non_alphanumeric': 0.11265709156193895, 'fraction_numerical': 0.018402154398563735, 'mean_word_length': 5.702255639097745, 'pattern_counts': {'":': 1, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '396730', 'license': 'Apache-2.0', 'n_tokens_mistral': 1537, 'n_tokens_neox': 1448, 'n_words': 414, 'path': 'Python Serialization.ipynb', 'repo_name': 'csantill/PySerialization', 'size': '4466'}