<jupyter_start><jupyter_text>WWLayerIteratorThis Notebook explains how to use the internal WeightWatcher Layer IteratorsIncludes the WWStackerLayerIterator- Stacks all weight matrices into a single, large, rectangular matrix- Probably should normalize each layer in some consistent way *(not done yet)<jupyter_code># Suppress the powerlaw package warnings
# "powerlaw.py:700: RuntimeWarning: divide by zero encountered in true_divide"
# "powerlaw.py:700: RuntimeWarning: invalid value encountered in true_divide"
import warnings
warnings.simplefilter(action='ignore', category=RuntimeWarning)
import numpy as np
import pandas as pd

from tqdm import tqdm

import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline

%load_ext watermark
%watermark<jupyter_output>2021-09-05T21:04:13-07:00

CPython 3.8.5
IPython 7.18.1

compiler   : Clang 10.0.0 
system     : Darwin
release    : 17.7.0
machine    : x86_64
processor  : i386
CPU cores  : 12
interpreter: 64bit<jupyter_text>Import WeightWatcherset custom Logging at WARN Level<jupyter_code>import logging

import weightwatcher as ww
import torchvision.models as models

logger = logging.getLogger(ww.__name__)
logger.setLevel(logging.WARNING)

ww.__version__<jupyter_output><empty_output><jupyter_text>How to create a WWLayerIterator<jupyter_code>model = models.vgg11(pretrained=True)
watcher = ww.WeightWatcher(model=model)
ww_layer_iterator = watcher.make_layer_iterator(model=model)<jupyter_output><empty_output><jupyter_text>The Iterator lets you loop over WWLayer instances- The WWLayer instance (object) is a wrapper to the underlying framework layers- The intent is to only access the WWLayer instance and not the underlying framework methods- This lets weightwatcher apply different functions / transformations on each layer individually<jupyter_code>for ww_layer in ww_layer_iterator:
    print(ww_layer)<jupyter_output>WWLayer 2  None PYTORCH CONV2D  skipped False
WWLayer 5  None PYTORCH CONV2D  skipped False
WWLayer 8  None PYTORCH CONV2D  skipped False
WWLayer 10  None PYTORCH CONV2D  skipped False
WWLayer 13  None PYTORCH CONV2D  skipped False
WWLayer 15  None PYTORCH CONV2D  skipped False
WWLayer 18  None PYTORCH CONV2D  skipped False
WWLayer 20  None PYTORCH CONV2D  skipped False
WWLayer 25  None PYTORCH DENSE  skipped False
WWLayer 28  None PYTORCH DENSE  skipped False
WWLayer 31  None PYTORCH DENSE  skipped False<jupyter_text>The WWLayerIterator constructor method takes- layers=[LAYER_ID, ...] to specify filters, as in the watcher.analyze(..,) and watcher.describe(...) methods- other parameters, like ww2x and channels, are specified in the parameters dict<jupyter_code>logger = logging.getLogger('weightwatcher') 
logger.setLevel(logging.WARNING)


layers = [60]
DEFAULT_PARAMS = {'glorot_fix': False, 'normalize':False, 'conv2d_norm':True, 'randomize': True, 'savefig':False, 
                  'rescale':True , 'deltaEs':False, 'intra':False, 'channels':None, 'conv2d_fft':False, 
                  'ww2x':False}

params = DEFAULT_PARAMS
ww_layer_iterator = watcher.make_layer_iterator(model=model, layers=layers, params=params)<jupyter_output><empty_output><jupyter_text>Now only 1 layer is processed<jupyter_code>for ww_layer in ww_layer_iterator:
    print(ww_layer)<jupyter_output><empty_output><jupyter_text>WWLayer  InstancesWhen a WWLayer instance is created, the weight matrices for the layer are extracted from the underlying framework tensor (i.e. layer.weights and layer.biases) and placed into WMats- WMats = [W,W,W,...] contains 1 or more W matrices, of the same shape NxM, N > M- evals: the *combined* evals for each layer- rf = 1 or (k)x(k)  the size of the 'receptive field'- layer_type =  an internal enum:    known layer types so far include:LAYER_TYPE.DENSE | LAYER_TYPE.CONV1D | LAYER_TYPE.CONV2D | LAYER_TYPE.FLATTENED |  LAYER_TYPE.EMBEDDING | LAYER_TYPE.NORM- channel_str:  string for channel type :  "FIRST" | "LAST" | "UNKNOWN"  WeightWatcher Apply Methods The various apply_xxx() methods use basic metaprogramming to set additional instance variables- apply_filters()- apply_normalize_Wmats()- apply_esd()- apply_random_esd()- apply_plot_esd()- apply_fit_powerlaw()- apply_norm_metrics()- apply_plot_deltaEs()- apply_mp_fit()- apply_svdsmoothing()i.e the apply_esd() method runs SVD on all the WMats, then combines them into a single ESD    def apply_esd(self, ww_layer, params=AULT_PARAMS):            """run full SVD on layer weight matrices, compute ESD on combined eigenvalues, combine all..."""...    ww_layer.evals = evals    ww_layer.add_column("has_esd", True)    ww_layer.add_column("num_evals", len(evals))    ww_layer.add_column("sv_max", sv_max)    ww_layer.add_column("rank_loss", rank_loss)    ww_layer.add_column("lambda_max", np.max(evals))              return ww_layer                 There are different Iterators for different ways of walking the Model Layers<jupyter_code>type(ww_layer_iterator)
from weightwatcher.weightwatcher import WWLayerIterator
from weightwatcher.constants import LAYER_TYPE

class WWStackedLayerIterator(WWLayerIterator):
    """Iterator variant that sticaks all weight matrics into a single WWLayer"""
    from copy import deepcopy

    def ww_stacked_iter_(self):
        from copy import deepcopy
        
        # find the maximum dimensions so we can pad the matrices
        ww_stacked_layer = None
        Wmats = []
        for ww_layer in self.ww_layer_iter_():
            
            # Here, Ijust lazizy copy an older layer
            # really, we should creat the WWLayer using the init() constructor
            if ww_stacked_layer is None:
                ww_stacked_layer =  deepcopy(ww_layer)
                ww_stacked_layer.the_type =  LAYER_TYPE.COMBINED
                ww_stacked_layer.layer_id = 0  
                ww_stacked_layer.name = "Example Stacked Layer"
            Wmats.extend(ww_layer.Wmats)
            
        # Note: Here the matrices should be padded so that they are all the same width
        #  ww_stacked_layer.Wmats = pad(Wmats)
        # 
        
        Nmax  = np.max([np.max(W.shape) for W in Wmats])
        Mmin  = np.min([np.min(W.shape) for W in Wmats])
                        
        Wmats_padded = []
        for W in Wmats:             
            Height, Width = W.shape[0], W.shape[1]
            if Height > Width:
                W = W.T
                Height, Width = W.shape[0], W.shape[1]
                
            W = np.pad(W, ((0, 0), (0, Nmax-Width)) )
            W = W/np.linalg.norm(W)
            Wmats_padded.append(W)
                
        W_stacked = np.vstack(Wmats_padded)
        N, M = W_stacked.shape[0],  W_stacked.shape[1]
        if N < M:
            W_stacked = W_stacked.T
            N, M = W_stacked.shape[0],  W_stacked.shape[1]
                    
        ww_stacked_layer.Wmats = [W_stacked]
    
        # Then, the layer shape has to be set
        # Just setting dummy variable here
        ww_stacked_layer.N = N
        ww_stacked_layer.M = M
        ww_stacked_layer.rf = 1
        #...
    
        
        yield ww_stacked_layer
                
    def make_layer_iter_(self):
        return self.ww_stacked_iter_()
layer_iter = WWStackedLayerIterator(model=model)
for layer in layer_iter:
    print(layer)
layer.N, layer.M<jupyter_output><empty_output><jupyter_text>Notice: The final matrix is quite large so the SVD will take some time<jupyter_code>W = layer.Wmats[0]<jupyter_output><empty_output><jupyter_text>svals = np.linalg.svd(W, compute_uv=False)<jupyter_code>from sklearn.decomposition import TruncatedSVD
svd = TruncatedSVD(n_components=5000)
svd.fit(W)
svals = svd.singular_values_
plt.hist(svals, bins=100);
plt.title("VGG11 SVs, all layer W's naively padded and stacked")
plt.xlabel(r"Singular Values,  $\sigma_{i}$")
plt.ylabel(r"Density, $\rho(\sigma)$")
evals = svals*svals
plt.hist(np.log10(evals), bins=100);
plt.title("VGG11 ESD, all layer W's naively padded and stacked")
plt.xlabel(r"Log EigenValues,  $\log\;\lambda_{i}$")
evals_nz = evals[evals>0.01]
plt.hist(np.log10(evals_nz), bins=100);
plt.title("VGG11 non-zero ESD , all layer W's naively padded and stacked")
plt.xlabel(r"Log EigenValues,  $\log\;\lambda,\;\;\lambda>0.01$")
plt.loglog(evals);
plt.title("VGG11 Stacked ESD, log log plot")

plt.loglog(evals_nz);
plt.title("VGG11 Stacked non-zero ESD, log log plot")
import powerlaw
results = powerlaw.Fit(evals_nz)
results.alpha
layer.evals = evals_nz<jupyter_output><empty_output><jupyter_text>The savefig option is not working correctly...<jupyter_code>params = {'glorot_fix': False, 'normalize':False, 'conv2d_norm':False, 'randomize': True, 
                  'savedir':'ww-img', 'savefig':True, 'rescale':True, 'plot':True,
                  'deltaEs':False, 'intra':False, 'channels':None, 'conv2d_fft':False, 
                  'ww2x':False, 'vectors':False, 'smooth':None}
watcher.apply_fit_powerlaw(layer, params=params)
W = layer.Wmats[0]
Wrand = W.flatten()
np.random.shuffle(Wrand)
np.random.shuffle(Wrand)
np.random.shuffle(Wrand)
np.random.shuffle(Wrand)
np.random.shuffle(Wrand)

W = Wrand.reshape(W.shape)
W = W.astype(float)

svd = TruncatedSVD(n_components=5000)
svd.fit(W)
svals = svd.singular_values_
rand_evals =  svals*svals
layer.rand_evals = rand_evals
plt.hist(np.log10(layer.rand_evals), bins=100, color='red', alpha=0.5, density=True, label='random');
plt.hist(np.log10(evals), bins=100, color='green', alpha=0.5, density=True, label='stacked');
plt.legend()
plt.xlabel("Log10 Eigenvalues")
plt.title("log10 Eigenvalues: n\n Stacked Layers X  vs Randomized(X) ")
layer.evals = layer.rand_evals
watcher.apply_fit_powerlaw(layer, params=params)
details =  watcher.describe(model)
details

Ntot = int(np.sum(details.N.to_numpy()))
Mtot = int(np.sum(details.M.to_numpy()*np.sqrt(details.rf.to_numpy())))
Ntot, Mtot
Ntot/ Mtot, Ntot/3
layer.N = Ntot
layer.M = Ntot/20
watcher.apply_mp_fit(layer, params=params)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10019055260254739, 'fraction_numerical': 0.016648280012034902, 'mean_word_length': 3.9026548672566372, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 43, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '22398', 'license': 'Apache-2.0', 'n_tokens_mistral': 3602, 'n_tokens_neox': 3375, 'n_words': 856, 'path': 'WWLayerIterator.ipynb', 'repo_name': 'nsfzyzz/WeightWatcher', 'size': '9971'}