<jupyter_start><jupyter_text>FastSparse> Customizable Fastai+PyTorch implementation of sparse model training methods (SET, SNFS, RigL).  _Warning: this repo is undergoing active development_  Getting Started  Install `pip install fastsparse`  Sparse Algorithms This network implements the following sparse algorithms:| Abbr. | Sparse Algorithm | in FastSparse | Notes || :- | :- | :- | :- ||  | static sparsity baseline | omit `DynamicSparseTrainingCallback` |  || SET | [Sparse Evolutionary Training](https://arxiv.org/abs/1901.09181) (Jan 2019) | `DynamicSparseTrainingCallback(**SET_presets)` |  || SNFS | [Sparse Networks From Scratch](https://arxiv.org/abs/1907.04840) (Jul 2019) | `DynamicSparseTrainingCallback(**SNFS_presets)` | \*redistribution not implemented || RigL | [Rigged Lottery](https://arxiv.org/abs/1911.11134) (Nov 2019) | `DynamicSparseTrainingCallback(**RigL_presets)` |  |\*Authors of the RigL paper demonstrate that using SNFS + Erdos-Renyi-Kernel distribution - redistribution outperforms SNFS + uniform sparsity + redistribution (at least on the measured benchmarks).  Fastai demo With just 4 additional lines of code, you can train your model using the latest dynamic sparse training techniques. This example achieves >99% accuracy on MNIST using a ResNet34 with only 1% of the weights.```python (0) install the library ! pip install fastsparse from fastai.vision.all import * (1) import this packageimport fastsparse as sparsepath = untar_data(URLs.MNIST)dls = ImageDataLoaders.from_folder(path, 'training', 'testing')learn = cnn_learner(dls, resnet34, metrics=error_rate, pretrained=False) (2) sparsify initial model + enforce maskssparse_hooks = sparse.sparsify_model(learn.model,                                      model_sparsity=0.99,                                     sparse_f=sparse.erdos_renyi_sparsity) (3) schedule dynamic mask updatescbs = [sparse.DynamicSparseTrainingCallback(**sparse.SNFS_presets,                                             batches_per_update=32)]learn.fit_one_cycle(5, cbs=cbs) (4) remove hooks that enforce maskssparse_hooks.remove()```Simply omit the `DynamicSparseTrainingCallback` to train a fixed-sparsity model as a baseline.  PyTorch demo (*not implemented yet*) ```pythonimport torchfrom torchvision import modelsdata = ...model = ...opt = ...opt = DynamicSparseTrainingOptimizerWrapper(model, opt, **RigL_kwargs) Modified training step sparse_opt.step(...) will determine whether to:  (A) take a regular opt step, or  (B) update network connectivitydef sparse_train_step(model, xb, yb, loss_func, sparse_opt, step, pct_train):    preds = model(xb)    loss = loss_func(preds, yb)    loss.backward()    sparse_opt.step(step, pct_train)    sparse_opt.zero_grad()```  Save/Reload demo Here is an example of saving a model and reloading it to resume training.<jupyter_code>from fastai.vision.all import *
from fastsparse import *

path = untar_data(URLs.MNIST_TINY)
dls = ImageDataLoaders.from_folder(untar_data(URLs.MNIST_TINY))
learn = cnn_learner(dls, resnet18, metrics=accuracy, pretrained=False)
sparse_hooks = sparsify_model(learn.model, model_sparsity=0.9, sparse_f=erdos_renyi_sparsity)
dst_kwargs = {**SNFS_presets, **{'batches_per_update': 8}}
cbs = DynamicSparseTrainingCallback(**dst_kwargs)

learn.fit_flat_cos(5, cbs=cbs)

# (0) save model as usual (masks are stored automatically)
save_model('sparse_tiny_mnist', learn.model, learn.opt)
### manually restart notebook ###

# (1) then recreate learner as usual

from fastai.vision.all import *
from fastsparse import *

path = untar_data(URLs.MNIST_TINY)
dls = ImageDataLoaders.from_folder(untar_data(URLs.MNIST_TINY))
learn = cnn_learner(dls, resnet18, metrics=accuracy, pretrained=False)

# (2) re-sparsify model (this adds the masks to the parameters)
sparse_hooks = sparsify_model(learn.model, model_sparsity=0.9, sparse_f=erdos_renyi_sparsity) # <-- initial sparsity + enforce masks

# (3) load model as usual
load_model('sparse_tiny_mnist', learn.model, learn.opt)

# (5) check validation loss & accuracy to verify we've loaded it successfully
val_loss, val_acc = learn.validate()
print(f'validation loss: {val_loss}, validation accuracy: {val_acc}')

# (4) optionally, continue training; otherwise remove sparsity-preserving hooks
sparse_hooks.remove()<jupyter_output>/home/dc/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py:53: UserWarning: Could not load the optimizer state.
  if with_opt: warn("Could not load the optimizer state.")
{'fraction_non_alphanumeric': 0.10367074527252503, 'fraction_numerical': 0.017575083426028922, 'mean_word_length': 5.2444444444444445, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '569535', 'license': 'Apache-2.0', 'n_tokens_mistral': 1500, 'n_tokens_neox': 1384, 'n_words': 387, 'path': 'index.ipynb', 'repo_name': 'dcato98/fastsparse', 'size': '4495'}