<jupyter_start><jupyter_text>Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models This notebook shows an example for the threat model and attack described in "Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models". This example deviates from the other "honest-but-curious" server models and investigates a malicious server that may send malicious server updates. The attack succeeds for a range of common transformer architectures and works merely by sending a single malicious query to the user model.In this notebook, we attack the tiny transformer model discussed in "Advances and Open Problems in Federated Learning" (https://arxiv.org/abs/1912.04977). The model architecture is unchanged.Paper URL: https://arxiv.org/abs/2201.12675  Abstract:A central tenet of Federated learning (FL), which trains models without centralizing user data, is privacy. However, previous work has shown that the gradient updates used in FL can leak user information. While the most industrial uses of FL are for text applications (e.g. keystroke prediction), nearly all attacks on FL privacy have focused on simple image classifiers. We propose a novel attack that reveals private user text by deploying malicious parameter vectors, and which succeeds even with mini-batches, multiple users, and long sequences. Unlike previous attacks on FL, the attack exploits characteristics of both the Transformer architecture and the token embedding, separately extracting tokens and positional embeddings to retrieve high-fidelity text. This work suggests that FL on text, which has historically been resistant to privacy attacks, is far more vulnerable than previously thought.  Startup<jupyter_code>try:
    import breaching
except ModuleNotFoundError:
    # You only really need this safety net if you want to run these notebooks directly in the examples directory
    # Don't worry about this if you installed the package or moved the notebook to the main directory.
    import os; os.chdir("..")
    import breaching
    
import torch
%load_ext autoreload
%autoreload 2

# Redirects logs directly into the jupyter notebook
import logging, sys
logging.basicConfig(level=logging.INFO, handlers=[logging.StreamHandler(sys.stdout)], format='%(message)s')
logger = logging.getLogger()<jupyter_output><empty_output><jupyter_text>Initialize cfg object and system setup: This will load the full configuration object. This includes the configuration for the use case and threat model as `cfg.case` and the hyperparameters and implementation of the attack as `cfg.attack`. All parameters can be modified below, or overriden with `overrides=` as if they were cmd-line arguments.<jupyter_code>cfg = breaching.get_config(overrides=["attack=decepticon", "case=10_causal_lang_training", 
                                     "case/server=malicious-transformer"])
          
device = torch.device('cpu')
torch.backends.cudnn.benchmark = cfg.case.impl.benchmark
setup = dict(device=device, dtype=getattr(torch, cfg.case.impl.dtype))
setup<jupyter_output>Investigating use case causal_lang_training with server type malicious_transformer_parameters.<jupyter_text>Modify config options here You can use `.attribute` access to modify any of these configurations for the attack, or the case:<jupyter_code>cfg.case.user.num_data_points = 8 # How many sentences?
cfg.case.user.user_idx = 1 # From which user?
cfg.case.data.shape = [32] # This is the sequence length

cfg.case.server.has_external_data = True  # Not strictly necessary, could also use random text (see Appendix)
cfg.case.data.tokenizer = "word-level"

# Attack hyperparameters:

# this option requires installation of `k-means-constrained` which can be tricky:
# If this doesn't work for you, falling back to "dynamic-threshold" is still a decent option.
cfg.attack.sentence_algorithm = "k-means"<jupyter_output><empty_output><jupyter_text>Instantiate all parties The following lines generate "server, "user" and "attacker" objects and print an overview of their configurations.<jupyter_code>user, server, model, loss_fn = breaching.cases.construct_case(cfg.case, setup)
attacker = breaching.attacks.prepare_attack(server.model, server.loss, cfg.attack, setup)
breaching.utils.overview(server, user, attacker)<jupyter_output>Reusing dataset wikitext (/home/jonas/data/wikitext/wikitext-103-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
Using custom data configuration default
Reusing dataset ag_news (/home/jonas/data/ag_news/default/0.0.0/bc2bcb40336ace1a0374767fc29bb0296cdaf8a6da7298436239c54d79180548)
Reusing dataset wikitext (/home/jonas/data/wikitext/wikitext-103-v1/1.0.0/a241db52902eaf2c6aa732210bead40c090019a499ceb13bcbfa3f8ab646a126)
Model architecture transformer3 loaded with 10,800,433 parameters and 0 buffers.
Overall this is a data ratio of   42189:1 for target shape [8, 32] given that num_queries=1.
User (of type UserSingleStep) with settings:
    Number of data points: 8

    Threat model:
    User provides labels: False
    User provides buffers: False
    User provides number of data points: True

    Data:
    Dataset: wikitext
    user: 1
    
        
Server (of type MaliciousTransformerServer) with settings:
    Threat model: Malicious (Parameters)
    Number of[...]<jupyter_text>Simulate an attacked FL protocol This exchange is a simulation of a single query in a federated learning protocol. The server sends out a `server_payload` and the user computes an update based on their private local data. This user update is `shared_data` and contains, for example, the parameter gradient of the model in the simplest case. `true_user_data` is also returned by `.compute_local_updates`, but of course not forwarded to the server or attacker and only used for (our) analysis.<jupyter_code>server_payload = server.distribute_payload()
shared_data, true_user_data = user.compute_local_updates(server_payload)  
user.print(true_user_data)<jupyter_output>[CLS] the tower building of the little rock arsenal, also known as u. s. arsenal building, is a building located in macarthur park in downtown little rock,
arkansas. built in 1 8 4 0, it was part of little rock's first military installation. since its decommissioning, the tower building has housed two museums
. it was home to the arkansas museum of natural history and antiquities from 1 9 4 2 to 1 9 9 7 and the macarthur museum of arkansas military history since
2 0 0 1. it has also been the headquarters of the little rock [UNK] club since 1 8 9 4. [SEP] [CLS] the building receives its name from its
distinct [UNK] tower. besides being the last remaining structure of the original little rock arsenal and one of the oldest buildings in central arkansas, it was also the birthplace of
general douglas macarthur, who became the supreme commander of us forces in the south pacific during world war ii. it was also the starting place of the camden expedition.
in 2 0 1 1 it was named as one of the[...]<jupyter_text>Reconstruct user data: Now we launch the attack, reconstructing user data based on only the `server_payload` and the `shared_data`. For this attack, we also share secret information from the malicious server with the attack (`server.secrets`), which here is the location and structure of the imprint block.<jupyter_code>reconstructed_user_data, stats = attacker.reconstruct([server_payload], [shared_data], server.secrets, 
                                                      dryrun=cfg.dryrun)<jupyter_output>Recovered tokens tensor([[    0,     1,     2,     5,     6,     7,     8,    10,    11,    12,
            14,    16,    17,    18,    19,    22,    25,    26,    29,    30,
            31,    32,    34,    35,    38,    40,    43,    50,    51,    56,
            61,    62],
        [   63,    64,    71,    72,    75,    91,    98,   104,   108,   153,
           154,   184,   245,   262,   277,   279,   291,   294,   310,   319,
           351,   400,   405,   456,   494,   566,   572,   605,   649,   652,
           794,   845],
        [  872,   901,   926,   942,   963,   992,  1085,  1262,  1404,  1407,
          1495,  1671,  1936,  2109,  2172,  2189,  2209,  2423,  2778,  2976,
          3162,  3410,  3675,  3688,  3794,  3856,  4045,  5246,  5351,  5503,
          5658,  5675],
        [    5,     5,     5,     5,     5,     5,     5,     5,     5,     5,
            11,    11,   329,  6073,  6557,  6572,  8102,  8388,  8685,  8757,
          9579, 10989, 13466, 15519, 15970[...]<jupyter_text>Next we'll evaluate metrics, comparing the `reconstructed_user_data` to the `true_user_data`.<jupyter_code>metrics = breaching.analysis.report(reconstructed_user_data, true_user_data, [server_payload], 
                                    server.model, order_batch=True, compute_full_iip=False, 
                                    cfg_case=cfg.case, setup=setup)<jupyter_output>METRICS: | Accuracy: 0.9414 | S-BLEU: 0.92 | FMSE: 2.3505e-02 | 
 G-BLEU: 0.90 | ROUGE1: 0.95| ROUGE2: 0.92 | ROUGE-L: 0.95| Token Acc: 98.83% | Label Acc: 98.83%<jupyter_text>And finally, we also plot the reconstructed data:<jupyter_code>user.print(reconstructed_user_data)<jupyter_output>[CLS] the tower building of the little rock arsenal, also known at u. s. arsenal building, is a building located in macarthur park in downtown little rock of
arkansas. built in 1 8 4 0, it was part of little rock's first military installation. since its decommissioning, the tower building has housed two the
. it was home to the way museum of natural history and antiquities from 1 9 4 as to 1 9 9 7 and the macarthur museum of arkansas military history its
2 0 0 1. it has also been the headquarters of the little rock [UNK] club 1 1 8 9 4. [SEP] [CLS] the building receives its name from.
distinct [UNK] tower. besides being the last remaining structure of the original little rock arsenal and one of the oldest buildings in central arkansas, it was also the birthplace since
general douglas macarthur, who became the supreme commander of us forces in the south pacific during world war ii. it was also the starting place of museums camden expedition since
in 2 0 1 1 it was named as one of the to[...]
{'fraction_non_alphanumeric': 0.06867246263553775, 'fraction_numerical': 0.05880629090553873, 'mean_word_length': 3.8088304368248003, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '400425', 'license': 'MIT', 'n_tokens_mistral': 3360, 'n_tokens_neox': 2910, 'n_words': 1240, 'path': 'examples/Decepticons -  Analytic Attack - Transformer Model on Wikitext.ipynb', 'repo_name': 'ZHTushar23/breaching', 'size': '10237'}