<jupyter_start><jupyter_text>Making Synthetic TI reportsThe goal here is to fine-tune/train a language model on Threat Intelligence reports, and then use that model to generate new, fake, TI reports. I'm doing this mostly because I think it will be funny.For this experiment, we're doing this with a very basic LSTM, rather than any of the fancier GPT or transformer models. (walk before you run and all that.)In any case, the process here is: * collect a ton of TI reports. This is what the https://github.com/g-clef/ThreatIntelCollector is for. * parse them all to extract the text (that's what the prefect job in this repo is for) * make a fastai databundle out of them * instantiate a language learner * train it * predict text This should be fairly straightforward, and I imagine this is going to be wonky at best. Ways to improve it that I can see right off the top: * replace the vendor URLs at the bottom of every page * parse out other URLs/hashes/DNS names/file paths in the report? * try gpt-2 (will it fit in my video card?) * try other fancier transformer libraries For now, though, let's just do the default/easy one.<jupyter_code>from fastai.text.all import *
from fastai import *<jupyter_output><empty_output><jupyter_text>So, first I'm going to pull the data from the prefect job in (I copy the relevant files to a local drive on my ML rig so that I'm not paying a network round-trip price for every file), then make a DataLoader out of that data, setting aside 10% for a "valid" dataset. Note: this will also do some standard tokenization of the text (using Spacy, I believe)<jupyter_code>base_dir = Path("/home/g-clef/local_ml_data_copy/ti-reports/bec011c7-0344-49dd-8daa-891f62a8c8f2")
dls_lm = TextDataLoaders.from_folder(base_dir, is_lm=True, valid_pct=0.1)
dls_lm.show_batch(max_n=5)<jupyter_output><empty_output><jupyter_text>Next I'm going to make an NLP language learning model, in this case using a pre-trained LSTM model, and pointing that learning model at the dataloader from before.<jupyter_code>learner = language_model_learner(dls_lm, AWD_LSTM, metrics=[accuracy, Perplexity()], path=base_dir, wd=0.1).to_fp16()<jupyter_output><empty_output><jupyter_text>By default, fastai language models are created in a "frozen" state, which means that only the head of the model will change with training, not the underlying layers. We'll see how well that works by just running a single epoch of training<jupyter_code>learner.fit_one_cycle(1)
learner.save("one_epoch")<jupyter_output><empty_output><jupyter_text>To train the whole model (fine-tuning the already-trained model), I have to "unfreeze" the model, and train again. I don't know off the top of my head how long to train it for, but I'll take a guess at 10 epochs.<jupyter_code>learner.unfreeze()
learner.fit_one_cycle(10)<jupyter_output><empty_output><jupyter_text>It's looking like it's starting to level off here, so 10 epochs may be enough. Let's save it so I don't have to re-do that 200 minutes of training time, and then see how it does.<jupyter_code>learner.save("eleven_epochs")
learner.save_encoder("first_finetuning")<jupyter_output><empty_output><jupyter_text>To make predictions, we give the model a starting point (or starting words, in this case), and ask it to predict a number of words forward. Fortunately, lots of the models begin with a title, so we'll make one up. You can also add a "temperature" variable here, which is a way of (to my understanding) weighting the distribution of the words to be more or less "spiky". I'm not sure how that will effect the final results here, so I'm going to leave it alone for the first try.<jupyter_code>first_try = learner.predict("Underpants Gnomes - A Targetted Intrusion", 2000)
print(first_try)<jupyter_output>Xxunk Xxunk - a Targetted Intrusion on a Unioncryptoupdater`memory_exec2 
▁ Internetopena 


▁ Whatsapp 
▁ Pwn2own 

▁ tty - security 

▁ Posted on November 18 , 2012 .  Keep notified and encouraged to follow full categories 

▁ high-ﬁdelity 85321dee31100bd3ece5b586ac3e6557 부트 difﬁcult 

▁ Detecting Malware 


▁ Running tslow.pyc ! 

▁ Linux Backdoor [ mechanized ] 


▁ First Determines : False Positives 

▁ Successful Kill Chain : The RLO 


▁ Although the cyberthreat is considered in a just very segmented manner , which would better hearing the next attacks relevant 
▁ to APT , it may be even more difficult and thoroughly 1761 . Just like all the Linux threat 
▁ actors , this case envyscout Additionally , the overall form of obfuscation 

▁ with a slight bb28 splits out three odds of choice : it does n’t need to apply a different cybersecurity 

▁ capability when Sofacy works . We will expect these types of attacks to be conducted out of that 

▁ multiple kinds — among other governm[...]<jupyter_text>That is...bad. Almost funny, but still bad. A number of things strike me:  1. Why does each line start with `_`? I should probably cut that out of the original files if it's in there somewhere. 2. The code samples/assembly language snipped that are scattered throughout are clearly confusing things. I should probably cut those out of the original files also. 3. The rest of it isn't *terrible*, just nonsensicle. Curious to see how well it does with just the two fixes above before moving to other models (since I suspect those two fixes will make the other examples perform better also). So, first step: let's find out where all those underscores are coming from. Huh. Those underscores don't exist in the original files. It appears that is some sort of artifact of either Spacy or FastAI's tokenizer. I could try to change that behavior, but I may just leave it alone & remove them at the end.  Next up: let's find the code examples and assembly and remove those. That will mean re-running the prefect side of this.The results of that weren't perfect (`def` still shows up in a few places, for example), but it's much lower than it was, which I am hopeful will make the results easier to read, and less likely to spew out random assembly language instructions. So, let's start from the top with this new batch.<jupyter_code>base_dir = Path("/home/g-clef/local_ml_data_copy/ti-reports/1655302d-a401-4d87-a223-dbb06648bb8f")
dls_lm = TextDataLoaders.from_folder(base_dir, is_lm=True, valid_pct=0.1)
dls_lm.show_batch(max_n=5)
learner = language_model_learner(dls_lm, AWD_LSTM, metrics=[accuracy, Perplexity()], path=base_dir, wd=0.1).to_fp16()
learner.fit_one_cycle(1)
learner.save("no_code_first_epoch")
learner.unfreeze()
learner.fit_one_cycle(10)
learner.save("no_code_eleven_epochs")
learner.save_encoder("no_code_first_finetuning")
second_try = learner.predict("Underpants Gnomes - A Targeted Intrusion", 2000)
print(second_try)<jupyter_output>Xxunk Xxunk - a Targeted Intrusion with Network Uncovers Mirrorthief Group Targets Leveraging 

▁ VPN Domains 

▁ Meet 주소 , Tiered Architecture and foreign - policy Operations at Over 

▁ The Atlantic Treaty Organization 

▁ March 11 , 2018 at 10:21:44 PM 

▁ Vulnerability in Internet Explorer 10 

▁ 5638 - at - xpl0it - analysis 

▁ t+m - yoroi - blog 
▁ By Supplying 82053737 , a augustus 11.0 
▁ producer 

▁ October 2 , 2021 

▁ Evasive Tools of muddywater Unknown Unknown Threat Actor 

▁ Early 02 , 2020 






▁ Witnessed in the first wave of Ill - dated March 27 - 2020 attacks by espionage 
▁ in the Palestinian Authority . 


▁ Introduction 


▁ The Middle East contains an increase in cyber 
▁ capabilities , distributed denial - of - service ( dos attacks and the use of malicious 
▁ Ipsum Denial - of - service , Keybase Malware , spoofed emails and proper infection chain . 


▁ Copies of the malicious files ( jan . 13 , 2021 ) , ILLUSTRATING 

▁ darkhalo , Tortoiseshell ’s Playbook[...]<jupyter_text>This is a lot better. It's still bad, but not nearly as bad as before. The text is a little better, the IOC's are more sensical (though still wrong). It seemed to level off at about 41% accuracy, though, which isn't great. I'm happy with this, but I think this is as far as the LSTM will take me. Next stop: transformers. Ok, I lied. One last thing: I want to check to see if lots more rounds of training will improve things, or if it's done (and/or overfit). Will just let this run in the background while I do other things.<jupyter_code>learner = learner.load("no_code_eleven_epochs")
learner.unfreeze()

learner.fit_one_cycle(30)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06745182012847965, 'fraction_numerical': 0.020580537711158697, 'mean_word_length': 4.688092016238159, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '359464', 'license': 'Apache-2.0', 'n_tokens_mistral': 2715, 'n_tokens_neox': 2508, 'n_words': 1174, 'path': 'jupyter/LSTM experiments.ipynb', 'repo_name': 'g-clef/synthetic-ti-reports', 'size': '8406'}