<jupyter_start><jupyter_text>**How to save this notebook to your personal Drive**To copy this notebook to your Google Drive, go to File and select "Save a copy in Drive", where it will automatically open the copy in a new tab for you to work in. This notebook will be saved into a folder on your personal Drive called "Colab Notebooks".Still stumped? Check out  this video for help**Note: If you get an error saying that there is no module "pps_tools", go to Runtime, and select Restart runtime. Also make sure that your runtime type is Python 3** This notebook describes how to use the provided tools to interface with the data. It goes over the process of installing the tools, retrieving the data, and opening the data within a notebook. Importing the tools and data[Video Tutorial on how to import tools into Colab](https://www.youtube.com/watch?v=N7F6SISMIaM)Before beginning, you will likely find it useful to import the math and plotting libraries that are already included in Colab, such as ```numpy```, ```pylab```, and ```math```. In the case of ```pylab```, you will not need to include the usual ```%matplotlib inline``` or ```%matplotlib notebook```  when working in Colab, because Colab displays all outputs within the notebook by default. However, you will need to include it if you are working on a local application.<jupyter_code>import numpy as np
import matplotlib.pylab as plt<jupyter_output><empty_output><jupyter_text>Now we can bring in the custom tools created specifically for Particle Physics Playground. This is a bit more involved of a process.The first tools we need are for reading the data files themselves. The files we use are stored as ```.hdf5``` files, which are often used in particle physics research. We unpack this data using ```h5hep```, a file format based on ```hdf5```.  These tools are located in [this GitHub repository.](https://github.com/mattbellis/h5hep) First we need to get these files into a place where Colab can find them, so we use the following commands to pull them into Google's servers. Then we run a command to install them:<jupyter_code>###### This cell need only be run once per session ##############
###### Make sure your runtime type is Python 3 #########

# Import h5hep from Github. This is to allow us to read these
# particular files. 
!pip install git+https://github.com/mattbellis/h5hep.git<jupyter_output><empty_output><jupyter_text>Now that the files exist where Colab can find them, we can import them into the notebook like we do with the standard libraries.<jupyter_code>import h5hep<jupyter_output><empty_output><jupyter_text>The next set of tools we need are called Particle Physics Simplified, a custom package made specifically for Particle Physics Playground which include tools used to unpack data files from the `hdf5` type, tools to organize the data into navigable lists, and tools to download data files from online locations. If you want to see the tools and how they work, you can find them in [this GitHub repository](https://github.com/mattbellis/particle_physics_simplified). We just clone them with git, install them, and import them the same as we did above.<jupyter_code># Import custom tools package from Github. These are some simple accessor functions
# to make it easier to work with these data files. 
!pip install git+https://github.com/mattbellis/particle_physics_simplified.git

import pps_tools as pps<jupyter_output><empty_output><jupyter_text>Included in the tools we just imported are several download tools that are used to download files into Colab from Google Drive. All of the files you will need for the activies can be found in [ this Google Drive folder](https://drive.google.com/open?id=1qK8HL070jpQ6AJllsGHDDdUbjerDqG-c). We can bring in any individual file by calling it in with the command `pps_tools.download_from_drive`, and then the name of the file you want. Each activity should direct you as to which file or files you will be needing; for an example, we will use the file used for the CMS Muons activity. For more details on how to download files, please refer to the [codebkg01_DownloadData](https://colab.research.google.com/drive/174AUxy9qTqQ1XndbJW_Wryz9m95QZRIk) notebook.<jupyter_code># Download the data
pps.download_from_drive('dimuons_1000_collisions.hdf5')<jupyter_output><empty_output><jupyter_text>Now you should have all the tools you need to start interfacing with and analyzing the data. Interfacing with the data[Video overview on data interfacing](https://www.youtube.com/watch?v=tI2foOcuRVM)Reading data and navigating listsThe ```h5hep``` tools we used to unpack the data files puts the data all into lists that are accessible with dictionaries. You can view all the of the dictionary entries the data is tagged with using the following command (this is entirely optional, and simply gives you an idea of what kind of data the file may have contained).<jupyter_code># Print the keys to see what is in the dictionary   OPTIONAL
#for key in event.keys():
#  print(key)<jupyter_output><empty_output><jupyter_text>To organize the data in a way that makes it easy to find what you need, you will need to use the ```hep``` tools we imported. This can be done in several ways. Simple wayThe first way uses less commands, but gives you less control over how much data you are using. This command organizes ALL of the data, so if you are using a large data file, this can take a long time. However, it is usually the best choice if you want to use all the data. NOTE: This command will need to change depending on which experiment your activity is aligned with. For instance, the top quark activity uses CMS tools, so for the ```experiment``` argument in ```get_collisions```, we put ```'CMS'```. The other possible arguments would be ```'CLEO'``` and ```'BaBar'```.<jupyter_code>infile = 'data/dimuons_1000_collisions.hdf5'

collisions = pps.get_collisions(infile,experiment='CMS',verbose=False)
print(len(collisions), " collisions")  # This line is optional, and simply tells you how many events are in the file.<jupyter_output><empty_output><jupyter_text>This returns a list called ```collisions``` which has all of the collision events as entries. Each event is in turn its own list whose entries are the different types of particles involved in that collision. These are also lists, containing each individual particle of that particular type as entries, which are also lists of the four-momentum and other characteristics of each particle. This can be a bit complicated until you learn to work with it, so we'll try a visualization as an example:<jupyter_code>#collisions
second_collision = collisions[1]   # the second event
print("First event: ",second_collision)
all_muons = second_collision['muons']    # all of the jets in the first event
print("All muons: ",all_muons)
first_muon = all_muons[0]    # the first jet in the first event
print("First muon: ",first_muon)   
muon_energy = first_muon['e']      # the energy of the first photon
print("First muon's energy: ",muon_energy)<jupyter_output><empty_output><jupyter_text>You might notice that each individual event is callable from all collisions by is entry number, as are the individual particles from within their lists of particle types. However, the particle types themselves are only callable from the event list by their names. The characteristics of each particle are also only callable from their lists by the name of the characteristic. The exact dictionary entry needed to call them can be referenced by printing ```event.keys``` as above. Because ```get_collisions``` puts ALL of the data in a list, to do you analysis, you can simply call everything you need from this one list. For instance, if we wanted to find the energies of all the muons in the entire list of collisions, we could do so using loops:<jupyter_code>energies = []

for collision in collisions:          # loops over all the events in the file
  muons = collision['muons']      # gets the list of all muons in the event
  
  for muon in muons:           # loops over each muon in the current event
    e = muon['e']                # gets the energy of the muon
    
    energies.append(e)             # puts the energy in a list<jupyter_output><empty_output><jupyter_text>You can then show this list of energies in a histogram:<jupyter_code>plt.hist(energies,bins=50,range=(0,100));<jupyter_output><empty_output><jupyter_text>More involved wayAlternatively, you can use the following series of commands to organize the data. It is a little more involved, but gives you more control over the data. For instance, it gives you the ability to only use some of the events rather than all of them, which also would decrease some of the computing time.<jupyter_code>infile = 'data/dimuons_1000_collisions.hdf5' 

alldata = pps.get_all_data(infile,verbose=False)
nentries = pps.get_number_of_entries(alldata)

print("# entries: ",nentries)   # This optional line tells you how many events are in the file<jupyter_output><empty_output><jupyter_text>The above commands do not actually make the data directly usable, we need one more step for that, which is the ```get_collision``` function. This function is different from the ```get_collisions``` function used in the simpler method in that it only pulls out the information of a single event rather than all of them. This means that to get information from multiple events, you will need to use this command in a loop, for which you can define a range that determines what events you actualy want to use.NOTE: Depending on which activity you are doing, you will have to change the ```experiment``` argument to ```'CMS'```, ```'CLEO'```, or ```'BaBar'```. You will also need to change the ```entry_number``` argument to be the same variable you call in the loop.<jupyter_code>for entry in range(nentries):      # This range will loop over ALL of the events
    collision = pps.get_collision(alldata,entry_number=entry,experiment='CMS')

for entry in range(0,int(nentries/2)):     # This range will loop over the first half of the events
    collision = pps.get_collision(alldata,entry_number=entry,experiment='CMS')
    
for entry in range(int(nentries/2),nentries):      # This range will loop over the second half of the events
    collision = pps.get_collision(alldata,entry_number=entry,experiment='CMS')<jupyter_output><empty_output><jupyter_text>Other than that `get_collision` only gets the information from one event rather than all of them, it essentially organizes the information in the same way that `get_collisions` does. You can interact with this data the same way you would for any individual event from the big list of events that `get_collisions` would give you. For instance, to find the energies of all photons in the events we were looking at like we did for the simpler method, it would look very similar. However, you will notice that because we already have to loop over each event to use the `get_collision` function, we can simply nest the rest of our code within this loop.<jupyter_code>energies = []

for event in range(0,int(nentries/3)):        # Loops over first 3rd of all events
  
  collision = pps.get_collision(alldata,entry_number=event,experiment='CMS')    # organizes the data so you can interface with it
  muons = collision['muons']         # gets the list of all photons in the current event
  
  for muon in muons:                 # loops over all photons in the event
    e = muon['e']                      # gets the energy of the photon
    
    energies.append(e)                   # adds the energy to a list<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06883777031101922, 'fraction_numerical': 0.005083139484793659, 'mean_word_length': 4.7722526106414715, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '58117', 'license': 'MIT', 'n_tokens_mistral': 3179, 'n_tokens_neox': 3094, 'n_words': 1595, 'path': 'codebkg00_DataInterfacing.ipynb', 'repo_name': 'Samuel0X/simplified-particle-physics', 'size': '11607'}