<jupyter_start><jupyter_text>Particles and decaysin the Scikit-HEP project<NAME>University of Cincinnati PyHEP 2019 Workshop, Abingdon, 16-18 October 2019Notebook: https://github.com/eduardo-rodrigues/2019-10-16_PyHEP2019Workshop PDG particle data and MC identification codeswith the Particle package **Pythonic interface to**- Particle Data Group (PDG) particle data table.- Particle MC identification codes, with inter-MC converters.- With various extra goodies.  Package motivation - particle data- The [PDG](http://pdg.lbl.gov/) provides a downloadable table of particle masses, widths, charges and Monte Carlo particle ID numbers (PDG IDs).  - Most recent file [here](http://pdg.lbl.gov/2019/html/computer_read.html).- It also provided an experimental file with extended information(spin, quark content, P and C parities, etc.) until 2008 only, see [here](http://pdg.lbl.gov/2008/html/computer_read.html) (not widely known!).- But *anyone* wanting to use these data, the only readily available,has to parse the file programmatically.- Why not make a Python package to deal with all these data, for everyone?  Package motivation - MC identification codes- The C++ HepPID and HepPDT libraries provide functions for processing particle ID codesin the standard particle (aka PDG) numbering scheme.- Different event generators have their separate set of particle IDs: Pythia, EvtGen, etc.- Again, why not make a package providing all functionality/conversions, Python-ically, for everyone?  Package, in short- Particle - loads extended **PDG data tables** and implements search and manipulations / display.- PDGID - find out as much as possible from the PDG ID number. **No table lookup**.- Converters for MC IDs used in Pythia and Geant.- Basic usage via the command line.- Fexible / advanced usage programmatically.  1. Command line usageSearch and query ...<jupyter_code>!python -m particle --version
!python -m particle -h<jupyter_output>usage: particle [-h] [--version] {search,pdgid} ...

Particle command line display utility. Has two modes.

positional arguments:
  {search,pdgid}  Subcommands
    search        Look up particles by PID or name (Ex.: python -m particle
                  search D+ D-)
    pdgid         Print info from PID (Ex.: python -m particle pdgid 11 13)

optional arguments:
  -h, --help      show this help message and exit
  --version       show program's version number and exit<jupyter_text>PDGID Print all information from a PDG ID:<jupyter_code>!python -m particle pdgid 211<jupyter_output><PDGID: 211>
A              None
C              None
J              0.0
L              0
P              -1
S              0
Z              None
abspid         211
charge         1.0
has_bottom     False
has_charm      False
has_down       True
has_fundamental_anti False
has_strange    False
has_top        False
has_up         True
is_Qball       False
is_Rhadron     False
is_SUSY        False
is_baryon      False
is_diquark     False
is_dyon        False
is_hadron      True
is_lepton      False
is_meson       True
is_nucleus     False
is_pentaquark  False
is_valid       True
j_spin         1
l_spin         1
s_spin         1
three_charge   3<jupyter_text>Particle Search a particle by its PDG ID - return description summary of particle:<jupyter_code>!python -m particle search 211<jupyter_output>Name: pi+            ID: 211          Latex: $\pi^{+}$
Mass  = 139.57061 ± 0.00024 MeV
Lifetime = 26.033 ± 0.005 ns
Q (charge)        = +       J (total angular) = 0.0      P (space parity) = -
C (charge parity) = ?       I (isospin)       = 1        G (G-parity)     = -
    SpinType: SpinType.PseudoScalar
    Quarks: uD
    Antiparticle name: pi- (antiparticle status: ChargeInv)<jupyter_text>Search a particle by its name - either return the description summary of matching particle ...<jupyter_code>!python -m particle search "pi(1400)+"<jupyter_output>Name: pi(1)(1400)+   ID: 9000213      Latex: $\pi_{1}(1400)^{+}$
Mass  = 1354 ± 25 MeV
Width = 330 ± 35 MeV
Q (charge)        = +       J (total angular) = 1.0      P (space parity) = -
C (charge parity) = ?       I (isospin)       = 1        G (G-parity)     = -
    SpinType: SpinType.Vector
    Quarks: Maybe non-qQ
    Antiparticle name: pi(1)(1400)- (antiparticle status: ChargeInv)<jupyter_text>... or a list of particles matching the keyword in their names:<jupyter_code>!python -m particle search "pi+"<jupyter_output><Particle: name="pi+", pdgid=211, mass=139.57061 ± 0.00024 MeV>
<Particle: name="pi(2)(1670)+", pdgid=10215, mass=1670.6 + 2.9 - 1.2 MeV>
<Particle: name="pi(1300)+", pdgid=100211, mass=1300 ± 100 MeV>
<Particle: name="pi(1)(1400)+", pdgid=9000213, mass=1354 ± 25 MeV>
<Particle: name="pi(1800)+", pdgid=9010211, mass=1810 + 9 - 11 MeV>
<Particle: name="pi(1)(1600)+", pdgid=9010213, mass=1660 + 15 - 11 MeV><jupyter_text>Bonus feature: zipappPackage provides a [zipapp](https://docs.python.org/3/library/zipapp.html) version - **one file** that runs on **any computer with Python**, no other dependencies! Find it [attached to releases](https://github.com/scikit-hep/particle/releases). Example:```bash./particle.pyz search gamma``` All dependencies are installed inside the zipapp, and the data lookup is handled in a zip-safe way inside particle. Python 3 is used to make the zipapp, but including the backports makes it work on Python 2 as well.  2. `PDGID` class and MC ID classes- Classes `PDGID`, `PythiaID`, `GeantID`.- Converters in module `particle.converters`: `Geant2PDGIDBiMap`, etc.  PDG IDs module overview- Process and query PDG IDs, and more – no look-up table needed.  - Current version of package reflects the latest version of the    HepPID & HepPDT utility functions defined in the C++ HepPID and HepPDT versions 3.04.01  - It contains more functionality than that available in the C++ code … and minor fixes too.- Definition of a PDGID class, PDG ID literals,and set of standalone HepPID functions to query PDG IDs(is_meson, has_bottom, j_spin, charge, etc.).   - All PDGID class functions are available standalone.  PDGID class- Wrapper class `PDGID` for PDG IDs.- Behaves like an int, with extra goodies.- Large spectrum of properties and methods, with a Pythonic interface, and yet more!<jupyter_code>from particle import PDGID
pid = PDGID(211)
pid
PDGID(99999999)
from particle.pdgid import is_meson

pid.is_meson, is_meson(pid)<jupyter_output><empty_output><jupyter_text>To print all `PDGID` properties:<jupyter_code>print(pid.info())<jupyter_output>A              None
C              None
J              0.0
L              0
P              -1
S              0
Z              None
abspid         211
charge         1.0
has_bottom     False
has_charm      False
has_down       True
has_fundamental_anti False
has_strange    False
has_top        False
has_up         True
is_Qball       False
is_Rhadron     False
is_SUSY        False
is_baryon      False
is_diquark     False
is_dyon        False
is_hadron      True
is_lepton      False
is_meson       True
is_nucleus     False
is_pentaquark  False
is_valid       True
j_spin         1
l_spin         1
s_spin         1
three_charge   3<jupyter_text>MC ID classes and converters- Classes for MC IDs used in Pythia and Geant: `PythiaID` and `GeantID`.- ID converters in module `particle.converters`: `Geant2PDGIDBiMap`, etc.<jupyter_code>from particle import PythiaID, GeantID

pyid = PythiaID(10221)

pyid.to_pdgid()<jupyter_output><empty_output><jupyter_text>Conversions are directly available via mapping classes.E.g., bi-directional map Pythia ID - PDG ID:<jupyter_code>from particle.converters import Pythia2PDGIDBiMap

Pythia2PDGIDBiMap[PDGID(9010221)]
Pythia2PDGIDBiMap[PythiaID(10221)]<jupyter_output><empty_output><jupyter_text>3. `Particle` classThere are lots of ways to create a particle.<jupyter_code>from particle import Particle<jupyter_output><empty_output><jupyter_text>From a PDG ID<jupyter_code>Particle.from_pdgid(211)<jupyter_output><empty_output><jupyter_text>From a name<jupyter_code>Particle.from_string('pi+')<jupyter_output><empty_output><jupyter_text>SearchingSimple and natural API to deal with the PDG particle data table,with powerful 1-line search and look-up utilities!- `Particle.find(…)` – search a single match (exception raised if multiple particles match the search specifications).- `Particle.findall(…)` – search a list of candidates.- Search methods that can query any particle property!<jupyter_code>Particle.find('J/psi')<jupyter_output><empty_output><jupyter_text>You can specify search terms as keywords - _any particle property_:<jupyter_code>Particle.find(latex_name=r'\phi(1020)')<jupyter_output><empty_output><jupyter_text>Some properties have enums available. For example, you can directly check the numeric charge:<jupyter_code>Particle.findall('pi', charge=-1)<jupyter_output><empty_output><jupyter_text>Or you can use the enum (for charge, this is 3 times the charge, hence the name `three_charge`)<jupyter_code>from particle import Charge

Particle.findall('pi', three_charge=Charge.p)<jupyter_output><empty_output><jupyter_text>Or use a **lambda function** for the ultimate in generality! For example, to find all the neutral particles with a bottom quark between 5.2 and 5.3 GeV:<jupyter_code>from hepunits import GeV, s  # Units are good. Use them.
Particle.findall(lambda p:
                     p.pdgid.has_bottom
                     and p.charge==0
                     and 5.2*GeV < p.mass < 5.3*GeV
                )<jupyter_output><empty_output><jupyter_text>Another lambda function example: You can use the width or the lifetime:<jupyter_code>Particle.findall(lambda p: p.lifetime > 1000*s)<jupyter_output><empty_output><jupyter_text>If you want infinite lifetime, you could just use the keyword search instead:<jupyter_code>Particle.findall(lifetime=float('inf'))<jupyter_output><empty_output><jupyter_text>Trivially find all pseudoscalar charm mesons:<jupyter_code>from particle import SpinType

Particle.findall(lambda p: p.pdgid.is_meson and p.pdgid.has_charm and p.spin_type==SpinType.PseudoScalar)<jupyter_output><empty_output><jupyter_text>Display Nice display in Jupyter notebooks, as well as `str` and `repr` support:<jupyter_code>p = Particle.from_pdgid(-415)
p
print(p)
print(repr(p))<jupyter_output><Particle: name="D(2)*(2460)-", pdgid=-415, mass=2465.4 ± 1.3 MeV><jupyter_text>Full descriptions:<jupyter_code>print(p.describe())<jupyter_output>Name: D(2)*(2460)-   ID: -415         Latex: $D_{2}^{*}(2460)^{-}$
Mass  = 2465.4 ± 1.3 MeV
Width = 46.7 ± 1.2 MeV
Q (charge)        = -       J (total angular) = 2.0      P (space parity) = +
C (charge parity) = ?       I (isospin)       = 1/2      G (G-parity)     = ?
    SpinType: SpinType.Tensor
    Quarks: Cd
    Antiparticle name: D(2)*(2460)+ (antiparticle status: ChargeInv)<jupyter_text>You may find LaTeX or HTML to be more useful in your program; both are supported:<jupyter_code>print(p.latex_name, '\n', p.html_name)<jupyter_output>D_{2}^{*}(2460)^{-} 
 D<SUB>2</SUB><SUP>*</SUP>(2460)<SUP>-</SUP><jupyter_text>It is easy to get hold of the whole list of particle (instances) as a list:<jupyter_code>print('# of particles in loaded data table:', len(Particle.all()))
Particle.all()<jupyter_output># of particles in loaded data table: 536<jupyter_text>Particle propertiesYou can do things to particles, like **invert** them:<jupyter_code>~p<jupyter_output><empty_output><jupyter_text>There are a plethora of properties you can access:<jupyter_code>p.spin_type<jupyter_output><empty_output><jupyter_text>You can quickly access the PDGID of a particle:<jupyter_code>p.pdgid
PDGID(p)<jupyter_output><empty_output><jupyter_text>4. LiteralsThey provide a handy way to manipulate things with human-readable names!Package defines literals for most common particles, with easily recognisable names.- Literals are dynamically generated on import for both `PDGID` and `Particle` classes. **PDGID literals**<jupyter_code>from particle.pdgid import literals as lid
lid.phi_1020<jupyter_output><empty_output><jupyter_text>**Particle literals**<jupyter_code>from particle.particle import literals as lpart
lpart.phi_1020<jupyter_output><empty_output><jupyter_text>5. Data files, stored in `particle/data/`- PDG particle data files  - Original PDG data files, which are in a fixed-width format  - Code uses “digested forms” of these, stored as CSV, for optimised querying  - Latest PDG data (2019) used by default  - Advanced usage: user can load older PDG table, load a “user table” with new particles, append to default table- Other data files  - CSV file for mapping of PDG IDs to particle LaTeX names  Dump table contentsThe `Particle.dump_table(...)` method is rather flexible.(No need to dig into the package installation directory to inspect the particle data table ;-).)<jupyter_code>help(Particle.dump_table)
fields = ['pdgid', 'pdg_name', 'mass', 'mass_upper', 'mass_lower', 'three_charge']
    
Particle.dump_table(exclusive_fields=fields, n_rows=10)<jupyter_output>pdgid  pdg_name       mass    mass_upper    mass_lower    three_charge
-------  ----------  -------  ------------  ------------  --------------
      1  d              4.67           0.5           0.2              -1
     -1  d              4.67           0.5           0.2               1
      2  u              2.16           0.5           0.3               2
     -2  u              2.16           0.5           0.3              -2
      3  s             93             11             5                -1
     -3  s             93             11             5                 1
      4  c           1270             20            20                 2
     -4  c           1270             20            20                -2
      5  b           4180             30            20                -1
     -5  b           4180             30            20                 1<jupyter_text>Table with all b-flavoured hadrons (in _reStructuredText_ format):<jupyter_code>Particle.dump_table(filter_fn=lambda p: p.pdgid.has_bottom, exclusive_fields=fields, tablefmt='rst')<jupyter_output>=======  ==============  =========  ============  ============  ==============
  pdgid  pdg_name             mass    mass_upper    mass_lower    three_charge
=======  ==============  =========  ============  ============  ==============
    511  B                5279.64           0.13          0.13               0
   -511  B                5279.64           0.13          0.13               0
    513  B*               5324.7            0.22          0.22               0
   -513  B*               5324.7            0.22          0.22               0
    515  B(2)*(5747)      5739.5            0.7           0.7                0
   -515  B(2)*(5747)      5739.5            0.7           0.7                0
    521  B                5279.33           0.13          0.13               3
   -521  B                5279.33           0.13          0.13              -3
    523  B*               5324.7            0.22          0.22               3
   -523  B*               5324.7            0.22    [...]<jupyter_text>6. Advanced usageYou can:* Extend or replace the default particle data table in `Particle`.* Adjust properties for a particle.* Make custom particles. Decays and decay chains with the DecayLanguage package`DecayLanguage` is designed for the manipulation of decay structures in Python. The current package has:- Amplitude Analysis decay language:  - Input based on AmpGen generator, output format for GooFit C++- Decay file parsers:  - Read DecFiles, such as the LHCb master DecFile  - Manipulate adn visualize them in Python  Package motivation- Ability to describe decay-tree-like structures- Provide a translation of decay amplitude models from AmpGen to GooFit  - Idea is to generalise this to other decay descriptions - Any experiment uses event generators which, among many things, need to describe particle decay chains - Programs such as EvtGen rely on so-called .dec decay files - Many experiments need decay data files - Why not make a Python package to deal with decay files, for everyone?  Package, in short- Tools to parse decay files and programmatically manipulate them, query, display information.  - Descriptions and parsing built atop the [Lark parser](https://github.com/lark-parser/lark/).- Tools to translate decay amplitude models from AmpGen to GooFit, and manipulate them.  1. Decay files *Master file” DECAY.DECGigantic file defining decay modes for all relevant particles, including decay model specifications. User .dec files- Needed to produce specific MC samples.- Typically contain a single decay chain (except if defining inclusive samples). **Example user decay file:** Decay file for [B_c+ -> (B_s0 -> K+ K-) pi+]ccAlias      B_c+sig        B_c+Alias      B_c-sig        B_c-ChargeConj B_c+sig        B_c-sigAlias      MyB_s0         B_s0Alias      Myanti-B_s0    anti-B_s0ChargeConj MyB_s0         Myanti-B_s0Decay B_c+sig  1.000     MyB_s0     pi+     PHOTOS PHSP;EnddecayCDecay B_c-sigDecay MyB_s0    1.000     K+     K-     SSD_CP 20.e12 0.1 1.0 0.04 9.6 -0.8 8.4 -0.6;EnddecayCDecay Myanti-B_s0  2. Decay file parsing- **Parsing should be simple**  - Expert users can configure parser choice and settings, etc. - **Parsing should be (reasonably) fast!**After parsing, many queries are possible!<jupyter_code>from decaylanguage import DecFileParser<jupyter_output><empty_output><jupyter_text>The LHCb "master decay file"It's a big file! ~ 450 particle decays defined, thousands of decay modes, over 11k lines in total.<jupyter_code>dfp = DecFileParser('data/DECAY_LHCB.DEC')
%%time
dfp.parse()
dfp<jupyter_output><empty_output><jupyter_text>Let's parse and play with a small decay file:<jupyter_code>with open('data/Dst.dec') as f:
    print(f.read())
dfp_Dst = DecFileParser('data/Dst.dec')
dfp_Dst
dfp_Dst.parse()
dfp_Dst<jupyter_output><empty_output><jupyter_text>It can be handy to **parse from a multi-line string** rather than a file:<jupyter_code>s = """
# Decay file for [B_c+ -> (B_s0 -> K+ K-) pi+]cc

Alias      B_c+sig        B_c+
Alias      B_c-sig        B_c-
ChargeConj B_c+sig        B_c-sig
Alias      MyB_s0         B_s0
Alias      Myanti-B_s0    anti-B_s0
ChargeConj MyB_s0         Myanti-B_s0

Decay B_c+sig
  1.000     MyB_s0     pi+     PHOTOS PHSP;
Enddecay
CDecay B_c-sig

Decay MyB_s0
    1.000     K+     K-     SSD_CP 20.e12 0.1 1.0 0.04 9.6 -0.8 8.4 -0.6;
Enddecay
CDecay Myanti-B_s0
"""
dfp = DecFileParser.from_string(s)
dfp.parse()
dfp<jupyter_output><empty_output><jupyter_text>Decay file information<jupyter_code>dfp_Dst.print_decay_modes('D*+')
dfp_Dst.list_decay_mother_names()
dfp_Dst.list_decay_modes('D*+')<jupyter_output><empty_output><jupyter_text>Info such as particle aliases<jupyter_code>dfp.dict_aliases()
dfp.dict_charge_conjugates()<jupyter_output><empty_output><jupyter_text>3.  Display of decay chainsThe parser can provide a simple representation of any decay chain found in the input decay file(s).<jupyter_code>dc = dfp_Dst.build_decay_chains('D+')
dc
from decaylanguage import DecayChainViewer
DecayChainViewer(dc)
dc = dfp_Dst.build_decay_chains('D*+')
DecayChainViewer(dc)
dc = dfp_Dst.build_decay_chains('D*+', stable_particles=['D+', 'D0', 'pi0'])
DecayChainViewer(dc)<jupyter_output><empty_output><jupyter_text>Charge conjugation<jupyter_code>dc_cc = dfp_Dst.build_decay_chains('D*-', stable_particles=['D-', 'anti-D0', 'pi0'])
DecayChainViewer(dc_cc)<jupyter_output><empty_output><jupyter_text>Parsing several filesTypically useful when the user decay file needs information from the master decay file.<jupyter_code>s = u"""
Alias      MyXic+              Xi_c+
Alias      MyantiXic-          anti-Xi_c-
ChargeConj MyXic+              MyantiXic-

Decay Xi_cc+sig
  1.000       MyXic+    pi-    pi+       PHSP;
Enddecay
CDecay anti-Xi_cc-sig

Decay MyXic+
  1.000       p+    K-    pi+       PHSP;
Enddecay
CDecay MyantiXic-

End
"""
dfp = DecFileParser.from_string(s)
dfp.parse()
dfp<jupyter_output>C:\home\sw\Anaconda3\lib\site-packages\decaylanguage\dec\dec.py:445: UserWarning: 
Corresponding 'Decay' statement for 'CDecay' statement(s) of following particle(s) not found:
anti-Xi_cc-sig.
Skipping creation of these charge-conjugate decay trees.
  warnings.warn(msg)<jupyter_text>Note the subtletly: 3, not 4 decays, are found! This is because the file contains no statement`ChargeConj anti-Xi_cc-sigXi_cc+sig`, hence the parser cannot know to which particle (matching `Decay` statement) the charge-conjugate decay of `anti-Xi_cc-sig` relates to (code does not rely on position of statements to guess ;-)).<jupyter_code>d = dfp.build_decay_chains('Xi_cc+sig')
DecayChainViewer(d)<jupyter_output><empty_output><jupyter_text>As said in the warning, the information provided is not enough for the anti-Xi_cc-sig to make sense:<jupyter_code>from decaylanguage.dec.dec import DecayNotFound

try:
    d = dfp.build_decay_chains('anti-Xi_cc-sig')
except DecayNotFound:
    print("Decays of particle 'anti-Xi_cc-sig' not found in .dec file!")<jupyter_output>Decays of particle 'anti-Xi_cc-sig' not found in .dec file!<jupyter_text>But the missing information is easily providing **parsing two files simultaneously ...!** (Any number of files is allowed.)<jupyter_code>from tempfile import NamedTemporaryFile

with NamedTemporaryFile(delete=False) as tf:
    tf.write(s.encode('utf-8'))
    
dfp = DecFileParser(tf.name, 'data/DECAY_LHCB.DEC')
dfp.parse()
dc = dfp.build_decay_chains('Xi_cc+sig')

DecayChainViewer(dc)
dc_cc = dfp.build_decay_chains('anti-Xi_cc-sig')

DecayChainViewer(dc_cc)<jupyter_output><empty_output><jupyter_text>4. Representation of decay chainsThe universal (and digital) representation of decay chains is of interest well outside the context of decay file parsing!  Building blocks- A daughters list - list of final-state particles.- A decay mode - typically a branching fraction and a list of final-state particles (may also contain _any_ metadata such as decay model and optional decay-model parameters, as defined for example in .dec decay files).- A decay chain - can be seen as a mother particle and a list of decay modes.<jupyter_code>from decaylanguage.decay.decay import DaughtersDict, DecayMode, DecayChain<jupyter_output><empty_output><jupyter_text>**Daughters list** (actually a ``Counter`` dictionary, internally):<jupyter_code># Constructor from a dictionary
dd = DaughtersDict({'K+': 1, 'K-': 2, 'pi+': 1, 'pi0': 1})

# Constructor from a list of particle names
dd = DaughtersDict(['K+', 'K-', 'K-', 'pi+', 'pi0'])

# Constructor from a string representing the final state
dd = DaughtersDict('K+ K- pi0')
dd<jupyter_output><empty_output><jupyter_text>Decay Modes<jupyter_code># A 'default' and hence empty, decay mode
dm = DecayMode()

# Decay mode with minimal input information
dd = DaughtersDict('K+ K-')
dm = DecayMode(0.5, dd)

# Decay mode with decay model information and user metadata
dm = DecayMode(0.2551,                                              # branching fraction
               'pi- pi0 nu_tau',                                    # final-state particles
               model='TAUHADNU',                                    # decay model
               model_params=[-0.108, 0.775, 0.149, 1.364, 0.400],   # decay-model parameters
               study='toy', year=2019                               # user metadata
              )
dm
print(dm.describe())<jupyter_output>Daughters: pi- pi0 nu_tau , BF: 0.2551         
    Decay model: TAUHADNU [-0.108, 0.775, 0.149, 1.364, 0.4]
    Extra info:
        study: toy
        year: 2019<jupyter_text>Various manipulations are available:<jupyter_code>dm = DecayMode.from_pdgids(0.5, [321, -321])
print(dm)

dm = DecayMode(1.0, 'K+ K+ pi-')
dm.charge_conjugate()<jupyter_output><DecayMode: daughters=K+ K-, BF=0.5><jupyter_text>Decay chains<jupyter_code>dm1 = DecayMode(0.0124, 'K_S0 pi0', model='PHSP')
dm2 = DecayMode(0.692, 'pi+ pi-')
dm3 = DecayMode(0.98823, 'gamma gamma')
dc = DecayChain('D0', {'D0':dm1, 'K_S0':dm2, 'pi0':dm3})

dc<jupyter_output><empty_output><jupyter_text>Flatten the decay chain, i.e. replace all intermediate, decaying particles, with their final states:- The BF is now the *visible BF*<jupyter_code>dc.flatten()
dc.print_as_tree()
#dcv = DecayChainViewer(dc.to_dict())
#dcv.graph.write_pdf('test.pdf')

DecayChainViewer(dc.to_dict())<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10268717948717948, 'fraction_numerical': 0.0352, 'mean_word_length': 3.244471530558941, 'pattern_counts': {'":': 0, '<': 199, '<?xml version=': 0, '>': 202, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '49', 'dir': 'jupyter-structured-clean-dedup', 'id': '308633', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 8577, 'n_tokens_neox': 7857, 'n_words': 2416, 'path': 'ParticlesAndDecays.ipynb', 'repo_name': 'eduardo-rodrigues/2019-10-16_PyHEP2019Workshop', 'size': '24386'}