<commit_before><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
  <meta http-equiv="CONTENT-TYPE" content="text/html; charset=UTF-8">
  <title>glideinWMS - Glidein Frontend</title>
  <meta name="CREATED" content="0;0">
  <meta name="CHANGED" content="20100521;20081031;14115200">    
  <link rel="stylesheet" type="text/css" href="../common/glideinWMS.css" media="screen, projection" />
</head>

<body lang="en-US" dir="ltr">
    <h1>
        <a href="index.html">GlideinWMS</a>
	<span>The Glidein-based Workflow Management System</span>
    </h1>
    <ul class="breadcrumbs">
        <li><a href="../index.html">Home</a></li>
        <li><a href="./index.html">Glidein Frontend</a></li>
        <li>Installation</li>
    </ul>
    <div class="clear" />
    <div class="leftmenu">
        <ul class="components">
	<li> <a href="../index.html">Home</a></li>
	<li> <a href="../download.html">Download</a></li>
	<li> <a href="../frontend/index.html">Glidein Frontend</a></li>
    <li> <a href="../corral.html">Corral Frontend</a></li>
	<li> <a href="../factory/index.html">WMS Factory</a></li>
	<li> <a href="../components/index.html" >Components</a></li>
	<li> <a href="../components/faq.html" class="last">FAQ</a></li>
        </ul>
   <div class="search">
        <div id="cse-search-form" style="width: 100%;">Loading</div>
        <script src="http://www.google.com/jsapi" type="text/javascript"></script>
        <script src="../common/glidein.js" type="text/javascript"></script>
        <script type="text/javascript">
            google.load('search', '1', {language : 'en', style : google.loader.themes.MINIMALIST});
            google.setOnLoadCallback(onLoad);
        </script>
    </div>
    </div>
    <div id="searchcontent"><h2>Search Results</h2><div id="cse"></div> </div>
<div class="content">
    <div class="heading">
    <img align="right" width="280px" border="0px"
        src="../images/simple_diagram.png" usemap="#rightimage">
    <map name="rightimage">
        <area shape="rect" coords="90,3,177,60" href="../frontend/index.html" />
        <area shape="rect" coords="5,88,118,146" href="../components/collector_install.html" />
        <area shape="rect" coords="134,88,275,146" href="../factory/index.html" />
        <area shape="rect" coords="189,3,274,60" target="_blank" href="../corral.html" />
    </map>

    <h2>Glidein Frontend</h2>
	<ul class="subcomponents">
	     <li> <a href="./index.html">Overview</a></li>
	     <li> Install</li>
	     <li> <a href="./configuration.html">Configuration</a></li>
	     <li> <a href="./design.html">Design</a></li>
         <li> <a href="./monitoring.html">Monitoring</a></li>
	     <li class="last"> <a href="./troubleshooting.html">Troubleshooting</a></li>
	</ul>
        <h3>Installation</h3>
	</div>
<div class="jump">
<u>Jump to:</u>
<ol>
    <li><a href="#description">Description</a></li>
    <li><a href="#hardware">Hardware requirements</a></li>
    <li><a href="#software">Needed software</a></li>

    <li><a href="#before_install">Before you begin...</a></li>
    <li><a href="#system">Condor installation</a></li>
    <li><a href="#frontend">Frontend installation</a></li>
</ol>
</div>
<div class="related">
<u><b>Installation Procedure</b></u><br/><br/>

<a href="../install.html">Installation Overview</a><br/><br/>
<a href="../install.html#osg">Connecting to OSG Factory</a><br/><br/>

Preceded by: <a href="../components/pool_install.html">User Pool Schedd</a><br/>
Followed by: <a href="../components/index.html">Other Components</a><br/>
Install as: <b>Vo Frontend User</b><br/>
</div>
<div class="section">

            <h2><a name="description">1. Description</a></h2>
        <p><img src="../images/overview_frontend.png" name="graphics1" align="right" border="0"></p>
            <p>The Glidein Frontend will match user jobs with glidein factory ads. It is responsible for the submission of new glideins.</p>

            <p>
                The VO frontend also keeps part of the configuration of a glidein, and can also provide the 
                <a href="../factory/install.html">glidein Factory</a> with the pilot proxies.
            </p>

    <p> Note: if you are installing only the frontend
    and planning to connect to the OSG factory, please refer to this section:
    <a href="../install.html#osg">Connecting to OSG Factory</a></p>

    </div>




    <div class="section">

            <h2><a name="hardware">2. Hardware requirements</a></h2>
            <table class="requirements" style="width: 500px;">
            <tr class="head">
                <td>CPUs</td>

                <td>Memory</td>
                <td>Disk</td>
            </tr>
            <tr>
                <td>1</td>
                <td>1GB</td>
                <td>~20GB</td>

            </tr>
            </table>
            
            <p>
                This machine needs <b>a fast CPU</b> and a moderate amount of <b>memory</b> (<b>1GB </b>should be enough).<br>

                The <b>disk</b> needed is just for binaries, config files and log files (<b>20GB</b> should be enough)
            </p>
    </div>




    <div class="section">

    <h2><a name="software">3. Needed software</a></h2>
    <table class="requirements">
    <tr class="head">
        <td>Software</td>
        <td>Notes</td>
        <td>Install Before glideinWMS</td>

    </tr>
    <tr>
        <td>Linux OS</td>
        <td>
            A reasonably recent <a target="_blank" href="http://www.cs.wisc.edu/condor/manual/v7.0/1_5Availability.html">Condor-supported OS</a>
            Linux OS (RH/SL4 and RH/SL5 tested at press time).
        </td>
        <td class="checked"> X </td>

    </tr>
    <tr>
        <td><a target="_blank" href="http://www.python.org/">Python</a> interpreter</td>
        <td>v2.3.4 or above</td>
        <td class="checked"> X </td>
    </tr>

    <tr>
        <td>The perl-Time-HiRes rpm.</td>
        <td>This rpm may already be included in perl, depending on the perl version</td>
        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="https://twiki.grid.iu.edu/twiki/bin/view/Documentation/WebHome">OSG client software</a>.</td>

        <td>This can be installed prior to glideinWMS, but the installer can install it inline with the glideinWMS install</td>
        <td> &nbsp; </td>
    </tr>
    <tr>
        <td>
            A HTTP server, like <a target="_blank" href="http://www.apache.org/">Apache</a> or
            <a target="_blank" href="http://www.stllinux.org/meeting_notes/2001/0719/tux/intro.html">TUX</a>.
        </td>

        <td>This should be installed prior to glideinWMS (see <a href="#software_httpd">below</a>)</td>
        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="http://www.cs.wisc.edu/condor/downloads/">Condor distribution</a> as a tarball.</td>

        <td>The installer will use the tarball to install and configure Condor inline</td>
        <td> &nbsp; </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="http://oss.oetiker.ch/rrdtool/">RRDTool package</a></td>
        <td>v1.2.18 or later (see <a href="#software_rrdtool">additional install notes</a>)</td>

        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="http://freshmeat.net/projects/m2crypto">M2Crypto python library</a></td>
        <td>v0.17 or later (see <a href="#software_m2crypto">additional install notes</a>)</td>

        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="http://sourceforge.net/projects/javascriptrrd/">javascriptRRD</a> package</td>
        <td>v0.5 or later with flot (see <a href="#software_javascriptRRD">additional install notes</a>)</td>

        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a href="index.html">glideinWMS software</a>.</td>
        <td>&nbsp;</td>
        <td> &nbsp; </td>

    </tr>
    </table>

    <br>
    <h4>NOTE:</h4>
    <ul>
        <li>Condor version v7.3.1 has a known issue with incorrect return/exit codes of condor_status and condor_q</li>
        <li>

            If you are using Condor version v7.3.2 disable VOMS checking in condor_config file used by Condor daemons other 
            than that used by user schedd. VOMS checking adds unrequired overhead. To do so, set <br>
            <blockquote>USE_VOMS_ATTRIBUTES = False </blockquote> 
            or for individual condor daemons like collector<br>
            <blockquote>COLLECTOR.USE_VOMS_ATTRIBUTES = False </blockquote> 
        </li>
    </ul>
    
    <h3><a name="software_httpd">3.2 HTTP Server</a></h3>

    <p>
        The glidein Frontend needs a HTTP server, like <a target="_blank" href="http://www.apache.org/">Apache</a> or 
        <a target="_blank" href="http://www.stllinux.org/meeting_notes/2001/0719/tux/intro.html">TUX</a>.  The server should be installed on 
        the same node, but a different node can be used as long as the web area is writable from this one. Servers often come 
        pre-installed with HTTP server software, so if you have one running, just reuse it.  Otherwise, the installer can help 
        you install one (as root).  (See <a href="../components/prerequisites.html#system_httpd">GlideinWMS Component Install</a>) 
    </p>
    
    <h3><a name="software_rrdtool">3.3 RRDTool</a></h3>
    <p>

        You will also need the python module for <a target="_blank" href="http://oss.oetiker.ch/rrdtool/">RRDTool</a> (v1.2.18 or later). Many 
        systems come with packages for it; if possible use that.  Otherwise see 
        <a href="../components/prerequisites.html#system_rrdtool">additional install notes</a> for alternative installs.
    </p>

    <h3><a name="software_m2crypto">3.4 M2Crypto</a></h3>
    <p>
        You will need the <a target="_blank" href="http://freshmeat.net/projects/m2crypto">M2Crypto python library</a>. A few systems include it in the 
        software distribution; if possible install the system one.  Otherwise see 
        <a href="../components/prerequisites.html#system_m2crypto">additional install notes</a> for alternative installs.
    </p>

    <h3><a name="software_javascriptRRD">3.5 javascriptRRD</a></h3>
    <p>
        You will need the <a target="_blank" href="http://sourceforge.net/projects/javascriptrrd/">javascriptRRD package</a>.  It contains the 
        javascript libraries needed by the monitoring.  Just download the tarball (with flot), and untar it.  You will need to 
        point the installer to this directory.
    </p>
    </div>




    <div class="section">

    <h2><a name="before_install">4. Before you begin...</a></h2>
    <h3><a name="before_install_users">4.1 Required Users</a></h3>
    <p>
        The installer will ask you for several non-privileged users during the install process.  These should be created prior to running the
        glideinWMS installer.  
    </p>
    
    <table class="requirements">
    <tr class="head">

        <td>User</td>
        <td>Notes</td>
    </tr>
    <tr>
        <td>Vo Frontend User</td>
        <td>
            The Glidein Frontend should be installed as a non-privilged user.  <strong>Note:</strong> It is recommended that you install
            Condor as this user as well.
        </td>

    </tr>
    </table>
    <br>
    
   <h3><a name="before_install_certs">4.2 Required Certificates/Proxies</a></h3>
    <p>
        The installer will ask for several DNs for GSI authentication.  You have the option of using a service certificate or a proxy.  These should be
        created and put in place before running the installer.  The following is a list of DNs the installer will ask for:
        <ul>
            <li>WMS Collector cert/proxy DN</li>

            <li>User Pool Collector cert/proxy DN</li>
            <li>User Submitter cert/proxy DN</li>
            <li>Glidein Frontend Condor cert/proxy DN (cannot use a cert here)</li>
        </ul>
        <strong>Note:</strong> In some places the installer will also ask for nicknames to go with the DNs.  For the most part the name given doesn't 
        really matter.  There is one case where is does matter.  If you are using privilege separation, then, on the WMS Collector, the nickname for 
        each Glidein Frontend must be the username that you created for the frontend.<br> 
        <strong>Note 2:</strong> The installer will ask if these are trusted Condor Daemons.  Answer 'y'. 
    </p>

    
    <h3><a name="before_install_dirs">4.3 Required Directories</a></h3>
    <p>
        When installing the Glidein Frontend you will be presented with a question asking for the directory location for various items.  The example below puts<commit_msg>Documentation: Frontend condor client user explained.<commit_after><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
  <meta http-equiv="CONTENT-TYPE" content="text/html; charset=UTF-8">
  <title>glideinWMS - Glidein Frontend</title>
  <meta name="CREATED" content="0;0">
  <meta name="CHANGED" content="20100521;20081031;14115200">    
  <link rel="stylesheet" type="text/css" href="../common/glideinWMS.css" media="screen, projection" />
</head>

<body lang="en-US" dir="ltr">
    <h1>
        <a href="index.html">GlideinWMS</a>
	<span>The Glidein-based Workflow Management System</span>
    </h1>
    <ul class="breadcrumbs">
        <li><a href="../index.html">Home</a></li>
        <li><a href="./index.html">Glidein Frontend</a></li>
        <li>Installation</li>
    </ul>
    <div class="clear" />
    <div class="leftmenu">
        <ul class="components">
	<li> <a href="../index.html">Home</a></li>
	<li> <a href="../download.html">Download</a></li>
	<li> <a href="../frontend/index.html">Glidein Frontend</a></li>
    <li> <a href="../corral.html">Corral Frontend</a></li>
	<li> <a href="../factory/index.html">WMS Factory</a></li>
	<li> <a href="../components/index.html" >Components</a></li>
	<li> <a href="../components/faq.html" class="last">FAQ</a></li>
        </ul>
   <div class="search">
        <div id="cse-search-form" style="width: 100%;">Loading</div>
        <script src="http://www.google.com/jsapi" type="text/javascript"></script>
        <script src="../common/glidein.js" type="text/javascript"></script>
        <script type="text/javascript">
            google.load('search', '1', {language : 'en', style : google.loader.themes.MINIMALIST});
            google.setOnLoadCallback(onLoad);
        </script>
    </div>
    </div>
    <div id="searchcontent"><h2>Search Results</h2><div id="cse"></div> </div>
<div class="content">
    <div class="heading">
    <img align="right" width="280px" border="0px"
        src="../images/simple_diagram.png" usemap="#rightimage">
    <map name="rightimage">
        <area shape="rect" coords="90,3,177,60" href="../frontend/index.html" />
        <area shape="rect" coords="5,88,118,146" href="../components/collector_install.html" />
        <area shape="rect" coords="134,88,275,146" href="../factory/index.html" />
        <area shape="rect" coords="189,3,274,60" target="_blank" href="../corral.html" />
    </map>

    <h2>Glidein Frontend</h2>
	<ul class="subcomponents">
	     <li> <a href="./index.html">Overview</a></li>
	     <li> Install</li>
	     <li> <a href="./configuration.html">Configuration</a></li>
	     <li> <a href="./design.html">Design</a></li>
         <li> <a href="./monitoring.html">Monitoring</a></li>
	     <li class="last"> <a href="./troubleshooting.html">Troubleshooting</a></li>
	</ul>
        <h3>Installation</h3>
	</div>
<div class="jump">
<u>Jump to:</u>
<ol>
    <li><a href="#description">Description</a></li>
    <li><a href="#hardware">Hardware requirements</a></li>
    <li><a href="#software">Needed software</a></li>

    <li><a href="#before_install">Before you begin...</a></li>
    <li><a href="#system">Condor installation</a></li>
    <li><a href="#frontend">Frontend installation</a></li>
</ol>
</div>
<div class="related">
<u><b>Installation Procedure</b></u><br/><br/>

<a href="../install.html">Installation Overview</a><br/><br/>
<a href="../install.html#osg">Connecting to OSG Factory</a><br/><br/>

Preceded by: <a href="../components/pool_install.html">User Pool Schedd</a><br/>
Followed by: <a href="../components/index.html">Other Components</a><br/>
Install as: <b>Vo Frontend User</b><br/>
</div>
<div class="section">

            <h2><a name="description">1. Description</a></h2>
        <p><img src="../images/overview_frontend.png" name="graphics1" align="right" border="0"></p>
            <p>The Glidein Frontend will match user jobs with glidein factory ads. It is responsible for the submission of new glideins.</p>

            <p>
                The VO frontend also keeps part of the configuration of a glidein, and can also provide the 
                <a href="../factory/install.html">glidein Factory</a> with the pilot proxies.
            </p>

    <p> Note: if you are installing only the frontend
    and planning to connect to the OSG factory, please refer to this section:
    <a href="../install.html#osg">Connecting to OSG Factory</a></p>

    </div>




    <div class="section">

            <h2><a name="hardware">2. Hardware requirements</a></h2>
            <table class="requirements" style="width: 500px;">
            <tr class="head">
                <td>CPUs</td>

                <td>Memory</td>
                <td>Disk</td>
            </tr>
            <tr>
                <td>1</td>
                <td>1GB</td>
                <td>~20GB</td>

            </tr>
            </table>
            
            <p>
                This machine needs <b>a fast CPU</b> and a moderate amount of <b>memory</b> (<b>1GB </b>should be enough).<br>

                The <b>disk</b> needed is just for binaries, config files and log files (<b>20GB</b> should be enough)
            </p>
    </div>




    <div class="section">

    <h2><a name="software">3. Needed software</a></h2>
    <table class="requirements">
    <tr class="head">
        <td>Software</td>
        <td>Notes</td>
        <td>Install Before glideinWMS</td>

    </tr>
    <tr>
        <td>Linux OS</td>
        <td>
            A reasonably recent <a target="_blank" href="http://www.cs.wisc.edu/condor/manual/v7.0/1_5Availability.html">Condor-supported OS</a>
            Linux OS (RH/SL4 and RH/SL5 tested at press time).
        </td>
        <td class="checked"> X </td>

    </tr>
    <tr>
        <td><a target="_blank" href="http://www.python.org/">Python</a> interpreter</td>
        <td>v2.3.4 or above</td>
        <td class="checked"> X </td>
    </tr>

    <tr>
        <td>The perl-Time-HiRes rpm.</td>
        <td>This rpm may already be included in perl, depending on the perl version</td>
        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="https://twiki.grid.iu.edu/twiki/bin/view/Documentation/WebHome">OSG client software</a>.</td>

        <td>This can be installed prior to glideinWMS, but the installer can install it inline with the glideinWMS install</td>
        <td> &nbsp; </td>
    </tr>
    <tr>
        <td>
            A HTTP server, like <a target="_blank" href="http://www.apache.org/">Apache</a> or
            <a target="_blank" href="http://www.stllinux.org/meeting_notes/2001/0719/tux/intro.html">TUX</a>.
        </td>

        <td>This should be installed prior to glideinWMS (see <a href="#software_httpd">below</a>)</td>
        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="http://www.cs.wisc.edu/condor/downloads/">Condor distribution</a> as a tarball.</td>

        <td>The installer will use the tarball to install and configure Condor inline</td>
        <td> &nbsp; </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="http://oss.oetiker.ch/rrdtool/">RRDTool package</a></td>
        <td>v1.2.18 or later (see <a href="#software_rrdtool">additional install notes</a>)</td>

        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="http://freshmeat.net/projects/m2crypto">M2Crypto python library</a></td>
        <td>v0.17 or later (see <a href="#software_m2crypto">additional install notes</a>)</td>

        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a target="_blank" href="http://sourceforge.net/projects/javascriptrrd/">javascriptRRD</a> package</td>
        <td>v0.5 or later with flot (see <a href="#software_javascriptRRD">additional install notes</a>)</td>

        <td class="checked"> X </td>
    </tr>
    <tr>
        <td>The <a href="index.html">glideinWMS software</a>.</td>
        <td>&nbsp;</td>
        <td> &nbsp; </td>

    </tr>
    </table>

    <br>
    <h4>NOTE:</h4>
    <ul>
        <li>Condor version v7.3.1 has a known issue with incorrect return/exit codes of condor_status and condor_q</li>
        <li>

            If you are using Condor version v7.3.2 disable VOMS checking in condor_config file used by Condor daemons other 
            than that used by user schedd. VOMS checking adds unrequired overhead. To do so, set <br>
            <blockquote>USE_VOMS_ATTRIBUTES = False </blockquote> 
            or for individual condor daemons like collector<br>
            <blockquote>COLLECTOR.USE_VOMS_ATTRIBUTES = False </blockquote> 
        </li>
    </ul>
    
    <h3><a name="software_httpd">3.2 HTTP Server</a></h3>

    <p>
        The glidein Frontend needs a HTTP server, like <a target="_blank" href="http://www.apache.org/">Apache</a> or 
        <a target="_blank" href="http://www.stllinux.org/meeting_notes/2001/0719/tux/intro.html">TUX</a>.  The server should be installed on 
        the same node, but a different node can be used as long as the web area is writable from this one. Servers often come 
        pre-installed with HTTP server software, so if you have one running, just reuse it.  Otherwise, the installer can help 
        you install one (as root).  (See <a href="../components/prerequisites.html#system_httpd">GlideinWMS Component Install</a>) 
    </p>
    
    <h3><a name="software_rrdtool">3.3 RRDTool</a></h3>
    <p>

        You will also need the python module for <a target="_blank" href="http://oss.oetiker.ch/rrdtool/">RRDTool</a> (v1.2.18 or later). Many 
        systems come with packages for it; if possible use that.  Otherwise see 
        <a href="../components/prerequisites.html#system_rrdtool">additional install notes</a> for alternative installs.
    </p>

    <h3><a name="software_m2crypto">3.4 M2Crypto</a></h3>
    <p>
        You will need the <a target="_blank" href="http://freshmeat.net/projects/m2crypto">M2Crypto python library</a>. A few systems include it in the 
        software distribution; if possible install the system one.  Otherwise see 
        <a href="../components/prerequisites.html#system_m2crypto">additional install notes</a> for alternative installs.
    </p>

    <h3><a name="software_javascriptRRD">3.5 javascriptRRD</a></h3>
    <p>
        You will need the <a target="_blank" href="http://sourceforge.net/projects/javascriptrrd/">javascriptRRD package</a>.  It contains the 
        javascript libraries needed by the monitoring.  Just download the tarball (with flot), and untar it.  You will need to 
        point the installer to this directory.
    </p>
    </div>




    <div class="section">

    <h2><a name="before_install">4. Before you begin...</a></h2>
    <h3><a name="before_install_users">4.1 Required Users</a></h3>
    <p>
        The installer will ask you for several non-privileged users during the install process.  These should be created prior to running the
        glideinWMS installer.  
    </p>
    
    <table class="requirements">
    <tr class="head">

        <td>User</td>
        <td>Notes</td>
    </tr>
    <tr>
        <td>Vo Frontend User</td>
        <td>
            The Glidein Frontend should be installed as a non-priviliged user.<br/>  
            <strong>Note:</strong>If you are not re-using an existing condor client and instead installing a <a href="#system">condor client</a> specific for the frontend, you should install it as this user as well.  
        </td>

    </tr>
    </table>
    <br>
    
   <h3><a name="before_install_certs">4.2 Required Certificates/Proxies</a></h3>
    <p>
        The installer will ask for several DNs for GSI authentication.  You have the option of using a service certificate or a proxy.  These should be
        created and put in place before running the installer.  The following is a list of DNs the installer will ask for:
        <ul>
            <li>WMS Collector cert/proxy DN</li>

            <li>User Pool Collector cert/proxy DN</li>
            <li>User Submitter cert/proxy DN</li>
            <li>Glidein Frontend Condor cert/proxy DN (cannot use a cert here)</li>
        </ul>
        <strong>Note:</strong> In some places the installer will also ask for nicknames to go with the DNs.  For the most part the name given doesn't 
        really matter.  There is one case where is does matter.  If you are using privilege separation, then, on the WMS Collector, the nickname for 
        each Glidein Frontend must be the username that you created for the frontend.<br> 
        <strong>Note 2:</strong> The installer will ask if these are trusted Condor Daemons.  Answer 'y'. 
    </p>

    
    <h3><a name="before_install_dirs">4.3 Required Directories</a></h3>
    <p>
        When installing the Glidein Frontend you will be presented with a question asking for the directory location for various items.  The example below puts
{'fraction_non_alphanumeric': 0.16310181224235826, 'fraction_numerical': 0.014777941977133079, 'mean_word_length': 2.7112137393563285, 'pattern_counts': {'":': 0, '<': 926, '<?xml version=': 0, '>': 926, 'https://': 2, 'lorem ipsum': 0, 'www.': 16, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4790901', 'n_tokens_mistral': 8771, 'n_tokens_neox': 8303, 'n_words': 2254}