<commit_before><!DOCTYPE html>
<html>
  <head>
    <title>mtarep</title>
    <script src="/bootstrap/js/jquery-1.8.3.min.js"></script>
    <script src="/bootstrap/js/bootstrap.js"></script>
    <link rel="stylesheet" href="/bootstrap/css/bootstrap.css">
  </head>

  <body style="background-color: #575757; padding-top: 50px;">

    <div class="navbar navbar-fixed-top">
      <div class="navbar-inner">
        <div class="container-fluid">
          <a class="brand"><strong>mtarep - reputation monitor</strong></a>
          <ul class="nav pull-right">
            <li class="dropdown">
              <a href="#" class="dropdown-toggle" data-toggle="dropdown"><img src="/bootstrap/img/list.png"></a>
              <ul class="dropdown-menu">
                <li><a href="/" style="color: #cccccc;">Home</a></li>
                <li><a href="/help" style="color: #cccccc;">Help</a></li>
                <li><a href="/graphs" style="color: #cccccc;">Graphs</a></li>
                <li><a href="/version" style="color: #cccccc;">Version</a></li>
              </ul>
            </li>
          </ul>
        </div>
      </div>
    </div>

    <br>
    <div class="container-fluid">
      <div class="row-fluid">
        <div class="span1"></div>
        <div class="span10">
          <br>
          <br>
          <div class="tabbable" style="margin-bottom: 18px;">
            <ul class="nav nav-tabs">
              <li class="active"><a href="#tab1" data-toggle="tab"><strong>Overview</strong></a></li>
              <li class=""><a href="#tab2" data-toggle="tab"><strong>Web Interface</strong></a></li>
              <li class=""><a href="#tab3" data-toggle="tab"><strong>Graphing</strong></a></li>
              <li class=""><a href="#tab4" data-toggle="tab"><strong>REST Api</strong></a></li>
              <li class=""><a href="#tab5" data-toggle="tab"><strong>Configuration</strong></a></li>
              <li class=""><a href="#tab6" data-toggle="tab"><strong>Current RBLs</strong></a></li>
              <li class=""><a href="#tab7" data-toggle="tab"><strong>Brightmail Reputation</strong></a></li>
            </ul>
            <div class="tab-content" style="color: #000000; padding-bottom: 9px; border-bottom: 1px solid #ddd;">
              <div class="tab-pane active" id="tab1">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>The following data is collected and displayed for each postfix MTA you monitor with mtarep:</strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 330px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">Return Path Sender Score<br>Microsoft Smart Network Data Services<br>Postfix SMTP Rejections<br>Cloudmark CSI<br>Brightmail Reputation Service<br>RBL/DNSBL Listings</font></p>
                <br>
                <p style="color: #D0D0D0; font-size: 120%;"><strong>All the data is collected and inserted into a redis data store whenever the collector completes a run. The collector (collector.rb) can be scheduled via cron. The sinatra webapp (app.rb) handles generating the HTML and retrieving the data from redis.<br><br>All table cells w/ <span style="color: red; font-size: 120%;">red</span> text can be clicked for removal instructions.<br><br>Any modifications to the main mtarep.yml configuration file will be picked up automatically by mtarep before it's subsequent collection run. The collector's cron scheduled collection interval is explained in more detail in the README for mtarep.</strong></p>
              </div>
              <div class="tab-pane" id="tab2">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>When a provider block or rbl cell contains a listing, it becomes clickable. For provider blocks, when you click a block link a modal appears containing the most recent smtp rejection message from your server's maillog (location and access are configurable in the mtarep.yml file), as well details about the listing and a direct link to that provider's block removal form or instructions. The same scenario occurs for rbl listings but without the maillog info, as mtarep queries rbls directly via a dns lookup.<br><br>Each issue modal contains an acknowledgement button that can be clicked once you begin working an mtarep reported issue. When the acknowledgement button is clicked, a unique key is inserted back into redis that holds basic details about the issue, including a timestamp and the mtarep authenticated http username. Once acknowledged, the acknowledgement button is replaced with a timestamp and the mtarep authenticated http username that acknowledged the issue. This prevents multiple users working the same issue unknowingly.<br><br>Each column header in the web interface allows for rows sorting (eg: sort by rbl listings, microsoft SNDS filtering or trap hits, hostname, etc..).</strong></p>
              </div>
              <div class="tab-pane" id="tab3">
              <p style="color: #D0D0D0; font-size: 120%;"><strong>To enable graphing you will need to configure a few things outside the scope of mtarep.<br><br>Sent, bounced, expired and feedback loop bar graphs can be configured for the domains of your choosing via the 'graph_domains' YAML array collection in the mtarep.yml configuration file. The graphing is provided by highcharts.js. Individual bar graphs can be removed from view by clicking each type of graph in the graphing legend. This allows more granular detail on specific bar graphs, which can be particularly helpful if your sent total bar graph obfuscates the shit out of the bounce, fbl or expired graphs.<br><br>Each individual bar graph is calculated from midnight on the current day and continues to be calculated until 11:59pm on that same day. The data used to calculate the sent, bounced, fbl and expired bar graphs is not *collected* by mtarep. However, bar graphs will be calculated and rendered by mtarep if the appropriate data exists in the same redis db used by mtarep.<br><br><br>The mtarep graphing code will search your mtarep redis backend for HINCRBY based keys in the format of:</strong></p><p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 160px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">mycompany:20140208:expired<br>mycompany:20140208:fbl<br>mycompany:20140208:bounced<br>mycompany:20140208:sent</font></p>
<p style="color: #D0D0D0;"><i>Where '20140208' is the current date and 'mycompany' is some type of identifier (perhaps your sending domain, company name or client name?)</i></p>
<p style="color: #D0D0D0; font-size: 120%;"><strong>These redis keys should contain an incrementing total for each unique domain that has been sent, bounced, expired or feedback-loop received. The exact redis operation is 'hincrby'.<br><br><br>Using the redis-rb ruby client you could do something like this:</strong></p>
<p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 410px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">require 'redis'<br><br>@redis = Redis.new(:host => 'your redis server')<br><br>key = ['mycompany']<br>key << Time.now.strftime("%Y%m%d") <br> key << 'bounced'<br><br>@redis.hincrby(key.join(':'), 'gmail.com', 1)</font></p>
<p style="color: #D0D0D0; font-size: 120%;"><strong>The above code would increment a counter for each bounced email to 'gmail.com' under the redis key 'mycompany:20140208:bounced'. This could be accomplished by creating a custom redis output plugin with logstash to filter and ship this data directly from your postfix mail logs to either a pubsub broker like rabbitmq or perhaps even directly to your redis datastore.</strong></p>
              </div>
              <div class="tab-pane" id="tab4">
              <p style="color: #D0D0D0; font-size: 120%;"><strong>Mtarep comes with a very basic REST api. I expect to add more functionality to the api as needed. At the moment, the api can only be queried for single host data.<br><br>For example, to retrieve mtarep data for a public fqdn of 'mta-test1.domain.tld', you could do something like this:<br></strong></p><p style="padding: 6px; color: #CD7814; background-color: #495674; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">require 'rest-client'<br>require 'json'<br><br>response = RestClient.get("http://username:password@mtarep.domain.tld/api/#{public_fqdn}")<br>data_hash = JSON.parse(response)<br><br>=> {"hostname"=>"mta-test1.domain.tld", "senderscore"=>"98", "sndscolor"=>"green", "sndstraps"=>"2", "brightmail"=>"good", "provblocks"=>"no blocks", "listings"=>"unlisted"}</font></p><br><p style="color: #D0D0D0; font-size: 120%;"><strong>A JSON blob of the requested public fqdn's mtarep data is returned.</strong></p>
              </div>
              <div class="tab-pane" id="tab5">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>Mtarep comes with example configuration files for rackup (config.ru), thin (thin.yml) and mtarep itself (mtarep.yml). Please copy the example configs without the '.example' appendage and adjust each config according to your environment.<br><br>The individual mtarep.yml configuration file settings are as follows:</strong></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>error_log</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The absolute path to your main mtarep error log file. All directories in the path must already exist. If the error log file does not exist, it will be created.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>redis_server</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The DNS resolvable hostname of your redis server. Your redis server will be accessed over the default port of 6379 and the primary 'db0' database. Currently non standard ports and redis databases are not supported.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>snds_key</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">Your organization's microsoft smart network data services access key (SNDS). If you do not use microsoft SNDS, you should signup immediately. There is a wealth of info regarding your inbox placement.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>maillog_path</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The absolute path to your current postfix mail log file on each postfix MTA that you are using mtarep to report major provider rejections/blocks for. Currently only postfix log formats are supported.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>ssh_key</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The absolute path to the ssh key file (on the server running mtarep) you want to use to access your remote postfix MTA mail logs with. The ssh key you provide here must have permissions to the remote 'maillog_path' specified above, and for the 'ssh_user' you specify below.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>ssh_user</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The ssh username associated with the 'ssh_key' you specified above.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>http_auth_file</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The absolute path to your mtarep app's http authentication file. This is used for authentcating login access credentials (username/password) to the mtarep web interface<br><br>The format of this file must be:<br><br>&nbsp;&nbsp;<span style="color: #FF3C3C;">username:{SHA}ME2JP/+546KPSPZQxQirw0qkUsQRyYWM= </span><br><br>Currently only a SHA1 base64 digest is supported (Digest::SHA1.base64digest('password')).</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>mta_map</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">This setting can contain either a YAML array collection or a YAML hash collection.<br><br>A YAML array collection is used when you want mtarep to use the same fqdn/hostname to lookup public fqdn/hostname/ip reputation data as well as the fqdn/hostname to ssh to for mail log parsing (ESP and ISP (provider) SMTP rejection log entries). Do not use this type of map if you ssh to your MTAs using a different fqdn/hostname or host alias from it's public mail fqdn/hostname.<br><br>The YAML hash collection support for this configuration setting should be used if you ssh to your MTAs using an internal hostname or alias that is different from the host's public fqdn hostname.<br><br>For example, if you have an MTA with a public HELO hostname of mta1.mydomain.com and you also access the MTA with ssh using that same hostname, then you want to use a YAML array collection here. If you have an MTA with a public HELO hostname of mta1.mydomain.com but you ssh to the MTA using the shortname 'mta1', then you want to use a YAML hash collection here (mta1: 'mta1.mydomain.com', etc..).</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>graph_domains</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The domains that you want mtarep to render sent, bounced, feedback-loop and expired bat graph totals for.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>rbls</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">A YAML array collection of RBL/DNSBL's you want mtarep to check your MTA public IP addresses against. The format of these must be the RBL hostnames that are queried for listings (eg: bl.spamcop.net).</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>provider_block_strings</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">A YAML hash collection of external email provider names (the key) and a string (the value) that indicates a provider is blocking your MTA. These key/values are used by mtarep to search your remote MTA mail logs for particular rejection text patterns. The rejection patterns for several major email providers are already included in the mtarep.yml.example file.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>removal_links</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">A YAML hash collection of external email provider names and rbl lists (the key) and a corresponding URL (the value) to that provider or rbl's listing info and/or removal form.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>assistance_links</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">A YAML hash collection of any custom documentation you maintain that outlines steps your organization has for resolving a specific mtarep reported issue. These links are used in the modal of a clicked issue in mtarep.</font></p>
              </div>
              <div class="tab-pane" id="tab6">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>Mtarep is currently configured to check each monitored MTA ip address against the following RBL lists:</strong></p><br>
                <% rbl_list = @redis.lrange('rbls', 0, -1) %>
                <p style="padding: 6px; color: #CD7814; max-width: 220px; background-color: #495674; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco"><%= rbl_list.join('<br>') %></font></p>
              </div>
              <div class="tab-pane" id="tab7">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>The possible values in the Brightmail reputation service column are:</strong>
                <ul>
                  <li style="color: #D0D0D0; font-size: 104%;"><strong><span style="color: #90EE90;">Good:</span></strong>&nbsp;&nbsp;<strong>Brightmail has detected that only legitimate email has originated from the address</strong></li>
                  <li style="color: #D0D0D0; font-size: 104%;"><strong><span style="color: #9E9E9E;">Neutral:</span></strong>&nbsp;&nbsp;<strong>Brightmail data does not indicate a positive or negative reputation</strong></li>
                  <li style="color: #D0D0D0; font-size: 104%;"><strong><span style="color: #FF3C3C;">Bad:</span></strong>&nbsp;&nbsp;<strong>Brightmail has detected suspicious activity, such as spam or malicious code, originating from the address</strong></li>
                </ul>
              </div>
            </div>
          </div>
        </div>
        <div class="span1"></div>
      </div>
    </div>

  </body>
</html>
<commit_msg>expanded width of redis graph key example code block<commit_after><!DOCTYPE html>
<html>
  <head>
    <title>mtarep</title>
    <script src="/bootstrap/js/jquery-1.8.3.min.js"></script>
    <script src="/bootstrap/js/bootstrap.js"></script>
    <link rel="stylesheet" href="/bootstrap/css/bootstrap.css">
  </head>

  <body style="background-color: #575757; padding-top: 50px;">

    <div class="navbar navbar-fixed-top">
      <div class="navbar-inner">
        <div class="container-fluid">
          <a class="brand"><strong>mtarep - reputation monitor</strong></a>
          <ul class="nav pull-right">
            <li class="dropdown">
              <a href="#" class="dropdown-toggle" data-toggle="dropdown"><img src="/bootstrap/img/list.png"></a>
              <ul class="dropdown-menu">
                <li><a href="/" style="color: #cccccc;">Home</a></li>
                <li><a href="/help" style="color: #cccccc;">Help</a></li>
                <li><a href="/graphs" style="color: #cccccc;">Graphs</a></li>
                <li><a href="/version" style="color: #cccccc;">Version</a></li>
              </ul>
            </li>
          </ul>
        </div>
      </div>
    </div>

    <br>
    <div class="container-fluid">
      <div class="row-fluid">
        <div class="span1"></div>
        <div class="span10">
          <br>
          <br>
          <div class="tabbable" style="margin-bottom: 18px;">
            <ul class="nav nav-tabs">
              <li class="active"><a href="#tab1" data-toggle="tab"><strong>Overview</strong></a></li>
              <li class=""><a href="#tab2" data-toggle="tab"><strong>Web Interface</strong></a></li>
              <li class=""><a href="#tab3" data-toggle="tab"><strong>Graphing</strong></a></li>
              <li class=""><a href="#tab4" data-toggle="tab"><strong>REST Api</strong></a></li>
              <li class=""><a href="#tab5" data-toggle="tab"><strong>Configuration</strong></a></li>
              <li class=""><a href="#tab6" data-toggle="tab"><strong>Current RBLs</strong></a></li>
              <li class=""><a href="#tab7" data-toggle="tab"><strong>Brightmail Reputation</strong></a></li>
            </ul>
            <div class="tab-content" style="color: #000000; padding-bottom: 9px; border-bottom: 1px solid #ddd;">
              <div class="tab-pane active" id="tab1">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>The following data is collected and displayed for each postfix MTA you monitor with mtarep:</strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 330px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">Return Path Sender Score<br>Microsoft Smart Network Data Services<br>Postfix SMTP Rejections<br>Cloudmark CSI<br>Brightmail Reputation Service<br>RBL/DNSBL Listings</font></p>
                <br>
                <p style="color: #D0D0D0; font-size: 120%;"><strong>All the data is collected and inserted into a redis data store whenever the collector completes a run. The collector (collector.rb) can be scheduled via cron. The sinatra webapp (app.rb) handles generating the HTML and retrieving the data from redis.<br><br>All table cells w/ <span style="color: red; font-size: 120%;">red</span> text can be clicked for removal instructions.<br><br>Any modifications to the main mtarep.yml configuration file will be picked up automatically by mtarep before it's subsequent collection run. The collector's cron scheduled collection interval is explained in more detail in the README for mtarep.</strong></p>
              </div>
              <div class="tab-pane" id="tab2">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>When a provider block or rbl cell contains a listing, it becomes clickable. For provider blocks, when you click a block link a modal appears containing the most recent smtp rejection message from your server's maillog (location and access are configurable in the mtarep.yml file), as well details about the listing and a direct link to that provider's block removal form or instructions. The same scenario occurs for rbl listings but without the maillog info, as mtarep queries rbls directly via a dns lookup.<br><br>Each issue modal contains an acknowledgement button that can be clicked once you begin working an mtarep reported issue. When the acknowledgement button is clicked, a unique key is inserted back into redis that holds basic details about the issue, including a timestamp and the mtarep authenticated http username. Once acknowledged, the acknowledgement button is replaced with a timestamp and the mtarep authenticated http username that acknowledged the issue. This prevents multiple users working the same issue unknowingly.<br><br>Each column header in the web interface allows for rows sorting (eg: sort by rbl listings, microsoft SNDS filtering or trap hits, hostname, etc..).</strong></p>
              </div>
              <div class="tab-pane" id="tab3">
              <p style="color: #D0D0D0; font-size: 120%;"><strong>To enable graphing you will need to configure a few things outside the scope of mtarep.<br><br>Sent, bounced, expired and feedback loop bar graphs can be configured for the domains of your choosing via the 'graph_domains' YAML array collection in the mtarep.yml configuration file. The graphing is provided by highcharts.js. Individual bar graphs can be removed from view by clicking each type of graph in the graphing legend. This allows more granular detail on specific bar graphs, which can be particularly helpful if your sent total bar graph obfuscates the shit out of the bounce, fbl or expired graphs.<br><br>Each individual bar graph is calculated from midnight on the current day and continues to be calculated until 11:59pm on that same day. The data used to calculate the sent, bounced, fbl and expired bar graphs is not *collected* by mtarep. However, bar graphs will be calculated and rendered by mtarep if the appropriate data exists in the same redis db used by mtarep.<br><br><br>The mtarep graphing code will search your mtarep redis backend for HINCRBY based keys in the format of:</strong></p><p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 220px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">mycompany:20140208:expired<br>mycompany:20140208:fbl<br>mycompany:20140208:bounced<br>mycompany:20140208:sent</font></p>
<p style="color: #D0D0D0;"><i>Where '20140208' is the current date and 'mycompany' is some type of identifier (perhaps your sending domain, company name or client name?)</i></p>
<p style="color: #D0D0D0; font-size: 120%;"><strong>These redis keys should contain an incrementing total for each unique domain that has been sent, bounced, expired or feedback-loop received. The exact redis operation is 'hincrby'.<br><br><br>Using the redis-rb ruby client you could do something like this:</strong></p>
<p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 410px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">require 'redis'<br><br>@redis = Redis.new(:host => 'your redis server')<br><br>key = ['mycompany']<br>key << Time.now.strftime("%Y%m%d") <br> key << 'bounced'<br><br>@redis.hincrby(key.join(':'), 'gmail.com', 1)</font></p>
<p style="color: #D0D0D0; font-size: 120%;"><strong>The above code would increment a counter for each bounced email to 'gmail.com' under the redis key 'mycompany:20140208:bounced'. This could be accomplished by creating a custom redis output plugin with logstash to filter and ship this data directly from your postfix mail logs to either a pubsub broker like rabbitmq or perhaps even directly to your redis datastore.</strong></p>
              </div>
              <div class="tab-pane" id="tab4">
              <p style="color: #D0D0D0; font-size: 120%;"><strong>Mtarep comes with a very basic REST api. I expect to add more functionality to the api as needed. At the moment, the api can only be queried for single host data.<br><br>For example, to retrieve mtarep data for a public fqdn of 'mta-test1.domain.tld', you could do something like this:<br></strong></p><p style="padding: 6px; color: #CD7814; background-color: #495674; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">require 'rest-client'<br>require 'json'<br><br>response = RestClient.get("http://username:password@mtarep.domain.tld/api/#{public_fqdn}")<br>data_hash = JSON.parse(response)<br><br>=> {"hostname"=>"mta-test1.domain.tld", "senderscore"=>"98", "sndscolor"=>"green", "sndstraps"=>"2", "brightmail"=>"good", "provblocks"=>"no blocks", "listings"=>"unlisted"}</font></p><br><p style="color: #D0D0D0; font-size: 120%;"><strong>A JSON blob of the requested public fqdn's mtarep data is returned.</strong></p>
              </div>
              <div class="tab-pane" id="tab5">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>Mtarep comes with example configuration files for rackup (config.ru), thin (thin.yml) and mtarep itself (mtarep.yml). Please copy the example configs without the '.example' appendage and adjust each config according to your environment.<br><br>The individual mtarep.yml configuration file settings are as follows:</strong></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>error_log</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The absolute path to your main mtarep error log file. All directories in the path must already exist. If the error log file does not exist, it will be created.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>redis_server</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The DNS resolvable hostname of your redis server. Your redis server will be accessed over the default port of 6379 and the primary 'db0' database. Currently non standard ports and redis databases are not supported.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>snds_key</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">Your organization's microsoft smart network data services access key (SNDS). If you do not use microsoft SNDS, you should signup immediately. There is a wealth of info regarding your inbox placement.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>maillog_path</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The absolute path to your current postfix mail log file on each postfix MTA that you are using mtarep to report major provider rejections/blocks for. Currently only postfix log formats are supported.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>ssh_key</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The absolute path to the ssh key file (on the server running mtarep) you want to use to access your remote postfix MTA mail logs with. The ssh key you provide here must have permissions to the remote 'maillog_path' specified above, and for the 'ssh_user' you specify below.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>ssh_user</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The ssh username associated with the 'ssh_key' you specified above.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>http_auth_file</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The absolute path to your mtarep app's http authentication file. This is used for authentcating login access credentials (username/password) to the mtarep web interface<br><br>The format of this file must be:<br><br>&nbsp;&nbsp;<span style="color: #FF3C3C;">username:{SHA}ME2JP/+546KPSPZQxQirw0qkUsQRyYWM= </span><br><br>Currently only a SHA1 base64 digest is supported (Digest::SHA1.base64digest('password')).</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>mta_map</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">This setting can contain either a YAML array collection or a YAML hash collection.<br><br>A YAML array collection is used when you want mtarep to use the same fqdn/hostname to lookup public fqdn/hostname/ip reputation data as well as the fqdn/hostname to ssh to for mail log parsing (ESP and ISP (provider) SMTP rejection log entries). Do not use this type of map if you ssh to your MTAs using a different fqdn/hostname or host alias from it's public mail fqdn/hostname.<br><br>The YAML hash collection support for this configuration setting should be used if you ssh to your MTAs using an internal hostname or alias that is different from the host's public fqdn hostname.<br><br>For example, if you have an MTA with a public HELO hostname of mta1.mydomain.com and you also access the MTA with ssh using that same hostname, then you want to use a YAML array collection here. If you have an MTA with a public HELO hostname of mta1.mydomain.com but you ssh to the MTA using the shortname 'mta1', then you want to use a YAML hash collection here (mta1: 'mta1.mydomain.com', etc..).</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>graph_domains</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">The domains that you want mtarep to render sent, bounced, feedback-loop and expired bat graph totals for.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>rbls</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">A YAML array collection of RBL/DNSBL's you want mtarep to check your MTA public IP addresses against. The format of these must be the RBL hostnames that are queried for listings (eg: bl.spamcop.net).</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>provider_block_strings</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">A YAML hash collection of external email provider names (the key) and a string (the value) that indicates a provider is blocking your MTA. These key/values are used by mtarep to search your remote MTA mail logs for particular rejection text patterns. The rejection patterns for several major email providers are already included in the mtarep.yml.example file.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>removal_links</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">A YAML hash collection of external email provider names and rbl lists (the key) and a corresponding URL (the value) to that provider or rbl's listing info and/or removal form.</font></p><br>
                <p style="color: #D0D0D0; font-size: 140%;"><strong><i>assistance_links</i></strong></p>
                <p style="padding: 6px; color: #CD7814; background-color: #495674; max-width: 720px; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco">A YAML hash collection of any custom documentation you maintain that outlines steps your organization has for resolving a specific mtarep reported issue. These links are used in the modal of a clicked issue in mtarep.</font></p>
              </div>
              <div class="tab-pane" id="tab6">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>Mtarep is currently configured to check each monitored MTA ip address against the following RBL lists:</strong></p><br>
                <% rbl_list = @redis.lrange('rbls', 0, -1) %>
                <p style="padding: 6px; color: #CD7814; max-width: 220px; background-color: #495674; border: #9B9B9B 4px solid; border-radius:10px;"><font face="monaco"><%= rbl_list.join('<br>') %></font></p>
              </div>
              <div class="tab-pane" id="tab7">
                <p style="color: #D0D0D0; font-size: 120%;"><strong>The possible values in the Brightmail reputation service column are:</strong>
                <ul>
                  <li style="color: #D0D0D0; font-size: 104%;"><strong><span style="color: #90EE90;">Good:</span></strong>&nbsp;&nbsp;<strong>Brightmail has detected that only legitimate email has originated from the address</strong></li>
                  <li style="color: #D0D0D0; font-size: 104%;"><strong><span style="color: #9E9E9E;">Neutral:</span></strong>&nbsp;&nbsp;<strong>Brightmail data does not indicate a positive or negative reputation</strong></li>
                  <li style="color: #D0D0D0; font-size: 104%;"><strong><span style="color: #FF3C3C;">Bad:</span></strong>&nbsp;&nbsp;<strong>Brightmail has detected suspicious activity, such as spam or malicious code, originating from the address</strong></li>
                </ul>
              </div>
            </div>
          </div>
        </div>
        <div class="span1"></div>
      </div>
    </div>

  </body>
</html>

{'fraction_non_alphanumeric': 0.13619856374409525, 'fraction_numerical': 0.035824481882969716, 'mean_word_length': 3.9153133807542613, 'pattern_counts': {'":': 0, '<': 865, '<?xml version=': 0, '>': 875, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 2, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '177546', 'n_tokens_mistral': 11985, 'n_tokens_neox': 10874, 'n_words': 3844}