<commit_before><html>
<head>
<title>Gigablast Administrator Documentaion</title>
<meta http-equiv="Content-Type" content="text/html;charset=utf8" />
</head>
<body text=#000000 bgcolor=#ffffff link=#000000 vlink=#000000 alink=#000000 >
<style>body,td,p,.h{font-family:arial,sans-serif; font-size: 15px;} </style>


<center>

<img border=0 width=500 height=122 src=/logo-med.jpg>
<br><br>
</center>

<h1>Gigablast Administrator Documentation</h1>
Developer documentation is <a href=/developer.html>here</a>.
<br><br>
A work-in-progress <a href=/compare.html>comparison to SOLR</a>.

<br>
<br>
<h1>Table of Contents</h1>
<br>
<a href=#quickstart>Quick Start</a><br><br>
<a href=#src>Build from Source</a><br><br>
<a href=#features>Features</a><br><br>
<a href=/searchfeed.html>XML/REST Search Feed API</a><br><br>


<!--<a href=#weighting>Weighting Query Terms</a> - how to pass in your own query term weights<br><br>-->

<a href=#requirements>Hardware Requirements</a> - what is required to run gigablast
<br>
<br>

<a href=#perf>Performance Specifications</a> - various statistics.
<br>
<br>

<a href=#scaling>Scaling the Cluster</a> - how to add more gb instances.
<br>
<br>


<!--<a href=#files>List of Files</a> - the necessary files to run Gigablast
<br>
<br>-->
<a href=#cmdline>Command Line Options</a> - various command line options (coming soon)<br><br>

<!--
<a href=#clustermaint>Cluster Maintenance</a> - running Gigablast on a cluster of computers.<br><br><a href=#trouble>Troubleshooting</a> - how to fix problems
-->

<!--<br><br><a href=#disaster>Disaster Recovery</a> - dealing with a crashed host-->
<!--<br><br>
<a href=#security>The Security System</a> - how to control access-->
<br><br>

<!--<a href=#build>Building an Index</a> - how to start building your index<br><br>
<a href=#spider>The Spider</a> - all about Gigabot, Gigablast's crawling agent<br><br>-->

<!--<a href=#quotas>Document Quotas</a> - how to limit documents into the index<br><br>-->
<a href=#injecting>Injecting Documents</a> - inserting documents directly into Gigablast<br><br><a href=#deleting>Deleting Documents</a> - removing documents from the index

<br><br><a href=#metas>Indexing User-Defined Meta Tags</a> - how Gigablast indexes user-defined meta tags

<!--<br><br><a href=#bigdocs>Indexing Big Documents</a> - what controls the maximum size of a document that can be indexed?-->

<!--<br><br><a href=#rolling>Rolling the New Index</a> - merging the realtime files into the base file-->
<br><br>
<a href=#dmoz>Building a DMOZ Based Directory</a> - build a web directory based on open DMOZ data

<br><br>
<a href=#logs>The Log System</a> - how Gigablast logs information

<br><br>
<a href=#optimizing>Optimizing</a> - optimizing Gigablast's spider and query performance
<!--
<br><br>
<a href=#config>gb.conf</a> - describes the gb configuration file
<br><br>
<a href=#hosts>hosts.conf</a> - the file that describes all participating hosts in the network
<br><br>

<a href=#stopwords>Stopwords</a> - list of common words generally ignored at query time<br><br>
<a href=#phrasebreaks>Phrase Breaks</a> - list of punctuation that breaks a phrase<br><br>
-->

<br><br><a name=quickstart></a>
<h1>Quick Start</h1>
&lt;<i>Last Updated May 2014</i>&gt;
<br>
<br>

<!--Until I get the binary packages ready, <a href=#src>build from the source code</a>, it should only take about 30 seconds to type the three commands.-->

Requirements: You will need an Intel or AMD system running Linux and at least 4GB of RAM to run one instance/host of gb.<br><br>

Install the <a href=http://www.gigablast.com/gb-1.0-0.i686.rpm>Gigablast package for <b>RedHat</b></a> or the <a href=http://www.gigablast.com/gb_1.0-1_amd64.deb>Gigablast package for <b>Ubuntu</b>/Debian</a>.

<br><br>
Type <b>gb -d</b> (as root) to run it in the background as a daemon.

<br><br>
Once installed visit your <a href=http://127.0.0.1:8000/>local port 8000</a> to access the search engine controls and begin configuration. It could take up to 20 seconds to start the search engine for the first time. 

<br><br>

<table><tr><td colspan=2><b>Installed Files</b></td></tr>
<tr><td><nobr>/var/gigablast/data0/</nobr></td><td>Directory of Gigablast binary and data files</td></tr>
<tr><td>/etc/init.d/gb</td><td>start up script link</td></tr>
<tr><td>/etc/init/gb.conf</td><td>Ubuntu upstart conf file so you can type 'start gb' or 'stop gb', but that will only work on local instances of gb.</td></tr>
<tr><td>/usr/bin/gb</td><td>Link to /var/gigablast/data0/gb</td></tr>
</table>


<br><br>
If you run into an bugs let me know so i can fix them right away: <EMAIL>.


<br>
<br>
<a name=src></a>
<h1>Build From Source</h1>
&lt;<i>Last Updated May 2014</i>&gt;
<br>
<br>

Requirements: You will need an Intel or AMD system running Linux and at least 4GB of RAM.<br><br>

If you run into an bugs let me know so i can fix them right away: <EMAIL>.
<br><br>
<!--
You will need the following packages installed<br>
<ul>
<li>For Ubuntu do a <b>apt-get install make g++ gcc-multilib</b>
<br>
For RedHat do a <b>yum install gcc-c++</b>
-->
<!--<li>apt-get install g++
<li>apt-get install gcc-multilib <i>(for 32-bit compilation support)</i>
-->
<!--<li>apt-get install libssl-dev <i>(for the includes, 32-bit libs are here)</i>-->
<!--<li>apt-get install libplot-dev <i>(for the includes, 32-bit libs are here)</i>-->
<!--<li>apt-get install lib32stdc++6-->
<!--<li>apt-get install ia32-libs-->
<!--<li>I supply libstdc++.a but you might need the include headers and have to do <b>apt-get install lib32stdc++6</b> or something.
</ul>
-->

<b>1.</b> For <u>Ubuntu</u> do <b>apt-get install make g++ gcc-multilib lib32stdc++6</b>
<br>
<b>2.</b> For <u>RedHat</u> do <b>yum install gcc-c++</b>
<br>
<b>3.</b> Download the <a href=https://github.com/gigablast/open-source-search-engine>Gigablast source code</a> using <b>wget --no-check-certificate "https://github.com/gigablast/open-source-search-engine/archive/master.zip"</b>, unzip it and cd into it. (optionally use <b>git clone https://github.com/gigablast/open-source-search-engine.git ./github</b> if you have <i>git</i> installed.)
<br>
<b>4.</b> Run <b>make</b> to compile. (e.g. use 'make -j 4' to compile on four cores)
<br>
<b>5.</b> Run <b>./gb -d</b>  to start a single gigablast node which listens on port 8000 running in daemon mode (-d).
<br>
<b>6.</b> The first time you run gb, wait about 30 seconds for it to build some files. Check the log file to see when it completes.
<br><commit_msg>updated email.<commit_after><html>
<head>
<title>Gigablast Administrator Documentaion</title>
<meta http-equiv="Content-Type" content="text/html;charset=utf8" />
</head>
<body text=#000000 bgcolor=#ffffff link=#000000 vlink=#000000 alink=#000000 >
<style>body,td,p,.h{font-family:arial,sans-serif; font-size: 15px;} </style>


<center>

<img border=0 width=500 height=122 src=/logo-med.jpg>
<br><br>
</center>

<h1>Gigablast Administrator Documentation</h1>
Developer documentation is <a href=/developer.html>here</a>.
<br><br>
A work-in-progress <a href=/compare.html>comparison to SOLR</a>.

<br>
<br>
<h1>Table of Contents</h1>
<br>
<a href=#quickstart>Quick Start</a><br><br>
<a href=#src>Build from Source</a><br><br>
<a href=#features>Features</a><br><br>
<a href=/searchfeed.html>XML/REST Search Feed API</a><br><br>


<!--<a href=#weighting>Weighting Query Terms</a> - how to pass in your own query term weights<br><br>-->

<a href=#requirements>Hardware Requirements</a> - what is required to run gigablast
<br>
<br>

<a href=#perf>Performance Specifications</a> - various statistics.
<br>
<br>

<a href=#scaling>Scaling the Cluster</a> - how to add more gb instances.
<br>
<br>


<!--<a href=#files>List of Files</a> - the necessary files to run Gigablast
<br>
<br>-->
<a href=#cmdline>Command Line Options</a> - various command line options (coming soon)<br><br>

<!--
<a href=#clustermaint>Cluster Maintenance</a> - running Gigablast on a cluster of computers.<br><br><a href=#trouble>Troubleshooting</a> - how to fix problems
-->

<!--<br><br><a href=#disaster>Disaster Recovery</a> - dealing with a crashed host-->
<!--<br><br>
<a href=#security>The Security System</a> - how to control access-->
<br><br>

<!--<a href=#build>Building an Index</a> - how to start building your index<br><br>
<a href=#spider>The Spider</a> - all about Gigabot, Gigablast's crawling agent<br><br>-->

<!--<a href=#quotas>Document Quotas</a> - how to limit documents into the index<br><br>-->
<a href=#injecting>Injecting Documents</a> - inserting documents directly into Gigablast<br><br><a href=#deleting>Deleting Documents</a> - removing documents from the index

<br><br><a href=#metas>Indexing User-Defined Meta Tags</a> - how Gigablast indexes user-defined meta tags

<!--<br><br><a href=#bigdocs>Indexing Big Documents</a> - what controls the maximum size of a document that can be indexed?-->

<!--<br><br><a href=#rolling>Rolling the New Index</a> - merging the realtime files into the base file-->
<br><br>
<a href=#dmoz>Building a DMOZ Based Directory</a> - build a web directory based on open DMOZ data

<br><br>
<a href=#logs>The Log System</a> - how Gigablast logs information

<br><br>
<a href=#optimizing>Optimizing</a> - optimizing Gigablast's spider and query performance
<!--
<br><br>
<a href=#config>gb.conf</a> - describes the gb configuration file
<br><br>
<a href=#hosts>hosts.conf</a> - the file that describes all participating hosts in the network
<br><br>

<a href=#stopwords>Stopwords</a> - list of common words generally ignored at query time<br><br>
<a href=#phrasebreaks>Phrase Breaks</a> - list of punctuation that breaks a phrase<br><br>
-->

<br><br><a name=quickstart></a>
<h1>Quick Start</h1>
&lt;<i>Last Updated May 2014</i>&gt;
<br>
<br>

<!--Until I get the binary packages ready, <a href=#src>build from the source code</a>, it should only take about 30 seconds to type the three commands.-->

Requirements: You will need an Intel or AMD system running Linux and at least 4GB of RAM to run one instance/host of gb.<br><br>

Install the <a href=http://www.gigablast.com/gb-1.0-0.i686.rpm>Gigablast package for <b>RedHat</b></a> or the <a href=http://www.gigablast.com/gb_1.0-1_amd64.deb>Gigablast package for <b>Ubuntu</b>/Debian</a>.

<br><br>
Type <b>gb -d</b> (as root) to run it in the background as a daemon.

<br><br>
Once installed visit your <a href=http://127.0.0.1:8000/>local port 8000</a> to access the search engine controls and begin configuration. It could take up to 20 seconds to start the search engine for the first time. 

<br><br>

<table><tr><td colspan=2><b>Installed Files</b></td></tr>
<tr><td><nobr>/var/gigablast/data0/</nobr></td><td>Directory of Gigablast binary and data files</td></tr>
<tr><td>/etc/init.d/gb</td><td>start up script link</td></tr>
<tr><td>/etc/init/gb.conf</td><td>Ubuntu upstart conf file so you can type 'start gb' or 'stop gb', but that will only work on local instances of gb.</td></tr>
<tr><td>/usr/bin/gb</td><td>Link to /var/gigablast/data0/gb</td></tr>
</table>


<br><br>
If you run into an bugs let me know so i can fix them right away: <EMAIL>.


<br>
<br>
<a name=src></a>
<h1>Build From Source</h1>
&lt;<i>Last Updated May 2014</i>&gt;
<br>
<br>

Requirements: You will need an Intel or AMD system running Linux and at least 4GB of RAM.<br><br>

If you run into an bugs let me know so i can fix them right away: <EMAIL>.
<br><br>
<!--
You will need the following packages installed<br>
<ul>
<li>For Ubuntu do a <b>apt-get install make g++ gcc-multilib</b>
<br>
For RedHat do a <b>yum install gcc-c++</b>
-->
<!--<li>apt-get install g++
<li>apt-get install gcc-multilib <i>(for 32-bit compilation support)</i>
-->
<!--<li>apt-get install libssl-dev <i>(for the includes, 32-bit libs are here)</i>-->
<!--<li>apt-get install libplot-dev <i>(for the includes, 32-bit libs are here)</i>-->
<!--<li>apt-get install lib32stdc++6-->
<!--<li>apt-get install ia32-libs-->
<!--<li>I supply libstdc++.a but you might need the include headers and have to do <b>apt-get install lib32stdc++6</b> or something.
</ul>
-->

<b>1.</b> For <u>Ubuntu</u> do <b>apt-get install make g++ gcc-multilib lib32stdc++6</b>
<br>
<b>2.</b> For <u>RedHat</u> do <b>yum install gcc-c++</b>
<br>
<b>3.</b> Download the <a href=https://github.com/gigablast/open-source-search-engine>Gigablast source code</a> using <b>wget --no-check-certificate "https://github.com/gigablast/open-source-search-engine/archive/master.zip"</b>, unzip it and cd into it. (optionally use <b>git clone https://github.com/gigablast/open-source-search-engine.git ./github</b> if you have <i>git</i> installed.)
<br>
<b>4.</b> Run <b>make</b> to compile. (e.g. use 'make -j 4' to compile on four cores)
<br>
<b>5.</b> Run <b>./gb -d</b>  to start a single gigablast node which listens on port 8000 running in daemon mode (-d).
<br>
<b>6.</b> The first time you run gb, wait about 30 seconds for it to build some files. Check the log file to see when it completes.
<br>
{'fraction_non_alphanumeric': 0.1774925051887155, 'fraction_numerical': 0.017526327926819893, 'mean_word_length': 4.586088449978532, 'pattern_counts': {'":': 0, '<': 617, '<?xml version=': 0, '>': 617, 'https://': 6, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3342144', 'n_tokens_mistral': 4989, 'n_tokens_neox': 4608, 'n_words': 1486}