<commit_before>INSTALL.txt

This file provides instructions on how to install the whips
program on your local *nix machine.  There are currently no plans to
support a Windows installation.

PREREQUISITES
=============

WHIPS depends on the following 3rd party pieces of software

1) python (version 2.5.x or newer, but NOT version 3)

   The software used here was all written for python version 2.5.x. or later
   Though there is a version 3 of python, it is not backwards
   compatible and should not be used.

   Python may already be installed on your system.  Try executing

   	  which python

   to see if this is the case.  If you have it, try

      	  python --version

   and if the version number is greater than 2.5 you can skip this step.

   If you do have to install python, or simply want a fresh copy, then
   do the following:

   1.1: Obtain the source code from the official python website
      http://python.org/download/releases/

   1.2: Untar the source
   	  
	  tar -xzvf python-2.x.y.tar.gz

   1.3: Set any desired compilation environmental variables.  These
      include all variables commonly recognized by autoconf scripts,
      including:
	  
	CC for the C compiler
	CFLAGS for the C compiler flags
	CXX for the C++ compiler
	CXXFLAGS for the C++ compiler flags
	and many others

      If you have no idea what any of this means don't worry about it,
      it's more than likely safe to ignore it.

   1.4: Change to the unpacked directory and run the configuration
      script.  The script accepts numerous arguments, the most
      important among them the --prefix argument that sets the base
      directory for the install.

      	  cd python-2.x.y
	  ./configure --prefix=/path/to/base/directory

   1.5: Make the program and install it to the base directory

   	  make install

   1.6: Confirm that your default python is now the one you just
      installed.  Run the following commands to regenerate the list of
      commands as if you'd just logged in, then look at the full path
      to the default copy of python.

          rehash
	  which python
 
      This should print out

          /path/to/base/directory/bin/python

      If it does not, you need to adjust your path in your shrc file
      (~/.tcshrc, ~/.cshrc, ~/.bashrc, depending on your shell) to
      point to the new copy of python.  Do NOT attempt to use an alias
      to point to the correct copy of python as this can cause serious
      errors with python's installation tools.
      
2) Install HDF4

   The software was orignally written with HDF 4.2.6.  It should be
   backwards compatible beyond that but no attempts have been made to
   determine how far back it goes.

   HDF4 software is necessary to read in the MOPITT data.

   The following instructions detail how to install from source.
   You're welcome to try to install the binaries available.

   2.1: Download the HDF4 source code from
      http://wwww.hdfgroup.org/ftp/HDF/HDF_Current/src/

   2.2: Untar the source code

   	  tar -xzvf hdf-4.2.6.tar.gz

   2.3: Install JPEG.  The HDF4 installation depends on the JPEG
      libraries, so we'll need to install those before we can proceed
      with installing HDF4.

      2.3.1: Download the JPEG (v66 or newer) source code.  A link is
         provided at http://www.hdfgroup.org/release4/obtain.html

      2.3.2: Untar JPEG

      	     tar -xzvf jpegsrc.v66.tar.gz

      2.3.3: Move to the untarred jpeg directory and run the configure
         script.  Again, make sure to set the --prefix flag to the
         base directory of the install

	     cd jpeg-6b
	     ./configure --prefix=/path/to/base/directory

      2.3.4: Build the libraries

      	     make

      2.3.5: Install the libraries to the base directory.  Note that
         unlike most well-behaved distributions, this actually has a
         chance of failing.  If it complains about a file called
         'cjpeg.1', try explicitly creating the filetree in which it
         resides, then re-running the install.

	     make install

   2.4: Install zlib.  The HDF4 installation depends on this library
      as well.  We build it from source, though binaries may work
      better for your system.

      2.4.1: Obtain zlib.  Note that though it is linked from the HDF4
         site, the link was broken as of the last time I checked.  The
         source code can be obtained directly from http://zlib.net

      2.4.2: Untar the zlib source distribution
     
	     tar -xzvf zlib-1.2.6.tar.gz

      2.4.3: Change to folder and configure the zlib install.  As
         always, set the base directory using the --prefix flag

	     cd zlib-1.2.6
	     ./configure --prefix=/path/to/base/directory

      2.4.4: Test the installation if desired

      	     make test

      2.4.5: If the tests pass, install the libraries to the base
         directory.

	     make install

   2.5: Set some environmental variables that will be needed during
      the install.  Note that the F77 variable might not be necessary, but
      CFLAGS does need to include the -fPIC flag

      	     setenv F77 gfortran
	     setenv CFLAGS -fPIC $CFLAGS

   2.6: Configure HDF for installation.  Note that unlike most
      configurations, this step requires more than just the --prefix
      flag.  Make sure to give the correct path the base directory
      under which the zlib and jpeg installs (not the source code!)
      live.  The --disable-netcdf is also critical, but may prevent
      you from using this HDF build for other purposes.

      	     cd hdf-4.2.6
             ./configure --with-zlib=/path/to/base/directory \
	     		 --with-jpeg=/path/to/base/directory \
			 --disable-netcdf \<commit_msg>Added to HDF4 installation documentation<commit_after>INSTALL.txt

This file provides instructions on how to install the whips
program on your local *nix machine.  There are currently no plans to
support a Windows installation.

PREREQUISITES
=============

WHIPS depends on the following 3rd party pieces of software

1) python (version 2.5.x or newer, but NOT version 3)

   The software used here was all written for python version 2.5.x. or later
   Though there is a version 3 of python, it is not backwards
   compatible and should not be used.

   Python may already be installed on your system.  Try executing

   	  which python

   to see if this is the case.  If you have it, try

      	  python --version

   and if the version number is greater than 2.5 you can skip this step.

   If you do have to install python, or simply want a fresh copy, then
   do the following:

   1.1: Obtain the source code from the official python website
      http://python.org/download/releases/

   1.2: Untar the source
   	  
	  tar -xzvf python-2.x.y.tar.gz

   1.3: Set any desired compilation environmental variables.  These
      include all variables commonly recognized by autoconf scripts,
      including:
	  
	CC for the C compiler
	CFLAGS for the C compiler flags
	CXX for the C++ compiler
	CXXFLAGS for the C++ compiler flags
	and many others

      If you have no idea what any of this means don't worry about it,
      it's more than likely safe to ignore it.

   1.4: Change to the unpacked directory and run the configuration
      script.  The script accepts numerous arguments, the most
      important among them the --prefix argument that sets the base
      directory for the install.

      	  cd python-2.x.y
	  ./configure --prefix=/path/to/base/directory

   1.5: Make the program and install it to the base directory

   	  make install

   1.6: Confirm that your default python is now the one you just
      installed.  Run the following commands to regenerate the list of
      commands as if you'd just logged in, then look at the full path
      to the default copy of python.

          rehash
	  which python
 
      This should print out

          /path/to/base/directory/bin/python

      If it does not, you need to adjust your path in your shrc file
      (~/.tcshrc, ~/.cshrc, ~/.bashrc, depending on your shell) to
      point to the new copy of python.  Do NOT attempt to use an alias
      to point to the correct copy of python as this can cause serious
      errors with python's installation tools.
      
2) Install HDF4

   The software was orignally written with HDF 4.2.6.  It should be
   backwards compatible beyond that but no attempts have been made to
   determine how far back it goes.

   HDF4 software is necessary to read in the MOPITT data.

   The following instructions detail how to install from source.
   You're welcome to try to install the binaries available.

   2.1: Download the HDF4 source code from
      http://wwww.hdfgroup.org/ftp/HDF/HDF_Current/src/

   2.2: Untar the source code

   	  tar -xzvf hdf-4.2.6.tar.gz

   2.3: Install JPEG.  The HDF4 installation depends on the JPEG
      libraries, so we'll need to install those before we can proceed
      with installing HDF4.

      2.3.1: Download the JPEG (v66 or newer) source code.  A link is
         provided at http://www.hdfgroup.org/release4/obtain.html

      2.3.2: Untar JPEG

      	     tar -xzvf jpegsrc.v66.tar.gz

      2.3.3: Move to the untarred jpeg directory and run the configure
         script.  Again, make sure to set the --prefix flag to the
         base directory of the install

	     cd jpeg-6b
	     ./configure --prefix=/path/to/base/directory

      2.3.4: Build the libraries

      	     make

      2.3.5: Install the libraries to the base directory.  Note that
         unlike most well-behaved distributions, this actually has a
         chance of failing.  If it complains about a file called
         'cjpeg.1', try explicitly creating the filetree in which it
         resides, then re-running the install.  Note that we have to
	 run two separate commands to actually get the libraries.

	      make install
	      make install-lib

   2.4: Install zlib.  The HDF4 installation depends on this library
      as well.  We build it from source, though binaries may work
      better for your system.

      2.4.1: Obtain zlib.  Note that though it is linked from the HDF4
         site, the link was broken as of the last time I checked.  The
         source code can be obtained directly from http://zlib.net

      2.4.2: Untar the zlib source distribution
     
	     tar -xzvf zlib-1.2.6.tar.gz

      2.4.3: Change to folder and configure the zlib install.  As
         always, set the base directory using the --prefix flag

	     cd zlib-1.2.6
	     ./configure --prefix=/path/to/base/directory

      2.4.4: Test the installation if desired

      	     make test

      2.4.5: If the tests pass, install the libraries to the base
         directory.

	     make install

   2.5: Set some environmental variables that will be needed during
      the install.  Note that the F77 variable might not be necessary, but
      CFLAGS does need to include the -fPIC flag

      	     setenv F77 gfortran
	     setenv CFLAGS -fPIC 

      On 64-bit versions of OSX (and possibly other operating systems),
      the compilers may get confused.  If this is the case, it can be
      rectified by adjusting the above to the following:

      	     setenv F77 gfortran
	     setenv CFLAGS "-fPIC -m64"
	     setenv FFLAGS -m64

   2.6: Configure HDF for installation.  Note that unlike most
      configurations, this step requires more than just the --prefix
      flag.  Make sure to give the correct path the base directory
      under which the zlib and jpeg installs (not the source code!)
      live.  The --disable-netcdf is also critical, but may prevent
      you from using this HDF build for other purposes.

      	     cd hdf-4.2.6
             ./configure --with-zlib=/path/to/base/directory \
	     		 --with-jpeg=/path/to/base/directory \
			 --disable-netcdf \
{'fraction_non_alphanumeric': 0.05402444163506111, 'fraction_numerical': 0.017614833544037083, 'mean_word_length': 2.3415939172064206, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '853534', 'n_tokens_mistral': 3482, 'n_tokens_neox': 3251, 'n_words': 1626}