<commit_before><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta name="Description" content="HIPI - Hadoop Image Processing Interface is a Hadoop MapReduce library for performing image processing tasks in the Hadoop distributed computation framework." />
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
    
    <link rel="stylesheet" type="text/css" href="include/main.css" />
    <link rel="stylesheet" type="text/css" href="include/javasyntax.css" />
    <title>HIPI - Hadoop Image Processing Interface :: Introduction</title>
  </head>
  
  <body>
    <div class="header">
      <h1>HIPI - Hadoop Image Processing Framework</h1>
    </div>
    <div class="navigation_menu">
      <ul>
	<li><a href="index.html">Introduction</a></li>
	<li><a href="gettingstarted.html">Getting Started</a></li>
	<li><a href="documentation.html">Documentation</a></li>
	<li><a href="examples.html">Examples</a></li>
	<li><a href="downloads.html">Downloads</a></li>
	<li><a href="about.html">About</a></li>
      </ul>
    </div>
    
    <!-- Begin Content -->
    <div class="content">
      <h2>Overview</h2>
      <div class="section">
	HIPI is a library for <a class="external_link"
	href="http://hadoop.apache.org/mapreduce">Hadoop's
	MapReduce</a> framework that provides an API for performing
	image processing tasks in a distributed computing
	environment. Here is an overview of our system:
	
	<img class="centered_image" src="images/hipi_pipeline.png" width="700" alt=""/>
	
	The input type used in HIPI is referred to as a <a
	href="">HipiImageBundle</a> (HIB). A HIB is a set of images
	combined into one large file along with some metadata
	describing the layout of the images. A HIB can be created from
	an already existing set of images or directly through some
	other source (e.g. our <a href="">DistributedDownloader</a>). <br /><br />

	In order to improve the efficiency of some jobs, HIPI allows a
	user to specify a culling function that discards images that
	do not meet a specified set of criteria (e.g. the image must
	be less than 10 megapixels). The user-specified <a
	href="">CullMapper</a> class is then invoked on each image
	that passes the culling test. Images are presented to this
	class as a <a href="">FloatImage</a> and an associated <a
	href="">ImageHeader</a>. Although HIPI does not directly
	modify any of the default Hadoop MapReduce behavior once the
	Mapper's take over, a user can modify execution parameters
	specific to image processing tasks through the <a
	href="">HipiJob</a> object during setup.  <br /><br />

	Great care has been taken to ensure that the implementations
	of the necessary components of Hadoop MapReduce are efficient
	and effective for image processing tasks. Please see our <a
	href="experiments.html">Experiments</a> section for further
	analysis of HIPI.
      </div>
      
      <h2>The Four Main Classes</h2>
      <div class="section">
	The classes most frequently used by a practitioner are the <a
	href="">HipiImageBundle</a>, the <a href="">FloatImage</a>,
	and the <a href="">CullMapper</a> classes. These classes
	provide most of the functionality an average user will need
	and are described briefly below. The full API for HIPI can be
	found on our <a href="documentation.html">documentation</a>
	page.
	
	<h3>HIPI Image Bundle (HIB)</h3> 
	
	A HIPI Image Bundle (HIB) is a collection of images that are
	stored together in one file, somewhat analagous to .tar files
	in UNIX. HIBs are implemented via the <a
	href="">HipiImageBundle</a> class and can be used directly in
	the Hadoop MapReduce framework. Several common operations can
	be performed on HIBs, including:
	<div class="todo">TODO: Show small code snippets for each of these</div>
	<ul>
	  <li>Create a HIB</li>
	  <li>Add images to an already existing HIB</li>
	  <li>Merge two HIBs</li>
	  <li>Read the images from a HIB (random access is permitted)</li>
	</ul>
	<div class="important">Important Note:</div>
	HIBs must be opened in either FILE_MODE_READ or
	FILE_MODE_WRITE mode (see <a
	href="">AbstractImageBundle::open</a>), and once they have
	been opened in one mode, the mode cannot be switched.  <br
	/><br />

	You can create a HIPI Image Bundle from a set of files using
	the operations listed above, or via some external source
	(e.g. Flickr) via
	our <a href="">DistributedDownloader</a>. Please see
	the <a href="">HIB javadoc page</a> for more details on all of
	the HIB operations, and the <a href="">Experiments</a> page
	for information regarding the design of HIBs.
	
	<h3>Float Image</h3>
	
	The primary input to Map tasks in HIPI is
	the <a href="">FloatImage</a> class. It is a simple
	representation of an image file as a set of pixels specified
	with single floating-point precision. Several common
	operations can be performed on a FloatImage:
	<div class="todo">TODO: Show small code snippets for each of these</div>
	<ul>
	  <li>Get/set pixels values</li>
	  <li>Convert to grayscale</li>
	  <li>Scalar multiplication</li>
	  <li>Addition</li>
	  <li>Image cropping</li>
	</ul>

	A <a href="">FloatImage</a> is actually just a three-dimensional set of
	floating-point values. Thus it can also be treated as a matrix
	(or tensor) and used accordingly. The function
	<a href="">FloatImage::getData</a> converts the FloatImage
	into its array representation and can be used directly with
	the popular BLAS/LAPACK Java implementation
	<a class="external_link"
	href="http://icl.cs.utk.edu/f2j/">f2j</a> to perform standard
	matrix operations. Creating a FloatImage from an array of
	pixels can be performed via a special form of
	the <a href="">FloatImage constructor</a>. <br /><br />

	As mentioned in the <a href="#overview">Overview</a> section,
	HIPI takes as input a <a href="#hib">HIPI Image Bundle</a> and
	sends records (key-value pairs) to user-specified Mapper
	functions with the key being an object of
	type <a href="">ImageHeader</a> and the value being an object
	of type <a href="">FloatImage</a>. For each image contained in
	the HIB that passes the optional <a href="#culling">culling
	step</a> one such record will be generated.
	
	<h3>Cull Mapper</h3>
	
	HIPI provides a way for users to discard, or cull, images that
	do not meet a set of criteria. The CullMapper class defines a
	special type of Hadoop Mapper that contains an extra function
	CullMapper::cull that can be used to test an ImageHeader
	associated with a FloatImage for arbitrary criteria. An
	example of this can be found in our Experiments page under
	Culling Test.<br /><br />

	In the example below, the CullMapper is being used (which
	inherits directly from Mapper). The function CullMapper::cull
	returns a boolean indicating whether the image described by
	ImageHeader should be processed (true) or should be discarded
	(false). In this example, images must contain an EXIF tag
	indicating they were taken with a Canon PowerShot S500 digital
	camera and the images must have dimensions 2592 by 1944:

	<pre id="Classes">
    <font id="Public">public</font> <font id="Static">static</font> <font id="Class">class</font> MyMapper <font id="Extends">extends</font> CullMapper&lt;ImageHeader, FloatImage, NullWritable, FloatImage&gt;
    {       
        <font id="Public">public</font> <font id="Boolean">boolean</font> cull(ImageHeader key) <font id="Throws">throws</font> IOException, InterruptedException {
            <font id="If">if</font>(key.getEXIFInformation(<font id="StringLiteral">"Model"</font>).equals(<font id="StringLiteral">"Canon PowerShot S500"</font>) &amp;&amp; key.width == <font id="IntegerLiteral">2592</font> &amp;&amp; key.height == <font id="IntegerLiteral">1944</font>)
                <font id="Return">return</font> <font id="True">true</font>;
            <font id="Else">else</font>
                <font id="Return">return</font> <font id="False">false</font>;
        }
        
        <font id="Public">public</font> <font id="Void">void</font> map(ImageHeader key, FloatImage value, Context context) <font id="Throws">throws</font> IOException, InterruptedException {
	...
	</pre>
Often when
	dealing with image sets, you want a way of filtering images so
	that only images with certain properties are
	consider. Applications such
	as <a href="http://graphics.cs.cmu.edu/projects/im2gps/">im2gps</a>,
	for instance, only need to consider images with proper GIS
	data in the image headers and exif data. HIPI provides a Cull
	class that provides an interface for filtering the data. This
	class is passed in through the HipiJob class (described
	below). In addition to saving runtime (images that do not pass
	the cull are not decoded into FloatImages), the culler also
	makes writing applications in the Map class much simpler since
	users can assume that images in the Mapper are only the images
	which pass the filter specified by the culler.
	
	<h3>HIPI Job</h3>
	
      </div>
      
      <h2>HIPI Platforms</h2>
      <div class="section">
	HIPI stands for Hadoop Image Processing Interface because it
	is meant to run on Apache Hadoop. Take a look at our <a
	href="gettingstarted.html"> Getting Started</a> page for
	details on how to install and set up Hadoop. Check out the <a
	href="http://hadoop.apache.org/">Apache Hadoop</a> website for
	more details on Hadoop.
      </div>
      
      <h2>Examples</h2>
      <div class="section">
	Examples can be found on the <a
	href="examples.html">examples</a> page. We provide several
	examples, including trivial and non-trivial applications of
	HIPI being used on Hadoop's platform.
      </div><commit_msg>sean: website checkpoint<commit_after><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta name="Description" content="HIPI - Hadoop Image Processing Interface is a Hadoop MapReduce library for performing image processing tasks in the Hadoop distributed computation framework." />
    <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
    
    <link rel="stylesheet" type="text/css" href="include/main.css" />
    <link rel="stylesheet" type="text/css" href="include/javasyntax.css" />
    <title>HIPI - Hadoop Image Processing Interface :: Introduction</title>
  </head>
  
  <body>
    <div class="header">
      <h1>HIPI - Hadoop Image Processing Framework</h1>
    </div>
    <div class="navigation_menu">
      <ul>
	<li><a href="index.html">Introduction</a></li>
	<li><a href="gettingstarted.html">Getting Started</a></li>
	<li><a href="documentation.html">Documentation</a></li>
	<li><a href="examples.html">Examples</a></li>
	<li><a href="downloads.html">Downloads</a></li>
	<li><a href="about.html">About</a></li>
      </ul>
    </div>
    
    <!-- Begin Content -->
    <div class="content">
      <h2>Overview</h2>
      <div class="section">
	HIPI is a library for <a class="external_link"
	href="http://hadoop.apache.org/mapreduce">Hadoop's
	MapReduce</a> framework that provides an API for performing
	image processing tasks in a distributed computing
	environment. Here is an overview of our system:
	
	<img class="centered_image" src="images/hipi_pipeline.png" width="700" alt=""/>
	
	The input type used in HIPI is referred to as a <a
	href="">HipiImageBundle</a> (HIB). A HIB is a set of images
	combined into one large file along with some metadata
	describing the layout of the images. A HIB can be created from
	an already existing set of images or directly through some
	other source (e.g. our <a href="">DistributedDownloader</a>). <br /><br />

	In order to improve the efficiency of some jobs, HIPI allows a
	user to specify a culling function that discards images that
	do not meet a specified set of criteria (e.g. the image must
	be less than 10 megapixels). The user-specified <a
	href="">CullMapper</a> class is then invoked on each image
	that passes the culling test. Images are presented to this
	class as a <a href="">FloatImage</a> and an associated <a
	href="">ImageHeader</a>. Although HIPI does not directly
	modify any of the default Hadoop MapReduce behavior once the
	Mapper's take over, a user can modify execution parameters
	specific to image processing tasks through the <a
	href="">HipiJob</a> object during setup.  <br /><br />

	Great care has been taken to ensure that the implementations
	of the necessary components of Hadoop MapReduce are efficient
	and effective for image processing tasks. Please see our <a
	href="experiments.html">Experiments</a> section for further
	analysis of HIPI.
      </div>
      
      <h2>The Four Main Classes</h2>
      <div class="section">
	The classes most frequently used by a practitioner are the <a
	href="">HipiImageBundle</a>, the <a href="">FloatImage</a>,
	and the <a href="">CullMapper</a> classes. These classes
	provide most of the functionality an average user will need
	and are described briefly below. The full API for HIPI can be
	found on our <a href="documentation.html">documentation</a>
	page.
	
	<h3>HIPI Image Bundle (HIB)</h3> 
	
	A HIPI Image Bundle (HIB) is a collection of images that are
	stored together in one file, somewhat analagous to .tar files
	in UNIX. HIBs are implemented via the <a
	href="">HipiImageBundle</a> class and can be used directly in
	the Hadoop MapReduce framework. Several common operations can
	be performed on HIBs, including:
	<div class="todo">TODO: Show small code snippets for each of these</div>
	<ul>
	  <li>Create a HIB</li>
	  <li>Add images to an already existing HIB</li>
	  <li>Merge two HIBs</li>
	  <li>Read the images from a HIB (random access is permitted)</li>
	</ul>
	<div class="important">Important Note:</div>
	HIBs must be opened in either FILE_MODE_READ or
	FILE_MODE_WRITE mode (see <a
	href="">AbstractImageBundle::open</a>), and once they have
	been opened in one mode, the mode cannot be switched.  <br
	/><br />

	You can create a HIPI Image Bundle from a set of files using
	the operations listed above, or via some external source
	(e.g. Flickr) via
	our <a href="">DistributedDownloader</a>. Please see
	the <a href="">HIB javadoc page</a> for more details on all of
	the HIB operations, and the <a href="">Experiments</a> page
	for information regarding the design of HIBs.
	
	<h3>Float Image</h3>
	
	The primary input to Map tasks in HIPI is
	the <a href="">FloatImage</a> class. It is a simple
	representation of an image file as a set of pixels specified
	with single floating-point precision. Several common
	operations can be performed on a FloatImage:
	<div class="todo">TODO: Show small code snippets for each of these</div>
	<ul>
	  <li>Get/set pixels values</li>
	  <li>Convert to grayscale</li>
	  <li>Scalar multiplication</li>
	  <li>Addition</li>
	  <li>Image cropping</li>
	</ul>

	A <a href="">FloatImage</a> is actually just a three-dimensional set of
	floating-point values. Thus it can also be treated as a matrix
	(or tensor) and used accordingly. The function
	<a href="">FloatImage::getData</a> converts the FloatImage
	into its array representation and can be used directly with
	the popular BLAS/LAPACK Java implementation
	<a class="external_link"
	href="http://icl.cs.utk.edu/f2j/">f2j</a> to perform standard
	matrix operations. Creating a FloatImage from an array of
	pixels can be performed via a special form of
	the <a href="">FloatImage constructor</a>. <br /><br />

	As mentioned in the <a href="#overview">Overview</a> section,
	HIPI takes as input a <a href="#hib">HIPI Image Bundle</a> and
	sends records (key-value pairs) to user-specified Mapper
	functions with the key being an object of
	type <a href="">ImageHeader</a> and the value being an object
	of type <a href="">FloatImage</a>. For each image contained in
	the HIB that passes the optional <a href="#culling">culling
	step</a> one such record will be generated.
	
	<h3>Cull Mapper</h3>
	
	HIPI provides a way for users to discard, or cull, images that
	do not meet a set of criteria. The CullMapper class defines a
	special type of Hadoop Mapper that contains an extra function
	CullMapper::cull that can be used to test an ImageHeader
	associated with a FloatImage for arbitrary
	criteria. <br /><br />

	In the example below, the CullMapper is being used (which
	inherits directly from Mapper) to process images taken with a
	Canon PowerShot S500 digital camera and having dimensions 2592
	by 1944. The function CullMapper::cull returns a boolean
	indicating whether the image described by ImageHeader should
	be processed (true) or should be discarded (false);

	<pre id="Classes">
    <font id="Public">public</font> <font id="Static">static</font> <font id="Class">class</font> MyMapper <font id="Extends">extends</font> CullMapper&lt;ImageHeader, FloatImage, NullWritable, FloatImage&gt;
    {       
        <font id="Public">public</font> <font id="Boolean">boolean</font> cull(ImageHeader key) <font id="Throws">throws</font> IOException, InterruptedException {
            <font id="If">if</font>(key.getEXIFInformation(<font id="StringLiteral">"Model"</font>).equals(<font id="StringLiteral">"Canon PowerShot S500"</font>) &amp;&amp; key.width == <font id="IntegerLiteral">2592</font> &amp;&amp; key.height == <font id="IntegerLiteral">1944</font>)
                <font id="Return">return</font> <font id="True">true</font>;
            <font id="Else">else</font>
                <font id="Return">return</font> <font id="False">false</font>;
        }
        
        <font id="Public">public</font> <font id="Void">void</font> map(ImageHeader key, FloatImage value, Context context) <font id="Throws">throws</font> IOException, InterruptedException {
	...
	</pre>
	
	<h3>HIPI Job</h3>
	
      </div>

      <h2>HIPI Platforms</h2>
      <div class="section">
	HIPI stands for Hadoop Image Processing Interface because it
	is meant to run on Apache Hadoop. Take a look at our <a
	href="gettingstarted.html"> Getting Started</a> page for
	details on how to install and set up Hadoop. Check out the <a
	href="http://hadoop.apache.org/">Apache Hadoop</a> website for
	more details on Hadoop.
      </div>
      
      <h2>Examples</h2>
      <div class="section">
	Examples can be found on the <a
	href="examples.html">examples</a> page. We provide several
	examples, including trivial and non-trivial applications of
	HIPI being used on Hadoop's platform.
      </div>
{'fraction_non_alphanumeric': 0.11345831757523367, 'fraction_numerical': 0.006483332432870495, 'mean_word_length': 4.068455640744797, 'pattern_counts': {'":': 0, '<': 425, '<?xml version=': 0, '>': 425, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3775191', 'n_tokens_mistral': 5917, 'n_tokens_neox': 5548, 'n_words': 2199}