<commit_before><html>
<head>
 <style type="text/css">
/* <![CDATA[ */
@import "http://www.tigris.org/branding/css/tigris.css";
@import "http://www.tigris.org/branding/css/inst.css";
/*  ]]> */
 </style>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>JSaPar Introduction</title>
</head>
<body>
<h2>Java Schema Parser</h2>
The javadoc within the package contains more comprehensive documentation regarding the classes mentioned below. <br/><br/>
The JSaPar package is a java library that provides a parser for flat and CSV (Comma Separated Values) files. 
The concept is that a schema class denotes the way a file should be parsed or written. The schema class can be built by specifying a xml-document or it can be constructed programmatically by using java code.
The output of the parser is usually a org.jsapar.Document object that contains a list of org.jsapar.Line objects which contains a list of org.jsapar.Cell objects. <br/> <br/>
Supported file formats:
<ul>
<li><b>Fixed width </b><i>- Also refered to as flat file. Each cell is described only by its positions within the line. The type of the line is denoted by its position within the file.</i></li>
<li><b>Fixed width contol value </b><i>- The same as Fixed width above except that each line type is denoted by a control value in the leading characters of each line.</i></li>
<li><b>CSV </b><i>- (Comma Separated Values) Each cell is limited by a separator character (or characters). The type of the line is denoted by its position within the file.</i></li>
<li><b>CSV contol value </b><i>- The same as CSV above except that each line type is denoted by a control value in the leading cell of each line.</i></li>
</ul>
 
<h2>Events for each line</h2>
For very large files there can be a problem to build the complete org.jsapar.Document in the memory before further processing. 
It may simply take up to much memory. 
In that case you may choose to get an event for each line that is parsed instead. 
You do that by registering a sub-class of org.jsapar.ParsingEventListener to the org.jsapar.input.Parser. 
That way you can process one line at a time, thus freeing memory as you go along.   
<h2>Converter</h2>
If you are only interesting in converting a file of one format into another, you can use the org.jsapar.io.Converter where you specify the input and the output schema for the conversion. 
The converter uses the event mechanism under the hood, thus it reads, converts and writes one line at a time. 
This means it is very lean regarding memory usage. 
<h2>Building java objects</h2>
Use the method org.jsapar.Parser.buildJava() in order to build java objects for each line in a file (or input). 
Note that in order to be able to use this feature, the schema have to be carefully written. 
For instance, the line type (name) of the line within the schema have to contain the complete class name of the java class to build for each line. 
<h2>Converting java objects into a file</h2>
Use the class org.jsapar.input.JavaBuilder in order to convert java objects into a org.jsapar.Document, which can be used to produce the output file according to a schema.
<h2>Using xml as input</h2>
It is possilbe to build a org.jsapar.Document by using a xml document according to the XMLDocumentFormat.xsl (http://jsapar.tigris.org/XMLDocumentFormat/1.0). 
Use the class org.jsapar.input.XmlDocumentParser in order to convert a xml document into a org.jsapar.Document.  
<h2>Examples</h2>
The files for the examples below are provided in the <tt>samples</tt> folder of the project. The JUnit test <tt>org.jsapar.JSaParExamplesTest.java</tt> 
contains a more comprehensive set of examples of how to use the package. 
<p>
Example of reading <b>CSV file</b> into a Document object according to an xml-schema:
<table style="font-family: monospace" border="1" >
	<tr>
		<td><tt>
        Reader schemaReader = new FileReader("samples/01_CsvSchema.xml");<br/>
        Xml2SchemaBuilder xmlBuilder = new Xml2SchemaBuilder();<br/>
        Reader fileReader = new FileReader("samples/01_Names.csv");<br/>
        Parser parser = new Parser(xmlBuilder.build(schemaReader));<br/>
        Document document = parser.build(fileReader);<br/>
        fileReader.close();<br/>
		</tt></td>
	</tr>
</table>
</p>

<p>
Example of converting a <b>Fixed width file</b> into a <b>CSV file</b> according to two xml-schemas:
<table style="font-family: monospace" border="1" >
	<tr>
		<td><tt>
        File outFile = new File("samples/02_FixedWidthSchema.xml");<br/>
        Reader inSchemaReader = new FileReader("samples/01_CsvSchema.xml");<br/>
        Reader outSchemaReader = new FileReader(outFile);<br/>
        Xml2SchemaBuilder xmlBuilder = new Xml2SchemaBuilder();<br/>
        Reader inReader = new FileReader("samples/01_Names.csv");<br/>
        Writer outWriter = new FileWriter("samples/02_Names_out.txt");<br/>
        Converter converter = new Converter(xmlBuilder.build(inSchemaReader), xmlBuilder.build(outSchemaReader));<br/>
        converter.convert(inReader, outWriter);<br/>
        inReader.close();<br/>
        outWriter.close();<br/>
		</tt></td>
	</tr>
</table>
</p>

<p>
Example of converting a <b>CSV file</b> into a list of <b>Java objects</b> according to an xml-schema:
<table style="font-family: monospace" border="1" >
	<tr>
		<td><tt>
        Reader schemaReader = new FileReader("samples/07_CsvSchemaToJava.xml");<br/>
        Xml2SchemaBuilder xmlBuilder = new Xml2SchemaBuilder();<br/>
        Reader fileReader = new FileReader("samples/07_Names.csv");<br/>
        Parser parser = new Parser(xmlBuilder.build(schemaReader));<br/>
        List<CellParseError> parseErrors = new LinkedList<CellParseError>()<br/>
        List<TestPerson> people = parser.buildJava(fileReader, parseErrors);<br/>
        fileReader.close();<br/>
		</tt></td>
	</tr>
</table>
</p>


<commit_msg>Updated introduction<commit_after><html>
<head>
 <style type="text/css">
/* <![CDATA[ */
@import "http://www.tigris.org/branding/css/tigris.css";
@import "http://www.tigris.org/branding/css/inst.css";
/*  ]]> */
 </style>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>JSaPar Introduction</title>
</head>
<body>
<h2>Java Schema Parser</h2>
The javadoc within the package contains more comprehensive documentation regarding the classes mentioned below. <br/><br/>
The JSaPar package is a java library that provides a parser for flat and CSV (Comma Separated Values) files. 
The concept is that a schema class denotes the way a file should be parsed or written. The schema class can be built by specifying a xml-document or it can be constructed programmatically by using java code.
The output of the parser is usually a org.jsapar.Document object that contains a list of org.jsapar.Line objects which contains a list of org.jsapar.Cell objects. <br/> <br/>
Supported file formats:
<ul>
<li><b>Fixed width </b><i>- Also refered to as flat file. Each cell is described only by its positions within the line. The type of the line is denoted by its position within the file.</i></li>
<li><b>Fixed width contol value </b><i>- The same as Fixed width above except that each line type is denoted by a control value in the leading characters of each line.</i></li>
<li><b>CSV </b><i>- (Comma Separated Values) Each cell is limited by a separator character (or characters). The type of the line is denoted by its position within the file.</i></li>
<li><b>CSV contol value </b><i>- The same as CSV above except that each line type is denoted by a control value in the leading cell of each line.</i></li>
</ul>
 
<h2>Events for each line</h2>
For very large files there can be a problem to build the complete org.jsapar.Document in the memory before further processing. 
It may simply take up to much memory. 
In that case you may choose to get an event for each line that is parsed instead. 
You do that by registering a sub-class of org.jsapar.ParsingEventListener to the org.jsapar.input.Parser. 
That way you can process one line at a time, thus freeing memory as you go along.   
<h2>Converter</h2>
If you are only interesting in converting a file of one format into another, you can use the org.jsapar.io.Converter where you specify the input and the output schema for the conversion. 
The converter uses the event mechanism under the hood, thus it reads, converts and writes one line at a time. 
This means it is very lean regarding memory usage. 
<h2>Building java objects</h2>
Use the method org.jsapar.Parser.buildJava() in order to build java objects for each line in a file (or input). 
Note that in order to be able to use this feature, the schema have to be carefully written. 
For instance, the line type (name) of the line within the schema have to contain the complete class name of the java class to build for each line. 
<h2>Converting java objects into a file</h2>
Use the class org.jsapar.input.JavaBuilder in order to convert java objects into a org.jsapar.Document, which can be used to produce the output file according to a schema.
<h2>Using xml as input</h2>
It is possilbe to build a org.jsapar.Document by using a xml document according to the XMLDocumentFormat.xsl (http://jsapar.tigris.org/XMLDocumentFormat/1.0). 
Use the class org.jsapar.input.XmlDocumentParser in order to convert a xml document into a org.jsapar.Document.  
<h2>Examples</h2>
The files for the examples below are provided in the <tt>samples</tt> folder of the project. The JUnit test <tt>org.jsapar.JSaParExamplesTest.java</tt> 
contains a more comprehensive set of examples of how to use the package. 
<p>
Example of reading <b>CSV file</b> into a Document object according to an xml-schema:
<table style="font-family: monospace" border="1" >
	<tr>
		<td><tt>
        Reader schemaReader = new FileReader("samples/01_CsvSchema.xml");<br/>
        Xml2SchemaBuilder xmlBuilder = new Xml2SchemaBuilder();<br/>
        Reader fileReader = new FileReader("samples/01_Names.csv");<br/>
        Parser parser = new Parser(xmlBuilder.build(schemaReader));<br/>
        Document document = parser.build(fileReader);<br/>
        fileReader.close();<br/>
		</tt></td>
	</tr>
</table>
</p>

<p>
Example of converting a <b>Fixed width file</b> into a <b>CSV file</b> according to two xml-schemas:
<table style="font-family: monospace" border="1" >
	<tr>
		<td><tt>
        File outFile = new File("samples/02_FixedWidthSchema.xml");<br/>
        Reader inSchemaReader = new FileReader("samples/01_CsvSchema.xml");<br/>
        Reader outSchemaReader = new FileReader(outFile);<br/>
        Xml2SchemaBuilder xmlBuilder = new Xml2SchemaBuilder();<br/>
        Reader inReader = new FileReader("samples/01_Names.csv");<br/>
        Writer outWriter = new FileWriter("samples/02_Names_out.txt");<br/>
        Converter converter = new Converter(xmlBuilder.build(inSchemaReader), xmlBuilder.build(outSchemaReader));<br/>
        converter.convert(inReader, outWriter);<br/>
        inReader.close();<br/>
        outWriter.close();<br/>
		</tt></td>
	</tr>
</table>
</p>

<p>
Example of converting a <b>CSV file</b> into a list of <b>Java objects</b> according to an xml-schema:
<table style="font-family: monospace" border="1" >
	<tr>
		<td><tt>
        Reader schemaReader = new FileReader("samples/07_CsvSchemaToJava.xml");<br/>
        Xml2SchemaBuilder xmlBuilder = new Xml2SchemaBuilder();<br/>
        Reader fileReader = new FileReader("samples/07_Names.csv");<br/>
        Parser parser = new Parser(xmlBuilder.build(schemaReader));<br/>
        List<CellParseError> parseErrors = new LinkedList<CellParseError>()<br/>
        List<TestPerson> people = parser.buildJava(fileReader, parseErrors);<br/>
        fileReader.close();<br/>
		</tt></td>
	</tr>
</table>
If you want to run this example, you will need the class org.jsapar.TstPerson within your classpath. 
The class is not included in the jar file or in the binary package but it can be found in the source package. 
As an alternative you can create your own TstPerson class and modify the schema 07_CsvSchemaToJava.xml to use that class instead. 
The class should contain a default constructor plus getters and setters for all the attributes used in the schema. 
</p>



{'fraction_non_alphanumeric': 0.10664921894168643, 'fraction_numerical': 0.007687903819416047, 'mean_word_length': 4.0549813972716, 'pattern_counts': {'":': 0, '<': 251, '<?xml version=': 0, '>': 251, 'https://': 0, 'lorem ipsum': 0, 'www.': 4, 'xml': 37}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '622027', 'n_tokens_mistral': 3716, 'n_tokens_neox': 3566, 'n_words': 1481}