<commit_before><?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <title>CS161 Labs - Lab 4: Counting Words</title>
        <link rel="stylesheet" type="text/css" href="../css/default.css" />
	<link rel="stylesheet" type="text/css" href="../css/syntax.css" />
    <script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
    </head>
    <body>
        <div id="header">
            <div id="logo">
                <a href="../">CS161 Labs</a>
            </div>
            <div id="navigation">
                <a href="../">Home</a>
                <a href="../about.html">About</a>
                <a href="../contact.html">Contact</a>
                <a href="../archive.html">Archive</a>
            </div>
        </div>

        <div id="content">
            <h1>Lab 4: Counting Words</h1>

            <div class="info">
    Posted on October 26, 2015
    
</div>

<h1 id="announcements">Announcements</h1>
<p>Office hours 1-2pm on Wednesday and Thursday.</p>
<h1 id="counting-words">Counting Words</h1>
<p>In this lab we will produce a sorted word count from input text. The input will be a text stream as in the <code>rot13</code> program and you will output a list of words with their associated counts such that the words are sorted (<code>String</code> is in typeclass <code>Ord</code> and so that ordering is acceptable) and all in lower-case. E.g. the text</p>
<pre><code>A parser for things
Is a function from strings
To lists of pairs
Of things and strings</code></pre>
<p>should output:</p>
<pre><code>a 2
and 1
for 1
from 1
function 1
is 1
lists 1
of 2
pairs 1
parser 1
strings 2
things 2
to 1</code></pre>
<p>Implement this as a program with the exact name <code>WordCount.hs</code> such that it can be run as <code>./WordCount</code> like the rot13 program. It should be in your <code>lab4/</code> directory of your laboratory repository. A word is defined as a contiguous sequence of letters with the characters <code>'\''</code> and <code>'-'</code>. Note that <code>'\'':[] == &quot;'&quot;</code>.</p>
<h1 id="collecting-permutations">Collecting Permutations</h1>
<p>Some words are permutations of each other: e.g. <code>[&quot;outs&quot;, &quot;oust&quot;]</code> or <code>[&quot;portions&quot;, &quot;positron&quot;]</code>. Write a second program which outputs the permutations in sorted order for those words which have multiple permutations, e.g. ignore words that appear once, or only appear as one permutation. E.g. given text</p>
<pre><code>On the Outer Pyre steps the three purest usurpers swore to their vile
pursuers they were no prey. The route was worse from the original form they
expected there as the live volcano nearby often erupts.</code></pre>
<p>the program should output</p>
<pre><code>erupts, purest
form, from
live, vile
no, on
outer, route
prey, pyre
pursuers, usurpers
swore, worse
there, three</code></pre>
<p>The words on a single line should be sorted as well as the lines themselves in the text output. Use the definition of a word given above. This program should be implemented with the exact name <code>CollectPermutations.hs</code> in your <code>lab4/</code> folder of your subversion repository. It should take in text just like the <code>rot13</code> program.</p>
<p>To perform the sort you will likely want to use the <code>Ord</code> typeclass</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">instance</span> <span class="dt">Ord</span>  <span class="fu">....</span>  <span class="kw">where</span>
   <span class="fu">...</span> <span class="fu">&lt;=</span> <span class="fu">...</span> <span class="fu">=</span> <span class="fu">...</span></code></pre>
<h1 id="to-hand-in">To hand in</h1>
<p>Your <code>lab4/</code> directory should have <code>CollectPermutations.hs</code> and <code>WordCount.hs</code>, each of which can be compiled to take in text input and output the desired text.</p>

        </div>
        <div id="footer">
            Site proudly generated by
            <a href="http://jaspervdj.be/hakyll">Hakyll</a>
        </div>
    </body>
</html><commit_msg>added extra credit<commit_after><?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
        <title>CS161 Labs - Lab 4: Counting Words</title>
        <link rel="stylesheet" type="text/css" href="../css/default.css" />
	<link rel="stylesheet" type="text/css" href="../css/syntax.css" />
    <script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
    </head>
    <body>
        <div id="header">
            <div id="logo">
                <a href="../">CS161 Labs</a>
            </div>
            <div id="navigation">
                <a href="../">Home</a>
                <a href="../about.html">About</a>
                <a href="../contact.html">Contact</a>
                <a href="../archive.html">Archive</a>
            </div>
        </div>

        <div id="content">
            <h1>Lab 4: Counting Words</h1>

            <div class="info">
    Posted on October 26, 2015
    
</div>

<h1 id="announcements">Announcements</h1>
<p>Office hours 1-2pm on Wednesday and Thursday.</p>
<h1 id="counting-words">Counting Words</h1>
<p>In this lab we will produce a sorted word count from input text. The input will be a text stream as in the <code>rot13</code> program and you will output a list of words with their associated counts such that the words are sorted (<code>String</code> is in typeclass <code>Ord</code> and so that ordering is acceptable) and all in lower-case. E.g. the text</p>
<pre><code>A parser for things
Is a function from strings
To lists of pairs
Of things and strings</code></pre>
<p>should output:</p>
<pre><code>a 2
and 1
for 1
from 1
function 1
is 1
lists 1
of 2
pairs 1
parser 1
strings 2
things 2
to 1</code></pre>
<p>Implement this as a program with the exact name <code>WordCount.hs</code> such that it can be run as <code>./WordCount</code> like the rot13 program. It should be in your <code>lab4/</code> directory of your laboratory repository. A word is defined as a contiguous sequence of letters with the characters <code>'\''</code> and <code>'-'</code>. Note that <code>'\'':[] == &quot;'&quot;</code>.</p>
<h1 id="collecting-permutations">Collecting Permutations</h1>
<p>Some words are permutations of each other: e.g. <code>[&quot;outs&quot;, &quot;oust&quot;]</code> or <code>[&quot;portions&quot;, &quot;positron&quot;]</code>. Write a second program which outputs the permutations in sorted order for those words which have multiple permutations, e.g. ignore words that appear once, or only appear as one permutation. E.g. given text</p>
<pre><code>On the Outer Pyre steps the three purest usurpers swore to their vile
pursuers they were no prey. The route was worse from the original form they
expected there as the live volcano nearby often erupts.</code></pre>
<p>the program should output</p>
<pre><code>erupts, purest
form, from
live, vile
no, on
outer, route
prey, pyre
pursuers, usurpers
swore, worse
there, three</code></pre>
<p>The words on a single line should be sorted as well as the lines themselves in the text output. Use the definition of a word given above. This program should be implemented with the exact name <code>CollectPermutations.hs</code> in your <code>lab4/</code> folder of your subversion repository. It should take in text just like the <code>rot13</code> program.</p>
<p>To perform the sort you will likely want to use the <code>Ord</code> typeclass</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">instance</span> <span class="dt">Ord</span>  <span class="fu">....</span>  <span class="kw">where</span>
   <span class="fu">...</span> <span class="fu">&lt;=</span> <span class="fu">...</span> <span class="fu">=</span> <span class="fu">...</span></code></pre>
<h1 id="to-hand-in">To hand in</h1>
<p>Your <code>lab4/</code> directory should have <code>CollectPermutations.hs</code> and <code>WordCount.hs</code>, each of which can be compiled to take in text input and output the desired text.</p>
<h1 id="extra-credit">Extra Credit</h1>
<p>Two extra credit assignments:</p>
<ol style="list-style-type: decimal">
<li>Use a <a href="https://en.wikipedia.org/wiki/Trie">trie</a> as the internal datastructure for the word counts</li>
<li><p>Write a program <code>WordSegmenter.hs</code> that uses <code>/usr/share/dict/words</code> to segment input text: e.g. the input</p>
<p>echo “Itisappropriatetostartthestudyofmathematicallogicbyinquiringwhatmathematicallogicis” | ./WordSegmenter</p></li>
</ol>
<p>should output:</p>
<pre><code>It is appropriate to start the study of mathematical logic by inquiring what
mathematical logic is</code></pre>
<p>You can start second extra credit program with a file</p>
<pre class="sourceCode haskell"><code class="sourceCode haskell"><span class="kw">import </span><span class="dt">System.IO</span>
<span class="kw">import </span><span class="dt">WordSegmenterLib</span>
       
main <span class="fu">=</span> <span class="kw">do</span>
    handle <span class="ot">&lt;-</span> openFile <span class="st">&quot;/usr/share/dict/words&quot;</span> <span class="dt">ReadMode</span>
    contents <span class="ot">&lt;-</span> hGetContents handle
    hClose handle</code></pre>
<p>and you’ll need to add in logic to take input as well as implement a file <code>WordSegmenterLib.hs</code> which will contain the module <code>WordSegmenterLib</code> that contains your segmentation logic.</p>

        </div>
        <div id="footer">
            Site proudly generated by
            <a href="http://jaspervdj.be/hakyll">Hakyll</a>
        </div>
    </body>
</html>
{'fraction_non_alphanumeric': 0.15798204527712725, 'fraction_numerical': 0.01249024199843872, 'mean_word_length': 3.749304911955514, 'pattern_counts': {'":': 0, '<': 343, '<?xml version=': 2, '>': 343, 'https://': 1, 'lorem ipsum': 0, 'www.': 4, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2187940', 'n_tokens_mistral': 3408, 'n_tokens_neox': 3228, 'n_words': 1091}