<jupyter_start><jupyter_text>Assignment Data Structure and Solution We can use this Jupyter Notebook to analysis the data structure of different file and variables to help us to solve these three model construction and analysis.  First Class(CommandLine)**Calss Function**:   1. Judge the command line to accord with the stationary Formate.   2. Print Help Inforamtion if the command formate is wrong.   3. Default:      1)'index_nostoplist_nostemming.txt' / queries_nostoplist_nostemming.txt'.       2) For the First dataset(index?), it's the key vocabularly in the text.       3) For the second dataset(queries?), it's the summary of all the search queris.        **Example Command Formate**:```> python ir_engine.py -o result.txt```<jupyter_code>#==============================================================================
# Importing
import sys, getopt, re
import time
#==============================================================================
# Command line processing
class CommandLine:
    def __init__(self):
        opts, args = getopt.getopt(sys.argv[1:], 'hspw:o:')
        opts = dict(opts)
        self.exit = True

        if '-h' in opts:
            self.printHelp()
            return

        if len(args) > 0:
            print("*** ERROR: no arg files - only options! ***", file=sys.stderr)
            self.printHelp()
            return

        if '-w' in opts:
            if opts['-w'] in ('binary', 'tf', 'tfidf'):
                self.termWeighting = opts['-w']
            else:
                warning = (
                    "*** ERROR: term weighting label (opt: -w LABEL)! ***\n"
                    "    -- value (%s) not recognised!\n"
                    "    -- must be one of: binary / tf / tfidf"
                    )  % (opts['-w'])
                print(warning, file=sys.stderr)
                self.printHelp()
                return
        else:
            self.termWeighting = 'binary'

        if '-o' in opts:
            self.outfile = opts['-o']
        else:
            print("*** ERROR: must specify output file (opt: -o FILE) ***",
                  file=sys.stderr)
            self.printHelp()
            return

        if '-s' in opts and '-p' in opts:
            self.indexFile   = 'index_withstoplist_withstemming.txt'
            self.queriesFile = 'queries_withstoplist_withstemming.txt'
        elif '-s' in opts:
            self.indexFile   = 'index_withstoplist_nostemming.txt'
            self.queriesFile = 'queries_withstoplist_nostemming.txt'
        elif '-p' in opts:
            self.indexFile   = 'index_nostoplist_withstemming.txt'
            self.queriesFile = 'queries_nostoplist_withstemming.txt'
        else:
            self.indexFile   = 'index_nostoplist_nostemming.txt'
            self.queriesFile = 'queries_nostoplist_nostemming.txt'
            
        self.exit = False

    def printHelp(self):
        progname = sys.argv[0]
        progname = progname.split('/')[-1] # strip off extended path
        help = __doc__.replace('<PROGNAME>', progname, 1)
        print(help, file=sys.stderr)<jupyter_output><empty_output><jupyter_text>Second Class(IndexLoader)**Calss Function**:   1. Use the **Regular Expression** to extract the information from original file and index in a new dictionary.   2. The New Generator dictionary has 2 dimensions.**Data Structure(Python Dictionary)**:   1. {(term name)->{(term name)-> Count}}<jupyter_code>#==============================================================================
# Load (precomputed) Index File for (preprocessed) Document Collection
class IndexLoader:
    def __init__(self, indexFile):
        self.index = {}
        docidCountRE = re.compile('(\d+):(\d+)')
        f = open(indexFile, 'r')
        for line in f:
            term = line.split(' ', 1)[0]
            self.index[term] = {}
            for (docid, count) in docidCountRE.findall(line):
                docid = int(docid)
                self.index[term][docid] = int(count)
    def getIndex(self):
        return self.index<jupyter_output><empty_output><jupyter_text>Third Class(Queries)**Calss Function**:   1. Use the **Regular Expression** to extract the information from original file and index in a new dictionary.   2. The New Generator dictionary has 2 or more dimensions.   3. This class is to extract the queries and get the same dataset formate(python dictionary) as IndexLoader**Data Structure(Dictionary)**:   1. {(qid name)->{(qid name)-> count_num}}<jupyter_code>class Queries:
    def __init__(self, queriesFile):
        self.qStore = {}
        termCountRE = re.compile('(\w+):(\d+)')
        f = open(queriesFile, 'r')
        for line in f:
            qid = int(line.split(' ', 1)[0])
            self.qStore[qid] = {}
            for (term, count) in termCountRE.findall(line):
                self.qStore[qid][term] = int(count)
    
    def getQuery(self, qid):
        if qid in self.qStore:
            return self.qStore[qid]
        else:
            print("*** ERROR: unknown query identifier (\"%s\") ***" % qid, file=sys.stderr)
            if type(qid) == type(''):
                print('WARNING: query identifiers should be of type: integer', file=sys.stderr)
                print('         -- your query identifier is of type: string', file=sys.stderr)
            print(' -- program exiting', file=sys.stderr)
    
    def qids(self):
        return sorted(self.qStore)<jupyter_output><empty_output><jupyter_text>Forth Class(ResultStore)**Calss Function**:**Data Structure(Dictionary)**:<jupyter_code>class ResultStore:
    def __init__(self, outfile):
        self.outfile = outfile
        self.results = []

    def store(self, qid, docids):
        if len(docids) > 10:
            docids = docids[:10]
        self.results.append((qid, docids))

    def output(self):
        with open(self.outfile, 'w') as out:
            for (qid, docids) in self.results:
                for docid in docids:
                    print(qid, docid, file=out)<jupyter_output><empty_output><jupyter_text>Target Class(Retrieve)<jupyter_code># TODO: Finish the assignment part
class Retrieve:
    # Create new Retrieve object storing index and termWeighting scheme
    def __init__(self,index, termWeighting):
        self.index = index
        self.termWeighting = termWeighting

    # Method performing retrieval for specified query
    def forQuery(self, query):
        return range(1,11)<jupyter_output><empty_output><jupyter_text>**Below Code is the Formate of Command(Please customize with on the basis of your requirements)****How to generate command line As you wish**`sys.argv` is the python reserved variables and applied to `ArgumentParser.parse_args(args=None, namespace=None)` to be the one of the parameters. This is a list of strings, each of which is obviously derived from command line arguments, separated by spaces. We can redefine this list by below command.We should delete the key parameters like `python` and put the remains into the list.<jupyter_code># Example Command Formate:
#  > python ir_engine.py -o result.txt
sys.argv = ['ir_engine.py', '-o', 'result.txt']
config = CommandLine()
index = IndexLoader(config.indexFile).getIndex()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1431136897361514, 'fraction_numerical': 0.004282359441911866, 'mean_word_length': 2.6751269035532994, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 34, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '145715', 'license': 'MIT', 'n_tokens_mistral': 2110, 'n_tokens_neox': 2024, 'n_words': 613, 'path': 'Assignment/assignment_ipynb/assignment.ipynb', 'repo_name': 'DoHerasYang/Text_Processing_U.She', 'size': '7239'}