An Inverted Index is a data structure used to create full text search.



Task

Given a set of text files, implement a program to create an inverted index.

Also create a user interface to do a search using that inverted index which returns a list of files that contain the query term / terms.

The search index can be in memory.




if File.exist? "index.dat"
  @data = Marshal.load open("index.dat")
else
  @data = {}
end
 
# Let's give the string class the ability to tokenize itsself into lowercase
# words with no punctuation.
class String
  def index_sanitize
    self.split.collect do |token|
      token.downcase.gsub(/\W/, '')
    end
  end
end
 
# Just implementing a simple inverted index here.
ARGV.each do |filename|
  open filename do |file|
    file.read.index_sanitize.each do |word|
      @data[word] ||= []
      @data[word] << filename unless @data[word].include? filename
    end
  end
end
 
open("index.dat", "w") do |index|
  index.write Marshal.dump(@data)
end