Q:

How to parse very big files in python?

I have a very big tsv file: 1.5 GB. i want to parse this file. Im using the following function:
def readEvalFileAsDictInverse(evalFile):
  eval = open(evalFile, "r")
  evalIDs = {}
  for row in eval:
    ids = row.split("\t")
    if ids[0] not in evalIDs.keys():
      evalIDs[ids[0]] = []
    evalIDs[ids[0]].append(ids[1])
  eval.close()

  return evalIDs 

It is take more than 10 hours and it is still working. I dont know  how to accelerate this step and if there is another method to parse such as file

A:

several issues here:

testing for keys with if ids[0] not in evalIDs.keys() takes forever in python 2, because keys() is a list. .keys() is rarely useful anyway. A better way already is if ids[0] not in evalIDs, but, but...
why not use a collections.defaultdict instead?
why not use csv module?
overriding eval built-in (well, not really an issue seeing how dangerous it is)

my proposal:
import csv, collections

def readEvalFileAsDictInverse(evalFile):
  with open(evalFile, "r") as handle:
     evalIDs = collections.defaultdict(list)
     cr = csv.reader(handle,delimiter='\t')
     for ids in cr:
        evalIDs[ids[0]].append(ids[1]]

the magic evalIDs[ids[0]].append(ids[1]] creates a list if doesn't already exist. It's also portable and very fast whatever the python version and saves a if
I don't think it could be faster with default libraries, but a pandas solution probably would.

A:

Maybe, you can make it somewhat faster; change it: 
if ids[0] not in evalIDs.keys():
      evalIDs[ids[0]] = []
evalIDs[ids[0]].append(ids[1])

to
evalIDs.setdefault(ids[0],[]).append(ids[1])

The 1st solution searches 3 times in the "evalID" dictionary.

