I have a very large unsorted file, 1000GB, of ID pairsID:ABC123 ID:ABC124 ID:ABC123 ID:ABC124 ID:ABC123 ID:ABA122ID:ABC124 ID:ABC123 ID:ABC124 ID:ABC126I would like to filter the file for 1) duplicatesexampleABC123 ABC124ABC123 ABC1242) reverse pairs (discard the second occurrence)exampleABC123 ABC124ABC124 ABC123After filtering, the example file above would look likeID:ABC123 ID:ABC124 ID:ABC123 ID:ABA122ID:ABC124 ID:ABC126Currently, my solution is thisvar hash;foreach line in file      get ID1       get ID2      if exists in hash ID1ID2 or exists in hash ID2ID1        //next line, as we have already have this in our collection      else         add to hash ID1 ID2                  print the linewhich gives me the desired results for smaller lists, but takes up too much memory for larger lists, as I am storing the hash in memory.I am looking for a solution that will take less memory to implement.Some thoughts I have are1) save the hash to a file, instead of memory2) multiple passes over the file3) sorting and uniquing the file with unix sort -u -k1,2 , but that takes many hours and it also doesn't reduce the file that much.So even if I run that sort,  Most of the filesize is retained.  when the hash is searched for ID2ID1 , I still have to load a huge file into memory
How to filter a very, very large file
algorithms;efficiency;memory management;data sets;information retrieval