There is a file with a couple of identical and repetitive entries, which looks something like this:123 abc nhjk123 abc cftr123 abc xdrt123 def nhjk123 def cftr123 def xdrtIf the combination of (coloumns) Field1 and Field2 match, then only the first occurrence where they both match needs to be retained. So since 123 and abc of the first line match with 123 and abc of the second line, only the first line is to be retained. On further comparision, since the match is true for the first and third line too, again only the first line is to be retained.However,For the first and fourth line 123 and 123 matches but abc and def do not match, so both lines are to be retained.So the final output should look like this:123 abc nhjk123 def nhjk
Remove repetitive entries from a file
files;duplicate
One way to do it is with the -u flag to sort although this may end up not preserving original file order:sort -k1,1 -k2,2 -u fileIf you need the de-duping done with file order preservedawk '!a[$1, $2]++' file