I have previously asked a question about keeping the first instance of duplicates and the solution provided has been working perfectly until now (see Keeping first instance of duplicates).However, I now have the scenario where the values in Columns C and D could be the same but in different orders and I only want to keep one (either order will do).Example input:A B C D E F G1 2 T TACA 3 2 Q9 3 A C 9 3 P8 3 I R 8 2 Q9 3 C A 9 3 P4 8 C T 7 4 P9 3 T G 9 3 PDesired output:A B C D E F G1 2 T TACA 3 2 Q9 3 A C 9 3 P8 3 I R 8 2 Q4 8 C T 7 4 P9 3 T G 9 3 PUsing: sort -k3,4 -k5,5r -k1,1r file | sort -k1,1 -k3,4 -u | sort -k1,1r (or similar) keeps both the 9 3 A C 9 3 P and 9 3 C A 9 3 P versions but I only want to keep one of these. The added complication is that for values for Column A could have multiple acceptable Column C and D values (such as row 2 9 3 A C 9 3 P and row 5 9 3 T G 9 3 P in the Desired Output example above) so cannot search for duplicates based on Column A alone.Thanks!
Keeping first instance of duplicates when column values are interchangeable
bash;shell script;sort;duplicate