Okay so I want remove duplicate lines but it's a bit more complicated than that..I have a file named users.txt, example of file is: users:email@email.com users1:email@email.comNow due to a bug in my system people were able to register with the same email as someone else, so I want to remove if lines have the same email more than once, example of issue: user:display:email@email.com user2:email@email.com user3:email@email.com user4:email@email.comNotice how user, user2, user3, user4 all have the same email.. well I want to remove user2, user3, user4 but keep user.. or vice versa ( first one to be picked up by request ) remove any other lines containing same email..so if email@email.com is in 20 lines remove 19 spam@spam.com is in 555 lines remove 554 and so fourth..
Remove duplicate lines with a twist
shell script;text processing;awk;uniq
This calls for Awk.  Since the field you want to check is the first field of each line, just reference $1.awk -F: '! ($1 in seen) {print; seen[$1]}' users.txtYou can golf this to reduce it considerably:awk -F: '!a[$1]++' users.txtThe longer form is more or less self-explanatory; you build an associative array using each email address as an index, without bothering to assign a value.  Then you can just check if the email address has been seen before (i.e., if the associative array has a particular email address as an index already), and print the whole line if not.The shorter form is actually doing more or less the same thing, but requires more explanation for the shorter code.The postfix ++ operator acts on a variable after the expression is evaluated, so we'll come back to that later.In Awk, 0 means false and non-zero means true.  ! is for negation and reverses the truth value.Appearing as it does outside of curly brackets, the expression is interpreted as a boolean expression, with an associated action (in curly brackets) to be performed if the expression is true.  Since no action is explicitly stated, the default (implicit) action of printing the whole line is used, if the expression evaluates to true (non-zero).Essentially, this retrieves the value in the associative array a which is pointed to be the email address (first field) as its indexor creates that value initialized as 0 if not already present, interprets a 0 as false or non-zero as true, inverts this truth value and prints the whole line if the result is truthy, and then increments the value stored in the associative array at that point.A common enough Awk idiom, actually, but I wouldn't fault you for using the longer more explicit version.  :)