I do text manipulation a lot, and one of the operations I use frequently is sorting - often removing any duplicates as well.So I normally use the commands sort or sort -u either from command line or in scripts, macros, etc - if I'm not working in LibreOffice Writer or Calc, of course (which unfortunately don't have an option to remove duplicates while sorting, or I don't know ;-)Now I have a plain text file containing a large collection of symbols, emoticons, shapes, lines, non-standard ASCII letters and numbers, etc. with many duplicates.It was easy to convert them a into one character per line text.However, sorting and removing duplicates apparently is not as simple as one would think:Using the command sort -u file.txt > file-sorted.txt unfortunately reduces the 2078 lines down to just 359, removing about a thousand unique characters I guess - I can see that there are many many that are filtered out in error.So my conclusion is that sort -u command is only good for standard alphanumeric characters.Any ideas and suggestions?PS: Here is a sample text of 40 characters from the file I'm trying to process:     o O                 '  ' *  *       .  .   There are only a few duplicates here but although sort command processes the text with a few issues, but without any loss, both sort -u and uniq have exactly the same output, cutting it down to 11 with many characters wiped out.
What's the best way to sort?
text processing;sort
Try using something with proper Unicode support, such as Python:$ python3 -c 'import sys; print(\n.join(sorted(set(c for l in sys.stdin.readlines() for c in l.split()))))' < bar'*.Oo$ python3 -c 'import sys; print(len(set(c for l in sys.stdin.readlines() for c in l.split())))' < bar30