I am currently writing a webcrawl bot. It generates a list of URLs, and I need it to remove duplicates, and sort the lines alphabetically. My code looks like this:#! /bin/bashURL=google.comwhile [ 1 ]; do  wget --output-document=dl.html $URL  links=($(grep -Po '(?<=href=)[^]*' dl.html))  printf %s\n ${links[@]} >> results.db  sort results.db | uniq -u  URL=$(shuf -n 1 results.db)  echo $URLdoneSpefifcially the line:sort results.db | uniq -u
Sorting not seeming to work
shell script
POSIX says of uniq -u:Suppress the writing of lines that are repeated in the input.which means that any line which is repeated (even the original line) will be filtered out.  What you meant was probably (done with POSIX also):sort -u results.dbFor sort -u, POSIX saysUnique: suppress all but one in each set of lines having equal keys. If used with the -c option, check that there are no lines with duplicate keys, in addition to checking that the input file is sorted.In either case, the following lineURL=$(shuf -n 1 results.db)probably assumes that the purpose of sort/uniq is to update results.db (it won't).  You would have to modify the script a little more for that:sort -u results.db >results.db2 && mv results.db2 results.dbor (as suggested by @drewbenn), combine it with the previous line.  However, since that appends to the file (combining the commands as shown in his answer won't eliminate the duplicates between the latest printf and the file's contents), a separate command sort/mv looks closer to the original script.If you want to ensure that $URL is not empty, that's (actually another question), and done by the [ test, e.g.,  [ -n $URL ] && wget --output-document=dl.html $URLthough simply exiting from the loop would be simpler:[ -z $URL ] && break