I need to download some pages from a site.What am I trying to do is using sed on the site source code to get the links, passing them one by one to curl and then output the downloaded file on the right file in right directory.I will try to get more explicit.In the page source there's lines like this one:... href=view-source: http://www.site.org/the/file-42.php>/the/file-42.php </a>&gt; </span><span> OutDir and some more things ...I'm getting what I need (link - filename - dirname) like this:for i in `cat ~/site_source_file.htm `; do    echo $i | grep http://www.site.org |    sed -n 's|^.*\(http://\(www.site.org/the/file-[0-9]*\)\.php\).*.php </a>&gt; </span><span> \(.*\)|\1 > \3/\2|p' |        xargs -r done;which output is something like this:http://www.site.org/the/file-42.php > OutDir/the/file-42What I need to do is redirect http://www.site.org/the/file-42.php content to a file named /the/file-42 on a directory named OutDirSo instead of xargs -r alone I thought usign xargs -r curl to redirect curl output to a file.But it doesn't work.Do you have any suggestion how can I redirect 'curl' output to a file this way?
Redirecting sed to curl then to file
shell script;sed;io redirection;curl;xargs
The reason your stratagem of using sed<->xargs<->curl doesn't work is that the > is interpreted by the shell and not xargs.There are a few things you could do here: 1) curl -o like as shown below:for i in `cat ~/site_source_file.htm `; do    echo $i | grep http://www.site.org |    sed -n 's|^.*\(http://\(www.site.org/the/file-[0-9]*\)\.php\).*.php </a>&gt; </span><span> \(.*\)|curl \1 -o \3/\2|p' |    bash doneIf you want to use xargs then you could: for i in `cat ~/site_source_file.htm `; do    echo $i | grep http://www.site.org |    sed -n 's|^.*\(http://\(www.site.org/the/file-[0-9]*\)\.php\).*.php </a>&gt; </span><span> \(.*\)|\1 \3/\2|p' |    xargs -r -n 2 sh -c 'shift $1; curl $1 > $2' 2 1done;