Suppose if i have a situation in which some files need to be copied and it takes long time soi would have to go for parallel processing of the file copy.for example it may look like thisfor i in ipaddresslistdocp x y &  //running in back ground or some other process in backgrounddonewait  //will cause till all the files are copied or whatever process if all are finishednow i have used wait  so that all the backgroud processes are completedbut there are cases in it like 1) some file copy may happen earlier so if i have to do some processing on these filesi have to wait till all the files are copied.2) if the copy process (or any other program running in back ground) writes to a log filethe log file could be garbled with each background process trying to write to the fileat the same time.is there any workaround for such things,1)i mean if i can know that this process has finished then i could start doing the rest of the processing for that particular instance of the process(say file copy) if its completed.also, moreover writing to log files could be in orderly fashion.please make suggestions
multiple background processes in a script
linux;shell script;process;process management
If some job needs to be started after some file has been copied, just make it part of the background job:(cp this /there && start job that needs this in /there) &(cp that /here && start job that needs that in /here) &wait(the last & is not necessary).Now for more complex dependencies, you could use GNU make -j.make -j2 -f /dev/fd/3 3<< 'EOF'all: j1 j2 j3.PHONY: cp1 cp2 cp3 j1 j2 j3 allcp1:    cp this /therecp2:    cp that /herecp3:    cp this /herej1: cp1    start job that needs this in /therej2: cp2    start job that needs that in /herej3: cp1 cp3    start job that needs this in /here and /thereEOF-j2 would run up to 2 jobs at any given time, and dependencies would be respected.Now to avoid garbling of log files you have two main optionsdon't interleave them, that is append the content of each job one after the other.try to ensure they interleave nicely, possibly tagging each line of each job to make it easier to see what job which line belongs to.For 1, the easiest is to store each job output in a separate file and to merge them afterwards:(cp this /there && start job that needs this in /there) > j1.log 2>&1 &(cp that /here && start job that needs that in /here) > j2.log 2>&1 &waitcat j1.log j2.log > jobs.logAnother option is to use pipes to gather the output of each job and have cat merge them. Shell process substitution as available in ksh, zsh or bash can help us with that and even take care of the backgrounding:j1() { cp this /there && start job that needs this in /there; }j2() { cp that /here && start job that needs that in /here; }cat <(j1 2>&1) <(j2 2>&1) > jobs.logj1, j2 and cat will be started concurrently and inter connected with pipes.However note that cat will only start reading from the second pipe (that is written to by j2) after j1 has finished. That means that if j2 writes more logging than the size of the pipe (for instance, on Linux, typically 64kiB) j2 will be blocked until j1 finishes.That can be avoided by using sponge from moreutils, like:cat <(j1 2>&1) <(j2 2>&1 | sponge) > jobs.logThough that would mean all the output of j2 would be stored in memory, and cat will only start writing the output of j2 in jobs.log after j2 has finished, in which case using pv -qB 100M for instance may be preferable:cat <(j1 2>&1) <(j2 2>&1 | pv -qB 100M) > jobs.logThat way j2 would only get paused (if j1 has not finished yet) after 100M (plus two pipe contents) of logs have been output, and pv wouldn't wait for j2 to finish before outputting to stdout.Note that for all the above, you need to beware that once you redirect the output of most commands to a file or pipe (anything but a tty), the behavior is affected. The commands, or rather the stdio API of the libc they call (printf, fputs, fwrite...) detects that the output is not going to a terminal and perform an optimisation by outputting in big chunks (several kilo-bytes), while they don't do that for standard error. That means the order of the output and error messages will be affected. If that's an issue, on GNU systems or FreeBSD (at least) and for dynamically linked commands, you can use the stdbuf command:stdbuf -oL j1 > j1.log 2>&1instead of:j1 > j1.log 2>&1to ensure that the stdio output is line-buffered (each line of output will be written separately as soon as they are complete).For option 2, writes to a pipe of less than PIPE_BUF bytes which on Linux is 4096 bytes so much larger than your average line of log, are guaranteed to be atomic, that is if two processes write to the same pipe at the same time, their 2 writes are guaranteed not to be intertwined. There's no such guarantee on regular files but I seriously doubt that 2 writes of less than 4kiB could end up intertwined on any OS or filesystem.So if it weren't for that buffering described above and if log lines were output individually as a whole and separately, you'd have a guarantee that the lines of the output wouldn't have a piece of line of this job and a piece of line of that other job.However, nothing prevents a command to do a flush in between two parts of a line being written (like printf(foo); fflush(stdout); printf(bar\n);) and there's no buffering on stderr.Another problem is that once the lines of all the jobs are interleaved, it's going to be hard to tell which line is for which job.You can solve both problems by doing something like:tag() { stdbuf -oL sed s%^%$1: %; }{  j1 2>&1 | tag j1 &  j2 2>&1 | tag j2} | cat > jobs.log(note that we don't need the wait (and it wouldn't work anyway in most shells), because cat will not finish until there's nobody writing to the pipe anymore, so not until j1 and j2 have terminated).Above we use | cat to have pipe with its atomicity guarantee. We pipe the output of each command to a command that tags every line with the job name. j1 and j2 can write their output however they want, sed (because of stdbuf -oL) will output the lines (with the tag prefix) as a whole and separately, which will guarantee the output not to be mangled.The same note as above still applies: we're not applying stdbuf -oL to the commands in j1 and j2 so they will most likely buffer their output which may therefore be written long after it has been produced. That is even worse than in the previous case, because if you see:j1: doing Aj1: doing Bj2: doing CThat does mean that j1 did A before doing B, but not that it did any of them before j2 doing C. So once again, you may need to apply stdbuf -oL to more commands if it's an issue.Note that you can't apply stdbuf to shell functions like j1 or j2 above, but at least with GNU and FreeBSD stdbuf, you can use this to set stdbuf globally or on a per-subshell basis:stdbuf_LD_PRELOAD=$(stdbuf sh -c 'export -p LD_PRELOAD')line_buffered_output() {  eval $stdbuf_LD_PRELOAD  export _STDBUF_O=L}j1() (line_buffered_output; cp this /there && start...)