I am writing a batch script to sort through gigs and gigs of data. All of the data is text but the script will take a long long time to execute. I would like to give some visual indication that the script is running.  I found the program called pv which allows you to create progress bars and other nice CLI progress indicators.  My question is what type of impact does something like this have on performance and are there any other alternatives without me having to reinvent the wheel. I have googled it to death and haven't found anything which is surprising as I would assume you would only show progress on long tasks where performance is important.  But I suppose I am using clock cycles and you could also use it to measure disk I/O or network data transfers where using a few cycles for a progress indication wouldn't matter.Any ideas?P.S. I have also considered using the echo -ne trick to create my own but in order to make it feasible I would have to use the % operator on every loop and only take action ever 100th or so loop but that is A LOT of wasted calculations... 
Pipe Viewer - Progress monitor performance consequence
text processing;performance;cpu;pv
There will typically (lacking zero-copy trickery) be measurable overhead due to the extra IPC: copying the data from one process to another, rather than the workhorse process reading files directly. A pipe may also result in loss of performance (or functionality) for other reasons: with piped input a process cannot seek() on its input, and cannot mmap() it.Generally though, the main performance bottlenecks are probably disk I/O, and CPU compute time (which presumably is intensive in your case). These may be very much larger than the IPC overhead, but there are many variables here (CPU type, disk type and fileystem type, available physical RAM, OS and version, libc and version  at least).You can get a rough idea of performance with some quick tests, taking care to flush the disk cache before each one (I'm using linux, I use this method) in between each test.# time ( pv -pt somethinglarge.iso | sha256sum )[...]real    0m8.066suser    0m5.146s sys     0m1.075s# time ( sha256sum somethinglarge.iso )[...]real    0m7.913suser    0m5.064ssys     0m0.309sNote the similar real and user and  times, and the marked increase in system time for the piped case due to the extra copying.On some OSes, specifically Linux, you may be able to read per-process I/O stats from /proc (see 3.3) (you'll need CONFIG_TASKSTATS enabled in the kernel for this). This isn't as easy or as slick as pv, but it's low-overhead. pidstat uses this, it can be used to show real-time throughput (rate) on a PID, but it's less useful as a completion indicator.A similar linux option (this one doesn't need CONFIG_TASKSTATS), given a process and file descriptor, you can track the file descriptor's offset in /proc/PID/fdinfo/FD (the pos: field). Here's a toy script that shows this:FILE=/tmp/some-large-inputSZ=$(stat -c %s $FILE)# start slow process in background( some-slow-command $FILE ) &PID=$!FD=/proc/$PID/fdinfo/3       # some experimentation required# or iterate over /proc/$PID/fd/* with readlink # start %-ometer in background, exits when FD disappears(while nawk '/^pos:/{printf(%i\n,$2*100/'$SZ')}' $FD 2>/dev/null ; do    sleep 5  # adjustdone | dialog --gauge $PID: processing $FILE ($SZ bytes) 10 60) &wait $PIDif [ $? -eq 0 ]; then    echo 100 | dialog --gauge $PID: completed $FILE ($SZ bytes) 10 60else     echo ...fi(Caveat: not accurate for small files, libc stdio buffering will skew the results.)Other options that occur to me right now:use lsof to monitor a processes fd offsetsnot exactly lightweight, but multi-platform, and you can start it on any long-running process after the fact, which you cannot do with pv (it's not pretty either, since lsof refuses to give both the size and offset in one go)something hackish with LD_PRELOAD and some stubs which track data read/write, this too is multi-platform, but I think you'd have to write your own (I don't know any that does exactly this, but here's a related answer of mine)Update: someone has gone to the trouble of writing a general purpose  transfer monitor tool cv for use with coreutils commands on Linux. It uses similar logic to the /proc fdinfo approach (as demonstrated in the shell hack above). It also has a background mode where it scans /proc and reports on transfers in progress as it finds them. See related question Is it possible to see cp speed and percent copied?