V

>/dev/null, but do it fast

If a script or program produces lots of output, generally, a reduction of the amount of output will make things run faster. The most radical reduction often seen at the command line is a redirection to /dev/null. However, this can be done in various ways with rather varying results.

My reference script ;-) is

$ while ((c<12345678)) ; do ((c++)) ; echo -n . ; done

If I run this on one of our newer servers, accessed by means of a remote SSH shell, it produces first of all lots of futile dots, and takes almost 2 minutes:

$ time bash -c "while ((c<12345678)) ; do ((c++)) ; echo -n . ; done"
(... lots of dots not shown ...)
real    1m58.311s
user    1m43.310s
sys     0m14.613s

Now, let's >/dev/null:

$ time bash -c "while ((c<12345678)) ; do ((c++)) ; echo -n . >/dev/null ; done"
real    2m52.775s
user    2m15.276s
sys     0m36.298s

That's hard to believe, isn't it? I have some idea of why this actually takes longer, but I don't know for sure. Anyway, there is room for improvement, of course:

$ time bash -c "while ((c<12345678)) ; do ((c++)) ; echo -n . ; done" >/dev/null
real    1m37.386s
user    1m33.458s
sys     0m3.668s
 
$ time bash -c "while ((c<12345678)) ; do ((c++)) ; echo -n . ; done >/dev/null"
real    1m36.513s
user    1m32.246s
sys     0m3.998s

The last 2 variations I have been running a few times to see if the small difference persists. The above quoted examples reflect some sort of observed average. But, try for yourself and let me know what you get.

Discussion

Johann Klasek, 2012-05-25 21:30

In my eyes this is not really hard to believe, because the try with the redirection inside the loop leads to at least two additional systemcalls (which are expensive in time) in each iteration. One can check this easy with strace-ing it …

The difference of the last two tries seems to lay somewhere in a kind of statistic variance. The shell "time" even on a idle unix system could not expected to be that precise as needed for real benchmarking … It is stated they are run a few times - probably not sufficent. To get the measurement more exact, the result of each variant should be taken from the average of 10 or more runs (depending on how many digits are to be taken serious).

Andreas Schamanek, 2012-06-12 10:49, 2012-06-12 11:08

Of course, system calls are a main factor. But system calls are everywhere :-) IMHO, here, they are not the best means of explanation. Mind you that system architecture, network speed, buffering and also the choice of terminal emulation might have a much larger impact.

Johann Klasek, 2012-07-19 18:58

Just to get me right:

My statement was concerning the /dev/null variants, where terminal emulation, networkspeed and other system depend stuff does not play any role … in these cases *makes a difference* if an open systemcall for a redirection is done inside or outside the loop! Every variant with stdout to a terminal is simple meaningless and not comparable at all (at least in the above cases).

My other concern was the similar times for the last 2 examples. They are fairly the same. Any variances came from the measurements (which are inherently inaccurate).

gupta deva, 2012-09-24 10:44

OK … quick test running in xterm locally on a vmware ubuntu-flavoured distribution loop is also smaller (c<123456)


(1. example) without /dev/null

real 0m5.337s user 0m1.524s sys 0m0.400s


(2. example) with /dev/null

real 0m2.547s user 0m1.808s sys 0m0.404s

m( ;-)

Andreas Schamanek, 2012-09-24 10:52

I think it's strange that the user time in the 2nd example is larger than the one from the 1st. You might want to run the loops several times and maybe with larger loops to see what your system is capable of :-)

Johann Klasek, 2012-10-03 13:36

My guess for this is that in the 2nd example the interpreter (bash) has more to do like parsing and processing the redirection (spending more time in the C library to handle buffered standard I/O).

More interesting is the fact that the system time does not differ. Maybe on a virtual machine handles devices like /dev/null or a pseudo terminal device in the same manner (at least regarding time) or has no measurable overhead (for this number of iterations)? 8-o

Enter your comment. Wiki syntax is allowed:
O S​ H G M
 
 
blog/111020_dev_null_but_fast_please.txt · Last modified: 2012-02-27 19:57 by andreas