parallel processing and bash reduce

It’s sad that after all this time, one can look at any random article on parallel programming and find some variation of:

for i = 1 ... n
      create thread i
             do something
end for

as if that was the only way to express parallel computation. This is such an awkward way of looking at problems.  I think many problems come from the sloppy “non-determinism” of the operating systems and multi-core machines.  One of the few  interesting ideas seen in the last 20 years on parallel processing is the Google map-reduce scheme ( . But what I find impressive is bashreduce. This is really a clever trick and a great validation of the UNIX toolset design (as if it needed another validation).

Comment (1)

  1. Ole Tange wrote::

    GNU Parallel http://www.gnu.org/software/parallel/ solves a different class of problems than map-reduce. For bashscripts it is very useful, though.

    See the basic usage on http://www.youtube.com/watch?v=LlXDtd_pRaY

    Sunday, June 13, 2010 at 18:44 #