parallel processing and bash reduce

It’s sad that after all this time, one can look at any random article on parallel programming and find some variation of:

for i = 1 ... n
      create thread i
             do something
end for

as if that was the only way to express parallel computation. This is such an awkward way of looking at problems.  I think many problems come from the sloppy “non-determinism” of the operating systems and multi-core machines.  One of the few  interesting ideas seen in the last 20 years on parallel processing is the Google map-reduce scheme ( . But what I find impressive is bashreduce. This is really a clever trick and a great validation of the UNIX toolset design (as if it needed another validation).

This entry was posted in data center, operating systems, security+fault-tolerance, software engineering and tagged , . Bookmark the permalink.

One Response to parallel processing and bash reduce

  1. Ole Tange says:

    GNU Parallel http://www.gnu.org/software/parallel/ solves a different class of problems than map-reduce. For bashscripts it is very useful, though.

    See the basic usage on http://www.youtube.com/watch?v=LlXDtd_pRaY

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>