[Beowulf] python & Lush on a cluster (Newbie question)

[Robert G. Brown]

On Tue, 25 Jan 2005, Rolando Espinoza La Fuente wrote:

> Hi :)
> 
> (my english isn't very good...)
> 
> I'll build a basic beowulf cluster for numerical "research" (i hope),
> what do you think about using python and lush for programming on the
> cluster?
> 
> Anybody has comments about lush?
> 
> Better way (language... than C/Fortran) for programming (numerical
> apps) on the cluster?

Boy, you don't know the risks you run asking for advice on languages on
this list.  Folks have, um, "strong" is too weak a word -- opinions.

As a general rule, though, if you are serious enough about numerical
research to build a cluster in the first place to speed up the
computations, you will USUALLY be better off using a proper compiled
language such as C, C++, or Fortran than using any sort of interpreted
language.  I will avoid endorsing any one of these three at the expense
of the other two, but well-written code in any of them will generally
blow away equally well written code in an interpreted language (with a
few possible exceptions).  You'd have to run tests to get some idea of
the difference, but at a guess an interpreter will be around an order of
magnitude slower.  This means you'd need some ten nodes running in
efficient parallel just to break even.

Additionally, "real" parallel numerical programming requires library
support that is generally not available for anything but real compiled
languages.  To use e.g. MPI or PVM to write a distributed program,
you'll pretty much need one of these three.  Some of the advanced
commercial compilers have a certain amount of built-in support for
parallel programs as well.  Numerical libraries, e.g. the GSL, are
only likely to be available for and run efficiently within compiled
code, although I've heard rumors of ports into some interpreted
languages as well.

> Thanks in advance.
> 
> PD: If you don't know  about lush:
> 
> Lush is an object-oriented programming language designed for
> researchers, experimenters, and engineers interested in large-scale
> numerical and graphic applications.
> Lush's main features includes:
>     * A very clean, simple, and easy to learn Lisp-like syntax.
>     * A compiler that produces very efficient C code and relies 
>        on the C compiler to produce efficient native code 
>       (no inefficient bytecode or virtual machine)
> .     ....
> more info: http://lush.sourceforge.net/

You already know more about lush than I do, obviously.  Here I cannot
help you.  Perhaps this is a possible solution, but as a generally
cynical person about translation engines I doubt it.  In particular, I'd
want to see those claims demonstrated in real benchmark code.  Getting
maximal performance out of a system often requires some fairly subtle
tricks, tricks that translators are unlikely to be able to figure out
and implement safely.  My experiences with tools "like" this (e.g. f2c)
are that they perform a sort of linear translation of the language
elements into e.g. C code fragments and assemble them into something
that is logically identical but often impossible to read (on the
translated side) and not terribly efficiently translated.

Now fortran is of course an upper-level compiled language not THAT
dissimilar to C, and yet f2c produces illegible code with all sorts of
wrapper-enclosed subroutine calls to do the translation of various
fortran functions into something that can be called in C, with
automatically generated variables, with goto statements and crude loop
structure.  f2c is (was) a fairly mature product and has been around a
long time.

This is the basis of my cynicism.  Would a translator of a "lisp like"
upper level language that is very DISsimilar to C be capable of doing a
decent job of producing legible C code that isn't heavily instrumented
with black box calls and perverse loop and conditional constructs?  What
does it do for arrays and pointers?  Is it likely to do better than the
ugly job done by f2c, when fortran actually CAN be translated to legible
C pretty easily by humans but not, apparently, by computers?

If you decide to forge on ahead with lush, report back in a few months
and tell us how things are going -- I for one am very curious as to how
you'll do. If possible, run some comparative benchmarks in native C and
in lush-based C.  See if you can make lush create PVM or MPI-based
parallel applications.

   rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu