[Beowulf] Re: Re: Home beowulf - NIC latencies

[James Cownie]

> A last remark. I really think that the argument of using the same
> swiss-army-knive MPI implementation such as ScaMPI or Intel MPI or
> even MPI/Pro to infere interconnect characteristics is even worse that
> looking at latency and bandwidth alone. These implementations are
> never going to be designed to use all hardware efficiently, their
> design is either historic (Scali used to provided software for SCI
> alone) or politicaly motivated (Intel is using uDapl, hummm, wonder
> why), or both. They are by-products of the MPI forum's failure to make
> the Standard practical (compatible ABI).

As someone who was on the MPI Forum, and sat through an awful lot of
meetings, I'd like to provide some justification for _why_ we didn't try
to make a binary standard.

1) At the time (over ten years ago), we would have been happy to have
   _one_ MPI implementation on a given machine, and we weren't expecting
   to have multiple MPIs on the same hardware. (It was by no means a
   foregone conclusion that MPI would succeeed).

2) We didn't expect MPI to move into a commercial environment in which
   the people running the code wouldn't have the sources, and wouldn't
   be optimising for _their_ machine, which obviously requires
   recompilation, making an ABI irrelevant.

3) Not having a binary interface allows optimisations in the C MPI
   interface (such as using macros rather than functions in some
   places).

4) A binary interface based on no MPI implementation experience would
   likely be worse than no binary interface.

5) MPI is supposed to be machine and architecture independent, specifying
   a binary interface under those circumstances is hard. 

   Maybe you can do it if you leverage the C ABI, however it's not clear
   that that is ideal, since that either changes with time, or suffers
   from poor vision of the future too (e.g. look at the required
   alignment of double in the x86 ABI).

6) It was a hard enough job to agree on the source level
   specification. If we'd tried to add an ABI we'd probably still be
   stuck in the Bristol Suites :-) 

You seem to think (maybe subconsciously) that the MPI forum added
features the standard just to make life hard for implementors and
to kill performance ;-)

I can assure you that that was not the case, and that the standard was a
compromise between features which users really wanted and what the
implementors felt they could reasonably provide. If the standard had not
provided things the users wanted (like wildcard receive), then it's
quite possible that his whole discussion would be moot because MPI would
by now be of only historical interest since the user community would
have ignored it.

If you _really_ believe that there is so much performance benefit for
your customers in having an MPI-light with the restrictions you outlined
which only runs on your hardware, then no-one's stopping you from
providing it. 

The market will decide...

-- 
-- Jim
--
James Cownie	<jcownie at etnus.com>
Etnus, LLC.     +44 117 9071438
http://www.etnus.com