Performance Variations using MPI/Myrico

[Thomas Davis]

Patrick Geoffray wrote:
> 
> Thomas Davis wrote:
> >
> > We are looking for sites that run Intel Linux/SMP(dual)/MPI/Myricom 2k,
> > and have experienced performance variations.  IE, you've ran the NAS/FT
> > parallel benchmark, densely packed (using all CPU's on the nodes), and
> > noted that the runs come up different each time - and the difference is
> > not minor (as much as 80%).
> 
> Hi Thomas,
> 
> I think I know the machine you are thinking about :-)
> I have a lot of documentation (trace file, profiles, timings) for the NAS
> FT benchmark on this cluster. I would be happy to show some screenshot of
> the MPI trace but I am afraid it would be a too big file to send on the
> list.
> 

Patrick, I'm sure you know too..  Like I said in a previous list
message, everyone else we've talked to is baffled too.

> 
> > 3) Any ideas on what could cause this much variation?
> 
> I have some ideas, but nothing I would bet on. Mainly cache trashing : the
> memory copy operation is improved with SSE by using the prefecthing
> support, and this prefetch bypass the L2 cache. Without SSE, the L2 cache
> is happilly flushed as a processor is doing a copy. As the FFT code
> include a copy step, who knows... :-)
> 
> Ahh, I am eager to see dual athlon on the market...
> 

We are eager too.. Believe me!  You try selling PIII's to people used to
Cray/SP memory performance some day..  :-)

-- 
------------------------+--------------------------------------------------
Thomas Davis		| ASG Cluster guy
tadavis at lbl.gov		| 
(510) 486-4524		| "80 nodes and chugging Captain!"