From eugen at leitl.org  Wed Feb  2 06:10:21 2011
From: eugen at leitl.org (Eugen Leitl)
Date: Wed, 2 Feb 2011 15:10:21 +0100
Subject: [Beowulf] HPC-Europa
Message-ID: <20110202141021.GN23560@leitl.org>

----- Forwarded message from Andy Alexander <andrew.alexander at ED.AC.UK> -----

From: Andy Alexander <andrew.alexander at ED.AC.UK>
Date:         Wed, 2 Feb 2011 13:53:47 +0000
To: MOLECULAR-DYNAMICS-NEWS at JISCMAIL.AC.UK
Subject: HPC-Europa
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.10)
	Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5
Reply-To: Andy Alexander <andrew.alexander at ED.AC.UK>

Please see below (next deadline is 28th February). If anyone is interested 
in access to EPCC in Edinburgh through this programme and would like to 
investigate potential local scientific contacts, please let me know 
(a.alexander at ed.ac.uk).

---------------------------------

HPC-Europa: research visits using High Performance Computing

HPC-Europa is an EC-funded programme which allows scientists to carry out 
short "transnational access" research visits to collaborate with a  
research department working in a similar field.

Applicants can be working in any discipline, but MUST require the use of  
High Performance Computing facilities (for large-scale computer modelling, 
etc).

Visitors work closely with a "host" research group working in a similar  
field of research - over 200 research groups in 7 countries are currently 
associated with the programme as "host" research groups (see webpage for 
more details).

Visits can last between 3 weeks and 3 months.

HPC-Europa pays travel and living costs and provides accommodation.

Following EC guidelines, applications are particularly encouraged from the 
new EU member countries and those who do not have access to similar  
computing facilities at their home institute.

The programme is open to researchers of all levels, from postgraduate  
students to the most senior professors.

The next closing date for applications is MONDAY 28th FEBRUARY.

Closing dates are held every 3 months, and the HPC-Europa programme  
continues until December 2012.

Further information and the on-line application form can be found at  
http://www.hpc-europa.eu/

Any questions not answered by the information on the webpage can be  
addressed to the HPC-Europa helpdesk at access at hpc-europa.org

------------------------------------------------


-- 
Dr Andrew J. Alexander
School of Chemistry
University of Edinburgh                tel: +44 131 650 4741
Edinburgh   UK   EH9 3JJ               fax: +44 131 650 4743

The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.

To join or leave the molecular-dynamics-news email list, go to:
http://www.jiscmail.ac.uk/lists/molecular-dynamics-news.html

----- End forwarded message -----
-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE


From madskaddie at gmail.com  Wed Feb  2 12:29:37 2011
From: madskaddie at gmail.com (madskaddie at gmail.com)
Date: Wed, 2 Feb 2011 20:29:37 +0000
Subject: [Beowulf] Advice on 4 CPU node configurations
Message-ID: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>

  Hi all

  We are designing a new HPC system for our lab. We are shifting ideas
around (our budget is of 30K euro).

  We do CFD and mainly run two types of stuff:

  - Finite volume, with neighbour (hallo cell) comunication (so,
communication with neighbours only)

  - Direct Numerical Simulations, with pseudo-spectral methods, imposing
ALL_TO_ALL communication

  Because there are users of a proprietary finite volume code we are in
need of at least one machine with a good RAM figure (48GB or above).

  So, we would like to tap into the beowulfers pound of knowledge and get
your opinions/comments/experience on...

  - 4 CPU configurations in the same mobo, like this one:

   http://www.supermicro.com/Aplus/system/2U/2042/AS-2042G-TRF.cfm

  here, one of our questions has to do with bandwith bottlenecks (with this
solution we are thinking we could use DDR or QDR Infiniband to join
different nodes)

  - Some comments on AMD 12 or 8 core Opteron 6100+ CPUs

  - Registred vs unbuf memmory

  - Intel alternatives?

  anything else we should think/look at ?


Thanks,


Gil Brand?o

-- 
"
It can't continue forever. The nature of exponentials is that you push
them out and eventually disaster happens.
"
Gordon Moore? (Intel co-founder and author of the Moore's law)


From john.hearns at mclaren.com  Thu Feb  3 01:45:18 2011
From: john.hearns at mclaren.com (Hearns, John)
Date: Thu, 3 Feb 2011 09:45:18 -0000
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
Message-ID: <68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>

> 
>   We are designing a new HPC system for our lab. We are shifting ideas
> around (our budget is of 30K euro).
> 
> 
>   anything else we should think/look at ?
> 

I would be looking at small SMP machines from Bull and SGI if I were
you.
http://www.bull.com/extreme-computing/bullxs.html
www.sgi.com   small UV machines


Also look at ScaleMP and Numascale
Here's a damn good article from Doug Eadline:
http://www.linux-mag.com/id/7947

I must admit though I don't know how far a budget of 30K takes you
there!

The contents of this email are confidential and for the exclusive use of the intended recipient.  If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy.


From prentice at ias.edu  Thu Feb  3 06:42:18 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Thu, 03 Feb 2011 09:42:18 -0500
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
Message-ID: <4D4ABECA.1050905@ias.edu>

Hearns, John wrote:
>>   We are designing a new HPC system for our lab. We are shifting ideas
>> around (our budget is of 30K euro).
>>

30K EUR won't get you much, especially if you have to by IB  or 10 GbE
networking hardware. For that little money, I would recommend buying a
4-socket server with 8- or 12-core CPUs, and load it up with as much RAM
as possible. You might be able to by 2 or 3, depending on how much you
max them out.

If you go this route, make sure it has redundant disks, power supplies
and all this, since you won't have inherent redunancy of duplicate
cluster nodes.

I recently ordered a couple of Dell PowerEdge R815s with 32-cores (4
sockets) and 128 GB of RAM, and my users have been flocking to them.
Unfortunately, one of these servers keeps reporting SBEs since October
and that still hasn't resolved to my satisfaction (documented elsewhere
on this list), so I wouldn't recommend that particular model.

-- 
Prentice


From crhea at mayo.edu  Thu Feb  3 12:30:41 2011
From: crhea at mayo.edu (Cris Rhea)
Date: Thu, 3 Feb 2011 14:30:41 -0600
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <mailman.1.1296763201.24756.beowulf@beowulf.org>
References: <mailman.1.1296763201.24756.beowulf@beowulf.org>
Message-ID: <20110203203041.GD20154@kaizen.mayo.edu>

> From: Prentice Bisbal <prentice at ias.edu>
> Subject: Re: [Beowulf] Advice on 4 CPU node configurations
> 
> Hearns, John wrote:
> >>   We are designing a new HPC system for our lab. We are shifting ideas
> >> around (our budget is of 30K euro).
> >>
> 
> 30K EUR won't get you much, especially if you have to by IB  or 10 GbE
> networking hardware. For that little money, I would recommend buying a
> 4-socket server with 8- or 12-core CPUs, and load it up with as much RAM
> as possible. You might be able to by 2 or 3, depending on how much you
> max them out.
> 
> If you go this route, make sure it has redundant disks, power supplies
> and all this, since you won't have inherent redunancy of duplicate
> cluster nodes.
> 
> I recently ordered a couple of Dell PowerEdge R815s with 32-cores (4
> sockets) and 128 GB of RAM, and my users have been flocking to them.
> Unfortunately, one of these servers keeps reporting SBEs since October
> and that still hasn't resolved to my satisfaction (documented elsewhere
> on this list), so I wouldn't recommend that particular model.
> 
> -- 
> Prentice


I recently purchased a few servers from Advanced HPC: Supermicro RM-206
servers with 4 AMD sockets (currently using 4 x 12-core at 2.3GHz)
and 512GB (yes, 0.5TB) memory.

Heck of a machine for about the same price we paid for a Dell 905 
a couple years ago.

While not your typical "cluster node", the 48 CPU cores and
huge memory are a big win for "piggy" applications. 

Obviously, you can bottleneck on the network connection or the local
disks, but if the app can fit on a single machine, rather than having
to be split across several, it can be a win.

--- Cris

-- 
 Cristopher J. Rhea
 Mayo Clinic - Research Computing Facility
 200 First St SW, Rochester, MN 55905
 crhea at Mayo.EDU
 (507) 284-0587


From landman at scalableinformatics.com  Fri Feb  4 17:56:44 2011
From: landman at scalableinformatics.com (Joe Landman)
Date: Sat, 05 Feb 2011 01:56:44 -0000
Subject: [Beowulf] Anybody using Redhat HPC Solution in their Beowulf
In-Reply-To: <4CC1BA64.5050200@gmail.com>
References: <4CC11EA8.8030602@gmail.com>	<AANLkTik=Ej5C5fjBp0NnDYF__QOZVcNwm3QJSXuoRfwo@mail.gmail.com>
	<4CC1BA64.5050200@gmail.com>
Message-ID: <4CC63B5F.1080006@scalableinformatics.com>

On 10/22/2010 12:23 PM, Richard Chang wrote:

> I am satisfied with Rocks, but my group people wanted something well
> supported that means, RHEL instead of CentOS and "some other PAID"
> cluster manager instead of Rocks, especially, when we have the
> budget.
>
> Somehow, I have not been able to convince my folks that Rocks is a
> much better solution than the PAID ones.

Hmmm ... define "better".  Rocks has its strengths and weaknesses. It is 
fine as a basic distribution.  We've used all manner of cluster 
distributions, rolled our own, leveraged others.

It all comes down to what you want and need to do.  If you have very 
windows-y admins who are classically terrified of anything that looks 
like a CLI, Rocks might not be the most appropriate system.  If you have 
a complex networking or storage environment, it may take a bit more 
effort to encode that into various cluster distros.

Generally the free systems, Perceus, Rocks, XCat2, Onesis, ... do pretty 
good jobs, but "some assembly required."  Completely turnkey distros 
exist on the paid side, with varying degrees of admin friendliness. 
There are stacks from Bright Computing, Penguin, Platform, RedHat, 
Streamline-Computing, and a few others.

(disclosure: we've built/recommended/sold systems with many of these 
tools in the past, free and commercial)

Admin time/effort/friendliness is not something to be discounted. 
Better is in the eye of the beholder.

I do recommend at least one "roll your own" setup at least once, so you 
get a better appreciation for what the packages do.

Just my $0.021 USD (sorry, inflation is creeping up)

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 866 888 3112
cell : +1 734 612 4615


From robh at dongle.org.uk  Mon Feb  7 07:10:20 2011
From: robh at dongle.org.uk (Robert Horton)
Date: Mon, 07 Feb 2011 15:10:20 +0000
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
Message-ID: <1297091420.1795.27.camel@moelwyn>

On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
> Also look at ScaleMP and Numascale
> Here's a damn good article from Doug Eadline:
> http://www.linux-mag.com/id/7947
> 
> I must admit though I don't know how far a budget of 30K takes you
> there! 

My understanding is that the Numascale cards cost "a bit more" than IB
cards but you don't need a switch (it's a 3D taurus I think) so it would
be worth looking into.

Rob



From Craig.Tierney at noaa.gov  Mon Feb  7 08:31:21 2011
From: Craig.Tierney at noaa.gov (Craig Tierney)
Date: Mon, 07 Feb 2011 09:31:21 -0700
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <1297091420.1795.27.camel@moelwyn>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
Message-ID: <4D501E59.3010807@noaa.gov>

On 2/7/11 8:10 AM, Robert Horton wrote:
> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>> Also look at ScaleMP and Numascale
>> Here's a damn good article from Doug Eadline:
>> http://www.linux-mag.com/id/7947
>>
>> I must admit though I don't know how far a budget of 30K takes you
>> there!
>
> My understanding is that the Numascale cards cost "a bit more" than IB
> cards but you don't need a switch (it's a 3D taurus I think) so it would
> be worth looking into.
>
> Rob
>

Numascale was born out of the Dolphinics Interconnect.  The interconnect
already provided cach-coherent global shared memory, so they worked on making
it work at the node level.  The unique feature of this card is that it never
required a switch, and the topology was a Torus.  You could buy 2D or 3D
versions.  The maximum scale back then was 12 nodes in a dimension.  How
this works out with Numascale, I don't know.

I think evaluation systems are out in the wild and the are supposedly
really close to having something for sale.

Craig





> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From hahn at mcmaster.ca  Mon Feb  7 09:15:02 2011
From: hahn at mcmaster.ca (Mark Hahn)
Date: Mon, 7 Feb 2011 12:15:02 -0500 (EST)
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D501E59.3010807@noaa.gov>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn> <4D501E59.3010807@noaa.gov>
Message-ID: <Pine.LNX.4.64.1102071210310.29202@coffee.psychology.mcmaster.ca>

> I think evaluation systems are out in the wild and the are supposedly
> really close to having something for sale.

if anyone has done measurements of bandwidth, latency and rate,
please post!


From deadline at eadline.org  Tue Feb  8 10:34:36 2011
From: deadline at eadline.org (Douglas Eadline)
Date: Tue, 8 Feb 2011 13:34:36 -0500 (EST)
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <1297091420.1795.27.camel@moelwyn>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
Message-ID: <42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>

Although I wrote about the SMP and the Numascale hardware,
I have not yet had a chance to use it.

To me there are two issues worth considering. First,
if you need a big SMP this might be a low
cost solution. Second, should your application
not scale best in a node-based ccNuma system (ScaleMP
or Numascale), MPI is still an option. Indeed, no need
to rewrite the codes. And, management is probably
much easier.

Of course for large systems, clusters work best.

--
Doug


> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>> Also look at ScaleMP and Numascale
>> Here's a damn good article from Doug Eadline:
>> http://www.linux-mag.com/id/7947
>>
>> I must admit though I don't know how far a budget of 30K takes you
>> there!
>
> My understanding is that the Numascale cards cost "a bit more" than IB
> cards but you don't need a switch (it's a 3D taurus I think) so it would
> be worth looking into.
>
> Rob
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


--
Doug

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.



From gus at ldeo.columbia.edu  Tue Feb  8 10:54:26 2011
From: gus at ldeo.columbia.edu (Gus Correa)
Date: Tue, 08 Feb 2011 13:54:26 -0500
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
Message-ID: <4D519162.5040800@ldeo.columbia.edu>

How about the wiring?
Since it is switchless, how much cabling would it require?
The cables don't seem to be fiber.
How do you implement the torus (or other) topology?
With daisy-chains?
A 2D/3D (wraparound) mesh?
Other?
I could only find a datasheet for the card and cables
on their site, no pictures of a cluster or servers
connected.

Gus Correa

Douglas Eadline wrote:
> Although I wrote about the SMP and the Numascale hardware,
> I have not yet had a chance to use it.
> 
> To me there are two issues worth considering. First,
> if you need a big SMP this might be a low
> cost solution. Second, should your application
> not scale best in a node-based ccNuma system (ScaleMP
> or Numascale), MPI is still an option. Indeed, no need
> to rewrite the codes. And, management is probably
> much easier.
> 
> Of course for large systems, clusters work best.
> 
> --
> Doug
> 
> 
>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>>> Also look at ScaleMP and Numascale
>>> Here's a damn good article from Doug Eadline:
>>> http://www.linux-mag.com/id/7947
>>>
>>> I must admit though I don't know how far a budget of 30K takes you
>>> there!
>> My understanding is that the Numascale cards cost "a bit more" than IB
>> cards but you don't need a switch (it's a 3D taurus I think) so it would
>> be worth looking into.
>>
>> Rob
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>> --
>> This message has been scanned for viruses and
>> dangerous content by MailScanner, and is
>> believed to be clean.
>>
> 
> 
> --
> Doug
> 



From deadline at eadline.org  Tue Feb  8 13:14:03 2011
From: deadline at eadline.org (Douglas Eadline)
Date: Tue, 8 Feb 2011 16:14:03 -0500 (EST)
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D519162.5040800@ldeo.columbia.edu>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D519162.5040800@ldeo.columbia.edu>
Message-ID: <51093.192.168.93.213.1297199643.squirrel@mail.eadline.org>

I assume it is similar to Dolphin 1D, 2D, 3D torus.

--
Doug

> How about the wiring?
> Since it is switchless, how much cabling would it require?
> The cables don't seem to be fiber.
> How do you implement the torus (or other) topology?
> With daisy-chains?
> A 2D/3D (wraparound) mesh?
> Other?
> I could only find a datasheet for the card and cables
> on their site, no pictures of a cluster or servers
> connected.
>
> Gus Correa
>
> Douglas Eadline wrote:
>> Although I wrote about the SMP and the Numascale hardware,
>> I have not yet had a chance to use it.
>>
>> To me there are two issues worth considering. First,
>> if you need a big SMP this might be a low
>> cost solution. Second, should your application
>> not scale best in a node-based ccNuma system (ScaleMP
>> or Numascale), MPI is still an option. Indeed, no need
>> to rewrite the codes. And, management is probably
>> much easier.
>>
>> Of course for large systems, clusters work best.
>>
>> --
>> Doug
>>
>>
>>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>>>> Also look at ScaleMP and Numascale
>>>> Here's a damn good article from Doug Eadline:
>>>> http://www.linux-mag.com/id/7947
>>>>
>>>> I must admit though I don't know how far a budget of 30K takes you
>>>> there!
>>> My understanding is that the Numascale cards cost "a bit more" than IB
>>> cards but you don't need a switch (it's a 3D taurus I think) so it
>>> would
>>> be worth looking into.
>>>
>>> Rob
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
>>> Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>>
>>
>>
>> --
>> Doug
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


-- 
Doug

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.



From landman at scalableinformatics.com  Tue Feb  8 13:37:44 2011
From: landman at scalableinformatics.com (Joe Landman)
Date: Tue, 08 Feb 2011 16:37:44 -0500
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <51093.192.168.93.213.1297199643.squirrel@mail.eadline.org>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>	<1297091420.1795.27.camel@moelwyn>	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>	<4D519162.5040800@ldeo.columbia.edu>
	<51093.192.168.93.213.1297199643.squirrel@mail.eadline.org>
Message-ID: <4D51B7A8.4060808@scalableinformatics.com>

On 02/08/2011 04:14 PM, Douglas Eadline wrote:
> I assume it is similar to Dolphin 1D, 2D, 3D torus.

(in best Homer Simpson voice) Mmmmm 2D torus ....

http://www.youtube.com/watch?v=8-4P1WPE-Qg


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615


From james.p.lux at jpl.nasa.gov  Tue Feb  8 14:49:05 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Tue, 8 Feb 2011 14:49:05 -0800
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D51B7A8.4060808@scalableinformatics.com>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D519162.5040800@ldeo.columbia.edu>
	<51093.192.168.93.213.1297199643.squirrel@mail.eadline.org>
	<4D51B7A8.4060808@scalableinformatics.com>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0C64295@ALTPHYEMBEVSP20.RES.AD.JPL>

Why not a Klein bottle?

Jim Lux
+1(818)354-2075 

> -----Original Message-----
> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Joe Landman
> Sent: Tuesday, February 08, 2011 1:38 PM
> To: beowulf at beowulf.org
> Subject: Re: [Beowulf] Advice on 4 CPU node configurations
> 
> On 02/08/2011 04:14 PM, Douglas Eadline wrote:
> > I assume it is similar to Dolphin 1D, 2D, 3D torus.
> 
> (in best Homer Simpson voice) Mmmmm 2D torus ....
> 
> http://www.youtube.com/watch?v=8-4P1WPE-Qg
> 
> 
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman at scalableinformatics.com
> web  : http://scalableinformatics.com
>         http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf


From gus at ldeo.columbia.edu  Tue Feb  8 16:06:14 2011
From: gus at ldeo.columbia.edu (Gus Correa)
Date: Tue, 08 Feb 2011 19:06:14 -0500
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0C64295@ALTPHYEMBEVSP20.RES.AD.JPL>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>	<1297091420.1795.27.camel@moelwyn>	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>	<4D519162.5040800@ldeo.columbia.edu>	<51093.192.168.93.213.1297199643.squirrel@mail.eadline.org>	<4D51B7A8.4060808@scalableinformatics.com>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0C64295@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <4D51DA76.3000409@ldeo.columbia.edu>

Lux, Jim (337C) wrote:
> Why not a Klein bottle?

Because it wouldn't hold Homer's beer.

But my question was
how much cabling does it take to wire a 2D or 3D doughnut/torus
on the back of racks of servers with these Numascale cards,
and no switch.
(Or do they use switches?)
How much cabling?
Does it scale well?
Or will it look like our router on the 3rd floor?
Anyway, it's not as fun as Homer.

Thanks,
Gus Correa

> 
> Jim Lux
> +1(818)354-2075 
> 
>> -----Original Message-----
>> From: beowulf-bounces at beowulf.org [mailto:beowulf-bounces at beowulf.org] On Behalf Of Joe Landman
>> Sent: Tuesday, February 08, 2011 1:38 PM
>> To: beowulf at beowulf.org
>> Subject: Re: [Beowulf] Advice on 4 CPU node configurations
>>
>> On 02/08/2011 04:14 PM, Douglas Eadline wrote:
>>> I assume it is similar to Dolphin 1D, 2D, 3D torus.
>> (in best Homer Simpson voice) Mmmmm 2D torus ....
>>
>> http://www.youtube.com/watch?v=8-4P1WPE-Qg
>>
>>
>> --
>> Joseph Landman, Ph.D
>> Founder and CEO
>> Scalable Informatics Inc.
>> email: landman at scalableinformatics.com
>> web  : http://scalableinformatics.com
>>         http://scalableinformatics.com/sicluster
>> phone: +1 734 786 8423 x121
>> fax  : +1 866 888 3112
>> cell : +1 734 612 4615
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> http://www.beowulf.org/mailman/listinfo/beowulf
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



From Craig.Tierney at noaa.gov  Tue Feb  8 19:40:04 2011
From: Craig.Tierney at noaa.gov (Craig Tierney)
Date: Tue, 08 Feb 2011 20:40:04 -0700
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D519162.5040800@ldeo.columbia.edu>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D519162.5040800@ldeo.columbia.edu>
Message-ID: <4D520C94.4050505@noaa.gov>

On 2/8/11 11:54 AM, Gus Correa wrote:
> How about the wiring?
> Since it is switchless, how much cabling would it require?
> The cables don't seem to be fiber.
> How do you implement the torus (or other) topology?
> With daisy-chains?
> A 2D/3D (wraparound) mesh?
> Other?
> I could only find a datasheet for the card and cables
> on their site, no pictures of a cluster or servers
> connected.
>

I believe the cables are still copper.  The connectors
looked similar at SC10 when I tried out Dolphinics
circa 2004.

For a 3D card there are 6 ports, so you get two connections
in the X,Y, and Z directions.  So the 3D torus is just basically
wiring the nodes serially along each axis.  I don't recall
if any of the dimensions wrap around.   But if you use
36 nodes (3x3x4) you can easily build a 864 core/1.7TB shared memory
node in a rack (24 cores/node, 48GB/node).

How it performs is the real question.

Craig



> Gus Correa
>
> Douglas Eadline wrote:
>> Although I wrote about the SMP and the Numascale hardware,
>> I have not yet had a chance to use it.
>>
>> To me there are two issues worth considering. First,
>> if you need a big SMP this might be a low
>> cost solution. Second, should your application
>> not scale best in a node-based ccNuma system (ScaleMP
>> or Numascale), MPI is still an option. Indeed, no need
>> to rewrite the codes. And, management is probably
>> much easier.
>>
>> Of course for large systems, clusters work best.
>>
>> --
>> Doug
>>
>>
>>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>>>> Also look at ScaleMP and Numascale
>>>> Here's a damn good article from Doug Eadline:
>>>> http://www.linux-mag.com/id/7947
>>>>
>>>> I must admit though I don't know how far a budget of 30K takes you
>>>> there!
>>> My understanding is that the Numascale cards cost "a bit more" than IB
>>> cards but you don't need a switch (it's a 3D taurus I think) so it would
>>> be worth looking into.
>>>
>>> Rob
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>>
>>> --
>>> This message has been scanned for viruses and
>>> dangerous content by MailScanner, and is
>>> believed to be clean.
>>>
>>
>>
>> --
>> Doug
>>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From eugen at leitl.org  Wed Feb  9 04:21:11 2011
From: eugen at leitl.org (Eugen Leitl)
Date: Wed, 9 Feb 2011 13:21:11 +0100
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D51DA76.3000409@ldeo.columbia.edu>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D519162.5040800@ldeo.columbia.edu>
	<51093.192.168.93.213.1297199643.squirrel@mail.eadline.org>
	<4D51B7A8.4060808@scalableinformatics.com>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0C64295@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D51DA76.3000409@ldeo.columbia.edu>
Message-ID: <20110209122111.GQ23560@leitl.org>

On Tue, Feb 08, 2011 at 07:06:14PM -0500, Gus Correa wrote:

> how much cabling does it take to wire a 2D or 3D doughnut/torus
> on the back of racks of servers with these Numascale cards,

A cubic carbon (diamond) lattice would use 4 links
for a 3d lattice, but for cubic primitive you'd go
4 links for 2d and 6 links for 3d. 

Cables can be short within a rack if your grid is
wrapped around. Of course if you start to interconnect
nodes in adjacent racks with short cables for 3d, the
result would drive a spider green with envy. So
below the floor they would go, and pretty thick cable
bundles they're going to be, even if they're plain CAT 5e.

> and no switch.
> (Or do they use switches?)
> How much cabling?
> Does it scale well?
> Or will it look like our router on the 3rd floor?
> Anyway, it's not as fun as Homer.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE


From prentice at ias.edu  Wed Feb  9 06:38:50 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 09 Feb 2011 09:38:50 -0500
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
Message-ID: <4D52A6FA.1040701@ias.edu>

I don't think Numascale/ScaleMP has much of a cost advantage anymore.

About 6 months ago, I purchased a couple of Dell PowerEdge R815s with
128 GB of RAM and 32 cores. We looked at similar RAM configurations a
couple of years ago. and the cost premium for that much RAM was
prohibitive (the price for additional RAM seemed to go up exponentially
with the amount of RAM) so we stayed at 32 GB.  This time around, the
premium for the additional RAM seemed marginal. Not sure how large you
can go and keep that "marginal" relationship, but it's much larger than
it was a couple of years ago.

And don't forget to figure in the cost of the interconnect. I don't know
what Numascale requires, but ScaleMP require IB, which can add
significant costs if you don't have an existing IB fabric to use.

Prentice


Douglas Eadline wrote:
> Although I wrote about the SMP and the Numascale hardware,
> I have not yet had a chance to use it.
> 
> To me there are two issues worth considering. First,
> if you need a big SMP this might be a low
> cost solution. Second, should your application
> not scale best in a node-based ccNuma system (ScaleMP
> or Numascale), MPI is still an option. Indeed, no need
> to rewrite the codes. And, management is probably
> much easier.
> 
> Of course for large systems, clusters work best.
> 
> --
> Doug
> 
> 
>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>>> Also look at ScaleMP and Numascale
>>> Here's a damn good article from Doug Eadline:
>>> http://www.linux-mag.com/id/7947
>>>
>>> I must admit though I don't know how far a budget of 30K takes you
>>> there!
>> My understanding is that the Numascale cards cost "a bit more" than IB
>> cards but you don't need a switch (it's a 3D taurus I think) so it would
>> be worth looking into.
>>
>> Rob
>>


From ntmoore at gmail.com  Wed Feb  9 09:20:21 2011
From: ntmoore at gmail.com (Nathan Moore)
Date: Wed, 9 Feb 2011 11:20:21 -0600
Subject: [Beowulf] pseudo-beowulfery
Message-ID: <AANLkTi=9yat2CvFTPiJXinTGd4e_TeOc12_w9zbmehD3@mail.gmail.com>

This isn't quite a beowulf article, but the socio-cultural attitude is
close.  I figure you-all might enjoy it:

http://www.winonadailynews.com/news/local/education/article_fb3f6890-340b-11e0-8309-001cc4c03286.html


LEWISTON, Minn. ? At Lewiston-Altura High School students don't just
use the school's computers, they build them. Virtually all of the
Lewiston-Altura school districts 300-plus PCs have been built or
refurbished by ninth-grade students in Joel Ellinghuysen's
"Introduction to Technology" class.
For the past five years, with the close cooperation of district
information technology manager Gene Berg, students have learned to
install memory, swap out hard drives, replace motherboards and hook up
power supplies while saving the district tens of thousands of dollars
in the process.
Ellinghuysen acknowledged it might seem a bit crazy to wheel in 30
working PCs and let a class of 14- and 15-year-olds fiddle around with
their insides in a most intimate manner.
Each year about one-fifth of the districts's PC inventory is either
rebuilt or replaced with the ninth graders' handiwork.
"We took a chance the first time around," Berg said. "By now they've
done over 300."
"Don't call us crazy," Ellinghuysen said, "call us innovative."
Before a student picks up a screwdriver, Ellinghuysen has covered the
basics of what they will find inside the computer case and reviewed
the basic techniques and precautions they will practice in working
with electronic components.
"They're a little scared to be working with $500 t0 $600 worth of
equipment," he said of the students' initial reaction. "They'll ask,
?You're really going to trust us with that?'"
Support and supervision are key to building confidence and assuring success.
"We're constantly checking for understanding as we go," he said.
Berg said that the neophyte electronic techs are almost always up to the task.
"It's really not that hard once they get over the fear factor of doing
it," he said. "For the most part they just dive in."
Student Jordan Drier said with each step they would "watch and learn"
then repeat what they had seen.
"It was a lot easier than I thought," he said.
Austyn Neldner said he felt he had a much better understanding of how
computers work after rebuilding one himself.
"I'm confident I could do it on my own," he said.
This is where textbook meets tools.
"I believe very much in hands-on learning," Ellinghuysen said. "Where
else could these kids get this opportunity?"
The students actually benefit in more ways than one, both Ellinghuysen
and Berg stress.
This year's class renovated 45 computers for the district elementary
schools, upgrading the memory, motherboard and hard drive as well as
replacing the hard drive mounting bay with one that allows a
technician to change hard drives with a turn of a key - a feature that
allows the computers to be used for online administration of required
state standardized tests. The enterprise-grade parts - much faster and
more durable than those found in the consumer grade machines sold in
retail stores - necessary for the upgrade cost the district about $575
per computer, Berg said. By contrast, an off-the-shelf machine of
comparable capability manufactured by Dell or Hewlett-Packard is
priced at $1,300 - a savings of $34,875 to the school district.
Berg said that by purchasing a case and off-the-shelf components,
students can build a brand new computer for between $550 and $600.
"We get a high performance machine for the price of something you buy
at Wal-Mart," he said.


From Craig.Tierney at noaa.gov  Wed Feb  9 10:21:37 2011
From: Craig.Tierney at noaa.gov (Craig Tierney)
Date: Wed, 09 Feb 2011 11:21:37 -0700
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D52A6FA.1040701@ias.edu>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D52A6FA.1040701@ias.edu>
Message-ID: <4D52DB31.4010203@noaa.gov>

On 2/9/11 7:38 AM, Prentice Bisbal wrote:
> I don't think Numascale/ScaleMP has much of a cost advantage anymore.
>
> About 6 months ago, I purchased a couple of Dell PowerEdge R815s with
> 128 GB of RAM and 32 cores. We looked at similar RAM configurations a
> couple of years ago. and the cost premium for that much RAM was
> prohibitive (the price for additional RAM seemed to go up exponentially
> with the amount of RAM) so we stayed at 32 GB.  This time around, the
> premium for the additional RAM seemed marginal. Not sure how large you
> can go and keep that "marginal" relationship, but it's much larger than
> it was a couple of years ago.
>

There is only so much memory you can put in a box, regardless of the
cost.  ScaleMP/Numascale let you get around that issue.  What if I need
a TB of RAM?  Yes, I might be able to convert my algorithms, but if I can
add $3k per node to get cache-coherent memory, why not go that route?


> And don't forget to figure in the cost of the interconnect. I don't know
> what Numascale requires, but ScaleMP require IB, which can add
> significant costs if you don't have an existing IB fabric to use.
>

ScaleMP is a software solution, Numascale is a cache-coherent hardware solution.
The list cost of the 3D torus version Numascale card with 4GB Ram (highest end card).
Cables are similar to IB, about $120 each.

It would be really cool to build SMP system to sit next to my cluster.  When users
need large memory, or want to do things with UPC or other similar languages,
an SMP system like this would be handy.

- I am not advocating for Numascale, or any other products here.  I am just
trying to provide information to the community.

Craig



> Prentice
>
>
> Douglas Eadline wrote:
>> Although I wrote about the SMP and the Numascale hardware,
>> I have not yet had a chance to use it.
>>
>> To me there are two issues worth considering. First,
>> if you need a big SMP this might be a low
>> cost solution. Second, should your application
>> not scale best in a node-based ccNuma system (ScaleMP
>> or Numascale), MPI is still an option. Indeed, no need
>> to rewrite the codes. And, management is probably
>> much easier.
>>
>> Of course for large systems, clusters work best.
>>
>> --
>> Doug
>>
>>
>>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>>>> Also look at ScaleMP and Numascale
>>>> Here's a damn good article from Doug Eadline:
>>>> http://www.linux-mag.com/id/7947
>>>>
>>>> I must admit though I don't know how far a budget of 30K takes you
>>>> there!
>>> My understanding is that the Numascale cards cost "a bit more" than IB
>>> cards but you don't need a switch (it's a 3D taurus I think) so it would
>>> be worth looking into.
>>>
>>> Rob
>>>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From prentice at ias.edu  Wed Feb  9 11:37:38 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 09 Feb 2011 14:37:38 -0500
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D52DB31.4010203@noaa.gov>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>	<1297091420.1795.27.camel@moelwyn>	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>	<4D52A6FA.1040701@ias.edu>
	<4D52DB31.4010203@noaa.gov>
Message-ID: <4D52ED02.3030800@ias.edu>

Craig Tierney wrote:
> On 2/9/11 7:38 AM, Prentice Bisbal wrote:
>> I don't think Numascale/ScaleMP has much of a cost advantage anymore.
>>
>> About 6 months ago, I purchased a couple of Dell PowerEdge R815s with
>> 128 GB of RAM and 32 cores. We looked at similar RAM configurations a
>> couple of years ago. and the cost premium for that much RAM was
>> prohibitive (the price for additional RAM seemed to go up exponentially
>> with the amount of RAM) so we stayed at 32 GB.  This time around, the
>> premium for the additional RAM seemed marginal. Not sure how large you
>> can go and keep that "marginal" relationship, but it's much larger than
>> it was a couple of years ago.
>>
> 
> There is only so much memory you can put in a box, regardless of the
> cost.  ScaleMP/Numascale let you get around that issue.  What if I need
> a TB of RAM?  Yes, I might be able to convert my algorithms, but if I can
> add $3k per node to get cache-coherent memory, why not go that route?

Agreed. I was going to add the caveat that there are limits to how much
RAM you can add to a single box, or there eventually be a point the
amount of RAM in a single box vs. price is no longer nonlinear, but then
I got lazy. ;)

> 
> 
>> And don't forget to figure in the cost of the interconnect. I don't know
>> what Numascale requires, but ScaleMP require IB, which can add
>> significant costs if you don't have an existing IB fabric to use.
>>
> 
> ScaleMP is a software solution, Numascale is a cache-coherent hardware solution.
> The list cost of the 3D torus version Numascale card with 4GB Ram (highest end card).
> Cables are similar to IB, about $120 each.
> 
> It would be really cool to build SMP system to sit next to my cluster.  When users
> need large memory, or want to do things with UPC or other similar languages,
> an SMP system like this would be handy.
> 
> - I am not advocating for Numascale, or any other products here.  I am just
> trying to provide information to the community.
> 
> Craig
> 
> 
> 
>> Prentice
>>
>>
>> Douglas Eadline wrote:
>>> Although I wrote about the SMP and the Numascale hardware,
>>> I have not yet had a chance to use it.
>>>
>>> To me there are two issues worth considering. First,
>>> if you need a big SMP this might be a low
>>> cost solution. Second, should your application
>>> not scale best in a node-based ccNuma system (ScaleMP
>>> or Numascale), MPI is still an option. Indeed, no need
>>> to rewrite the codes. And, management is probably
>>> much easier.
>>>
>>> Of course for large systems, clusters work best.
>>>
>>> --
>>> Doug
>>>
>>>
>>>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>>>>> Also look at ScaleMP and Numascale
>>>>> Here's a damn good article from Doug Eadline:
>>>>> http://www.linux-mag.com/id/7947
>>>>>
>>>>> I must admit though I don't know how far a budget of 30K takes you
>>>>> there!
>>>> My understanding is that the Numascale cards cost "a bit more" than IB
>>>> cards but you don't need a switch (it's a 3D taurus I think) so it would
>>>> be worth looking into.
>>>>
>>>> Rob
>>>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Prentice Bisbal
Linux Software Support Specialist/System Administrator
School of Natural Sciences
Institute for Advanced Study
Princeton, NJ


From xingqiuyuan at gmail.com  Wed Feb  9 12:21:41 2011
From: xingqiuyuan at gmail.com (xingqiu yuan)
Date: Wed, 9 Feb 2011 15:21:41 -0500
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D52ED02.3030800@ias.edu>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D52A6FA.1040701@ias.edu> <4D52DB31.4010203@noaa.gov>
	<4D52ED02.3030800@ias.edu>
Message-ID: <AANLkTimDdH0MsrGwqv7kFHLho9nKkH8A1+5CZP8KLh0m@mail.gmail.com>

On Wed, Feb 9, 2011 at 2:37 PM, Prentice Bisbal <prentice at ias.edu> wrote:

> Craig Tierney wrote:
> > On 2/9/11 7:38 AM, Prentice Bisbal wrote:
> >> I don't think Numascale/ScaleMP has much of a cost advantage anymore.
> >>
> >> About 6 months ago, I purchased a couple of Dell PowerEdge R815s with
> >> 128 GB of RAM and 32 cores. We looked at similar RAM configurations a
> >> couple of years ago. and the cost premium for that much RAM was
> >> prohibitive (the price for additional RAM seemed to go up exponentially
> >> with the amount of RAM) so we stayed at 32 GB.  This time around, the
> >> premium for the additional RAM seemed marginal. Not sure how large you
> >> can go and keep that "marginal" relationship, but it's much larger than
> >> it was a couple of years ago.
> >>
> >
> > There is only so much memory you can put in a box, regardless of the
> > cost.  ScaleMP/Numascale let you get around that issue.  What if I need
> > a TB of RAM?  Yes, I might be able to convert my algorithms, but if I can
> > add $3k per node to get cache-coherent memory, why not go that route?
>
> >Agreed. I was going to add the caveat that there are limits to how much
> >RAM you can add to a single box, or there eventually be a point the
> >amount of RAM in a single box vs. price is no longer nonlinear, but then
> >I got lazy. ;)
>


Don't understand why you need TB of RAM? Don't forget the fact that $3K can
almost buy one-node!!!



> >
> >
> >> And don't forget to figure in the cost of the interconnect. I don't know
> >> what Numascale requires, but ScaleMP require IB, which can add
> >> significant costs if you don't have an existing IB fabric to use.
> >>
> >
> > ScaleMP is a software solution, Numascale is a cache-coherent hardware
> solution.
> > The list cost of the 3D torus version Numascale card with 4GB Ram
> (highest end card).
> > Cables are similar to IB, about $120 each.
> >
> > It would be really cool to build SMP system to sit next to my cluster.
>  When users
> > need large memory, or want to do things with UPC or other similar
> languages,
> > an SMP system like this would be handy.
> >
> > - I am not advocating for Numascale, or any other products here.  I am
> just
> > trying to provide information to the community.
> >
> > Craig
> >
> >
> >
> >> Prentice
> >>
> >>
> >> Douglas Eadline wrote:
> >>> Although I wrote about the SMP and the Numascale hardware,
> >>> I have not yet had a chance to use it.
> >>>
> >>> To me there are two issues worth considering. First,
> >>> if you need a big SMP this might be a low
> >>> cost solution. Second, should your application
> >>> not scale best in a node-based ccNuma system (ScaleMP
> >>> or Numascale), MPI is still an option. Indeed, no need
> >>> to rewrite the codes. And, management is probably
> >>> much easier.
> >>>
> >>> Of course for large systems, clusters work best.
> >>>
> >>> --
> >>> Doug
> >>>
> >>>
> >>>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
> >>>>> Also look at ScaleMP and Numascale
> >>>>> Here's a damn good article from Doug Eadline:
> >>>>> http://www.linux-mag.com/id/7947
> >>>>>
> >>>>> I must admit though I don't know how far a budget of 30K takes you
> >>>>> there!
> >>>> My understanding is that the Numascale cards cost "a bit more" than IB
> >>>> cards but you don't need a switch (it's a 3D taurus I think) so it
> would
> >>>> be worth looking into.
> >>>>
> >>>> Rob
> >>>>
> >> _______________________________________________
> >> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> Computing
> >> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> >
>
> --
> Prentice Bisbal
> Linux Software Support Specialist/System Administrator
> School of Natural Sciences
> Institute for Advanced Study
> Princeton, NJ
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110209/4b10b8d0/attachment.html>

From prentice at ias.edu  Wed Feb  9 12:58:32 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 09 Feb 2011 15:58:32 -0500
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <AANLkTimDdH0MsrGwqv7kFHLho9nKkH8A1+5CZP8KLh0m@mail.gmail.com>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>	<1297091420.1795.27.camel@moelwyn>	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>	<4D52A6FA.1040701@ias.edu>	<4D52DB31.4010203@noaa.gov>	<4D52ED02.3030800@ias.edu>
	<AANLkTimDdH0MsrGwqv7kFHLho9nKkH8A1+5CZP8KLh0m@mail.gmail.com>
Message-ID: <4D52FFF8.9080409@ias.edu>



xingqiu yuan wrote:
> 
> 
> On Wed, Feb 9, 2011 at 2:37 PM, Prentice Bisbal <prentice at ias.edu
> <mailto:prentice at ias.edu>> wrote:
> 
>     Craig Tierney wrote:
>     > On 2/9/11 7:38 AM, Prentice Bisbal wrote:
>     >> I don't think Numascale/ScaleMP has much of a cost advantage anymore.
>     >>
>     >> About 6 months ago, I purchased a couple of Dell PowerEdge R815s with
>     >> 128 GB of RAM and 32 cores. We looked at similar RAM configurations a
>     >> couple of years ago. and the cost premium for that much RAM was
>     >> prohibitive (the price for additional RAM seemed to go up
>     exponentially
>     >> with the amount of RAM) so we stayed at 32 GB.  This time around, the
>     >> premium for the additional RAM seemed marginal. Not sure how
>     large you
>     >> can go and keep that "marginal" relationship, but it's much
>     larger than
>     >> it was a couple of years ago.
>     >>
>     >
>     > There is only so much memory you can put in a box, regardless of the
>     > cost.  ScaleMP/Numascale let you get around that issue.  What if I
>     need
>     > a TB of RAM?  Yes, I might be able to convert my algorithms, but
>     if I can
>     > add $3k per node to get cache-coherent memory, why not go that route?
> 
>     >Agreed. I was going to add the caveat that there are limits to how much
>     >RAM you can add to a single box, or there eventually be a point the
>     >amount of RAM in a single box vs. price is no longer nonlinear, but
>     then
>     >I got lazy. ;)
> 
> 
> 
> Don't understand why you need TB of RAM? Don't forget the fact that $3K
> can almost buy one-node!!! 
> 

Certain problems require more RAM than others. I worked with an
astrophysicist whose work involved modelling the motion of planets. He
didn't even need RAM for his work - all of the data would fit in the
processor's cache.

On the other hand, genomics research is very data intensive, and can
involve searching for one really long string in millions (billions?) of
other long strings. This will require much more memory. If you can fit
all those other strings into RAM, the pattern searching will go much
faster.

-- 
Prentice


From Craig.Tierney at noaa.gov  Wed Feb  9 16:37:36 2011
From: Craig.Tierney at noaa.gov (Craig Tierney)
Date: Wed, 09 Feb 2011 17:37:36 -0700
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <AANLkTimDdH0MsrGwqv7kFHLho9nKkH8A1+5CZP8KLh0m@mail.gmail.com>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D52A6FA.1040701@ias.edu> <4D52DB31.4010203@noaa.gov>
	<4D52ED02.3030800@ias.edu>
	<AANLkTimDdH0MsrGwqv7kFHLho9nKkH8A1+5CZP8KLh0m@mail.gmail.com>
Message-ID: <4D533350.5010404@noaa.gov>

On 2/9/11 1:21 PM, xingqiu yuan wrote:
>
>
> On Wed, Feb 9, 2011 at 2:37 PM, Prentice Bisbal <prentice at ias.edu
> <mailto:prentice at ias.edu>> wrote:
>
>     Craig Tierney wrote:
>     >  On 2/9/11 7:38 AM, Prentice Bisbal wrote:
>     > > I don't think Numascale/ScaleMP has much of a cost advantage anymore.
>     > >
>     > > About 6 months ago, I purchased a couple of Dell PowerEdge R815s with
>     > > 128 GB of RAM and 32 cores. We looked at similar RAM configurations a
>     > > couple of years ago. and the cost premium for that much RAM was
>     > > prohibitive (the price for additional RAM seemed to go up
>     exponentially
>     > > with the amount of RAM) so we stayed at 32 GB.  This time around, the
>     > > premium for the additional RAM seemed marginal. Not sure how
>     large you
>     > > can go and keep that "marginal" relationship, but it's much
>     larger than
>     > > it was a couple of years ago.
>     > >
>     >
>     >  There is only so much memory you can put in a box, regardless of the
>     >  cost.  ScaleMP/Numascale let you get around that issue.  What if I
>     need
>     >  a TB of RAM?  Yes, I might be able to convert my algorithms, but
>     if I can
>     >  add $3k per node to get cache-coherent memory, why not go that route?
>
>     >Agreed. I was going to add the caveat that there are limits to how much
>     >RAM you can add to a single box, or there eventually be a point the
>     >amount of RAM in a single box vs. price is no longer nonlinear, but then
>     >I got lazy. ;)
>
>
>
> Don't understand why you need TB of RAM? Don't forget the fact that $3K
> can almost buy one-node!!!
>
>

Reasons for a TB of RAM include databases, a lot of users that need 10's of
GB, but also having the additional cores provides some flexibility.

I can't say for certain a setup like this would be beneficial, but at $3k per
node, I would be willing to try it out!

You are right about the node price.  Well, I would put the cost much higher.
You could put 48GB in a node without much of a price penalty, and then you
are probably paying $5-6K per node.  Also, the $3k/node is list and who
ever pays list? :-)

Craig




>
>     >
>     >
>     > > And don't forget to figure in the cost of the interconnect. I
>     don't know
>     > > what Numascale requires, but ScaleMP require IB, which can add
>     > > significant costs if you don't have an existing IB fabric to use.
>     > >
>     >
>     >  ScaleMP is a software solution, Numascale is a cache-coherent
>     hardware solution.
>     >  The list cost of the 3D torus version Numascale card with 4GB Ram
>     (highest end card).
>     >  Cables are similar to IB, about $120 each.
>     >
>     >  It would be really cool to build SMP system to sit next to my
>     cluster.  When users
>     >  need large memory, or want to do things with UPC or other similar
>     languages,
>     >  an SMP system like this would be handy.
>     >
>     >  - I am not advocating for Numascale, or any other products here.
>       I am just
>     >  trying to provide information to the community.
>     >
>     >  Craig
>     >
>     >
>     >
>     > > Prentice
>     > >
>     > >
>     > > Douglas Eadline wrote:
>     > >> Although I wrote about the SMP and the Numascale hardware,
>     > >> I have not yet had a chance to use it.
>     > >>
>     > >> To me there are two issues worth considering. First,
>     > >> if you need a big SMP this might be a low
>     > >> cost solution. Second, should your application
>     > >> not scale best in a node-based ccNuma system (ScaleMP
>     > >> or Numascale), MPI is still an option. Indeed, no need
>     > >> to rewrite the codes. And, management is probably
>     > >> much easier.
>     > >>
>     > >> Of course for large systems, clusters work best.
>     > >>
>     > >> --
>     > >> Doug
>     > >>
>     > >>
>     > >>> On Thu, 2011-02-03 at 09:45 +0000, Hearns, John wrote:
>     > >>>> Also look at ScaleMP and Numascale
>     > >>>> Here's a damn good article from Doug Eadline:
>     > >>>> http://www.linux-mag.com/id/7947
>     > >>>>
>     > >>>> I must admit though I don't know how far a budget of 30K takes you
>     > >>>> there!
>     > >>> My understanding is that the Numascale cards cost "a bit more"
>     than IB
>     > >>> cards but you don't need a switch (it's a 3D taurus I think) so
>     it would
>     > >>> be worth looking into.
>     > >>>
>     > >>> Rob
>     > >>>
>     > > _______________________________________________
>     > > Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     > > To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>     >  _______________________________________________
>     >  Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     >  To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>     >
>
>     --
>     Prentice Bisbal
>     Linux Software Support Specialist/System Administrator
>     School of Natural Sciences
>     Institute for Advanced Study
>     Princeton, NJ
>     _______________________________________________
>     Beowulf mailing list, Beowulf at beowulf.org
>     <mailto:Beowulf at beowulf.org> sponsored by Penguin Computing
>     To change your subscription (digest mode or unsubscribe) visit
>     http://www.beowulf.org/mailman/listinfo/beowulf
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From samuel at unimelb.edu.au  Wed Feb  9 19:23:36 2011
From: samuel at unimelb.edu.au (Christopher Samuel)
Date: Thu, 10 Feb 2011 14:23:36 +1100
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <AANLkTimDdH0MsrGwqv7kFHLho9nKkH8A1+5CZP8KLh0m@mail.gmail.com>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D52A6FA.1040701@ias.edu> <4D52DB31.4010203@noaa.gov>
	<4D52ED02.3030800@ias.edu>
	<AANLkTimDdH0MsrGwqv7kFHLho9nKkH8A1+5CZP8KLh0m@mail.gmail.com>
Message-ID: <4D535A38.9070302@unimelb.edu.au>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/02/11 07:21, xingqiu yuan wrote:

> Don't understand why you need TB of RAM?

I've heard 1TB of RAM quoted by the bioinfomatics people
as the amount of RAM needed to do de-novo reassembly of
the human genome with Velvet (which is a single threaded
application).

cheers,
Chris
- -- 
    Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
         http://www.vlsci.unimelb.edu.au/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1TWjgACgkQO2KABBYQAh9sfgCfbD2bNa1iHs+CBynXi7j913jN
n70An1JQ4byJGGDHGdhwx7R/6G+F8HNr
=V3Ko
-----END PGP SIGNATURE-----


From stuartb at 4gh.net  Thu Feb 10 10:52:52 2011
From: stuartb at 4gh.net (Stuart Barkley)
Date: Thu, 10 Feb 2011 13:52:52 -0500 (EST)
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <4D52A6FA.1040701@ias.edu>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D52A6FA.1040701@ias.edu>
Message-ID: <alpine.BSF.2.00.1102101041120.3826@freeman.4gh.net>

On Wed, 9 Feb 2011 at 09:38 -0000, Prentice Bisbal wrote:

> About 6 months ago, I purchased a couple of Dell PowerEdge R815s
> with 128 GB of RAM and 32 cores. We looked at similar RAM
> configurations a couple of years ago. and the cost premium for that
> much RAM was prohibitive (the price for additional RAM seemed to go
> up exponentially with the amount of RAM) so we stayed at 32 GB.
> This time around, the premium for the additional RAM seemed
> marginal. Not sure how large you can go and keep that "marginal"
> relationship, but it's much larger than it was a couple of years
> ago.

How much does the RAM speed matter to people?

We have a lot of 2 CPU (8 core) systems and have been considering
memory upgrades (or just purchasing new systems with more memory since
we are still growing).

Currently we are running 1 4GB DIMM per channel at 1333 MHz.  If we
add a second 4GB DIMM per channel then the memory will run at 1066 MHz
(and to 800 MHz with 3 DIMMs per channel).  Alternatively, we could
replace the 4GB modules with 8GB ones (keeping the old ones for spares
or to side-grade some other machines to 48GB/1066 MHz).

Have people been worrying about the lower memory bus rate with
multiple DIMMs on each channel?  Do "ordinary" applications notice the
difference?

I know it depends upon the application and I have no idea if this will
matter to our applications.  Mostly, I just happy things are running
at all rather than fighting every ounce of performance.

Stuart Barkley
-- 
I've never been lost; I was once bewildered for three days, but never lost!
                                        --  Daniel Boone


From Craig.Tierney at noaa.gov  Thu Feb 10 12:18:10 2011
From: Craig.Tierney at noaa.gov (Craig Tierney)
Date: Thu, 10 Feb 2011 13:18:10 -0700
Subject: [Beowulf] Advice on 4 CPU node configurations
In-Reply-To: <alpine.BSF.2.00.1102101041120.3826@freeman.4gh.net>
References: <AANLkTimcMBs6ZHZKmbb-=1ZjDTJFzPz_nQ1HRaxqCE0i@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B12874DEA@milexchmb1.mil.tagmclarengroup.com>
	<1297091420.1795.27.camel@moelwyn>
	<42442.192.168.93.213.1297190076.squirrel@mail.eadline.org>
	<4D52A6FA.1040701@ias.edu>
	<alpine.BSF.2.00.1102101041120.3826@freeman.4gh.net>
Message-ID: <4D544802.3000207@noaa.gov>

On 2/10/11 11:52 AM, Stuart Barkley wrote:
> On Wed, 9 Feb 2011 at 09:38 -0000, Prentice Bisbal wrote:
>
>> About 6 months ago, I purchased a couple of Dell PowerEdge R815s
>> with 128 GB of RAM and 32 cores. We looked at similar RAM
>> configurations a couple of years ago. and the cost premium for that
>> much RAM was prohibitive (the price for additional RAM seemed to go
>> up exponentially with the amount of RAM) so we stayed at 32 GB.
>> This time around, the premium for the additional RAM seemed
>> marginal. Not sure how large you can go and keep that "marginal"
>> relationship, but it's much larger than it was a couple of years
>> ago.
>
> How much does the RAM speed matter to people?
>
> We have a lot of 2 CPU (8 core) systems and have been considering
> memory upgrades (or just purchasing new systems with more memory since
> we are still growing).
>
> Currently we are running 1 4GB DIMM per channel at 1333 MHz.  If we
> add a second 4GB DIMM per channel then the memory will run at 1066 MHz
> (and to 800 MHz with 3 DIMMs per channel).  Alternatively, we could
> replace the 4GB modules with 8GB ones (keeping the old ones for spares
> or to side-grade some other machines to 48GB/1066 MHz).
>
> Have people been worrying about the lower memory bus rate with
> multiple DIMMs on each channel?  Do "ordinary" applications notice the
> difference?
>

Our weather/hurricane codes are memory bound, so yes we worry about
it.  We aren't installing so much memory that we are paying a price
premium to ensure only one channel is filled, so it isn't a big deal.

Have we tested it?  No.  But we haven't had to.

Craig



> I know it depends upon the application and I have no idea if this will
> matter to our applications.  Mostly, I just happy things are running
> at all rather than fighting every ounce of performance.
>
> Stuart Barkley


From mathog at caltech.edu  Thu Feb 10 12:21:54 2011
From: mathog at caltech.edu (David Mathog)
Date: Thu, 10 Feb 2011 12:21:54 -0800
Subject: [Beowulf] Advice on 4 CPU node configurations
Message-ID: <E1Pnd1W-0003Vi-26@mendel.bio.caltech.edu>

Christopher Samuel wrote:
> I've heard 1TB of RAM quoted by the bioinfomatics people
> as the amount of RAM needed to do de-novo reassembly of
> the human genome with Velvet (which is a single threaded
> application).

That would be an expensive thing to do though, much more efficient to
paint the new reads onto the existing consensus and then go back and
deal with any discrepancies.

Anyway, the Velvet memory usage estimator is described here:

http://listserver.ebi.ac.uk/pipermail/velvet-users/2009-July/000474.html

Velvet is for assembling from short reads.  Short reads are easy to get
in large numbers but are cruddy for fully assembling large genomes de
novo, since genomes are full of high copy DNA longer than the reads.  So
the de novo sequence would come out in a lot of disconnected chunks
which would have to mapped back onto the reference sequence anyway.

Assuming reads of 100 ( bp), a genome of 3000 (in MB, rounded down to
the nearest billion), numreads = genome size/read size * 20/1000000
(reads in millions, 20X over sequenced) plugging into that formula gives:

 -109635 + 18977*100 + 86326*3000 + 233353*(3000000000/100)*20/1000000 -
51092*31

        399194013 Kb

Roughly 381 Gb (if I didn't typo anything).  Most of that is in the
hashes, where each base from a read is included in 31 different hashes,
once in each position 1->31 (the hash is calculated on a sliding window
that increments by 1, not by 31).  Effectively the hash takes the
60Gbases of raw sequence and expands it 31X.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech


From deadline at eadline.org  Mon Feb 14 08:59:31 2011
From: deadline at eadline.org (Douglas Eadline)
Date: Mon, 14 Feb 2011 11:59:31 -0500 (EST)
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <E1Pnd1W-0003Vi-26@mendel.bio.caltech.edu>
References: <E1Pnd1W-0003Vi-26@mendel.bio.caltech.edu>
Message-ID: <59778.192.168.93.213.1297702771.squirrel@mail.eadline.org>


For those who are not aware, tonight is the face-off between
man and machine (it is not live, however). This is no chess game.
Today through Wednesday, on the game show Jeopardy (CBS Network)
the best human players will attempt to beat Watson, the 2880 core
IBM Jeopardy computer.

Small write up on ClusterMonkey with some links:

  http://www.clustermonkey.net//content/view/290/1/

I predict one more defeat for the carbon based life forms.

--
Doug

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.



From james.p.lux at jpl.nasa.gov  Mon Feb 14 11:52:19 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Mon, 14 Feb 2011 11:52:19 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <59778.192.168.93.213.1297702771.squirrel@mail.eadline.org>
References: <E1Pnd1W-0003Vi-26@mendel.bio.caltech.edu>,
	<59778.192.168.93.213.1297702771.squirrel@mail.eadline.org>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A4717A@ALTPHYEMBEVSP20.RES.AD.JPL>

should be interesting..

having been on the show( cue Al Yankovic "I lost on Jeopardy")...a lot of it is timing in pressing the button when the (hidden to the tv audience) light goes on.  the published descriptions are a bit sketchy on how Watson pushes the button.  it could have microsecond reaction times.

typically, you've read the answer and already know your response and then you wait for the light. (or at least, have a good idea you'll know it within 10 seconds or so)

You buzz in, and have a couple second to gather your thoughts to come up with the response . 

(I was the only contestant to get the Final Jeopardy right.. but not enough to overcome the leader.. he had more than twice as much cash as i did)


________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of Douglas Eadline [deadline at eadline.org]
Sent: Monday, February 14, 2011 08:59
To: beowulf at beowulf.org
Subject: [Beowulf] IBM's Watson on Jeopardy tonight

For those who are not aware, tonight is the face-off between
man and machine (it is not live, however). This is no chess game.
Today through Wednesday, on the game show Jeopardy (CBS Network)
the best human players will attempt to beat Watson, the 2880 core
IBM Jeopardy computer.

Small write up on ClusterMonkey with some links:

  http://www.clustermonkey.net//content/view/290/1/

I predict one more defeat for the carbon based life forms.

--
Doug

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From deadline at eadline.org  Tue Feb 15 06:25:41 2011
From: deadline at eadline.org (Douglas Eadline)
Date: Tue, 15 Feb 2011 09:25:41 -0500 (EST)
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0A4717A@ALTPHYEMBEVSP20.RES.AD.JPL
	>
References: <E1Pnd1W-0003Vi-26@mendel.bio.caltech.edu>,
	<59778.192.168.93.213.1297702771.squirrel@mail.eadline.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A4717A@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <53153.192.168.93.213.1297779941.squirrel@mail.eadline.org>

Wow, I would be interested in your view.

BTW, I just posted some thoughts on
http://clustermonkey.net (I'll be posting
after each day) I think IBM is doing
a good job of educating viewers about
what is going on (and what is not)

I was pleased to note that Watson got the Beowulf
question right (Who is Grendel?)

--
Doug



> should be interesting..
>
> having been on the show( cue Al Yankovic "I lost on Jeopardy")...a lot of
> it is timing in pressing the button when the (hidden to the tv audience)
> light goes on.  the published descriptions are a bit sketchy on how Watson
> pushes the button.  it could have microsecond reaction times.
>
> typically, you've read the answer and already know your response and then
> you wait for the light. (or at least, have a good idea you'll know it
> within 10 seconds or so)
>
> You buzz in, and have a couple second to gather your thoughts to come up
> with the response .
>
> (I was the only contestant to get the Final Jeopardy right.. but not
> enough to overcome the leader.. he had more than twice as much cash as i
> did)
>
>
> ________________________________________
> From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf
> Of Douglas Eadline [deadline at eadline.org]
> Sent: Monday, February 14, 2011 08:59
> To: beowulf at beowulf.org
> Subject: [Beowulf] IBM's Watson on Jeopardy tonight
>
> For those who are not aware, tonight is the face-off between
> man and machine (it is not live, however). This is no chess game.
> Today through Wednesday, on the game show Jeopardy (CBS Network)
> the best human players will attempt to beat Watson, the 2880 core
> IBM Jeopardy computer.
>
> Small write up on ClusterMonkey with some links:
>
>   http://www.clustermonkey.net//content/view/290/1/
>
> I predict one more defeat for the carbon based life forms.
>
> --
> Doug
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


--
Doug

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.



From mathog at caltech.edu  Tue Feb 15 13:55:56 2011
From: mathog at caltech.edu (David Mathog)
Date: Tue, 15 Feb 2011 13:55:56 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
Message-ID: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>

Jim Lux pointed out in an earlier post the reaction time element on
buzzing in from the (invisible to the audience) "go" light.  I only
watched about 10 minutes of this, but my impression was that the machine
did have a reaction time advantage.  Alex T. was a little vague on how
the machine was fed the question and if he explained what its equivalent
of the "go" light is I missed it.  It could be decisive - sending the
entire question to the machine at the instant it became visible to the
humans would give the machine an advantage, since it could digest the
message before the contestants eyes could even track across the first
line of text, let alone until they heard the entire question. In any
case, for the part I watched the machine was beating the humans to the
buzzer pretty consistently.  It was cute, but a little silly, that the
machine had to push the button with a mechanical plunger.  Unless they
built a typical human reaction delay into that mechanism one would
expect the plunger to be much faster than a thumb.

In terms of score/watt the humans had the machine beat by a mile :-).

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech


From eagles051387 at gmail.com  Wed Feb 16 00:13:39 2011
From: eagles051387 at gmail.com (Jonathan Aquilina)
Date: Wed, 16 Feb 2011 09:13:39 +0100
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
Message-ID: <4D5B8733.1010505@gmail.com>

http://edition.cnn.com/2011/TECH/innovation/02/14/jeopardy.ibm.watson/index.html

this is quite an interesting read.

question becomes are computers just as smart as we are?

I'm quite surprised that the computer doesn't have a massive lead

On 2/15/11 10:55 PM, David Mathog wrote:
> Jim Lux pointed out in an earlier post the reaction time element on
> buzzing in from the (invisible to the audience) "go" light.  I only
> watched about 10 minutes of this, but my impression was that the machine
> did have a reaction time advantage.  Alex T. was a little vague on how
> the machine was fed the question and if he explained what its equivalent
> of the "go" light is I missed it.  It could be decisive - sending the
> entire question to the machine at the instant it became visible to the
> humans would give the machine an advantage, since it could digest the
> message before the contestants eyes could even track across the first
> line of text, let alone until they heard the entire question. In any
> case, for the part I watched the machine was beating the humans to the
> buzzer pretty consistently.  It was cute, but a little silly, that the
> machine had to push the button with a mechanical plunger.  Unless they
> built a typical human reaction delay into that mechanism one would
> expect the plunger to be much faster than a thumb.
>
> In terms of score/watt the humans had the machine beat by a mile :-).
>
> David Mathog
> mathog at caltech.edu
> Manager, Sequence Analysis Facility, Biology Division, Caltech
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



From geoff at galitz.org  Wed Feb 16 00:24:20 2011
From: geoff at galitz.org (Geoff Galitz)
Date: Wed, 16 Feb 2011 09:24:20 +0100
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5B8733.1010505@gmail.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>
Message-ID: <50888AA2846A4ECF9332AFB80EFB87D5@USERPC>

> http://edition.cnn.com/2011/TECH/innovation/02/14/jeopardy.ibm.watson/index.html
>
> this is quite an interesting read.
>
> question becomes are computers just as smart as we are?
>
> I'm quite surprised that the computer doesn't have a massive lead




Science Friday did a segment on this which is quite good and, of course, 
more informative for us techi/science guys:

http://www.sciencefriday.com/program/archives/201102112 (listen to the 
podcast segment)

Smart is always a tough thing to measure, even in people.  Computers have a 
very long way to go if you want to measure them to humans.



 



From eagles051387 at gmail.com  Wed Feb 16 00:26:53 2011
From: eagles051387 at gmail.com (Jonathan Aquilina)
Date: Wed, 16 Feb 2011 09:26:53 +0100
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
Message-ID: <4D5B8A4D.80101@gmail.com>

To be tied with a human though i feel means they are on par with humans.

On 2/16/11 9:24 AM, Geoff Galitz wrote:
>> http://edition.cnn.com/2011/TECH/innovation/02/14/jeopardy.ibm.watson/index.html
>>
>> this is quite an interesting read.
>>
>> question becomes are computers just as smart as we are?
>>
>> I'm quite surprised that the computer doesn't have a massive lead
>
>
>
> Science Friday did a segment on this which is quite good and, of course,
> more informative for us techi/science guys:
>
> http://www.sciencefriday.com/program/archives/201102112 (listen to the
> podcast segment)
>
> Smart is always a tough thing to measure, even in people.  Computers have a
> very long way to go if you want to measure them to humans.
>
>
>
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf



From geoff at galitz.org  Wed Feb 16 00:53:05 2011
From: geoff at galitz.org (Geoff Galitz)
Date: Wed, 16 Feb 2011 09:53:05 +0100
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5B8A4D.80101@gmail.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>	<4D5B8733.1010505@gmail.com><50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>
Message-ID: <D90CDD14CCC74FF2A86F8DF1A0D243A7@USERPC>



> To be tied with a human though i feel means they are on par with humans.
>

I've always thought that our enthusiasm (or for some, fear) for technology 
has clouded our opinion on AI to human parity.  In the case of Deep Blue and 
Watson, the system was designed to do one or a small set of operations well, 
but they are no where as well rounded and multi-functional as we are.  The 
human brain for example:

- processes  input from multiple input sensors and sensor types
- regulates many body functions
- records data in realtime
- retrieves data
- can self-heal
- can adapt to new circumstances

and then there are the higher level functions:

- language
- music
- abstract thinking

And this is all done in a space that is a fraction of the size of Watson or 
Deep Blue.


I'd be quiet interested in working on a project that actually tried 
integrate all of those functions together.  That would be fun!



 



From eugen at leitl.org  Wed Feb 16 01:41:59 2011
From: eugen at leitl.org (Eugen Leitl)
Date: Wed, 16 Feb 2011 10:41:59 +0100
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5B8A4D.80101@gmail.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>
Message-ID: <20110216094159.GO23560@leitl.org>

On Wed, Feb 16, 2011 at 09:26:53AM +0100, Jonathan Aquilina wrote:
> To be tied with a human though i feel means they are on par with humans.

Here's a slightly different take on the matter

http://www.sciencemag.org/content/early/2011/02/09/science.1200970.abstract


Perspectives 
To put our findings in perspective, the 6.4*10 18  
instructions per second that human kind can carry out on its 
general-purpose computers in 2007 are in the same ballpark 
area as the maximum number of nerve impulses executed by 
one human brain per second (10 17 ) (36). The 2.4*10 21  bits 
stored by humanity in all of its technological devices in 2007 
is approaching order of magnitude of the roughly 10 23  bits 
stored in the DNA of a human adult (37), but it is still 
minuscule compared to the 10 90  bits stored in the observable 
universe (38). However, in contrast to natural information 
processing, the world?s technological information processing 
capacities are quickly growing at clearly exponential rates. 
 
/ www.sciencexpress.org / 10 February 2011 / Page 1 / 10.1126/science.1200970 
We estimate the world?s technological capacity to store, 
communicate, and compute information, tracking 60 
analog and digital technologies during the period from 
1986 to 2007. In 2007, humankind was able to store 2.9 x 
10
20
 optimally compressed bytes, communicated almost 2 
x 10
21
 bytes, and carry out 6.4 x 10
18
 instructions per 
second on general-purpose computers. General-purpose 
computing capacity grew at an annual rate of 58%. The 
world?s capacity for bidirectional telecommunication 
grew at 28% per year, closely followed by the increase in 
globally stored information (23%). Humankind?s capacity 
for unidirectional information diffusion through 
broadcasting channels has experienced comparatively 
modest annual growth (6%). Telecommunication has been 
dominated by digital technologies since 1990 (99.9% in 
digital format in 2007) and the majority of our 
technological memory has been in digital format since the 
early 2000s (94% digital in 2007). 
Leading social scientists have recognized that we are living 
through an age in which ?the generation of wealth, the 
exercise of power, and the creation of cultural codes came to 
depend on the technological capacity of societies and 
individuals, with information technologies as the core of this 
capacity? (1). Despite this insight, most evaluations of 
society?s technological capacity to handle information are 
based on either qualitative assessments or indirect 
approximations, such as the stock of installed devices or the 
economic value of related products and services (2?9). 
Previous work 
Some pioneering studies have taken a more direct 
approach to quantify the amount of information that society 
processes with its information and communication 
technologies (ICT). Following pioneering work in Japan (10), 
Pool (11) estimated the growth trends of the ?amount of 
words? transmitted by 17 major communications media in the 
United States from 1960 to 1977. This study was the first to 
show empirically the declining relevance of print media with 
respect to electronic media. In 1997, Lesk (12) asked ?how 
much information is there in the world?? and presented a 
brief outline on how to go about estimating the global 
information storage capacity. A group of researchers at the 
University of California, at Berkeley, took up the 
measurement challenge between 2000 and 2003 (13). Their 
focus on ?uniquely created? information resulted in the 
conclusion that ?most of the total volume of new information 
flows is derived from the volume of voice telephone traffic, 
most of which is unique content? (97%); as broadcasted 
television and most information storage mainly consists of 
duplicate information, these omnipresent categories 
contributed relatively little. A storage company hired a 
private sector research firm (International Data Corporation, 
IDC) to estimate the global hardware capacity of digital ICT 
for the years 2007-2008 (14). For digital storage, IDC 
estimates that in 2007 ?all the empty or usable space on hard 
drives, tapes, CDs, DVDs, and memory (volatile and 
nonvolatile) in the market equaled 264 exabytes? (14). 
During 2008, an industry and university collaboration 
explicitly focused on information consumption (15), 
measured in hardware capacity, words, and hours. The results 
are highly reliant on media time-budget studies, which 
estimate how many hours people interact with a media 
device. Not surprisingly, the result obtained with this 
methodology was that computer games and movies represent 
99.2% of the total amount of data ?consumed?. 
Scope of our exercise 
To reconcile these different results, we focus on the 
world?s technological capacity to handle information. We do 
not account for uniqueness of information, since it is very 
difficult to differentiate between truly new and merely 
recombined, duplicate information. Instead we assume that all 
information has some relevance for some individual. Aside 
from the traditional focus on the transmission through space 
(communication) and time (storage), we also consider the 
computation of information. We define storage as the 
The World?s Technological Capacity to Store, Communicate, and Compute 
Information 
Martin Hilbert 1*
 and Priscila L?pez 2
 
1 University of Southern California (USC), Annenberg School of Communication; United Nations ECLAC.  2 Open University of 
Catalonia (UOC). 
*To whom correspondence should be addressed. E-mail: mhilbert at usc.edu 
 
 on February 16, 2011 www.sciencemag.org Downloaded from  
 
/ www.sciencexpress.org / 10 February 2011 / Page 2 / 10.1126/science.1200970 
maintenance of information over a considerable amount of 
time for explicit later retrieval and estimate the installed 
(available) capacity. We do not consider volatile storage in 
the respective inventory (such as RAM), since the ultimate 
end of volatile memory is computation, not storage per se. 
Communication is defined as the amount of information that 
is effectively received or sent by the user, while being 
transmitted over a considerable distance (outside the local 
area). This includes those transmissions whose main purpose 
consists in the overcoming of distances, not the local sharing 
of information (such as the distribution of copies at a 
meeting, or communication through private local area 
networks). We take inventory of the effective communication 
capacity (the actual amount of bits transmitted). We define 
computation as the meaningful transformation of information 
and estimate the installed (available) capacity. 
More precisely, as shown in Fig. 1, we distinguishamong: 
storage of information in bits; unidirectional diffusion 
through broadcasting in bits per second; bidirectional 
telecommunication in bits per second; computation of 
information by general purpose computers in instructions per 
second (or MIPS); and (the estimated computational capacity 
of a selected sample of application-specific devices (MIPS). 
While previous studies tracked some two or three dozen 
categories of ICT over three consecutive years at most, our 
study encompasses worldwide estimates for 60 categories (21 
analog and 39 digital) and spans over two decades (1986-
2007). 
We obtain the technological capacity by basically 
multiplying the number of installed technological devices 
with their respective performances. All estimates are yearly 
averages, but we adjust for the fact that the installed 
technological stock of a given year is the result of an 
accumulation process of previous years, whereas each year?s 
technologies contribute with different performance rates. We 
used 1,120 sources and explain our assumptions in detail in 
Supporting Online Material (16). The statistics we rely on 
include databases from international organizations [such as 
(17?22)], historical inventories from individuals for 
commercial or academic purposes [such as (23?26)], publicly 
available statistics from private research firms [such as (27, 
28)], as well as a myriad of sales and product specifications 
from equipment producers. We filled in occasional blanks 
with either linear or exponential interpolations, depending on 
the nature of the process in question. Frequently we compared 
diverse sources for the same phenomena and strove for 
reasonable middle grounds in case of contradictions. In cases 
where specific country data were not available, we aimed for 
a globally balanced outlook by creating at least two 
international profiles, one for the ?developed? member 
countries of the Organisation for Economic Co-operation and 
Development (OECD), and another one for the rest of the 
world. 
Information, not hardware with redundant data 
Although the estimation of the global hardware capacity 
for information storage and communication is of interest for 
the ICT industry (14), we are more interested in the amount 
of information that is handled by this hardware. Therefore, 
we converted the data contained in storage and 
communication hardware capacity into informational bits by 
normalizing on compression rates. This addresses the fact that 
information sources have different degrees of redundancy. 
The redundancy (or predictability) of the source is primarily 
determined by the content in question, such as text, images, 
audio or video (29, 30). Considering the kind of content, we 
measured information as if all redundancy were removed with 
the most efficient compression algorithms available in 2007 
(we call this level of compression ?optimally compressed?). 
Shannon (29) showed that the uttermost compression of 
information approximates the entropy of the source, which 
unambiguously quantifies the amount of information 
contained in the message. In an information theoretic sense 
(30), information is defined as the opposite of uncertainty. 
Shannon (29) defined one bit as the amount of information 
that reduces uncertainty by half (regarding a given probability 
space, such as letters from an alphabet or pixels from a color 
scale). This definition is independent of the specific task or 
content. For example, after normalization on optimally 
compressed bits we can say things like ?a 6 square-cm 
newspaper image is worth a 1000 words?, because both 
require the same average number of binary yes/no decisions 
to resolve the same amount of uncertainty. 
Normalization on compression rates is essential for 
comparing the informational performance of analog and 
digital technologies. It is also indispensable for obtaining 
meaningful time series of digital technologies, since more 
efficient compression algorithms enable us to handle more 
information with the same amount of hardware. For example, 
we estimate that a hard disk with a hardware performance of 
1 MB for video storage was holding the equivalent of 1 
optimally compressed MB in 2007 (?optimally compressed? 
with MPEG-4), but only 0.45 optimally compressed MB in 
2000 (compressed with MPEG-1), 0.33 in 1993 (compressed 
with cinepack) and merely 0.017 optimally compressed MB 
in 1986 (supposing that no compression algorithms were 
used). Given that statistics on the most commonly used 
compression algorithms are scarce, we limited our 
estimations of information storage and communication to the 
years 1986, 1993, 2000 and 2007 (see Supporting Online 
Material, 16, B Compression). 
Conventionally bits are abbreviated with a small ?b? (such 
as in kilobits per second: kbps) and bytes (equal to 8 bits) 
with a capital ?B? (such as in Megabyte: MB). Standard 
 on February 16, 2011 www.sciencemag.org Downloaded from  
 
/ www.sciencexpress.org / 10 February 2011 / Page 3 / 10.1126/science.1200970 
decimal prefixes are used: kilo (10 3 ), mega (10 6 ), giga (10 9 ), 
tera (10 12 ), peta (10 15 ), exa (10 18 ), zetta (10 21 ). 
Storage 
We estimated how much information could possibly have 
been stored by the 12 most widely used families of analog 
storage technologies and the 13 most prominent families of 
digital memory, from paper-based advertisement to the 
memory chips installed on a credit card (Fig. 2). The total 
amount of information grew from 2.6 optimally compressed 
exabytes in 1986, to 15.8 in 1993, over 54.5 in 2000, to 295 
optimally compressed exabytes in 2007. This is equivalent to 
less than one 730 MB CD-ROM per person in 1986 (539 MB 
per person), roughly 4 CD-ROM per person of 1993, 12 in 
the year 2000 and almost 61 CD-ROM per person in 2007. 
Piling up the imagined 404 billion CD-ROM from 2007 
would create a stack from the earth to the moon and a quarter 
of this distance beyond (with 1.2 mm thickness per CD). 
Our estimate is significantly larger than the previously 
cited hardware estimate from IDC for the same year (14) 
(IDC estimates 264 exabytes of digital hardware, not 
normalized for compression, while we count 276 optimally 
compressed exabytes on digital devices, which occupy 363 
exabytes of digital hardware). While our study is more 
comprehensive, we are not in a position to fully analyze all 
differences, since IDC?s methodological assumptions and 
statistics are based on inaccessible and proprietary company 
sources. 
Before the digital revolution, the amount of stored 
information was dominated by the bits stored in analog 
videotapes, such as VHS cassettes (Fig. 2). In 1986, vinyl 
Long-Play records still made up a significant part (14%), as 
did analog audio cassettes (12%) and photography (5% and 
8%). It was not until the year 2000 that digital storage made a 
significant contribution to our technological memory, 
contributing 25% of the total in 2000. Hard disks make up the 
lion share of storage in 2007 (52% in total),optical storage 
contributed more than a quarter (28%) and digital tape 
roughly 11%. Paper-based storage solutions captured a 
decreasing share of the total (0.33% in 1986 and 0.007% in 
2007), even though their capacity was steadily increasing in 
absolute terms (from 8.7 to 19.4 optimally compressed 
petabytes). 
Communication 
We divided the world?s technological communication 
capacity into two broad groups: one includes technological 
systems that provide only unidirectional downstream capacity 
to diffuse information (referred to as broadcasting), and one 
provides bidirectional upstream and downstream channels 
(telecommunication). The ongoing technological convergence 
between broadcasting and telecommunication is blurring this 
distinction, as exemplified by the case of digital TV, which 
we counted as broadcasting, even though it incorporates a 
small, but existent upstream channel (e.g. video-on-demand). 
The inventories of Figs. 3 and 4 account for only those bits 
that are actually communicated. In the case of 
telecommunication, the sum of the effective usages of all 
users is quite similar to the total installed capacity (any 
difference represents an over- or future investment). This is 
because most backbone networks are shared and only used 
sporadically by an individual user. If all users demanded their 
promised bandwidth simultaneously, the network would 
collapse. This is not the case for individual broadcast 
subscribers, who could continuously receive incoming 
information. To meaningfully compare the carrying capacities 
of each, we applied effective consumption rates to the 
installed capacity of broadcasting (calling it the effective 
capacity). This reduced the installed capacity by a stable 
factor (by 9 in 1986; 9.1 in 1993; 8.7 in 2000; and 8.4 in 
2007), implying an average individual broadcast consumption 
of roughly 2 hours and 45 minutes per 24 hours. It did not 
significantly change the relative distribution of the diverse 
technologies (Fig. 3). 
Fig. 3 displays the capacity of 6 analog and 5 digital 
broadcast technologies, including newspapers and personal 
navigation devices (GPS). In 1986, the world?s technological 
receivers picked up around 432 exabytes of optimally 
compressed information, 715 in 1993, 1.2 optimally 
compressed zettabytes in 2000 and 1.9 in 2007. Cable and 
satellite TV steadily gained importance but analog, ?over-the-
air,? terrestrial television still dominated the evolutionary 
trajectory. Digital satellite television led the pack into the 
digital age, receiving 50% of all digital broadcast signals in 
2007. Only a quarter of all broadcasting information was in 
digital format in 2007. The share of radio declined gradually 
from 7.2% in 1986 to 2.2% in 2007. 
Fig. 4 presents effective capacity of the 3 most common 
bidirectional analog telecommunication technologies and 
their 4 most prominent digital heirs. The 281 petabytes of 
optimally compressed information from 1986 were 
overwhelmingly dominated by fixed line telephony, whereas 
postal letters contributed only 0.34%. The year 1993 was 
characterized by the digitization of the fixed phone network 
(471 optimally compressed petabytes). We estimate the year 
1990 to be the turning point from analog to digital 
supremacy. The Internet revolution began shortly after the 
year 2000. In only 7 years, the introduction of broadband 
Internet effectively multiplied the world?s telecommunication 
capacity by a factor of 29, from 2.2 optimally compressed 
exabytes in 2000, to 65 in 2007. The most widespread 
telecommunication technology was the mobile phone, with 
3.4 billion devices in 2007 (versus 1.2 billion fixed line 
phones and 0.6 billion Internet subscriptions). Nevertheless, 
the fixed-line phone is still the solution of choice for voice 
 on February 16, 2011 www.sciencemag.org Downloaded from  
 
/ www.sciencexpress.org / 10 February 2011 / Page 4 / 10.1126/science.1200970 
communication (1.5% of the total). The mobile phone 
network became increasingly dominated by data traffic in 
2007 (1.1% for mobile data versus 0.8% for mobile voice). 
When compared withbroadcasting, telecommunications 
makes a modest, but rapidly growing part of the global 
communications landscape (3.3% of their sum in 2007, up 
from 0.07% in 1986). Althoughthere are only 8% more 
broadcast devices in the world than telecommunication 
equipment (6.66 billion vs. 6.15 billion in 2007), the average 
broadcasting device communicates 27 times more 
information per day than the average telecommunications 
gadget. This result might be unexpected, especially 
considering the omnipresence of the Internet, but can be 
understood when considering that an average Internet 
subscription effectively uses its full bandwidth for only 
around 9 minutes per day (during an average 1 hour and 36 
minutes daily session). 
Computation 
>From a theoretical standpoint, a ?computation? is the 
repeated transmission of information through space 
(communication) and time (storage), guided by an 
algorithmic procedure (31). The problem is that the applied 
algorithmic procedure influences the overall performance of a 
computer, both in terms of hardware design and in terms of 
the contributions of software. As a result, the theoretical, 
methodological, and statistical bases for our estimates for 
computation are less solid than the ones for storage and 
communication. In contrast to Shannon?s bit (29, 30), there is 
no generally accepted theory that provides us with an ultimate 
performance measure for computers. There are several ways 
to measure computational hardware performance. We chose 
MIPS (Million or Mega Instructions Per Second) as our 
hardware performance metric, which was imposed upon us by 
the reality of available statistics. Regarding the contributions 
of software, it would theoretically be possible to normalize 
the resulting hardware capacity for algorithmic efficiency 
(such as measured by O-notation) (32). This would recognize 
the constant progress of algorithms, which continuously make 
more efficient use of existing hardware. However, the 
weighted contribution of each algorithm would require 
statistics on respective execution intensities of diverse 
algorithms on different computational devices. We are not 
aware of such statistics. As a result of these limitations, our 
estimates refer to the installed hardware capacity of 
computers. 
We distinguished between two broad groups of computers. 
The first group includes all computers whose functionality is 
directly guided by their human users. We call this group 
?general-purpose computers? and include 6 technological 
families (Fig. 5). The second group carries out automated 
computations that are incidental to the primary task, such as 
in electronic appliances or visual interfaces. The user may 
have a range of predefined choices regarding their 
functionality, butcannot change the automated logic of these 
embedded systems. We call this group ?application-specific 
computers?. 
Although general-purpose computers are also equipped 
with application-specific parts (mobile phones come with 
digital signal processors, and PCs contain microcontroller 
units, etc.), we only include the capacity of humanly guidable 
microprocessors in the respective inventory. The calculator 
laid the cornerstone for modern microprocessors and was still 
the dominant way to compute information in 1986 (41% of 
3.0x 10 8 general-purpose -MIPS). The landscape changed 
quickly during the early 1990s, as personal computers and 
servers and mainframe computers pushed the evolutionary 
trajectory to 4.4 x 10 9 -MIPS. The personal computer 
extended its dominance during the year 2000 (86% of a total 
of 2.9 10 11 -MIPS), to be rivaled in 2007 by videogame 
consoles (1.6 x 10 12  MIPS or 25% of the total of 6.4 x 10 12 
MIPS) and increasingly relevant mobile phones (3.7 x 10 11 
MIPS or 6% of the 2007 total). Nowadays, clusters of 
videogame consoles are occasionally used as supercomputer 
substitutes for scientific purposes and other data intensive 
computational tasks (33). The relatively small role of 
supercomputers (less than 0.5% throughout) and professional 
servers and mainframes might come as a surprise. It can 
partially be explained by the fact that the inventory of Fig. 5 
presents the installed capacity, independent of effective usage 
rates. We also carried out some estimations based on the 
effective gross usage of the computers, which considers the 
time users interact with computers (not the net computational 
time). As a result we get between 5.8% and 9.1% of the 
installed capacity (16, table SA4). The share of servers and 
mainframes grows to 89% in 1986 and 11% in 2007, and 
supercomputers contribute 4% to the effective capacity in 
2007. 
The data also allows us to look at respective growth rates. 
Until the early 1990s, the annual growth rate was quite stable, 
at roughly 40% (Fig. 6). The 1990s show outstanding growth, 
reaching a peak of 88% in 1998. Since then, the technological 
progress has slowed. In recent times, every new year allows 
humankind to carry out roughly 60% of the computations that 
could have possibly been executed by all existing general-
purpose computers before that year. 
Our inventory of application-specific computations is the 
least complete one. The entire group of application-specific 
computers is very large and diverse (for example, dice cups 
and roulette wheels are application-specific, analog, random 
number generators) and it is often not straightforward to 
translate their performance into MIPS. The main goal of our 
inventory of this group was to show that the computational 
hardware capacity of application-specific computers is larger 
than the computational capacity of general-purpose 
 on February 16, 2011 www.sciencemag.org Downloaded from  
 
/ www.sciencexpress.org / 10 February 2011 / Page 5 / 10.1126/science.1200970 
computers (16) (table SA3). To achieve this we focused on a 
sample that includes three prominent groups: digital signal 
processors (DSP), which translate between analog and digital 
signals (including CD, DVD and PVR devices, cameras and 
camcorders, modems and setup boxes, GPS, portable media, 
printer and fax, radio, fixed and mobile phones); 
microcontrollers (MCU) (which regulate electronics and 
appliances); and graphic processing units (GPU) (an 
increasingly powerful microprocessor for visual displays). 
Although microcontrollers dominated our sample of 
application-specific computing support in 1986 (90% of the 
4.3 x 10 8  application-specific MIPS from our sample), graphic 
processing units clearly made up the lion share in 2007 (97% 
of 1.9 x 10 14 MIPS). 
Comparisons and growth rates 
The world?s technological capacity to compute 
information has by far experienced the highest growth (Table 
1). The per capita capacity of our sample of application-
specific machine mediators grew with a compound annual 
growth rate of 83% between 1986 and 2007 and humanly 
guided general-purpose computers grew at58% per year. The 
world?s technological capacity to telecommunicate only grew 
half as fast (CAGR of 28%). This might seem a little 
surprising, as the advancement of telecommunications, and 
especially the Internet, is often celebrated as the epitome of 
the digital revolution. The results from Table 1 challenge this 
idea and move the world?sability to compute information into 
the spotlight. The storage of information in vast technological 
memories has experienced a growth rate almost similar to 
telecommunication (CAGR of 23% per capita over two 
decades). The lower growth rate results from the relatively 
high base level provided by prevalent analog storage devices. 
The main characteristic of the storage trajectory is the 
digitalization of previously analog information (from 0.8% 
digital in 1986 to 94% in 2007). The global capacity to 
broadcast information has experienced the least progress, at 
6% CAGR per capita. Broadcasting is also the only 
information operation that is still dominated by analog ICT. 
As a result, the capacity to store information has grown at a 
much faster rate than the combined growth rate of tele- and 
broadcast communication. In 1986 it would have been 
possible to fill the global storage capacity with the help of all 
effectively used communication technologies in roughly 2.2 
days (539/241.16). In 1993 it would have taken almost 8 
days, in the year 2000 roughly 2.5 weeks, and in 2007 almost 
8 weeks. 
The compound annual growth rates represent the temporal 
average of periods which were experiencing different patterns 
of technological change. General-purpose computation had its 
peak growth around the turn of the millennia (Fig. 6).Storage 
capacity slowed down around the year 2000, but accelerated 
growth has been occurring in recent years (CAGR of 27% for 
1986-1993, 18% for 1993-2000 and 26% for 2000-2007; 
Table 1). The introduction of broadband has led to a 
continuous acceleration of t telecommunication (CAGR of 
6% for 1986-1993, 23% for 1993-2000 and 60% for 2000-
2007; Table 1), whereasbroadcasting had a relatively stable 
rate of change (CAGRs of 5.7%, 5.6% and 6.1% for 1986-
1993, 1993-2000 and 2000-2007, respectively; Table 1). 
The growth rates also allow us to look at the application of 
Moore?s laws (34) for the technological information 
processing capacity of humankind. Machines? application-
specific capacity to compute information per capita has 
roughly doubled every 14 months over the past decades in our 
sample, while the per capita capacity of the world?s general-
purpose computers has doubled every 18 months. The global 
telecommunication capacity per capita doubled every 34 
months, while the world?s storage capacity per capita 
required roughly 40 months. Per capita broadcast information 
has doubled roughly every 12.3 years. Of course, such 
averages disguise the varying nature of technological 
innovation avenues (35). 
Perspectives 
To put our findings in perspective, the 6.4*10 18  
instructions per second that human kind can carry out on its 
general-purpose computers in 2007 are in the same ballpark 
area as the maximum number of nerve impulses executed by 
one human brain per second (10 17 ) (36). The 2.4*10 21  bits 
stored by humanity in all of its technological devices in 2007 
is approaching order of magnitude of the roughly 10 23  bits 
stored in the DNA of a human adult (37), but it is still 
minuscule compared to the 10 90  bits stored in the observable 
universe (38). However, in contrast to natural information 
processing, the world?s technological information processing 
capacities are quickly growing at clearly exponential rates. 
References and Notes 
1. M. Castells, Volume III (Wiley-Blackwell, 2000). 
2. D. Bell (Basic Books, 1976). 
3. M.U. Porat, (National Science Foundation, Washington, 
DC, 1977). 
4. Y. Masuda (Transaction Publishers, 1980). 
5. C. Perez, Futures 15, 357 (1983). 
6. T. Forester (The MIT Press, 1985). 
7. C. Freeman, F. Lou?? (Oxford Univ. Press, USA, 2002). 
8. M. Castells, Volume I (Wiley-Blackwell, 2009). 
9. E. Brynjolfsson, A. Saunders, (The MIT Press, 2009). 
10. Y. Ito, Mass Communication Review Yearbook, 2, 671 
(1981). 
11. I.D.S. Pool, Science 221, 609 (1983). 
12. M. Lesk, lesk.com/mlesk (1997). 
13. P. Lyman, H. Varian (University of California, Berkeley, 
2003). 
14. J. Gantz, et al. (International Data Corporation, 2008). 
 on February 16, 2011 www.sciencemag.org Downloaded from  
 
/ www.sciencexpress.org / 10 February 2011 / Page 6 / 10.1126/science.1200970 
15. R. Bohn, J. Short (University of California, San Diego, 
2009). 
16. Materials and methods are available as supporting 
material on Science Online. 
17. International Telecommunication Union, United Nations 
ICT Indicators Database (2010). 
18. Faostat, United Nations ForesSTAT (2010). 
19. Universal Postal Union, United Nations Postal Statistics 
(2007). 
20. International Federation of the Phonographic Industry 
(1995-2004). 
21. Japanese Recording-Media Industries Association (2007). 
22. TOP500, Supercomputers (2009). 
23. J. Porter, disktrend.com (2005). 
24. R. Longbottom, roylongbottom.org.uk (2006). 
25. J. McCallum, The computer engineering handbook, 136 
(2006). 
26. T. Coughlin (Coughlin Associates, 2007). 
27. Morgan Stanley (2006). 
28. International Data Corporation (IDC) (2008). 
29. C. A. Shannon, Bell. Syst. Tech. J. 27, 379-423, 623 
(1948). 
30. T.M. Cover, J.A. Thomas (Wiley-Interscience, ed. 2, 
2006). 
31. A.M. Turing, Proc. of the London Math. Soc. s2-42, 230 
(1937). 
32. T. Cormen, C. Leiserson, R. Rivest & C. Stein (McGraw-
Hill Science/Engineering/Math, 2003). 
33. B. Gardiner, Wired Magazine Tech Biz IT, 10.17.07 
(2007). 
34. Moore's law measures technological progress of computer 
performance by counting the numbers of transistors on an 
integrated circuit, which has approximately doubled every 
two years since the 1960s. G.E. Moore. Proc. of SPIE, 2-
17 (1995). 
35. D. Sahal, Research Policy 14, 61 (1985). 
36. Assuming 100 billion neurons * 1,000 connections per 
neuron * max 1,000 nerve impulses per second. 
37. Considering a quaternary DNA alphabet, in which each 
base pair can store 4 bits * 3 billion DNA base pairs per 
human cell * 60 trillion cells per adult human . 
38. S. Lloyd. Phys. Rev. Lett. 88, 237901 (2002). 
39. We would like to thank the Information Society Program 
of United Nations ECLAC (in Chile) for its support, Tom 
Coughlin, John McCallum, Don Franz, Miguel Gonzalez, 
Cristian Vasquez, Len Adleman, Manuel Castells and the 
statisticians from UPU (Universal Post Union) and ITU 
(International Telecommunications Union), as well as 
numerous colleagues who motivated us by doubting the 
feasibility of this undertaking. 
 
 
 
 
Supporting Online Material 
www.sciencemag.org/cgi/contebt/full/science.1200970/DC1 
Materials and Methods 
Figs. A1 to E12 
Tables SA1 to SE24 
References and Notes 
29 November 2010; accepted 1 February 2011 
Published online 10 February 2011; 10.1126/science.1200970 
Fig. 1. The three basic information operations and theirmost 
prominent technologies. 
Fig. 2. World?s technological installed capacity to store 
information. Based on Supporting Online Material (16) Table 
SA1. 
Fig. 3. World?s technological effective capacity to broadcast 
information, in optimally compressed Megabytes (MB) per 
year, for 1986, 1993, 2000 and 2007, semi-logarithmic plot. 
Based on Supporting Online Material (16) Table SA2. 
Fig. 4. World?s technological effective capacity to 
telecommunicate information, Based on Supporting Online 
Material (16) Table SA2. 
Fig. 5. World?s technological installed capacity to compute 
information on general-purpose computers, in millions 
instructions per second (MIPS), Based on Supporting Online 
Material (16) Table SA3. 
Fig. 6. Annual growth of installed general-purpose 
computational capacity as percentage of all previous 
computations since 1977 (yeart / ?[1977, yeart-1]). Based on 
Supporting Online Material (16) Table SA3. 


From Glen.Beane at jax.org  Wed Feb 16 02:32:53 2011
From: Glen.Beane at jax.org (Glen Beane)
Date: Wed, 16 Feb 2011 05:32:53 -0500
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5B8A4D.80101@gmail.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com> <50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>
Message-ID: <0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>





On Feb 16, 2011, at 3:27 AM, Jonathan Aquilina <eagles051387 at gmail.com> wrote:

> To be tied with a human though i feel means they are on par with humans.
> 

At answering questions on Jeopardy or playing chess (deep blue). These are designed to do a very specific task well. 


> On 2/16/11 9:24 AM, Geoff Galitz wrote:
>>> http://edition.cnn.com/2011/TECH/innovation/02/14/jeopardy.ibm.watson/index.html
>>> 
>>> this is quite an interesting read.
>>> 
>>> question becomes are computers just as smart as we are?
>>> 
>>> I'm quite surprised that the computer doesn't have a massive lead
>> 
>> 
>> 
>> Science Friday did a segment on this which is quite good and, of course,
>> more informative for us techi/science guys:
>> 
>> http://www.sciencefriday.com/program/archives/201102112 (listen to the
>> podcast segment)
>> 
>> Smart is always a tough thing to measure, even in people.  Computers have a
>> very long way to go if you want to measure them to humans.
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From james.p.lux at jpl.nasa.gov  Wed Feb 16 06:00:53 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Wed, 16 Feb 2011 06:00:53 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5B8733.1010505@gmail.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>,
	<4D5B8733.1010505@gmail.com>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471A2@ALTPHYEMBEVSP20.RES.AD.JPL>


________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of Jonathan Aquilina [eagles051387 at gmail.com]
Sent: Wednesday, February 16, 2011 00:13
To: beowulf at beowulf.org
Subject: Re: [Beowulf] IBM's Watson on Jeopardy tonight

http://edition.cnn.com/2011/TECH/innovation/02/14/jeopardy.ibm.watson/index.html

this is quite an interesting read.

question becomes are computers just as smart as we are?

I'm quite surprised that the computer doesn't have a massive lead

----

Kind got into gear last night.. whipped the human's behinds big time.

It occurs to me that in the real game, among humans, there's a lot of psychology.. there's betting strategy, for one (although the show staff does give the contestants helpful information about good strategies).  But there's also a whole dynamic of "when do you buzz in if you think you "might" know the answer"

And, Watson's reaction time is unemotional.. The humans will have some effect.  Clearly, since they're champions, they're good at the timing, but when I was on the show, that was a HUGE factor. The guy who won was a stand-up comic and was very, very good at anticipating the light. There's also an advantage in being the winner from the previous day's game (actually filmed a few minutes before.. they film 5 shows in a day) because you've got the rhythm.

Jim

From james.p.lux at jpl.nasa.gov  Wed Feb 16 06:22:06 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Wed, 16 Feb 2011 06:22:06 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com> <50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>


______
On Feb 16, 2011, at 3:27 AM, Jonathan Aquilina <eagles051387 at gmail.com> wrote:

> To be tied with a human though i feel means they are on par with humans.
>

At answering questions on Jeopardy or playing chess (deep blue). These are designed to do a very specific task well.

--------------

Indeed, but I don't think that detracts much from the accomplishment.  It's readily acknowledged that the machine has different capabilities than the human.  Most people can't do N-body gravitational simulations in their head, but that's a pretty straightforward task for a machine like Watson. 

I think it will be a while before a machine has the wide span of capabilities of a human (particularly in terms of the ability to manipulate the surroundings), and, as someone pointed out the energy consumption is quite different (as is the underlying computational rate... lots of fairly slow neurons with lots of parallelism vs relatively few really fast transistors)

Watson is a particularly impressive natural language understanding system, with a fairly constrained user interface.  But for other aspects of "life" or "human-ness" other researchers are doing pieces of the puzzle (perhaps not to the level of Watson or Deep Blue).  There are self organizing and self replicating robots (granted, with the "intelligence" of less than a flatworm).

A big advance will be when the machine not only can build its own tools, but can independently figure out what tools it needs to build.  The former is more a matter of physical manipulation, the latter requires creativity and thought.  That's a pretty high bar; there are humans that can't figure out what tool is needed to solve a problem, independently, without having seen someone else do the task.


Jim

From cbergstrom at pathscale.com  Wed Feb 16 06:29:15 2011
From: cbergstrom at pathscale.com (=?ISO-8859-1?Q?=22C=2E_Bergstr=F6m=22?=)
Date: Wed, 16 Feb 2011 21:29:15 +0700
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <4D5BDF3B.6010104@pathscale.com>

Lux, Jim (337C) wrote:
> I think it will be a while before a machine has the wide span of capabilities of a human (particularly in terms of the ability to manipulate the surroundings), and, as someone pointed out the energy consumption is quite different (as is the underlying computational rate... lots of fairly slow neurons with lots of parallelism vs relatively few really fast transistors)
>   
Doesn't this then raise the question of why we aren't modeling computers 
and programming models after the brain? ;)




From asabigue at fing.edu.uy  Wed Feb 16 06:52:01 2011
From: asabigue at fing.edu.uy (ariel sabiguero yawelak)
Date: Wed, 16 Feb 2011 12:52:01 -0200
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5BDF3B.6010104@pathscale.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>	<4D5B8733.1010505@gmail.com>	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>
Message-ID: <4D5BE491.2010107@fing.edu.uy>

I believe that it is because, on the one hand, we don't accept fuzzy 
results, and on the other hand, we don't know how to train the millions 
of ANNs required to mimic a mammal's brain.
The way in which biology deals with failures, faults and defects is far 
beyond our full comprehension. How to program a brain? and similar 
questions are beyond our grasp too, yet, we re-program ourselves day 
after day intuitively.

In some way, the speed at which multicores are evolving (more and 
simpler cores instead of a single, yet powerful one) indicates that 
parallel processing is the way to go -I think I read it somewhere in 
this list-. Maybe the answer that the evolution found for carbon-based 
processing is different than the one for future silicon-based life forms 
(or at least, intelligent processing). A dead company used to say "the 
computer is the network", and for our brains it seems so. Will it be 
true for really-massive processors? Will they shrink until they only sum 
and bias inputs without any programing inside? Will we discuss 
multi-million-core-SMP? If so, I doubt that it will be single-bus-based 
solution.

I'm not sure I'll live until we find an answer, but is a nice long term 
research topic....

ariel

El 16/02/11 12:29, "C. Bergstr?m" escribi?:
> Lux, Jim (337C) wrote:
>> I think it will be a while before a machine has the wide span of capabilities of a human (particularly in terms of the ability to manipulate the surroundings), and, as someone pointed out the energy consumption is quite different (as is the underlying computational rate... lots of fairly slow neurons with lots of parallelism vs relatively few really fast transistors)
>>
> Doesn't this then raise the question of why we aren't modeling computers
> and programming models after the brain? ;)
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>


From Bill.Rankin at sas.com  Wed Feb 16 07:18:59 2011
From: Bill.Rankin at sas.com (Bill Rankin)
Date: Wed, 16 Feb 2011 15:18:59 +0000
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5BE491.2010107@fing.edu.uy>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com> <4D5BE491.2010107@fing.edu.uy>
Message-ID: <76097BB0C025054786EFAB631C4A2E3C09503031@MERCMBX03R.na.SAS.com>

> ariel sabiguero yawelak wrote:
> 
> A dead company used to say "the
> computer is the network", and for our brains it seems so. 

Actually for HPC and esp. clusters that (IMNSHO) is even more true today.

My desktop just serves as my window into that network.  Loss of a CPU or an entire server I can usually deal with.  Loss of the network would be terminal on many different levels.

-b




From james.p.lux at jpl.nasa.gov  Wed Feb 16 07:20:42 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Wed, 16 Feb 2011 07:20:42 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5BE491.2010107@fing.edu.uy>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>,<4D5BE491.2010107@fing.edu.uy>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471B0@ALTPHYEMBEVSP20.RES.AD.JPL>

Aside from how brains are "programmed" (a fascinating question)

One big difference between biological systems and non-bio is the handling of errors and faults.  Biological systems tend to assume that (many) failures occur and use a strategy of massive redundancy with fuzzy collective action.  To date, there's been very little work on large scale computing systems that are more than just fault tolerant.  Partly it's the fact that the problem is difficult.  partly it's a market driven thing: the people who want to anticipate and deal with faults tend to want "provably failure tolerant" so that drives things like error correcting codes, Triple Modular Redundancy (TMR) and a whole host of schemes for failover, hot standby, etc.

Our friends at google are probably one of the best examples of a large computing environment which is very aware of "good enough" and which has enough scale to know that at any given time, some fraction of their computers are dead/wrong/faulted.

But even there, the redundancy is at a pretty high level, compared to the very fine grained redundancy in any biological system.

Even very simple biological systems with very simple individual cellular algorithms/control rules display emergent behavior which is quite complex.  Consider the recent article about the annual plague of Portuguese Man o'Wars in Florida.  (A floating colony of different animals, resembling a jellyfish, but actually not one organism)
________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of ariel sabiguero yawelak [asabigue at fing.edu.uy]
Sent: Wednesday, February 16, 2011 06:52
To: beowulf at beowulf.org
Subject: Re: [Beowulf] IBM's Watson on Jeopardy tonight

I believe that it is because, on the one hand, we don't accept fuzzy
results, and on the other hand, we don't know how to train the millions
of ANNs required to mimic a mammal's brain.
The way in which biology deals with failures, faults and defects is far
beyond our full comprehension. How to program a brain? and similar
questions are beyond our grasp too, yet, we re-program ourselves day
after day intuitively.

In some way, the speed at which multicores are evolving (more and
simpler cores instead of a single, yet powerful one) indicates that
parallel processing is the way to go -I think I read it somewhere in
this list-. Maybe the answer that the evolution found for carbon-based
processing is different than the one for future silicon-based life forms
(or at least, intelligent processing). A dead company used to say "the
computer is the network", and for our brains it seems so. Will it be
true for really-massive processors? Will they shrink until they only sum
and bias inputs without any programing inside? Will we discuss
multi-million-core-SMP? If so, I doubt that it will be single-bus-based
solution.

I'm not sure I'll live until we find an answer, but is a nice long term
research topic....

ariel

El 16/02/11 12:29, "C. Bergstr?m" escribi?:
> Lux, Jim (337C) wrote:
>> I think it will be a while before a machine has the wide span of capabilities of a human (particularly in terms of the ability to manipulate the surroundings), and, as someone pointed out the energy consumption is quite different (as is the underlying computational rate... lots of fairly slow neurons with lots of parallelism vs relatively few really fast transistors)
>>
> Doesn't this then raise the question of why we aren't modeling computers
> and programming models after the brain? ;)
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From rgb at phy.duke.edu  Wed Feb 16 07:21:37 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Wed, 16 Feb 2011 10:21:37 -0500 (EST)
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5BDF3B.6010104@pathscale.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>
Message-ID: <alpine.LFD.2.02.1102161000050.3627@lilith>

On Wed, 16 Feb 2011, "C. Bergstr?m" wrote:

> Lux, Jim (337C) wrote:
>> I think it will be a while before a machine has the wide span of capabilities of a human (particularly in terms of the ability to manipulate the surroundings), and, as someone pointed out the energy consumption is quite different (as is the underlying computational rate... lots of fairly slow neurons with lots of parallelism vs relatively few really fast transistors)
>>
> Doesn't this then raise the question of why we aren't modeling computers
> and programming models after the brain? ;)

We are, but that problem is, well, "hard".  As in grand challenge hard.
There are other problems -- brains are highly non-deterministic (in that
they often selectively amplify tiny nearly random signals, as in "having
an idea" or "reacting completely differently depending on our hormonal
state, our immediate past history, and the phase of the moon").  Brains
are extremely non-Markovian with all sorts of multiple-time-scale memory
and with plenty of functional structures we honestly don't understand
much at all.  We don't even know how brains ENCODE memory -- when I've
got Sheryl Crow running through my head in what SEEM to be remarkably
good fidelity, there is absolutely no way to determine where that
detailed, replayable music is stored or how it is encoded or how it is
being reproduced or how the reproduction is being conceived in the
background of my mind right now as I'm "doing" something else with my
fingers and my attention is only partly on these words, that are
appearing out of -- exactly where?  They're being synthesized, but how?

Our computers, on the other hand, tend to be mostly deterministic and
usually serial unless (as everybody on this list understands very well)
we work HARD to parallelize their function.  Our programs tend to be
insensitive to noise and to not use random numbers (unless we
DELIBERATELY use random numbers in the programs) and if we do the
latter, we are ultimately sampling an ensemble of possible dynamical
outcomes with little theoretical reason to believe that our sample will
be a "good" one.  Not that this isn't true of humans as well, but it is
both a sometimes strength and a frequent weakness.

Humans can "almost" remember somebody's name one day, for example, and
then another day know it immediately, and another day still not even
recognize that they once knew it.  Computers are more systematic and as
they are currently architected and built either remember it or don't,
based on fairly deterministic criteria and the ability to do a
systematic search instead of a vague, parallel, "latching" of
holographically encoded memory into a holographically encoded attention
center for a being that IS little more than that attention center and
associated memory and hardware plus various bidirectional interfaces.
Computers may fail, but it isn't the same KIND of failure that humans
have.

To be honest, I think the Jeapordy example is a silly one.  It's a
canned problem.  Why bother?  We do the test every day.  Let's race --
everybody here has probably read, I dunno, Moby Dick at one time or
another.

So now, off to the races.  Open up a Google window.  Type in:

    What is the name of the third mate in Moby Dick?

Now press return.  I don't really care how soon.  Look!  Google beat the
shit out of Watson and all of the Jeapordy contestants (it didn't even
take a full second, human reactions couldn't possibly have matched it as
EVEN IF YOU KNEW THE ANSWER (which once upon a time you HAVE seen) your
mouth would still be forming the word "Flask" long after Google has the
answer on the screen -- four or five times!

So this is a dumb test.  Let >>me<< make up the questions, and >>don't<<
let the participants see them ahead of time, and Google will beat Watson
and all of the humans put together hands down, generally nearly zero
time to infinite time (they just don't know).  Everybody knows computers
win at this sort of thing, and have since Alta Vista days.

Now, let's try to subject "Watson" to a real Turing test.  What's that
Watson?  How do you >>feel<< about being tested to see if you're human?

Not much, I'd wager.

    rgb

>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu


From james.p.lux at jpl.nasa.gov  Wed Feb 16 07:32:40 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Wed, 16 Feb 2011 07:32:40 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <alpine.LFD.2.02.1102161000050.3627@lilith>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com> <50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>,
	<alpine.LFD.2.02.1102161000050.3627@lilith>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471B3@ALTPHYEMBEVSP20.RES.AD.JPL>


________________________________________
From: Robert G. Brown [rgb at phy.duke.edu]
Sent: Wednesday, February 16, 2011 07:21
To: "C. Bergstr?m"
Cc: Lux, Jim (337C); beowulf at beowulf.org; Glen Beane
Subject: Re: [Beowulf] IBM's Watson on Jeopardy tonight

Now, let's try to subject "Watson" to a real Turing test.  What's that
Watson?  How do you >>feel<< about being tested to see if you're human?

-----------------------------
Check out this: http://www.theatlantic.com/magazine/archive/2011/03/mind-vs-machine/8386/

Jim

From eagles051387 at gmail.com  Wed Feb 16 07:35:33 2011
From: eagles051387 at gmail.com (Jonathan Aquilina)
Date: Wed, 16 Feb 2011 16:35:33 +0100
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5BDF3B.6010104@pathscale.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>
Message-ID: <4D5BEEC5.6060707@gmail.com>

On 2/16/11 3:29 PM, "C. Bergstr?m" wrote:
> Lux, Jim (337C) wrote:
>> I think it will be a while before a machine has the wide span of 
>> capabilities of a human (particularly in terms of the ability to 
>> manipulate the surroundings), and, as someone pointed out the energy 
>> consumption is quite different (as is the underlying computational 
>> rate... lots of fairly slow neurons with lots of parallelism vs 
>> relatively few really fast transistors)
> Doesn't this then raise the question of why we aren't modeling 
> computers and programming models after the brain? ;)
>
>
Isn't that the whole purpose of artificial intelligence and neural networks?


From cbergstrom at pathscale.com  Wed Feb 16 07:38:38 2011
From: cbergstrom at pathscale.com (=?ISO-8859-15?Q?=22C=2E_Bergstr=F6m=22?=)
Date: Wed, 16 Feb 2011 22:38:38 +0700
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <alpine.LFD.2.02.1102161000050.3627@lilith>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>
	<alpine.LFD.2.02.1102161000050.3627@lilith>
Message-ID: <4D5BEF7E.8060004@pathscale.com>

Robert G. Brown wrote:
> On Wed, 16 Feb 2011, "C. Bergstr?m" wrote:
>
>> Lux, Jim (337C) wrote:
>>> I think it will be a while before a machine has the wide span of 
>>> capabilities of a human (particularly in terms of the ability to 
>>> manipulate the surroundings), and, as someone pointed out the energy 
>>> consumption is quite different (as is the underlying computational 
>>> rate... lots of fairly slow neurons with lots of parallelism vs 
>>> relatively few really fast transistors)
>>>
>> Doesn't this then raise the question of why we aren't modeling computers
>> and programming models after the brain? ;)
>
> We are, but that problem is, well, "hard".  As in grand challenge hard.
I wonder how you'd really describe the human brain learning in terms of 
a programming model....
> There are other problems -- brains are highly non-deterministic (in that
> they often selectively amplify tiny nearly random signals, as in "having
> an idea" or "reacting completely differently depending on our hormonal
> state, our immediate past history, and the phase of the moon").  Brains
> are extremely non-Markovian with all sorts of multiple-time-scale memory
> and with plenty of functional structures we honestly don't understand
> much at all.  We don't even know how brains ENCODE memory -- when I've
> got Sheryl Crow running through my head in what SEEM to be remarkably
> good fidelity,
I wonder just how good it is?  If I had to guess I'd say pretty bad.  
(no offense)  I wonder how much actual "space" it takes up.  I'd bet 
from a physical size perspective we're possibly ahead of nature in terms 
of data storage density.  (Without the packaging/cases.. etc)
> there is absolutely no way to determine where that
> detailed, replayable music is stored or how it is encoded or how it is
> being reproduced or how the reproduction is being conceived in the
> background of my mind right now as I'm "doing" something else with my
> fingers and my attention is only partly on these words, that are
> appearing out of -- exactly where?  They're being synthesized, but how?
I think motor functions are better understood than pure thought.
>
> Our computers, on the other hand, tend to be mostly deterministic and
> usually serial unless (as everybody on this list understands very well)
> we work HARD to parallelize their function.  Our programs tend to be
> insensitive to noise and to not use random numbers (unless we
> DELIBERATELY use random numbers in the programs) and if we do the
> latter, we are ultimately sampling an ensemble of possible dynamical
> outcomes with little theoretical reason to believe that our sample will
> be a "good" one.  Not that this isn't true of humans as well, but it is
> both a sometimes strength and a frequent weakness.
I don't believe in random, but it may appear that way.
>
> Humans can "almost" remember somebody's name one day, for example, and
> then another day know it immediately,
cache hit
> and another day still not even
> recognize that they once knew it.
cache miss


From james.p.lux at jpl.nasa.gov  Wed Feb 16 07:57:14 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Wed, 16 Feb 2011 07:57:14 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5BEEC5.6060707@gmail.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com> <50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>,<4D5BEEC5.6060707@gmail.com>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471B6@ALTPHYEMBEVSP20.RES.AD.JPL>


________________________________________
From: Jonathan Aquilina [eagles051387 at gmail.com]
Sent: Wednesday, February 16, 2011 07:35
To: "C. Bergstr?m"
Cc: Lux, Jim (337C); Glen Beane; beowulf at beowulf.org
Subject: Re: [Beowulf] IBM's Watson on Jeopardy tonight

On 2/16/11 3:29 PM, "C. Bergstr?m" wrote:
> Doesn't this then raise the question of why we aren't modeling
> computers and programming models after the brain? ;)
>
>
Isn't that the whole purpose of artificial intelligence and neural networks?


--- 
Yes and no..
That field covers a lot of territory, and the specific territory changes with time and fashion.  Neural Nets (under a different name) were the rage in the 50s and 60s until the paper came out proving that they couldn't solve a big class of problems.  All the funding and publications went away, while the AI community chased natural language and LISP for a while, until the idea of multiple layers came up in the 80s, and bingo neural nets were fashionable again (with the added cachet of venture capital.. oh yeah...good work on a ANN could get you a Ferrari, how cool is that if you're a struggling AI PhD candidate on a $500/month stipend)

From tjrc at sanger.ac.uk  Wed Feb 16 08:03:56 2011
From: tjrc at sanger.ac.uk (Tim Cutts)
Date: Wed, 16 Feb 2011 16:03:56 +0000
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471B0@ALTPHYEMBEVSP20.RES.AD.JPL>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>, <4D5BE491.2010107@fing.edu.uy>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471B0@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <3273CE4C-2CFB-46F6-A4E6-660FF2B909F7@sanger.ac.uk>


On 16 Feb 2011, at 15:20, Lux, Jim (337C) wrote:

> Aside from how brains are "programmed" (a fascinating question)
> 
> One big difference between biological systems and non-bio is the handling of errors and faults.  Biological systems tend to assume that (many) failures occur and use a strategy of massive redundancy with fuzzy collective action.

<biologist mode on>
Actually, I'd quibble with that statement about redundancy.  On the system level, there's very little redundancy in most [animal] organisms.  You only have one heart.  You only have one aorta, stomach, oesophagus and so on.  Even when some things are paired, the loss of either one (such as severing a carotid or femoral artery) can in many cases result in total organism failure.  :-)

Biological systems tend to be plastic and self-healing rather than redundant, I would have said.  At the sub-cellular level, redundancy is much more widespread of course, with various protein families able to cover for each other, resulting in redundant biochemical pathways in many cases, although again not all.  Consider the DNA repair disorder xeroderma pigmentosum; it's a failure of one particular DNA repair mechanism, and there are seven genetic complementation groups for the disease - in other words there are seven distinct genes involved in the process, and the failure of any one of them results in the failure of the entire process.  There's redundancy in the fact that you have two copies of most chromosomes, of course, although you and I, being male, are a bit buggered if there's anything wrong with our X chromosome.  Ask any haemophiliac.

Plants of course do much better, and do tend to be massively redundant, both at the structural level of the whole organism, and also at the genetic level (they often have much larger genomes than us, and massively redundant).  Ever tried eradicating bindweed from your garden?  That stuff is just impossible to kill.

But as to mental processing, I think one only has to look at what happens in brain injury cases like strokes to see that the processing is not redundant; a stroke can remove some particular processing ability.  The brain however, has a lot of plasticity, especially when young, and other parts of the brain can learn to take over the functions.  But I don't think that's quite the same as true redundancy; it's like re-programming another set of flash chips to take over.

Having said all that, I haven't been actively doing any biology research for 15 years, so I'm a bit out of touch.

Regards,

Tim



-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 


From prentice at ias.edu  Wed Feb 16 08:10:24 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 16 Feb 2011 11:10:24 -0500
Subject: [Beowulf] 9 traits of the veteran Unix admin
Message-ID: <4D5BF6F0.8080103@ias.edu>

Anyone see this yet? He's pretty dead on, IMHO, especially 5,6,9


http://www.infoworld.com/t/unix/nine-traits-the-veteran-unix-admin-276

--
Prentice



From james.p.lux at jpl.nasa.gov  Wed Feb 16 08:22:34 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Wed, 16 Feb 2011 08:22:34 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <3273CE4C-2CFB-46F6-A4E6-660FF2B909F7@sanger.ac.uk>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>,<4D5BE491.2010107@fing.edu.uy>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471B0@ALTPHYEMBEVSP20.RES.AD.JPL>,
	<3273CE4C-2CFB-46F6-A4E6-660FF2B909F7@sanger.ac.uk>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471B9@ALTPHYEMBEVSP20.RES.AD.JPL>


________________________________________
From: Tim Cutts [tjrc at sanger.ac.uk]
Sent: Wednesday, February 16, 2011 08:03
To: Lux, Jim (337C)
Cc: ariel sabiguero yawelak; beowulf at beowulf.org
Subject: Re: [Beowulf] IBM's Watson on Jeopardy tonight

On 16 Feb 2011, at 15:20, Lux, Jim (337C) wrote:

> Aside from how brains are "programmed" (a fascinating question)
>
> One big difference between biological systems and non-bio is the handling of errors and faults.  Biological systems tend to assume that (many) failures occur and use a strategy of massive redundancy with fuzzy collective action.

<biologist mode on>
Actually, I'd quibble with that statement about redundancy.  On the system level, there's very little redundancy in most [animal] organisms.  You only have one heart.  You only have one aorta, stomach, oesophagus and so on.  Even when some things are paired, the loss of either one (such as severing a carotid or femoral artery) can in many cases result in total organism failure.  :-)

>>> I was thinking at a smaller scale.. you have many bone cells in a bone, so if there's a microfracture, that's accommodated within the "design".  Likewise, I can kill off some brain cells and still remain reasonably functional.

Biological systems tend to be plastic and self-healing rather than redundant, I would have said.

>>> a much better description than mine.. plastic.. works in the mechanical sense too.. 


  At the sub-cellular level, redundancy is much more widespread of course, with various protein families able to cover for each other, resulting in redundant biochemical pathways in many cases, although again not all.  Consider the DNA repair disorder xeroderma pigmentosum; it's a failure of one particular DNA repair mechanism, and there are seven genetic complementation groups for the disease - in other words there are seven distinct genes involved in the process, and the failure of any one of them results in the failure of the entire process.  There's redundancy in the fact that you have two copies of most chromosomes, of course, although you and I, being male, are a bit buggered if there's anything wrong with our X chromosome.  Ask any haemophiliac.

Plants of course do much better, and do tend to be massively redundant, both at the structural level of the whole organism, and also at the genetic level (they often have much larger genomes than us, and massively redundant).  Ever tried eradicating bindweed from your garden?  That stuff is just impossible to kill.

But as to mental processing, I think one only has to look at what happens in brain injury cases like strokes to see that the processing is not redundant; a stroke can remove some particular processing ability.  The brain however, has a lot of plasticity, especially when young, and other parts of the brain can learn to take over the functions.  But I don't think that's quite the same as true redundancy; it's like re-programming another set of flash chips to take over.

>>> But I think in a stroke, you're not talking about a generalized distributed failure, but of the failure of a particular "region" of cells.  One might think of the brain, which although remarkably uniform in internal structure, does have specialized areas.. some areas because that's where sensory or motor nerves wind up (e.g. your optic pathways wind up at the occiput, so that's where vision processing is done).  I think of it as more a collection of organs that all happen to look the same but do different things.  
>> you raise an interesting point, though. Biological systems handle distributed failures better than localized failures.  Sort of like in error correction coding, there are codes that deal with burst errors and those that deal with distributed errors.  In coding, there are interleaving schemes to change one into the other, too.
>> and of course, the brain (and other parts of your body, to a certain extent) is remarkably plastic, and even moreso at the higher "system level" (e.g. one can invent a wheelchair)



From john.hearns at mclaren.com  Wed Feb 16 08:22:31 2011
From: john.hearns at mclaren.com (Hearns, John)
Date: Wed, 16 Feb 2011 16:22:31 -0000
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <3273CE4C-2CFB-46F6-A4E6-660FF2B909F7@sanger.ac.uk>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu><4D5B8733.1010505@gmail.com>	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC><4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org><ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL><4D5BDF3B.6010104@pathscale.com>,
	<4D5BE491.2010107@fing.edu.uy><ECE7A93BD093E1439C20020FBE87C47FEDE0A471B0@ALTPHYEMBEVSP20.RES.AD.JPL>
	<3273CE4C-2CFB-46F6-A4E6-660FF2B909F7@sanger.ac.uk>
Message-ID: <68A57CCFD4005646957BD2D18E60667B1287549C@milexchmb1.mil.tagmclarengroup.com>

> 
> On 16 Feb 2011, at 15:20, Lux, Jim (337C) wrote:
> 
> 
> <biologist mode on>
> Actually, I'd quibble with that statement about redundancy.  On the
> system level, there's very little redundancy in most [animal]
> organisms.  You only have one heart.  You only have one aorta,
stomach,
> oesophagus and so on.  Even when some things are paired, the loss of
> either one (such as severing a carotid or femoral artery) can in many
> cases result in total organism failure.  :-)

Re. carotid arteries, that is not actually true.
Back when I was but a cub HPC type, working in medical imaging, a
colleague was doing
MRI Angiography studies of the Circle of Willis. This is a 'ring main'
at the base of the brain,
where the carotids connect in a ring. Biology has built in redundancy
there, and so if you lose a carotid your brain
will continue to get blood (assuming you don't bleed to death).
Some people have a congenital malformation in the circle of Willis, and
it is not complete.
Brain surgery for such people is very risky, so finding this out is
important.


Re. hearts, there was a fascinating documentary in the Horizon series on
Monday (Vaelntine's day).
A lab somewhere has managed to remove all the cells from a heart,
leaving the connective tissue and valves making what
they call 'ghost heart'. They are experimenting with repopulating the
heart tissue with stem cells, so your transplant heart
would be effectively your own, and not rejected.
Moving this back on topic, as usual there were a lot of whizzy
visualisations, it s a pity that documentaries like these show the
cpictures but don't explain in depth what the colour scales mean.


The contents of this email are confidential and for the exclusive use of the intended recipient.  If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy.


From prentice at ias.edu  Wed Feb 16 08:36:19 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 16 Feb 2011 11:36:19 -0500
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5B8733.1010505@gmail.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>
Message-ID: <4D5BFD03.2030409@ias.edu>

Jonathan Aquilina wrote:
> http://edition.cnn.com/2011/TECH/innovation/02/14/jeopardy.ibm.watson/index.html
> 
> this is quite an interesting read.
> 
> question becomes are computers just as smart as we are?

Don't confuse knowledge for intelligence ("smarts") They are two
completely separate (but often related things). Knowledge is the
collection of facts, but intelligence is the ability to reason or
"figure things out"

Note this is just my description, and not anything super-scientific, so
please don't split hairs over my description.

Prentice



> 
> I'm quite surprised that the computer doesn't have a massive lead
> 
> On 2/15/11 10:55 PM, David Mathog wrote:
>> Jim Lux pointed out in an earlier post the reaction time element on
>> buzzing in from the (invisible to the audience) "go" light.  I only
>> watched about 10 minutes of this, but my impression was that the machine
>> did have a reaction time advantage.  Alex T. was a little vague on how
>> the machine was fed the question and if he explained what its equivalent
>> of the "go" light is I missed it.  It could be decisive - sending the
>> entire question to the machine at the instant it became visible to the
>> humans would give the machine an advantage, since it could digest the
>> message before the contestants eyes could even track across the first
>> line of text, let alone until they heard the entire question. In any
>> case, for the part I watched the machine was beating the humans to the
>> buzzer pretty consistently.  It was cute, but a little silly, that the
>> machine had to push the button with a mechanical plunger.  Unless they
>> built a typical human reaction delay into that mechanism one would
>> expect the plunger to be much faster than a thumb.
>>
>> In terms of score/watt the humans had the machine beat by a mile :-).
>>
>> David Mathog
>> mathog at caltech.edu
>> Manager, Sequence Analysis Facility, Biology Division, Caltech


-- 
Prentice


From mathog at caltech.edu  Wed Feb 16 08:42:44 2011
From: mathog at caltech.edu (David Mathog)
Date: Wed, 16 Feb 2011 08:42:44 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
Message-ID: <E1PpkSi-0006f4-62@mendel.bio.caltech.edu>

Glen Beane <Glen.Beane at jax.org> wrote:

> On Feb 16, 2011, at 3:27 AM, Jonathan Aquilina
<eagles051387 at gmail.com> wrote:
> 
> > To be tied with a human though i feel means they are on par with humans.
> > 
>
> At answering questions on Jeopardy or playing chess (deep blue). These
are designed to do a very specific task well. 

It was too bad they didn't reveal how the computer set its dollar values
for daily doubles.  They were very, very different from what a human
would have wagered.  My impression was that one factor was how well it
was doing in that category, so if it had already answered some questions
right it would bet more than when it hit the daily double on the first
question.  There is apparently a rule in Jeopardy that a wager cannot
bet less than $5 on a daily double, and since the regular questions are
all multiples of ten, this may have factored in somehow to result in
machine wagers that ended in 5.

It was also interesting that the second answer on the machine's list was
often clearly ruled out by the question, yet would have a probability
like 30%.  The only example of this I remember gave a year in the 1600's
and asked who the Lucasian Chair was at that time.  The machine did pick
Newton, but its second best answer was Stephen Hawking, who was
obviously not around at the time.  A human would have put one of
Newton's contemporaries in that spot, and probably not remembering the
occupants of that chair before and after Newton (neither of whom was
remotely as famous), might have listed somebody like Robert Hooke, who
was better known than either of them, and more likely to be remembered
by somebody with a good background in Science, but not a PhD in the
history of mathematics.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech


From james.p.lux at jpl.nasa.gov  Wed Feb 16 08:51:51 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Wed, 16 Feb 2011 08:51:51 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <E1PpkSi-0006f4-62@mendel.bio.caltech.edu>
References: <E1PpkSi-0006f4-62@mendel.bio.caltech.edu>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471BF@ALTPHYEMBEVSP20.RES.AD.JPL>


It was too bad they didn't reveal how the computer set its dollar values
for daily doubles.  They were very, very different from what a human
would have wagered.  My impression was that one factor was how well it
was doing in that category, so if it had already answered some questions
right it would bet more than when it hit the daily double on the first
question.  There is apparently a rule in Jeopardy that a wager cannot
bet less than $5 on a daily double, and since the regular questions are
all multiples of ten, this may have factored in somehow to result in
machine wagers that ended in 5.

>>>  when you are a contestant, the help you get from the staff is twofold:
a) how to buzz in effectively
b) how to bet
-- they can't help with the questions, obviously, and, in any case, you wouldn't be there if you weren't already an information sponge. (my kids' term)

They go over betting strategy a LOT.. when to jump to the bottom of the board.  What to bet on a daily double or final jeopardy? if you're in second or third and way behind, bet the farm less a dollar.. if you go negative you go home without anything.. stay at $1, and you get whatever prizes there are..  If you're in close 2nd, it's more dicey.. you have to think about what the first place player is going to bet.. you want to be optimistic.. if you get it right, you want a chance of winning, so you have to bet at least 1 dollar more than the amount you're behind...

The strategy on daily doubles is different than on final, because it's only YOU choosing the bet, while on final, everyone bets.

--------------------

It was also interesting that the second answer on the machine's list was
often clearly ruled out by the question, yet would have a probability
like 30%.  The only example of this I remember gave a year in the 1600's
and asked who the Lucasian Chair was at that time.  The machine did pick
Newton, but its second best answer was Stephen Hawking, who was
obviously not around at the time.  A human would have put one of
Newton's contemporaries in that spot, and probably not remembering the
occupants of that chair before and after Newton (neither of whom was
remotely as famous), might have listed somebody like Robert Hooke, who
was better known than either of them, and more likely to be remembered
by somebody with a good background in Science, but not a PhD in the
history of mathematics.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From james.p.lux at jpl.nasa.gov  Wed Feb 16 08:52:05 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Wed, 16 Feb 2011 08:52:05 -0800
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <E1PpkSi-0006f4-62@mendel.bio.caltech.edu>
References: <E1PpkSi-0006f4-62@mendel.bio.caltech.edu>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471C0@ALTPHYEMBEVSP20.RES.AD.JPL>



It was also interesting that the second answer on the machine's list was
often clearly ruled out by the question, yet would have a probability
like 30%.  The only example of this I remember gave a year in the 1600's
and asked who the Lucasian Chair was at that time.  The machine did pick
Newton, but its second best answer was Stephen Hawking, who was
obviously not around at the time.  

---------
I was struck by that too.. the 2nd and 3rd choices were sometimes totally unreasonable.  Maybe that's more a flaw in the display than indicative of the underlying algorithms.  After all, knowing what the alternate choices are doesn't change the progress of the game.  For all we know it was a late add on to enhance the entertainment value.

Jim

From rgb at phy.duke.edu  Wed Feb 16 09:40:05 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Wed, 16 Feb 2011 12:40:05 -0500 (EST)
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471B3@ALTPHYEMBEVSP20.RES.AD.JPL>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>,
	<alpine.LFD.2.02.1102161000050.3627@lilith>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471B3@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <alpine.LFD.2.02.1102161239450.3531@lilith>

On Wed, 16 Feb 2011, Lux, Jim (337C) wrote:

> Check out this: http://www.theatlantic.com/magazine/archive/2011/03/mind-vs-machine/8386/

Indeed! :-)

    rgb

>
> Jim

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu




From rgb at phy.duke.edu  Wed Feb 16 10:06:35 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Wed, 16 Feb 2011 13:06:35 -0500 (EST)
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5BEF7E.8060004@pathscale.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5BDF3B.6010104@pathscale.com>
	<alpine.LFD.2.02.1102161000050.3627@lilith>
	<4D5BEF7E.8060004@pathscale.com>
Message-ID: <alpine.LFD.2.02.1102161240390.3531@lilith>

On Wed, 16 Feb 2011, "C. Bergstr?m" wrote:

>> We are, but that problem is, well, "hard".  As in grand challenge hard.
> I wonder how you'd really describe the human brain learning in terms of a 
> programming model....

Well, AI researchers tend to have two answers.  One is semantic and the
other is microscopic.  The semantic description is
functional/operational, broken down in terms of e.g. conditionals and
logical elements, and doesn't come close to explaining consciousness
(see Searle's "Chinese Room" objection):

   http://en.wikipedia.org/wiki/Chinese_room

The microscopic model is basically neural networks, but NNAI hasn't made
it much past the level of a flatworm or an ant in AI-based models.  NNs
are marvelously useful nonlinear function approximators and hence are
quite capable of building Bayesian reasoning processes through inference
(if appropriately architected) but there the results are semantically or
functionally much like the human brain, its more like we think that SOME
parts of what the human brain does are SOMEWHAT mediated by networks
that have enormous structure (most of it occult) that does all sorts of
things (most of them unknown) to produce results (that nobody can
observe from the OUTSIDE of the brain save highly indirectly, with a nod
to FMRI and certain wetware neural implants which recently have had some
success in studying dynamic brain funciton in situ as near-exceptions).

Like I said, a hard problem, both philosophically, mathematically,
statistically, computationally, biologically, psychologically, and
semantically/information theoretically.  Partly because the solution
involves at least all of these fields in synthesis (and, if you are a
religious sort, you can throw in idle speculation about higher
dimensional Universes where some fraction of "intelligence" resides in
and/or utilizes physics in the additional dimensions).

>> There are other problems -- brains are highly non-deterministic (in that
>> they often selectively amplify tiny nearly random signals, as in "having
>> an idea" or "reacting completely differently depending on our hormonal
>> state, our immediate past history, and the phase of the moon").  Brains
>> are extremely non-Markovian with all sorts of multiple-time-scale memory
>> and with plenty of functional structures we honestly don't understand
>> much at all.  We don't even know how brains ENCODE memory -- when I've
>> got Sheryl Crow running through my head in what SEEM to be remarkably
>> good fidelity,
> I wonder just how good it is?  If I had to guess I'd say pretty bad.  (no 
> offense)  I wonder how much actual "space" it takes up.  I'd bet from a 
> physical size perspective we're possibly ahead of nature in terms of data 
> storage density.  (Without the packaging/cases.. etc)

It's actually damn good.  I can't remember the exact numbers and have to
teach and can't look them up, but my recollection is that the total
sensory bandwidth and bandwidth of the memory chanels in the human brain
are estimated at terabits -- just think of your visual field.  One thing
that is clear is that the brain is extremely efficient at information
storage and representation, using all sorts of tricks to compress
information.  However, it is highly error prone.

> I think motor functions are better understood than pure thought.

Sure, and they are rather boring.  Circuitry to activate a mechanical
process is straightforward (although sometimes nontrivial).  But where
the impulses come from to intentionally activate motor function, ah,
that's one of the many rubs.

>> Humans can "almost" remember somebody's name one day, for example, and
>> then another day know it immediately,
> cache hit
>> and another day still not even
>> recognize that they once knew it.
> cache miss

But it isn't that simple.  Not at all.  First of all, there is no cache,
this is all long term storage retrieval as the brain's immediate, short
and intermediate term memory (where immediate is the only one that is
arguably "cache") is all differentiated on much shorter timescales than
weeks or months.  Second of all, human retrieval is in some sense
associative and spontaneous, rather than a lookup process.  We KNOW how
computers look things up, and it is nothing at all like the way people
remember things.  Human memory is perhaps very slightly like using
hashes to store and retrieve information in computing, but in another
sense it is nothing at all like it.  The computer, one way or another,
stores a datum >>at a location<< and retrieves it by establishing a map
to the location.  The brain does not.  You do not have a location in
your brain that corresponds to your memory of e.g. what you ate for
dinner last night, or the value of pi.  That information is stored all
over the place, and the neurons that help store it may well be
simultaneously helping to store other information at the same time.
There also isn't a single pathway to that information (enter the correct
hash key, decode it into an address) -- there are multiple pathways and
some of them can be triggered by irrelevant stimuli.

There is also enormous variability in ability and function.  Some humans
remember names like magic, and can remember thousands of people by name.
Others (like me) have to think hard to remember the names of my own kids
and nephews and neices, are hopeless with student names.  Which isn't a
function of intelligence -- politicians are often great at names and
dumb as oxen, and I'm, well, a physicists/polymath sort of guy who STILL
can't remember the value of e and who has to derive everything he
teaches because it is EASIER than trying to remember it all.

Don't get me wrong -- I think we have true AI about licked, but I think
semantic AI is a false pathway to a Chinese Room.

    rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu


From rgb at phy.duke.edu  Wed Feb 16 10:13:45 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Wed, 16 Feb 2011 13:13:45 -0500 (EST)
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <4D5BF6F0.8080103@ias.edu>
References: <4D5BF6F0.8080103@ias.edu>
Message-ID: <alpine.LFD.2.02.1102161308260.3531@lilith>

On Wed, 16 Feb 2011, Prentice Bisbal wrote:

> Anyone see this yet? He's pretty dead on, IMHO, especially 5,6,9
>
>
> http://www.infoworld.com/t/unix/nine-traits-the-veteran-unix-admin-276

I think he's a bit narrow about vi -- I understand it, you have to be
competent with vi because vi is the only editor that (used to) be in
/sbin and hence might well be the only editor you have to work with
(except for ed) on a crashed system.  But real sysadmins don't
necessarily LIKE to work with vi, and emacs isn't the only alternative.
One of the best admins I've known worked with joe, a wordstar clone, and
was very happy with it.  I personally am crippled without jove (a
C-based binary, non-lisp small emacs ("Jonathan's Own Version of Emacs).
It is lightning fast, can do the integrated compiler/editor debugging
thing that emacs does, is highly customizable (but not as much as
emacs), and is NOT all things to all humans (although I have run
multiple shells in multiple jove windows instead of using e.g. screen or
multiple xterms, I don't view it as an operating environment).

Am I a veteran?  Well, I started doing Unix sysadmin in 1986.  Not the
veteranist of all veterans, but not exactly a tyro either...;-)

Most of the rest, sure.

    rgb

>
> --
> Prentice
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu




From prentice at ias.edu  Wed Feb 16 11:02:21 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Wed, 16 Feb 2011 14:02:21 -0500
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <59778.192.168.93.213.1297702771.squirrel@mail.eadline.org>
References: <E1Pnd1W-0003Vi-26@mendel.bio.caltech.edu>
	<59778.192.168.93.213.1297702771.squirrel@mail.eadline.org>
Message-ID: <4D5C1F3D.2030101@ias.edu>

Doug,

Jeopardy is in ABC in my neck of the woods. I believe it's syndicated,
so it might be on different networks in different cities.

Douglas Eadline wrote:
> For those who are not aware, tonight is the face-off between
> man and machine (it is not live, however). This is no chess game.
> Today through Wednesday, on the game show Jeopardy (CBS Network)
> the best human players will attempt to beat Watson, the 2880 core
> IBM Jeopardy computer.
> 
> Small write up on ClusterMonkey with some links:
> 
>   http://www.clustermonkey.net//content/view/290/1/
> 
> I predict one more defeat for the carbon based life forms.
> 
> --
> Doug
> 

-- 
Prentice


From deadline at eadline.org  Wed Feb 16 12:43:23 2011
From: deadline at eadline.org (Douglas Eadline)
Date: Wed, 16 Feb 2011 15:43:23 -0500 (EST)
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5C1F3D.2030101@ias.edu>
References: <E1Pnd1W-0003Vi-26@mendel.bio.caltech.edu>
	<59778.192.168.93.213.1297702771.squirrel@mail.eadline.org>
	<4D5C1F3D.2030101@ias.edu>
Message-ID: <60803.192.168.93.213.1297889003.squirrel@mail.eadline.org>

I noticed that as well. It seems CBS produces the show
but as you say it is syndicated. It is on ABC in my town as well.

--
Doug


> Doug,
>
> Jeopardy is in ABC in my neck of the woods. I believe it's syndicated,
> so it might be on different networks in different cities.
>
> Douglas Eadline wrote:
>> For those who are not aware, tonight is the face-off between
>> man and machine (it is not live, however). This is no chess game.
>> Today through Wednesday, on the game show Jeopardy (CBS Network)
>> the best human players will attempt to beat Watson, the 2880 core
>> IBM Jeopardy computer.
>>
>> Small write up on ClusterMonkey with some links:
>>
>>   http://www.clustermonkey.net//content/view/290/1/
>>
>> I predict one more defeat for the carbon based life forms.
>>
>> --
>> Doug
>>
>
> --
> Prentice
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


-- 
Doug

-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.



From nixon at nsc.liu.se  Thu Feb 17 02:51:14 2011
From: nixon at nsc.liu.se (Leif Nixon)
Date: Thu, 17 Feb 2011 11:51:14 +0100
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <alpine.LFD.2.02.1102161308260.3531@lilith> (Robert G. Brown's
	message of "Wed, 16 Feb 2011 13:13:45 -0500 (EST)")
References: <4D5BF6F0.8080103@ias.edu>
	<alpine.LFD.2.02.1102161308260.3531@lilith>
Message-ID: <m37hczx7bx.fsf@stuor.nsc.liu.se>

"Robert G. Brown" <rgb at phy.duke.edu> writes:

> I personally am crippled without jove (a C-based binary, non-lisp
> small emacs

Non-lisp Emacs? What's the point? I bet you drink decaf coffee as well.

-- 
Leif Nixon - Security officer
National Supercomputer Centre - Swedish National Infrastructure for Computing
Nordic Data Grid Facility - European Grid Infrastructure


From rgb at phy.duke.edu  Thu Feb 17 06:04:19 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 17 Feb 2011 09:04:19 -0500 (EST)
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <m37hczx7bx.fsf@stuor.nsc.liu.se>
References: <4D5BF6F0.8080103@ias.edu>
	<alpine.LFD.2.02.1102161308260.3531@lilith>
	<m37hczx7bx.fsf@stuor.nsc.liu.se>
Message-ID: <alpine.LFD.2.02.1102170856070.3085@lilith>

On Thu, 17 Feb 2011, Leif Nixon wrote:

> "Robert G. Brown" <rgb at phy.duke.edu> writes:
>
>> I personally am crippled without jove (a C-based binary, non-lisp
>> small emacs
>
> Non-lisp Emacs? What's the point? I bet you drink decaf coffee as well.

Half-decaf.  And I use half-and-half, too, to avoid the >>bloat<< caused
by full-caf double espressos with real cream... ;-)

OTOH, I can build jove out of a tarball a bit less than half a
>>megabyte<< in size, part of which is taken up by xjove, an embedded
project that nobody every bothered to finish.

How big are the emacs sources these days?  I mean, the installed binary
and common packages alone are 22 MB in F14... and was that 73 MB
installed?  Why, it was, wasn't it.  :-)

Every now and then I wish somebody would hack jove to handle tex/latex
make errors (but not enough to do it myself) but then I think nah...
best leave it as it is, nice and tight, no colored letters, no
info-of-evil integration, small and efficient...

    rgb

>
> -- 
> Leif Nixon - Security officer
> National Supercomputer Centre - Swedish National Infrastructure for Computing
> Nordic Data Grid Facility - European Grid Infrastructure
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu




From Michael.Frese at NumerEx-LLC.com  Thu Feb 17 06:14:47 2011
From: Michael.Frese at NumerEx-LLC.com (Michael H. Frese)
Date: Thu, 17 Feb 2011 07:14:47 -0700
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <4D5BF6F0.8080103@ias.edu>
References: <4D5BF6F0.8080103@ias.edu>
Message-ID: <6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>

Let the Editor Wars begin!


Mike

At 09:10 AM 2/16/2011, Prentice Bisbal wrote:
>Anyone see this yet? He's pretty dead on, IMHO, especially 5,6,9
>
>
>http://www.infoworld.com/t/unix/nine-traits-the-veteran-unix-admin-276
>
>--
>Prentice
>
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>To change your subscription (digest mode or unsubscribe) visit 
>http://www.beowulf.org/mailman/listinfo/beowulf




From james.p.lux at jpl.nasa.gov  Thu Feb 17 06:17:18 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Thu, 17 Feb 2011 06:17:18 -0800
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
References: <4D5BF6F0.8080103@ias.edu>,
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471CE@ALTPHYEMBEVSP20.RES.AD.JPL>

"the best editor"
"what is ???"
________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of Michael H. Frese [Michael.Frese at NumerEx-LLC.com]
Sent: Thursday, February 17, 2011 06:14
To: Prentice Bisbal; Beowulf Mailing List
Subject: Re: [Beowulf] 9 traits of the veteran Unix admin

Let the Editor Wars begin!



From cbergstrom at pathscale.com  Thu Feb 17 06:20:05 2011
From: cbergstrom at pathscale.com (=?ISO-8859-1?Q?=22C=2E_Bergstr=F6m=22?=)
Date: Thu, 17 Feb 2011 21:20:05 +0700
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
Message-ID: <4D5D2E95.3070901@pathscale.com>

Michael H. Frese wrote:
> Let the Editor Wars begin!
>   
Does that mean it's time to start a beowulf adapted drinking game[1] for 
each post about which editor is best going forward?

[1] http://blog.joelesler.net/the-snort-drinking-game


From nixon at nsc.liu.se  Thu Feb 17 06:22:49 2011
From: nixon at nsc.liu.se (Leif Nixon)
Date: Thu, 17 Feb 2011 15:22:49 +0100
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <alpine.LFD.2.02.1102170856070.3085@lilith> (Robert G. Brown's
	message of "Thu, 17 Feb 2011 09:04:19 -0500 (EST)")
References: <4D5BF6F0.8080103@ias.edu>
	<alpine.LFD.2.02.1102161308260.3531@lilith>
	<m37hczx7bx.fsf@stuor.nsc.liu.se>
	<alpine.LFD.2.02.1102170856070.3085@lilith>
Message-ID: <m3mxlupwp2.fsf@stuor.nsc.liu.se>

"Robert G. Brown" <rgb at phy.duke.edu> writes:

> How big are the emacs sources these days?  I mean, the installed binary
> and common packages alone are 22 MB in F14... and was that 73 MB
> installed?  Why, it was, wasn't it.  :-)

And it's all full of goodness!

-- 
Leif Nixon - Security officer
National Supercomputer Centre - Swedish National Infrastructure for Computing
Nordic Data Grid Facility - European Grid Infrastructure


From james.p.lux at jpl.nasa.gov  Thu Feb 17 06:28:48 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Thu, 17 Feb 2011 06:28:48 -0800
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <4D5D2E95.3070901@pathscale.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>,
	<4D5D2E95.3070901@pathscale.com>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471D0@ALTPHYEMBEVSP20.RES.AD.JPL>

Certainly, our list namesake was no stranger to drinking games...
And off topic digressions in the middle of a story, for that matter.
________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of "C. Bergstr?m" [cbergstrom at pathscale.com]
Sent: Thursday, February 17, 2011 06:20
To: Michael H. Frese
Cc: Beowulf Mailing List
Subject: Re: [Beowulf] 9 traits of the veteran Unix admin

Michael H. Frese wrote:
> Let the Editor Wars begin!
>
Does that mean it's time to start a beowulf adapted drinking game[1] for
each post about which editor is best going forward?


From prentice at ias.edu  Thu Feb 17 06:40:36 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Thu, 17 Feb 2011 09:40:36 -0500
Subject: [Beowulf] Wired article on AI
Message-ID: <4D5D3364.8010508@ias.edu>

Since Watson on Jeopardy! Lead to a discussion on AI, did anyone else
read Wired's cover story from last month on AI? If not, here's the link
to it (at bottom).

The main gist of it is this: AI is all around us today, but it's nothing
like we expected it to be. Instead of trying to model human
intelligence, which had long been the goal of AI, modern AI systems are
built to accomplish a specific task: you cars stability control and
collision avoidance systems, for example.

While Watson isn't mentioned in the article, it's relevant to him (it?)
since he's not AI in the classic definition, but is in the modern
definintion - he's a computer system built to handle a specific task
well - Jeopardy!

http://www.wired.com/magazine/2010/12/ff_ai_essay_airevolution/

-- 
Prentice


From bob at drzyzgula.org  Thu Feb 17 07:33:14 2011
From: bob at drzyzgula.org (Bob Drzyzgula)
Date: Thu, 17 Feb 2011 10:33:14 -0500
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
Message-ID: <20110217153314.GA6611@mx1.drzyzgula.org>

There's a saying among photographers that the best camera
is the one you have with you. I think a similar argument
can be made for editors -- the best editor is the one
you know how to use.

While I am firmly in the vi camp -- the only emacs
command I ever really managed to learn was ^X^C -- there
are historical reasons for this.  I learned Unix on a
standalone system (a Sun 2/120 with SunOS 1.1) without
any access to the Internet, and vi was the only editor
available on a stock install of that system that was worth
learning. By the time I had access to emacs a lot of my
vi knowledge had kind of moved into my brain stem.

While I tried to learn emacs a couple of times, it just
seemed like it took forever for the thing to load. I expect
with modern computers this is less of a concern, but at the
time, for a systems person who had to do small amounts of
work on a large number of systems rather than a lot of work
on a single system, this was a major issue.  Actually, I've
always assumed that any applications programmer/systems
programmer correlation with emacs/vi preferance stems
largely from this one factor (rgb: yes, I know, jove; but
sorry, you're in a very small club there). In any event,
on top of this I found emacs very alien and disconcerting,
so I gave up on it.

But none of this means that vi is "better" than emacs,
just that it is better for me, because I already know how
to use it. I could write a treatise about why I like vi,
but none of my arguments would ever be any more compelling
to an emacs user any than their affinity for the disaster
that is emacs :-) would be to me.

--Bob Drzyzgula


On 17/02/11 07:14 -0700, Michael H. Frese wrote:
> Let the Editor Wars begin!
> 
> 
> Mike
> 
> At 09:10 AM 2/16/2011, Prentice Bisbal wrote:
> >Anyone see this yet? He's pretty dead on, IMHO, especially 5,6,9
> >
> >
> >http://www.infoworld.com/t/unix/nine-traits-the-veteran-unix-admin-276
> >
> >--
> >Prentice
> >
> >_______________________________________________
> >Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> >To change your subscription (digest mode or unsubscribe) visit 
> >http://www.beowulf.org/mailman/listinfo/beowulf
> 
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From cbergstrom at pathscale.com  Thu Feb 17 07:44:14 2011
From: cbergstrom at pathscale.com (=?ISO-8859-1?Q?=22C=2E_Bergstr=F6m=22?=)
Date: Thu, 17 Feb 2011 22:44:14 +0700
Subject: [Beowulf] [OT] EMACS vs VI
In-Reply-To: <20110217153314.GA6611@mx1.drzyzgula.org>
References: <4D5BF6F0.8080103@ias.edu>	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
Message-ID: <4D5D424E.3060806@pathscale.com>

Bob Drzyzgula wrote:
> There's a saying among photographers that the best camera
> is the one you have with you. I think a similar argument
> can be made for editors -- the best editor is the one
> you know how to use.
>
> While I am firmly in the vi camp -- the only emacs
> command I ever really managed to learn was ^X^C -- there
> are historical reasons for this.  I learned Unix on a
> standalone system (a Sun 2/120 with SunOS 1.1) without
> any access to the Internet, and vi was the only editor
> available on a stock install of that system that was worth
> learning. By the time I had access to emacs a lot of my
> vi knowledge had kind of moved into my brain stem.
>
> While I tried to learn emacs a couple of times, it just
> seemed like it took forever for the thing to load.
fwiw some people still *really* care about emacs performance even on 
modern hw.  We (PathScale) have a few vocal users and plan to spin some 
cycles to speed it up a bit.  (If you're interested to help test/give 
feedback ping me off list.)

+1 vim

./C


/me takes 2 shots


From rgb at phy.duke.edu  Thu Feb 17 07:47:55 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 17 Feb 2011 10:47:55 -0500 (EST)
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <4D5D2E95.3070901@pathscale.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<4D5D2E95.3070901@pathscale.com>
Message-ID: <alpine.LFD.2.02.1102171043460.3085@lilith>

On Thu, 17 Feb 2011, "C. Bergstr?m" wrote:

> Michael H. Frese wrote:
>> Let the Editor Wars begin!
>>
> Does that mean it's time to start a beowulf adapted drinking game[1] for
> each post about which editor is best going forward?
>
> [1] http://blog.joelesler.net/the-snort-drinking-game
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

No, I think that we would have to just agree to create
the-beowulf-drinking-game.  Is this open source, I wonder?  Will we
violate any sort of copyright?

Then we can >>apply<< it to editor letters, "How do I build a Beowulf"
questions, "Does Beowulf run on Windows" questions, compiler wars, the
eternal "If I install Beowulf on my computer, will my (fill in the
serial coded blank) run faster?" question etc.

I'm game.  Any excuse to drink beer is a good one, now that I make my
own (needless to say, perfect) beer...

    rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu


From bob at drzyzgula.org  Thu Feb 17 07:48:55 2011
From: bob at drzyzgula.org (Bob Drzyzgula)
Date: Thu, 17 Feb 2011 10:48:55 -0500
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <4D5D424E.3060806@pathscale.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
Message-ID: <20110217154855.GB6611@mx1.drzyzgula.org>


On 17/02/11 22:44 +0700, "C. Bergstr?m" wrote:
> Bob Drzyzgula wrote:
>>
>> While I tried to learn emacs a couple of times, it just
>> seemed like it took forever for the thing to load.
> fwiw some people still *really* care about emacs performance even on  
> modern hw.  We (PathScale) have a few vocal users and plan to spin some  
> cycles to speed it up a bit.  (If you're interested to help test/give  
> feedback ping me off list.)

It is one of my standing, and I'm sure tiresome, jokes
that when someone asks me how fast a new system seems
to be, I respond with something along the lines of
"well, vi just screams". Vi, of course, having "screamed"
on a MC68010 with 2MB of memory.

--Bob


From bug at sas.upenn.edu  Thu Feb 17 07:55:33 2011
From: bug at sas.upenn.edu (Gavin W. Burris)
Date: Thu, 17 Feb 2011 10:55:33 -0500
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <20110217153314.GA6611@mx1.drzyzgula.org>
References: <4D5BF6F0.8080103@ias.edu>	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
Message-ID: <4D5D44F5.60505@sas.upenn.edu>

I'm an emacs-to-vim convert.  Join me, brothers and sisters!

But seriously.... I used to spend way too much time customizing every
aspect of ever aplication I used.  This included my .emacs.el file.
I've stopped doing this because I have real programming to do, and I
have to choose my battles.

Now I'm more of the minimal-changes, as-default-as-possible camp.  Out
of the box, vim does everything I need it to do, and I don't waste hours
every time an update breaks my many-layers-of-dependencies emacs
environment.  Trying to do everything in emacs was one of the worst
timesyncs I've every engaged in.

Keep it simple, learn your editors use of regular expressions, and use
automation shortcuts.

Cheers.

On 02/17/2011 10:33 AM, Bob Drzyzgula wrote:
> There's a saying among photographers that the best camera
> is the one you have with you. I think a similar argument
> can be made for editors -- the best editor is the one
> you know how to use.ul 16, 2009 9:00 AM
> 
> While I am firmly in the vi camp -- the only emacs
> command I ever really managed to learn was ^X^C -- there
> are historical reasons for this.  I learned Unix on a
> standalone system (a Sun 2/120 with SunOS 1.1) without
> any access to the Internet, and vi was the only editor
> available on a stock install of that system that was worth
> learning. By the time I had access to emacs a lot of my
> vi knowledge had kind of moved into my brain stem.
> 
> While I tried to learn emacs a couple of times, it just
> seemed like it took forever for the thing to load. I expect
> with modern computers this is less of a concern, but at the
> time, for a systems person who had to do small amounts of
> work on a large number of systems rather than a lot of work
> on a single system, this was a major issue.  Actually, I've
> always assumed that any applications programmer/systems
> programmer correlation with emacs/vi preferance stems
> largely from this one factor (rgb: yes, I know, jove; but
> sorry, you're in a very small club there). In any event,
> on top of this I found emacs very alien and disconcerting,
> so I gave up on it.
> 
> But none of this means that vi is "better" than emacs,
> just that it is better for me, because I already know how
> to use it. I could write a treatise about why I like vi,
> but none of my arguments would ever be any more compelling
> to an emacs user any than their affinity for the disaster
> that is emacs :-) would be to me.
> 
> --Bob Drzyzgula
> 
> 
> On 17/02/11 07:14 -0700, Michael H. Frese wrote:
>> Let the Editor Wars begin!
>>
>>
>> Mike
>>
>> At 09:10 AM 2/16/2011, Prentice Bisbal wrote:
>>> Anyone see this yet? He's pretty dead on, IMHO, especially 5,6,9
>>>
>>>
>>> http://www.infoworld.com/t/unix/nine-traits-the-veteran-unix-admin-276
>>>
>>> --
>>> Prentice
>>>
>>> _______________________________________________
>>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>>> To change your subscription (digest mode or unsubscribe) visit 
>>> http://www.beowulf.org/mailman/listinfo/beowulf
>>
>>
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Gavin W. Burris
Senior Systems Programmer
Information Security and Unix Systems
School of Arts and Sciences
University of Pennsylvania


From nixon at nsc.liu.se  Thu Feb 17 07:56:47 2011
From: nixon at nsc.liu.se (Leif Nixon)
Date: Thu, 17 Feb 2011 16:56:47 +0100
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <alpine.LFD.2.02.1102171043460.3085@lilith>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<4D5D2E95.3070901@pathscale.com>
	<alpine.LFD.2.02.1102171043460.3085@lilith>
Message-ID: <AANLkTimAHN95rzc1fMcmDX94+PQ2xOuu3NbdkbMdzQ6=@mail.gmail.com>

2011/2/17 Robert G. Brown <rgb at phy.duke.edu>:
>
> I'm game. ?Any excuse to drink beer is a good one, now that I make my
> own (needless to say, perfect) beer...

Sans alcohol, I presume.

-- 
Leif Nixon? ? ? ? ? ? ? ? ? ? ?? -? ? ? ? ? ? Systems expert
------------------------------------------------------------
National Supercomputer Centre? ? -? ? ? Linkoping University
------------------------------------------------------------


From rgb at phy.duke.edu  Thu Feb 17 07:59:58 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 17 Feb 2011 10:59:58 -0500 (EST)
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <m3mxlupwp2.fsf@stuor.nsc.liu.se>
References: <4D5BF6F0.8080103@ias.edu>
	<alpine.LFD.2.02.1102161308260.3531@lilith>
	<m37hczx7bx.fsf@stuor.nsc.liu.se>
	<alpine.LFD.2.02.1102170856070.3085@lilith>
	<m3mxlupwp2.fsf@stuor.nsc.liu.se>
Message-ID: <alpine.LFD.2.02.1102171048220.3085@lilith>

On Thu, 17 Feb 2011, Leif Nixon wrote:

> "Robert G. Brown" <rgb at phy.duke.edu> writes:
>
>> How big are the emacs sources these days?  I mean, the installed binary
>> and common packages alone are 22 MB in F14... and was that 73 MB
>> installed?  Why, it was, wasn't it.  :-)
>
> And it's all full of goodness!

I'll bet it is!

And somewhere in the world, just imagine, there is actually a geek -- in
fact, I'd like to think of him as >>the<< ubergeek, an actual
evolutionary advance in ordinary geeks (at least, he would be if he were
ever able to reproduce), the geek whose blood serum is 2/3 Jolt cola and
1/3 Mountain Dew and who last actually got up from his workstation
console in 1994, the first year one was able to order new computer
hardware online for home delivery from the Web (using emacs ported to
the Commodore Amiga, of course, to actually >>access<< the web) -- who
knows >>every feature<< of emacs, how to use every single thing that it
can do.  In fact, like many emacs users, he doesn't bother with silly
things like mail clients, word processors, web browsers, editors,
windowing interfaces, or command line interfaces OUTSIDE of emacs,
because he never actually exits emacs.  Secretly, he doesn't really see
the point of consoles, xterms, shells and most software, because emacs
can simply replace them all.

I'm guessing that he has figured out how to hotwire init so that it just
forks emacs and skips everything else.  That is, assuming that he hasn't
actually hacked emacs directly into the kernel.  Who needs init, anyway?
One can start anything just as easily from inside emacs as by forking
off of init, right...?

;-)

     rgb

>
> -- 
> Leif Nixon - Security officer
> National Supercomputer Centre - Swedish National Infrastructure for Computing
> Nordic Data Grid Facility - European Grid Infrastructure
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu




From nixon at nsc.liu.se  Thu Feb 17 08:07:58 2011
From: nixon at nsc.liu.se (Leif Nixon)
Date: Thu, 17 Feb 2011 17:07:58 +0100
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <alpine.LFD.2.02.1102171048220.3085@lilith>
References: <4D5BF6F0.8080103@ias.edu>
	<alpine.LFD.2.02.1102161308260.3531@lilith>
	<m37hczx7bx.fsf@stuor.nsc.liu.se>
	<alpine.LFD.2.02.1102170856070.3085@lilith>
	<m3mxlupwp2.fsf@stuor.nsc.liu.se>
	<alpine.LFD.2.02.1102171048220.3085@lilith>
Message-ID: <AANLkTinbAhYpVC6L6+sjKKobEG=rifmDH1HXv_NibHHb@mail.gmail.com>

On 17 February 2011 16:59, Robert G. Brown <rgb at phy.duke.edu> wrote:

> I'm guessing that he has figured out how to hotwire init so that it just
> forks emacs and skips everything else. ?That is, assuming that he hasn't
> actually hacked emacs directly into the kernel. ?Who needs init, anyway?
> One can start anything just as easily from inside emacs as by forking
> off of init, right...?

Actually, what you do is you supply "init=/usr/bin/emacs" as a kernel
boot argument. Works just fine.

-- 
Leif Nixon? ? ? ? ? ? ? ? ? ? ?? -? ? ? ? ? ? Systems expert
------------------------------------------------------------
National Supercomputer Centre? ? -? ? ? Linkoping University
------------------------------------------------------------


From rgb at phy.duke.edu  Thu Feb 17 08:13:23 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 17 Feb 2011 11:13:23 -0500 (EST)
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <9E1EAF5C-B573-44EB-AB3C-7EF1E47FC8EB@gmail.com>
References: <4D5BF6F0.8080103@ias.edu>
	<alpine.LFD.2.02.1102161308260.3531@lilith>
	<m37hczx7bx.fsf@stuor.nsc.liu.se>
	<alpine.LFD.2.02.1102170856070.3085@lilith>
	<9E1EAF5C-B573-44EB-AB3C-7EF1E47FC8EB@gmail.com>
Message-ID: <alpine.LFD.2.02.1102171100280.3085@lilith>

On Thu, 17 Feb 2011, Lawrence Stewart wrote:

> I have a nice "me" or microemacs, written by Dave Conroy, back in the day.  The tarball is 121K.
> I asked him how to change the keybindings and he said "use the change configuration command".
> "It's called "cc" on the system."

Jove is one notch up from that.  It does parse a .joverc and come with
an integrated tutorial and actual documentation.  But otherwise they are
probably very similar.

It does have its own wikipedia page:

   http://en.wikipedia.org/wiki/JOVE

and I originally used it on PDPs in 1986, on Suns by 1987, back when it
was still quite young.  That's why I can't, actually, give it up even
though it is a mild PITA to keep it compiling as libc and termcap and
some of the other things the original BSD Unix sources relied on have
gradually morphed.  The current 2011 version is up to 4.16.0.73, and is
distributed in Debian and trivially buildable into an RPM from a
specfile in the tarball.  I wish it were in Fedora; I have to build my
own as one of the first things I do after each upgrade or install,
pecking away in vi as required until then, cursing...;-)

I see microemacs also has its own wikipedia page.  And ooo, Linus
Torvalds uses a linear descendent of microemacs!  I think that does it!
Small/compiled emacs variants are an acceptable alternative to vi on the
Real Systems Administrator checksheet.

Real Emacs is OK (I suppose, scoff scoff) for developers who work on fat
systems and like to have their hands held with color coding superimposed
on e.g. html or latex or c code -- I personally find it rather
distracting and have never been able to see light green on white worth a
damn, but the real reason I don't use emacs is that its keybindings
aren't the same as jove's...

    rgb

>
> -L
>
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu




From landman at scalableinformatics.com  Thu Feb 17 08:19:32 2011
From: landman at scalableinformatics.com (Joe Landman)
Date: Thu, 17 Feb 2011 11:19:32 -0500
Subject: [Beowulf] [OT] EMACS vs VI
In-Reply-To: <4D5D424E.3060806@pathscale.com>
References: <4D5BF6F0.8080103@ias.edu>	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
Message-ID: <4D5D4A94.4070507@scalableinformatics.com>

shades of editor wars from ...  decades ago ...

e2 and e3 (a programming editor at IBM TJ Watson, never made it out the 
door as far as I remember, though OS2 had some build variant of it built 
in), was amazing.  Pretty good by todays standards.

If I'm remote, no higher bandwidth link, I'll use vim, pico/nano, or vi. 
  If there is a higher bandwidth connection, I've used nedit (wrote a 
thesis in an earlier version), and am largely switching to kate .  I 
just could never grok emacs.  I could get TeX and LaTeX, and many other 
arcane things.  I just couldn't get my mind around emacs.  Or vi for 
that matter.  I am ok with it, but after 20+ years using it, I am 
*still* dangerous with it.  I am thankful for the undo feature in vim.

kate is the closest thing to a good editor I've used since e2.  e2 just 
rocked.  This was 1985-ish or so, and it still was good by todays 
standards.  Never made it out the door though.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615


From prentice at ias.edu  Thu Feb 17 08:27:12 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Thu, 17 Feb 2011 11:27:12 -0500
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471D0@ALTPHYEMBEVSP20.RES.AD.JPL>
References: <4D5BF6F0.8080103@ias.edu>	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>,
	<4D5D2E95.3070901@pathscale.com>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471D0@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <4D5D4C60.7000300@ias.edu>

We would need to add "Drink *heavily* when crazy dutchman hijacks any
thread to go completely off-topic to criticize the actions of the United
States government"


Lux, Jim (337C) wrote:
> Certainly, our list namesake was no stranger to drinking games...
> And off topic digressions in the middle of a story, for that matter.
> ________________________________________
> From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of "C. Bergstr?m" [cbergstrom at pathscale.com]
> Sent: Thursday, February 17, 2011 06:20
> To: Michael H. Frese
> Cc: Beowulf Mailing List
> Subject: Re: [Beowulf] 9 traits of the veteran Unix admin
> 
> Michael H. Frese wrote:
>> Let the Editor Wars begin!
>>
> Does that mean it's time to start a beowulf adapted drinking game[1] for
> each post about which editor is best going forward?
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
> 

-- 
Prentice


From rgb at phy.duke.edu  Thu Feb 17 08:38:13 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 17 Feb 2011 11:38:13 -0500 (EST)
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <20110217154855.GB6611@mx1.drzyzgula.org>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
	<20110217154855.GB6611@mx1.drzyzgula.org>
Message-ID: <alpine.LFD.2.02.1102171117250.3085@lilith>

On Thu, 17 Feb 2011, Bob Drzyzgula wrote:

>
> On 17/02/11 22:44 +0700, "C. Bergstr?m" wrote:
>> Bob Drzyzgula wrote:
>>>
>>> While I tried to learn emacs a couple of times, it just
>>> seemed like it took forever for the thing to load.
>> fwiw some people still *really* care about emacs performance even on
>> modern hw.  We (PathScale) have a few vocal users and plan to spin some
>> cycles to speed it up a bit.  (If you're interested to help test/give
>> feedback ping me off list.)
>
> It is one of my standing, and I'm sure tiresome, jokes
> that when someone asks me how fast a new system seems
> to be, I respond with something along the lines of
> "well, vi just screams". Vi, of course, having "screamed"
> on a MC68010 with 2MB of memory.

Well, that >>is<< the advantage of relatively compact compiled code over
lisp, right?

Addressing your last post, a lot of sysadmins and top-gun systems
programmers and so on do use non-vi compiled editors, notably jove, me,
joe, although as you note anybody who started the game in the 80s had to
learn vi because (among other things) it was all that you could be more
or less assured of having access to on a new system install or a crashed
system with no /usr.  Even older guys could still use ed on an actual
teletype (not a tty interface, mind you, I'm talking the actual printer
terminal).  I did my share of that, but I really hated it, to be honest.

vi had the advantage of letting you do fullscreen, and lots of early
terminal games used vi key commands to navigate and hence were actually
vi learning games as well -- good old hjkl.

The two reasons I came to dislike vi and abandoned it for jove were:

   a) Moving in and out of insert mode is a pita.  Yes, a lot of the time
one just uses direct edit commands in command mode instead of navigation
and retyping (shades of ed) but for someone who learned to >>type<<
before they learned to edit, this was actually counterintuitive.  jove
was more "typewriter like" and less QED/ED like.  Note well that in my
terminal timeshare days I wrote a complete basic front end for QED that
turned it INTO a fullscreen editor because editing a program sucked
using line commands (I was probably one of the only humans in the world
that could edit my fortran using QED under TSO with cursor keys and a
full screen of text and block movement and so on, even though I did have
to roll my own to get it:-).

   b) Jove runs (ran) make from inside and was smart enough to permit one
to run through the syntax errors a la full emacs.  No popping in and out
with suspends.  In fact, at one point in time I too kept three internal
windows open, one for source, one for the make/errors to run in, and one
for a shell so I could actually compile, debug, and runtime test the
program without leaving jove.  But eventually tcsh and bash got good
enough that the advantages of a CLI inside jove (or emacs) to give you
editing and replayability wasn't worth the hassle, and I started to
suspend once again to runtime debug or just run the code.

I'm sure that by now vim or whatever has long since subsumed these
advantages so that there is no real need for me to stick with jove, but
as you said (I think) -- once you've used an editor for a decade or
more, it is >>very very difficult to change<<.  I still more or less
constantly pop up strange printscreen windows and so on inside
browser-based editors because I forget and I pop a Ctrl-P in trying to
move up a line.  My fingers move like lightning, and I can navigate
instantly without my fingers leaving the home keys...

I had the damnedest time when IBM moved the Ctrl-shift and introduced
the Alt key.  Dark evil, carpal tunnel inducing change.  I still resent
it -- nobody ever uses caps lock and there it is, wasting primary
keyboard space to the left of the a one lousy cm away, where the Ctrl is
now a full fifth-left-wrist-pivot and two row down movement.  I'd type
>>even faster<< if it were still where it belonged, but remapping the
keyboard (although not too much of a problem if you only work on one
machine) makes you crippled every time you change to a different machine
without the map.

Naturally, I'd be happy to challenge >>anybody<< to a typing
speed/prolixity dual... and a lot of my speed advantage is, in fact, due
to my use of jove....

     rgb

>
> --Bob
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu


From bob at drzyzgula.org  Thu Feb 17 08:39:08 2011
From: bob at drzyzgula.org (Bob Drzyzgula)
Date: Thu, 17 Feb 2011 11:39:08 -0500
Subject: [Beowulf] [OT] EMACS vs VI
In-Reply-To: <4D5D4A94.4070507@scalableinformatics.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
	<4D5D4A94.4070507@scalableinformatics.com>
Message-ID: <AANLkTinHw8vX+hSN4xHV7sp8O==N_7Qm7pxLg_BJDX7u@mail.gmail.com>

e2 & e3 -- were those the ones that, when you searched for a string it
would draw ovals around the results? I was fond of those as well, for
the limited time I had anything to do with OS/2. For a while, on our
Sun systems in the late '80s but before emacs gained any popularity,
we offered the RAND editor e19. It was largely function-key driven and
didn't know how to use termcap, so it was a PITA to port to a new
terminal, but on a VT2xx clone it was pretty nice for the time -- it
was pretty easy for a non-technical user.

On a lot of platforms in the early days there weren't a lot of
choices, you just used what was there. In the early '80s, I worked at
some places were Wylbur was just How One Did Things; but it was
actually pretty powerful. If you were working on VM/CMS you were
probably editing in xedit and scripting in Rexx. And if you were
working interactively on OS/MVS, the best you could hope for was
probably SPF/PDF.

--Bob

On Thu, Feb 17, 2011 at 11:19 AM, Joe Landman
<landman at scalableinformatics.com> wrote:
> shades of editor wars from ... ?decades ago ...
>
> e2 and e3 (a programming editor at IBM TJ Watson, never made it out the
> door as far as I remember, though OS2 had some build variant of it built
> in), was amazing. ?Pretty good by todays standards.
>
> If I'm remote, no higher bandwidth link, I'll use vim, pico/nano, or vi.
> ?If there is a higher bandwidth connection, I've used nedit (wrote a
> thesis in an earlier version), and am largely switching to kate . ?I
> just could never grok emacs. ?I could get TeX and LaTeX, and many other
> arcane things. ?I just couldn't get my mind around emacs. ?Or vi for
> that matter. ?I am ok with it, but after 20+ years using it, I am
> *still* dangerous with it. ?I am thankful for the undo feature in vim.
>
> kate is the closest thing to a good editor I've used since e2. ?e2 just
> rocked. ?This was 1985-ish or so, and it still was good by todays
> standards. ?Never made it out the door though.
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman at scalableinformatics.com
> web ?: http://scalableinformatics.com
> ? ? ? ?http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax ?: +1 866 888 3112
> cell : +1 734 612 4615
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>


From rgb at phy.duke.edu  Thu Feb 17 08:43:00 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 17 Feb 2011 11:43:00 -0500 (EST)
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <AANLkTimAHN95rzc1fMcmDX94+PQ2xOuu3NbdkbMdzQ6=@mail.gmail.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<4D5D2E95.3070901@pathscale.com>
	<alpine.LFD.2.02.1102171043460.3085@lilith>
	<AANLkTimAHN95rzc1fMcmDX94+PQ2xOuu3NbdkbMdzQ6=@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1102171140290.3085@lilith>

On Thu, 17 Feb 2011, Leif Nixon wrote:

> 2011/2/17 Robert G. Brown <rgb at phy.duke.edu>:
>>
>> I'm game. ?Any excuse to drink beer is a good one, now that I make my
>> own (needless to say, perfect) beer...
>
> Sans alcohol, I presume.

No, merely with >>a fraction<< of the alcohol in, say, Everclear...;-)

Although the batch I'm currently fermenting should be quite strong -- as
much as 7-8% if it ferments dry.  Not quite brandywine, but higher than
most commercial beers.  But usually I shoot for the usual 5-6%, with a
hint of sweetness counterbalancing the bitterness and aroma of the hops
and the rich biscuit of the barley...;-)

    rgb

>
> -- 
> Leif Nixon? ? ? ? ? ? ? ? ? ? ?? -? ? ? ? ? ? Systems expert
> ------------------------------------------------------------
> National Supercomputer Centre? ? -? ? ? Linkoping University
> ------------------------------------------------------------
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu


From james.p.lux at jpl.nasa.gov  Thu Feb 17 08:56:44 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Thu, 17 Feb 2011 08:56:44 -0800
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <alpine.LFD.2.02.1102171140290.3085@lilith>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<4D5D2E95.3070901@pathscale.com>	<alpine.LFD.2.02.1102171043460.3085@lilith>
	<AANLkTimAHN95rzc1fMcmDX94+PQ2xOuu3NbdkbMdzQ6=@mail.gmail.com>,
	<alpine.LFD.2.02.1102171140290.3085@lilith>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471E0@ALTPHYEMBEVSP20.RES.AD.JPL>

And you have a whole "cluster" of brewing vessels that you make small changes in to do a suitable Monte Carlo approach?

________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of Robert G. Brown [rgb at phy.duke.edu]
Sent: Thursday, February 17, 2011 08:43
To: Leif Nixon
Cc: Beowulf Mailing List
Subject: Re: [Beowulf] 9 traits of the veteran Unix admin

On Thu, 17 Feb 2011, Leif Nixon wrote:

> 2011/2/17 Robert G. Brown <rgb at phy.duke.edu>:
>>
>> I'm game.  Any excuse to drink beer is a good one, now that I make my
>> own (needless to say, perfect) beer...
>
> Sans alcohol, I presume.

No, merely with >>a fraction<< of the alcohol in, say, Everclear...;-)

Although the batch I'm currently fermenting should be quite strong -- as
much as 7-8% if it ferments dry.  Not quite brandywine, but higher than
most commercial beers.  But usually I shoot for the usual 5-6%, with a
hint of sweetness counterbalancing the bitterness and aroma of the hops
and the rich biscuit of the barley...;-)

    rgb

>
> --
> Leif Nixon                       -            Systems expert
> ------------------------------------------------------------
> National Supercomputer Centre    -      Linkoping University
> ------------------------------------------------------------
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown                        http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu



From bob at drzyzgula.org  Thu Feb 17 08:59:03 2011
From: bob at drzyzgula.org (Bob Drzyzgula)
Date: Thu, 17 Feb 2011 11:59:03 -0500
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <alpine.LFD.2.02.1102171117250.3085@lilith>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
	<20110217154855.GB6611@mx1.drzyzgula.org>
	<alpine.LFD.2.02.1102171117250.3085@lilith>
Message-ID: <AANLkTinmUEbwmubKdH=UR86FwgSky8Tp+8Q6c4JU7xcL@mail.gmail.com>

On Thu, Feb 17, 2011 at 11:38 AM, Robert G. Brown <rgb at phy.duke.edu> wrote:
>
> Even older guys could still use ed on an actual
> teletype (not a tty interface, mind you, I'm talking the actual printer
> terminal).

Of course. Been there, done that, on a dial-up terminal with a thermal
printer at 110 baud over an acoustic coupler. If you know how to use
sed you probably can suss out ed. I had to know how to use edlin in
DOS, too. But if you had vi at all, you also had ex, so that was a
much better option.


> ?I still more or less
> constantly pop up strange printscreen windows and so on inside
> browser-based editors because I forget and I pop a Ctrl-P in trying to
> move up a line. ?My fingers move like lightning, and I can navigate
> instantly without my fingers leaving the home keys...

hjkl, at least, just leave you with text you have to edit out. My
problem is always that ESC can have some very unwanted consequences.

> I had the damnedest time when IBM moved the Ctrl-shift and introduced
> the Alt key.

http://pckeyboards.stores.yahoo.net/linux101.html

> Naturally, I'd be happy to challenge >>anybody<< to a typing
> speed/prolixity dual... and a lot of my speed advantage is, in fact, due
> to my use of jove....

I've certainly had dyed-in-the-wool emacs users be shocked at how fast
I could make complex edits to system files in vi. Again, it's all a
matter of what you've trained your fingers to do.

--Bob


From stewart at serissa.com  Thu Feb 17 08:59:29 2011
From: stewart at serissa.com (Lawrence Stewart)
Date: Thu, 17 Feb 2011 11:59:29 -0500
Subject: [Beowulf] [OT] EMACS vs VI
In-Reply-To: <AANLkTinHw8vX+hSN4xHV7sp8O==N_7Qm7pxLg_BJDX7u@mail.gmail.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
	<4D5D4A94.4070507@scalableinformatics.com>
	<AANLkTinHw8vX+hSN4xHV7sp8O==N_7Qm7pxLg_BJDX7u@mail.gmail.com>
Message-ID: <D5C591EC-17DF-4B55-AFB2-1EE00D31765F@serissa.com>


On Feb 17, 2011, at 11:39 AM, Bob Drzyzgula wrote:

> On a lot of platforms in the early days there weren't a lot of
> choices, you just used what was there. In the early '80s, I worked at
> some places were Wylbur was just How One Did Things; but it was
> actually pretty powerful. If you were working on VM/CMS you were
> probably editing in xedit and scripting in Rexx. And if you were
> working interactively on OS/MVS, the best you could hope for was
> probably SPF/PDF.
> 
> --Bob

My advisor at Stanford had a sabbatical at Yorktown Heights sometime
in the mid '80s, and he reported having to use SOS on whatever 370 they had.
He said the UI was horrible, but the global search and replace was really fast.

I think I was using TECO on Tenex and ed on Unix around then.

-L




From egan at sense.net  Thu Feb 17 09:55:26 2011
From: egan at sense.net (Egan Ford)
Date: Thu, 17 Feb 2011 10:55:26 -0700
Subject: [Beowulf] [OT] EMACS vs VI
In-Reply-To: <4D5D424E.3060806@pathscale.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
Message-ID: <AANLkTikwMyehB0g=m0zamuVZ6yw8ZfJofdjrYhx9Mv9K@mail.gmail.com>

> Bob Drzyzgula wrote:
> > There's a saying among photographers that the best camera
> > is the one you have with you. I think a similar argument
> > can be made for editors -- the best editor is the one
> > you know how to use.
> >
> > While I am firmly in the vi camp
> >
> > While I tried to learn emacs a couple of times, it just
> > seemed like it took forever for the thing to load.

I am also in the vi camp.  Mostly because it is what I have been using
since the late '80s.  I remember building emacs for AIX in the early
'90s.  I was told of its amazing capabilities and that I should move
to it.  It felt like building and operating system.  It took forever
to build and if I striped out the debugging code it would coredump.
And, it was generally slow and a resource hog.  I never really learned
how to use emacs.

I like vi for its simplicity.  No function keys, no arrow keys
(required), no meta key.  Just the ESC and CTRL keys to work its
magic.   It's fast and efficient on any keyboard.  IMHO, of course.

Bill Joy had many achievements, IMHO vi tops the list.


From akshar.bhosale at gmail.com  Thu Feb 17 10:31:25 2011
From: akshar.bhosale at gmail.com (akshar bhosale)
Date: Fri, 18 Feb 2011 00:01:25 +0530
Subject: [Beowulf] maui: how to set different walltime for different users
Message-ID: <AANLkTinXX_tgtamDnykstGXHowPY5dFRhVHrnmUUB1wb@mail.gmail.com>

hi,

we have a cluster on 16 nodes where we run torque+maui.

We have set max walltime of 4 days for all jobs.  we want to set different
max walltimes for different users. e.g. user abc wants 5 days as max
walltime, user xyzwants 55 days as max walltime for a single job. we dont
want to create new queue.
kindly help
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110218/d1fc7ee6/attachment.html>

From mathog at caltech.edu  Thu Feb 17 10:34:58 2011
From: mathog at caltech.edu (David Mathog)
Date: Thu, 17 Feb 2011 10:34:58 -0800
Subject: [Beowulf] 9 traits of the veteran Unix admin
Message-ID: <E1Pq8gs-0007Ai-1S@mendel.bio.caltech.edu>

Editors are like cars, there are lots of them 

  http://en.wikipedia.org/wiki/Comparison_of_text_editors

and one person's preference may be incomprehensible to another.  
So do we really need to talk about editors? 

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech


From bill at Princeton.EDU  Thu Feb 17 10:35:38 2011
From: bill at Princeton.EDU (Bill Wichser)
Date: Thu, 17 Feb 2011 13:35:38 -0500
Subject: [Beowulf] maui: how to set different walltime for different
	users
In-Reply-To: <AANLkTinXX_tgtamDnykstGXHowPY5dFRhVHrnmUUB1wb@mail.gmail.com>
References: <AANLkTinXX_tgtamDnykstGXHowPY5dFRhVHrnmUUB1wb@mail.gmail.com>
Message-ID: <4D5D6A7A.5080407@princeton.edu>


We do this by individual requests to extend walltime and as sysadmin use 
the "qalter -l walltime=720:00:00 ,jobid>" to increase to 720 hours 
say.  If it isn't too many jobs involved in the requests, we can easily 
handle this.  Otherwise, a queue per user or time slot, protected with 
ACL, is going to be the only way around this.

Bill

akshar bhosale wrote:
> hi,
>
> we have a cluster on 16 nodes where we run torque+maui.
>
> We have set max walltime of 4 days for all jobs.  we want to set 
> different max walltimes for different users. e.g. user abc wants 5 
> days as max walltime, user xyzwants 55 days as max walltime for a 
> single job. we dont want to create new queue.
> kindly help
> ------------------------------------------------------------------------
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>   


From peter.st.john at gmail.com  Thu Feb 17 10:42:50 2011
From: peter.st.john at gmail.com (Peter St. John)
Date: Thu, 17 Feb 2011 13:42:50 -0500
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <alpine.LFD.2.02.1102171048220.3085@lilith>
References: <4D5BF6F0.8080103@ias.edu>
	<alpine.LFD.2.02.1102161308260.3531@lilith>
	<m37hczx7bx.fsf@stuor.nsc.liu.se>
	<alpine.LFD.2.02.1102170856070.3085@lilith>
	<m3mxlupwp2.fsf@stuor.nsc.liu.se>
	<alpine.LFD.2.02.1102171048220.3085@lilith>
Message-ID: <AANLkTinuSORV2-urHAD=2FXM-fmppfzKX7gJTWeU7WMh@mail.gmail.com>

"I'm guessing that he [the ubergeek] has figured out how to hotwire init so
that it just forks emacs and skips everything else..." [RGB]

I think essentially they had that: Lisp used to be inefficient (as, aimed at
the way logicians think, not at the way machines think) on general purpose
hardware like the PDP, so in the 70's Lisp workstations were developed (at
MIT, the same time by the same people as emacs, which wasn't ported to unix
until 81, the year I learned C & vi). When in the early 90's I debated with
one of the Unix Haters, Judy Anderson (of Stanford and PARC), I asked, "OK,
so what is better than Unix?" and was surprised: "nothing that isn't
proprietary" (their beloved lisp workstations). So who was it that quipped,
"Emacs would be a great operating system if only it had a decent editor"?
For them, the editor, the language, and the firmware were all each other.

It's all good. You can be close to the machine (C), the logician (Lisp), the
engineer (fortran). You can express yourself with long sentences from a
small alphabet/vocabulary (assembler, symbolic logic) or short sentences
from a large vocabulary (APL, differential topology); use tools that do one
job well (vi) or that are mutable into anything as needed (emacs). It's all
good. We all have preferences, we are all finite, and different needs are
served.

In the Religion Wars there is imbalance between "Berkely, Bell, Unix, vi, C"
versus "Stanford, MIT, emacs, Lisp". Really we need a "GNU FOSS Sourceforge
Red Hat" OS that integrates with emacs and Lisp, so we can all make the
choice for ourselves. "Microsoft SCO Adobe Visual Studio Flash" wont' do it
for us :-)

Peter



On Thu, Feb 17, 2011 at 10:59 AM, Robert G. Brown <rgb at phy.duke.edu> wrote:

> On Thu, 17 Feb 2011, Leif Nixon wrote:
>
> > "Robert G. Brown" <rgb at phy.duke.edu> writes:
> >
> >> How big are the emacs sources these days?  I mean, the installed binary
> >> and common packages alone are 22 MB in F14... and was that 73 MB
> >> installed?  Why, it was, wasn't it.  :-)
> >
> > And it's all full of goodness!
>
> I'll bet it is!
>
> And somewhere in the world, just imagine, there is actually a geek -- in
> fact, I'd like to think of him as >>the<< ubergeek, an actual
> evolutionary advance in ordinary geeks (at least, he would be if he were
> ever able to reproduce), the geek whose blood serum is 2/3 Jolt cola and
> 1/3 Mountain Dew and who last actually got up from his workstation
> console in 1994, the first year one was able to order new computer
> hardware online for home delivery from the Web (using emacs ported to
> the Commodore Amiga, of course, to actually >>access<< the web) -- who
> knows >>every feature<< of emacs, how to use every single thing that it
> can do.  In fact, like many emacs users, he doesn't bother with silly
> things like mail clients, word processors, web browsers, editors,
> windowing interfaces, or command line interfaces OUTSIDE of emacs,
> because he never actually exits emacs.  Secretly, he doesn't really see
> the point of consoles, xterms, shells and most software, because emacs
> can simply replace them all.
>
> I'm guessing that he has figured out how to hotwire init so that it just
> forks emacs and skips everything else.  That is, assuming that he hasn't
> actually hacked emacs directly into the kernel.  Who needs init, anyway?
> One can start anything just as easily from inside emacs as by forking
> off of init, right...?
>
> ;-)
>
>     rgb
>
> >
> > --
> > Leif Nixon - Security officer
> > National Supercomputer Centre - Swedish National Infrastructure for
> Computing
> > Nordic Data Grid Facility - European Grid Infrastructure
> > _______________________________________________
> > Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> > To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> >
>
> Robert G. Brown                        http://www.phy.duke.edu/~rgb/
> Duke University Dept. of Physics, Box 90305
> Durham, N.C. 27708-0305
> Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110217/3c009146/attachment.html>

From peter.st.john at gmail.com  Thu Feb 17 10:48:36 2011
From: peter.st.john at gmail.com (Peter St. John)
Date: Thu, 17 Feb 2011 13:48:36 -0500
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <20110217154855.GB6611@mx1.drzyzgula.org>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
	<20110217154855.GB6611@mx1.drzyzgula.org>
Message-ID: <AANLkTikAEzvbqA5ApU7t=XjvoV0tm3c+PmLGaa0CPsSs@mail.gmail.com>

I booted System V (from "Microport", a competitor of SCO Xenix) on an i286
(IBM AT, which I bought retail at an IBM Product Center, had 1.2 MB floppy
drive!) with 512K RAM. pwd, ls, worked fine, but my third command, "vi temp"
hung. There are limits even for vi. I had to expand the real memory.
Peter

2011/2/17 Bob Drzyzgula <bob at drzyzgula.org>

>
> On 17/02/11 22:44 +0700, "C. Bergstr?m" wrote:
> > Bob Drzyzgula wrote:
> >>
> >> While I tried to learn emacs a couple of times, it just
> >> seemed like it took forever for the thing to load.
> > fwiw some people still *really* care about emacs performance even on
> > modern hw.  We (PathScale) have a few vocal users and plan to spin some
> > cycles to speed it up a bit.  (If you're interested to help test/give
> > feedback ping me off list.)
>
> It is one of my standing, and I'm sure tiresome, jokes
> that when someone asks me how fast a new system seems
> to be, I respond with something along the lines of
> "well, vi just screams". Vi, of course, having "screamed"
> on a MC68010 with 2MB of memory.
>
> --Bob
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110217/7b77e318/attachment.html>

From rgb at phy.duke.edu  Thu Feb 17 11:36:48 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 17 Feb 2011 14:36:48 -0500 (EST)
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471E0@ALTPHYEMBEVSP20.RES.AD.JPL>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<4D5D2E95.3070901@pathscale.com>
	<alpine.LFD.2.02.1102171043460.3085@lilith>
	<AANLkTimAHN95rzc1fMcmDX94+PQ2xOuu3NbdkbMdzQ6=@mail.gmail.com>,
	<alpine.LFD.2.02.1102171140290.3085@lilith>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471E0@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <alpine.LFD.2.02.1102171417240.3155@lilith>

On Thu, 17 Feb 2011, Lux, Jim (337C) wrote:

> And you have a whole "cluster" of brewing vessels that you make small
> changes in to do a suitable Monte Carlo approach?

Actually, I'm converting to a superbrewer -- I had a two node brewery,
and will probably keep my two five gallon 1 cycle per week nodes around
to do cereal brewing, but my new fifteen gallon node will permit me to
brew even more beer in the same time without the cereal penalty.  Of
course, the five gallon nodes were simple to configure -- I just
installed a Gnu fermentation lock to keep the source open.  The
superbrewing node I actually have to assemble out of parts, including
special network pipes that will allow me to dump cruft (trub) mid-cycle,
sample progress through a special port, and bottle the final product
directly out of the node instead of having to pass through a "master"
five gallon bottling node that I have on the side (that really doesn't
do much when the cluster is full and running).

I suppose I could put them all together into a very large heterogeneous
cluster and make a 30 gpw cluster (that would be twelve to thirteen
cases) but my primary mashing vessel (with an absolute maximum 12 gallon
of fermentable liquid limit) then becomes a bottle.. um, neck in the
process, and I have no idea how I'd manage the requisite boil.  In other
words.  Besides, I only HAVE around 20 cases of bottles to fill, and if
I get more the administrative penalty of using our laundry room wall as
a beer storage unit will become, um, "severe".

In other words, before I reach that point I will have to acknowledge
that I've passed a critical scaling threshold and upgrade everything,
and will probably need improved infrastructure in the form of a
seriously air-conditioned and darkened corner of the garage devoted to
just the brew-cluster, or the whole project might get kicked
off-premises by my wife.

Its these sorts of things that separate out the casual small brewcluster
builder from the professional -- its easy to throw together a one, two,
or even three node operation, but at some point you need special
cooling, a floor that can support a metric ton or so of liquid (8 pounds
per gallon plus container weights adds up fast), special heat sources
capable of boiling 30 or so gallons at a time, and a mill and 55 pound
stacks of barley to keep the whole process flowing.  And ultimately, I
can only drink 1-3 beers a day, sustained, before other problems surface
affecting my belly and my liver.  My more modest cluster is more than
sufficient for keeping up with that, with a surplus production large
enough to cover sporadic out-of-band demand on the resource such as
bringing my beer to parties, letting my older sons drink some two,
providing some for my wife, serving guests.  I do look forward to the
finishing the new superbrewing unit as doing 12+ gallons all at once
will actually permit me to brew only once every 2-3 weeks and still
gradually fill all of my bottles so that they get the maximum aging time
before consumption.

    rgb

> ________________________________________
> From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of Robert G. Brown [rgb at phy.duke.edu]
> Sent: Thursday, February 17, 2011 08:43
> To: Leif Nixon
> Cc: Beowulf Mailing List
> Subject: Re: [Beowulf] 9 traits of the veteran Unix admin
>
> On Thu, 17 Feb 2011, Leif Nixon wrote:
>
>> 2011/2/17 Robert G. Brown <rgb at phy.duke.edu>:
>>>
>>> I'm game.  Any excuse to drink beer is a good one, now that I make my
>>> own (needless to say, perfect) beer...
>>
>> Sans alcohol, I presume.
>
> No, merely with >>a fraction<< of the alcohol in, say, Everclear...;-)
>
> Although the batch I'm currently fermenting should be quite strong -- as
> much as 7-8% if it ferments dry.  Not quite brandywine, but higher than
> most commercial beers.  But usually I shoot for the usual 5-6%, with a
> hint of sweetness counterbalancing the bitterness and aroma of the hops
> and the rich biscuit of the barley...;-)
>
>    rgb
>
>>
>> --
>> Leif Nixon                       -            Systems expert
>> ------------------------------------------------------------
>> National Supercomputer Centre    -      Linkoping University
>> ------------------------------------------------------------
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>>
>
> Robert G. Brown                        http://www.phy.duke.edu/~rgb/
> Duke University Dept. of Physics, Box 90305
> Durham, N.C. 27708-0305
> Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu




From rgb at phy.duke.edu  Thu Feb 17 11:52:40 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Thu, 17 Feb 2011 14:52:40 -0500 (EST)
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <AANLkTinmUEbwmubKdH=UR86FwgSky8Tp+8Q6c4JU7xcL@mail.gmail.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
	<20110217154855.GB6611@mx1.drzyzgula.org>
	<alpine.LFD.2.02.1102171117250.3085@lilith>
	<AANLkTinmUEbwmubKdH=UR86FwgSky8Tp+8Q6c4JU7xcL@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1102171438050.3155@lilith>

On Thu, 17 Feb 2011, Bob Drzyzgula wrote:

> On Thu, Feb 17, 2011 at 11:38 AM, Robert G. Brown <rgb at phy.duke.edu> wrote:
>>
>> Even older guys could still use ed on an actual
>> teletype (not a tty interface, mind you, I'm talking the actual printer
>> terminal).
>
> Of course. Been there, done that, on a dial-up terminal with a thermal
> printer at 110 baud over an acoustic coupler. If you know how to use
> sed you probably can suss out ed. I had to know how to use edlin in
> DOS, too. But if you had vi at all, you also had ex, so that was a
> much better option.

Yup.  I never really learned ex, but I used it a few times.  And of
course vi on the inside has ex-like commands.

>> ?I still more or less
>> constantly pop up strange printscreen windows and so on inside
>> browser-based editors because I forget and I pop a Ctrl-P in trying to
>> move up a line. ?My fingers move like lightning, and I can navigate
>> instantly without my fingers leaving the home keys...
>
> hjkl, at least, just leave you with text you have to edit out. My
> problem is always that ESC can have some very unwanted consequences.

Sure, although in jove ESC is used only for repeat-key (and is usually
followed by a repeat count and the ESC-5-kkkkkey); the meta-key is
mapped to Alt at this point, so one never actually uses ESC unless you
mean it or really mis-hit ` or 1.

Makes it really easy to create 72-char

#=======================================================================

separators, though...

>> I had the damnedest time when IBM moved the Ctrl-shift and introduced
>> the Alt key.
>
> http://pckeyboards.stores.yahoo.net/linux101.html

I actually kept my old keyboards until they fell apart and then remapped
keys both, but if you are a sysadmin you have to be able to sit down at
a user's keyboard and work smoothly.  Smoothly does not involve turning
on caps lock every time you try to move the cursor or page up or down or
delete to end of line or...

So you have to pretty much go with what is standard, even if it is
painful and annoying.

>> Naturally, I'd be happy to challenge >>anybody<< to a typing
>> speed/prolixity dual... and a lot of my speed advantage is, in fact, due
>> to my use of jove....
>
> I've certainly had dyed-in-the-wool emacs users be shocked at how fast
> I could make complex edits to system files in vi. Again, it's all a
> matter of what you've trained your fingers to do.

And I appreciate that.  Similarly, I can do some things very quickly in
jove, but they might well be different things.  Jove definitely doesn't
directly grok regexps and can only do variations of global replace with
or without asking you per swap.  But sed is only an Alt-X S, Alt-S away,
and I keep a few sed scripts around that predefine some particularly
useful global swap variants.

As you say, it is what your fingers know how to do, and how you use the
WHOLE unix/linux CLI toolbox to do work.  vi is arguably closer to the
Unix Way of many small tools that in concert do big jobs in an
expert-friendly environment; it is like an oven that will good your
food, but you have to do all the prep and provide the pots and pans.
Emacs is the diametric opposite -- one tool that will make, bake, and
eat your meatloaf for you (and then do the dishes afterward in the
kitchen sink that some kindly soul wrote in lisp and then added).  jove
splits the difference -- it can make and bake well enough and has
various clever features like a built in rotissary and a convection
blower, but still needs some help from tools and pots and pans on the
side and it very definitely won't eat your food or do the dishes.  No
kitchen sink in jove, no way to put one there.

Now, after mutilating THAT metaphor, back to work...;-)

    rgb

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu


From bob at drzyzgula.org  Thu Feb 17 12:06:03 2011
From: bob at drzyzgula.org (Bob Drzyzgula)
Date: Thu, 17 Feb 2011 15:06:03 -0500
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <alpine.LFD.2.02.1102171438050.3155@lilith>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<20110217153314.GA6611@mx1.drzyzgula.org>
	<4D5D424E.3060806@pathscale.com>
	<20110217154855.GB6611@mx1.drzyzgula.org>
	<alpine.LFD.2.02.1102171117250.3085@lilith>
	<AANLkTinmUEbwmubKdH=UR86FwgSky8Tp+8Q6c4JU7xcL@mail.gmail.com>
	<alpine.LFD.2.02.1102171438050.3155@lilith>
Message-ID: <20110217200603.GE6611@mx1.drzyzgula.org>

On 17/02/11 14:52 -0500, Robert G. Brown wrote:
> On Thu, 17 Feb 2011, Bob Drzyzgula wrote:
>
>> Of course. Been there, done that, on a dial-up terminal with a thermal
>> printer at 110 baud over an acoustic coupler. If you know how to use
>> sed you probably can suss out ed. I had to know how to use edlin in
>> DOS, too. But if you had vi at all, you also had ex, so that was a
>> much better option.
>
> Yup.  I never really learned ex, but I used it a few times.  And of
> course vi on the inside has ex-like commands.

vi, at least originally, was just a full-screen front end on
ex, so vi is more than ex-like, it *is* ex. If you install
vim today, ex is just a symbolic link to vim, and when vim
detects that it has been started up as ex it just goes into
ex mode; if you type the "visual" command, it reverts to
full screen vi mode.

> I actually kept my old keyboards until they fell apart and then remapped
> keys both, but if you are a sysadmin you have to be able to sit down at
> a user's keyboard and work smoothly.  Smoothly does not involve turning
> on caps lock every time you try to move the cursor or page up or down or
> delete to end of line or...
>
> So you have to pretty much go with what is standard, even if it is
> painful and annoying.

In fact; I never went to the trouble of getting one of
those. On any given day I use might use a half-dozen
different keyboards, from a little netbook to a giant
Microsoft "Natural" keyboard. After you do that for enough
years it is all the same.

--Bob


From ellis at runnersroll.com  Thu Feb 17 14:27:17 2011
From: ellis at runnersroll.com (Ellis H. Wilson III)
Date: Thu, 17 Feb 2011 17:27:17 -0500
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <alpine.LFD.2.02.1102171438050.3155@lilith>
References: <4D5BF6F0.8080103@ias.edu>	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>	<20110217153314.GA6611@mx1.drzyzgula.org>	<4D5D424E.3060806@pathscale.com>	<20110217154855.GB6611@mx1.drzyzgula.org>	<alpine.LFD.2.02.1102171117250.3085@lilith>	<AANLkTinmUEbwmubKdH=UR86FwgSky8Tp+8Q6c4JU7xcL@mail.gmail.com>
	<alpine.LFD.2.02.1102171438050.3155@lilith>
Message-ID: <4D5DA0C5.7060504@runnersroll.com>

On 02/17/11 14:52, Robert G. Brown wrote:
> Now, after mutilating THAT metaphor, back to work...;-)

I've been waiting on the absurd metaphors to come out ;).  I think my 
favorite of all time (granted it's not really a metaphor, but 
nonetheless) is something about sacrificing a goat over the computer to 
get vi to work.  I'll have to go through the archives at some point to 
find that little gem.

I have to say however I am certainly in support of vi(m), and it's not 
for historical reasons - I think I might have just barely nicked the 90s 
for seeing a command line for the first time as we didn't have a 
computer in my house growing up until 95.  When I started running a Real 
Operating System (tm) in 2003/2004 I bounced around a bit but ultimately 
settled with vim.  I did destroy a series of important documents on my 
way to understanding it and subsequently realizing there was a tutorial 
built into the program.  Speaking of undo, that was one "feature" I only 
stumbled on some 2 or 3 years after beginning with vim, :).

Anyhow, what this thread originally should have linked to is:
"9 traits of the annual Beowulf List editor wars"
Now that would be an interesting read!

ellis


From scrusan at ur.rochester.edu  Thu Feb 17 14:40:24 2011
From: scrusan at ur.rochester.edu (Steve Crusan)
Date: Thu, 17 Feb 2011 17:40:24 -0500
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <4D5DA0C5.7060504@runnersroll.com>
Message-ID: <C9830E08.EDD0%scrusan@ur.rochester.edu>

On 2/17/11 5:27 PM, "Ellis H. Wilson III" <ellis at runnersroll.com> wrote:

> On 02/17/11 14:52, Robert G. Brown wrote:
>> Now, after mutilating THAT metaphor, back to work...;-)
> 
> I've been waiting on the absurd metaphors to come out ;).  I think my
> favorite of all time (granted it's not really a metaphor, but
> nonetheless) is something about sacrificing a goat over the computer to
> get vi to work.  I'll have to go through the archives at some point to
> find that little gem.
> 
> I have to say however I am certainly in support of vi(m), and it's not
> for historical reasons - I think I might have just barely nicked the 90s
> for seeing a command line for the first time as we didn't have a
> computer in my house growing up until 95.  When I started running a Real
> Operating System (tm) in 2003/2004 I bounced around a bit but ultimately
> settled with vim.  I did destroy a series of important documents on my
> way to understanding it and subsequently realizing there was a tutorial
> built into the program.  Speaking of undo, that was one "feature" I only
> stumbled on some 2 or 3 years after beginning with vim, :).
> 
> Anyhow, what this thread originally should have linked to is:
> "9 traits of the annual Beowulf List editor wars"
> Now that would be an interesting read!

That is a drinking game in itself.

A) 1 shots: if you've destroyed something important because of a wrong vi(m)
keystroke

B) 2 shots: if you've been using vi(m) since the 80's, and it took 10+ years
to discover a standard feature

C) 3 shot: there's a vim tutorial?

   
> 
> ellis
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf



----------------------
Steve Crusan
System Administrator
Center for Research Computing
University of Rochester
https://www.crc.rochester.edu/



From ellis at runnersroll.com  Thu Feb 17 14:40:51 2011
From: ellis at runnersroll.com (Ellis H. Wilson III)
Date: Thu, 17 Feb 2011 17:40:51 -0500
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <C9830E08.EDD0%scrusan@ur.rochester.edu>
References: <C9830E08.EDD0%scrusan@ur.rochester.edu>
Message-ID: <4D5DA3F3.5010800@runnersroll.com>

On 02/17/11 17:40, Steve Crusan wrote:
> That is a drinking game in itself.
>
> A) 1 shots: if you've destroyed something important because of a wrong vi(m)
> keystroke

This rule has the makings of a nice drinking game (exponential increase 
of mistakes and resulting shots as shots are taken).

> B) 2 shots: if you've been using vi(m) since the 80's, and it took 10+ years
> to discover a standard feature

I was born in the 80s so this rule really wouldn't work too well for me.

> C) 3 shot: there's a vim tutorial?

Fair enough :).  It is stated clearly on the main screen when you open 
vim without a parameter so I have no defense here.  That was certainly 
one of those things you notice only after you know about it.

The real question is, which editor will we use to write the rules up in?

ellis


From j.wender at science-computing.de  Fri Feb 18 01:28:56 2011
From: j.wender at science-computing.de (Jan Wender)
Date: Fri, 18 Feb 2011 10:28:56 +0100
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <4D5DA0C5.7060504@runnersroll.com>
References: <4D5BF6F0.8080103@ias.edu>	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>	<20110217153314.GA6611@mx1.drzyzgula.org>	<4D5D424E.3060806@pathscale.com>	<20110217154855.GB6611@mx1.drzyzgula.org>	<alpine.LFD.2.02.1102171117250.3085@lilith>	<AANLkTinmUEbwmubKdH=UR86FwgSky8Tp+8Q6c4JU7xcL@mail.gmail.com>	<alpine.LFD.2.02.1102171438050.3155@lilith>
	<4D5DA0C5.7060504@runnersroll.com>
Message-ID: <4D5E3BD8.4030909@science-computing.de>

Ellis H. Wilson III schrieb:
> I've been waiting on the absurd metaphors to come out ;).  I think
> my favorite of all time (granted it's not really a metaphor, but
> nonetheless) is something about sacrificing a goat over the computer
> to get vi to work.  I'll have to go through the archives at some
> point to find that little gem.

 From my (ancient) signature quotes file:

SCSI is not magic. There are fundamental technical reasons why it is
necessary to sacrifice a young goat to your SCSI every chain now and 
then. (John Woods)

And:
A witty saying proves nothing.                               (Voltaire)

SCNR, Jan
-- 
---- Company Information ----
Vorstand/Board of Management: Dr. Bernd Finkbeiner, Dr. Roland Niemeier,
Dr. Arno Steitz, Dr. Ingrid Zech
Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: Michel
Lepert
Sitz/Registered Office: Tuebingen Registergericht/Registration Court:
Stuttgart
Registernummer/Commercial Register No.: HRB 382196

-------------- next part --------------
A non-text attachment was scrubbed...
Name: j_wender.vcf
Type: text/x-vcard
Size: 372 bytes
Desc: not available
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110218/78486122/attachment.vcf>

From eagles051387 at gmail.com  Fri Feb 18 02:21:46 2011
From: eagles051387 at gmail.com (Jonathan Aquilina)
Date: Fri, 18 Feb 2011 11:21:46 +0100
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>	<4D5B8A4D.80101@gmail.com>,
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <4D5E483A.3070106@gmail.com>

On 2/16/11 3:22 PM, Lux, Jim (337C) wrote:
> particularly in terms of the ability to manipulate the surroundings), and, as someone pointed out the energy consumption is quite different (as is the underlying computational rate... lots of fairly slow neurons with lots of parallelism vs relatively few really fast transistors)
>
> Watson is a particularly impressive natural language understanding system, with a fairly constrained user interface.  But for other aspects of "life" or "human-ness" other researchers are doing pieces of the puzzle (perhaps not to the level of Watson or Deep Blue).  There are self organizing and self replicating robots (granted, with the "intelligence" of less than a flatworm).
>
> A big advance will be when the machine not only can build its own tools, but can independently figure out what tools it needs to build.  The former is more a matter of physical manipulation, the latter requires creativity and thought.  That's a pretty high bar; there are humans that can't figure out what tool is needed to solve a problem, independently, without having seen someone else do the task.

To throw something interesting into the loop here.

What about the scaled down versions of watson in the cars that take part 
in the darpa challenge. that basically drive themselves? aren't those a 
lot like humans since they are taking in information from many sensors. 
If AI is used i think we possibly could have a human brain in respect to 
driving a car.


From eugen at leitl.org  Fri Feb 18 04:06:48 2011
From: eugen at leitl.org (Eugen Leitl)
Date: Fri, 18 Feb 2011 13:06:48 +0100
Subject: [Beowulf] IBM's Watson on Jeopardy tonight
In-Reply-To: <4D5E483A.3070106@gmail.com>
References: <E1PpSsG-0006IP-Er@mendel.bio.caltech.edu>
	<4D5B8733.1010505@gmail.com>
	<50888AA2846A4ECF9332AFB80EFB87D5@USERPC>
	<0627DE5B-7CAD-4A72-8E8F-8D60167FC43B@jax.org>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471A6@ALTPHYEMBEVSP20.RES.AD.JPL>
	<4D5E483A.3070106@gmail.com>
Message-ID: <20110218120648.GB23560@leitl.org>

On Fri, Feb 18, 2011 at 11:21:46AM +0100, Jonathan Aquilina wrote:

> What about the scaled down versions of watson in the cars that take part 
> in the darpa challenge. that basically drive themselves? aren't those a 

Realtime navigation with machine vision is a considerably harder
problem than DeepQA.

> lot like humans since they are taking in information from many sensors. 
> If AI is used i think we possibly could have a human brain in respect to 
> driving a car.

Operating in the text domain is entirely useless in the context
of driving a car. It's one of the classic reasons why we're
as far removed from human-machine equivalence: isolated capability
peaks do not a landscape make.

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE


From atp at piskorski.com  Fri Feb 18 05:36:23 2011
From: atp at piskorski.com (Andrew Piskorski)
Date: Fri, 18 Feb 2011 08:36:23 -0500
Subject: [Beowulf] [OT] EMACS vs VI
In-Reply-To: <4D5D424E.3060806@pathscale.com>
References: <4D5D424E.3060806@pathscale.com>
Message-ID: <20110218133623.GA93466@piskorski.com>

On Thu, Feb 17, 2011 at 10:44:14PM +0700, C. Bergstr?m wrote:

> fwiw some people still *really* care about emacs performance even on 
> modern hw.  We (PathScale) have a few vocal users and plan to spin some 
> cycles to speed it up a bit.

[Well as long as we're almost totally off topic...]

As a constant (X and Gnu) Emacs user for the last 11 years, I can say
that only rarely has its serial speed ever been a problem, and then
usually solved by Don't Do That.  E.g., turn off font-lock-mode in any
shell buffers that might receive many thousands of lines of output;
don't try to read massive mail folders in VM, use mutt under screen
instead.

Emacs's glaring weakness is its pervasive single-threadedness.  Its
whole UI will just lock up completely in many cases, with the cpu
pegged at 100%, when it can't keep up with output from some child
process.  E.g., R spitting out some sort of absurdly large backtrace,
which ess-mode tries to parse or markup in some fashion.  In those
cases, the only solution is generally to switch to a different shell
and kill the child process; then Emacs will recover.

Emacs needs more concurrency.  Although ironically, continuing to run
on just 1 cpu core would be adequate; it's its lack of UI
responsiveness under stress that's just stupid.  Firefox seems to have
been gradually getting better at that, although version 3.6.13 is
still far from perfect.  AFAICT Emacs hasn't made any similar
progress, but I admit I haven't looked.  It's good enough in other
ways that I merely put up with its few grating flaws.

Although now that I think of it, since the slow output-parsing stuff
in Emacs is probably implemented by running one or more regular
expression matches on every single line, it might be helpful to
retrofit an efficient linear time regexp engine like Russell Cox's
RE2:

   http://swtch.com/~rsc/regexp/regexp3.html
   "Regular Expression Matching in the Wild", by Russ Cox, March 2010
   http://code.google.com/p/re2/
   http://swtch.com/~rsc/regexp/

   http://sljit.sourceforge.net/regex_perf.html
   Performance comparison of regular expression engines

-- 
Andrew Piskorski <atp at piskorski.com>
http://www.piskorski.com/


From bob at drzyzgula.org  Fri Feb 18 05:54:28 2011
From: bob at drzyzgula.org (Bob Drzyzgula)
Date: Fri, 18 Feb 2011 08:54:28 -0500
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <20110218133623.GA93466@piskorski.com>
References: <4D5D424E.3060806@pathscale.com>
	<20110218133623.GA93466@piskorski.com>
Message-ID: <20110218135428.GB27380@mx1.drzyzgula.org>

On 18/02/11 08:36 -0500, Andrew Piskorski wrote:
> 
> [Well as long as we're almost totally off topic...]
> 
> Emacs's glaring weakness is its pervasive single-threadedness.  Its
> whole UI will just lock up completely in many cases, with the cpu
> pegged at 100%, when it can't keep up with output from some child
> process.  E.g., R spitting out...

I suspect that it is quite rare for vi users to have
problems related to a lack of proper multithreading
support. 

--Bob


From rgb at phy.duke.edu  Fri Feb 18 06:55:10 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Fri, 18 Feb 2011 09:55:10 -0500 (EST)
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <4D5DA3F3.5010800@runnersroll.com>
References: <C9830E08.EDD0%scrusan@ur.rochester.edu>
	<4D5DA3F3.5010800@runnersroll.com>
Message-ID: <alpine.LFD.2.02.1102180953510.3257@lilith>

On Thu, 17 Feb 2011, Ellis H. Wilson III wrote:

> The real question is, which editor will we use to write the rules up in?

edlin, of course, in DOS running under Wine.

     rgb

(But this is my last post on the subject, as I suspect we are torturing
at least some people on the list...;-)

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu




From Michael.Frese at NumerEx-LLC.com  Fri Feb 18 07:20:26 2011
From: Michael.Frese at NumerEx-LLC.com (Michael H. Frese)
Date: Fri, 18 Feb 2011 08:20:26 -0700
Subject: [Beowulf] EMACS vs VI
In-Reply-To: <alpine.LFD.2.02.1102180953510.3257@lilith>
References: <C9830E08.EDD0%scrusan@ur.rochester.edu>
	<4D5DA3F3.5010800@runnersroll.com>
	<alpine.LFD.2.02.1102180953510.3257@lilith>
Message-ID: <6.2.5.6.2.20110218081944.05f5cfb8@NumerEx-LLC.com>

Yes, Please.

Let the Editor Wars END!

Please!


Mike


At 07:55 AM 2/18/2011, Robert G. Brown wrote:
>On Thu, 17 Feb 2011, Ellis H. Wilson III wrote:
>
> > The real question is, which editor will we use to write the rules up in?
>
>edlin, of course, in DOS running under Wine.
>
>      rgb
>
>(But this is my last post on the subject, as I suspect we are torturing
>at least some people on the list...;-)
>
>Robert G. Brown                        http://www.phy.duke.edu/~rgb/
>Duke University Dept. of Physics, Box 90305
>Durham, N.C. 27708-0305
>Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu
>
>
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>To change your subscription (digest mode or unsubscribe) visit 
>http://www.beowulf.org/mailman/listinfo/beowulf




From Bill.Rankin at sas.com  Fri Feb 18 08:12:24 2011
From: Bill.Rankin at sas.com (Bill Rankin)
Date: Fri, 18 Feb 2011 16:12:24 +0000
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <alpine.LFD.2.02.1102171417240.3155@lilith>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<4D5D2E95.3070901@pathscale.com>	<alpine.LFD.2.02.1102171043460.3085@lilith>
	<AANLkTimAHN95rzc1fMcmDX94+PQ2xOuu3NbdkbMdzQ6=@mail.gmail.com>,
	<alpine.LFD.2.02.1102171140290.3085@lilith>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471E0@ALTPHYEMBEVSP20.RES.AD.JPL>
	<alpine.LFD.2.02.1102171417240.3155@lilith>
Message-ID: <76097BB0C025054786EFAB631C4A2E3C095042B6@MERCMBX03R.na.SAS.com>

> Besides, I only HAVE around 20 cases of bottles to fill, and if
> I get more the administrative penalty of using our laundry room wall as
> a beer storage unit will become, um, "severe".
>
> In other words, before I reach that point I will have to acknowledge
> that I've passed a critical scaling threshold and upgrade everything,
> and will probably need improved infrastructure in the form of a
> seriously air-conditioned and darkened corner of the garage devoted to
> just the brew-cluster, or the whole project might get kicked
> off-premises by my wife.


One of the great consistencies in our field, and indeed life, is that adequate machine room space is always at a premium and unfortunately tends to take second place to other concerns during the initial planning phases.

:-)

-b




From rgb at phy.duke.edu  Fri Feb 18 08:54:51 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Fri, 18 Feb 2011 11:54:51 -0500 (EST)
Subject: [Beowulf] 9 traits of the veteran Unix admin
In-Reply-To: <76097BB0C025054786EFAB631C4A2E3C095042B6@MERCMBX03R.na.SAS.com>
References: <4D5BF6F0.8080103@ias.edu>
	<6.2.5.6.2.20110217071324.0572e360@NumerEx-LLC.com>
	<4D5D2E95.3070901@pathscale.com>
	<alpine.LFD.2.02.1102171043460.3085@lilith>
	<AANLkTimAHN95rzc1fMcmDX94+PQ2xOuu3NbdkbMdzQ6=@mail.gmail.com>,
	<alpine.LFD.2.02.1102171140290.3085@lilith>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A471E0@ALTPHYEMBEVSP20.RES.AD.JPL>
	<alpine.LFD.2.02.1102171417240.3155@lilith>
	<76097BB0C025054786EFAB631C4A2E3C095042B6@MERCMBX03R.na.SAS.com>
Message-ID: <alpine.LFD.2.02.1102181152560.3257@lilith>

On Fri, 18 Feb 2011, Bill Rankin wrote:

>> Besides, I only HAVE around 20 cases of bottles to fill, and if
>> I get more the administrative penalty of using our laundry room wall as
>> a beer storage unit will become, um, "severe".
>>
>> In other words, before I reach that point I will have to acknowledge
>> that I've passed a critical scaling threshold and upgrade everything,
>> and will probably need improved infrastructure in the form of a
>> seriously air-conditioned and darkened corner of the garage devoted to
>> just the brew-cluster, or the whole project might get kicked
>> off-premises by my wife.
>
>
> One of the great consistencies in our field, and indeed life, is that
> adequate machine room space is always at a premium and unfortunately
> tends to take second place to other concerns during the initial planning
> phases.

Amen, brother.  Seriously, not just in beer-metaphorsville.  If we build
it, then >>surely<< somebody will give us somewhere to put it, don't you
think?  Someplace handy with 30 KW of power and ten tons of AC and a
structurally reinforced floor?

    rgb

> :-)
>
> -b
>
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu




From thakur at mcs.anl.gov  Fri Feb 18 13:18:56 2011
From: thakur at mcs.anl.gov (Rajeev Thakur)
Date: Fri, 18 Feb 2011 15:18:56 -0600
Subject: [Beowulf] SC11 Call for Papers
Message-ID: <109A4EF2-2F78-4EC6-8939-DB375C272BD9@mcs.anl.gov>

SC11: Call for Papers 

SC11, the premier annual international conference on high-performance computing, networking, and storage, will be held in Seattle, Washington, November 12-18, 2011. The Technical Papers Program at SC is the lead component for presenting the most timely and highest quality work in all areas in this field. The conference committee solicits 10 page, two-column submissions of excellent scientific quality. The committee will rigorously review all submissions using originality, technical soundness, timeliness, and impact as the predominant acceptance criteria. SC11 anticipates an acceptance rate of 20-25% and will value papers that focus on sustained performance and/or data intensive science. Awards will be presented for Best Paper and Best Student Paper. Extended versions of papers selected for the Best Paper and Best Student Paper Awards may be published in the journal Scientific Programming.

Abstracts Due: Friday, April 1, 2011
Papers Due: Friday, April 8, 2011
Notification: Friday, July 1, 2011

Further information: http://sc11.supercomputing.org/?pg=papers.html
Questions: papers at info.supercomputing.org

SC11 Technical Papers Chairs
Franck Cappello and Rajeev Thakur




From akshar.bhosale at gmail.com  Wed Feb 23 11:07:11 2011
From: akshar.bhosale at gmail.com (akshar bhosale)
Date: Thu, 24 Feb 2011 00:37:11 +0530
Subject: [Beowulf] Fwd: maui: how to set different walltime for different
	users
In-Reply-To: <AANLkTinXX_tgtamDnykstGXHowPY5dFRhVHrnmUUB1wb@mail.gmail.com>
References: <AANLkTinXX_tgtamDnykstGXHowPY5dFRhVHrnmUUB1wb@mail.gmail.com>
Message-ID: <AANLkTikboJk7rhm249_4ax9xT0QwFm+PAw4SfhH=BUP7@mail.gmail.com>

---------- Forwarded message ----------
From: akshar bhosale <akshar.bhosale at gmail.com>
Date: Fri, Feb 18, 2011 at 12:01 AM
Subject: maui: how to set different walltime for different users
To: Beowulf Mailing List <beowulf at beowulf.org>


hi,

we have a cluster on 16 nodes where we run torque+maui.

We have set max walltime of 4 days for all jobs.  we want to set different
max walltimes for different users. e.g. user abc wants 5 days as max
walltime, user xyzwants 55 days as max walltime for a single job. we dont
want to create new queue.
kindly help
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110224/f0f23828/attachment.html>

From reuti at staff.uni-marburg.de  Wed Feb 23 11:14:00 2011
From: reuti at staff.uni-marburg.de (Reuti)
Date: Wed, 23 Feb 2011 20:14:00 +0100
Subject: [Beowulf] Fwd: maui: how to set different walltime for
	different users
In-Reply-To: <AANLkTikboJk7rhm249_4ax9xT0QwFm+PAw4SfhH=BUP7@mail.gmail.com>
References: <AANLkTinXX_tgtamDnykstGXHowPY5dFRhVHrnmUUB1wb@mail.gmail.com>
	<AANLkTikboJk7rhm249_4ax9xT0QwFm+PAw4SfhH=BUP7@mail.gmail.com>
Message-ID: <A4F0E0E0-0FF3-40EC-B25B-E949D6646848@staff.uni-marburg.de>

Am 23.02.2011 um 20:07 schrieb akshar bhosale:

> ---------- Forwarded message ----------
> From: akshar bhosale <akshar.bhosale at gmail.com>
> Date: Fri, Feb 18, 2011 at 12:01 AM
> Subject: maui: how to set different walltime for different users
> To: Beowulf Mailing List <beowulf at beowulf.org>
> 
> 
> hi,
> 
> we have a cluster on 16 nodes where we run torque+maui.
> 
> We have set max walltime of 4 days for all jobs.  we want to set different max walltimes for different users. e.g. user abc wants 5 days as max walltime, user xyzwants 55 days as max walltime for a single job. we dont want to create new queue.
> kindly help

I can't say for Maui, but in GridEngine you can define an RQS (resource quota set) for users or groups to limit the wallclock time. I would expect, that this is possible in Maui too.

-- Reuti

From akshar.bhosale at gmail.com  Wed Feb 23 17:57:09 2011
From: akshar.bhosale at gmail.com (akshar bhosale)
Date: Thu, 24 Feb 2011 07:27:09 +0530
Subject: [Beowulf] Fwd: maui: how to set different walltime for
	different users
In-Reply-To: <A4F0E0E0-0FF3-40EC-B25B-E949D6646848@staff.uni-marburg.de>
References: <AANLkTinXX_tgtamDnykstGXHowPY5dFRhVHrnmUUB1wb@mail.gmail.com>
	<AANLkTikboJk7rhm249_4ax9xT0QwFm+PAw4SfhH=BUP7@mail.gmail.com>
	<A4F0E0E0-0FF3-40EC-B25B-E949D6646848@staff.uni-marburg.de>
Message-ID: <AANLkTikSo7+yS2JCtz5vTOjovEvUt13XzOyj_0=Hi0kN@mail.gmail.com>

thanks..i could not find resource quota set in maui...can anybody help me
out.

-akshar

On Thu, Feb 24, 2011 at 12:44 AM, Reuti <reuti at staff.uni-marburg.de> wrote:

> Am 23.02.2011 um 20:07 schrieb akshar bhosale:
>
> > ---------- Forwarded message ----------
> > From: akshar bhosale <akshar.bhosale at gmail.com>
> > Date: Fri, Feb 18, 2011 at 12:01 AM
> > Subject: maui: how to set different walltime for different users
> > To: Beowulf Mailing List <beowulf at beowulf.org>
> >
> >
> > hi,
> >
> > we have a cluster on 16 nodes where we run torque+maui.
> >
> > We have set max walltime of 4 days for all jobs.  we want to set
> different max walltimes for different users. e.g. user abc wants 5 days as
> max walltime, user xyzwants 55 days as max walltime for a single job. we
> dont want to create new queue.
> > kindly help
>
> I can't say for Maui, but in GridEngine you can define an RQS (resource
> quota set) for users or groups to limit the wallclock time. I would expect,
> that this is possible in Maui too.
>
> -- Reuti
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110224/e28a9275/attachment.html>

From samuel at unimelb.edu.au  Thu Feb 24 20:59:56 2011
From: samuel at unimelb.edu.au (Christopher Samuel)
Date: Fri, 25 Feb 2011 15:59:56 +1100
Subject: [Beowulf] New Iranian clusters - GPU based ?
Message-ID: <4D67374C.9050604@unimelb.edu.au>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi folks,

Apparently Iran claims to have 2 new supers, the larger
at 89TF.

http://www.computerworld.com/s/article/9211058/Iran_claims_two_new_supercomputers

Looking at the photos they look like SuperMicro boxes,
some 2I with lots of disks and some that look remarkably
like GPU boxes - compare this picture:

http://www.mehrnews.com/mehr_media/image/2011/02/623838_orig.jpg

and this SuperMicro GPU 1U box:

http://www.supermicro.com/products/system/1U/6016/SYS-6016GT-TF.cfm?GPU=TM1

They look the same to me though of course there's no way to
tell if GPUs are in them, but the spacing between them does
seem to imply that they kick out a fair amount of heat..

Interesting!

cheers,
Chris
- -- 
    Christopher Samuel - Senior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
         http://www.vlsci.unimelb.edu.au/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1nN0wACgkQO2KABBYQAh9MpACbB60nnQfotRWW79+EGuzOyN3V
rRIAoI9ZO0L8HZfXy/zgY+4L3Tt+wkdg
=H+gH
-----END PGP SIGNATURE-----


From hahn at mcmaster.ca  Thu Feb 24 22:08:36 2011
From: hahn at mcmaster.ca (Mark Hahn)
Date: Fri, 25 Feb 2011 01:08:36 -0500 (EST)
Subject: [Beowulf] thunderbolt?
Message-ID: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>

anyone have non-vapid info about thunderbolt?
I can't really figure it out - it appears to be somewhat dumbed down
(not even as network-like as USB).  is the advantage that it's 
supposed to be cheap (compared to 10G ethernet, I suppose)?
or cooler (10GT is down to <10W, isn't it?)

the mention of it transporting PCIE makes me wonder whether it might
actually be interesting for HPC.  that is, if it becomes standard for 
every cheap motherboard to have a few ports, and you can do RDMA over it...

thanks, mark hahn


From geoff at galitz.org  Thu Feb 24 22:25:14 2011
From: geoff at galitz.org (Geoff Galitz)
Date: Fri, 25 Feb 2011 07:25:14 +0100
Subject: [Beowulf] thunderbolt?
In-Reply-To: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
Message-ID: <1A194533E25D4CC79CCDCA3DB962E857@USERPC>

> anyone have non-vapid info about thunderbolt?
> I can't really figure it out - it appears to be somewhat dumbed down
> (not even as network-like as USB).  is the advantage that it's
> supposed to be cheap (compared to 10G ethernet, I suppose)?
> or cooler (10GT is down to <10W, isn't it?)


Still mostly overview, but some info is here:

http://www.intel.com/technology/io/thunderbolt/325136-001US_secured.pdf
http://www.intel.com/technology/io/thunderbolt/index.htm

Looking around at the info on the web, it seems some data is still 
inconsistent.  It appears that support for optics was announced, and then 
delayed according to EETimes.

http://www.eetimes.com/electronics-news/4213501/Ten-things-to-know-about-Thunderbolt

It appears they are keeping some details under wraps for some reason.











 



From ljdursi at scinet.utoronto.ca  Fri Feb 25 05:05:24 2011
From: ljdursi at scinet.utoronto.ca (Jonathan Dursi)
Date: Fri, 25 Feb 2011 08:05:24 -0500
Subject: [Beowulf] thunderbolt?
In-Reply-To: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
Message-ID: <FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>


On 2011-02-25, at 1:08AM, Mark Hahn wrote:

> anyone have non-vapid info about thunderbolt?

Not really - best I've seen so far is ars

http://arstechnica.com/apple/news/2011/02/thunderbolt-smokes-usb-firewire-with-10gbps-throughput.ars

but that's not great either.   It seems like it plugs directly into the PCIe bus (_and_ the 
displayport stuff?) and requires no CPU intervention to transfer data - those two things,
plus the eventual upgrade to optics + 100Gb/s, does seem like it would it would be of 
interest.   Consumer grade stuff suddenly coming with something that  looks a bit like
Inifiniband, (that you could also also easily open a console up over?) does  seem like
it'd be useful.   The question is, I guess, if it can be switched/routed easily.

Anyway, several people around here are planning on upgrading their MacBooks, so
I guess we'll get a chance to play with it soon enough.

   - Jonathan

-- 
Jonathan Dursi <ljdursi at scinet.utoronto.ca>      SciNet, Compute/Calcul Canada








From ellis at runnersroll.com  Fri Feb 25 06:56:44 2011
From: ellis at runnersroll.com (Ellis H. Wilson III)
Date: Fri, 25 Feb 2011 09:56:44 -0500
Subject: [Beowulf] thunderbolt?
In-Reply-To: <FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
	<FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>
Message-ID: <4D67C32C.9090506@runnersroll.com>

On 02/25/11 08:05, Jonathan Dursi wrote:
> Not really - best I've seen so far is ars
> 
> http://arstechnica.com/apple/news/2011/02/thunderbolt-smokes-usb-firewire-with-10gbps-throughput.ars

>From the page:
"That 10Gbps is much faster than most current I/O technologies. With two
devices pushing data at the maximum rate, you could back up a full
Blu-ray movie in 30 seconds, or sync 64GB of music to a portable device
in about a minute."

I love completely meaningless comparisons like this, which assume the
medium your pushing or pulling your data to or from is capable of
respectively reading or writing at that rate.  For instance, where is
this massive Blu-ray movie going to?  The hard drive on the laptop?
Unless one has a magic raid array in your laptop or is willing to burn
out your $1000 ssd drive at a particularly fast rate, writing greater
than 1.2GB per second is something like 10x faster than typical consumer
HDDs can work with.  Additionally, why isn't there any comparisons with
eSata?  This seems to me like another way to draw ignorant people in for
proprietary multimedia devices and keep them from moving off the Mac
platform (assuming PCs don't get this).  What a surprise.

> interest.   Consumer grade stuff suddenly coming with something that  looks a bit like
> Inifiniband, (that you could also also easily open a console up over?) does  seem like
> it'd be useful.   The question is, I guess, if it can be switched/routed easily.

I guess I'm lost on why this would be really useful, especially from a
beowulfery perspective.  It's not like any sane Beowulfer would pay a
premium for Macs just to have this interconnect when they could just get
some old IB hardware.

Best,

ellis


From prentice at ias.edu  Fri Feb 25 07:07:01 2011
From: prentice at ias.edu (Prentice Bisbal)
Date: Fri, 25 Feb 2011 10:07:01 -0500
Subject: [Beowulf] New Iranian clusters - GPU based ?
In-Reply-To: <4D67374C.9050604@unimelb.edu.au>
References: <4D67374C.9050604@unimelb.edu.au>
Message-ID: <4D67C595.8000904@ias.edu>

Christopher Samuel wrote:
> Hi folks,
> 
> Apparently Iran claims to have 2 new supers, the larger
> at 89TF.

Big deal. We have one that can win on Jeopardy! ;)

> 
> http://www.computerworld.com/s/article/9211058/Iran_claims_two_new_supercomputers
> 
> Looking at the photos they look like SuperMicro boxes,
> some 2I with lots of disks and some that look remarkably
> like GPU boxes - compare this picture:
> 
> http://www.mehrnews.com/mehr_media/image/2011/02/623838_orig.jpg
> 
> and this SuperMicro GPU 1U box:
> 
> http://www.supermicro.com/products/system/1U/6016/SYS-6016GT-TF.cfm?GPU=TM1
> 
> They look the same to me though of course there's no way to
> tell if GPUs are in them, but the spacing between them does
> seem to imply that they kick out a fair amount of heat..

...or they spaced them out just so they'd take up more racks and look
more impressive.

> 
> Interesting!
> 
> cheers,
> Chris
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf

-- 
Prentice Bisbal
Linux Software Support Specialist/System Administrator
School of Natural Sciences
Institute for Advanced Study
Princeton, NJ


From ljdursi at scinet.utoronto.ca  Fri Feb 25 07:07:05 2011
From: ljdursi at scinet.utoronto.ca (Jonathan Dursi)
Date: Fri, 25 Feb 2011 10:07:05 -0500
Subject: [Beowulf] thunderbolt?
In-Reply-To: <4D67C32C.9090506@runnersroll.com>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
	<FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>
	<4D67C32C.9090506@runnersroll.com>
Message-ID: <2B5018E4-A554-4932-A74D-A923345DFF2A@scinet.utoronto.ca>

On 2011-02-25, at 9:56AM, Ellis H. Wilson III wrote:
> 
> I guess I'm lost on why this would be really useful, especially from a
> beowulfery perspective.  It's not like any sane Beowulfer would pay a
> premium for Macs just to have this interconnect when they could just get
> some old IB hardware.

It's not an Apple thing, it's an Intel thing; presumably it'll be 
rolled out on PC hardware soon enough too (and I think if this had
first been introduced on (say) Dell boxes we wouldn't be seeing the 
knee-jerk skepticism from some corners that we are.)  It looks like
it comes built into the chipsets, eg, you wouldn't need a separate 
"thunderbolt" board.  

Whether it scales up remains to be seen, of course, but hooking up 
a bunch of machines to something approximating a shared PCIe bus 
with no additional hardware seems like it could be genuinely 
interesting, if it works like that.   If not, well hey, it's a fast adapter
bus, and faster data transfer is good, even if just to peripherals.

   - Jonathan
-- 
Jonathan Dursi <ljdursi at scinet.utoronto.ca>  SciNet, Compute/Calcul Canada








From john.hearns at mclaren.com  Fri Feb 25 07:10:12 2011
From: john.hearns at mclaren.com (Hearns, John)
Date: Fri, 25 Feb 2011 15:10:12 -0000
Subject: [Beowulf] New Iranian clusters - GPU based ?
In-Reply-To: <4D67C595.8000904@ias.edu>
References: <4D67374C.9050604@unimelb.edu.au> <4D67C595.8000904@ias.edu>
Message-ID: <68A57CCFD4005646957BD2D18E60667B1292EB0E@milexchmb1.mil.tagmclarengroup.com>


> >
> > They look the same to me though of course there's no way to
> > tell if GPUs are in them, but the spacing between them does
> > seem to imply that they kick out a fair amount of heat..
> 
> ...or they spaced them out just so they'd take up more racks and look
> more impressive.
> 
There's an Infiniband switch in one of the photos, if I'm not wrong.

The contents of this email are confidential and for the exclusive use of the intended recipient.  If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy.


From john.hearns at mclaren.com  Fri Feb 25 07:17:21 2011
From: john.hearns at mclaren.com (Hearns, John)
Date: Fri, 25 Feb 2011 15:17:21 -0000
Subject: [Beowulf] thunderbolt?
In-Reply-To: <4D67C32C.9090506@runnersroll.com>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca><FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>
	<4D67C32C.9090506@runnersroll.com>
Message-ID: <68A57CCFD4005646957BD2D18E60667B1292EB14@milexchmb1.mil.tagmclarengroup.com>



> 
> I guess I'm lost on why this would be really useful, especially from a
> beowulfery perspective.  It's not like any sane Beowulfer would pay a
> premium for Macs just to have this interconnect when they could just
> get
> some old IB hardware.


Well, Beowulfery should be following off-the-shelf components.
Remember that Infiniband started life as a general interconnect, for
attaching devices outwith the chassis, in a converged network style. It
only found its niche in HPC as a 'by-product' when HPC types hunting for
the most blazingly fast interconnects latched onto it (pun intended).
Remember that Intel dropped Infiniband and there was a bit of a stooshie
in the HPC world at the time.
Thank goodness other manufacturers kept running with it.

As other people have said, even if this bus does not have a central
switch if you could (say) rack four boxes together and connect them in a
mesh,
and run RDMA over it you could achieve an single process space machine
for not much money.




stooshie - a fuss, a fracas




The contents of this email are confidential and for the exclusive use of the intended recipient.  If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy.


From james.p.lux at jpl.nasa.gov  Fri Feb 25 07:17:23 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Fri, 25 Feb 2011 07:17:23 -0800
Subject: [Beowulf] thunderbolt?
In-Reply-To: <4D67C32C.9090506@runnersroll.com>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
	<FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>,
	<4D67C32C.9090506@runnersroll.com>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A4722E@ALTPHYEMBEVSP20.RES.AD.JPL>


________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of Ellis H. Wilson III [ellis at runnersroll.com]
Sent: Friday, February 25, 2011 06:56
To: beowulf at beowulf.org
Subject: Re: [Beowulf] thunderbolt?

On 02/25/11 08:05, Jonathan Dursi wrote:
> Not really - best I've seen so far is ars
>
> http://arstechnica.com/apple/news/2011/02/thunderbolt-smokes-usb-firewire-with-10gbps-throughput.ars

>From the page:
"That 10Gbps is much faster than most current I/O technologies. With two
devices pushing data at the maximum rate, you could back up a full
Blu-ray movie in 30 seconds, or sync 64GB of music to a portable device
in about a minute."

I love completely meaningless comparisons like this, which assume the
medium your pushing or pulling your data to or from is capable of
respectively reading or writing at that rate.  For instance, where is
this massive Blu-ray movie going to?  

>>> not to mention that Digital Rights Management means you're unlikely to be allowed to "back-up" that copyrighted Blu-Ray video.
(at least they're not talking about "transferring a Library of Congress" or stacks of encyclopedias)




> interest.   Consumer grade stuff suddenly coming with something that  looks a bit like
> Inifiniband, (that you could also also easily open a console up over?) does  seem like
> it'd be useful.   The question is, I guess, if it can be switched/routed easily.

I guess I'm lost on why this would be really useful, especially from a
beowulfery perspective.  It's not like any sane Beowulfer would pay a
premium for Macs just to have this interconnect when they could just get
some old IB hardware.


>> optical interconnects are nice.  If high rate optical becomes a "consumer product" that's hewing to the original beowulf concept of leveraging cheap commodity hardware to build a supercomputer.  Yes, I know that cluster computing has fallen from the true path of Beowulf, but as our namesake hero says, that was the mead talking.



From ellis at runnersroll.com  Fri Feb 25 07:53:31 2011
From: ellis at runnersroll.com (Ellis H. Wilson III)
Date: Fri, 25 Feb 2011 10:53:31 -0500
Subject: [Beowulf] thunderbolt?
In-Reply-To: <2B5018E4-A554-4932-A74D-A923345DFF2A@scinet.utoronto.ca>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
	<FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>
	<4D67C32C.9090506@runnersroll.com>
	<2B5018E4-A554-4932-A74D-A923345DFF2A@scinet.utoronto.ca>
Message-ID: <4D67D07B.2030000@runnersroll.com>

On 02/25/11 10:07, Jonathan Dursi wrote:
> On 2011-02-25, at 9:56AM, Ellis H. Wilson III wrote:
>>
>> I guess I'm lost on why this would be really useful, especially from a
>> beowulfery perspective.  It's not like any sane Beowulfer would pay a
>> premium for Macs just to have this interconnect when they could just get
>> some old IB hardware.
> 
> It's not an Apple thing, it's an Intel thing; presumably it'll be 
> rolled out on PC hardware soon enough too (and I think if this had
> first been introduced on (say) Dell boxes we wouldn't be seeing the 
> knee-jerk skepticism from some corners that we are.)  It looks like
> it comes built into the chipsets, eg, you wouldn't need a separate 
> "thunderbolt" board.

I noticed the non-exclusivity after my post and although I admit I'd
prefer to see this rolled out first by a company that doesn't pride
itself on closed and overpriced hardware and software, my skepticism
isn't primarily founded on the technology's origins.  I still think
(certainly for the average consumer) this type of product is largely
useless, since at the end of the day you either need to have all that
data cached in RAM (with this tech and 8 gigs of RAM you're looking at
like 5-10 seconds of transfer, presuming the source can read data to you
at that maximum rate) or on the HDD/SSD.  Until manufacturers start
concentrating on the massive I/O performance gap that has been growing
since the late 70s any further advancements on these fronts is in my
opinion, fruitless.

> Whether it scales up remains to be seen, of course, but hooking up 
> a bunch of machines to something approximating a shared PCIe bus 
> with no additional hardware seems like it could be genuinely 
> interesting, if it works like that.   If not, well hey, it's a fast adapter
> bus, and faster data transfer is good, even if just to peripherals.

Peripherals which probably can't operate at 1.2GB/s (provided you don't
have a SAN in your home).

ellis


From james.p.lux at jpl.nasa.gov  Fri Feb 25 08:20:20 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Fri, 25 Feb 2011 08:20:20 -0800
Subject: [Beowulf] thunderbolt?
In-Reply-To: <4D67D07B.2030000@runnersroll.com>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
	<FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>
	<4D67C32C.9090506@runnersroll.com>
	<2B5018E4-A554-4932-A74D-A923345DFF2A@scinet.utoronto.ca>,
	<4D67D07B.2030000@runnersroll.com>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A47231@ALTPHYEMBEVSP20.RES.AD.JPL>


> interesting, if it works like that.   If not, well hey, it's a fast adapter
> bus, and faster data transfer is good, even if just to peripherals.

Peripherals which probably can't operate at 1.2GB/s (provided you don't
have a SAN in your home).

---
This *is* the beowulf list.. I wouldn't make many assumptions about the computational horsepower at people's abodes...


A peripheral that most people have in their home with that kind of data rate is a DVI (or HDMI) interface to their TV.  The Clock pair runs at 25-165MHz, RGB Data on 3 pairs at 10 times that rate.  24bits/pixel at a rate high enough to display 60 frames sec at 2.5 megapixel/frame.

I think it works out to around 4Gbps...


From cap at nsc.liu.se  Fri Feb 25 11:17:15 2011
From: cap at nsc.liu.se (Peter =?iso-8859-1?q?Kjellstr=F6m?=)
Date: Fri, 25 Feb 2011 20:17:15 +0100
Subject: [Beowulf] New Iranian clusters - GPU based ?
In-Reply-To: <68A57CCFD4005646957BD2D18E60667B1292EB0E@milexchmb1.mil.tagmclarengroup.com>
References: <4D67374C.9050604@unimelb.edu.au> <4D67C595.8000904@ias.edu>
	<68A57CCFD4005646957BD2D18E60667B1292EB0E@milexchmb1.mil.tagmclarengroup.com>
Message-ID: <201102252017.22704.cap@nsc.liu.se>

On Friday, February 25, 2011 04:10:12 pm Hearns, John wrote:
> > > They look the same to me though of course there's no way to
> > > tell if GPUs are in them, but the spacing between them does
> > > seem to imply that they kick out a fair amount of heat..
> > 
> > ...or they spaced them out just so they'd take up more racks and look
> > more impressive.
> 
> There's an Infiniband switch in one of the photos, if I'm not wrong.

Yes, that's an old 288-port (max) Cisco DDR switch. The port count goes quite 
well with the rest (8 racks with 8 chassis of what looks to be 4in2U => 256 
nodes).

The racklayout is also reasonable for an old style open computer room (32 
nodes => ~10KW per rack).

If the 32 1U boxes (also seen) are GPU boxes and the 89TF is total sum then 
(assuming dual socket magnycour) even the FLOPs seem plausible.

Worth noting however is that the cabling seems at best incomplete (some 
chassis not cabled at all, others without IB, ...:

 http://www.mehrnews.com/mehr_media/image/2011/02/623839_orig.jpg

/Friday-bored-out-of-his-mind-Peter
 
-- 
-= Peter Kjellstr?m
-= National Supercomputer Centre
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part.
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110225/74a87eae/attachment.sig>

From stuartb at 4gh.net  Fri Feb 25 15:08:09 2011
From: stuartb at 4gh.net (Stuart Barkley)
Date: Fri, 25 Feb 2011 18:08:09 -0500 (EST)
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
Message-ID: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>

We have a couple of clusters with headless, diskless and stateless
nodes using CentOS 5.  One of our users just ran onto a problem with
/dev/random blocking due to the lack of entropy.

I had the user change the program to use /dev/urandom and this has
handled the immediate problem.

/proc/sys/kernel/random/entropy_avail shows 0 across the compute
nodes even just after boot.

It appears that our Ethernet and Infiniband drivers don't add any
entropy to the random pool.

hw_random/intel-rng doesn't seem to work on our systems.

Some questions:

Do others have this problem?  What do you do?

Do you just refer users to /dev/urandom?

Do you modify network drivers to introduce entropy?

Are there other suggested methods of adding entropy to /dev/random?

Are there ways to introduce entropy from the random number generator
on some Intel systems?  Did Intel remove this from more recent chips?

How reliable is /dev/urandom without initial entropy?  We boot from
stateless disk images and don't carry any entropy over from previous
boots.  /dev/urandom appears to be different across several servers
just after boot, but I have not found any other initialization of the
entropy pool.  I haven't checked that single systems get different
results on different boots.  I'm concerned about users getting poor
random numbers from what should be good sources.

Thanks for any suggestions,
Stuart Barkley
-- 
I've never been lost; I was once bewildered for three days, but never lost!
                                        --  Daniel Boone


From samuel at unimelb.edu.au  Fri Feb 25 16:11:11 2011
From: samuel at unimelb.edu.au (Chris Samuel)
Date: Sat, 26 Feb 2011 11:11:11 +1100
Subject: [Beowulf] New Iranian clusters - GPU based ?
In-Reply-To: <201102252017.22704.cap@nsc.liu.se>
References: <4D67374C.9050604@unimelb.edu.au>
	<68A57CCFD4005646957BD2D18E60667B1292EB0E@milexchmb1.mil.tagmclarengroup.com>
	<201102252017.22704.cap@nsc.liu.se>
Message-ID: <201102261111.11791.samuel@unimelb.edu.au>

On Sat, 26 Feb 2011 06:17:15 AM Peter Kjellstr?m wrote:

> If the 32 1U boxes (also seen) are GPU boxes and the
> 89TF is total sum then  (assuming dual socket magnycour)
> even the FLOPs seem plausible.

I've done a guesstimate on what the system is likely to be
like on my personal blog here:

http://www.csamuel.org/2011/02/25/irans-new-gpu-powered-supercomputers

The 1U nodes are almost certainly the Supermicro 1U GPU
boxes (though of course there's no way of knowing if the
GPUs are still in them, much less whether they are M1060's
or M2070's).

In summary I'm guessing they've quoted Rpeak and not Rmax. ;-)

cheers,
Chris
-- 
Christopher Samuel          Senior Systems Administrator
VLSCI - Victorian Life Sciences Computational Initiative
Email: samuel at unimelb.edu.au Phone: +61 (0)3 903 55545
          http://www.vlsci.unimelb.edu.au/


From gdjacobs at gmail.com  Fri Feb 25 18:55:15 2011
From: gdjacobs at gmail.com (Geoffrey Jacobs)
Date: Fri, 25 Feb 2011 20:55:15 -0600
Subject: [Beowulf] thunderbolt?
In-Reply-To: <ECE7A93BD093E1439C20020FBE87C47FEDE0A47231@ALTPHYEMBEVSP20.RES.AD.JPL>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
	<FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>
	<4D67C32C.9090506@runnersroll.com>
	<2B5018E4-A554-4932-A74D-A923345DFF2A@scinet.utoronto.ca>
	<4D67D07B.2030000@runnersroll.com>
	<ECE7A93BD093E1439C20020FBE87C47FEDE0A47231@ALTPHYEMBEVSP20.RES.AD.JPL>
Message-ID: <AANLkTinv1Yu=QtHqZGjuv+PuZzoxG8==K2BuVEktgO8q@mail.gmail.com>

On Fri, Feb 25, 2011 at 10:20 AM, Lux, Jim (337C)
<james.p.lux at jpl.nasa.gov>wrote:

>
> > interesting, if it works like that.   If not, well hey, it's a fast
> adapter
> > bus, and faster data transfer is good, even if just to peripherals.
>
> Peripherals which probably can't operate at 1.2GB/s (provided you don't
> have a SAN in your home).
>
> ---
> This *is* the beowulf list.. I wouldn't make many assumptions about the
> computational horsepower at people's abodes...
>
>
> A peripheral that most people have in their home with that kind of data
> rate is a DVI (or HDMI) interface to their TV.  The Clock pair runs at
> 25-165MHz, RGB Data on 3 pairs at 10 times that rate.  24bits/pixel at a
> rate high enough to display 60 frames sec at 2.5 megapixel/frame.
>
> I think it works out to around 4Gbps...
> _______________________________________________
>

That's the limit for single link. High end consumer panels can do 2560x1600
which works out to ~5600 mbits/sec (pumped by a dual link DVI signal). HDMI
can supply about 8000 mbits/sec, and DP can do double that.

I wonder what actual performance is going to be like.

-- 
MORE CORE AVAILABLE, BUT NOT FOR YOU
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110225/bd6b988b/attachment.html>

From peter.st.john at gmail.com  Fri Feb 25 19:18:48 2011
From: peter.st.john at gmail.com (Peter St. John)
Date: Fri, 25 Feb 2011 22:18:48 -0500
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
Message-ID: <AANLkTimrKRD53S2GZiqJHtTmKgHUxw77GsQCCCb6vXJn@mail.gmail.com>

Stuart,
This is a bit nuts but (considering the hardware random number generator
issue, the user's application needs, and carrying an entropy pool across
boots, as out of bound) there's something that you might try. I don't know
enough about how this (to me, new-fangled) entropy pool works, but I noted
that the file "random" is world writable, so:

peter at hattie:/dev$ ls -l random
crw-rw-rw- 1 root root 1, 8 2011-02-25 19:31 random
peter at hattie:/dev$ cat /proc/sys/kernel/random/entropy_avail
228
peter at hattie:/dev$ date --rfc-3339=ns | sed 's/.*\.//' | sed 's/-.*//' >>
random
peter at hattie:/dev$ cat /proc/sys/kernel/random/entropy_avail
173
peter at hattie:/dev$ date --rfc-3339=ns | sed 's/.*\.//' | sed 's/-.*//'
483200591

The date gives the time to nanosecond precision; the sed strips out the
leading stuff before the decimal point (hours, minutes) and the trailing
stuff (offset for timezone) leaving the number of nanoseconds since the last
second of time.
I do not know flushing this onto "random" does anything at all; in my
environment, the available pool size jiggles no matter what I do, as it's
probably using the milliseconds (or smaller) between keystrokes, so I can't
cat it without jiggling it. However, it seems harmless to try if nobody has
a better suggestion, to see if it un-nulls your flatlined entropy pool at
boot time (from a script).

Nutty huh?

Peter

On Fri, Feb 25, 2011 at 6:08 PM, Stuart Barkley <stuartb at 4gh.net> wrote:

> We have a couple of clusters with headless, diskless and stateless
> nodes using CentOS 5.  One of our users just ran onto a problem with
> /dev/random blocking due to the lack of entropy.
>
> I had the user change the program to use /dev/urandom and this has
> handled the immediate problem.
>
> /proc/sys/kernel/random/entropy_avail shows 0 across the compute
> nodes even just after boot.
>
> It appears that our Ethernet and Infiniband drivers don't add any
> entropy to the random pool.
>
> hw_random/intel-rng doesn't seem to work on our systems.
>
> Some questions:
>
> Do others have this problem?  What do you do?
>
> Do you just refer users to /dev/urandom?
>
> Do you modify network drivers to introduce entropy?
>
> Are there other suggested methods of adding entropy to /dev/random?
>
> Are there ways to introduce entropy from the random number generator
> on some Intel systems?  Did Intel remove this from more recent chips?
>
> How reliable is /dev/urandom without initial entropy?  We boot from
> stateless disk images and don't carry any entropy over from previous
> boots.  /dev/urandom appears to be different across several servers
> just after boot, but I have not found any other initialization of the
> entropy pool.  I haven't checked that single systems get different
> results on different boots.  I'm concerned about users getting poor
> random numbers from what should be good sources.
>
> Thanks for any suggestions,
> Stuart Barkley
> --
> I've never been lost; I was once bewildered for three days, but never lost!
>                                        --  Daniel Boone
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110225/e8087fe5/attachment.html>

From john.hearns at mclaren.com  Sat Feb 26 05:54:18 2011
From: john.hearns at mclaren.com (Hearns, John)
Date: Sat, 26 Feb 2011 13:54:18 -0000
Subject: [Beowulf] thunderbolt?
In-Reply-To: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
Message-ID: <68A57CCFD4005646957BD2D18E60667B1292EB44@milexchmb1.mil.tagmclarengroup.com>

> anyone have non-vapid info about thunderbolt?
> I can't really figure it out - it appears to be somewhat dumbed down
> (not even as network-like as USB).  is the advantage that it's
> supposed to be cheap (compared to 10G ethernet, I suppose)?
> or cooler (10GT is down to <10W, isn't it?)
Interesting article from The Register:

http://www.theregister.co.uk/2011/02/24/intel_thunderbolt_copper_embrace
_explained/


100 metres possible over optical in the future, and it was intended as a
LAN replacement.
I wonder when we will see these features?

The contents of this email are confidential and for the exclusive use of the intended recipient.  If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy.


From hahn at mcmaster.ca  Sat Feb 26 14:13:14 2011
From: hahn at mcmaster.ca (Mark Hahn)
Date: Sat, 26 Feb 2011 17:13:14 -0500 (EST)
Subject: [Beowulf] thunderbolt?
In-Reply-To: <4D67C32C.9090506@runnersroll.com>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>
	<FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>
	<4D67C32C.9090506@runnersroll.com>
Message-ID: <Pine.LNX.4.64.1102261657240.7392@coffee.psychology.mcmaster.ca>

> I guess I'm lost on why this would be really useful, especially from a
> beowulfery perspective.  It's not like any sane Beowulfer would pay a
> premium for Macs just to have this interconnect when they could just get
> some old IB hardware.

Thunderbolt has nothing to do with Macs, of course.  at least Intel makes 
it sound as if they'll be pushing it into the commodity (PC) market, just 
as they did with PCI*/USB/etc.

there's a lot to not like about IB.  the fact that it's very nearly a 
one-company "ecosystem".  the fact that even the cheapest SDR IB setup
will set you back >$300/port (adapter, cable, switch), and "real" configs
are more like $1k/port.  or the just IB's gratuitous non-ethernet-ness.

what makes it interesting to me is that it appears to be headed towards 
commoditization/ubiquity like ethernet is (and like IB will never be.)
Intel talks about carying PCIE over thunderbolt - does that means that 
there may be a market for switched PCIE fabrics?  would that provide a 
lower-overhead RDMA mechanism?  some of the Intel verbiage mentions 
multiport controllers with builtin switching - does that mean that they're
thinking about, say, a 6-port adapter that can implement a switchless fabric?

it's quite difficult to understand at this point, expecially when the product
appears to have originally been a lab project in silicon photonics.




From hahn at mcmaster.ca  Sat Feb 26 14:32:18 2011
From: hahn at mcmaster.ca (Mark Hahn)
Date: Sat, 26 Feb 2011 17:32:18 -0500 (EST)
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
Message-ID: <Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>

> nodes using CentOS 5.  One of our users just ran onto a problem with
> /dev/random blocking due to the lack of entropy.

/dev/random should be reserved for generating non-ephemeral keys.

> Do others have this problem?  What do you do?

I've only ever heard of it on servers that do a lot of ssl transactions,
and were configured to use /dev/random for transient keys or nonces.

> Do you modify network drivers to introduce entropy?
> Are there other suggested methods of adding entropy to /dev/random?

good questions.  I haven't been following the state of kernel entropy 
gathering - I guess I assumed that they'd worked out a basic set of 
reasonable sources and had a (eg) /proc interface for enabling others
that would be site-specific (such as eth).

> Are there ways to introduce entropy from the random number generator
> on some Intel systems?  Did Intel remove this from more recent chips?

appears so:
http://software.intel.com/en-us/forums/showthread.php?t=66236

> How reliable is /dev/urandom without initial entropy?  We boot from

my understanding is urandom is a crypto hash of the entropy pool:
if the entropy pool never changes or is perfectly guessable, 
urandom is only as good as the hash.  since the crypto hash is not 
broken in general, I'd consider it plenty good.


> stateless disk images and don't carry any entropy over from previous
> boots.

boot scripts normally save and restore entropy, so why can't they do so
to/from some server of yours?  or even just have a boot script that stirs
in some unique per-node data?  (say low-order rdtsc salted with the node's 
MAC.)

this is a good question - not a burning issue I think, but something to 
not forget about.  we're starting to use NFS-root login nodes, for instance,
which could conceivably run into entropy issues.


From peter.st.john at gmail.com  Sat Feb 26 16:56:16 2011
From: peter.st.john at gmail.com (Peter St. John)
Date: Sat, 26 Feb 2011 19:56:16 -0500
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
	<Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
Message-ID: <AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>

After scanning a bunch of man pages (incidentally, "man urandom" explains
the difference between random and urandom; "man random" does not) and
experimenting to produce my reply (above), I found this when google pointed
into wiki (sheesh):

"It is also possible to write to /dev/random. This allows any user to mix
random data into the pool. Non-random data is harmless, because only a
privileged user can issue the ioctl <http://en.wikipedia.org/wiki/Ioctl> needed
to increase the entropy estimate. The current amount of entropy and the size
of the Linux kernel entropy pool are available in /proc/sys/kernel/random/
."
( http://en.wikipedia.org/wiki//dev/random )

So, yes re Mark's:

".. or even just have a boot script that stirs in some unique per-node data?
 (say low-order rdtsc salted with the node's
MAC.) ..."

So (from the wiki) piping dumb data into /dev/random is harmless since the
entropy measure wouldn't be fooled, so then yes, just anyone can pipe some
bytes in anytime. So yeah, the rtdsc, I just meant my 9 digits of
nanoseconds as something easy to try at boot time, and shuffling that with
the MAC is a good idea. (Since a light-nanosecond is about what 10 cm? the
lengths of cables in the server room would be enough to give every node
different boot times, in nanoseconds, right? or no, because your cables are
all standard lengths, but coiled as needed?). So just sticking in a script
that flushes such stuff into /dev/random at boot time (or anytime) should be
practicable and easy.

Peter



On Sat, Feb 26, 2011 at 5:32 PM, Mark Hahn <hahn at mcmaster.ca> wrote:

> > nodes using CentOS 5.  One of our users just ran onto a problem with
> > /dev/random blocking due to the lack of entropy.
>
> /dev/random should be reserved for generating non-ephemeral keys.
>
> > Do others have this problem?  What do you do?
>
> I've only ever heard of it on servers that do a lot of ssl transactions,
> and were configured to use /dev/random for transient keys or nonces.
>
> > Do you modify network drivers to introduce entropy?
> > Are there other suggested methods of adding entropy to /dev/random?
>
> good questions.  I haven't been following the state of kernel entropy
> gathering - I guess I assumed that they'd worked out a basic set of
> reasonable sources and had a (eg) /proc interface for enabling others
> that would be site-specific (such as eth).
>
> > Are there ways to introduce entropy from the random number generator
> > on some Intel systems?  Did Intel remove this from more recent chips?
>
> appears so:
> http://software.intel.com/en-us/forums/showthread.php?t=66236
>
> > How reliable is /dev/urandom without initial entropy?  We boot from
>
> my understanding is urandom is a crypto hash of the entropy pool:
> if the entropy pool never changes or is perfectly guessable,
> urandom is only as good as the hash.  since the crypto hash is not
> broken in general, I'd consider it plenty good.
>
>
> > stateless disk images and don't carry any entropy over from previous
> > boots.
>
> boot scripts normally save and restore entropy, so why can't they do so
> to/from some server of yours?  or even just have a boot script that stirs
> in some unique per-node data?  (say low-order rdtsc salted with the node's
> MAC.)
>
> this is a good question - not a burning issue I think, but something to
> not forget about.  we're starting to use NFS-root login nodes, for
> instance,
> which could conceivably run into entropy issues.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110226/3ab241fe/attachment.html>

From ellis at runnersroll.com  Sun Feb 27 06:16:25 2011
From: ellis at runnersroll.com (Ellis H. Wilson III)
Date: Sun, 27 Feb 2011 09:16:25 -0500
Subject: [Beowulf] thunderbolt?
In-Reply-To: <Pine.LNX.4.64.1102261657240.7392@coffee.psychology.mcmaster.ca>
References: <Pine.LNX.4.64.1102250103320.11878@coffee.psychology.mcmaster.ca>	<FFD043BD-0A03-4A6F-B055-DC78A107E8C0@scinet.utoronto.ca>	<4D67C32C.9090506@runnersroll.com>
	<Pine.LNX.4.64.1102261657240.7392@coffee.psychology.mcmaster.ca>
Message-ID: <4D6A5CB9.9040301@runnersroll.com>

On 02/26/11 17:13, Mark Hahn wrote:
> what makes it interesting to me is that it appears to be headed towards
> commoditization/ubiquity like ethernet is (and like IB will never be.)

Yes, I felt very "there is a world market for maybe five computers" 
after I sent my original email.  Not only does this become interesting 
when another vendor places the technology in card form that can be used 
in nearly any architecture, but it becomes more interesting as you 
mention when the economies of scale work in our favor.

In any event, my cynicism was directed at the consumer community who 
likely won't be able to fully utilize the technology for sometime (and 
probably won't realize they aren't).  I have no doubt Beowulfers will 
find a way to leverage the technology especially if it's cheap (perhaps 
even in their own home, as it's been suggested).

ellis


From rgb at phy.duke.edu  Sun Feb 27 08:48:28 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Sun, 27 Feb 2011 11:48:28 -0500 (EST)
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
	<Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
	<AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1102271131580.4648@lilith>

On Sat, 26 Feb 2011, Peter St. John wrote:

I've been following this conversation halfheartedly because of midterm
grades due last Friday, but I suppose I should chime in.

   /dev/random is, as noted, an entropy generator.  Usually it has enough
entropy after a boot and some network activity to start generating
random numbers.  However, it is always >>enormously slow<< and can only
generate a very few random numbers before it has to wait for more
entropy to be generated.  To run dieharder tests on it I had to do
things like stroke the mouse pad as though it was a kitty's neck for an
hour or so.  You can actually get it to where you can watch it making
new random numbers AS you do bursty things that it uses as part of the
entropy pool.  It is not, I repeat not, for production runs using very
large numbers of rands unless you want to die of old age before an e.g.
Monte Carlo run finishes.

   /dev/urandom is a mix of a software generator and /dev/random.
Entropy based rands are used to "diffuse" entropy into a software
generator, keeping it from being predictable (and giving it an infinite
period, as it were).  However, you then do have to worry about the
quality of the random numbers, and it is still slow.  dieharder doesn't
reveal in over flaws with the stream, but slow is in and of itself a
good reason not to use it for much.

The solution for nearly anyone needing large numbers of fast, high
quality random numbers is going to be:  Use a fast, high quality random
number generator from e.g. the Gnu Scientific Library, and >>seed<< it
from /dev/random, ensuring uniqueness of the seed(s) across the cluster
by any one of several means (storing used seeds in a shared table and
avoiding them in case you have a "birthdays" collision out of the four
billion 32 bit seeds /dev/random can produce).  The table is small, the
lookup occurs outside of the main loop as there is no good reason to
reseed a good rng in the middle of a computation (and for many
generators, there is at least one good reason not to).

So, what are the good generators?  Mersenne Twisters are good
(mt19937_1999, for example). Good and fast.  gfsr4 is pretty good,
pretty fast.  taus/taus2 are both good and fast.  ranlxd2 is good, but
not as fast.  KISS (not yet in the GSL, but I have a GSL-compatible
implementation if anybody needs it in the current dieharder that
eventually I'll send in to Brian for the GSL) is very good and very fast
(fastest).  Any of these, seeded from /dev/random (and yes, dieharder
has code to do that too) will work just fine, if just fine is "the best
you can do given the state of the art".

You too can test generators for yourself (and learn a lot about random
number generators and null hypotheses and testing) with dieharder,
available here:

  http://www.phy.duke.edu/~rgb/General/dieharder.php

I do have a slightly more current version with kiss, "superkiss" (a
mixture of kiss generators) and a straight up supergenerator that lets
you use a mixture of GSL-based generators, each separately seeded, in a
frame that returns numbers out of a shuffle table.  If the generators it
uses are fast, it is still acceptably fast, and if nothing else it hides
and obscures any sort of flaws any single generator might have.

     rgb

> After scanning a bunch of man pages (incidentally, "man urandom" explains
> the difference between random and urandom; "man random" does not) and
> experimenting to produce my reply (above), I found this when google pointed
> into wiki (sheesh):
> "It is also possible to write to?/dev/random. This allows any user to mix
> random data into the pool. Non-random data is harmless, because only a
> privileged user can issue the?ioctl?needed to increase the entropy estimate.
> The current amount of entropy and the size of the Linux kernel entropy pool
> are available in?/proc/sys/kernel/random/."?
> (?http://en.wikipedia.org/wiki//dev/random?)
> 
> So, yes re Mark's:
> 
> ".. or even just have a boot script that stirs?in some unique per-node data?
> ?(say low-order rdtsc salted with the node's
> MAC.) ..."
> 
> So (from the wiki) piping dumb data into /dev/random is harmless since the
> entropy measure wouldn't be fooled, so then yes, just anyone can pipe some
> bytes in anytime. So yeah, the rtdsc, I just meant my 9 digits of
> nanoseconds as something easy to try at boot time, and shuffling that with
> the MAC is a good idea. (Since a light-nanosecond is about what 10 cm? the
> lengths of cables in the server room would be enough to give every node
> different boot times, in nanoseconds, right? or no, because your cables are
> all standard lengths, but coiled as needed?). So just sticking in a script
> that flushes such stuff into /dev/random at boot time (or anytime) should be
> practicable and easy.
> 
> Peter
> 
> 
> 
> On Sat, Feb 26, 2011 at 5:32 PM, Mark Hahn <hahn at mcmaster.ca> wrote:
>       > nodes using CentOS 5. ?One of our users just ran onto a
>       problem with
>       > /dev/random blocking due to the lack of entropy.
> 
> /dev/random should be reserved for generating non-ephemeral keys.
> 
> > Do others have this problem? ?What do you do?
> 
> I've only ever heard of it on servers that do a lot of ssl
> transactions,
> and were configured to use /dev/random for transient keys or nonces.
> 
> > Do you modify network drivers to introduce entropy?
> > Are there other suggested methods of adding entropy to /dev/random?
> 
> good questions. ?I haven't been following the state of kernel entropy
> gathering - I guess I assumed that they'd worked out a basic set of
> reasonable sources and had a (eg) /proc interface for enabling others
> that would be site-specific (such as eth).
> 
> > Are there ways to introduce entropy from the random number generator
> > on some Intel systems? ?Did Intel remove this from more recent
> chips?
> 
> appears so:
> http://software.intel.com/en-us/forums/showthread.php?t=66236
> 
> > How reliable is /dev/urandom without initial entropy? ?We boot from
> 
> my understanding is urandom is a crypto hash of the entropy pool:
> if the entropy pool never changes or is perfectly guessable,
> urandom is only as good as the hash. ?since the crypto hash is not
> broken in general, I'd consider it plenty good.
> 
> 
> > stateless disk images and don't carry any entropy over from previous
> > boots.
> 
> boot scripts normally save and restore entropy, so why can't they do
> so
> to/from some server of yours? ?or even just have a boot script that
> stirs
> in some unique per-node data? ?(say low-order rdtsc salted with the
> node's
> MAC.)
> 
> this is a good question - not a burning issue I think, but something
> to
> not forget about. ?we're starting to use NFS-root login nodes, for
> instance,
> which could conceivably run into entropy issues.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
> Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
> 
> 
> 
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu


From heiko.bauke at snafu.de  Mon Feb 28 01:41:03 2011
From: heiko.bauke at snafu.de (Heiko Bauke)
Date: Mon, 28 Feb 2011 10:41:03 +0100
Subject: [Beowulf] Random numbers (was Re: /dev/random entropy on
 stateless/headless nodes)
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
	<Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
	<AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
	<alpine.LFD.2.02.1102271131580.4648@lilith>
Message-ID: <20110228104103.6e195d2b@keitel105.dhcp.mpi-hd.mpg.de>

Hi,

On Sun, 27 Feb 2011 11:48:28 -0500 (EST)
"Robert G. Brown" <rgb at phy.duke.edu> wrote:

> The solution for nearly anyone needing large numbers of fast, high
> quality random numbers is going to be:  Use a fast, high quality
> random number generator from e.g. the Gnu Scientific Library, and
> >>seed<< it from /dev/random, ensuring uniqueness of the seed(s)
> >>across the cluster

I agree that for Monte Carlo simulations a fast, high quality (pseudo)
random number generator (PRNG) is more appropriate than /dev/{u}random.
However, seeding a PRNG randomly is imho a missconception. Even though
Monte Carlo algorithms utilize a pseudo random resource the final
result of a Monte Carlo simulation should be deterministic and
reproducible. Therefore, for scientific Monte Carlo applications one
should use a known seed. Parallel Monte Carlo applications may derive
streams of pseudo random numbers from a common base sequence by
splitting and leap frogging, see also
http://arxiv.org/abs/cond-mat/0609584 


	Regards,

	Heiko


-- 
-- Number Crunch Blog @ http://numbercrunch.de
--  Cluster Computing @ http://www.clustercomputing.de
--     Random numbers @ http://trng.berlios.de
--        Heiko Bauke @ http://www.mpi-hd.mpg.de/personalhomes/bauke




From peter.st.john at gmail.com  Mon Feb 28 02:15:34 2011
From: peter.st.john at gmail.com (Peter St. John)
Date: Mon, 28 Feb 2011 05:15:34 -0500
Subject: [Beowulf] Random numbers (was Re: /dev/random entropy on
 stateless/headless nodes)
In-Reply-To: <20110228104103.6e195d2b@keitel105.dhcp.mpi-hd.mpg.de>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
	<Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
	<AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
	<alpine.LFD.2.02.1102271131580.4648@lilith>
	<20110228104103.6e195d2b@keitel105.dhcp.mpi-hd.mpg.de>
Message-ID: <AANLkTimSsBBzL+O-BiYPo_9sMEvd=GyNB=8nnYPMeyuX@mail.gmail.com>

The use of the system resource /dev/random is more for cryptography, than
for scientific simulations. Some application like SSH is blocking (in
Stuart's situation) when the entropy pool is empty.
Peter

On Mon, Feb 28, 2011 at 4:41 AM, Heiko Bauke <heiko.bauke at snafu.de> wrote:

> Hi,
>
> On Sun, 27 Feb 2011 11:48:28 -0500 (EST)
> "Robert G. Brown" <rgb at phy.duke.edu> wrote:
>
> > The solution for nearly anyone needing large numbers of fast, high
> > quality random numbers is going to be:  Use a fast, high quality
> > random number generator from e.g. the Gnu Scientific Library, and
> > >>seed<< it from /dev/random, ensuring uniqueness of the seed(s)
> > >>across the cluster
>
> I agree that for Monte Carlo simulations a fast, high quality (pseudo)
> random number generator (PRNG) is more appropriate than /dev/{u}random.
> However, seeding a PRNG randomly is imho a missconception. Even though
> Monte Carlo algorithms utilize a pseudo random resource the final
> result of a Monte Carlo simulation should be deterministic and
> reproducible. Therefore, for scientific Monte Carlo applications one
> should use a known seed. Parallel Monte Carlo applications may derive
> streams of pseudo random numbers from a common base sequence by
> splitting and leap frogging, see also
> http://arxiv.org/abs/cond-mat/0609584
>
>
>        Regards,
>
>        Heiko
>
>
> --
> -- Number Crunch Blog @ http://numbercrunch.de
> --  Cluster Computing @ http://www.clustercomputing.de
> --     Random numbers @ http://trng.berlios.de
> --        Heiko Bauke @ http://www.mpi-hd.mpg.de/personalhomes/bauke
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> http://www.beowulf.org/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110228/aef7da04/attachment.html>

From john.hearns at mclaren.com  Mon Feb 28 02:27:22 2011
From: john.hearns at mclaren.com (Hearns, John)
Date: Mon, 28 Feb 2011 10:27:22 -0000
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net><Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
	<AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
Message-ID: <68A57CCFD4005646957BD2D18E60667B1292EB7C@milexchmb1.mil.tagmclarengroup.com>

So (from the wiki) piping dumb data into /dev/random is harmless since
the entropy measure wouldn't be fooled, so then yes, just anyone can
pipe some bytes in anytime. So yeah, the rtdsc, I just meant my 9 digits
of nanoseconds as something easy to try at boot time, and shuffling that
with the MAC is a good idea. (Since a light-nanosecond is about what 10
cm? the lengths of cables in the server room would be enough to give
every node different boot times, in nanoseconds, right? or no, because
your cables are all standard lengths, but coiled as needed?). 

 

Me Sir! Please! Me Sir!

 

As a cub high energy physicist, in the 'counting room' for an experiment
one day, my supervisor was discussing coaxial delay lines with me.

Light travels at a foot per nanosecond, so that enables you to choose
appropriate lengths of coax delay lines for coincidence counting
experiments.

 

 

Sigh - I guess we have gone all metric since then with 10cm per
nanosecond!

 


The contents of this email are confidential and for the exclusive use of the intended recipient.  If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110228/0d901e9f/attachment.html>

From eugen at leitl.org  Mon Feb 28 03:35:39 2011
From: eugen at leitl.org (Eugen Leitl)
Date: Mon, 28 Feb 2011 12:35:39 +0100
Subject: [Beowulf] SeaMicro drops another Atom bomb on the server market
Message-ID: <20110228113539.GV23560@leitl.org>


(not commodity, but still interesting)

http://venturebeat.com/2011/02/27/seamicro-64bit-atom-servers/

SeaMicro drops another Atom bomb on the server market

February 27, 2011 | Dean Takahashi

Back in June, server startup SeaMicro dropped its Atom bomb on the server
market, launching an extremely energy-efficient server using Intel?s Atom
microprocessors. That enabled SeaMicro to get customers who delivered web
pages to tens of millions of internet users across four continents. Now
SeaMicro is dropping another Atom bomb.

The Santa Clara, Calif.-based company is announcing a second generation of
servers today that use a new 64-bit Atom microprocessor from Intel, improving
the amount of computing power that SeaMicro?s servers can use per watt of
power consumed.

?The response has been extraordinary,? said Andrew Feldman (pictured at top),
chief executive of SeaMicro. ?The sucking sound in the market is
unbelievable. Everybody wants low-power computing.?

As we wrote last year, SeaMicro found a way to turn the server industry on
its head. Computer makers were once obsessed with building servers that had
as many cores, or brains, as possible. But those microprocessors used a lot
of electricity, throwing off a lot of heat in a confined space and consuming
so much power that the electricity bill became bigger than the cost of the
equipment.

So SeaMicro created the SM10000, with a tiny server board that was two inches
by three inches (pictured). Using just one power efficient Atom chip set,
SeaMicro could jam eight servers in a 5-inch by 11-inch circuit board. It put
64 of those circuit boards into 10 slots in a rack chassis. It could
therefore put 512 Atom processors in a space that was a quarter of the size
of a normal server rack. And it used a quarter of the power, weighed a third
of the equipment it replaced, and cost a lot less at $130,000 per machine.
SeaMicro started shipping the SM10000 in August and it has sold a lot of the
systems to customers including Skype, Mozilla, Rogers Communications,
Oakridge National Laboratories, France Telecom and China Netcom Broadband.
Those companies are using the servers in internet data centers where serving
large volumes of web pages to lots of users is the task at hand.

That?s pretty stunning execution for a hardware startup, considering that
most enterprises are shy about using small companies and prefer to work with
established vendors. But those customers got tired of using the computing
equivalent of a space shuttle to go shopping at the grocery store.

SeaMicro also attacked the power consumption in the rest of the system, which
accounts for about two-thirds of the power consumed by a server. The company
did so with a very clever trick known as virtualization.

Today, virtualization is used with servers. It is a layer of software that
rests between an application and the servers that it runs on. If an
application needs only two servers, the virtualization software finds two
available servers to run the application. If the application gets busy and
needs 10 servers, the virtualization software finds 10 available servers to
do the job. The application is no longer tied to specific servers; the
virtualization software frees up the overall system and gets more utilization
out of the available servers, as needed to run applications.

SeaMicro did the same thing, but it applied the concept of virtualization to
the  inside of a server. Feldman designed custom chips that could take the
tasks that were handled by everything beyond the Intel microprocessor and its
chip set. The custom chips virtualize all of those other components so that
it finds the resource when it?s needed. It essentially tricks the
microprocessor into thinking that the rest of the system is there when it
needs it.

SeaMicro virtualized a lot of functions that took up a lot of space inside
each server in a rack. It also did the same with functions such as storage,
networking, server management and load balancing. Full told, SeaMicro
eliminates 90 percent of the components from a system board. SeaMicro calls
this CPU/IO virtualization. With it, SeaMicro shrinks the size of the system
board from a pizza box to the size of a credit card.

By boiling down the rest of the system into a couple of chips, SeaMicro can
get rid of a lot of the components in a system, thereby getting rid of space,
cost, and power consumption.

Feldman acknowledges there were some roadblocks for customers who wanted to
buy the first-generation servers. Since the Atom chips were 32-bit
processors, the biggest potential customers couldn?t buy the SeaMicro
technology because they needed 64-bit processing.

Also, the 32-bit processors could only address as much as 2 gigabytes of main
memory at any given time. That was too small for companies with the largest
databases. Now, with the new 64-bit dual-core Intel Atom n570 coming soon,
SeaMicro can now build 64-bit servers with 4 gigabytes of main memory per
processor.

Feldman says SeaMicro?s new SM10000-64 rack server can now lower its power
consumption per machine as much as 15 percent to 20 percent with the new Atom
chips, which support virtualization technology. SeaMicro kept the same
10-slot rack that it used before, but now it has 256 dual-core chips in it
with a total of 512 cores, rather than the 512 single-core chips. The machine
is more powerful and uses less power.

Customers who spend about $4 million on SeaMicro equipment can wind up saving
about $16 million in lower computing and power costs.

SeaMicro has about 75 employees now and is adding about one a week. It has
raised more than $40 million. It also got a $9.3 million grant from the
Department of Energy. Feldman says the company is profitable and is not
looking for a new round of funding.


From eugen at leitl.org  Mon Feb 28 03:38:47 2011
From: eugen at leitl.org (Eugen Leitl)
Date: Mon, 28 Feb 2011 12:38:47 +0100
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <68A57CCFD4005646957BD2D18E60667B1292EB7C@milexchmb1.mil.tagmclarengroup.com>
References: <AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B1292EB7C@milexchmb1.mil.tagmclarengroup.com>
Message-ID: <20110228113847.GW23560@leitl.org>

On Mon, Feb 28, 2011 at 10:27:22AM -0000, Hearns, John wrote:

> Sigh - I guess we have gone all metric since then with 10cm per
> nanosecond!

It's interesting that at 10 GBit 300 m SR fiber is an optical FIFO
for a 1500 MTU, like a modern version of mercury delay line
storage.

Once they figure out optical gate crossbars and suitable header
layout (e.g. precomputed path, header bits directly selecting
crossbar ports), purely photonic cut-through can't be far behind. 

-- 
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE


From james.p.lux at jpl.nasa.gov  Mon Feb 28 06:24:24 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Mon, 28 Feb 2011 06:24:24 -0800
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <68A57CCFD4005646957BD2D18E60667B1292EB7C@milexchmb1.mil.tagmclarengroup.com>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net><Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
	<AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>,
	<68A57CCFD4005646957BD2D18E60667B1292EB7C@milexchmb1.mil.tagmclarengroup.com>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A47264@ALTPHYEMBEVSP20.RES.AD.JPL>

foot per ns in free space.  60-80% of that in a transmission line, depending on sqrt(LC)  (mostly depends on dielectric.. solid vs foam vs air), except for some exotic delay line coax which has a center conductor wound as a spiral, so the L/unit length is really big.

so more like 20-25 cm/nanosecond.  10cm/ns would be a very slow line
________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of Hearns, John [john.hearns at mclaren.com]
Sent: Monday, February 28, 2011 02:27
To: beowulf at beowulf.org
Subject: Re: [Beowulf] /dev/random entropy on stateless/headless nodes

So (from the wiki) piping dumb data into /dev/random is harmless since the entropy measure wouldn't be fooled, so then yes, just anyone can pipe some bytes in anytime. So yeah, the rtdsc, I just meant my 9 digits of nanoseconds as something easy to try at boot time, and shuffling that with the MAC is a good idea. (Since a light-nanosecond is about what 10 cm? the lengths of cables in the server room would be enough to give every node different boot times, in nanoseconds, right? or no, because your cables are all standard lengths, but coiled as needed?).

Me Sir! Please! Me Sir!

As a cub high energy physicist, in the ?counting room? for an experiment one day, my supervisor was discussing coaxial delay lines with me.
Light travels at a foot per nanosecond, so that enables you to choose appropriate lengths of coax delay lines for coincidence counting experiments.


Sigh ? I guess we have gone all metric since then with 10cm per nanosecond!


The contents of this email are confidential and for the exclusive use of the intended recipient. If you receive this email in error you should not copy it, retransmit it, use it or disclose its contents but should return it to the sender immediately and delete your copy.



From james.p.lux at jpl.nasa.gov  Mon Feb 28 06:29:03 2011
From: james.p.lux at jpl.nasa.gov (Lux, Jim (337C))
Date: Mon, 28 Feb 2011 06:29:03 -0800
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <20110228113847.GW23560@leitl.org>
References: <AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
	<68A57CCFD4005646957BD2D18E60667B1292EB7C@milexchmb1.mil.tagmclarengroup.com>,
	<20110228113847.GW23560@leitl.org>
Message-ID: <ECE7A93BD093E1439C20020FBE87C47FEDE0A47265@ALTPHYEMBEVSP20.RES.AD.JPL>

prop delay is always an issue ...see stories about ATM message size being chosen to be small enough that packet length < propagation delay across France, making echo cancellers easier.  European PTTs wanted 32 bytes, Ma Bell wanted 64 bytes

and, of course, how TCP breaks down if you try to use it, to, say, the Moon
________________________________________
From: beowulf-bounces at beowulf.org [beowulf-bounces at beowulf.org] On Behalf Of Eugen Leitl [eugen at leitl.org]
Sent: Monday, February 28, 2011 03:38
To: beowulf at beowulf.org
Subject: Re: [Beowulf] /dev/random entropy on stateless/headless nodes

On Mon, Feb 28, 2011 at 10:27:22AM -0000, Hearns, John wrote:

> Sigh - I guess we have gone all metric since then with 10cm per
> nanosecond!

It's interesting that at 10 GBit 300 m SR fiber is an optical FIFO
for a 1500 MTU, like a modern version of mercury delay line
storage.

Once they figure out optical gate crossbars and suitable header
layout (e.g. precomputed path, header bits directly selecting
crossbar ports), purely photonic cut-through can't be far behind.

--
Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org
______________________________________________________________
ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org
8B29F6BE: 099D 78BA 2FD3 B014 B08A  7779 75B0 2443 8B29 F6BE
_______________________________________________
Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf


From stuartb at 4gh.net  Mon Feb 28 06:59:57 2011
From: stuartb at 4gh.net (Stuart Barkley)
Date: Mon, 28 Feb 2011 09:59:57 -0500 (EST)
Subject: [Beowulf] /dev/random entropy on stateless/headless nodes
In-Reply-To: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
Message-ID: <alpine.BSF.2.00.1102261807140.29803@freeman.4gh.net>

Thanks for the various suggestions.  I'm fairly well versed on the
theory of /dev/{,u}random but was surprised when our diskless nodes
had no entropy source in practice.

My current concern is letting users get some good randomness for
initializing their own random number generators (gsl, etc).

I'll recommend users use /dev/urandom instead of /dev/random.  This
alone should be sufficient for non-cryptographic purposes and appears
to be the correct system usage.

I'll leave the science issues (<http://arxiv.org/abs/1102.4666> and
<http://arxiv.org/abs/cond-mat/0609584>) to the scientists.  I was
surprised to see that gsl only supports a single integer for seeding
their algorithms.  Perhaps its related.

A secondary concern is nodes where the cryptographic randomness of
/dev/random is required (mostly ssl web servers, but I will need to
inventory).

<http://www.pinkas.net/PAPERS/gpr06.pdf> gives a description of the
Linux kernel internals (as of 2004).  It indicates that the kernel
already does some baseline initialization (probably) sufficient for
/dev/urandom.

Since initial state isn't explicitly stated, throwing node name/mac
addresses/boot time and maybe a few other things into the entropy pool
at boot time should be good enough to prime /dev/urandom to prevent
repetition on reboot.  Something like:

    (date; hostname; uptime; ifconfig; qstat) > /dev/random

in a start up script should at least not harm anything.

I would like to see letting network interrupts be enabled to feed
entropy into /dev/random.  This currently looks like it involves
building custom device drivers (with IRQF_SAMPLE_RANDOM and/or
SA_SAMPLE_RANDOM enabled).

egd/prngd look like reasonable ways to provide a local network service
to provide random data to also feed into things that need it.  I think
they are overkill for now (and I haven't fully reviewed the internals
either).

Thanks for the ideas,
Stuart
-- 
I've never been lost; I was once bewildered for three days, but never lost!
                                        --  Daniel Boone


From mathog at caltech.edu  Mon Feb 28 09:37:19 2011
From: mathog at caltech.edu (David Mathog)
Date: Mon, 28 Feb 2011 09:37:19 -0800
Subject: [Beowulf] Random numbers (was Re: /dev/random entropy on
	stateless/headless nodes)
Message-ID: <E1Pu727-0003a8-Q0@mendel.bio.caltech.edu>

On _some_ systems one decent source of randomness, albeit at a low bit
rate, is lm_sensors. The machines I am referring to alternate randomly
between two different fan speed values or have varying low bits in their
temperature readings (at a constant workload, including idle).

If it is ok to go outside the node for this information, then rather
than mess about with transmission time delays the simplest thing would
be to run a pseudo random number generator on one node and have the
other nodes retrieve blocks of numbers from it.  If the server hands out
the generator output sequentially they will all receive different
blocks, and these can be stuffed into the ioctl to keep /dev/urandom happy.

Probably less work to keep the cluster behind a firewall and dispense
with applications (or application modes) that need /dev/urandom.

Regards,

David Mathog
mathog at caltech.edu
Manager, Sequence Analysis Facility, Biology Division, Caltech


From rgb at phy.duke.edu  Mon Feb 28 10:46:11 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Mon, 28 Feb 2011 13:46:11 -0500 (EST)
Subject: [Beowulf] Random numbers (was Re: /dev/random entropy on
 stateless/headless nodes)
In-Reply-To: <AANLkTimSsBBzL+O-BiYPo_9sMEvd=GyNB=8nnYPMeyuX@mail.gmail.com>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
	<Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
	<AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
	<alpine.LFD.2.02.1102271131580.4648@lilith>
	<20110228104103.6e195d2b@keitel105.dhcp.mpi-hd.mpg.de>
	<AANLkTimSsBBzL+O-BiYPo_9sMEvd=GyNB=8nnYPMeyuX@mail.gmail.com>
Message-ID: <alpine.LFD.2.02.1102281029180.3235@lilith>

On Mon, 28 Feb 2011, Peter St. John wrote:

> The use of the system resource /dev/random is more for cryptography, than
> for scientific simulations. Some application like SSH is blocking (in
> Stuart's situation) when the entropy pool is empty.Peter

Find attached a nearly complete solution.  This is a "one liner"
(almost) xinetd daemon that will read and spit out a four byte uint from
/dev/urandom over a network socket connection to port 8885.  Install the
(bash) script on your head node, which hopefully is on an ethernet and
has a keyboard and has enough boot-time entropy to make /dev/urandom
work.

Installation and testing instructions are included, sort of, but I am
NOT providing a tutorial on security and ports and allowed networks, so
if you don't want everybody on the internet to be able to get random
numbers from your server, you'll have to figure out how to take the
appropriate steps on your own (it's not difficult, lines in the xinetd
config and/or additions to the iptables firewall rules will do it).

The only other thing you'll need (that I'm not providing) is a script or
binary on the client side that will connect to the daemon, send it
CR/LF, read the uint return, and write it to e.g. /dev/random if that's
what it takes to build up enough entropy.  Note well that there isn't
any reason I can think of to do this a uint at a time other than it was
a convenient increment for reseeding, nothing more.  So you can increase
the size of the dd block to a full page or even more if you like, for
substantially greater efficiency.

Note well that this is just one way to do it.  Perl would let you be
more sophisticated (bash is a crude tool and can't cope with binary).  C
is far more elegant, and you can write a C-based daemon that would seed
from /dev/random, use a really good rng, and optionally maintain a
self-avoiding table of rands it has returned if you want to use the
return as a seed.  Even with the current daemon, you can make it read in
the requested number of random bytes and return it.  Tweak away.

    rgb

> 
> On Mon, Feb 28, 2011 at 4:41 AM, Heiko Bauke <heiko.bauke at snafu.de> wrote:
>       Hi,
>
>       On Sun, 27 Feb 2011 11:48:28 -0500 (EST)
>       "Robert G. Brown" <rgb at phy.duke.edu> wrote:
>
>       > The solution for nearly anyone needing large numbers of fast,
>       high
>       > quality random numbers is going to be: ?Use a fast, high
>       quality
>       > random number generator from e.g. the Gnu Scientific Library,
>       and
>       > >>seed<< it from /dev/random, ensuring uniqueness of the
>       seed(s)
>       > >>across the cluster
>
>       I agree that for Monte Carlo simulations a fast, high quality
>       (pseudo)
>       random number generator (PRNG) is more appropriate than
>       /dev/{u}random.
>       However, seeding a PRNG randomly is imho a missconception. Even
>       though
>       Monte Carlo algorithms utilize a pseudo random resource the
>       final
>       result of a Monte Carlo simulation should be deterministic and
>       reproducible. Therefore, for scientific Monte Carlo applications
>       one
>       should use a known seed. Parallel Monte Carlo applications may
>       derive
>       streams of pseudo random numbers from a common base sequence by
>       splitting and leap frogging, see also
>       http://arxiv.org/abs/cond-mat/0609584
> 
>
>       ? ? ? ?Regards,
>
>       ? ? ? ?Heiko
> 
>
>       --
>       -- Number Crunch Blog @ http://numbercrunch.de
>       -- ?Cluster Computing @ http://www.clustercomputing.de
>       -- ? ? Random numbers @ http://trng.berlios.de
>       -- ? ? ? ?Heiko Bauke @
>       http://www.mpi-hd.mpg.de/personalhomes/bauke
> 
>
>       _______________________________________________
>       Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin
>       Computing
>       To change your subscription (digest mode or unsubscribe) visit
>       http://www.beowulf.org/mailman/listinfo/beowulf
> 
> 
> 
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu
-------------- next part --------------
A non-text attachment was scrubbed...
Name: random_daemon.tar.gz
Type: application/octet-stream
Size: 3082 bytes
Desc: Small xinetd random number daemon.
URL: <http://www.beowulf.org/pipermail/beowulf/attachments/20110228/693f3c7e/attachment.obj>

From rgb at phy.duke.edu  Mon Feb 28 12:03:33 2011
From: rgb at phy.duke.edu (Robert G. Brown)
Date: Mon, 28 Feb 2011 15:03:33 -0500 (EST)
Subject: [Beowulf] Random numbers (was Re: /dev/random entropy on
 stateless/headless nodes)
In-Reply-To: <20110228104103.6e195d2b@keitel105.dhcp.mpi-hd.mpg.de>
References: <alpine.BSF.2.00.1102251708280.98544@freeman.4gh.net>
	<Pine.LNX.4.64.1102261715520.7392@coffee.psychology.mcmaster.ca>
	<AANLkTinrDKWde+2VYAYu0XdE9CTU=H-NcO=AippwC+_p@mail.gmail.com>
	<alpine.LFD.2.02.1102271131580.4648@lilith>
	<20110228104103.6e195d2b@keitel105.dhcp.mpi-hd.mpg.de>
Message-ID: <alpine.LFD.2.02.1102281346210.3235@lilith>

On Mon, 28 Feb 2011, Heiko Bauke wrote:

> Hi,
>
> On Sun, 27 Feb 2011 11:48:28 -0500 (EST)
> "Robert G. Brown" <rgb at phy.duke.edu> wrote:
>
>> The solution for nearly anyone needing large numbers of fast, high
>> quality random numbers is going to be:  Use a fast, high quality
>> random number generator from e.g. the Gnu Scientific Library, and
>>>> seed<< it from /dev/random, ensuring uniqueness of the seed(s)
>>>> across the cluster
>
> I agree that for Monte Carlo simulations a fast, high quality (pseudo)
> random number generator (PRNG) is more appropriate than /dev/{u}random.
> However, seeding a PRNG randomly is imho a missconception. Even though
> Monte Carlo algorithms utilize a pseudo random resource the final
> result of a Monte Carlo simulation should be deterministic and
> reproducible. Therefore, for scientific Monte Carlo applications one
> should use a known seed. Parallel Monte Carlo applications may derive
> streams of pseudo random numbers from a common base sequence by
> splitting and leap frogging, see also
> http://arxiv.org/abs/cond-mat/0609584

Interesting paper.

Well, SOMETIMES one cares if they are reproducible, sometimes not.  I
agree in that I always save the seed in the node/run metadata to be ABLE
to check for iid uniqueness and be ABLE to repeat a run, although quite
honestly I can't recall ever actually repeating a run with the same seed
a single time in CPU-centuries of computation time.  Why would one wish
to?  When you're average tens of thousands of runs, you aren't going to
be rerunning the sample average generated from seed = 3023216177 very
often, although perhaps when debugging the simulation code it can be
useful (in which case I always use simple seeds such as seed =
1,2,3...).

As for seeding randomly, I think (even after reading your paper:-) that
how important this is in a parallel computation depends on the
characteristics of the random number generator you are using.  R250, of
course, is shooting a crippled duck in a barrel, but it isn't so clear
that a mersenne twister or (super) KISS generator will have the same
problem -- those are pretty good generators, and pass all of the diehard
tests and most of L'Ecuyer's tests all the way out to where the tests
themselves start to fail for numerical reasons, and (depending on the
implementation) they may well have very long periods (as in MT19937 has
a period roughly 10^6000 and is equi-distributed in 623 dimensions).
Given that MT19937 is also damn fast (though KISS is faster) it isn't an
unreasonable choice.

Generators with very long periods compared to the number of random
numbers consumed in a parallel computation (and that otherwise pass all
of the cranked-up dieharder tests) are still very likely to give one
sufficiently independent results with random seeds "out of the (GSL)
box" so to speak.  However, I completely agree that this is a useful
thing to be able to explicitly test, and the latest version of dieharder
lets one construct a "supergenerator" at the command line that might
consist, e.g. of four MTs started with adjacent or random seeds
providing returns in round robin fashion (or via a shuffle table) and
then use it as direct input to dieharder.  Dieharder will then (one
hopes) be able to detect bitlevel or hyperplanar correlations not just
within a sequence but between sequences generated by various types of
correlated or decorrelated seeds.

It would also be very interesting to try feeding one of your MRG or
YARN-type generators into dieharder and running the full series of tests
on it (there are actually several diehard tests in dieharder that look
for hyperplanar correlations as well as the long tail "monkey" tests).
If I ever have time I'll try to give it a shot.  Is the library and/or a
cli available anyplace easy to find?

    rgb

>
>
> 	Regards,
>
> 	Heiko
>
>
> -- 
> -- Number Crunch Blog @ http://numbercrunch.de
> --  Cluster Computing @ http://www.clustercomputing.de
> --     Random numbers @ http://trng.berlios.de
> --        Heiko Bauke @ http://www.mpi-hd.mpg.de/personalhomes/bauke
>
>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
>

Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu