[Beowulf] 3d rendering cluster

[Vincent Diepeveen]

At 12:34 PM 5/23/2005 +0200, Paul K Egell-Johnsen wrote:
>Hi,
>I'm tasked to find prices for a solution involving a rack mounted 3d
>rendering farm, but after searching a bit on the net (and more
>importantly on this list) I've yet to find some simple answers to the
>following question(s):


>What is the most important part for a 3d rendering farm; memory to
>each processor, processor speed and/or access to SAN?
>
>The kicker is that we're also going to use the same hardware for 2d
>composition in AfterFX and synth/fx processing for sound production.
>Ie. when we're through rendering we're not going to let that hardware
>rest but will utilise it in other parts of the workflow.

it depends heavily upon the type of 3d rendering you do.
first of all the software needs to be distributed. If it isn't then
a quad opteron dual core for sure is going to be faster.

If you have your own software which works distributed then benchmarking is
important. Usually after the processor you choose, the network is most
important decision. A combination of one way pingpong latency, distributed
shared memory (DSM) and having a good bandwidth is real important.

>I know that for fx processing prosessor speed is paramount, and for
>sample playback memory size is important, as well as local access to
>the samples, but what about 3d? We're going to use Discreets 3d Studio
>Max w/Brazil.

You might ship them an email and ask them whether they are distributed and
if so, whether they have done benchmarks at different systems.

Do not forget that working under linux is important for beowulf. Not 100%
necessary but sure adviced. 

Most commercial software just has windows support.

>We're aiming for off the shell 1U units or blades. Any pros/cons would
>be apprecieated, as for price they are pretty equal. I've been looking
>mostly at Intel solutions, since they are easily available from Dell
>and IBM.
>
>Furthermore, but this is off topic I guess, would a larger iron from
>SGI or SUN be better than a cluster for this kind of use (though we'd
>be loosing sound fx and AfterFX processing)?

It depends upon the number of nodes you plan to use. For like 2 nodes or so, 
which together form a brick of 4 cpu's, SGI sure is having superior custom
interconnects.

Note they are also expensive. 

At 64 processors, they are definitely too expensive for what they deliver.
Perhaps price has gone down a bit, but it used to be $1 million for a 64
processor altix3000. Not sure whether you get the fastest 1.6ghz processor
for that price. Might be you just get 1.5ghz for that.

Please consider that for 3d things like sorting and all kind of handling
and maintaining operations are important. It's not just the 16-32 bits
floats which are so important.

So dual core opteron is going to be a real cheap processor that's going to
be superior for this task.

I do not know about price for a 32 node itanium2 system, but for sure
building it yourself can be done real real cheap.

Say you get the dual core opterons for like $823 a piece. You need 64 of
them then. That's a bit less than $53000 just for processors. Then say
$1000 for each node in total. So the total price for the hardware excluding
network will be roughly $85000. Add to that the network costs. I see at
quadrics homepage that they charge for 32 nodes roughly $16000 for a switch
32 way. So i guess that will be another $50k. Dolphin will be similar price
(no DSM possibilities though) and pingpong latency. So you'll have for
$130k roughly a 128 core system then of 1.8Ghz opterons.

The one way pingpong latency of such cluster is just as good as it is from
an altix3000 machine (64 processors), no matter what SGI wants to believe
you on paper.

Just watch my exact words: "one way ping-pong latency from this node to the
furthest away node". That's one of the things that matters.

For Sun, double the price of a selfbuild cluster and you know what they
charge for that performance usually. 

Please note that most economic to buy is a 8 node system, you can assemble
such a 32 processor or 64 core system. I see at www.quadrics.com $3700 now
for switch mentionned. Nearly all manufacturers charge roughly $3800-$5000
for a 8-way switch.

pdsh as software to start the jobs over the network and there you go.

Vincent

>Investment timeframe is 6 months, and we're looking at < USD 50K

You can build for that price 2 clusters from 32 cores (8 node dual opteron
dual core). 

Please realize the huge advantage of self build small clusters running linux.

They have a 99% uptime usually.

>I'm sure this is an asked and answered question, but I've been going
>through this the last few days, doing searches and still nothing in
>the first 30 results on google..., even when constraining on a
>particular site, like the mailing list archives.

Every company wants to sell their product. 

Please realize that software in the end will cause you to take a certain
decision. That's the advantage i have in that respect. 

I program my own software and can take advantage of things like DSM :)

>Best regards,
>Paul K Egell-Johnsen
>
>_______________________________________________
>Beowulf mailing list, Beowulf at beowulf.org
>To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
>
>