Block Sizes

[Robert G. Brown]

On Thu, 31 Oct 2002, Leandro Tavares Carneiro wrote:

> We can't use ramdisks because the ammount of data nedeed to be load is very 
> huge, something about 100Gb of information needed to process the data. These 
> are tables with a lot of information, and they are loaded when demanded, and 
> wich tables are loaded depends of the data is going to process.
> Now, we are trying to run this on an SGI machine, but this application will 
> run also on our clusters, but with a different parallelism.
> The development team is searching to how to write an routine in C for read 
> with different block sizes, like a "dd" do, but they are Fortran especialists, 
> and they are searching for help...

I was afraid of that.  I'm not really sure what dd does, but you can
easily grab the source rpm for dd from any rpm repository and find out.
I tend to work through the standard C interface (e.g. man 2 open, man 2
read) and try NOT to use ioctl's or fcntl's to muck with file descriptor
properties unless I really really need to.  dd's blocksize could just be
the granularity with which data is read in with a read (as in

 read(fd,&mybuf,32768)

for reading 32K into mybuf) or it could use features in e.g. fcntl I've
never had an urge to explore.  The thing is, I strongly suspect that
this "32K" read just reads in 8 4K pages.  Or maybe it uses
fread/fwrite.  Dunno.

So (depending on how the fortran arrays are stored and referenced, a
thing I hope devoutly never to have to learn:-) it could be as simple as
a loop on

 read(fd,address_of_array,blocksize)

where one increments address_of_array by (type adjusted) blocksize every
pass and where one tests for and exits the loop on things like eof or
other errors.  That's what I'd do, anyway.  I'd guess read to be more
elementary than fread (and hence possibly faster) -- in fact fread could
just be this very loop implemented in read.

This MIGHT require the data to have been written out of your fortran
array by the inverse of this process to get the array contents in
exactly the right order.

   rgb

-- 
Robert G. Brown	                       http://www.phy.duke.edu/~rgb/
Duke University Dept. of Physics, Box 90305
Durham, N.C. 27708-0305
Phone: 1-919-660-2567  Fax: 919-660-2525     email:rgb at phy.duke.edu