From chris at csamuel.org  Mon Feb  3 01:26:03 2025
From: chris at csamuel.org (Chris Samuel)
Date: Sun, 2 Feb 2025 17:26:03 -0800
Subject: [Beowulf] Monitoring communication + processing time
In-Reply-To: <CAEGBW9d-Cyv9526B79YwcFG_aCGFnPf+c=WS3MrTOkEZ+bRCjA@mail.gmail.com>
References: <CAEGBW9d-Cyv9526B79YwcFG_aCGFnPf+c=WS3MrTOkEZ+bRCjA@mail.gmail.com>
Message-ID: <51f7eb99-c7ff-481d-989e-7aafcd41b2d6@csamuel.org>

On 15/1/25 5:04 pm, Alexandre Ferreira Ramos via Beowulf wrote:

> Does anyone have a hint about how we should proceed for this monitoring?

LLNL also has an MPI profiling library: https://github.com/LLNL/mpiP

I've not tried it myself, but I like the idea of it.

All the best,
Chris

From mdidomenico4 at gmail.com  Wed Feb  5 14:33:07 2025
From: mdidomenico4 at gmail.com (Michael DiDomenico)
Date: Wed, 5 Feb 2025 09:33:07 -0500
Subject: [Beowulf] malloc on filesystem
Message-ID: <CABOsP2P+vsCXn=BVVP8KcHbqnXrLOqvsNVVj4t+iup2FTL6YEA@mail.gmail.com>

this might sound like a bit of an oddify, but does anyone know if
there's a library out there that will let me override malloc calls to
memory and direct them to a filesystem instead?  ie using the
filesystem as memory instead of ram for a program.  ideally something
i can LD_PRELOAD on top of a static binary.  understandably this is
generally a silly thing to do, but you know, users...

google is failing me, my search terms likely aren't right.  i'm
looking through some of the older checkpointing codes at the moment.
maybe someone cut shortcut my search

From stewart at serissa.com  Wed Feb  5 14:46:24 2025
From: stewart at serissa.com (Serissa)
Date: Wed, 5 Feb 2025 09:46:24 -0500
Subject: [Beowulf] malloc on filesystem
In-Reply-To: <CABOsP2P+vsCXn=BVVP8KcHbqnXrLOqvsNVVj4t+iup2FTL6YEA@mail.gmail.com>
References: <CABOsP2P+vsCXn=BVVP8KcHbqnXrLOqvsNVVj4t+iup2FTL6YEA@mail.gmail.com>
Message-ID: <509D751B-DEF0-4600-892A-4D96C7848080@serissa.com>

If you are willing to mmap the whole file, then dlmalloc can do this.  The issue is that it expects the storage pool to be accessible with pointers.
I am not aware of an allocator that uses function calls to read and write its own metadata (so that you can abstract the metadata), but if you find one I'd like the reference!  In my previous job we wanted such a thing so that host code could run an allocator for GPU memory without load store access to the memory being allocated, and we couldn't find one. There were too many places in dlmalloc that did pointer access so it was too hard to figure out which ones were accessing the storage pool and which were local.

-Larry

> On Feb 5, 2025, at 9:33?AM, Michael DiDomenico <mdidomenico4 at gmail.com> wrote:
> 
> ?this might sound like a bit of an oddify, but does anyone know if
> there's a library out there that will let me override malloc calls to
> memory and direct them to a filesystem instead?  ie using the
> filesystem as memory instead of ram for a program.  ideally something
> i can LD_PRELOAD on top of a static binary.  understandably this is
> generally a silly thing to do, but you know, users...
> 
> google is failing me, my search terms likely aren't right.  i'm
> looking through some of the older checkpointing codes at the moment.
> maybe someone cut shortcut my search
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf


From hearnsj at gmail.com  Wed Feb  5 16:17:08 2025
From: hearnsj at gmail.com (John Hearns)
Date: Wed, 5 Feb 2025 16:17:08 +0000
Subject: [Beowulf] malloc on filesystem
In-Reply-To: <CABOsP2P+vsCXn=BVVP8KcHbqnXrLOqvsNVVj4t+iup2FTL6YEA@mail.gmail.com>
References: <CABOsP2P+vsCXn=BVVP8KcHbqnXrLOqvsNVVj4t+iup2FTL6YEA@mail.gmail.com>
Message-ID: <CAPqNE2VFwdanw9zX6oDvLwydk0Kywi8KtGbG=CNkPpWrebk9DQ@mail.gmail.com>

Is this any use  https://en.wikipedia.org/wiki/Zram

On Wed, 5 Feb 2025 at 15:50, Michael DiDomenico <mdidomenico4 at gmail.com>
wrote:

> this might sound like a bit of an oddify, but does anyone know if
> there's a library out there that will let me override malloc calls to
> memory and direct them to a filesystem instead?  ie using the
> filesystem as memory instead of ram for a program.  ideally something
> i can LD_PRELOAD on top of a static binary.  understandably this is
> generally a silly thing to do, but you know, users...
>
> google is failing me, my search terms likely aren't right.  i'm
> looking through some of the older checkpointing codes at the moment.
> maybe someone cut shortcut my search
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20250205/6190d521/attachment.htm>

From jcownie at gmail.com  Thu Feb  6 11:17:59 2025
From: jcownie at gmail.com (Jim Cownie)
Date: Thu, 6 Feb 2025 11:17:59 +0000
Subject: [Beowulf] Monitoring communication + processing time
In-Reply-To: <51f7eb99-c7ff-481d-989e-7aafcd41b2d6@csamuel.org>
References: <CAEGBW9d-Cyv9526B79YwcFG_aCGFnPf+c=WS3MrTOkEZ+bRCjA@mail.gmail.com>
 <51f7eb99-c7ff-481d-989e-7aafcd41b2d6@csamuel.org>
Message-ID: <BD6F0AF2-32F8-4667-89D0-7BE46283A642@gmail.com>

There are a number of open source MPI profiling libraries which Google can no doubt find for you; as recommended below, mpiP looks sane (though I haven't tried it myself)
Or, you can use the MPI Profiling interface to intercept MPI calls and time them yourself, though this is in effect writing your own MPI profiler, so seems somewhat unnecessary. If you do go this route, you should be able to do it as a separate add-on that doesn't require any application source code changes.

MPI has (at my insistence :-)) had a profiling interface since MPI-1, so this is not new technology. 

-- Jim
James Cownie <jcownie at gmail.com>
Mob: +44 780 637 7146

> On 3 Feb 2025, at 01:26, Chris Samuel <chris at csamuel.org> wrote:
> 
> On 15/1/25 5:04 pm, Alexandre Ferreira Ramos via Beowulf wrote:
> 
>> Does anyone have a hint about how we should proceed for this monitoring?
> 
> LLNL also has an MPI profiling library: https://github.com/LLNL/mpiP
> 
> I've not tried it myself, but I like the idea of it.
> 
> All the best,
> Chris
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20250206/37ad8f9c/attachment.htm>

From Josef.Weidendorfer at in.tum.de  Thu Feb  6 14:56:12 2025
From: Josef.Weidendorfer at in.tum.de (Weidendorfer, Josef)
Date: Thu, 6 Feb 2025 15:56:12 +0100
Subject: [Beowulf] Monitoring communication + processing time
In-Reply-To: <BD6F0AF2-32F8-4667-89D0-7BE46283A642@gmail.com>
References: <CAEGBW9d-Cyv9526B79YwcFG_aCGFnPf+c=WS3MrTOkEZ+bRCjA@mail.gmail.com>
 <51f7eb99-c7ff-481d-989e-7aafcd41b2d6@csamuel.org>
 <BD6F0AF2-32F8-4667-89D0-7BE46283A642@gmail.com>
Message-ID: <AB58F3B9-123B-45F0-80E5-7FBFDC62610A@in.tum.de>

Have a look at the tools page of VI-HPS:
     https://www.vi-hps.org/tools/tools.html
Most is open source, some is commercial.
It includes mpiP and OpenSpeedShop, but there is also Scalasca, TAU, Vampir ?

Josef

> Am 06.02.2025 um 12:17 schrieb Jim Cownie <jcownie at gmail.com>:
> 
> There are a number of open source MPI profiling libraries which Google can no doubt find for you; as recommended below, mpiP looks sane (though I haven't tried it myself)
> Or, you can use the MPI Profiling interface to intercept MPI calls and time them yourself, though this is in effect writing your own MPI profiler, so seems somewhat unnecessary. If you do go this route, you should be able to do it as a separate add-on that doesn't require any application source code changes.
> 
> MPI has (at my insistence :-)) had a profiling interface since MPI-1, so this is not new technology. 
> 
> -- Jim
> James Cownie <jcownie at gmail.com>
> Mob: +44 780 637 7146
> 
>> On 3 Feb 2025, at 01:26, Chris Samuel <chris at csamuel.org> wrote:
>> 
>> On 15/1/25 5:04 pm, Alexandre Ferreira Ramos via Beowulf wrote:
>> 
>>> Does anyone have a hint about how we should proceed for this monitoring?
>> 
>> LLNL also has an MPI profiling library: https://github.com/LLNL/mpiP
>> 
>> I've not tried it myself, but I like the idea of it.
>> 
>> All the best,
>> Chris
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
> 
> 
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf