From pc7 at sanger.ac.uk  Tue May 19 12:14:24 2026
From: pc7 at sanger.ac.uk (Peter Clapham)
Date: Tue, 19 May 2026 12:14:24 +0000
Subject: [Beowulf] HPC meets agentic AI tools, any advice / thoughts?
Message-ID: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>

OK, so sticking my neck out a little here

How are people covering the risks from AI agentic tools across their HPC platforms

Ducks and listens?

Pete

Sent from Outlook for Mac

----------------------------------------------------------------------
The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20260519/6f548846/attachment.htm>

From mahmood.sayed at gmail.com  Tue May 19 12:40:47 2026
From: mahmood.sayed at gmail.com (Mahmood Sayed)
Date: Tue, 19 May 2026 08:40:47 -0400
Subject: [Beowulf] HPC meets agentic AI tools, any advice / thoughts?
In-Reply-To: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>
References: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>
Message-ID: <4AD31B46-C1DA-4195-A12B-10FA50CCDE20@gmail.com>

We?ve implemented a hard data protection zone for all critical and clinical data - a backup copy on warm storage not accessible by any agents  Let the agents run amok on a non-critical data. Also implemented spawn limits and cycle timers. 

> On May 19, 2026, at 8:15?AM, Peter Clapham <pc7 at sanger.ac.uk> wrote:
> 
> ?
> OK, so sticking my neck out a little here
> 
> How are people covering the risks from AI agentic tools across their HPC platforms
> 
> Ducks and listens?
> 
> Pete
> 
> Sent from Outlook for Mac
> The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20260519/25191eec/attachment.htm>

From dag at sonsorol.org  Tue May 19 13:24:03 2026
From: dag at sonsorol.org (Chris Dag)
Date: Tue, 19 May 2026 09:24:03 -0400
Subject: [Beowulf] HPC meets agentic AI tools, any advice / thoughts?
In-Reply-To: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>
References: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>
Message-ID: <CAP-RRXx7ZX-DeG3-QJYm+ZdsPMQ6Uz6POMK_RoK90bWa3uwMDw@mail.gmail.com>

I'm finding the agentic stuff (Claude Code within VSCode IDE) surprisingly
useful for HPC operations, support and troubleshooting- especially when
paired with a RAG that I stuffed full of HPC specific documentation on
slurm, relion, cryosparc, schrodinger, posit workbench and AWS parallelcluster
details -- basically the RAG is full of the stuff I need to do every day.

My guardrails are usually:

 - The https://hpc-mcp.apps.bioteam.cloud/ custom RAG; I force my agent to
challenge itself against the RAG for every determination, config suggestion
or parameter setting

 - A read-only mode for running slurm commands or otherwise poking at the
filesystem and logs; sudo acts require manual human review before proceeding

 - A limited write-allowed mode so it can submit slurm jobs as me, run a
schrodinger job with a license request or pull a schrodinger job failure
postmortem file out of the cluster

 - When I want the agent to actually "do stuff" on an HPC system I'll have
it write out a local ansible playbook or bash script or .md file containing
instructions. I'll run local linters and static security scanners against
bash, terraform and ansible files and then have the agent stage them to git
or the HPC filesystem where I can manually review and run them.  For more
complex deployments involving lots of playbooks or terraform I'll invoke a
5-agent "review committee" to audit the files before they go anywhere.

 - In all cases I manually run the terraform, bash script or ansible
playbook

 - In all cases, my instructions force the LLM to query my custom hpc-docs
MCP to challenge and verify all commands. 90% of the time, it finds a
mistake or hallucination involving parallelcluster, slurm, or schrodinger
specific settings or commands that the RAG will catch and force the agent
to fix.

 - I use Claude Teams so that Anthropic does not train on our data.  I've got
a M4 Pro mac mini connected to tailscale running a local LLM that I'll
sometimes offload jobs to but it's nowhere near as good as the frontier
models and super slow

My hpc-docs rag is online here with lots of technical documentation on
sources, ingest and architecture -- https://hpc-mcp.apps.bioteam.cloud/

the actual RAG content itself is gated behind our Okta SSO server because
I've stuffed the rag with content that is not actually public (some
consulting notes, some vendor stuff that is not public)  so it's not
actually useful by anyone else but I'd love to learn what others are doing
in this space.

I admit I'm kinda terrified what user-space people would do if not paranoid
and careful. I think main risk is data loss or mangling on the local HPC
but also the data leakage risk of end-users talking to remote LLMs and
sending info there they should not.

my $.02 only




On Tue, May 19, 2026 at 8:14?AM Peter Clapham <pc7 at sanger.ac.uk> wrote:

> OK, so sticking my neck out a little here
>
> How are people covering the risks from AI agentic tools across their HPC
> platforms
>
> Ducks and listens?
>
> Pete
>
> Sent from Outlook for Mac
> ------------------------------
> The Wellcome Sanger Institute is operated by Genome Research Limited, a
> charity registered in England with number 1021457 and a company registered
> in England with number 2742969, whose registered office is Wellcome Sanger
> Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20260519/fbc1584d/attachment-0001.htm>

From hearnsj at gmail.com  Tue May 19 17:54:31 2026
From: hearnsj at gmail.com (John Hearns)
Date: Tue, 19 May 2026 18:54:31 +0100
Subject: [Beowulf] HPC meets agentic AI tools, any advice / thoughts?
In-Reply-To: <CAP-RRXx7ZX-DeG3-QJYm+ZdsPMQ6Uz6POMK_RoK90bWa3uwMDw@mail.gmail.com>
References: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>
 <CAP-RRXx7ZX-DeG3-QJYm+ZdsPMQ6Uz6POMK_RoK90bWa3uwMDw@mail.gmail.com>
Message-ID: <CAPqNE2XyKrWvG+F1sh3NJgo21vp=bwuaQF=obOphh2+dFjZSug@mail.gmail.com>

Posit workbench? Damn.. I thought you were talking about Posit number system

On Tue, 19 May 2026 at 14:24, Chris Dag via Beowulf <beowulf at beowulf.org>
wrote:

> I'm finding the agentic stuff (Claude Code within VSCode IDE)
> surprisingly useful for HPC operations, support and troubleshooting-
> especially when paired with a RAG that I stuffed full of HPC specific
> documentation on slurm, relion, cryosparc, schrodinger, posit workbench and
> AWS parallelcluster details -- basically the RAG is full of the stuff I
> need to do every day.
>
> My guardrails are usually:
>
>  - The https://hpc-mcp.apps.bioteam.cloud/ custom RAG; I force my agent
> to challenge itself against the RAG for every determination, config
> suggestion or parameter setting
>
>  - A read-only mode for running slurm commands or otherwise poking at the
> filesystem and logs; sudo acts require manual human review before
> proceeding
>
>  - A limited write-allowed mode so it can submit slurm jobs as me, run a
> schrodinger job with a license request or pull a schrodinger job failure
> postmortem file out of the cluster
>
>  - When I want the agent to actually "do stuff" on an HPC system I'll have
> it write out a local ansible playbook or bash script or .md file
> containing instructions. I'll run local linters and static security
> scanners against bash, terraform and ansible files and then have the agent
> stage them to git or the HPC filesystem where I can manually review and
> run them.  For more complex deployments involving lots of playbooks or
> terraform I'll invoke a 5-agent "review committee" to audit the files
> before they go anywhere.
>
>  - In all cases I manually run the terraform, bash script or ansible
> playbook
>
>  - In all cases, my instructions force the LLM to query my custom hpc-docs
> MCP to challenge and verify all commands. 90% of the time, it finds a
> mistake or hallucination involving parallelcluster, slurm, or schrodinger
> specific settings or commands that the RAG will catch and force the agent
> to fix.
>
>  - I use Claude Teams so that Anthropic does not train on our data.  I've got
> a M4 Pro mac mini connected to tailscale running a local LLM that I'll
> sometimes offload jobs to but it's nowhere near as good as the frontier
> models and super slow
>
> My hpc-docs rag is online here with lots of technical documentation on
> sources, ingest and architecture -- https://hpc-mcp.apps.bioteam.cloud/
>
> the actual RAG content itself is gated behind our Okta SSO server because
> I've stuffed the rag with content that is not actually public (some
> consulting notes, some vendor stuff that is not public)  so it's not
> actually useful by anyone else but I'd love to learn what others are doing
> in this space.
>
> I admit I'm kinda terrified what user-space people would do if not
> paranoid and careful. I think main risk is data loss or mangling on the
> local HPC but also the data leakage risk of end-users talking to remote
> LLMs and sending info there they should not.
>
> my $.02 only
>
>
>
>
> On Tue, May 19, 2026 at 8:14?AM Peter Clapham <pc7 at sanger.ac.uk> wrote:
>
>> OK, so sticking my neck out a little here
>>
>> How are people covering the risks from AI agentic tools across their HPC
>> platforms
>>
>> Ducks and listens?
>>
>> Pete
>>
>> Sent from Outlook for Mac
>> ------------------------------
>> The Wellcome Sanger Institute is operated by Genome Research Limited, a
>> charity registered in England with number 1021457 and a company registered
>> in England with number 2742969, whose registered office is Wellcome Sanger
>> Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
>> _______________________________________________
>> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
>> To change your subscription (digest mode or unsubscribe) visit
>> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>>
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20260519/f107e5ad/attachment.htm>

From deadline at eadline.org  Tue May 19 21:15:29 2026
From: deadline at eadline.org (Douglas Eadline)
Date: Tue, 19 May 2026 17:15:29 -0400
Subject: [Beowulf] HPC meets agentic AI tools, any advice / thoughts?
In-Reply-To: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>
References: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>
Message-ID: <cab1f103c40d20f3f22f07a11cbd118b.squirrel@mail.eadline.org>


The TPC (Trillion Parameter Consortium: AI for Science) meetings
are full of people doing science with AI.
IIRC, this topic came up at the last meeting TPC25.

This years meeting TPC26 is in Baltimore

  https://tpc26.org/

Unfortunately, I will not be able to attend, but anyone in the
DC area (or east coast) it may be worth a visit to talk
with like minded individuals (like SC there is a large portion
implementers)

--
Doug



> OK, so sticking my neck out a little here
>
> How are people covering the risks from AI agentic tools across their HPC
> platforms
>
> Ducks and listens

>
> Pete
>
> Sent from Outlook for Mac
>
> ----------------------------------------------------------------------
> The Wellcome Sanger Institute is operated by Genome Research Limited, a
> charity registered in England with number 1021457 and a company registered
> in England with number 2742969, whose registered office is Wellcome Sanger
> Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
>


-- 
Doug


From pc7 at sanger.ac.uk  Wed May 20 10:17:39 2026
From: pc7 at sanger.ac.uk (Peter Clapham)
Date: Wed, 20 May 2026 10:17:39 +0000
Subject: [Beowulf] HPC meets agentic AI tools,
 any advice / thoughts? [EXT]
In-Reply-To: <cab1f103c40d20f3f22f07a11cbd118b.squirrel@mail.eadline.org>
References: <AM4PR07MB11085B2DDC5586584CD1911A5CE002@AM4PR07MB11085.eurprd07.prod.outlook.com>
 <cab1f103c40d20f3f22f07a11cbd118b.squirrel@mail.eadline.org>
Message-ID: <VI0PR07MB11084C5FCA2DA2E75943F30CACE012@VI0PR07MB11084.eurprd07.prod.outlook.com>

Many thanks Doug,
Good spot. I?ll see what I can find. The community currently seems split between completely segregated access, saying no to AI-agentic code tooling tools or advancing with hope as a strategy.

So exciting times.

Thanks again to all who have responded so far, much appreciated. If anyone has any other pointers, they will be greatly appreciated.

Best
Pete

Sent from Outlook for Mac
From: Douglas Eadline <deadline at eadline.org>
Date: Tuesday, 19 May 2026 at 22:15
To: Peter Clapham <pc7 at sanger.ac.uk>
Cc: beowulf at beowulf.org <beowulf at beowulf.org>
Subject: Re: [Beowulf] HPC meets agentic AI tools, any advice / thoughts? [EXT]

This Message Is From an Untrusted Sender
You have not previously corresponded with this sender.
Report Suspicious<https://us-phishalarm-ewt.proofpoint.com/EWT/v1/K3L-m5jYGn9kIMs!rK_4WDTPs8p0YTH2smqzHfm409Ioo2plsf_DeaIjV64kjyznErphYkFKdhrSXUTA5oD1C9EAkxBM3ZaYTebVfx2RfGBEdbaU2ZiLxoRWkS3Q_FMsLw$>


The TPC (Trillion Parameter Consortium: AI for Science) meetings
are full of people doing science with AI.
IIRC, this topic came up at the last meeting TPC25.

This years meeting TPC26 is in Baltimore

  https://urldefense.proofpoint.com/v2/url?u=https-3A__tpc26.org_&d=DwIDAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=ciV1Zl7RPeuv0_H4fzZoLN0BkNBXGKdvhavel3j9l08&m=zV7-umF3F2mlC_scBPxMaBqQS0xKoGkAl0yW33rslwyjV1SkZdfba_N_RS1Myk33&s=wTYljIl7QhHllPmFVdCzMbZs_klVbWiMh0ieDVGVYA0&e=

Unfortunately, I will not be able to attend, but anyone in the
DC area (or east coast) it may be worth a visit to talk
with like minded individuals (like SC there is a large portion
implementers)

--
Doug



> OK, so sticking my neck out a little here
>
> How are people covering the risks from AI agentic tools across their HPC
> platforms
>
> Ducks and listens?
>
> Pete
>
> Sent from Outlook for Mac
>
> ----------------------------------------------------------------------
> The Wellcome Sanger Institute is operated by Genome Research Limited, a
> charity registered in England with number 1021457 and a company registered
> in England with number 2742969, whose registered office is Wellcome Sanger
> Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit
> https://urldefense.proofpoint.com/v2/url?u=https-3A__beowulf.org_cgi-2Dbin_mailman_listinfo_beowulf&d=DwIDAw&c=D7ByGjS34AllFgecYw0iC6Zq7qlm8uclZFI0SqQnqBo&r=ciV1Zl7RPeuv0_H4fzZoLN0BkNBXGKdvhavel3j9l08&m=zV7-umF3F2mlC_scBPxMaBqQS0xKoGkAl0yW33rslwyjV1SkZdfba_N_RS1Myk33&s=RP-4D-QjiVUL7p6DMSqxof65czM9trGtscdlK6lGEVM&e=
>


--
Doug


----------------------------------------------------------------------
The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://beowulf.org/pipermail/beowulf/attachments/20260520/fbb1c8bd/attachment-0001.htm>

