[Beowulf] Dealing with masquerade attacks (Was: CLuster - Mpich - tstmachines - Heeelp !!!!!!!!)
landman at scalableinformatics.com
Sat Jul 29 06:18:16 PDT 2006
Leif Nixon wrote:
> Mark Hahn <hahn at physics.mcmaster.ca> writes:
>> this is wandering pretty far afield. a cluster, to my way of thinking,
>> is intended to act as a single resource, and as such is a single trust
> I used to think that, as well. However, expensively bought experience
> has taught me otherwise.
> Events the last two years  have shown that if you have a cluster
s/cluster/any system whatsoever/
We have logs from 1997 onwards (ok, we purged the old ones) that get
> that is somehow reachable from the Internet there is a non-negligible
> risk that an intruder at some point will log in on it using stolen
> credentials. I know for a fact that a large fraction of Swedish
> academic clusters have had such visits.
As have a fair number of academic sites (and industrial sites). We have
seen everything from sniffed ftp/telnet passwords (early days), through
brute force cracking of passwords, as well as capturing "secure keys"
from USB fobs which have been a fad of a well meaning group of admins
> You cannot trust your users, because that user over there might
> actually be a pimply-faced kid holding a freshly stolen password in
> his sweaty palms.
Or worse, they may do something dumb. Like give out their passwords.
> I don't see the world doing away with password or private-key-on-disk
> authentication any time soon, so this problem is here to stay, I'm
> afraid. We have to learn to live with it.
Or private key on a USB fob, or key-loggers, or, ...
> In general, we have to acknowledge that our security will always be
> slightly broken. This means that you can't put all your effort and
> trust in a perimeter style defense, because it will never be perfect
> and one day somebody will penetrate it. You need defense-in-depth.
> (That old, worn phrase)
True. At the same time, your layers need to make sense. I don't want
to light off an argument here on password strength, but various
sniffing/cracking tools won't be affected by passwords of effectively
random characters and relatively short length. Even longer passphrases
are sniffable with keyloggers.
You need tools which can withstand a sniffing/logging/interception
attack and allow you to recover. A One-Time-Password-In-Everything
(http://en.wikipedia.org/wiki/One-time_password) method is probably
advisable for most internet facing systems. They are two factor
methods, and if done right, are quite a bit more secure for
authentication purposes than passphrase based systems alone.
Fundamentally the statement of depth of security, while cliche, is quite
> Which in this case boils down to: Yes, you do need internal security
> barriers in your cluster.
I would phrase it that you need to limit the damage one can do. Assume
they will compromise your system. Don't worry about how, simply assume
they get in as a normal user. What can they do?
If you can containerize the users (VM their main login, limit access to
file systems, have them work in a temporary directory, etc...) you would
reduce the risk that a malicious login could compromise much. If on
each login you force them to use a two factor authentication (with a
physical timed key device) to get at their files (disallow an automount
or a FUSE based file system on a different machine), after using a two
factor authentication to log in, you impose yet another limitation of
what they can do.
This of course needs to be balanced against the users desire to gain
easy access to their account and ease of use of the system.
In which case, the best security of the system comes from very good backups.
> As you note, hardening a cluster to untrusted external users "would
> take quite a bit of effort", but even when it would be unrealistic to
> go full-out virtualized and compartmentalized, you should still keep
> these issues in mind when designing a cluster.
> Ask yourself, what happens if an intruder gets access to one of the
> machines in the cluster? It's very hard to totally stop the intrusion
> from spreading across the cluster, but you *can* make life harder for
> the intruder, which might just buy you enough time to detect the
> intrusion in its early stages.
Hmmm. If the compute nodes are not nodes that can be logged into in the
first place, simply processing elements to be used, then this is moot
(ala Scyld). If the nodes need to be logged into in order to be used
you are constrained by the need to have automated login systems.
> So, for example, do you really need unlimited passwordless access
> across the entire cluster, or can you limit it in useful ways? Perhaps
> you can hook PAM up to PBS, so users only can access nodes they are
> scheduled on? Pay special attention to how root is allowed to access
> other machines. Export NFS filesystems read-only and mount them
> nosuid, unless you really need rw/suid. And, of course, never leave a
and don't forget root_squash.
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax : +1 734 786 8452
cell : +1 734 612 4615
More information about the Beowulf