[Beowulf] log_mtts_per_seg in mlx4 driver

[Ryan Novosielski]

Read the source code/answered my own question. Likely bogus since 2012, confirmed at least in 3.10.0-957.21.3 — this was the relevant fix:

/*
         * We want to scale the number of MTTs with the size of the
         * system memory, since it makes sense to register a lot of
         * memory on a system with a lot of memory.  As a heuristic,
         * make sure we have enough MTTs to cover twice the system
         * memory (with PAGE_SIZE entries).
         *
         * This number has to be a power of two and fit into 32 bits
         * due to device limitations, so cap this at 2^31 as well.
         * That limits us to 8TB of memory registration per HCA with
         * 4KB pages, which is probably OK for the next few months.
         */
        si_meminfo(&si);
        request->num_mtt =
                roundup_pow_of_two(max_t(unsigned, request->num_mtt,
                                         min(1UL << (31 - log_mtts_per_seg),
                                             si.totalram >> (log_mtts_per_seg - 1))));

> On Jul 9, 2019, at 4:12 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote:
> 
> Hi all,
> 
> There seems to be a whole lot of misinformation out there about the appropriate setting for the log_mtts_per_seg parameter in the mlx4 driver. Some folks suggest that it’s different between the RHEL/kernel.org provided driver and Mellanox OFED, some suggest it’s been fixed such that the default can assign 2x the total memory on a node since at least RHEL6.6 (and so one would assume even longer ago in OFED), and some places seem to still be carrying it forward because “maybe it matters.” 
> 
> Is anyone here sure of the current state? I’ll probably read the source code if not, but I’d like to spare myself the hassle.
> 
> --
> ____
> || \\UTGERS,  	 |---------------------------*O*---------------------------
> ||_// the State	 |         Ryan Novosielski - novosirj at rutgers.edu
> || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
> ||  \\    of NJ	 | Office of Advanced Research Computing - MSB C630, Newark
>     `'
> 
> _______________________________________________
> Beowulf mailing list, Beowulf at beowulf.org sponsored by Penguin Computing
> To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf