[Rpm-maint] [rpm-software-management/rpm] RFC: Limit build parallelism by available memory too, add tunables (#804) (#821)

Panu Matilainen notifications at github.com
Thu Aug 22 12:36:39 UTC 2019


This PR is to address two different but related issues:
1) On 32bit systems, available virtual memory is in practise very limited and easy to exhaust on systems with many CPUs. (RhBug:1729382)
2) There are systems with lot of CPUs (virtual or otherwise) but relatively limited memory, where just looking at available CPUs causes severe trashing. (RhBug:1118734)
    
This effectively caps the number of threads on 32bit to just four, but tunable by changing the thread size estimate. Number of processes is capped to gigabytes of memory or number of CPU's, whichever is smaller, and tunable by changing the process size estimate.

The series adds a new macro primitive for getting system memory information and builds bunch of heuristics on top of those. There's a lot of subtleties involved and no doubt some of them I've gotten wrong here, so this PR is not intended for immediate merging, but more as a basis for discussion and other feedback.

What bothers me personally here is that it adds quite a bit of brittle heuristics that we never needed before, heuristics that will inevitably go wrong. The 32bit thread issue could be handled by just slapping a hard limit, with just a couple of lines of code. OTOH, the many cpus but little memory -case is legit and solving does require heuristics no matter what. And since heuristics will go wrong sooner or later, there needs to be tunables, which is why so much of this is done in macro level despite being somewhat painful.
You can view, comment on, or merge this pull request online at:

  https://github.com/rpm-software-management/rpm/pull/821

-- Commit Summary --

  * Move smp-related macros from platform to main macros
  * Add some macro primitives for retrieving system memory information
  * Limit build parallelism by available memory too, add tunables (#804)

-- File Changes --

    M build/parseSpec.c (2)
    M doc/manual/macros (7)
    M macros.in (27)
    M platform.in (11)
    M rpmio/macro.c (67)

-- Patch Links --

https://github.com/rpm-software-management/rpm/pull/821.patch
https://github.com/rpm-software-management/rpm/pull/821.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/821
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20190822/2a79b44d/attachment.html>


More information about the Rpm-maint mailing list