[Rpm-maint] MT compression in rpmbuild ?

[Florian Festi]

On 03/26/2015 12:42 PM, Håkan Olsson wrote:
> Hello,
> 
> I’m looking for various ways forward in speeding up RPM build times.  In our various build pipelines (Continuous Integration, …) a common resource usage scenario is nice parallel compilations (-j10 or up, etc), after which the build spends several minutes in rpmbuild, with just one core running at 100%. Developers are in various ways skipping the RPM generation step in many cases due to this, not really what we want.

Welcome to the list!

> The little investigation I’ve done points to not very many (or any?) compression libraries themselves capable of multi-threading, what’s available seems to be mostly/only applications such as pxz, pbzip2, pigz using the standard libraries.  With rpmbuild using a library rather than e.g. streaming output to a child process replacing this appears to require a coding effort.

My guess is that choosing the right tools/libraries is one of the more
difficult parts. It might be possible to just do compression in a
separate process for some formats, though.

The compression handling code is actually one of the less ugly corners.
It can be found in rpmio/rpmio.c [1]. It implements an abstract file
descriptor data type. The patches adding lzma support[2] give a rough
idea of what is needed to add a new compression format (although some
things have changed since then. E.g. the name of the format changed to xz).

> I have a support case with RedHat open into investigating ways forward and one of the suggestions was to ask here.  Possibly this has been discussed before?

I got some question forwarded. So if you got some answers from
"Engineering" they are probably from me.

> (e.g. “deltarpm” has been mentioned as an issue, but at least in my case this is not a concern - we won’t be using it.)

Well, it is one reason for requiring bit for bit reproducibility of the
compression stage. But having bit for bit reproducibility is an value on
its own. So this is something to test for when choosing the library/tool.

Florian

[1] https://github.com/rpm-software-management/rpm/blob/master/rpmio/rpmio.c

[2]341266ae26e6685d1d870b28533469c653355bb7..
   edc5c0ef84a93d19acecaaf9a07f4ea3e9d8fe39


-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters