[Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

[Jonathan Dieter]

On Mon, 2018-08-06 at 16:36 +0000, Zbigniew Jędrzejewski-Szmek wrote:
> Hi dnf and libsolv developers,
> 
> this mail is a continuation of an FPC [1] and a FESCo [2] tickets.
> 
> A proposal was made is to disallow packages in Fedora from using file
> deps, and to optimize dnf to not load filelists.xml. File deps would
> still be supported, because external packages and users want to use
> them, but they would not be allowed for distro packages.
> 
> Not downloading or loading filelists.xml which are required for file
> deps would provide significant bandwidth savings (~47 MB compressed)
> and noticeable runtime savings (~10s at dnf startup) in many common
> cases.
> 
> So this is something that is worth exploring, but it's not clear if
> it
> is at all feasible. It seems that dnf would need to support loading
> filelists.xml lazily. In the mailing list discussions, some people
> said that this would be hard, some people said that it would be
> possible… What is the situation here? IIUC, dnf would need to restart
> the resolution of a transaction mid-flight once it encounters a file
> dep,
> which would require support across the different layers.
> If Fedora commits to making use of this, would it be possible to
> implement this in dnf? What kind of changes would be required?
> 
> [1] https://pagure.io/packaging-committee/issue/714
> [2] https://pagure.io/fesco/issue/1955
> 
> Zbyszek, on behalf of FESCo (but not that this writeup is based
> on my understanding, so all errors are mine.)

This is a bit off on a tangent, but we are working on zchunk (see https
://fedoraproject.org/wiki/Changes/Zchunk_Metadata) which should make
this whole problem far less painful.  As things are going, it's highly
unlikely that we'll be completely done for F29 (we have working code
that needs review), but it should be good to go for F30.

Jonathan