[Rpm-ecosystem] lazy loading of filelists.xml to speed up dnf

[Zbigniew Jędrzejewski-Szmek]

> On Wed, Aug 8, 2018 at 7:09 PM Pascal Terjan <pterjan at gmail.com> wrote:
> > $ GET http://ftp.free.fr/mirrors/mageia.org/distrib/cauldron/x86_64/media/media_info/file-deps
> > /bin/csh
> > /bin/grep
> > /bin/perl
> > /usr/bin/ln
> > /usr/bin/rm
> > /sbin/service
> > /usr/bin/chattr
> > /usr/bin/guile
> > /usr/bin/openssl
> > /usr/bin/pear
> > /usr/bin/texhash
> > /usr/bin/tr
> > /usr/bin/which
> > /usr/sbin/groupadd
> > /usr/sbin/groupdel
> > /usr/sbin/useradd
> > /usr/sbin/userdel
This gives us the Mandriva/Mageia/Mandrake behaviour.

For Fedora, we need to look at createrepo_c. There was some
uncertainty whether e.g. /usr/libexec paths are in primary.xml. It
turns out they are *not*, and the whitelist is anything that matches
/etc|/usr/lib/sendmail|bin/ [1]. So we have paths like
/usr/share/awstats/wwwroot/cgi-bin/awredir.pl and 
/var/www/moodle/web/admin/tool/recyclebin/classes/base_bin.php (sic!)
in primary.xml.

It seems that this behaviour is accidental and arbitrary. Adding the
list of pattern to primary.xml seems like a good first step. I hope
we can later clean up up the patterns to only match '^/usr/s?bin/'...

[1] https://github.com/rpm-software-management/createrepo_c/blob/master/src/misc.h#L110-L118

> So the primary.xml already includes all that. If you actually look in
> the primary.xml.gz files in the Mageia rpm-md data, those are already
> there. The problem is that there are people who actually request files
> outside of the base whitelist as a means to be able to request
> "things" without knowing how they are packaged, because the file path
> is the consistent thing across distros. This is supported in YUM and
> DNF, just slightly differently.
> 
> In this case, the wish is to restore the YUM behavior. The idea is
> that stacking this on top of the Zchunk deltarepo extension will yield
> incredible boosts for everything.
Yes!

Zbyszek