From ludwig.nussel at suse.de Mon Jan 3 16:01:56 2022 From: ludwig.nussel at suse.de (Ludwig Nussel) Date: Mon, 3 Jan 2022 17:01:56 +0100 Subject: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr In-Reply-To: <874k73grw9.fsf@oldenburg.str.redhat.com> References: <871r2dpked.fsf@oldenburg.str.redhat.com> <874k73grw9.fsf@oldenburg.str.redhat.com> Message-ID: Florian Weimer wrote: > * Colin Walters: > [...] >> What I *did* wake up one day and say I'm going to do is make upgrades >> transactional and offline by default and hence safe. No one should >> ever fear starting an operating system upgrade while their laptop is >> at 30% battery. Administrators running important servers must be able >> to easily roll back when the kernel *or* systemd (or something) else >> regresses, because it's software, it regresses all the time despite >> our best efforts. > > I appreciate these efforts. > > Although transactional-update (as documented below) seems have one major > regression: transactional RPM updates now require reboots. 8-( Yes, that's a PITA. However, I'm sure if we wanted to invest some brains the mechanism could be made smarter to figure out what is actually safe to apply live somehow. >> So yes again, this does matter. And it matters because whether you're >> doing "image based upgrades" like ostree or just "client side offline >> updates" like the >> https://kubic.opensuse.org/documentation/man-pages/transactional-update.8.html >> thing - it's very important *what data specifically* is >> versioned/snapshotted and what isn't. On an ostree system for >> example, it's completely normal that there are *two* rpm databases >> (one you're running, one that's pending in the new root). >> >> All the data in the rpmdb is about content that's in `/usr`. > > That totally ignores Software Collections and packages from ISVs. If > the expected future that RPM is going to be an OS-internal software > deployment mechanism (and not be used for third-party software), it > should not be a side effect of this change, but an explicit decision. AFAIK at least for SLE Micro which uses the same tech as openSUSE MicroOS ie transactional updates, customers are expected to install extra software as container. The OS is not meant to install rpms from sources other than it's own repo. So for that product it was an explicit decision that rpm is only for OS stuff. However, that policy only applies to the host. Inside containers rpm can be used in whatever creative way as usual. Anyway, what's happening is that in order to make OS updates more robust, a stricter separation of what is OS and what isn't gets implemented. The read-only bits of the OS have converged into /usr nowadays?. It's expected to be kind of self contained. Systems can now even boot with empty /etc and /var as everything needed in there gets bootstrapped from content in /usr. If rpm is used to manage /usr content, all required information for that needs to be in /usr too and rpms have no business outside /usr anymore. Since rpm keeps package information in the rpmdb, the obvious next step is to demand moving the rpmdb to /usr. I think a reason why that doesn't feel right is because the rpmdb is a single binary blob that mixes different kinds of data. Maybe it could be split up. Package headers could be considered original data, strictly bound to the files the header refers to. So for OS files the headers would belong into /usr. Name indexes etc however are derivative data that can be reconstructed from headers. As such could be stored in /var and generated when needed?, eg when actually installing packages or after rollback. I've suggested storing headers separately here: https://github.com/rpm-software-management/rpm/issues/1151 I acknowledge that the title is misleading though as that doesn't mean no database, just different format :-) If the headers are stored where the files are that would open the possibility to define additional prefixes such as /opt or /usr/local with their own headers. The database in /var could merge the information to have a unified view. That would allow to have /usr read-only while still being able to install 3rd party add-ons in separate locations. cu Ludwig [1] https://lnussel.github.io/2020/12/16/fslayout/ [2] File flags (eg skipped files due to --excludedocs) are computed and stored on installation time. Would have to think how and where to store that. Or maybe not support them in certain scenarios. -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Ivo Totev HRB 36809 (AG N?rnberg) From lhrazky at redhat.com Tue Jan 11 16:42:24 2022 From: lhrazky at redhat.com (=?UTF-8?Q?Luk=C3=A1=C5=A1_Hr=C3=A1zk=C3=BD?=) Date: Tue, 11 Jan 2022 17:42:24 +0100 Subject: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr In-Reply-To: References: <40332836-968f-2945-0347-94c26f1d65b8@suse.com> <813f0139339ce954e387e65fe8fd8d88dc7bae96.camel@redhat.com> Message-ID: <65bfb53a2fe97639be6a742ee44a4ddcce695ac8.camel@redhat.com> Chris, apologies for the late reply. On Mon, 2021-12-13 at 13:02 -0500, Chris Murphy wrote: > On Mon, Dec 13, 2021 at 10:07 AM Luk?? Hr?zk? wrote: > > > > On Fri, 2021-12-10 at 17:08 +0100, Daniel Mach wrote: > > > > I understand your point about auditing, but does DNF have to handle > > > everything? > > > There should be better options of tracking filesystem rollbacks than > > > DNF's history database. > > > > > > > Maybe, but the history DB still contains a log of events as they > > happened over time, I don't think that belongs to /usr. > > How about: > > For read-write /usr store any database describing /usr state in > /usr/lib/sysimage > For read-only /usr store any database describing /usr state in > /var/lib/dnf with a /usr/lib/sysimage symlink pointing to it > /var is always read-write so any database describing /var state goes > in /var/lib/dnf That seems unnecessarily complex, and also: - if there's no symlink on read-only /usr/lib/sysimage, we wouldn't be able to create one - if you consider /usr and /var loosely coupled, linking from /usr/lib/sysimage to /var means the link could break any time or the content just not be what you expect anyway > This suggests that anything that can describe state, should have > separate databases for /usr and /var. I don't know to what degree dnf > touches various top level FHS directories: /var /usr /etc /home and so > on. But I wonder if the history databases should separately track the > things being touched? Note dnf only deals with installing RPMs, we have no means to track changes to particular files and thus track changes across various directories under root. My tentative conclusion from the disucssion so far is that we'll put the system state files to /usr/lib/sysimage in dnf 5 (I dislike the "sysimage" part of the path but we're not going to break the established convention I don't think). And the history db will likely stay in /var/lib. This adheres to KISS and anything more involved seems to not solve issues, only move them around while adding complexity. Cheers, Lukas From lists at colorremedies.com Wed Jan 12 07:34:26 2022 From: lists at colorremedies.com (Chris Murphy) Date: Wed, 12 Jan 2022 00:34:26 -0700 Subject: [Rpm-ecosystem] [Rpm-maint] RFC: Relocate RPM and DNF databases to /usr In-Reply-To: <65bfb53a2fe97639be6a742ee44a4ddcce695ac8.camel@redhat.com> References: <40332836-968f-2945-0347-94c26f1d65b8@suse.com> <813f0139339ce954e387e65fe8fd8d88dc7bae96.camel@redhat.com> <65bfb53a2fe97639be6a742ee44a4ddcce695ac8.camel@redhat.com> Message-ID: On Tue, Jan 11, 2022 at 9:42 AM Luk?? Hr?zk? wrote: > > Chris, apologies for the late reply. > > On Mon, 2021-12-13 at 13:02 -0500, Chris Murphy wrote: > > On Mon, Dec 13, 2021 at 10:07 AM Luk?? Hr?zk? wrote: > > > > > > On Fri, 2021-12-10 at 17:08 +0100, Daniel Mach wrote: > > > > > > I understand your point about auditing, but does DNF have to handle > > > > everything? > > > > There should be better options of tracking filesystem rollbacks than > > > > DNF's history database. > > > > > > > > > > Maybe, but the history DB still contains a log of events as they > > > happened over time, I don't think that belongs to /usr. > > > > How about: > > > > For read-write /usr store any database describing /usr state in > > /usr/lib/sysimage > > For read-only /usr store any database describing /usr state in > > /var/lib/dnf with a /usr/lib/sysimage symlink pointing to it > > /var is always read-write so any database describing /var state goes > > in /var/lib/dnf > > That seems unnecessarily complex, and also: > - if there's no symlink on read-only /usr/lib/sysimage, we wouldn't be > able to create one > > - if you consider /usr and /var loosely coupled, linking from > /usr/lib/sysimage to /var means the link could break any time or the > content just not be what you expect anyway > > > This suggests that anything that can describe state, should have > > separate databases for /usr and /var. I don't know to what degree dnf > > touches various top level FHS directories: /var /usr /etc /home and so > > on. But I wonder if the history databases should separately track the > > things being touched? > > Note dnf only deals with installing RPMs, we have no means to track > changes to particular files and thus track changes across various > directories under root. > > My tentative conclusion from the disucssion so far is that we'll put > the system state files to /usr/lib/sysimage in dnf 5 (I dislike the > "sysimage" part of the path but we're not going to break the > established convention I don't think). And the history db will likely > stay in /var/lib. This adheres to KISS and anything more involved seems > to not solve issues, only move them around while adding complexity. The Fedora 36 change proposal thread also now a sub-thread proposing a new top-level directory for this purpose, /state. Easiest way to find it it go to the main thread then search for /state https://lists.fedoraproject.org/archives/list/devel at lists.fedoraproject.org/thread/G6LFNGOXZXFWZM3IWT6MLZWGMNSIZCWM/#XG4AT7NLMYZD2UHPG35JLPYWDSR5UAZT If dnf history remains in /var/lib/dnf it needs to be snapshot and rolled back with /usr in order for things like dnf history, and its subcommands. If it were to go in /state along with rpmdb, it might work out better? Although that suggests /state needs to be snapshot and rolled back along with /usr as well. -- Chris Murphy