[Rpm-maint] [Rpm-ecosystem] RFC: Relocate RPM and DNF databases to /usr

[Lukáš Hrázký]

On Fri, 2021-12-10 at 13:53 -0500, Chris Murphy wrote:
> {
> Apologies for not sending to rpm-ecosystem@ first off [1], then b.
> since I wasn't subscribed to rpm-ecosystem@ my reply [2] was dropped,
> messing up the threading.
> [1] http://lists.rpm.org/pipermail/rpm-maint/2021-December/018662.html
> [2] http://lists.rpm.org/pipermail/rpm-maint/2021-December/018687.html
> }
> 
> On Fri, Dec 10, 2021 at 7:39 AM Lukáš Hrázký <lhrazky at redhat.com> wrote:
> > 
> > On Fri, 2021-12-10 at 06:16 -0500, Neal Gompa wrote:
> > > (Adding the correct Daniel Mach email address, he moved from Red Hat
> > > to SUSE last month...)
> > > 
> > > On Thu, Dec 9, 2021 at 10:11 AM Chris Murphy <lists at colorremedies.com> wrote:
> > > > 
> > > > I'm pretty sure I'm generally opposed to the idea of logs being rolled
> > > > back (in a snapshot and rollback regime). Logs should always carry
> > > > forward. Ideally, indicate relevant things like rollbacks in the log,
> > > > because rolling back a log is a form of data loss. The history
> > > > database sounds more like a log than not. So I'm getting squeamish
> > > > about locating that somewhere else. Whereas system state information,
> > > > if it's primarily describing /usr, sounds like it needs to get
> > > > somewhere in /usr.
> > > > 
> > > 
> > > The history database is basically a transaction database like the
> > > rpmdb is. It maps rpmdb transactions to dnf actions.
> > 
> > I'm not sure this is true at all. I haven't examined the rpmdb ever,
> > but AFAIK it contains the list of currently installed rpms (and their
> > data), but not the history as they came and went (i.e. a log).
> > 
> > OTOH dnf history database is exactly that. The current system state is
> > in dnf 4 also computed as the aggregation of the transactions in the
> > history database, which is IMO very wrong and we're fixing it in dnf 5.
> 
> What's the timeframe for dnf 5 as it relates to the Fedora release cycle?
> 

The aim is to release dnf 5 in Fedora 38. Long before that we should
have a sort-of tech preview (as soon as it's somewhat consistently
working) and transition it into alpha/beta releases for testing, we'll
still be figuring out how to do that exactly.

> > > So if you're
> > > rolling back the rpm-managed content, it still makes sense to roll
> > > back the history database, since that's the state of the system. It's
> > > not terribly useful if the matching rpmdb transactions don't exist in
> > > the rpmdb.
> > 
> > Conceptually I agree with Chris on this: We shouldn't roll back the log
> > of transactions if we'd be rolling back the system itself. That way you
> > in effect erase that the system was running in a different state for an
> > amount of time, imagine an auditing situation.
> 
> Yeah, the auditing case is important to consider. The FHS can be
> aggravating in the context of snapshot and rollback regimes, in
> particular deciding whether a /usr /var and /etc are tied to each
> other for all time or if they can be mix and match. And that brings up
> how to name the snapshots (I've varied between simple version control
> by tacking on a .1 .2 .3 counter; to using ISO 8601 datetime stamps)
> and whether and how to do system assembly at startup [3].
> 
> [3] need for a discoverable subvolumes specification (thread crosses two months)
> https://lists.freedesktop.org/archives/systemd-devel/2021-November/047059.html
> https://lists.freedesktop.org/archives/systemd-devel/2021-December/047176.html
> 
> For my non-emphemeral systems:
> 
> * /usr /var /etc are directories contained in btrfs subvolume
> "root-x86-64:fedora.35"
> * read-only snapshots have an ISO 8601 datetime format tacked onto the end
> 
> * /var/log is a directory contained in subvolume "varlog-x86-64:fedora.35"
> * /var/lib/libvirt/images is a directory contained in subvolume
> "varlibvirtimages-x86-64:fedora.35"
> * no snapshots are ever made of these; by virtue of themselves being
> subvolumes, any snapshot of "root-x86-64:fedora.35" doesn't contain
> /var/log or /var/lib/libvirt/images so just like /home, they only ever
> carry forward in time.
> 
> Therefore in my case, snapshot and rollback doesn't affect any logs,
> including the systemd journal. They only ever carry forward in time.
> Same for VM images and for ~/ since that lives in /home and is a
> separate subvolume too by default in Fedora. But both RPM database and
> DNF history do get rolled back in this case. Which works out OK
> because usr/ var/ etc/ are all in one root. A rollback means DNF
> history still accurately describes that root's history and state.
> 
> As soon as there's mix and match any of usr/ var/ and etc/ it gets trickier.

So, is this basically the summary of your motivation for this change?
That rollbacks of your /usr is always in sync with the dnf-stored
system state?

FWIW we could do with a practical evaluation of impact and issues this
is causing now, are we talking desktops/servers/containers, etc.?

So far this looks like it can wait for dnf 5.

> > There are also issues with sqlite creating some lock files even if just
> > reading, which wouldn't be possible on a RO FS (IIRC how it was
> > explained to me), so there are other reasons why the history db will
> > likely not move to /usr.
> 
> Detect it and use overlayfs on a volatile /run directory for this use
> case? I'm not sure the use case really calls for a persistent file
> that survives a crash for reading read-only files?

AFAICT it doesn't require a persistent file. But if you're suggesting
for dnf itself to set up that overlayfs, I may be misjudging, but it
seems way out of bounds of what dnf should be doing.

But the situation may be solvable in other ways, e.g. storing those
files in a writable temporary location. I haven't investigated.

Tbh. given the ugliness and complexity added by history DB and how it
stores the system state (you seem to agree the transaction log
shouldn't be on /usr, but those two are stored together now) I doubt
we'll resolve this before dnf 5, unless there's a contribution which
neatly ties the loose ends (again, dnf 4 is ugly in a lot of ways...)

> > If there's anything in history db that is required for modularity to be
> > functional on a system, it should be moved to the dnf system state (in
> > dnf 5), history db should just be a log and nothing else (at most we
> > could use it to reconstruct the system state if it gets damaged while
> > the history db remains intact).
> 
> 
> I wonder if the dnf history ought to go in /var/log and be exempt like
> the systemd journal? Or maybe for auditing purposes it should write to
> both its own history file as well as into the systemd journal?

I guess for auditing you could do with a simple text log of the
transactions. The history DB supports querying and also undo/redo, so
it's a more sophisticated solution. There have been requests for that
simple log (in addition to the history DB) exactly because of
simplicity. But I don't think we'd want to store history DB in
/var/log.

/var/lib still seems like the arguably best place for it to me.

Cheers,
Lukas