[Rpm-ecosystem] DNF team would like to take over libdnf ownership

Neal Gompa ngompa13 at gmail.com
Mon Oct 30 23:59:02 UTC 2017

On Mon, Oct 30, 2017 at 3:15 PM, Daniel Mach <dmach at redhat.com> wrote:
> On Mon, Oct 30, 2017 at 12:43 PM, Neal Gompa <ngompa13 at gmail.com> wrote:
>> On Mon, Oct 30, 2017 at 6:48 AM, Daniel Mach <dmach at redhat.com> wrote:
>> > Hi everyone,
>> > My name is Daniel Mach and I'm leading the DNF team.
>> >
>> >
>> > TL;DR:
>> >
>> > The DNF team wants to take libdnf over and re-start development, which
>> > would
>> > possibly include using C++ (only to limited and reasonable extent). The
>> > goal
>> > is to get existing DNF code into libdnf. The current APIs would remain
>> > intact.
>> > We're looking for your use cases.
>> >
>> Fantastic!
>> >
>> > More details follow:
>> >
>> > We are getting requests to implement more features in microdnf,
>> > such as modularity[1][2], unify repo cache and overall business logic.
>> > Porting big portions of existing DNF code (Python) into a C library
>> > seems to be the only feasible option to avoid code duplication.
>> >
>> > In the first place, we'd like to clarify libdnf ownership and
>> > responsibilities.
>> > Libdnf is currently co-maintained by several teams (DNF, PackageKit,
>> > rpm-ostree), which makes it an unmaintained orphan.
>> > There are gaps in roadmap, development and response time to the customer
>> > issues.
>> > The code suffers with a technical debt, especially the libhif/hawkey
>> > merge
>> > hasn't been finished.
>> >
>> > From organizational perspective, we are proposing following:
>> > * DNF team to own libdnf and be fully responsible for it.
>> > * DNF team to guarantee existing C and Python APIs.
>> This would probably be a mistake, given how the existing APIs are. I
>> would suggest picking which base of APIs you want to guarantee and
>> work from there.
> That's right, it's something close to what we're planning.
> Carefully choose what to support and what to discard,
> communicate with stakeholders, avoid problems.
> I just wanted to make a clear statement that users can expect
> an evolution rather than a big bang throwing away existing APIs
> and breaking everything.

That's good. Especially as we're looking at potentially using libdnf
for DrakX in Mageia (unless we decide to glue Python to Perl through

>> > * DNF team to discuss any API changes with the stakeholders in timely
>> > manner.
>> This means you should include soname bumps every time API/ABI changes
>> occur.
> Either that or supporting multiple soname versions.
> I don't want to make any promises, but we're looking even into that.

How would that even work? Unless you're talking about symbol
versioning? I'll note that glibc is the only libc that properly
supports symbol versioning, and other libc implementations for Linux
(and other OSes) either always export only the latest or ignore symbol
versioning entirely.

>> > * PackageKit to become a libdnf stakeholder.
>> > * rpm-ostree to become a libdnf stakeholder.
>> >
>> > We are looking for a way how to boost the development speed by using a
>> > more
>> > high-level programming language than C. We are considering C++ at the
>> > moment,
>> > because it offers some features we (mainly Python programmers) would
>> > appreciate,
>> > including strongly typed lists and dictionaries, less work with pointers
>> > and
>> > last, but not least - native objects. It also allows us to wrap and
>> > re-use
>> > existing C code without major changes.
>> >
>> I don't have a problem with C++. In fact, I'm more versed in C++ than
>> I am in C, so this is pretty nice for me. That said, I do have a
>> couple of concerns about it:
>> * Binding from C++ to languages is trickier than C, so do you plan to
>> provide a C API for this? Or maybe do you plan to wire in SWIG[1] to
>> dynamically generate them as libsolv does?
> SWIG it is for Python (and possibly other languages in the future).
> C will probably need a hand-written wrapper.
>> * Using C++ makes glib2 usage mostly pointless, as glib2 is by and
>> large replaced by the C++ STL. The only advantage of GLib2 is GObject
>> Introspection, but if you're using C++, you might as well do something
>> else for bindings generation (like SWIG).
> Replacing glib2 with C++ STL would be nice,
> but I'm sure we'll be using both for some time,
> before we refactor the code.
> Even then I'm not completely against using glib2 if it brings any benefits.
> Also if other tools in minimal distro use glib2, not using helps no-one...

I don't know of any tools in a minimal environment (think in
initramfs!) that use glib2.

That said, I don't particularly object to glib2, as I think it might
be possible to link everything statically (including glib2).

There *are* nice things you get with glib2, but a lot of it is pretty
redundant with C++11 / C++ 14.

>> * Please avoid Boost. The Zypper stack uses it and it's a major pain
>> (it's why Yocto dropped Zypper years ago).
> Boost on the blacklist already :)


>> [1]: http://www.swig.org/
>> > Short-term plan: Fix libdnf technical debt
>> > * Finish libhif and hawkey merge.
>> > * Remove unused code.
>> > * Fix code duplicates.
>> > * Hide private functions which were published by mistake.
>> >
>> This conflicts with guaranteeing all C/Python APIs. Again, I suggest
>> picking what you want to stabilize and go from there.
> Yes. Unfortunately there is some dead code and features nobody uses.
>> > Mid-term plan:
>> > * Merge DNF Base class with libdnf's context.
>> > * Re-shuffle existing C code into new structures, but still keep
>> > existing
>> > APIs.
>> > * Implement modularity[1]
>> >
>> > Long-term plan:
>> > * Full rewrite/cleanup
>> > * Move major part of DNF's code into libdnf
>> > * New, well-defined, fully supported API
>> >
>> > The Holy Grail:
>> > * libdnf becomes "The Software Management Library"
>> > * libdnf has fully documented and tested API
>> > * libdnf has bindings to infinite amount of languages :)
>> Though if we move from C to C++ (and consequently drop glib2
>> dependency), then GObject introspection goes away, which is what could
>> give us this (sorta). SWIG (as mentioned above) can offer this for us,
>> but you need to build this in from the beginning...
>> >
>> > I'd like to ask you for your feedback on the plan.
>> > If you have any use cases you'd like implemented, please open an
>> > issue[3] or
>> > a bug[4].
>> > I understand that the C++ part sounds controversial, but as long as we
>> > keep
>> > the
>> > C API available and intact, nothing should change for the existing
>> > users.
>> > Please keep the discussion technical.
>> >
>> The big question for me is if you decide to use C++, do you plan to
>> keep the glib2 dependency? From my point of view, glib2 is a wasteful
>> and unnecessary extra dependency for C++ libraries and applications.
> See above.
>> Also, where do librepo and libcomps fit into this plan? Both libraries
>> are also maintained similarly to libdnf, and they've caused their own
>> share of issues.
> They are intentionally out of scope for now.
> We need to focus on libdnf in the first place and avoid scope creep.

So I ask this because one of the biggest pain points Fedora users have
today is that PackageKit's DNF backend doesn't support groups (either
by RPM Group tag as Mageia uses or by composition group metadata as
Fedora uses). In the past, the reason for this is because libcomps was
difficult to deal with and no one wanted to leverage it. The libsolv
library has its own comps processing logic that we could leverage. In
addition, libsolv supports processing AppStream metadata and
generating solvables that can be queried and installed in a similar
manner. It'd be nice if this stuff was wired up in libdnf so that
consumers of the library can use it.

I already wired up the Group tag a long time ago, so all that's really
left is the comps groups.

>> In addition, it would be good if libdnf is capable of being used as a
>> static link library, mainly for building a statically linked version
>> of microdnf that would work in the initramfs for system recovery and
>> such.
> I had a similar idea couple years back, thanks for reminder!
> I'm putting this as a low priority item on my list.

There are a few interested parties (aside from Mageia) who are
interested in the idea of providing a useful basic package manager
tool for recovering from a bad situation when the package manager
binaries are screwed up on the installed environment. I think this
shouldn't be hard to accomplish, provided we accurately capture what
static libs are needed and make it possible to generate a fully-static
static link library.

真実はいつも一つ!/ Always, there's only one truth!

More information about the Rpm-ecosystem mailing list