[Rpm-ecosystem] Testing the Dependency Chain

[Honza Šilhan]

> From: "Pavel Odvody" <podvody at redhat.com>
> > 
> > Pavel, what is the aim of the tests?
> > 1) Have a guide for package maintainers, the reference
> > 2) make sure all DNF versions always honor the guide
> > 3) test that RPM works too
> > 
> > For 1) we can do that in Behave (easily readable) and convert it to
> > libsolv's testcases.
> > (btw the newest Fedora libsolv build contains the `testsolv` executable.)
> > 
> 
> I'm not sure about the factual merits, people unfamiliar with Behave are
> going to struggle with parsing it's sentences anyway, it's like writing
> software in COBOL for the sake of it being "self-documenting" :)
> OTOH, if we had some syntax highlight for Behave test cases it could be
> nice.
> Anyway, I'm going to try to rewrite a few tests into Behave to see how
> it turns out.

That's the strength of the Behave is you can specify your own "language".
IMO more understandable then json file for newbies.

The example of Behave test definition could be like :
```
Given an available package B and package A recommending package B
When a package A is marked for installation
Then a package A and package B is installed
```

> > If we want to support 2) too then the test should run on DNF
> > level. Hawkey adds some flags to libsolv by default and some of them
> > are added by different DNF command. So you would test DNF by
> > `dnf ... --assumeno --debugsolver` then inspect "debug data/solver.result".
> > Injecting the different env of installed packages would be harder, maybe
> > by setting different --installroot.
> > 
> > Having tested all rpm stack 3) you should continue with what you are doing.
> 
> All 3, the aim is to be:
> 
>    1) Technology neutral (can test dnf, libsolv, pkcon, rpm ...)
>    2) Closest to the natural usage as possible (command line,
> lightweight bindings)

1) is in contradiction with 2). From the example above the sentence will be
different then.
1) "When a package A is marked for installation"
~
2) "When `dnf install A` is executed"

> > It would be a good idea to have these tests written the same way as planned
> > DNF functional tests. If we don't use Beaker then docker will be a good
> > choice
> > for creating a separate environment without the need of setup and teardown
> > (just kill and run again the container). It should not be a problem to run
> > it under the Jenkins - I've seen some posts mentioning running docker
> > there.
> > 
> 
> Definitely, do you have some documentation/roadmap regarding the
> functional tests?

The new team member should work on this from the beginning of the September
while he will be changing the Fedora docs from yum to DNF.

If you have your own working test suite and you don't need to invest additional time
to extending it's backend functionality, continue with covering the most test cases as
possible (in whatever definition you like). Then if you have a free time you can define
the Behave parsing steps. We can then use your tests + steps definition in Behave once
we choose the backend ~ mid of the September (we can schedule the meeting with you).


Honza