[Rpm-ecosystem] Testing the Dependency Chain

[Pavel Odvody]

On Tue, 2015-08-25 at 06:09 -0400, Honza Šilhan wrote:
> > From: "Pavel Odvody" <podvody at redhat.com>
> > > 
> > > Pavel, what is the aim of the tests?
> > > 1) Have a guide for package maintainers, the reference
> > > 2) make sure all DNF versions always honor the guide
> > > 3) test that RPM works too
> > > 
> > > For 1) we can do that in Behave (easily readable) and convert it to
> > > libsolv's testcases.
> > > (btw the newest Fedora libsolv build contains the `testsolv` executable.)
> > > 
> > 
> > I'm not sure about the factual merits, people unfamiliar with Behave are
> > going to struggle with parsing it's sentences anyway, it's like writing
> > software in COBOL for the sake of it being "self-documenting" :)
> > OTOH, if we had some syntax highlight for Behave test cases it could be
> > nice.
> > Anyway, I'm going to try to rewrite a few tests into Behave to see how
> > it turns out.
> 
> That's the strength of the Behave is you can specify your own "language".
> IMO more understandable then json file for newbies.
> 
> The example of Behave test definition could be like :
> ```
> Given an available package B and package A recommending package B
> When a package A is marked for installation
> Then a package A and package B is installed
> ```

This complex format that would require natural language processing input
parsing from which we could in the end even generate spec files :)

What I have right now is something like this:

Feature: Richdeps/Behave test

Scenario: Install TestA
 Given I use the repository "repo-1"
 When I "install" a package "TestA" with "dnf"
 Then package "TestA, TestB" should be "installed"
 And package "TestC" should be "absent"

Actions: 
  remove, install

Pkg. managers: 
  dnf, rpm, pkcon

States: 
  installed, absent, removed

> 
> > > If we want to support 2) too then the test should run on DNF
> > > level. Hawkey adds some flags to libsolv by default and some of them
> > > are added by different DNF command. So you would test DNF by
> > > `dnf ... --assumeno --debugsolver` then inspect "debug data/solver.result".
> > > Injecting the different env of installed packages would be harder, maybe
> > > by setting different --installroot.
> > > 
> > > Having tested all rpm stack 3) you should continue with what you are doing.
> > 
> > All 3, the aim is to be:
> > 
> >    1) Technology neutral (can test dnf, libsolv, pkcon, rpm ...)
> >    2) Closest to the natural usage as possible (command line,
> > lightweight bindings)
> 
> 1) is in contradiction with 2). From the example above the sentence will be
> different then.
> 1) "When a package A is marked for installation"
> ~
> 2) "When `dnf install A` is executed"
> 
By technology neutral I mean that you can mix & match dnf/rpm/pkcon
calls in the same test case specification. 
We need to discriminate somehow which technology to run, either we can
separate test cases, or we can do it at runtime. 
Doing it at runtime means that I can test the following scenario:
 install pkgA with rpm, then install pkgB with dnf, and finally install
pkgC with pkcon
Which nicely captures the gist of what happens during daily usage of
related packaging technologies. So I'd dare to say that 1) implies 2).

On the other hand, if the test specification itself is technology
neutral, I suppose we might let the test runner decided which package
manager to use and issue a specific call. This way the same test suite
could be used to test each package manager separately, or mix & match
them depending on the level of composition/inheritance supported by
Behave. Guess I'll need to dive more into Behave internals, if anyone
has the knowledge how such composition/inheritance system could be
implemented, please share :)

> > > It would be a good idea to have these tests written the same way as planned
> > > DNF functional tests. If we don't use Beaker then docker will be a good
> > > choice
> > > for creating a separate environment without the need of setup and teardown
> > > (just kill and run again the container). It should not be a problem to run
> > > it under the Jenkins - I've seen some posts mentioning running docker
> > > there.
> > > 
> > 
> > Definitely, do you have some documentation/roadmap regarding the
> > functional tests?
> 
> The new team member should work on this from the beginning of the September
> while he will be changing the Fedora docs from yum to DNF.
> 
> If you have your own working test suite and you don't need to invest additional time
> to extending it's backend functionality, continue with covering the most test cases as
> possible (in whatever definition you like). Then if you have a free time you can define
> the Behave parsing steps. We can then use your tests + steps definition in Behave once
> we choose the backend ~ mid of the September (we can schedule the meeting with you).
> 

I'm starting to like the textual specification, while I still need to
stare at the template in a split window it's very comfy to write.
That being said, I'm rewriting the JSON specifications into Behave
features. 

Thanks

> 
> Honza
> _______________________________________________
> Rpm-ecosystem mailing list
> Rpm-ecosystem at lists.rpm.org
> http://lists.rpm.org/mailman/listinfo/rpm-ecosystem


-- 
Pavel Odvody <podvody at redhat.com>
Software Engineer - EMEA ENG Developer Experience
5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.rpm.org/pipermail/rpm-ecosystem/attachments/20150828/edb6de8b/attachment.asc>