[Rpm-ecosystem] Testing the Dependency Chain

[Pavel Odvody]

On Mon, 2015-08-24 at 09:21 -0400, Honza Šilhan wrote:
> > From: "Pavel Odvody" <podvody at redhat.com>
> > 
> > On Mon, 2015-08-24 at 11:29 +0200, Michael Schroeder wrote:
> > > As you're basically testing libsolv, you might as well use libsolv's
> > > testcase support.
> >
> > That looks interesting, Michael.
> > Can this framework do without actually writing all the specfiles?
> > 
> > Is there a documentation for the *.t file syntax or do I have to
> > reverse-engineer that from tests/* ? :)
> 
> The syntax is pretty straightforward. There is some docs of flags [1] or look
> into code for SOLVER_(FLAG) constants.
Thanks for the link.

> (The generated testcases from `dnf ... --debugsolver` can be seen in
> "debugdata/testcase.t")
> 
> Pavel, what is the aim of the tests?
> 1) Have a guide for package maintainers, the reference
> 2) make sure all DNF versions always honor the guide
> 3) test that RPM works too
> 
> For 1) we can do that in Behave (easily readable) and convert it to
> libsolv's testcases.
> (btw the newest Fedora libsolv build contains the `testsolv` executable.)
> 

I'm not sure about the factual merits, people unfamiliar with Behave are
going to struggle with parsing it's sentences anyway, it's like writing
software in COBOL for the sake of it being "self-documenting" :)
OTOH, if we had some syntax highlight for Behave test cases it could be
nice. 
Anyway, I'm going to try to rewrite a few tests into Behave to see how
it turns out.

> If we want to support 2) too then the test should run on DNF
> level. Hawkey adds some flags to libsolv by default and some of them
> are added by different DNF command. So you would test DNF by
> `dnf ... --assumeno --debugsolver` then inspect "debug data/solver.result".
> Injecting the different env of installed packages would be harder, maybe
> by setting different --installroot.
> 
> Having tested all rpm stack 3) you should continue with what you are doing.

All 3, the aim is to be:

   1) Technology neutral (can test dnf, libsolv, pkcon, rpm ...)
   2) Closest to the natural usage as possible (command line,
lightweight bindings)

> 
> It would be a good idea to have these tests written the same way as planned
> DNF functional tests. If we don't use Beaker then docker will be a good choice
> for creating a separate environment without the need of setup and teardown
> (just kill and run again the container). It should not be a problem to run
> it under the Jenkins - I've seen some posts mentioning running docker there. 
> 

Definitely, do you have some documentation/roadmap regarding the
functional tests?

> [1] https://github.com/openSUSE/libsolv/blob/master/doc/libsolv-bindings.txt
> 
> Honza
> _______________________________________________
> Rpm-ecosystem mailing list
> Rpm-ecosystem at lists.rpm.org
> http://lists.rpm.org/mailman/listinfo/rpm-ecosystem


-- 
Pavel Odvody <podvody at redhat.com>
Software Engineer - EMEA ENG Developer Experience
5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.rpm.org/pipermail/rpm-ecosystem/attachments/20150824/5f00d3d4/attachment.asc>