[Rpm-ecosystem] Testing the Dependency Chain

Pavel Odvody podvody at redhat.com
Fri Aug 21 16:21:37 UTC 2015


Hello,

I've setup a test suite for cozy testing of various setups that the
dependency system in RPM provides. Please refer to [1] for further
information about implementation details of the test suite.

=Test results and how to interpret them
The fact that test succeeded/failed needs to be properly interpreted as
the test context might be a bit different in each particular case so the
expectations about the behavior are properly aligned with what happens
under the hood. One example is the failed case #2 below, where it can be
noted that the test failed because the expectation is that Recommends
are not installed by default, and by 'failing' the test proved that
Recommends actually *are* installed by default.

Generally I'd call this "a game of subtle side effects", where each part
of the dependency chain adds something to the resulting equation and
until we compile some matrix/chart with all the influences that each of
those components have on the whole we're swimming in a gray area, since
we have no idea what the result is going to be (hunch does not count).

What I'd like to propose as a solution is test-driven formalization,
where we write tests to test each formal aspect of the dependency chain 
until we get 100% coverage. With that we can generate awesome
documentation for package maintainers so that they don't need to fear it
like a voodoo.

=How to test?
First we need to build the container image that will serve as a basis
for our test environment.
  
  $ git clone https://github.com/shaded-enmity/richdeps-docker
  $ cd richdeps-docker/
  $ docker build -t richdeps:1.0.0 .

The git repo already contains one testing repository and a few test cases.
So let's try the first one:

  $ ./test-launcher.py tests/test1.json
  Loading test configuration from:
   tests/test1.json
  Starting container:
   docker run -i -v /home/podvody/Repos/richdeps-docker/repos/test-1:/repo:Z richdeps:1.0.0

  OK

The test succeeded as expected, we installed a package TestA which
requires (TestB | TestC), and recommends TestC -- packages TestA and
TestC were installed.

The output of the second test is quite lengthy since it fails:

  $ ./test-launcher.py tests/test2.json
  (see output at: http://pastebin.com/hb61xrNX)

The test failed since we installed TestB before TestA, and 
installing TestA then also installed TestC, which was only in
Recommended, and the (TestB | TestC) requirement was already satisfied.

=Repos and packages
All repos and packages we created manually, I'm currently looking into 
using rpmfluff[2] to make the test suite somewhat automated.


Suggestions, discussion, pull requests with test cases and repos are
very welcome :)

[1]: https://github.com/shaded-enmity/richdeps-docker
[2]: https://fedorahosted.org/rpmfluff/
-- 
Pavel Odvody <podvody at redhat.com>
Software Engineer - EMEA ENG Developer Experience
5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.rpm.org/pipermail/rpm-ecosystem/attachments/20150821/1bad275e/attachment.asc>


More information about the Rpm-ecosystem mailing list