[Rpm-ecosystem] [v2] Testing the Dependency Chain (w/ Behave)

Radek Holy rholy at redhat.com
Fri Sep 11 14:14:00 UTC 2015


----- Original Message -----
> From: "Pavel Odvody" <podvody at redhat.com>
> To: rpm-ecosystem at lists.rpm.org
> Sent: Friday, September 11, 2015 2:48:30 PM
> Subject: Re: [Rpm-ecosystem] [v2] Testing the Dependency Chain (w/ Behave)
> 
> On Fri, 2015-09-11 at 08:11 -0400, Radek Holy wrote:
> > 
> > ----- Original Message -----
> > > From: "Pavel Odvody" <podvody at redhat.com>
> > > To: rpm-ecosystem at lists.rpm.org
> > > Sent: Friday, September 11, 2015 1:48:03 PM
> > > Subject: Re: [Rpm-ecosystem] [v2] Testing the Dependency Chain (w/
> > > Behave)
> > > 
> > > On Fri, 2015-09-11 at 07:12 -0400, Radek Holy wrote:
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Pavel Odvody" <podvody at redhat.com>
> > > > > To: rpm-ecosystem at lists.rpm.org
> > > > > Sent: Monday, September 7, 2015 3:20:50 PM
> > > > > Subject: Re: [Rpm-ecosystem] [v2] Testing the Dependency Chain
> > > > > (w/
> > > > > Behave)
> > > > > 
> > > > > Update notice!
> > > > > 
> > > > > Rich dependencies now share common syntax across RPM and
> > > > > libsolv:
> > > > > - the Dockerfile was updated to rebuild both from git master
> > > > > - all testing repositories were ported to the new syntax
> > > > > 
> > > > > The operators are: or, and, if, else
> > > > > 
> > > > > New test cases are coming, stay tuned!
> > > > > 
> > > > > On Fri, 2015-08-28 at 19:05 +0200, Pavel Odvody wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > I've pushed an updated branch to the git repo [1], which now
> > > > > > implements
> > > > > > the tests on top of the Behave framework. One test is
> > > > > > currently
> > > > > > converted to the new specification (test-1).
> > > > > > 
> > > > > > Example run:
> > > > > > 
> > > > > > $ ./test-launcher.py test-1
> > > > > > Running test:
> > > > > >  test-1
> > > > > > Starting container:
> > > > > >  docker run -i -v /richdeps-docker/repo:/build:Z -v /richdeps
> > > > > > -docker/features:/behave:Z pavelo/richdeps:1.0.0 test-1
> > > > > > 
> > > > > > Feature: Richdeps/Behave test # behave/test-1.feature:1
> > > > > >   TestA requires (TestB OR TestC), TestA recommends TestC
> > > > > >   Scenario: Install TestA from repository "test-1"    #
> > > > > > behave/test
> > > > > > -1.feature:4
> > > > > >     Given I use the repository "test-1"               #
> > > > > > behave/steps/test_behave.py:51
> > > > > >     When I "install" a package "TestA" with "dnf"     #
> > > > 
> > > > 
> > > > I think you can use "context.config.userdata" in order to make it
> > > > technology agnostic (i.e. get rid of 'with "dnf"'). Content of
> > > > this
> > > > dictionary can be defined in the behave config as well as from
> > > > command line.
> > > 
> > > The goal is to be able to mix&match package managers within the
> > > same
> > > test case specification, for example to test package erasure
> > > through
> > > RPM after installing with DNF etc.
> > > Both RPM and libsolv have their own implementation of the
> > > dependency
> > > resolution so I guess it makes sense to test both.
> > > I suppose it could be used to "memoize" the last package manager
> > > used,
> > > allowing for constructs like:
> > > 
> > > When I "install" a package "PkgA" with "dnf"
> > >  And I "install" a package "PkgB"
> > > Then package "PkgA, PkgB" should be "installed"
> > > 
> > > But this dims the expressiveness of Behave and essentially is a
> > > shift
> > > from explicit to implicit definition (which is especially frowned
> > > upon
> > > in the land of Python).
> > 
> > What problems do you expect to find/prevent with these tests? In such
> > case, it doesn't make much sense to run these tests during the
> > continuous integration of a single package manager. Then I need to
> > ask who is going to run/maintain these tests?
> > 
> > If you insist on this style of tests, as an alternative, I can also
> > imagine something like:
> > 
> >     When I "install" a package "PkgA" with the user-defined package
> > manager
> >      And I "remove" a package "PkgA" with another package manager
> >     Then I am not really surprised that it works
> > 
> 
> These tests are not meant for unit nor integration testing phase, and
> are better fit to V&V phase.

"V&V phase" of what? When / How often does its V&V phase occur?

> I wonder how is that different from the specification of test-2 below:
> 
> Scenario:
>   Given I use the repository "test-1"
>   When I "install" a package "TestB" with "rpm"
>   Then package "TestB" should be "installed"
>   When I "install" a package "TestA" with "dnf"
>   Then package "TestA, TestC" should be "installed"

The difference is that there is no binary mentioned in the test. You can run the test on whatever program you want without a need to duplicate/modify the tests by simply running "behave --define manager=foo".

> RPM has no notion of repository and conversely libsolv has no notion of
> "single package" and since both technologies use their own dependency
> resolution mechanism to act upon a *single* database, it makes sense to
> offer that level of granularity to test writers.

Well, it's up to the step definitions to translate the sentence into correct command line arguments of the given package manager. In case of DNF, it may execute something like "dnf repository-packages test-1 install TestB", while in case of RPM, it may locate the package and execute something like "rpm -i url://of/the/package". You are testing the dependency resolution algorithm, right? Not the possibility to specify a package by a name or an URL. It has nothing to do with dependency resolution.

> The split of "WHEN install X with Y" and "THEN check if X is installed"
> is to catch situations where given tool returned a wrong return code
> for the action that took effect.

So what problems do you expect to find/prevent with tests "mixing&matching package managers"? I think that you are trying to test too much. I don't think that it's reasonable to expect the e.g. DNF would choose different packages (this is what the tests are supposed to cover, right?) if "TestB" is installed with "rpm" instead of "dnf". Did I miss the answer?

BTW, it's recommended to have just one THEN clause in each scenario. I think it's common to any testing framework that each test should test just one aspect. In this case, I think that the implementation of the WHEN step could directly check whether the package have been really installed and raise an exception otherwise. Then, there would be just 'Then package "TestC" should be "installed"'. But the test (I mean the mix&match) does not make sense to me.

> 
> > > 
> > > 
> > > > 
> > > > 
> > > > > > behave/steps/test_behave.py:61
> > > > > >     Then package "TestA, TestC" should be "installed" #
> > > > > > behave/steps/test_behave.py:71
> > > > > >     And package "TestB" should be "absent"            #
> > > > > > behave/steps/test_behave.py:71
> > > > > > 
> > > > > > 1 feature passed, 0 failed, 0 skipped
> > > > > > 1 scenario passed, 0 failed, 0 skipped
> > > > > > 4 steps passed, 0 failed, 0 skipped, 0 undefined
> > > > > > Took 0m5.615s
> > > > > > OK
> > > > > > 
> > > > > > 
> > > > > > [1]:
> > > > > > https://github.com/shaded-enmity/richdeps-docker/tree/behave-
> > > > > > int
> > > > > > egration
> > > > > > _______________________________________________
> > > > > > Rpm-ecosystem mailing list
> > > > > > Rpm-ecosystem at lists.rpm.org
> > > > > > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
> > > > > 
> > > > > --
> > > > > Pavel Odvody <podvody at redhat.com>
> > > > > Software Engineer - EMEA ENG Developer Experience
> > > > > 5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
> > > > > Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
> > > > > 
> > > > > 
> > > > > 
> > > > > _______________________________________________
> > > > > Rpm-ecosystem mailing list
> > > > > Rpm-ecosystem at lists.rpm.org
> > > > > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
> > > > > 
> > > > 
> > > 
> > > --
> > > Pavel Odvody <podvody at redhat.com>
> > > Software Engineer - EMEA ENG Developer Experience
> > > 5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
> > > Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > Rpm-ecosystem mailing list
> > > Rpm-ecosystem at lists.rpm.org
> > > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
> > > 
> > 
> 
> --
> Pavel Odvody <podvody at redhat.com>
> Software Engineer - EMEA ENG Developer Experience
> 5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
> Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
> 
> 
> 
> _______________________________________________
> Rpm-ecosystem mailing list
> Rpm-ecosystem at lists.rpm.org
> http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
> 

-- 
Radek Holý
Associate Software Engineer
Software Management Team
Red Hat Czech


More information about the Rpm-ecosystem mailing list