[Rpm-ecosystem] [v2] Testing the Dependency Chain (w/ Behave)
Pavel Odvody
podvody at redhat.com
Fri Sep 11 16:25:11 UTC 2015
On Fri, 2015-09-11 at 10:14 -0400, Radek Holy wrote:
> ----- Original Message -----
> > From: "Pavel Odvody" <podvody at redhat.com>
> > To: rpm-ecosystem at lists.rpm.org
> > Sent: Friday, September 11, 2015 2:48:30 PM
> > Subject: Re: [Rpm-ecosystem] [v2] Testing the Dependency Chain (w/
> > Behave)
> >
> > On Fri, 2015-09-11 at 08:11 -0400, Radek Holy wrote:
> > >
> > > ----- Original Message -----
> > > > From: "Pavel Odvody" <podvody at redhat.com>
> > > > To: rpm-ecosystem at lists.rpm.org
> > > > Sent: Friday, September 11, 2015 1:48:03 PM
> > > > Subject: Re: [Rpm-ecosystem] [v2] Testing the Dependency Chain
> > > > (w/
> > > > Behave)
> > > >
> > > > On Fri, 2015-09-11 at 07:12 -0400, Radek Holy wrote:
> > > > >
> > > > > ----- Original Message -----
> > > > > > From: "Pavel Odvody" <podvody at redhat.com>
> > > > > > To: rpm-ecosystem at lists.rpm.org
> > > > > > Sent: Monday, September 7, 2015 3:20:50 PM
> > > > > > Subject: Re: [Rpm-ecosystem] [v2] Testing the Dependency
> > > > > > Chain
> > > > > > (w/
> > > > > > Behave)
> > > > > >
> > > > > > Update notice!
> > > > > >
> > > > > > Rich dependencies now share common syntax across RPM and
> > > > > > libsolv:
> > > > > > - the Dockerfile was updated to rebuild both from git
> > > > > > master
> > > > > > - all testing repositories were ported to the new syntax
> > > > > >
> > > > > > The operators are: or, and, if, else
> > > > > >
> > > > > > New test cases are coming, stay tuned!
> > > > > >
> > > > > > On Fri, 2015-08-28 at 19:05 +0200, Pavel Odvody wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > I've pushed an updated branch to the git repo [1], which
> > > > > > > now
> > > > > > > implements
> > > > > > > the tests on top of the Behave framework. One test is
> > > > > > > currently
> > > > > > > converted to the new specification (test-1).
> > > > > > >
> > > > > > > Example run:
> > > > > > >
> > > > > > > $ ./test-launcher.py test-1
> > > > > > > Running test:
> > > > > > > test-1
> > > > > > > Starting container:
> > > > > > > docker run -i -v /richdeps-docker/repo:/build:Z -v
> > > > > > > /richdeps
> > > > > > > -docker/features:/behave:Z pavelo/richdeps:1.0.0 test-1
> > > > > > >
> > > > > > > Feature: Richdeps/Behave test # behave/test-1.feature:1
> > > > > > > TestA requires (TestB OR TestC), TestA recommends TestC
> > > > > > > Scenario: Install TestA from repository "test-1" #
> > > > > > > behave/test
> > > > > > > -1.feature:4
> > > > > > > Given I use the repository "test-1" #
> > > > > > > behave/steps/test_behave.py:51
> > > > > > > When I "install" a package "TestA" with "dnf" #
> > > > >
> > > > >
> > > > > I think you can use "context.config.userdata" in order to
> > > > > make it
> > > > > technology agnostic (i.e. get rid of 'with "dnf"'). Content
> > > > > of
> > > > > this
> > > > > dictionary can be defined in the behave config as well as
> > > > > from
> > > > > command line.
> > > >
> > > > The goal is to be able to mix&match package managers within the
> > > > same
> > > > test case specification, for example to test package erasure
> > > > through
> > > > RPM after installing with DNF etc.
> > > > Both RPM and libsolv have their own implementation of the
> > > > dependency
> > > > resolution so I guess it makes sense to test both.
> > > > I suppose it could be used to "memoize" the last package
> > > > manager
> > > > used,
> > > > allowing for constructs like:
> > > >
> > > > When I "install" a package "PkgA" with "dnf"
> > > > And I "install" a package "PkgB"
> > > > Then package "PkgA, PkgB" should be "installed"
> > > >
> > > > But this dims the expressiveness of Behave and essentially is a
> > > > shift
> > > > from explicit to implicit definition (which is especially
> > > > frowned
> > > > upon
> > > > in the land of Python).
> > >
> > > What problems do you expect to find/prevent with these tests? In
> > > such
> > > case, it doesn't make much sense to run these tests during the
> > > continuous integration of a single package manager. Then I need
> > > to
> > > ask who is going to run/maintain these tests?
> > >
> > > If you insist on this style of tests, as an alternative, I can
> > > also
> > > imagine something like:
> > >
> > > When I "install" a package "PkgA" with the user-defined
> > > package
> > > manager
> > > And I "remove" a package "PkgA" with another package manager
> > > Then I am not really surprised that it works
> > >
> >
> > These tests are not meant for unit nor integration testing phase,
> > and
> > are better fit to V&V phase.
>
> "V&V phase" of what? When / How often does its V&V phase occur?
Of "dependency chain" as technology invariant, run with each update of
tested component.
>
> > I wonder how is that different from the specification of test-2
> > below:
> >
> > Scenario:
> > Given I use the repository "test-1"
> > When I "install" a package "TestB" with "rpm"
> > Then package "TestB" should be "installed"
> > When I "install" a package "TestA" with "dnf"
> > Then package "TestA, TestC" should be "installed"
>
> The difference is that there is no binary mentioned in the test. You
> can run the test on whatever program you want without a need to
> duplicate/modify the tests by simply running "behave --define
> manager=foo".
I don't understand. In your example you had used 2 package managers,
how are you going to discriminate which to execute and where, without
mentioning them in the test spec?
When I "{ACTION1}" a package "{PKG1}" with "{MGR1}"
When I "{ACTION2}" a package "{PKG2}" with "{MGR2}"
External parametrization would be great if there was many package
managers to test, and you'd need to test only 1 at a time, not to
mention that the test is not now self-contained, but needs correct
external parameters to run. So apart from behave "feature" files you'd
also need additional test metadata that would specify how to invoke
particular test.
>
> > RPM has no notion of repository and conversely libsolv has no
> > notion of
> > "single package" and since both technologies use their own
> > dependency
> > resolution mechanism to act upon a *single* database, it makes
> > sense to
> > offer that level of granularity to test writers.
>
> Well, it's up to the step definitions to translate the sentence into
> correct command line arguments of the given package manager. In case
> of DNF, it may execute something like "dnf repository-packages test-1
> install TestB", while in case of RPM, it may locate the package and
> execute something like "rpm -i url://of/the/package". You are testing
> the dependency resolution algorithm, right? Not the possibility to
> specify a package by a name or an URL. It has nothing to do with
> dependency resolution.
That's *exactly* what it does [1].
One particular example is RPM install, which is like "install this
package(s) and bail if not possible", say that TestA requires TestB,
but RPM does not search through the "repo" for TestB when installing
TestA.
[1]: https://github.com/shaded-enmity/richdeps-docker/blob/master/features/steps/test_behave.py
>
> > The split of "WHEN install X with Y" and "THEN check if X is
> > installed"
> > is to catch situations where given tool returned a wrong return
> > code
> > for the action that took effect.
>
> So what problems do you expect to find/prevent with tests
> "mixing&matching package managers"? I think that you are trying to
> test too much. I don't think that it's reasonable to expect the e.g.
> DNF would choose different packages (this is what the tests are
> supposed to cover, right?) if "TestB" is installed with "rpm" instead
> of "dnf". Did I miss the answer?
Well, I think otherwise, because the result is indeed different, just
think what happens when TestB itself has some Requires. With this
testing methodology you can easily replicate what happened in real
-world, where you get some packages installed through package kit, some
through DNF, some through RPM -- ordering of these event matters.
You can also use RPM to prepare the system before you run a libsolv
query.
>
> BTW, it's recommended to have just one THEN clause in each scenario.
> I think it's common to any testing framework that each test should
> test just one aspect. In this case, I think that the implementation
> of the WHEN step could directly check whether the package have been
> really installed and raise an exception otherwise. Then, there would
> be just 'Then package "TestC" should be "installed"'. But the test (I
> mean the mix&match) does not make sense to me.
Well, THEN and bunch of AND's and BUT's. In my case I use the THEN
clause to diff the current package set with one obtained during GIVEN
clause. It could all happen during the WHEN clause, but then the WHEN
clause would be like 200 chars wide and the THEN clause would be
essentially a no-op.
All in all, I'd like to move from the theoretical plane, if you think
that the methodology could benefit from the changes you propose, please
create some PoC / pull request, or we can brainstorm this topic next
week to find a solution that might be usable in both cases without
sacrificing much.
>
> >
> > > >
> > > >
> > > > >
> > > > >
> > > > > > > behave/steps/test_behave.py:61
> > > > > > > Then package "TestA, TestC" should be "installed" #
> > > > > > > behave/steps/test_behave.py:71
> > > > > > > And package "TestB" should be "absent" #
> > > > > > > behave/steps/test_behave.py:71
> > > > > > >
> > > > > > > 1 feature passed, 0 failed, 0 skipped
> > > > > > > 1 scenario passed, 0 failed, 0 skipped
> > > > > > > 4 steps passed, 0 failed, 0 skipped, 0 undefined
> > > > > > > Took 0m5.615s
> > > > > > > OK
> > > > > > >
> > > > > > >
> > > > > > > [1]:
> > > > > > > https://github.com/shaded-enmity/richdeps-docker/tree/beh
> > > > > > > ave-
> > > > > > > int
> > > > > > > egration
> > > > > > > _______________________________________________
> > > > > > > Rpm-ecosystem mailing list
> > > > > > > Rpm-ecosystem at lists.rpm.org
> > > > > > > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
> > > > > >
> > > > > > --
> > > > > > Pavel Odvody <podvody at redhat.com>
> > > > > > Software Engineer - EMEA ENG Developer Experience
> > > > > > 5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
> > > > > > Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Rpm-ecosystem mailing list
> > > > > > Rpm-ecosystem at lists.rpm.org
> > > > > > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
> > > > > >
> > > > >
> > > >
> > > > --
> > > > Pavel Odvody <podvody at redhat.com>
> > > > Software Engineer - EMEA ENG Developer Experience
> > > > 5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
> > > > Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Rpm-ecosystem mailing list
> > > > Rpm-ecosystem at lists.rpm.org
> > > > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
> > > >
> > >
> >
> > --
> > Pavel Odvody <podvody at redhat.com>
> > Software Engineer - EMEA ENG Developer Experience
> > 5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
> > Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
> >
> >
> >
> > _______________________________________________
> > Rpm-ecosystem mailing list
> > Rpm-ecosystem at lists.rpm.org
> > http://lists.rpm.org/mailman/listinfo/rpm-ecosystem
> >
>
--
Pavel Odvody <podvody at redhat.com>
Software Engineer - EMEA ENG Developer Experience
5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <http://lists.rpm.org/pipermail/rpm-ecosystem/attachments/20150911/2cd2c94e/attachment-0001.asc>
More information about the Rpm-ecosystem
mailing list