[Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)

nim-nim notifications at github.com
Fri May 15 12:38:27 UTC 2020


@ffesti Thank you for sharing a different analysis and point of view. I’ll correct some things here (I don’t fundamentally disagree with what you wrote, but you made some shortcuts that would block a real-world design)

> The current font and go macros are a pain to implement but - obviously - don't require something that can't be done. 

Actually this is slowly getting to the point I’ve written enough helpers for common needs in redhat-rpm-config that implementing a new macro set is easy. I will make a new dump of common helpers after the `%new_package` part is processed. That will get us to the point where a macro implementor can write things like:
(rpm macro side)
```rpm
# Run tests in the check section for a font (sub)package. Arguments:
# -z <number>         read the zth block of definitions, for example
#                     %{fontfamily<number>}
# -v                  be verbose
%fontcheck(z:v) %{lua:
local      fedora =  require "fedora.common"
local       fonts =  require "fedora.srpm.fonts"
local   fonts-rpm =  require "fedora.rpm.fonts"
local      suffix =  rpm.expand("%{?-z*}")
local     verbose = (rpm.expand("%{-v}") ~= "")
fedora.suffixloop(fonts-rpm.check, suffix, fonts.suffixes(), {verbose})
}
```
(and lua side)

```lua
-- Core of %fontcheck
local function check(suffix, verbose)
  fonts.env(suffix, verbose)
  print(rpm.expand([[
grep -E '^"%{_fontconfig_templatedir}/.+\.conf"' '%{currentfontfiles}' \
  | xargs -I{} -- sh -c "xmllint --loaddtd --valid     --nonet '%{buildroot}{}' \
  >/dev/null && echo %{buildroot}{}: OK"
grep -E '^"%{_datadir}/metainfo/.+\.xml"'        '%{currentfontfiles}' \
  | xargs -I{} --        appstream-util validate-relax --nonet '%{buildroot}{}'
]]))
end
```

and don’t worry at all about the heavy lifting done by the helpers to make that just work in presence of multiple subpackages. All the ugliness here is pure domain-specific code, the rpm-induced templating ugliness is hidden from the macro writer.

> But they rely on all the actual data being punched into the spec file by the user. 

If that was the case, they would need much longer specs packager side. A huge part of the complexity in the forge, go and fonts macros is computing domain-specific sane defaults from partial packager information (that’s why `%forgemeta`, `%gometa` and `%fontmeta` are complex. They fill in the blanks using complex domain-specific rules so the rest of the macro code and the packager in its spec do not have to worry if info X was filled or not.)

I don’t see this blank filling need going away. Even assuming upstream provided perfect metadata that does not need correction or overriding Fedora-side (and, we all know upstreams are not perfect), there will always be additional metadata that Fedora requires, but a domain-specific component system forgot to handle. The legal (licensing) aspect was already given in example. That’s not the only one.

Now, it is true that currently rpm constrains things in such a way, it is not possible to feed upstream info to this blank filling process, even when it is present as upstream metadata in the source archives. 

> Using data from the buildroot is currently not possible at all - with the exception of globs in file lists and dependency generators. To work around this we need to be able to create (sub) packages after the build.

Good automatic package generation would require moving the evaluation point of things used to construct headers (and sources!) at least after `%prep`, the same way dynamic build requires had to move to a section that follows `%prep`. Not sure if it needs to go as far as after `%build`.

In a fully automated mode,

1. the preamble is optional

2. a first logic pass computes upstream sources in `%sourcelist` (since everything is moving to git nowadays, I have more specs that use the `%forge` macros now than specs that do not)

3. then you have explicit Fedora patches in `%patchlist`

4. starting from `%prep`, you need domain-specific processing, either via packager-specified explicit `%fooprep` calls, or via some automated detection process.

    For the reasons I exposed before, I prefer explicit calls till we understand the ordering requirements better. At this point you’re already in domain-specific generation logic but you may not know the components that will be created by this process yet.

   Thus `%prep`, `%generate_buildrequires` ` %build` `%install` using domain-specific logic, mixing the logic blocks if necessary.

5. After the `install`, however, you need ventilate the installed files between subpackages, which means you can not avoid defining the corresponding package headers any longer. 

6. That means you may not know the final install package list, naming and versionning before a section between `%install` and `%files`.

7. And, you also may not know the final srpm name and versioning before this section. Because if you want to keep sane, the general case will be to name your srpm after the most critical of the generated install packages, unless the packager deliberately demands to use a srpm-specific name. And, if the packager does not demand a  srpm-specific name, and forgets to tell which of the generated subpackages is the important one, the only sane default is to take the first of them as most important.

  That means you use a temporary srpm filename (probably just the spec name, changing the file extension) till the build progresses to this point.

8. As explained, as the domain logic progresses, it can result in computing additional fedora-provided source files. And, those need to be present before the section that will use them, and after the section that computed them. So, lots of optional `%sourcelists` between `%prep`, `%generate_buildrequires` ` %build` `%install` (some would add `%check` here)

So, all of this is doable, adding more build phases and sections, and we proved with `%generate_buildrequires` that could work great as long as rpm and mock people worked together.

However that will require careful communication, since removing the roadblocks this way, also removes the side-effects of those roadblocks, that existed for so long, that people have started to take as the natural and eternal rpm state.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/329#issuecomment-629212094
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20200515/9dba9fc5/attachment.html>


More information about the Rpm-maint mailing list