[Rpm-maint] How to generate GNU make dependency file for a SPEC file automatically?

Mon May 14 12:15:29 UTC 2018

Hi Jeff,
I was taking a look at the "tar --to-command" option but unfortunately it
seems that even latest TAR versions do not support "cpio" archive type.
I think that hacking directly the rpmbuild utility would be indeed the most
elegant solution to the problem, but I'm not sure how open/difficult is to
provide a patch to a basic utility like rpmbuild (since it's used by
million of people around)!!

Francesco

2018-05-14 9:38 GMT+02:00 Francesco <francesco.montorsi at gmail.com>:

> Hi Jeff,
>
> thanks for the useful pointers. I definitively think that my current
> Python proof-of-concept can be improved, mostly in 2 areas:
> 1) the need of RPM unpacking, as you pointed out;
> 2) I found out it fails miserably to do its job when the RPM contains a
> compiled ELF: the md5sum of the original file copied into the build root
> and the md5sum of the file extracted from RPM CPIO archive are different! I
> guess that rpmbuild is doing some operation on the file before it gets
> archived. Do you have any suggestions on how to disable such behavior from
> rpmbuild?
>
> Anyway I think that an elegant solution to both these 2 problems would be
> to have this (path,md5sum) generation available inside "rpmbuild" directly:
> I think it would be the best place where to put the md5sum computation of a
> packaged file, just before it gets compressed.
> Do you think it would be acceptable a patch that adds a new flag
> --generate-md5sum=<output file> to rpmbuild?
>
> Thanks a lot!
> Francesco
>
>
>
> 2018-05-14 0:34 GMT+02:00 Jeff Johnson <n3npq at me.com>:
>
>>
>>
>> On May 13, 2018, at 2:31 PM, Jeff Johnson <n3npq at me.com> wrote:
>>
>>
>>
>> On May 11, 2018, at 7:31 AM, Francesco <francesco.montorsi at gmail.com>
>> wrote:
>>
>> Hi Jeff, Hi all,
>> I'm resuming this (old) topic in case somebody is interested, I created a
>> very small and simple Python utility
>>
>>   https://github.com/f18m/rpm-make-rules-dependency-lister
>>
>> that does what I was describing in this topic: it allows to connect in a
>> slightly smarter way GNU make and rpmbuild,
>> and to avoid unnecessary RPM re-packaging operations
>>
>> HTH,
>> Francesco
>>
>>
>> Nice!
>>
>> Hmmm ... it looks like you want (path,md5) pairs in GNU md5sum format. Is
>> that correct?
>>
>> If so, You might be able to simplify the code, using a utility like
>>
>> https://www.guyrutenberg.com/2008/10/24/tarsum-calculate-che
>> cksum-for-files-inside-tar-archive/
>>
>> An rpm payload is just a compressed cpio ball.
>>
>> Feed that to something like the above utility to spew the Makefile
>> dependencies.
>>
>> Disclaimer: I haven't looked at the code, nor am I Python programmer. I'm
>> just trying to remove the need to unpack the *.rpm payload.
>>
>> hth
>>
>> 73 de Jeff
>>
>>
>> Here is another approach, using common system utilities:
>>
>> https://stackoverflow.com/questions/16423484/get-checksum-
>> of-files-inside-tar-archive-on-remote-host
>>
>> There is an example there that doesn't need ssh, but you will need to
>>  use rpm2cpio into cpio (modern GNU tar handles cpio?), as well as create
>> the (path,md5) pair using awk/sed/etc.
>>
>> Again: nothing wrong with your script, I'm just trying to avoid the rpm
>> unpacking, as well as to simplify the need for Python as a build
>> prerequisite in the Makefile rules.
>>
>> hth
>>
>> 73 de Jeff
>>
>>
>>
>>
>> 2018-03-27 3:31 GMT+02:00 Jeff Johnson <n3npq at me.com>:
>>
>>>
>>>
>>> On Mar 26, 2018, at 6:15 PM, Francesco <francesco.montorsi at gmail.com>
>>> wrote:
>>>
>>> Hi Jeff,
>>> thanks for your reply.
>>>
>>>
>>>
>>> 2018-03-26 19:04 GMT+02:00 Jeff Johnson <n3npq at me.com>:
>>>
>>>>
>>>> There isn't an explicit tool to convert rpmbuild dependencies to
>>>> Makefile dependencies afaik.
>>>>
>>>> OTOH, it's not impossibly hard to script a couple of missing pieces
>>>> that are needed:
>>>>
>>>> 1) Automate by rule generating a *.src.rpm from a (possibly modified)
>>>> *.spec.
>>>>
>>>> 2) Automate by rule installing a *.src.rpm into a local build
>>>> directory. This requires configuring some rpm macros to map rpmbuild
>>>> inputs/outputs into a single directory.
>>>>
>>>> 3) Automate by rule rebuilding of binary *.rpm from a newer *.spec in
>>>> the local build directory
>>>>
>>>> You can find some useful rules using wild cards and % template rules in
>>>> rpm5 sources: see the
>>>> tests/Makefile.am file for the pattern rules.
>>>>
>>>
>>> Honestly I'm not sure to understand what you mean... when you say
>>> "Automate by rule" you mean writing a GNU make rule, right?
>>>
>>>
>>> Yes.
>>>
>>> Then it's clear to me step 1 and 2 but is not clear why should I get a
>>> newer .spec file in the local build directory only if some of my sources
>>> have changed: if step 1 and 2 are always executed by GNU make (which btw
>>> would be against my goal of running unneeded packaging operations) then I
>>> guess that the mtime of the .spec file in the local build directory will
>>> always change from run to run... what am I missing?
>>>
>>>
>>> Sorry to be obscure. The pattern rules I wrote had 2 *.spec files: the
>>> top level *.spec determined when a *.src.rpm should be made, which
>>> triggered an install not a subdirectories, with a 2nd *.spec (copy) that
>>> triggers a rpmbuild by rule.
>>>
>>> Does that make sense? worksforme, for the purposes I needed, QA testing
>>> of a just built version of rpm within the buil tree ...
>>>
>>> Also I'm particularly interested in binary-only RPMs (I need such a tool
>>> for a commercial software)... to give you an idea, most of my RPM spec
>>> install sections look like:
>>>
>>> %install
>>> make -C ../mysources mytarget_install DESTDIR=%{buildroot}
>>>
>>>
>>> That's pretty generic, yes. Meanwhile the real problem is that rpmbuild
>>> within Makefiles is kinda awkward.
>>>
>>>
>>>
>>>> Alternatively, one could attempt generating a Makefile include rule.
>>>>
>>>> Extract (and filter) rpmbuild dependencies from a spec (or *.src.rpm)
>>>> file. RPM can query a spec file (or *.src.rpm), convert those (filtered)
>>>> dependencies to package names, and convert the package names to file names
>>>> that have a time stamp that can be included into a Makefile.
>>>>
>>>>
>>> Sorry, I'm not sure I get this either... I know RPM allows you to query
>>> dependencies but you can only query the package names listed under
>>> "Requires:" isn't it?
>>>
>>>
>>> Basically yes. But to automate within a Makefile (like dependencies on
>>> *.h)' the build requirements need to be mapped onto a file name that has a
>>> time stamp.
>>>
>>>
>>> Btw I have sketched out a possible "solution" for my problem: first time
>>> "make" is run, I call rpmbuild and build my binary RPM.
>>> Then (automatically via GNU make rules) I unzip that RPM into some
>>> temporary folder, I see what's inside and go search for MD5-sum matching
>>> files in the source build folder. Every time I find inside the source build
>>> folder a file that is inside the RPM, I write that into a .d file (same
>>> name of the RPM spec).
>>> The GNU makefile has an "include $(MY_SPEC_FILE_LIST:.spec=.d)"
>>> statement, so that it's aware of the auto-discovered dependencies of the
>>> spec file.
>>> Next time I run "make", it will be able to understand if the RPM is up
>>> to date or needs to be regenerated because some of the file it packages has
>>> been updated (perhaps as a result of the build process).
>>>
>>>
>>> Post a sample Makefile please. Lots of people have tried to use rpmbuild
>>> in Makefiles, and most of the solutions I have seen are rather awkward.
>>>
>>> Of course this process is not 100% accurate: in the %install section
>>> some temporary file may be generated and copied inside the RPM build root.
>>> Or files (e.g. config files) could be renamed when they get copied inside
>>> the build root. And maybe there are other cases as well.
>>>
>>>
>>> My pattern rules have some *ahem* issues as well, but are good enough
>>> for "make test" automation.
>>>
>>> (aside)
>>> The issues happen while developing: the pattern rules implicitly fire a
>>> (perhaps buggy) rpm. Good enough, but sometimes  a pita.
>>>
>>> However that approach might cover a wide range of use cases... what do
>>> you think?
>>>
>>> It would be nice to have some mechanism like that inside rpmbuild so
>>> that I can do
>>>       rpmbuild -MD myspec.spec --output=myspec.d
>>> and it generates such dependency file for GNU make...
>>>
>>>
>>> There aren't any clean or obvious solutions by rpmbuild design. Oh well
>>> ...
>>>
>>> hth
>>>
>>> 73 de Jeff
>>>
>>> Thanks!
>>>
>>> Francesco
>>>
>>>
>>>
>>>
>> _______________________________________________
>> Rpm-maint mailing list
>> Rpm-maint at lists.rpm.org
>> http://lists.rpm.org/mailman/listinfo/rpm-maint
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20180514/defa8e37/attachment.html>