[Rpm-maint] How to generate GNU make dependency file for a SPEC file automatically?

Mon May 14 07:38:38 UTC 2018

Hi Jeff,

thanks for the useful pointers. I definitively think that my current Python
proof-of-concept can be improved, mostly in 2 areas:
1) the need of RPM unpacking, as you pointed out;
2) I found out it fails miserably to do its job when the RPM contains a
compiled ELF: the md5sum of the original file copied into the build root
and the md5sum of the file extracted from RPM CPIO archive are different! I
guess that rpmbuild is doing some operation on the file before it gets
archived. Do you have any suggestions on how to disable such behavior from
rpmbuild?

Anyway I think that an elegant solution to both these 2 problems would be
to have this (path,md5sum) generation available inside "rpmbuild" directly:
I think it would be the best place where to put the md5sum computation of a
packaged file, just before it gets compressed.
Do you think it would be acceptable a patch that adds a new flag
--generate-md5sum=<output file> to rpmbuild?

Thanks a lot!
Francesco

2018-05-14 0:34 GMT+02:00 Jeff Johnson <n3npq at me.com>:

>
>
> On May 13, 2018, at 2:31 PM, Jeff Johnson <n3npq at me.com> wrote:
>
>
>
> On May 11, 2018, at 7:31 AM, Francesco <francesco.montorsi at gmail.com>
> wrote:
>
> Hi Jeff, Hi all,
> I'm resuming this (old) topic in case somebody is interested, I created a
> very small and simple Python utility
>
>   https://github.com/f18m/rpm-make-rules-dependency-lister
>
> that does what I was describing in this topic: it allows to connect in a
> slightly smarter way GNU make and rpmbuild,
> and to avoid unnecessary RPM re-packaging operations
>
> HTH,
> Francesco
>
>
> Nice!
>
> Hmmm ... it looks like you want (path,md5) pairs in GNU md5sum format. Is
> that correct?
>
> If so, You might be able to simplify the code, using a utility like
>
> https://www.guyrutenberg.com/2008/10/24/tarsum-calculate-
> checksum-for-files-inside-tar-archive/
>
> An rpm payload is just a compressed cpio ball.
>
> Feed that to something like the above utility to spew the Makefile
> dependencies.
>
> Disclaimer: I haven't looked at the code, nor am I Python programmer. I'm
> just trying to remove the need to unpack the *.rpm payload.
>
> hth
>
> 73 de Jeff
>
>
> Here is another approach, using common system utilities:
>
> https://stackoverflow.com/questions/16423484/get-
> checksum-of-files-inside-tar-archive-on-remote-host
>
> There is an example there that doesn't need ssh, but you will need to  use
> rpm2cpio into cpio (modern GNU tar handles cpio?), as well as create the
> (path,md5) pair using awk/sed/etc.
>
> Again: nothing wrong with your script, I'm just trying to avoid the rpm
> unpacking, as well as to simplify the need for Python as a build
> prerequisite in the Makefile rules.
>
> hth
>
> 73 de Jeff
>
>
>
>
> 2018-03-27 3:31 GMT+02:00 Jeff Johnson <n3npq at me.com>:
>
>>
>>
>> On Mar 26, 2018, at 6:15 PM, Francesco <francesco.montorsi at gmail.com>
>> wrote:
>>
>> Hi Jeff,
>> thanks for your reply.
>>
>>
>>
>> 2018-03-26 19:04 GMT+02:00 Jeff Johnson <n3npq at me.com>:
>>
>>>
>>> There isn't an explicit tool to convert rpmbuild dependencies to
>>> Makefile dependencies afaik.
>>>
>>> OTOH, it's not impossibly hard to script a couple of missing pieces that
>>> are needed:
>>>
>>> 1) Automate by rule generating a *.src.rpm from a (possibly modified)
>>> *.spec.
>>>
>>> 2) Automate by rule installing a *.src.rpm into a local build directory.
>>> This requires configuring some rpm macros to map rpmbuild inputs/outputs
>>> into a single directory.
>>>
>>> 3) Automate by rule rebuilding of binary *.rpm from a newer *.spec in
>>> the local build directory
>>>
>>> You can find some useful rules using wild cards and % template rules in
>>> rpm5 sources: see the
>>> tests/Makefile.am file for the pattern rules.
>>>
>>
>> Honestly I'm not sure to understand what you mean... when you say
>> "Automate by rule" you mean writing a GNU make rule, right?
>>
>>
>> Yes.
>>
>> Then it's clear to me step 1 and 2 but is not clear why should I get a
>> newer .spec file in the local build directory only if some of my sources
>> have changed: if step 1 and 2 are always executed by GNU make (which btw
>> would be against my goal of running unneeded packaging operations) then I
>> guess that the mtime of the .spec file in the local build directory will
>> always change from run to run... what am I missing?
>>
>>
>> Sorry to be obscure. The pattern rules I wrote had 2 *.spec files: the
>> top level *.spec determined when a *.src.rpm should be made, which
>> triggered an install not a subdirectories, with a 2nd *.spec (copy) that
>> triggers a rpmbuild by rule.
>>
>> Does that make sense? worksforme, for the purposes I needed, QA testing
>> of a just built version of rpm within the buil tree ...
>>
>> Also I'm particularly interested in binary-only RPMs (I need such a tool
>> for a commercial software)... to give you an idea, most of my RPM spec
>> install sections look like:
>>
>> %install
>> make -C ../mysources mytarget_install DESTDIR=%{buildroot}
>>
>>
>> That's pretty generic, yes. Meanwhile the real problem is that rpmbuild
>> within Makefiles is kinda awkward.
>>
>>
>>
>>> Alternatively, one could attempt generating a Makefile include rule.
>>>
>>> Extract (and filter) rpmbuild dependencies from a spec (or *.src.rpm)
>>> file. RPM can query a spec file (or *.src.rpm), convert those (filtered)
>>> dependencies to package names, and convert the package names to file names
>>> that have a time stamp that can be included into a Makefile.
>>>
>>>
>> Sorry, I'm not sure I get this either... I know RPM allows you to query
>> dependencies but you can only query the package names listed under
>> "Requires:" isn't it?
>>
>>
>> Basically yes. But to automate within a Makefile (like dependencies on
>> *.h)' the build requirements need to be mapped onto a file name that has a
>> time stamp.
>>
>>
>> Btw I have sketched out a possible "solution" for my problem: first time
>> "make" is run, I call rpmbuild and build my binary RPM.
>> Then (automatically via GNU make rules) I unzip that RPM into some
>> temporary folder, I see what's inside and go search for MD5-sum matching
>> files in the source build folder. Every time I find inside the source build
>> folder a file that is inside the RPM, I write that into a .d file (same
>> name of the RPM spec).
>> The GNU makefile has an "include $(MY_SPEC_FILE_LIST:.spec=.d)"
>> statement, so that it's aware of the auto-discovered dependencies of the
>> spec file.
>> Next time I run "make", it will be able to understand if the RPM is up to
>> date or needs to be regenerated because some of the file it packages has
>> been updated (perhaps as a result of the build process).
>>
>>
>> Post a sample Makefile please. Lots of people have tried to use rpmbuild
>> in Makefiles, and most of the solutions I have seen are rather awkward.
>>
>> Of course this process is not 100% accurate: in the %install section some
>> temporary file may be generated and copied inside the RPM build root. Or
>> files (e.g. config files) could be renamed when they get copied inside the
>> build root. And maybe there are other cases as well.
>>
>>
>> My pattern rules have some *ahem* issues as well, but are good enough for
>> "make test" automation.
>>
>> (aside)
>> The issues happen while developing: the pattern rules implicitly fire a
>> (perhaps buggy) rpm. Good enough, but sometimes  a pita.
>>
>> However that approach might cover a wide range of use cases... what do
>> you think?
>>
>> It would be nice to have some mechanism like that inside rpmbuild so that
>> I can do
>>       rpmbuild -MD myspec.spec --output=myspec.d
>> and it generates such dependency file for GNU make...
>>
>>
>> There aren't any clean or obvious solutions by rpmbuild design. Oh well
>> ...
>>
>> hth
>>
>> 73 de Jeff
>>
>> Thanks!
>>
>> Francesco
>>
>>
>>
>>
> _______________________________________________
> Rpm-maint mailing list
> Rpm-maint at lists.rpm.org
> http://lists.rpm.org/mailman/listinfo/rpm-maint
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20180514/74c18b43/attachment.html>