From fweimer at redhat.com Wed Dec 21 10:17:20 2022 From: fweimer at redhat.com (Florian Weimer) Date: Wed, 21 Dec 2022 11:17:20 +0100 Subject: Memoizing macro values Message-ID: <87y1r12h8v.fsf@oldenburg.str.redhat.com> In Fedora, we have this: /usr/lib/rpm/macros.d/macros.python:%python_sitelib %(RPM_BUILD_ROOT= %{__python} -Esc "import sysconfig; print(sysconfig.get_path('purelib', vars={'platbase': '%{_prefix}', 'base': '%{_prefix}'}))") /usr/lib/rpm/macros.d/macros.python3:%python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; print(sysconfig.get_path('purelib', vars={'platbase': '%{_prefix}', 'base': '%{_prefix}'}))") This is a %define-style macro, so the macro gets re-evaluated for every expansion. It can make spec file operations really, really slow. It's possible to work around this in spec files like this: %{?python3_sitearch: %global python3_sitearch %{python3_sitearch}} But this is quite ugly, and I wonder if there is a nice way to do it directly in the macros file instead. I think I know how do it with Lua, but maybe there is a better way? Thanks, Florian From mhroncok at redhat.com Wed Dec 21 10:40:15 2022 From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=) Date: Wed, 21 Dec 2022 11:40:15 +0100 Subject: Memoizing macro values In-Reply-To: <87y1r12h8v.fsf@oldenburg.str.redhat.com> References: <87y1r12h8v.fsf@oldenburg.str.redhat.com> Message-ID: On 21. 12. 22 11:17, Florian Weimer wrote: > In Fedora, we have this: > > /usr/lib/rpm/macros.d/macros.python:%python_sitelib %(RPM_BUILD_ROOT= %{__python} -Esc "import sysconfig; print(sysconfig.get_path('purelib', vars={'platbase': '%{_prefix}', 'base': '%{_prefix}'}))") > /usr/lib/rpm/macros.d/macros.python3:%python3_sitelib %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; print(sysconfig.get_path('purelib', vars={'platbase': '%{_prefix}', 'base': '%{_prefix}'}))") > > This is a %define-style macro, so the macro gets re-evaluated for every > expansion. It can make spec file operations really, really slow. > > It's possible to work around this in spec files like this: > > %{?python3_sitearch: %global python3_sitearch %{python3_sitearch}} > > But this is quite ugly, and I wonder if there is a nice way to do it > directly in the macros file instead. I think I know how do it with Lua, > but maybe there is a better way? This works (using random to demonstarte the behavior: %_python3_random %(RPM_BUILD_ROOT= %{__python3} -Ic "import random; print(random.randint(0,100))") %python3_random %{?!_python3_random_cached:%{global _python3_random_cached %_python3_random}}%_python3_random_cached $ rpm --eval '%python3_random %python3_random %python3_random %python3_random %python3_random' 42 42 42 42 42 Note that %python3_sitlib must be evaluated after Python is installed so maybe there must be some "non-empty" conditional added as well. -- Miro Hron?ok -- Phone: +420777974800 IRC: mhroncok From mhroncok at redhat.com Wed Dec 21 11:00:50 2022 From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=) Date: Wed, 21 Dec 2022 12:00:50 +0100 Subject: Memoizing macro values In-Reply-To: References: <87y1r12h8v.fsf@oldenburg.str.redhat.com> Message-ID: On 21. 12. 22 11:40, Miro Hron?ok wrote: > On 21. 12. 22 11:17, Florian Weimer wrote: >> In Fedora, we have this: >> >> /usr/lib/rpm/macros.d/macros.python:%python_sitelib %(RPM_BUILD_ROOT= >> %{__python} -Esc "import sysconfig; print(sysconfig.get_path('purelib', >> vars={'platbase': '%{_prefix}', 'base': '%{_prefix}'}))") >> /usr/lib/rpm/macros.d/macros.python3:%python3_sitelib %(RPM_BUILD_ROOT= >> %{__python3} -Ic "import sysconfig; print(sysconfig.get_path('purelib', >> vars={'platbase': '%{_prefix}', 'base': '%{_prefix}'}))") >> >> This is a %define-style macro, so the macro gets re-evaluated for every >> expansion.? It can make spec file operations really, really slow. >> >> It's possible to work around this in spec files like this: >> >> %{?python3_sitearch: %global python3_sitearch %{python3_sitearch}} >> >> But this is quite ugly, and I wonder if there is a nice way to do it >> directly in the macros file instead.? I think I know how do it with Lua, >> but maybe there is a better way? > > This works (using random to demonstarte the behavior: > > %_python3_random %(RPM_BUILD_ROOT= %{__python3} -Ic "import random; > print(random.randint(0,100))") > %python3_random %{?!_python3_random_cached:%{global _python3_random_cached > %_python3_random}}%_python3_random_cached > > $ rpm --eval '%python3_random %python3_random %python3_random %python3_random > %python3_random' > 42 42 42 42 42 > > Note that %python3_sitlib must be evaluated after Python is installed so maybe > there must be some "non-empty" conditional added as well. This should do: %_python3_random %(RPM_BUILD_ROOT= %{__python3} -Ic "import random; print(i if (i := random.randint(0,100)) > 75 else '')") %python3_random %["%?_python3_random_cached" == "" ? "%{global _python3_random_cached %_python3_random}" : ""]%_python3_random_cached $ rpm --eval '%python3_random %python3_random %python3_random %python3_random %python3_random' 87 87 -- Miro Hron?ok -- Phone: +420777974800 IRC: mhroncok From fweimer at redhat.com Wed Dec 21 11:10:33 2022 From: fweimer at redhat.com (Florian Weimer) Date: Wed, 21 Dec 2022 12:10:33 +0100 Subject: Memoizing macro values In-Reply-To: ("Miro =?utf-8?Q?Hron=C4=8Dok=22's?= message of "Wed, 21 Dec 2022 12:00:50 +0100") References: <87y1r12h8v.fsf@oldenburg.str.redhat.com> Message-ID: <87tu1p2es6.fsf@oldenburg.str.redhat.com> * Miro Hron?ok: > On 21. 12. 22 11:40, Miro Hron?ok wrote: >> On 21. 12. 22 11:17, Florian Weimer wrote: >>> In Fedora, we have this: >>> >>> /usr/lib/rpm/macros.d/macros.python:%python_sitelib >>> %(RPM_BUILD_ROOT= %{__python} -Esc "import sysconfig; >>> print(sysconfig.get_path('purelib', vars={'platbase': '%{_prefix}', >>> 'base': '%{_prefix}'}))") >>> /usr/lib/rpm/macros.d/macros.python3:%python3_sitelib >>> %(RPM_BUILD_ROOT= %{__python3} -Ic "import sysconfig; >>> print(sysconfig.get_path('purelib', vars={'platbase': '%{_prefix}', >>> 'base': '%{_prefix}'}))") >>> >>> This is a %define-style macro, so the macro gets re-evaluated for every >>> expansion.? It can make spec file operations really, really slow. >>> >>> It's possible to work around this in spec files like this: >>> >>> %{?python3_sitearch: %global python3_sitearch %{python3_sitearch}} >>> >>> But this is quite ugly, and I wonder if there is a nice way to do it >>> directly in the macros file instead.? I think I know how do it with Lua, >>> but maybe there is a better way? >> This works (using random to demonstarte the behavior: >> %_python3_random %(RPM_BUILD_ROOT= %{__python3} -Ic "import random; >> print(random.randint(0,100))") >> %python3_random %{?!_python3_random_cached:%{global >> _python3_random_cached %_python3_random}}%_python3_random_cached >> $ rpm --eval '%python3_random %python3_random %python3_random >> %python3_random %python3_random' >> 42 42 42 42 42 >> Note that %python3_sitlib must be evaluated after Python is >> installed so maybe there must be some "non-empty" conditional added >> as well. Yes, that's similar to what I'd have done with Lua. > This should do: > > %_python3_random %(RPM_BUILD_ROOT= %{__python3} -Ic "import random; > print(i if (i := random.randint(0,100)) > 75 else '')") > %python3_random %["%?_python3_random_cached" == "" ? "%{global > _python3_random_cached %_python3_random}" : > ""]%_python3_random_cached > > $ rpm --eval '%python3_random %python3_random %python3_random > %python3_random %python3_random' > 87 87 Could you bring this to Fedora (rawhide in 37 preferably)? I think it will really help package maintainers. I did apply the workaround to samba because it took a minute or so to process the spec file (they follow the packaging guidelines to the letter and have hundreds of lines with %python3_sitearch), but it would be nice to implement this centrally. Another package that benefits is python-pynn, and there I don't quite see the source of the slowness. Thanks, Florian From mhroncok at redhat.com Wed Dec 21 11:21:30 2022 From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=) Date: Wed, 21 Dec 2022 12:21:30 +0100 Subject: Memoizing macro values In-Reply-To: <87tu1p2es6.fsf@oldenburg.str.redhat.com> References: <87y1r12h8v.fsf@oldenburg.str.redhat.com> <87tu1p2es6.fsf@oldenburg.str.redhat.com> Message-ID: <1149c171-6812-3b38-6d38-c9a6350cac97@redhat.com> On 21. 12. 22 12:10, Florian Weimer wrote: > Could you bring this to Fedora (rawhide in 37 preferably)? Rawhide for sure, but there is another change in progress now and I'd like to ship that first. I have this on my TODO list and will eventually get to this. -- Miro Hron?ok -- Phone: +420777974800 IRC: mhroncok From mhroncok at redhat.com Wed Dec 21 11:30:35 2022 From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=) Date: Wed, 21 Dec 2022 12:30:35 +0100 Subject: Memoizing macro values In-Reply-To: <1149c171-6812-3b38-6d38-c9a6350cac97@redhat.com> References: <87y1r12h8v.fsf@oldenburg.str.redhat.com> <87tu1p2es6.fsf@oldenburg.str.redhat.com> <1149c171-6812-3b38-6d38-c9a6350cac97@redhat.com> Message-ID: <5488c01c-df6e-7994-2b99-667b2b2368fb@redhat.com> On 21. 12. 22 12:21, Miro Hron?ok wrote: > On 21. 12. 22 12:10, Florian Weimer wrote: >> Could you bring this to Fedora (rawhide in 37 preferably)? > > Rawhide for sure, but there is another change in progress now and I'd like to > ship that first. I have this on my TODO list and will eventually get to this. Fedora RFE: https://bugzilla.redhat.com/2155505 -- Miro Hron?ok -- Phone: +420777974800 IRC: mhroncok