[Rpm-maint] PATCH: Extend brp-python-bytecompile to support multiple parallel Python stacks

[David Malcolm]

Python's runtime compiles .py source files to bytecode "code objects",
and saves a cache of this data as .pyc and .pyo files on the filesystem.
It looks for .pyc files before parsing a .py file, using the .pyc file
if it can, to avoid having to parse the file.

rpm.org has a script: "rpm/scripts/brp-python-bytecompile".  We invoke
this during postprocessing of an rpm build in Fedora and derived
distributions, so that we can ship pre-parsed bytecode files for .py
files in our package payloads.

The python runtime requires that a 4-byte "magic" ABI value stored in
the header of the pyc file equals that of the python runtime.  (this is
implemented in Python/import.c:check_compiled_module; see also the
importlib/_bootstrap.py implementation within Python 3; similarly a
4-byte mtime value is embedded, which has to match the mtime of the .py
file).

If there's an ABI mismatch, the .pyc file is treated as stale, and
the .py file is parsed.  If this happens for a packaged python module,
the .pyc files aren't writable by non-root, and so python will typically
fail to write out an updated cache (if running as non-root), and will
constantly have to reparse the code, without any visible indication
apart from a big loss of performance.  Additionally, the Python syntax
changed a lot between Python 2 and 3, and picking the wrong interpreter
will often lead to syntax errors when byte-compiling the file.

I want to support deploying multiple python versions via RPM on one
machine.  Hence I want to make sure that when we bytecompile a .py file,
we use the correct python interpreter.

Currently, brp-python-bytecompile can take a single optional argument: a
path to the python interpreter, using /usr/bin/python as the default.

My first thought was to fixup our macros to pass along a "__python"
variable as an argument to the invocation of brp-python-bytecompile
[1].  

However doing so assumes that every .py file within an rpmbuild is to be
processed using the same python runtime.  I don't think this is
sophisticated enough.  I'd like to be able to support having an srpm in
which multiple subpackages are emitted, each for a different python
runtime, all within the same build.  To handle this, some .py files
within RPM_BUILD_ROOT are handled by one runtime, and some by another.

I'm attaching a patch that covers this case.

For every directory of the form /usr/lib/python$VERSION/, the .py files
below it are assumed to be associated with the python interpreter
at /usr/bin/python$VERSION.  For all other .py files, they are
associated with a default python runtime, which is either the argument
to the script, or defaulting to /usr/bin/python (similar to the existing
behavior of the script).

I've also submitted a possible rpmlint test for verifying this in built
RPMs here:
https://www.zarb.org/pipermail/rpmlint-discuss/2009-October/000775.html

Thoughts?
Dave

[1] In Fedora we do this using "__os_install_post", which is defined
in /usr/lib/rpm/redhat/macros (from the redhat-rpm-config package),
which has the invocation of /usr/lib/rpm/brp-python-bytecompile

-------------- next part --------------
A non-text attachment was scrubbed...
Name: extend-brp-python-bytecompile-to-support-multiple-pythons.patch
Type: text/x-patch
Size: 2771 bytes
Desc: not available
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20091026/49727adf/attachment.bin>