[Rpm-maint] PATCH: Extend brp-python-bytecompile to support multiple parallel Python stacks

[Panu Matilainen]

On Mon, 26 Oct 2009, David Malcolm wrote:

> Python's runtime compiles .py source files to bytecode "code objects",
> and saves a cache of this data as .pyc and .pyo files on the filesystem.
> It looks for .pyc files before parsing a .py file, using the .pyc file
> if it can, to avoid having to parse the file.
>
> rpm.org has a script: "rpm/scripts/brp-python-bytecompile".  We invoke
> this during postprocessing of an rpm build in Fedora and derived
> distributions, so that we can ship pre-parsed bytecode files for .py
> files in our package payloads.
>
> The python runtime requires that a 4-byte "magic" ABI value stored in
> the header of the pyc file equals that of the python runtime.  (this is
> implemented in Python/import.c:check_compiled_module; see also the
> importlib/_bootstrap.py implementation within Python 3; similarly a
> 4-byte mtime value is embedded, which has to match the mtime of the .py
> file).
>
> If there's an ABI mismatch, the .pyc file is treated as stale, and
> the .py file is parsed.  If this happens for a packaged python module,
> the .pyc files aren't writable by non-root, and so python will typically
> fail to write out an updated cache (if running as non-root), and will
> constantly have to reparse the code, without any visible indication
> apart from a big loss of performance.  Additionally, the Python syntax
> changed a lot between Python 2 and 3, and picking the wrong interpreter
> will often lead to syntax errors when byte-compiling the file.
>
> I want to support deploying multiple python versions via RPM on one
> machine.  Hence I want to make sure that when we bytecompile a .py file,
> we use the correct python interpreter.
>
> Currently, brp-python-bytecompile can take a single optional argument: a
> path to the python interpreter, using /usr/bin/python as the default.
>
> My first thought was to fixup our macros to pass along a "__python"
> variable as an argument to the invocation of brp-python-bytecompile
> [1].
>
> However doing so assumes that every .py file within an rpmbuild is to be
> processed using the same python runtime.  I don't think this is
> sophisticated enough.  I'd like to be able to support having an srpm in
> which multiple subpackages are emitted, each for a different python
> runtime, all within the same build.  To handle this, some .py files
> within RPM_BUILD_ROOT are handled by one runtime, and some by another.
>
> I'm attaching a patch that covers this case.
>
> For every directory of the form /usr/lib/python$VERSION/, the .py files
> below it are assumed to be associated with the python interpreter
> at /usr/bin/python$VERSION.  For all other .py files, they are
> associated with a default python runtime, which is either the argument
> to the script, or defaulting to /usr/bin/python (similar to the existing
> behavior of the script).
>
> I've also submitted a possible rpmlint test for verifying this in built
> RPMs here:
> https://www.zarb.org/pipermail/rpmlint-discuss/2009-October/000775.html
>
> Thoughts?

Seems reasonable I guess, but there's at least one issue with the 
patch: it doesn't work correctly on x86_64 and the like due to hardcoded 
/usr/lib/python* paths. Both lib and lib64 path variants need to be 
handled by anything dealing with python library path.

 	- Panu -