[Rpm-maint] PATCH: Extend brp-python-bytecompile to support multiple parallel Python stacks

[David Malcolm]

On Tue, 2009-10-27 at 13:47 +0200, Panu Matilainen wrote:
> On Mon, 26 Oct 2009, David Malcolm wrote:
> 
> > Python's runtime compiles .py source files to bytecode "code objects",
> > and saves a cache of this data as .pyc and .pyo files on the filesystem.
> > It looks for .pyc files before parsing a .py file, using the .pyc file
> > if it can, to avoid having to parse the file.
> >
> > rpm.org has a script: "rpm/scripts/brp-python-bytecompile".  We invoke
> > this during postprocessing of an rpm build in Fedora and derived
> > distributions, so that we can ship pre-parsed bytecode files for .py
> > files in our package payloads.
> >
> > The python runtime requires that a 4-byte "magic" ABI value stored in
> > the header of the pyc file equals that of the python runtime.  (this is
> > implemented in Python/import.c:check_compiled_module; see also the
> > importlib/_bootstrap.py implementation within Python 3; similarly a
> > 4-byte mtime value is embedded, which has to match the mtime of the .py
> > file).
> >
> > If there's an ABI mismatch, the .pyc file is treated as stale, and
> > the .py file is parsed.  If this happens for a packaged python module,
> > the .pyc files aren't writable by non-root, and so python will typically
> > fail to write out an updated cache (if running as non-root), and will
> > constantly have to reparse the code, without any visible indication
> > apart from a big loss of performance.  Additionally, the Python syntax
> > changed a lot between Python 2 and 3, and picking the wrong interpreter
> > will often lead to syntax errors when byte-compiling the file.
> >
> > I want to support deploying multiple python versions via RPM on one
> > machine.  Hence I want to make sure that when we bytecompile a .py file,
> > we use the correct python interpreter.
> >
> > Currently, brp-python-bytecompile can take a single optional argument: a
> > path to the python interpreter, using /usr/bin/python as the default.
> >
> > My first thought was to fixup our macros to pass along a "__python"
> > variable as an argument to the invocation of brp-python-bytecompile
> > [1].
> >
> > However doing so assumes that every .py file within an rpmbuild is to be
> > processed using the same python runtime.  I don't think this is
> > sophisticated enough.  I'd like to be able to support having an srpm in
> > which multiple subpackages are emitted, each for a different python
> > runtime, all within the same build.  To handle this, some .py files
> > within RPM_BUILD_ROOT are handled by one runtime, and some by another.
> >
> > I'm attaching a patch that covers this case.
> >
> > For every directory of the form /usr/lib/python$VERSION/, the .py files
> > below it are assumed to be associated with the python interpreter
> > at /usr/bin/python$VERSION.  For all other .py files, they are
> > associated with a default python runtime, which is either the argument
> > to the script, or defaulting to /usr/bin/python (similar to the existing
> > behavior of the script).
> >
> > I've also submitted a possible rpmlint test for verifying this in built
> > RPMs here:
> > https://www.zarb.org/pipermail/rpmlint-discuss/2009-October/000775.html
> >
> > Thoughts?
> 
> Seems reasonable I guess, but there's at least one issue with the 
> patch: it doesn't work correctly on x86_64 and the like due to hardcoded 
> /usr/lib/python* paths. Both lib and lib64 path variants need to be 
> handled by anything dealing with python library path.

Thanks.

Updated patch attached; it assumes that .py files below both
  - /usr/lib/python$VERSION  and
  - /usr/lib64/python/python$VERSION
can be handled by
  - /usr/bin/python$VERSION


-------------- next part --------------
A non-text attachment was scrubbed...
Name: extend-brp-python-bytecompile-to-support-multiple-pythons-v2.patch
Type: text/x-patch
Size: 2776 bytes
Desc: not available
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20091027/f6a61b1e/attachment.bin>