[Rpm-maint] [rpm-software-management/rpm] [JFYI] Introduction of "rpms.lock.yaml" file (Discussion #2908)

[Ondrej Nosek]

# [JFYI] Introduction of _rpms.lock.yaml_ file


## Context

My team is currently working on the implementation of a hermetic build process for containers that use RPMs. The build process runs in a network-isolated build environment. To be able to implement this, we need to pre-fetch all required RPMs and a full chain of their transitive dependencies to be available during the build process (except for packages that are already installed in the parent container image). As part of this requirement, we also want to strive towards reproducibility. To prefetch all required RPMs, including dependencies, and to be able to pre-fetch the same set of them when we re-run the build with the same input parameters, we need a "lock" file similar to one known from Python - _requirements.txt_ that is programmatically generated from an input file called _requirements.in_.

To be transparent and to give you a chance to provide feedback as RPM ecosystem SMEs I would like to present to you the format of the lock file we designed.

For more details about our requirements for the container build process, you can see SLSA requirements, especially these:

* [https://slsa.dev/spec/v0.1/requirements#hermetic](https://slsa.dev/spec/v0.1/requirements#hermetic)
* [https://slsa.dev/spec/v0.1/requirements#reproducible](https://slsa.dev/spec/v0.1/requirements#reproducible)
* [https://slsa.dev/spec/v0.1/requirements#dependencies-complete](https://slsa.dev/spec/v0.1/requirements#dependencies-complete)


## rpms.lock.yaml

A file that contains a list of fully resolved dependencies (their URLs) that cachi2 ([https://github.com/containerbuildsystem/cachi2/](https://github.com/containerbuildsystem/cachi2/)) will need to download for a hermetic build. This file contains a different list of RPMs per architecture. Only the RPMs listed in this file will be available during the build process as the build process has no access to the internet.

Note: This file contains only RPMs that will be installed on top of the parent image - i.e. RPMs that are required but are already installed in the parent container image are not included in this file.

This will be generated and maintained programmatically based on an input file (_rpms.in.yaml_) that is out of the scope of this doc.


### 📔 Notes about format design


* YAML format is extensible, allows to write schema to allow simple validation, is platform independent, and easily consumable by computers as well as humans.
* “repoid” is optional. If defined it will be propagated to the container build so RPMs that will be installed during the container build process will have this repoid listed as an origin repo ("From repo" when you run `dnf info $PKG`). This is beneficial for example for a container vulnerability scanning tool Clair.
* “sources” are optional. If provided, they will be collected by the cachi2 during the container build process to allow generation of source containers.


### ⚙️ Example

```
lockfileVersion: 1
arches:
  - arch: x86_64
    packages:
      - url: https://mirrors.nic.cz/pub/fedora/linux/updates/38/Everything/x86_64/Packages/v/vim-enhanced-9.1.031-1.fc38.x86_64.rpm
        repoid: updates
      - url: https://mirrors.nic.cz/pub/fedora/linux/updates/38/Everything/x86_64/Packages/v/vim-common-9.1.031-1.fc38.x86_64.rpm
        repoid: updates
      - url: https://mirrors.nic.cz/pub/fedora/linux/updates/38/Everything/x86_64/Packages/v/vim-filesystem-9.1.031-1.fc38.noarch.rpm
        repoid: updates
      - url: https://mirrors.nic.cz/pub/fedora/linux/releases/38/Everything/x86_64/os/Packages/g/gpm-libs-1.20.7-42.fc38.x86_64.rpm
        repoid: fedora
    sources:
      - url: https://mirrors.nic.cz/pub/fedora/linux/updates/38/Everything/source/tree/Packages/v/vim-9.1.031-1.fc38.src.rpm
        repoid: updates-source
      - url: https://mirrors.nic.cz/pub/fedora/linux/releases/38/Everything/source/tree/Packages/g/gpm-1.20.7-42.fc38.src.rpm
        repoid: fedora-source
  - arch: aarch64
    packages:
... SNIP ...
```


We understand that managing such a lock file manually is going to be very cumbersome and difficult. The long-term plan is to have a tool that will be able to automate it. This however is not within the scope of our minimal viable product.


## Possible future extensions

These are some possible extensions that we can envision may become relevant at some point in future and can be easily added because of the YAML format, but they are not planned right now as our use case doesn't need them.



* Support for metalink and mirrorlists.
* Support for modules.
* Checksums of the files.
* Mapping of RPMs to the source RPMs.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2908
You are receiving this because you are subscribed to this thread.

Message ID: <rpm-software-management/rpm/repo-discussions/2908 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20240214/49876063/attachment.html>