[Rpm-maint] [rpm-software-management/rpm] Backport multi-threaded zstd to 4.14.x to support multi-threaded zstd compression on centos 8 (PR #2130)

[Alexander]

Summary:
zstd supports multi-threaded compression and is already available in rpm 4.17.
Unfortunately, this feature is not available on centos 8, that has 4.14.x only.
Adding this feature requires upgrading dependency on zstd lib and backporting the intergration.
 

Changes:
- Backport of https://github.com/rpm-software-management/rpm/pull/1303/files
- Couldn't backport [autodetection for the number of threads](https://github.com/rpm-software-management/rpm/pull/1324) as the `getncpus` macro is not supported in 4.14.x. I replaced it with `sysconf(_SC_NPROCESSORS_ONLN)` that is [already used](https://github.com/rpm-software-management/rpm/blob/rpm-4.14.x/rpmio/rpmio.c#L799) in `xz`.
- 4.14.x will require libzstd version >= 1.3.8.


Testing:
1. Created a test rpmbuild project https://gist.github.com/angry-cellophane/cfa31454f2376388079c74af7d459aa0
2. Built rpm with my changes and zstd enabled - `./configure && make`
3. Ran rpmbuild with different payloads, unpacked the rpm after each run with rpm2cpio and checked the rpm's size.

w19T0.zstdio
```
❱❱❱ time ./rpmbuild --define '_source_payload w19T0.zstdio'  -bs -v ~/rpmbuild/SPECS/test.spec
Wrote: /home/****/rpmbuild/SRPMS/test-0.0.1-1.src.rpm
./rpmbuild --define '_source_payload w19T0.zstdio' -bs -v   277.81s user 1.49s system 803% cpu 34.775 total

❱❱❱ du -sh SRPMS/*
68M	SRPMS/test-0.0.1-1.src.rpm

❱❱❱ rpm2cpio SRPMS/test-0.0.1-1.src.rpm | cpio -idmv
binary_412MB
test.spec
text_file
843347 blocks
```


w19.zstdio
```
[❱❱❱ time ./rpmbuild --define '_source_payload w19.zstdio'  -bs -v ~/rpmbuild/SPECS/test.spec
Wrote: /home/****/rpmbuild/SRPMS/test-0.0.1-1.src.rpm
./rpmbuild --define '_source_payload w19.zstdio' -bs -v   240.95s user 0.55s system 99% cpu 4:01.54 total

❱❱❱ du -sh SRPMS/*
68M	SRPMS/test-0.0.1-1.src.rpm
[14:14:32] [cost 0.063s] du -sh SRPMS/*

❱❱❱ rpm2cpio SRPMS/test-0.0.1-1.src.rpm | cpio -idmv
binary_412MB
test.spec
text_file
843347 blocks
```


w19T1.zstdio
```
❱❱❱ time ./rpmbuild --define '_source_payload w19T1.zstdio'  -bs -v ~/rpmbuild/SPECS/test.spec
Wrote: /home/****/rpmbuild/SRPMS/test-0.0.1-1.src.rpm
./rpmbuild --define '_source_payload w19T1.zstdio' -bs -v   246.21s user 0.51s system 100% cpu 4:06.45 total

❱❱❱ du -sh SRPMS/*
68M	SRPMS/test-0.0.1-1.src.rpm

❱❱❱ rpm2cpio SRPMS/test-0.0.1-1.src.rpm | cpio -idmv
binary_412MB
test.spec
text_file
843347 blocks
```


w19T4.zstdio
```
❱❱❱ time ./rpmbuild --define '_source_payload w19T4.zstdio'  -bs -v ~/rpmbuild/SPECS/test.spec
Wrote: /home/****/rpmbuild/SRPMS/test-0.0.1-1.src.rpm
./rpmbuild --define '_source_payload w19T4.zstdio' -bs -v   255.46s user 0.65s system 353% cpu 1:12.46 total

❱❱❱ du -sh SRPMS/*
68M	SRPMS/test-0.0.1-1.src.rpm

❱❱❱ rpm2cpio SRPMS/test-0.0.1-1.src.rpm | cpio -idmv
binary_412MB
test.spec
text_file
843347 blocks
```

w2T0.xzdio (for comparison and to double check)
```
❱❱❱ time ./rpmbuild --define '_source_payload w2T0.xzdio'  -bs -v ~/rpmbuild/SPECS/test.spec
Wrote: /home/****/rpmbuild/SRPMS/test-0.0.1-1.src.rpm
./rpmbuild --define '_source_payload w2T0.xzdio' -bs -v   50.81s user 0.43s system 99% cpu 51.245 tota

❱❱❱ du -sh SRPMS/*
73M	SRPMS/test-0.0.1-1.src.rpm

❱❱❱ rpm2cpio SRPMS/test-0.0.1-1.src.rpm | cpio -idmv
binary_412MB
test.spec
text_file
843347 blocks
```
You can view, comment on, or merge this pull request online at:

  https://github.com/rpm-software-management/rpm/pull/2130

-- Commit Summary --

  * Backport multi-threaded zstd to 4.14.x as it's the latest version

-- File Changes --

    M configure.ac (2)
    M macros.in (11)
    M rpmio/rpmio.c (87)

-- Patch Links --

https://github.com/rpm-software-management/rpm/pull/2130.patch
https://github.com/rpm-software-management/rpm/pull/2130.diff

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/2130
You are receiving this because you are subscribed to this thread.

Message ID: <rpm-software-management/rpm/pull/2130 at github.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20220725/affae357/attachment.html>