[Rpm-maint] [rpm-software-management/rpm] rpm processes hangs if we have a corrupt db (#602)

[Sangamesh Mallayya]

Hi all,
I have a scenario where lot of rpm processes are being created if we have a corrupt database which eventually causing performance issue.

If their is a corruption any new rpm process runs will try to acquire the lock and then calls select with timeout value.

So, before the previous process timeout expires and release the lock, if another rpm process is started it will be waiting the for the lock.
Sequence here is, acquire the lock (after first process timeout expires and released the lock) and then wait for sometime using timeout value as we have corrupted rpm database.
If we run more rpm processes (through cron etc..),  original process completion gets delayed and more and more processes keep on running.

This continues until we don't fix the database manually.
But, if in case of automated environment where some cron jobs would be running really can cause a performance issue.
 

Here we can see that process 13107464 initially was waiting for the lock and then later goes to wait till the timeout expires.

bash-4.4# ps -aef | grep rpm | cut -d' ' -f6 | xargs procstack
**13107464:** /usr/opt/freeware/bin/rpm -qa
0xd0121694  __fcntl(??, ??, ??) + 0x1d4
0xd0121930  fcntl(0x0, 0xd, 0x2ff21e30, 0xd0942b30, 0x3b8c, 0x0, 0x504dcc00, 0x0) + 0x30
0xd08c6e80  serialize_env(??) + 0xc0
0xd08c60bc  db_init(??, ??) + 0x47c
0xd08c4d10  db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38  dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534  pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4  openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8  rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd08cf7b4  loadKeyringFromDB(??) + 0x54
0xd08cfd90  loadKeyring(??) + 0x50
0xd08d27b0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x330
0xd092c13c  initFilterIterator(??, ??) + 0x3c
0xd092d6bc  rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c  rpmcliQuery(??, ??, ??) + 0xcc
0x100008ac  main() + 0x4ac
0x10000190  __start() + 0x68
15401230: /usr/opt/freeware/bin/rpm -qa
0xd0121694  __fcntl(??, ??, ??) + 0x1d4
0xd0121930  fcntl(0x0, 0xd, 0x2ff21e30, 0xd0942b30, 0x3b8c, 0x0, 0x504dd800, 0x0) + 0x30
0xd08c6e80  serialize_env(??) + 0xc0
0xd08c60bc  db_init(??, ??) + 0x47c
0xd08c4d10  db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38  dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534  pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4  openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8  rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd08cf7b4  loadKeyringFromDB(??) + 0x54
0xd08cfd90  loadKeyring(??) + 0x50
0xd08d27b0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x330
0xd092c13c  initFilterIterator(??, ??) + 0x3c
0xd092d6bc  rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c  rpmcliQuery(??, ??, ??) + 0xcc
0x10000190  __start() + 0x68
22806956: /usr/opt/freeware/bin/rpm -qa
0xd027582c  __fd_select(??, ??, ??, ??, ??) + 0xcc
0xd08c5fa8  db_init(??, ??) + 0x368
0xd08c4d10  db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38  dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534  pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4  openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8  rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd092c13c  initFilterIterator(??, ??) + 0x3c
0xd092d6bc  rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c  rpmcliQuery(??, ??, ??) + 0xcc
0x10000190  __start() + 0x68

bash-4.4# ps -aef | grep rpm | cut -d' ' -f6 | xargs procstack
**13107464**: /usr/opt/freeware/bin/rpm -qa
0xd027582c  __fd_select(??, ??, ??, ??, ??) + 0xcc
0xd0f0a40c  select() + 0xc
0xd0f0a2b0  __os_sleep() + 0x30
0xd0f87c68  __env_attach() + 0x228
0xd0f849f4  __env_attach_regions() + 0x134
0xd0f86088  __env_open() + 0x2a8
0xd0f86260  __env_open_pp() + 0x120
0xd08c5fa8  db_init(??, ??) + 0x368
0xd08c4d10  db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38  dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534  pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4  openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8  rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd08cf7b4  loadKeyringFromDB(??) + 0x54
0xd08cfd90  loadKeyring(??) + 0x50
0xd08d27b0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x330
0xd092c13c  initFilterIterator(??, ??) + 0x3c
0xd092d6bc  rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c  rpmcliQuery(??, ??, ??) + 0xcc
0x100008ac  main() + 0x4ac
0x10000190  __start() + 0x68
15401230: /usr/opt/freeware/bin/rpm -qa
0xd0121694  __fcntl(??, ??, ??) + 0x1d4
0xd0121930  fcntl(0x0, 0xd, 0x2ff21e30, 0xd0942b30, 0x3b8c, 0x0, 0x504dd800, 0x0) + 0x30
0xd08c6e80  serialize_env(??) + 0xc0
0xd08c60bc  db_init(??, ??) + 0x47c
0xd08c4d10  db3_dbiOpen(??, ??, ??, ??) + 0xd0
0xd08c8c38  dbiOpen(??, ??, ??, ??) + 0x38
0xd0902534  pkgdbOpen(??, ??, ??) + 0xb4
0xd09019a4  openDatabase(??, ??, ??, ??, ??, ??) + 0x104
0xd08d19c8  rpmtsOpenDB(??, ??) + 0x68
0xd08d24d0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x50
0xd08cf7b4  loadKeyringFromDB(??) + 0x54
0xd08cfd90  loadKeyring(??) + 0x50
0xd08d27b0  rpmtsInitIterator at AF92_52(??, ??, ??, ??, ??, ??) + 0x330
0xd092c13c  initFilterIterator(??, ??) + 0x3c
0xd092d6bc  rpmcliArgIter(??, ??, ??) + 0x1fc
0xd092d32c  rpmcliQuery(??, ??, ??) + 0xcc
0x10000190  __start() + 0x68
bash-4.4# 

Please let me know if this is a know issue or bug in rpm or db package.

Thanks,
Sangamesh

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/602
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20181120/4cce2906/attachment.html>