<p>To be clear we hit thus hundreds of times a day on Linux servers running RPM from rawhide.</p>
<p>Hundreds. Of. Times. A. Day.</p>
<p>We have remediations that pause all config management and then run db_recovers and if necessary rpm rebuilds.</p>
<p>Because the rpmdb is completely fucked -- <code>rpm -qa</code> doesn't work, as seen before.</p>
<pre><code>$ rpm -q libdb
libdb-5.3.21-19.el7.x86_64
$ rpm -q rpm
rpm-4.13.0-1.el7.centos.fb3.x86_64
</code></pre>
<p>There are two causes we've found - one is RPMs fault, one isn't. The most common is exactly the one described here. This one is RPMs fault.</p>
<p>The other is heavily IO loaded boxes were the transactions take too long and get killed by another process and the rpmdb can't recover from that. This isn't RPMs fault... though it being more resilient would be nice.</p>

<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />You are receiving this because you are subscribed to this thread.<br />Reply to this email directly, <a href="https://github.com/rpm-software-management/rpm/issues/232#issuecomment-308292205">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/ANb802gYMSamgohTG_IjOIznPSxIjTc1ks5sDzT8gaJpZM4NzFoB">mute the thread</a>.<img alt="" height="1" src="https://github.com/notifications/beacon/ANb808RmrpbqNf2y6hsH3D7cBcy6mjl5ks5sDzT8gaJpZM4NzFoB.gif" width="1" /></p>
<div itemscope itemtype="http://schema.org/EmailMessage">
<div itemprop="action" itemscope itemtype="http://schema.org/ViewAction">
  <link itemprop="url" href="https://github.com/rpm-software-management/rpm/issues/232#issuecomment-308292205"></link>
  <meta itemprop="name" content="View Issue"></meta>
</div>
<meta itemprop="description" content="View this Issue on GitHub"></meta>
</div>

<script type="application/json" data-scope="inboxmarkup">{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/rpm-software-management/rpm","title":"rpm-software-management/rpm","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/rpm-software-management/rpm"}},"updates":{"snippets":[{"icon":"PERSON","message":"@jaymzh in #232: To be clear we hit thus hundreds of times a day on Linux servers running RPM from rawhide.\r\n\r\nHundreds. Of. Times. A. Day.\r\n\r\nWe have remediations that pause all config management and then run db_recovers and if necessary rpm rebuilds.\r\n\r\nBecause the rpmdb is completely fucked -- `rpm -qa` doesn't work, as seen before.\r\n\r\n```\r\n$ rpm -q libdb\r\nlibdb-5.3.21-19.el7.x86_64\r\n$ rpm -q rpm\r\nrpm-4.13.0-1.el7.centos.fb3.x86_64\r\n```\r\n\r\nThere are two causes we've found - one is RPMs fault, one isn't. The most common is exactly the one described here. This one is RPMs fault.\r\n\r\nThe other is heavily IO loaded boxes were the transactions take too long and get killed by another process and the rpmdb can't recover from that. This isn't RPMs fault... though it being more resilient would be nice."}],"action":{"name":"View Issue","url":"https://github.com/rpm-software-management/rpm/issues/232#issuecomment-308292205"}}}</script>