[Rpm-maint] digest_beecrypt cleanup

[Michael Schroeder]

Hi Panu & al,

the attached patch cleans up the code in digest_beecrypt:

- free all MPIs, no more memory leaks
- do not use mp?sethex, use mp?setbin instead
- check return values of mp?setbin
- do DSA2 hash truncation (untested though)
- when doing RSA padding, use the length of the key instead
  of the sig, as the sig can start with zeros

I also changed the internal interface a bit, the length of the
MPIs os now checked before calling ->setmpi. Thus there's no
longer the need to have an "pend" parameter everywhere in the
digest code.

While I did that change i noticed a potential problem in digest_nss'
pgpSetSigMpiDSA function: it calculates "qbits" from the signature MPIs.
This is IMHO wrong, as the signature MPIs can be smaller. So there's
a small chance that qbits is < DSA_MIN_Q_BITS. I don't think there should
be a check at all (there's no check in pgpSetSigMpiRSA()), if there's a
check it should only check against the max size.
(Of course the nss interface assumes that the two DSA's signature
MPIs have the same size, so there must be some padding if one is smaller.)

(The check was added with commit #fe5a1e5d)

Cheers,
  Michael.

-- 
Michael Schroeder                                   mls at suse.de
SUSE LINUX Products GmbH,  GF Jeff Hawn, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}
-------------- next part --------------
A non-text attachment was scrubbed...
Name: beedigest.diff
Type: text/x-patch
Size: 14461 bytes
Desc: not available
URL: <http://lists.rpm.org/pipermail/rpm-maint/attachments/20130827/765209b8/attachment.bin>