Date: Thu, 27 Feb 2003 23:27:03 +0100 From: Adrian Steinmann <ast@marabu.ch> To: "Sam Leffler" <sam@errno.com> Cc: <stable@FreeBSD.ORG> Subject: Re: Is OpenSSL 0.9.7a really using cryptodev hardware? Message-ID: <200302272227.XAA01208@marabu.marabu.ch> In-Reply-To: <036f01c2de84$82a70b30$52557f42@errno.com> References: <200302271357.OAA00975@marabu.marabu.ch> <036f01c2de84$82a70b30$52557f42@errno.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Using the lthe -p option on larger samples, this is what I get:
# cryptotest -p 1024 4096
2.833 sec, 2048 3des crypts, 4096 bytes, 2961324 byte/sec, 22.6 Mb/sec
dispatch->invoke: avg 8174 ns : min 5280 ns : max 93839 ns [2048 samps]
invoke->done: avg 667909 ns : min 630956 ns : max 791275 ns [2048 samps]
done->cb: avg 59435 ns : min 42720 ns : max 453957 ns [2048 samps]
cb->finis: avg 11255 ns : min 8519 ns : max 78119 ns [2048 samps]
this looks much better. I was using the cryptotest -z without a
parameter, and that was just 2 samples so the numbers were indeed "wumpus".
I was actually not questioning cryptotest which indeed exercises
the hifn and shows how it can be accessed via /dev/crypto.
My problem is that I cannot reproduce those same speeds with openssl,
even if I specify the engine cryptodev (the only one actually
available on my soekris) As expected, cryptodev disappears from the
list when I invoke
openssl engine -vvv -c -t
after /dev/crypto is gone, yet the speeds are identical in before
and after (and enginetest doesn't report any engines in both cases).
OpenSSL 0.9.7a Feb 19 2003 with /dev/crypto:
$ openssl speed des
type 8 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
des cbc 572.92k 605.14k 614.04k 616.71k 608.82k
des ede3 165.00k 171.29k 171.84k 174.22k 172.88k
(same numbers after rm -f /dev/crypto)
The speeds look slightly worse than OpenSSL 0.9.6g which doesn't
support cryptodev:
OpenSSL 0.9.6g 9 Aug 2002 (without cryptodev support, and no /dev/crypto):
des cbc 1236.05k 1343.45k 1356.76k 1364.72k 1327.12k
des ede3 450.66k 465.74k 463.10k 466.68k 466.32k
I suspect engine overhead is in 0.9.7, but then "at the last minute"
openssl doesn't go through the hardware (which would explain the
lack of ioctl to /dev/crypto when ktracing openssl).
When I run cryptostats before and after, say,
$ openvpn --test-crypto --secret /tmp/s
I see no access to the hifn either, which could be an indication
that the problem is in the common shared library
libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x280df000)
Adrian
_______________________________________________
Adrian Steinmann Apollostrasse 21 8032 Zurich
Tel +41 1 380 30 83 Mailto:ast@marabu.ch
I originally wrote:
> I'm running RELENG_4 cvsup-ed 2003/02/25UTC
>
> $ openssl version
> OpenSSL 0.9.7a Feb 19 2003
> $ ldd /usr/bin/openssl:
> libssl.so.3 => /usr/lib/libssl.so.3 (0x280b0000)
> libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x280df000)
> libc.so.4 => /usr/lib/libc.so.4 (0x281d6000)
>
> On my Soekris net4501 I have:
>
> device crypto
> device cryptodev
> device hifn
>
> configured in the kernel and running:
> hifn0 mem 0xa0001000-0xa0001fff,0xa0000000-0xa0000fff irq 10 at device
16.0 on pci0
> hifn0: Hifn 7951, rev 0, 128KB sram, 193 sessions
>
> and /dev/crypto exists as well, openssl reports it as available:
>
> $ openssl engine -vvv -c -t
> (cryptodev) BSD cryptodev engine
> [RSA, DSA, DH, DES-CBC, DES-EDE3-CBC]
> [ available ]
> ... (the other are not available)
>
> When I run cryptotest -z (from /usr/src/tools/tools/crypto)
> I get these speeds:
> # sysctl -w debug.crypto_timing=1
> $ cryptotest -z|grep 8192
> 0.474 sec, 2 des crypts, 8192 bytes, 34600 byte/sec,
0.3 Mb/sec
> 0.471 sec, 2 3des crypts, 8192 bytes, 34753 byte/sec,
0.3 Mb/sec
>
FWIW you can also use cryptotest -zp to turn on+off profiling over the time
of the run. Also, w/ no additional argument you're doing only 1 run of each
block size so your results are unrealistic (but still they look wrong too).
> and cryptostats reports these:
> $ cryptostats
> 1171 symmetric crypto ops (0 errors, 0 times driver blocked)
> 0 key ops (0 errors, 0 times driver blocked)
> 0 crypto dispatch thread activations
> 1171 crypto return thread activations
>
This last line indicates you're slightly out of date in that I made a recent
MFC to eliminate the "thread activations" required for processing operations
going through /dev/crypto.
> dispatch->invoke: avg 7764 ns : min 0 ns : max 102959 ns [823
samps]
> invoke->done: avg 224321569 ns : min 0 ns : max 226578803 ns [823
samps]
> done->cb: avg 8647603 ns : min 0 ns : max 13619770 ns [823
samps]
> cb->finis: avg 27031 ns : min 0 ns : max 120359 ns [823
samps]
>
These numbers are kinda wumpus because you manually turned profling on+off.
cryptotest -p does the right thing to reinitialize the profiling counters so
that min+max are correct. But your numbers, even with 823 samples, are
still very slow:
dispatch->invoke 7.8us
invoke->done 224ms
done->cb 8.6ms
cb->finis 27us
I'm not sure what size data blocks you're passing here (probably an
assortment) but 224ms is basically the time spent in the driver (and h/w)
doing the actual crypto operation. Everything else is essentially the
overhead of using /dev/crypto. If you do a fixed size run then you can use
this to calculate the effect performance of the 7951; e.g.
cryptotest -p 1024 4096
will run only 4K data blocks through and then you can use the invoke->done
time to calculate out the raw performance available (assuming nothing else
of note is running on the machine).
> However, when I do the same test with openssl the numbers look
> identical with and without /dev/crypto:
>
> $ openssl speed -engine cryptodev des
> engine "cryptodev" set.
> ...
> The 'numbers' are in 1000s of bytes per second processed.
> type 16 bytes 64 bytes 256 bytes 1024 bytes 8192
bytes
> des cbc 576.97k 612.60k 610.02k 617.09k
609.10k
> des ede3 174.44k 181.80k 185.19k 183.17k
183.55k
I'm not sure if cryptodev is the engine name (it was changed at one point).
If you don't specify engine you should get the h/w device if it is
available.
> # rm -f /dev/crypto
> $ openssl speed -engine cryptodev des
> reports an invalid engine "cryptodev" and then continues with same numbers
>
> cryptostats doesn't report any addironal hifn processing, and when
>
> # sysctl -w debug.hifn=1
>
> is set, no debugging is seen on console during openssl speed either.
>
> Furthermore,
>
> /usr/src/crypto/openssl/crypto/engine/enginetest.c
>
> reports no available engines:
>
> $ ./enginetest
>
> enginetest beginning
>
> listing available engine types
> end of list
> ...
>
Is this before or after you removed /dev/crypto? I'm not sure if the list
will include the h/w crypto engine unless it can open /dev/crypto.
> This all leads me to suspect that the -stable openssl 0.9.7a doesn't
> have the complete cryptodev engine compiled in. Is openssl failing
> to put cryptodev into the engine list when it does speed? I've also
> tried encryption runs but they all don't seem to use the hifn for
> encryption.
>
> (I've compared ktraces for both cryptotest and openssl, openssl
> does three ioctl() on /dev/crypto during speed, whereas cryptotest
> keeps on doing the ioctl() for the encryption).
I'm using it w/o any issues and it seems to do the right thing. But I
haven't tried things on a soekris box in a long time; not that it should
matter. I didn't see results for just openssl w/o -engine cryptodev; did
you try that?
Sam
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200302272227.XAA01208>