From owner-freebsd-stable Thu Feb 27 14:32:19 2003 Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5803837B401 for ; Thu, 27 Feb 2003 14:32:13 -0800 (PST) Received: from oneplusone.ch (oneplusone.ch [212.55.208.170]) by mx1.FreeBSD.org (Postfix) with ESMTP id 74BB143F75 for ; Thu, 27 Feb 2003 14:32:10 -0800 (PST) (envelope-from ast@marabu.ch) Received: from oneplusone.ch (localhost [127.0.0.1]) by oneplusone.ch (8.12.6/8.12.3) with ESMTP id h1RMW3ew002796; Thu, 27 Feb 2003 23:32:03 +0100 (MET) (envelope-from ast@marabu.ch) Received: (from uucp@localhost) by oneplusone.ch (8.12.6/8.12.3/Submit) with UUCP id h1RMW2JS002795; Thu, 27 Feb 2003 23:32:02 +0100 (MET) Received: from marabu.marabu.ch (marabu.marabu.ch [192.168.21.3]) by marabu.ch (8.11.6/8.11.6) with ESMTP id h1RMR8F34275; Thu, 27 Feb 2003 23:27:08 +0100 (MET) (envelope-from ast@marabu.ch) Received: by marabu.marabu.ch (8.7.5/20001028-ast-8.3) id XAA01208; Thu, 27 Feb 2003 23:27:05 +0100 (CET) Message-Id: <200302272227.XAA01208@marabu.marabu.ch> MIME-Version: 1.0 (NeXT Mail 3.3 v124.8483.6) Content-Type: text/plain In-Reply-To: <036f01c2de84$82a70b30$52557f42@errno.com> X-Nextstep-Mailer: Mail 3.3 (Enhance 2.0b6) Received: by NeXT.Mailer (1.124.8483.6) From: Adrian Steinmann Date: Thu, 27 Feb 2003 23:27:03 +0100 To: "Sam Leffler" Subject: Re: Is OpenSSL 0.9.7a really using cryptodev hardware? Cc: References: <200302271357.OAA00975@marabu.marabu.ch> <036f01c2de84$82a70b30$52557f42@errno.com> X-Organization: Webgroup Consulting AG, Apollostrasse 21, 8032 Zurich X-Phone-Numbers: Switzerland, Tel +41 1 380 30 83 Fax +41 1 380 30 85 Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Using the lthe -p option on larger samples, this is what I get: # cryptotest -p 1024 4096 2.833 sec, 2048 3des crypts, 4096 bytes, 2961324 byte/sec, 22.6 Mb/sec dispatch->invoke: avg 8174 ns : min 5280 ns : max 93839 ns [2048 samps] invoke->done: avg 667909 ns : min 630956 ns : max 791275 ns [2048 samps] done->cb: avg 59435 ns : min 42720 ns : max 453957 ns [2048 samps] cb->finis: avg 11255 ns : min 8519 ns : max 78119 ns [2048 samps] this looks much better. I was using the cryptotest -z without a parameter, and that was just 2 samples so the numbers were indeed "wumpus". I was actually not questioning cryptotest which indeed exercises the hifn and shows how it can be accessed via /dev/crypto. My problem is that I cannot reproduce those same speeds with openssl, even if I specify the engine cryptodev (the only one actually available on my soekris) As expected, cryptodev disappears from the list when I invoke openssl engine -vvv -c -t after /dev/crypto is gone, yet the speeds are identical in before and after (and enginetest doesn't report any engines in both cases). OpenSSL 0.9.7a Feb 19 2003 with /dev/crypto: $ openssl speed des type 8 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes des cbc 572.92k 605.14k 614.04k 616.71k 608.82k des ede3 165.00k 171.29k 171.84k 174.22k 172.88k (same numbers after rm -f /dev/crypto) The speeds look slightly worse than OpenSSL 0.9.6g which doesn't support cryptodev: OpenSSL 0.9.6g 9 Aug 2002 (without cryptodev support, and no /dev/crypto): des cbc 1236.05k 1343.45k 1356.76k 1364.72k 1327.12k des ede3 450.66k 465.74k 463.10k 466.68k 466.32k I suspect engine overhead is in 0.9.7, but then "at the last minute" openssl doesn't go through the hardware (which would explain the lack of ioctl to /dev/crypto when ktracing openssl). When I run cryptostats before and after, say, $ openvpn --test-crypto --secret /tmp/s I see no access to the hifn either, which could be an indication that the problem is in the common shared library libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x280df000) Adrian _______________________________________________ Adrian Steinmann Apollostrasse 21 8032 Zurich Tel +41 1 380 30 83 Mailto:ast@marabu.ch I originally wrote: > I'm running RELENG_4 cvsup-ed 2003/02/25UTC > > $ openssl version > OpenSSL 0.9.7a Feb 19 2003 > $ ldd /usr/bin/openssl: > libssl.so.3 => /usr/lib/libssl.so.3 (0x280b0000) > libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x280df000) > libc.so.4 => /usr/lib/libc.so.4 (0x281d6000) > > On my Soekris net4501 I have: > > device crypto > device cryptodev > device hifn > > configured in the kernel and running: > hifn0 mem 0xa0001000-0xa0001fff,0xa0000000-0xa0000fff irq 10 at device 16.0 on pci0 > hifn0: Hifn 7951, rev 0, 128KB sram, 193 sessions > > and /dev/crypto exists as well, openssl reports it as available: > > $ openssl engine -vvv -c -t > (cryptodev) BSD cryptodev engine > [RSA, DSA, DH, DES-CBC, DES-EDE3-CBC] > [ available ] > ... (the other are not available) > > When I run cryptotest -z (from /usr/src/tools/tools/crypto) > I get these speeds: > # sysctl -w debug.crypto_timing=1 > $ cryptotest -z|grep 8192 > 0.474 sec, 2 des crypts, 8192 bytes, 34600 byte/sec, 0.3 Mb/sec > 0.471 sec, 2 3des crypts, 8192 bytes, 34753 byte/sec, 0.3 Mb/sec > FWIW you can also use cryptotest -zp to turn on+off profiling over the time of the run. Also, w/ no additional argument you're doing only 1 run of each block size so your results are unrealistic (but still they look wrong too). > and cryptostats reports these: > $ cryptostats > 1171 symmetric crypto ops (0 errors, 0 times driver blocked) > 0 key ops (0 errors, 0 times driver blocked) > 0 crypto dispatch thread activations > 1171 crypto return thread activations > This last line indicates you're slightly out of date in that I made a recent MFC to eliminate the "thread activations" required for processing operations going through /dev/crypto. > dispatch->invoke: avg 7764 ns : min 0 ns : max 102959 ns [823 samps] > invoke->done: avg 224321569 ns : min 0 ns : max 226578803 ns [823 samps] > done->cb: avg 8647603 ns : min 0 ns : max 13619770 ns [823 samps] > cb->finis: avg 27031 ns : min 0 ns : max 120359 ns [823 samps] > These numbers are kinda wumpus because you manually turned profling on+off. cryptotest -p does the right thing to reinitialize the profiling counters so that min+max are correct. But your numbers, even with 823 samples, are still very slow: dispatch->invoke 7.8us invoke->done 224ms done->cb 8.6ms cb->finis 27us I'm not sure what size data blocks you're passing here (probably an assortment) but 224ms is basically the time spent in the driver (and h/w) doing the actual crypto operation. Everything else is essentially the overhead of using /dev/crypto. If you do a fixed size run then you can use this to calculate the effect performance of the 7951; e.g. cryptotest -p 1024 4096 will run only 4K data blocks through and then you can use the invoke->done time to calculate out the raw performance available (assuming nothing else of note is running on the machine). > However, when I do the same test with openssl the numbers look > identical with and without /dev/crypto: > > $ openssl speed -engine cryptodev des > engine "cryptodev" set. > ... > The 'numbers' are in 1000s of bytes per second processed. > type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes > des cbc 576.97k 612.60k 610.02k 617.09k 609.10k > des ede3 174.44k 181.80k 185.19k 183.17k 183.55k I'm not sure if cryptodev is the engine name (it was changed at one point). If you don't specify engine you should get the h/w device if it is available. > # rm -f /dev/crypto > $ openssl speed -engine cryptodev des > reports an invalid engine "cryptodev" and then continues with same numbers > > cryptostats doesn't report any addironal hifn processing, and when > > # sysctl -w debug.hifn=1 > > is set, no debugging is seen on console during openssl speed either. > > Furthermore, > > /usr/src/crypto/openssl/crypto/engine/enginetest.c > > reports no available engines: > > $ ./enginetest > > enginetest beginning > > listing available engine types > end of list > ... > Is this before or after you removed /dev/crypto? I'm not sure if the list will include the h/w crypto engine unless it can open /dev/crypto. > This all leads me to suspect that the -stable openssl 0.9.7a doesn't > have the complete cryptodev engine compiled in. Is openssl failing > to put cryptodev into the engine list when it does speed? I've also > tried encryption runs but they all don't seem to use the hifn for > encryption. > > (I've compared ktraces for both cryptotest and openssl, openssl > does three ioctl() on /dev/crypto during speed, whereas cryptotest > keeps on doing the ioctl() for the encryption). I'm using it w/o any issues and it seems to do the right thing. But I haven't tried things on a soekris box in a long time; not that it should matter. I didn't see results for just openssl w/o -engine cryptodev; did you try that? Sam To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message