From owner-svn-src-all@FreeBSD.ORG Sun Aug 25 13:10:04 2013 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 1C0979E1; Sun, 25 Aug 2013 13:10:04 +0000 (UTC) (envelope-from roberto@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:1900:2254:2068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 0915D2CE1; Sun, 25 Aug 2013 13:10:04 +0000 (UTC) Received: from svn.freebsd.org ([127.0.1.70]) by svn.freebsd.org (8.14.7/8.14.7) with ESMTP id r7PDA3xM090498; Sun, 25 Aug 2013 13:10:03 GMT (envelope-from roberto@svn.freebsd.org) Received: (from roberto@localhost) by svn.freebsd.org (8.14.7/8.14.5/Submit) id r7PDA3Yq090496; Sun, 25 Aug 2013 13:10:03 GMT (envelope-from roberto@svn.freebsd.org) Message-Id: <201308251310.r7PDA3Yq090496@svn.freebsd.org> From: Ollivier Robert Date: Sun, 25 Aug 2013 13:10:03 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-9@freebsd.org Subject: svn commit: r254856 - stable/9/sys/crypto/aesni X-SVN-Group: stable-9 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Aug 2013 13:10:04 -0000 Author: roberto Date: Sun Aug 25 13:10:03 2013 New Revision: 254856 URL: http://svnweb.freebsd.org/changeset/base/254856 Log: Merge r226837 & r226839 from head: r226837: Improve AES-NI performance for AES-XTS: - Operate on uint64_t types when doing XORing, etc. instead of uint8_t. - Don't bzero() temporary block for every AES block. Do it once for entire data block. - AES-NI is available only on little endian architectures. Simplify code that takes block number from IV. r226839: Update Copyright. Modified: stable/9/sys/crypto/aesni/aesni_wrap.c Directory Properties: stable/9/sys/ (props changed) Modified: stable/9/sys/crypto/aesni/aesni_wrap.c ============================================================================== --- stable/9/sys/crypto/aesni/aesni_wrap.c Sun Aug 25 12:58:34 2013 (r254855) +++ stable/9/sys/crypto/aesni/aesni_wrap.c Sun Aug 25 13:10:03 2013 (r254856) @@ -1,6 +1,6 @@ /*- * Copyright (c) 2010 Konstantin Belousov - * Copyright (c) 2010 Pawel Jakub Dawidek + * Copyright (c) 2010-2011 Pawel Jakub Dawidek * All rights reserved. * * Redistribution and use in source and binary forms, with or without @@ -87,33 +87,33 @@ aesni_decrypt_ecb(int rounds, const void #define AES_XTS_ALPHA 0x87 /* GF(2^128) generator polynomial */ static void -aesni_crypt_xts_block(int rounds, const void *key_schedule, uint8_t *tweak, - const uint8_t *from, uint8_t *to, int do_encrypt) +aesni_crypt_xts_block(int rounds, const void *key_schedule, uint64_t *tweak, + const uint64_t *from, uint64_t *to, uint64_t *block, int do_encrypt) { - uint8_t block[AES_XTS_BLOCKSIZE]; - u_int i, carry_in, carry_out; + int carry; - for (i = 0; i < AES_XTS_BLOCKSIZE; i++) - block[i] = from[i] ^ tweak[i]; + block[0] = from[0] ^ tweak[0]; + block[1] = from[1] ^ tweak[1]; if (do_encrypt) - aesni_enc(rounds - 1, key_schedule, block, to, NULL); + aesni_enc(rounds - 1, key_schedule, (uint8_t *)block, (uint8_t *)to, NULL); else - aesni_dec(rounds - 1, key_schedule, block, to, NULL); + aesni_dec(rounds - 1, key_schedule, (uint8_t *)block, (uint8_t *)to, NULL); - for (i = 0; i < AES_XTS_BLOCKSIZE; i++) - to[i] ^= tweak[i]; + to[0] ^= tweak[0]; + to[1] ^= tweak[1]; /* Exponentiate tweak. */ - carry_in = 0; - for (i = 0; i < AES_XTS_BLOCKSIZE; i++) { - carry_out = tweak[i] & 0x80; - tweak[i] = (tweak[i] << 1) | (carry_in ? 1 : 0); - carry_in = carry_out; - } - if (carry_in) - tweak[0] ^= AES_XTS_ALPHA; - bzero(block, sizeof(block)); + carry = ((tweak[0] & 0x8000000000000000ULL) > 0); + tweak[0] <<= 1; + if (tweak[1] & 0x8000000000000000ULL) { + uint8_t *twk = (uint8_t *)tweak; + + twk[0] ^= AES_XTS_ALPHA; + } + tweak[1] <<= 1; + if (carry) + tweak[1] |= 1; } static void @@ -121,32 +121,33 @@ aesni_crypt_xts(int rounds, const void * const void *tweak_schedule, size_t len, const uint8_t *from, uint8_t *to, const uint8_t iv[AES_BLOCK_LEN], int do_encrypt) { + uint64_t block[AES_XTS_BLOCKSIZE / 8]; uint8_t tweak[AES_XTS_BLOCKSIZE]; - uint64_t blocknum; size_t i; /* * Prepare tweak as E_k2(IV). IV is specified as LE representation * of a 64-bit block number which we allow to be passed in directly. */ - bcopy(iv, &blocknum, AES_XTS_IVSIZE); - for (i = 0; i < AES_XTS_IVSIZE; i++) { - tweak[i] = blocknum & 0xff; - blocknum >>= 8; - } +#if BYTE_ORDER == LITTLE_ENDIAN + bcopy(iv, tweak, AES_XTS_IVSIZE); /* Last 64 bits of IV are always zero. */ bzero(tweak + AES_XTS_IVSIZE, AES_XTS_IVSIZE); +#else +#error Only LITTLE_ENDIAN architectures are supported. +#endif aesni_enc(rounds - 1, tweak_schedule, tweak, tweak, NULL); len /= AES_XTS_BLOCKSIZE; for (i = 0; i < len; i++) { - aesni_crypt_xts_block(rounds, data_schedule, tweak, from, to, - do_encrypt); + aesni_crypt_xts_block(rounds, data_schedule, (uint64_t *)tweak, + (const uint64_t *)from, (uint64_t *)to, block, do_encrypt); from += AES_XTS_BLOCKSIZE; to += AES_XTS_BLOCKSIZE; } bzero(tweak, sizeof(tweak)); + bzero(block, sizeof(block)); } static void