From owner-svn-src-head@freebsd.org Tue May 21 03:52:49 2019 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A258F15A176A; Tue, 21 May 2019 03:52:49 +0000 (UTC) (envelope-from ngie@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3C0167302B; Tue, 21 May 2019 03:52:49 +0000 (UTC) (envelope-from ngie@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D6C8330B; Tue, 21 May 2019 03:52:48 +0000 (UTC) (envelope-from ngie@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x4L3qmWV090981; Tue, 21 May 2019 03:52:48 GMT (envelope-from ngie@FreeBSD.org) Received: (from ngie@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x4L3qm5X090980; Tue, 21 May 2019 03:52:48 GMT (envelope-from ngie@FreeBSD.org) Message-Id: <201905210352.x4L3qm5X090980@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: ngie set sender to ngie@FreeBSD.org using -f From: Enji Cooper Date: Tue, 21 May 2019 03:52:48 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r348042 - head/tests/sys/opencrypto X-SVN-Group: head X-SVN-Commit-Author: ngie X-SVN-Commit-Paths: head/tests/sys/opencrypto X-SVN-Commit-Revision: 348042 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 3C0167302B X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 May 2019 03:52:49 -0000 Author: ngie Date: Tue May 21 03:52:48 2019 New Revision: 348042 URL: https://svnweb.freebsd.org/changeset/base/348042 Log: Fix encoding issues with python 3 In python 3, the default encoding was switched from ascii character sets to unicode character sets in order to support internationalization by default. Some interfaces, like ioctls and packets, however, specify data in terms of non-unicode encodings formats, either in host endian (`fcntl.ioctl`) or network endian (`dpkt`) byte order/format. This change alters assumptions made by previous code where it was all data objects were assumed to be basestrings, when they should have been treated as byte arrays. In order to achieve this the following are done: * str objects with encodings needing to be encoded as ascii byte arrays are done so via `.encode("ascii")`. In order for this to work on python 3 in a type agnostic way (as it anecdotally varied depending on the caller), call `.encode("ascii")` only on str objects with python 3 to cast them to ascii byte arrays in a helper function name `str_to_ascii(..)`. * `dpkt.Packet` objects needing to be passed in to `fcntl.ioctl(..)` are done so by casting them to byte arrays via `bytes()`, which calls `dpkt.Packet__str__` under the covers and does the necessary str to byte array conversion needed for the `dpkt` APIs and `struct` module. In order to accomodate this change, apply the necessary typecasting for the byte array literal in order to search `fop.name` for nul bytes. This resolves all remaining python 2.x and python 3.x compatibility issues on amd64. More work needs to be done for the tests to function with i386, in general (this is a legacy issue). PR: 237403 MFC after: 1 week Tested with: python 2.7.16 (amd64), python 3.6.8 (amd64) Modified: head/tests/sys/opencrypto/cryptodev.py Modified: head/tests/sys/opencrypto/cryptodev.py ============================================================================== --- head/tests/sys/opencrypto/cryptodev.py Tue May 21 03:06:10 2019 (r348041) +++ head/tests/sys/opencrypto/cryptodev.py Tue May 21 03:52:48 2019 (r348042) @@ -136,16 +136,22 @@ def _getdev(): _cryptodev = _getdev() +def str_to_ascii(val): + if sys.version_info[0] >= 3: + if isinstance(val, str): + return val.encode("ascii") + return val + def _findop(crid, name): fop = FindOp() fop.crid = crid - fop.name = name + fop.name = str_to_ascii(name) s = array.array('B', fop.pack_hdr()) ioctl(_cryptodev, CIOCFINDDEV, s, 1) fop.unpack(s) try: - idx = fop.name.index('\x00') + idx = fop.name.index(b'\x00') name = fop.name[:idx] except ValueError: name = fop.name @@ -218,11 +224,11 @@ class Crypto: if self._maclen is not None: m = array.array('B', [0] * self._maclen) cop.mac = m.buffer_info()[0] - ivbuf = array.array('B', iv) + ivbuf = array.array('B', str_to_ascii(iv)) cop.iv = ivbuf.buffer_info()[0] #print('cop:', cop) - ioctl(_cryptodev, CIOCCRYPT, str(cop)) + ioctl(_cryptodev, CIOCCRYPT, bytes(cop)) s = array_tobytes(s) if self._maclen is not None: @@ -236,6 +242,7 @@ class Crypto: caead.op = op caead.flags = CRD_F_IV_EXPLICIT caead.flags = 0 + src = str_to_ascii(src) caead.len = len(src) s = array.array('B', src) caead.src = caead.dst = s.buffer_info()[0] @@ -246,6 +253,7 @@ class Crypto: if self._maclen is None: raise ValueError('must have a tag length') + tag = str_to_ascii(tag) if tag is None: tag = array.array('B', [0] * self._maclen) else: @@ -259,7 +267,7 @@ class Crypto: caead.ivlen = len(iv) caead.iv = ivbuf.buffer_info()[0] - ioctl(_cryptodev, CIOCCRYPTAEAD, str(caead)) + ioctl(_cryptodev, CIOCCRYPTAEAD, bytes(caead)) s = array_tobytes(s) @@ -267,6 +275,7 @@ class Crypto: def perftest(self, op, size, timeo=3): inp = array.array('B', (random.randint(0, 255) for x in range(size))) + inp = str_to_ascii(inp) out = array.array('B', inp) # prep ioctl @@ -293,8 +302,9 @@ class Crypto: start = time.time() reps = 0 + cop = bytes(cop) while not exit[0]: - ioctl(_cryptodev, CIOCCRYPT, str(cop)) + ioctl(_cryptodev, CIOCCRYPT, cop) reps += 1 end = time.time()