From owner-svn-src-head@FreeBSD.ORG Sun Jan 25 23:08:47 2009 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E6081065713; Sun, 25 Jan 2009 23:08:47 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 7B7CF8FC2A; Sun, 25 Jan 2009 23:08:47 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id n0PN8l6S030021; Sun, 25 Jan 2009 23:08:47 GMT (envelope-from delphij@svn.freebsd.org) Received: (from delphij@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id n0PN8lvg030020; Sun, 25 Jan 2009 23:08:47 GMT (envelope-from delphij@svn.freebsd.org) Message-Id: <200901252308.n0PN8lvg030020@svn.freebsd.org> From: Xin LI Date: Sun, 25 Jan 2009 23:08:47 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r187700 - head/lib/libc/string X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jan 2009 23:08:48 -0000 Author: delphij Date: Sun Jan 25 23:08:47 2009 New Revision: 187700 URL: http://svn.freebsd.org/changeset/base/187700 Log: Rewrite of MI strlen(3) in a way that can better utilize modern hardware by reducing branches and doing word-sized operation. The idea is taken from J.T. Conklin's x86_64 optimized version of strlen(3) for NetBSD, and reimplemented in C by me. Discussed on: -arch@ Modified: head/lib/libc/string/strlen.c Modified: head/lib/libc/string/strlen.c ============================================================================== --- head/lib/libc/string/strlen.c Sun Jan 25 21:27:31 2009 (r187699) +++ head/lib/libc/string/strlen.c Sun Jan 25 23:08:47 2009 (r187700) @@ -1,6 +1,6 @@ /*- - * Copyright (c) 1990, 1993 - * The Regents of the University of California. All rights reserved. + * Copyright (c) 2009 Xin LI + * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions @@ -10,14 +10,11 @@ * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. - * 4. Neither the name of the University nor the names of its contributors - * may be used to endorse or promote products derived from this software - * without specific prior written permission. * - * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) @@ -27,21 +24,87 @@ * SUCH DAMAGE. */ -#if defined(LIBC_SCCS) && !defined(lint) -static char sccsid[] = "@(#)strlen.c 8.1 (Berkeley) 6/4/93"; -#endif /* LIBC_SCCS and not lint */ #include __FBSDID("$FreeBSD$"); +#include +#include #include +/* + * Portable strlen() for 32-bit and 64-bit systems. + * + * Rationale: it is generally much more efficient to do word length + * operations and avoid branches on modern computer systems, as + * compared to byte-length operations with a lot of branches. + * + * The expression: + * + * ((x - 0x01....01) & ~x & 0x80....80) + * + * would evaluate to a non-zero value iff any of the bytes in the + * original word is zero. However, we can further reduce ~1/3 of + * time if we consider that strlen() usually operate on 7-bit ASCII + * by employing the following expression, which allows false positive + * when high bit of 1 and use the tail case to catch these case: + * + * ((x - 0x01....01) & 0x80....80) + * + * This is more than 5.2 times as compared to the raw implementation + * on Intel T7300 under EM64T mode for strings longer than word length. + */ + +/* Magic numbers for the algorithm */ +#if LONG_BIT == 32 +static const unsigned long mask01 = 0x01010101; +static const unsigned long mask80 = 0x80808080; +#elif LONG_BIT == 64 +static const unsigned long mask01 = 0x0101010101010101; +static const unsigned long mask80 = 0x8080808080808080; +#else +#error Unsupported word size +#endif + +#define LONGPTR_MASK (sizeof(long) - 1) + +/* + * Helper macro to return string length if we caught the zero + * byte. + */ +#define testbyte(x) \ + do { \ + if (p[x] == '\0') \ + return (p - str + x); \ + } while (0) + size_t -strlen(str) - const char *str; +strlen(const char *str) { - const char *s; + const char *p; + const unsigned long *lp; + + /* Skip the first few bytes until we have an aligned p */ + for (p = str; (uintptr_t)p & LONGPTR_MASK; p++) + if (*p == '\0') + return (p - str); + + /* Scan the rest of the string using word sized operation */ + for (lp = (const unsigned long *)p; ; lp++) + if ((*lp - mask01) & mask80) { + p = (const char *)(lp); + testbyte(0); + testbyte(1); + testbyte(2); + testbyte(3); +#if (LONG_BIT >= 64) + testbyte(4); + testbyte(5); + testbyte(6); + testbyte(7); +#endif + } - for (s = str; *s; ++s); - return(s - str); + /* NOTREACHED */ + return 0; }