From owner-svn-src-head@freebsd.org Thu Oct 6 19:46:44 2016 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C5529BD3B2C; Thu, 6 Oct 2016 19:46:44 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 84FDCF2D; Thu, 6 Oct 2016 19:46:44 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id u96JkhsM041910; Thu, 6 Oct 2016 19:46:43 GMT (envelope-from bapt@FreeBSD.org) Received: (from bapt@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id u96Jkhww041909; Thu, 6 Oct 2016 19:46:43 GMT (envelope-from bapt@FreeBSD.org) Message-Id: <201610061946.u96Jkhww041909@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bapt set sender to bapt@FreeBSD.org using -f From: Baptiste Daroussin Date: Thu, 6 Oct 2016 19:46:43 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r306782 - in head: contrib/netbsd-tests/lib/libc/locale usr.bin/localedef X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Oct 2016 19:46:44 -0000 Author: bapt Date: Thu Oct 6 19:46:43 2016 New Revision: 306782 URL: https://svnweb.freebsd.org/changeset/base/306782 Log: localedef: Fix ctype dump (fixed wide spread errors) This commit is from John Marino in dragonfly with the following commit log: ==== This was a CTYPE encoding error involving consecutive points of the same ctype. It was reported by myself to Illumos over a year ago but I was unsure if it was only happening on BSD. Given the cause, the bug is also present on Illumos. Basically, if consecutive points were of the exact same ctype, they would be defined as a range regardless. For example, all of these would be considered equivalent: ... , (converts to .. ) , , (converts to .. ) , ... (converts to .. ) So all the points that shouldn't have been defined got "bridged" by the extreme points. The effects were recently reported to FreeBSD on PR 213013. There are countless places were the ctype flags are misdefined, so this is a major fix that has to be MFC'd. ==== This reveals a bad change I did on the testsuite: while 0x07FF is a valid unicode it is not used yet (reserved for future use) PR: 213013 Submitted by: marino@ Reported by: Kurtis Rader Obtained from: Dragonfly MFC after: 1 month Modified: head/contrib/netbsd-tests/lib/libc/locale/t_mbstowcs.c head/usr.bin/localedef/ctype.c Modified: head/contrib/netbsd-tests/lib/libc/locale/t_mbstowcs.c ============================================================================== --- head/contrib/netbsd-tests/lib/libc/locale/t_mbstowcs.c Thu Oct 6 19:41:09 2016 (r306781) +++ head/contrib/netbsd-tests/lib/libc/locale/t_mbstowcs.c Thu Oct 6 19:46:43 2016 (r306782) @@ -88,7 +88,7 @@ static struct test { 0xFFFF, 0x5D, 0x5B, 0x10000, 0x10FFFF, 0x5D, 0x0A }, #ifdef __FreeBSD__ - { 1, -1, -1, 1, 1, -1, 1, 1, 1, 1, -1, 1, 1, -1, -1, + { 1, -1, -1, 1, 1, -1, -1, 1, 1, 1, -1, 1, 1, -1, -1, #else { 1, -1, -1, 1, 1, -1, -1, 1, 1, -1, -1, 1, 1, -1, -1, #endif Modified: head/usr.bin/localedef/ctype.c ============================================================================== --- head/usr.bin/localedef/ctype.c Thu Oct 6 19:41:09 2016 (r306781) +++ head/usr.bin/localedef/ctype.c Thu Oct 6 19:46:43 2016 (r306782) @@ -407,9 +407,9 @@ dump_ctype(void) continue; } - if ((last_ct != NULL) && (last_ct->ctype == ctn->ctype)) { + if ((last_ct != NULL) && (last_ct->ctype == ctn->ctype) && + (last_ct->wc + 1 == wc)) { ct[rl.runetype_ext_nranges-1].max = wc; - last_ct = ctn; } else { rl.runetype_ext_nranges++; ct = realloc(ct, @@ -417,8 +417,8 @@ dump_ctype(void) ct[rl.runetype_ext_nranges - 1].min = wc; ct[rl.runetype_ext_nranges - 1].max = wc; ct[rl.runetype_ext_nranges - 1].map = ctn->ctype; - last_ct = ctn; } + last_ct = ctn; if (ctn->tolower == 0) { last_lo = NULL; } else if ((last_lo != NULL) &&