From owner-freebsd-stable@freebsd.org Sun Nov 6 20:57:04 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF642C33E7A for ; Sun, 6 Nov 2016 20:57:04 +0000 (UTC) (envelope-from stb@lassitu.de) Received: from gilb.zs64.net (gilb.zs64.net [IPv6:2a00:14b0:4200:32e0::1ea]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gilb.zs64.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 53B37ED5; Sun, 6 Nov 2016 20:57:04 +0000 (UTC) (envelope-from stb@lassitu.de) Received: by gilb.zs64.net (Postfix, from stb@lassitu.de) id 9B2C61E21F7; Sun, 6 Nov 2016 20:57:01 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 10.1 \(3251\)) Subject: Re: Uppercase RE matching problems in FreeBSD 11 From: Stefan Bethke In-Reply-To: <20161106110729.z2px7mzlhcwxvrvu@ivaldir.etoilebsd.net> Date: Sun, 6 Nov 2016 21:57:00 +0100 Cc: Greg Rivers , freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <29451103-E8DB-4656-A5BB-AEB924A728D6@lassitu.de> References: <20161106110729.z2px7mzlhcwxvrvu@ivaldir.etoilebsd.net> To: Baptiste Daroussin X-Mailer: Apple Mail (2.3251) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Nov 2016 20:57:04 -0000 > Am 06.11.2016 um 12:07 schrieb Baptiste Daroussin : >=20 > On Sat, Nov 05, 2016 at 08:23:25PM -0500, Greg Rivers wrote: >> I happened to run an old script today that uses sed(1) to extract the = system >> boot time from the kern.boottime sysctl MIB. On 11.0 this no longer = works as >> expected: >>=20 >> $ sysctl kern.boottime >> kern.boottime: { sec =3D 1478380714, usec =3D 145351 } Sat Nov 5 = 16:18:34 2016 >> $ sysctl kern.boottime | sed -e 's/.*\([A-Z].*\)$/\1/' >> v 5 16:18:34 2016 >>=20 >> sed passes over 'S' and 'N' until it hits 'v', which it considers = uppercase >> apparently. This is with LANG=3Den_US.UTF-8. If I set LANG=3DC, it = works as >> expected: >>=20 >> $ sysctl kern.boottime | LANG=3DC sed -e 's/.*\([A-Z].*\)$/\1/' >> Nov 5 16:18:34 2016 >>=20 >> Testing every lowercase character separately gives even more = inconsistent >> results: >>=20 >> $ cat <> Here sed thinks every lowercase character except for 'a' is = uppercase! This >> differs from the first test where sed did not think 'o' is uppercase. = Again, >> the above behaves as expected with LANG=3DC. >>=20 >> Does anyone have any insight into this? This is likely to break a lot = of >> existing code. >>=20 >=20 > Yes A-Z only means uppercase in an ASCII only world in a unicode world = it means > AaBb... Z because there are way more characters that simple A-Z. In = FreeBSD 11 > we have a unicode collation instead of falling back in on LC_COLLATE=3DC= which > means ascii only >=20 > For regrexp for example one should use the classes: :upper: or = :lower:. That is rather surprising. Is there a normative reference for the = treatment of bracket expressions and character classes when using = locales other than C and/or encodings like UTF-8? Stefan --=20 Stefan Bethke Fon +49 151 14070811