From owner-freebsd-current@FreeBSD.ORG Sat Apr 5 01:35:58 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ABA4C9EA for ; Sat, 5 Apr 2014 01:35:58 +0000 (UTC) Received: from mail-la0-f47.google.com (mail-la0-f47.google.com [209.85.215.47]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FB55B78 for ; Sat, 5 Apr 2014 01:35:57 +0000 (UTC) Received: by mail-la0-f47.google.com with SMTP id pn19so3057691lab.34 for ; Fri, 04 Apr 2014 18:35:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=TNpNtlmDbysLXSY7THx2/+Np1fr8YT8omJNfkifmUyQ=; b=D0Urvz+TLaXKTpHFUlkhvWVHEI9aUmd4kQzALtvVT4kOZevAJuEx/GMw2eEXJtBaLv CYJTg6cVr1D3X3LK8SgXK+tKbkOPSVhv87D6Gk3ceenWoK9/aXam3bbETLrN+lh9T645 Wn8WiG3SyeYgcWRp/JUiAvnfy4aKgr5d2ctMNgqPlkVVQBPfom9jodhNVR7GXUdB7rKN SE00Nvt2bs+dcWqu5DbDAoIYTs6UnWSDH7uvwCebx+yTAJkv3Sk+ZF7r4UOaPrZpTBhD 9CKALEpTDYcTjU/8EHM653MhTW08vx+yhzawk+9AMvAwDLp5yzcSyfvQToiosVmSBwUK 7Iww== X-Gm-Message-State: ALoCoQlqwSQcxzuMGg4asNVHzg9zZh3Bb84lne/5YiRUczNVMbHGWcS39w1I4aO5seW+eSfUyFdS X-Received: by 10.112.142.105 with SMTP id rv9mr538987lbb.42.1396661749533; Fri, 04 Apr 2014 18:35:49 -0700 (PDT) Received: from [192.168.1.2] ([89.169.173.68]) by mx.google.com with ESMTPSA id n9sm6748859lbg.6.2014.04.04.18.35.48 for (version=TLSv1.2 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 04 Apr 2014 18:35:48 -0700 (PDT) Message-ID: <533F5DF5.9020803@freebsd.org> Date: Sat, 05 Apr 2014 05:35:49 +0400 From: Andrey Chernov User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: Gleb Smirnoff , i18n@freebsd.org Subject: Re: login.conf --> UTF-8 References: <1396457629.2280.2.camel@powernoodle.corp.yahoo.com> <20140402171546.GL44326@FreeBSD.org> <533C8269.7040305@freebsd.org> <20140404124634.GC44326@glebius.int.ru> In-Reply-To: <20140404124634.GC44326@glebius.int.ru> X-Enigmail-Version: 1.7a1pre Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: "freebsd-current@freebsd.org" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Apr 2014 01:35:58 -0000 On 04.04.2014 16:46, Gleb Smirnoff wrote: > On Thu, Apr 03, 2014 at 01:34:33AM +0400, Andrey Chernov wrote: > A> On 02.04.2014 21:15, Gleb Smirnoff wrote: > A> > S> + :lang=en_US.UTF-8:\ > A> > S> + :charset=UTF-8: > A> > > A> > And I'd like to do same change for the 'russian' login class > A> > in /etc/login.conf. > A> > A> Please everybody remember that we don't have UTF-8 collation > A> implemented, just fallback to bytecode comparison. > > Any objections on checking in FreeBSD-compatible[1] UTF-8 collation > implementation from Alex Tutubalin? > > http://blog.lexa.ru/2008/03/03/freebsd_utf8_russian_collate_vtoraja_popitka.html > Even his "version 2" have my objections. I already reply Alex about this in 2008. In short: 1) It is error there: almost all single chars above ASCII should be "chains", i.t. two bytes minimum, since there almost no intersections with ISO8859-1 as UTF-8 subset. 2) The table itself is very incomplete, f.e. not covering either whole KOI8-R, nor ISO8859-5, nor CP866. It is made from CP1251 with all its restrictions. So, switching from f.e. KOI8-R to UTF-8 will cause sorting regression. Russian UTF-8 collation should be able to sort all major Russian charsets mentioned, i.e. we need combined table. 3) "charmap map.ISO8859-1" declaration is missing (needed mainly for using pure ASCII chars mnemonic names). Even in case above mentioned errors will be removed and the code will be committed afterwards, we should understand that this way (implementing multibyte collation via single byte one) even while being possible is a big hack and slowing sorting down up to 10 times. Proper "Unicode collation algorithm" is already implemented by ICU and other projects. See http://unicode.org/reports/tr10/ It will be better if someone adopt it instead of hacks. -- http://ache.vniz.net/