From owner-svn-src-stable-11@freebsd.org Thu Dec 6 11:49:54 2018 Return-Path: Delivered-To: svn-src-stable-11@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 19DAD1329015; Thu, 6 Dec 2018 11:49:54 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B41648D625; Thu, 6 Dec 2018 11:49:53 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 95858230D9; Thu, 6 Dec 2018 11:49:53 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id wB6Bnr8Y074996; Thu, 6 Dec 2018 11:49:53 GMT (envelope-from yuripv@FreeBSD.org) Received: (from yuripv@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id wB6BnqDW074992; Thu, 6 Dec 2018 11:49:52 GMT (envelope-from yuripv@FreeBSD.org) Message-Id: <201812061149.wB6BnqDW074992@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: yuripv set sender to yuripv@FreeBSD.org using -f From: Yuri Pankov Date: Thu, 6 Dec 2018 11:49:52 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r341630 - in stable/11: share/ctypedef tools/tools/locale tools/tools/locale/etc X-SVN-Group: stable-11 X-SVN-Commit-Author: yuripv X-SVN-Commit-Paths: in stable/11: share/ctypedef tools/tools/locale tools/tools/locale/etc X-SVN-Commit-Revision: 341630 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: B41648D625 X-Spamd-Result: default: False [-1.13 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-0.46)[-0.456,0]; NEURAL_HAM_SHORT(-0.61)[-0.609,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-0.07)[-0.070,0] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: svn-src-stable-11@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for only the 11-stable src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Dec 2018 11:49:54 -0000 Author: yuripv Date: Thu Dec 6 11:49:52 2018 New Revision: 341630 URL: https://svnweb.freebsd.org/changeset/base/341630 Log: MFC r339311, r339313: Restore some of the ctype definitions reported in the PR from pre-CLDR data, namely 0xE000-0xF8FF private use area, and 0xFF00-0xFFF half- and fullwidth punctuation. While here, update tools/tools/locale/README based on my experience rebuilding the locale data. PR: 225692 Reviewed by: bapt, cem (previous version) Differential Revision: https://reviews.freebsd.org/D17471 Modified: stable/11/share/ctypedef/en_US.UTF-8.src stable/11/tools/tools/locale/README stable/11/tools/tools/locale/etc/common.UTF-8.src stable/11/tools/tools/locale/etc/manual-input.UTF-8 (contents, props changed) Directory Properties: stable/11/ (props changed) Modified: stable/11/share/ctypedef/en_US.UTF-8.src ============================================================================== --- stable/11/share/ctypedef/en_US.UTF-8.src Thu Dec 6 10:53:11 2018 (r341629) +++ stable/11/share/ctypedef/en_US.UTF-8.src Thu Dec 6 11:49:52 2018 (r341630) @@ -6241,6 +6241,12 @@ graph ;...;;...; ********************************************************************** +* 0xE000 - 0xF8FF Private Use Area (from pre-CLDR data) +********************************************************************** + +graph ;...; + +********************************************************************** * 0xFB50 - 0xFDFF Arabic Presentation Forms (differential) ********************************************************************** @@ -6277,6 +6283,17 @@ punct ;...; ********************************************************************** blank + +********************************************************************** +* 0xFF00 - 0xFFFF Half- and Fullwidth Punctuation (from pre-CLDR data) +********************************************************************** + +punct ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...; ********************************************************************** * 0x10300 - 0x1032F Old Italic Modified: stable/11/tools/tools/locale/README ============================================================================== --- stable/11/tools/tools/locale/README Thu Dec 6 10:53:11 2018 (r341629) +++ stable/11/tools/tools/locale/README Thu Dec 6 11:49:52 2018 (r341630) @@ -2,17 +2,37 @@ To generate the locales: -Tools needed: java, perl and devel/p5-Tie-IxHash +Tools needed: + java (openjdk >= 8) + perl + converters/p5-Text-Iconv + devel/p5-Tie-IxHash + textproc/p5-XML-Parser -fetch cldr data from: http://cldr.unicode.org -extract in a directory ~/unicode/cldr/v27.0.1 for example -fetch unidata from http://www.unicode.org/Public/zipped/ (latest version) -extract in a directory ~/unicode/UNIDATA/8.0.0 for example +Fetch CLDR data from: http://unicode.org/Public/cldr/. You need all of the +core.zip, keyboards.zip, and tools.zip. -either modify tools/tools/locales/etc/unicode.conf or export variables: -CLDRDIR="~/unicode/cldr/v27.0.1" -UNIDATADIR="~/unicode/UNIDATA/8.0.0" +Extract: + mkdir -p ~/unicode/cldr/v33.0 + cd ~/unicode/cldr/v33.0 + unzip ~/core.zip ~/keyboards.zip ~/tools.zip -run: -make POSIX -make install +Fetch unidata (UCD.zip) from http://www.unicode.org/Public/zipped/latest. + +Extract: + mkdir -p ~/unicode/UNIDATA/11.0.0 + cd ~/unicode/UNIDATA/11.0.0 + unzip ~/UCD.zip + +Either modify tools/tools/locales/etc/unicode.conf or export variables: + CLDRDIR=~/unicode/cldr/v33.0; export CLDRDIR + UNIDATADIR=~/unicode/UNIDATA/9.0.0; export UNIDATADIR + +Build the CLDR tools: + cd $CLDRDIR/tools/java + ant jar + +Run: + make POSIX + make + make install Modified: stable/11/tools/tools/locale/etc/common.UTF-8.src ============================================================================== --- stable/11/tools/tools/locale/etc/common.UTF-8.src Thu Dec 6 10:53:11 2018 (r341629) +++ stable/11/tools/tools/locale/etc/common.UTF-8.src Thu Dec 6 11:49:52 2018 (r341630) @@ -6241,6 +6241,12 @@ graph ;...;;...; ********************************************************************** +* 0xE000 - 0xF8FF Private Use Area (from pre-CLDR data) +********************************************************************** + +graph ;...; + +********************************************************************** * 0xFB50 - 0xFDFF Arabic Presentation Forms (differential) ********************************************************************** @@ -6277,6 +6283,17 @@ punct ;...; ********************************************************************** blank + +********************************************************************** +* 0xFF00 - 0xFFFF Half- and Fullwidth Punctuation (from pre-CLDR data) +********************************************************************** + +punct ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...; ********************************************************************** * 0x10300 - 0x1032F Old Italic Modified: stable/11/tools/tools/locale/etc/manual-input.UTF-8 ============================================================================== --- stable/11/tools/tools/locale/etc/manual-input.UTF-8 Thu Dec 6 10:53:11 2018 (r341629) +++ stable/11/tools/tools/locale/etc/manual-input.UTF-8 Thu Dec 6 11:49:52 2018 (r341630) @@ -877,6 +877,12 @@ graph ;...;;...; ********************************************************************** +* 0xE000 - 0xF8FF Private Use Area (from pre-CLDR data) +********************************************************************** + +graph ;...; + +********************************************************************** * 0xFB50 - 0xFDFF Arabic Presentation Forms (differential) ********************************************************************** @@ -913,6 +919,17 @@ punct ;...; ********************************************************************** blank + +********************************************************************** +* 0xFF00 - 0xFFFF Half- and Fullwidth Punctuation (from pre-CLDR data) +********************************************************************** + +punct ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...;;/ + ;...; ********************************************************************** * 0x10300 - 0x1032F Old Italic