From owner-svn-src-all@freebsd.org Sun Dec 13 22:26:03 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7EFD94C660D; Sun, 13 Dec 2020 22:26:03 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CvJyJ59F3z3Kgh; Sun, 13 Dec 2020 22:26:00 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BD8611757C; Sun, 13 Dec 2020 22:25:55 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0BDMPtes051312; Sun, 13 Dec 2020 22:25:55 GMT (envelope-from yuripv@FreeBSD.org) Received: (from yuripv@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0BDMPss0051309; Sun, 13 Dec 2020 22:25:54 GMT (envelope-from yuripv@FreeBSD.org) Message-Id: <202012132225.0BDMPss0051309@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: yuripv set sender to yuripv@FreeBSD.org using -f From: Yuri Pankov Date: Sun, 13 Dec 2020 22:25:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r368619 - in stable/12/tools/tools/locale: . etc/final-maps tools X-SVN-Group: stable-12 X-SVN-Commit-Author: yuripv X-SVN-Commit-Paths: in stable/12/tools/tools/locale: . etc/final-maps tools X-SVN-Commit-Revision: 368619 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 13 Dec 2020 22:26:05 -0000 Author: yuripv Date: Sun Dec 13 22:25:54 2020 New Revision: 368619 URL: https://svnweb.freebsd.org/changeset/base/368619 Log: MFC r368390: update wcwidth data from utf8proc Character width data being out of date is a constant source of weird rendering issues and wasted time trying to diagnose those, e.g. as reported by Jeremy Chadwick: https://gitlab.com/muttmua/mutt/-/issues/67 Sadly, there is no real ("standard") wcwidth data source, so this tries to rectify the problem using the utf8proc one (through its C API) which would hopefully benefeat both FreeBSD and utf8proc through bug reports (if any). PR: 251767 Added: stable/12/tools/tools/locale/tools/getwidths.c - copied unchanged from r368390, head/tools/tools/locale/tools/getwidths.c stable/12/tools/tools/locale/tools/mkwidths.pl - copied unchanged from r368390, head/tools/tools/locale/tools/mkwidths.pl Modified: stable/12/tools/tools/locale/Makefile stable/12/tools/tools/locale/README stable/12/tools/tools/locale/etc/final-maps/widths.txt (contents, props changed) Directory Properties: stable/12/ (props changed) Modified: stable/12/tools/tools/locale/Makefile ============================================================================== --- stable/12/tools/tools/locale/Makefile Sun Dec 13 21:32:20 2020 (r368618) +++ stable/12/tools/tools/locale/Makefile Sun Dec 13 22:25:54 2020 (r368619) @@ -224,7 +224,7 @@ posix/${enc}.cm: .for area in ${BASE_LOCALES_OF_INTEREST} posixsrc: build-tools posix/${area}.UTF-8.src .ORDER: build-tools posix/${area}.UTF-8.src -posix/${area}.UTF-8.src: +posix/${area}.UTF-8.src: mkdir -p posix && \ ${JAVA_CLDR} org.unicode.cldr.posix.GeneratePOSIX \ -d posix -m ${area} -c UTF-8 @@ -237,5 +237,21 @@ posix/${area}.${encoding}.src: ${JAVA_CLDR} org.unicode.cldr.posix.GeneratePOSIX \ -d posix -m ${area} -c ${encoding} .endfor + +# generate widths.txt using the data from libut8proc +GETWIDTHS=${TOOLSDIR}/getwidths +MKWIDTHS=${TOOLSDIR}/mkwidths.pl +WIDTHS= ${ETCDIR}/final-maps/widths.txt + +U8CFLAGS!=pkgconf --cflags libutf8proc +U8LIBS!=pkgconf --libs libutf8proc +CFLAGS+=${U8CFLAGS} +LDFLAGS+=${U8LIBS} + +CLEANFILES+=${TOOLSDIR}/getwidths + +widths: ${WIDTHS} +${WIDTHS}: posixcm ${GETWIDTHS} + ${GETWIDTHS} | ${MKWIDTHS} ${.OBJDIR}/posix/UTF-8.cm ${.TARGET} .include Modified: stable/12/tools/tools/locale/README ============================================================================== --- stable/12/tools/tools/locale/README Sun Dec 13 21:32:20 2020 (r368618) +++ stable/12/tools/tools/locale/README Sun Dec 13 22:25:54 2020 (r368619) @@ -55,4 +55,8 @@ Targets: make install Install the build results into $LOCALESRCDIR. + + make widths + Generate widths.txt. Requires pkgconf and utf8proc + packages to be installed. [EOF] Modified: stable/12/tools/tools/locale/etc/final-maps/widths.txt ============================================================================== --- stable/12/tools/tools/locale/etc/final-maps/widths.txt Sun Dec 13 21:32:20 2020 (r368618) +++ stable/12/tools/tools/locale/etc/final-maps/widths.txt Sun Dec 13 22:25:54 2020 (r368619) @@ -1,26 +1,73 @@ -# -# Width Table. Credit for the input into this table, which is derived from -# the Unicode standards, is due to Markus Kuhn's implementation of wcwidth -# which can be found at http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c -# -# Some key differences: The numbers here are translated to the values found -# in the UTF-8 character map, and where symbolic names are missing for -# values from that table, we have omitted the values here. This means that -# characters which lack a symbolic name but are not 1 screen column wide -# are going to be mis-handled. This emphasizes the importance of having -# symbolic names for all characters that are to be handled properly. -# -# Also, to support use with different encodings, we avoid the use of ranges -# as some encodings may not have the same contiguous ranges as stock Unicode. -# -# This table must be processed *after* the charmap, as the symbolic names -# need to be defined there. -# -# Everything after this comment was generated automatically using the -# the mkwidths.py python script. To make corrections, fix the widths-0.txt -# or widths-2.txt files, and re-run mkwidths.py. But preserve this header. -# +# Warning: Do not edit. This file is automatically generated from the +# tools in /usr/src/tools/tools/locale. The data is obtained from the +# utf8proc 2.5.0. +# ----------------------------------------------------------------------------- WIDTH + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 @@ -137,6 +184,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -194,12 +242,20 @@ WIDTH 0 0 0 + 0 + 0 0 0 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 @@ -220,6 +276,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -229,7 +286,6 @@ WIDTH 0 0 0 - 0 0 0 0 @@ -291,9 +347,86 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 + 0 + 0 + 0 0 + 0 + 0 + 0 0 0 0 @@ -302,25 +435,49 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 0 + 0 + 0 0 0 0 0 + 0 + 0 + 0 0 0 0 + 0 + 0 0 + 0 + 0 + 0 0 0 0 0 + 0 + 0 + 0 + 0 0 + 0 0 0 + 0 0 0 + 0 0 + 0 + 0 + 0 0 0 0 @@ -328,11 +485,17 @@ WIDTH 0 0 0 + 0 0 0 + 0 0 0 + 0 0 + 0 + 0 + 0 0 0 0 @@ -340,23 +503,64 @@ WIDTH 0 0 0 + 0 + 0 + 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 0 + 0 0 + 0 0 0 0 + 0 + 0 + 0 + 0 + 0 0 0 + 0 + 0 + 0 0 + 0 + 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 + 0 + 0 + 0 + 0 0 0 0 @@ -366,22 +570,73 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 + 0 0 0 + 0 + 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 @@ -405,6 +660,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -418,6 +674,8 @@ WIDTH 0 0 0 + 0 + 0 0 0 0 @@ -432,6 +690,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -439,6 +698,9 @@ WIDTH 0 0 0 + 0 + 0 + 0 0 0 0 @@ -484,176 +746,162 @@ WIDTH 0 0 0 + 0 + 0 0 0 0 0 + 0 0 + 0 + 0 + 0 0 0 + 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 0 + 0 0 0 0 @@ -667,6 +915,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -674,7 +923,17 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 0 0 0 @@ -690,30 +949,116 @@ WIDTH 0 0 0 + 0 + 0 + 0 0 0 0 0 + 0 + 0 + 0 + 0 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 0 + 0 0 + 0 0 0 0 0 0 + 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 0 0 0 @@ -723,6 +1068,85 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***