From owner-svn-src-head@freebsd.org Sun Dec 6 16:44:42 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F11334A23C2; Sun, 6 Dec 2020 16:44:42 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cpsjk3mBBz4Wvk; Sun, 6 Dec 2020 16:44:42 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6FD3B13611; Sun, 6 Dec 2020 16:44:42 +0000 (UTC) (envelope-from yuripv@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 0B6Gig5J005947; Sun, 6 Dec 2020 16:44:42 GMT (envelope-from yuripv@FreeBSD.org) Received: (from yuripv@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 0B6GifBe005944; Sun, 6 Dec 2020 16:44:41 GMT (envelope-from yuripv@FreeBSD.org) Message-Id: <202012061644.0B6GifBe005944@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: yuripv set sender to yuripv@FreeBSD.org using -f From: Yuri Pankov Date: Sun, 6 Dec 2020 16:44:41 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r368390 - in head/tools/tools/locale: . etc/final-maps tools X-SVN-Group: head X-SVN-Commit-Author: yuripv X-SVN-Commit-Paths: in head/tools/tools/locale: . etc/final-maps tools X-SVN-Commit-Revision: 368390 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Dec 2020 16:44:43 -0000 Author: yuripv Date: Sun Dec 6 16:44:41 2020 New Revision: 368390 URL: https://svnweb.freebsd.org/changeset/base/368390 Log: update wcwidth data from utf8proc Character width data being out of date is a constant source of weird rendering issues and wasted time trying to diagnose those, e.g. as reported by Jeremy Chadwick: https://gitlab.com/muttmua/mutt/-/issues/67 Sadly, there is no real ("standard") wcwidth data source, so this tries to rectify the problem using the utf8proc one (through its C API) which would hopefully benefeat both FreeBSD and utf8proc through bug reports (if any). Reviewed by: bapt Differential Revision: https://reviews.freebsd.org/D27259 Added: head/tools/tools/locale/tools/getwidths.c (contents, props changed) head/tools/tools/locale/tools/mkwidths.pl (contents, props changed) Modified: head/tools/tools/locale/Makefile head/tools/tools/locale/README head/tools/tools/locale/etc/final-maps/widths.txt (contents, props changed) Modified: head/tools/tools/locale/Makefile ============================================================================== --- head/tools/tools/locale/Makefile Sun Dec 6 16:22:26 2020 (r368389) +++ head/tools/tools/locale/Makefile Sun Dec 6 16:44:41 2020 (r368390) @@ -225,7 +225,7 @@ posix/${enc}.cm: .for area in ${BASE_LOCALES_OF_INTEREST} posixsrc: build-tools posix/${area}.UTF-8.src .ORDER: build-tools posix/${area}.UTF-8.src -posix/${area}.UTF-8.src: +posix/${area}.UTF-8.src: mkdir -p posix && \ ${JAVA_CLDR} org.unicode.cldr.posix.GeneratePOSIX \ -d posix -m ${area} -c UTF-8 @@ -238,5 +238,21 @@ posix/${area}.${encoding}.src: ${JAVA_CLDR} org.unicode.cldr.posix.GeneratePOSIX \ -d posix -m ${area} -c ${encoding} .endfor + +# generate widths.txt using the data from libut8proc +GETWIDTHS=${TOOLSDIR}/getwidths +MKWIDTHS=${TOOLSDIR}/mkwidths.pl +WIDTHS= ${ETCDIR}/final-maps/widths.txt + +U8CFLAGS!=pkgconf --cflags libutf8proc +U8LIBS!=pkgconf --libs libutf8proc +CFLAGS+=${U8CFLAGS} +LDFLAGS+=${U8LIBS} + +CLEANFILES+=${TOOLSDIR}/getwidths + +widths: ${WIDTHS} +${WIDTHS}: posixcm ${GETWIDTHS} + ${GETWIDTHS} | ${MKWIDTHS} ${.OBJDIR}/posix/UTF-8.cm ${.TARGET} .include Modified: head/tools/tools/locale/README ============================================================================== --- head/tools/tools/locale/README Sun Dec 6 16:22:26 2020 (r368389) +++ head/tools/tools/locale/README Sun Dec 6 16:44:41 2020 (r368390) @@ -55,4 +55,8 @@ Targets: make install Install the build results into $LOCALESRCDIR. + + make widths + Generate widths.txt. Requires pkgconf and utf8proc + packages to be installed. [EOF] Modified: head/tools/tools/locale/etc/final-maps/widths.txt ============================================================================== --- head/tools/tools/locale/etc/final-maps/widths.txt Sun Dec 6 16:22:26 2020 (r368389) +++ head/tools/tools/locale/etc/final-maps/widths.txt Sun Dec 6 16:44:41 2020 (r368390) @@ -1,26 +1,73 @@ -# -# Width Table. Credit for the input into this table, which is derived from -# the Unicode standards, is due to Markus Kuhn's implementation of wcwidth -# which can be found at http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c -# -# Some key differences: The numbers here are translated to the values found -# in the UTF-8 character map, and where symbolic names are missing for -# values from that table, we have omitted the values here. This means that -# characters which lack a symbolic name but are not 1 screen column wide -# are going to be mis-handled. This emphasizes the importance of having -# symbolic names for all characters that are to be handled properly. -# -# Also, to support use with different encodings, we avoid the use of ranges -# as some encodings may not have the same contiguous ranges as stock Unicode. -# -# This table must be processed *after* the charmap, as the symbolic names -# need to be defined there. -# -# Everything after this comment was generated automatically using the -# the mkwidths.py python script. To make corrections, fix the widths-0.txt -# or widths-2.txt files, and re-run mkwidths.py. But preserve this header. -# +# Warning: Do not edit. This file is automatically generated from the +# tools in /usr/src/tools/tools/locale. The data is obtained from the +# utf8proc 2.5.0. +# ----------------------------------------------------------------------------- WIDTH + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 @@ -137,6 +184,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -194,12 +242,20 @@ WIDTH 0 0 0 + 0 + 0 0 0 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 @@ -220,6 +276,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -229,7 +286,6 @@ WIDTH 0 0 0 - 0 0 0 0 @@ -291,9 +347,86 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 + 0 + 0 + 0 0 + 0 + 0 + 0 0 0 0 @@ -302,25 +435,49 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 0 + 0 + 0 0 0 0 0 + 0 + 0 + 0 0 0 0 + 0 + 0 0 + 0 + 0 + 0 0 0 0 0 + 0 + 0 + 0 + 0 0 + 0 0 0 + 0 0 0 + 0 0 + 0 + 0 + 0 0 0 0 @@ -328,11 +485,17 @@ WIDTH 0 0 0 + 0 0 0 + 0 0 0 + 0 0 + 0 + 0 + 0 0 0 0 @@ -340,23 +503,64 @@ WIDTH 0 0 0 + 0 + 0 + 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 0 + 0 0 + 0 0 0 0 + 0 + 0 + 0 + 0 + 0 0 0 + 0 + 0 + 0 0 + 0 + 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 + 0 + 0 + 0 + 0 0 0 0 @@ -366,22 +570,73 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 + 0 0 0 + 0 + 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 @@ -405,6 +660,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -418,6 +674,8 @@ WIDTH 0 0 0 + 0 + 0 0 0 0 @@ -432,6 +690,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -439,6 +698,9 @@ WIDTH 0 0 0 + 0 + 0 + 0 0 0 0 @@ -484,176 +746,162 @@ WIDTH 0 0 0 + 0 + 0 0 0 0 0 + 0 0 + 0 + 0 + 0 0 0 + 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 - 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 2 + 0 + 0 0 0 0 @@ -667,6 +915,7 @@ WIDTH 0 0 0 + 0 0 0 0 @@ -674,7 +923,17 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 0 0 0 @@ -690,30 +949,116 @@ WIDTH 0 0 0 + 0 + 0 + 0 0 0 0 0 + 0 + 0 + 0 + 0 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 0 0 0 0 + 0 0 + 0 0 0 0 0 0 + 0 0 + 0 + 0 + 0 + 0 + 0 0 + 0 + 0 0 0 0 @@ -723,6 +1068,85 @@ WIDTH 0 0 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 + 0 *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***