From owner-svn-src-all@freebsd.org Wed Jan 16 12:04:32 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 011171485E6C; Wed, 16 Jan 2019 12:04:32 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 246456A980; Wed, 16 Jan 2019 12:04:31 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8CB6C1EC9; Wed, 16 Jan 2019 12:04:30 +0000 (UTC) (envelope-from bapt@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x0GC4U5N019202; Wed, 16 Jan 2019 12:04:30 GMT (envelope-from bapt@FreeBSD.org) Received: (from bapt@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x0GC4TGm019196; Wed, 16 Jan 2019 12:04:29 GMT (envelope-from bapt@FreeBSD.org) Message-Id: <201901161204.x0GC4TGm019196@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: bapt set sender to bapt@FreeBSD.org using -f From: Baptiste Daroussin Date: Wed, 16 Jan 2019 12:04:29 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-11@freebsd.org Subject: svn commit: r343091 - stable/11/lib/libedit X-SVN-Group: stable-11 X-SVN-Commit-Author: bapt X-SVN-Commit-Paths: stable/11/lib/libedit X-SVN-Commit-Revision: 343091 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 246456A980 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.96)[-0.965,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2019 12:04:32 -0000 Author: bapt Date: Wed Jan 16 12:04:29 2019 New Revision: 343091 URL: https://svnweb.freebsd.org/changeset/base/343091 Log: MFC r340933: libedit: improve multibyte support Until this commit libedit only supported UTF-8 for multibyte charset Improve it to support other multibyte charsets Tested with eucJP and SJIS charsets. Note that this change as been review and committed in upstream libedit as well via christos@NetBSD Submitted by: naito.yuichiro _at_ gmail.com Reviewed by: bapt, pfg, yuripv, 0mp Differential Revision: https://reviews.freebsd.org/D17903 Modified: stable/11/lib/libedit/chartype.c stable/11/lib/libedit/chartype.h stable/11/lib/libedit/el.c stable/11/lib/libedit/el.h stable/11/lib/libedit/read.c Directory Properties: stable/11/ (props changed) Modified: stable/11/lib/libedit/chartype.c ============================================================================== --- stable/11/lib/libedit/chartype.c Wed Jan 16 12:04:24 2019 (r343090) +++ stable/11/lib/libedit/chartype.c Wed Jan 16 12:04:29 2019 (r343091) @@ -37,6 +37,7 @@ __RCSID("$NetBSD: chartype.c,v 1.23 2016/02/28 23:02:2 __FBSDID("$FreeBSD$"); #include +#include #include #include @@ -182,17 +183,13 @@ ct_decode_argv(int argc, const char *argv[], ct_buffer protected size_t ct_enc_width(Char c) { - /* UTF-8 encoding specific values */ - if (c < 0x80) - return 1; - else if (c < 0x0800) - return 2; - else if (c < 0x10000) - return 3; - else if (c < 0x110000) - return 4; - else - return 0; /* not a valid codepoint */ + mbstate_t ps = (mbstate_t){{0}}; + size_t len; + char cbuf[MB_LEN_MAX]; + len = ct_wcrtomb(cbuf, c, &ps); + if (len == (size_t)-1) + return (0); + return (len); } protected ssize_t Modified: stable/11/lib/libedit/chartype.h ============================================================================== --- stable/11/lib/libedit/chartype.h Wed Jan 16 12:04:24 2019 (r343090) +++ stable/11/lib/libedit/chartype.h Wed Jan 16 12:04:29 2019 (r343091) @@ -56,6 +56,7 @@ #define ct_wctob wctob #define ct_wctomb wctomb +#define ct_wcrtomb wcrtomb #define ct_wctomb_reset wctomb(0,0) #define ct_wcstombs wcstombs #define ct_mbstowcs mbstowcs @@ -109,6 +110,7 @@ Width(wchar_t c) #define ct_wctob(w) ((int)(w)) #define ct_wctomb error +#define ct_wcrtomb error #define ct_wctomb_reset #define ct_wcstombs(a, b, c) (strncpy(a, b, c), strlen(a)) #define ct_mbstowcs(a, b, c) (strncpy(a, b, c), strlen(a)) Modified: stable/11/lib/libedit/el.c ============================================================================== --- stable/11/lib/libedit/el.c Wed Jan 16 12:04:24 2019 (r343090) +++ stable/11/lib/libedit/el.c Wed Jan 16 12:04:29 2019 (r343091) @@ -99,10 +99,6 @@ el_init_fd(const char *prog, FILE *fin, FILE *fout, FI * Initialize all the modules. Order is important!!! */ el->el_flags = 0; - if (setlocale(LC_CTYPE, NULL) != NULL){ - if (strcmp(nl_langinfo(CODESET), "UTF-8") == 0) - el->el_flags |= CHARSET_IS_UTF8; - } if (terminal_init(el) == -1) { el_free(el->el_prog); @@ -293,7 +289,7 @@ FUN(el,set)(EditLine *el, int op, ...) void *ptr = va_arg(ap, void *); rv = hist_set(el, func, ptr); - if (!(el->el_flags & CHARSET_IS_UTF8)) + if (MB_CUR_MAX == 1) el->el_flags &= ~NARROW_HISTORY; break; } Modified: stable/11/lib/libedit/el.h ============================================================================== --- stable/11/lib/libedit/el.h Wed Jan 16 12:04:24 2019 (r343090) +++ stable/11/lib/libedit/el.h Wed Jan 16 12:04:29 2019 (r343091) @@ -56,7 +56,6 @@ #define NO_TTY 0x02 #define EDIT_DISABLED 0x04 #define UNBUFFERED 0x08 -#define CHARSET_IS_UTF8 0x10 #define NARROW_HISTORY 0x40 typedef unsigned char el_action_t; /* Index to command array */ Modified: stable/11/lib/libedit/read.c ============================================================================== --- stable/11/lib/libedit/read.c Wed Jan 16 12:04:24 2019 (r343090) +++ stable/11/lib/libedit/read.c Wed Jan 16 12:04:29 2019 (r343091) @@ -363,13 +363,7 @@ read_char(EditLine *el, wchar_t *cp) goto again; } case (size_t)-2: - /* - * We don't support other multibyte charsets. - * The second condition shouldn't happen - * and is here merely for additional safety. - */ - if ((el->el_flags & CHARSET_IS_UTF8) == 0 || - cbp >= MB_LEN_MAX) { + if (cbp >= MB_LEN_MAX) { errno = EILSEQ; *cp = L'\0'; return -1;