From nobody Tue Jul 5 11:58:11 2022 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 79E521CFE87E; Tue, 5 Jul 2022 11:58:12 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Lch5J1y4hz4ty1; Tue, 5 Jul 2022 11:58:12 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1657022292; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3edxidsbJkaz03mjwPYjZvSmXssHxC/q5Cn2Cu9ndtk=; b=IL+FeezEWkjTiWM6OrWAjI/HTT07CRT/3hIk7l7PpesQE0WTzTkHXSQiPxh/EC/f8eiAat pePF1jo9HKL/3eZiA3WIO61GN5G4rNJqAqyufpDmcEljTFYVbZFItRMO3BfYEIA1V4HwnD davkKPMaOdbxuJbzFjpyuzGrahzkP56BYHBIuvhv636Qo2lvlCueionLS1greraAOjm5QD GvyzwfPOwIXbPLuzJcaWV5Th+Ejn/1vxDTrs6uWE8A37BsmqEhM4POmMg0fVFoShbIb5XM hfuXt86eILl0lJg6r+Q+wWXg0BjqBOKCxN+xrhsbFvayOinvUNdmVCz6M71kKA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 1086924FB3; Tue, 5 Jul 2022 11:58:12 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 265BwB3j012182; Tue, 5 Jul 2022 11:58:11 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 265BwBmB012181; Tue, 5 Jul 2022 11:58:11 GMT (envelope-from git) Date: Tue, 5 Jul 2022 11:58:11 GMT Message-Id: <202207051158.265BwBmB012181@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Hans Petter Selasky Subject: git: d4dd3c8f8b5b - stable/13 - vt: Improve multi lingual word separation. List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: hselasky X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: d4dd3c8f8b5b050a24da6fc2b885c301843214a5 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1657022292; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=3edxidsbJkaz03mjwPYjZvSmXssHxC/q5Cn2Cu9ndtk=; b=g8cALH/4UOSxANiUGc2J+KLphNWV2cgzREBLVQzFnOlbnXp1gHF+HQyVWdXOBWMdMTJ//m awNyqE6GJwD8RrWRWK5sK4Py+fzPQzMdmEb2b8M9qFalm/1Cia4dOUPWaI6x1ndZp8HiIJ snNMlHiQ0yAAAYowI10sz5FsxmyTCCqViJiD8nKUmdbtBADWmnLJUoAGs2rAHDnhGPPqmq kHESKcIFwrZC/gUXn9ta8njzeocYB6NSeI/3ZwJaS90o5zMZCBaoZQT8SFau6V5qOfQuRk 2ubaUkj1mCTegdccLSNr0YIkUliBF0AwIOgIZ9ywpI2eGZIwdCX696uaJSWBgQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1657022292; a=rsa-sha256; cv=none; b=BQhWgAtCmRWXi+9nAZupnVJW/eglEQwtQDlgv51lkV1SMqnvW+vbCcA0WMaiPv/bbzH0ar hMOsDwrnZEWGUWEgCqf3dPBgfEmd1JBZjGgiyvnlKMADN+64MWqQBP6G8Eov95Ruh56Qvy ir0L/nWVNWH0LwkK2l6QU8O4kus1rvFU0+Sqy/QSjetvNcMRmo5zIzZGQ6RbW+nQXv7bFf Ve0xG+av8dEiqr/Nl2v7c183x7Dzd73ATWZP3K3DN1iU+Nv8+Icve0tBUfP3aGDvf9p4b2 hLVLDw/2KmUthfNZkJjGnxEAYmQhgllgtrHHqzGa4IizQ2g6uEqQe11FOBtYXg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by hselasky: URL: https://cgit.FreeBSD.org/src/commit/?id=d4dd3c8f8b5b050a24da6fc2b885c301843214a5 commit d4dd3c8f8b5b050a24da6fc2b885c301843214a5 Author: Hans Petter Selasky AuthorDate: 2022-06-25 09:17:44 +0000 Commit: Hans Petter Selasky CommitDate: 2022-07-05 11:42:28 +0000 vt: Improve multi lingual word separation. Suggested by: Tomoaki AOKI Differential Revision: https://reviews.freebsd.org/D35552 PR: 263084 Sponsored by: NVIDIA Networking (cherry picked from commit 9971e6aff1bef3d456172c41a3df3ce7266517cf) --- sys/dev/vt/vt_buf.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/sys/dev/vt/vt_buf.c b/sys/dev/vt/vt_buf.c index fa6c7c8fec5f..b83db85f1cdb 100644 --- a/sys/dev/vt/vt_buf.c +++ b/sys/dev/vt/vt_buf.c @@ -747,6 +747,29 @@ vtbuf_get_marked_len(struct vt_buf *vb) return (sz * sizeof(term_char_t)); } +static bool +tchar_is_word_separator(term_char_t ch) +{ + /* List of unicode word separator characters: */ + switch (TCHAR_CHARACTER(ch)) { + case 0x0020: /* SPACE */ + case 0x180E: /* MONGOLIAN VOWEL SEPARATOR */ + case 0x2002: /* EN SPACE (nut) */ + case 0x2003: /* EM SPACE (mutton) */ + case 0x2004: /* THREE-PER-EM SPACE (thick space) */ + case 0x2005: /* FOUR-PER-EM SPACE (mid space) */ + case 0x2006: /* SIX-PER-EM SPACE */ + case 0x2008: /* PUNCTUATION SPACE */ + case 0x2009: /* THIN SPACE */ + case 0x200A: /* HAIR SPACE */ + case 0x200B: /* ZERO WIDTH SPACE */ + case 0x3000: /* IDEOGRAPHIC SPACE */ + return (true); + default: + return (false); + } +} + void vtbuf_extract_marked(struct vt_buf *vb, term_char_t *buf, int sz) { @@ -779,7 +802,7 @@ vtbuf_extract_marked(struct vt_buf *vb, term_char_t *buf, int sz) if (r != e.tp_row) { /* Trim trailing word separators, if any. */ for (; i != j; i--) { - if (TCHAR_CHARACTER(buf[i - 1]) != ' ') + if (!tchar_is_word_separator(buf[i - 1])) break; } /* Add newline character as expected by TTY. */ @@ -824,7 +847,7 @@ vtbuf_set_mark(struct vt_buf *vb, int type, int col, int row) vtbuf_wth(vb, row); r = vb->vb_rows[vb->vb_mark_start.tp_row]; for (i = col; i >= 0; i --) { - if (TCHAR_CHARACTER(r[i]) == ' ') { + if (tchar_is_word_separator(r[i])) { vb->vb_mark_start.tp_col = i + 1; break; } @@ -833,7 +856,7 @@ vtbuf_set_mark(struct vt_buf *vb, int type, int col, int row) if (i == -1) vb->vb_mark_start.tp_col = 0; for (i = col; i < vb->vb_scr_size.tp_col; i++) { - if (TCHAR_CHARACTER(r[i]) == ' ') { + if (tchar_is_word_separator(r[i])) { vb->vb_mark_end.tp_col = i; break; }