From nobody Sat Oct 21 14:29:33 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SCP3f1C11z4y559; Sat, 21 Oct 2023 14:29:34 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SCP3f0Xtnz4FmG; Sat, 21 Oct 2023 14:29:34 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697898574; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=eM6VAX6iW5nrMfI7ujgkgOc3GF5nTG1LvTfyWtEm/Zs=; b=GJK1sjXn6TfmqcbCwC1hhImbgkd90HxxII/SGV4usOKSpi+zXuymP3KA7LM2Th/KI+CfUR bW2Z2S29ZVXRfnuJcfUB4xCsW9uZ4V2YpY23H8/9EbxryeWL+YA+5BvOsl5OcpP4awzv0f AW/6WHNDNHIuHJz+4QhuUlr5+GNSI2lM4SLJNYJ4XyM+b8GKhrHWWtcazMFjWOm1ZEOvvJ uJIqincoqhiT+x4GdgVlJxfn5tKECV8DILhemOCyrPYHNftWoVFVhq4JnOszeRCGod3tOf T2fmXp4FMT7hOyQvixygEjFUyJs0vxv9ZAgjiMQOAgjKKWYA93swpYa+kGX7Iw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1697898574; a=rsa-sha256; cv=none; b=EXc7pJH9rh+pdlareh7wk4Z1fY55zHFJpYN5HE1pMqdCBjyh+Zw9drgNtouJJGj+aI6cLl HKuOMN9uREf0DFpLgQFcZK+pfY2YmsYj7UDs6YUPDFz4PG+rDW4ARr/PrUY1jHohyqfbkd sjZWNyB09zHoRoEIMg//2Yh/csZNx3eRp/6jrTnS/E2Ne1qVgeOh3dR4xt4naXA/SvomSz 6tfzz9FN0d5kU4/NvNCmgBCQleCGamILLfmILCzEdCwifO1mpt4ZWZfX6240Jz9lRskTsr xPDt5eVJnv2Eb2oA+Lf0F2JEyybNmuLwK5bYJlIxujgdwwZSn3Ztv1b/yQZ3mg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697898574; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=eM6VAX6iW5nrMfI7ujgkgOc3GF5nTG1LvTfyWtEm/Zs=; b=LSa8CJMVfsB1BFjXPCNHZyoCQWDPaJ7uKdFG78cgrQ30KNoiTRqHU9USCJboLcrvJjGkw5 tjXC+X7l4aYkJGf+vKf6AUFcqg/JW+a8MG2CN5S/B5xWU+njrAmy526WlfJeTFicQM5QMz hN5K2A2h/5oSLJ6ESaJxKUVB7g0Woa22ncLs3AGMljknma0bmYMQcQrJutUYzsPj4brLQY w/pu0Fuk1VC6buS4AJ42IpY5icqmpqHKNb8qbS4ZZa1ZKXNkfr7jZfSMTTij9XpcCYP5Ym hLxIU4RxISVhxPHEpxKA+gtLtd9j6XuQKpGv7SVjrg5QNsSY7YZFNM7oKkzaqw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SCP3d6Wjgz4SP; Sat, 21 Oct 2023 14:29:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 39LETXQa071575; Sat, 21 Oct 2023 14:29:33 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 39LETXmx071572; Sat, 21 Oct 2023 14:29:33 GMT (envelope-from git) Date: Sat, 21 Oct 2023 14:29:33 GMT Message-Id: <202310211429.39LETXmx071572@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Christos Margiolis Subject: git: 817701123233 - stable/14 - tty: fix improper backspace behaviour for UTF8 characters when in canonical mode List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: christos X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: 81770112323383f1bcb93c7d038fe195b5872b8f Auto-Submitted: auto-generated The branch stable/14 has been updated by christos: URL: https://cgit.FreeBSD.org/src/commit/?id=81770112323383f1bcb93c7d038fe195b5872b8f commit 81770112323383f1bcb93c7d038fe195b5872b8f Author: Bojan Novković AuthorDate: 2023-10-07 18:00:11 +0000 Commit: Christos Margiolis CommitDate: 2023-10-21 14:28:34 +0000 tty: fix improper backspace behaviour for UTF8 characters when in canonical mode This patch adds additional logic in ttydisc_rubchar() to properly handle backspace behaviour for UTF-8 characters. Currently, typing in a backspace after a UTF8 character will delete only one byte from the byte sequence, leaving garbled output in the tty's output queue. With this change all of the character's bytes are deleted. This change is only active when the IUTF8 flag is set (see 19054eb6053189144aa962b2ecc1bf5087758a3e "(s)tty: add support for IUTF8 input flag") The code uses the teken_wcwidth() function to properly handle character column widths for different code points, and adds the teken_utf8_bytes_to_codepoint() function that converts a UTF-8 byte sequence to a codepoint, as specified in RFC3629. Reported by: christos Reviewed by: christos, imp MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D42067 (cherry picked from commit 9e589b0938579f3f4d89fa5c051f845bf754184d) --- sys/kern/tty_ttydisc.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++ sys/teken/teken_wcwidth.h | 30 +++++++++++++++++++ 2 files changed, 104 insertions(+) diff --git a/sys/kern/tty_ttydisc.c b/sys/kern/tty_ttydisc.c index 665275ee93e7..eae7162e31c0 100644 --- a/sys/kern/tty_ttydisc.c +++ b/sys/kern/tty_ttydisc.c @@ -43,6 +43,9 @@ #include #include +#include +#include + /* * Standard TTYDISC `termios' line discipline. */ @@ -78,8 +81,13 @@ SYSCTL_ULONG(_kern, OID_AUTO, tty_nout, CTLFLAG_RD, /* Character is alphanumeric. */ #define CTL_ALNUM(c) (((c) >= '0' && (c) <= '9') || \ ((c) >= 'a' && (c) <= 'z') || ((c) >= 'A' && (c) <= 'Z')) +/* Character is UTF8-encoded. */ +#define CTL_UTF8(c) (!!((c) & 0x80)) +/* Character is a UTF8 continuation byte. */ +#define CTL_UTF8_CONT(c) (((c) & 0xc0) == 0x80) #define TTY_STACKBUF 256 +#define UTF8_STACKBUF 4 void ttydisc_open(struct tty *tp) @@ -800,6 +808,72 @@ ttydisc_rubchar(struct tty *tp) ttyoutq_write_nofrag(&tp->t_outq, "\b\b\b\b\b\b\b\b", tablen); return (0); + } else if ((tp->t_termios.c_iflag & IUTF8) != 0 && + CTL_UTF8(c)) { + uint8_t bytes[UTF8_STACKBUF] = { 0 }; + int curidx = UTF8_STACKBUF - 1, cwidth = 1, + nb = 0; + teken_char_t codepoint; + + /* Save current byte. */ + bytes[curidx] = c; + curidx--; + nb++; + /* Loop back through inq until we hit the + * leading byte. */ + while (CTL_UTF8_CONT(c) && nb < UTF8_STACKBUF) { + ttyinq_peekchar(&tp->t_inq, &c, "e); + ttyinq_unputchar(&tp->t_inq); + bytes[curidx] = c; + curidx--; + nb++; + } + /* + * Shift array so that the leading + * byte ends up at idx 0. + */ + if (nb < UTF8_STACKBUF) + memmove(&bytes[0], &bytes[curidx + 1], + nb * sizeof(uint8_t)); + /* Check for malformed UTF8 characters. */ + if (nb == UTF8_STACKBUF && + CTL_UTF8_CONT(bytes[0])) { + /* + * Place all bytes back into the inq and + * delete the last byte only. + */ + ttyinq_write(&tp->t_inq, bytes, + UTF8_STACKBUF, 0); + } else { + /* Find codepoint and width. */ + codepoint = + teken_utf8_bytes_to_codepoint(bytes, + nb); + if (codepoint != + TEKEN_UTF8_INVALID_CODEPOINT) { + cwidth = teken_wcwidth( + codepoint); + } else { + /* + * Place all bytes back into the + * inq and fall back to + * default behaviour. + */ + ttyinq_write(&tp->t_inq, bytes, + nb, 0); + } + } + tp->t_column -= cwidth; + /* + * Delete character by punching + * 'cwidth' spaces over it. + */ + if (cwidth == 1) + ttyoutq_write_nofrag(&tp->t_outq, + "\b \b", 3); + else if (cwidth == 2) + ttyoutq_write_nofrag(&tp->t_outq, + "\b\b \b\b", 6); } else { /* * Remove a regular character by diff --git a/sys/teken/teken_wcwidth.h b/sys/teken/teken_wcwidth.h index f57a185c2433..f5a23dbc9679 100644 --- a/sys/teken/teken_wcwidth.h +++ b/sys/teken/teken_wcwidth.h @@ -8,6 +8,8 @@ * Latest version: http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c */ +#define TEKEN_UTF8_INVALID_CODEPOINT -1 + struct interval { teken_char_t first; teken_char_t last; @@ -116,3 +118,31 @@ static int teken_wcwidth(teken_char_t ucs) (ucs >= 0x20000 && ucs <= 0x2fffd) || (ucs >= 0x30000 && ucs <= 0x3fffd))); } + +/* + * Converts an UTF-8 byte sequence to a codepoint as specified in + * https://datatracker.ietf.org/doc/html/rfc3629#section-3 . The function + * expects the 'bytes' array to start with the leading character. + */ +static teken_char_t +teken_utf8_bytes_to_codepoint(uint8_t bytes[4], int nbytes) +{ + + /* Check for malformed characters. */ + if (bitcount(bytes[0] & 0xf0) != nbytes) + return (TEKEN_UTF8_INVALID_CODEPOINT); + + switch (nbytes) { + case 1: + return (bytes[0] & 0x7f); + case 2: + return (bytes[0] & 0xf) << 6 | (bytes[1] & 0x3f); + case 3: + return (bytes[0] & 0xf) << 12 | (bytes[1] & 0x3f) << 6 | (bytes[2] & 0x3f); + case 4: + return (bytes[0] & 0x7) << 18 | (bytes[1] & 0x3f) << 12 | + (bytes[2] & 0x3f) << 6 | (bytes[3] & 0x3f); + default: + return (TEKEN_UTF8_INVALID_CODEPOINT); + } +}