From nobody Sat Oct 21 14:35:48 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SCPBs1Sgtz4y5kG; Sat, 21 Oct 2023 14:35:49 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SCPBs0TvCz4J4s; Sat, 21 Oct 2023 14:35:49 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697898949; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5P8Qfg2D8EPhBr4gVyxTQiteXR7uTn24jTz/igvhoyY=; b=VdegxjelU7HH9f/FcGvaReBqcz863sA8wX+u+F7KclygDZvuaAWX3BF6ycdtO26xnPbGVE onRrUoTULdQ7DEqhQSw9k2f5ld5kXxYXHDi+hgrdaK/HYIE/T+IOQ91XL6/rySZCg3ob7o rSf3DCUR5ielDsUNGo1Od1G+sPbzazPIFbA0EwyVzA84MwPbQmZt1VkhjJUIgXmvl/0yDp T+put5c36DGfJustHF8lb+tAeEW0dDUND4gs7ra9J5KvAXN6RRtJH40CXkuwzvSpNAJslv N6FqjBakPC+pv6Os499wHD9JPQcCMlJBCBAFb2FYweL9uzHbPUgh7asO1xkMXA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1697898949; a=rsa-sha256; cv=none; b=jxMdvbLfKx/ehHdYLXPYoBFt8Q2oDVHyVgCAewg7+0yApXJwFyBsN1PCgiaSm2sR2kZokK EnAlXdCjd/sE9XC7l4k3vBCWW/A+MqoL8lgtkefLu++M8u/gtzDJIecz7RYxue31PbQP+G iAzx1eLJ/oEAqDfh8oX/gmF/47ZZMcs+j+TqIxzF/6uELm2/DEEl82DbcXnCKHvhdpsQG+ IBjPt5egrDLROXQHJ4l7BFg0UZ5a9SktORngm7zeERkAky7t3yrHCOTq0P78pPptC/lUgg g3sD/dcyuFnoMGlLACQoQvzw1BPPSkg7aLZC6lOurhaP62jM8NDTSSTQDpNy1A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697898949; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=5P8Qfg2D8EPhBr4gVyxTQiteXR7uTn24jTz/igvhoyY=; b=rAvrp327tSZvm5mgYIXOmzYOFSOD3UBV1iJGUjtgUBwQgadqu/fvnx+TnFjwZPhNL3GbsJ 2oT0KBsG+FdHFBXO6PmsEUpIc6t1D1OtqTfecBZmUlbk01h0x5aRZ3AAJDfhv1P71ish22 7rErypPz0U5QeUCV2KJWPBynhsc+UB4SCxCRqyaPkQN0hmONn8xF0hF3uJY67A4VN7nIAa 7SQ2uigAepR+lrcNiFqTmDPD7beIq1NMR+0yn33/NlkIJQM6gQAOV5FTzpNHimQxP4CpbW rWS/CQbvj3UE5eyr2w3tJZiiU8Epj9Py6hSpKOinVMssVhi4DP7mGgmIe0mlDw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SCPBr6cqfz4TC; Sat, 21 Oct 2023 14:35:48 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 39LEZmnv087652; Sat, 21 Oct 2023 14:35:48 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 39LEZms3087649; Sat, 21 Oct 2023 14:35:48 GMT (envelope-from git) Date: Sat, 21 Oct 2023 14:35:48 GMT Message-Id: <202310211435.39LEZms3087649@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Christos Margiolis Subject: git: 376c2ff8981a - stable/13 - tty/teken: fix UTF8 sequence validation logic List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: christos X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 376c2ff8981a9b818e7a61149feeb4961ed970d2 Auto-Submitted: auto-generated The branch stable/13 has been updated by christos: URL: https://cgit.FreeBSD.org/src/commit/?id=376c2ff8981a9b818e7a61149feeb4961ed970d2 commit 376c2ff8981a9b818e7a61149feeb4961ed970d2 Author: Bojan Novković AuthorDate: 2023-10-13 05:14:36 +0000 Commit: Christos Margiolis CommitDate: 2023-10-21 14:34:22 +0000 tty/teken: fix UTF8 sequence validation logic This patch fixes UTF-8 sequence validation logic in teken_utf8_bytes_to_codepoint() and fixes fallback behaviour in ttydisc_rubchar() when an invalid UTF8 sequence is encountered. The code previously used __bitcount() to extract sequence length information from the leading byte. However, this assumption breaks for certain code points that have additional bits set in the first half of the leading byte (e.g. Cyrillic characters). This lead to incorrect behaviour when deleting those characters using backspaces. The code now checks the number of consecutive set bits in the leading byte starting from the MSB, as per RFC 3629. Reviewed by: christos MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D42147 (cherry picked from commit 2fed1c579c52d63b72fc08ffcc652ba0183f9254) --- sys/kern/tty_ttydisc.c | 12 +++++++----- sys/teken/teken_wcwidth.h | 23 ++++++++++++++++++++--- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/sys/kern/tty_ttydisc.c b/sys/kern/tty_ttydisc.c index eae7162e31c0..c46579cefbb1 100644 --- a/sys/kern/tty_ttydisc.c +++ b/sys/kern/tty_ttydisc.c @@ -844,23 +844,25 @@ ttydisc_rubchar(struct tty *tp) */ ttyinq_write(&tp->t_inq, bytes, UTF8_STACKBUF, 0); + ttyinq_unputchar(&tp->t_inq); } else { /* Find codepoint and width. */ codepoint = teken_utf8_bytes_to_codepoint(bytes, nb); - if (codepoint != - TEKEN_UTF8_INVALID_CODEPOINT) { - cwidth = teken_wcwidth( - codepoint); - } else { + if (codepoint == + TEKEN_UTF8_INVALID_CODEPOINT || + (cwidth = teken_wcwidth( + codepoint)) == -1) { /* * Place all bytes back into the * inq and fall back to * default behaviour. */ + cwidth = 1; ttyinq_write(&tp->t_inq, bytes, nb, 0); + ttyinq_unputchar(&tp->t_inq); } } tp->t_column -= cwidth; diff --git a/sys/teken/teken_wcwidth.h b/sys/teken/teken_wcwidth.h index bc40c23a6eb7..14af2c6f9530 100644 --- a/sys/teken/teken_wcwidth.h +++ b/sys/teken/teken_wcwidth.h @@ -128,15 +128,32 @@ static inline teken_char_t teken_utf8_bytes_to_codepoint(uint8_t bytes[4], int nbytes) { - /* Check for malformed characters. */ - if (__bitcount(bytes[0] & 0xf0) != nbytes) + /* + * Check for malformed characters by comparing 'nbytes' + * to the byte length of the character. + * + * The table in section 3 of RFC 3629 defines 4 different + * values indicating the length of a UTF-8 byte sequence. + * + * 0xxxxxxx -> 1 byte + * 110xxxxx -> 2 bytes + * 1110xxxx -> 3 bytes + * 11110xxx -> 4 bytes + * + * The length is determined by the higher-order bits in + * the leading octet (except in the first case, where an MSB + * of 0 means a byte length of 1). Here we flip the 4 upper + * bits and count the leading zeros using __builtin_clz() + * to determine the number of bytes. + */ + if (__builtin_clz(~(bytes[0] & 0xf0) << 24) != nbytes) return (TEKEN_UTF8_INVALID_CODEPOINT); switch (nbytes) { case 1: return (bytes[0] & 0x7f); case 2: - return (bytes[0] & 0xf) << 6 | (bytes[1] & 0x3f); + return (bytes[0] & 0x1f) << 6 | (bytes[1] & 0x3f); case 3: return (bytes[0] & 0xf) << 12 | (bytes[1] & 0x3f) << 6 | (bytes[2] & 0x3f); case 4: