From nobody Sun Oct 22 11:57:39 2023 X-Original-To: dev-commits-src-branches@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SCxdv4Ms9z4xCM9; Sun, 22 Oct 2023 11:57:39 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SCxdv3YW5z3WbQ; Sun, 22 Oct 2023 11:57:39 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697975859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Mh2TxfTxz7YEQDdrp4VXQgYGpwXWkX0LmAutcHISj54=; b=pDtPXAKi6nVishM0aadcQ28wNaJuWonqZHSSezjZtI27QuVPQm7PPxBb0vAZGWR7nZgoJ3 Wiek1JN7+zGjGKVEMZNn9bLdD4hMMJqPz7bbMK0+sckWAHOfGUZ4x7iZcbvf0lDSafqILJ nmMjkjzEQWp4RY/KIR6PPQP+gKy931GSjzynisxr8nf+CI+wEkeBraJ1KNDxZ0vCC2ZVhT 2bhAIKw3jIiaN2FACfKxCvMNDmxxyefa1Hvy7ekAQm4aU1Vsv/fXB2Ge/1GaFFO4iYC2I/ RgiPkSCZrt8L/JillGqfNLzs/zFsZy+bxx5QoUgc78NaQck1knhKI7e8gx7R2g== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1697975859; a=rsa-sha256; cv=none; b=lrLx32MABBdVWnl8jHpWk2eGfJbnlhimKFaNmrnFI8D8oMlcz7Yfn7s+264qBb2pYRDa9g dUi9jn/pWZ0aegW5bQk+fzAEhKOxzSeOESnC2WT93tSgGKxGYteRG3SclgWK9f15Kz4kAZ CiraiwNQmVIG97farVBOQWuUl5ode/WJqWTijjdcRzZSQUrxhO8wxgrQtKpNbexdD1HAF+ 6tcnT4KWsWaZLoOJpOOD2XuH7ExtgJqeTUFk8G9HShW5kstHAteSLbGkkuNkg9l9Q6vfZU 5WS18K8rmqxByfiDAhYV4+vjCnDicbepUhKD38fZprokZqqTBRwPoZ0s4gVmSg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697975859; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Mh2TxfTxz7YEQDdrp4VXQgYGpwXWkX0LmAutcHISj54=; b=a9aW7QO7nGP8ecRSr+wDFiSTvfRTiZPkV0FxEKCEzkESifiEi6SznpTkGxCXT+qKzbu6sZ oFTr9IAfhPR1jJLfmKBW0B7HRgsl5U/V5TSdiam/3H310FWsCAT9p96lvapwRYOLLqkD+W uxDOH9aZNm11/5LL+AapStlHOy2cwc2YY03rKRDX98AA+d6ZR9m5araNefc0bOviQuqPzY xl8wu+Drs/4XFP6Qili9UUSnJatoaTlWFFUrTYTcjb9U974sVc0q7z6OUnbnu6cynm34lG GY5qteCCgylVEiWrpCBYQW86FmJgOv9TjLTLP2F7nquX+yGQMmaxlZmz7jN7qA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4SCxdv2XXrzwV9; Sun, 22 Oct 2023 11:57:39 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 39MBvdOB015395; Sun, 22 Oct 2023 11:57:39 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 39MBvdGU015392; Sun, 22 Oct 2023 11:57:39 GMT (envelope-from git) Date: Sun, 22 Oct 2023 11:57:39 GMT Message-Id: <202310221157.39MBvdGU015392@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Christos Margiolis Subject: git: b80d4b2f93e5 - releng/14.0 - tty/teken: fix UTF8 sequence validation logic List-Id: Commits to the stable branches of the FreeBSD src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-branches List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-branches@freebsd.org X-BeenThere: dev-commits-src-branches@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: christos X-Git-Repository: src X-Git-Refname: refs/heads/releng/14.0 X-Git-Reftype: branch X-Git-Commit: b80d4b2f93e54b81041a518ea4c0a8be2985a658 Auto-Submitted: auto-generated The branch releng/14.0 has been updated by christos: URL: https://cgit.FreeBSD.org/src/commit/?id=b80d4b2f93e54b81041a518ea4c0a8be2985a658 commit b80d4b2f93e54b81041a518ea4c0a8be2985a658 Author: Bojan Novković AuthorDate: 2023-10-13 05:14:36 +0000 Commit: Christos Margiolis CommitDate: 2023-10-22 11:57:00 +0000 tty/teken: fix UTF8 sequence validation logic This patch fixes UTF-8 sequence validation logic in teken_utf8_bytes_to_codepoint() and fixes fallback behaviour in ttydisc_rubchar() when an invalid UTF8 sequence is encountered. The code previously used __bitcount() to extract sequence length information from the leading byte. However, this assumption breaks for certain code points that have additional bits set in the first half of the leading byte (e.g. Cyrillic characters). This lead to incorrect behaviour when deleting those characters using backspaces. The code now checks the number of consecutive set bits in the leading byte starting from the MSB, as per RFC 3629. Reviewed by: christos MFC after: 2 weeks Differential Revision: https://reviews.freebsd.org/D42147 (cherry picked from commit 2fed1c579c52d63b72fc08ffcc652ba0183f9254) Approved by: re (gjb) --- sys/kern/tty_ttydisc.c | 12 +++++++----- sys/teken/teken_wcwidth.h | 23 ++++++++++++++++++++--- 2 files changed, 27 insertions(+), 8 deletions(-) diff --git a/sys/kern/tty_ttydisc.c b/sys/kern/tty_ttydisc.c index eae7162e31c0..c46579cefbb1 100644 --- a/sys/kern/tty_ttydisc.c +++ b/sys/kern/tty_ttydisc.c @@ -844,23 +844,25 @@ ttydisc_rubchar(struct tty *tp) */ ttyinq_write(&tp->t_inq, bytes, UTF8_STACKBUF, 0); + ttyinq_unputchar(&tp->t_inq); } else { /* Find codepoint and width. */ codepoint = teken_utf8_bytes_to_codepoint(bytes, nb); - if (codepoint != - TEKEN_UTF8_INVALID_CODEPOINT) { - cwidth = teken_wcwidth( - codepoint); - } else { + if (codepoint == + TEKEN_UTF8_INVALID_CODEPOINT || + (cwidth = teken_wcwidth( + codepoint)) == -1) { /* * Place all bytes back into the * inq and fall back to * default behaviour. */ + cwidth = 1; ttyinq_write(&tp->t_inq, bytes, nb, 0); + ttyinq_unputchar(&tp->t_inq); } } tp->t_column -= cwidth; diff --git a/sys/teken/teken_wcwidth.h b/sys/teken/teken_wcwidth.h index bc40c23a6eb7..14af2c6f9530 100644 --- a/sys/teken/teken_wcwidth.h +++ b/sys/teken/teken_wcwidth.h @@ -128,15 +128,32 @@ static inline teken_char_t teken_utf8_bytes_to_codepoint(uint8_t bytes[4], int nbytes) { - /* Check for malformed characters. */ - if (__bitcount(bytes[0] & 0xf0) != nbytes) + /* + * Check for malformed characters by comparing 'nbytes' + * to the byte length of the character. + * + * The table in section 3 of RFC 3629 defines 4 different + * values indicating the length of a UTF-8 byte sequence. + * + * 0xxxxxxx -> 1 byte + * 110xxxxx -> 2 bytes + * 1110xxxx -> 3 bytes + * 11110xxx -> 4 bytes + * + * The length is determined by the higher-order bits in + * the leading octet (except in the first case, where an MSB + * of 0 means a byte length of 1). Here we flip the 4 upper + * bits and count the leading zeros using __builtin_clz() + * to determine the number of bytes. + */ + if (__builtin_clz(~(bytes[0] & 0xf0) << 24) != nbytes) return (TEKEN_UTF8_INVALID_CODEPOINT); switch (nbytes) { case 1: return (bytes[0] & 0x7f); case 2: - return (bytes[0] & 0xf) << 6 | (bytes[1] & 0x3f); + return (bytes[0] & 0x1f) << 6 | (bytes[1] & 0x3f); case 3: return (bytes[0] & 0xf) << 12 | (bytes[1] & 0x3f) << 6 | (bytes[2] & 0x3f); case 4: