From nobody Thu Nov 2 19:43:32 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SLvSX4xLlz50FHC for ; Thu, 2 Nov 2023 19:43:40 +0000 (UTC) (envelope-from christos@freebsd.org) Received: from margiolis.net (mail.margiolis.net [95.179.159.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SLvSW3xJDz4DyV; Thu, 2 Nov 2023 19:43:39 +0000 (UTC) (envelope-from christos@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=margiolis.net header.s=mail header.b=Ux1Gf6BT; spf=softfail (mx1.freebsd.org: 95.179.159.8 is neither permitted nor denied by domain of christos@freebsd.org) smtp.mailfrom=christos@freebsd.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=mail; bh=D6/PLR9CoIS+Wnx 5qjQFHBwmJDBH02hjny5u0+eZ/5c=; h=subject:cc:to:from:date; d=margiolis.net; b=Ux1Gf6BTqVfaDhT2j2WHbgP/Sh232eIj2XUmCm8CokXuOOf3Gi1 JUrD/fr2lBrAI7ytvFHnORSVVBZQbScqTvMi33o5mi8/fZ9J8rDJW55knNZN/ppVKgAr7E DN+xTNbhuiCaWpN7ivJ66UBIu5VmMyD5ucfiO1g/0eSqMWhT14= Received: from tpad (host-84-205-228-1.cpe.syzefxis.ote.gr [84.205.228.1]) by margiolis.net (OpenSMTPD) with ESMTPSA id 87bca5e6 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Thu, 2 Nov 2023 19:43:32 +0000 (UTC) Date: Thu, 2 Nov 2023 21:43:32 +0200 From: Christos Margiolis To: freebsd-arch@freebsd.org Cc: bojan.novkovic@fer.hr, imp@freebsd.org Subject: HEADS UP: IUTF8 to be enabled by default Message-ID: List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-2.80 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[margiolis.net:s=mail]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[margiolis.net:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; ASN(0.00)[asn:20473, ipnet:95.179.144.0/20, country:US]; FREEFALL_USER(0.00)[christos]; ARC_NA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4SLvSW3xJDz4DyV X-Spamd-Bar: -- Hello again and sorry for the poorly worded previous email, To give a bit more context, during EuroBSDCon 2023, me and Bojan Novković started working on a patch to fix backspacing of UTF-8 characters in the tty driver. What was happening is if you typed a >1 byte UTF-8 character and then backspaced it, the driver would actually delete only 1 byte from the character, instead of all its bytes, which ended up leaving garbage in the buffer since the character wasn't fully deleted. To test this, run cat(1), type a UTF-8 character (e.g é, è, à, non-latin characters, etc), press backspace only once, and look at the output: $ cat αα α� Bojan then implemented a new IUTF8 flag for stty [1], which enables proper handling for UTF-8 backspacing in the tty driver [2]. In the Phabricator review of the tty(4) patch [3], I proposed the idea of having the IUTF8 flag enabled by default. imp@ mentioned that since the default locale is UTF-8, having the flag set by default shouldn't be a problem. Two possible solutions I have thought of: 1. Add IUTF8 to TTYDEF_IFLAG in sys/sys/ttydefaults.h. 2. Add a check in tty_init_termios() whether the current locale is UTF-8 (how?), and enable it there. What do you think? Could this change cause any side-effects we haven't thought about? Christos [1] https://cgit.freebsd.org/src/commit/?id=128f63cedc14ae21b35f74e11e2fe1a5659c58e8 [2] https://cgit.freebsd.org/src/commit/?id=9e589b0938579f3f4d89fa5c051f845bf754184d [3] https://reviews.freebsd.org/D42067