From nobody Mon Oct 16 17:12:40 2023 X-Original-To: arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S8Nw85THDz4xbSV for ; Mon, 16 Oct 2023 17:12:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S8Nw845stz3WJs for ; Mon, 16 Oct 2023 17:12:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1697476360; a=rsa-sha256; cv=none; b=qtHrUyluGli1oWk9A6T0OxMVgoAagb/E5Kkwhgb8rXmCjGs8N2IQK4pLX+FWaYT955Rueg DFCqgoVvS9wqp/mQpDx+gdRCbnp/PbtLanN4RgOtxzNAw+Wa9ITdnEPs8tDVPjaYsPYsUw Awu+f8pucqDNn7KnhirifaZn8v1xRKeCtmnDDqfK2jwojlCYXVRcDfjQUE8ak7CdVG7aUF s12u4wSaUsRTXHJQx225d20TPt/3ohaDSczAGTZ+8bB6zMLazOI9TUOOUrUOtYYXATp6jJ RLM4TzhenIZn1IxgUKX7Fz1CV5IZyBrB3CqYxknFUgXNTl0Rn8e3/2kULdCqSg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1697476360; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sXQrwinc85fAWaqDvHrM06RDrgwT30A3qx/xTNn2htM=; b=daYIfco+x6AVlVHctLJlm0pmR5Ha1OjF1KC7C3vcDvHqPehcSQR16Hoy73EuDApahBUfHa 6XyzGCXSWsMysBx9BL/p+4Z3+5LrLzREOgRPDCo0pVQPCtf2fqCwP0TgbNWfSENEeR2t4H BlyPGa2lORqxQ5PEpArlrX8m/mYPDIcMZ149XddM8W1jGUl14Gadcgzf0Bs86HHs5qqrhV a6c71v7r2zX0RjFDPkNEX9oLcWhYcNxkXj0nOgyA1oBM2KBI5/ZPJnmHBtiAvfwqMbqb2x L51K39aBNXDBUXdUCEcSiULVj74rUbsfs5WlFcOCQ8gTQr4WXUA+aBKdGV2pvg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S8Nw83BGGzpjJ for ; Mon, 16 Oct 2023 17:12:40 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 39GHCeTJ002465 for ; Mon, 16 Oct 2023 17:12:40 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 39GHCemX002464 for arch@FreeBSD.org; Mon, 16 Oct 2023 17:12:40 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: arch@FreeBSD.org Subject: [Bug 270481] intro(9) needs an update/rewrite Date: Mon, 16 Oct 2023 17:12:40 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Documentation X-Bugzilla-Component: Manual Pages X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mhorne@freebsd.org X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: mhorne@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_status resolution Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D270481 Mitchell Horne changed: What |Removed |Added ---------------------------------------------------------------------------- Status|In Progress |Closed Resolution|--- |FIXED --=20 You are receiving this mail because: You are on the CC list for the bug.= From nobody Thu Nov 2 09:19:18 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SLdcL0y2wz4yc4k for ; Thu, 2 Nov 2023 09:19:30 +0000 (UTC) (envelope-from christos@freebsd.org) Received: from margiolis.net (mail.margiolis.net [95.179.159.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SLdcJ5xgQz3GZR; Thu, 2 Nov 2023 09:19:28 +0000 (UTC) (envelope-from christos@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=margiolis.net header.s=mail header.b=MfPwxIMJ; spf=softfail (mx1.freebsd.org: 95.179.159.8 is neither permitted nor denied by domain of christos@freebsd.org) smtp.mailfrom=christos@freebsd.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=mail; bh=Udcvk6jc90OLlQB 0FRXUvhpJtmVupmXjT7Liiq9IZkM=; h=subject:cc:to:from:date; d=margiolis.net; b=MfPwxIMJtjVLUZ+gDucWR+AYRFea/6C+7KJYCr9a9GaRRUTHYBp xSQVot7IOvwB1qvzTA31kU0O6ELJdpQytTTXaFKRQJH4450nKnS7XeKMnFMadc+45tPPpo AWe2G4fvlG8S9EemsFVXcdjI2Dk4bHZCiIfHXzzoC6hnt1jzBc= Received: from tpad (host-84-205-228-1.cpe.syzefxis.ote.gr [84.205.228.1]) by margiolis.net (OpenSMTPD) with ESMTPSA id 1aaf8795 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Thu, 2 Nov 2023 09:19:25 +0000 (UTC) Date: Thu, 2 Nov 2023 11:19:18 +0200 From: Christos Margiolis To: freebsd-arch@freebsd.org Cc: bojan.novkovic@fer.hr, imp@freebsd.org Subject: Re: IUTF8 on by default Message-ID: List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-1.80 / 15.00]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[margiolis.net:s=mail]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[margiolis.net:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; DMARC_NA(0.00)[freebsd.org]; FREEFALL_USER(0.00)[christos]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:20473, ipnet:95.179.144.0/20, country:US]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4SLdcJ5xgQz3GZR X-Spamd-Bar: - Hello, What do you think about this? Would there be any significant side-effects or breakages if we enable this option by default? For context, look at the PR description. Christos Warner Losh wrote: > I'd post this to arch@. I think it's a good idea, but I don't know the > larger implications. > > Warner > > On Mon, Oct 30, 2023 at 12:46 PM Christos Margiolis > wrote: > > > I was thinking about the discussion we had in D42067 [1] about whether > > the IUTF8 flag should be enabled by default. Warner suggested that since > > the default locale is UTF-8, setting IUTF8 by default shouldn't be too > > bad. > > > > Do you think we should go ahead with this change and have it enabled by > > default in sys/sys/ttydefaults.h, or add a check in tty_init_termios() > > whether the current locale is UTF-8, and manually enable it there? > > > > Christos > > > > [1] https://reviews.freebsd.org/D42067 From nobody Thu Nov 2 19:43:32 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SLvSX4xLlz50FHC for ; Thu, 2 Nov 2023 19:43:40 +0000 (UTC) (envelope-from christos@freebsd.org) Received: from margiolis.net (mail.margiolis.net [95.179.159.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SLvSW3xJDz4DyV; Thu, 2 Nov 2023 19:43:39 +0000 (UTC) (envelope-from christos@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=margiolis.net header.s=mail header.b=Ux1Gf6BT; spf=softfail (mx1.freebsd.org: 95.179.159.8 is neither permitted nor denied by domain of christos@freebsd.org) smtp.mailfrom=christos@freebsd.org; dmarc=none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=mail; bh=D6/PLR9CoIS+Wnx 5qjQFHBwmJDBH02hjny5u0+eZ/5c=; h=subject:cc:to:from:date; d=margiolis.net; b=Ux1Gf6BTqVfaDhT2j2WHbgP/Sh232eIj2XUmCm8CokXuOOf3Gi1 JUrD/fr2lBrAI7ytvFHnORSVVBZQbScqTvMi33o5mi8/fZ9J8rDJW55knNZN/ppVKgAr7E DN+xTNbhuiCaWpN7ivJ66UBIu5VmMyD5ucfiO1g/0eSqMWhT14= Received: from tpad (host-84-205-228-1.cpe.syzefxis.ote.gr [84.205.228.1]) by margiolis.net (OpenSMTPD) with ESMTPSA id 87bca5e6 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Thu, 2 Nov 2023 19:43:32 +0000 (UTC) Date: Thu, 2 Nov 2023 21:43:32 +0200 From: Christos Margiolis To: freebsd-arch@freebsd.org Cc: bojan.novkovic@fer.hr, imp@freebsd.org Subject: HEADS UP: IUTF8 to be enabled by default Message-ID: List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-2.80 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MID_RHS_NOT_FQDN(0.50)[]; R_DKIM_ALLOW(-0.20)[margiolis.net:s=mail]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[margiolis.net:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DMARC_NA(0.00)[freebsd.org]; ASN(0.00)[asn:20473, ipnet:95.179.144.0/20, country:US]; FREEFALL_USER(0.00)[christos]; ARC_NA(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4SLvSW3xJDz4DyV X-Spamd-Bar: -- Hello again and sorry for the poorly worded previous email, To give a bit more context, during EuroBSDCon 2023, me and Bojan Novković started working on a patch to fix backspacing of UTF-8 characters in the tty driver. What was happening is if you typed a >1 byte UTF-8 character and then backspaced it, the driver would actually delete only 1 byte from the character, instead of all its bytes, which ended up leaving garbage in the buffer since the character wasn't fully deleted. To test this, run cat(1), type a UTF-8 character (e.g é, è, à, non-latin characters, etc), press backspace only once, and look at the output: $ cat αα α� Bojan then implemented a new IUTF8 flag for stty [1], which enables proper handling for UTF-8 backspacing in the tty driver [2]. In the Phabricator review of the tty(4) patch [3], I proposed the idea of having the IUTF8 flag enabled by default. imp@ mentioned that since the default locale is UTF-8, having the flag set by default shouldn't be a problem. Two possible solutions I have thought of: 1. Add IUTF8 to TTYDEF_IFLAG in sys/sys/ttydefaults.h. 2. Add a check in tty_init_termios() whether the current locale is UTF-8 (how?), and enable it there. What do you think? Could this change cause any side-effects we haven't thought about? Christos [1] https://cgit.freebsd.org/src/commit/?id=128f63cedc14ae21b35f74e11e2fe1a5659c58e8 [2] https://cgit.freebsd.org/src/commit/?id=9e589b0938579f3f4d89fa5c051f845bf754184d [3] https://reviews.freebsd.org/D42067 From nobody Fri Nov 3 07:15:29 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SMBpy26Wmz50F0y for ; Fri, 3 Nov 2023 07:15:38 +0000 (UTC) (envelope-from garyj@gmx.de) Received: from mout.gmx.net (mout.gmx.net [212.227.15.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mout.gmx.net", Issuer "Telekom Security ServerID OV Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SMBpx4cH6z3HYf; Fri, 3 Nov 2023 07:15:37 +0000 (UTC) (envelope-from garyj@gmx.de) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1698995735; x=1699600535; i=garyj@gmx.de; bh=FWT1WiH6Q+DNgITFvetTuif6aBwqYT1/DDjo0euntrM=; h=X-UI-Sender-Class:Date:From:To:Cc:Subject:In-Reply-To:References: Reply-To; b=sf5nLb1Qyd7P4Uh80AgrvdAVJ9HvT/fBXVtHGq+7pBeMcz88qfhltt4yHNgH5aQP aWpdZqt6I/lKbMsVIfK+HR19MhY6QNd6+HfYpM7LW2Xg2konKEKWCi6DoDNl87rdC eVdDDXxT288Lp8LdIGzX+KPhuiwQ6MNDbvL10UbiyEr6z24eY784lC1DBx0Fouts4 pRva2eSHZzHph/JsG4xqtX9FZKJ4DedoUPT2bCDiOpD2pr6AHZKbcKDr8RJXUvq4F IyAkLAiTzY1LCd9NPh9CPsXlTpkpvTOZlJPn5NWKcHManMyXwVVNfxtgss5UFfOKm cKUbyUGKqHXA/4ZRNg== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from ernst.home ([217.226.57.134]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1Mirna-1rb9QB0QHE-00eu2i; Fri, 03 Nov 2023 08:15:35 +0100 Date: Fri, 3 Nov 2023 07:15:29 +0000 From: Gary Jennejohn To: Christos Margiolis Cc: freebsd-arch@freebsd.org, bojan.novkovic@fer.hr, imp@freebsd.org Subject: Re: HEADS UP: IUTF8 to be enabled by default Message-ID: <20231103081529.016be29d@ernst.home> In-Reply-To: References: Reply-To: garyj@gmx.de X-Mailer: Claws Mail 3.19.1 (GTK+ 2.24.33; amd64-portbld-freebsd14.0) List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K1:GlfeU2caXTLBJpUleA+fWNL/sgjZlJXQWX3rKlj0viaVCPrk+9e hquDMWP1d6qSRXdXZhh06V88pIuLBYrCG3MYabf67BGv62BnoHL1uS7FTsm8UXQ02trhzGR +a7i6bytdwYCBP3QuWMnUlvmM0hsoE4iEuDXAzmidwnqYsfe6zWLgP+ftZQbtnnFzjfQ0mR 6x2tab4G9SroT59Ws2ofw== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:wo+O13zraLg=;pVE1JprvRxqGB6hXmmolOTEReEG LgSVIZX2YgkKrAI6exErv/ro6h5V+PYK/GA6/oNmiNqJr0NgZD+t550H4UF2hsDOOnZIKbyCb 7vfaSeBW5Hmi6ddLaholhZ1hLC8//SnFezwQKzoptkQVnM+vPneOx+X0Qa5pTd0gVROHGWqpW VnbP94AxBMAZLsevYN/NUsysGbgMR5EprgQZAxecQaHA/WXQ0uNxT9voqHWbvT/ttUoClWtNu TA01/LvPKk/R9TnZqRALQ9T6pQ1BstTm6jgop1O7XcKPVXpfIQV/MYtOwJLs2emfEeFY1JWBb FQR2xr/Qb1twKvrgjehnG+2seWX4iyd+b5R5alscciYloDtjlUSohCUQrT/dRhxJn3OIwFrhE I7yX2XtWxgWtAGGmizLYWHQJJFpgvJDQ1Lqn0gCwW6VDew+y6gFVmyi5ILcI1B24r2jdjBCjV 7P7+iaaZPeU1QA+8hyDwzxEb5rGfV92qQAG/6mcXxPeg2QJ/4uKJA4Jn9+F/IytyLiNVRagsT /aFpTjIiv6fmEwa/zgMS3atNzvUUAGsjpUgIUHJ5IiZEIKsPv08vk+1MxH80tQueYEssPmX1m a/+o4ZzUrhfyZ2xcqpqHK5VJxW7rWYahDS0rdjA1g7rb9Hf8g5HsV9e7q1R2YIZmP29sdmBex 2GrAe+9CNZSz2Av1BY9myeHUWup9K7d7oLK9Jj07oOahEHCAmodHDHQkjZfF/BPKV3aRcvPTE 0IGMIv6L5/8obwBNN6VbqRMa87RR02nMmZaaUqk4cvBVDJkbR+WbWukI/uju094ee6T6iV5T0 uPovRO46Bz3OOmdQ7lmcVjMMxvXv7J8JFB1k1JIUoHTTqcndcqWf482vsyYKFncx7OYwD7si4 Ei9yy2P0v56sM+EbyQkrRm1IqgDHt3ybYBBnteKcrlCfkPbtEIkzJdLl8X97V2kd+v2T1qBF7 SIPDQkaQIZjIRGmssVd2kSZLJ54= X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8560, ipnet:212.227.0.0/16, country:DE] X-Rspamd-Queue-Id: 4SMBpx4cH6z3HYf On Thu, 2 Nov 2023 21:43:32 +0200 Christos Margiolis wrote: > Hello again and sorry for the poorly worded previous email, > > To give a bit more context, during EuroBSDCon 2023, me and Bojan > Novkovi? started working on a patch to fix backspacing of UTF-8 > characters in the tty driver. What was happening is if you typed a >1 > byte UTF-8 character and then backspaced it, the driver would actually > delete only 1 byte from the character, instead of all its bytes, which > ended up leaving garbage in the buffer since the character wasn't fully > deleted. To test this, run cat(1), type a UTF-8 character (e.g =E9, =E8,= =E0, > non-latin characters, etc), press backspace only once, and look at the > output: > > $ cat > ?? > ?? > > Bojan then implemented a new IUTF8 flag for stty [1], which enables > proper handling for UTF-8 backspacing in the tty driver [2]. > > In the Phabricator review of the tty(4) patch [3], I proposed the idea > of having the IUTF8 flag enabled by default. imp@ mentioned that since > the default locale is UTF-8, having the flag set by default shouldn't be > a problem. > > Two possible solutions I have thought of: > > 1. Add IUTF8 to TTYDEF_IFLAG in sys/sys/ttydefaults.h. > 2. Add a check in tty_init_termios() whether the current locale is > UTF-8 (how?), and enable it there. > Use getenv("LANG") and check whether UTF-8 is part of the string? My LANG is set to C.UTF-8, for example. > What do you think? Could this change cause any side-effects we haven't > thought about? > > Christos > > [1] https://cgit.freebsd.org/src/commit/?id=3D128f63cedc14ae21b35f74e11e= 2fe1a5659c58e8 > [2] https://cgit.freebsd.org/src/commit/?id=3D9e589b0938579f3f4d89fa5c05= 1f845bf754184d > [3] https://reviews.freebsd.org/D42067 > =2D- Gary Jennejohn From nobody Fri Nov 3 12:51:38 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SMLGl1mzNz5037v for ; Fri, 3 Nov 2023 12:51:43 +0000 (UTC) (envelope-from christos@freebsd.org) Received: from margiolis.net (mail.margiolis.net [95.179.159.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SMLGk4kLSz4Xkl; Fri, 3 Nov 2023 12:51:42 +0000 (UTC) (envelope-from christos@freebsd.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=mail; bh=bZ9QWl6n5qMsKij P5nEpWFAIdEd8r6LahJVkb2t4/Wo=; h=in-reply-to:references:subject:cc:to: from:date; d=margiolis.net; b=euV6UYyrnGqE2ySg5InpRrwiPNSzsWT4sbfFCCXc eUu1WQh7xqC/sxgptB7nRZW8bjDKR9ilXZOd7OumYqtSf5WyzgwWg8mcX+fvzehlVT6TZZ wWHamwxUvM4UweOY3CULdXNb2iqpn/3IrxHs1O+JZjNS6w4OofMudRgi39gMY= Received: from pleb (ppp-94-66-59-115.home.otenet.gr [94.66.59.115]) by margiolis.net (OpenSMTPD) with ESMTPSA id 987ff268 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Fri, 3 Nov 2023 12:51:40 +0000 (UTC) Date: Fri, 3 Nov 2023 14:51:38 +0200 From: Christos Margiolis To: Gary Jennejohn Cc: freebsd-arch@freebsd.org, bojan.novkovic@fer.hr, imp@freebsd.org Subject: Re: HEADS UP: IUTF8 to be enabled by default Message-ID: <6k62glovuqzn6p5bt6hmqw3lwmcigvnnsjv5y66zpi2lpzi7xs@rtlhbrj447sp> References: <20231103081529.016be29d@ernst.home> List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20231103081529.016be29d@ernst.home> X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:20473, ipnet:95.179.144.0/20, country:US] X-Rspamd-Queue-Id: 4SMLGk4kLSz4Xkl Gary Jennejohn wrote: > Use getenv("LANG") and check whether UTF-8 is part of the string? > > My LANG is set to C.UTF-8, for example. tty_init_termios() is a kernel function, so getenv(3) cannot be used. Christos From nobody Fri Nov 3 13:23:02 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SMLz96FlQz5083C for ; Fri, 3 Nov 2023 13:23:17 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SMLz94db0z4c0l for ; Fri, 3 Nov 2023 13:23:17 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-9dd3f4a0f5aso67147866b.1 for ; Fri, 03 Nov 2023 06:23:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1699017795; x=1699622595; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=8fi0gT9Bin8I6uIoomxlDYzxGRWctvV/YrA1CSx+ZrI=; b=NGDkLlylvm8pN5RhAKLcnK46ff3RLiNGouDM/IYz68QxCHOYuWNCaBCJNchbqQzL25 tOyTJOAqG7MpG+siJJVE3DUWpttTNw18FO7PMUgKPcI4Zi5npjmj/r3tJ/NXtBzoecMi 42MXggKsegYlaGgymzd2MMHP5020UVljaTw2mQ0Y30jYwYmwrzb+YRTHtGBgzN+MQJmf DapU6+m0SRnU+8IuVyGfEFTYL0u4MiuKY4lUChV8Xeq30mpya/loPZ57uNWclvhhUR7o TSACnxNDTHC7fjEnZi6C/Jj4Awdu2uCcPy1b7KSznj1/PX5avFCVW+yIprYfZ8xBmRL9 SuJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699017795; x=1699622595; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8fi0gT9Bin8I6uIoomxlDYzxGRWctvV/YrA1CSx+ZrI=; b=Z07Q4gpRkKOdK87TxA2+jgIJmCiawO3diOlRasg0zH750kmkU9OzrUwPiN8hv8Imgm AVIbaIm9KnvR5l3LEh8+4Y3AVzNcBtleL2Rk4j/SntxBVG58dPmmW9C71WZ2V/xY0iCO +gcNj/qwe1YgwknuNfg4/A7gX+klBs2YMJhbvH78GW/pjFhQgMxSXT1wjYP6dvk1BtZw FkAZBOKTDxhrdnsAmSV7G1qGoCJulERS8h5WaYBtvrm5xQrkru3Xj7K+lX36MfEIGKKD S4gmjIto5PqxWZGV9r0eY+TkVQAk/9i1LDrFpQWR6Weaq/HsKNx6SnXAdGvGBfPYEuKR 0l3A== X-Gm-Message-State: AOJu0YxjWowNEq1OZVUT4c+JW3MNIIA6BO5ZsZB45fzVaKj936OeQCVR Rbz8vpbLMnVLcO++eOiTKZ9ysz4EFx9MMEgHRBariA== X-Google-Smtp-Source: AGHT+IGkq325+qHJVzT4muU3U82+Wg/BrY6bwryrARU/1ZG6yGD1QoCKNgjd1N6PvscYfEt26jC5zsWz8Ui4LNwMipw= X-Received: by 2002:a17:907:944e:b0:9ae:4054:5d2a with SMTP id dl14-20020a170907944e00b009ae40545d2amr6277903ejc.16.1699017794580; Fri, 03 Nov 2023 06:23:14 -0700 (PDT) List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 References: <20231103081529.016be29d@ernst.home> In-Reply-To: <20231103081529.016be29d@ernst.home> From: Warner Losh Date: Fri, 3 Nov 2023 07:23:02 -0600 Message-ID: Subject: Re: HEADS UP: IUTF8 to be enabled by default To: garyj@gmx.de Cc: Christos Margiolis , "freebsd-arch@freebsd.org" , bojan.novkovic@fer.hr, Warner Losh Content-Type: multipart/alternative; boundary="0000000000001d05db06093f6892" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4SMLz94db0z4c0l --0000000000001d05db06093f6892 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, Nov 3, 2023, 1:15 AM Gary Jennejohn wrote: > On Thu, 2 Nov 2023 21:43:32 +0200 > Christos Margiolis wrote: > > > Hello again and sorry for the poorly worded previous email, > > > > To give a bit more context, during EuroBSDCon 2023, me and Bojan > > Novkovi? started working on a patch to fix backspacing of UTF-8 > > characters in the tty driver. What was happening is if you typed a >1 > > byte UTF-8 character and then backspaced it, the driver would actually > > delete only 1 byte from the character, instead of all its bytes, which > > ended up leaving garbage in the buffer since the character wasn't fully > > deleted. To test this, run cat(1), type a UTF-8 character (e.g =C3=A9, = =C3=A8, =C3=A0, > > non-latin characters, etc), press backspace only once, and look at the > > output: > > > > $ cat > > ?? > > ?? > > > > Bojan then implemented a new IUTF8 flag for stty [1], which enables > > proper handling for UTF-8 backspacing in the tty driver [2]. > > > > In the Phabricator review of the tty(4) patch [3], I proposed the idea > > of having the IUTF8 flag enabled by default. imp@ mentioned that since > > the default locale is UTF-8, having the flag set by default shouldn't b= e > > a problem. > > > > Two possible solutions I have thought of: > > > > 1. Add IUTF8 to TTYDEF_IFLAG in sys/sys/ttydefaults.h. > > 2. Add a check in tty_init_termios() whether the current locale is > > UTF-8 (how?), and enable it there. > > > > Use getenv("LANG") and check whether UTF-8 is part of the string? > This string is set too late for the default. Also, drivers don't have access to process data. Warner My LANG is set to C.UTF-8, for example. > > > What do you think? Could this change cause any side-effects we haven't > > thought about? > > > > Christos > > > > [1] > https://cgit.freebsd.org/src/commit/?id=3D128f63cedc14ae21b35f74e11e2fe1a= 5659c58e8 > > [2] > https://cgit.freebsd.org/src/commit/?id=3D9e589b0938579f3f4d89fa5c051f845= bf754184d > > [3] https://reviews.freebsd.org/D42067 > > > > > -- > Gary Jennejohn > --0000000000001d05db06093f6892 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Fri, Nov 3, 2023, 1:15 AM Gary Jennejohn <garyj@gmx.de> wrote:
On Thu, 2 Nov 2023 21:43:32 +0200
Christos Margiolis <christos@freebsd.org> wrote:

> Hello again and sorry for the poorly worded previous email,
>
> To give a bit more context, during EuroBSDCon 2023, me and Bojan
> Novkovi? started working on a patch to fix backspacing of UTF-8
> characters in the tty driver. What was happening is if you typed a >= ;1
> byte UTF-8 character and then backspaced it, the driver would actually=
> delete only 1 byte from the character, instead of all its bytes, which=
> ended up leaving garbage in the buffer since the character wasn't = fully
> deleted. To test this, run cat(1), type a UTF-8 character (e.g =C3=A9,= =C3=A8, =C3=A0,
> non-latin characters, etc), press backspace only once, and look at the=
> output:
>
> $ cat
> ??<backspace>
> ??
>
> Bojan then implemented a new IUTF8 flag for stty [1], which enables > proper handling for UTF-8 backspacing in the tty driver [2].
>
> In the Phabricator review of the tty(4) patch [3], I proposed the idea=
> of having the IUTF8 flag enabled by default. imp@ mentioned that since=
> the default locale is UTF-8, having the flag set by default shouldn= 9;t be
> a problem.
>
> Two possible solutions I have thought of:
>
> 1. Add IUTF8 to TTYDEF_IFLAG in sys/sys/ttydefaults.h.
> 2. Add a check in tty_init_termios() whether the current locale is
>=C2=A0 =C2=A0 UTF-8 (how?), and enable it there.
>

Use getenv("LANG") and check whether UTF-8 is part of the string?=

= This string is set too late for the default. Also, drivers don't have a= ccess to process data.

W= arner

My LANG is set to C.UTF-8, for example.

> What do you think? Could this change cause any side-effects we haven&#= 39;t
> thought about?
>
> Christos
>
> [1] https://cgit.freebsd.org/src/commit/?id=3D128f63cedc14ae21b35f74e11e2fe1a= 5659c58e8
> [2] https://cgit.freebsd.org/src/commit/?id=3D9e589b0938579f3f4d89fa5c051f845= bf754184d
> [3] https://reviews.freebsd.org/D42067
>


--
Gary Jennejohn
--0000000000001d05db06093f6892-- From nobody Fri Nov 3 13:54:23 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SMMg92Ybtz50DZb for ; Fri, 3 Nov 2023 13:54:29 +0000 (UTC) (envelope-from christos@freebsd.org) Received: from margiolis.net (mail.margiolis.net [95.179.159.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SMMg90CX4z3CVH; Fri, 3 Nov 2023 13:54:28 +0000 (UTC) (envelope-from christos@freebsd.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; s=mail; bh=9fOpgXfsCNNCbn1 K/hg1klfKke9MQEwRjJAopOv0vl0=; h=in-reply-to:references:subject:cc:to: from:date; d=margiolis.net; b=e7jxeTKSr49G8Qh9bhZV9G+2fzrhCTXE2uHuUSyA kBijU4+zdswykusLsmMyYnkG9w8aUGzrLPWCPQWaYNGQkmZ1PJzT5dTH7nGsw1RpTTcxW3 RwyOpUe6IOU+DDvH1+s5X8zH/LZXQUiO0NZCmjMX9XZCQ++Vyajvv2nZR9Uoo= Received: from pleb (ppp-94-66-59-115.home.otenet.gr [94.66.59.115]) by margiolis.net (OpenSMTPD) with ESMTPSA id 2c01ee01 (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO); Fri, 3 Nov 2023 13:54:24 +0000 (UTC) Date: Fri, 3 Nov 2023 15:54:23 +0200 From: Christos Margiolis To: Warner Losh Cc: Gary Jennejohn , freebsd-arch@freebsd.org, bojan.novkovic@fer.hr, imp@freebsd.org Subject: Re: HEADS UP: IUTF8 to be enabled by default Message-ID: References: <20231103081529.016be29d@ernst.home> List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:20473, ipnet:95.179.144.0/20, country:US] X-Rspamd-Queue-Id: 4SMMg90CX4z3CVH Warner Losh wrote: > This string is set too late for the default. Also, drivers don't have > access to process data. Do you think we should just enable it in TTYDEF_IFLAG in sys/sys/ttydefaults.h? Christos From nobody Sat Nov 4 14:17:43 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SN07j2XCVz50bHB for ; Sat, 4 Nov 2023 14:17:53 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Received: from phk.freebsd.dk (phk.freebsd.dk [130.225.244.222]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4SN07h2kW7z4Rd6; Sat, 4 Nov 2023 14:17:52 +0000 (UTC) (envelope-from phk@critter.freebsd.dk) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of phk@critter.freebsd.dk designates 130.225.244.222 as permitted sender) smtp.mailfrom=phk@critter.freebsd.dk; dmarc=none Received: from critter.freebsd.dk (unknown [192.168.55.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by phk.freebsd.dk (Postfix) with ESMTPS id 43D1E892B2; Sat, 4 Nov 2023 14:17:44 +0000 (UTC) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.17.1/8.16.1) with ESMTPS id 3A4EHhMd015360 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 4 Nov 2023 14:17:43 GMT (envelope-from phk@critter.freebsd.dk) Received: (from phk@localhost) by critter.freebsd.dk (8.17.1/8.16.1/Submit) id 3A4EHhmw015359; Sat, 4 Nov 2023 14:17:43 GMT (envelope-from phk) Message-Id: <202311041417.3A4EHhmw015359@critter.freebsd.dk> To: Christos Margiolis cc: freebsd-arch@freebsd.org, bojan.novkovic@fer.hr, imp@freebsd.org Subject: Re: HEADS UP: IUTF8 to be enabled by default In-reply-to: From: "Poul-Henning Kamp" References: List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <15357.1699107463.1@critter.freebsd.dk> Date: Sat, 04 Nov 2023 14:17:43 +0000 X-Spamd-Result: default: False [-2.65 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.65)[-0.655]; FORGED_SENDER(0.30)[phk@phk.freebsd.dk,phk@critter.freebsd.dk]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[freebsd.dk]; FREEFALL_USER(0.00)[phk]; ARC_NA(0.00)[]; ASN(0.00)[asn:1835, ipnet:130.225.0.0/16, country:EU]; RCPT_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[phk@phk.freebsd.dk,phk@critter.freebsd.dk] X-Rspamd-Queue-Id: 4SN07h2kW7z4Rd6 X-Spamd-Bar: -- -------- Christos Margiolis writes: > Two possible solutions I have thought of: > > 1. Add IUTF8 to TTYDEF_IFLAG in sys/sys/ttydefaults.h. > 2. Add a check in tty_init_termios() whether the current locale is > UTF-8 (how?), and enable it there. > > What do you think? Could this change cause any side-effects we haven't > thought about? Ideally this should be part of libc's "unified session setup", which UNIX unfortunately never have had. With N different login shells, accounts which do not login to a traditional shell etc. AND considering that UTF8 have universally become the "lingua franka", I vote that it makes a lot more sense to simply make it the default (ie: your first suggestion) rather than add complexity all over the place, yet missing out some obscure cases. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence.