From owner-svn-src-all@freebsd.org Mon Jun 22 22:20:07 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 19B4E33E69E; Mon, 22 Jun 2020 22:20:07 +0000 (UTC) (envelope-from yuripv@yuripv.dev) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49rP3p5SlRz4Vwl; Mon, 22 Jun 2020 22:20:06 +0000 (UTC) (envelope-from yuripv@yuripv.dev) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id B8E5B5C0BFF; Mon, 22 Jun 2020 18:20:05 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 22 Jun 2020 18:20:05 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.dev; h= subject:to:cc:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm1; bh=P 27tDNr+k+qQMq1deG8Lbs21fMUPrWQdkKpUGxnfrYQ=; b=fUGurQCjd1J0fIJh8 p7V9B15oNoWAH1vN9w68iavcFjBW8mi2sebtkpumMR084SGEXfTS5Rh9fQE2ckeN HqnJdINnhg0dCHFINDIp3Qwk2xuJdXQ8br5GmRuv6A2cqPXoXAW8tHWkPtXYjy7b fcX0dBSe7mVMjstcYFn0OXLBklpMVf/jUZ3pYRF495s8OUxJqobeIDfqtUCwQqK4 cazwyDCBEjeTqkiCXdn+FpGpBq1fAPBL7M4W3AOGtEp6vxjvnV86/94A2FsJxrRq JYZoXSRCiDtaWEwDh7i5knAtUB1Rsxje3h09Q31nVdHQm+vyyRls4kN6JpQtnmGl YwGsg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=P27tDNr+k+qQMq1deG8Lbs21fMUPrWQdkKpUGxnfr YQ=; b=ujl4MJ/e/QfNX+n6CKSX9mo1Dd+QimASHrhBRoqL1ebH0yunuHGkMHBte MxXAyYPuG1RGAwUierjp1WBszhWzApoBPAenSKxA37kz2TlBMT4bx8LNrZKSbtOL /XD6mHAt1m+q+BH8yNepLjbjdYQ2GTYatG2n4tyd4JVG2xmICYcS/Ind7xPlAYKp diVgBDXCaRt3870Qm9ozfYO/yrckwDOC3/7lPNV+SwG5C9dVDC5LWT1c62UTis0o krFn9QdXVAdW4zd0h3dtbQjIFJ1gHtnwlC9G4Z3DuJ+DXte4gz1FjZWBfcMSVGDE vbsD9oQL4044H313pfqNtYyVKRrTw== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrudekfedgtdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefuvfhfhffkffgfgggjtgfgsehtje ertddtfeejnecuhfhrohhmpegjuhhrihcurfgrnhhkohhvuceohihurhhiphhvseihuhhr ihhpvhdruggvvheqnecuggftrfgrthhtvghrnhepieekteeftdeftdekhfffueeljeefff fgffffuefhhefgheejvdegfeehieevudfgnecuffhomhgrihhnpehfrhgvvggsshgurdho rhhgnecukfhppeeluddrvdegtddruddvgedrudefjeenucevlhhushhtvghrufhiiigvpe dtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeihuhhrihhpvheshihurhhiphhvrdguvghv X-ME-Proxy: Received: from [192.168.1.6] (unknown [91.240.124.137]) by mail.messagingengine.com (Postfix) with ESMTPA id 66156306741D; Mon, 22 Jun 2020 18:20:04 -0400 (EDT) Subject: Re: svn commit: r362148 - head/contrib/nvi/common To: Gleb Smirnoff , Yuri Pankov , Zhihao Yuan Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org References: <202006131411.05DEB2mP097868@repo.freebsd.org> <20200622221144.GA31842@FreeBSD.org> From: Yuri Pankov Message-ID: <3fe4705c-e036-6999-b6b0-6e05f7cf8321@yuripv.dev> Date: Tue, 23 Jun 2020 01:20:01 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <20200622221144.GA31842@FreeBSD.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 49rP3p5SlRz4Vwl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2020 22:20:07 -0000 Gleb Smirnoff wrote: > Yuri, Zhihao, > > this commit totally broke Russian input for me in nvi. After > exiting edit mode, nvi immediately converts all text to ???????. > > I don't have any special settings in my environment. All I have > is "russian" class for my user which yields in these environment > variables: > > declare -x LANG="ru_RU.UTF-8" > declare -x MM_CHARSET="UTF-8" > declare -x XTERM_LOCALE="ru_RU.UTF-8" > > I'm already digging into that problem, but may be you have > a clue immediately. My bad, yes, I see the problem, looking into it. > On Sat, Jun 13, 2020 at 02:11:02PM +0000, Yuri Pankov wrote: > Y> Author: yuripv > Y> Date: Sat Jun 13 14:11:02 2020 > Y> New Revision: 362148 > Y> URL: https://svnweb.freebsd.org/changeset/base/362148 > Y> > Y> Log: > Y> nvi: fallback to ISO8859-1 as last resort > Y> > Y> Current logic of using user's locale encoding that is UTF-8 doesn't make > Y> much sense if we already failed the looks_utf8() check and skipped > Y> encoding set using "fileencoding" as being UTF-8 as well; fallback to > Y> ISO8859-1 in that case. > Y> > Y> Reviewed by: Zhihao Yuan > Y> Differential Revision: https://reviews.freebsd.org/D24919 > Y> > Y> Modified: > Y> head/contrib/nvi/common/exf.c > Y> > Y> Modified: head/contrib/nvi/common/exf.c > Y> ============================================================================== > Y> --- head/contrib/nvi/common/exf.c Sat Jun 13 09:16:07 2020 (r362147) > Y> +++ head/contrib/nvi/common/exf.c Sat Jun 13 14:11:02 2020 (r362148) > Y> @@ -1237,7 +1237,10 @@ file_encinit(SCR *sp) > Y> } > Y> > Y> /* > Y> - * Detect UTF-8 and fallback to the locale/preset encoding. > Y> + * 1. Check for valid UTF-8. > Y> + * 2. Check if fallback fileencoding is set and is NOT UTF-8. > Y> + * 3. Check if user locale's encoding is NOT UTF-8. > Y> + * 4. Use ISO8859-1 as last resort. > Y> * > Y> * XXX > Y> * A manually set O_FILEENCODING indicates the "fallback > Y> @@ -1246,9 +1249,13 @@ file_encinit(SCR *sp) > Y> */ > Y> if (looks_utf8(buf, blen) > 1) > Y> o_set(sp, O_FILEENCODING, OS_STRDUP, "utf-8", 0); > Y> - else if (!O_ISSET(sp, O_FILEENCODING) || > Y> - !strcasecmp(O_STR(sp, O_FILEENCODING), "utf-8")) > Y> + else if (O_ISSET(sp, O_FILEENCODING) && > Y> + strcasecmp(O_STR(sp, O_FILEENCODING), "utf-8") != 0) > Y> + /* Use fileencoding as is */ ; > Y> + else if (strcasecmp(codeset(), "utf-8") != 0) > Y> o_set(sp, O_FILEENCODING, OS_STRDUP, codeset(), 0); > Y> + else > Y> + o_set(sp, O_FILEENCODING, OS_STRDUP, "iso8859-1", 0); > Y> > Y> conv_enc(sp, O_FILEENCODING, 0); > Y> #endif