From owner-freebsd-questions@freebsd.org Sun May 17 00:15:50 2020 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A6E7F2DB239 for ; Sun, 17 May 2020 00:15:50 +0000 (UTC) (envelope-from ypankov@fastmail.com) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49PjNP5vM8z4CHM for ; Sun, 17 May 2020 00:15:49 +0000 (UTC) (envelope-from ypankov@fastmail.com) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.west.internal (Postfix) with ESMTP id 81B6D79D; Sat, 16 May 2020 20:15:48 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Sat, 16 May 2020 20:15:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fastmail.com; h= subject:from:to:cc:references:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm2; bh=J brS1DHmfhyJHDkYTBKjckHSqBR3K8hv/hnDpdlflng=; b=Y4t3VcXdUzMjS8v0a PLEixFYTLstk5/e5ZY8jm3DwpOp18V37D+/aGD3KKFeleeiL/cJL2Gfy+HDE9iao fQ4brya/pEh5vTwgKvbHI/6qneOyuZIwPNU3hRyTkomt8A59FM9n6o095mfulzwK bodyhKySLXWitBaijv8DBJksgpi8JJWULuGefL4fG8/3oj7252b67NgOOB7czlUB EXLmmFiRfioGi0wkuYznjJ42lSgW8tBloZP5AYa3OmoMc0uNbP7bZljSdisT6XsN sPreH+Ob4JQUz3xdOb7EviUtsrMoo+b3/OSuF1AuKAJMYh0Yuzcy2/wUn0grChb/ XXD1A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=JbrS1DHmfhyJHDkYTBKjckHSqBR3K8hv/hnDpdlfl ng=; b=DjIt/Xx8OEMg/ZyI72IeIYqwmiOd0C/KJVMEs2XOJ/kg6QIIibwBMprgq 2Lf2vE6aQNUM9JRoFhh3LUSk85amGbgArLwqed147BOvQfzr3bvsaBSYkaBBGue8 UODjQtoWrzFVc775pW+UT123iEjooAlRCpeJGoXJw2+BBYetuXO2+GtwPUDc8Pel mqlKRORKf5onYDe0FPvqa3i9Fa1DKNH6ZwznwuHUQj7/eFr/rU3MzrOecY6h+BYq y/CG98Yvf+/szfjsODLLJ7emSG3JNe4tIQCDuqEj5XlwC/D5b5WSnvqpqydioT7a cnvuwhkm0dmJg7M2i9f40Yyulvt2w== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedruddtuddgfeduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefuhffvfhfkffgfgggjtgfgsehtke ertddtfeejnecuhfhrohhmpegjuhhrihcurfgrnhhkohhvuceohihprghnkhhovhesfhgr shhtmhgrihhlrdgtohhmqeenucggtffrrghtthgvrhhnpeekfeevfeejheejleeltdeuge eihfetueetleegkeejvdejiedvieevtdeuteehudenucfkphephedrudefledrudekhedr ieegnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhephi hprghnkhhovhesfhgrshhtmhgrihhlrdgtohhm X-ME-Proxy: Received: from [192.168.1.6] (unknown [5.139.185.64]) by mail.messagingengine.com (Postfix) with ESMTPA id CF005328005E; Sat, 16 May 2020 20:15:46 -0400 (EDT) Subject: Re: (character) Conversion error (in vi) ? From: Yuri Pankov To: "Ronald F. Guilmette" Cc: freebsd-questions@freebsd.org References: <72173.1589672025@segfault.tristatelogic.com> Message-ID: <910485e6-a5bf-8da1-55bb-2bc632d657e3@fastmail.com> Date: Sun, 17 May 2020 03:15:45 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 49PjNP5vM8z4CHM X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=fastmail.com header.s=fm2 header.b=Y4t3VcXd; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=DjIt/Xx8; dmarc=pass (policy=none) header.from=fastmail.com; spf=pass (mx1.freebsd.org: domain of ypankov@fastmail.com designates 64.147.123.24 as permitted sender) smtp.mailfrom=ypankov@fastmail.com X-Spamd-Result: default: False [-2.10 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24:c]; FREEMAIL_FROM(0.00)[fastmail.com]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[fastmail.com:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[fastmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[fastmail.com]; IP_SCORE(0.00)[ip: (-9.82), ipnet: 64.147.123.0/24(-4.92), asn: 11403(-2.69), country: US(-0.05)]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[fastmail.com:s=fm2,messagingengine.com:s=fm2]; RCVD_IN_DNSWL_LOW(-0.10)[24.123.147.64.list.dnswl.org : 127.0.5.1]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[24.123.147.64.rep.mailspike.net : 127.0.0.19] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 May 2020 00:15:50 -0000 Yuri Pankov wrote: > Ronald F. Guilmette wrote: >> In message <06696e14-e7e5-f212-4ef0-b89e7e78cde5@fastmail.com>, >> Yuri Pankov wrote: >> >>> I'm interested in more details on this as I could be somewhat involved >>> (see PR 202290).  What release did you upgrade from/to?  Could you >>> provide a simple test case that shows the issue? >> >> Yuri, >> >> I've just read PR 202290 and I have added to it a one line attachment >> which is a test case that demonstrates the issue/problem I have been >> experiencing, which sounds like it is most probably the same issue >> as in PR 202290.  (But I would like your opinion on that.) >> >> The test case is just a From: line from am email sent to me by a >> european correspondant of mine.  (Note that I use NHM as a mail >> client, and that it in turn invokes vi to edit new outbound >> messages and replies, which is where I frequently encounter this >> issue/problem.) >> >> I'm too embarassed to tell you what versions of FreeBSD I last did >> an upgrade from.  Let's just say that it was certifiably ancient. >> My last full upgrade was to 12.0-RELEASE and that date on my local >> vi is as follows: >> >> -r-xr-xr-x  6 root  wheel  461872 Dec  6  2018 /usr/bin/vi >> >> I guess that I simply need to upgrade that in order to get your >> fix for this issue (?) >> >> (Yes, I tend to be altogether too lax in keeping this particular >> system upgraded.) > > No, it's not that bug after all.  The issue is that (n)vi now (for quite > some time :-) defaults to UTF-8 when it can't reliably detect the file > encoding, so you'll just have to help it a bit adding the following to > ~/.nexrc: > > set fileencoding=iso8859-1 > > This way (n)vi will check if file encoding looks like UTF-8, and if not, > it will use ISO8859-1 as fallback. For the sake of correctness, re-reading the code, my reply was not entirely precise: (n)vi first checks if file is valid UTF-8, and if it isn't and fallback encoding (as shown above) is not set, uses your locale's encoding which I guess is UTF-8, hence failing.