From owner-svn-src-head@freebsd.org Mon Jun 22 22:56:09 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DAF3C33EFC4 for ; Mon, 22 Jun 2020 22:56:09 +0000 (UTC) (envelope-from yuripv@yuripv.dev) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49rPsP3JRSz4XXN; Mon, 22 Jun 2020 22:56:09 +0000 (UTC) (envelope-from yuripv@yuripv.dev) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.nyi.internal (Postfix) with ESMTP id AA79B5C01EE; Mon, 22 Jun 2020 18:46:59 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Mon, 22 Jun 2020 18:46:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yuripv.dev; h= subject:to:cc:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=fm1; bh=G fVclCuApheCkbKpPYpbmdYC62G1lcpwG8/oZyxNgS8=; b=MCU100109yAvuIc1o I6uQPngm7rA+s5opKsyKt7BWTB/KSgW2rlx7tca+S6oTsg2YNhcFChtbx0ZxM4ih AsMMDWpMAWTM5nGyRFejjkT/HL9oA9xgFwTMshFrZslPoK/Hb34RTi+dcncl/B0L GkTi0+CO1c/bUAznECWOOxPKJvFE43Ugcwn+k+JQXLtAT1uyWyhMjHorFDAObSYh QeQXBuI/eA+ogM+ab+sHWGy/bRrJ9erM81eyDB9zYZFhVQXo8Fd2kE5VR3eg3S1F YWirFH6r4Icviylpw657LR4aCrVkXy8DSbry4vRezM1JzsktqOK+q4mt50B/zmrF /vx8w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; bh=GfVclCuApheCkbKpPYpbmdYC62G1lcpwG8/oZyxNg S8=; b=r5x6rX6uZp12FBoQ3F0apucmdNr5UfcqTg8BLdRrFkh5ZI7y7NP1usYxX PXzkVNOuZrhbIZ10vTH9k1MSHO5tdSyRHInyQj+tYIve6IAn0N1Nk5/8f5fSz6k4 el0Mi9DLMtDp4VADjL7VJrwuiW1Re7B1BT4Gdw6o80z2rDJ54ALV2YS+K70Xica4 GQrKsgNDd+buqjCagxulbZssYJsxv4nfECZwceoHfc3J72G7rmuMywjKTlkUBj/f py0k46mL0i5bJgmcFipTUXxXtmb0fi+J2UOdID48sT9o6/ztdBwuH5HEe2wK6xJc lJFqj1RRN6JUnrCy7lC29rebdVTlQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedrudekfedgudefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefuvfhfhffkffgfgggjtgfgsehtke ertddtfeejnecuhfhrohhmpegjuhhrihcurfgrnhhkohhvuceohihurhhiphhvseihuhhr ihhpvhdruggvvheqnecuggftrfgrthhtvghrnhepudeuffegtdehffdtffefkefhgfelie eitefghfeugeelfeduffegtdeufeekgfdvnecukfhppeeluddrvdegtddruddvgedrudef jeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeihuh hrihhpvheshihurhhiphhvrdguvghv X-ME-Proxy: Received: from [192.168.1.6] (unknown [91.240.124.137]) by mail.messagingengine.com (Postfix) with ESMTPA id B2E1630673FD; Mon, 22 Jun 2020 18:46:58 -0400 (EDT) Subject: Re: svn commit: r362148 - head/contrib/nvi/common To: Zhihao Yuan , Gleb Smirnoff Cc: Yuri Pankov , svn-src-head@freebsd.org References: <202006131411.05DEB2mP097868@repo.freebsd.org> <20200622221144.GA31842@FreeBSD.org> <3fe4705c-e036-6999-b6b0-6e05f7cf8321@yuripv.dev> <20200622222448.GB31842@FreeBSD.org> From: Yuri Pankov Message-ID: Date: Tue, 23 Jun 2020 01:46:57 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 49rPsP3JRSz4XXN X-Spamd-Bar: ++++++++++++++ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yuripv.dev header.s=fm1 header.b=MCU10010; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=r5x6rX6u; dmarc=none; spf=pass (mx1.freebsd.org: domain of yuripv@yuripv.dev designates 66.111.4.27 as permitted sender) smtp.mailfrom=yuripv@yuripv.dev X-Spamd-Result: default: False [14.51 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(0.00)[+ip4:66.111.4.27:c]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[yuripv.dev:+,messagingengine.com:+]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.27:from]; ARC_NA(0.00)[]; RECEIVED_SPAMHAUS_XBL(5.00)[91.240.124.137:received]; R_DKIM_ALLOW(0.00)[yuripv.dev:s=fm1,messagingengine.com:s=fm3]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_SPAM_SHORT(0.26)[0.263]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[yuripv.dev]; NEURAL_SPAM_MEDIUM(0.97)[0.966]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.88)[0.884]; URIBL_BLACK(7.50)[yuripv.dev:dkim]; RWL_MAILSPIKE_VERYGOOD(0.00)[66.111.4.27:from]; GREYLIST(0.00)[pass,body] X-Spam: Yes X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2020 22:56:09 -0000 Zhihao Yuan wrote: > On Mon, Jun 22, 2020 at 5:24 PM Gleb Smirnoff > wrote: > > > My first attempt was this fix: > > --- common/exf.c        (revision 362200) > +++ common/exf.c        (working copy) > @@ -1252,7 +1252,8 @@ file_encinit(SCR *sp) >         else if (O_ISSET(sp, O_FILEENCODING) && >             strcasecmp(O_STR(sp, O_FILEENCODING), "utf-8") != 0) >                 /* Use fileencoding as is */ ; > -       else if (strcasecmp(codeset(), "utf-8") != 0) > +       else if (strncasecmp(codeset() + strlen(codeset()) - 5, > "utf-8", 5) != > +           0) >                 o_set(sp, O_FILEENCODING, OS_STRDUP, codeset(), 0); >         else >                 o_set(sp, O_FILEENCODING, OS_STRDUP, "iso8859-1", 0); > > But it appeared to be not the case. To my surprise, codeset() > which is wrapper around nl_langinfo() in my case returns US-ASCII. > > > That sounds strange. > >   1. Can you set LC_CTYPE as well and see >     if anything changes? >   2. Can you revert to the previous version >     and see what nl_langinfo gives? > > There is another issue... I'm sorry.  I totally forgot what > looks_utf8 actually does. > > Here is its behavior (encoding.c): > >  Returns >  -1: invalid UTF-8 >   0: uses odd control characters, so doesn't look like text >   1: 7-bit text >   2: definitely UTF-8 text (valid high-bit set bytes) > > So if looks_utf8() > 1, it means the file itself is UTF-8 > for sure.  If you opened a file with 7-bit text or with > control characters, :set fileencoding should set > the encoding intended to write.  But the HEAD > behaviors is that you can't input Unicode. > > I'm reverting upstream. Yes, I will revert for now.