Date: Sat, 4 Feb 2023 10:47:27 +0100 From: Eivind Nicolay Evensen <eivinde@terraplane.org> To: Eugene Grosbein <eugen@grosbein.net> Cc: freebsd-stable@freebsd.org Subject: Re: Grep with non-ascii Message-ID: <20230204104727.72bb3715@elg.hjerdalen.lokalnett> In-Reply-To: <c50fe93c-a53f-1e63-d8b3-cd346d1050a7@grosbein.net> References: <20230203110642.70e4a076@elg.hjerdalen.lokalnett> <819a4336-9689-bdbe-a90d-8f1d7b842662@grosbein.net> <20230203151853.02732bd6@elg.hjerdalen.lokalnett> <c50fe93c-a53f-1e63-d8b3-cd346d1050a7@grosbein.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Den Sat, 4 Feb 2023 08:41:17 +0700 skrev Eugene Grosbein <eugen@grosbein.net>: > 03.02.2023 21:18, Eivind Nicolay Evensen wrote: > > > Den Fri, 3 Feb 2023 19:12:32 +0700 > > skrev Eugene Grosbein <eugen@grosbein.net>: > > > >> 03.02.2023 17:06, Eivind Nicolay Evensen wrote: > >>> Hello. > >>> > >>> I just noticed this today: > >>> > >>> elg!ene[~]> printf "bø\nhei\nøl\n" | grep ø > >>> grep: trailing backslash (\) > >>> elg!ene[~]> echo $LC_CTYPE $LANG > >>> nb_NO.ISO8859-1 nb_NO.ISO8859-1 > >>> > >>> While I have the result I envisioned with gnugrep: > >>> > >>> elg!ene[~]> printf "bø\nhei\nøl\n" | ggrep ø > >>> bø > >>> øl > >>> > >>> Also, on OpenIndiana, linux and Netbsd, grep gives the proper > >>> result. > >>> > >>> Is lib/libc/regex the right place to look into this if I > >>> find the time, or does anybody know this enough to know the > >>> problem? > >> > >> Try single quotes instead of double quotes. > >> And pleace specify system version and shell name, and shell version > >> if its not in base system. > > > > This is > > elg!ene[~]> uname -a > > FreeBSD elg.hjerdalen.lokalnett 13.2-PRERELEASE FreeBSD > > 13.2-PRERELEASE #1: Tue Jan 31 11:23:29 CET 2023 > > ene@elg.hjerdalen.lokalnett:/usr/obj/usr/src/amd64.amd64/sys/ENE-spurv > > amd64 > > > > Using the tcsh that comes with it. But I don't think the quotes > > matter much because of this: > > > > elg!ene[~]> grep ø > > grep: trailing backslash (\) > > > > The output was more just to have something to look for, like > > with ggrep but anyway: > > > > elg!ene[~]> printf 'bø\nhei\nøl\n' |grep ø > > grep: trailing backslash (\) > > > > And obviously: > > > > elg!ene[~]> printf 'bø\nhei\nøl\n' > > bø > > hei > > øl > > > > And it seems to be the same for any 8859-1 character not part > > of ascii: > > > > elg!ene[~]> grep ä > > grep: trailing backslash (\) > > elg!ene[~]> grep ß > > grep: trailing backslash (\) > > elg!ene[~]> grep ç > > grep: trailing backslash (\) > > I checked it with ru_RU.KOI8-R locale and same problem manifested, > with every Cyrillic letter. The following line shows codes and > characters of affected positions in last half of 8-bit character > table. > > $ jot -w '%o' - 128 255 1 | xargs -n2 -I^ printf '^ \^\n' | while > read octal char; do grep -q "$char" /etc/motd 2>/dev/null; [ $? -gt 1 > ] && echo $octal $char; done > > Note that this problem does not exist in 12.4 or earlier FreeBSD > versions, so this is recent regression. Surely that's due to grep > command being GNU grep in 12.4 but BSD grep in 13.x That makes sense, since I know for certain I have grepped for Norwegian words containing æøå without seeing this problem before. And I switched from 11 to 13 very late, and only because I wanted to use hardware unsupported by the old one, so that would explain why it took me so long to discover. -- Eivind Nicolay Evensen
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20230204104727.72bb3715>