From owner-freebsd-bugs@freebsd.org Wed May 2 12:39:09 2018 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0117CFA86EB for ; Wed, 2 May 2018 12:39:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 62A456BDB9 for ; Wed, 2 May 2018 12:39:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1BDCFFA86EA; Wed, 2 May 2018 12:39:08 +0000 (UTC) Delivered-To: bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BEFCFFA86E9 for ; Wed, 2 May 2018 12:39:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 51E546BDB8 for ; Wed, 2 May 2018 12:39:07 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 824F920AE8 for ; Wed, 2 May 2018 12:39:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w42Cd6N1076724 for ; Wed, 2 May 2018 12:39:06 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w42Cd6te076720 for bugs@FreeBSD.org; Wed, 2 May 2018 12:39:06 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 227922] man -k/-f does not search man pages/mandoc.db from packages/ports (MANPATH problem) Date: Wed, 02 May 2018 12:39:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 11.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: jdc@koitsu.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 May 2018 12:39:09 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D227922 Bug ID: 227922 Summary: man -k/-f does not search man pages/mandoc.db from packages/ports (MANPATH problem) Product: Base System Version: 11.1-STABLE Hardware: amd64 OS: Any Status: New Severity: Affects Many People Priority: --- Component: bin Assignee: bugs@FreeBSD.org Reporter: jdc@koitsu.org It appears that man -k (a.k.a. apropos) and man -f (a.k.a. whatis) have been broken for several years now. Rather than just give some terse explanation, I'm going to provide proof/evidence first, followed by the fixes last. Looks broken: $ man -k IPC::Open3 apropos: nothing appropriate $ man -f IPC::Open3 whatis: nothing appropriate $ apropos -k IPC::Open3 apropos: nothing appropriate $ whatis -f IPC::Open3 whatis: nothing appropriate Man page does exist: $ find /usr/local/lib/perl5 -name "IPC::Open3*" -print /usr/local/lib/perl5/5.26/perl/man/man3/IPC::Open3.3.gz What does manpath say? $ manpath /usr/share/man:/usr/local/man:/usr/share/openssl/man:/usr/local/lib/perl5/s= ite_perl/man:/usr/local/lib/perl5/5.26/perl/man Looks correct. Double verification that the perl pkg did the right thing: $ cat /usr/local/etc/man.d/perl5.conf MANPATH /usr/local/lib/perl5/site_perl/man MANPATH /usr/local/lib/perl5/5.26/perl/man And mandoc.db from makewhatis? Looks OK... $ find /usr/local -name "mandoc.db" -print 2>/dev/null /usr/local/lib/perl5/site_perl/man/mandoc.db /usr/local/lib/perl5/5.26/perl/man/mandoc.db /usr/local/man/mandoc.db What if we specify the MANPATH directly? Wow, it works: $ apropos -M /usr/local/lib/perl5/5.26/perl/man -k IPC::Open3 IPC::Open3(3) - open a process for reading, writing, and error handling usi= ng open3() $ whatis -M /usr/local/lib/perl5/5.26/perl/man -f IPC::Open3 IPC::Open3(3) - open a process for reading, writing, and error handling usi= ng open3() Okay, let's dig around with truss and see what's happening in apropos and whatis, since they're C programs (man is not -- man is a shell script): $ file /usr/bin/man /usr/bin/apropos /usr/bin/whatis /usr/bin/man: POSIX shell script, ASCII text executable /usr/bin/apropos: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1 (1101515), FreeBSD-style, not stripped /usr/bin/whatis: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.1 (1101515), FreeBSD-style, not stripped $ truss -f -- apropos -k IPC::Open3 2>&1 | grep man.d $ truss -f -- whatis -f IPC::Open3 2>&1 | grep man.d $ truss -f -- whatis -f IPC::Open3 2>&1 | grep opendir $ truss -f -- apropos -k IPC::Open3 2>&1 | grep opendir $ truss -f -- apropos -k IPC::Open3 2>&1 | egrep 'chdir|mandoc.db' 6011: chdir("/usr/share/man") =3D 0 (0x0) 6011: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00) =3D 3 (0x3) 6011: chdir("/usr/local/man") =3D 0 (0x0) 6011: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00) =3D 3 (0x3) 6011: chdir("/home/jdc") =3D 0 (0x0) $ truss -f -- whatis -f IPC::Open3 2>&1 | egrep 'chdir|mandoc.db' 6018: chdir("/usr/share/man") =3D 0 (0x0) 6018: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00) =3D 3 (0x3) 6018: chdir("/usr/local/man") =3D 0 (0x0) 6018: openat(AT_FDCWD,"mandoc.db",O_RDONLY,00) =3D 3 (0x3) 6018: chdir("/home/jdc") =3D 0 (0x0) $ truss -f -- apropos -k IPC::Open3 2>&1 | grep \.conf 5594: lstat("/etc/libmap.conf",{ mode=3D-rw-r--r-- ,inode=3D1926215,size=3D107,blksize=3D32768 }) =3D 0 (0x0) 5594: openat(AT_FDCWD,"/etc/libmap.conf",O_RDONLY|O_CLOEXEC,00) =3D 3 (0x3) 5594: readlink("/etc/malloc.conf",0x7fffffffd520,1024) ERR#2 'No such file= or directory' 5594: open("/etc/man.conf",O_RDONLY,0666) ERR#2 'No such file or directory' $ truss -f -- whatis -f IPC::Open3 2>&1 | grep \.conf 5623: lstat("/etc/libmap.conf",{ mode=3D-rw-r--r-- ,inode=3D1926215,size=3D107,blksize=3D32768 }) =3D 0 (0x0) 5623: openat(AT_FDCWD,"/etc/libmap.conf",O_RDONLY|O_CLOEXEC,00) =3D 3 (0x3) 5623: readlink("/etc/malloc.conf",0x7fffffffd520,1024) ERR#2 'No such file= or directory' 5623: open("/etc/man.conf",O_RDONLY,0666) ERR#2 'No such file or directory' I see no evidence that apropos or whatis understand /usr/local/etc/man.d/*.conf. The man pages for these utilities confirm it: $ man apropos ... NAME apropos, whatis =E2=80=93 search manual page databases ... By default, apropos searches for makewhatis(8) databases in the default paths stipulated by man(1) and uses case-insensitive substring matching (the =3D operator) over manual names and descriptions (the Nm and Nd m= acro keys). Multiple terms imply pairwise -o. ... FILES mandoc.db name of the makewhatis(8) keyword database /etc/man.conf default man(1) configuration file ... What about man's own man page? $ man man ... -d Print extra debugging information. Repeat for increased verbosity. Does not display the manual page. ... FILES /etc/man.conf System configuration file. /usr/local/etc/man.d/*.conf Local configuration files. ... So it should be working, but isn't. Let's try that debugging flag: $ man -dddddd -k IPC::Open3 apropos: nothing appropriate $ man -dddddd -f IPC::Open3 whatis: nothing appropriate Not helpful. At this stage I'm thinking: do I delve into 20KB+ of shell script for man, = or do I start looking at C code for apropos and whatis? I decided to look at = man first. What I found: 1. Use of man -k or -f ends up calling do_man (because argv[0] is man, not apropos or whatis like it used to be in old days). 2. do_man immediately calls man_parse_args, which uses shell getopt to parse argv flags and so on. Within this same code: if -f is set, run do_whatis w= ith arguments. If -k is set, run do_apropos with arguments. 3. Looking at do_apropos we find this magical beast: 957 do_apropos() { 958 [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos= ) ] && \ 959 exec apropos "$@" 960 search_whatis apropos "$@" 961 } That's... interesting. What is with the stat calls? Why are we comparing inode numbers of these two binaries, and then call exec on apropos if they don't match? 213317 gordon do_apropos() { 285836 bapt [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ] && \ 285835 bapt exec apropos "$@" 213317 gordon search_whatis apropos "$@" 213317 gordon } What are these two commits? r285836 | bapt | 2015-07-24 02:20:02 -0700 (Fri, 24 Jul 2015) | 4 lines inode should be different to actually mean mandocdb is in used Sponsored by: gandi.net ------------------------------------------------------------------------ r285835 | bapt | 2015-07-24 02:10:03 -0700 (Fri, 24 Jul 2015) | 9 lines Fix man -k with mandocdb If apropos(1) and whatis(1) are not hardlinks to man(1) that means the syst= em is using mandocdb, then man -k should spawn apropos(1) and/or whatis(1) direct= ly Reported by: kevlo Tested by: kevlo Sponsored by: gandi.net So they happened over 3 years ago. There's no PR/GNATS/bug number in the commit. Should I search mailing lists? At this point I opt to say no. The commit message says that the intended goal was to "fix man -k", so did this break things or did something else cause the breakage (ex. /usr/local/etc/man.d/*.conf introduction?). My guess is the Gandi folks are building their systems with WITHOUT_MANDOCDB=3Dtrue or WITHOUT_MAN_UTILS=3D= true (not sure which) in /etc/src.conf. Anyway, let's look into the man.1 man p= age to see if we can find out when THAT feature got added. 213317 gordon .Sh FILES 213317 gordon .Bl -tag -width indent -compact 213317 gordon .It Pa /etc/man.conf 213317 gordon System configuration file. 213317 gordon .It Pa /usr/local/etc/man.d/*.conf 213317 gordon Local configuration files. 213317 gordon .El Long before r285835 and r285836, so /usr/local/etc/man.d/*.conf is not responsible for this. No wonder I didn't see this breakage in stable/9. So now that we know who to blame and understand WHY there's breakage, let's figure out how to fix it. apropos and whatis both comprehend the MANPATH environment variable if pres= ent. On FreeBSD it isn't by default. FreeBSD has the manpath command, esp. man= path -q which is used in periodic scripts like weekly/320.whatis. man's man page says it comprehends /usr/local/etc/man.d/*.conf, so surely it has the smarts somewhere, but it isn't getting called when using man -k or man -f. I chan= ged #! /bin/sh to #! /bin/sh -x and witnessed proof of this. So back to the man shell script we go, starting from step 4... 4. In do_man(), if man -k or man -f aren't used, the do_apropos and do_what= is aren't called. So what happens in that case? Interesting function name: 969 man_setup 5. man_setup() does some magical things relating to environment variables within the script itself (NOTHING is export-ed). Near the end of it, I see: 628 build_manpath 629 man_setup_locale 630 man_setup_width Hey, that build_manpath() looks like it might be relevant. 6. build_manpath() parse all kinds of fun stuff. Most importantly, it call= s a function called parse_configs, which does in fact comprehend /usr/local/etc/man.d/*.conf and so on. Near the end, build_manpath() sets = the env var MANPATH to what should be a proper manpath. What if we were to call build_manpath ourselves and export MANPATH? Let's = try this code in do_apropos() and then try man -k: 957 do_apropos() { 958 build_manpath 959 export MANPATH 960 [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos= ) ] && \ 961 exec apropos "$@" 962 search_whatis apropos "$@" 963 } $ ./man.sh -k IPC::Open3 IPC::Open3(3) - open a process for reading, writing, and error handling usi= ng open3() It works! Mostly. There is some locale-based support in man.sh but it's mainly used by manpath (which is also this script) and the OLD whatis used to do: call search_what= is() back in the old days. 7. search_whatis() does this right at the start: 817 search_whatis() { ... 825 build_manpath 826 build_manlocales 827 setup_pager Very similar. build_manlocales() sets an env var called MANLOCALES that is only used within search_whatis() itself. setup_pager() isn't relevant to m= an -k output. So while I can't get the locale bits to work for this because some code from search_whatis() would need to be made its own function (or maybe apropos/wh= atis modified to support MANLOCALE env var), what I CAN do is get man -f and man= -k working with this simple change: $ svnlite diff Index: man.sh =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- man.sh (revision 333170) +++ man.sh (working copy) @@ -955,6 +955,8 @@ whatis_usage() { # Supported commands do_apropos() { + build_manpath + export MANPATH [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/apropos) ] &= & \ exec apropos "$@" search_whatis apropos "$@" @@ -992,6 +994,8 @@ do_manpath() { } do_whatis() { + build_manpath + export MANPATH [ $(stat -f %i /usr/bin/man) -ne $(stat -f %i /usr/bin/whatis) ] &&= \ exec whatis "$@" search_whatis whatis "$@" Confirmation: $ ./man.sh -k IPC::Open3 IPC::Open3(3) - open a process for reading, writing, and error handling usi= ng open3() $ ./man.sh -f IPC::Open3 IPC::Open3(3) - open a process for reading, writing, and error handling usi= ng open3() In summary: I'm amazed that man -k/-f has essentially been broken for almos= t 3 years (for ports/pkgs, not base system) and nobody's complained. Like, TRU= LY amazed/baffled. So what are apropos/whatis now? Well, they're from OpenBSD, and the source= is in src/contrib/mdocml. This is code for mandoc, makewhatis, apropos, and whatis. I just don't see a native way to make this code work with /usr/local/etc/man.d/*.conf so man (the shell script) will have to be the "gateway" for now. Also: I just want to give a shoutout to Bug 223555 because it looks like apropos/whatis may on stable/10 been the old shell script version which use= rs have come to rely on. Maybe the tail end of this PR will help explain that. --=20 You are receiving this mail because: You are the assignee for the bug.=