Date: Thu, 28 Feb 2019 10:52:45 -0800 From: Conrad Meyer <cem@freebsd.org> To: =?UTF-8?B?TMOhc3psw7MgS8Ohcm9seWk=?= <laszlo@karolyi.hu> Cc: freebsd-bugs@freebsd.org, christos@netbsd.org Subject: Re: Blacklistd not recognizing probing attemtps Message-ID: <CAG6CVpWr7%2BBnfi%2BHxk2DeUHaQ2JeWhjp_=PzKzohHWCbbbC5Fg@mail.gmail.com> In-Reply-To: <cf0f5a82-b15e-64b9-53de-c0bf7de9ce10@karolyi.hu> References: <cf0f5a82-b15e-64b9-53de-c0bf7de9ce10@karolyi.hu>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi L=C3=A1szl=C3=B3, On Tue, Feb 26, 2019 at 5:53 PM L=C3=A1szl=C3=B3 K=C3=A1rolyi <laszlo@karol= yi.hu> wrote: > I'm on 12.0-RELEASE-p3 and I have configured blacklistd with sshd to > lock out those random IPs that are probing my server. The problem is, I > noticed that in many cases, blacklistd does not put the offending IP on > its list. > > I've contacted Christos Zoulas in email to see if he has anything to > tell about it, and after putting blacklistd in debug mode and > reproducing the issue, he suggested to contact you with the it. So here > it is. I'll paste a couple lines from the sshd log, I get these, which > aren't registered for some reason: I think there are a couple suspicious issues in FreeBSD blacklist(d) and sshd's blacklist support, some of which explains what you see. First, FreeBSD blacklistd ignores BAD_USER failures entirely.[1.1] This is not ideal. That includes things like "Invalid user FOO from XXX port YYY" in your logs. Those attempts will never result in IPs being filtered. 2 of 5 blacklist failure notifications in ssh are BAD_USER types (i.e., nops). The rest are ordinary AUTH_FAIL. Second, libblacklist opens reporting connections SOCK_CLOEXEC (or FD_CLOEXEC if the sock macro is unavailable), and the FreeBSD sshd libblacklist patches only invoke BLACKLIST_INIT() in one location. The problem is that sshd loves to fork. The patch happens to initialize the BL reporting socket in the primary process associated with any given connection (post-accept fork), but sshd forks multiple times *after* this in privsep mode for preauth (i.e., the authentication process), and even something postauth. When it forks, the logging socket is closed by the operating system. *Privsep is enabled by default in sshd.* BLACKLIST_NOTIFY -> blacklist_r -> bl_send has a mechanism to detect failures, reset the connection, and retry. However, it will (a) spuriously close an invalid fd that no longer represents the cloexec socket, and (b), execute in a context where the _PATH_BLSOCK (/var/run/blacklistd.sock) is no longer available due to chroot, capsicum, or other sandboxing. Third, sshd creates the login grace timeout handler, then begins initial packet version exchange, then actually initializes libblacklist. There is a BLACKLIST_NOTIFY in the login grace timeout handler. This means an abusive client that stalls initial version exchange can't be reported, because the timeout handler runs before libblacklist is initialized. (It also means we can't notify libblacklist if initial version exchange detects something bad =E2=80=94 bu= t we don't try to at this time, AFAICT.) The 1st seems like an issue local to FreeBSD =E2=80=94 upstream blacklistd does not appear to have a BAD_USER concept at all[1.2]. Importantly, upstream treats all of these as AUTH_FAIL. The 2nd issue is shared with NetBSD, as far as I can tell. The details are a little different, but both close arbitrary invalid fds, and both may be chrooted when attempting to reinitialize a forked sshd process's blacklist reporting socket. NetBSD has the same cloexec behavior[2.1] and also defaults to using privsep[2.2][2.3]. NetBSD's pfilter.c[2.4] looks slightly different from our blacklist.c[2.5]. However, they are not significantly different. The 3rd issue is sort of unique to FreeBSD: NetBSD initializes libblacklist much earlier in sshd:main(), via server_accept_loop(). In fact, NetBSD initializes blacklist (pfilter) prior to the fork() for the client connection. This means that in NetBSD, the initial blacklist socket is closed before any sshd authentication begins at all! bl_send() -> bl_reset / bl_init may recover in some contexts, but maybe not others. Another important distinction is that NetBSD has a catch-all pfilter_notify() in cleanup_exit() when the exitcode matches 255; FreeBSD's sshd is missing that one. Best regards, Conrad [1.1]: https://github.com/freebsd/freebsd/blob/master/contrib/blacklist/bin= /blacklistd.c#L263 [1.2]: https://github.com/NetBSD/src/blob/trunk/external/bsd/blacklist/incl= ude/bl.h#L40 [2.1]: https://github.com/NetBSD/src/blob/trunk/external/bsd/blacklist/lib/= bl.c#L130-L161 [2.2]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh= /dist/sshd.c#L232 [2.3]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh= /dist/servconf.c#L526 [2.4]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh= /dist/pfilter.c#L32-L38 [2.5]: https://github.com/freebsd/freebsd/blob/master/crypto/openssh/blackl= ist.c#L83-L96
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAG6CVpWr7%2BBnfi%2BHxk2DeUHaQ2JeWhjp_=PzKzohHWCbbbC5Fg>