From owner-freebsd-bugs@freebsd.org Thu Feb 28 18:53:04 2019 Return-Path: Delivered-To: freebsd-bugs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60A941513477 for ; Thu, 28 Feb 2019 18:53:04 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: from mail-it1-f174.google.com (mail-it1-f174.google.com [209.85.166.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7194E81DA1 for ; Thu, 28 Feb 2019 18:53:03 +0000 (UTC) (envelope-from cse.cem@gmail.com) Received: by mail-it1-f174.google.com with SMTP id w18so16264689itj.4 for ; Thu, 28 Feb 2019 10:53:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc:content-transfer-encoding; bh=vY5oPQYD97qIUeKThONSR5gHd8DsNBZjcgnoHnIrj1o=; b=Rjm4sLRTDvSqAVBIonJS2YEuC7Zz1iVJBAo++sU0EjODEIYOKfLfVONa5oSxzXpoRe rBnvnESrB8RoOemkGhJbmb0DQzTmdQ9DNTezGVCbRh6c8pigHy75qV7Cko7emmbuKp+8 rtws5MxiPlVyoxLqxoibF8UsH2b2n1xRkAkrGqlIju69QZU5UuTj5vgZWFps5MLEv/3U 9C4/VaiOvC6GbN5sepxNvehi7PcQZ2+sPwUnUPwjc71SeznDosy4lNUpdkXyp8MTLp54 /IdfvfdryA8rwwLl81Y33dijzFvwVVoBUJKo8ONuKvGdyyOp+3DI09lC5nMAZC6MZ2rg rOfw== X-Gm-Message-State: APjAAAXyLzmzWqFYD5dk05Uceyj6I49DncD2fFJU/2ypeJklH4nuWzbH 1jvS/cV2y9JBI3aAxwapnBkdHWTY X-Google-Smtp-Source: AHgI3IbSazGm0S152GLfp3FwKBU6rWwHhikvlM8K81t39etcF+g7wW6HK8YUUmolqgJuydiFEq89vQ== X-Received: by 2002:a24:5ec1:: with SMTP id h184mr830341itb.4.1551379977113; Thu, 28 Feb 2019 10:52:57 -0800 (PST) Received: from mail-io1-f45.google.com (mail-io1-f45.google.com. [209.85.166.45]) by smtp.gmail.com with ESMTPSA id p11sm7793863ios.15.2019.02.28.10.52.56 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Feb 2019 10:52:56 -0800 (PST) Received: by mail-io1-f45.google.com with SMTP id x9so17438626iog.12 for ; Thu, 28 Feb 2019 10:52:56 -0800 (PST) X-Received: by 2002:a5e:9412:: with SMTP id q18mr449095ioj.233.1551379976080; Thu, 28 Feb 2019 10:52:56 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: Reply-To: cem@freebsd.org From: Conrad Meyer Date: Thu, 28 Feb 2019 10:52:45 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Blacklistd not recognizing probing attemtps To: =?UTF-8?B?TMOhc3psw7MgS8Ohcm9seWk=?= Cc: freebsd-bugs@freebsd.org, christos@netbsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 7194E81DA1 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of csecem@gmail.com designates 209.85.166.174 as permitted sender) smtp.mailfrom=csecem@gmail.com X-Spamd-Result: default: False [-5.81 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[cem@freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; REPLYTO_ADDR_EQ_FROM(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.93)[-0.927,0]; FORGED_SENDER(0.30)[cem@freebsd.org,csecem@gmail.com]; IP_SCORE(-2.88)[ip: (-8.47), ipnet: 209.85.128.0/17(-3.81), asn: 15169(-2.03), country: US(-0.07)]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_NEQ_ENVFROM(0.00)[cem@freebsd.org,csecem@gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-bugs@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[174.166.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2019 18:53:04 -0000 Hi L=C3=A1szl=C3=B3, On Tue, Feb 26, 2019 at 5:53 PM L=C3=A1szl=C3=B3 K=C3=A1rolyi wrote: > I'm on 12.0-RELEASE-p3 and I have configured blacklistd with sshd to > lock out those random IPs that are probing my server. The problem is, I > noticed that in many cases, blacklistd does not put the offending IP on > its list. > > I've contacted Christos Zoulas in email to see if he has anything to > tell about it, and after putting blacklistd in debug mode and > reproducing the issue, he suggested to contact you with the it. So here > it is. I'll paste a couple lines from the sshd log, I get these, which > aren't registered for some reason: I think there are a couple suspicious issues in FreeBSD blacklist(d) and sshd's blacklist support, some of which explains what you see. First, FreeBSD blacklistd ignores BAD_USER failures entirely.[1.1] This is not ideal. That includes things like "Invalid user FOO from XXX port YYY" in your logs. Those attempts will never result in IPs being filtered. 2 of 5 blacklist failure notifications in ssh are BAD_USER types (i.e., nops). The rest are ordinary AUTH_FAIL. Second, libblacklist opens reporting connections SOCK_CLOEXEC (or FD_CLOEXEC if the sock macro is unavailable), and the FreeBSD sshd libblacklist patches only invoke BLACKLIST_INIT() in one location. The problem is that sshd loves to fork. The patch happens to initialize the BL reporting socket in the primary process associated with any given connection (post-accept fork), but sshd forks multiple times *after* this in privsep mode for preauth (i.e., the authentication process), and even something postauth. When it forks, the logging socket is closed by the operating system. *Privsep is enabled by default in sshd.* BLACKLIST_NOTIFY -> blacklist_r -> bl_send has a mechanism to detect failures, reset the connection, and retry. However, it will (a) spuriously close an invalid fd that no longer represents the cloexec socket, and (b), execute in a context where the _PATH_BLSOCK (/var/run/blacklistd.sock) is no longer available due to chroot, capsicum, or other sandboxing. Third, sshd creates the login grace timeout handler, then begins initial packet version exchange, then actually initializes libblacklist. There is a BLACKLIST_NOTIFY in the login grace timeout handler. This means an abusive client that stalls initial version exchange can't be reported, because the timeout handler runs before libblacklist is initialized. (It also means we can't notify libblacklist if initial version exchange detects something bad =E2=80=94 bu= t we don't try to at this time, AFAICT.) The 1st seems like an issue local to FreeBSD =E2=80=94 upstream blacklistd does not appear to have a BAD_USER concept at all[1.2]. Importantly, upstream treats all of these as AUTH_FAIL. The 2nd issue is shared with NetBSD, as far as I can tell. The details are a little different, but both close arbitrary invalid fds, and both may be chrooted when attempting to reinitialize a forked sshd process's blacklist reporting socket. NetBSD has the same cloexec behavior[2.1] and also defaults to using privsep[2.2][2.3]. NetBSD's pfilter.c[2.4] looks slightly different from our blacklist.c[2.5]. However, they are not significantly different. The 3rd issue is sort of unique to FreeBSD: NetBSD initializes libblacklist much earlier in sshd:main(), via server_accept_loop(). In fact, NetBSD initializes blacklist (pfilter) prior to the fork() for the client connection. This means that in NetBSD, the initial blacklist socket is closed before any sshd authentication begins at all! bl_send() -> bl_reset / bl_init may recover in some contexts, but maybe not others. Another important distinction is that NetBSD has a catch-all pfilter_notify() in cleanup_exit() when the exitcode matches 255; FreeBSD's sshd is missing that one. Best regards, Conrad [1.1]: https://github.com/freebsd/freebsd/blob/master/contrib/blacklist/bin= /blacklistd.c#L263 [1.2]: https://github.com/NetBSD/src/blob/trunk/external/bsd/blacklist/incl= ude/bl.h#L40 [2.1]: https://github.com/NetBSD/src/blob/trunk/external/bsd/blacklist/lib/= bl.c#L130-L161 [2.2]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh= /dist/sshd.c#L232 [2.3]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh= /dist/servconf.c#L526 [2.4]: https://github.com/NetBSD/src/blob/trunk/crypto/external/bsd/openssh= /dist/pfilter.c#L32-L38 [2.5]: https://github.com/freebsd/freebsd/blob/master/crypto/openssh/blackl= ist.c#L83-L96