Date: Tue, 14 Nov 2017 09:31:36 +0100 From: Cos Chan <rosettas@gmail.com> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-questions <freebsd-questions@freebsd.org>, Michael Ross <gmx@ross.cx>, Kurt Lidl <lidl@freebsd.org> Subject: Re: How to setup IPFW working with blacklistd Message-ID: <CAKV%2BxLAt4Ciqmg2w1iJK42jq6f%2BnumASKMQ=UL6dT%2BCdGYujVQ@mail.gmail.com> In-Reply-To: <CAKV%2BxLDicLze3Dvd2i7HGWJUxCdSLjvhuWWZUJ65pMi%2Bx483=A@mail.gmail.com> References: <mailman.87.1509969603.28633.freebsd-questions@freebsd.org> <20171106235944.U9710@sola.nimnet.asn.au> <CAKV%2BxLCizjt5M%2BmJmTZj-cr=D6rhXRwDjCkE=6Q-VQX73iY%2B4A@mail.gmail.com> <20171107033226.M9710@sola.nimnet.asn.au> <CAKV%2BxLBWgU6zmc7tQNA=0%2B=2aF23C1QfJ2i3q1gKYDttwsCTkg@mail.gmail.com> <20171107162914.G9710@sola.nimnet.asn.au> <CAKV%2BxLDQQcG3bvo1b2nUAu7oOVhdNzDDrPWTVp2qOmkWVV89BQ@mail.gmail.com> <20171108012948.A9710@sola.nimnet.asn.au> <CAKV%2BxLCQ9NE6%2BEg6NvHZuEED8Cf6ZX74unvk9ajfLyG-yA2rXA@mail.gmail.com> <CAKV%2BxLAkfiQCLXfgZOtQGUXOW8gYN7sjOD5uWezv-N%2BTBjybMQ@mail.gmail.com> <20171111213759.I72828@sola.nimnet.asn.au> <CAKV%2BxLDicLze3Dvd2i7HGWJUxCdSLjvhuWWZUJ65pMi%2Bx483=A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Nov 13, 2017 at 3:17 PM, Cos Chan <rosettas@gmail.com> wrote:
>
>
> On Sat, Nov 11, 2017 at 1:42 PM, Ian Smith <smithi@nimnet.asn.au> wrote:
>
>> On Thu, 9 Nov 2017 14:25:52 +0100, Cos Chan wrote:
>>
>> > Dear All
>> >
>> > Thanks Ian's great help, I have solved problem to post banned entries
>> from
>> > blacklistd to ipfw.
>>
>> Well, we're some of the way there :) We really need Kurt Lidl's eyes on
>> this to make real progress, and indications are that my and your emails
>> cc'ing him were still being deferred for some reason - maybe he's away?
>>
>> > The original message was received at Tue, 7 Nov 2017 10:12:05 -0500
>> (EST)
>> > from mx2.freebsd.org [8.8.178.116]
>> >
>> > ----- Transcript of session follows -----
>> > <lidl@pix.net>... Deferred: Operation timed out with hydra.pix.net.
>> > Warning: message still undelivered after 4 hours
>> > Will keep trying until message is 1 week, 3 days old
>>
>>
>> > To my knowledge the problem is:
>> >
>> > I setup sshd+blacklistd without ipfw at first. Then I got problem the
>> entry
>> > was never reached nfail number (is it a bug?).
>>
>> The first issue was because of a severe deficiency in blacklistd-helper,
>> in that it doesn't actually check that the chosen firewall is running,
>> and it then fails to detect commands for that firewall that do not (can
>> not) succeed as any sort of error! More about that below.
>>
>> The second, however, was mainly that you missed that nfail set to '*'
>> means that the host is NOT to be blocked, no matter how many auth or
>> other failures that (in this case) sshd reports.
>>
>> That also answers another question you had .. "nnn/-1" indicates that
>> nfail=* ie never to be blocked. These still get accumulated in the
>> database, but are not applied as ipfw block rule table entries.
>>
>>
>> > so I have to change the nfail to * to get the entry into banned list.
>>
>> In combination with other factors - like whether ipfw was running at the
>> time - that got blacklistd to record reported failures to its database,
>> but not to execute the 'add' commands to blacklistd-helper, so that
>> address was not in fact blocked, and subsequent attempts kept trying.
>>
>> > But while I setup ipfw, the nfail=* would not activate
>> blacklistd-helper so
>> > no entry in blacklist banned list were added to ipfw.
>>
>> Yes, nfail=* means NEVER block these failed addreses. blacklistd.conf(5)
>>
>> > I have modify the blacklistd nfail to 2, sshd MaxAuthTries to 3. The
>> > blacklist entries working fine.
>>
>> With ipfw running, yes :) But it should have failed - noisily - sooner.
>>
>> When ipfw is running, issuing this will show you the addresses blocked:
>>
>> # ipfw table port22 list
>>
>
> until now it seems working on list updating. but I am not sure if it is
> really working fine.
>
> here is one strange record:
>
> $ sudo blacklistctl dump -b | grep 1662
> 193.201.224.218/32:22 OK 1662/1 2017/11/13 00:31:04
>
> This IP was blocked in ipfw from last week. while I checked it last week
> Friday it was 800+/1 in blacklist and until today it become 1662.
>
> To my knowledge the ipfw should block the connection, the times of banned
> IP should be not increased?
>
> I could see more entries with more than 3/1, for example:
>
> 89.160.221.132/32:22 OK 18/1 2017/11/13 00:01:21
> 60.125.42.119/32:22 OK 3/1 2017/11/12 16:13:53
> 166.62.35.180/32:22 OK 3/1 2017/11/10 06:36:25
> 202.162.221.51/32:22 OK 6/1 2017/11/10 00:42:14
> 168.0.114.130/32:22 OK 3/1 2017/11/10 23:40:30
> 95.145.71.165/32:22 OK 3/1 2017/11/11 07:07:07
> 123.161.206.210/32:22 OK 3/1 2017/11/12 18:14:00
> 203.146.208.208/32:22 OK 6/1 2017/11/10 10:16:21
> 149.56.223.241/32:22 OK 1/1 2017/11/12 06:09:16
> 121.169.217.98/32:22 OK 9/1 2017/11/12 21:59:57
> 211.251.237.162/32:22 OK 2/1 2017/11/13 12:08:07
> 103.99.0.116/32:22 OK 30/1 2017/11/10 14:56:07
>
> These records I am not sure if they were not increased after added to ipfw
> list. but the 1662 times one, I am sure it was increased after ipfw had the
> ip in list.
>
add the ipfw rules:
$ sudo ipfw list
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any ip6 icmp6types 1
01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
02022 deny tcp from table(port22) to any dst-port 22
65000 allow ip from any to any
65535 deny ip from any to any
>
>
>> > BUT I found another problem.
>> >
>> > The output of blacklist dump is strange:
>> >
>> > $ sudo blacklistctl dump
>> > address/ma:port id nfail last access
>> > 96.227.104.132/32:22 0/2 1970/01/01 01:00:00
>> > 89.245.78.187/32:22 0/2 1970/01/01 01:00:00
>> > 116.193.162.203/32:22 1/2 2017/11/09 11:48:05
>> >
>> > Since the blacklistd accepts instruction from sshd. how could be 0/2
>> > entries presented there? I am sure my successful logins were not added
>> to
>> > blacklistd.
>>
>> 1970/01/01 01:00:00 is just the UNIX '0' timestamp, in this case plus
>> one hour (your TZ offset). It here means 'no previous entry'. Not sure
>> about that 0/2, but there are several different codes returned by sshd
>> including success, failed auth and 'abusive behaviour' .. I'm not sure
>> which ones your reports (including in off-list mail) indicate.
>>
>> As for the mysterious 'n-1' behaviour you mentioned offlist for nfail,
>> in /usr/src/contrib/blacklist/bin/blacklistd.c there's this:
>>
>> switch (bi->bi_type) {
>> case BL_ABUSE:
>> /*
>> * If the application has signaled abusive behavior,
>> * set the number of fails to be one less than the
>> * configured limit. Fallthrough to the normal BL_ADD
>> * processing, which will increment the failure count
>> * to the threshhold, and block the abusive address.
>> */
>> if (c.c_nfail != -1)
>> dbi.count = c.c_nfail - 1;
>> /*FALLTHROUGH*/
>> case BL_ADD:
>> dbi.count++;
>> dbi.last = ts.tv_sec;
>> if (dbi.id[0]) {
>> /*
>> * We should not be getting this since the rule
>> * should have blocked the address. A possible
>> * explanation is that someone removed that rule,
>> * and another would be that we got another
>> attempt
>> * before we added the rule. In anycase, we remove
>> * and re-add the rule because we don't want to
>> add
>> * it twice, because then we'd lose track of it.
>> */
>> (*lfun)(LOG_DEBUG, "rule exists %s", dbi.id);
>> (void)run_change("rem", &c, dbi.id, 0);
>> dbi.id[0] = '\0';
>> }
>> if (c.c_nfail != -1 && dbi.count >= c.c_nfail) {
>> int res = run_change("add", &c, dbi.id, sizeof(
>> dbi.id));
>> if (res == -1)
>> goto out;
>> sockaddr_snprintf(rbuf, sizeof(rbuf), "%a",
>> (void *)&rss);
>> (*lfun)(LOG_INFO,
>> "blocked %s/%d:%d for %d seconds",
>> rbuf, c.c_lmask, c.c_port, c.c_duration);
>>
>> }
>> break;
>>
>> But if the 'add' command via blacklistd-helper fails, it will never add
>> the 1 .. I'm not certain about this, but it could explain what you see,
>> although I can't discern whether sshd is reporting BL_ADD or BL_ABUSE.
>>
>> You might instead try MaxAuthTries 4 .. sshd_config(5) says:
>>
>> MaxAuthTries
>> Specifies the maximum number of authentication attempts
>> permitted
>> per connection. Once the number of failures reaches half
>> this
>> value, additional failures are logged. The default is 6.
>>
>> Half of 3 as an integer is only 1, but half of 4 is 2. See if it helps?
>>
>
> I didnt change the MaxAuthTries, since I found something interesting from
> the different logs concerning that issue:
>
> From blacklistctl dump:
>
> $ sudo blacklistctl dump
> address/ma:port id nfail last access
> 78.203.146.34/32:22 0/1 1970/01/01 01:00:00
> 195.225.116.21/32:22 0/1 1970/01/01 01:00:00
> 123.31.26.123/32:22 0/1 1970/01/01 01:00:00
> 112.148.101.13/32:22 0/1 1970/01/01 01:00:00
> 93.23.6.18/32:22 0/1 1970/01/01 01:00:00
> 5.102.197.124/32:22 0/1 1970/01/01 01:00:00
> 193.154.127.32/32:22 0/1 1970/01/01 01:00:00
> 113.232.216.41/32:22 0/1 1970/01/01 01:00:00
>
> From sshd log:
>
> Nov 10 17:57:41 res sshd[49839]: Invalid user pi from 193.154.127.32
> Nov 10 17:57:41 res sshd[49840]: Invalid user pi from 193.154.127.32
> Nov 10 17:57:41 res sshd[49840]: input_userauth_request: invalid user pi
> [preauth]
> Nov 10 17:57:41 res sshd[49839]: input_userauth_request: invalid user pi
> [preauth]
> ...
> Nov 11 03:50:47 res sshd[57896]: Invalid user support from 123.31.26.123
> Nov 11 03:50:47 res sshd[57896]: input_userauth_request: invalid user
> support [preauth]
> Nov 11 03:50:47 res sshd[57896]: error: Received disconnect from
> 123.31.26.123 port 55811:3: com.jcraft.jsch.JSchException: Auth fail
> [preauth]
> Nov 11 03:50:49 res sshd[57898]: Invalid user admin from 123.31.26.123
> Nov 11 03:50:49 res sshd[57898]: input_userauth_request: invalid user
> admin [preauth]
> Nov 11 03:50:49 res sshd[57898]: error: Received disconnect from
> 123.31.26.123 port 57823:3: com.jcraft.jsch.JSchException: Auth fail
> [preauth]
> Nov 11 03:50:51 res sshd[57900]: Invalid user admin from 123.31.26.123
> Nov 11 03:50:51 res sshd[57900]: input_userauth_request: invalid user
> admin [preauth]
> Nov 11 03:50:51 res sshd[57900]: error: Received disconnect from
> 123.31.26.123 port 59819:3: com.jcraft.jsch.JSchException: Auth fail
> [preauth]
> Nov 11 03:50:53 res sshd[57902]: Invalid user ubnt from 123.31.26.123
> Nov 11 03:50:53 res sshd[57902]: input_userauth_request: invalid user ubnt
> [preauth]
> Nov 11 03:50:53 res sshd[57902]: error: Received disconnect from
> 123.31.26.123 port 61795:3: com.jcraft.jsch.JSchException: Auth fail
> [preauth]
> Nov 11 03:50:55 res sshd[57904]: Invalid user PlcmSpIp from 123.31.26.123
> Nov 11 03:50:55 res sshd[57904]: input_userauth_request: invalid user
> PlcmSpIp [preauth]
> Nov 11 03:50:55 res sshd[57904]: error: Received disconnect from
> 123.31.26.123 port 61920:3: com.jcraft.jsch.JSchException: Auth fail
> [preauth]
> Nov 11 03:50:57 res sshd[57906]: Invalid user admin from 123.31.26.123
> Nov 11 03:50:57 res sshd[57906]: input_userauth_request: invalid user
> admin [preauth]
> Nov 11 03:50:57 res sshd[57906]: error: Received disconnect from
> 123.31.26.123 port 61949:3: com.jcraft.jsch.JSchException: Auth fail
> [preauth]
>
> I see 2 problems:
>
> Problem 1:
> The IP 193.154.127.32 didn't reach sshd maximum authentication (=3), it
> tried only 2 times.
> But in my opinion it should be recorded to blacklistd as 2/1 instead of
> 0/1.
>
> Problem 2:
> The IP 123.31.26.123 was trying to use different user name to login more
> than 3 times. it was also recorded in blacklistd as 0/1.
>
> In my opinion the above 2 all should be banned by blacklistd.
>
>
>>
>> > I am trying to find out the reason from log but I dont know how to see
>> > blacklistd log. man page said that is to syslogd but what the facility
>> it
>> > is? or some other ways to get out log?
>>
>> Not sure of the facility but when using the -v switch, as you have been,
>> logging goes to stderr instead of syslog. Without -v you should see it
>> logging to /var/log/messages. If not, try adding to /etc/syslog.conf:
>>
>> !blacklistd
>> *.* /var/log/myblacklistd.log
>>
>> then '# touch /var/log/myblacklistd.log && service syslogd restart'
>>
>
> Unfortunately I started the logging later than Nov 11 03:50:57, so I didnt
> get the log of "0/1" records yet.
>
got the log for one new "0/1" entry:
$ sudo blacklistctl dump
address/ma:port id nfail last access
24.7.90.146/32:22 0/1 1970/01/01 01:00:00
...
$ sudo cat auth.log | grep 24.7.90.146
Nov 14 02:13:58 res sshd[6212]: Invalid user pi from 24.7.90.146
Nov 14 02:13:58 res sshd[6215]: Invalid user pi from 24.7.90.146
Nov 14 02:13:59 res sshd[6215]: Connection closed by 24.7.90.146 port 34746
[preauth]
Nov 14 02:13:59 res sshd[6212]: Connection closed by 24.7.90.146 port 34742
[preauth]
$ cat myblacklistd.log | grep 'Nov 14'
...
Nov 14 02:09:11 res blacklistd[5590]: blocked 202.51.74.55/32:22 for -1
seconds
Nov 14 02:11:06 res blacklistd[5590]: rule exists OK
Nov 14 02:11:06 res blacklistd[5590]: blocked 202.51.74.55/32:22 for -1
seconds
Nov 14 02:14:43 res blacklistd[5590]: blocked 66.232.147.46/32:22 for -1
seconds
Nov 14 02:16:40 res blacklistd[5590]: rule exists OK
could not see operation against that IP from blacklistd.log
>
>
>>
>> Ok, problems with blacklistd-helper; the first bit verbatim, tabs lost:
>>
>> #!/bin/sh
>> #echo "run $@" 1>&2
>> #set -x
>> # $1 command
>> # $2 rulename
>> # $3 protocol
>> # $4 address
>> # $5 mask
>> # $6 port
>> # $7 id
>>
>> pf=
>> if [ -f "/etc/ipfw-blacklist.rc" ]; then
>> pf="ipfw"
>> . /etc/ipfw-blacklist.rc
>> ipfw_offset=${ipfw_offset:-2000}
>> fi
>>
>> if [ -z "$pf" ]; then
>> for f in npf pf ipf; do
>> if [ -f "/etc/$f.conf" ]; then
>> pf="$f"
>> break
>> fi
>> done
>> fi
>>
>> if [ -z "$pf" ]; then
>> echo "$0: Unsupported packet filter" 1>&2
>> exit 1
>> fi
>>
>> Earlier you said you'd run it without /etc/ipfw-blacklist.rc existing.
>> In that case - UNLESS you had either /etc/pf.conf or /etc/ipf.conf lying
>> around from before? it should have failed with 'exit 1' .. though it's
>> not clear from browsing the code that even that would cause it to quit.
>>
>
> No, there are not /etc/pf.conf and /etc/ipf.conf.
>
>
>>
>> So once /etc/ipfw-blacklist.rc exists, that's a flag indicating you
>> intend using ipfw, however there's NO check that ipfw is running ..
>>
>> Then - ignoring the pf) and ipf) sections - though I suspect they'd have
>> the same issue unless really running - here's the ipfw add bit, no tabs:
>>
>> add)
>> case "$pf" in
>> [..]
>> ipfw)
>> # use $ipfw_offset+$port for rule number
>> rule=$(($ipfw_offset + $6))
>> tname="port$6"
>> /sbin/ipfw table $tname create type addr 2>/dev/null
>>
>> Unless ipfw is running, enabled, that will fail - silently.
>>
>> /sbin/ipfw -q table $tname add "$addr/$mask"
>>
>> Ditto, perhaps with a message to stderr - that's simply ignored.
>>
>> # if rule number $rule does not already exist, create it
>> /sbin/ipfw show $rule >/dev/null 2>&1 || \
>> /sbin/ipfw add $rule drop $3 from \
>> table"("$tname")" to any dst-port $6 >/dev/null
>> && \
>> echo OK
>> ;;
>>
>> When both of these ipfw commands also fail, it'll only fail to echo OK.
>>
>> Not that failing to echo OK seems to matter to the calling code, but
>> the OK is kept as 'id' which is passed to the rem)ove code, but is
>> unused except by the npf firewall .. 'netbsd packet filter' I guess.
>>
>> I can certainly suggest patches for at least the ipfw sections - and
>> really, if the introductory code checks ipfw is working that should be
>> enough - but I'm unsure whether 'exit 1' after an error message is all
>> that's needed to get blacklistd to whinge loudly and refuse to continue?
>>
>> This should be turned into a PR via bugzilla, but since I'm not running
>> 11.x here, I can only really contribute if you do so and add me as a cc.
>>
>
> Sorry I dont know how to describe the problem in bugzilla since I dont
> really understand what you said.
> I have to learn more about the script :)
>
>
>>
>> Please try to avoid top-posting on replies, thanks.
>
>
> Sure, I will.
>
>
>>
>> cheers, Ian
>>
>
>
>
> --
> with kind regards
>
--
with kind regards
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKV%2BxLAt4Ciqmg2w1iJK42jq6f%2BnumASKMQ=UL6dT%2BCdGYujVQ>