Date: Tue, 14 Nov 2017 15:38:51 +0100 From: Cos Chan <rosettas@gmail.com> To: Ian Smith <smithi@nimnet.asn.au> Cc: freebsd-questions <freebsd-questions@freebsd.org>, Michael Ross <gmx@ross.cx>, Kurt Lidl <lidl@freebsd.org> Subject: Re: How to setup IPFW working with blacklistd Message-ID: <CAKV%2BxLD_KE938JnmjDE=CmfZ7bOJ1CaqvWuQ%2B0jDzQNWM%2B6yLg@mail.gmail.com> In-Reply-To: <CAKV%2BxLAt4Ciqmg2w1iJK42jq6f%2BnumASKMQ=UL6dT%2BCdGYujVQ@mail.gmail.com> References: <mailman.87.1509969603.28633.freebsd-questions@freebsd.org> <20171106235944.U9710@sola.nimnet.asn.au> <CAKV%2BxLCizjt5M%2BmJmTZj-cr=D6rhXRwDjCkE=6Q-VQX73iY%2B4A@mail.gmail.com> <20171107033226.M9710@sola.nimnet.asn.au> <CAKV%2BxLBWgU6zmc7tQNA=0%2B=2aF23C1QfJ2i3q1gKYDttwsCTkg@mail.gmail.com> <20171107162914.G9710@sola.nimnet.asn.au> <CAKV%2BxLDQQcG3bvo1b2nUAu7oOVhdNzDDrPWTVp2qOmkWVV89BQ@mail.gmail.com> <20171108012948.A9710@sola.nimnet.asn.au> <CAKV%2BxLCQ9NE6%2BEg6NvHZuEED8Cf6ZX74unvk9ajfLyG-yA2rXA@mail.gmail.com> <CAKV%2BxLAkfiQCLXfgZOtQGUXOW8gYN7sjOD5uWezv-N%2BTBjybMQ@mail.gmail.com> <20171111213759.I72828@sola.nimnet.asn.au> <CAKV%2BxLDicLze3Dvd2i7HGWJUxCdSLjvhuWWZUJ65pMi%2Bx483=A@mail.gmail.com> <CAKV%2BxLAt4Ciqmg2w1iJK42jq6f%2BnumASKMQ=UL6dT%2BCdGYujVQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Nov 14, 2017 at 9:31 AM, Cos Chan <rosettas@gmail.com> wrote:
>
>
> On Mon, Nov 13, 2017 at 3:17 PM, Cos Chan <rosettas@gmail.com> wrote:
>
>>
>>
>> On Sat, Nov 11, 2017 at 1:42 PM, Ian Smith <smithi@nimnet.asn.au> wrote:
>>
>>> On Thu, 9 Nov 2017 14:25:52 +0100, Cos Chan wrote:
>>>
>>> > Dear All
>>> >
>>> > Thanks Ian's great help, I have solved problem to post banned entries
>>> from
>>> > blacklistd to ipfw.
>>>
>>> Well, we're some of the way there :) We really need Kurt Lidl's eyes on
>>> this to make real progress, and indications are that my and your emails
>>> cc'ing him were still being deferred for some reason - maybe he's away?
>>>
>>> > The original message was received at Tue, 7 Nov 2017 10:12:05 -0500
>>> (EST)
>>> > from mx2.freebsd.org [8.8.178.116]
>>> >
>>> > ----- Transcript of session follows -----
>>> > <lidl@pix.net>... Deferred: Operation timed out with hydra.pix.net.
>>> > Warning: message still undelivered after 4 hours
>>> > Will keep trying until message is 1 week, 3 days old
>>>
>>>
>>> > To my knowledge the problem is:
>>> >
>>> > I setup sshd+blacklistd without ipfw at first. Then I got problem the
>>> entry
>>> > was never reached nfail number (is it a bug?).
>>>
>>> The first issue was because of a severe deficiency in blacklistd-helper,
>>> in that it doesn't actually check that the chosen firewall is running,
>>> and it then fails to detect commands for that firewall that do not (can
>>> not) succeed as any sort of error! More about that below.
>>>
>>> The second, however, was mainly that you missed that nfail set to '*'
>>> means that the host is NOT to be blocked, no matter how many auth or
>>> other failures that (in this case) sshd reports.
>>>
>>> That also answers another question you had .. "nnn/-1" indicates that
>>> nfail=* ie never to be blocked. These still get accumulated in the
>>> database, but are not applied as ipfw block rule table entries.
>>>
>>>
>>> > so I have to change the nfail to * to get the entry into banned list.
>>>
>>> In combination with other factors - like whether ipfw was running at the
>>> time - that got blacklistd to record reported failures to its database,
>>> but not to execute the 'add' commands to blacklistd-helper, so that
>>> address was not in fact blocked, and subsequent attempts kept trying.
>>>
>>> > But while I setup ipfw, the nfail=* would not activate
>>> blacklistd-helper so
>>> > no entry in blacklist banned list were added to ipfw.
>>>
>>> Yes, nfail=* means NEVER block these failed addreses. blacklistd.conf(5)
>>>
>>> > I have modify the blacklistd nfail to 2, sshd MaxAuthTries to 3. The
>>> > blacklist entries working fine.
>>>
>>> With ipfw running, yes :) But it should have failed - noisily - sooner.
>>>
>>> When ipfw is running, issuing this will show you the addresses blocked:
>>>
>>> # ipfw table port22 list
>>>
>>
>> until now it seems working on list updating. but I am not sure if it is
>> really working fine.
>>
>> here is one strange record:
>>
>> $ sudo blacklistctl dump -b | grep 1662
>> 193.201.224.218/32:22 OK 1662/1 2017/11/13 00:31:04
>>
>> This IP was blocked in ipfw from last week. while I checked it last week
>> Friday it was 800+/1 in blacklist and until today it become 1662.
>>
>> To my knowledge the ipfw should block the connection, the times of banned
>> IP should be not increased?
>>
>> I could see more entries with more than 3/1, for example:
>>
>> 89.160.221.132/32:22 OK 18/1 2017/11/13 00:01:21
>> 60.125.42.119/32:22 OK 3/1 2017/11/12 16:13:53
>> 166.62.35.180/32:22 OK 3/1 2017/11/10 06:36:25
>> 202.162.221.51/32:22 OK 6/1 2017/11/10 00:42:14
>> 168.0.114.130/32:22 OK 3/1 2017/11/10 23:40:30
>> 95.145.71.165/32:22 OK 3/1 2017/11/11 07:07:07
>> 123.161.206.210/32:22 OK 3/1 2017/11/12 18:14:00
>> 203.146.208.208/32:22 OK 6/1 2017/11/10 10:16:21
>> 149.56.223.241/32:22 OK 1/1 2017/11/12 06:09:16
>> 121.169.217.98/32:22 OK 9/1 2017/11/12 21:59:57
>> 211.251.237.162/32:22 OK 2/1 2017/11/13 12:08:07
>> 103.99.0.116/32:22 OK 30/1 2017/11/10 14:56:07
>>
>> These records I am not sure if they were not increased after added to
>> ipfw list. but the 1662 times one, I am sure it was increased after ipfw
>> had the ip in list.
>>
>
> add the ipfw rules:
>
> $ sudo ipfw list
> 00100 allow ip from any to any via lo0
> 00200 deny ip from any to 127.0.0.0/8
> 00300 deny ip from 127.0.0.0/8 to any
> 00400 deny ip from any to ::1
> 00500 deny ip from ::1 to any
> 00600 allow ipv6-icmp from :: to ff02::/16
> 00700 allow ipv6-icmp from fe80::/10 to fe80::/10
> 00800 allow ipv6-icmp from fe80::/10 to ff02::/16
> 00900 allow ipv6-icmp from any to any ip6 icmp6types 1
> 01000 allow ipv6-icmp from any to any ip6 icmp6types 2,135,136
> 02022 deny tcp from table(port22) to any dst-port 22
> 65000 allow ip from any to any
> 65535 deny ip from any to any
>
the more logs might be useful:
$ sudo tail security
Nov 14 15:09:07 res kernel: ipfw: 2022 Deny TCP 182.93.152.171:6920
192.168.11.15:22 in via em0
Nov 14 15:09:21 res kernel: ipfw: 2022 Deny TCP 123.125.203.196:6920
192.168.11.15:22 in via em0
Nov 14 15:10:11 res kernel: ipfw: 2022 Deny TCP 182.93.152.171:6920
192.168.11.15:22 in via em0
Nov 14 15:10:33 res kernel: ipfw: 2022 Deny TCP 83.12.107.106:6920
192.168.11.15:22 in via em0
Nov 14 15:11:08 res last message repeated 15 times
Nov 14 15:12:32 res last message repeated 4 times
Nov 14 15:21:10 res kernel: ipfw: 2022 Deny TCP 201.147.183.55:60299
192.168.11.15:22 in via em0
Nov 14 15:21:17 res last message repeated 3 times
Nov 14 15:25:38 res kernel: ipfw: 2022 Deny TCP 105.226.55.239:48315
192.168.11.15:22 in via em0
Nov 14 15:26:18 res last message repeated 12 times
$ sudo tail auth.log
Nov 14 15:07:24 res sshd[9029]: input_userauth_request: invalid user admin
[preauth]
Nov 14 15:10:33 res sshd[9052]: Invalid user omni from 83.12.107.106
Nov 14 15:10:33 res sshd[9052]: input_userauth_request: invalid user omni
[preauth]
Nov 14 15:25:37 res sshd[9144]: reverse mapping checking getaddrinfo for
105-226-55-239.south.dsl.telkomsa.net [105.226.55.239] failed - POSSIBLE
BREAK-IN ATTEMPT!
Nov 14 15:25:37 res sshd[9144]: Invalid user admin from 105.226.55.239
Nov 14 15:25:37 res sshd[9144]: input_userauth_request: invalid user admin
[preauth]
Nov 14 15:26:08 res sshd[9152]: Received disconnect from 121.18.238.123
port 42391:11: [preauth]
Nov 14 15:26:08 res sshd[9152]: Disconnected from 121.18.238.123 port 42391
[preauth]
The IP 105.226.55.239 looks like banned by IPFW, but still connected to
sshd?
>
>
>>
>>
>>> > BUT I found another problem.
>>> >
>>> > The output of blacklist dump is strange:
>>> >
>>> > $ sudo blacklistctl dump
>>> > address/ma:port id nfail last access
>>> > 96.227.104.132/32:22 0/2 1970/01/01 01:00:00
>>> > 89.245.78.187/32:22 0/2 1970/01/01 01:00:00
>>> > 116.193.162.203/32:22 1/2 2017/11/09 11:48:05
>>> >
>>> > Since the blacklistd accepts instruction from sshd. how could be 0/2
>>> > entries presented there? I am sure my successful logins were not
>>> added to
>>> > blacklistd.
>>>
>>> 1970/01/01 01:00:00 is just the UNIX '0' timestamp, in this case plus
>>> one hour (your TZ offset). It here means 'no previous entry'. Not sure
>>> about that 0/2, but there are several different codes returned by sshd
>>> including success, failed auth and 'abusive behaviour' .. I'm not sure
>>> which ones your reports (including in off-list mail) indicate.
>>>
>>> As for the mysterious 'n-1' behaviour you mentioned offlist for nfail,
>>> in /usr/src/contrib/blacklist/bin/blacklistd.c there's this:
>>>
>>> switch (bi->bi_type) {
>>> case BL_ABUSE:
>>> /*
>>> * If the application has signaled abusive behavior,
>>> * set the number of fails to be one less than the
>>> * configured limit. Fallthrough to the normal BL_ADD
>>> * processing, which will increment the failure count
>>> * to the threshhold, and block the abusive address.
>>> */
>>> if (c.c_nfail != -1)
>>> dbi.count = c.c_nfail - 1;
>>> /*FALLTHROUGH*/
>>> case BL_ADD:
>>> dbi.count++;
>>> dbi.last = ts.tv_sec;
>>> if (dbi.id[0]) {
>>> /*
>>> * We should not be getting this since the rule
>>> * should have blocked the address. A possible
>>> * explanation is that someone removed that rule,
>>> * and another would be that we got another
>>> attempt
>>> * before we added the rule. In anycase, we
>>> remove
>>> * and re-add the rule because we don't want to
>>> add
>>> * it twice, because then we'd lose track of it.
>>> */
>>> (*lfun)(LOG_DEBUG, "rule exists %s", dbi.id);
>>> (void)run_change("rem", &c, dbi.id, 0);
>>> dbi.id[0] = '\0';
>>> }
>>> if (c.c_nfail != -1 && dbi.count >= c.c_nfail) {
>>> int res = run_change("add", &c, dbi.id, sizeof(
>>> dbi.id));
>>> if (res == -1)
>>> goto out;
>>> sockaddr_snprintf(rbuf, sizeof(rbuf), "%a",
>>> (void *)&rss);
>>> (*lfun)(LOG_INFO,
>>> "blocked %s/%d:%d for %d seconds",
>>> rbuf, c.c_lmask, c.c_port, c.c_duration);
>>>
>>> }
>>> break;
>>>
>>> But if the 'add' command via blacklistd-helper fails, it will never add
>>> the 1 .. I'm not certain about this, but it could explain what you see,
>>> although I can't discern whether sshd is reporting BL_ADD or BL_ABUSE.
>>>
>>> You might instead try MaxAuthTries 4 .. sshd_config(5) says:
>>>
>>> MaxAuthTries
>>> Specifies the maximum number of authentication attempts
>>> permitted
>>> per connection. Once the number of failures reaches half
>>> this
>>> value, additional failures are logged. The default is 6.
>>>
>>> Half of 3 as an integer is only 1, but half of 4 is 2. See if it helps?
>>>
>>
>> I didnt change the MaxAuthTries, since I found something interesting from
>> the different logs concerning that issue:
>>
>> From blacklistctl dump:
>>
>> $ sudo blacklistctl dump
>> address/ma:port id nfail last access
>> 78.203.146.34/32:22 0/1 1970/01/01 01:00:00
>> 195.225.116.21/32:22 0/1 1970/01/01 01:00:00
>> 123.31.26.123/32:22 0/1 1970/01/01 01:00:00
>> 112.148.101.13/32:22 0/1 1970/01/01 01:00:00
>> 93.23.6.18/32:22 0/1 1970/01/01 01:00:00
>> 5.102.197.124/32:22 0/1 1970/01/01 01:00:00
>> 193.154.127.32/32:22 0/1 1970/01/01 01:00:00
>> 113.232.216.41/32:22 0/1 1970/01/01 01:00:00
>>
>> From sshd log:
>>
>> Nov 10 17:57:41 res sshd[49839]: Invalid user pi from 193.154.127.32
>> Nov 10 17:57:41 res sshd[49840]: Invalid user pi from 193.154.127.32
>> Nov 10 17:57:41 res sshd[49840]: input_userauth_request: invalid user pi
>> [preauth]
>> Nov 10 17:57:41 res sshd[49839]: input_userauth_request: invalid user pi
>> [preauth]
>> ...
>> Nov 11 03:50:47 res sshd[57896]: Invalid user support from 123.31.26.123
>> Nov 11 03:50:47 res sshd[57896]: input_userauth_request: invalid user
>> support [preauth]
>> Nov 11 03:50:47 res sshd[57896]: error: Received disconnect from
>> 123.31.26.123 port 55811:3: com.jcraft.jsch.JSchException: Auth fail
>> [preauth]
>> Nov 11 03:50:49 res sshd[57898]: Invalid user admin from 123.31.26.123
>> Nov 11 03:50:49 res sshd[57898]: input_userauth_request: invalid user
>> admin [preauth]
>> Nov 11 03:50:49 res sshd[57898]: error: Received disconnect from
>> 123.31.26.123 port 57823:3: com.jcraft.jsch.JSchException: Auth fail
>> [preauth]
>> Nov 11 03:50:51 res sshd[57900]: Invalid user admin from 123.31.26.123
>> Nov 11 03:50:51 res sshd[57900]: input_userauth_request: invalid user
>> admin [preauth]
>> Nov 11 03:50:51 res sshd[57900]: error: Received disconnect from
>> 123.31.26.123 port 59819:3: com.jcraft.jsch.JSchException: Auth fail
>> [preauth]
>> Nov 11 03:50:53 res sshd[57902]: Invalid user ubnt from 123.31.26.123
>> Nov 11 03:50:53 res sshd[57902]: input_userauth_request: invalid user
>> ubnt [preauth]
>> Nov 11 03:50:53 res sshd[57902]: error: Received disconnect from
>> 123.31.26.123 port 61795:3: com.jcraft.jsch.JSchException: Auth fail
>> [preauth]
>> Nov 11 03:50:55 res sshd[57904]: Invalid user PlcmSpIp from 123.31.26.123
>> Nov 11 03:50:55 res sshd[57904]: input_userauth_request: invalid user
>> PlcmSpIp [preauth]
>> Nov 11 03:50:55 res sshd[57904]: error: Received disconnect from
>> 123.31.26.123 port 61920:3: com.jcraft.jsch.JSchException: Auth fail
>> [preauth]
>> Nov 11 03:50:57 res sshd[57906]: Invalid user admin from 123.31.26.123
>> Nov 11 03:50:57 res sshd[57906]: input_userauth_request: invalid user
>> admin [preauth]
>> Nov 11 03:50:57 res sshd[57906]: error: Received disconnect from
>> 123.31.26.123 port 61949:3: com.jcraft.jsch.JSchException: Auth fail
>> [preauth]
>>
>> I see 2 problems:
>>
>> Problem 1:
>> The IP 193.154.127.32 didn't reach sshd maximum authentication (=3), it
>> tried only 2 times.
>> But in my opinion it should be recorded to blacklistd as 2/1 instead of
>> 0/1.
>>
>> Problem 2:
>> The IP 123.31.26.123 was trying to use different user name to login more
>> than 3 times. it was also recorded in blacklistd as 0/1.
>>
>> In my opinion the above 2 all should be banned by blacklistd.
>>
>>
>>>
>>> > I am trying to find out the reason from log but I dont know how to see
>>> > blacklistd log. man page said that is to syslogd but what the
>>> facility it
>>> > is? or some other ways to get out log?
>>>
>>> Not sure of the facility but when using the -v switch, as you have been,
>>> logging goes to stderr instead of syslog. Without -v you should see it
>>> logging to /var/log/messages. If not, try adding to /etc/syslog.conf:
>>>
>>> !blacklistd
>>> *.* /var/log/myblacklistd.log
>>>
>>> then '# touch /var/log/myblacklistd.log && service syslogd restart'
>>>
>>
>> Unfortunately I started the logging later than Nov 11 03:50:57, so I
>> didnt get the log of "0/1" records yet.
>>
>
>
> got the log for one new "0/1" entry:
>
> $ sudo blacklistctl dump
> address/ma:port id nfail last access
> 24.7.90.146/32:22 0/1 1970/01/01 01:00:00
> ...
>
> $ sudo cat auth.log | grep 24.7.90.146
> Nov 14 02:13:58 res sshd[6212]: Invalid user pi from 24.7.90.146
> Nov 14 02:13:58 res sshd[6215]: Invalid user pi from 24.7.90.146
> Nov 14 02:13:59 res sshd[6215]: Connection closed by 24.7.90.146 port
> 34746 [preauth]
> Nov 14 02:13:59 res sshd[6212]: Connection closed by 24.7.90.146 port
> 34742 [preauth]
>
> $ cat myblacklistd.log | grep 'Nov 14'
> ...
> Nov 14 02:09:11 res blacklistd[5590]: blocked 202.51.74.55/32:22 for -1
> seconds
> Nov 14 02:11:06 res blacklistd[5590]: rule exists OK
> Nov 14 02:11:06 res blacklistd[5590]: blocked 202.51.74.55/32:22 for -1
> seconds
> Nov 14 02:14:43 res blacklistd[5590]: blocked 66.232.147.46/32:22 for -1
> seconds
> Nov 14 02:16:40 res blacklistd[5590]: rule exists OK
>
> could not see operation against that IP from blacklistd.log
>
>
>>
>>
>>>
>>> Ok, problems with blacklistd-helper; the first bit verbatim, tabs lost:
>>>
>>> #!/bin/sh
>>> #echo "run $@" 1>&2
>>> #set -x
>>> # $1 command
>>> # $2 rulename
>>> # $3 protocol
>>> # $4 address
>>> # $5 mask
>>> # $6 port
>>> # $7 id
>>>
>>> pf=
>>> if [ -f "/etc/ipfw-blacklist.rc" ]; then
>>> pf="ipfw"
>>> . /etc/ipfw-blacklist.rc
>>> ipfw_offset=${ipfw_offset:-2000}
>>> fi
>>>
>>> if [ -z "$pf" ]; then
>>> for f in npf pf ipf; do
>>> if [ -f "/etc/$f.conf" ]; then
>>> pf="$f"
>>> break
>>> fi
>>> done
>>> fi
>>>
>>> if [ -z "$pf" ]; then
>>> echo "$0: Unsupported packet filter" 1>&2
>>> exit 1
>>> fi
>>>
>>> Earlier you said you'd run it without /etc/ipfw-blacklist.rc existing.
>>> In that case - UNLESS you had either /etc/pf.conf or /etc/ipf.conf lying
>>> around from before? it should have failed with 'exit 1' .. though it's
>>> not clear from browsing the code that even that would cause it to quit.
>>>
>>
>> No, there are not /etc/pf.conf and /etc/ipf.conf.
>>
>>
>>>
>>> So once /etc/ipfw-blacklist.rc exists, that's a flag indicating you
>>> intend using ipfw, however there's NO check that ipfw is running ..
>>>
>>> Then - ignoring the pf) and ipf) sections - though I suspect they'd have
>>> the same issue unless really running - here's the ipfw add bit, no tabs:
>>>
>>> add)
>>> case "$pf" in
>>> [..]
>>> ipfw)
>>> # use $ipfw_offset+$port for rule number
>>> rule=$(($ipfw_offset + $6))
>>> tname="port$6"
>>> /sbin/ipfw table $tname create type addr 2>/dev/null
>>>
>>> Unless ipfw is running, enabled, that will fail - silently.
>>>
>>> /sbin/ipfw -q table $tname add "$addr/$mask"
>>>
>>> Ditto, perhaps with a message to stderr - that's simply ignored.
>>>
>>> # if rule number $rule does not already exist, create it
>>> /sbin/ipfw show $rule >/dev/null 2>&1 || \
>>> /sbin/ipfw add $rule drop $3 from \
>>> table"("$tname")" to any dst-port $6 >/dev/null
>>> && \
>>> echo OK
>>> ;;
>>>
>>> When both of these ipfw commands also fail, it'll only fail to echo OK.
>>>
>>> Not that failing to echo OK seems to matter to the calling code, but
>>> the OK is kept as 'id' which is passed to the rem)ove code, but is
>>> unused except by the npf firewall .. 'netbsd packet filter' I guess.
>>>
>>> I can certainly suggest patches for at least the ipfw sections - and
>>> really, if the introductory code checks ipfw is working that should be
>>> enough - but I'm unsure whether 'exit 1' after an error message is all
>>> that's needed to get blacklistd to whinge loudly and refuse to continue?
>>>
>>> This should be turned into a PR via bugzilla, but since I'm not running
>>> 11.x here, I can only really contribute if you do so and add me as a cc.
>>>
>>
>> Sorry I dont know how to describe the problem in bugzilla since I dont
>> really understand what you said.
>> I have to learn more about the script :)
>>
>>
>>>
>>> Please try to avoid top-posting on replies, thanks.
>>
>>
>> Sure, I will.
>>
>>
>>>
>>> cheers, Ian
>>>
>>
>>
>>
>> --
>> with kind regards
>>
>
>
>
> --
> with kind regards
>
--
with kind regards
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKV%2BxLD_KE938JnmjDE=CmfZ7bOJ1CaqvWuQ%2B0jDzQNWM%2B6yLg>