Date: Mon, 16 Mar 2009 12:11:22 -0700 From: Chris St Denis <chris@smartt.com> To: =?ISO-8859-1?Q?Mikko_Ty=F6l=E4j=E4rvi?= <mbsd@pacbell.net> Cc: freebsd-hackers@freebsd.org Subject: Re: Bug in tcp wrappers? Message-ID: <49BEA45A.5060603@smartt.com> In-Reply-To: <20090315144440.N24160@antec.home> References: <49BA9E63.3040000@smartt.com> <20090315144440.N24160@antec.home>
next in thread | previous in thread | raw e-mail | index | archive | help
Mikko Työläjärvi wrote:
>
> Hi Chris,
>
> On Fri, 13 Mar 2009, Chris St Denis wrote:
>
>> I think I've found a bug in libwrap/tcpwrappers.
>
> I think so too :) See below.
>
>> Before filing an actual bug report I want to get some feedback here
>> first.
>>
>> A hosts.allow file with ~1000 ips on a single line (Haven't
>> experimented with
>> other quantities yet), causes network daemons that use libwrap stop
>> accepting
>> incoming network connections and use 100% cpu on an incoming connection.
>> This problem appeared because sshguard placed a large number of IPs
>> in my
>> hosts.allow file triggering this bug.
>>
>> I've left the affected daemons for a long period of time (once about
>> 8 hours)
>> and they don't seem to come back, so I think this is more than just
>> it taking
>> a while to loop through a 1000 item array of IPs
>>
>>
>> The production system that was affected is FreeBSD 7.0-32bit
>> Test system is FreeBSD 7.1-32bit
>>
>> Example hosts.allow file (IPs are randomly generated for purposes of
>> example)
>>
>> sshd : 112.110.123.63 113.11.2.126 113.11.8.6 113.19.19.22
>> 113.197.48.68 <snipped 990+ IPs> 116.48.108.244 116.48.11.19 : deny
>> ALL : ALL : allow
>>
>> top output of affected system. sshd wcpu slowly crawls up to 100%
>> over about
>> 30 seconds or so.
>>
>> crash# top
>> last pid: 692; load averages: 0.08, 0.04, 0.04
>> up
>> 0+00:12:13 15:42:30
>> 24 processes: 2 running, 22 sleeping
>> CPU: 49.7% user, 0.0% nice, 0.2% system, 0.2% interrupt, 49.9% idle
>> Mem: 9304K Active, 6004K Inact, 21M Wired, 32K Cache, 10M Buf, 947M
>> Free
>> Swap: 1995M Total, 1995M Free
>>
>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
>> COMMAND
>> 691 root 1 103 0 5760K 3660K CPU1 1 0:04 33.98% sshd
>> 672 root 1 4 0 8436K 3888K sbwait 1 0:00 0.00% sshd
>> 677 cstdenis 1 20 0 4460K 2288K pause 0 0:00 0.00% csh
>> 682 root 1 20 0 5484K 2632K pause 0 0:00 0.00% csh
>> 675 cstdenis 1 44 0 8436K 3896K select 0 0:00 0.00% sshd
>> <snip>
>>
>> A backtrace shows
>>
>> crash# gdb /usr/sbin/sshd 691
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and
>> you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB. Type "show warranty" for
>> details.
>> This GDB was configured as "i386-marcel-freebsd"...
>> Attaching to program: /usr/sbin/sshd, process 691
>> Reading symbols from /usr/lib/libssh.so.4...done.
>> Loaded symbols for /usr/lib/libssh.so.4
>> Reading symbols from /lib/libutil.so.7...done.
>> Loaded symbols for /lib/libutil.so.7
>> Reading symbols from /lib/libz.so.4...done.
>> Loaded symbols for /lib/libz.so.4
>> Reading symbols from /usr/lib/libwrap.so.5...done.
>> Loaded symbols for /usr/lib/libwrap.so.5
>> <snip other symbols for breviry>
>> Reading symbols from /libexec/ld-elf.so.1...done.
>> Loaded symbols for /libexec/ld-elf.so.1
>> 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
>> /usr/src/lib/libc/stdio/fgets.c:56
>> 56 {
>> (gdb) bt
>> #0 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
>> /usr/src/lib/libc/stdio/fgets.c:56
>> #1 0x281124ee in xgets (ptr=0xbfbfe67b "", len=1, fp=0x283b8040) at
>> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/misc.c:38
>> #2 0x28111410 in table_match (table=0x28112c5c "/etc/hosts.allow",
>> request=0xbfbfeb14)
>> at
>> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:162
>> #3 0x28111540 in hosts_access (request=0xbfbfeb14) at
>> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:132
>> #4 0x08052b39 in main (ac=2, av=0xbfbfeecc) at
>> /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:1843
>> (gdb) bt
>> #0 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
>> /usr/src/lib/libc/stdio/fgets.c:56
>> #1 0x281124ee in xgets (ptr=0xbfbfe67b "", len=1, fp=0x283b8040) at
>> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/misc.c:38
>> #2 0x28111410 in table_match (table=0x28112c5c "/etc/hosts.allow",
>> request=0xbfbfeb14)
>> at
>> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:162
>> #3 0x28111540 in hosts_access (request=0xbfbfeb14) at
>> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:132
>> #4 0x08052b39 in main (ac=2, av=0xbfbfeecc) at
>> /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:1843
>> (gdb) q
>> The program is running. Quit anyway (and detach it)? (y or n) y
>> Detaching from program: /usr/sbin/sshd, process 691
>>
>>
>> A few questions
>> 1. Is this a known issue of any sort? I've done some searching on it,
>> but
>> haven't found anything of interest.
>> 2. Should this be reported to FreeBSD bug tracker, or to libwrap (or
>> both)?
>> Basically, is FreeBSD's libwrap (more or less) in sync with the main
>> one, or
>> is it completely separate?
>
> When given an input line of more than 2k bytes, libwrap ends up in an
> infinite loop in xgets(), calling fgets() with a read length of one.
> As fgets() reads the length minus one characters, it will keep
> "reading" and returning zero length strings.
>
> Thus your server processes will remain stuck until aborted.
>
> This Q&D patch makes libwrap behave as documented in hosts_access(5):
>
> --- misc.c.orig 2009-03-15 14:06:11.000000000 -0700
> +++ misc.c 2009-03-15 14:06:49.000000000 -0700
> @@ -48,6 +48,8 @@
> ptr += got;
> len -= got;
> ptr[0] = 0;
> + if (len <= 1)
> + return start;
> }
> return (ptr > start ? start : 0);
> }
>
>
> The documented behavior is:
>
> "An error is reported when ... when the length of an access control
> rule exceeds the capacity of an internal buffer; ..."
>
> This is only sligtly better, as the code will now try to parse the
> remainder of the line as a rule, and either fail or, due to some
> syntactic quirk, get a false match. From a security standpoint, both
> are bad.
>
> I don't think you'll get a false "allow" match in your case, but unless
> you have a default "deny" rule somewhere at the end, access may be
> granted when it shouldn't.
>
> Please do file a FreeBSD bug. Is there even an upstream maintainer of
> tcp wrappers? A quick search seems to indicate that it is more or
> less abandoned, albeit adopted by several projects.
>
> The immediate workarounds I can think of for you are:
>
> - Somehow teach sshguard to write rules on multiple lines, each
> shorter than 2k. Splitting lines using backslashes will not help,
> as xgets() is concatenating continued lines into a single buffer
> (the one that is too small) anyway.
>
> - Apply the patch above, change the definition of BUFLEN in tcpdchk.c
> and hosts_access.c to a "sufficiently large" value and rebuild
> libwrap. Of course, there is no "sufficiently large" value; with
> the current libwrap code, you'll always run the risk of lines being
> too long.
>
> The real fix involves rewriting chunks of the libwrap code, or finding
> a version where someone has already done so.
>
> $.02,
> /Mikko
Thanks. I have created PR 132705.
http://www.freebsd.org/cgi/query-pr.cgi?pr=132705
My immediate workaround was even simpler than that. I just turned off
sshGuard. It's just there to provide an additional level of security
which isn't really needed. I may put it back using one of the firewall
modules instead of hosts.allow in the future. I guess it was just never
designed for the kind of distributed brute force ssh and ftp attacks
that have been occurring more in the last several months.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?49BEA45A.5060603>