Date: Sun, 15 Mar 2009 14:44:49 -0700 (PDT) From: =?ISO-8859-1?Q?Mikko_Ty=F6l=E4j=E4rvi?= <mbsd@pacbell.net> To: Chris St Denis <chris@smartt.com> Cc: freebsd-hackers@freebsd.org Subject: Re: Bug in tcp wrappers? Message-ID: <20090315144440.N24160@antec.home> In-Reply-To: <49BA9E63.3040000@smartt.com> References: <49BA9E63.3040000@smartt.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Chris,
On Fri, 13 Mar 2009, Chris St Denis wrote:
> I think I've found a bug in libwrap/tcpwrappers.
I think so too :) See below.
> Before filing an actual bug report I want to get some feedback here
> first.
>
> A hosts.allow file with ~1000 ips on a single line (Haven't experimented with
> other quantities yet), causes network daemons that use libwrap stop accepting
> incoming network connections and use 100% cpu on an incoming connection.
> This problem appeared because sshguard placed a large number of IPs in my
> hosts.allow file triggering this bug.
>
> I've left the affected daemons for a long period of time (once about 8 hours)
> and they don't seem to come back, so I think this is more than just it taking
> a while to loop through a 1000 item array of IPs
>
>
> The production system that was affected is FreeBSD 7.0-32bit
> Test system is FreeBSD 7.1-32bit
>
> Example hosts.allow file (IPs are randomly generated for purposes of example)
>
> sshd : 112.110.123.63 113.11.2.126 113.11.8.6 113.19.19.22
> 113.197.48.68 <snipped 990+ IPs> 116.48.108.244 116.48.11.19 : deny
> ALL : ALL : allow
>
> top output of affected system. sshd wcpu slowly crawls up to 100% over about
> 30 seconds or so.
>
> crash# top
> last pid: 692; load averages: 0.08, 0.04, 0.04
> up
> 0+00:12:13 15:42:30
> 24 processes: 2 running, 22 sleeping
> CPU: 49.7% user, 0.0% nice, 0.2% system, 0.2% interrupt, 49.9% idle
> Mem: 9304K Active, 6004K Inact, 21M Wired, 32K Cache, 10M Buf, 947M Free
> Swap: 1995M Total, 1995M Free
>
> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
> COMMAND
> 691 root 1 103 0 5760K 3660K CPU1 1 0:04 33.98% sshd
> 672 root 1 4 0 8436K 3888K sbwait 1 0:00 0.00% sshd
> 677 cstdenis 1 20 0 4460K 2288K pause 0 0:00 0.00% csh
> 682 root 1 20 0 5484K 2632K pause 0 0:00 0.00% csh
> 675 cstdenis 1 44 0 8436K 3896K select 0 0:00 0.00% sshd
> <snip>
>
> A backtrace shows
>
> crash# gdb /usr/sbin/sshd 691
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "i386-marcel-freebsd"...
> Attaching to program: /usr/sbin/sshd, process 691
> Reading symbols from /usr/lib/libssh.so.4...done.
> Loaded symbols for /usr/lib/libssh.so.4
> Reading symbols from /lib/libutil.so.7...done.
> Loaded symbols for /lib/libutil.so.7
> Reading symbols from /lib/libz.so.4...done.
> Loaded symbols for /lib/libz.so.4
> Reading symbols from /usr/lib/libwrap.so.5...done.
> Loaded symbols for /usr/lib/libwrap.so.5
> <snip other symbols for breviry>
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
> /usr/src/lib/libc/stdio/fgets.c:56
> 56 {
> (gdb) bt
> #0 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
> /usr/src/lib/libc/stdio/fgets.c:56
> #1 0x281124ee in xgets (ptr=0xbfbfe67b "", len=1, fp=0x283b8040) at
> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/misc.c:38
> #2 0x28111410 in table_match (table=0x28112c5c "/etc/hosts.allow",
> request=0xbfbfeb14)
> at
> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:162
> #3 0x28111540 in hosts_access (request=0xbfbfeb14) at
> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:132
> #4 0x08052b39 in main (ac=2, av=0xbfbfeecc) at
> /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:1843
> (gdb) bt
> #0 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at
> /usr/src/lib/libc/stdio/fgets.c:56
> #1 0x281124ee in xgets (ptr=0xbfbfe67b "", len=1, fp=0x283b8040) at
> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/misc.c:38
> #2 0x28111410 in table_match (table=0x28112c5c "/etc/hosts.allow",
> request=0xbfbfeb14)
> at
> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:162
> #3 0x28111540 in hosts_access (request=0xbfbfeb14) at
> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:132
> #4 0x08052b39 in main (ac=2, av=0xbfbfeecc) at
> /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:1843
> (gdb) q
> The program is running. Quit anyway (and detach it)? (y or n) y
> Detaching from program: /usr/sbin/sshd, process 691
>
>
> A few questions
> 1. Is this a known issue of any sort? I've done some searching on it, but
> haven't found anything of interest.
> 2. Should this be reported to FreeBSD bug tracker, or to libwrap (or both)?
> Basically, is FreeBSD's libwrap (more or less) in sync with the main one, or
> is it completely separate?
When given an input line of more than 2k bytes, libwrap ends up in an
infinite loop in xgets(), calling fgets() with a read length of one.
As fgets() reads the length minus one characters, it will keep
"reading" and returning zero length strings.
Thus your server processes will remain stuck until aborted.
This Q&D patch makes libwrap behave as documented in hosts_access(5):
--- misc.c.orig 2009-03-15 14:06:11.000000000 -0700
+++ misc.c 2009-03-15 14:06:49.000000000 -0700
@@ -48,6 +48,8 @@
ptr += got;
len -= got;
ptr[0] = 0;
+ if (len <= 1)
+ return start;
}
return (ptr > start ? start : 0);
}
The documented behavior is:
"An error is reported when ... when the length of an access control
rule exceeds the capacity of an internal buffer; ..."
This is only sligtly better, as the code will now try to parse the
remainder of the line as a rule, and either fail or, due to some
syntactic quirk, get a false match. From a security standpoint, both
are bad.
I don't think you'll get a false "allow" match in your case, but unless
you have a default "deny" rule somewhere at the end, access may be
granted when it shouldn't.
Please do file a FreeBSD bug. Is there even an upstream maintainer of
tcp wrappers? A quick search seems to indicate that it is more or
less abandoned, albeit adopted by several projects.
The immediate workarounds I can think of for you are:
- Somehow teach sshguard to write rules on multiple lines, each
shorter than 2k. Splitting lines using backslashes will not help,
as xgets() is concatenating continued lines into a single buffer
(the one that is too small) anyway.
- Apply the patch above, change the definition of BUFLEN in tcpdchk.c
and hosts_access.c to a "sufficiently large" value and rebuild
libwrap. Of course, there is no "sufficiently large" value; with
the current libwrap code, you'll always run the risk of lines being
too long.
The real fix involves rewriting chunks of the libwrap code, or finding
a version where someone has already done so.
$.02,
/Mikko
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20090315144440.N24160>