From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 16 19:11:19 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45F4D10656D3 for ; Mon, 16 Mar 2009 19:11:19 +0000 (UTC) (envelope-from chris@smartt.com) Received: from barium.smartt.com (barium.smartt.com [69.67.187.30]) by mx1.freebsd.org (Postfix) with ESMTP id 2A0C28FC16 for ; Mon, 16 Mar 2009 19:11:18 +0000 (UTC) (envelope-from chris@smartt.com) Received: from [69.31.174.220] (unknown [69.31.174.220]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by barium.smartt.com (Postfix) with ESMTPSA id 974F310E516; Mon, 16 Mar 2009 12:11:21 -0700 (PDT) Message-ID: <49BEA45A.5060603@smartt.com> Date: Mon, 16 Mar 2009 12:11:22 -0700 From: Chris St Denis User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) MIME-Version: 1.0 To: =?ISO-8859-1?Q?Mikko_Ty=F6l=E4j=E4rvi?= References: <49BA9E63.3040000@smartt.com> <20090315144440.N24160@antec.home> In-Reply-To: <20090315144440.N24160@antec.home> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-hackers@freebsd.org Subject: Re: Bug in tcp wrappers? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Mar 2009 19:11:20 -0000 Mikko Työläjärvi wrote: > > Hi Chris, > > On Fri, 13 Mar 2009, Chris St Denis wrote: > >> I think I've found a bug in libwrap/tcpwrappers. > > I think so too :) See below. > >> Before filing an actual bug report I want to get some feedback here >> first. >> >> A hosts.allow file with ~1000 ips on a single line (Haven't >> experimented with >> other quantities yet), causes network daemons that use libwrap stop >> accepting >> incoming network connections and use 100% cpu on an incoming connection. >> This problem appeared because sshguard placed a large number of IPs >> in my >> hosts.allow file triggering this bug. >> >> I've left the affected daemons for a long period of time (once about >> 8 hours) >> and they don't seem to come back, so I think this is more than just >> it taking >> a while to loop through a 1000 item array of IPs >> >> >> The production system that was affected is FreeBSD 7.0-32bit >> Test system is FreeBSD 7.1-32bit >> >> Example hosts.allow file (IPs are randomly generated for purposes of >> example) >> >> sshd : 112.110.123.63 113.11.2.126 113.11.8.6 113.19.19.22 >> 113.197.48.68 116.48.108.244 116.48.11.19 : deny >> ALL : ALL : allow >> >> top output of affected system. sshd wcpu slowly crawls up to 100% >> over about >> 30 seconds or so. >> >> crash# top >> last pid: 692; load averages: 0.08, 0.04, 0.04 >> up >> 0+00:12:13 15:42:30 >> 24 processes: 2 running, 22 sleeping >> CPU: 49.7% user, 0.0% nice, 0.2% system, 0.2% interrupt, 49.9% idle >> Mem: 9304K Active, 6004K Inact, 21M Wired, 32K Cache, 10M Buf, 947M >> Free >> Swap: 1995M Total, 1995M Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 691 root 1 103 0 5760K 3660K CPU1 1 0:04 33.98% sshd >> 672 root 1 4 0 8436K 3888K sbwait 1 0:00 0.00% sshd >> 677 cstdenis 1 20 0 4460K 2288K pause 0 0:00 0.00% csh >> 682 root 1 20 0 5484K 2632K pause 0 0:00 0.00% csh >> 675 cstdenis 1 44 0 8436K 3896K select 0 0:00 0.00% sshd >> >> >> A backtrace shows >> >> crash# gdb /usr/sbin/sshd 691 >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and >> you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "i386-marcel-freebsd"... >> Attaching to program: /usr/sbin/sshd, process 691 >> Reading symbols from /usr/lib/libssh.so.4...done. >> Loaded symbols for /usr/lib/libssh.so.4 >> Reading symbols from /lib/libutil.so.7...done. >> Loaded symbols for /lib/libutil.so.7 >> Reading symbols from /lib/libz.so.4...done. >> Loaded symbols for /lib/libz.so.4 >> Reading symbols from /usr/lib/libwrap.so.5...done. >> Loaded symbols for /usr/lib/libwrap.so.5 >> >> Reading symbols from /libexec/ld-elf.so.1...done. >> Loaded symbols for /libexec/ld-elf.so.1 >> 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at >> /usr/src/lib/libc/stdio/fgets.c:56 >> 56 { >> (gdb) bt >> #0 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at >> /usr/src/lib/libc/stdio/fgets.c:56 >> #1 0x281124ee in xgets (ptr=0xbfbfe67b "", len=1, fp=0x283b8040) at >> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/misc.c:38 >> #2 0x28111410 in table_match (table=0x28112c5c "/etc/hosts.allow", >> request=0xbfbfeb14) >> at >> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:162 >> #3 0x28111540 in hosts_access (request=0xbfbfeb14) at >> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:132 >> #4 0x08052b39 in main (ac=2, av=0xbfbfeecc) at >> /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:1843 >> (gdb) bt >> #0 0x28373225 in fgets (buf=0xbfbfe67b "", n=1, fp=0x283b8040) at >> /usr/src/lib/libc/stdio/fgets.c:56 >> #1 0x281124ee in xgets (ptr=0xbfbfe67b "", len=1, fp=0x283b8040) at >> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/misc.c:38 >> #2 0x28111410 in table_match (table=0x28112c5c "/etc/hosts.allow", >> request=0xbfbfeb14) >> at >> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:162 >> #3 0x28111540 in hosts_access (request=0xbfbfeb14) at >> /usr/src/lib/libwrap/../../contrib/tcp_wrappers/hosts_access.c:132 >> #4 0x08052b39 in main (ac=2, av=0xbfbfeecc) at >> /usr/src/secure/usr.sbin/sshd/../../../crypto/openssh/sshd.c:1843 >> (gdb) q >> The program is running. Quit anyway (and detach it)? (y or n) y >> Detaching from program: /usr/sbin/sshd, process 691 >> >> >> A few questions >> 1. Is this a known issue of any sort? I've done some searching on it, >> but >> haven't found anything of interest. >> 2. Should this be reported to FreeBSD bug tracker, or to libwrap (or >> both)? >> Basically, is FreeBSD's libwrap (more or less) in sync with the main >> one, or >> is it completely separate? > > When given an input line of more than 2k bytes, libwrap ends up in an > infinite loop in xgets(), calling fgets() with a read length of one. > As fgets() reads the length minus one characters, it will keep > "reading" and returning zero length strings. > > Thus your server processes will remain stuck until aborted. > > This Q&D patch makes libwrap behave as documented in hosts_access(5): > > --- misc.c.orig 2009-03-15 14:06:11.000000000 -0700 > +++ misc.c 2009-03-15 14:06:49.000000000 -0700 > @@ -48,6 +48,8 @@ > ptr += got; > len -= got; > ptr[0] = 0; > + if (len <= 1) > + return start; > } > return (ptr > start ? start : 0); > } > > > The documented behavior is: > > "An error is reported when ... when the length of an access control > rule exceeds the capacity of an internal buffer; ..." > > This is only sligtly better, as the code will now try to parse the > remainder of the line as a rule, and either fail or, due to some > syntactic quirk, get a false match. From a security standpoint, both > are bad. > > I don't think you'll get a false "allow" match in your case, but unless > you have a default "deny" rule somewhere at the end, access may be > granted when it shouldn't. > > Please do file a FreeBSD bug. Is there even an upstream maintainer of > tcp wrappers? A quick search seems to indicate that it is more or > less abandoned, albeit adopted by several projects. > > The immediate workarounds I can think of for you are: > > - Somehow teach sshguard to write rules on multiple lines, each > shorter than 2k. Splitting lines using backslashes will not help, > as xgets() is concatenating continued lines into a single buffer > (the one that is too small) anyway. > > - Apply the patch above, change the definition of BUFLEN in tcpdchk.c > and hosts_access.c to a "sufficiently large" value and rebuild > libwrap. Of course, there is no "sufficiently large" value; with > the current libwrap code, you'll always run the risk of lines being > too long. > > The real fix involves rewriting chunks of the libwrap code, or finding > a version where someone has already done so. > > $.02, > /Mikko Thanks. I have created PR 132705. http://www.freebsd.org/cgi/query-pr.cgi?pr=132705 My immediate workaround was even simpler than that. I just turned off sshGuard. It's just there to provide an additional level of security which isn't really needed. I may put it back using one of the firewall modules instead of hosts.allow in the future. I guess it was just never designed for the kind of distributed brute force ssh and ftp attacks that have been occurring more in the last several months.