From owner-freebsd-hackers@FreeBSD.ORG Tue Jul 13 08:18:38 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78423106566C; Tue, 13 Jul 2010 08:18:38 +0000 (UTC) (envelope-from ladrielt@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id EA13D8FC21; Tue, 13 Jul 2010 08:18:37 +0000 (UTC) Received: by vws19 with SMTP id 19so84472vws.13 for ; Tue, 13 Jul 2010 01:18:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=+fuvzAqQ1STTTnTmCEPhhh0TC0WI9azF4VP6wKRECKQ=; b=aj+QW7Xz5NVFh05DtPHyiqh4iBQ5YTwQXxrJ5G6HSlhhiiegy6RDnyy+9Ku7c+2WLi z2YWaYVeuJBHDdBMbBkP6Y6WSAZ3I8LSCAZd39ufo7r9Fp2BQvGzYR0xysDPWRDS7jVw F704BqifeH75+eg/O3Zng8sUwVqVLzp+3FMwo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=Mu1/+Dn+VVrKyyuumTFx/ayIHLPMJ8vxRYW5ds9PxMHu5kH9mr7rNr+WRpMWaO0PQp ypUDfxNahhLHPFLPwUhluN9vHd4MmbrABBHXymCITNwqJoHItT0yYKVF9Mt7EonBq9Bg dJEPHDnYjw0SxujDM0oKD+087zamcV7h/yg1g= MIME-Version: 1.0 Received: by 10.229.233.137 with SMTP id jy9mr9061439qcb.5.1279007670534; Tue, 13 Jul 2010 00:54:30 -0700 (PDT) Received: by 10.229.81.74 with HTTP; Tue, 13 Jul 2010 00:54:30 -0700 (PDT) In-Reply-To: References: <4B79297D.9080403@FreeBSD.org> Date: Tue, 13 Jul 2010 13:54:30 +0600 Message-ID: From: l adrielt To: Ali Mashtizadeh Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Maxim Sobolev , Jack Vogel , FreeBSD Hackers , freebsd-net@freebsd.org Subject: Re: Sudden mbuf demand increase and shortage under the load X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jul 2010 08:18:38 -0000 Hi, I'm not sure but reading the advisory that just came out today sounds like it could have something to do with your mbuf issues. -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D FreeBSD-SA-10:07.mbuf Security Adviso= ry The FreeBSD Projec= t Topic: Lost mbuf flag resulting in data corruption Category: core Module: kern Announced: 2010-07-13 Credits: Ming Fu Affects: FreeBSD 7.x and later. Corrected: 2010-07-13 02:45:17 UTC (RELENG_8, 8.1-PRERELEASE) 2010-07-13 02:45:17 UTC (RELENG_8_1, 8.1-RELEASE) 2010-07-13 02:45:17 UTC (RELENG_8_0, 8.0-RELEASE-p4) 2010-07-13 02:45:17 UTC (RELENG_7, 7.3-STABLE) 2010-07-13 02:45:17 UTC (RELENG_7_3, 7.3-RELEASE-p2) 2010-07-13 02:45:17 UTC (RELENG_7_1, 7.1-RELEASE-p13) CVE Name: CVE-2010-2693 For general information regarding FreeBSD Security Advisories, including descriptions of the fields above, security branches, and the following sections, please visit . I. Background An mbuf is a basic unit of memory management in the FreeBSD kernel inter-process communication and networking subsystem. Network packets and socket buffers are dependent on mbufs for their storage. Data can be embedded directly in mbufs, or mbufs can instead reference external buffers. The sendfile(2) system call uses external mbuf storage to directly map the contents of a file into a chain of mbufs for transmission purposes. The mbuf object supports a read-only flag that must be honored to prevent modification or writes to buffer data in cases like these. II. Problem Description The read-only flag is not correctly copied when a mbuf buffer reference is duplicated. When the sendfile(2) system call is used to transmit data over the loopback interface, this can result in the backing pages for the transmitted file being modified, causing data corruption. III. Impact This data corruption can be exploited by an local attacker to escalate their privilege by carefully controlling the corruption of system files. It should be noted that the attacker can corrupt any file they have read access to. NOTE: While systems without untrusted local users are not affected by the security aspects of this issue, the potential for data corruption implies that this should still be treated as a critical erratum. IV. Workaround No workaround is available. V. Solution Perform one of the following: 1) Upgrade your vulnerable system to 7-STABLE or 8-STABLE, or to the RELENG_8_1, RELENG_8_0, RELENG_7_3, or RELENG_7_1 security branch dated after the correction date. 2) To update your vulnerable system via a source code patch: The following patches have been verified to apply to FreeBSD 7.1, 7.3, 8.0 and 8.1 systems. a) Download the relevant patch from the location below, and verify the detached PGP signature using your PGP utility. # fetch http://security.FreeBSD.org/patches/SA-10:07/mbuf.patch # fetch http://security.FreeBSD.org/patches/SA-10:07/mbuf.patch.asc b) Apply the patch. On 7/11/10, Ali Mashtizadeh wrote: > Hi Maxim, > > I experienced the same issue recently on 8-STABLE branch and it seems > it has been fixed since 8.1-RC2 and above. I couldn't track down the > root cause in the code nor could I find a commit that seems to be the > obvious fix. > > Thanks, > ~ Ali > > 2010/2/15 Maxim Sobolev : >> Hi, >> >> Our company have a FreeBSD based product that consists of the numerous >> interconnected processes and it does some high-PPS UDP processing (30-50= K >> PPS is not uncommon). We are seeing some strange periodic failures under >> the >> load in several such systems, which usually evidences itself in IPC (eve= n >> through unix domain sockets) suddenly either breaking down or pausing an= d >> restoring only some time later (like 5-10 minutes). The only sign of >> failure >> I managed to find was the increase of the "requests for mbufs denied" in >> the >> netstat -m and number of total mbuf clusters (nmbclusters) raising up to >> the >> limit. >> >> I have tried to raise some network-related limits (most notably maxusers >> and >> nmbclusters), but it has not helped with the issue - it's still happenin= g >> from time to time to us. Below you can find output from the netstat -m f= ew >> minutes right after that shortage period - you see that somehow the syst= em >> has allocated huge amount of memory for the network (700MB), with only >> tiny >> amount of that being actually in use. This is for the >> kern.ipc.nmbclusters: >> 302400. Eventually the system reclaims all that memory and goes back to >> its >> normal use of 30-70MB. >> >> This problem is killing us, so any suggestions are greatly appreciated. = My >> current hypothesis is that due to some issues either with the network >> driver >> or network subsystem itself, the system goes insane and "eats" up all >> mbufs >> up to nmbclusters limit. But since mbufs are shared between network and >> local IPC, IPC goes down as well. >> >> We observe this issue with systems using both em(4) driver and igb(4) >> driver. I believe both drivers share the same design, however I am not >> sure >> if this is some kind of design flaw in the driver or part of a larger >> problem with the network subsystem. >> >> This happens on amd64 7.2-RELEASE and 7.3-PRERELEASE alike, with 8GB of >> memory. I have not tried upgrading to 8.0, this is production system so >> upgrading will not be easy. =C2=A0I don't believe there are some differe= nces >> that >> let us hope that this problem will go away after upgrade, but I can try = it >> as the last resort. >> >> As I said, this is very critical issue, so I can provide any additional >> debug information upon request. We are ready to go as far as paying >> somebody >> reasonable amount of money for tracking down and resolving the issue. >> >> Regards, >> -- >> Maksym Sobolyev >> Sippy Software, Inc. >> Internet Telephony (VoIP) Experts >> T/F: +1-646-651-1110 >> Web: http://www.sippysoft.com >> MSN: sales@sippysoft.com >> Skype: SippySoft >> >> >> [ssp-root@ds-467 /usr/src]$ netstat -m >> 17061/417669/434730 mbufs in use (current/cache/total) >> 10420/291980/302400/302400 mbuf clusters in use (current/cache/total/max= ) >> 10420/0 mbuf+clusters out of packet secondary zone in use (current/cache= ) >> 19/1262/1281/51200 4k (page size) jumbo clusters in use >> (current/cache/total/max) >> 0/0/0/25600 9k jumbo clusters in use (current/cache/total/max) >> 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) >> 25181K/693425K/718606K bytes allocated to network (current/cache/total) >> 1246681/129567494/67681640 requests for mbufs denied >> (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >> 0/0/0 sfbufs in use (current/peak/max) >> 0 requests for sfbufs denied >> 0 requests for sfbufs delayed >> 0 requests for I/O initiated by sendfile >> 0 calls to protocol drain routines >> >> [FEW MINUTES LATER] >> >> [ssp-root@ds-467 /usr/src]$ netstat -m >> 10001/84574/94575 mbufs in use (current/cache/total) >> 6899/6931/13830/302400 mbuf clusters in use (current/cache/total/max) >> 6899/6267 mbuf+clusters out of packet secondary zone in use >> (current/cache) >> 2/1151/1153/51200 4k (page size) jumbo clusters in use >> (current/cache/total/max) >> 0/0/0/25600 9k jumbo clusters in use (current/cache/total/max) >> 0/0/0/12800 16k jumbo clusters in use (current/cache/total/max) >> 16306K/39609K/55915K bytes allocated to network (current/cache/total) >> 1246681/129567494/67681640 requests for mbufs denied >> (mbufs/clusters/mbuf+clusters) >> 0/0/0 requests for jumbo clusters denied (4k/9k/16k) >> 0/0/0 sfbufs in use (current/peak/max) >> 0 requests for sfbufs denied >> 0 requests for sfbufs delayed >> 0 requests for I/O initiated by sendfile >> 0 calls to protocol drain routines >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > > > -- > Ali Mashtizadeh > =D8=B9=D9=84=DB=8C =D9=85=D8=B4=D8=AA=DB=8C =D8=B2=D8=A7=D8=AF=D9=87 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " >