From owner-freebsd-net@FreeBSD.ORG Wed Jan 16 14:19:09 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 695D8D26 for ; Wed, 16 Jan 2013 14:19:09 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm23-vm2.bullet.mail.ne1.yahoo.com (nm23-vm2.bullet.mail.ne1.yahoo.com [98.138.91.211]) by mx1.freebsd.org (Postfix) with ESMTP id 1FCE3B68 for ; Wed, 16 Jan 2013 14:19:08 +0000 (UTC) Received: from [98.138.90.52] by nm23.bullet.mail.ne1.yahoo.com with NNFMP; 16 Jan 2013 14:19:01 -0000 Received: from [98.138.89.251] by tm5.bullet.mail.ne1.yahoo.com with NNFMP; 16 Jan 2013 14:19:01 -0000 Received: from [127.0.0.1] by omp1043.mail.ne1.yahoo.com with NNFMP; 16 Jan 2013 14:19:01 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 892358.66938.bm@omp1043.mail.ne1.yahoo.com Received: (qmail 52964 invoked by uid 60001); 16 Jan 2013 14:19:01 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1358345941; bh=fcmZ0cIIWGc6QgARu78Ez+nA+rAY9U9CDOttPS/VASc=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=vClkeOqWu/2Wra6NDivKZyT3jcZkdnTdkrQQ6YoJDJi9Tetz+TwTkfuNnpsE/7L1W3laD//CzL3WH5F9+eMo1GwsUvTGUcMgRdOIqtAQvCLThymgb9p4F9+FeE9tpCrBuZpOrjw7ma11nkJDTjmzMZIaWYsXl3XayIE7rVdyx5A= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=ps2e0pRJuVl+LwyCT1fmqCTPKfRjE7zfQeuZ+zLLmYbN9USlbC5k+eITbcD+W9XFo5GoRdxfIKoDhl6b0nGPkLXlO9tCREf/aWIZTYR57iuDhNcM5YN2io7VvsimpgLjutBJBx57X79lceccVAo6gfyXXC9IQhlRfIFC0Td+myk=; X-YMail-OSG: 1c7djLIVM1nSv5H5XjR8YHrPr2kNnJpMkOB9SM4XOpfrhAd 4HaQE82x2.w.RIn5TMCMyIWdjrCBZJV2n8fuEL7_e_pcCKPub7yBVZT9Oys2 dKoqcKLYUqc4aFEKpHvxp3FfN7H0gKpXojIZtNZUPyZYhySa8re1CNR9Ibnu N4URWzQqEbLrLt.9LLisdvlaqbuePuKSS9XgpUneuuXy3FheMWQvpZN4_jTb yQh188MJMuS.hnpas.8RhIVrFYjQ7E9XDLSIsKW8DyewX1bjVUWnwpbev9UJ 1aqeDhnWMkU6EIR.s4fFqBipowyEaVLiIv8Pk2lfmvZXHDYFC0uB80MLbBdt tu_eNq8U77YTjTP34FUco3TvksmaxZSkkA8PwUyf.UKYqNeNEdH3kopxTqaQ K3YGdQdBf060giWzJTaRQocb58bdmOk5R0qEa_.rLxwmpBFQLqWgj.ddD6fI 2y9KemR_SWWYIxCJi9BlyePT5f_u05LfuanrszcO3bhDEGPafpIFeZO7QVku phpmZQx7kL3DekSihY7Q_dAF3ksWQ Received: from [174.48.128.27] by web121604.mail.ne1.yahoo.com via HTTP; Wed, 16 Jan 2013 06:19:01 PST X-Rocket-MIMEInfo: 001.001, CgotLS0gT24gVHVlLCAxLzE1LzEzLCBMdWlnaSBSaXp6byA8cml6em9AaWV0LnVuaXBpLml0PiB3cm90ZToKCj4gRnJvbTogTHVpZ2kgUml6em8gPHJpenpvQGlldC51bmlwaS5pdD4KPiBTdWJqZWN0OiB0d28gcHJvYmxlbXMgaW4gZGV2L2UxMDAwL2lmX2xlbS5jOjpsZW1faGFuZGxlX3J4dHgoKQo.IFRvOiBoZWFkQGZyZWVic2Qub3JnLCAiZnJlZWJzZC1uZXRAZnJlZWJzZC5vcmciIDxmcmVlYnNkLW5ldEBmcmVlYnNkLm9yZz4KPiBDYzogIkphY2sgVm9nZWwiIDxqZnZvZ2VsQGdtYWlsLmNvbT4KPiBEYXQBMAEBAQE- X-Mailer: YahooMailClassic/15.1.2 YahooMailWebService/0.8.130.494 Message-ID: <1358345941.38671.YahooMailClassic@web121604.mail.ne1.yahoo.com> Date: Wed, 16 Jan 2013 06:19:01 -0800 (PST) From: Barney Cordoba Subject: Re: two problems in dev/e1000/if_lem.c::lem_handle_rxtx() To: Luigi Rizzo In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2013 14:19:09 -0000 =0A=0A--- On Tue, 1/15/13, Luigi Rizzo wrote:=0A=0A> F= rom: Luigi Rizzo =0A> Subject: two problems in dev/e100= 0/if_lem.c::lem_handle_rxtx()=0A> To: head@freebsd.org, "freebsd-net@freebs= d.org" =0A> Cc: "Jack Vogel" = =0A> Date: Tuesday, January 15, 2013, 8:23 PM=0A> Hi,=0A> i found a couple = of problems in=0A> =A0 =A0 =A0 =A0=0A> dev/e1000/if_lem.c::lem_handle_rxtx(= ) ,=0A> (compare with dev/e1000/if_em.c::em_handle_que() for better=0A> und= erstanding):=0A> =0A> 1. in if_em.c::em_handle_que(), when em_rxeof() excee= ds the=0A> =A0 rx_process_limit, the task is rescheduled so it can=0A> comp= lete the work.=0A> =A0 Conversely, in if_lem.c::lem_handle_rxtx() the=0A> l= em_rxeof() is=0A> =A0 only run once, and if there are more pending packets= =0A> the only=0A> =A0 chance to drain them is to receive (many) more=0A> in= terrupts.=0A> =0A> =A0 This is a relatively serious problem, because the=0A= > receiver has=0A> =A0 a hard time recovering.=0A> =0A> =A0 I'd like to com= mit a fix to this same as it is done=0A> in e1000.=0A> =0A> 2. in if_em.c::= em_handle_que(), interrupts are reenabled=0A> unconditionally,=0A> =A0=A0= =A0whereas lem_handle_rxtx() only enables=0A> them if IFF_DRV_RUNNING is se= t.=0A> =0A> =A0=A0=A0I cannot really tell what is the correct=0A> way here,= so I'd like=0A> =A0=A0=A0to put a comment there unless there is a=0A> good= suggestion on=0A> =A0=A0=A0what to do.=0A> =0A> =A0=A0=A0Accesses to the i= ntr register are=0A> race-prone anyways=0A> =A0=A0=A0(disabled in fastintr,= enabled in the rxtx=0A> task without=0A> =A0=A0=A0holding any lock, and ge= nerally accessed=0A> under EM_CORE_LOCK=0A> =A0=A0=A0in other places), and = presumably=0A> enabling/disabling the=0A> =A0=A0=A0interrupts around activa= tions of the taks=0A> is just an=0A> =A0=A0=A0optimization (and on a VM, it= is actually=0A> a pessimization=0A> =A0=A0=A0due to the huge cost of VM ex= its).=0A> =0A> cheers=0A> luigi=0A=0AThis is not really a big deal; this is= how things works for a million =0Ayears before we had task queues.=0A=0AIn= tel controllers have built in interrupt moderation (unless you're on an=0AI= SA bus or something), so interrupt storms aren't possible. Typical default= =0Ais 6K ints per second, so you can't get another interrupt for 1/6000th o= f=0Aa second whether there's more work to do or not.=0A=0AThe "work" parame= ter should be an indicator that something is happening too=0Aslow, which ca= n happen with a shaper that's taking a lot more time than=0Anormal to proce= ss packets. =0A=0ASystems should have a maximum pps engineered into its tun= ing depending on=0Athe cpu to avoid live-lock on legacy systems. the defaul= t work limit of=0A100 is too low on a gigabit system. =0A=0Aqueuing tasks a= ctually creates more overhead in the system, not less. The=0Aissue is wheth= er the process_limit * interrupt_moderation is set to a =0Apps that's suita= ble for your system. =0A=0ASetting low work limits isn't really a good idea= unless you have some other=0Atime sensitive kernel task. Usually networkin= g is a priority, so setting=0Aarbitrary work limits makes less sense than q= ueuing an additional task,=0Awhich defeats the purpose of interrupt moderat= ion.=0A=0ABC