From owner-freebsd-net@FreeBSD.ORG Wed Aug 21 19:11:12 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 45E8A492 for ; Wed, 21 Aug 2013 19:11:12 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pa0-x22a.google.com (mail-pa0-x22a.google.com [IPv6:2607:f8b0:400e:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 200E2234E for ; Wed, 21 Aug 2013 19:11:12 +0000 (UTC) Received: by mail-pa0-f42.google.com with SMTP id lj1so1168811pab.1 for ; Wed, 21 Aug 2013 12:11:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=Ykc7WUzz89lClxriySZKZlK/jt+dMeFJhggjw2wD6Hs=; b=IRlsnIryOJ0pKIU1BO6MWwTO/isO+T3x4lEqIZzoCsW2cKmDish/mScciMnNPDF0ON SwN2fQC7LznaZgrPpC8AG9x/iyTZiMDQHX59TgFNhqvKnXP/msf8zG2NqSzFz2DMdhu7 ZsdQtxYUdUDDTshgTIlm5gzSI2MtT2N1+NeX6u0JHiAw4yhAuPupccl0l5HrCFQZibOS BrTZspVsLCaS7HP9m7RTtb1LfyQKEWvIVHs5H+msolnx+ZRDCkffQXOkFl7IqaPQsxUb DIJT/pk8rUjC8hGUy/0gEzDKqQMBu9m+n0+3E1xMipEeyE1Ab0QYmhENdrEaBywPGGij RMBA== X-Received: by 10.66.227.39 with SMTP id rx7mr11194028pac.44.1377112271816; Wed, 21 Aug 2013 12:11:11 -0700 (PDT) Received: from [10.192.166.0] (stargate.chelsio.com. [67.207.112.58]) by mx.google.com with ESMTPSA id s5sm9919129pbo.38.1969.12.31.16.00.00 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 21 Aug 2013 12:11:10 -0700 (PDT) Sender: Navdeep Parhar Message-ID: <521510CC.3040104@FreeBSD.org> Date: Wed, 21 Aug 2013 12:11:08 -0700 From: Navdeep Parhar User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130819 Thunderbird/17.0.8 MIME-Version: 1.0 To: "freebsd-net@freebsd.org" Subject: Please review: LRO entry last-active timestamp. Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Aug 2013 19:11:12 -0000 I'd like to add a last-active timestamp to the structure that tracks the LRO state in a NIC's rx handler. This is r254336 in user/np/cxl_tuning that will be merged to head if there are no objections. http://svnweb.freebsd.org/base?view=revision&revision=254336 http://svnweb.freebsd.org/base/user/np/cxl_tuning/sys/netinet/tcp_lro.c?r1=254336&r2=254335&pathrev=254336 http://svnweb.freebsd.org/base/user/np/cxl_tuning/sys/netinet/tcp_lro.h?r1=254336&r2=254335&pathrev=254336 ----- Add a last-modified timestamp to each LRO entry and provide an interface to flush all inactive entries. Drivers decide when to flush and what the inactivity threshold should be. Network drivers that process an rx queue to completion can enter a livelock type situation when the rate at which packets are received reaches equilibrium with the rate at which the rx thread is processing them. When this happens the final LRO flush (normally when the rx routine is done) does not occur. Pure ACKs and segments with total payload < 64K can get stuck in an LRO entry. Symptoms are that TCP tx-mostly connections' performance falls off a cliff during heavy, unrelated rx on the interface. Flushing only inactive LRO entries works better than any of these alternates that I tried: - don't LRO pure ACKs - flush _all_ LRO entries periodically (every 'x' microseconds or every 'y' descriptors) - stop rx processing in the driver periodically and schedule remaining work for later. ----- Regards, Navdeep