From owner-svn-src-all@FreeBSD.ORG Thu Nov 6 17:21:14 2014 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6AC83A38; Thu, 6 Nov 2014 17:21:14 +0000 (UTC) Received: from mail-wg0-x22f.google.com (mail-wg0-x22f.google.com [IPv6:2a00:1450:400c:c00::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EEEE7CA3; Thu, 6 Nov 2014 17:21:13 +0000 (UTC) Received: by mail-wg0-f47.google.com with SMTP id a1so1746951wgh.6 for ; Thu, 06 Nov 2014 09:21:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Kj3IqJqICx1qVhI2ZCd8OGbX0fmuKixCG2w4bS2ZVG4=; b=UZ4MC77VWTv0N4pFt8iXaha2gwaRwIifkWWqPPFz7qy3dhYtcSlWhwt9ApBE7+nkLj ADuHbwQX6QKkZz5MAsIVAUTLPRR61i/c8ALFLS6LS72oYNDZLvVkQYOSsNLc217Ti4Pl EEqOw7VOrAjODk1uHZWx4ebe3CCYfim+LKuRbr7C4KNqod/0GZIl+X2U5QIA+6JAWWCW zSrPhFq3JlNyNWPSnV6AQnSttD3xnzxA443HnqGhuPVBgerW4bYXEEM2+sp9tvhJ8M4r 4OkFpztM1Ljcg9F9HNesgZWqyKxjEiNag8/XUNqlJ0Ko5Msh6WL/sAwWB9sm67glMruN 0Mvw== MIME-Version: 1.0 X-Received: by 10.194.47.226 with SMTP id g2mr7680386wjn.68.1415294471527; Thu, 06 Nov 2014 09:21:11 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.136 with HTTP; Thu, 6 Nov 2014 09:21:11 -0800 (PST) In-Reply-To: <545B55FA.6070408@FreeBSD.org> References: <201410150122.s9F1Mudu083306@svn.freebsd.org> <545B55FA.6070408@FreeBSD.org> Date: Thu, 6 Nov 2014 09:21:11 -0800 X-Google-Sender-Auth: 4OHi_iflqgKlv_KTEiG2Mk_BfdI Message-ID: Subject: Re: svn commit: r273112 - head/sys/dev/ixgbe From: Adrian Chadd To: "Alexander V. Chernikov" Content-Type: text/plain; charset=UTF-8 Cc: "svn-src-head@freebsd.org" , "svn-src-all@freebsd.org" , "src-committers@freebsd.org" X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Nov 2014 17:21:14 -0000 Hm, are there any other hardware counters that we can use to see what was dropped? I didn't see performance drops doing this; in fact I saw performance increases with FC disabled and DROP_EN set. Thanks, -adrian On 6 November 2014 03:05, Alexander V. Chernikov wrote: > On 15.10.2014 05:22, Adrian Chadd wrote: >> >> Author: adrian >> Date: Wed Oct 15 01:22:56 2014 >> New Revision: 273112 >> URL: https://svnweb.freebsd.org/changeset/base/273112 >> >> Log: >> Set the DROP_EN bit before the RX queue is brought up and active. >> He noticed issues setting this bit in SRRCTL after the queue was up, >> so doing it from the sysctl handler isn't enough and may not actually >> work correctly. > > While investigating/measuring ixgbe performance w/ and without flow control > I noticed that disabling fc entirely (e.g. disabling 512K skid buffer and > turning drop_en on) > not only makes things a bit worse, but also makes NIC stop accounting > tail-drops as errors > at all. > It is a bit unhandy with HW counters, since you, for example, see 2M packets > being received > in netstat, but in reality system receives only 1M (And there is no easy way > to determine exact > count). > >> This commit doesn't remove the sysctl path or try to change its >> behaviour. I'll talk with others about how to finish fixing that >> before I tackle that. >> PR: kern/194311 >> Submitted by: luigi >> MFC after: 3 days >> Sponsored by: Norse Corp, Inc >> >> Modified: >> head/sys/dev/ixgbe/ixgbe.c >> >> Modified: head/sys/dev/ixgbe/ixgbe.c >> >> ============================================================================== >> --- head/sys/dev/ixgbe/ixgbe.c Wed Oct 15 01:16:11 2014 (r273111) >> +++ head/sys/dev/ixgbe/ixgbe.c Wed Oct 15 01:22:56 2014 (r273112) >> @@ -4377,6 +4377,20 @@ ixgbe_initialize_receive_units(struct ad >> srrctl &= ~IXGBE_SRRCTL_BSIZEPKT_MASK; >> srrctl |= bufsz; >> srrctl |= IXGBE_SRRCTL_DESCTYPE_ADV_ONEBUF; >> + >> + /* >> + * Set DROP_EN iff we have no flow control and >1 queue. >> + * Note that srrctl was cleared shortly before during >> reset, >> + * so we do not need to clear the bit, but do it just in >> case >> + * this code is moved elsewhere. >> + */ >> + if (adapter->num_queues > 1 && >> + adapter->hw.fc.requested_mode == ixgbe_fc_none) { > > This code seems to turn drop_en always on HEAD due to > hw->fc.requested_mode is inherited from adapter->fc at the end of > ixgbe_init_locked(), _after_ calling ixgbe_initialize_receive_units() > >> + srrctl |= IXGBE_SRRCTL_DROP_EN; >> + } else { >> + srrctl &= ~IXGBE_SRRCTL_DROP_EN; >> + } >> + >> IXGBE_WRITE_REG(hw, IXGBE_SRRCTL(i), srrctl); >> /* Setup the HW Rx Head and Tail Descriptor Pointers */ >> >> >