From owner-freebsd-net@freebsd.org Wed Oct 21 19:07:37 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21FAEA1B20A for ; Wed, 21 Oct 2015 19:07:37 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-wi0-x236.google.com (mail-wi0-x236.google.com [IPv6:2a00:1450:400c:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5A57FA3 for ; Wed, 21 Oct 2015 19:07:36 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by wicfx6 with SMTP id fx6so105267110wic.1 for ; Wed, 21 Oct 2015 12:07:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft_com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=CrUiBbKYG60OKz11QpwUNuJJ78PNnyX2qcB2L7G5RyA=; b=x2EmBMuDI/vIppuTtH0j8nBhCM5hMsDf3VHBv7XaPmtaq2bwU1B2Y4Sxuc1PlEOmi3 KNFXu5S13BKZFMZ52N5DZNelP4sFQCQhv4BX4CCnfSaXn9mT8PIsMc/B0WqZ7GVrxMty e7Mpid/yMAYJsKYsLw2hWlxRqTyCJxscZGlEn190hLNiqYdTKIWyK73sAhm/dExy8eIz GADGjZmHtHHJ1wW0JxfFhDpAWzRHln3xtoIwTVUjB3rFhJ/tXrXkTyCbmOwaUlRw3/61 wI5KM/dnzkcYoWfTktTUN4ymmKRCRWXu10ir23DoJ8n4nrquJHKKF1QcNzg5Ccw8Af1y BuWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=CrUiBbKYG60OKz11QpwUNuJJ78PNnyX2qcB2L7G5RyA=; b=ZVKZTeHE3+EIQ18HbF0tbWOBwTU9nTgOTqgivfOUoiALZEhuodexLPQJCpcn2sVqrO 0mJ5Ns62UikyKeiGgG8juSZuI1HNmEeOvjViknbBot1dQ3HV0yHzJH45JuuQbcDOhTVN N0wrnlN1vpVCpOolvK+LcQ2G2gU/g8rpqGRfSjqWzFupuUQTVpPEWZi8Rb6clyKIKDzq /RziS6H7swxbFbtdwoL9woO2tsV/+L9A+oZGvdguZgrIYhxnMROoOvI9udJetSgFgBEa SWdM+OcS55u13HvhnzJJzSvTz5E2UuP626m+TW+zYeTVkzzHMczJGYGZ0X5vrjo9OsdD 22dQ== X-Gm-Message-State: ALoCoQlgbCj/pAktmIPR7Mx5wXZD6BYfdqbkuyNVMiNjfCqGMk7ulJARI7GvVz5lW12pty3ADYhZ MIME-Version: 1.0 X-Received: by 10.180.99.8 with SMTP id em8mr12345288wib.8.1445454454904; Wed, 21 Oct 2015 12:07:34 -0700 (PDT) Sender: sobomax@sippysoft.com Received: by 10.27.11.228 with HTTP; Wed, 21 Oct 2015 12:07:34 -0700 (PDT) In-Reply-To: <206504538.Si4KCGV8IQ@ralph.baldwin.cx> References: <2050066.YiOUntKNB6@ralph.baldwin.cx> <206504538.Si4KCGV8IQ@ralph.baldwin.cx> Date: Wed, 21 Oct 2015 12:07:34 -0700 X-Google-Sender-Auth: dmlnxWj_HtG2PhncuEEFAptbh2o Message-ID: Subject: Re: Some MSI are not routed correctly From: Maxim Sobolev To: John Baldwin Cc: FreeBSD Net Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Oct 2015 19:07:37 -0000 Oh, bingo! Just checked packets count and found the following: $ sysctl -a | grep dev.igb.1.queue | grep packets dev.igb.1.queue3.rx_packets: 2997 dev.igb.1.queue3.tx_packets: 21045801676 dev.igb.1.queue2.rx_packets: 3084 dev.igb.1.queue2.tx_packets: 21265692009 dev.igb.1.queue1.rx_packets: 3016 dev.igb.1.queue1.tx_packets: 21496134503 dev.igb.1.queue0.rx_packets: 48868 dev.igb.1.queue0.tx_packets: 21729900371 $ sysctl -a | grep dev.igb.0.queue | grep packets dev.igb.0.queue3.rx_packets: 40760861870 dev.igb.0.queue3.tx_packets: 21068449957 dev.igb.0.queue2.rx_packets: 40724698310 dev.igb.0.queue2.tx_packets: 21288469372 dev.igb.0.queue1.rx_packets: 40739376158 dev.igb.0.queue1.tx_packets: 21519768656 dev.igb.0.queue0.rx_packets: 40602824141 dev.igb.0.queue0.tx_packets: 21754065014 Apparently all incoming packets are going through igb0, while outbound get distributed. This means the upstream switch is not doing proper load balancing between two ports. We'll take it to the DC to fix. Thanks John, for helping to drill that down! On Wed, Oct 21, 2015 at 11:31 AM, John Baldwin wrote: > On Wednesday, October 21, 2015 11:29:17 AM Maxim Sobolev wrote: > > Yes, I do. However, please note that for some reason they are not using > > nearly as much CPU time as the other 4 for some reason. > > > > 11 root -92 - 0K 1104K WAIT 3 95.3H 28.96% > > intr{irq267: igb0:que} > > 11 root -92 - 0K 1104K WAIT 1 95.5H 24.41% > > intr{irq265: igb0:que} > > 11 root -92 - 0K 1104K CPU2 2 95.2H 23.73% > > intr{irq266: igb0:que} > > 11 root -92 - 0K 1104K WAIT 0 95.2H 23.05% > > intr{irq264: igb0:que} > > 11 root -92 - 0K 1104K WAIT 6 286:37 1.12% > > intr{irq271: igb1:que} > > 11 root -92 - 0K 1104K WAIT 7 278:05 1.12% > > intr{irq272: igb1:que} > > 11 root -92 - 0K 1104K WAIT 5 284:26 1.07% > > intr{irq270: igb1:que} > > 11 root -92 - 0K 1104K WAIT 4 290:41 0.98% > > intr{irq269: igb1:que} > > > > CPU 0: 0.0% user, 0.0% nice, 0.9% system, 24.9% interrupt, 74.2% idle > > CPU 1: 0.5% user, 0.0% nice, 0.0% system, 26.3% interrupt, 73.2% idle > > CPU 2: 0.0% user, 0.0% nice, 1.4% system, 25.4% interrupt, 73.2% idle > > CPU 3: 0.0% user, 0.0% nice, 0.5% system, 23.9% interrupt, 75.6% idle > > CPU 4: 0.9% user, 0.0% nice, 2.3% system, 2.3% interrupt, 94.4% idle > > CPU 5: 1.4% user, 0.0% nice, 4.2% system, 4.2% interrupt, 90.1% idle > > CPU 6: 1.4% user, 0.0% nice, 3.8% system, 1.4% interrupt, 93.4% idle > > CPU 7: 2.8% user, 0.0% nice, 0.0% system, 3.8% interrupt, 93.4% idle > > > > 34263 igb0:que 0 > > 32308 igb0:que 1 > > 35022 igb0:que 2 > > 34593 igb0:que 3 > > 14931 igb1:que 0 > > 13059 igb1:que 1 > > 12971 igb1:que 2 > > 13032 igb1:que 3 > > > > So I guess interrupts are routed correctly after all, but for some reason > > driver takes some 5 times less time to process it on cpus 4-7 > > (per-interrupt). Weird. > > Are the pps rates the same? It seems like the interrupt rates on igb0 > are double those of igb1? > > -- > John Baldwin >