From owner-freebsd-net@freebsd.org Sat Jan 23 19:12:31 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5326CA8E5C2 for ; Sat, 23 Jan 2016 19:12:31 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x230.google.com (mail-lf0-x230.google.com [IPv6:2a00:1450:4010:c07::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCE5314D1 for ; Sat, 23 Jan 2016 19:12:30 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lf0-x230.google.com with SMTP id 17so64605286lfz.1 for ; Sat, 23 Jan 2016 11:12:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=nvuBrkx1fLNWXhdJwgwJls+frjLQG5XOMxPjbN2TvFI=; b=dOQi+/vhNd8W4xvZ2zVFFttw9DozrO8mnp+MgrKuu+GZs+VGT/MRJyCpGMUR5OUHHC R48VSuxx5Sm56/G0NzC3/f7XiOHQRomZqHWqXvuFwktEyGPb9RAgb4Ggwp8cScZjW81n KSl6r4wnZioiiZ0nd1aH3ZtzLwnLYtvEEbhq6F695+thUA883OF6pVyAX2rdjpPsi2vV Y0GFT5z5iSE0RrCULAHj7hy0o2bsWOa8aSsSV2Y1hL2o4QYVRQLLR4en13F6MkHdsJTv earJScNM+Q2hHP8+X4G73ZTnW868HvhuJksHkLdck4BChs352bUK2M+NIMKy17O9nOpk gqkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=nvuBrkx1fLNWXhdJwgwJls+frjLQG5XOMxPjbN2TvFI=; b=ltyId2n9gsfFqXROA93t8BhSdFxRLoYNGXHDclIMwNVC2Y2pW9cicfz5E9nsCy93DF Kiq+vVbsY5QGUS3Yb7+FUBMbPcK+rG2a9+LcE7woNn+LfTninVhY48PTlVfPVNchvgrE 685KfSjLpkjnnV6wvKNgVGj2uVHF+iJF16kXoEqZd2ait0UPEmeKCrpRfCOEpNrQxzty 1Ka9DikfMhOv4N0tPfvJimVw4btN5apBC/QInrY+MtXlkkBX1ik37YXRBC4Xa2oCTPUO blo5Jof5pMSVHOum2OZZBGT/fEJ7m8EW0alMp+bcXp74XOIbI6uSatz9oSd29sIFA4LS SyXw== X-Gm-Message-State: AG10YOTHC5A0NaiZBdWBEpAYQe8/HrZ1CqilC01YveVw0XxfDooozOFbJyIuM2m5ByICfiXtFldmADYfNqBV+A== MIME-Version: 1.0 X-Received: by 10.25.143.17 with SMTP id r17mr3541937lfd.37.1453576348879; Sat, 23 Jan 2016 11:12:28 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 11:12:28 -0800 (PST) In-Reply-To: <20160123183836.GB4574@ox> References: <20160123053428.2091EA0121@smtp.hushmail.com> <20160123154052.GA4574@ox> <20160123171300.0F448A0121@smtp.hushmail.com> <20160123174840.32B1DA0121@smtp.hushmail.com> <20160123183836.GB4574@ox> Date: Sat, 23 Jan 2016 11:12:28 -0800 X-Google-Sender-Auth: Cfk9s2Kijid6nFrLiI6IEjHa33w Message-ID: Subject: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Marcus Cenzatti , Luigi Rizzo , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Jan 2016 19:12:31 -0000 On Sat, Jan 23, 2016 at 10:38 AM, Navdeep Parhar wrote: > On Sat, Jan 23, 2016 at 03:48:39PM -0200, Marcus Cenzatti wrote: > ... >> >> woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps. > > Run "netstat -hdw1 -i cxl" on the receiver during your test. Navdeep, does this give any info on the ncxl port rather than the cxl port connected to the host stack ? ... > Do you know if the transmitter will pad up so as not to put runts on the > wire? If not then you might want to bump up the size of the frame > explicitly (there's some pkt-gen knob for this). > ix/ixl do automatic padding, and in any case pkt-gen by default generates valid packet sizes (and so it does with the variable-size tests I suggested). Is there any parameter that controls interrupt moderation ? In any case we need to know the numbers when sending to the ncxl MAC address as opposed to broadcast. I suspects one of these problems: - the interrupt moderation interval is too long thus limiting the rate to one ring/interval. Unlikely though, even with 1k slots, the measured 1.2 Mpps corresponds to almost 1ms which is too long - the receiver cannot cope with the input load and somehow takes a long time to recover from congestion. If this is the case, running the sender at a lower rate might reach a peak throughput > 1.2 Mpps when the receiver can still keep up, and then drop to the low rate under congestion. - and of course bus errors, when the device is connected on a PCIe slot with only 1-2 data lanes. This actually happens a lot, physical connector sizes do not reflect the number of active PCIe lanes. cheers luigi