From owner-freebsd-net@freebsd.org Wed Mar 28 04:11:55 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1441F57AFA for ; Wed, 28 Mar 2018 04:11:55 +0000 (UTC) (envelope-from christian.baltini@gmail.com) Received: from mail-ot0-x230.google.com (mail-ot0-x230.google.com [IPv6:2607:f8b0:4003:c0f::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 505F773AE6 for ; Wed, 28 Mar 2018 04:11:55 +0000 (UTC) (envelope-from christian.baltini@gmail.com) Received: by mail-ot0-x230.google.com with SMTP id m7-v6so1246865otd.1 for ; Tue, 27 Mar 2018 21:11:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=0g8MH1ynW/4W9eJzHH+/RRnfZLSu1yp/h63WSc8vgG0=; b=HgNg8MOknC4KoU3BPJ6Gul2iunHTpoe4P5yY6bi7BxCuOZ38w5AlvmQZdmeyd9xyVh oBxzTPuZDTibiyMeODSFsJGLhgg8Aumpxi9jWOZqreOTNRJjIij6+OtUppZgbmaDfzlp dQiwSs8NPB/XRluNsGKE+OE38cSInQEoxjDAqNrrXS75vJIh8UyH4t8LpZQVDw5LhLHv m7NrG9K5YAmis38XvC3PtfSLEh0rY4own48y9NDaZg06vnZD9dZk40Wmwha7A149mG0T XMtSgApfjjOG1Rsu6QoIpUvh8toXDo//H8bstT557qbZolWut4pV1wpmIgl05bryzCYq VX7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=0g8MH1ynW/4W9eJzHH+/RRnfZLSu1yp/h63WSc8vgG0=; b=EYQU3aweNdF9oShWbUZ9KC+pzxN7kn0/0/UemjjnFegFE76xWiIv6EqQ/acU00xZbd HI6hv/h6xX0TTzectgiOrjFWAxutowCZiFOp+tEqV8Riorn2MkX/+tzy6fLDVioZ0upk cOeaNhbam8qPn7/kFOYTro7/5PVcxpo8Ls5ctMc4vqJZFcDKckrKmmkos2b0HMYkR384 IoPJLlKnLotQ097zp/t1zT177oTx3XJ/6nY/nWD5Zrup41Z8+YR6Gmv4VdFya4YhnaMA Xv5DkbzmHHIiJ/lxZCIwWzgS5QyCIgFFKKtYyfNMmt3C0JYb5D21s1dc+XD+VmXpZkGy xh6A== X-Gm-Message-State: AElRT7HKdCXRCdAGKYqB/aNU4jDxi3QCVIZ7wklfROtOueacs7OZXS/e Mj0PHyR+dGaV9eRSM5I3kx6cNBRYkjO0f8tdS355a21lPPg= X-Google-Smtp-Source: AIpwx4/U+JkOp0puWr/ggi2mAy2gvgVZHcH91BMMBiKVs+FW0Cux6CZ9BF9Dd6y0n1cBLFhrmlu6oLCrw9aaiqLpFGw= X-Received: by 2002:a9d:618c:: with SMTP id g12-v6mr63332otk.225.1522210314535; Tue, 27 Mar 2018 21:11:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.200.144 with HTTP; Tue, 27 Mar 2018 21:11:54 -0700 (PDT) From: christian russell Date: Tue, 27 Mar 2018 21:11:54 -0700 Message-ID: Subject: Current state of Intel XL710 40G NIC ixl performance To: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Mar 2018 04:11:55 -0000 I am having trouble getting an Intel XL710-DA2 NIC to get even close to line rate. It is a 4x10 Gbps card. The box is running FreeBSD 11 (FreeNAS in particular). We have tried both 1.7 and 1.9 driver revisions with similar results. The NVM version is 5.05. The card is in a confirmed 8x slot on a SuperMicro X10DRL-i with two Xeon E5-2600 processors and 256 GB DDR4 RAM. After upping the interrupt threshold to 9000 dmesg doesn't log anything unusual. We have added the tunes that are standard for 10 Gbps configurations. On a single-client basis the fastest rates we see are around 5 Gbps. Hitting this server from multiple boxes we see peaks of 20 Gbps at the very highest. More frequently things top off around 13 Gbps. These numbers are coming from iperf tests. We are seeing similar numbers with direct point-to-point as well as switched topologies. These threads from 2015 describe similar issues but fizzled out: https://lists.freebsd.org/pipermail/freebsd-net/2015-May/042273.html https://lists.freebsd.org/pipermail/freebsd-net/2015-October/043584.html Is there very particular tuning required to get these cards working at proper speed? Any insights? >From Googling around it appears frustration with this card and FreeBSD is pretty common. Thanks in advance. Christian