From owner-freebsd-virtualization@freebsd.org Mon Nov 7 14:13:01 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE6E8C33676 for ; Mon, 7 Nov 2016 14:13:01 +0000 (UTC) (envelope-from rajil.s@gmail.com) Received: from mail-pf0-x22f.google.com (mail-pf0-x22f.google.com [IPv6:2607:f8b0:400e:c00::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BD4C3D82 for ; Mon, 7 Nov 2016 14:13:01 +0000 (UTC) (envelope-from rajil.s@gmail.com) Received: by mail-pf0-x22f.google.com with SMTP id d2so91955824pfd.0 for ; Mon, 07 Nov 2016 06:13:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=+2dpCnzPdwwQaOekqhJBffvq4B0Qqr8nmR11IYrmIrs=; b=h1Eps55VswWWdpXLkytFxwfX6oF/6a7mOyf2j7dKB+WjVvz3qPfzrQcCjRyHW3CuW0 SiecQbAEGxIS3Zfzr+o9wFz3qyldkwIJKwoVFHLG84UVeKPHk3x+jSTw8Mk+yowwg2em X8sdoRNDweKDnIOberUv3CVjzuv6GjtA7TEvchkQmfO5tu47VIvI6VuyJLuvYNPzWXT1 Amb9FJhdYhMS+r8hPuOSwTRFYb3Tug5FseRcUgH2OSmQyoa0Gs6RsL4/LUh3DAww9NjG dX1HjwPKyDpDbTtHEMTdwmb3jJUSAFwwSnWLYHH3Ctye4tR8A4kpT3WLG/lN7+u1UD1r ozoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=+2dpCnzPdwwQaOekqhJBffvq4B0Qqr8nmR11IYrmIrs=; b=idlqfZ1Ga1UQLgF3LXXOlxauI2IhRGRomGPFYVu3+tlbEbHlLoCLJwtdvEBJxJIegt kR+06y+NgziBAyrfcz2+BJLJwDbt5vCfSEGQRjFTn+PQXFRJDewIZz2AhYUuqb0/4ShK c2iWU9p2WJqVz86Hh6GCclc1RDNh5J/D+up3Y+QGcnIJXdATH+dWNGNlynOdKRjD+777 /RD0DZ758u6xD0gz8Dx/aB4fajXeqZ3bCxSeUyv0EUcJ84pGxN9F1lDZKAUMRXyYNqbJ EE9Vskm//Ig35l3KimiExLe3Z8ScY6Yid5tVbB6MZ3JikceaxYijX6WW0jzXK8+c59Ox A9kw== X-Gm-Message-State: ABUngvf3tu3gVen/K79GOJdhDkQG/ht8RGmoYx/RDKDcWjcxj6bGAeQIquZZHp+vPUwNqA== X-Received: by 10.99.142.201 with SMTP id k192mr10965867pge.174.1478527981029; Mon, 07 Nov 2016 06:13:01 -0800 (PST) Received: from ?IPv6:2601:2c2:200:12c7:202:c9ff:fe4e:1dc0? ([2601:2c2:200:12c7:202:c9ff:fe4e:1dc0]) by smtp.gmail.com with ESMTPSA id v6sm41156762pab.14.2016.11.07.06.12.59 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 07 Nov 2016 06:13:00 -0800 (PST) Subject: Re: bhyve, ixgbe and debian 8 To: freebsd-virtualization@FreeBSD.org References: <124ae089-6215-7632-44cf-319b5f88c0b5@gmail.com> <8c44e3c4-fcc8-d288-26f0-df742b1c79a7@gmail.com> From: Rajil Saraswat Message-ID: Date: Mon, 7 Nov 2016 08:12:55 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: <8c44e3c4-fcc8-d288-26f0-df742b1c79a7@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Nov 2016 14:13:02 -0000 On 11/07/2016 07:41 AM, Rajil Saraswat wrote: > On 11/07/2016 07:08 AM, Rajil Saraswat wrote: >> Hello, >> >> I have PCI passthrough working now for the Intel X540-AT2 NIC and is >> seen inside debian 8 vm. >> >> # lspci >> >> 00:00.0 Host bridge: Network Appliance Corporation Device 1275 >> 00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device >> 00:05.0 Ethernet controller: Red Hat, Inc Virtio network device >> 00:06.0 Ethernet controller: Intel Corporation Ethernet Controller >> 10-Gigabit X540-AT2 (rev 01) >> 00:1f.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] >> >> >> However, on loading the module the kernel gives the error >> >> #modprobe ixgbe >> >> [ 34.929546] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - >> version 3.19.1-k >> [ 34.929549] ixgbe: Copyright (c) 1999-2014 Intel Corporation. >> [ 34.929706] ixgbe 0000:00:06.0: can't derive routing for PCI INT A >> [ 34.929708] ixgbe 0000:00:06.0: PCI INT A: no GSI >> [ 34.930354] ixgbe 0000:00:06.0: Adapter removed >> [ 34.931710] ixgbe 0000:00:06.0: can't derive routing for PCI INT A >> [ 34.931717] ixgbe: probe of 0000:00:06.0 failed with error -5 >> >> Any idea what could be the issue? >> >> Thanks, >> >> Rajil >> > I tried with the second onboard NIC and it worked the first time. iperf > showed 10g speeds. After restarting the VM, i get an error for both > NICs. This a dual CPU motherboard Supermicro X10DriT with a pair of > onboard NICs. > > [ 0.628758] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - > version 3.19.1-k > [ 0.628760] ixgbe: Copyright (c) 1999-2014 Intel Corporation. > [ 0.628899] ixgbe 0000:00:06.0: can't derive routing for PCI INT A > [ 0.628901] ixgbe 0000:00:06.0: PCI INT A: no GSI > [ 0.629519] ixgbe 0000:00:06.0: Adapter removed > [ 0.630143] ixgbe 0000:00:06.0: can't derive routing for PCI INT A > [ 0.630147] ixgbe: probe of 0000:00:06.0 failed with error -5 > [ 0.630199] ixgbe 0000:00:06.1: can't derive routing for PCI INT B > [ 0.630200] ixgbe 0000:00:06.1: PCI INT B: no GSI > [ 0.630789] ixgbe 0000:00:06.1: Adapter removed > [ 0.639469] ixgbe 0000:00:06.1: can't derive routing for PCI INT B > [ 0.639473] ixgbe: probe of 0000:00:06.1 failed with error -5 > After a cold reboot of the server, the devices are again seen in debian. I guess somehow the devices become unusable when the vm is restarted. [ 1.146946] ixgbe 0000:00:06.0: PCI Express bandwidth of 32GT/s available [ 1.146949] ixgbe 0000:00:06.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%) [ 1.146953] ixgbe 0000:00:06.0: This is not sufficient for optimal performance of this card. [ 1.146955] ixgbe 0000:00:06.0: For optimal performance, at least 60GT/s of bandwidth is required. [ 1.146956] ixgbe 0000:00:06.0: A slot with more lanes and/or higher speed is suggested. [ 1.189458] ixgbe 0000:00:06.0: MAC: 3, PHY: 3, PBA No: 030C00-000 [ 1.189461] ixgbe 0000:00:06.0: 0c:c4:7a:de:cc:c0 [ 1.339231] ixgbe 0000:00:06.0: Intel(R) 10 Gigabit Network Connection [ 1.339364] ixgbe 0000:00:06.1: can't derive routing for PCI INT B [ 1.339366] ixgbe 0000:00:06.1: PCI INT B: no GSI [ 1.348140] input: PS/2 Generic Mouse as /devices/platform/i8042/serio1/input/input2 [ 1.566630] tsc: Refined TSC clocksource calibration: 2198.264 MHz [ 1.793929] ixgbe 0000:00:06.1: irq 47 for MSI/MSI-X [ 1.793950] ixgbe 0000:00:06.1: irq 48 for MSI/MSI-X [ 1.795079] ixgbe 0000:00:06.1: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1 [ 1.859258] ixgbe 0000:00:06.1: PCI Express bandwidth of 32GT/s available [ 1.859261] ixgbe 0000:00:06.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%) [ 1.859262] ixgbe 0000:00:06.1: This is not sufficient for optimal performance of this card. [ 1.859264] ixgbe 0000:00:06.1: For optimal performance, at least 60GT/s of bandwidth is required. [ 1.859265] ixgbe 0000:00:06.1: A slot with more lanes and/or higher speed is suggested. [ 1.902549] ixgbe 0000:00:06.1: MAC: 3, PHY: 3, PBA No: 030C00-000 [ 1.902554] ixgbe 0000:00:06.1: 0c:c4:7a:de:cc:c1 [ 2.055631] ixgbe 0000:00:06.1: Intel(R) 10 Gigabit Network Connection # cat /proc/interrupts CPU0 0: 45 IO-APIC-edge timer 1: 9 IO-APIC-edge i8042 4: 3756 IO-APIC-edge serial 8: 0 IO-APIC-edge rtc0 9: 0 IO-APIC-fasteoi acpi 12: 138 IO-APIC-edge i8042 40: 0 PCI-MSI-edge virtio0-config 41: 5667 PCI-MSI-edge virtio0-requests 42: 0 PCI-MSI-edge virtio1-config 43: 139 PCI-MSI-edge virtio1-input.0 44: 1 PCI-MSI-edge virtio1-output.0 45: 120 PCI-MSI-edge eth1-TxRx-0 46: 0 PCI-MSI-edge eth1 47: 1170 PCI-MSI-edge eth2-TxRx-0 48: 3 PCI-MSI-edge eth2 NMI: 0 Non-maskable interrupts LOC: 8265 Local timer interrupts SPU: 0 Spurious interrupts PMI: 0 Performance monitoring interrupts IWI: 0 IRQ work interrupts RTR: 0 APIC ICR read retries RES: 0 Rescheduling interrupts CAL: 0 Function call interrupts TLB: 0 TLB shootdowns TRM: 0 Thermal event interrupts THR: 0 Threshold APIC interrupts MCE: 0 Machine check exceptions MCP: 1 Machine check polls HYP: 0 Hypervisor callback interrupts ERR: 0 MIS: 0