From owner-svn-src-all@freebsd.org Sat Feb 2 20:29:30 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6F76D14CE663; Sat, 2 Feb 2019 20:29:30 +0000 (UTC) (envelope-from pkelsey@gmail.com) Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DF01096AAA; Sat, 2 Feb 2019 20:29:29 +0000 (UTC) (envelope-from pkelsey@gmail.com) Received: by mail-pl1-f179.google.com with SMTP id u6so4954879plm.8; Sat, 02 Feb 2019 12:29:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=JTRq8g944PxpiBoPyMUQAuILltD0cX5ZfuI3CyEI7xc=; b=KrPHFHT6AIVMyAZBvAj8BCL/RkGOW5EV+O77A6UDG9GTVERP4sff+k6tgnBTx70uM9 b9HFn+oLpStoipnEBJFVUrBEJQidXhlIwNDicDw4F+9FnHAh15xEGw5UrkPom99mxNn3 2GbRpt9+kiiBMUi072L8UQhgalH47kdW7OJgKXgZwQPIQvrV+sxrsld6eS9zdwxoM+0z vz5IVs2AHrxX2QujfuLo99yK7dhvyVLClJ3p9NstQbJm9p4MVNhCN6ODxxaknmZUFykt xp8TZ9acLeAYvABIf2o5hlTpRfRKenUXnNGP4P2NdQd5YroKNIbKVbjPotQKtPhDMDJU HgSA== X-Gm-Message-State: AJcUukfwX/Km9g93QrpxvAaSS+EAeF8jT/M1oNpLG45Kq5SayG0t1vLk 8humlCEVXtzWA3vWEgoUt3dIVonvf/9hdx4mElmma9ZI X-Google-Smtp-Source: ALg8bN5xo5Pc6VFkhHZNMu1OYcCGsV4kf5vHzoavuXEPsYxaR4Vc/v1wV7q9Oq5n4Zw1NRZBLUMTBxutHsudhBbEHWY= X-Received: by 2002:a17:902:128c:: with SMTP id g12mr43286482pla.146.1549139362797; Sat, 02 Feb 2019 12:29:22 -0800 (PST) MIME-Version: 1.0 References: <201901220111.x0M1BHS1025063@repo.freebsd.org> <2c0e0179-63e7-1e2f-ba7f-6a373c927d88@yuripv.net> In-Reply-To: <2c0e0179-63e7-1e2f-ba7f-6a373c927d88@yuripv.net> From: Patrick Kelsey Date: Sat, 2 Feb 2019 15:29:09 -0500 Message-ID: Subject: Re: svn commit: r343291 - in head/sys: dev/vmware/vmxnet3 net To: Yuri Pankov Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org X-Rspamd-Queue-Id: DF01096AAA X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.85 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.85)[-0.851,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Feb 2019 20:29:30 -0000 On Sat, Feb 2, 2019 at 9:28 AM Yuri Pankov wrote: > Patrick Kelsey wrote: > > Author: pkelsey > > Date: Tue Jan 22 01:11:17 2019 > > New Revision: 343291 > > URL: https://svnweb.freebsd.org/changeset/base/343291 > > > > Log: > > onvert vmx(4) to being an iflib driver. > > > > Also, expose IFLIB_MAX_RX_SEGS to iflib drivers and add > > iflib_dma_alloc_align() to the iflib API. > > > > Performance is generally better with the tunable/sysctl > > dev.vmx..iflib.tx_abdicate=1. > > > > Reviewed by: shurd > > MFC after: 1 week > > Relnotes: yes > > Sponsored by: RG Nets > > Differential Revision: https://reviews.freebsd.org/D18761 > > This breaks vmx interfaces for me in ESXi 6.7 (output below). The > review mentions setting hw.pci.honor_msi_blacklist="0" and it helps > indeed -- worth mentioning in UPDATING? > > vmx0: port 0x3000-0x300f mem > 0xfe903000-0xfe903fff,0xfe902000-0xfe902fff,0xfe900000-0xfe901fff at > device 0.0 on pci3 > vmx0: Using 512 tx descriptors and 256 rx descriptors > vmx0: msix_init qsets capped at 8 > vmx0: intr CPUs: 20 queue msgs: 24 admincnt: 1 > vmx0: Using 8 rx queues 8 tx queues > vmx0: attempting to allocate 9 MSI-X vectors (25 supported) > vmx0: failed to allocate 9 MSI-X vectors, err: 6 - using MSI > vmx0: attempting to allocate 1 MSI vectors (1 supported) > msi: routing MSI IRQ 25 to local APIC 6 vector 48 > vmx0: using IRQ 25 for MSI > vmx0: Using an MSI interrupt > msi: Assigning MSI IRQ 25 to local APIC 25 vector 48 > msi: Assigning MSI IRQ 25 to local APIC 24 vector 48 > vmx0: bpf attached > vmx0: Ethernet address: 00:00:00:00:00:33 > vmx0: netmap queues/slots: TX 1/512, RX 1/512 > vmx0: device enable command failed! > vmx0: link state changed to UP > vmx0: device enable command failed! > > Setting hw.pci.honor_msi_blacklist="0" should only be necessary if you want to operate with more than one queue. If hw.pci.honor_msi_blacklist="0" is not set, then MSI-X will not be available, and MSI will be used, which reduces the number of queues that can be configured for use to 1. This case should work correctly. I am able to reproduce the behavior you described above on ESXi 6.7 using the latest snapshot release (based on r343598). The error that appears in the ESXi logs will be similar to: 2019-02-02T15:14:02.986Z| vcpu-1| I125: VMXNET3 user: failed to activate 'Ethernet0', status: 0xbad0001 which vaguely means 'the device did not like something about the configuration it was given'. I will see if I can determine the root cause. Given that enabling MSI-X seems to work around the problem, and based on other issues I encountered during development, I currently suspect there is a problem with the interrupt index that is being configured for the transmit queue in the device configuration structure when using MSI. -Patrick