From owner-freebsd-net@freebsd.org Wed Jan 29 21:31:10 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ECC891FF76E for ; Wed, 29 Jan 2020 21:31:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 487GrG21xjz4D4y for ; Wed, 29 Jan 2020 21:31:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by mail-lj1-f169.google.com with SMTP id y6so1045614lji.0 for ; Wed, 29 Jan 2020 13:31:10 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=VxnkWfCqndfeB5XxxE248eMspySmN3CKd9h5v6VL1eA=; b=BYlO6JexNliUrLhX8OhsDv5dNTghldU/3EKEARqgNgQAe5SarnASm1EHWCLvfVfleg Ol7/4XArZ8DEi67mM7MS6dUGpwfNqd+I84XFAkOYb3J+yOMjLBvg83U1VzVI1NX/0Y/4 /JV6xWT/1scEsJrNQCxil1rMM17eaZ4NtK0m6wp7NWiIdtKCCczAmZq34AVBaV+Wcmg8 /A2NSJ11i1iMeo5Eqi+XwKqH5SP55WfMvdbS1k5lRydJ0BljVXIpqbmD/3UwthQHpFhC TQDOHZZG1Kh1m5w+mhcOCrrnxtS7y3NYa4AVrcu5IzXvvbP6VIKr6fzc+DwR0Rl3IcK2 lKWA== X-Gm-Message-State: APjAAAVLYQIKi+aDzWnX9v3OB2dO6Xs0WD34WNFw3hHjyVBWRs2J9ZK7 tPv3xZPr70UehhbkUd3/9uMAc/SD X-Google-Smtp-Source: APXvYqzTSQha76e+c/0CYK1AvAKqXzYwr43BY5R6QLAyTHVgaPmY6IJXiAseuKaPwGxOvhaViXalDQ== X-Received: by 2002:a2e:978c:: with SMTP id y12mr672984lji.167.1580333467981; Wed, 29 Jan 2020 13:31:07 -0800 (PST) Received: from mail-lj1-f175.google.com (mail-lj1-f175.google.com. [209.85.208.175]) by smtp.gmail.com with ESMTPSA id 4sm1632516lfj.75.2020.01.29.13.31.07 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2020 13:31:07 -0800 (PST) Received: by mail-lj1-f175.google.com with SMTP id o11so1003759ljc.6 for ; Wed, 29 Jan 2020 13:31:07 -0800 (PST) X-Received: by 2002:a2e:844e:: with SMTP id u14mr686118ljh.183.1580333466679; Wed, 29 Jan 2020 13:31:06 -0800 (PST) MIME-Version: 1.0 From: Eric Joyner Date: Wed, 29 Jan 2020 13:30:55 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Issue with epoch_drain_callbacks and unloading iavf(4) [using iflib] To: freebsd-net@freebsd.org X-Rspamd-Queue-Id: 487GrG21xjz4D4y X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ricera10@gmail.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=ricera10@gmail.com X-Spamd-Result: default: False [-3.06 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_IN_DNSWL_NONE(0.00)[169.208.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.06)[ip: (-0.42), ipnet: 209.85.128.0/17(-3.05), asn: 15169(-1.78), country: US(-0.05)]; FORGED_SENDER(0.30)[erj@freebsd.org,ricera10@gmail.com]; DMARC_NA(0.00)[freebsd.org]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[erj@freebsd.org,ricera10@gmail.com]; RCVD_TLS_ALL(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jan 2020 21:31:11 -0000 Hi freebsd-net, We've encountered an issue with unloading the iavf(4) driver on FreeBSD 12.1 (and stable). On a VM with two iavf(4) interfaces, if we send heavy traffic to iavf1 and try to kldunload the driver, the kldunload process hangs on iavf0 until iavf1 stops receiving traffic. After some debugging, it looks like epoch_drain_callbacks() [via if_detach_internal()] tries to switch CPUs to run on one that iavf1 is using for RX processing, but since iavf1 is busy, it can't make the switch, so cpu_switch() just hangs and nothing happens until iavf1's RX thread stops being busy. I can work around this by inserting a kern_yield(PRI_USER) somewhere in one of the iavf txrx functions that iflib calls into (e.g. iavf_isc_rxd_available), but that's not a proper fix. Does anyone know what to do to prevent this from happening? Wildly guessing, does maybe epoch_drain_callbacks() need a higher priority than the PI_SOFT used in the group taskqueues used in iflib's RX processing? - Eric