From owner-freebsd-net@freebsd.org Thu Jan 30 01:29:10 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 515DF1FF5CE for ; Thu, 30 Jan 2020 01:29:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 487N6s2H0hz4Ss5 for ; Thu, 30 Jan 2020 01:29:08 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by mail-qk1-f173.google.com with SMTP id g195so1345600qke.13 for ; Wed, 29 Jan 2020 17:29:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=GLI5xkKWCiGcMn9+w09d44JdZaA+pfei5ltHUrP1CWs=; b=B8rj0BzTVIN0ygyTHIyuKZr6BdRVwneFr2dsOZXc4HXIvWt7c3xcOyW5tdm4bQ93WL fqo01gc5Og2QoxNVSUUtwVDLGfQL5UHoeLV8GFdC1tPNBPRk4FmBBVUflkypM9TwYtcj WL1sYEQ4Dnn2VXQPHpcpwMNd/3V+D7Cgdbf3ngPlT3cXTdVLxk+GI5qaL7xviUaKZSMK 5NBhKIi+YwsHnttdI9tvwLUXZFn7BpTC8QlENgP4/9NW/7zhi07/CANXXw7PiK1ZxYXL WbgmLyyX6EN+BJY8nT1jswpkIrR5qauzlpdyQYNRmZdFccwmlmrKD/3g043V6CLBnfFN LOBw== X-Gm-Message-State: APjAAAURgwnL5CH9JQoZaNGPEaxNwQSmS5DrrdcaRbOQ/Bpc3g5bY7qV 80l5PsqSu/YSMRsgq0FOrE1nM+3mJuo= X-Google-Smtp-Source: APXvYqycakIl97r5eDIJ3Z4CHAReQXLPUDRyrmwLFeJhKum+f0xjoWtCN3/ffS7mNdBlEihQrksB7Q== X-Received: by 2002:ae9:c205:: with SMTP id j5mr2807274qkg.58.1580347747977; Wed, 29 Jan 2020 17:29:07 -0800 (PST) Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com. [209.85.222.170]) by smtp.gmail.com with ESMTPSA id s20sm1913228qkg.131.2020.01.29.17.29.07 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Jan 2020 17:29:07 -0800 (PST) Received: by mail-qk1-f170.google.com with SMTP id w25so1423677qki.3 for ; Wed, 29 Jan 2020 17:29:07 -0800 (PST) X-Received: by 2002:ae9:e892:: with SMTP id a140mr2825142qkg.47.1580347747208; Wed, 29 Jan 2020 17:29:07 -0800 (PST) MIME-Version: 1.0 References: <0e2e97f2-df75-3c6f-9bdd-e8c2ab7bf79e@selasky.org> In-Reply-To: From: Eric Joyner Date: Wed, 29 Jan 2020 17:28:31 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Issue with epoch_drain_callbacks and unloading iavf(4) [using iflib] To: Hans Petter Selasky Cc: freebsd-net@freebsd.org X-Rspamd-Queue-Id: 487N6s2H0hz4Ss5 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ricera10@gmail.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=ricera10@gmail.com X-Spamd-Result: default: False [-2.54 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; URI_COUNT_ODD(1.00)[5]; RCVD_COUNT_THREE(0.00)[4]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[173.222.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.54)[ip: (-2.84), ipnet: 209.85.128.0/17(-3.05), asn: 15169(-1.78), country: US(-0.05)]; FORGED_SENDER(0.30)[erj@freebsd.org,ricera10@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[173.222.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[erj@freebsd.org,ricera10@gmail.com]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jan 2020 01:29:10 -0000 On Wed, Jan 29, 2020 at 5:12 PM Hans Petter Selasky wrote: > On 2020-01-29 22:44, Eric Joyner wrote: > > On Wed, Jan 29, 2020 at 1:41 PM Hans Petter Selasky > wrote: > > > >> On 2020-01-29 22:30, Eric Joyner wrote: > >>> Hi freebsd-net, > >>> > >>> We've encountered an issue with unloading the iavf(4) driver on FreeBSD > >>> 12.1 (and stable). On a VM with two iavf(4) interfaces, if we send > heavy > >>> traffic to iavf1 and try to kldunload the driver, the kldunload process > >>> hangs on iavf0 until iavf1 stops receiving traffic. > >>> > >>> After some debugging, it looks like epoch_drain_callbacks() [via > >>> if_detach_internal()] tries to switch CPUs to run on one that iavf1 is > >>> using for RX processing, but since iavf1 is busy, it can't make the > >> switch, > >>> so cpu_switch() just hangs and nothing happens until iavf1's RX thread > >>> stops being busy. > >>> > >>> I can work around this by inserting a kern_yield(PRI_USER) somewhere in > >> one > >>> of the iavf txrx functions that iflib calls into (e.g. > >>> iavf_isc_rxd_available), but that's not a proper fix. Does anyone know > >> what > >>> to do to prevent this from happening? > >>> > >>> Wildly guessing, does maybe epoch_drain_callbacks() need a higher > >> priority > >>> than the PI_SOFT used in the group taskqueues used in iflib's RX > >> processing? > >>> > >> > >> Hi, > >> > >> Which scheduler is this? ULE or BSD? > >> > >> EPOCH(9) expects some level of round-robin scheduling on the same > >> priority level. Setting a higher priority on EPOCH(9) might cause epoch > >> to start spinning w/o letting the lower priority thread which holds the > >> EPOCH() section to finish. > >> > >> --HPS > >> > >> > > Hi Hans, > > > > kern.sched.name gives me "ULE" > > > > Hi Eric, > > epoch_drain_callbacks() depends on that epoch_call_task() gets execution > which is executed from a GTASKQUEUE at PI_SOFT. Also > epoch_drain_callbacks() runs at the priority of the calling thread, and > if this is lower than PI_SOFT, and a gtaskqueue is spinning heavily, > then that won't work. > > For a single CPU system you will be toast in this situation regardless > if there is no free time on a CPU for EPOCH(). > > In general if epoch_call_task() doesn't get execution time, you will > have a problem. > > Maybe add a flag to iflib which stops the grouptask's before detaching > the network interface? > > --HPS > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > Hi Hans, Maybe add a flag to iflib which stops the grouptask's before detaching > the network interface? That was something we considered, but it only seemed like that would work for kldunload. This would result in undesired behavior if you used "devctl detach iavf0" to just remove iavf0, and didn't expect that to affect iavf1. But, if stopping traffic on iavf1 in order to unload iavf0 is acceptable, then that might be a solution. - Eric