From owner-freebsd-net@freebsd.org Mon Mar 30 20:19:28 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C8EAA26659B for ; Mon, 30 Mar 2020 20:19:28 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48rkMD19bMz3Lgm for ; Mon, 30 Mar 2020 20:19:19 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by mail-lj1-f170.google.com with SMTP id t17so19600114ljc.12 for ; Mon, 30 Mar 2020 13:19:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4zxw9LFdynN0mR9maJOFwnEgppelG9WVVDX0hLLj5sw=; b=Nxewxk7AOGzlsdWvy6zgowSeYPDKKCoJ4/o5BRIgV9iCO103hxx4g01V1jkH0EoyLb 62cbMBr36RDQ/ETLEd6ZQ4NZayZgmsT4ifmC5yQ2pRT5Mmp5H+tC2W/Vbno3tV5HDUyv pGi0CGkGZLry7ljd8q+tAytCQ14R+UgfUnteqarHRp5/AtCoHGOlLXjTu/geaZOaFs9o Tlhb0Xa5onJ7Pi47iNC4Qv1/DWslyqz6KNWDHZHQF1zRG1BWzEPrv0oh9k2QYyJajgfo 4vKvZxoTCZ6n0dN57qdcAPRYgGWi4aguAtw+3EGiIx89X9l3ywM9ncOFk/ibEMYCPkDT TQzw== X-Gm-Message-State: AGi0PuZRtW9em6d9KZ4e/ty879gxy386L6pbrN04wnxVaJVcFwOkMiLy I5W2tIy0NzPQIrwkciCAPaC6xWpd X-Google-Smtp-Source: APiQypLklY5T+g1i1aXmxNEmjjGSvDzufUK9Jxg3Ov1icztj12O0qPN3v7sqwaR984Whudnfj3xIFw== X-Received: by 2002:a05:651c:2c7:: with SMTP id f7mr8172822ljo.152.1585599546327; Mon, 30 Mar 2020 13:19:06 -0700 (PDT) Received: from mail-lj1-f173.google.com (mail-lj1-f173.google.com. [209.85.208.173]) by smtp.gmail.com with ESMTPSA id v19sm9540940lfg.9.2020.03.30.13.19.04 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 30 Mar 2020 13:19:05 -0700 (PDT) Received: by mail-lj1-f173.google.com with SMTP id t17so19599974ljc.12 for ; Mon, 30 Mar 2020 13:19:04 -0700 (PDT) X-Received: by 2002:a2e:7805:: with SMTP id t5mr8421224ljc.144.1585599544232; Mon, 30 Mar 2020 13:19:04 -0700 (PDT) MIME-Version: 1.0 References: <0e2e97f2-df75-3c6f-9bdd-e8c2ab7bf79e@selasky.org> <20200130030911.GA15281@spy> <20200212222219.GE83892@raichu> <20200328225150.GA82767@raichu> In-Reply-To: <20200328225150.GA82767@raichu> From: Eric Joyner Date: Mon, 30 Mar 2020 13:18:52 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Issue with epoch_drain_callbacks and unloading iavf(4) [using iflib] To: Mark Johnston Cc: freebsd-net@freebsd.org, Hans Petter Selasky , John Baldwin , shurd , Drew Gallatin , Gleb Smirnoff X-Rspamd-Queue-Id: 48rkMD19bMz3Lgm X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of ricera10@gmail.com designates 209.85.208.170 as permitted sender) smtp.mailfrom=ricera10@gmail.com X-Spamd-Result: default: False [-2.32 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_THREE(0.00)[4]; MIME_TRACE(0.00)[0:+,1:+,2:~]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[170.208.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.32)[ip: (-0.40), ipnet: 209.85.128.0/17(-0.71), asn: 15169(-0.45), country: US(-0.05)]; FORGED_SENDER(0.30)[erj@freebsd.org,ricera10@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[170.208.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[erj@freebsd.org,ricera10@gmail.com]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Mar 2020 20:19:29 -0000 On Sat, Mar 28, 2020 at 3:52 PM Mark Johnston wrote: > On Wed, Mar 11, 2020 at 04:32:40PM -0700, Eric Joyner wrote: > > Mark, > > > > I did get some time to get back and retry this; however your second patch > > still doesn't solve the problem. Looking into it a bit, it looks like the > > kldunload process isn't hitting the code you've changed; it's hanging in > > epoch_wait_preempt() in if_detach_internal(), which is immediately before > > epoch_drain_callbacks(). > > > > I did a kernel dump while it was hanging, and this is the backtrace for > the > > kldunload process: > > I see. I think the callback can be made much simpler and avoid the > problematic sched_bind() calls. I wrote a patch that allows waiting > threads to lend scheduling priority to a preempted thread blocked in an > epoch section, based on some code I wrote to implement preemptible SMR > sections. If waiting for a running thread, the callback just spins. > > This might be enough to solve your problem, I posted the two lightly > tested patches here: > https://reviews.freebsd.org/D24214 > https://reviews.freebsd.org/D24215 > > If we hit a situation where a reader is preempted and then its CPU is > hogged by a high-priority kernel thread, this still won't be enough, but > I suspect it'll solve your case. Would you be able to test? > Yeah, I'll try them out. - Eric