From owner-dev-commits-src-main@freebsd.org Tue Jan 12 06:43:34 2021 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AB6CF4CB619; Tue, 12 Jan 2021 06:43:34 +0000 (UTC) (envelope-from rlibby@gmail.com) Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DFLd16wRjz4VQY; Tue, 12 Jan 2021 06:43:33 +0000 (UTC) (envelope-from rlibby@gmail.com) Received: by mail-qt1-f182.google.com with SMTP id b9so970089qtr.2; Mon, 11 Jan 2021 22:43:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ViXucKVoLwRbdgSLgkHvg32u3OGnBkzMWxGr1XLRrYA=; b=mrCKR27Gf9e8OCf48LX1z/TcylQw330qBNvRb0V7UKN0HvghG8q0eMLKEXuEx9aqyN DU5RkoHds36Xrq7j4fCL7Icd3AfcT0VEaNwcOEeCr39cnWufp31Y2MzmrmRR8DJkceu0 9WO5HNw/Stt4/Dbbo2TIVyv5tLOUU423WAnTCjcKSPhV2b0MvlG5CSQ4rupgdNQQ3ksZ zCNFcVb2OEaOLoRnicBHOhicAAPHacj8StNQ3it8XlCS6Sv0dDiMh3+HOeoeyqkfq8xx zds80NqUm4hsuI5OXbwzYZ0wt6RcbYbYKZFNan8pHcwxA/sAYIebDrx7BiSU3rvywszG Xt+w== X-Gm-Message-State: AOAM532j+0dQoaMT1bkDRLObf/Qah+zluFJyyrFO0DR99mUNUkg7VUZA pxOAYxIUfv0hjfUKonnHYIzwTOO1sTbHiA== X-Google-Smtp-Source: ABdhPJxpMV4sSomvSgb/JNGuZKIOb/BSG49tUJLkbZJddM/dHdjXidvOF4YgFNs9JNLvp1t7lokn+g== X-Received: by 2002:aed:2ea1:: with SMTP id k30mr3260481qtd.32.1610433812565; Mon, 11 Jan 2021 22:43:32 -0800 (PST) Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com. [209.85.219.42]) by smtp.gmail.com with ESMTPSA id a5sm786525qtn.57.2021.01.11.22.43.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 11 Jan 2021 22:43:32 -0800 (PST) Received: by mail-qv1-f42.google.com with SMTP id bd6so491677qvb.9; Mon, 11 Jan 2021 22:43:32 -0800 (PST) X-Received: by 2002:a0c:e383:: with SMTP id a3mr2912280qvl.45.1610433811832; Mon, 11 Jan 2021 22:43:31 -0800 (PST) MIME-Version: 1.0 References: <202101120110.10C1A6mg069684@gitrepo.freebsd.org> In-Reply-To: From: Ryan Libby Date: Mon, 11 Jan 2021 22:43:20 -0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: git: 7eaea04a5bb1 - main - amd64: compare TLB shootdown target to all_cpus To: Mateusz Guzik , Andrew Gallatin Cc: src-committers , dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4DFLd16wRjz4VQY X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rlibby@gmail.com designates 209.85.160.182 as permitted sender) smtp.mailfrom=rlibby@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RWL_MAILSPIKE_GOOD(0.00)[209.85.160.182:from]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[4]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FORGED_SENDER(0.30)[rlibby@freebsd.org,rlibby@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[209.85.160.182:from]; R_DKIM_NA(0.00)[]; FROM_NEQ_ENVFROM(0.00)[rlibby@freebsd.org,rlibby@gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[rlibby]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; SPAMHAUS_ZRD(0.00)[209.85.160.182:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.160.182:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[dev-commits-src-all,dev-commits-src-main] X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for the main branch of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2021 06:43:34 -0000 On Mon, Jan 11, 2021 at 10:22 PM Ryan Libby wrote: > > On Mon, Jan 11, 2021 at 9:59 PM Mateusz Guzik wrote: > > > > This makes my 2 core vm crash on boot: > > There's some known broken logic in bitset(9) for comparisons and some > other operations. I started some tests and a patch to fix it and clean > up some stuff, but I haven't polished it off: > > https://github.com/rlibby/freebsd/commits/bitset > > Long story short, it isn't careful not to examine bits in the last word > that are past the end of the set. > Ah, except MAXCPU is a multiple of the word size on amd64, so this shouldn't be the issue. Sorry for the noise. > > > > Launching APs: 1 > > Timecounter "TSC-low" frequency 1346899854 Hz quality 1000 > > panic: IPI scoreboard is zero, initiator 1 target 1 > > cpuid = 1 > > time = 1 > > KDB: stack backtrace: > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffffff80fb7600 > > vpanic() at vpanic+0x181/frame 0xffffffff80fb7650 > > panic() at panic+0x43/frame 0xffffffff80fb76b0 > > smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x434/frame > > 0xffffffff80fb7730 > > pmap_invalidate_range() at pmap_invalidate_range+0x16f/frame 0xffffffff80fb77a0 > > vm_thread_stack_create() at vm_thread_stack_create+0x45/frame 0xffffffff80fb78e0 > > kstack_import() at kstack_import+0x4e/frame 0xffffffff80fb7910 > > cache_alloc() at cache_alloc+0x3bf/frame 0xffffffff80fb7970 > > cache_alloc_retry() at cache_alloc_retry+0x25/frame 0xffffffff80fb79b0 > > vm_thread_new() at vm_thread_new+0x3d/frame 0xffffffff80fb79d0 > > thread_alloc() at thread_alloc+0x50/frame 0xffffffff80fb7a00 > > kthread_add() at kthread_add+0x42/frame 0xffffffff80fb7aa0 > > kproc_kthread_add() at kproc_kthread_add+0xaf/frame 0xffffffff80fb7bc0 > > intr_event_add_handler() at intr_event_add_handler+0x227/frame > > 0xffffffff80fb7c30 > > start_softintr() at start_softintr+0xea/frame 0xffffffff80fb7c60 > > mi_startup() at mi_startup+0xec/frame 0xffffffff80fb7cb0 > > btext() at btext+0x2c > > KDB: enter: panic > > > > > > On 1/12/21, Andrew Gallatin wrote: > > > The branch main has been updated by gallatin: > > > > > > URL: > > > https://cgit.FreeBSD.org/src/commit/?id=7eaea04a5bb1dc86c43ce046311e1c1a042994d3 > > > > > > commit 7eaea04a5bb1dc86c43ce046311e1c1a042994d3 > > > Author: Andrew Gallatin > > > AuthorDate: 2021-01-12 01:03:37 +0000 > > > Commit: Andrew Gallatin > > > CommitDate: 2021-01-12 01:09:32 +0000 > > > > > > amd64: compare TLB shootdown target to all_cpus > > > > > > On amd64, the pmap code passes all_cpus to > > > smp_targeted_tlb_shootdown() when unmapping from the > > > kernel pmap. This function has an optimized path to send IPIs > > > to all but itself, which it intends to do when the target > > > is all cpus. However, we need to compare the target cpu mask > > > with all_cpus, rather than using CPU_ISFULLSET(). Comparing with > > > CPU_ISFULLSET() will only work when we have MAXCPU cpus active in > > > the system, otherwise, we'll be sending repeated IPIs, rather than > > > a single IPI to all CPUs but ourself. > > > > > > Fixing this should reduce the time spent in native_lapic_ipi_wait() > > > as we will be sending ipis in parallel, rather than one-by-one. > > > This is confirmed by dtrace. > > > > > > Reviewed by: alc, jhb, kib, markj > > > Sponsored by: Netflix > > > Differential Revision: https://reviews.freebsd.org/D28102 > > > --- > > > sys/amd64/amd64/mp_machdep.c | 4 ++-- > > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > > > diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c > > > index 63777014e151..794a11bf1276 100644 > > > --- a/sys/amd64/amd64/mp_machdep.c > > > +++ b/sys/amd64/amd64/mp_machdep.c > > > @@ -673,7 +673,7 @@ smp_targeted_tlb_shootdown(cpuset_t mask, pmap_t pmap, > > > vm_offset_t addr1, > > > /* > > > * Check for other cpus. Return if none. > > > */ > > > - if (CPU_ISFULLSET(&mask)) { > > > + if (!CPU_CMP(&mask, &all_cpus)) { > > > if (mp_ncpus <= 1) > > > goto local_cb; > > > } else { > > > @@ -719,7 +719,7 @@ smp_targeted_tlb_shootdown(cpuset_t mask, pmap_t pmap, > > > vm_offset_t addr1, > > > * (zeroing slot) and reading from it below (wait for > > > * acknowledgment). > > > */ > > > - if (CPU_ISFULLSET(&mask)) { > > > + if (!CPU_CMP(&mask, &all_cpus)) { > > > ipi_all_but_self(IPI_INVLOP); > > > other_cpus = all_cpus; > > > CPU_CLR(PCPU_GET(cpuid), &other_cpus); > > > > > > > > > -- > > Mateusz Guzik