From owner-freebsd-current@freebsd.org Tue Sep 12 03:35:10 2017 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB3ACE13B90 for ; Tue, 12 Sep 2017 03:35:10 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt0-x243.google.com (mail-qt0-x243.google.com [IPv6:2607:f8b0:400d:c0d::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9EB6D82FF1; Tue, 12 Sep 2017 03:35:10 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qt0-x243.google.com with SMTP id q8so6702504qtb.1; Mon, 11 Sep 2017 20:35:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=WFbdaYnXAoN5zcs0WPhMFDP3itZlaERM9hWE+c+hyXg=; b=AnhZnrXmpui6wOtZdzfTUPHhiAeGOPHMbwR17WnXl43q8AlXWp4amciP9/MzFhhk98 06W0FKth/JF3ld9EU0LqL00hgyfXUrqJTePxQ13fADInovSp/fIg8F9DKGoKYVPCHSC5 cdfcSq2wB/BDUnmuERz8X3WTVuRlbh1cHzp5iByfiUDRZLix9Kt+TBWK9ETgjdK0YguG 0HKPcL2ObbSMZ5yd0kZvrg/V0n1krZ6IEyMdkgkll6AGctz6vjM7DOGdJajrl7pAv3V0 5RqPlUu4qf5tQBR5RwDT3n8C6AdOWGKau9e4PSgNCpErny/45aMASxeQpCmS3Cef0f7O ArFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=WFbdaYnXAoN5zcs0WPhMFDP3itZlaERM9hWE+c+hyXg=; b=WIYyG1T3Af0wPfCYem/mxgNu/Y8BEpvcuNCaABffFg96Be8c9tM7+XV6HaJhWJCoZ3 OXZjnYqln+giK7KpKsSgJc3dOpAScynOw5boUBJI5BxyOCanUijwVAjVBScwP4mhrniF jDS/Z6C99JMwGd3szIVkrcFU2C7c0WbPlZokfth6KAJO5WOro52ZG6QMH8EwoO1QXQM5 p5o9j8U66k9kKKoO2YqiGVit1qqzfZoA9N1mGYE09jeSV0G0vzLhucJl1E/v6rVEOTKn gdPzddbQG1SKcK4vIhYQhOP2mXkqAnpElNt0y8V8gSe6MgG82imxAJfcDOERHrzPic43 KzXA== X-Gm-Message-State: AHPjjUjFZHYjlSLBI5lnZP0Hyc5SI9Pom6eStpUwhQlivdN03x5yvbvt s6Vf4UZjy6b3wgkr X-Google-Smtp-Source: AOwi7QBAh8x8d2du1TR7kpA7gJw+X709REN66qUX5jR4RaewYLdbfLzRWHbLmwY4qA2JDDgU8LcpeQ== X-Received: by 10.200.43.123 with SMTP id 56mr18412681qtv.15.1505187309599; Mon, 11 Sep 2017 20:35:09 -0700 (PDT) Received: from wkstn-mjohnston.west.isilon.com (c-76-104-203-143.hsd1.wa.comcast.net. [76.104.203.143]) by smtp.gmail.com with ESMTPSA id b77sm7012137qkg.26.2017.09.11.20.35.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 11 Sep 2017 20:35:08 -0700 (PDT) Sender: Mark Johnston Date: Mon, 11 Sep 2017 20:35:23 -0700 From: Mark Johnston To: "Andrey V. Elsukov" Cc: Raphael Kubo da Costa , freebsd-current@FreeBSD.org, cem@FreeBSD.org Subject: Re: r323412: Panic on boot (slab->us_keg == keg) Message-ID: <20170912033523.GA89424@wkstn-mjohnston.west.isilon.com> References: <86vakp1vsq.fsf@orwell> <8ccad12b-3d31-6478-7884-c003bc314990@yandex.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8ccad12b-3d31-6478-7884-c003bc314990@yandex.ru> User-Agent: Mutt/1.8.3 (2017-05-23) X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2017 03:35:11 -0000 On Mon, Sep 11, 2017 at 09:15:51PM +0300, Andrey V. Elsukov wrote: > On 11.09.2017 15:23, Andrey V. Elsukov wrote: > > --- trap 0xc, rip = 0xffffffff80d84870, rsp = 0xffffffff82193970, rbp = > > 0xffffffff821939b0 --- > > zone_import() at zone_import+0x110/frame 0xffffffff821939b0 > > zone_alloc_item() at zone_alloc_item+0x36/frame 0xffffffff821939f0 > > uma_startup() at uma_startup+0x1d0/frame 0xffffffff82193ae0 > > vm_page_startup() at vm_page_startup+0x34e/frame 0xffffffff82193b30 > > vm_mem_init() at vm_mem_init+0x1a/frame 0xffffffff82193b50 > > mi_startup() at mi_startup+0x9c/frame 0xffffffff82193b70 > > btext() at btext+0x2c > > Uptime: 1s > > I bisected revisions, and the last working is r322988. > This machine is E5-2660 v4@ based. > > [...] > FreeBSD/SMP: 2 package(s) x 14 core(s) x 2 hardware threads > > Also I determined that it can successfully boot with disabled > hyper-threading. After the change to CACHE_LINE_SIZE, we have sizeof(struct uma_zone) == 448 and sizeof(struct uma_cache) == 64. With 56 CPUs, we therefore need 4032 bytes per UMA zone, plus 80 bytes for the slab header - "internal" zones always keep the slab header in the slab itself. That's slightly larger than one page, but the UMA zone zone's keg will have uk_ppera == 1. So, when allocating slabzone, keg_alloc_slab() will call startup_alloc(uk_ppera * PAGE_SIZE), which will allocate 4096 bytes for a structure that is 4032 + 80 = 4112 bytes in size. I think the bug is that keg_large_init() doesn't take sizeof(struct uma_slab) into account when setting uk_ppera for the keg. In particular, the bug isn't specific to the bootup process; it only affects internal zones with an item size in the range [4016, 4096]. The patch below should fix this - could you give it a try? diff --git a/sys/vm/uma_core.c b/sys/vm/uma_core.c index 44c91e66769a..48daeb18f9c3 100644 --- a/sys/vm/uma_core.c +++ b/sys/vm/uma_core.c @@ -1306,10 +1306,6 @@ keg_large_init(uma_keg_t keg) keg->uk_ipers = 1; keg->uk_rsize = keg->uk_size; - /* We can't do OFFPAGE if we're internal, bail out here. */ - if (keg->uk_flags & UMA_ZFLAG_INTERNAL) - return; - /* Check whether we have enough space to not do OFFPAGE. */ if ((keg->uk_flags & UMA_ZONE_OFFPAGE) == 0) { shsize = sizeof(struct uma_slab); @@ -1317,8 +1313,17 @@ keg_large_init(uma_keg_t keg) shsize = (shsize & ~UMA_ALIGN_PTR) + (UMA_ALIGN_PTR + 1); - if ((PAGE_SIZE * keg->uk_ppera) - keg->uk_rsize < shsize) - keg->uk_flags |= UMA_ZONE_OFFPAGE; + if ((PAGE_SIZE * keg->uk_ppera) - keg->uk_rsize < shsize) { + /* + * We can't do offpage if we're internal, in which case + * we need an extra page per allocation to contain the + * slab header. + */ + if ((keg->uk_flags & UMA_ZFLAG_INTERNAL) == 0) + keg->uk_flags |= UMA_ZONE_OFFPAGE; + else + keg->uk_ppera++; + } } if ((keg->uk_flags & UMA_ZONE_OFFPAGE) && diff --git a/sys/vm/vm_page.c b/sys/vm/vm_page.c index ee7b93bbd719..477a816b0bd2 100644 --- a/sys/vm/vm_page.c +++ b/sys/vm/vm_page.c @@ -475,7 +475,8 @@ vm_page_startup(vm_offset_t vaddr) * in proportion to the zone structure size. */ pages_per_zone = howmany(sizeof(struct uma_zone) + - sizeof(struct uma_cache) * (mp_maxid + 1), UMA_SLAB_SIZE); + sizeof(struct uma_slab) + sizeof(struct uma_cache) * (mp_maxid + 1), + UMA_SLAB_SIZE); if (pages_per_zone > 1) { /* Reserve more pages so that we don't run out. */ boot_pages = UMA_BOOT_PAGES_ZONES * pages_per_zone;