From owner-freebsd-current@FreeBSD.ORG Fri May 22 18:38:23 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BF872E8B for ; Fri, 22 May 2015 18:38:23 +0000 (UTC) Received: from mail-la0-f42.google.com (mail-la0-f42.google.com [209.85.215.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E6901BAA for ; Fri, 22 May 2015 18:38:22 +0000 (UTC) Received: by lami4 with SMTP id i4so18368581lam.0 for ; Fri, 22 May 2015 11:38:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to:cc :content-type; bh=t0EQHg+sL8fSWRSdaGl4ApVlGvf/5Up2Nxtsn00eDko=; b=R51G35cQPly2g2lgT17x8pYLgF//tgMpyUw53bZ39q9wzkU9t7Fo+X3JVPt4ADIZkY RnW3wsvDaERrk4cf7m6Sl2mmOacoPfvMmuM3fXGvZnusUxepFmB3NQPhwpCe8iogAvJd QCd/fpZ/MIGNR9OyjMJxsuY9GW5bEbEmwPlUwO2GCGnvuOABOTVV4upeL5J/P7aXpjNB 8vm4gjbnkD8Xbq/itEsngV9hK3waSWtNrUbmgbWIsBgZw1d5+4ZDZ35LA1QJisf2frAM a/q64RxN7Z09ua0IhH8ZTac/7raZUqUCWIXIMYHSDE2/TKMPA4WfnQ7l31qjAMJhpVPq kPUA== X-Gm-Message-State: ALoCoQlcJ93quCoC8Egl5GGq53HJZV81WUE74XUsTDmW4MsS5zjPt/UDcC1cyKhoyEyjmtcbBYIY MIME-Version: 1.0 X-Received: by 10.152.203.233 with SMTP id kt9mr7479135lac.21.1432319895279; Fri, 22 May 2015 11:38:15 -0700 (PDT) Received: by 10.25.201.72 with HTTP; Fri, 22 May 2015 11:38:15 -0700 (PDT) Date: Fri, 22 May 2015 20:38:15 +0200 Message-ID: Subject: Re: Fwd: UMA initialization failure with 48 core ARM64 From: =?UTF-8?Q?Micha=C5=82_Stanek?= To: Konstantin Belousov Cc: freebsd-current@freebsd.org, freebsd-arm@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 18:38:23 -0000 2015-05-22 16:36 GMT+02:00 Konstantin Belousov : > On Fri, May 22, 2015 at 04:14:13PM +0200, Micha?? Stanek wrote: > > Success! I am finally able to boot 48 cores. I have been trying out > > different values given to uma_prealloc() and uma_zone_reserve(). It > started > > working when I increased the parameter in uma_prealloc() 32 times and > left > > uma_zone_reserve() as it was originally. UMA_BOOT_PAGES also needs to be > > set to 512. > > > > Thank you very much for your help. Do you know how the value to > > uma_prealloc() should scale with the number of CPUs? If it is not > obvious, > > then maybe for now we should make a #define with a value to multiply > > BT_MAXALLOC by, with a comment that a higher number is required on > > platforms with many CPUs. What do you think the final fix should look > like? > > I suspect it is not only the number of CPUs which makes the play. Note > that the number of tags is already scaled with the number of CPUs. > > It is also the question of how much the given architecture needs to > allocate > before the normal uma/vmem mechanisms start working. My quess is that > arm64 > performes more kva_alloc()s on early stages than other architectures. > I am forwarding the result of my conversation with Konstantin Belousov. With his help I was able to boot 48 cores on an arm64 platform. I needed to set UMA_BOOT_PAGES=512 and increase the parameter given to uma_prealloc() in vmem_startup() 32 times (giving 32 * BT_MAXALLOC). It looks like this should be made configurable to avoid running out of space for initial allocations on some platforms. In our case, the panic happened still in SI_SUB_VM sysinit. Best regards, Michal Stanek