From owner-freebsd-arm@FreeBSD.ORG Fri May 22 19:05:44 2015 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ED39F8CE for ; Fri, 22 May 2015 19:05:44 +0000 (UTC) Received: from mail-la0-f50.google.com (mail-la0-f50.google.com [209.85.215.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 928A71ECC for ; Fri, 22 May 2015 19:05:44 +0000 (UTC) Received: by lagv1 with SMTP id v1so18701146lag.3 for ; Fri, 22 May 2015 12:05:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to:cc :content-type; bh=t0EQHg+sL8fSWRSdaGl4ApVlGvf/5Up2Nxtsn00eDko=; b=CcES9TGht3BKiZHoM1S6A89WRRG2sYf0aiNcp87v+jlKIkZZYyMG/iLPPDd+TyHzvt keb62/d5vSo6mUrq2U+WW7ykCd3TaB4UBQWuwi1R8lrj6jzRRNuWFIe5mFaYhmkgzeV0 9WkldchUkCfVoTCMTbP64xXY9xe2L1osdQNVOCq+XCHt5WEUElHgbOymDT1pM/JjYVnr F1386vxZjP5BJXzz23VzQBgfs0cTVRqNy8B0P4LlHCQPEbXXyb2+ixVi84+p0f4NAZeS pzztwVKLF1ykC9Wjl+TqEA+A+te3wuMPaan9Ut5Rfo2c3/DevkCMfT+OfPIjkNsRe9ps VR4w== X-Gm-Message-State: ALoCoQm2zpxzdBbiJLSOb+X9YVba50uLDjJ9rHHZcDZzGRn5EvCAMeklOLwDBPKk30TWpyEXvajj MIME-Version: 1.0 X-Received: by 10.152.203.233 with SMTP id kt9mr7479135lac.21.1432319895279; Fri, 22 May 2015 11:38:15 -0700 (PDT) Received: by 10.25.201.72 with HTTP; Fri, 22 May 2015 11:38:15 -0700 (PDT) Date: Fri, 22 May 2015 20:38:15 +0200 Message-ID: Subject: Re: Fwd: UMA initialization failure with 48 core ARM64 From: =?UTF-8?Q?Micha=C5=82_Stanek?= To: Konstantin Belousov Cc: freebsd-current@freebsd.org, freebsd-arm@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 May 2015 19:05:45 -0000 2015-05-22 16:36 GMT+02:00 Konstantin Belousov : > On Fri, May 22, 2015 at 04:14:13PM +0200, Micha?? Stanek wrote: > > Success! I am finally able to boot 48 cores. I have been trying out > > different values given to uma_prealloc() and uma_zone_reserve(). It > started > > working when I increased the parameter in uma_prealloc() 32 times and > left > > uma_zone_reserve() as it was originally. UMA_BOOT_PAGES also needs to be > > set to 512. > > > > Thank you very much for your help. Do you know how the value to > > uma_prealloc() should scale with the number of CPUs? If it is not > obvious, > > then maybe for now we should make a #define with a value to multiply > > BT_MAXALLOC by, with a comment that a higher number is required on > > platforms with many CPUs. What do you think the final fix should look > like? > > I suspect it is not only the number of CPUs which makes the play. Note > that the number of tags is already scaled with the number of CPUs. > > It is also the question of how much the given architecture needs to > allocate > before the normal uma/vmem mechanisms start working. My quess is that > arm64 > performes more kva_alloc()s on early stages than other architectures. > I am forwarding the result of my conversation with Konstantin Belousov. With his help I was able to boot 48 cores on an arm64 platform. I needed to set UMA_BOOT_PAGES=512 and increase the parameter given to uma_prealloc() in vmem_startup() 32 times (giving 32 * BT_MAXALLOC). It looks like this should be made configurable to avoid running out of space for initial allocations on some platforms. In our case, the panic happened still in SI_SUB_VM sysinit. Best regards, Michal Stanek