From owner-freebsd-mips@FreeBSD.ORG Mon Aug 27 15:24:25 2012 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 87C721065674 for ; Mon, 27 Aug 2012 15:24:25 +0000 (UTC) (envelope-from c.jayachandran@gmail.com) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by mx1.freebsd.org (Postfix) with ESMTP id 14BAC8FC1F for ; Mon, 27 Aug 2012 15:24:24 +0000 (UTC) Received: by wicr5 with SMTP id r5so2747495wic.13 for ; Mon, 27 Aug 2012 08:24:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2CRqMP01W2tz4cd+r3ibZ/i6BWeDoplZc3mxoCt4u8Q=; b=JVmO7rsaBNXEfExILtthhar+wPQvHzZBRFg72u4Rywz4PFXdCnrF/i3zb2kDXpmOiZ m5x26FPS6G2wcON0YFwLwPy57RuM44ZiH1YEM78YT79oF2LcDRUZLLfOgke6uquzmpLK 3mY6Xq6yh7Q2xPUeSuJ6MmAFsBv/ZAofRr3IvwjPS/KAVAeHXHUwH0HuVaDfcDadA9aj qaeGLWbu2/PNBm1DLoMiWuPRCKEVb7xWR9jo5kk8qm55WcEavAoP25hKyo3iMKn/Y/0R IG6GVSL04ickV2cl79SIzFxVLOAAFgUVnpUZ3EA3SUq04IXP8LGTMYw4sKCQ63D6iDcV ZSqw== MIME-Version: 1.0 Received: by 10.216.139.196 with SMTP id c46mr3464921wej.220.1346081063556; Mon, 27 Aug 2012 08:24:23 -0700 (PDT) Received: by 10.216.115.3 with HTTP; Mon, 27 Aug 2012 08:24:23 -0700 (PDT) In-Reply-To: <50325DC3.3090201@rice.edu> References: <50228F5C.1000408@rice.edu> <50269AD4.9050804@rice.edu> <5029635A.4050209@rice.edu> <502D2271.6080105@rice.edu> <50325DC3.3090201@rice.edu> Date: Mon, 27 Aug 2012 20:54:23 +0530 Message-ID: From: "Jayachandran C." To: Alan Cox Content-Type: text/plain; charset=ISO-8859-1 Cc: mips@freebsd.org Subject: Re: mips pmap patch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Aug 2012 15:24:25 -0000 On Mon, Aug 20, 2012 at 9:24 PM, Alan Cox wrote: > On 08/20/2012 05:36, Jayachandran C. wrote: >> >> On Thu, Aug 16, 2012 at 10:10 PM, Alan Cox wrote: >>> >>> On 08/15/2012 17:21, Jayachandran C. wrote: >>>> >>>> On Tue, Aug 14, 2012 at 1:58 AM, Alan Cox wrote: >>>>> >>>>> On 08/13/2012 11:37, Jayachandran C. wrote: >> >> [...] >>>>>> >>>>>> I could not test for more than an hour on 32-bit due to another >>>>>> problem (freelist 1 containing direct-mapped pages runs out of pages >>>>>> after about an hour of compile test). This issue has been there for a >>>>>> long time, I am planning to look at it when I get a chance. >>>>>> >>>>> What exactly happens? panic? deadlock? >>>> >>>> The build slows down to a crawl and hangs when it runs out of pages in >>>> the freelist. >>> >>> >>> I'd like to see the output of "sysctl vm.phys_segs" and "sysctl >>> vm.phys_free" from this machine. Even better would be running "sysctl >>> vm.phys_free" every 60 seconds during the buildworld. Finally, I'd like >>> to >>> know whether or not either "ps" or "top" shows any threads blocked on the >>> "swwrt" wait channel once things slow to a crawl. >> >> I spent some time looking at this issue. I use a very large kernel >> image with built-in root filesystem, and this takes about 120 MB out >> of the direct mapped area. The remaining pages (~64 MB) are not enough >> for the build process. If I increase free memory in this area either >> by reducing the rootfs size of by adding a few more memory segments to >> this area, the build goes through fine. > > > I'm still curious to see what "sysctl vm.phys_segs" says. It sounds like > roughly half of the direct map region is going to DRAM and half to > memory-mapped I/O devices. Is that correct? Yes, about half of the direct mapped region in 32-bit is taken by flash, PCIe and other memory mapped IO. I also made the problem even worse by not reclaiming some bootloader areas in the direct mapped region, which reduced the available direct mapped memory. Here's the output of sysctls: root@testboard:/root # sysctl vm.phys_segs vm.phys_segs: SEGMENT 0: start: 0x887e000 end: 0xc000000 domain: 0 free list: 0x887a407c SEGMENT 1: start: 0x1d000000 end: 0x1fc00000 domain: 0 free list: 0x887a407c SEGMENT 2: start: 0x20000000 end: 0xbc0b3000 domain: 0 free list: 0x887a3f38 SEGMENT 3: start: 0xe0000000 end: 0xfffff000 domain: 0 free list: 0x887a3f38 root@testboard:/root # sysctl vm.phys_free vm.phys_free: FREE LIST 0: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 | POOL 2 -- -- -- -- -- -- -- -- 8 ( 1024K) | 2877 | 0 | 0 7 ( 512K) | 0 | 1 | 0 6 ( 256K) | 1 | 0 | 0 5 ( 128K) | 0 | 1 | 0 4 ( 64K) | 0 | 1 | 0 3 ( 32K) | 0 | 1 | 0 2 ( 16K) | 0 | 1 | 0 1 ( 8K) | 0 | 0 | 0 0 ( 4K) | 0 | 0 | 0 FREE LIST 1: ORDER (SIZE) | NUMBER | POOL 0 | POOL 1 | POOL 2 -- -- -- -- -- -- -- -- 8 ( 1024K) | 66 | 0 | 0 7 ( 512K) | 1 | 1 | 0 6 ( 256K) | 0 | 0 | 0 5 ( 128K) | 0 | 0 | 0 4 ( 64K) | 0 | 1 | 0 3 ( 32K) | 0 | 0 | 0 2 ( 16K) | 0 | 0 | 0 1 ( 8K) | 1 | 1 | 0 0 ( 4K) | 0 | 1 | 0 >> I also found that when the build slows down, most of the pages taken >> from freelist 1 are allocated by the UMA subsystem, which seems to >> keep quite a few pages allocated. > > > At worst, it may be necessary to disable the use of uma_small_alloc() for > this machine configuration. At best, uma_small_alloc() could be revised > opportunistically use pages in the direct map region, but have the ability > to fall back to pages that have to be mapped. I think this probably is not a bug, but a configuration problem (we cannot have such a huge built-in root filesystem when the direct mapped area is at this low). Anyway, I have checked in code to recover more areas from the bootloader, and this mostly solves the issue for me. The above output is taken before the check-in. Regards, JC.