From owner-freebsd-questions@freebsd.org Sun Oct 9 07:13:50 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24A60C0517E for ; Sun, 9 Oct 2016 07:13:50 +0000 (UTC) (envelope-from michaelsprivate@gmail.com) Received: from mail-yw0-x231.google.com (mail-yw0-x231.google.com [IPv6:2607:f8b0:4002:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CBC272E3 for ; Sun, 9 Oct 2016 07:13:49 +0000 (UTC) (envelope-from michaelsprivate@gmail.com) Received: by mail-yw0-x231.google.com with SMTP id u124so54545464ywg.3 for ; Sun, 09 Oct 2016 00:13:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=uSimNQU1e4EGIgYz/kslw7M9SqfWTb3CPnnqRX8uRbQ=; b=N40i58Gp0zRu49W0nTNeZKhI40QZdzstDgBi+/EF/j0WVHvLWATlecY7ulJZrR2p7g zHzOsmWO4AuC4hsxJIQegPIDz1ZbdC7oKh2p/LLXhFqb72T0pa0ZqXjw0/MwCevZ1MCp jzAGubUafD+vpU/gUmO3kbdOl7x7dFEYGq8/0cWZ2W6Q/Ojkrg6jz4AZdSLEwGmkEWlD IuHkoeFftkkMsy8X8fLUW8LUlFMmSIHKr9C2pXeB0Rhoy7utv+qqi2I1EncZbd0GJvRX qFxaMoAyW7PT7wb/mH17w7pdTQxNFMeb27LKXrqRELzA938U6z8wFI7pYY47dPwaSRv+ EP5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=uSimNQU1e4EGIgYz/kslw7M9SqfWTb3CPnnqRX8uRbQ=; b=eXKYuIO6m7WV7qSx2CyndJ+eef8arGKlYIQK0u/d6Yy+2n23ml08RDmXXSX1umXfr0 O/ZGuBbXXbdqamvnHlDePzbhzR/8eKqVgLzgbZIt5xT7VzITj578SCRUm3hMRtKgArER ipINgKC1sSuA60SaLYHda+EEkV0QjVLrknWJQu5FPiTANqFHfQLQpoM3e/w2XFySDRb+ URKoXn98tk1CVnhhdDTMwifMY6o5hK8HD0oBydHi8TIOFPuMXafF1tFMXLHEF6JvBj8L DsJ6Tw+/RC7llH542P95DZJ4CYKJw1nQoukuzFrytWI95XJWJM9tcPz8eBMt1I8At/00 2U9A== X-Gm-Message-State: AA6/9RkiJEL3wpcKRHVuHAGMBf0P6uwBTwmaQOkcGw1DrSkHS/KaFspFGL18DrmTCyIvxGwo4re/nwfg0E3tTA== X-Received: by 10.129.55.205 with SMTP id e196mr24034330ywa.344.1475997229103; Sun, 09 Oct 2016 00:13:49 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.6.85 with HTTP; Sun, 9 Oct 2016 00:13:28 -0700 (PDT) In-Reply-To: <9DB5B54C-2AE4-4200-AB78-F86913E1E433@mail.sermon-archive.info> References: <1CD13C1C-5344-4909-A061-F25FBB86AFF9@lafn.org> <9DB5B54C-2AE4-4200-AB78-F86913E1E433@mail.sermon-archive.info> From: Michael Schuster Date: Sun, 9 Oct 2016 09:13:28 +0200 Message-ID: Subject: Re: Swap exhaustion To: Doug Hardie Cc: FreeBSD - Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Oct 2016 07:13:50 -0000 On Sun, Oct 9, 2016 at 3:33 AM, Doug Hardie wrote: > I have been trying periodically to resolve this issue. I modified the > application to log (syslog) every memory location it allocates. > Interestingly enough they are all around 0x100F380. However, the segments > that are increasing in number all have the highest order address bit set, > i.e., 0x83ac00000. These are shown by procstat as type "df". The number > of these increases with time. The sizes vary from what appears to be small > to over 5000 resident pages. They never seem to go away unless I restart > the process. Right after a restart there are 7 "df"s. However, with time > there can eventually be hundreds. Since these do not have any file backing > them, they eat up swap space and thus cause the system to run out of swap > and start killing processes. > > What things allocate memory at the top of the address space? The > application mallocs and mmaps all allocate much lower in the address space. > Not a direct answer, but a few thoughts: 1) think about disabling memory overcommit (temporarily) ( https://wiki.freebsd.org/SystemTuning). Perhaps this can shed some light on the culprit. 2) this is something I'd use dtrace to analyse - as a start, I'd watch everything remotely malloc()-ish (both syscall and libc) and trigger on return values you're interested in. If that doesn't help, loosen the restrictions on your tracing until something interesting shows up... HTH Michael -- Michael Schuster http://recursiveramblings.wordpress.com/ recursion, n: see 'recursion'