From owner-freebsd-performance@FreeBSD.ORG Tue Mar 30 11:36:49 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7820216A4CE for ; Tue, 30 Mar 2004 11:36:49 -0800 (PST) Received: from mail1.simplenet.com (mail1.simplenet.com [209.132.1.30]) by mx1.FreeBSD.org (Postfix) with ESMTP id 68C8543D2F for ; Tue, 30 Mar 2004 11:36:49 -0800 (PST) (envelope-from tt-list@simplenet.com) Received: from TraverPC.simplenet.com (64.170.12.242) by mail1.simplenet.com (7.0.016) (authenticated as tt-list@simplenet.com) id 4069459E00000F21 for freebsd-performance@freebsd.org; Tue, 30 Mar 2004 11:35:53 -0800 Message-Id: <6.0.1.1.0.20040330113631.01ef7ec0@mail1.simplenet.com> X-Sender: tt-list@simplenet.com@mail1.simplenet.com X-Mailer: QUALCOMM Windows Eudora Version 6.0.1.1 Date: Tue, 30 Mar 2004 11:36:48 -0800 To: freebsd-performance@freebsd.org From: Tim Traver Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: shmem release X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Mar 2004 19:36:49 -0000 Hi all, Ok, I am running a 4.7 FreeBSD box that is a web server running apache. It looks like some module that I have is leaking memory, and eventually, apache crashes on restarts becuase of this error : shmget() failed: No space left on device which means it can't get any more memory, which I understand. When I look at the top list, it shows me something like this : Mem: 140M Active, 879M Inact, 151M Wired, 181M Cache, 199M Buf, 660M Free But when you look at the processes that are still up, they hardly take up any memory. So, my question is this. Is there a way to free up Inactive memory from crashed processes ??? Without just rebooting the box ??? I know that I need to find the source of the leaking and crashing to begin with, but in the mean time, if it happens, I'd like to free up the memory manually, so I can get the box running again... Thanks, Tim. SimpleNet's Back ! http://www.simplenet.com From owner-freebsd-performance@FreeBSD.ORG Tue Mar 30 12:10:59 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6BB4A16A4CE for ; Tue, 30 Mar 2004 12:10:59 -0800 (PST) Received: from mail.trippynames.com (unknown [216.143.148.128]) by mx1.FreeBSD.org (Postfix) with ESMTP id 562F443D3F for ; Tue, 30 Mar 2004 12:10:59 -0800 (PST) (envelope-from sean@chittenden.org) Received: from localhost (localhost [127.0.0.1]) by mail.trippynames.com (Postfix) with ESMTP id 435D4A85A6; Tue, 30 Mar 2004 12:10:32 -0800 (PST) Received: from mail.trippynames.com ([127.0.0.1]) by localhost (rand.nxad.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 79236-10; Tue, 30 Mar 2004 12:10:29 -0800 (PST) Received: from [192.168.102.100] (dsl081-160-109.sea1.dsl.speakeasy.net [64.81.160.109]) by mail.trippynames.com (Postfix) with ESMTP id B46E0A7CA2; Tue, 30 Mar 2004 12:10:28 -0800 (PST) In-Reply-To: <6.0.1.1.0.20040330113631.01ef7ec0@mail1.simplenet.com> References: <6.0.1.1.0.20040330113631.01ef7ec0@mail1.simplenet.com> Mime-Version: 1.0 (Apple Message framework v613) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <5686BD90-8286-11D8-89F3-000A95C705DC@chittenden.org> Content-Transfer-Encoding: 7bit From: Sean Chittenden Date: Tue, 30 Mar 2004 12:10:49 -0800 To: Tim Traver X-Mailer: Apple Mail (2.613) cc: freebsd-performance@freebsd.org Subject: Re: shmem release X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Mar 2004 20:10:59 -0000 > Ok, I am running a 4.7 FreeBSD box that is a web server running apache. > > It looks like some module that I have is leaking memory, and > eventually, apache crashes on restarts becuase of this error : > > shmget() failed: No space left on device Look in your logs and see if there's an error message being logged that could help you identify what module it is. Otherwise, get a crash dump of a crashed process and run gdb on it to see if the module that's crashing is obvious from the back trace. > which means it can't get any more memory, which I understand. > > When I look at the top list, it shows me something like this : > > Mem: 140M Active, 879M Inact, 151M Wired, 181M Cache, 199M Buf, 660M > Free > > But when you look at the processes that are still up, they hardly take > up any memory. > > So, my question is this. > > Is there a way to free up Inactive memory from crashed processes ??? You don't need to worry about that. http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/admin.html#TOP- MEMORY-STATES There's a better article that I remember reading that went into great detail explaining this, but I can't seem to dig it up. > Without just rebooting the box ??? Inactive, buffered, cached, and free memory is memory that the system will use to fulfill memory allocation requests (ie, don't worry about it, your box has plenty of memory available). > I know that I need to find the source of the leaking and crashing to > begin with, but in the mean time, if it happens, I'd like to free up > the memory manually, so I can get the box running again... shmget() means you've run out of semaphores, not RAM. From postgresql's post-install-notes: To allow many simultaneous connections to your PostgreSQL server, you should raise the SystemV shared memory limits in your kernel. Here are example values for allowing up to 180 clients (tinkering in postgresql.conf also needed, of course): options SYSVSHM options SYSVSEM options SYSVMSG options SHMMAXPGS=65536 options SEMMNI=40 options SEMMNS=240 options SEMUME=40 options SEMMNU=120 You don't have a RAM issue or memory leak, but you're running into a limit for the number semaphores. Dime to dollar you've got mod_ssl installed and its using semaphore locking and not using a file. Google is your friend, this is a pretty common configuration problem/task that many run into. -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Tue Mar 30 12:34:41 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C93D716A4CF for ; Tue, 30 Mar 2004 12:34:41 -0800 (PST) Received: from mail1.simplenet.com (mail1.simplenet.com [209.132.1.30]) by mx1.FreeBSD.org (Postfix) with ESMTP id 99DA243D41 for ; Tue, 30 Mar 2004 12:34:41 -0800 (PST) (envelope-from tt-list@simplenet.com) Received: from TraverPC.simplenet.com (64.170.12.242) by mail1.simplenet.com (7.0.016) (authenticated as tt-list@simplenet.com) id 4069459E00001146; Tue, 30 Mar 2004 12:33:36 -0800 Message-Id: <6.0.1.1.0.20040330123219.01f5fec0@mail1.simplenet.com> X-Sender: tt-list@simplenet.com@mail1.simplenet.com X-Mailer: QUALCOMM Windows Eudora Version 6.0.1.1 Date: Tue, 30 Mar 2004 12:34:32 -0800 To: Sean Chittenden From: Tim Traver In-Reply-To: <5686BD90-8286-11D8-89F3-000A95C705DC@chittenden.org> References: <6.0.1.1.0.20040330113631.01ef7ec0@mail1.simplenet.com> <5686BD90-8286-11D8-89F3-000A95C705DC@chittenden.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Content-Filtered-By: Mailman/MimeDel 2.1.1 cc: freebsd-performance@freebsd.org Subject: Re: shmem release X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Mar 2004 20:34:41 -0000 Thanks Sean, that points me more in the right direction... I have changed the mutex values for mod_ssl, and we'll see if that causes the problem to go away, and I'll look into the amount of shmmaxpgs is set up on the box... Tim. At 12:10 PM 3/30/2004, Sean Chittenden wrote: >>Ok, I am running a 4.7 FreeBSD box that is a web server running apache. >> >>It looks like some module that I have is leaking memory, and >>eventually, apache crashes on restarts becuase of this error : >> >>shmget() failed: No space left on device > >Look in your logs and see if there's an error message being logged that >could help you identify what module it is. Otherwise, get a crash dump >of a crashed process and run gdb on it to see if the module that's >crashing is obvious from the back trace. > >>which means it can't get any more memory, which I understand. >> >>When I look at the top list, it shows me something like this : >> >>Mem: 140M Active, 879M Inact, 151M Wired, 181M Cache, 199M Buf, 660M >>Free >> >>But when you look at the processes that are still up, they hardly take >>up any memory. >> >>So, my question is this. >> >>Is there a way to free up Inactive memory from crashed processes ??? > >You don't need to worry about that. > >http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/admin.html#TOP- >MEMORY-STATES > >There's a better article that I remember reading that went into great >detail explaining this, but I can't seem to dig it up. > >>Without just rebooting the box ??? > >Inactive, buffered, cached, and free memory is memory that the system >will use to fulfill memory allocation requests (ie, don't worry about >it, your box has plenty of memory available). > >>I know that I need to find the source of the leaking and crashing to >>begin with, but in the mean time, if it happens, I'd like to free up >>the memory manually, so I can get the box running again... > >shmget() means you've run out of semaphores, not RAM. From >postgresql's post-install-notes: > >To allow many simultaneous connections to your PostgreSQL server, you >should raise the SystemV shared memory limits in your kernel. Here are >example values for allowing up to 180 clients (tinkering in >postgresql.conf also needed, of course): > options SYSVSHM > options SYSVSEM > options SYSVMSG > options SHMMAXPGS=65536 > options SEMMNI=40 > options SEMMNS=240 > options SEMUME=40 > options SEMMNU=120 > >You don't have a RAM issue or memory leak, but you're running into a >limit for the number semaphores. Dime to dollar you've got mod_ssl >installed and its using semaphore locking and not using a file. Google >is your friend, this is a pretty common configuration problem/task that >many run into. -sc > >-- >Sean Chittenden > >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" SimpleNet's Back ! http://www.simplenet.com