From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 04:32:00 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01A02106566C; Sun, 6 Mar 2011 04:32:00 +0000 (UTC) (envelope-from anderson@ttel.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4C1A68FC08; Sun, 6 Mar 2011 04:31:59 +0000 (UTC) Received: by gxk7 with SMTP id 7so1430064gxk.13 for ; Sat, 05 Mar 2011 20:31:58 -0800 (PST) Received: by 10.90.126.17 with SMTP id y17mr3099507agc.64.1299385918593; Sat, 05 Mar 2011 20:31:58 -0800 (PST) Received: from [10.0.7.145] (r74-193-70-198.pfvlcmta01.grtntx.tl.dh.suddenlink.net [74.193.70.198]) by mx.google.com with ESMTPS id w4sm1732043anw.16.2011.03.05.20.31.55 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 05 Mar 2011 20:31:57 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Eric Anderson In-Reply-To: Date: Sat, 5 Mar 2011 22:31:53 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> To: Eygene Ryabinkin X-Mailer: Apple Mail (2.1082) X-Mailman-Approved-At: Sun, 06 Mar 2011 06:40:46 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 04:32:00 -0000 On Mar 5, 2011, at 6:23 AM, Eygene Ryabinkin wrote: > Fri, Mar 04, 2011 at 10:02:45PM -0600, Eric Anderson wrote: >> I have a moderately threaded userland program (all C) I am working >> on (using pthreads, freebsd 8.1 64bit). It seems to leak memory >> (using standard malloc/free) badly. I am using pcap to capture >> packets and process them. I have a handful of libs statically linked >> in (pcap is one, the rest don't seem to matter - I can remove them >> and still see the leak). =20 >>=20 >> Does anyone know of issues regarding malloc/free on multithreaded >> userland apps? =20 >=20 > I personally hadn't heard about them, but you can definitely run > your program under Valgrind's memcheck tool -- it often does the > good amount of work and detects many leaks. Valrgind is ported > to FreeBSD, devel/valgrind. I tried valgrind from ports, however there were some missing syscalls = and such, and when I went to fill them in, could not figure out which it = was, and just could not make it fly. :( I also tried to make hoard work, but it seems to not be happy in ports = on 64bit FreeBSD. Thanks! Eric From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 04:53:58 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E8E1106566B for ; Sun, 6 Mar 2011 04:53:58 +0000 (UTC) (envelope-from anderson@ttel.com) Received: from mail-yi0-f54.google.com (mail-yi0-f54.google.com [209.85.218.54]) by mx1.freebsd.org (Postfix) with ESMTP id D57538FC08 for ; Sun, 6 Mar 2011 04:53:57 +0000 (UTC) Received: by yie12 with SMTP id 12so1419612yie.13 for ; Sat, 05 Mar 2011 20:53:57 -0800 (PST) Received: by 10.236.184.130 with SMTP id s2mr826783yhm.71.1299387236937; Sat, 05 Mar 2011 20:53:56 -0800 (PST) Received: from [10.0.7.145] (r74-193-70-198.pfvlcmta01.grtntx.tl.dh.suddenlink.net [74.193.70.198]) by mx.google.com with ESMTPS id u49sm820073yhn.37.2011.03.05.20.53.55 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 05 Mar 2011 20:53:56 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Eric Anderson In-Reply-To: <4D726887.5080800@rojer.pp.ru> Date: Sat, 5 Mar 2011 22:53:54 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> To: Deomid Ryabkov X-Mailer: Apple Mail (2.1082) X-Mailman-Approved-At: Sun, 06 Mar 2011 06:40:58 +0000 Cc: freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 04:53:58 -0000 On Mar 5, 2011, at 10:44 AM, Deomid Ryabkov wrote: > On 03/05/2011 04:02 AM, Eric Anderson wrote: >> Hi all, >>=20 >> I have a moderately threaded userland program (all C) I am working on = (using pthreads, freebsd 8.1 64bit). It seems to leak memory (using = standard malloc/free) badly. > as opposed to what? OpenBSD? Linux? Windows? why do you think your = problem is specific to FreeBSD (as evidenced by your post to a = FreeBSD-specific list) or is related to threaded programs? OpenSolaris and Mac OS X. I didn't really assume or state it was = specific to FreeBSD, just that this scenario was on FreeBSD. I happen to = do most of the development and testing on OS X and FreeBSD, and I've = enjoyed the FreeBSD community for a very long time. >=20 >> I am using pcap to capture packets and process them. I have a = handful of libs statically linked in (pcap is one, the rest don't seem = to matter - I can remove them and still see the leak). >>=20 >> Does anyone know of issues regarding malloc/free on multithreaded = userland apps? > hell yeah. it goes like this: you malloc() then forget to free() - = boom, you have a memory leak. >=20 > you're welcome. Thanks, very insightful. >=20 >=20 > sarcasm aside, those questions still remain: why do you think = os/libraries are the problem and not your code? Because I am tracking all malloc and free calls within the application = code (aside from libraries) and I can account for all malloc'ed memory = and freed memory in both count and by bytes, yet looking at ps output = shows a very different story, and if I leave it run long enough, will = consume all memory and swap in the system and then be killed off. I = wrapped malloc/free in a function, and record all memory alloc'ed and = free'd. The only memory I cannot track is memory alloced and freed by = libraries I am pulling in (well, can't track easily anyway without = hacking through all of their source code). > you can't post all of it, ok, and we don't want all of it either. can = you isolate a specific example of where valid usage of a library causes = a leak? Not really - if I could, I would have fixed it by now. It's a = non-trivial issue - which is why I am beginning to suspect something = more complicated than a "oops I forgot to free" kind of error. Plus, I = have seen a few people elsewhere on various forums/mailing lists with = similar issues claiming that switching to the Hoard allocator fixed the = problem (which doesn't seem to be happy with 32bit FreeBSD - tried it). = I have also seen various comments about pthreads and memory allocators = having apparent leaks at some threading level, but not sure. Thanks, Eric From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 12:07:26 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8C048106566B for ; Sun, 6 Mar 2011 12:07:26 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4FFF18FC12 for ; Sun, 6 Mar 2011 12:07:26 +0000 (UTC) Received: by qwj8 with SMTP id 8so2861618qwj.13 for ; Sun, 06 Mar 2011 04:07:25 -0800 (PST) Received: by 10.224.8.205 with SMTP id i13mr337514qai.214.1299413245182; Sun, 06 Mar 2011 04:07:25 -0800 (PST) MIME-Version: 1.0 Received: by 10.224.46.161 with HTTP; Sun, 6 Mar 2011 04:06:45 -0800 (PST) In-Reply-To: <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> From: Vlad Galu Date: Sun, 6 Mar 2011 14:06:45 +0200 Message-ID: To: Eric Anderson Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Deomid Ryabkov , freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 12:07:26 -0000 On Sun, Mar 6, 2011 at 6:53 AM, Eric Anderson wrote: > On Mar 5, 2011, at 10:44 AM, Deomid Ryabkov wrote: > > > On 03/05/2011 04:02 AM, Eric Anderson wrote: > >> Hi all, > >> > >> I have a moderately threaded userland program (all C) I am working on > (using pthreads, freebsd 8.1 64bit). It seems to leak memory (using > standard malloc/free) badly. > > as opposed to what? OpenBSD? Linux? Windows? why do you think your > problem is specific to FreeBSD (as evidenced by your post to a > FreeBSD-specific list) or is related to threaded programs? > > OpenSolaris and Mac OS X. I didn't really assume or state it was specific > to FreeBSD, just that this scenario was on FreeBSD. I happen to do most of > the development and testing on OS X and FreeBSD, and I've enjoyed the > FreeBSD community for a very long time. > > > > > > >> I am using pcap to capture packets and process them. I have a handful > of libs statically linked in (pcap is one, the rest don't seem to matter - I > can remove them and still see the leak). > >> > >> Does anyone know of issues regarding malloc/free on multithreaded > userland apps? > > hell yeah. it goes like this: you malloc() then forget to free() - boom, > you have a memory leak. > > > > you're welcome. > > > Thanks, very insightful. > > > > > > > > sarcasm aside, those questions still remain: why do you think > os/libraries are the problem and not your code? > > Because I am tracking all malloc and free calls within the application code > (aside from libraries) and I can account for all malloc'ed memory and freed > memory in both count and by bytes, yet looking at ps output shows a very > different story, and if I leave it run long enough, will consume all memory > and swap in the system and then be killed off. I wrapped malloc/free in a > function, and record all memory alloc'ed and free'd. The only memory I > cannot track is memory alloced and freed by libraries I am pulling in (well, > can't track easily anyway without hacking through all of their source code). > > > > > you can't post all of it, ok, and we don't want all of it either. can you > isolate a specific example of where valid usage of a library causes a leak? > > > Not really - if I could, I would have fixed it by now. It's a non-trivial > issue - which is why I am beginning to suspect something more complicated > than a "oops I forgot to free" kind of error. Plus, I have seen a few > people elsewhere on various forums/mailing lists with similar issues > claiming that switching to the Hoard allocator fixed the problem (which > doesn't seem to be happy with 32bit FreeBSD - tried it). I have also seen > various comments about pthreads and memory allocators having apparent leaks > at some threading level, but not sure. > > Thanks, > Eric > > Had there been a memleak in jemalloc, I'm sure more people would have spotted it by now. How many pcap_t structs do you use in your app? libpcap is not threadsafe. FWIW I've been running a pcap/threaded application continuously for the past couple of years and the memory usage has been constant. Also, put a couple of printf()s before and after pcap_dispatch()/pcap_loop() (if you use any of them), to make sure they don't block waiting for your callback to return (on some platforms it doesn't, so you never get to free memory in outer frames). This might not necessarily be your case, but it's worth taking a look at. > > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > -- Good, fast & cheap. Pick any two. From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 12:09:57 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95CD9106566B for ; Sun, 6 Mar 2011 12:09:57 +0000 (UTC) (envelope-from dudu@dudu.ro) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5841D8FC12 for ; Sun, 6 Mar 2011 12:09:56 +0000 (UTC) Received: by qwj8 with SMTP id 8so2862305qwj.13 for ; Sun, 06 Mar 2011 04:09:56 -0800 (PST) Received: by 10.224.8.205 with SMTP id i13mr338732qai.214.1299413396268; Sun, 06 Mar 2011 04:09:56 -0800 (PST) MIME-Version: 1.0 Received: by 10.224.46.161 with HTTP; Sun, 6 Mar 2011 04:09:16 -0800 (PST) In-Reply-To: References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> From: Vlad Galu Date: Sun, 6 Mar 2011 14:09:16 +0200 Message-ID: To: Eric Anderson Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Deomid Ryabkov , freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 12:09:57 -0000 On Sun, Mar 6, 2011 at 2:06 PM, Vlad Galu wrote: > > > On Sun, Mar 6, 2011 at 6:53 AM, Eric Anderson wrote: > >> On Mar 5, 2011, at 10:44 AM, Deomid Ryabkov wrote: >> >> > On 03/05/2011 04:02 AM, Eric Anderson wrote: >> >> Hi all, >> >> >> >> I have a moderately threaded userland program (all C) I am working on >> (using pthreads, freebsd 8.1 64bit). It seems to leak memory (using >> standard malloc/free) badly. >> > as opposed to what? OpenBSD? Linux? Windows? why do you think your >> problem is specific to FreeBSD (as evidenced by your post to a >> FreeBSD-specific list) or is related to threaded programs? >> >> OpenSolaris and Mac OS X. I didn't really assume or state it was specific >> to FreeBSD, just that this scenario was on FreeBSD. I happen to do most of >> the development and testing on OS X and FreeBSD, and I've enjoyed the >> FreeBSD community for a very long time. >> >> >> >> > >> >> I am using pcap to capture packets and process them. I have a handful >> of libs statically linked in (pcap is one, the rest don't seem to matter - I >> can remove them and still see the leak). >> >> >> >> Does anyone know of issues regarding malloc/free on multithreaded >> userland apps? >> > hell yeah. it goes like this: you malloc() then forget to free() - boom, >> you have a memory leak. >> > >> > you're welcome. >> >> >> Thanks, very insightful. >> >> >> > >> > >> > sarcasm aside, those questions still remain: why do you think >> os/libraries are the problem and not your code? >> >> Because I am tracking all malloc and free calls within the application >> code (aside from libraries) and I can account for all malloc'ed memory and >> freed memory in both count and by bytes, yet looking at ps output shows a >> very different story, and if I leave it run long enough, will consume all >> memory and swap in the system and then be killed off. I wrapped malloc/free >> in a function, and record all memory alloc'ed and free'd. The only memory I >> cannot track is memory alloced and freed by libraries I am pulling in (well, >> can't track easily anyway without hacking through all of their source code). >> >> >> >> > you can't post all of it, ok, and we don't want all of it either. can >> you isolate a specific example of where valid usage of a library causes a >> leak? >> >> >> Not really - if I could, I would have fixed it by now. It's a non-trivial >> issue - which is why I am beginning to suspect something more complicated >> than a "oops I forgot to free" kind of error. Plus, I have seen a few >> people elsewhere on various forums/mailing lists with similar issues >> claiming that switching to the Hoard allocator fixed the problem (which >> doesn't seem to be happy with 32bit FreeBSD - tried it). I have also seen >> various comments about pthreads and memory allocators having apparent leaks >> at some threading level, but not sure. >> >> Thanks, >> Eric >> >> > Had there been a memleak in jemalloc, I'm sure more people would have > spotted it by now. How many pcap_t structs do you use in your app? libpcap > is not threadsafe. FWIW I've been running a pcap/threaded application > continuously for the past couple of years and the memory usage has been > constant. > > Also, put a couple of printf()s before and after > pcap_dispatch()/pcap_loop() (if you use any of them), to make sure they > don't block waiting for your callback to return (on some platforms it > doesn't, so you never get to free memory in outer frames). This might not > necessarily be your case, but it's worth taking a look at. > > That should've been pcap_dispatch()/pcap_next(). > >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org >> " >> > > > > -- > Good, fast & cheap. Pick any two. > -- Good, fast & cheap. Pick any two. From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 09:29:44 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D8813106566B for ; Sun, 6 Mar 2011 09:29:44 +0000 (UTC) (envelope-from martin.birgmeier@aon.at) Received: from email.aon.at (nat-warsl417-01.aon.at [195.3.96.119]) by mx1.freebsd.org (Postfix) with ESMTP id 393A78FC0C for ; Sun, 6 Mar 2011 09:29:43 +0000 (UTC) Received: (qmail 23047 invoked from network); 6 Mar 2011 09:03:03 -0000 Received: from smarthub95.highway.telekom.at (HELO email.aon.at) ([172.18.5.234]) (envelope-sender ) by fallback43.highway.telekom.at (qmail-ldap-1.03) with SMTP for ; 6 Mar 2011 09:03:03 -0000 Received: (qmail 31909 invoked from network); 6 Mar 2011 09:03:00 -0000 X-Spam-Checker-Version: SpamAssassin 3.2.0 (2007-05-01) on WARSBL503.highway.telekom.at X-Spam-Level: Received: from 188-23-212-212.adsl.highway.telekom.at (HELO gandalf.xyzzy) ([188.23.212.212]) (envelope-sender ) by smarthub95.highway.telekom.at (qmail-ldap-1.03) with SMTP for ; 6 Mar 2011 09:03:00 -0000 Received: from atpcdvvc.xyzzy (atpcdvvc.xyzzy [IPv6:fec0:0:0:4d42::84]) by gandalf.xyzzy (8.14.4/8.14.4) with ESMTP id p2692xdm007453 for ; Sun, 6 Mar 2011 10:03:00 +0100 (CET) (envelope-from martin.birgmeier@aon.at) Message-ID: <4D734DC3.3050709@aon.at> Date: Sun, 06 Mar 2011 10:02:59 +0100 From: Martin Birgmeier User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.15) Gecko/20110305 Thunderbird/3.1.9 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sun, 06 Mar 2011 12:13:49 +0000 Subject: All CPUs always running at profhz X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 09:29:44 -0000 I have systems running 7.4 and 8.2. The newer ones (in terms of age of hardware) are always running at profhz, i.e., when executing a "systat -vm 1" I see that all cores always get profhz interrupts per second. An older machine running 7.4 is running at hz (which for this machine I set to kern.hz=200 in loader.conf). Using ps -o ...,flags,... I see that no process has profiling turned on. Why is this so? Especially on laptops, I'd like to save power by lowering the interrupt rate. Regards, Martin -- Martin Birgmeier Vienna Austria From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 15:34:13 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93090106566B for ; Sun, 6 Mar 2011 15:34:13 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-ey0-f182.google.com (mail-ey0-f182.google.com [209.85.215.182]) by mx1.freebsd.org (Postfix) with ESMTP id 224128FC08 for ; Sun, 6 Mar 2011 15:34:12 +0000 (UTC) Received: by eyg7 with SMTP id 7so1204950eyg.13 for ; Sun, 06 Mar 2011 07:34:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=qYAzzLPqUhwcJvbbUXVxYEXWG063ZSobhiFEeYK4d7E=; b=ccUM40nzazaIHJPDX5S2bHqfLt/ZcVrVGiGl4HnhVnauJpW9D31siyQsU0gvbJc6jK 2QxcUEP1LAzeq1BJL0njBqMmOchQxffZtk1nW8ua9b5ghwVF8skcsTQyjPHNRwNiLEux 8pbsQBpnxtQAkzEjnA15MYX3qVyBTIl8otvUU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=uOUuXejPWvUztIapOJhYZ40SvkYm3oiPfJdaT5dqbya3iFKsYoiWTIfiHd5SioSahG xbI9Hr70ALeWUHUWYgg9V5lDn/vIExp8PINHL0dfaUN8hNlTkRKWG6OEtap5kPT0id4m d7L4eIBX0A51GreB2PoKtoHzLmHLiT5W5nAGo= MIME-Version: 1.0 Received: by 10.213.28.200 with SMTP id n8mr365167ebc.3.1299425651853; Sun, 06 Mar 2011 07:34:11 -0800 (PST) Received: by 10.213.23.13 with HTTP; Sun, 6 Mar 2011 07:34:11 -0800 (PST) In-Reply-To: References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> Date: Sun, 6 Mar 2011 10:34:11 -0500 Message-ID: From: Ryan Stone To: Vlad Galu Content-Type: text/plain; charset=ISO-8859-1 Cc: Deomid Ryabkov , Eric Anderson , freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 15:34:13 -0000 I would try playing with MALLOC_OPTIONS. I seriously doubt that there is an actual leak in jemalloc, but from my own experiences with it I suspect that there are certain multithreaded malloc/free sequences that interact badly with with the per-thread caching that jemalloc performs. The first thing I would try is setting MALLOC_OPTIONS=7h to disable the caching. From owner-freebsd-hackers@FreeBSD.ORG Sun Mar 6 17:35:04 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D76391065672 for ; Sun, 6 Mar 2011 17:35:04 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-ew0-f54.google.com (mail-ew0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 5FC9A8FC0A for ; Sun, 6 Mar 2011 17:35:04 +0000 (UTC) Received: by ewy28 with SMTP id 28so1243869ewy.13 for ; Sun, 06 Mar 2011 09:35:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=nps4GO6uTwFVcp+O1fApQAamiS7OAxYk76GjQ6SCdrw=; b=NiFN/Fsp4QhdCEMYVcQT0fKPFbVH8zBVB2URhfVWp7NSNgVFmtvTjwQo/oNjPOKt5t Bs1VLqoDpBNLnIBTQrxdLXEreAXszF1J+WEAqOaKfY1kJZyTG/EUF/eIqyou3Cd9GN8o T21bhBh1qJYRooty5bguECQelR24iZNzb/Kvw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=bBq2c7Ouz77FELqqHgdRF4+5QXVAB8l9wNVScWrL1rgf3ZLkSXRR7q+sy/cD/QLkIm ISDqVhyUtoGaka7KY7PsKxcvU2jFeoGvROuA6NwGKe0421a3yWAugmmsf2WIj8ja/V7c 1BCn9XOd1vXM7DLHeOaGxrJZ/AqIFxUI/UBTk= MIME-Version: 1.0 Received: by 10.213.109.72 with SMTP id i8mr1108177ebp.42.1299432903252; Sun, 06 Mar 2011 09:35:03 -0800 (PST) Received: by 10.213.23.13 with HTTP; Sun, 6 Mar 2011 09:35:03 -0800 (PST) In-Reply-To: References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> Date: Sun, 6 Mar 2011 12:35:03 -0500 Message-ID: From: Ryan Stone To: Vlad Galu Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Deomid Ryabkov , Eric Anderson , freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Mar 2011 17:35:04 -0000 On Sun, Mar 6, 2011 at 10:34 AM, Ryan Stone wrote: > I would try playing with MALLOC_OPTIONS. =A0I seriously doubt that there > is an actual leak in jemalloc, but from my own experiences with it I > suspect that there are certain multithreaded malloc/free sequences > that interact badly with with the per-thread caching that jemalloc > performs. =A0The first thing I would try is setting MALLOC_OPTIONS=3D7h t= o > disable the caching. > Wait, sorry, apparently this is a new option in HEAD. Under 8.1 MALLOC_OPTIONS=3Dg will disable the thread-specific caching. See the malloc(3) man page for the definitive list of available options. From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 7 02:58:47 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8F4D4106564A for ; Mon, 7 Mar 2011 02:58:47 +0000 (UTC) (envelope-from anderson@ttel.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 4B6088FC16 for ; Mon, 7 Mar 2011 02:58:46 +0000 (UTC) Received: by ywf9 with SMTP id 9so1669335ywf.13 for ; Sun, 06 Mar 2011 18:58:46 -0800 (PST) Received: by 10.236.110.50 with SMTP id t38mr2153727yhg.84.1299466726339; Sun, 06 Mar 2011 18:58:46 -0800 (PST) Received: from [10.0.7.145] (r74-193-70-198.pfvlcmta01.grtntx.tl.dh.suddenlink.net [74.193.70.198]) by mx.google.com with ESMTPS id y73sm1425614yhc.42.2011.03.06.18.58.44 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 06 Mar 2011 18:58:45 -0800 (PST) Sender: Eric Anderson Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Eric Anderson In-Reply-To: Date: Sun, 6 Mar 2011 20:58:43 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> To: Ryan Stone X-Mailer: Apple Mail (2.1082) Cc: Deomid Ryabkov , Vlad Galu , freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2011 02:58:47 -0000 On Mar 6, 2011, at 11:35 AM, Ryan Stone wrote: > On Sun, Mar 6, 2011 at 10:34 AM, Ryan Stone wrote: >> I would try playing with MALLOC_OPTIONS. I seriously doubt that = there >> is an actual leak in jemalloc, but from my own experiences with it I >> suspect that there are certain multithreaded malloc/free sequences >> that interact badly with with the per-thread caching that jemalloc >> performs. The first thing I would try is setting MALLOC_OPTIONS=3D7h = to >> disable the caching. >>=20 >=20 > Wait, sorry, apparently this is a new option in HEAD. Under 8.1 > MALLOC_OPTIONS=3Dg will disable the thread-specific caching. See the > malloc(3) man page for the definitive list of available options. Thanks - I think I tried that at one point with no change, but I'll = double check just in case. Thanks for the hints! Eric From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 7 02:56:15 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 16F961065670 for ; Mon, 7 Mar 2011 02:56:15 +0000 (UTC) (envelope-from anderson@ttel.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id CB2D68FC12 for ; Mon, 7 Mar 2011 02:56:14 +0000 (UTC) Received: by yxl31 with SMTP id 31so1673403yxl.13 for ; Sun, 06 Mar 2011 18:56:14 -0800 (PST) Received: by 10.150.131.2 with SMTP id e2mr3786192ybd.262.1299466573833; Sun, 06 Mar 2011 18:56:13 -0800 (PST) Received: from [10.0.7.145] (r74-193-70-198.pfvlcmta01.grtntx.tl.dh.suddenlink.net [74.193.70.198]) by mx.google.com with ESMTPS id l2sm1437208ybn.15.2011.03.06.18.56.12 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 06 Mar 2011 18:56:12 -0800 (PST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Eric Anderson In-Reply-To: Date: Sun, 6 Mar 2011 20:56:11 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <79592F5D-A099-4474-A818-4DD4C6F279D5@ttel.com> References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> To: Vlad Galu X-Mailer: Apple Mail (2.1082) X-Mailman-Approved-At: Mon, 07 Mar 2011 03:14:25 +0000 Cc: Deomid Ryabkov , freebsd-hackers@freebsd.org Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2011 02:56:15 -0000 On Mar 6, 2011, at 6:09 AM, Vlad Galu wrote: >=20 >=20 > On Sun, Mar 6, 2011 at 2:06 PM, Vlad Galu wrote: >=20 >=20 > On Sun, Mar 6, 2011 at 6:53 AM, Eric Anderson = wrote: > On Mar 5, 2011, at 10:44 AM, Deomid Ryabkov wrote: >=20 > > On 03/05/2011 04:02 AM, Eric Anderson wrote: > >> Hi all, > >> > >> I have a moderately threaded userland program (all C) I am working = on (using pthreads, freebsd 8.1 64bit). It seems to leak memory (using = standard malloc/free) badly. > > as opposed to what? OpenBSD? Linux? Windows? why do you think your = problem is specific to FreeBSD (as evidenced by your post to a = FreeBSD-specific list) or is related to threaded programs? >=20 > OpenSolaris and Mac OS X. I didn't really assume or state it was = specific to FreeBSD, just that this scenario was on FreeBSD. I happen to = do most of the development and testing on OS X and FreeBSD, and I've = enjoyed the FreeBSD community for a very long time. >=20 >=20 >=20 > > > >> I am using pcap to capture packets and process them. I have a = handful of libs statically linked in (pcap is one, the rest don't seem = to matter - I can remove them and still see the leak). > >> > >> Does anyone know of issues regarding malloc/free on multithreaded = userland apps? > > hell yeah. it goes like this: you malloc() then forget to free() - = boom, you have a memory leak. > > > > you're welcome. >=20 >=20 > Thanks, very insightful. >=20 >=20 > > > > > > sarcasm aside, those questions still remain: why do you think = os/libraries are the problem and not your code? >=20 > Because I am tracking all malloc and free calls within the application = code (aside from libraries) and I can account for all malloc'ed memory = and freed memory in both count and by bytes, yet looking at ps output = shows a very different story, and if I leave it run long enough, will = consume all memory and swap in the system and then be killed off. I = wrapped malloc/free in a function, and record all memory alloc'ed and = free'd. The only memory I cannot track is memory alloced and freed by = libraries I am pulling in (well, can't track easily anyway without = hacking through all of their source code). >=20 >=20 >=20 > > you can't post all of it, ok, and we don't want all of it either. = can you isolate a specific example of where valid usage of a library = causes a leak? >=20 >=20 > Not really - if I could, I would have fixed it by now. It's a = non-trivial issue - which is why I am beginning to suspect something = more complicated than a "oops I forgot to free" kind of error. Plus, I = have seen a few people elsewhere on various forums/mailing lists with = similar issues claiming that switching to the Hoard allocator fixed the = problem (which doesn't seem to be happy with 32bit FreeBSD - tried it). = I have also seen various comments about pthreads and memory allocators = having apparent leaks at some threading level, but not sure. >=20 > Thanks, > Eric >=20 >=20 > Had there been a memleak in jemalloc, I'm sure more people would have = spotted it by now. How many pcap_t structs do you use in your app? = libpcap is not threadsafe. FWIW I've been running a pcap/threaded = application continuously for the past couple of years and the memory = usage has been constant. >=20 > Also, put a couple of printf()s before and after = pcap_dispatch()/pcap_loop() (if you use any of them), to make sure they = don't block waiting for your callback to return (on some platforms it = doesn't, so you never get to free memory in outer frames). This might = not necessarily be your case, but it's worth taking a look at. > =20 >=20 > That should've been pcap_dispatch()/pcap_next(). Hmm - interestingly enough, I use a third party lib that handles some = parts of what pcap does.. I'll did into that lib and check these = things. Thanks for the idea! Eric From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 7 15:01:17 2011 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5955B1065670 for ; Mon, 7 Mar 2011 15:01:17 +0000 (UTC) (envelope-from bf1783@googlemail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 175ED8FC14 for ; Mon, 7 Mar 2011 15:01:16 +0000 (UTC) Received: by ywf9 with SMTP id 9so1867061ywf.13 for ; Mon, 07 Mar 2011 07:01:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=gamma; h=domainkey-signature:mime-version:reply-to:date:message-id:subject :from:to:cc:content-type; bh=cdDimqisoijpSRj87esLEl0SaHupK6PB2astnDUc3iI=; b=txpRWLUdv90dLqKUyk8gCW2rvAVfXUoEDj9WNBJhtbrvMV9ojpswBE5LKeihLIIm+1 jTUouW5KHQVanBFuJu5Fvfa/8ZYps3S6P8cy++ffu67Lt85HS2uHRDx7mOdaL1EQdhVq kdiqsdMUqynN+/EeOhF7S3O1Gqm55VKMn+cCU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=googlemail.com; s=gamma; h=mime-version:reply-to:date:message-id:subject:from:to:cc :content-type; b=YOR/ACavPqstB9CzBw9Oa5ae7563lOkbJ0TIDO4xim6J9j9UId7CULUNTbLWDCVqRD tVQsZKnj85ed/SkuSXsaayrtMDtoynxNxZkq7S2sgoFLvZjE6poKot9g4fTKwQATR2LD zuM3BLpECgqF5JFtQ3gFUoEmcfSAoXoMJ+jpM= MIME-Version: 1.0 Received: by 10.236.192.130 with SMTP id i2mr6083567yhn.37.1299510076281; Mon, 07 Mar 2011 07:01:16 -0800 (PST) Received: by 10.236.108.44 with HTTP; Mon, 7 Mar 2011 07:01:16 -0800 (PST) Date: Mon, 7 Mar 2011 15:01:16 +0000 Message-ID: From: "b. f." To: Martin Birgmeier Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-hackers@FreeBSD.org Subject: Re: All CPUs always running at profhz X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bf1783@gmail.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2011 15:01:17 -0000 > I have systems running 7.4 and 8.2. The newer ones (in terms of age of > hardware) are always running at profhz, i.e., when executing a "systat > -vm 1" I see that all cores always get profhz interrupts per second. > > An older machine running 7.4 is running at hz (which for this machine I > set to kern.hz=200 in loader.conf). > > Using ps -o ...,flags,... I see that no process has profiling turned on. > > Why is this so? > > Especially on laptops, I'd like to save power by lowering the interrupt > rate. I have not analyzed this problem. However, by using 9-CURRENT, together with http://people.freebsd.org/~mav/tm6292_idle.patch , removing DEVICE_POLLING from my kernel, and reducing kern.hz to 100, I was able to achieve reported interrupt rates of ~35-40 Hz on my UP machines when idle, and substantial reductions when the machines were loaded, as well. When used with other power-saving measures, this was a big help. b. From owner-freebsd-hackers@FreeBSD.ORG Mon Mar 7 23:10:10 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B534A1065673 for ; Mon, 7 Mar 2011 23:10:10 +0000 (UTC) (envelope-from jstuart@adaranet.com) Received: from barracuda.adaranet.com (smtp.adaranet.com [72.5.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 95DFB8FC20 for ; Mon, 7 Mar 2011 23:10:08 +0000 (UTC) X-ASG-Debug-ID: 1299538456-2c504f1d0001-P5m3U7 Received: from SJ-EXCH-1.adaranet.com ([10.10.1.29]) by barracuda.adaranet.com with ESMTP id kuGlfGCoAXAAYdnl for ; Mon, 07 Mar 2011 14:54:16 -0800 (PST) X-Barracuda-Envelope-From: jstuart@adaranet.com Received: from SJ-EXCH-1.adaranet.com ([fe80::7042:d8c2:5973:c523]) by SJ-EXCH-1.adaranet.com ([fe80::7042:d8c2:5973:c523%14]) with mapi; Mon, 7 Mar 2011 14:54:16 -0800 From: Jonathan Stuart X-Barracuda-BBL-IP: fe80::7042:d8c2:5973:c523 X-Barracuda-RBL-IP: fe80::7042:d8c2:5973:c523 To: "freebsd-hackers@freebsd.org" Date: Mon, 7 Mar 2011 14:54:47 -0800 X-ASG-Orig-Subj: UMA zone alloc on large UMA zone causing Page fault in kernel mode Thread-Topic: UMA zone alloc on large UMA zone causing Page fault in kernel mode Thread-Index: AcvdGqglAkD6l+yjSkWJqBxyxAvTIw== Message-ID: <32AB5C9615CC494997D9ABB1DB12783C024CDCB780@SJ-EXCH-1.adaranet.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Barracuda-Connect: UNKNOWN[10.10.1.29] X-Barracuda-Start-Time: 1299538456 X-Barracuda-URL: http://172.16.10.203:8000/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at adaranet.com X-Mailman-Approved-At: Tue, 08 Mar 2011 00:03:34 +0000 Cc: Patrick Mahan Subject: UMA zone alloc on large UMA zone causing Page fault in kernel mode X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Mar 2011 23:10:10 -0000 Hiya all, Does anyone have any idea why a large UMA zone created without the UMA_ZONE= _PAGEABLE flag would page fault in kernel mode when I uma_zalloc one item w= ith M_ZERO | M_WAITOK. The fault takes place during the bzero'ing of the memory.. the pointer *looks* valid, as well. This does not happen with= some smaller zones I've been using. The zone shows up in ddb's "show uma", and it's size 620756992. Do I need to use UMA_ZONE_NOFREE to keep it in memory? This was not my und= erstanding of that flag. Thanks for any/all help, if code or ddb output would be preferable please l= et me know. Kind Regards, Jonathan Stuart From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 8 03:27:29 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31638106566B for ; Tue, 8 Mar 2011 03:27:29 +0000 (UTC) (envelope-from freebsd-hackers@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id D77748FC0C for ; Tue, 8 Mar 2011 03:27:28 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Pwna3-0007e5-Dx for freebsd-hackers@freebsd.org; Tue, 08 Mar 2011 04:27:27 +0100 Received: from cpe-188-129-84-115.dynamic.amis.hr ([188.129.84.115]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 08 Mar 2011 04:27:27 +0100 Received: from ivoras by cpe-188-129-84-115.dynamic.amis.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 08 Mar 2011 04:27:27 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-hackers@freebsd.org From: Ivan Voras Date: Tue, 08 Mar 2011 04:27:07 +0100 Lines: 19 Message-ID: References: <40A52D4A-9397-4406-A7EC-E7CBBEFADD55@freebsd.org> <4D726887.5080800@rojer.pp.ru> <570723AB-7011-4185-89E1-E02F018DC0C3@ttel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: cpe-188-129-84-115.dynamic.amis.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.12) Gecko/20101102 Thunderbird/3.1.6 In-Reply-To: Subject: Re: Mem leak : malloc/free + pthreads = leakage? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Mar 2011 03:27:29 -0000 On 06/03/2011 18:35, Ryan Stone wrote: > On Sun, Mar 6, 2011 at 10:34 AM, Ryan Stone wrote: >> I would try playing with MALLOC_OPTIONS. I seriously doubt that there >> is an actual leak in jemalloc, but from my own experiences with it I >> suspect that there are certain multithreaded malloc/free sequences >> that interact badly with with the per-thread caching that jemalloc >> performs. The first thing I would try is setting MALLOC_OPTIONS=7h to >> disable the caching. >> > > Wait, sorry, apparently this is a new option in HEAD. Under 8.1 > MALLOC_OPTIONS=g will disable the thread-specific caching. See the > malloc(3) man page for the definitive list of available options. I can confirm this suspicion; I have an malloc-intensive multithreaded program on 8-STABLE which I need to run with MALLOC_OPTIONS g10f2n to reduce otherwise severe memory lossage by the allocator. I haven't tried it on 9, though. From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 8 13:29:06 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 973EF106566B for ; Tue, 8 Mar 2011 13:29:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 6F0158FC16 for ; Tue, 8 Mar 2011 13:29:06 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 0B92546B06; Tue, 8 Mar 2011 08:29:06 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.10]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A07FE8A01B; Tue, 8 Mar 2011 08:29:05 -0500 (EST) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Tue, 8 Mar 2011 08:29:04 -0500 User-Agent: KMail/1.13.5 (FreeBSD/7.4-CBSD-20110107; KDE/4.4.5; amd64; ; ) References: <32AB5C9615CC494997D9ABB1DB12783C024CDCB780@SJ-EXCH-1.adaranet.com> In-Reply-To: <32AB5C9615CC494997D9ABB1DB12783C024CDCB780@SJ-EXCH-1.adaranet.com> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201103080829.05135.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 08 Mar 2011 08:29:05 -0500 (EST) Cc: Patrick Mahan , Jonathan Stuart Subject: Re: UMA zone alloc on large UMA zone causing Page fault in kernel mode X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Mar 2011 13:29:06 -0000 On Monday, March 07, 2011 5:54:47 pm Jonathan Stuart wrote: > Hiya all, > > Does anyone have any idea why a large UMA zone created without the UMA_ZONE_PAGEABLE flag would page fault in kernel mode when I uma_zalloc one item with M_ZERO | M_WAITOK. The fault takes place during the bzero'ing of > the memory.. the pointer *looks* valid, as well. This does not happen with some smaller zones I've been using. > The zone shows up in ddb's "show uma", and it's size 620756992. > > Do I need to use UMA_ZONE_NOFREE to keep it in memory? This was not my understanding of that flag. No, that just prevents free slabs from being returned to the system. Is this on amd64? Some questions, things you can try if so: See if it is a direct-mapped address. If it is, then I'm at a loss as to why this would fault. Maybe verify that the physical addresses backed by that range are valid via SMAP? If it is not direct-mapped, grab my gdb scripts from http://www.freebsd.org/~jhb/gdb/. Then fire up kgdb on the crashdump, cd to a directory holding the scripts and do 'source gdb6'. Then you can use the 'kvm' command to display a rough layout of the kernel's address space. Make sure the virtual address range is backed by something valid. If it is then you might want to write some custom gdb scripts to find the right vm_map_entry in the kernel_map for your address range and find the backing VM object and starting offset. Then you can use gdb to examine the pages assigned to that VM object at that offset and ensure they are valid, etc. You might also try examining the PTE's directly as well. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Tue Mar 8 21:20:29 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C904B1065676 for ; Tue, 8 Mar 2011 21:20:29 +0000 (UTC) (envelope-from jstuart@adaranet.com) Received: from barracuda.adaranet.com (smtp.adaranet.com [72.5.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id A65308FC20 for ; Tue, 8 Mar 2011 21:20:29 +0000 (UTC) X-ASG-Debug-ID: 1299619228-2c5058720001-P5m3U7 Received: from SJ-EXCH-1.adaranet.com ([10.10.1.29]) by barracuda.adaranet.com with ESMTP id v10suob7SjvYIqpY; Tue, 08 Mar 2011 13:20:28 -0800 (PST) X-Barracuda-Envelope-From: jstuart@adaranet.com Received: from SJ-EXCH-1.adaranet.com ([fe80::7042:d8c2:5973:c523]) by SJ-EXCH-1.adaranet.com ([fe80::7042:d8c2:5973:c523%14]) with mapi; Tue, 8 Mar 2011 13:20:29 -0800 From: Jonathan Stuart X-Barracuda-BBL-IP: fe80::7042:d8c2:5973:c523 X-Barracuda-RBL-IP: fe80::7042:d8c2:5973:c523 To: John Baldwin , "freebsd-hackers@freebsd.org" Date: Tue, 8 Mar 2011 13:21:09 -0800 X-ASG-Orig-Subj: RE: UMA zone alloc on large UMA zone causing Page fault in kernel mode Thread-Topic: UMA zone alloc on large UMA zone causing Page fault in kernel mode Thread-Index: AcvdlM3ByxfVZlZiQGC2CD+YbHxOhAAQcAug Message-ID: <32AB5C9615CC494997D9ABB1DB12783C024CDCB861@SJ-EXCH-1.adaranet.com> References: <32AB5C9615CC494997D9ABB1DB12783C024CDCB780@SJ-EXCH-1.adaranet.com> <201103080829.05135.jhb@freebsd.org> In-Reply-To: <201103080829.05135.jhb@freebsd.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Barracuda-Connect: UNKNOWN[10.10.1.29] X-Barracuda-Start-Time: 1299619228 X-Barracuda-URL: http://172.16.10.203:8000/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at adaranet.com X-Mailman-Approved-At: Tue, 08 Mar 2011 23:06:50 +0000 Cc: Patrick Mahan Subject: RE: UMA zone alloc on large UMA zone causing Page fault in kernel mode X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Mar 2011 21:20:29 -0000 Thanks John. Yes, it is amd64. I decreased the size and we're working aro= und it that way for now.. I will do debugging in the future, because we hav= e some other zones that we need to be pretty large. Jonathan -----Original Message----- From: John Baldwin [mailto:jhb@freebsd.org] Sent: Tuesday, March 08, 2011 5:29 AM To: freebsd-hackers@freebsd.org Cc: Jonathan Stuart; Patrick Mahan Subject: Re: UMA zone alloc on large UMA zone causing Page fault in kernel = mode On Monday, March 07, 2011 5:54:47 pm Jonathan Stuart wrote: > Hiya all, > > Does anyone have any idea why a large UMA zone created without the UMA_ZO= NE_PAGEABLE flag would page fault in kernel mode when I uma_zalloc one item with M_ZERO | M_WAITOK. The fault takes place during the bzero'ing of > the memory.. the pointer *looks* valid, as well. This does not happen wi= th some smaller zones I've been using. > The zone shows up in ddb's "show uma", and it's size 620756992. > > Do I need to use UMA_ZONE_NOFREE to keep it in memory? This was not my u= nderstanding of that flag. No, that just prevents free slabs from being returned to the system. Is this on amd64? Some questions, things you can try if so: See if it is a direct-mapped address. If it is, then I'm at a loss as to w= hy this would fault. Maybe verify that the physical addresses backed by that range are valid via SMAP? If it is not direct-mapped, grab my gdb scripts from http://www.freebsd.org/~jhb/gdb/. Then fire up kgdb on the crashdump, cd t= o a directory holding the scripts and do 'source gdb6'. Then you can use the 'kvm' command to display a rough layout of the kernel's address space. Mak= e sure the virtual address range is backed by something valid. If it is then you might want to write some custom gdb scripts to find the right vm_map_entry in the kernel_map for your address range and find the backing VM object and starting offset. Then you can use gdb to examine the pages assigned to that VM object at that offset and ensure they are valid, etc. You might also try examining the PTE's directly as well. -- John Baldwin From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 10 15:35:31 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DBFD71065677 for ; Thu, 10 Mar 2011 15:35:31 +0000 (UTC) (envelope-from elman_syah@yahoo.com) Received: from smtp102-mob.biz.mail.ne1.yahoo.com (smtp102-mob.biz.mail.ne1.yahoo.com [98.138.88.35]) by mx1.freebsd.org (Postfix) with SMTP id 8A6608FC17 for ; Thu, 10 Mar 2011 15:35:31 +0000 (UTC) Received: (qmail 24033 invoked from network); 10 Mar 2011 15:08:51 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=DKIM-Signature:Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:X-rim-org-msg-ref-id:Message-ID:Content-Transfer-Encoding:Reply-To:X-Priority:Sensitivity:Importance:Subject:To:From:Date:Content-Type:MIME-Version; b=b31atZHEjBM3DjFR+XMgfVdPGUQ9VtnxjL7OKuI8TovRZUd+k+2cjGOdwzT5gDb0I2sAQGpwZAE/82ak0S2RoKcU9PxIje6u6PVDIHkLpiBBM6cyX6rK49WoQ6Zg0vhAhOHUahXH6Uk9DT9r6WGxUmrNfI76nZIcFIfWkYW11nc= ; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1299769731; bh=Q2NtDozxt23dOekVaTEzuYklgeJX8m23F5CxfefVbqU=; h=Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:X-rim-org-msg-ref-id:Message-ID:Content-Transfer-Encoding:Reply-To:X-Priority:Sensitivity:Importance:Subject:To:From:Date:Content-Type:MIME-Version; b=gKMSgkKemg2nMragn9GCulgZqLhE2Enm+J9mx6LGynxYlrcGawm1vQWmCHUPlgaTAgRjzCeHB+3cTd5VmMmtchuDIpBAlm4Lf1HinaIUhShISoOnd0oPlZrv3/Z3965gjW4aen2FwWlGXKPa4/D+dzx7MFp3rQZsSUErvaO8MCo= Received: from b13.c1.bise3.blackberry (elman_syah@216.9.249.73 with xymcookie) by smtp102-mob.biz.mail.ne1.yahoo.com with SMTP; 10 Mar 2011 07:08:50 -0800 PST X-Yahoo-SMTP: q_7iUPaswBAP9nwhgF_vqEtO2hoAqIY- X-YMail-OSG: qfetHpoVM1kS9NWBxo3Zw2.JBeyr7Li6YISYIIAJqLHYEqb ZBvKcGkD1Se7UsujZksFUGEkxWzar177fipObs0blqItfpq9Bp49giiwIwBB FfY.N.Ew6f5hh3T7Cdw_Mlas8Oe0473oPIN.Z9rPoUNKHycKkGO9TQFVF2Rw 5VFLLjMIMaApDEn9OrCbzrKivC5_wZsVuxspxUAoc4peSxNBTrZgwi9_gC8h fPQy_1jGLOF4- X-Yahoo-Newman-Property: ymail-3 X-rim-org-msg-ref-id: 1766981347 Message-ID: <1766981347-1299769728-cardhu_decombobulator_blackberry.rim.net-1948900434-@b1.c1.bise3.blackberry> Content-Transfer-Encoding: base64 X-Priority: Normal Sensitivity: Normal Importance: Normal To: freebsd-hackers@freebsd.org From: "juki" Date: Thu, 10 Mar 2011 15:08:50 +0000 Content-Type: text/plain; charset="Windows-1252" MIME-Version: 1.0 X-Mailman-Approved-At: Thu, 10 Mar 2011 16:25:25 +0000 Subject: Error amavisd on freebsd 8.2 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: elman_syah@yahoo.com List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Mar 2011 15:35:31 -0000 RGVhciBhbGwsDQoNCkkgdHJ5IGluc3RhbGwgdGhlIGFtYXZpc2QtbmV3IGZvciB0aGUgbmV3IG1h aWxzZXJ2ZXIsIGJ1dCB0aGUgYW1hdmlzZCBzZXJ2aWNlIGNhbid0IHJ1bm5pbmcsIEkgY2hlY2tl ZCBpbiBsb2cgYW5kIHRoZXJlIG5vdGhpbmcgZXJyb3IsIEkgdHJ5IHRvIGRlaW5zdGFsbCBhbWF2 aXNkIGluIHBvcnQgKC91c3IvcG9ydC9zZWN1cml0eS9hbWF2aXNkLW5ldykgYW5kIGJhY2sgdG8g cmVpbnN0YWxsIGFuZCBub3Qgc3VjY2Vzcy4gV2hhdCBjYW4gSSBkbyBmb3IgcmVzb2x2aW5nIGl0 Pz8/DQoNClRoYW5rcw0KUmVnYXJkcywgSnVraSANClBvd2VyZWQgYnkgVGVsa29tc2VsIEJsYWNr QmVycnmu From owner-freebsd-hackers@FreeBSD.ORG Thu Mar 10 17:44:53 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D542106564A for ; Thu, 10 Mar 2011 17:44:53 +0000 (UTC) (envelope-from jhelfman@experts-exchange.com) Received: from mail.experts-exchange.com (mail.experts-exchange.com [72.29.183.251]) by mx1.freebsd.org (Postfix) with ESMTP id EC2048FC16 for ; Thu, 10 Mar 2011 17:44:52 +0000 (UTC) Received: from mail.experts-exchange.com (localhost [127.0.0.1]) by mail.experts-exchange.com (Postfix) with ESMTP id AE404700EAA; Thu, 10 Mar 2011 09:44:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=e-e.com; h= user-agent:in-reply-to:content-transfer-encoding :content-disposition:content-type:content-type:mime-version :references:message-id:subject:subject:from:from:date:date :received:received:received; s=ee; t=1299779092; x=1301593492; bh=L5o0DqPPXC92eoVC0hs2b1ZApgfJ+BjCSBdjleNgzzA=; b=E/4NIIVnFduq eKq5V3RR64Y1YvV4E4We1P2b2daqdhmUO+h9sxPPKwYqx8s4gYdQBWWqwdeOYHdo vbDSj6UlSJpkLLpS9DWpq3OM3Q3DipdfCi/OfbmqPoNmtCFUYDD3J0K5ijp2PAM6 ycTNL5xTTun42V5ZKPyyH3/0gFFI0xk= X-Virus-Scanned: amavisd-new at experts-exchange.com Received: from mail.experts-exchange.com ([127.0.0.1]) by mail.experts-exchange.com (mail.experts-exchange.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1KSGe2V+1Amy; Thu, 10 Mar 2011 09:44:52 -0800 (PST) Received: from experts-exchange.com (unknown [72.29.180.81]) by mail.experts-exchange.com (Postfix) with SMTP id 7E9AC700EA9; Thu, 10 Mar 2011 09:44:52 -0800 (PST) Received: (nullmailer pid 50558 invoked by uid 1001); Thu, 10 Mar 2011 17:41:28 -0000 Date: Thu, 10 Mar 2011 09:41:28 -0800 From: Jason Helfman To: juki Message-ID: <20110310174127.GQ13527@eggman.experts-exchange.com> References: <1766981347-1299769728-cardhu_decombobulator_blackberry.rim.net-1948900434-@b1.c1.bise3.blackberry> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1766981347-1299769728-cardhu_decombobulator_blackberry.rim.net-1948900434-@b1.c1.bise3.blackberry> X-Operating-System: FreeBSD 8.2-RELEASE X-Living-The-Dream: I love the SLO Life! User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@freebsd.org Subject: Re: Error amavisd on freebsd 8.2 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Mar 2011 17:44:53 -0000 On Thu, Mar 10, 2011 at 03:08:50PM +0000, juki thus spake: >Dear all, > >I try install the amavisd-new for the new mailserver, but the amavisd service can't running, I checked in log and there nothing error, I try to deinstall amavisd in port (/usr/port/security/amavisd-new) and back to reinstall and not success. What can I do for resolving it??? > Errors would be good for helping you in resolve this problem, and you may want to consider following up with freebsd-ports@ on this issue, as you may yield more feedback, and potentially resolve your issue in a more timely manner. -jgh >Thanks >Regards, Juki >Powered by Telkomsel BlackBerry® >_______________________________________________ >freebsd-hackers@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" -- Jason Helfman System Administrator experts-exchange.com http://www.experts-exchange.com/M_4830110.html E4AD 7CF1 1396 27F6 79DD 4342 5E92 AD66 8C8C FBA5 From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 11 00:51:34 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 733CD106566B for ; Fri, 11 Mar 2011 00:51:34 +0000 (UTC) (envelope-from bahamasfranks@gmail.com) Received: from mail-bw0-f54.google.com (mail-bw0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 031EA8FC08 for ; Fri, 11 Mar 2011 00:51:33 +0000 (UTC) Received: by bwz12 with SMTP id 12so2625915bwz.13 for ; Thu, 10 Mar 2011 16:51:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to :content-type; bh=RNvgR6Hfe/AJ2nM4zVLkNGjT+z9RvROIJL2TTRT6yTI=; b=db5kw4obBqXKFILX3ah2mHrYSKHC7WMGSA/T0uhT0zM9qkG4zE3oCkj44jcCvbZWk0 Qy7P/nbwZObnVxrdY8vDtNdPYprTk4aM1Nd92CbBFTAp0Rx92qx9Npf/XUDxenJP0vZ/ ns+UY3dmGWLFFdslno2pt2MTtH9Y8ID9NkvYE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=Y1Gk5kuJQglljaLDbYwtDa5NJnKLAl0EN4g5Td+a5mkxZFrRVU8/fvNoiW+I69Egmj lFA9DLdFey1bH/4SxrzvmmhJvxFg8f6Tbu77/xKXIPsqTXoCgHO2CJQBL3wDSJqgYfjF zFZfaVwCpOzEWWvdMjbxm6iJIy0xeQHle1gBI= MIME-Version: 1.0 Received: by 10.204.20.65 with SMTP id e1mr3840850bkb.149.1299803110654; Thu, 10 Mar 2011 16:25:10 -0800 (PST) Received: by 10.204.57.66 with HTTP; Thu, 10 Mar 2011 16:25:10 -0800 (PST) Date: Thu, 10 Mar 2011 17:25:10 -0700 Message-ID: From: Steve Franks To: freebsd-hackers Content-Type: text/plain; charset=ISO-8859-1 Subject: looking for mature/efficient gui builder/toolkit/IDE for Python (or C for that matter) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2011 00:51:34 -0000 I'm interested in doing some graphical serial-port parsing software in Python (or possibly C which I'm actually more familiar with) - anyone care to render an opinion on the most direct route to a usable gui? I figure Python is probably somewhat the preferred language these days for GUIs given the large number of 'nix desktop apps that have been showing up in python of late... Last time I wrote a gui was in VisualC 6.0, so it's been awhile - with VisualC it took about the same amount of time to write all the coordinates for a GUI in the code as it did to draw it and hook up the code; hopefully things have gotten a bit more streamlined - hoping to spend most of my coding time on string parsing, not gui building... Thanks, Steve From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 11 01:25:25 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4E8D8106566B for ; Fri, 11 Mar 2011 01:25:25 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.10]) by mx1.freebsd.org (Postfix) with ESMTP id C3BA88FC1D for ; Fri, 11 Mar 2011 01:25:24 +0000 (UTC) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.4/8.14.4/NETPLEX) with ESMTP id p2B1PNVr008607; Thu, 10 Mar 2011 20:25:23 -0500 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.2.6 (mail.netplex.net [204.213.176.10]); Thu, 10 Mar 2011 20:25:23 -0500 (EST) Date: Thu, 10 Mar 2011 20:25:23 -0500 (EST) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Steve Franks In-Reply-To: Message-ID: References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-hackers Subject: Re: looking for mature/efficient gui builder/toolkit/IDE for Python (or C for that matter) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Daniel Eischen List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2011 01:25:25 -0000 On Thu, 10 Mar 2011, Steve Franks wrote: > I'm interested in doing some graphical serial-port parsing software in > Python (or possibly C which I'm actually more familiar with) - anyone > care to render an opinion on the most direct route to a usable gui? > > I figure Python is probably somewhat the preferred language these days > for GUIs given the large number of 'nix desktop apps that have been > showing up in python of late... > > Last time I wrote a gui was in VisualC 6.0, so it's been awhile - with > VisualC it took about the same amount of time to write all the > coordinates for a GUI in the code as it did to draw it and hook up the > code; hopefully things have gotten a bit more streamlined - hoping to > spend most of my coding time on string parsing, not gui building... After using Java for a few years (since JDK 1.3), I wouldn't use anything else for a GUI. It'll also easily run on other OS's. I don't use IDE's though, so I can't recommend one. Some people I work with use Netbeans though. There does seem to be support available for Python in Netbeans: http://wiki.netbeans.org/Python -- DE From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 11 05:42:54 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBA40106566C; Fri, 11 Mar 2011 05:42:54 +0000 (UTC) (envelope-from subbsd@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 9B0C28FC17; Fri, 11 Mar 2011 05:42:54 +0000 (UTC) Received: by ywf9 with SMTP id 9so1155523ywf.13 for ; Thu, 10 Mar 2011 21:42:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:date:message-id:subject:from:to:cc :content-type; bh=Hd/nnMBFbN7/kX0ap567rhX+t8LivzuFCF8OmY02oVY=; b=noS+Zh1ZQwXSTft/7aOwxH51jbay1rOzH9ta5vI1gvRXEku6ccFTUVrjJOnGHdVAfB YoTFKaNdCmWBqkbOf1mEsj4zcQbOiwTx5wnpuiB2ByLlZAAqzgobAOOAJnan1GtgfJW1 mrDbSd5yjJQDDyT5yGIO8MK1EDaU4i+u8pbeE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:cc:content-type; b=l6E+v6kQ3IhDJaRap/ness+488hTn/j6yBvw57sUSjBPYnFEfOWtNfwjqPykkNrDGJ jPE7mj/JiFIPV35V7yYBHYx8Au210pJhpFBJZ3HKUgBUh7rDYAlR8wFqleyCC6LKyJyn QFDpCaz8WLxYPumic0+GrRtJfItQuXyaZeOi8= MIME-Version: 1.0 Received: by 10.101.24.20 with SMTP id b20mr715604anj.16.1299820395004; Thu, 10 Mar 2011 21:13:15 -0800 (PST) Received: by 10.101.10.31 with HTTP; Thu, 10 Mar 2011 21:13:14 -0800 (PST) Date: Fri, 11 Mar 2011 08:13:14 +0300 Message-ID: From: Subbsd To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-virtualization@freebsd.org Subject: userland utilites for specific version use the current kernel API X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2011 05:42:55 -0000 Hi. i have a server with FreeBSD-current (9) kernel and core system that runs a dozen JAIL-s - some of them is FreeBSD 8x environment , and part is surrounded on FreeBSD 7x. My question - despite on COMPAT_FREEBSD{4-7} in the kernel, utilities do not use those structures in the nucleus, which belong to their release. For example, the utility in jail shows: sockstat: struct xtcpcb size mismatch USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS The COMPAT_FREEBSD is not transparent for utilites? is it possible to have the newer kernel with the minimum of the system, but have JAIL with earlier versions of FreeBSD environments, where it will work sockstat (and perhaps some others)? Thanks From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 11 16:25:50 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 246EA1065675 for ; Fri, 11 Mar 2011 16:25:50 +0000 (UTC) (envelope-from talon@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.freebsd.org (Postfix) with ESMTP id 9B3AE8FC0A for ; Fri, 11 Mar 2011 16:25:49 +0000 (UTC) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.14.4/jtpda-5.4) with ESMTP id p2BG4wEd040712 ; Fri, 11 Mar 2011 17:04:59 +0100 (CET) X-Ids: 164 Received: from niobe.lpthe.jussieu.fr (niobe.lpthe.jussieu.fr [134.157.10.41]) by parthe.lpthe.jussieu.fr (Postfix) with ESMTP id 9FC3320616; Fri, 11 Mar 2011 17:04:57 +0100 (CET) Received: by niobe.lpthe.jussieu.fr (Postfix, from userid 2005) id 2D659405F; Fri, 11 Mar 2011 17:05:02 +0100 (CET) Date: Fri, 11 Mar 2011 17:05:02 +0100 From: Michel Talon To: Steve Franks , freebsd-hackers Message-ID: <20110311160502.GA77107@lpthe.jussieu.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Miltered: at jchkmail.jussieu.fr with ID 4D7A482A.005 by Joe's j-chkmail (http : // j-chkmail dot ensmp dot fr)! X-j-chkmail-Enveloppe: 4D7A482A.005/134.157.10.1/parthe.lpthe.jussieu.fr/parthe.lpthe.jussieu.fr/ X-Mailman-Approved-At: Fri, 11 Mar 2011 16:43:16 +0000 Cc: Subject: Re: looking for mature/efficient gui builder/toolkit/IDE for Python X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2011 16:25:50 -0000 You will not find more easy and good looking for python as: http://en.wikipedia.org/wiki/PyQt -- Michel TALON From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 11 23:07:41 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A3392106566B for ; Fri, 11 Mar 2011 23:07:41 +0000 (UTC) (envelope-from lgj@usenix.org) Received: from lonestar.usenix.org (lonestar.usenix.org [131.106.3.102]) by mx1.freebsd.org (Postfix) with ESMTP id 704038FC14 for ; Fri, 11 Mar 2011 23:07:41 +0000 (UTC) Received: from negroni.usenix.org (negroni.usenix.org [131.106.3.145]) (authenticated bits=0) by lonestar.usenix.org (8.14.2/8.14.2) with ESMTP id p2BN7Aft018647 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Fri, 11 Mar 2011 15:07:41 -0800 (PST) From: Lionel Garth Jones Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Date: Fri, 11 Mar 2011 15:07:40 -0800 Message-Id: <7121A80A-FEB9-4CA2-92FF-1CE2343FCEAE@usenix.org> To: freebsd-hackers@freebsd.org Mime-Version: 1.0 (Apple Message framework v1082) X-Mailer: Apple Mail (2.1082) X-DCC-USENIX-Metrics: lonestar; whitelist X-Spam-Status: No, score=0.7 required=6.0 tests=ALL_TRUSTED, FH_DATE_PAST_20XX autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on lonestar X-Mailman-Approved-At: Fri, 11 Mar 2011 23:29:17 +0000 Subject: WIOV '11 Call for Papers Submission Deadline Approaching X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Mar 2011 23:07:41 -0000 We are writing to remind you that the submission deadline for the 3rd Workshop on I/O Virtualization (WIOV '11) is quickly approaching. Please submit all work by Monday, March 21, 2011, at 11:59 p.m. PST. http://www.usenix.org/wiov11/cfpb/ While I/O systems have a long history of using abstraction to shield applications from the intricacies of devices and storage systems, this workshop's focus is on the interplay of I/O with virtualization and cloud technologies. Our goal is to provide a forum to discuss the impact and challenges of I/O virtualization along multiple dimensions, including: the application domains in which virtualized solutions are used and the subsystems involved; platform or hardware support for I/O virtualization; the virtual machine monitor; the guest operating system; the processor, GPU, memory, and I/O subsystems. Also of interest are higher-level virtualization solutions, such as those concerning API virtualization, new virtual I/O APIs like those used for cloud storage, and innovative approaches to device virtualization in cloud computing and in the mobile domain. The committee also encourages the submission of short papers to update WIOV 11 attendees on new efforts or approaches. Please label such submissions accordingly. Topics of interest include but are not limited to: * Hardware support for I/O virtualization * Novel I/O device architectures for virtualization * New software approaches to I/O virtualization * Hardware and software methods for I/O device emulation * Para-virtualized I/O device driver design for virtualization * Virtual machine monitors' support for I/O * Security and trust considerations for virtualized I/O, including hardware support * Energy-efficient I/O virtualization * Virtualized I/O in the cloud: API virtualization, challenges of large-scale I/O (performance, availability) * Virtualization mechanisms for datacenter networking * I/O monitoring and black box methods or system support for such functions Paper submissions are due Monday, March 21, 2011, at 11:59 p.m. PST. Submissions guidelines and more information can be a found at http://www.usenix.org/wiov11/cfpb/ We look forward to receiving your submissions! On behalf of the WIOV '11 Program Committee, Sanjay Kumar, Intel Labs Himanshu Raj, Microsoft Research Karsten Schwan, Georgia Institute of Technology WIOV '11 Program Co-Chairs wiov11chairs@usenix.org ----------------------------------------------------------------------- WIOV '11 Call for Papers 3rd Workshop on I/O Virtualization (WIOV '11) June 14, 2011, Portland, OR Part of the 2011 USENIX Federated Conferences Week, June 12-17, 2011 http://www.usenix.org/wiov11/cfpb/ Submission deadline: March 21, 2011, 11:59 p.m. PST ------------------------------------------------------------------------ From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 12 07:41:23 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 44A99106566C for ; Sat, 12 Mar 2011 07:41:23 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id 16CF48FC15 for ; Sat, 12 Mar 2011 07:41:22 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p2C7fKjX058719 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 11 Mar 2011 23:41:21 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4D7B23A7.5040708@freebsd.org> Date: Fri, 11 Mar 2011 23:41:27 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.14) Gecko/20110221 Thunderbird/3.1.8 MIME-Version: 1.0 To: Subbsd References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org, freebsd-virtualization@freebsd.org Subject: Re: userland utilites for specific version use the current kernel API X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Mar 2011 07:41:23 -0000 On 3/10/11 9:13 PM, Subbsd wrote: > Hi. > > i have a server with FreeBSD-current (9) kernel and core system that > runs a dozen JAIL-s - some of them is FreeBSD 8x environment , and > part is surrounded on FreeBSD 7x. > My question - despite on COMPAT_FREEBSD{4-7} in the kernel, utilities > do not use those structures in the nucleus, which belong to their > release. For example, the utility in > jail shows: > > sockstat: struct xtcpcb size mismatch > USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS > > The COMPAT_FREEBSD is not transparent for utilites? is it possible to > have the newer kernel with the minimum of the system, but have JAIL > with earlier versions of FreeBSD environments, where it will work > sockstat (and perhaps some others)? Thanks > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > yes there is a small set of system utilities for which this is true. the answer is generally to compile up some staticly linked versions of those tools and supply them in the jail as replacements for the standard ones. a very quick way to do this is to use the ones in /rescue. (note they are all the same binary, don't make multiple copies..) From owner-freebsd-hackers@FreeBSD.ORG Sat Mar 12 20:00:33 2011 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC950106564A for ; Sat, 12 Mar 2011 20:00:33 +0000 (UTC) (envelope-from avg@freebsd.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id B451B8FC13 for ; Sat, 12 Mar 2011 20:00:32 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id WAA29609; Sat, 12 Mar 2011 22:00:31 +0200 (EET) (envelope-from avg@freebsd.org) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1PyUzG-000NXr-T4; Sat, 12 Mar 2011 22:00:30 +0200 Message-ID: <4D7BD0DE.1020301@freebsd.org> Date: Sat, 12 Mar 2011 22:00:30 +0200 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.15) Gecko/20110308 Lightning/1.0b2 Thunderbird/3.1.9 MIME-Version: 1.0 To: Lawrence Stewart References: <4AD6067E.2010503@icyb.net.ua> <4D410609.3000104@freebsd.org> In-Reply-To: <4D410609.3000104@freebsd.org> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: heci: a new driver for review and testing X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Mar 2011 20:00:33 -0000 on 27/01/2011 07:43 Lawrence Stewart said the following: > Hi Andriy, > > On 10/15/09 04:12, Andriy Gapon wrote: >> >> Some time ago I posted some ideas about HECI/MEI driver for FreeBSD: >> http://docs.freebsd.org/cgi/mid.cgi?4968E9A1.3080006 >> >> I actually got around to implementing it (in initial/basic form): >> http://people.freebsd.org/~avg/heci.tgz > > An old thread I know, but I just noticed my desktop has AMT support and > was investigating if I could get access to the Serial Over Lan feature. > I stumbled across your HECI driver and thought I'd give it a spin. It > loads and attaches fine and unloads without issue as well. I tested with > device "HECI_DEV_ID_ICH9_82Q35" on a HP DC7800 minitower machine. I get > no output when I run your "heci-qst" program though. > > Few more details: > > heci0@pci0:0:3:0: class=0x078000 card=0x2819103c chip=0x29b48086 > rev=0x02 hdr=0x00 > vendor = 'Intel Corporation' > device = 'Intel(R) Management Engine Interface (HECI) (Q35-Chipset)' > class = simple comms > > lstewart@lstewart> uname -a > FreeBSD lstewart 8.2-PRERELEASE FreeBSD 8.2-PRERELEASE #0 r217387M: Fri > Jan 14 15:52:16 EST 2011 > lstewart@lstewart:/usr/obj/usr/src/sys/DESKTOP amd64 It's quite possible that your AMT firmware doesn't have a QST module. Upon loading of heci driver it prints a list of discovered module guids. QST guid is 6B5205B9-8185-4519-B889-D98724B58607. But I have mostly gave up on development of this driver (although I keep using it). I think that it's better to port either OpenSolaris or Linux heci driver to FreeBSD. BTW, Intel has released QST SDK source code: http://software.intel.com/en-us/articles/Intel_Quiet_System_Technology_Software_Development_Kit/ -- Andriy Gapon