From owner-freebsd-hackers@FreeBSD.ORG Fri Jul 26 07:45:56 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 23F9AEE8 for ; Fri, 26 Jul 2013 07:45:56 +0000 (UTC) (envelope-from h.tugrul.erdogan@gmail.com) Received: from mail-ve0-x232.google.com (mail-ve0-x232.google.com [IPv6:2607:f8b0:400c:c01::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D80092A3E for ; Fri, 26 Jul 2013 07:45:55 +0000 (UTC) Received: by mail-ve0-f178.google.com with SMTP id ox1so396508veb.9 for ; Fri, 26 Jul 2013 00:45:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=7BejTPjwrJ7F+apJV+1KPAXaQbYv8Aad5zGpmzlbcnc=; b=Z+YjynAvIQSOACXF1xL9b96/MVUM4+qv9mw95V+G98Ev0G59QJGO/dtC0N90poOeVv AQ4/u2WCY3JZZe023+7AqgA/SDdxXxoHEkMDHcdEiATLgKpouvT81Xm2vNY71R0wu5Zj N0yAt4DzkKkFxgGaq6MV6w32vDqQwOpI5EmefFheP8UyaPMt3N5QK9kj1a7jwYL/2GRf htK8J2zEgLfYvyQ5+CO8jLg4udu0BIusqJOYm6KQ7p204XI7N8DBzgLGE8Cw06Tl77KR A1EOYBe28rjHlwvmt9sFDZulpfUDriGZyboMoeKVg0k38Fy2bUt5w1Fv4BIu+hEKFA3n +rfQ== MIME-Version: 1.0 X-Received: by 10.220.175.84 with SMTP id w20mr2563370vcz.84.1374824754948; Fri, 26 Jul 2013 00:45:54 -0700 (PDT) Received: by 10.58.54.103 with HTTP; Fri, 26 Jul 2013 00:45:54 -0700 (PDT) In-Reply-To: References: Date: Fri, 26 Jul 2013 10:45:54 +0300 Message-ID: Subject: Re: panic: kmem_map too small at heavy packet traffic From: Tugrul Erdogan To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jul 2013 07:45:56 -0000 Specifically, I am taking this panic when doing ip spoof attack while syn-proxy activated. The output of system arguments below: kern.malloc_count: 315 vm.md_malloc_wait: 0 vfs.bufmallocspace: 0 vfs.maxmallocbufspace: 86269952 vm.kmem_size: 16686845952 vm.kmem_size_min: 0 vm.kmem_size_max: 329853485875 vm.kmem_size_scale: 1 vm.kmem_map_size: 543973376 vm.kmem_map_free: 15974895616 kern.maxvnodes: 350097 kern.minvnodes: 87524 vfs.numvnodes: 112329 vfs.wantfreevnodes: 87524 vfs.freevnodes: 87502 [root@ ~]# pfctl -si No ALTQ support in kernel ALTQ related functions disabled Status: Enabled for 0 days 00:17:39 Debug: Urgent State Table Total Rate current entries 5142886 searches 26982141 25478.9/s inserts 29055053 27436.3/s removals 24218654 22869.4/s Counters match 24901305 23514.0/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 18 0.0/s proto-cksum 0 0.0/s state-mismatch 0 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 29378439 27741.7/s [root@~]# panic: kmem_malloc(-1 814 425 600): kmem_map too small: 543 956 992 to tal allocated cpuid = 8 Uptime: 1d18h2m14s (ada0:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada0:ahcich1:0:0:0): CAM status: CCB request is in progress (ada0:ahcich1:0:0:0): Error 5, Retries exhausted (ada0:ahcich1:0:0:0): Synchronize cache failed (ada1:ahcich2:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada1:ahcich2:0:0:0): CAM status: CCB request is in progress (ada1:ahcich2:0:0:0): Error 5, Retries exhausted (ada1:ahcich2:0:0:0): Synchronize cache failed Dumping 8243 out of 16357 MB:..1%..11% On Thu, Jul 25, 2013 at 5:57 PM, Tugrul Erdogan wrote: > howdy all, > > At my work, I am using 10.0-CURRENT on Intel(R) Xeon(R) E5620 with 16GB > ram. I am taking > > "panic: kmem_malloc(-548663296): kmem_map too small: 539459584 total allocated" > > message with configuration below: > > [root@ ~]# sysctl vm.kmem_size_min vm.kmem_size_max vm.kmem_size vm.kmem_size_scale > > vm.kmem_size_min: 0 > vm.kmem_size_max: 329853485875 > vm.kmem_size: 16686845952 > vm.kmem_size_scale: 1 > [root@ ~]# sysctl hw.physmem hw.usermem hw.realmem > hw.physmem: 17151787008 > hw.usermem: 8282652672 > > hw.realmem: 18253611008 > [root@ ~]# sysctl hw.pagesize hw.pagesizes hw.availpages > hw.pagesize: 4096 > hw.pagesizes: 4096 2097152 0 > hw.availpages: 4187448 > > > When I compare vmstat and netstat output of boot time result and subsequent result, the major difference are seemed at: > > pf_temp 0 0K - 79309736 128 | pf_temp 1077640 134705K - 84330076 128 > > and after the panic at the core dump file the major vmstat difference is: > > temp 110 15K - 76212305 16,32,64,128,256 | temp 117 6742215K - 655115 16,32,64,128,2 > > When I explore the source code of kernel (at vm_kern.c and vm_map.c), I > see that the panic can occur with the cases at below: > > * negative malloc size parameter > > * longer than free buffer respect to kmem_map min_offset and max_offset > values > > * try to allocate when the root entry of map is the rightmost entry of map > > * try to allocate bigger than map's max_free value > > I think the panic occurs at mbuf creation process when calling malloc() as > a result of couldn't be able to allocate memory; but I don't understand why > one of this panic case activating? The memory is almost empty but the > device is saying kmem_map small when using about 0.5GB memory purely. How > can i solve this panic problem? > > Thank you all for your time. > > -- Best Wishes, > > Tugrul Erdogan > >