From owner-freebsd-stable@freebsd.org Wed Oct 31 17:13:33 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57A7910DF990 for ; Wed, 31 Oct 2018 17:13:33 +0000 (UTC) (envelope-from juraj.vanco@intel.com) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "fmsmga103.fm.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF48E7F1F6 for ; Wed, 31 Oct 2018 17:13:32 +0000 (UTC) (envelope-from juraj.vanco@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 31 Oct 2018 10:13:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,447,1534834800"; d="scan'208";a="83045260" Received: from irsmsx106.ger.corp.intel.com ([163.33.3.31]) by fmsmga008.fm.intel.com with ESMTP; 31 Oct 2018 10:13:24 -0700 Received: from irsmsx103.ger.corp.intel.com ([169.254.3.253]) by IRSMSX106.ger.corp.intel.com ([169.254.8.163]) with mapi id 14.03.0415.000; Wed, 31 Oct 2018 17:13:24 +0000 From: "Vanco, Juraj" To: Konstantin Belousov , "Bennett, Ciunas" CC: "freebsd-stable@freebsd.org" Subject: RE: Possible memory leak in the kernel (contigmalloc) Thread-Topic: Possible memory leak in the kernel (contigmalloc) Thread-Index: AdRrtbzKvvQI2iPZToSPdiaEvmHIXQBqhpgAALREjJAAAgPuAAABiKNgABAShYAALsIvoA== Date: Wed, 31 Oct 2018 17:13:23 +0000 Message-ID: <996B37956731CF40B1A674B085ACCE9440F747CA@IRSMSX103.ger.corp.intel.com> References: <770FD3608C9E864796AB46CB37B561B1BDFCF0CD@IRSMSX101.ger.corp.intel.com> <20181026201230.GV5335@kib.kiev.ua> <770FD3608C9E864796AB46CB37B561B1BE0B4096@irsmsx105.ger.corp.intel.com> <20181030101150.GN5335@kib.kiev.ua> <770FD3608C9E864796AB46CB37B561B1BE0B5562@irsmsx105.ger.corp.intel.com> <20181030183557.GP5335@kib.kiev.ua> In-Reply-To: <20181030183557.GP5335@kib.kiev.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiZDUwNmYxMGItYzUzYi00ZDg3LWE1YTAtZjExODg4NDQ5ZDhiIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiMTNMdTNhUkdYOTFTbGZxNUVscDZcL1o4TzREU0tIXC8zSEVDYmIzZ09hNVwvVW0rRnNZSUhTd2wwcVdsUW1QWjVLbiJ9 dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [163.33.239.180] Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 31 Oct 2018 17:13:33 -0000 Hi Konstatin, what we can directly see, is that the number of inactive pages increases an= d this happens after running single loop of contigmalloc -> contigfree even= with the repeating size of allocated blocks. The only source of inactive page could come from kernel memory management c= ontrol structures. Even if such structure becomes once inactive (cached pages), I do not see a= ny reason that kernel cannot reclaim these inactive cached pages when again= the contigmalloc asks for the same space of memory. I think this cycle should not take all the amount of physical memory withou= t being able to reclaim back when it's out of memory: int array0[10] =3D {2097152, 1024, 2101248, 1024, 2097152, 2101248, 210= 1248, 2097152, 2101248, 1024}; int array1[10] =3D {1024, 2101248, 2097152, 2101248, 2101248, 2097152, = 1024, 2101248, 1024, 2097152}; void *mem_blocks[10]; for (i =3D 0; i < 10; i++) mem_blocks[i] =3D contigmalloc(array0[i], TEST_MEM, 0, 0, ~0, PAGE_= SIZE, 0); for (i =3D 0; i < 10; i++) contigfree(mem_blocks[i], array0[i], TEST_MEM); for (i =3D 0; i < 10; i++) mem_blocks[i] =3D contigmalloc(array1[i], TEST_MEM, 0, 0, ~0, PAGE_= SIZE, 0); for (i =3D 0; i < 10; i++) contigfree(mem_blocks[i], array1[i], TEST_MEM); jv -----Original Message----- From: owner-freebsd-stable@freebsd.org [mailto:owner-freebsd-stable@freebsd= .org] On Behalf Of Konstantin Belousov Sent: Tuesday, October 30, 2018 6:36 PM To: Bennett, Ciunas Cc: freebsd-stable@freebsd.org Subject: Re: Possible memory leak in the kernel (contigmalloc) On Tue, Oct 30, 2018 at 11:15:47AM +0000, Bennett, Ciunas wrote: > Hi, > The only other activity that is running is the csh script that is insert= ing and removing the kernel module. > I am not using the VM for any other purpose but to run this test. > In my tests the correlation between memory allocation and moving to inact= ive list can be seen. > I don't think any other process is creating the inactive memory. But it is created. More, since anon private memory is freed (not deactivat= ed) on the process exit, something is accumulating the memory. The other possibility is that the memory is the caching pages from vnodes, = but for this buffers must be created and then reclaimed, which would sugges= t even more activity on the system. > Thanks. > = > -----Original Message----- > From: Konstantin Belousov [mailto:kostikbel@gmail.com] > Sent: Tuesday, October 30, 2018 10:12 AM > To: Bennett, Ciunas > Cc: freebsd-stable@freebsd.org > Subject: Re: Possible memory leak in the kernel (contigmalloc) > = > On Tue, Oct 30, 2018 at 09:46:59AM +0000, Bennett, Ciunas wrote: > > Hi, > > = > > I was debugging the issue by viewing the free ques "sysctl = > > vm.phys_free" and also using "show page" in ddb. The inactive memory = > > is never being released back into the free que. I thought that when = > > inactive memory reaches a certain threshold that the kernel will = > > start reclaiming and move it to the free list? In my program this is = > > not happening, the program uses free memory (contigmalloc), and then = > > it is put into the inactive que (contiigfree) when the program frees it. > Contigfree() does not release memory into inactive queue. By definition,= inactive pages have valid content, which is not possible for the pages fre= ed by contigfree(). > contigfree()->kmem_free()->kmem_unback()->vm_page_free(). > = > = > > This inactive memory is never released by the kernel, and the = > > inactive que grows until all the memory is in this que. I have = > > attached a xml sheet that shows the memory usage in the system. > Inactive memory is not freed, it makes no sense as far as there is = > valid content, which is either not recoverable (anon memory or dirty = > file > pages) or high-cost to restore (clean file pages, need to re-read from di= sk). Inactive is processed by pagedaemon when system has memory deficit, a= nd either inactive pages are written to swap, or written to their file back= ing storage. > = > I suspect that you have other activities on your system going on, which c= ause creation of the inactive memory and unrecoverable fragmentation. > Note that contigmalloc() tries to do defragmentation to satisfy requests,= but this is not always possible. > = > = > > Ciunas. > > = > > -----Original Message----- > > From: Konstantin Belousov [mailto:kostikbel@gmail.com] > > Sent: Friday, October 26, 2018 9:13 PM > > To: Bennett, Ciunas > > Cc: freebsd-stable@freebsd.org > > Subject: Re: Possible memory leak in the kernel (contigmalloc) > > = > > On Wed, Oct 24, 2018 at 04:27:52PM +0000, Bennett, Ciunas wrote: > > > Hello, > > > = > > > I have encountered an issue with a kernel application that I have = > > > written, the issue might be caused by a memory leak in the kernel. > > > The application allocates and deallocates contiguous memory using > > > contigmalloc() and contigfree(). The application will fail after a = > > > period of time because there is not enough free contiguous memory = > > > left. There could be an issue with the freeing of memory when = > > > using the contigfree() function. > > > > > = > > It is unlikely that there is an issue with a leak, but I would be not s= urprised if your allocation/free pattern would cause fragmentation on free = lists that results in contigmalloc(9) failures after. > > = > > Look at the vmstat -z/vmstat -m output to see uma and malloc stats. > > More interesting for your case can be the output from > > sysctl vm.phys_free > > which provides information about the free queues and order of free page= s on them. > > -------------------------------------------------------------- > > Intel Research and Development Ireland Limited Registered in Ireland = > > Registered Office: Collinstown Industrial Park, Leixlip, County = > > Kildare Registered Number: 308263 > > = > > = > > This e-mail and any attachments may contain confidential material = > > for the sole use of the intended recipient(s). Any review or = > > distribution by others is strictly prohibited. If you are not the = > > intended recipient, please contact the sender and delete all copies. > = > = > -------------------------------------------------------------- > Intel Research and Development Ireland Limited Registered in Ireland = > Registered Office: Collinstown Industrial Park, Leixlip, County = > Kildare Registered Number: 308263 > = > = > This e-mail and any attachments may contain confidential material for = > the sole use of the intended recipient(s). Any review or distribution = > by others is strictly prohibited. If you are not the intended = > recipient, please contact the sender and delete all copies. > = _______________________________________________ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" -------------------------------------------------------------- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the s= ole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact = the sender and delete all copies.