From owner-freebsd-drivers@freebsd.org Tue Mar 27 09:54:29 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2455CF67565 for ; Tue, 27 Mar 2018 09:54:29 +0000 (UTC) (envelope-from oleksandr.kononov@intel.com) Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "orsmga104.jf.intel.com", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 83B8F7A8AC for ; Tue, 27 Mar 2018 09:54:28 +0000 (UTC) (envelope-from oleksandr.kononov@intel.com) X-Amp-Result: UNSCANNABLE X-Amp-File-Uploaded: False Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga104.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 27 Mar 2018 02:53:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.48,366,1517904000"; d="c'?scan'208,217";a="27881244" Received: from irsmsx104.ger.corp.intel.com ([163.33.3.159]) by fmsmga008.fm.intel.com with ESMTP; 27 Mar 2018 02:53:16 -0700 Received: from irsmsx101.ger.corp.intel.com ([169.254.1.176]) by IRSMSX104.ger.corp.intel.com ([169.254.5.171]) with mapi id 14.03.0319.002; Tue, 27 Mar 2018 10:53:12 +0100 From: "Kononov, Oleksandr" To: "freebsd-drivers@freebsd.org" CC: "Vanco, Juraj" Subject: FreeBSD 11.1 contigfree performance issue Thread-Topic: FreeBSD 11.1 contigfree performance issue Thread-Index: AdPFrddNkK/KuVv6RIG+MvPYgkscuw== Date: Tue, 27 Mar 2018 09:53:11 +0000 Message-ID: <865AA1660A1A014C99D99B800FA40800813681@IRSMSX101.ger.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: yes X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMjI5YTlkMWItNWU0Yy00NTU5LTljZjMtNGQ4YjUwN2M2MDZhIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjIuNS4xOCIsIlRydXN0ZWRMYWJlbEhhc2giOiJ4NjFUdUJxWGtIbk40Q0l0eHR2RnV2Z0poUHkyKys4QXd0OWZtaElTMnhScWEySmQ0VjFLMmhRQ1U3c1wvVWZRMiJ9 dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: multipart/mixed; boundary="_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_" MIME-Version: 1.0 X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2018 09:54:29 -0000 --_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_ Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable I am using FreeBSD 11.1 -RELEASE-amd64 running on a single 32 core CPU and = am having issues with contigmalloc performance. Timing the function using rdtsc shows that it uses up on average about 10 m= illion cycles on that function along. Using the same code, FreeBSD and timing method I ran it on anther machine o= n two CPU's with a total of 32 cores. This gave about 12 thousand cycles on that function. Digging through the source code (on the single CPU) I found that smp_targeted_tlb_shootdown function in /usr/src/sys/x86/x86/mp_x86.c cause the majority of performance hit due to some cores remaining in a paus= ed state longer after the interrupt was send to them. I attached a sample code and Makefile in this email. Steps to recreate (and show rdtsc cycles): $ make $ kldload ./test.ko $ dmesg If anyone has any idea what is the cause of this issue, it would be greatly= appreciated. -------------------------------------------------------------- Intel Research and Development Ireland Limited Registered in Ireland Registered Office: Collinstown Industrial Park, Leixlip, County Kildare Registered Number: 308263 This e-mail and any attachments may contain confidential material for the s= ole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact = the sender and delete all copies. --_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_ Content-Type: application/octet-stream; name="Makefile" Content-Description: Makefile Content-Disposition: attachment; filename="Makefile"; size=317; creation-date="Tue, 27 Mar 2018 09:51:01 GMT"; modification-date="Tue, 27 Mar 2018 09:20:17 GMT" Content-Transfer-Encoding: base64 IyBOb3RlOiBJdCBpcyBpbXBvcnRhbnQgdG8gbWFrZSBzdXJlIHlvdSBpbmNsdWRlIHRoZSA8YnNk Lmttb2QubWs+IG1ha2VmaWxlIGFmdGVyIGRlY2xhcmluZyB0aGUgS01PRCBhbmQgU1JDUyB2YXJp YWJsZXMuCgojIERlY2xhcmUgTmFtZSBvZiBrZXJuZWwgbW9kdWxlCktNT0QgPSAgIHRlc3QKIyBF bnVtZXJhdGUgU291cmNlIGZpbGVzIGZvciBrZXJuZWwgbW9kdWxlClNSQ1MgPSAgIHRlc3QuYwoK IyBJbmNsdWRlIGtlcm5lbCBtb2R1bGUgbWFrZWZpbGUKIyAvdXNyL3NyYy9zaGFyZS9tay9ic2Qu a21vZC5tawouaW5jbHVkZSA8YnNkLmttb2QubWs+Cgo= --_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_ Content-Type: text/plain; name="test.c" Content-Description: test.c Content-Disposition: attachment; filename="test.c"; size=1523; creation-date="Tue, 27 Mar 2018 09:50:54 GMT"; modification-date="Tue, 27 Mar 2018 09:20:26 GMT" Content-Transfer-Encoding: base64 I2luY2x1ZGUgPHN5cy9wYXJhbS5oPgojaW5jbHVkZSA8c3lzL21vZHVsZS5oPgojaW5jbHVkZSA8 c3lzL2tlcm5lbC5oPgojaW5jbHVkZSA8c3lzL3N5c3RtLmg+CiNpbmNsdWRlIDxzeXMvbWFsbG9j Lmg+CgpzdGF0aWMgdm9pZCBydW5UZXN0KHZvaWQpOwpNQUxMT0NfREVDTEFSRShNX0ZPTyk7Ck1B TExPQ19ERUZJTkUoTV9GT08sICJtZm9vIiwgImZvbyBtZW0iKTsKCi8qIFRoZSBmdW5jdGlvbiBj YWxsZWQgYXQgbG9hZC91bmxvYWQuICovCnN0YXRpYyBpbnQgZXZlbnRfaGFuZGxlcihzdHJ1Y3Qg bW9kdWxlICptb2R1bGUsIGludCBldmVudCwgdm9pZCAqYXJnKSB7CiAgICBpbnQgZSA9IDA7IC8q IEVycm9yLCAwIGZvciBub3JtYWwgcmV0dXJuIHN0YXR1cyAqLwoKICAgIHN3aXRjaCAoZXZlbnQp IHsKICAgIGNhc2UgTU9EX0xPQUQ6CglydW5UZXN0KCk7CiAgICAgICAgYnJlYWs7CiAgICBjYXNl IE1PRF9VTkxPQUQ6CglwcmludGYoIkRvbmVcbiIpOwoJdXByaW50ZigiRG9uZVxuIik7CiAgICAg ICAgYnJlYWs7CiAgICBkZWZhdWx0OgogICAgICAgIGUgPSBFT1BOT1RTVVBQOyAvKiBFcnJvciwg T3BlcmF0aW9uIE5vdCBTdXBwb3J0ZWQgKi8KICAgICAgICBicmVhazsKICAgIH0KCiAgICByZXR1 cm4oZSk7Cn0KCnN0YXRpYyB2b2lkIHJ1blRlc3QoKSB7CgoJY29uc3QgaW50IHBhZ2VTaXplID0g NDA5NjsKICAgICAgICB1bnNpZ25lZCBsb25nIHNpemU7CiAgICAgICAgdm9pZCAqcHRyID0gTlVM TDsKCXVuc2lnbmVkIGxvbmcgbG9uZyB0MSwgdDI7CgogICAgICAgIGZvcihzaXplID0gMTsgc2l6 ZSA8PSAocGFnZVNpemUgLTEpOyBzaXplIDw8PSAxKSB7CgoJCXQxID0gcmR0c2MoKTsKICAgICAg ICAgICAgICAgIHB0ciA9IGNvbnRpZ21hbGxvYyhzaXplLCBNX0ZPTywgTV9XQUlUT0ssIDAsIH4w LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHBhZ2VTaXplLCAwKTsKCQl0MiA9 IHJkdHNjKCk7CgkJcHJpbnRmKCJjb250aWdtYWxsb2MgY3ljbGVzOiAlbGx1XG4iLCB0Mi10MSk7 CgogICAgICAgICAgICAgICAgaWYocHRyID09IE5VTEwpCiAgICAgICAgICAgICAgICAgICAgICAg IHJldHVybjsKCgkJdDEgPSByZHRzYygpOwogICAgICAgICAgICAgICAgY29udGlnZnJlZShwdHIs IHNpemUsIE1fRk9PKTsKCQl0MiA9IHJkdHNjKCk7CgkJcHJpbnRmKCJjb250aWdmcmVlIGN5Y2xl czogJWxsdVxuXG4iLCB0Mi10MSk7CgkJdXByaW50ZigiTG9vcCBjb21wbGV0ZVxuIik7CiAgICAg ICAgfQp9CgovKiBUaGUgc2Vjb25kIGFyZ3VtZW50IG9mIERFQ0xBUkVfTU9EVUxFLiAqLwpzdGF0 aWMgbW9kdWxlZGF0YV90IHRlc3RfY29uZiA9IHsKICAgICJ0ZXN0IiwgICAgLyogbW9kdWxlIG5h bWUgKi8KICAgICBldmVudF9oYW5kbGVyLCAgLyogZXZlbnQgaGFuZGxlciAqLwogICAgIE5VTEwg ICAgICAgICAgICAvKiBleHRyYSBkYXRhICovCn07CgpERUNMQVJFX01PRFVMRSh0ZXN0LCB0ZXN0 X2NvbmYsIFNJX1NVQl9EUklWRVJTLCBTSV9PUkRFUl9NSURETEUpOwo= --_005_865AA1660A1A014C99D99B800FA40800813681IRSMSX101gercorpi_-- From owner-freebsd-drivers@freebsd.org Tue Mar 27 10:26:12 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CC11F6AFFA for ; Tue, 27 Mar 2018 10:26:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E1A177C321 for ; Tue, 27 Mar 2018 10:26:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w2RAQ0Co030191 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 27 Mar 2018 13:26:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w2RAQ0Co030191 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w2RAQ0qZ030189; Tue, 27 Mar 2018 13:26:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 27 Mar 2018 13:26:00 +0300 From: Konstantin Belousov To: "Kononov, Oleksandr" Cc: "freebsd-drivers@freebsd.org" , "Vanco, Juraj" Subject: Re: FreeBSD 11.1 contigfree performance issue Message-ID: <20180327102600.GY76926@kib.kiev.ua> References: <865AA1660A1A014C99D99B800FA40800813681@IRSMSX101.ger.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <865AA1660A1A014C99D99B800FA40800813681@IRSMSX101.ger.corp.intel.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2018 10:26:12 -0000 On Tue, Mar 27, 2018 at 09:53:11AM +0000, Kononov, Oleksandr wrote: > > I am using FreeBSD 11.1 -RELEASE-amd64 running on a single 32 core CPU and am having issues with contigmalloc performance. > Timing the function using rdtsc shows that it uses up on average about 10 million cycles on that function along. > Using the same code, FreeBSD and timing method I ran it on anther machine on two CPU's with a total of 32 cores. > This gave about 12 thousand cycles on that function. > > Digging through the source code (on the single CPU) I found that > > smp_targeted_tlb_shootdown function in /usr/src/sys/x86/x86/mp_x86.c > cause the majority of performance hit due to some cores remaining in a paused state longer after the > interrupt was send to them. > > > I attached a sample code and Makefile in this email. > > Steps to recreate (and show rdtsc cycles): > > $ make > $ kldload ./test.ko > $ dmesg > > If anyone has any idea what is the cause of this issue, it would be greatly appreciated. You noted that CPUs have very long time coming out of the idle state. What is the output of sysctl dev.cpu ? In particular, look at the cx_* MIBs and if you have configured deep sleep modes, try to step it back. Also look at what idle method (cx_method) is used, and if MWAIT is problematic, change to the legacy idling, see acpi(4) and debug.acpi.disable="mwait" knob. What are those CPUs ? Wouldn't such high latency of waking up cores make the interactive performance of the machine a mess already. In principle, we can avoid sending shutdowns to the idle cores, instead doing global TLB flush on wakeup of the core if some percpu flag is set. But as I noted in the paragraph before, it should be quite bad even with IPI latency fixed, for other reasons. From owner-freebsd-drivers@freebsd.org Tue Mar 27 16:50:39 2018 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 123FFF69DCF for ; Tue, 27 Mar 2018 16:50:39 +0000 (UTC) (envelope-from lakhanshiva@gmail.com) Received: from mail-oi0-x244.google.com (mail-oi0-x244.google.com [IPv6:2607:f8b0:4003:c06::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 99153716CD for ; Tue, 27 Mar 2018 16:50:38 +0000 (UTC) (envelope-from lakhanshiva@gmail.com) Received: by mail-oi0-x244.google.com with SMTP id 71-v6so19774815oie.12 for ; Tue, 27 Mar 2018 09:50:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=9eYe0s4/BJCV1HWJGuQAynUf8k6k9aJVrKs9iXu7HBY=; b=XMF9qADUaEndKIN/KIuDKuiwp3JT9JDhUf7ZcZEDxDXjm02AKmtoPV7S3WKzaJBfH7 7MjHCUVD2HRiVctR7jdAnaBBHOsRGalCTxFGt88OfPGiMEbXUj8Kear7GAdqE0sqlzIP 82mpp/G9if4igoOwK4didTc9SWDtDqE2Lj2N0IeCK4JK/N6cWwAp3YTEj7N3ad3IDGt1 x0MW8TUExRzGlchVePVRhJTbcb9z4X6/jFXf/SkbfsnVl6EIZ8YE4BAe44ckxSwcn6hG OBJNgezJqeag3bUIcOixs4azgkww/VpLUjsHxF2PfR25HR7Aj5j/qD0SD1e5HqxvI3cY Jqyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=9eYe0s4/BJCV1HWJGuQAynUf8k6k9aJVrKs9iXu7HBY=; b=QHf3kI9OqpXXtCfZSJXpPG2MsotT1kwKrEd7Ea7d0gmtYJYwp1RpXLW8P40qrnAO9V H/9ZqS9YnLHc8osrrgR5ZST+IQP6Vgvp6ZUU1pVcNTzLFXXJb0Pbj9P9Ph6bK46TCXND mRflztSqGsXPFw7MG+QJNWOPZd/+kI8l3nlfrmR0JhN7eJESIpRpoXJ9i2Nwiek56H3m gcBkjTTGjLCTbzHWpfA9hNJizXHiHxjfj2+lD3x0KBaq2M2qc6HyK01jAe/mGgWHkV1v /A9ve8O7BfH9y14mlJGMO4CEMURlsQVkDhPHg7T2dfes6D+5ODhQJZXNxSV+34HX93a/ 3kCQ== X-Gm-Message-State: AElRT7HV2WsHD6hXjO5VSScctcBwTrjDWKeEAZ/DEA4w/JZl6xxHe1Oq WPP03w/2P2wYhyb2Kw0+0Z4tF8UqX8I1IuaEUDEhr9TF X-Google-Smtp-Source: AIpwx49gnN2nfMXcfIsG7Ma2fFGBTtl5dW5e6WkJSEej+J+vlpEYq51dh6EFweiCn6/A75S7jyzQijnIZVOOFoHUb4s= X-Received: by 10.202.48.21 with SMTP id w21mr51531oiw.131.1522169437568; Tue, 27 Mar 2018 09:50:37 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a9d:44a8:0:0:0:0:0 with HTTP; Tue, 27 Mar 2018 09:50:36 -0700 (PDT) From: Lakhan Shiva Date: Tue, 27 Mar 2018 22:20:36 +0530 Message-ID: Subject: Convert all PCI drivers to table driven and use MODULE_PNP_INFO To: freebsd-drivers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Mar 2018 16:50:39 -0000 Hi Everyone! My name is Lakhan Kamireddy. I am a second year graduate student at University of Colorado Boulder. I am pursuing the project to "Convert all PCI driver attachments to be table driven and mark with PNP_INFO" A little bio about me: I am doing MSEE at University of Colorado Boulder. Kindly find my proposal for GSOC here: https://docs.google.com/document/d/1bB_2Hgadg-3CEgashcM85-MzIOqRXT9wIEEfAstbPlU/edit?usp=sharing I would like to start working on converting PCI drivers that already have a table but font have a MODULE_PNP_INFO. I would like to know more about the branching strategy and the process to check in first piece of code. I am uising FreeBSD VM current image. I will be using SVN for checking out and checking in code from the VM. Regards, Lakhan