From owner-freebsd-hackers@freebsd.org Mon Nov 19 12:47:19 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 531A711029DA for ; Mon, 19 Nov 2018 12:47:19 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: from mail-ot1-x342.google.com (mail-ot1-x342.google.com [IPv6:2607:f8b0:4864:20::342]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 41BC0740B6; Mon, 19 Nov 2018 12:47:18 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: by mail-ot1-x342.google.com with SMTP id a11so23842567otr.10; Mon, 19 Nov 2018 04:47:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=s88MjzisMAdlXVyBoiQJJI73aj+mBbz9PchaDenqvKk=; b=YSw9AB0+rz/+RodAmCexR4YS1gF71nAvRrOUZPr2UlDgWCnCpCDvyI3A/G6drLD9Ez d77tCHJ6pMzeBAql2SfWKv/xLoqbOgOkWb1iOgO8rjuYDT3jv/Bzoq1Yo7MuSeCPbSQl zLZe0omp8Ie3pvlepPRtlbpAlMeTi11gWgxLfH6onPFhGEsSZp7u2xFSaeos4hnXjPj0 4XlkI9Zn/5Pq+q9O657kShzr9bwAOE7GWT/VdCBKRz72gyAVbzz8ApJ8DdOXDkpHPz2O 2Pp46hzSvwkoVJR33pOzOc8exikSlKLApIFr0OqoelPfd7S/K00ntMYn7F39/I55gal9 KFFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=s88MjzisMAdlXVyBoiQJJI73aj+mBbz9PchaDenqvKk=; b=WF2UkHNfIaCrsoXcuVUvmDT8Efw8BlmHhLtxGF3Pa9RKeOYaTf3rQ+kppbpcFQvScc zq4hIidKvghzLshkBn5Lzac/BgylIV6NnQBRbwOtMmvsu/0ASXr4LLzA9lwy9/JmbXRt DCI5XYSqAwjy1JBkqOXnjDgC8xE+QRwDc6NgxJLznYLm896mE9l2ws+glkBOeNgBpiPf CitdOKsyV5n5mSideJHXHcFNCaqLzG6FgQG1g3yrbcIyyTGgF5o8nALnuoLO1CCvd+28 ArrLM+HfuS2yDB+vx2GjlczLy0DYgDEgTmgo13sQaMc/zH9PeaQwDYaOkYw/udKmOwUb Neog== X-Gm-Message-State: AA+aEWZAxYOJqcpxYePXZhMkT/EyjqsBsslqBmRoL/LPIN3mScHR/H7i +6CsfFQCvz0887M5gMHvbncK+ROue099YYa5zsw+kNMxfwI= X-Google-Smtp-Source: AFSGD/VpaWHgW+GhriZWbF7NF5TJ+/W23cmDLGrjFvBWgkwiF0hduAt7ZiK85DqvQ6YpVlJ7lON2Ouht3xgw6TDV2q0= X-Received: by 2002:a9d:1541:: with SMTP id z1mr181249otz.305.1542631637387; Mon, 19 Nov 2018 04:47:17 -0800 (PST) MIME-Version: 1.0 References: <7bb8463e-aa5d-bade-7ae7-6c612bae65f9@gmail.com> <0f7e583f-9ad9-f210-4bdb-1ace13ab8c5f@freebsd.org> In-Reply-To: <0f7e583f-9ad9-f210-4bdb-1ace13ab8c5f@freebsd.org> From: Steevan Rodrigues Date: Mon, 19 Nov 2018 18:17:05 +0530 Message-ID: Subject: Re: FreeBSD 11.x thread creation time is 9000+ microseconds on Intel Xeon Gold series CPU To: se@freebsd.org Cc: mad@madpilot.net, freebsd-hackers@freebsd.org, =?UTF-8?Q?V=C3=A1clav_Haisman?= X-Rspamd-Queue-Id: 41BC0740B6 X-Spamd-Result: default: False [-3.07 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.954,0]; R_DKIM_ALLOW(-0.20)[gmail.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(-0.31)[ip: (2.78), ipnet: 2607:f8b0::/32(-2.52), asn: 15169(-1.70), country: US(-0.09)]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.80)[-0.804,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Server: mx1.freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Nov 2018 12:47:19 -0000 I tried FreeBSD 12 Beta 3 version on this server with Xeon Gold 5115 CPU. All these problems have disappeared. Thread creation time has improved greatly . It is now below 100 usec ( in FreeBSD 11.x 9000+ usec) Also I had another issue with FreeBSD 11.x which related to contigmalloc and contigfree . Actually contigfree was taking too much time on FreeBSD 11.x on this same server with Xeon Gold 5115 CPU. In FreeBSD 12 Beta3 also contigfree takes much more time compared to contigmalloc. However when I compare the values to FreeBSD 11.x number I can see huge improvement in FreeBSD 12 Beta 3 . Because of this contigfree issue my driver unload used to take 5 to 20 minutes in FreeBSD 11.x. Now my driver takes only a few seconds to load and a few seconds to unload in FreeBSD 12. BEta 3. Hence it looks like the problem is with FreeBSD 11.x. I am still waiting for final release version of FreeBSD 12.0 Regards, Steevan On Tue, Nov 6, 2018 at 6:18 PM Stefan Esser wrote: > Am 06.11.18 um 12:05 schrieb Steevan Rodrigues: > > Thanks to all of you for the quick responses. > > > > I see this issue only on a particular server. > > I do not see this issue on another server with slightly older processor > > Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (dual socket 6 core per > socket) > > In this system the same program takes only about 800 usec > > > > I wonder whether it is problem with 11.x FreeBSD release on this > particular > > processor. > > I came across two results (given below) from Phoronix OSbench test suite > > which also show poor thread creation time for FreeBSD 11.x > > > > https://openbenchmarking.org/result/1804094-AR-1804096AR58 > > https://openbenchmarking.org/result/1807114-RA-WINDOWSSE47 > > > > In this look for "Create Threads" row. > > This list shows, that FreeBSD-12 is fast, only FreeBSD-11 shows the long > thread creation time (16000 vs 36 in whatever units), even though > FreeBSD-12 > was running with KPTI-Patches. > > This lets me think that the issue that caused these delays has been fixed, > but not merged to FreeBSD-11, yet.# > > I guess that the cause for the delays is the synchronization IPIs between > sockets, and that these might be very high due to some cores in deep sleep > modes (with corresponding long wake-up delays). > > Regards, STefan >