From nobody Sun Mar 19 01:01:49 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PfKMV04Ycz40FM8 for ; Sun, 19 Mar 2023 01:01:58 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Received: from www121.sakura.ne.jp (www121.sakura.ne.jp [153.125.133.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PfKMS38Pjz4HZd for ; Sun, 19 Mar 2023 01:01:55 +0000 (UTC) (envelope-from junchoon@dec.sakura.ne.jp) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of junchoon@dec.sakura.ne.jp has no SPF policy when checking 153.125.133.21) smtp.mailfrom=junchoon@dec.sakura.ne.jp; dmarc=none Received: from kalamity.joker.local (123-1-88-210.area1b.commufa.jp [123.1.88.210]) (authenticated bits=0) by www121.sakura.ne.jp (8.16.1/8.16.1/[SAKURA-WEB]/20201212) with ESMTPA id 32J11oxT003717 for ; Sun, 19 Mar 2023 10:01:51 +0900 (JST) (envelope-from junchoon@dec.sakura.ne.jp) Date: Sun, 19 Mar 2023 10:01:49 +0900 From: Tomoaki AOKI To: freebsd-hackers@freebsd.org Subject: Re: splash(4) support in vt Message-Id: <20230319100149.df3c27607af87db372011c92@dec.sakura.ne.jp> In-Reply-To: References: <20230316092007.6a2695e995f5e4c589140886@bidouilliste.com> Organization: Junchoon corps X-Mailer: Sylpheed 3.7.0 (GTK+ 2.24.33; amd64-portbld-freebsd13.2) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-1.59 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; AUTH_NA(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:7684, ipnet:153.125.128.0/18, country:JP]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_SPF_NA(0.00)[no SPF record]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DMARC_NA(0.00)[sakura.ne.jp]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4PfKMS38Pjz4HZd X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On Sat, 18 Mar 2023 22:56:09 +0300 Gleb Popov wrote: > Thanks for the pointers! > > It just occurred to me that splash only hides kernel messages, but > after init(8) starts the image disappears and the textual boot > sequence is resumed. What are the approaches to solve this? Should we > handoff rendering to some usermode program? Would that be seamless? So maybe a R/W tunable that enables/disables screen output is needed. And init(8) SHALL not change it (leave it for rc.d script or something). If admins/vendors wants splash kept until desktop environment (mate or anything admins/vendors want), disable it on /boot/loader.conf and enable it after DE starts up (and/or X/Wayland is terminated). Possibly, switching between vtys to forcibly enable the writable tunable would help. This way, users can use vtys (including console [vty1]) after startup finishes or crashes, as X is historically tied to vty9. (Not sure for Wayland, though.) Without something like that, users cannot see console/vtys after logging out from DE or switching to any vty while DE is running, if no serial console is available. Note that sc worked as expected because sc doesn't touch graphic flamebuffers at all (pure, hardware text mode), but vt (including underlying efifb) renders texts to graphics FB, thus breaks splash on any screen output. And maybe some changes would be needed to log currently not logged console outputs. (Driver errors?) Without it, admins has no way to check such outputs when any error happenes. -- Tomoaki AOKI From nobody Mon Mar 20 16:49:50 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PgLLq1C2Nz3yxnH for ; Mon, 20 Mar 2023 16:49:55 +0000 (UTC) (envelope-from jrm@ftfl.ca) Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PgLLn424Tz4GDC for ; Mon, 20 Mar 2023 16:49:53 +0000 (UTC) (envelope-from jrm@ftfl.ca) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of jrm@ftfl.ca designates 209.85.160.176 as permitted sender) smtp.mailfrom=jrm@ftfl.ca; dmarc=none Received: by mail-qt1-f176.google.com with SMTP id s12so13795184qtq.11 for ; Mon, 20 Mar 2023 09:49:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679330992; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=0sTnUu0e/3tPYoCsFOLrQHbS9ms8ZKnBh4hFgjXq1LI=; b=M1m9oevz4LL+8cg9ea5BN2a0kpFx13I0OHFTfNWXhfC/xEPY1Kwmi3Dru5U8F+EOya 3BKiFGtfkhfqf2pAqDsskxAh+5/yOLDj1IGoVceu19JwA3TKSC/KeG0LlVC7DrmRs4LO Q97uQ8BYIDsxb96H7SY6QuZOsuNKVlwlU8364hrby0ZtVfnXn/ZawJPFdELrbzaD/q0A 57YM54/e3m23QAqwTpunJYu2XTKh9MzJ+Mf3HX9AdadX4vdhlyjWCMqQuX4GT8NgR6ct Wx084c0NDl7g00HHcNn+B2bQBT8MqV6twzi/TtafIJRLEAWAjYRNWIXz9JNeTMyGImXr 8lWA== X-Gm-Message-State: AO0yUKV+57eAFQJNQL602LGHRfIVXYX8xgtPx7l40ORbsQ2ho+KOB69m fmYHdjNTFas6WWRzRL/Vy9o615wqvbA663biE546JA== X-Google-Smtp-Source: AK7set9lmzkIYDq4EF1VbjoxC25EPpDsn/lpP95zT1h0jVgDX0qUaL7uFJYbHECAOcfsZQuFxML9KA== X-Received: by 2002:ac8:5c13:0:b0:3e1:5060:5c8f with SMTP id i19-20020ac85c13000000b003e150605c8fmr5349751qti.33.1679330991900; Mon, 20 Mar 2023 09:49:51 -0700 (PDT) Received: from phe.ftfl.ca.ftfl.ca (drmons0544w-156-34-187-123.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.187.123]) by smtp.gmail.com with ESMTPSA id q3-20020a05620a024300b00738e8e81dc9sm852203qkn.75.2023.03.20.09.49.51 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 20 Mar 2023 09:49:51 -0700 (PDT) From: Joseph Mingrone To: Ashutosh Anand Cc: "freebsd-hackers@FreeBSD.org" Subject: Re: GSoC 2023: Implement MPLS support for FreeBSD In-Reply-To: (Ashutosh Anand's message of "Fri, 10 Mar 2023 02:46:13 +0530") References: Date: Mon, 20 Mar 2023 13:49:50 -0300 Message-ID: <86lejrjrkh.fsf@phe.ftfl.ca> User-Agent: Gnus/5.13 (Gnus v5.13) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[jrm@FreeBSD.org,jrm@ftfl.ca]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FREEFALL_USER(0.00)[jrm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[209.85.160.176:from]; TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[jrm@FreeBSD.org,jrm@ftfl.ca]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.160.176:from]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4PgLLn424Tz4GDC X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Fri, 2023-03-10 at 02:46, Ashutosh Anand wrote: > Hi! > I am Ashutosh Anand, a final year undergrad at NITK, India. I am one of the > prospective student for the following GSoC project. > I have been personally interested in implementing this project for FreeBSD. > My initial contact with the mentor (Alexander Chernikov) was successful, > however, *for the past 3-4 days my communication with him has been off*. > Since there were no other assigned mentors for this project I decided to > write this email, *to query if I could continue researching and building my > proposal for this project*. > *Apologize* for the inconvenience caused. Hoping for a positive reply! > Regards, > Ashutosh Anand > CSE student at NITK Hi Ashutosh (and Alexander)! Alexander (copied) signed up to be a mentor this year, so presumably he is available and willing to mentor. I will try to get clarification. Regards, Joe From nobody Mon Mar 20 16:53:25 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PgLQw6crHz3yxrH for ; Mon, 20 Mar 2023 16:53:28 +0000 (UTC) (envelope-from jrm@ftfl.ca) Received: from mail-qv1-f49.google.com (mail-qv1-f49.google.com [209.85.219.49]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PgLQw1bnKz4HQm for ; Mon, 20 Mar 2023 16:53:28 +0000 (UTC) (envelope-from jrm@ftfl.ca) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of jrm@ftfl.ca designates 209.85.219.49 as permitted sender) smtp.mailfrom=jrm@ftfl.ca; dmarc=none Received: by mail-qv1-f49.google.com with SMTP id m6so8028820qvq.0 for ; Mon, 20 Mar 2023 09:53:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679331206; h=mime-version:user-agent:message-id:date:references:in-reply-to :subject:cc:to:from:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=t5OmmzCmaRFe8x9tiLvMJwWi+RUyYU0JK6kpxYKfLLQ=; b=X/yFmQxC9fxFKj3D1TD4aZfs3cjx2MDbNpZ0S8CUVSkNDotOH9rlDdG9l7ph6JXtSU pujsGgmYB/y6dG1R1eUvEQGgK5mSQ4dxa64hWCrauB0K9LdO+WyXsCacF26gD/tkh6Ch F/1eh6xBJGlzaJTCl8i71JEnH0vFCd2K8ciJhegWhAu7gRQvd6UUhdrvynsZvu0+E1V9 ZtykCWR+IkPV+ZVqMrRbKdZ4M1Qe1QKevCVu7tBUrjlbBVXp0Z+RGMUifZvpUxwiD5f8 vIxTvcWo93oYOGEgN53AbCeGeX34y05+24VXe3h3zAnRJ8VYcTcjzkjOvwyKvmXMbFgw aIOw== X-Gm-Message-State: AO0yUKXRHJZkgXcDf2ih87kGqwFuLrBTy7FEh7CGEh2KrsPXdT2FWJQd deWMv7bARWtn3yCq8p/9y9pGjhvajZhwz+5A6WE6xw== X-Google-Smtp-Source: AK7set9/I17+pc1QWNANhxN/q4oTpdwllff0Mx9hcaUuvfaktdHGLal9/r6PYBkHmHb2jJrcz3PuXg== X-Received: by 2002:a05:6214:76c:b0:5c8:b343:f749 with SMTP id f12-20020a056214076c00b005c8b343f749mr9108484qvz.22.1679331206660; Mon, 20 Mar 2023 09:53:26 -0700 (PDT) Received: from phe.ftfl.ca.ftfl.ca (drmons0544w-156-34-187-123.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.187.123]) by smtp.gmail.com with ESMTPSA id g2-20020a37b602000000b0071aacb2c76asm7477138qkf.132.2023.03.20.09.53.25 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 20 Mar 2023 09:53:26 -0700 (PDT) From: Joseph Mingrone To: Gleb Popov Cc: freebsd-hackers Subject: Re: splash(4) support in vt In-Reply-To: (Gleb Popov's message of "Wed, 15 Mar 2023 19:58:59 +0300") References: Date: Mon, 20 Mar 2023 13:53:25 -0300 Message-ID: <86edpjjrei.fsf@phe.ftfl.ca> User-Agent: Gnus/5.13 (Gnus v5.13) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[jrm@FreeBSD.org,jrm@ftfl.ca]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[209.85.219.49:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.219.49:from]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[jrm]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_NEQ_ENVFROM(0.00)[jrm@FreeBSD.org,jrm@ftfl.ca] X-Rspamd-Queue-Id: 4PgLQw1bnKz4HQm X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N On Wed, 2023-03-15 at 19:58, Gleb Popov wrote: > Hello hackers! > I found an old post [1] regarding splash(4) support for vt. The > company I'm working for wants to invest some time into making the > FreeBSD boot process as shiny as possible. Can someone knowledgeable > on the topic give us some insight on where to start? > Our ultimate task is being able to render a GIF logo right from the > boot splash screen and until X session kicks in. > Thanks in advance! > [1] https://lists.freebsd.org/pipermail/freebsd-hackers/2018-September/053331.html Hi Gleb, It sounds like this might make a good GSoC project. What do you think? Joe From nobody Mon Mar 20 17:05:32 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PgLjP2Gz2z3yxW3 for ; Mon, 20 Mar 2023 17:06:01 +0000 (UTC) (envelope-from 6yearold@gmail.com) Received: from mail-vs1-f43.google.com (mail-vs1-f43.google.com [209.85.217.43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PgLjN6pMwz4K8x; Mon, 20 Mar 2023 17:06:00 +0000 (UTC) (envelope-from 6yearold@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-vs1-f43.google.com with SMTP id s1so11098321vsk.5; Mon, 20 Mar 2023 10:06:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679331960; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GgyOLFhBenvd84eoi4nptV7TL5y8HycJ1ou1j4k6qUc=; b=0TYhA6m5Rn5NWkDZg8FVe//PuB2/tNExlhwljd62w9fkNGReI5Gjpl9WsxV9UMrZHh WgkxGaSIF67MlNsnpSf/Xlu0nluaOOReahA87wfOa32pDHdq/MDo0PpUC1tmmz+efXa8 cf3xXxAJdJnA8ukPX/b28sOuoNZNhiip+rrVI6QQr+ii3j9l7whHVG1tNq/Uu8QleSY8 HG6YOnFwnCLqEsNb4ZiezW3xlchEdz18I+D4Ont7NzpIh8+w3fEvU2dXzWx7rifYqnIZ 8Wxpto/9TBx16vdM+i7LFWXGeaUE8i7qrUO2B+gJYqx6abnX/tj9Tzd6uRQQLUn1wTUx fZSw== X-Gm-Message-State: AO0yUKVehor1iClcoJ7lVroOXDj6a+CA7vANhqV9zyAT05fIsrOqJtTT MeRDUzPrDcopxAVqFdzmtKcKP/SEnXMdJw== X-Google-Smtp-Source: AK7set8nNXr+gxSkMRtMbLsv6daKHsIcyI42uAc7cQcj5IY8hW1cqXuh31zS/ArAYJkNJAXSVYStvg== X-Received: by 2002:a67:ff8f:0:b0:426:2838:532a with SMTP id v15-20020a67ff8f000000b004262838532amr1518515vsq.4.1679331959709; Mon, 20 Mar 2023 10:05:59 -0700 (PDT) Received: from mail-vs1-f48.google.com (mail-vs1-f48.google.com. [209.85.217.48]) by smtp.gmail.com with ESMTPSA id j13-20020ab015cd000000b006903f96c1a8sm836736uae.9.2023.03.20.10.05.59 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 20 Mar 2023 10:05:59 -0700 (PDT) Received: by mail-vs1-f48.google.com with SMTP id d2so4865134vso.9; Mon, 20 Mar 2023 10:05:59 -0700 (PDT) X-Received: by 2002:a67:e0d7:0:b0:425:dd2d:1c0 with SMTP id m23-20020a67e0d7000000b00425dd2d01c0mr4629985vsl.0.1679331959112; Mon, 20 Mar 2023 10:05:59 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <86edpjjrei.fsf@phe.ftfl.ca> In-Reply-To: <86edpjjrei.fsf@phe.ftfl.ca> From: Gleb Popov Date: Mon, 20 Mar 2023 20:05:32 +0300 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: splash(4) support in vt To: Joseph Mingrone Cc: freebsd-hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4PgLjN6pMwz4K8x X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mon, Mar 20, 2023 at 7:53=E2=80=AFPM Joseph Mingrone w= rote: > > Hi Gleb, > > It sounds like this might make a good GSoC project. What do you think? > > Joe I'm already being paid for working on FreeBSD, so I only need information and guidance. Let's not waste GSoC resources on this. From nobody Mon Mar 20 18:33:24 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PgNfS6jqTz404qp for ; Mon, 20 Mar 2023 18:33:36 +0000 (UTC) (envelope-from melifaro@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PgNfS68zSz4W3G; Mon, 20 Mar 2023 18:33:36 +0000 (UTC) (envelope-from melifaro@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679337216; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yuiqa2ImiMsDcFkYbY0VpsRadD0eOTVCdi6a23zJhi8=; b=KUyp+GgtXBEWU+IC0Z3k/KPCU3Bh9UHlDxF1hYJ4mEfNMYsXgAX20zdrjbKYukULkVdlGD /DEvPfVoQ36nMWQzmldraK953tjpw9ix3Uiw5dHePuRYvIpenFFEfjmJg985jRVDe+CaIG 3LL8hdcaxCV2cF8VMwXpVHC474WhaK0DTdrWe+L21zTGI5zY0QItcd48Iz5WF3vxh+VaDl rPNeIdWmulm19AJQSChR+wvmn+oPhNC5T1ZNbgyHXjJ05jVcjYSJdfnhNQmxeYINxZLfuS Xwz5CWnBzCP5lPy3AWmIAvNc2ZMoc9QP9RfGohnrj1FZ6pSiAfstRmlDOzpedw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679337216; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Yuiqa2ImiMsDcFkYbY0VpsRadD0eOTVCdi6a23zJhi8=; b=HjoEa6sRQ89HmNpi3DxgCWbXV8OofZt7UfaRaho+2AzJr63RRgAzbKSzyhGfg/qEbQdPAx G77QnscMBDjG4KgaWGOjYu7siu0nl8lavDUaqhMxBTpeN24sCyE8kWGeILUiJDUxMucl2f ffFzhHSp+NGPONEKklYGr54O/+wGJUzpoyJ0CV85SqLxymiiN4W5kE1IdVVoPk8PUF7ObF NEncwxU6XoTuHxWFM1onbLlXQIWfoQVZP62K7xidbWH91iXbTRjuuj9dyI6jAVnZjuDgux fBpKol9Xhaay9EnIg4ybFtbxj5H1DEyIjyimYiLwv75z35Tckrs4oULMYZMGZA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679337216; a=rsa-sha256; cv=none; b=GhvZvYjQTXPJim7YzPpxP6M600Xr1KG9HPtVLdQgyP5vkfOzWk4pj84o8oOIUv+iaWY+Lb y2Hg2MkCwLvEUbj1v/kiL+rM0Rb6YofW7e0anlbkZ+FxnMJ388lqEtOXCx+AF1TKGMLi+S pbgloMB6fD3+/388edGJItG5cnDYL9heAEiCaUPnOWZ9gNyVzBc27ITHgXb/BHcVtvfKJF xlh76AMqCyQau3vroZcH/wYFuuuQlhdU0G71XNTpWl5M13U3w9hOdjXjzE3PU/opE6RYbl r/2mDic5+u9gZ4kSmsIZxU3l/BdGdq0v0S0IT6QFuzGhOwmNAwCwkusb/kkLpQ== Received: from smtpclient.apple (unknown [IPv6:2a02:8084:d6bb:510:2574:8220:1538:851c]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: melifaro/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PgNfS2R4fzGZS; Mon, 20 Mar 2023 18:33:36 +0000 (UTC) (envelope-from melifaro@freebsd.org) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: GSoC 2023: Implement MPLS support for FreeBSD From: Alexander Chernikov In-Reply-To: <86lejrjrkh.fsf@phe.ftfl.ca> Date: Mon, 20 Mar 2023 18:33:24 +0000 Cc: Ashutosh Anand , "freebsd-hackers@FreeBSD.org" Content-Transfer-Encoding: quoted-printable Message-Id: <3F1A93CB-C1A2-415A-95CB-3B3C691E74AF@freebsd.org> References: <86lejrjrkh.fsf@phe.ftfl.ca> To: Joseph Mingrone X-Mailer: Apple Mail (2.3731.400.51.1.1) X-ThisMailContainsUnwantedMimeParts: N > On 20 Mar 2023, at 16:49, Joseph Mingrone wrote: >=20 > On Fri, 2023-03-10 at 02:46, Ashutosh Anand = wrote: >=20 >> Hi! >=20 >> I am Ashutosh Anand, a final year undergrad at NITK, India. I am one = of the >> prospective student for the following GSoC project. >=20 >=20 >=20 >> I have been personally interested in implementing this project for = FreeBSD. >> My initial contact with the mentor (Alexander Chernikov) was = successful, >> however, *for the past 3-4 days my communication with him has been = off*. >=20 >=20 >=20 >> Since there were no other assigned mentors for this project I decided = to >> write this email, *to query if I could continue researching and = building my >> proposal for this project*. >=20 >=20 >=20 >> *Apologize* for the inconvenience caused. Hoping for a positive = reply! >=20 >=20 >=20 >> Regards, >=20 >> Ashutosh Anand >> CSE student at NITK >=20 > Hi Ashutosh (and Alexander)! >=20 > Alexander (copied) signed up to be a mentor this year, so presumably = he > is available and willing to mentor. I will try to get clarification. Yep, that=E2=80=99s my fault. I was busy with other things and only = managed to reply on March 10. We had a productive conversation afterwards. There are no outstanding = questions that I=E2=80=99m aware of - and happy to answer them if there = are some. /Alenxader >=20 > Regards, >=20 > Joe >=20 From nobody Tue Mar 21 05:28:07 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pgg9c6Jl5z40p0J for ; Tue, 21 Mar 2023 05:28:04 +0000 (UTC) (envelope-from loo00013@umn.edu) Received: from mta-p6.oit.umn.edu (mta-p6.oit.umn.edu [134.84.196.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pgg9c0tJwz3M2v for ; Tue, 21 Mar 2023 05:28:04 +0000 (UTC) (envelope-from loo00013@umn.edu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=umn.edu header.s=google header.b="Ldxl/fAk"; spf=pass (mx1.freebsd.org: domain of loo00013@umn.edu designates 134.84.196.206 as permitted sender) smtp.mailfrom=loo00013@umn.edu; dmarc=pass (policy=reject) header.from=umn.edu Received: from localhost (unknown [127.0.0.1]) by mta-p6.oit.umn.edu (Postfix) with ESMTP id 4Pgg9b31DMz9vBry for ; Tue, 21 Mar 2023 05:28:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at umn.edu Received: from mta-p6.oit.umn.edu ([127.0.0.1]) by localhost (mta-p6.oit.umn.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g8CIq8Z-ggVg for ; Tue, 21 Mar 2023 00:28:03 -0500 (CDT) Received: from mail-io1-f70.google.com (mail-io1-f70.google.com [209.85.166.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mta-p6.oit.umn.edu (Postfix) with ESMTPS id 4Pgg9b1HjJz9vBrp for ; Tue, 21 Mar 2023 00:28:03 -0500 (CDT) DMARC-Filter: OpenDMARC Filter v1.3.2 mta-p6.oit.umn.edu 4Pgg9b1HjJz9vBrp DKIM-Filter: OpenDKIM Filter v2.11.0 mta-p6.oit.umn.edu 4Pgg9b1HjJz9vBrp Received: by mail-io1-f70.google.com with SMTP id c83-20020a6bb356000000b00758333e1ddfso973148iof.14 for ; Mon, 20 Mar 2023 22:28:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=umn.edu; s=google; t=1679376482; h=content-transfer-encoding:subject:from:to:content-language :user-agent:mime-version:date:message-id:from:to:cc:subject:date :message-id:reply-to; bh=awJxDuIOv1kwrBuWfXX8roJQebNLBtAiyR+JGR6PcmU=; b=Ldxl/fAkV44o6lia32W0wAO6conuhXlERX3KwDFSkZKlJeykG7I5EvwIvcU9qY8Pm9 QqAkFJ+YGGA92Kr8jqtsaa6lFLbaImXQzVOkTTzJh0ku2NO22/Sx7dOtC6q7GMIj66RE 1gowrzi/ecKduYjJYOHbuTJ//KEcxz5qUpp6OyhLQNVNiGrd7fbIqHQHJJEg+YfMT2pn pqRPSjmnAMZDV7j9UKXKa1BZ817rXvZtfY1QQ9qB6Stx1ArhxjZ0oLL8qcjuTvhGPGKI 8YwcR8eNj+7VjeZ2jOBY0ZC/SX8Ww2/f9ff1lOq8ueH0pcT0YRwE2Z5w9Ka23g/XUAHA LvLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679376482; h=content-transfer-encoding:subject:from:to:content-language :user-agent:mime-version:date:message-id:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=awJxDuIOv1kwrBuWfXX8roJQebNLBtAiyR+JGR6PcmU=; b=Tr2hAqcsxxgWCOOJL0SnRiPwsg0sHGGcX8LkXsmtcSU/WOr4J6H6cxnOLowvkfib0v IZ3JmFBDsTfdxakUC6s1fztvXHWN7RULS/LRtNYNzCcTCH0DYK4aKUksyAC1y6PJynyA /sK+sAwJRLUOxPKrLjuO+WNnWdk4ZxlaIm9Wnlu18p4/IRWnDsJmhGSHCDLeEYHdqEHP sSFWsbx2PVs5EMlKyvYwxWzTcKSQMS7pdMnQx8UjCbbo3cGRcdTbLa2xotgFNfqPBBNa E9Uc02yOsjXnPppO6NyagGmwD/xKe7UJgfQQrWSlmyGwUqh4nRW6FtH+DRrSQ+NDfd4l QU+A== X-Gm-Message-State: AO0yUKVNWUeTFyofnZtIcSef0mq57Qr/lrhTc8EufoiSufNJyYD/WR4y tFe7rQFqPRaDh9VMvksm+5zpMvd/SRofzC4MlWL3iH4BXFaSWDF9iWFNZHTn5fCGd0qltylG7Kt k0InL9WWgCHVwXXbqNWgw+fK4NsbjuRxOFbUXG2M2i3WF4BX5sJl40gVU19EDxeSuJXI/PL34QD ZjIvuLnQA= X-Received: by 2002:a6b:7905:0:b0:752:dce2:ac4e with SMTP id i5-20020a6b7905000000b00752dce2ac4emr884955iop.5.1679376482315; Mon, 20 Mar 2023 22:28:02 -0700 (PDT) X-Google-Smtp-Source: AK7set+JjCSmEjTIpmdSbh3PKPdw1hwlHzoAg5S/Oeo6uPIjOPd51zivGhV24G/bLZheFzEtRffp/w== X-Received: by 2002:a6b:7905:0:b0:752:dce2:ac4e with SMTP id i5-20020a6b7905000000b00752dce2ac4emr884946iop.5.1679376481878; Mon, 20 Mar 2023 22:28:01 -0700 (PDT) Received: from [192.168.0.67] (97-116-81-233.mpls.qwest.net. [97.116.81.233]) by smtp.gmail.com with ESMTPSA id b23-20020a026f57000000b00406431d0fb5sm3522312jae.72.2023.03.20.22.28.01 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 20 Mar 2023 22:28:01 -0700 (PDT) Message-ID: <6032f3f0-6e37-9e1b-ffe8-481c1ce42d93@umn.edu> Date: Tue, 21 Mar 2023 00:28:07 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Content-Language: en-US To: freebsd-hackers@FreeBSD.org From: Shaun Loo Subject: GSoC 2023 MFSBSD Integration Project Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-5.69 / 15.00]; DWL_DNSWL_LOW(-1.00)[umn.edu:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DMARC_POLICY_ALLOW(-0.50)[umn.edu,reject]; R_DKIM_ALLOW(-0.20)[umn.edu:s=google]; RCVD_IN_DNSWL_MED(-0.20)[134.84.196.206:from]; R_SPF_ALLOW(-0.20)[+ip4:134.84.196.192/27]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_FIVE(0.00)[6]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.166.70:received]; DKIM_TRACE(0.00)[umn.edu:+]; TO_DN_NONE(0.00)[]; ARC_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:217, ipnet:134.84.0.0/16, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Pgg9c0tJwz3M2v X-Spamd-Bar: ----- X-ThisMailContainsUnwantedMimeParts: N Hi all, I'm Shaun, a student who has been poking around with FreeBSD since the end of last year (installing onto various systems, recompiling the kernel for fun, simple stuff), and I figure I'd join FreeBSD's GSoC program this year and contribute to the project. I picked out the project "Integrate MFSBSD into the release building tools." The basic idea is to integrate MFSBSD, which generates an image that contains a minimal FreeBSD installation that is completely loaded into memory, into the release build tools. This will allow MFSBSD images of weekly snapshots to be generated automatically, rather than waiting for someone to manually generate one when new versions of FreeBSD are released, as well as facilitating testing of these images. I have already started some preliminary work on it (building MFSBSD for the upcoming 14.0 and familiarzing myself with FreeBSD's build tools) to get a feel for what I'll be trying to integrate into the release building tools. I've contacted Brad Davis , who was originally listed as the mentor for the project on the GSoC Ideas wiki page. Unfortunately, I have yet to have received a response. I've checked the src tree and it seems like their last commit was dated to October 2022. I've talked about this in the freebsd-soc IRC channel, and jrm has kindly pointed me to this mailing list as well as Martin Matuska , who maintains MFSBSD. I've tried contacting Martin over email yesterday and have not gotten a response. I'm wondering if anyone on this list is able and willing to be my mentor for this project, or if there are any other build tools-related projects I could work on for GSoC 2023? Thanks, Shaun Loo From nobody Tue Mar 21 08:02:00 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PgkbV228Vz40yGS for ; Tue, 21 Mar 2023 08:02:14 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PgkbT3hGhz3rmV for ; Tue, 21 Mar 2023 08:02:13 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x536.google.com with SMTP id cy23so56146301edb.12 for ; Tue, 21 Mar 2023 01:02:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; t=1679385731; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=WGKSYJ51HIYq+4C0GqyB6qOtlTGP3XNu3JWqWLlDFLA=; b=KwZu0vsincx9EIGxg4HPBqzjXIVpUsndUilIzaxqrNB+Vgsmev6KbaUPqlZSjkTFeX i82h16ZG/I+COKx3oKDkxvKD6b82UeD1oBjNR7ARVvADlWw4s4ap75EECSQEoVhEKkos QvoWPKlbr0+5oHISBMaezgFWsXHtFdX9/BgZHS5AEtxWvg09+tEcyS6tAtsiZH+VeKEV ka2sMvzUxGj4G5tYQbeq0Gn1F57YLxnVspcVTnRXG6C/Sql/2jt15HFrFcfXmMIfskMV 9lgxJv2/pdKPKppyzbHw2hDiY+sLkQ9kjrxs+gdlSiqCzRkkNnP55vgsDJF1IwfoYW5l u0/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679385731; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WGKSYJ51HIYq+4C0GqyB6qOtlTGP3XNu3JWqWLlDFLA=; b=UUi29A1+x4yjHz4n1TcjsoymC1wPPeGcWjbqEfNkFdLAC1hZECPv6RpepR92UqpQJe JBPwWL7uSE2DiPUO31gxPgJEX5DxN/60T4Z7gKnDi5Eh4CfRXVdaPlr1zqwz/5RJ15Vv lGuxbm3f39NdKUPYywsiEHB8gmoNmgDZw/JBlkRoXbZ9ZaSLaej8F8nKiODFpY4lYRXz aYkTnb92Yn3UpG0hZb96zrfSLf/o90r1/s3k0fQTkhRfxNS2ur9tmefb4hz39VN9g/rR 5VCfjyMUF0wMRm126yjTBNnaE90nvi5zzWU9Bz81lXDtw+mCnV0fQwlx0EuFLTNx7K/w fy2Q== X-Gm-Message-State: AO0yUKVHjRAQajERaaT01Tt0e3W3JI9FWl9C8nmqeAQwmLhSPQDNTHGA EVpZJFoleEPDg/WHHuVUTtaJ65yU2l0+m8KNOJpyXg== X-Google-Smtp-Source: AK7set8QUtpKTlWsKPNfdAFuM8HHSMboJPkqibKVEg9VJ4Exfi+HuAQR6mcvbeieOlnzKacIyh6EA4pVuasM2JbgOTI= X-Received: by 2002:a50:9507:0:b0:4fa:5b7d:ebb4 with SMTP id u7-20020a509507000000b004fa5b7debb4mr1137227eda.7.1679385731304; Tue, 21 Mar 2023 01:02:11 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <6032f3f0-6e37-9e1b-ffe8-481c1ce42d93@umn.edu> In-Reply-To: <6032f3f0-6e37-9e1b-ffe8-481c1ce42d93@umn.edu> From: Warner Losh Date: Tue, 21 Mar 2023 02:02:00 -0600 Message-ID: Subject: Re: GSoC 2023 MFSBSD Integration Project To: Shaun Loo Cc: FreeBSD Hackers Content-Type: multipart/alternative; boundary="000000000000f4a48005f764758e" X-Rspamd-Queue-Id: 4PgkbT3hGhz3rmV X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000f4a48005f764758e Content-Type: text/plain; charset="UTF-8" On Tue, Mar 21, 2023, 6:28 AM Shaun Loo wrote: > Hi all, > > I'm Shaun, a student who has been poking around with FreeBSD since the > end of last year (installing onto various systems, recompiling the > kernel for fun, simple stuff), and I figure I'd join FreeBSD's GSoC > program this year and contribute to the project. > > I picked out the project "Integrate MFSBSD into the release building > tools." The basic idea is to integrate MFSBSD, which generates an > image that contains a minimal FreeBSD installation that is completely > loaded into memory, into the release build tools. This will allow > MFSBSD images of weekly snapshots to be generated automatically, > rather than waiting for someone to manually generate one when new > versions of FreeBSD are released, as well as facilitating testing > of these images. > > I have already started some preliminary work on it (building > MFSBSD for the upcoming 14.0 and familiarzing myself with FreeBSD's > build tools) to get a feel for what I'll be trying to integrate into > the release building tools. > > I've contacted Brad Davis , who was originally > listed as the mentor for the project on the GSoC Ideas wiki page. > Unfortunately, I have yet to have received a response. I've checked the > src tree and it seems like their last commit was dated to October 2022. > > I've talked about this in the freebsd-soc IRC channel, and jrm has > kindly pointed me to this mailing list as well as Martin Matuska > , who maintains MFSBSD. I've tried contacting > Martin over email yesterday and have not gotten a response. > > I'm wondering if anyone on this list is able and willing to be my > mentor for this project, or if there are any other build tools-related > projects I could work on for GSoC 2023? > I am likely mentoring another project, but I can answer questions about build related issues. Warner > --000000000000f4a48005f764758e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Tue, Mar 21, 2023, 6:28 AM Shaun Loo <loo00013@umn= .edu> wrote:
Hi all,

I'm Shaun, a student who has been poking around with FreeBSD since the<= br> end of last year (installing onto various systems, recompiling the
kernel for fun, simple stuff), and I figure I'd join FreeBSD's GSoC=
program this year and contribute to the project.

I picked out the project "Integrate MFSBSD into the release building tools." The basic idea is to integrate MFSBSD, which generates an
image that contains a minimal FreeBSD installation that is completely
loaded into memory, into the release build tools. This will allow
MFSBSD images of weekly snapshots to be generated automatically,
rather than waiting for someone to manually generate one when new
versions of FreeBSD are released, as well as facilitating testing
of these images.

I have already started some preliminary work on it (building
MFSBSD for the upcoming 14.0 and familiarzing myself with FreeBSD's
build tools) to get a feel for what I'll be trying to integrate into the release building tools.

I've contacted Brad Davis <brd at FreeBSD dot org>, who was origi= nally
listed as the mentor for the project on the GSoC Ideas wiki page.
Unfortunately, I have yet to have received a response. I've checked the=
src tree and it seems like their last commit was dated to October 2022.

I've talked about this in the freebsd-soc IRC channel, and jrm has
kindly pointed me to this mailing list as well as Martin Matuska
<mm at FreeBSD dot org>, who maintains MFSBSD. I've tried contact= ing
Martin over email yesterday and have not gotten a response.

I'm wondering if anyone on this list is able and willing to be my
mentor for this project, or if there are any other build tools-related
projects I could work on for GSoC 2023?

I am likely mentoring another projec= t, but I can answer questions about build related issues.

Warner=C2=A0
--000000000000f4a48005f764758e-- From nobody Tue Mar 21 15:11:30 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pgw786K7Zz40QQX for ; Tue, 21 Mar 2023 15:11:48 +0000 (UTC) (envelope-from ken@freebsd.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mithlond.kdm.org", Issuer "mithlond.kdm.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pgw774MwMz3P2s for ; Tue, 21 Mar 2023 15:11:47 +0000 (UTC) (envelope-from ken@freebsd.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=softfail (mx1.freebsd.org: 96.89.93.250 is neither permitted nor denied by domain of ken@freebsd.org) smtp.mailfrom=ken@freebsd.org; dmarc=none Received: from smtpclient.apple (mbp2021.int.kdm.org [10.0.0.25]) (authenticated bits=0) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPSA id 32LFBejS024008 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 21 Mar 2023 11:11:40 -0400 (EDT) (envelope-from ken@freebsd.org) From: Ken Merry Content-Type: multipart/alternative; boundary="Apple-Mail=_D56B27C3-1350-4263-B0C8-3DAB4A3FCAD4" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Getting v_wire_count from a kernel core dump? Message-Id: <66742036-C8DF-4A13-9D4A-CDA71217E574@freebsd.org> Date: Tue, 21 Mar 2023 11:11:30 -0400 To: hackers@freebsd.org X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [96.89.93.250]); Tue, 21 Mar 2023 11:11:41 -0400 (EDT) X-Spamd-Result: default: False [-1.60 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:7922, ipnet:96.64.0.0/11, country:US]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FREEFALL_USER(0.00)[ken]; RCVD_COUNT_TWO(0.00)[2]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all:c]; TO_DN_NONE(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Rspamd-Queue-Id: 4Pgw774MwMz3P2s X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_D56B27C3-1350-4263-B0C8-3DAB4A3FCAD4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I have kernel core dumps from several machines out in the field = (customer sites) that were out of memory panics, and I=E2=80=99m trying = to figure out, from the kernel core dumps, whether we=E2=80=99re dealing = with a potential page leak. For context, these machines are running stable/13 from April 2021, but = they do have the fix for this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256507 Which is this commit in stable/13: = https://cgit.freebsd.org/src/commit/?id=3D6094749a1a5dafb8daf98deab23fc968= 070bc695 On a running system, I can get a rough idea whether there is a page leak = by looking at the VM system page counters: # sysctl vm.stats |grep count vm.stats.vm.v_cache_count: 0 vm.stats.vm.v_user_wire_count: 0 vm.stats.vm.v_laundry_count: 991626 vm.stats.vm.v_inactive_count: 39733216 vm.stats.vm.v_active_count: 11821309 vm.stats.vm.v_wire_count: 11154113 vm.stats.vm.v_free_count: 1599981 vm.stats.vm.v_page_count: 65347213 So the first 5 numbers add up to 65300245 in this case, for a difference = of 46968. =20 Am I off base here as far as the various counts adding up to the page = count? (e.g. is the wire count just an additional attribute of a page = and not another separate state like active, inactive or laundry?) Looking at the kernel core dump for one of the systems I see: kgdb) print vm_cnt $1 =3D {v_swtch =3D 0xfffffe022158f2f8, v_trap =3D 0xfffffe022158f2f0, v_syscall =3D 0xfffffe022158f2e8, v_intr =3D 0xfffffe022158f2e0, v_soft =3D 0xfffffe022158f2d8, v_vm_faults =3D 0xfffffe022158f2d0, v_io_faults =3D 0xfffffe022158f2c8, v_cow_faults =3D = 0xfffffe022158f2c0, v_cow_optim =3D 0xfffffe022158f2b8, v_zfod =3D 0xfffffe022158f2b0, v_ozfod =3D 0xfffffe022158f2a8, v_swapin =3D 0xfffffe022158f2a0, v_swapout =3D 0xfffffe022158f298, v_swappgsin =3D 0xfffffe022158f290, v_swappgsout =3D 0xfffffe022158f288, v_vnodein =3D 0xfffffe022158f280, v_vnodeout =3D 0xfffffe022158f278, v_vnodepgsin =3D = 0xfffffe022158f270, v_vnodepgsout =3D 0xfffffe022158f268, v_intrans =3D = 0xfffffe022158f260, v_reactivated =3D 0xfffffe022158f258, v_pdwakeups =3D = 0xfffffe022158f250, v_pdpages =3D 0xfffffe022158f248, v_pdshortfalls =3D = 0xfffffe022158f240, v_dfree =3D 0xfffffe022158f238, v_pfree =3D 0xfffffe022158f230, v_tfree =3D 0xfffffe022158f228, v_forks =3D 0xfffffe022158f220, v_vforks =3D 0xfffffe022158f218, v_rforks =3D 0xfffffe022158f210, v_kthreads =3D 0xfffffe022158f208, v_forkpages =3D 0xfffffe022158f200, v_vforkpages =3D 0xfffffe022158f1f8, v_rforkpages =3D = 0xfffffe022158f1f0, v_kthreadpages =3D 0xfffffe022158f1e8, v_wire_count =3D = 0xfffffe022158f1e0, v_page_size =3D 4096, v_page_count =3D 65342843, v_free_reserved =3D = 85343, v_free_target =3D 1392195, v_free_min =3D 412056, v_inactive_target =3D = 2088292, v_pageout_free_min =3D 136, v_interrupt_free_min =3D 8, v_free_severe = =3D 248698} (kgdb) print vm_ndomains $2 =3D 4 (kgdb) print vm_dom[0].vmd_pagequeues[0].pq_cnt $3 =3D 6298704 (kgdb) print vm_dom[0].vmd_pagequeues[1].pq_cnt $4 =3D 3423939 (kgdb) print vm_dom[0].vmd_pagequeues[2].pq_cnt $5 =3D 629834 (kgdb) print vm_dom[0].vmd_pagequeues[3].pq_cnt $6 =3D 0 (kgdb) print vm_dom[1].vmd_pagequeues[0].pq_cnt $7 =3D 2301793 (kgdb) print vm_dom[1].vmd_pagequeues[1].pq_cnt $8 =3D 7130193 (kgdb) print vm_dom[1].vmd_pagequeues[2].pq_cnt $9 =3D 701495 (kgdb) print vm_dom[1].vmd_pagequeues[3].pq_cnt $10 =3D 0 (kgdb) print vm_dom[2].vmd_pagequeues[0].pq_cnt $11 =3D 464429 (kgdb) print vm_dom[2].vmd_pagequeues[1].pq_cnt $12 =3D 9123532 (kgdb) print vm_dom[2].vmd_pagequeues[2].pq_cnt $13 =3D 1037423 (kgdb) print vm_dom[2].vmd_pagequeues[3].pq_cnt $14 =3D 0 (kgdb) print vm_dom[3].vmd_pagequeues[0].pq_cnt $15 =3D 5444946 (kgdb) print vm_dom[3].vmd_pagequeues[1].pq_cnt $16 =3D 4466782 (kgdb) print vm_dom[3].vmd_pagequeues[2].pq_cnt $17 =3D 785195 (kgdb) print vm_dom[3].vmd_pagequeues[3].pq_cnt $18 =3D 0 (kgdb)=20 Adding up the page queue counts: 6298704 3423939 629834 ++p 10352477 2301793 7130193 701495 ++p 10133481 +p 20485958 464429 9123532 1037423 ++p 10625384 +p 31111342 5444946 4466782 785195 ++p 10696923 +p 41808265 So, about 23M pages short of v_page_count. =20 v_wire_count is a per-CPU counter, and on a running system it gets added = up. But trying to access it in the kernel core dump yields: (kgdb) print vm_cnt.v_wire_count $2 =3D (counter_u64_t) 0xfffffe022158f1e0 (kgdb) print *$2 Cannot access memory at address 0xfffffe022158f1e0 Anyone have any ideas whether I can figure out whether there is a page = leak from the core dump? Thanks, Ken =E2=80=94=20 Ken Merry ken@FreeBSD.ORG --Apple-Mail=_D56B27C3-1350-4263-B0C8-3DAB4A3FCAD4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 I have kernel = core dumps from several machines out in the field (customer sites) that = were out of memory panics, and I=E2=80=99m trying to figure out, from = the kernel core dumps, whether we=E2=80=99re dealing with a potential = page leak.

For context, these machines are running = stable/13 from April 2021, but they do have the fix for this = bug:

https:= //bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256507

Which is this commit in = stable/13:


On a running = system, I can get a rough idea whether there is a page leak by looking = at the VM system page counters:

# sysctl = vm.stats |grep count
vm.stats.vm.v_cache_count: = 0
vm.stats.vm.v_user_wire_count: = 0
vm.stats.vm.v_laundry_count: = 991626
vm.stats.vm.v_inactive_count: = 39733216
vm.stats.vm.v_active_count: = 11821309
vm.stats.vm.v_wire_count: = 11154113
vm.stats.vm.v_free_count: = 1599981
vm.stats.vm.v_page_count: = 65347213

So the first 5 numbers add up = to 65300245 in this case, for a difference of 46968. =  

Am I off base here as far as the various = counts adding up to the page count?  (e.g. is the wire count just = an additional attribute of a page and not another separate state like = active, inactive or laundry?)

Looking at the = kernel core dump for one of the systems I = see:

kgdb) print vm_cnt
$1 =3D = {v_swtch =3D 0xfffffe022158f2f8, v_trap =3D = 0xfffffe022158f2f0,
  v_syscall =3D 0xfffffe022158f2e8, = v_intr =3D 0xfffffe022158f2e0,
  v_soft =3D = 0xfffffe022158f2d8, v_vm_faults =3D 0xfffffe022158f2d0,
  = v_io_faults =3D 0xfffffe022158f2c8, v_cow_faults =3D = 0xfffffe022158f2c0,
  v_cow_optim =3D 0xfffffe022158f2b8, = v_zfod =3D 0xfffffe022158f2b0,
  v_ozfod =3D = 0xfffffe022158f2a8, v_swapin =3D 0xfffffe022158f2a0,
  = v_swapout =3D 0xfffffe022158f298, v_swappgsin =3D = 0xfffffe022158f290,
  v_swappgsout =3D = 0xfffffe022158f288, v_vnodein =3D 0xfffffe022158f280,
  = v_vnodeout =3D 0xfffffe022158f278, v_vnodepgsin =3D = 0xfffffe022158f270,
  v_vnodepgsout =3D = 0xfffffe022158f268, v_intrans =3D 0xfffffe022158f260,
  = v_reactivated =3D 0xfffffe022158f258, v_pdwakeups =3D = 0xfffffe022158f250,
  v_pdpages =3D 0xfffffe022158f248, = v_pdshortfalls =3D 0xfffffe022158f240,
  v_dfree =3D = 0xfffffe022158f238, v_pfree =3D 0xfffffe022158f230,
  = v_tfree =3D 0xfffffe022158f228, v_forks =3D = 0xfffffe022158f220,
  v_vforks =3D 0xfffffe022158f218, = v_rforks =3D 0xfffffe022158f210,
  v_kthreads =3D = 0xfffffe022158f208, v_forkpages =3D 0xfffffe022158f200,
  = v_vforkpages =3D 0xfffffe022158f1f8, v_rforkpages =3D = 0xfffffe022158f1f0,
  v_kthreadpages =3D = 0xfffffe022158f1e8, v_wire_count =3D = 0xfffffe022158f1e0,
  v_page_size =3D 4096, v_page_count = =3D 65342843, v_free_reserved =3D 85343,
  v_free_target = =3D 1392195, v_free_min =3D 412056, v_inactive_target =3D = 2088292,
  v_pageout_free_min =3D 136, = v_interrupt_free_min =3D 8, v_free_severe =3D 248698}
(kgdb) = print vm_ndomains
$2 =3D 4
(kgdb) print = vm_dom[0].vmd_pagequeues[0].pq_cnt
$3 =3D = 6298704
(kgdb) print = vm_dom[0].vmd_pagequeues[1].pq_cnt
$4 =3D = 3423939
(kgdb) print = vm_dom[0].vmd_pagequeues[2].pq_cnt
$5 =3D = 629834
(kgdb) print = vm_dom[0].vmd_pagequeues[3].pq_cnt
$6 =3D 0
(kgdb) = print vm_dom[1].vmd_pagequeues[0].pq_cnt
$7 =3D = 2301793
(kgdb) print = vm_dom[1].vmd_pagequeues[1].pq_cnt
$8 =3D = 7130193
(kgdb) print = vm_dom[1].vmd_pagequeues[2].pq_cnt
$9 =3D = 701495
(kgdb) print = vm_dom[1].vmd_pagequeues[3].pq_cnt
$10 =3D 0
(kgdb) = print vm_dom[2].vmd_pagequeues[0].pq_cnt
$11 =3D = 464429
(kgdb) print = vm_dom[2].vmd_pagequeues[1].pq_cnt
$12 =3D = 9123532
(kgdb) print = vm_dom[2].vmd_pagequeues[2].pq_cnt
$13 =3D = 1037423
(kgdb) print = vm_dom[2].vmd_pagequeues[3].pq_cnt
$14 =3D 0
(kgdb) = print vm_dom[3].vmd_pagequeues[0].pq_cnt
$15 =3D = 5444946
(kgdb) print = vm_dom[3].vmd_pagequeues[1].pq_cnt
$16 =3D = 4466782
(kgdb) print = vm_dom[3].vmd_pagequeues[2].pq_cnt
$17 =3D = 785195
(kgdb) print = vm_dom[3].vmd_pagequeues[3].pq_cnt
$18 =3D = 0
(kgdb) 


Add= ing up the page queue = counts:

6298704
3423939
= 629834
++p
10352477
2301793
713019= 3
701495
++p
10133481
+p
20485958
464429
9123532
1037423
+= +p
10625384
+p
31111342
5444946
4466782
785195
++p
10696923
+p
41808265

So, about 23M = pages short of v_page_count. =  

v_wire_count is a per-CPU counter, and = on a running system it gets added up.  But trying to access it in = the kernel core dump yields:

(kgdb) print = vm_cnt.v_wire_count
$2 =3D (counter_u64_t) = 0xfffffe022158f1e0
(kgdb) print *$2
Cannot access = memory at address = 0xfffffe022158f1e0

Anyone have any ideas = whether I can figure out whether there is a page leak from the core = dump?

Thanks,

Ken
<= div>=E2=80=94 
Ken = Merry
ken@FreeBSD.ORG



= --Apple-Mail=_D56B27C3-1350-4263-B0C8-3DAB4A3FCAD4-- From nobody Tue Mar 21 15:37:02 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PgwhR2zNJz40RlS for ; Tue, 21 Mar 2023 15:37:11 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PgwhQ3R1wz3hW3; Tue, 21 Mar 2023 15:37:10 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 32LFb21D036784 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Tue, 21 Mar 2023 17:37:05 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 32LFb21D036784 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 32LFb2g5036783; Tue, 21 Mar 2023 17:37:02 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 21 Mar 2023 17:37:02 +0200 From: Konstantin Belousov To: Ken Merry Cc: hackers@freebsd.org Subject: Re: Getting v_wire_count from a kernel core dump? Message-ID: References: <66742036-C8DF-4A13-9D4A-CDA71217E574@freebsd.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <66742036-C8DF-4A13-9D4A-CDA71217E574@freebsd.org> X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4PgwhQ3R1wz3hW3 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Tue, Mar 21, 2023 at 11:11:30AM -0400, Ken Merry wrote: > I have kernel core dumps from several machines out in the field (customer sites) that were out of memory panics, and I’m trying to figure out, from the kernel core dumps, whether we’re dealing with a potential page leak. > > For context, these machines are running stable/13 from April 2021, but they do have the fix for this bug: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=256507 > > Which is this commit in stable/13: > > https://cgit.freebsd.org/src/commit/?id=6094749a1a5dafb8daf98deab23fc968070bc695 > > On a running system, I can get a rough idea whether there is a page leak by looking at the VM system page counters: > > # sysctl vm.stats |grep count > vm.stats.vm.v_cache_count: 0 > vm.stats.vm.v_user_wire_count: 0 > vm.stats.vm.v_laundry_count: 991626 > vm.stats.vm.v_inactive_count: 39733216 > vm.stats.vm.v_active_count: 11821309 > vm.stats.vm.v_wire_count: 11154113 > vm.stats.vm.v_free_count: 1599981 > vm.stats.vm.v_page_count: 65347213 > > So the first 5 numbers add up to 65300245 in this case, for a difference of 46968. > > Am I off base here as far as the various counts adding up to the page count? (e.g. is the wire count just an additional attribute of a page and not another separate state like active, inactive or laundry?) > > Looking at the kernel core dump for one of the systems I see: > > kgdb) print vm_cnt > $1 = {v_swtch = 0xfffffe022158f2f8, v_trap = 0xfffffe022158f2f0, > v_syscall = 0xfffffe022158f2e8, v_intr = 0xfffffe022158f2e0, > v_soft = 0xfffffe022158f2d8, v_vm_faults = 0xfffffe022158f2d0, > v_io_faults = 0xfffffe022158f2c8, v_cow_faults = 0xfffffe022158f2c0, > v_cow_optim = 0xfffffe022158f2b8, v_zfod = 0xfffffe022158f2b0, > v_ozfod = 0xfffffe022158f2a8, v_swapin = 0xfffffe022158f2a0, > v_swapout = 0xfffffe022158f298, v_swappgsin = 0xfffffe022158f290, > v_swappgsout = 0xfffffe022158f288, v_vnodein = 0xfffffe022158f280, > v_vnodeout = 0xfffffe022158f278, v_vnodepgsin = 0xfffffe022158f270, > v_vnodepgsout = 0xfffffe022158f268, v_intrans = 0xfffffe022158f260, > v_reactivated = 0xfffffe022158f258, v_pdwakeups = 0xfffffe022158f250, > v_pdpages = 0xfffffe022158f248, v_pdshortfalls = 0xfffffe022158f240, > v_dfree = 0xfffffe022158f238, v_pfree = 0xfffffe022158f230, > v_tfree = 0xfffffe022158f228, v_forks = 0xfffffe022158f220, > v_vforks = 0xfffffe022158f218, v_rforks = 0xfffffe022158f210, > v_kthreads = 0xfffffe022158f208, v_forkpages = 0xfffffe022158f200, > v_vforkpages = 0xfffffe022158f1f8, v_rforkpages = 0xfffffe022158f1f0, > v_kthreadpages = 0xfffffe022158f1e8, v_wire_count = 0xfffffe022158f1e0, > v_page_size = 4096, v_page_count = 65342843, v_free_reserved = 85343, > v_free_target = 1392195, v_free_min = 412056, v_inactive_target = 2088292, > v_pageout_free_min = 136, v_interrupt_free_min = 8, v_free_severe = 248698} > (kgdb) print vm_ndomains > $2 = 4 > (kgdb) print vm_dom[0].vmd_pagequeues[0].pq_cnt > $3 = 6298704 > (kgdb) print vm_dom[0].vmd_pagequeues[1].pq_cnt > $4 = 3423939 > (kgdb) print vm_dom[0].vmd_pagequeues[2].pq_cnt > $5 = 629834 > (kgdb) print vm_dom[0].vmd_pagequeues[3].pq_cnt > $6 = 0 > (kgdb) print vm_dom[1].vmd_pagequeues[0].pq_cnt > $7 = 2301793 > (kgdb) print vm_dom[1].vmd_pagequeues[1].pq_cnt > $8 = 7130193 > (kgdb) print vm_dom[1].vmd_pagequeues[2].pq_cnt > $9 = 701495 > (kgdb) print vm_dom[1].vmd_pagequeues[3].pq_cnt > $10 = 0 > (kgdb) print vm_dom[2].vmd_pagequeues[0].pq_cnt > $11 = 464429 > (kgdb) print vm_dom[2].vmd_pagequeues[1].pq_cnt > $12 = 9123532 > (kgdb) print vm_dom[2].vmd_pagequeues[2].pq_cnt > $13 = 1037423 > (kgdb) print vm_dom[2].vmd_pagequeues[3].pq_cnt > $14 = 0 > (kgdb) print vm_dom[3].vmd_pagequeues[0].pq_cnt > $15 = 5444946 > (kgdb) print vm_dom[3].vmd_pagequeues[1].pq_cnt > $16 = 4466782 > (kgdb) print vm_dom[3].vmd_pagequeues[2].pq_cnt > $17 = 785195 > (kgdb) print vm_dom[3].vmd_pagequeues[3].pq_cnt > $18 = 0 > (kgdb) > > > Adding up the page queue counts: > > 6298704 > 3423939 > 629834 > ++p > 10352477 > 2301793 > 7130193 > 701495 > ++p > 10133481 > +p > 20485958 > 464429 > 9123532 > 1037423 > ++p > 10625384 > +p > 31111342 > 5444946 > 4466782 > 785195 > ++p > 10696923 > +p > 41808265 > > So, about 23M pages short of v_page_count. > > v_wire_count is a per-CPU counter, and on a running system it gets added up. But trying to access it in the kernel core dump yields: > > (kgdb) print vm_cnt.v_wire_count > $2 = (counter_u64_t) 0xfffffe022158f1e0 > (kgdb) print *$2 > Cannot access memory at address 0xfffffe022158f1e0 > > Anyone have any ideas whether I can figure out whether there is a page leak from the core dump? > Did you looked at UMA/malloc stats? It could be genuine VM leaking pages, but more often it is some kernel subsystem leaking its own allocations. For later, try both vmstat -z and vmstat -m on the kernel.debug + vmcore. Often the leakage is immediately obvious. From nobody Tue Mar 21 17:54:12 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pgzkm5jSNz40ZyQ for ; Tue, 21 Mar 2023 17:54:24 +0000 (UTC) (envelope-from ken@freebsd.org) Received: from mithlond.kdm.org (mithlond.kdm.org [96.89.93.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mithlond.kdm.org", Issuer "mithlond.kdm.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pgzkm310lz3xVr for ; Tue, 21 Mar 2023 17:54:24 +0000 (UTC) (envelope-from ken@freebsd.org) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (mbp2021.int.kdm.org [10.0.0.25]) (authenticated bits=0) by mithlond.kdm.org (8.15.2/8.14.9) with ESMTPSA id 32LHsM9M026077 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NO); Tue, 21 Mar 2023 13:54:22 -0400 (EDT) (envelope-from ken@freebsd.org) From: Ken Merry Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_2710E057-8BAD-4F01-9DD6-48E4ECCB364F" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Getting v_wire_count from a kernel core dump? Date: Tue, 21 Mar 2023 13:54:12 -0400 In-Reply-To: Cc: hackers@freebsd.org To: Konstantin Belousov References: <66742036-C8DF-4A13-9D4A-CDA71217E574@freebsd.org> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mithlond.kdm.org [96.89.93.250]); Tue, 21 Mar 2023 13:54:23 -0400 (EDT) X-Rspamd-Queue-Id: 4Pgzkm310lz3xVr X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7922, ipnet:96.64.0.0/11, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_2710E057-8BAD-4F01-9DD6-48E4ECCB364F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Mar 21, 2023, at 11:37, Konstantin Belousov = wrote: >=20 > On Tue, Mar 21, 2023 at 11:11:30AM -0400, Ken Merry wrote: >> I have kernel core dumps from several machines out in the field = (customer sites) that were out of memory panics, and I=E2=80=99m trying = to figure out, from the kernel core dumps, whether we=E2=80=99re dealing = with a potential page leak. >>=20 >> For context, these machines are running stable/13 from April 2021, = but they do have the fix for this bug: >>=20 >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256507 >>=20 >> Which is this commit in stable/13: >>=20 >> = https://cgit.freebsd.org/src/commit/?id=3D6094749a1a5dafb8daf98deab23fc968= 070bc695 >>=20 >> On a running system, I can get a rough idea whether there is a page = leak by looking at the VM system page counters: >>=20 >> # sysctl vm.stats |grep count >> vm.stats.vm.v_cache_count: 0 >> vm.stats.vm.v_user_wire_count: 0 >> vm.stats.vm.v_laundry_count: 991626 >> vm.stats.vm.v_inactive_count: 39733216 >> vm.stats.vm.v_active_count: 11821309 >> vm.stats.vm.v_wire_count: 11154113 >> vm.stats.vm.v_free_count: 1599981 >> vm.stats.vm.v_page_count: 65347213 >>=20 >> So the first 5 numbers add up to 65300245 in this case, for a = difference of 46968. =20 >>=20 >> Am I off base here as far as the various counts adding up to the page = count? (e.g. is the wire count just an additional attribute of a page = and not another separate state like active, inactive or laundry?) >>=20 >> Looking at the kernel core dump for one of the systems I see: >>=20 >> kgdb) print vm_cnt >> $1 =3D {v_swtch =3D 0xfffffe022158f2f8, v_trap =3D = 0xfffffe022158f2f0, >> v_syscall =3D 0xfffffe022158f2e8, v_intr =3D 0xfffffe022158f2e0, >> v_soft =3D 0xfffffe022158f2d8, v_vm_faults =3D 0xfffffe022158f2d0, >> v_io_faults =3D 0xfffffe022158f2c8, v_cow_faults =3D = 0xfffffe022158f2c0, >> v_cow_optim =3D 0xfffffe022158f2b8, v_zfod =3D 0xfffffe022158f2b0, >> v_ozfod =3D 0xfffffe022158f2a8, v_swapin =3D 0xfffffe022158f2a0, >> v_swapout =3D 0xfffffe022158f298, v_swappgsin =3D = 0xfffffe022158f290, >> v_swappgsout =3D 0xfffffe022158f288, v_vnodein =3D = 0xfffffe022158f280, >> v_vnodeout =3D 0xfffffe022158f278, v_vnodepgsin =3D = 0xfffffe022158f270, >> v_vnodepgsout =3D 0xfffffe022158f268, v_intrans =3D = 0xfffffe022158f260, >> v_reactivated =3D 0xfffffe022158f258, v_pdwakeups =3D = 0xfffffe022158f250, >> v_pdpages =3D 0xfffffe022158f248, v_pdshortfalls =3D = 0xfffffe022158f240, >> v_dfree =3D 0xfffffe022158f238, v_pfree =3D 0xfffffe022158f230, >> v_tfree =3D 0xfffffe022158f228, v_forks =3D 0xfffffe022158f220, >> v_vforks =3D 0xfffffe022158f218, v_rforks =3D 0xfffffe022158f210, >> v_kthreads =3D 0xfffffe022158f208, v_forkpages =3D = 0xfffffe022158f200, >> v_vforkpages =3D 0xfffffe022158f1f8, v_rforkpages =3D = 0xfffffe022158f1f0, >> v_kthreadpages =3D 0xfffffe022158f1e8, v_wire_count =3D = 0xfffffe022158f1e0, >> v_page_size =3D 4096, v_page_count =3D 65342843, v_free_reserved =3D = 85343, >> v_free_target =3D 1392195, v_free_min =3D 412056, v_inactive_target = =3D 2088292, >> v_pageout_free_min =3D 136, v_interrupt_free_min =3D 8, = v_free_severe =3D 248698} >> (kgdb) print vm_ndomains >> $2 =3D 4 >> (kgdb) print vm_dom[0].vmd_pagequeues[0].pq_cnt >> $3 =3D 6298704 >> (kgdb) print vm_dom[0].vmd_pagequeues[1].pq_cnt >> $4 =3D 3423939 >> (kgdb) print vm_dom[0].vmd_pagequeues[2].pq_cnt >> $5 =3D 629834 >> (kgdb) print vm_dom[0].vmd_pagequeues[3].pq_cnt >> $6 =3D 0 >> (kgdb) print vm_dom[1].vmd_pagequeues[0].pq_cnt >> $7 =3D 2301793 >> (kgdb) print vm_dom[1].vmd_pagequeues[1].pq_cnt >> $8 =3D 7130193 >> (kgdb) print vm_dom[1].vmd_pagequeues[2].pq_cnt >> $9 =3D 701495 >> (kgdb) print vm_dom[1].vmd_pagequeues[3].pq_cnt >> $10 =3D 0 >> (kgdb) print vm_dom[2].vmd_pagequeues[0].pq_cnt >> $11 =3D 464429 >> (kgdb) print vm_dom[2].vmd_pagequeues[1].pq_cnt >> $12 =3D 9123532 >> (kgdb) print vm_dom[2].vmd_pagequeues[2].pq_cnt >> $13 =3D 1037423 >> (kgdb) print vm_dom[2].vmd_pagequeues[3].pq_cnt >> $14 =3D 0 >> (kgdb) print vm_dom[3].vmd_pagequeues[0].pq_cnt >> $15 =3D 5444946 >> (kgdb) print vm_dom[3].vmd_pagequeues[1].pq_cnt >> $16 =3D 4466782 >> (kgdb) print vm_dom[3].vmd_pagequeues[2].pq_cnt >> $17 =3D 785195 >> (kgdb) print vm_dom[3].vmd_pagequeues[3].pq_cnt >> $18 =3D 0 >> (kgdb)=20 >>=20 >>=20 >> Adding up the page queue counts: >>=20 >> 6298704 >> 3423939 >> 629834 >> ++p >> 10352477 >> 2301793 >> 7130193 >> 701495 >> ++p >> 10133481 >> +p >> 20485958 >> 464429 >> 9123532 >> 1037423 >> ++p >> 10625384 >> +p >> 31111342 >> 5444946 >> 4466782 >> 785195 >> ++p >> 10696923 >> +p >> 41808265 >>=20 >> So, about 23M pages short of v_page_count. =20 >>=20 >> v_wire_count is a per-CPU counter, and on a running system it gets = added up. But trying to access it in the kernel core dump yields: >>=20 >> (kgdb) print vm_cnt.v_wire_count >> $2 =3D (counter_u64_t) 0xfffffe022158f1e0 >> (kgdb) print *$2 >> Cannot access memory at address 0xfffffe022158f1e0 >>=20 >> Anyone have any ideas whether I can figure out whether there is a = page leak from the core dump? >>=20 >=20 > Did you looked at UMA/malloc stats? It could be genuine VM leaking = pages, > but more often it is some kernel subsystem leaking its own = allocations. > For later, try both vmstat -z and vmstat -m on the kernel.debug + = vmcore. > Often the leakage is immediately obvious. So, vmstat -m doesn=E2=80=99t work on a kernel core dump, at least on = this vintage of stable/13: # vmstat -m -N /usr/lib/debug/boot/kernel/kernel.debug -M = 2023-03-15_21-23-27.vmcore.0 vmstat: memstat_kvm_malloc:=20 Type InUse MemUse Requests Size(s) As for vmstat -z, we=E2=80=99ve got a script that adds it all up, and = it=E2=80=99s ~80GB on a system with 256GB RAM. The largest processes in the system add up to approximately 24GB of RAM = used, although that is very difficult to measure precisely because = we=E2=80=99re using Postgres, and the processes that Postgres uses share = varying amounts of RAM. =20 But it doesn=E2=80=99t seem like we=E2=80=99re approaching 256GB. If I = can rule out a page leak, then the reason behind this change could be = the issue: = https://cgit.freebsd.org/src/commit/sys/vm?h=3Dstable/13&id=3D555baef969a1= 7a7cbcd6af3ee5bcf854ecd4de7c The ARC in our case is at about 65GB. Thanks, Ken =E2=80=94=20 Ken Merry ken@FreeBSD.ORG --Apple-Mail=_2710E057-8BAD-4F01-9DD6-48E4ECCB364F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On = Mar 21, 2023, at 11:37, Konstantin Belousov <kostikbel@gmail.com> = wrote:

On Tue, Mar 21, 2023 at = 11:11:30AM -0400, Ken Merry wrote:
I have kernel = core dumps from several machines out in the field (customer sites) that = were out of memory panics, and I=E2=80=99m trying to figure out, from = the kernel core dumps, whether we=E2=80=99re dealing with a potential = page leak.

For context, these machines are running stable/13 from = April 2021, but they do have the fix for this = bug:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D256507
=
Which is this commit in = stable/13:

https://cgit.freebsd.org/src/commit/?id=3D6094749a1a5daf= b8daf98deab23fc968070bc695

On a running system, I can get a rough = idea whether there is a page leak by looking at the VM system page = counters:

# sysctl vm.stats |grep = count
vm.stats.vm.v_cache_count: 0
vm.stats.vm.v_user_wire_count: = 0
vm.stats.vm.v_laundry_count: = 991626
vm.stats.vm.v_inactive_count: = 39733216
vm.stats.vm.v_active_count: = 11821309
vm.stats.vm.v_wire_count: = 11154113
vm.stats.vm.v_free_count: = 1599981
vm.stats.vm.v_page_count: 65347213

So the first 5 = numbers add up to 65300245 in this case, for a difference of 46968. =  

Am I off base here as far as the various counts adding up = to the page count?  (e.g. is the wire count just an additional = attribute of a page and not another separate state like active, inactive = or laundry?)

Looking at the kernel core dump for one of the = systems I see:

kgdb) print vm_cnt
$1 =3D {v_swtch =3D = 0xfffffe022158f2f8, v_trap =3D 0xfffffe022158f2f0,
 v_syscall =3D = 0xfffffe022158f2e8, v_intr =3D 0xfffffe022158f2e0,
 v_soft =3D = 0xfffffe022158f2d8, v_vm_faults =3D = 0xfffffe022158f2d0,
 v_io_faults =3D 0xfffffe022158f2c8, = v_cow_faults =3D 0xfffffe022158f2c0,
 v_cow_optim =3D = 0xfffffe022158f2b8, v_zfod =3D 0xfffffe022158f2b0,
 v_ozfod =3D = 0xfffffe022158f2a8, v_swapin =3D 0xfffffe022158f2a0,
 v_swapout = =3D 0xfffffe022158f298, v_swappgsin =3D = 0xfffffe022158f290,
 v_swappgsout =3D 0xfffffe022158f288, = v_vnodein =3D 0xfffffe022158f280,
 v_vnodeout =3D = 0xfffffe022158f278, v_vnodepgsin =3D = 0xfffffe022158f270,
 v_vnodepgsout =3D 0xfffffe022158f268, = v_intrans =3D 0xfffffe022158f260,
 v_reactivated =3D = 0xfffffe022158f258, v_pdwakeups =3D = 0xfffffe022158f250,
 v_pdpages =3D 0xfffffe022158f248, = v_pdshortfalls =3D 0xfffffe022158f240,
 v_dfree =3D = 0xfffffe022158f238, v_pfree =3D 0xfffffe022158f230,
 v_tfree =3D = 0xfffffe022158f228, v_forks =3D 0xfffffe022158f220,
 v_vforks =3D = 0xfffffe022158f218, v_rforks =3D 0xfffffe022158f210,
 v_kthreads = =3D 0xfffffe022158f208, v_forkpages =3D = 0xfffffe022158f200,
 v_vforkpages =3D 0xfffffe022158f1f8, = v_rforkpages =3D 0xfffffe022158f1f0,
 v_kthreadpages =3D = 0xfffffe022158f1e8, v_wire_count =3D = 0xfffffe022158f1e0,
 v_page_size =3D 4096, v_page_count =3D = 65342843, v_free_reserved =3D 85343,
 v_free_target =3D 1392195, = v_free_min =3D 412056, v_inactive_target =3D = 2088292,
 v_pageout_free_min =3D 136, v_interrupt_free_min =3D = 8, v_free_severe =3D 248698}
(kgdb) print vm_ndomains
$2 =3D = 4
(kgdb) print vm_dom[0].vmd_pagequeues[0].pq_cnt
$3 =3D = 6298704
(kgdb) print vm_dom[0].vmd_pagequeues[1].pq_cnt
$4 =3D = 3423939
(kgdb) print vm_dom[0].vmd_pagequeues[2].pq_cnt
$5 =3D = 629834
(kgdb) print vm_dom[0].vmd_pagequeues[3].pq_cnt
$6 =3D = 0
(kgdb) print vm_dom[1].vmd_pagequeues[0].pq_cnt
$7 =3D = 2301793
(kgdb) print vm_dom[1].vmd_pagequeues[1].pq_cnt
$8 =3D = 7130193
(kgdb) print vm_dom[1].vmd_pagequeues[2].pq_cnt
$9 =3D = 701495
(kgdb) print vm_dom[1].vmd_pagequeues[3].pq_cnt
$10 =3D = 0
(kgdb) print vm_dom[2].vmd_pagequeues[0].pq_cnt
$11 =3D = 464429
(kgdb) print vm_dom[2].vmd_pagequeues[1].pq_cnt
$12 =3D = 9123532
(kgdb) print vm_dom[2].vmd_pagequeues[2].pq_cnt
$13 =3D = 1037423
(kgdb) print vm_dom[2].vmd_pagequeues[3].pq_cnt
$14 =3D = 0
(kgdb) print vm_dom[3].vmd_pagequeues[0].pq_cnt
$15 =3D = 5444946
(kgdb) print vm_dom[3].vmd_pagequeues[1].pq_cnt
$16 =3D = 4466782
(kgdb) print vm_dom[3].vmd_pagequeues[2].pq_cnt
$17 =3D = 785195
(kgdb) print vm_dom[3].vmd_pagequeues[3].pq_cnt
$18 =3D = 0
(kgdb) 


Adding up the = page queue = counts:

6298704
3423939
629834
++p
10352477
2301793<= br>7130193
701495
++p
10133481
+p
20485958
464429
912= 3532
1037423
++p
10625384
+p
31111342
5444946
4466782=
785195
++p
10696923
+p
41808265

So, about 23M = pages short of v_page_count.  

v_wire_count is a per-CPU = counter, and on a running system it gets added up.  But trying to = access it in the kernel core dump yields:

(kgdb) print = vm_cnt.v_wire_count
$2 =3D (counter_u64_t) = 0xfffffe022158f1e0
(kgdb) print *$2
Cannot access memory at = address 0xfffffe022158f1e0

Anyone have any ideas whether I can = figure out whether there is a page leak from the core = dump?


Did you = looked at UMA/malloc stats?  It could be genuine VM leaking = pages,
but more often it is = some kernel subsystem leaking its own allocations.
For later, try both = vmstat -z and vmstat -m on the kernel.debug + vmcore.
Often the leakage is = immediately obvious.

So, vmstat = -m doesn=E2=80=99t work on a kernel core dump, at least on this vintage = of stable/13:

# vmstat -m -N = /usr/lib/debug/boot/kernel/kernel.debug -M = 2023-03-15_21-23-27.vmcore.0
vmstat: = memstat_kvm_malloc: 
        =  Type InUse MemUse Requests =  Size(s)

As for vmstat -z, we=E2=80=99ve = got a script that adds it all up, and it=E2=80=99s ~80GB on a system = with 256GB RAM.

The largest processes in the = system add up to approximately 24GB of RAM used, although that is very = difficult to measure precisely because we=E2=80=99re using Postgres, and = the processes that Postgres uses share varying amounts of RAM. =  

But it doesn=E2=80=99t seem like we=E2=80=99= re approaching 256GB.  If I can rule out a page leak, then the = reason behind this change could be the = issue:


The ARC in our case is at about = 65GB.

Thanks,

Ken
<= /div>
=E2=80=94 
Ken = Merry
ken@FreeBSD.ORG

= --Apple-Mail=_2710E057-8BAD-4F01-9DD6-48E4ECCB364F-- From nobody Tue Mar 21 21:37:30 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Ph4hW2jZpz40sQq for ; Tue, 21 Mar 2023 21:37:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-55.consmr.mail.gq1.yahoo.com (sonic307-55.consmr.mail.gq1.yahoo.com [98.137.64.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ph4hV1sF4z47j8 for ; Tue, 21 Mar 2023 21:37:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=FX05UPXH; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679434663; bh=yvBDuOMyAcC++2LEefvsrDpxN5rv2KZT1lDM/CTaVGA=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=FX05UPXHvckWt0ZQhUdINwJvM1p1vxdkIydJX08TFnClzAPrFrtak7EAVobeDnSapVEzxU1x5AyVsZnv1bWgWO+sW7G9NsakimBxHgWWnjc9LQIiKASNNcuEViV20u5FXjhexybNL6KrwcFuld4Erzl1R353PES1z6K1EW9AeltZfiZTZjVhceyUrTHVqMAw5jLY7RII+Bj7icbWwhryPSlJRWl5ok3HMjD57WeMAcUUUNS8Wmh2C0JFSv5k/wlNU9yzDiBi6E5DZQuAP6Oy3D2aEtcc4b8vGvkrtr4jpVCooc2TDy671gk8SRcGcSJr2fTmsL3w2ae8zPloDKcI0g== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679434663; bh=MpsmEFhkRWCNcfm+LGMhV6wRLXke3nkc5d1piZWTSCe=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=bo4ggrKSD3ZwvrPGSDZxIzsVvSWbwsEjU4HoOW7nsQSP/v9saTgChA+P6gsoFXB2Low5u46T+1p1NuU8jcUhgUW5ZL9rxTS004kOyVuUr6xkrc5qleWcxzJkY3quMMmratJjvKsUBfl2+A4u5RQSUZAIx81GtykPzux5W18uhUI6kEux1hKUHGIt4y1Dv4gf0wGJ1CCRJRcsFLU3X0e5XXtZ6VLuDf/rMg2fYEYJhFIK79zrCm57Q/u8J6LL/l8E+7mqzKYCWcFKiWermiYVvlcLk3gDFW7yzllJwUnSxJMH3drCoZHvjOCdjv9zN7SWh4Zr/XJ77sokUftzkq1uUg== X-YMail-OSG: irKfNRwVM1mZ16Ej.PZcZHv_Y4H8ghwZxLYiLgjS1FelfqoognZmOPPyItJgmZv 9upEQt_8ksri4yIlrCtSrynbuttI7cHlDTyIOJ1SOB7w49mv0t1lPkdG2t28kuj9q2zDNEGAHu62 Xworm2I52CZkggRYyxFuX4R0e_JnC5P1iZqW9v4UquMEGKUeuaXu3cETo0XwCrVbqnUfJ20_8AFd H9dQPj3Syt0NaPMsiikjXTLm9YwCAShhAtFRL4YZ3iMq6fmq_YPSuhssUiO_XcO1K7cexLp_DuTr kx2DF4pqaciChrFvXE2OfMO17M5A1wIefeX.V.alK65t_3z4wJOsMIPBv7XA_.RMPzPj6Pa8ebGm M9yMDMv6nL1UAwjMWBt21cjCcOdimO6.E2YakZ3v47NSwTSVsJ_wB.4rU91gw_Ts5VrMfRmQl43B F_rew_ym6Tt.2Athc41dHedfmmXIG7shM2wrAt.O1BsK8O8BFlS.wA6NeMlhfP_DafvsQizQuZCU 5KuyubjGulH5I0_e4KgzBuHfldKAkmy5OmYh6_jcDnmLuHPmGMOlZew_ogs9w8EzZPi6FdDfpexG zDm3ICfpQXNPt9OKd16PijFJDdzGv3ItidikytmgnEQIplCPp1SXq2B2RFkSTR_qr7Uwi8CRm1Lt KoRrm0qpCA6gXaJhA_7HBIcZINp10RaekZtIX2Rwn7_kgid2LeYkRPFjEIVA3BsjGHyVW7lG4Z_t .x2qlgEN2KuYbY8emAgIUgEmsWmSeALlmwak7XIsnHNDOEdXKoH4ogthdd.6v6HKkYi48aRzXwD9 XCtOw9QviVQHHCipkQyv_uy1pk4O2cZWSmI3dOTmkIiW5GTcqgQALnbc.7mtWk_6Oa6MFHOxhqgz wjwgEFPFAzRnH0vu10uVUP7nfOMWghskaB4v17bpIcBe_xx1bxmJcw.brLYKUowCR5fCLrIPH1z. 1G9NhrPd45ovkWOpM5x_OunslDrohNRzLG3NVCCdXl44Vl31i16dreBIDcJ3hPYMfdIbjmMtQl2i XJmjktgHVUc8Jsclstg_U1QFS22Opk7EjDPH5GYrCOI8K21KcAnPn2oDqXsuiyu4jckeEEzaZovh JiRa0qn_F4MPSYArVipbQOQImgdW0r5rsqC1.LI1O.FJZBcaW5ZY.6Z0i.bU7MGAmNQ8kaDjP1k0 bsN7L6.PyqAMToZvipkbj.qNskCwGSonedheAvXqM3I0UyXzA03r6i67K9GTOGAhGix8JaVaFAt3 kaytYbgElsbk7NlgeazvqQ7A8xhyHHpY6gLUbhidhIL9lCav96S2BAS0I1lLB.pq4c4.rNcxXBAD m6pI91mvtZFmd_kIPbnbt6iBiLtnP_PaPpuHYvynMPnqdpaZ_izQYhn_xRQ9IG3ZOl2kpWw4h.EU NF49Vts6sgr_goyLlCJBcovadhPLiwRqrbwPr9s0MbmGlHL0zTEoxKff0SkISqidb.Vizd6GyA3U q5w8kPgeWI5OtILaPiD9munyojRXRtivxbmhkA0kOKMdJw5qEmcCBrTd4ylunp_TiczlbrC73Y9P yCxN63jDj31HRWYIGl0.EoP5nryoTerYbR_SxshgPS9bYPHbub_Lw0vAJT.mn2qNHtUTQ4P2vYhs 8Z_az2Q_8rfOSAsRgcE.suA9qG0vYqTsmJayP9g5dvTSGyaG7F2wrgSlr7ehC87DxXqJHNhcAfdz msMSLTKbEESbtWX3OK2BdbdnT1i12UT2GnwCq3wSz0ErDm9BB3h59v2D2KUSNbSAPaA9B4VqANqT JzvC9aivpITmkdus.21sNN6WlW752nOs7KMMVIVjmZ7Jo6kCd_IxH2TGt4rj3IN3IEQVlD9rMywd SxHjM3u9JRFA2lgUfDb6LTIH6b0WLfk.tGcH.SeFpNtLi6D2bCMeV7sfQHkMTE2g8_Pv3eTWoh8s cHlkqAUqffPg0byppSvXTRMRH7jOxVr5HUAdZ.AzDLIJjG9f354SPRo33sj8k.E1iSWkdxMrgpKV _NpMw8TeGYeGftfw3_UQTIf3LP8tTvVdD6bRKUjsqd0dhLclMGJ9ohnix_dBuZYYbiGddxpOIpwx 37omfo5pJK.V94VKxn0K23o.iISRmmZ9c1l8qelCoiK4IIBXouAsoS_0dOMD5_3Ob7VOqRnmmxkB XfQ7d471Jab3My6eYX3DUZwcyO06SOBEZN2nyeZ2fziiZITAu5l0lBh7vRkdtXVSqElFhv4yV.Ml aeqtswwBvLQ_RpqzTa_Xl_Oy0iEN3nIlUlSlXdMHezRx.oYPnHqaO1d4ovub7QqzAhG0dMdeb6yI X X-Sonic-MF: X-Sonic-ID: c4bea01a-c9ed-4420-81ab-d59afd99b153 Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Tue, 21 Mar 2023 21:37:43 +0000 Received: by hermes--production-ne1-759c9b8c64-gl278 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 19f181ac1e6c900a5153a94efeb22be5; Tue, 21 Mar 2023 21:37:42 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: RE: Getting v_wire_count from a kernel core dump? Message-Id: <5AFA0CE6-48EF-4F41-ABE5-2B417A6BA271@yahoo.com> Date: Tue, 21 Mar 2023 14:37:30 -0700 To: ken@freebsd.org, FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <5AFA0CE6-48EF-4F41-ABE5-2B417A6BA271.ref@yahoo.com> X-Spamd-Result: default: False [-2.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.31:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Ph4hV1sF4z47j8 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Ken Merry wrote on Date: Tue, 21 Mar 2023 15:11:30 UTC : > I have kernel core dumps from several machines out in the field = (customer sites) that were out of memory panics, and I=E2=80=99m trying = to figure out, from the kernel core dumps, whether we=E2=80=99re dealing = with a potential page leak. >=20 > For context, these machines are running stable/13 from April 2021, but = they do have the fix for this bug: Originally I was going to ask which message(s) were produced. But then I noticed the mismatched time frame ("April 2021") for the general vintage of the kernel involved vs. the modern mnessaging. QUOTE author Mark Johnston 2022-01-14 20:03:53 +0000 committer Mark Johnston 2022-02-28 14:06:58 +0000 commit 13ba1d2836762d09f338158372806b90ce3d63ba (patch) tree 5547db830727964bc7cb6c68a60fb062f85a5010 /sys/vm/vm_pageout.c parent 19624b4c6b67881812e2bec0808a9d831cc61bb5 (diff) download src-13ba1d2836762d09f338158372806b90ce3d63ba.tar.gz src-13ba1d2836762d09f338158372806b90ce3d63ba.zip vm_pageout: Print a more accurate message to the console before an OOM = kill Previously we'd always print "out of swap space." This can be = misleading, as there are other reasons an OOM kill can be triggered. In = particular, it's entirely possible to trigger an OOM kill on a system = with plenty of free swap space. =20 END QUOTE The change leads to messages that look like: pid . . .(?), jid . . ., uid . . ., was killed: failed to reclaim memory pid . . .(?), jid . . ., uid . . ., was killed: a thread waited too long = to allocate a page pid . . .(?), jid . . ., uid . . ., was killed: out of swap space Each of the 3 have rather different implications. The last is actually a misnomer in that it really is one or both of: swap blk zone exhausted swap pctrie zone exhausted Those are kernel data structures, not directly swap space on media. You may want to pick up the messaging change. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Tue Mar 21 22:52:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Ph6Lp2Ty2z40x53 for ; Tue, 21 Mar 2023 22:52:34 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ph6Ln2RYYz4HDR for ; Tue, 21 Mar 2023 22:52:33 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32LMqG2A096172 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Tue, 21 Mar 2023 18:52:21 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: Date: Tue, 21 Mar 2023 18:52:16 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Content-Language: en-US To: FreeBSD Hackers From: George Mitchell Subject: Periodic rant about SCHED_ULE Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-3.16 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.86)[-0.857]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@FreeBSD.org]; TO_DN_ALL(0.00)[]; R_DKIM_NA(0.00)[]; DMARC_NA(0.00)[m5p.com]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; TAGGED_FROM(0.00)[freebsd] X-Rspamd-Queue-Id: 4Ph6Ln2RYYz4HDR X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Yes, you've all heard it before, but I've just reverified on FreeBSD 13.1-RELEASE-p7 that SCHED_ULE gives terrible performance for "make buildworld" in the presence of a totally compute-bound job (misc/dnetc from ports) running at nice 20. World builds in: 15597 seconds with SCHED_4BSD without dnetc 20477 seconds with SCHED_4BSD with dnetc 16006 seconds with SCHED_ULE without dnetc 50290 seconds with SCHED_ULE with dnetc. When I ask why SCHED_ULE is the default scheduler, I have been told more than once that there is some circumstance in which it gives superior performance. But no one seems to know what circumstance that is. Guess what! I propose that SCHED_4BSD be the default scheduler. -- George From nobody Wed Mar 22 10:17:51 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhPYY5DKBz40PVD for ; Wed, 22 Mar 2023 10:17:53 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhPYY4cDkz4JLF for ; Wed, 22 Mar 2023 10:17:53 +0000 (UTC) (envelope-from mandree@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679480273; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XoVcIzKvL39SxpApt7xKNP5SxL8ZuafrEyJlsOtjlpM=; b=LeYmBwsACC5/WbwMnSIAsdFjE6i46MxZtTL54mrhRoJWDhBENBXOAEkGWpqhgXOw1qQ7Bd KaPpNggIzUIq6SCEkvkN8UzMrOo4omMIOTX6vbiLg0WXzmYdCAghpLMclDT0avwmPrff6v ahyQMj69B4QO9FfZmfy0IBa9dSDqcrCbrx1zvjc3SizqvRG0pEPbYyuA4VZpYmJBjZL/v/ QMIB9dX7Ve/RjSZtJcfOXJHtUEOQ6balyACjrVho8bcyzhKFwg2jUhbirCXvXdMOnNAA9T TPJe0+hPAm4enTJBnMlf6Lvt59GRspouX70CTPQn74+DcnOxwkktvDqSeIb+wA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679480273; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XoVcIzKvL39SxpApt7xKNP5SxL8ZuafrEyJlsOtjlpM=; b=jQLhDgn2e1O5LGEVIVMkHAYM3pSksqePs9YjMjmEL0YIPqWCJLszUvIl0E3OM54V9cRXaL DGNSoOOOsCIwn2jyJe11Ch/PBS9DPpLUo6F1FCf1yu07rztCioMuxAPbHAmYlXUnOENLV/ kUjmveqjqGly10X4cJ7xulzZJWzH1XXPovhK//9R9cYutTv7647o0ZU/jhwq6md9cxDz0v G0FjQ4hvcsEw8cH7yjvirsbLyL/QP6c07h8KnsPtpKd077vZzJDhaFvfw+/ap1tHSnjbiX tnTScZp4iAA+XRiO2gZrWh5A72QqFiRY91J5oRwHYW3VfB/V60lCUd8peI9i6g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679480273; a=rsa-sha256; cv=none; b=OFuayM70Sj+xukA/2IkDNuCaCvI881RecAY3DFHbXVPrROt0l+TglFypyDQAKZJM1mc1Bq nWc/8pVC3jEK5IsRQYL06S4xRF3+fgjwz80cCrPA3CVcZnhd5IjJHrA32RDM6btpPQ5xdR Lt/tGGfZu278Vx1cYpqufGJfzZZ1UUBO9UqVDF8LfZOEBFo0B9OVeWFV8qGL3/89b+4XkS MKm0kVLr7sQ79ZUySogFGngvxhJv4ShbC35DuO8fmLES8u62lk8vHlt2TtjsVdIsowHztM SdoGCVCSa5/cCNKUrs75GU//uCcSiZUUuuUJpCvCllmQfkBbSfN6bbWJY/A4uw== Received: from mandree.no-ip.org (p54a03c23.dip0.t-ipconnect.de [84.160.60.35]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mandree/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PhPYY2mbwz1270 for ; Wed, 22 Mar 2023 10:17:53 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from [127.0.0.1] (localhost [127.0.0.1]) by ryzen.an3e.de (Postfix) with ESMTP id D798CDAF8B0 for ; Wed, 22 Mar 2023 11:17:51 +0100 (CET) Message-ID: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> Date: Wed, 22 Mar 2023 11:17:51 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-US To: freebsd-hackers@freebsd.org References: From: Matthias Andree Organization: FreeBSD.org In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N Am 21.03.23 um 23:52 schrieb George Mitchell: > Yes, you've all heard it before, but I've just reverified on FreeBSD > 13.1-RELEASE-p7 that SCHED_ULE gives terrible performance for "make > buildworld" in the presence of a totally compute-bound job (misc/dnetc > from ports) running at nice 20.  World builds in: > 15597 seconds with SCHED_4BSD without dnetc > 20477 seconds with SCHED_4BSD with dnetc > 16006 seconds with SCHED_ULE without dnetc > 50290 seconds with SCHED_ULE with dnetc. > When I ask why SCHED_ULE is the default scheduler, I have been told more > than once that there is some circumstance in which it gives superior > performance.  But no one seems to know what circumstance that is.  Guess > what!  I propose that SCHED_4BSD be the default scheduler.     -- George > Can you please also give figures how much CPU time has been allotted to dnetc in that respective situations? -- Matthias Andree From nobody Wed Mar 22 14:41:43 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhWQB501Jz40vT0 for ; Wed, 22 Mar 2023 14:41:54 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhWQB06m7z4cby for ; Wed, 22 Mar 2023 14:41:53 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32MEfhr1001415 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 22 Mar 2023 10:41:49 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> Date: Wed, 22 Mar 2023 10:41:43 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 To: freebsd-hackers@freebsd.org References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> Content-Language: en-US From: George Mitchell Subject: Re: Periodic rant about SCHED_ULE In-Reply-To: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-0.96 / 15.00]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.99)[-0.986]; NEURAL_SPAM_SHORT(0.22)[0.222]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; MIME_BASE64_TEXT(0.10)[]; DMARC_NA(0.00)[m5p.com]; RCVD_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TAGGED_FROM(0.00)[freebsd] X-Rspamd-Queue-Id: 4PhWQB06m7z4cby X-Spamd-Bar: / X-ThisMailContainsUnwantedMimeParts: N T24gMy8yMi8yMyAwNjoxNywgTWF0dGhpYXMgQW5kcmVlIHdyb3RlOg0KPiBBbSAyMS4wMy4y MyB1bSAyMzo1MiBzY2hyaWViIEdlb3JnZSBNaXRjaGVsbDoNCj4+IFllcywgeW91J3ZlIGFs bCBoZWFyZCBpdCBiZWZvcmUgWy4uLiBibGFoIGJsYWggYmxhaCAuLi5dDQo+IA0KPiBDYW4g eW91IHBsZWFzZSBhbHNvIGdpdmUgZmlndXJlcyBob3cgbXVjaCBDUFUgdGltZSBoYXMgYmVl biBhbGxvdHRlZCB0byANCj4gZG5ldGMgaW4gdGhhdCByZXNwZWN0aXZlIHNpdHVhdGlvbnM/ DQo+IA0KSSBsZXQgdGhlIHNjaGVkdWxlciBkbyB0aGUgdGltZSBhbGxvY2F0aW9uLiAgVGhl IHJlc3VsdCBpcyB0aGF0IGRuZXRjDQpnb2JibGVzIHdoYXRldmVyIHRpbWUgcmVtYWlucyBh dmFpbGFibGUgd2hlbiBoaWdoZXIgcHJpb3JpdHkgcHJvY2Vzc2VzDQooaS5lLiBldmVyeSBv dGhlciBwcm9jZXNzIG9uIHRoZSBzeXN0ZW0pIGhhdmUgbm90aGluZyB0byBkby4gIFdpdGgN ClNDSEVEXzRCU0QgdGhlIHJlc3VsdGluZyBpZGxlIHRpbWUgaXMgMCAoYXMgcmVwb3J0ZWQg YnkgdG9wKS4gIEkgZGlkDQpub3QgdGFrZSBub3RlIG9mIHRoZSBpZGxlIHRpbWUgd2hlbiBJ IHdhcyBkb2luZyB0aGUgU0NIRURfVUxFIHJ1bi4NCi0tIEdlb3JnZQ0K From nobody Wed Mar 22 16:20:27 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhYcH13Jjz411mg for ; Wed, 22 Mar 2023 16:20:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-24.consmr.mail.gq1.yahoo.com (sonic303-24.consmr.mail.gq1.yahoo.com [98.137.64.205]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhYcF3y44z3LhT for ; Wed, 22 Mar 2023 16:20:45 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=OpdR2pWh; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.205 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679502043; bh=h52jbG+Sztr9/eVJ14X6yTIKSVWRNffNRC+vSPcT1Dw=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=OpdR2pWh6+yKAfLnxcu7mEvW+E+O6VsTRBB5lIbU8GajHZixlZicnC0Z6ul0H6+nsS9jFaFSZDqVOvSlRETY15N6aY/zLp5XyZkaygJNZl5g4DU2t63hPsf+hm7x15BqNC/g/b049913TOTtixwF52VK/ruDqNFDJB9ujKPIvA2R2TduL63gmwlwuBxtsrsncY1fjow6ppa6vkGQ/BpYomcWMfNP5s6fVr3Um6zzGtdNH3zzuI9NevsXZ2MJJr5v/R4zOQEufhIY8qn+mIYoNp87sS+CRQ8I6kytXQqqyAHQiEJFFn5X79gZKQvyT1TR4YpQpbX8crnLbZPhy1lfrA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679502043; bh=07HPJfCP8ZHIeX0vRikH/dRSNT8nFJaNK8gV0FvHk3V=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=DS6vieZ86w5QBqgQLUFxKCvIy0PXwce2q8ZuT2hAn/mQp7wDnwo/EGiczEuRPvqsypEBv0ZtNGEdhX8AeoxPLw4WVCqY4imaBE4ruyfbcsBiroU73Asl9HeawfWuZ6UelneCZrPSTo10/V2ZXlZ4Io6QauzmAvQ1YCcZUMmmb3vspIMkTvWI5I7qWOHEB5QL30EcMaH/Zv5ook/n1dTIYtj2QrM3DLi84Zg0/Fv4aRaaLjsCDLTqtCPgVROAh6Y/PhqqaqseXW7+qtMk2wv/8lLVbKATegPfE5BABXVhk3xRrFT9PsD/Eu/WoMoR4XjduTEngqUshn2oSrc6wTVENA== X-YMail-OSG: Fv.Q8rwVM1mBLoyRSgXSIWFCSQOG9usO.vJ1o8PFbtwdNZvfGBjHcrr1x8GYmkU M0NV7kj0NWE3i6atG8p2XNMfSNYhO9NdV6NA1bQ.0sgF9gIFxdl0RfOSOjUFlx9iIHd70a.hAZmZ ZjmlvUIPHUbdW7s_dG_YkWttA9J1dB0.dRAPuVl5WogqX1DlRl18Sc82TVeG0_Z3wkaw0DM7iG1U Vj0zSL6WPlEQcYMmjTADBMKeMLDeAEIEJDwvx2gGLOLDMRRr2A4SEynhnLmIOHNQgSv0yu28KoS2 ANoRKULfo5pVOWKgcjiVlS9BPTGsL9haQQLEL6TGKvZ.WbrYxjO2x873.zAAhZzCQtcZuS2vVySQ XU20sfs54GKrfmn.T9iJUouTQqayC9rRK5oqP5s2RwOdUPcrnMasmBog0ylsJcnMw7DASYPbkqZh 0TEkjrTUsMwPc2CT1h_xj9F5SykVjXox12f9y190OmZ79zld3lFR8x6J_4LEAHg8prdlH6ICyloP oJoh7POe7C3jMIkgQqsQRDejd05gR1iCNItyJgXWFNMpYGQA0sojtQ_zm._0wpy3RKoxmZsO2dZX ozcABDTJJCUD.AU9XoIPC691bjBkvkbue4T.ab5oNtZvD3hmvTleG5x5MGuFamCY7EmQyB5ldvUX hsTW2fKXwp8614qNKmks8gP3K6mXgvN14ptKFnb6y.NZ6AZUDSk6arjWMIKKgUWSFGEFfAMWoAs5 TxNXxsuWfuTB5qaUTGWZ0QWzrszIM1ZSfSWAuiRoZUjjHfDu.oJdk74WOT3cFm4jga3xXdFInDcj sNvkwur.lq6GL.K7L0aYLF1kqeoTHutNRZd2Pour6NVxK03QzlcFKXPIszC9YkRAmaTSseeJ8aEi ZDwjH9NOoZDL.trQgQVs7Tm_Cvus9eydyMsZL9Kl5cwVSUrTVMwqGLsbySqI8EhXwTtWkO5wAMaU s5vN59iD33x8yl4KvzMCaMadnCCwaEguOpKmMi89_48EDC8KiccXfASicacJPhrHHsA9AWF5t3V5 w1wcuTeSQ_2eEQdZG_Hy0lGBtP99rymyKVTn8.j84Ggo1GusTWNSiTC1BgLzGSiEphOHi6f5IjT4 5mqUzlPsfqpyNe5nWA6DASytWNdCWwk3eQ4tJ.kSYaog1BqgPkCbHaH88KNj9XwZ1oVB7UdeGC4m VKxiBZzBDlHILDyVdO21aIL.fGbUDBqShIjq5YoIpgqG2gGqZIkKf8C4OfkbgO8O2SxafwD4lNkV 8rjEqZG2V93Kk7WaCXbsCxGTA4XnPFWoO6t31BfhwwUFBA6zNbeRUkwOyVyJRTTyBoudKOTVMj9u bh48pnyiwcoPiWiuQb4F4B9XYePsf7bWXdHkSVORlAhnlHz24Vk69dgfOOwDO79JttMigcTfONVf CqUMLxgLnjpUx6Y_nTRIsqEt1TWbA9PTRWXJ4Nj.IHzqYKoF6Dqg31Ig83jyCebpzi2.EoT7m_Vy TxzDq7ziwN6sWofEw7IpHH12Gkcs1Ro0yWKRCFBFye3LInb0i4WThLVJE_OjBkx6_DSH2Qv4i7oS 1mjN3clFL.k0PeRTGtk9aWEAQXvmEMUdGIFjaB9XdqOF8olCrOW_ZSLYhzCviyjGtOZfaQBeyQgs DeQlI3mHQPgQkhgA5hWp2Do9GxmmsN5ioUKvwUn2wLEyS1lNL8k0u3JZyNRmnyrLbcaVw0bzKewK CV01Wf.ge7lPl3jg57zADubvEemEWlmvXShO0GPYpUVC4olLYmw_PfkA8kuiNNczzKja_inr3Xfl AWd0vMK0RlFT_379Pdni70P8_rA4CUfprHFBUbA2VbxrCxnVwE6XsWC_rZw1LJqLcUfWB9yqOazl Ik5CEZkSGmdsq0szl.E_JQBHhkIoc6AJ3PLwpdpBHO10LkkygrcZpY8FroE1QKhstOESsfcS69Z_ 2WE498UOErz5jLZOJXOs_gB70ZJ79YMsFdDa1lnufqnlkIUDyDfsabgKmAbLq88l2cI8arovrAA_ 6U4OjVZ2WpihXeeTESv6vkcEzPrllDb23RpyRjlK7hNTHS.t2vLNURiqtqdbGqid5brcoghjAuou LNhYNXx1h_2nYjzoaWGbTTQjcE_q0tVb7KjIKcrkHsZQ0uk_4j4KsRo221jVPt0.UKocHxbDgDez CzbOOM2ONMRAlT6oEml5qEzOm7TKf7M0nhmtdp9t0uyGBpUUJ.qU_0Fe9tEzQbpceIjmxcUekEp4 oI7H.vF3.Z7Dvrfng5no6F_ndgjITErpOz2jVoNFCcDS_GGO6KkBihnL7jiPaDm.zmpo9IVhL4vf h X-Sonic-MF: X-Sonic-ID: 6d9f8a01-6eac-4a2f-84e1-c69d435074b1 Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Wed, 22 Mar 2023 16:20:43 +0000 Received: by hermes--production-ne1-759c9b8c64-7kw78 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 881877f8600727401990d04168bd6c5d; Wed, 22 Mar 2023 16:20:39 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: RE: Periodic rant about SCHED_ULE Message-Id: <44FB7EE7-295F-46B1-9128-945DCAC6E79E@yahoo.com> Date: Wed, 22 Mar 2023 09:20:27 -0700 To: George Mitchell , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <44FB7EE7-295F-46B1-9128-945DCAC6E79E.ref@yahoo.com> X-Spamd-Result: default: False [-3.16 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.66)[-0.657]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.205:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PhYcF3y44z3LhT X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N George Mitchell wrote on Date: Tue, 21 Mar 2023 22:52:16 UTC : > Yes, you've all heard it before, but I've just reverified on FreeBSD > 13.1-RELEASE-p7 that SCHED_ULE gives terrible performance for "make > buildworld" in the presence of a totally compute-bound job (misc/dnetc > from ports) running at nice 20. World builds in: > 15597 seconds with SCHED_4BSD without dnetc > 20477 seconds with SCHED_4BSD with dnetc > 16006 seconds with SCHED_ULE without dnetc > 50290 seconds with SCHED_ULE with dnetc. > When I ask why SCHED_ULE is the default scheduler, I have been told more > than once that there is some circumstance in which it gives superior > performance. But no one seems to know what circumstance that is. Guess > what! I propose that SCHED_4BSD be the default scheduler. -- George You might want to publish instructions that others could follow to try to repeat your results in a manor know to do so (at least in proportion to the number of hardware threads they have in their context and how performant their system happens to be for the type of activity). Does dnetc let you know how much progress it has made? If less time is spent on buildworld per elapsed time, was more time spent on dentc over the elasped time, getting significantly more dnetc work done? (So: a tradeoff on which gets the time?) Did the system end up using the hardware threads for wasted overhead activity (less useful-progress for both buildworld and dnetc instead of more useful total work done)? Did the system end up with more idle time instead of wasted overhead? Lots of questions could be formed and investigated. It is not even clear what the load averages were generally like vs. the number of hardware threads available. How many hardware threads was dnetc trying to use? For buildworld, what -jN was in use? How many hardware threads were provided by the system? For some systems, buildworld using all the hardware threads ( -jN ) would be I/O bound instead of CPU/RAM-subsystem bound. Giving folks a way to know they are repeating your tests appropriately, could give interested folks a way to answer their own questions. === Mark Millard marklmi at yahoo.com From nobody Wed Mar 22 16:48:17 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhZD96xz2z413G3 for ; Wed, 22 Mar 2023 16:48:25 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhZD91dWlz3kpZ for ; Wed, 22 Mar 2023 16:48:25 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Authentication-Results: mx1.freebsd.org; none Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.17.1/8.17.1) with ESMTPS id 32MGmHDW004065 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 22 Mar 2023 09:48:17 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.17.1/8.17.1/Submit) id 32MGmH4O004064; Wed, 22 Mar 2023 09:48:17 -0700 (PDT) (envelope-from sgk) Date: Wed, 22 Mar 2023 09:48:17 -0700 From: Steve Kargl To: Mark Millard Cc: George Mitchell , FreeBSD Hackers Subject: Re: Periodic rant about SCHED_ULE Message-ID: Reply-To: sgk@troutmask.apl.washington.edu References: <44FB7EE7-295F-46B1-9128-945DCAC6E79E.ref@yahoo.com> <44FB7EE7-295F-46B1-9128-945DCAC6E79E@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <44FB7EE7-295F-46B1-9128-945DCAC6E79E@yahoo.com> X-Rspamd-Queue-Id: 4PhZD91dWlz3kpZ X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[freebsd]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Wed, Mar 22, 2023 at 09:20:27AM -0700, Mark Millard wrote: > > Giving folks a way to know they are repeating your tests > appropriately, could give interested folks a way to answer > their own questions. This has been an issue for years (and now stretching into decades). It is trivial to show the problem with any numerically intensive MPI program. I've done this a few times, and reported the issues. Search the mailing list archives, e.g., https://lists.freebsd.org/pipermail/freebsd-hackers/2008-October/026375.html It appears to be (or was) an issue with cpu affinity. Caveat: I haven't tested this in a long time. I simple use 4BSD. -- Steve From nobody Wed Mar 22 17:09:20 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhZhc64sTz414H0 for ; Wed, 22 Mar 2023 17:09:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-54.consmr.mail.gq1.yahoo.com (sonic316-54.consmr.mail.gq1.yahoo.com [98.137.69.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhZhc3MD1z3mW4 for ; Wed, 22 Mar 2023 17:09:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679504974; bh=yHR4pQT02azDkcVikOZrFYnONK6N/fy6o6YtAvIxHhA=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=XsUoBmLmJ0phkp9Ae1QcSF0slJKx8/E0JNlpt6O6TH8ml9Nlgk0fPoIiE08c6kKV30s+xCCwfbjwCJqTu2eeWWYCz55a7VbcUdaz1mKVnvrGm4p0XhD4iQx72ZdE3GLP2Vt6dgBGZbO3City7N06WmE2c6yD2CnCnZC7mV4dxMRnnmXUoIhMlyFaS3m1jJ+5dGwJMPqWrm4R1UG8F0etFSEIcF40U4r0HzCv+a9H8dGC6yptcJuP1quT5u4yNWeodL+Jp39SPoajJEhxVBUMSRRmVQyHZkTuedfQFOZwontZzh4szMPFDj8c7mjZrD7TpR2hl/M1noPFTwcZFdGQjA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679504974; bh=owSzNioSKlRf23NsqGLxxANaewuGfLnvEqR7tZS8joE=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=A79k264jOcd62wAPJhqlXuJA6k/dAGNaeG0HiwToLoVz+WsT5VpqpjBoJojGEzixw9WOQe0JqSXNuHHZIcfxoXQHMlINZiiqxEEJ72uqJQ+cY9guNvPv1wVNN5xeYFE8QEK9zc8CCBaIuSWwH46fRMxhLmZJDYrTSYhipewmY9RQaq5mA5kfs7WZzaZQZ8l52AlvyYmTGe3igbJHJksHx+AfkbhOwOGV0gZi2wd5ZT/xNFPOHGDptuHJj0vxggiY6+Br/sFTQuGBVxj52Ddt0JWXfj3FDgM0S/sQW6I42ISFgqiEtgnfaltltMMijT1/Of1ftS7Todw7h9bNN0Eltw== X-YMail-OSG: 2fCi0ukVM1m3DBlfV7SDqTM_0JqrS.roPOnGa2VpE2PgItu9hezWNQW11gRU3J3 8aOdCPVAAnZNcBFm65SrbEiI23HfKAW6SC.03tBqPD.IfoKWjrMKq0fmHa_FMDtuF_.MQUak2Hx7 zs6GuaTlRzMpqlNch80HMCVqgKM0zATRV_Q5F5od8PDbsYjwcttNKTKeLG6T437qypF85nkfM12d TAjLdb1ErwcKMd9FfJcLL4crDJVWbPYzza0CFcd75oWsgI1ixGcwp.686SA_fgsw7kegTDtrSUoH R8sjza5OcfiXtalnC0VPaGHPmgMiyq6M9u3ToAs1H0CDIsKAsGEJ.2Ar9Tqp3A1_TyUkT9bH5JWS 8LTbjZhaRhgwAY_WaipF6Iv4yjD71JMPnN.sXp_8YQsiXDBRqsKrOnXLVhJdlBDXP1ESf1zSj_iI ZdYzIie9.6oZO4TJ_yHZHQfB4ykIa2_03n6iUrtIxL5qlIeiDbME7aH6CT0dy0IYDwHV84kz5kha GvrFed8E054UEF61oZSUCybDdpXT9Fmd48ogkKr4.8gp1KXK8aMzxbwMBlX3ybOhY0b1JeZpMpWs CIW519_Rdp2SE4WhBLO1MK7yJhp4.hmjQahJ9wZACXsIgUt.23yH9O8iEVlzjuBtsOz0Lf0Zuvwm xLCtm8OHNomaqc3ZNEZ5Phvp2FAm29vjj6VupUjUdVpJcTHF3wEBe2YX2Ksb8g_d3JC2s1bYLOzd RDmSVdqzA5MEyVCOMqK._UllIQUJBF80zntXmIrZVUgLTavtDirTz29pFcoZDYG_XKB5tP2SL9lU cssIVSoSWojglej5vhTLa5ij22yUKrrkpb9ovIG8xX1.Cw8F5UQ59ya9kLIg0nBfTf.GUWmdN0kX DMSgSup9A1VATgbYTRI.i0gsTq_0kaW5ozKlJ6PL1W830zNvJFPrNYQrSQ2w4lw.YzjeX3T55fiX T.IAd3Cnj6INBe9VdzZCiINVlnzHtm1iZhGzs1Ks6IqQ8lj0gKsiY8FvyR7Hf6C1dQMOlA0K2Oem NV2koRTPcI60M3CfVl29KOoTQm1MAPSjRpOGHUSU0RBi7i2VB4z8MyJXwW4RbjQFKaqQ3dxwNz8C cTyeuPi.DzmsdV.w_eTrC_DQCGCc3pn_9WsZgZi23gHEi12gp6Hrg48FsQCziMwECRz0FUHBzjMI XIfZN7Th7N1_181xFrf3QTdVpJEiIRhoRQp6fdRlH0NglHyOkQ.pO6CvoeCbuO6WeIO7jAmOGTYL iOrsx36CU8TTjWspjJDiJYL7vQzHJVwvntcGVE21vDl89tcQPNviCsOWGHscdZOSyc_VOS26.vvd h6vESUHncfIbS.xoar17kHkqmGVa7byFbW_PZoBMyK2Z2ePX6d.QL9w8Q1KIU77HIN5NA.HfMN0Q YNpTzL4E3zz09a.sGJMsufkJZMrfbkXzHY6Id4m0jBhpOW9y_s848g11_6PqzR4waJfg3A9HM0sz 1MYKmr7dJsL15tiZUZ9vBBN9RVoqEWtBqQX1H9a8XNUsDDGbVMmVpoIMk5JorSxK_OhgXbgtXiFb ypZjLV1cNqbQDEpw2v_.jqJFANeulb6WaM.RSnvoWpIYSENMQPR468RYtjqwFN4NR7BlduKvlOjJ VC9.Rj3Vgkx3f4E7.nYBL.utHBqM6ULUxGIF3p72_hu13Zw_8vhC0Y7SVzMU6WyzE6ZkgcsN2nAm COUkUTh6SOaS0Jn.lUyc2_WixzrUjDcl4gfy46mhEI.6rsxYMfRvcWawaqOazNPWeiOw1eSy92NL OWLpyGkfXH3IOrBTG7sKfZI7IS180331kf2zWtSIDfLBb9z1URoYi.kCrlz6NP35147zN7ovAdG_ swL4Qp5XH7IymHSBJ.3mcMvmqU7L0BZq0RQMUYtWIIAlBw81Lf2QRTu2rr6z80wTMfvwooUfBarD XirrWZu_DXJ7I9mTCs4p9lp.g0gJMPK0mjUkiumbM3XTaMZgg4JptNnUgsmZhZJMaLkV7Ligg0Np .3ilDAJpyqdnAGsXlcR7c.rib_aeUKAWqnihZPOLTQ9.m7YDoG4fORB5UW0uu_86FbMVf33pPRJv VUbr6LdFeOnAnBolfWZP.gHnJ3pgJY2RGyFNzUUQdKJ8fsckPD8IKlP7Ugv9e1OpCfSqc04zUWX_ kvvVqxrK_yLNosxnxPlNZY27yWoc3XEu.eD.zIUawY7t0kAxdt74X1XBZArLJDZoDM0R8veyqAUq r1eRQAwaGfuSLAEjz41mVBLX98zrVSmxo144wnVDvmYGms0vSL2BQHaIlhjcc4Da9z4zNJt1ipBG UcA-- X-Sonic-MF: X-Sonic-ID: 0f6d6d2d-6681-43a9-a045-9e521adbb371 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Wed, 22 Mar 2023 17:09:34 +0000 Received: by hermes--production-ne1-759c9b8c64-rg9jt (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 3698fc0ad65fa3c7ae7f6970933a9585; Wed, 22 Mar 2023 17:09:32 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Wed, 22 Mar 2023 10:09:20 -0700 Cc: George Mitchell , FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <79DA50DD-1B3B-490A-B639-37612203D23E@yahoo.com> References: <44FB7EE7-295F-46B1-9128-945DCAC6E79E.ref@yahoo.com> <44FB7EE7-295F-46B1-9128-945DCAC6E79E@yahoo.com> To: Steve Kargl X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4PhZhc3MD1z3mW4 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; TAGGED_RCPT(0.00)[freebsd] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mar 22, 2023, at 09:48, Steve Kargl = wrote: > On Wed, Mar 22, 2023 at 09:20:27AM -0700, Mark Millard wrote: >>=20 >> Giving folks a way to know they are repeating your tests >> appropriately, could give interested folks a way to answer >> their own questions. >=20 > This has been an issue for years (and now stretching into > decades). It is trivial to show the problem with any > numerically intensive MPI program. I've done this a few > times, and reported the issues. Search the mailing list > archives, e.g.,=20 >=20 > = https://lists.freebsd.org/pipermail/freebsd-hackers/2008-October/026375.ht= ml This referenced using: % time /OpenMPI/mpiexec -machinefile mf -n 8 ./Test_mpi |& tee sgk.log > It appears to be (or was) an issue with cpu affinity. >=20 > Caveat: I haven't tested this in a long time. I simple use 4BSD. >=20 Looks like a csh context, so . . . # csh # time /OpenMPI/mpiexec -machinefile mf -n 8 ./Test_mpi |& tee sgk.log /OpenMPI/mpiexec: Command not found. That still leaves me trying to figure out an accurate way to reproduce all the steps to have a comparable way to see for myself. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Mar 22 17:10:43 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhZk03Gt9z414f2 for ; Wed, 22 Mar 2023 17:10:48 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhZjz6lgFz3nb1 for ; Wed, 22 Mar 2023 17:10:47 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Authentication-Results: mx1.freebsd.org; none Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id 32MHAhYR047583; Wed, 22 Mar 2023 10:10:43 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id 32MHAhe9047582; Wed, 22 Mar 2023 10:10:43 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <202303221710.32MHAhe9047582@gndrsh.dnsmgr.net> Subject: Re: Periodic rant about SCHED_ULE In-Reply-To: To: sgk@troutmask.apl.washington.edu Date: Wed, 22 Mar 2023 10:10:43 -0700 (PDT) CC: Mark Millard , George Mitchell , FreeBSD Hackers X-Mailer: ELM [version 2.4ME+ PL121h (25)] List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 4PhZjz6lgFz3nb1 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[freebsd]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On Wed, Mar 22, 2023 at 09:20:27AM -0700, Mark Millard wrote: > > > > Giving folks a way to know they are repeating your tests > > appropriately, could give interested folks a way to answer > > their own questions. > > This has been an issue for years (and now stretching into > decades). It is trivial to show the problem with any > numerically intensive MPI program. I've done this a few > times, and reported the issues. Search the mailing list > archives, e.g., > > https://lists.freebsd.org/pipermail/freebsd-hackers/2008-October/026375.html > > It appears to be (or was) an issue with cpu affinity. > > Caveat: I haven't tested this in a long time. I simple use 4BSD. I dont even try ULE any more. I just used 4BSD, as did bde@freebsd.org, ULE seems to suck when your have interactive use and compute bound on the same box. I have seen interactive in the past take seconds to echo a command. IIRC ULE and zfs in a memory contrained environment dont play nicely togeather either. +1 on the return to 4BSD as the default scheduler -- Rod Grimes rgrimes@freebsd.org From nobody Wed Mar 22 17:36:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhbHz2JCzz415Rq for ; Wed, 22 Mar 2023 17:36:47 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhbHy1hpCz3r75 for ; Wed, 22 Mar 2023 17:36:46 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32MHadet002220 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 22 Mar 2023 13:36:44 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: <09fb0fc5-64a7-f77b-c98b-5d6dd011d2a1@m5p.com> Date: Wed, 22 Mar 2023 13:36:39 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE To: freebsd-hackers@freebsd.org References: <44FB7EE7-295F-46B1-9128-945DCAC6E79E.ref@yahoo.com> <44FB7EE7-295F-46B1-9128-945DCAC6E79E@yahoo.com> <79DA50DD-1B3B-490A-B639-37612203D23E@yahoo.com> Content-Language: en-US From: George Mitchell In-Reply-To: <79DA50DD-1B3B-490A-B639-37612203D23E@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-3.17 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_LONG(-0.97)[-0.973]; NEURAL_HAM_SHORT(-0.90)[-0.904]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; TAGGED_FROM(0.00)[freebsd]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[m5p.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4PhbHy1hpCz3r75 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/22/23 13:09, Mark Millard wrote: > On Mar 22, 2023, at 09:48, Steve Kargl wrote: > [...] > That still leaves me trying to figure out an accurate > way to reproduce all the steps to have a comparable > way to see for myself. > [...] Here are the very complicated instructions for reproducing the problem: 1. Install and start misc/dnetc from ports. 2. Run "make buildworld". Standard out conveniently reports how long it took (wall clock). -- George From nobody Wed Mar 22 17:37:45 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhbKG0qqqz415Rt for ; Wed, 22 Mar 2023 17:37:54 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhbKF2Ygtz3sJ3 for ; Wed, 22 Mar 2023 17:37:53 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32MHbkon002230 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 22 Mar 2023 13:37:52 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: <27f46bc2-54f8-f5aa-79ca-184e86d185d8@m5p.com> Date: Wed, 22 Mar 2023 13:37:45 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-US To: freebsd-hackers@freebsd.org References: <202303221710.32MHAhe9047582@gndrsh.dnsmgr.net> From: George Mitchell In-Reply-To: <202303221710.32MHAhe9047582@gndrsh.dnsmgr.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-3.17 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_LONG(-0.97)[-0.973]; NEURAL_HAM_SHORT(-0.90)[-0.901]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; TAGGED_FROM(0.00)[freebsd]; DMARC_NA(0.00)[m5p.com]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4PhbKF2Ygtz3sJ3 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/22/23 13:10, Rodney W. Grimes wrote: > [...] > I dont even try ULE any more. I just used 4BSD, as did bde@freebsd.org, > ULE seems to suck when your have interactive use and compute bound on > the same box. I have seen interactive in the past take seconds to > echo a command. IIRC ULE and zfs in a memory contrained environment > dont play nicely togeather either. > > +1 on the return to 4BSD as the default scheduler > Thanks for the support and confirming I am not crazy! -- George From nobody Wed Mar 22 17:42:32 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhbRP6lxkz415qm for ; Wed, 22 Mar 2023 17:43:13 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhbRN4jFVz3v2k for ; Wed, 22 Mar 2023 17:43:12 +0000 (UTC) (envelope-from karl@denninger.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of karl@denninger.net designates 104.236.120.189 as permitted sender) smtp.mailfrom=karl@denninger.net; dmarc=pass (policy=none) header.from=denninger.net Received: from denninger.net (097-081-026-048.res.spectrum.com [97.81.26.48]) by colo1.denninger.net (Postfix) with ESMTP id 92CD6211082 for ; Wed, 22 Mar 2023 13:42:36 -0400 (EDT) Received: from [192.168.10.25] (D15.Denninger.Net [192.168.10.25]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 103E5378F3D for ; Wed, 22 Mar 2023 13:42:36 -0400 (EDT) Message-ID: Date: Wed, 22 Mar 2023 13:42:32 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-US To: freebsd-hackers@freebsd.org References: <202303221710.32MHAhe9047582@gndrsh.dnsmgr.net> <27f46bc2-54f8-f5aa-79ca-184e86d185d8@m5p.com> From: Karl Denninger In-Reply-To: <27f46bc2-54f8-f5aa-79ca-184e86d185d8@m5p.com> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms010102080704090801020004" X-Spamd-Result: default: False [-5.66 / 15.00]; SIGNED_SMIME(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.76)[-0.761]; DMARC_POLICY_ALLOW(-0.50)[denninger.net,none]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:~]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; FREEFALL_USER(0.00)[karl]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4PhbRN4jFVz3v2k X-Spamd-Bar: ----- X-ThisMailContainsUnwantedMimeParts: N This is a cryptographically signed message in MIME format. --------------ms010102080704090801020004 Content-Type: multipart/alternative; boundary="------------XBIDUhzG3d5td0cD89AjjoqM" --------------XBIDUhzG3d5td0cD89AjjoqM Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 3/22/2023 13:37, George Mitchell wrote: > On 3/22/23 13:10, Rodney W. Grimes wrote: >> [...] >> I dont even try ULE any more.  I just used 4BSD, as did bde@freebsd.org, >> ULE seems to suck when your have interactive use and compute bound on >> the same  box.  I have seen interactive in the past take seconds to >> echo a command.  IIRC ULE and zfs in a memory contrained environment >> dont play nicely togeather either. >> >> +1 on the return to 4BSD as the default scheduler >> > Thanks for the support and confirming I am not crazy!          -- George > Am I correct that toggling between these without a kernel build (or even at runtime) is not possible?  Been a minute since I've cared.... :-) -- Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------XBIDUhzG3d5td0cD89AjjoqM Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 3/22/2023 13:37, George Mitchell wrote:
On 3/22/23 13:10, Rodney W. Grimes wrote:
[...]
I dont even try ULE any more.  I just used 4BSD, as did bde@freebsd.org,
ULE seems to suck when your have interactive use and compute bound on
the same  box.  I have seen interactive in the past take seconds to
echo a command.  IIRC ULE and zfs in a memory contrained environment
dont play nicely togeather either.

+1 on the return to 4BSD as the default scheduler

Thanks for the support and confirming I am not crazy!          -- George

Am I correct that toggling between these without a kernel build (or even at runtime) is not possible?  Been a minute since I've cared.... :-)

--
Karl Denninger
karl@denninger.net
The Market Ticker
[S/MIME encrypted email preferred]
--------------XBIDUhzG3d5td0cD89AjjoqM-- --------------ms010102080704090801020004 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DbowggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBxIwggT6oAMCAQICEgLG8yH4PQFdbd9x Ugmpzl1jXzANBgkqhkiG9w0BAQsFADB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlk YTEZMBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENB MSUwIwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBMB4XDTIyMDYyOTE2MTYz NloXDTI3MDYyODE2MTYzNlowOjELMAkGA1UEBhMCVVMxEjAQBgNVBAgMCVRlbm5lc3NlZTEX MBUGA1UEAwwOS2FybCBEZW5uaW5nZXIwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoIC AQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvW ZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B 3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTgy+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYI XgVVPgfZZrbJJb5HWOQpvvhILpPCD3xsYJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMi WapsatKm8mxuOOGOEBhAoTVTwUHlMNTg6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMb NQm1mWREQhw3axgGLSntjjnznJr5vsvXSYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZM qa20JLAF1YagutDiMRURU23iWS7bA9tMcXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5 CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1l y+5ZOZbxBAZZMod4y4b4FiRUhRI97r9lCxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY 2BlA7ExM8XShMd9bRPZrNTokPQPUCWCgCdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEE MDAuMCwGCCsGAQUFBzABhiBodHRwOi8vb2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNV HRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIFoDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYI KwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCGSAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBD bGllbnQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNV HSMEgcIwgb+AFF3AXsKnjdPND5+bxVECGKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQ MA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5 c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lz dGVtcyBMTEMgMjAxNyBDQYITAORIioIQzl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJs QGRlbm5pbmdlci5uZXQwDQYJKoZIhvcNAQELBQADggIBAKquc7cu0xc8FNtAQauZvocDzWQj 7HG9YvMbWnMi+ckhiA3rdW5NwWg0HBhBho1YlnqV+ntCVE2L8ezohHWm+KAdfXgpraL86Vsn 3ywNlZu/3COMpo2ALuHln8YQtH3Y8ebvzKMdlf2b5WB+7mOFIxXIr4AnNOLKCkq5ZhzC6JW6 Jvw3P0csiGa3UrfatYID5NvPgkaQvEgimEjG3psZqwQTL2Wxohvw783PrDt3wS0XeNhvQ61g 3QJFZKuv+bmGH3YBSPo1t6NUGAr+JozX5lDihB8JGkBt/NwdYec49a08uL0BbPaAJ7NjuIPG 7Y0Ak7PXZT37yx/Zla9PzLMJFgbelOkaatdzbblMZPDEVZ27l4lGMmV83Lm3YP17sdAyS/Wp mav7WmJUkQ9iuIKzSpdc82i9Mfujl1vbBtwtkHNPPtKuulIFM4ZwrPKjlVdLqTSqD8m9yHEi Y0PuAooq63OpJWF6hvMaiIPBWEAVIaDW9uG0MshLl9DnHnMyrJTfuC33Z9mOGMz7dRBjJd5Y W02xAzYnUuEBOpj+LQv5R8XIFMHFXktqEKvQrXeM2RU+PcZqKOBkTktxBLn3NI5VfA15Jk0c 5V5XcOqo3p2hvrwvXrinrb2pEREnoqmfrkXT3zOq5Y6ryRH8u734lGEF0dILXzoV4PM7XFit oTePoEjmMYIFBDCCBQACAQEwgZEwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGEx GTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTEl MCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQQISAsbzIfg9AV1t33FSCanO XWNfMA0GCWCGSAFlAwQCAwUAoIICQzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqG SIb3DQEJBTEPFw0yMzAzMjIxNzQyMzRaME8GCSqGSIb3DQEJBDFCBEDcOcqZRLjXvlCJjMaW nhLwjBOqMNmYl/K7sptkGWjYro4eP0VPDzGvygEQ7chrVnXGHUTaqI510aQ6AkL7ZlCDMGwG CSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEqMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAO BggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgw gaIGCSsGAQQBgjcQBDGBlDCBkTB7MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTEZ MBcGA1UECgwQQ3VkYSBTeXN0ZW1zIExMQzEYMBYGA1UECwwPQ3VkYSBTeXN0ZW1zIENBMSUw IwYDVQQDDBxDdWRhIFN5c3RlbXMgTExDIDIwMTcgSW50IENBAhICxvMh+D0BXW3fcVIJqc5d Y18wgaQGCyqGSIb3DQEJEAILMYGUoIGRMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9y aWRhMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMg Q0ExJTAjBgNVBAMMHEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0ECEgLG8yH4PQFdbd9x Ugmpzl1jXzANBgkqhkiG9w0BAQEFAASCAgCy27kmuXclS1QA8/H+TpvPklwXAsDk1HASMyPy zsiCwDCd74iNb1f1a0c3ybyzKC0taAUuA7/ombAmSDubO4flBLvhAkCNoMF2lIsR5gIg/ll0 W/esxAczBaRRA8VgaeWFnsRefsKMG8gB11r03wWf3pNDTqNEiTQCI/AbaYwRlAgnXKZREYKX zRRHIoopEYqIA9lUn9MYDkIluPFcHiWDQIP7EJPKsQbOd0Hc8YCb+tG27Eks/emqdhROmnqA 1qJX9z54SMFgGoBz1SxPJ/3QtCPl3fW4CY5aXidzuDlD664o1lZESUPu04IwCnlh6XoqjMMf R6Doaq/KWiFSNIpKIMfXG02YZ0CkZ5Trzx+Xt5zNPDtvkM0cTRsV77wqdct7293cKeaRiJvi u82Iq5hAqyd8TYaXYb/9dcOVub9JvZ5VkUoxPnr8+/sMvseEI7PMc9xXsLTL4ZVVtExDGnSP OI7Gl8h7m9oy5/+T6D03M34sYbWuUyg+PdLP26eLgPJj73G2aptAoFi4lvBbTe7Usembyug7 e873cp/ZkNEYPmY+AlWgOrglHYvq0XJi3mdhHjqWLP7RvEQhBQdEAwQliswnLQhUr2Cy2Tln t5tmh7EqvOIBILSJujMMN/7717bOsqakpGfc7STEBNIZ/+/1EtVfHUNmgRg9rj8gvL/wQgAA AAAAAA== --------------ms010102080704090801020004-- From nobody Wed Mar 22 18:03:37 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phbv924Nmz416ZQ for ; Wed, 22 Mar 2023 18:03:49 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phbv85SCkz3wtF for ; Wed, 22 Mar 2023 18:03:48 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32MI3clA002393 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 22 Mar 2023 14:03:43 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: Date: Wed, 22 Mar 2023 14:03:37 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-US To: freebsd-hackers@freebsd.org References: <202303221710.32MHAhe9047582@gndrsh.dnsmgr.net> <27f46bc2-54f8-f5aa-79ca-184e86d185d8@m5p.com> From: George Mitchell In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-3.06 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-0.99)[-0.988]; NEURAL_HAM_SHORT(-0.77)[-0.773]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; TAGGED_FROM(0.00)[freebsd]; DMARC_NA(0.00)[m5p.com]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4Phbv85SCkz3wtF X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/22/23 13:42, Karl Denninger wrote: > [...] > Am I correct that toggling between these without a kernel build (or even > at runtime) is not possible?  Been a minute since I've cared.... :-) > [...] Rebuilding the kernel is the only way I know. In an ideal world, the scheduler would be a loadable kernel module (if that's even possible), if we can't pull the switch back to 4BSD as the default. -- George From nobody Wed Mar 22 18:31:57 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhcWh2NhSz418cj for ; Wed, 22 Mar 2023 18:32:00 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhcWh1vD5z40wp for ; Wed, 22 Mar 2023 18:32:00 +0000 (UTC) (envelope-from mandree@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679509920; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lbCCU79BU15m0Y+BGVwjs7gMbn9LWtFQHHhWadk1HNk=; b=T249iH9DMqhWNHcxkKIHyQoQmSDrA4KCtvy5n1cvJDZpgWGHX/IJknsgoQoQYU31gBRBvi +wzkoUKIATG2U2+nLufgGSdt/PVX84Nz/pRp3s2K9tz3H1vzsFM3ot54uM7Q0M7qxJ36ha tSYiR6av4Y57ulrBq26qHHWg1sbG/naj3EG6Hb/vrXIs6o2QCFaQ/GgJbALNRUw7tO9T97 S4YJFRYhyns9SVg3oNl7fq2BlZ8Rc6yFl9/w3l55wQePy3re0Ds9hIC5iqeadrCXF8CVT6 w02nTU62iL9Q7j9TrUBBcAWJEFp/04asJ0mFPLVl9HzoXjW1vnypiSRZqvzAMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679509920; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lbCCU79BU15m0Y+BGVwjs7gMbn9LWtFQHHhWadk1HNk=; b=ApEgUWBnwh1DR6U7XUbg9P1YqyJcbWmknj1fj2g+cxZP7RvDZ206KR+oP3sT+O+v2VEbhv gpKG2VAuw7YnKdQeVFj0PwsP8wsxfqgLpq1M3S2C+MRO9p2YZGYKbmQEBE44PQpI/ruXll SKdGtco5z2QqX0iV40to9CnwBFQ6gI5YkmbybZHXHXaBkDYNPb0kbdTeidcZcnWgg9vXRM VU4AmfSrWi5XB+QCEZEMe2iYEc89vZ2cYWwv0Ev7AEU5SmBNHmNTg1Ldi+D9JgF7YccQAa 7mij0jAMviudidgOu/x/RyTG+IoImgVuWMzY9bpiwbj48OT0NzUoOeaaYK5IUg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679509920; a=rsa-sha256; cv=none; b=DwadY+XKJ6VGh/BeAT4BSjWFP0A/lqfYXXTtbWJ5bYhKGvA+fH+7oooY5KWTe60i842bev o1hE5ICeI+2DnQ3pyKkxYTIUR/+Pvp5Um9TceIOjW333Jcbcsjsbqx387sKsrSQ7UZiJjV gRs619u/854fAJeTJlDwA5UGRYmoAlDRbiaS+vnnelP12rq25YkkfaGX3IDpEPihFied9D kXnSWTKO+qwQ1ZC0RiaCnr5dN5nCBQ8Y3Qu2ma9PqJBHiP4f5aH3PWX0X1twx5TppDF5ww OdwxF4gr4LuEXOy6J6/sdVlP/2i6v+/4/EUD+M3yOv7gdgbnmjMm6Ol9XxF4TQ== Received: from mandree.no-ip.org (p200300d02712be00de65dff1a874e1cc.dip0.t-ipconnect.de [IPv6:2003:d0:2712:be00:de65:dff1:a874:e1cc]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mandree/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PhcWh0DMWz1Bpg for ; Wed, 22 Mar 2023 18:32:00 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from [127.0.0.1] (localhost [127.0.0.1]) by ryzen.an3e.de (Postfix) with ESMTP id 8C204DB08A0 for ; Wed, 22 Mar 2023 19:31:57 +0100 (CET) Message-ID: Date: Wed, 22 Mar 2023 19:31:57 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Content-Language: en-US To: freebsd-hackers@freebsd.org References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Matthias Andree Subject: Re: Periodic rant about SCHED_ULE Organization: FreeBSD.org In-Reply-To: <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N Am 22.03.23 um 15:41 schrieb George Mitchell: > On 3/22/23 06:17, Matthias Andree wrote: >> Am 21.03.23 um 23:52 schrieb George Mitchell: >>> Yes, you've all heard it before [... blah blah blah ...] >> >> Can you please also give figures how much CPU time has been allotted >> to dnetc in that respective situations? >> > I let the scheduler do the time allocation.  The result is that dnetc > gobbles whatever time remains available when higher priority processes > (i.e. every other process on the system) have nothing to do.  With > SCHED_4BSD the resulting idle time is 0 (as reported by top).  I did > not take note of the idle time when I was doing the SCHED_ULE run. You are not answering my question. I asked how much time did dnetc use in the time where you were doing your test compile. The next question is then where you get the idea that a scheduler must interpret 20 as "only if idle". Linux has scheduling classes with "idle" and "best-effort" priority classes, for one. Yes, there are reports that FreeBSD is not responsive by default - but this may make it get overall better throughput at the expense of responsiveness, because it might be doing fewer context switches. So just complaining about a longer buildworld without seeing how much dnetc did in the same wallclock time period is useless. Periodic rant's don't fix this lack of information. -- Matthias Andree FreeBSD ports committer From nobody Wed Mar 22 18:57:15 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phd500xPjz419tS for ; Wed, 22 Mar 2023 18:57:24 +0000 (UTC) (envelope-from alex@protasenko.com) Received: from mail.bkmks.com (mail.bkmks.com [45.79.157.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phd4y4dLMz43Vn for ; Wed, 22 Mar 2023 18:57:22 +0000 (UTC) (envelope-from alex@protasenko.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bkmks.com header.s=mail header.b=D6Oatsqx; spf=pass (mx1.freebsd.org: domain of alex@protasenko.com designates 45.79.157.50 as permitted sender) smtp.mailfrom=alex@protasenko.com; dmarc=none Received: from [192.168.2.11] (pool-173-54-194-2.nwrknj.fios.verizon.net [173.54.194.2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: aprotasenko@bkmks.com) by mail.bkmks.com (Postfix) with ESMTPSA id 4A4738DD414 for ; Wed, 22 Mar 2023 14:57:16 -0400 (EDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.bkmks.com 4A4738DD414 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=bkmks.com; s=mail; t=1679511436; bh=8JuGJrBE/tBqgCiHhn93uAuo4HH+43KAaqiT2tt+mZY=; h=Date:Subject:To:References:From:In-Reply-To:From; b=D6OatsqxAZ+ldWazXfWJyPdDpH4plqWbosvTX+XZNoIblnF3DjLQ4wXwg5S+VirTL aOqhKhYuQBnmwBcYNyD4g52/hbtVytMoLEuI7bPwpfQKOHRlfvIkMXjC7CeFpt4PQ4 +eGZYZ9tunaOoWAZQO6c3UdRuID5i5K8FsSFuHrg= Message-ID: Date: Wed, 22 Mar 2023 14:57:15 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-US To: freebsd-hackers@freebsd.org References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Alex Protasenko In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[bkmks.com:s=mail]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[bkmks.com:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:63949, ipnet:45.79.128.0/19, country:SG]; DMARC_NA(0.00)[protasenko.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[alex]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4Phd4y4dLMz43Vn X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N FWIW, to eliminate "stuttering" for interactive use (desktop), changing defaults for either one of these work wonderfully on recent AMD system (AM4 5700G): kern.sched.steal_thresh=1 or kern.eventtimer.timer=HPET My test case is autoscroll in firefox must be butter smooth, no freezes or jerks, and it works. So, maybe its worth tweaking some knobs to optimize particular workload, although how can one expect reproducible results with something like dnetc? On 3/22/23 14:31, Matthias Andree wrote: > Am 22.03.23 um 15:41 schrieb George Mitchell: >> On 3/22/23 06:17, Matthias Andree wrote: >>> Am 21.03.23 um 23:52 schrieb George Mitchell: >>>> Yes, you've all heard it before [... blah blah blah ...] >>> >>> Can you please also give figures how much CPU time has been allotted >>> to dnetc in that respective situations? >>> >> I let the scheduler do the time allocation.  The result is that dnetc >> gobbles whatever time remains available when higher priority processes >> (i.e. every other process on the system) have nothing to do. With >> SCHED_4BSD the resulting idle time is 0 (as reported by top).  I did >> not take note of the idle time when I was doing the SCHED_ULE run. > > You are not answering my question.  I asked how much time did dnetc > use in the time where you were doing your test compile. > > The next question is then where you get the idea that a scheduler must > interpret 20 as "only if idle". Linux has scheduling classes with > "idle" and "best-effort" priority classes, for one. > > Yes, there are reports that FreeBSD is not responsive by default - but > this may make it get overall better throughput at the expense of > responsiveness, because it might be doing fewer context switches.  So > just complaining about a longer buildworld without seeing how much > dnetc did in the same wallclock time period is useless.  Periodic > rant's don't fix this lack of information. > > From nobody Wed Mar 22 19:04:06 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhdDn1tz9z419q8 for ; Wed, 22 Mar 2023 19:04:09 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhdDm6X2Vz45PF; Wed, 22 Mar 2023 19:04:08 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Authentication-Results: mx1.freebsd.org; none Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.17.1/8.17.1) with ESMTPS id 32MJ47uq001819 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 22 Mar 2023 12:04:07 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.17.1/8.17.1/Submit) id 32MJ47KZ001818; Wed, 22 Mar 2023 12:04:07 -0700 (PDT) (envelope-from sgk) Date: Wed, 22 Mar 2023 12:04:06 -0700 From: Steve Kargl To: Matthias Andree Cc: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: Reply-To: sgk@troutmask.apl.washington.edu References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4PhdDm6X2Vz45PF X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: > > Yes, there are reports that FreeBSD is not responsive by default - but this > may make it get overall better throughput at the expense of responsiveness, > because it might be doing fewer context switches. So just complaining about > a longer buildworld without seeing how much dnetc did in the same wallclock > time period is useless. Periodic rant's don't fix this lack of information. > I reported the issue with ULE some 15 to 20 years ago. I gave up reporting the issue. The individuals with the requisite skills to hack on ULE did not; and yes, I lack those skills. The path of least resistance is to use 4BSD. % cat a.f90 ! ! Silly numerically intensive computation. ! program foo implicit none integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) integer i real(dp) x real(dp), allocatable :: a(:,:), b(:,:), c(:,:) call random_init(.true., .true.) allocate(a(n,n), b(n,n)) do i = 1, m call random_number(a) call random_number(b) c = matmul(a,b) x = sum(c) if (x < 0) stop 'Whoops' end do end program foo % gfortran11 -o z -O3 -march=native a.f90 % time ./z 42.16 real 42.04 user 0.09 sys % cat foo #! /bin/csh # # Launch NCPU+1 images with a 1 second delay # foreach i (1 2 3 4 5 6 7 8 9) ./z & sleep 1 end % ./foo In another xterm, you can watch the 9 images. % top st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 11:43:01 74 processes: 10 running, 64 sleeping CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G Free Swap: 16G Total, 16G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's exit after 55-ish seconds. If you try this experiment on ULE, you'll get NCPU-1 ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as the two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, this was/is due to ULE's cpu affinity where processes never migrate to another cpu. Admittedly, this was several years ago. Maybe ULE has gotten better, but George's rant seems to suggest otherwise. -- Steve From nobody Wed Mar 22 19:21:31 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phdd65Y82z41Btc for ; Wed, 22 Mar 2023 19:21:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-20.consmr.mail.gq1.yahoo.com (sonic305-20.consmr.mail.gq1.yahoo.com [98.137.64.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phdd55DHLz47Xs for ; Wed, 22 Mar 2023 19:21:45 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=k5sA5gYX; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679512903; bh=VzWRHCwOPowD1va23lmbG/Tk8YOTfQ2tiLY2YdzC8/c=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=k5sA5gYXdueXSjB6NEYzACbfjMq3uGU7bi3Uvka7rLEhUn1HJNOueFdXZlp3BfrfedOu6WYh31jWsLkPw3VFGXf9h67NnMjJCPa7vhxeRx5HBm+PQPfbFeNYJt/jXAv4X7tY6Y++y+xu1GScYbcCaGGXk5q3mYprU16cdKjmjAZ9aVFOHNb+WjxsWj8qGvTjsnC/TINAu9riEDufb1JoB4g10qKmpx+TOob1YLNID7Y/txlAhsDSgBr7rFAE3dvjpeWFnbZRg5OB1OpkzMdqIb+b9vXLrsGgswyFEyqMDHQlK3qOXffqLZl/wZ56b4OJYMEjAnShdhEfnMg24BsvVA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679512903; bh=DDCLoedBvUyYocRVkgXfLA7lv+Ye8CvD2mGh5M/YNIG=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=NSZMefN4MglxnKOIRzdlhglwJXhLoXyR79lMeiMIUN+fPUSg3+1HULPnwLYob+h+Vz1scBxL/INQYRFZW8w5NaV3lFmKSlFYeDqGLOZpWB+34XAA3oWHk8ZsQAq+weA/T1zvWeLdDqBtsxsLhEK1LiMg1ioJk+7/9V/q+U8f2v+ToU2Ewibzl7kn4Ynbss6BrGxfYh2wNa3iCN9RYa/1SOS9FXYyingh92K4xGE7p1xrcg6YygwQhgpnPePLnkW54BjLyFxwxT06HAsFvYQhWHNyY3pWjf913cs1i7v3RfUDEi5CWwMVxXMX8rByMVIpmMlbV2bdx9fh9YFRspP4Uw== X-YMail-OSG: 8d5oR3AVM1lT0YdlGXaqvxCT1rfJDZvK1YmHyyaQAnUdNhFiZOGQAJM3jWJfV.q KeSxh8GGj8.cTb7vUPKLwDFeIpecnyrrk7wa98Uafa4x9xDIklM6TteL7TWXKWDbm_1UnoCj6QXB LvY8NpI_tDnDEJmudCE7RGT.AXBhtmKuBQk..zBUimpb_c43jxiimxikJXjJgoLw759Vl1OH7iSm gOOy4e7ZooeGmU3IqSmUgLEgraQTcOf9U0w1NsUVZxOsRBJw5z29biV1ePhPMG9pd4uF5dH8h7.r e8GqrVw9RtknCcXo74QM3ctfTBD_WtOcGbhoJnBkOy6v3yBr1sD0Mgn4JBqckwTjLtO0lKJiAo4G NgSZ0mfc07LBefZne100a936BT.wAZKm7DbbzBF1jlHTCK9DbUybajEqg._H78NGUGJk5Sj2pzSC 7wcjgzyU57EeRwNcgPoBB.ryfsgDlYUGoU5Rl6bkzNP8.lJ4_hbH9MZz4KQiQnRAkVyxsNZlmg1q nkHylK.FGvE6leiZkZVuNDhyHAQa_WcPRFCMRc54EFTHMMUN5d4xH6.B2LxebrZ8IREJIh_56qLC 3M6p5J8QwG5RmWP00B3pvR5DdEf9hHn_cXmHPevY1wNCdI0sVbje02SJ3lF7_J.LABOlSbfSiZtp okA3i0Dqm5YkC_io2NEwTF5zrw7NzDnKZg_HbgWZ0uA.VbTsdwOoQxtYbyhC9PUqEEXEC7Eqd.kd 4zV8HqGkqkHofVNCexWlGV4OvbuSrykhf8LKunZCSEajB8Y0PT1m5qv9_l.g3kiuEXzpw5q9Tb5x DMgIq9usdvsoZNfpxchgI_Jo2q4_imJV0m4Ew0sf1GBFTYk3h.tcCYiVSD9zOjvlho.EyhBgQHQz 9FP6L0aIUTONqFYflfbNRxDTkeIkhrIu7R0p5YUwdW5lFaaek6jKwjSaWQagbWnbkU_dqePHeqMB jWmkl4__YLM_gRd1nm3rceRCZow7Cr4yQ06JopkP.XSCfvVe1OSg1O_SK5zE9of4zMixEdZQr4eh 9htT7SSvwakqbParnR41g72_t3FnlMp3Al_MqEzotUcteyFxYXtMR5WO34a73kL0bgwZuf1B9boL 8VcKU.DHIWGoIgimrNE4KZ7q6OuzTr_4Fjmk267VPhFP1BWhaBXnBQbrwzs.s0grRn3qiFWchNp7 ONhQ3huWanqQ.9VEtIl2rLmzx3iiM9Fk0q0UZlsTIU.EEiPuRdmSIHBAfv_651Rey_OirB1Hnn.5 tdDYaoN2qW42ZV5m8x3_Ay4NcOpzQRoh6Hemi8cLt9P7tupwxS2kTqPQXLxluQM6mwWdcAZTm4ja SjYAV6ifTtquEkIAotMTHLq3RM7sjhSukXnblBV_M_4tBh7Hy6mU9ILU_yHYbh0Z2uq7Z35T9p92 DXA6KMNVWOSCRhYyCsMgr6Qz.PspEb_UIoXmMEffQtaT.c56KtMCJP_j5.W0wINI_MSoXVFuVK24 ghKHxAHVXt_4SkbTo126C88njy5QuR9DmZUp0pNTINn6jmwfwaMIDhxSckb0TDgcrrFdXJhw9HGn G4pi2UT_C_VO.Z1HU0J0.fgDIJHyw29RqVpjNeu8Vad0FooRgvXq4y2bFwuvhfp6VMns7X.2qual cxvTSWyBl4DXWegMXMGjMpxxfnrZNfwsRsk4AvNNB.lPT3kZk_YIjliWuKulDq05lMpSQarkASgw 9f0ffHLnfbiQzDEFd0jY7p4fl6x1eFtpbhEgV7xs.3O_qMbhObZ8v0f2xJMEpqxyq7DI6jjTUFAw JT8KNZ5d7zID_63uoMLxYIDvl9No_pceEqV90CaRIJZ5NZQObnwDc4hqGtV1WVdZ6u8f.qJIjxdK TO4EjmpS5sUXNn8UZNufzywNQGw08IC29hfJe9UtfUV3g5K4.k1xRWonfksY0kkzecrGm7siaNMX FIu8qqC0tWRIthXeg27qwFgK81xv5V0l_QEvqjFMN2Yh.pd6vP2U80lfC.1hEUmRK9qV6Dsc4YH2 aRuswc_ss6L_hC0XuXunNdZ8htiQo3gnDKP0C9TuNumPdZVei2mwT78Pdy3wjxkuRLmFzlvTvjVH tvetkxUYQt6BDxsCJ4oZatyXHKW_w6_CPdQxnUWmks9b66klNUf0.rMaunJNsyaCqV5C_08qEnFU 6Wn914r6zmqU0JqAx7Y2X9djnNYSONu9Ax_m9Fl4P2d6zPOq4n1PSqQuTANK.bS6wa03RiCV7y65 OVPJb40zZ9qkYitK9picuZkxBUGpSjBw8Y38l1mjezdp9nBZwWtYRJuCzjwJpP2R5pDheH4cn9qX 8APE- X-Sonic-MF: X-Sonic-ID: 4134a486-e0f1-4a81-82fe-4045eac68235 Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Wed, 22 Mar 2023 19:21:43 +0000 Received: by hermes--production-gq1-6cf7749bc8-sm5v9 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 81550f138494fc82b9450062f292edd1; Wed, 22 Mar 2023 19:21:42 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE Message-Id: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> Date: Wed, 22 Mar 2023 12:21:31 -0700 To: george+freebsd@m5p.com, FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.83:from]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Phdd55DHLz47Xs X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N George Mitchell wrote on Date: Wed, 22 Mar 2023 17:36:39 UTC : > On 3/22/23 13:09, Mark Millard wrote: > > On Mar 22, 2023, at 09:48, Steve Kargl = wrote: > > [...] > > That still leaves me trying to figure out an accurate > > way to reproduce all the steps to have a comparable > > way to see for myself. > > [...] >=20 > Here are the very complicated instructions for reproducing the = problem: > 1. Install and start misc/dnetc from ports. Installing is likely easy, as likely would be building with default options (if any). I know nothing about starting misc/dnetc so that is research. (Possibly trivial, although if it has alternatives to control then I'd need to match that context too.) > 2. Run "make buildworld". So on the 32 hardware-thread (16 cores) amd64 machine that I have access to, the test is to only have buildworld use about one hardware thread, no matter what else is going on. I never would have guessed that the steps would not involve more like -j$(sysctl -n hw.ncpu) (so around -j32 in this context). So it is good that you provided your note or I'd not know if I'd done similarly or not when trying such. [Note: -j1 and lack of -j are not strictly equivalent in how make operates. As I remember, the distinction makes a notable difference in the number of subprocesses created directly by make (one per action "line" vs. one for the whole block?). So even using -j1 might make a difference vs. what you specified. I'd have to test to see.] > Standard out conveniently reports how long it took (wall clock). But nothing in your instructions indicate about how to get an idea much progress dnetc made during the various tests? I do not want to ignore dnetc progress as well (or I'd just not run such at all in a realistic situation of wanting to have the machine doing the 2 things in overlapping time frames). I'd be looking at the relationship between the progress for each of the 2 activities. FYI: I've never built with and run the alternate scheduler so if there is any appropriate background for that that would not be obvious on finding basic instructions, it would be appropriate to provide such notes. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Mar 22 19:23:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhdgN11jcz41BpN for ; Wed, 22 Mar 2023 19:23:44 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x2a.google.com (mail-oa1-x2a.google.com [IPv6:2001:4860:4864:20::2a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhdgM6Lb7z49Br; Wed, 22 Mar 2023 19:23:43 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oa1-x2a.google.com with SMTP id 586e51a60fabf-17aeb49429eso20151476fac.6; Wed, 22 Mar 2023 12:23:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679513023; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=GcwW6GunFt6vJyXRox5gqeYCqjP7ND39NoyPZHapQQA=; b=VHkhIV3Eawf3eQ5MBW00ntd6blw3yjR1MyZ0vTU+/mp0Yae4XmVlUiBW7/NhB4ORhD PgLvYcne/W+O1wMJKY0YYmMQ7w8tru1ppdcblFs5N4jZYJKhV4IWLa+cu8iBRChjmTwo Y2gitwyYb0W3PkggrNpKCVlaKwT/reRKZ0DfQyO7czID/IoMv7H7uc+TkQuK5fFiJ8oi K8Ou/34yFbR04CkyMXvkklS4x7GEgDxjQ4pCu/gp1F1NQ4r06msdncFmOWB48QCyYfGl 6BtPEIxbz2QPS5776ECcbYzUN6QUteNHynJf6ZngvkV2wi6hvDv5OixEDqcuK17UHMDv 37pg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679513023; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GcwW6GunFt6vJyXRox5gqeYCqjP7ND39NoyPZHapQQA=; b=cTjVfw/FMMCzHM3sSSwuqIrNRkQM75pUUhEMCBVSPq5/rfvdTjaiXIWi9KygSmUkY7 DuDCzF2qHFZWnxCkYNsq7SiSEuK3dGvKTyMZ6Tn6CmY8DWtPmwL8Ku9Cy17Otlj1Scfo /YErmoMWpkVku1zRIvxI/V1/M7CKoRq3po4uNr36sbkPy6/9wkIVuWG8For1DNBJP71T labEWW2iUs5LYJP3fIbOUuv1Fm2fSUQIcN+I4eHf/7GW6HpWrug/7aio7sG96Xy1V7G7 WWqzPayNOW+0AdrukLacXJoPnDO0pjKNs4Q0StvMzhJzqll4arHn8jEJsKyBAbR58ooU jbRA== X-Gm-Message-State: AAQBX9fUOJcNXDRnlW+/7DPs6xGK+Mrv0YZXCYbxA/v0nZfFGm9DxRe6 3sNc5xmMcOsAxi2qN2++vZSET6Fl7WEnRIVqzsGMDowK X-Google-Smtp-Source: AKy350bIVs5KP/r9XnotBS6z/8umamC4t9a+3cN9xVAin5qs8zV8mzdH2VcbX1pIknkEBe/hcLufzkj7YF2qu+QdrL4= X-Received: by 2002:a05:6870:1297:b0:17a:a52d:9df7 with SMTP id 23-20020a056870129700b0017aa52d9df7mr346958oal.4.1679513022832; Wed, 22 Mar 2023 12:23:42 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1922:0:b0:49c:b071:b1e3 with HTTP; Wed, 22 Mar 2023 12:23:42 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Wed, 22 Mar 2023 20:23:42 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: sgk@troutmask.apl.washington.edu Cc: Matthias Andree , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4PhdgM6Lb7z49Br X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 3/22/23, Steve Kargl wrote: > On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >> >> Yes, there are reports that FreeBSD is not responsive by default - but >> this >> may make it get overall better throughput at the expense of >> responsiveness, >> because it might be doing fewer context switches. So just complaining >> about >> a longer buildworld without seeing how much dnetc did in the same >> wallclock >> time period is useless. Periodic rant's don't fix this lack of >> information. >> > > I reported the issue with ULE some 15 to 20 years ago. > I gave up reporting the issue. The individuals with the > requisite skills to hack on ULE did not; and yes, I lack > those skills. The path of least resistance is to use > 4BSD. > > % cat a.f90 > ! > ! Silly numerically intensive computation. > ! > program foo > implicit none > integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) > integer i > real(dp) x > real(dp), allocatable :: a(:,:), b(:,:), c(:,:) > call random_init(.true., .true.) > allocate(a(n,n), b(n,n)) > do i = 1, m > call random_number(a) > call random_number(b) > c = matmul(a,b) > x = sum(c) > if (x < 0) stop 'Whoops' > end do > end program foo > % gfortran11 -o z -O3 -march=native a.f90 > % time ./z > 42.16 real 42.04 user 0.09 sys > % cat foo > #! /bin/csh > # > # Launch NCPU+1 images with a 1 second delay > # > foreach i (1 2 3 4 5 6 7 8 9) > ./z & > sleep 1 > end > % ./foo > > In another xterm, you can watch the 9 images. > > % top > st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 > 11:43:01 > 74 processes: 10 running, 64 sleeping > CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle > Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G Free > Swap: 16G Total, 16G Free > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU > COMMAND > 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z > 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z > 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z > 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z > 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z > 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z > 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z > 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z > 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z > > With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's exit > after 55-ish seconds. If you try this experiment on ULE, you'll get NCPU-1 > ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as the > two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, > this was/is due to ULE's cpu affinity where processes never migrate to > another cpu. Admittedly, this was several years ago. Maybe ULE has > gotten better, but George's rant seems to suggest otherwise. > While I have not tried openmpi yet, I can confirm there is a problem here, but the situtation is not as clear cut as one might think. sched_4bsd *round robins* all workers across all CPUs, which comes at a performance *hit* compared to ule if number of workers is <= CPU count -- here ule maintains affinity, avoiding cache busting. If you slap in $cpu_count + 1 workers, 4bsd keeps the round robin equally penalizing everyone, while ule mostly penalizes a select victim. By the end of it, you get lower total cpu time spent, but higher total real time. See below for an example. Two massive problems with 4bsd, apart from mandatory round robin which also happens to help by accident: 1. it has one *global* lock, meaning the scheduler itself just does not scale and this is visible at modest contemporary scales 2. it does not understand topology -- no accounting done for ht nor numa, but i concede the latter wont be a factor for most people That said, ULE definitely has performance bugs which need to be fixed. At least for the case below, 4BSD just "lucked" into sucking less simply because it is so primitive. I wrote a cpu burning program (memset 1 MB in a loop, with enough iterations to take ~20 seconds on its own). I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 distinct cores. The test runs *9* workers, here is a sample run: 4bsd: 23.18 real 20.81 user 0.00 sys 23.26 real 20.81 user 0.00 sys 23.30 real 20.81 user 0.00 sys 23.34 real 20.82 user 0.00 sys 23.41 real 20.81 user 0.00 sys 23.41 real 20.80 user 0.00 sys 23.42 real 20.80 user 0.00 sys 23.53 real 20.81 user 0.00 sys 23.60 real 20.80 user 0.00 sys 187.31s user 0.02s system 793% cpu 23.606 total ule: 20.67 real 20.04 user 0.00 sys 20.97 real 20.00 user 0.00 sys 21.45 real 20.29 user 0.00 sys 21.51 real 20.22 user 0.00 sys 22.77 real 20.04 user 0.00 sys 22.78 real 20.26 user 0.00 sys 23.42 real 20.04 user 0.00 sys 24.07 real 20.30 user 0.00 sys 24.46 real 20.16 user 0.00 sys 181.41s user 0.07s system 741% cpu 24.465 total It reliably uses 187s user time on 4BSD and 181s on ULE. At the same time it also reliably has massive time imblance between fastest/slowest in terms of total real time between workers *and* ULE reliably uses more total real time to finish the entire thing. iow this is a tradeoff, but most likely a bad one -- Mateusz Guzik From nobody Wed Mar 22 19:35:33 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phdx45lrlz41CP6 for ; Wed, 22 Mar 2023 19:35:36 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x35.google.com (mail-oa1-x35.google.com [IPv6:2001:4860:4864:20::35]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phdx33Dt9z4D9s; Wed, 22 Mar 2023 19:35:35 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=kREsM9pW; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2001:4860:4864:20::35 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-x35.google.com with SMTP id 586e51a60fabf-17aaa51a911so20511243fac.5; Wed, 22 Mar 2023 12:35:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679513734; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=DNqC16//EEe720Gn7k2KbqRc3GnXMEeEFesuBSbgXOk=; b=kREsM9pWRRLSWhyxUQ2t8mHWUvXoRiWdQmXgpzCV/+Bpel8Gb619jmmmfkUtZ6vcQT KS83w3mxlLbDqOe5QcSR8WPSALg3+gT/Xd5AYlQcbEbSjq2LKAAaGQ/9k5Dqz1Sh5wOA 7MMBbuBve+eb1Eh547t523kb+/N93vexUEgAamVfGPGvIRCmKTMB+xZADafcP/sZcgCm Rxq6oKqS7XggNLmEn2Cd/DLzQjQUWX/AlshBvfeP6mGw5ezWozAKDnjEWhU462lQCW3x bXyPt98tB6DYV9EfMFuUMdOUGy9ijJrp5xj+sDtDswoQTqCRMc7TiZ6kn9Rd03sgdewN rmyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679513734; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=DNqC16//EEe720Gn7k2KbqRc3GnXMEeEFesuBSbgXOk=; b=JSoV3v/0rmpjtI+Z0EVm7j/RZd/C5xkeS07cZ2KGnqJqdKY0V4BvXeD18sMfEYkNzh tYIAVVpvtlA7gNQgVfB7l7lWFb5OUr6RCi/JfXIPy5rY9X1PAqf4Gka6mZejPbh/9g46 uHeZIt85D6TQBOyQOZVnUI/gOnimzHt3sQJD0bNQN4usvMW4KkzlEbpE/BEANr15MF1/ QVecsXIFC6e1X6TgSUyGiYWekliZySUt2vY+P8qId8gF9fFXEU3jiRDN6RUy2nWDY8eh 7TeQkzwb8X8jU2qeqaxEeNcGBxeGYjvMx8FYCdGNQOwP+fF8dSmbzcny11H2CiYpRQVW UWbg== X-Gm-Message-State: AO0yUKWKx4LAxgMbm6CS4GE45mRRgQCaNaYaCqh/PA8wLRcmzltq7QRR hjalVk7OVgEKUfgJ1+5fVMcBx3oFfJnmKFnCvrBj2f/Z X-Google-Smtp-Source: AKy350Y+1xkiqhmFjZCeu8d8zO22BnZmZuMlKxM/7Qnwb29Rss3mw9wwkM6NSwSYDQT6r9EOBbNYaielbK1ZWS8dL5A= X-Received: by 2002:a05:6871:d96:b0:17a:b713:63e9 with SMTP id vi22-20020a0568710d9600b0017ab71363e9mr330405oab.4.1679513734071; Wed, 22 Mar 2023 12:35:34 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1922:0:b0:49c:b071:b1e3 with HTTP; Wed, 22 Mar 2023 12:35:33 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Wed, 22 Mar 2023 20:35:33 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: sgk@troutmask.apl.washington.edu Cc: Matthias Andree , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.92 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; NEURAL_HAM_SHORT(-0.92)[-0.920]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::35:from]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Phdx33Dt9z4D9s X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/22/23, Mateusz Guzik wrote: > On 3/22/23, Steve Kargl wrote: >> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >>> >>> Yes, there are reports that FreeBSD is not responsive by default - but >>> this >>> may make it get overall better throughput at the expense of >>> responsiveness, >>> because it might be doing fewer context switches. So just complaining >>> about >>> a longer buildworld without seeing how much dnetc did in the same >>> wallclock >>> time period is useless. Periodic rant's don't fix this lack of >>> information. >>> >> >> I reported the issue with ULE some 15 to 20 years ago. >> I gave up reporting the issue. The individuals with the >> requisite skills to hack on ULE did not; and yes, I lack >> those skills. The path of least resistance is to use >> 4BSD. >> >> % cat a.f90 >> ! >> ! Silly numerically intensive computation. >> ! >> program foo >> implicit none >> integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) >> integer i >> real(dp) x >> real(dp), allocatable :: a(:,:), b(:,:), c(:,:) >> call random_init(.true., .true.) >> allocate(a(n,n), b(n,n)) >> do i = 1, m >> call random_number(a) >> call random_number(b) >> c = matmul(a,b) >> x = sum(c) >> if (x < 0) stop 'Whoops' >> end do >> end program foo >> % gfortran11 -o z -O3 -march=native a.f90 >> % time ./z >> 42.16 real 42.04 user 0.09 sys >> % cat foo >> #! /bin/csh >> # >> # Launch NCPU+1 images with a 1 second delay >> # >> foreach i (1 2 3 4 5 6 7 8 9) >> ./z & >> sleep 1 >> end >> % ./foo >> >> In another xterm, you can watch the 9 images. >> >> % top >> st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 >> 11:43:01 >> 74 processes: 10 running, 64 sleeping >> CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle >> Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G >> Free >> Swap: 16G Total, 16G Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU >> COMMAND >> 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z >> 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z >> 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z >> 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z >> 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z >> 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z >> 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z >> 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z >> 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z >> >> With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's exit >> after 55-ish seconds. If you try this experiment on ULE, you'll get >> NCPU-1 >> ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as the >> two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, >> this was/is due to ULE's cpu affinity where processes never migrate to >> another cpu. Admittedly, this was several years ago. Maybe ULE has >> gotten better, but George's rant seems to suggest otherwise. >> > > While I have not tried openmpi yet, I can confirm there is a problem > here, but the situtation is not as clear cut as one might think. > > sched_4bsd *round robins* all workers across all CPUs, which comes at > a performance *hit* compared to ule if number of workers is <= CPU > count -- here ule maintains affinity, avoiding cache busting. If you > slap in $cpu_count + 1 workers, 4bsd keeps the round robin equally > penalizing everyone, while ule mostly penalizes a select victim. By > the end of it, you get lower total cpu time spent, but higher total > real time. See below for an example. > > Two massive problems with 4bsd, apart from mandatory round robin which > also happens to help by accident: > 1. it has one *global* lock, meaning the scheduler itself just does > not scale and this is visible at modest contemporary scales > 2. it does not understand topology -- no accounting done for ht nor > numa, but i concede the latter wont be a factor for most people > > That said, ULE definitely has performance bugs which need to be fixed. > At least for the case below, 4BSD just "lucked" into sucking less > simply because it is so primitive. > > I wrote a cpu burning program (memset 1 MB in a loop, with enough > iterations to take ~20 seconds on its own). > > I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 distinct > cores. > > The test runs *9* workers, here is a sample run: > > 4bsd: > 23.18 real 20.81 user 0.00 sys > 23.26 real 20.81 user 0.00 sys > 23.30 real 20.81 user 0.00 sys > 23.34 real 20.82 user 0.00 sys > 23.41 real 20.81 user 0.00 sys > 23.41 real 20.80 user 0.00 sys > 23.42 real 20.80 user 0.00 sys > 23.53 real 20.81 user 0.00 sys > 23.60 real 20.80 user 0.00 sys > 187.31s user 0.02s system 793% cpu 23.606 total > > ule: > 20.67 real 20.04 user 0.00 sys > 20.97 real 20.00 user 0.00 sys > 21.45 real 20.29 user 0.00 sys > 21.51 real 20.22 user 0.00 sys > 22.77 real 20.04 user 0.00 sys > 22.78 real 20.26 user 0.00 sys > 23.42 real 20.04 user 0.00 sys > 24.07 real 20.30 user 0.00 sys > 24.46 real 20.16 user 0.00 sys > 181.41s user 0.07s system 741% cpu 24.465 total > > It reliably uses 187s user time on 4BSD and 181s on ULE. At the same > time it also reliably has massive time imblance between > fastest/slowest in terms of total real time between workers *and* ULE > reliably uses more total real time to finish the entire thing. > > iow this is a tradeoff, but most likely a bad one > In case I did not make it clear enough: 1. both schedulers have problems 2. ... 4bsd has problems which are infeasible to fix 3. ... ule has problems which may or may not be easy, remains to be seen, but i can't promise i will have time to dig into it -- Mateusz Guzik From nobody Wed Mar 22 19:40:53 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phf3K66bKz41Cx4 for ; Wed, 22 Mar 2023 19:41:01 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phf3K4Ff5z4GnZ for ; Wed, 22 Mar 2023 19:41:01 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32MJesGt002753 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 22 Mar 2023 15:40:59 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> Date: Wed, 22 Mar 2023 15:40:53 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-US To: Mark Millard , FreeBSD Hackers References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> From: George Mitchell In-Reply-To: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Rspamd-Queue-Id: 4Phf3K4Ff5z4GnZ X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_FROM(0.00)[freebsd]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 3/22/23 15:21, Mark Millard wrote: > George Mitchell wrote on > Date: Wed, 22 Mar 2023 17:36:39 UTC : > [...] >> Here are the very complicated instructions for reproducing the problem: >> 1. Install and start misc/dnetc from ports. > > Installing is likely easy, as likely would be building > with default options (if any). I know nothing about > starting misc/dnetc so that is research. (Possibly > trivial, although if it has alternatives to control > then I'd need to match that context too.) service dnetc start > >> 2. Run "make buildworld". > > So on the 32 hardware-thread (16 cores) amd64 machine that > I have access to, the test is to only have buildworld use > about one hardware thread, no matter what else is going on. > I never would have guessed that the steps would not involve > more like -j$(sysctl -n hw.ncpu) (so around -j32 in this > context). So it is good that you provided your note or > I'd not know if I'd done similarly or not when trying such. > > [Note: -j1 and lack of -j are not strictly equivalent in > how make operates. As I remember, the distinction makes > a notable difference in the number of subprocesses created > directly by make (one per action "line" vs. one for the > whole block?). So even using -j1 might make a difference > vs. what you specified. I'd have to test to see.] I am literally running "make buildworld" with no additional options. > >> Standard out conveniently reports how long it took (wall clock). > > But nothing in your instructions indicate about how > to get an idea much progress dnetc made during the > various tests? [...] Honestly, I've never worried about this part. But dnetc logs its progress in /usr/local/distributed.net/dnetc.txt, though not in terms that are easy to relate to real-world progress. Oddly, when I run "make buildworld," I'm primarily interested in getting the world built. Perhaps others feel differently. > [...] > FYI: I've never built with and run the alternate > scheduler so if there is any appropriate background > for that that would not be obvious on finding basic > instructions, it would be appropriate to provide > such notes. > [...] You have to build a new kernel, using a config file in which you have replaced "options SCHED_ULE" with "options SCHED_4BSD". -- George From nobody Wed Mar 22 20:15:57 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phfqj5y7Vz41FtT for ; Wed, 22 Mar 2023 20:16:01 +0000 (UTC) (envelope-from se@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phfqj5JbPz4KsR; Wed, 22 Mar 2023 20:16:01 +0000 (UTC) (envelope-from se@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679516161; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CPym4lty5Rf/eaah1bf0Quez3DZd0p/hLzv8wedekNI=; b=akXTPHo5ivLvp1pyo+lgUsr1MdSsOMRhPUovASNDitxTE6jZ9L4HVBwlEEI73tOgNjcZhs Tu9L2gdrOzYmy0Fth3c/g2Bt9PLuk2BylsNgEFh8zp0BBNGkTfomfyvOCyialrka4R5e/b lhD8K8Dc1M7pU/UzqEjpt9xmfxhybitAW19+5a6Lr27na8GElKOL+cHXEZJJL1Wlj00jnE SB4O7VGNNC1zUSLfGEJDcdv+JAag2FVeAJxaBDdL8AyowRn7Y4otdhzVvH7n2PsRR4rGm8 2BxhiYUdh1XXYY0Ii+e/k8ssmoTD4TMfA+SJJzIvAFPMpc9H8iWjaltXxzLv7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679516161; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CPym4lty5Rf/eaah1bf0Quez3DZd0p/hLzv8wedekNI=; b=ccGLtWIJGUGwel/PbRAbatggaW3SH8kUuwYeEXCVO0u9+bNIlhDvpyObPK7VZ9bRr6pnK6 n41LMacTDHhtvapA7t0ZkqqcHZWqUBAf/rTuGh4hoe8qgO1ut3Rpgp+4q9M2/orkTgRJeS pjQRYSPc7mPXt7RQPNyk/t3vBATg4s0mMAkwDUVEcA6rDT6lcWCNPEj/yvaCsV+Cp6Ycs2 5vPF2PEXwcffBjdTx6zjPMSxHYomxNmsElKyWq67eau+2LzFnhBr3vCE8JLFPYHROSNxfa 9Jl5VXgrWmSSR7auNQUbUdQSAohdFj6kw004SOhWC2q13R0P1H73gwW7xKXtLw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679516161; a=rsa-sha256; cv=none; b=fR/DExP00hVvzCiatkSrkfcPX6IgPjQdUYgol02xghO7uy5TsAMk3u+oDqb7tErMstqc+2 XtGedmITs8xvDHgCyOyEB3zBxy+HTHv34D2bEu8KM07Mxne7iBdnRB2wbJbzWir3qE1JsB krrxuSVgi+DFv7UcLp9LIxN8kPZkV3S0eBOTftSZSmPw09z8dEi6KyHfzSan8e3WdJE06o Z7pTP0+CjcCdKaH3TCxVIHIiGx/2+/4EunovjdiwaUah0UgxA7g9deVrUvXvYSQ4YBxgXf ti7vWvH8dV7I2nbyaGGi0DrEQwjX4AGhIDxE93qW1kurc0UzQ5YBqT3ga6P3eg== Received: from [IPV6:2003:cd:5f1a:a300:a5cf:f0a3:530e:bda] (p200300cd5f1aa300a5cff0a3530e0bda.dip0.t-ipconnect.de [IPv6:2003:cd:5f1a:a300:a5cf:f0a3:530e:bda]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Phfqj0SZCzFWN; Wed, 22 Mar 2023 20:16:00 +0000 (UTC) (envelope-from se@FreeBSD.org) Message-ID: <7f26102c-7542-78f8-0c7b-ef3cdaa1a4a6@FreeBSD.org> Date: Wed, 22 Mar 2023 21:15:57 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 From: Stefan Esser Subject: Re: Periodic rant about SCHED_ULE Content-Language: de-DE, en-US To: Mateusz Guzik , sgk@troutmask.apl.washington.edu Cc: freebsd-hackers@freebsd.org References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N Am 22.03.23 um 20:23 schrieb Mateusz Guzik: > I wrote a cpu burning program (memset 1 MB in a loop, with enough > iterations to take ~20 seconds on its own). > > I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 distinct cores. > > The test runs *9* workers, here is a sample run: > > 4bsd: > 23.18 real 20.81 user 0.00 sys > 23.26 real 20.81 user 0.00 sys > 23.30 real 20.81 user 0.00 sys > 23.34 real 20.82 user 0.00 sys > 23.41 real 20.81 user 0.00 sys > 23.41 real 20.80 user 0.00 sys > 23.42 real 20.80 user 0.00 sys > 23.53 real 20.81 user 0.00 sys > 23.60 real 20.80 user 0.00 sys > 187.31s user 0.02s system 793% cpu 23.606 total > > ule: > 20.67 real 20.04 user 0.00 sys > 20.97 real 20.00 user 0.00 sys > 21.45 real 20.29 user 0.00 sys > 21.51 real 20.22 user 0.00 sys > 22.77 real 20.04 user 0.00 sys > 22.78 real 20.26 user 0.00 sys > 23.42 real 20.04 user 0.00 sys > 24.07 real 20.30 user 0.00 sys > 24.46 real 20.16 user 0.00 sys > 181.41s user 0.07s system 741% cpu 24.465 total > > It reliably uses 187s user time on 4BSD and 181s on ULE. At the same > time it also reliably has massive time imblance between > fastest/slowest in terms of total real time between workers *and* ULE > reliably uses more total real time to finish the entire thing. The problem is that user time depends on the number of CPU cycles, but real time on "when" the CPU is executing the last few cycles of the respective process. And in the case of a parallel program like the test program, where each thread takes the same number of cycles, but the thread with the highest real time determines the total real time taken (the other cores are idle for the last 0.3 to 3.8 seconds or 2 seconds on average, while real times with 4BSD are quite similar). > iow this is a tradeoff, but most likely a bad one Better balancing of the load would probably make ULE take less real time. The example of 9 identical tasks on 8 cores with 7 tasks getting 100% of a core and the other 2 sharing a core and getting 50% each could be resolved by moving a CPU bound process from the CPU with the highest load to a random CPU (probably not the one with the lowest load or limited to the same cluster or NUMA domain, since then it would stay in a subset of the available cores). Such a re-balancing could be performed at a relatively low rate (e.g. once per second) and only if all cores are running CPU bound tasks. This would probably not lead to an optimal distribution of threads to cores, but at least lead to a real time similar to 4BSD. From nobody Wed Mar 22 20:34:17 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhgF970xSz41GvT for ; Wed, 22 Mar 2023 20:34:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhgF85HQFz3C2d for ; Wed, 22 Mar 2023 20:34:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=TSsGtXpa; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679517274; bh=Mi1uZvwYTt2cmLU3UCzy5mL8Z7TMGYrk6yjVPa6idbM=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=TSsGtXpaIqzp1Hw0iq7QBv+95zCIa2D+OoimTqXt+tWDqGoj4K90UARdL7mM1VgRJrGzEEePYQjk2Z4OxUBicz6XjPM/nT+bUojkS1znnKT41ur83Bq2n1SoLrG0bMJjXGCshmJMD4kPKMZ3+h/a8WBwFYnsDg6CTKmqMORtXSMwPZlpTk5poUyg2D2SsFYQrGzyPdX/pbTMMBxb1yVCj1hB9vPzbBuKblTT0/9XFi8Lo0EpSq0nyy78Y7WLRWa4JT4dFJKcO7w52i5DYfusFE0MSB/7qZY5+GKt6IZhg8b/GyplixgCSdQ7SbKlIFZ3F339+XfgvXkR8enU4CtVTA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679517274; bh=m35eyy7dbOqLhXhIFFcLaEN7qkMvFn3qyqFdRu5cmp5=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=p27tSuv4qPgyf9Ku3gl/Z+CFR6LATr/56o7Eg3JJjADNl4s7Sxd3KbJkuBwBM3MSK2GuDT8Ln9eTbcapXANK51v2fNRGO9lUGvKOYGsFefa5mYfLdo0ClhoGaT2SR4zUQo4O9q95e0v/wAi+h87aB+5j7wZx56PIixASm+IQZSH10jTO/fNYOrAv/ceo74r0m7US+jOxv1Qy/ieat3y3ntPg0aBFAfNnWS2UQvlvR8Snncbm/V+6TLDVw5WLXK0ZnUTzwguYEG0M0fzn+IbcadGT35i6gOmBpEzGyiLENr3KXEiChApwxZ5MrlmW2DguDH0v7B9opsigckR5XRYylA== X-YMail-OSG: FOVTBhsVM1k9_HKkJgIaLk1gdYfk_6zdiU5EXBH_6x_jk4YNbXIAaZgLESXLW8a p1rCtqU296Q11zAHxWBe8tvCz4.gxEnEpEag7NVz48ACQ0u8pVRElanAGFh_HJyNolDPSL8gHV8w vTHw0ACX.HrTBZrblPPI6mMu8p3GDiGGYdxJCewwf8Oh0GKyUUBryLfLFJuMKzbO84bR8m1DMyTs ppYbWB5uQwxSbAs7WZCRq1ULBSEjsyBMdk9O.m.HY5Mx40ze55_85hqQIJUtHj4JYAUBKKtUPCIJ JCosvzROGDKehhPTk.nLv9FdHcZadsTlool38JtOyJAlAG65WDGwDQFJumyeRmXDn2wfn1YkbvFt aWo4UQJaBD7hHjbRPiUQVkrte_u7h6nnLRlDrjss18Eng8H_wWyFKMMayzQNbetc.kPIV28GH6Ln 4ssT6PIqiLyYbKcAwE0TvFDKHKxk0aI4OSUVRoCOaWMZq.zpemS2GrVS9roZsLiZN2zDSHmv3EHr b0rxjeC6Q9aihQ42o8zxTtEBd3xlY14pUs3CdhDTh13ZP74XsDn2rYT13zzp4dcVzuDJC1HjjfKn 3zx0Jlml5qsvWrPedZj4Anwjf8DxvkXRcd.yzOkNOJuXJLTh3k26dFTA8YObrpKooP4WTlbD5DPV W0pL0u9vmor0ePN_DQq4Eco5ZKu5vvlz553li7UOT2uEiq6AHf2Z4DWtLTjbb_5Y_eE9xLLy0gao 6M3au14825fMhskK3___XZ.E4e4R4t2bfVeYE4SfIxOgOIVJkCCvbeasC95RXxAI08x2uDKlfCCh dUNBWD1DE1SKYIqKXtnuvouM51BQRicb.xNlKj1tuSpsH9bqInPOfVNLAQ7q09xdoH_0my4mJCTB 0w0H6B919YvjaGorZYOWqi6i9_.XAzdtTWAFfhEsJiVh8D8nzXH3gXvuK07Lf93TkmSJtjjM5.gT UiufPNGZ8LUwTcWef48qNKzM5BJJRB6lgRVDH5XCXh08wCQtKXzKipy7H6lSiV3.8CZZ1qlqPzNx tWcd.wJniR.A32q1iEgK_QPaHfciUSUsFyplMoYY1jxMG1XBBi4Y.3NQ0O8fO3Ebq6bxVuU96WPZ OdeV636t5Ff8dH_cVlvNj02so7PJKZPTyEhL1xQIwwHlJ7OJr6YAPyr1PJdjeLZwfzVTCSGTFqOm _aOZxzUNnUcD6gi8.fA8sm54et3GXHB0iuOtVeoTxs8WcvaFnNKWYCeH2t5Nd5WMzFs00Z9pk0L_ mGtXBmaAoPaPwpRtrHC4McnWeWvKrd54xXb2MNsmtf0030gTckruR1RdUnnhkLAcR1l1ZPM.E43Y RXwDu9.DkYRkfNtuEn51XyGZ1UVEch4ONSRgNL6RPL.7idML9tyOnHqBsC08bp.kRaNx2TNmFA5V qH_30.JGwp7R0EVkh7HbfBEsr8BnBx0OHjhPj1KnXq8PS0gtPoqrQyh8KkkUcNWQnwW_pumNCvJs X7qzrZlO3sqTzjn7YmZ5X_z0KKXSlX4bXOGQlaF0hbOwm1ATsYu4FmsldkPkqkHKO6.P5UchtffU 59P5eNOz_TLP953SPwFnxKth8mBFLhwPP7WkoES3ed9ScGluLAPwonl9nhh7YMrNYH9U0CM7SDgF 4YOPEgD0EKWTTv.SaI3dzdlUNLJ94bPNg4t3_7k5UfJwhZFya27J3TFv62pFd7JDNw.JCUBz2cEK qoLXyPDGuR0covYzfY1caGyTVNm0_MWYmTeBCbzEjflcyOMqeDN2qe3YKLwdOnjpbSmNU2_6fw83 OIq3JLeoHT_TtnrspSdAsEl0VScEOAu0FXdRRWcdoefW.cZyyMdKy4_XS8RfCDUndpMTQZXYvAqJ zH2hD7bLk3g_DJTzfGgxJBsGE2Jur4iF05Szwuts1Cqvpnprmsba0X6X2iAyBzwEnlv2tX5NG409 lpKZE.NAXdQ1dYmlrZErCbX64f8sTkI.cvj0aljxnLhTkaCdVOa98Tq0TJCrII1RCmd3j6TK6yFN 0o0YV0Wjp0qhVTaY4oZZBVtae.J9MvOdGJKdR._bFhACIGgLBra0o37FmJZiiDYLqBcxbZyes8Gt 02xkjo3LSytq_ytTEz6hVFW5cqlzQNH9sAbxbrGnGIHiPweD.vWz6Y0Qptelhi944RRVLTuNSj79 KTv.4mUiMqSfb6IEwSmIImXOFVRCxO7SX7l0E5IT7I.6E4voZnvJv4GfScUCPYWlNI0GguSqNsVD bGArY.cFjjurjljx7tbKcEvbgAvV2jnUf5Z2I4tzsGtijn0L391BzQZRtA_dbYdzzgC.FTfu0uX3 m1S4- X-Sonic-MF: X-Sonic-ID: 4b74fea8-911f-47de-b74b-c6939f2b02ad Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Wed, 22 Mar 2023 20:34:34 +0000 Received: by hermes--production-bf1-777648578f-75chz (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 05e281b59b3dec1abae7697cc586a085; Wed, 22 Mar 2023 20:34:29 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> Date: Wed, 22 Mar 2023 13:34:17 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from] X-Rspamd-Queue-Id: 4PhgF85HQFz3C2d X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 22, 2023, at 12:40, George Mitchell = wrote: > On 3/22/23 15:21, Mark Millard wrote: >> George Mitchell wrote on >> Date: Wed, 22 Mar 2023 17:36:39 UTC : >> [...] >>> Here are the very complicated instructions for reproducing the = problem: >>> 1. Install and start misc/dnetc from ports. >> Installing is likely easy, as likely would be building >> with default options (if any). I know nothing about >> starting misc/dnetc so that is research. (Possibly >> trivial, although if it has alternatives to control >> then I'd need to match that context too.) >=20 > service dnetc start I built and installed misc/dnetc and got a binary blob that clearly was not built in my environment: # file /usr/local/distributed.net/dnetc /usr/local/distributed.net/dnetc: ELF 64-bit LSB executable, x86-64, = version 1 (FreeBSD), statically linked, for FreeBSD 10.1 (1001515), = FreeBSD-style, stripped Way older FreeBSD vintage than the locally available toolchains would normally build. Some might be cautious about such a thing. The man page reported that: QUOTE If you have never run the client before, it will initiate the = menu-driven configuration. Save and quit when done, the configuration file will = be saved in the same directory as the client. Now, simply restart the client. =46rom that point on it will use the saved configuration. END QUOTE I've not seen what the configuration asks about yet. >>> 2. Run "make buildworld". >> So on the 32 hardware-thread (16 cores) amd64 machine that >> I have access to, the test is to only have buildworld use >> about one hardware thread, no matter what else is going on. >> I never would have guessed that the steps would not involve >> more like -j$(sysctl -n hw.ncpu) (so around -j32 in this >> context). So it is good that you provided your note or >> I'd not know if I'd done similarly or not when trying such. >> [Note: -j1 and lack of -j are not strictly equivalent in >> how make operates. As I remember, the distinction makes >> a notable difference in the number of subprocesses created >> directly by make (one per action "line" vs. one for the >> whole block?). So even using -j1 might make a difference >> vs. what you specified. I'd have to test to see.] >=20 > I am literally running "make buildworld" with no additional options. So required for repeating your results, but likely making such results not be interesting relative to how I normally deal with buildworld buildkernel and the likel, no matter if there is other activity in an overlapping time frame or not: my time preferences are too strong to wait for a single hardware thread to do my normal builds, even with no competing activity on the builder. >>> Standard out conveniently reports how long it took (wall clock). >> But nothing in your instructions indicate about how >> to get an idea much progress dnetc made during the >> various tests? [...] >=20 > Honestly, I've never worried about this part. But dnetc logs its > progress in /usr/local/distributed.net/dnetc.txt, though not in terms > that are easy to relate to real-world progress. Oddly, when I run > "make buildworld," I'm primarily interested in getting the world = built. > Perhaps others feel differently. Off topic for the specifics of the actual benchmark that you run: Then why not use of -jN ? In my context, any buildworld using -j1 or no -j at all takes a huge amount of time longer than letting it use all the hardware threads (or so). (I've avoided having any I/O bound contexts for such.) It does not take additional load on the system for that to be true --including on the 4-core small arm boards when I happen to buildworld on such (rare). >> [...] >> FYI: I've never built with and run the alternate >> scheduler so if there is any appropriate background >> for that that would not be obvious on finding basic >> instructions, it would be appropriate to provide >> such notes. >> [...] >=20 > You have to build a new kernel, using a config file in which you have > replaced "options SCHED_ULE" with "options SCHED_4BSD". -- George Thanks for the notes. I've not decided if I'll do anything with the binary blob or not. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Mar 22 21:37:44 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phhf251SXz41Kr0 for ; Wed, 22 Mar 2023 21:37:46 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phhf24W14z3Jxk for ; Wed, 22 Mar 2023 21:37:46 +0000 (UTC) (envelope-from mandree@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679521066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KRhoI9lqO7S+my/Pr71ahXIlY9d6dzzX7WInT96y/7M=; b=jR4AmYMws45dajcsM6g38jW5ooa73/fKQoCGKm8G4Qr0FXWl3Wcztl2QMD989fXvhCrvSr j8RfCAtPa4B9WKpa/Jg8sdP36niNUw3nDYbWrqvBfgjdNB4dbC4xqbgELVxVownM8Ds7dM DoUjEMCP1sti2WiHZGjoMxYeFtiuqn8pNudqkAlChm8qxDvHip4zCvpBZHMI7Ajxuxx2Kc uAkq1f0M3nzkrHjYDm+ELtU/8rOWwO+EHaRc9i1XNP8dqMcHW51SEGWfTbUDmg80SG3p4j Z1tB5E82Szz1hLUAj1j+nLPX1mSE1m2GOw0CJHc+x6JPxMjvrA/z87PxU4GWsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679521066; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KRhoI9lqO7S+my/Pr71ahXIlY9d6dzzX7WInT96y/7M=; b=RvBqodSc5bwg+rze7FWTtfjANzFjBuht6RwhNk7sTjzaeZdspJlRV5U/r2h4QpmifNnQwS ZArNOYAyvq/4K/JymIWaXIMdMfbtL7+jc5fmSw85zlyfkc0HXhhq7LOwvk9G7F2ISVLhqq FQM+vwq/dWzql5asmEwMPDWeJd4bB/uKCzxDljtKSW0IiPahe5SF5GaP3Nkwm0jBSbp/7m oN80WZ/iPFgCByx2MGOqMqPMAcHEzMyHfBGZzWDOOGmVixJxodiyvaM6jMag2wjUEXFUz0 LwFEPaKQFnzgh3vAg4QdMg5dC57mV5DWzqMKBT+Q2aNR4nJU5+CYSx0rdMnz7w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679521066; a=rsa-sha256; cv=none; b=ezumRU0LeAShaltXN0dpY7UQ7XGma38sfw3/7Wyx04kSGenmozE+X17qSjaH215BE0z6hc 2sGLZJVkEiuizTB3VPMiwnuD6MzPxJ6IzUlkAC6/KnStrMGTqIM3OHHUp7zoWgYJWpyc8s bLWELWqOGgWUSTx9sZPRmjngC6xUUbKM4zSDLIexWQYkywtu2s95CZy5OKix7C/eWhUnJg 5aaXoGujnkgHkpqOiu9qbAXq/RehWv+Jb+3Zj31+KTuEqU+A3u7ZdbSzNySzrDnD6gxYoo QJ81bX20PEEILzpJIW3UMT3oYqGDrP42d5QIqtKeuvgdneshgWiMZHdHwzf1hQ== Received: from mandree.no-ip.org (p54a03c23.dip0.t-ipconnect.de [84.160.60.35]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: mandree/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Phhf22hzCzH0V for ; Wed, 22 Mar 2023 21:37:46 +0000 (UTC) (envelope-from mandree@FreeBSD.org) Received: from [127.0.0.1] (localhost [127.0.0.1]) by ryzen.an3e.de (Postfix) with ESMTP id 74999DB12B6 for ; Wed, 22 Mar 2023 22:37:44 +0100 (CET) Message-ID: Date: Wed, 22 Mar 2023 22:37:44 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 From: Matthias Andree Subject: Re: Periodic rant about SCHED_ULE To: freebsd-hackers@freebsd.org References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> Content-Language: en-US Organization: FreeBSD.org In-Reply-To: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N Am 22.03.23 um 20:21 schrieb Mark Millard: > FYI: I've never built with and run the alternate > scheduler so if there is any appropriate background > for that that would not be obvious on finding basic > instructions, it would be appropriate to provide > such notes. on amd64 13.X, I do, assuming sh/ksh/bash/mksh/pdksh: cat >/usr/src/sys/amd64/conf/GENERIC-4BSD <<_EOF include GENERIC ident GENERIC-4BSD nooptions SCHED_ULE options SCHED_4BSD _EOF cat >>/etc/src.conf <<_EOF WITHOUT_CLEAN=yes # remove after OSVERSION bump for poudriere KERNCONF=GENERIC-4BSD _EOF make -C /usr/src -j$(sysctl -n hw.ncpu) buildkernel installkernel Alternatively, you only use the first cat, omit the src.conf patches, and run this make command INSTEAD: make -C /usr/src -DNOCLEAN KERNCONF=GENERIC-4BSD \ -j$(sysctl -n hw.ncpu) buildkernel installkernel and then I reboot. The previous kernel (kernel.old) can be chosen from the beastie loader menu. -- Matthias Andree FreeBSD ports committer From nobody Wed Mar 22 22:39:24 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phk1V2Q8jz41PH1 for ; Wed, 22 Mar 2023 22:39:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-55.consmr.mail.gq1.yahoo.com (sonic316-55.consmr.mail.gq1.yahoo.com [98.137.69.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phk1S5XhPz3q0J for ; Wed, 22 Mar 2023 22:39:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ZvvRfsjF; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679524778; bh=gZI3kwFTWGc2+LFnyqWLmUnRd2wyIaxpVmuT7fsjqNc=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=ZvvRfsjFQ0/UKtjGz8SEsuGcCquc6P/Sakfo0UMPyjMIKlxxcSdWxvSnXA9fJYYQUw5K2u3JW4FuwHC6Xgz9V7CUjFZwMywjKvg3JK86JfM9ivL/LwgNHDfLzM16kI04W1B0uW+Wk2fWK7eTTjLr/Vd1nRyt2pwcY0XrhNtyjYF6iD8tAUQSEyRBQbJpJ4SSn9OkU6L7fmc7rNVPD0tjnl/3G4GEgPowbebTpd/QbgUMKN5N9KyKrSlSwQEjQsnqB+x6rW34uZdh8inO7z2up/WseH2SD/Mm0e9H/IIxuv7xfyslAt0DrSfu4UvzbuniDxcL5qpxiJTDDeCcBTKNig== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679524778; bh=yBn1FqytqSk0EAYTJM4O/zr7vKc1oLGp1aoPCPIEiTV=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=VFka3VPFclJLiel5LoPYWE8UJIDA0UWSJdSFhcs5XDRNmbrgpF01PtcwyR1AIXeqybCbVVG3Qb2zYr7rkfy574Nj33ke/pbxNItfNIz2kZUGFsDWhhF3CHBwVrzMjWa8Eof1S3AO9fOeBhZQp9HyrIO/ix9RMDlmi0kTUQMAE+oiF/7uUn9BlJ2HFTXGWH/gGoQ1ra6QO78qg2+jSNwptw1CLZAlWbQwHwN/S/PEeiPQA4dWKT2jZBRMynlM4CEfH3JorscJrzN3adzBH8s4vJonYi1QJyArkB+K87yoj0nEOk3MGLDotpLee3RnQw7DHrUuun8ccR1WpmEzbZd64g== X-YMail-OSG: rT1Ve3gVM1lOVDtp3dLy4BW5pz2_Vv7M9FEWsCHdrYiOe9fgEcTTSTBByIdyPf0 gSXVKYNPzn6W69dPGrLqla.zxvO2lsebEt81tc_S8z04Z0Y3ey9rQSytyVc23IMGCzZmvpfdj3O5 2.o1Xw7McSzMNbyIgN99oeBqCbQoMtpRtors9p_qEv_.r6WzP.Cf1KyYAiaEVOg5o3iggRCkl9xp xQeo0HOqfTpUemNsAACfpwpqapQaDnIabvHcZg__PnAcjwsqL02x2jUTvEy67iUJBBrjWlHx90h9 pF11zTX80e1lDbKPa015iLesXJZ5ssjwDIb2_bRV5PHtkA2CpO0nJ9mnZnLpJlkfSj_3qWX32VRu 6NtZmlkYYLNUZDUfJ1oYVrICw6EIbT8UcD86toYoMn7ry2XHS7yv6W8K1j8IkTVEKgbsAIXseIIb wRgvAM_wcZYTX8GICO8FiTZoRqNd5tVPIaenvM2ptnJYHTtc94zDGxu1fRS8zR0HJXN4sq_jGowy zHcisEay20JaTHL1AfiEihvpWUnGoCkF6yzi70B1YS9KlPWzXKzpOnSDnOS6HL.rPKLZ76lsI4Lk uQcybc5eLQPvgHDqOgWNwsaKnwnIoKiF1reMwaA74i.kDv8yDJC2tyfVaR9CgEUrKksRliA3F1Xl g4xF2jbeMKQofl9Y_ZJUoVbajiuMyTHz_6W67Iu1X1nNkfJdVC3hK2utuHc_I_QJOkg5vLQXoQZF Rtd3f00GnXQeaFpB2L8QLoEdcXnMhg_wxWUA.hVrdc6sdVOW.HLBQxzsmVjGbFFNTmlj1Nqs7orM tBWhTHhIyYrzYURyKGmclYzifi70RaSphUa3YUTNKo5sIa5FKTW9LVhFHvhejvK5HsW2oZZQeB9Z Dc1oL133TMqDuqhKLEjowt.wQUpp33PsEiEAECgQ9mreZahPtLyDtKfQ2IUxsWIUJSgenhzUv_p9 YAf2xfH5u73_Ut2T8Kjj_75hoXN9PUy80SqaF1rxUTk976CZ7P0fKZi3vHoImaDr0qn0St_fY0.x AAevRMusIlzN37WSIsZ7bO3GjUsYBR35Sa_xNl2qKmMqVBdojR4VSIlic.u1kZLrthdpmYPyK8gQ xfBsxkceS86rrTSAmWoX.AeQ7ds7665Ci.iSzt.qzpehG1WWpxCtIevyX5M.Zgat_lCPs47gS8xO 9CEPM746S73B9VzvPfIDBiUGykGWWogqWpj8qoHge.8ftCKK4rh_W3rOpMOY77cToodXyVJjcJ6i 78AghTC0JVtkyoL.MgrqGXuvDsTkud3.qAIPmo7UuTcX5HXV8QRbdglyt9pzxOvBy11LIduR2krd 6MoRA1CqyHPWvGwVspIAdt_NhSHbPL.BjcEq693YFNrGNApYKe81BXUrpdfnEbUHioFqHM3D5fSy neIoZkm2ut3ie_O9YLcz29hIyBAEckrd0qyXCQCY5nyHHWJGSfAvR6j.rLqa.3NRy2iRzxTn.KNR cBny9H4.KOcRXD14Zyrsbjlusg6JfFdj2U18FIGUv6z6TyQCeWLfWcAur1NvB6itGr6ZUzkRf9H4 BxJeK9q81gzc3F36FiqxYlSS4enOfvJfRO9Lax947vzN.JgEaANWCImGYHpuFVA_mMh5wVkZi9ib GxwtMjdHs7rlmuPEGhHhOIH3GcgPVMNAbDs1I2mKkLVtodGioat2p8rxB_kusnqC7avkeiOIOEHL .zykVLSquu8F3Dne3oCTEqwf64iaz0kAS14lq.ktu06LcloUQzhKVElaO3pNXZDpMoYHnagiL9Ld dCK7tQWYTwxpKqSXB.8A3JruhXIHaMmZ2cdSowS33ISI8_j8v1YAsSr3LcADpjoBUq9AhwxpzgXE aEbney94pnWB7stvEIQFTD1RqJ.Nk3CyNxaq4apMo54DSh2Uuh83EFgebELCT7ymMikoAl6XiDuV Weflt2kZWD__UObRhaystWQiW1T5v170EmqtwruBBQWtLIkiNMj65LYO8O_Ebzml7Jh3qDHbhmyl GdSIMQb.EU7YrwLCrd6mlcYPgvhno_11CgJiqF8djwiAmD._Ykb6eAvycFerP._.XfioN1PQwbcY eQF5I3goTJrh4I1y.AaXjnxSf3U_R4H6NEEAZHE3fW75PWUflwC3BR0MkzeIwO6yyWmr1nW8x7LK Qd0vFWlOo40I.kXoVXfHuM8MOLLbRwe.2ikJAx6pEgaU82k5fchZbdzv3qSW4195nMRribxm.YvF RLiM_qc6GD_oqYehcMln28fzPAOpK9NboBXAug.f.OEq8lje8h5Xyotu0TMdfaMnijIfkZoez3cA BHA8- X-Sonic-MF: X-Sonic-ID: eccea552-2096-4c8b-a6a4-d47d63371bc7 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Wed, 22 Mar 2023 22:39:38 +0000 Received: by hermes--production-ne1-759c9b8c64-2s6ww (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7c3ec5be265e86b0e2f7c9ed089cd386; Wed, 22 Mar 2023 22:39:35 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Wed, 22 Mar 2023 15:39:24 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.31:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.31:from] X-Rspamd-Queue-Id: 4Phk1S5XhPz3q0J X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 22, 2023, at 13:34, Mark Millard wrote: > On Mar 22, 2023, at 12:40, George Mitchell = wrote: >=20 >> On 3/22/23 15:21, Mark Millard wrote: >>> George Mitchell wrote on >>> Date: Wed, 22 Mar 2023 17:36:39 UTC : >>> [...] >>>> Here are the very complicated instructions for reproducing the = problem: >>>> 1. Install and start misc/dnetc from ports. >>> Installing is likely easy, as likely would be building >>> with default options (if any). I know nothing about >>> starting misc/dnetc so that is research. (Possibly >>> trivial, although if it has alternatives to control >>> then I'd need to match that context too.) >>=20 >> service dnetc start >=20 > I built and installed misc/dnetc and got a binary > blob that clearly was not built in my environment: >=20 > # file /usr/local/distributed.net/dnetc > /usr/local/distributed.net/dnetc: ELF 64-bit LSB executable, x86-64, = version 1 (FreeBSD), statically linked, for FreeBSD 10.1 (1001515), = FreeBSD-style, stripped >=20 > Way older FreeBSD vintage than the locally available toolchains > would normally build. Some might be cautious about such a thing. >=20 > The man page reported that: >=20 > QUOTE > If you have never run the client before, it will initiate the = menu-driven > configuration. Save and quit when done, the configuration file = will be > saved in the same directory as the client. Now, simply restart the > client. =46rom that point on it will use the saved configuration. > END QUOTE >=20 > I've not seen what the configuration asks about yet. I went through the configuration, basically just looking at it, other than providing an E-mail address. Then . . . $ sudo service dnetc start Password: Cannot 'start' dnetc. Set dnetc_enable to YES in /etc/rc.conf or use = 'onestart' instead of 'start'. $ sudo service dnetc onestart I just let it run without any extra competing activity, other than I had my patched version of top running. It records and reports various maximum-observed (MaxObs) figures, here the load averages being relevant. Top showed that dnetc started 32 processes, one per hardware thread. Mostly I saw: 100% nice and 0% idle. Letting it run and then looking at the load averages (and their matching MaxObs figures) after something like 60+ min (not carefully timed: was doing other things) showed: load averages: 31.97, 31.88, 31.66 MaxObs: 32.12, 31.97, 31.66 (Note: The machine had been up for over 2.75 days before starting this and had not been building much of anything during that time.) I've not yet experimented with having other, significant competing activity. >>>> 2. Run "make buildworld". >>> So on the 32 hardware-thread (16 cores) amd64 machine that >>> I have access to, the test is to only have buildworld use >>> about one hardware thread, no matter what else is going on. >>> I never would have guessed that the steps would not involve >>> more like -j$(sysctl -n hw.ncpu) (so around -j32 in this >>> context). So it is good that you provided your note or >>> I'd not know if I'd done similarly or not when trying such. >>> [Note: -j1 and lack of -j are not strictly equivalent in >>> how make operates. As I remember, the distinction makes >>> a notable difference in the number of subprocesses created >>> directly by make (one per action "line" vs. one for the >>> whole block?). So even using -j1 might make a difference >>> vs. what you specified. I'd have to test to see.] >>=20 >> I am literally running "make buildworld" with no additional options. >=20 > So required for repeating your results, but likely making > such results not be interesting relative to how I normally > deal with buildworld buildkernel and the likel, no matter > if there is other activity in an overlapping time frame or > not: my time preferences are too strong to wait for a single > hardware thread to do my normal builds, even with no > competing activity on the builder. >=20 >>>> Standard out conveniently reports how long it took (wall clock). >>> But nothing in your instructions indicate about how >>> to get an idea much progress dnetc made during the >>> various tests? [...] >>=20 >> Honestly, I've never worried about this part. But dnetc logs its >> progress in /usr/local/distributed.net/dnetc.txt, though not in terms >> that are easy to relate to real-world progress. Oddly, when I run >> "make buildworld," I'm primarily interested in getting the world = built. >> Perhaps others feel differently. >=20 > Off topic for the specifics of the actual benchmark > that you run: >=20 > Then why not use of -jN ? In my context, any buildworld > using -j1 or no -j at all takes a huge amount of time > longer than letting it use all the hardware threads (or > so). (I've avoided having any I/O bound contexts for > such.) It does not take additional load on the system > for that to be true --including on the 4-core small arm > boards when I happen to buildworld on such (rare). >=20 >=20 >>> [...] >>> FYI: I've never built with and run the alternate >>> scheduler so if there is any appropriate background >>> for that that would not be obvious on finding basic >>> instructions, it would be appropriate to provide >>> such notes. >>> [...] >>=20 >> You have to build a new kernel, using a config file in which you have >> replaced "options SCHED_ULE" with "options SCHED_4BSD". -- George >=20 > Thanks for the notes. >=20 > I've not decided if I'll do anything with the binary > blob or not. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Mar 22 22:57:08 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhkPv47RHz41QGY for ; Wed, 22 Mar 2023 22:57:23 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x533.google.com (mail-ed1-x533.google.com [IPv6:2a00:1450:4864:20::533]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhkPt53r0z3rvS for ; Wed, 22 Mar 2023 22:57:22 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=zYU4hzHx; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2a00:1450:4864:20::533) smtp.mailfrom=wlosh@bsdimp.com; dmarc=none Received: by mail-ed1-x533.google.com with SMTP id r11so79348021edd.5 for ; Wed, 22 Mar 2023 15:57:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; t=1679525840; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=pgBIm4Aw9LZRBdPUn+cHEy4hIxCSt9kQ6AfUHtc+1ic=; b=zYU4hzHxd0Z5hWh1B/1AhaMwVx1F9t+aiZP6xlncLWKeO5RMtIdWD50WGFEgZUQK0V f28EMT5Ivevg0TDFUpdiJc+S+J5u0Ei6TRertftTxFG/aY6XBMNz6VDWqquKoVqnq6s+ 3gWO+LsIc7kI9ViXD9qJYDCPExU8mn5JlPviza2Edp1eOy+xVvJ7Nq//N+uJVfCylh5u syvtOHliXWtHElez2+pq8qJIbb7H+eO4aHMi3VkFOJjYM8TxgeghWXSJAJ0406CEZ0Ab agnXomfTE/IJaMkChPDlcR1bojvJILIhAxy7h+GBfJAai0SUWHsqQCWtrF6EtXo0XNbY 8h7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679525840; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pgBIm4Aw9LZRBdPUn+cHEy4hIxCSt9kQ6AfUHtc+1ic=; b=w8d5VQ2ufR8PqUPF21qG0dPIF4bPTpFJng2mbNhFmwvQr4UJ2MhiSZq3XPHVba13Rh ostxZTSjlkD7AqfOwoZyboLlA4YLHKDdjOCDi3J/Y76xSEB5MkY6ziwT73BNFjN1L1rY jthaykRnmO8BrABaOXykfTg3Cf72Rg0CJSSxvqIpNKKQmIRfG7jV5b6KnpYB/mfZKi/Y 6Go261ZjcXHSGXhR+2CpvANC/IvsBIUOud2HT8Tue8PkM44TaotkLv1gjj+uLlqgjNma saPj/y2w3e3xlSUIhe9E/HHGJlCzZK4Cg0ZouFJGL4cZ73eLeAyoeSksWDgPGEY8Gp/t /1wg== X-Gm-Message-State: AO0yUKUiVlke28l4nvATg8qntTqLeHd1FuhqsTD1KzbrLo02/r58rhBe eelw3aJVkKRBR0Zh8gdqZ55GKO1NVqm3AmwAbYhBWw== X-Google-Smtp-Source: AK7set9hHxhiFYEWEglB6FT0+cvncB7tH07EZJ5WT5bFHSeYl2aBokx/xnVPhsNcNCGb/7g1ptiL0lmMkDDWcU9oHc0= X-Received: by 2002:a50:a698:0:b0:4fc:e9ef:e033 with SMTP id e24-20020a50a698000000b004fce9efe033mr4273642edc.7.1679525839672; Wed, 22 Mar 2023 15:57:19 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> In-Reply-To: <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> From: Warner Losh Date: Wed, 22 Mar 2023 16:57:08 -0600 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: George Mitchell Cc: Mark Millard , FreeBSD Hackers Content-Type: multipart/alternative; boundary="00000000000010aba205f78515c9" X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::533:from]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_CC(0.00)[yahoo.com,freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; TAGGED_RCPT(0.00)[freebsd]; DMARC_NA(0.00)[bsdimp.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4PhkPt53r0z3rvS X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --00000000000010aba205f78515c9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Mar 22, 2023 at 1:41=E2=80=AFPM George Mitchell wrote: > service dnetc start > I am literally running "make buildworld" with no additional options. > > So what are the results for make buildworld -j $(sysctl -n hw.ncpu )? ULE scales much better, but when there's too little to do it can make poor choices. ULE is better locked and don't fall over on high core count systems like BSD does at moderate load. Warner --00000000000010aba205f78515c9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, Mar 22, 2023 at 1:41=E2=80=AF= PM George Mitchell <george+f= reebsd@m5p.com> wrote:
service dnetc start
I am literally running "make buildworld" w= ith no additional options.


So what = are the results for make buildworld -j $(sysctl -n hw.ncpu )?
ULE scales much better, but when there's too little to do i= t can make poor choices.

ULE is better locked and = don't fall over on high core count systems like BSD does at moderate lo= ad.

Warner
--00000000000010aba205f78515c9-- From nobody Wed Mar 22 23:17:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phksc4WHYz41Rk9 for ; Wed, 22 Mar 2023 23:17:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-8.consmr.mail.gq1.yahoo.com (sonic307-8.consmr.mail.gq1.yahoo.com [98.137.64.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhksZ5SwCz3wKW for ; Wed, 22 Mar 2023 23:17:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=dlxPQnn7; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679527072; bh=MA5+jZW+3JnATkgSUnztu3vlCftHpE6VgiwgQObGfZQ=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=dlxPQnn72egK/IiO3JYWMyLzfoCr0tYb51GeOdJ+E2awaGj5nBY2cn+Wj0MdZgCuzmQ2HJdnlJkR0b/JhbAY7FTwMG2+vu5t0YfYD7w91dDfYzP6THVXSqcmQxL4T1Nbw/XXhZckztiSq5uLVPCKplZDYd/Dwa9w9TkXLNq/2Rv8MVvPjcoAbXu0VQy8rl7EdqzUu/YrQgCQgAqCuPpbct0txNsokl0xGVUg9FByUL4FQqAlOgGshGg7HDMtPLouG6qrlPActkEKpNv7AdSpTCrl+6Lhqpa7DRmRT4fdsUf193lY0OTl4czA3afojrehgclhshxb/b/wpOduGOG8/w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679527072; bh=yKux47FqC3llE8dP+bNPDs0n63/RoSQP6LntMi6ZICo=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=mXR1k8/lb/I6yMpEE3byaxqsLzLXh6DW3FRNxjk6ZxZHP5qtNt3PVpD6stITTdWcp/l7OnAblA+9mrCaBaspvJmy+IkeGulj0NPhsgMVLkzSyFhvJaEsFH88rhE51YlQ0wceb3DXTCgD9h7RdW/ixYa4dbgvLfroqk2Kk2HhwPVygRqZKXbZLZElSeHe7qbtY+ckLmsd3+3Gv0nlnTfnciKhtom8oHRUfG9wsOb+AJoL73FlyLhEiI5+AEhsaSnpHgCiCeHvJXIRjrV0OCzaGLC/FLE57DDqjalJsx9YRrNTnDcj1Qlon4ulAUfghs42ESrtBTc+dgLd+Lzk2cBmTQ== X-YMail-OSG: yp5eXB8VM1lJr6Kyv5dptkHnvRyla3SkL7hSNkULXJqI1lQzzxZgENTVg7.EaYN uDh0ez8ZFV_uiwofNLwibbuqBU8IcCzfmFem.DoEEH5mofEYE6tFYfzvef4dj1uzrosvCaClQIV7 CHLk3.aFNPil6Y6cKJljCM2nAMC59_dKtLLNJN7wbms7Lh9VekjXCukYouj1fRchz._JtcCcEfPy guR8HOp.k_PFlV8rrPbwe95Oc5sBchxXbVQo8HIkWsl0Txk1.MJ1s8XJ6P4T_x0JESVwAKLUMmlc RPQjK_hMoyevAP7XXBhKINEJ0gbwtFUAo1nTiHzW177BObWrGT.UpYhqIqMiPNgo_7y8b87lVhpu qwwqPOwEqYA_XLpiTaN4qudmCleirZQqIvjeJm6XXbNVS8V8J4Vp4eR6x6Y0mSaCeaYO2VyHecr0 c8e7sA0sl9d0Xv0PLftLTCVofbB74TymWxQ5IfpnWv.1EinmSACqgnOuWvltiw3ISETeNnjv.kGS 6Y3k0_A_Vm8Tt6mYC0V0R1Zjsx.yHdKugHXCBBeaN4Ir44dVl.jTiGa8GgLXQfgoqffrSaWOUtzk 8.tSUqt_dkvC5lCfj3R44Z4kO.Vc6dGxoHznhN3u7rOAoPMWg40opoAUAHbtn4W857SwrxJUVRyw YTqbqpmcZvAOQysR0JSSjjMKsGcdvahyExTEkX3V858UdI0eEEtcPQqHQ8gCCPAVlNr3CsSDd02P ze8ktTTw_uhlSzFjQiQUnvJRQcM13oXUCrYkd0qYhMO21wCtQPw8Hhru1WbvvZiRWep.jIzExQN4 jIlJPKNGvPLsZfHvTBXBDyNd.mZOqMaRxYirY16q0CBwGyjoA_lQWvm8DeRLegaRxOMdNNGNdGKz JowLGjFNKzNXMk2QMBZqJXo8dcGtDbDRS53lzYCNps6z3FCYF82V2GUWHWKCRlGPgnaaj.K2MtKc GuPMKg7jBVBD2xFFJUiNAqGs0NXBBh9b6L_xZ8olj.4CX_GAoZupw1g42Jv4nA9R4am.YUSBSG1r oKSlfPpLvR9fTOOGj3w3HvPtjeOYq2h8QjpFmiTA4J1zectFToTMFLD9kjtmO9WYwrz8pEAEg3KJ qud4oGH0lOyC3XWVrBg2rYwf051cQTXHefOnf3w61ZqN9hpVB78FkKvBE4X.ZHi4RgTaoS6tXdZh EDuIRNvJDD5nYI9_1DKpy.oFlXgTIf34dT6fMyV_QWQKr598OHYIqbAgnLGZ_i4vr49p4EGwhc_Q d.Nax4trDpfcwuxS8nqEpTtx_lr8GfEpP1ixaE.i5DNEHGiiOZ4l10DLlXof4EULZurYnrhCNmjK yMBF98fln8pnGR57F3r5dgveMAy93iA4R5Xm4EUgJSPnpA143A5PhXBB_BySSV8GVt3eNd63zq6w QFHt4iR8gQKSFdtzoSJTArkMz6Kdr6LT.xgp5i6Fu.7fzUhz980K1gia0A0JONVFFKNQZcSaY7ST N9mGz1gT.XL.O4zVLZLoMcGn_4MKzevM73Nto3iCz5oKhAHuGNBEmof9Ylw7xpUTTKjaxTmTJha_ rv7rnHocUEgPs2NNh4gyv2xWAg6aapGDuMlA11elEkuVqF909osp7Wnzfg3qWIC386Qz45dgdSEA N4r5Mpo.mo1K7WrYE0I_HBmD459nFU8tvPdM08XDsCt5yLlBTbE9fJVO0xKzuH5weOunUbakxt4i TynVG4PmpHhZvLqi5QF2wyjC2MQq1s1AU1bLghTyPHZjYocgfi.2X0.Vly9baQMPHOJh_12_IG2d CQBA4jrEaOm8vo8qykwyJlSfIAcoSek4D3z5rgOO_Gv6z3Q9VBm4banqw0jAlAySqwYbV9_84KK4 XLiwQHx.eU3n91kSKKpnRcL.jnPeuNlqLXBSIfWuJiKfojgzU1ula.VaI0.JjIeJqwaUf8uxNnHe LQuM8UVgKmbAb0cAhD935Th7XB4hD7fyXfvWs.Al_kd_SexZfzCyAb2O7abz29Thf._g3TbLOXnA QAFI5XD5aV_51pS2_msGzAE0s2sUo.Gfy4cy2Pi_RfQt9REA1cW1xxCYbkNoDlLFgTcruQX0uL3s E9r84L7BsGxgXleDnd3iWqGTblIGzaBcNJ4R4a4x2nht4xx1pySTjX1oXBMxYw1wdYEPgsVJf3t5 lLSHccuTRxGo781selm1NyoRJNxCzL8uvbbgk8O.gl_BN8KjX2lH7hA3IL6LXWEy7QYpepDBiehE oigQhGyWyTcAzMmnmARXdRgKzRlHzbxWt5s06bcI9YG_DgWd9YehE5xdirm6HZm1PVQLBxwVDjMx h X-Sonic-MF: X-Sonic-ID: 281e5939-a81b-4bd7-ac4c-efa8289d0731 Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Wed, 22 Mar 2023 23:17:52 +0000 Received: by hermes--production-ne1-759c9b8c64-l4sxl (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID bfffc2696ccfa47ac23588f2763e5cc0; Wed, 22 Mar 2023 23:17:50 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> Date: Wed, 22 Mar 2023 16:17:39 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.32:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.32:from] X-Rspamd-Queue-Id: 4PhksZ5SwCz3wKW X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 22, 2023, at 15:39, Mark Millard wrote: > On Mar 22, 2023, at 13:34, Mark Millard wrote: >=20 >> On Mar 22, 2023, at 12:40, George Mitchell = wrote: >>=20 >>> On 3/22/23 15:21, Mark Millard wrote: >>>> George Mitchell wrote on >>>> Date: Wed, 22 Mar 2023 17:36:39 UTC : >>>> [...] >>>>> Here are the very complicated instructions for reproducing the = problem: >>>>> 1. Install and start misc/dnetc from ports. >>>> Installing is likely easy, as likely would be building >>>> with default options (if any). I know nothing about >>>> starting misc/dnetc so that is research. (Possibly >>>> trivial, although if it has alternatives to control >>>> then I'd need to match that context too.) >>>=20 >>> service dnetc start >>=20 >> I built and installed misc/dnetc and got a binary >> blob that clearly was not built in my environment: >>=20 >> # file /usr/local/distributed.net/dnetc >> /usr/local/distributed.net/dnetc: ELF 64-bit LSB executable, x86-64, = version 1 (FreeBSD), statically linked, for FreeBSD 10.1 (1001515), = FreeBSD-style, stripped >>=20 >> Way older FreeBSD vintage than the locally available toolchains >> would normally build. Some might be cautious about such a thing. >>=20 >> The man page reported that: >>=20 >> QUOTE >> If you have never run the client before, it will initiate the = menu-driven >> configuration. Save and quit when done, the configuration file = will be >> saved in the same directory as the client. Now, simply restart the >> client. =46rom that point on it will use the saved configuration. >> END QUOTE >>=20 >> I've not seen what the configuration asks about yet. >=20 > I went through the configuration, basically just looking > at it, other than providing an E-mail address. Then . . . >=20 > $ sudo service dnetc start > Password: > Cannot 'start' dnetc. Set dnetc_enable to YES in /etc/rc.conf or use = 'onestart' instead of 'start'. >=20 > $ sudo service dnetc onestart >=20 > I just let it run without any extra competing activity, other > than I had my patched version of top running. It records and > reports various maximum-observed (MaxObs) figures, here > the load averages being relevant. >=20 > Top showed that dnetc started 32 processes, one per hardware > thread. Mostly I saw: 100% nice and 0% idle. >=20 > Letting it run and then looking at the load averages (and > their matching MaxObs figures) after something like 60+ min > (not carefully timed: was doing other things) showed: >=20 > load averages: 31.97, 31.88, 31.66 MaxObs: 32.12, 31.97, 31.66 >=20 > (Note: The machine had been up for over 2.75 days before > starting this and had not been building much of anything > during that time.) >=20 > I've not yet experimented with having other, significant > competing activity. >=20 >>>>> 2. Run "make buildworld". >>>> So on the 32 hardware-thread (16 cores) amd64 machine that >>>> I have access to, the test is to only have buildworld use >>>> about one hardware thread, no matter what else is going on. >>>> I never would have guessed that the steps would not involve >>>> more like -j$(sysctl -n hw.ncpu) (so around -j32 in this >>>> context). So it is good that you provided your note or >>>> I'd not know if I'd done similarly or not when trying such. >>>> [Note: -j1 and lack of -j are not strictly equivalent in >>>> how make operates. As I remember, the distinction makes >>>> a notable difference in the number of subprocesses created >>>> directly by make (one per action "line" vs. one for the >>>> whole block?). So even using -j1 might make a difference >>>> vs. what you specified. I'd have to test to see.] >>>=20 >>> I am literally running "make buildworld" with no additional options. >>=20 >> So required for repeating your results, but likely making >> such results not be interesting relative to how I normally >> deal with buildworld buildkernel and the likel, no matter >> if there is other activity in an overlapping time frame or >> not: my time preferences are too strong to wait for a single >> hardware thread to do my normal builds, even with no >> competing activity on the builder. >>=20 >>>>> Standard out conveniently reports how long it took (wall clock). >>>> But nothing in your instructions indicate about how >>>> to get an idea much progress dnetc made during the >>>> various tests? [...] >>>=20 >>> Honestly, I've never worried about this part. But dnetc logs its >>> progress in /usr/local/distributed.net/dnetc.txt, though not in = terms >>> that are easy to relate to real-world progress. Oddly, when I run >>> "make buildworld," I'm primarily interested in getting the world = built. >>> Perhaps others feel differently. >>=20 >> Off topic for the specifics of the actual benchmark >> that you run: >>=20 >> Then why not use of -jN ? In my context, any buildworld >> using -j1 or no -j at all takes a huge amount of time >> longer than letting it use all the hardware threads (or >> so). (I've avoided having any I/O bound contexts for >> such.) It does not take additional load on the system >> for that to be true --including on the 4-core small arm >> boards when I happen to buildworld on such (rare). >>=20 >>=20 >>>> [...] >>>> FYI: I've never built with and run the alternate >>>> scheduler so if there is any appropriate background >>>> for that that would not be obvious on finding basic >>>> instructions, it would be appropriate to provide >>>> such notes. >>>> [...] >>>=20 >>> You have to build a new kernel, using a config file in which you = have >>> replaced "options SCHED_ULE" with "options SCHED_4BSD". -- = George >>=20 >> Thanks for the notes. >>=20 >> I've not decided if I'll do anything with the binary >> blob or not. >=20 FYI: It is not your specific experiment, but I started my "extra load" experimenst with . . . I started a -j32 buildworld buildkernel with dnetc still running. I'm generally seeing around 55% Active and 42% nice, < 2% system (it was building libllvm at this point). At that time: load averages: 64.41, 60.52, 49.81 MaxObs: 64.47, 60.52, 49.81 =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Mar 23 01:03:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhnCY1jJyz40JcR for ; Thu, 23 Mar 2023 01:03:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhnCX0tctz46J3 for ; Thu, 23 Mar 2023 01:03:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=phAh7PiF; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679533413; bh=yTfU9CKUj/eBcDcyOXiYjc3zGn6cMji37M7a3LlJc4w=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=phAh7PiF1jVXX6aTvi8/qk6vSNNNibPzShJSTH5i/iBtqfMCT3L0pzEyFTNTNqjRDLJL99wYOm+QMTqONX+aZF5dO1ecp9wX7vT3ErSG+cvEy3oksByTC/l0c45Ez8gHB0aTVuRKgHi2F2cUve7Qq/bvbC1gE4bJOLBjxxiHhcEMt0Hxqceg9oaVv5vm6f23MEz1icxokeNgdAqIEF+ytvgs20uG1/v02zoIfycVLTrqFbksmwPqgg50ZA6VTS2ilwyJz2nAL05bdMEkF1GRmdDlrLzOjmD/2t/2/LwnVb8F2xlZw+qpoaZYwsvENtPG93TqOLchGW8qE4psBexKPw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679533413; bh=pJ3C+GVXrDpI2SaTMNBnmaguD4sQei9PupOMuoTcNFz=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=nmVpnxDiiF5WTl6BHD7U0u+AMHTfac/IjEjr3xtTezAxRvGNSi0jQEg9LqXeldUIcigiOBr+DT5OVJQyBxMiZdtmB4PBw8YgnM5k67WETUE9Ovn37MB4LVpqcCAFMXMBtcSwRKlG4p9NWFAX3nD2W8IP6eX7PXnzstOLFwNq0xbnHsBXbKHtyeMgGZNs3vWpdkWVTvyJQfMN0vkqL2KsawUs56n5B4gsmd4x9592XhNWS4rWsWgQsVHnmi7vmR+ptsvdCK7JO0oKDNaotcGYiFD/xvb/usz8DSA9hLpy81GCzWZEZi20aTUccdcUwOhfuOHUEV5wy1TrDnSFJnpSpQ== X-YMail-OSG: mcSh9lcVM1kLcOa7DqVF3M8e1e9_95oXMYXydcV4.OrtOMhVXu61Nch1ryJcfkx JPeBC1Cq44Dh2MDv_roEQsZ7vKotE82wzxQ_P_0l4H4ErZNz9sHBnj3bdKO5zXVI6KIDq_GSJDH4 vLbiKfF_ItXv4nfLTLksyaj7rAqossAIgbwQdGaRpUn6ssyXVHXVkTp5Eysm5VhNwsrkEnZBmg6O n.n4xrgr9U65Tut13TZXInNc5EdZ1ejF7Pll3oPE5I8t1auipaSd00GkHMg3sZnBpSJk_OW0j.K5 yS7FGapUq3nZ0LNUXOHo02bYut.7PfAOC11tGJzFeoJ7xwSwwHKVJcQ1b3P6oF9tZt90eDFBR0Jz yroS7nVa379KWQxfNN6Awukh_nzeIiLvzibeB8L6IDcVUtB0EGXLQ1CpVCGdHt6PXSBbDkcEz9kH H7K3w9JfonaqU5x08ZYlQyhiOHOKAKjElC3d0SJZjw5.u1r__XCp_rRm5RBYX3Kw9Ues4yz3aNMx vb1ywe8kRaAU4rnWAfrgYeDsxnq7I_t28zi8vYaWm_Z0jm4ChgZMYsCwhefP2XchKRSoyQ66dCKE hrvOu8vJe6G7O.KjHcKjzLvJ.CF_wG0BViiTEjnz2jGfB.9CpMLAiLZh8jIrxxXc35LdR596.jB3 fcJ.5_GLOHTfGNJJ_WQAJolu89E1VlX4NI.3xQGWKtaiqPrf_1j3LOhgIAhZgm.iJX29stdl.0vz KrurMeTT2K5wbsog7NhDtgQTgp6pgepTnzwaZe81nfCkDw1tKMUj9.YGJ2PeNhKq7vQuQ3E.CAXi dhsYs7xJjSbHX6DQ_dBfQB.LEFSUeHxgHwzcWnuCt2_w_cJbySkFtsrNnH1Jr9yON4UPPMPR7UeG oDqXXhWN1J0HF7_wGgs6eRtVBnQJ0gaBbRdVYN1tvH_SsvLcrtGf1fFI7t3B_B1LnVFSNCSOHyRP ikX3roMnWM9rBq9p4Ha18Ox_puGPFs9qxYvmpgjWLLeYYjgfZR8F2.tM1gmlnT33f44igW5u6bI5 o8eWyfFpWIPxoXZ53ZZ6xcOBhSRLJoBFHOubmAR9cg16jDLSgHdMgHqrESU4fYflc8_HoYLqEak8 ABU6PJ5X3wlKH9wEm7aCDz2hB1aRVJ6vVv2wdZbk2mt5HjHAQUy4pUHgoitQpYakqljtX2QCjtF_ Uf1fMlMB990j25svUiVdslyctx2CnbiR0kkkVmJb5oyXarIJjTPl9DSsElhONQUYCRTYWwMtk.p5 OkXIOHFNsG_fajdm26ymtUfnsX9sepijmJaHh2zkkO3sjn9MAwdbAgz1GHGd4JsExMaRbRzTGkr3 hKazKgLnE4ax5V03Q6tpqJuQVL9BTWsWA4suZECZ8Qri4nT3gUX0ckvuQIKiL874xKxtwXmx_5XH suiOAgt5xptHVeJeGNVCdCjbHMalUF7OYsPEaKnad_1A94ARCmk.2XQeswhfVZkxdhccckBD2tMD iMtcqHnidQgZU0LTowLywTaXjrz9c37_NP_9.pMpPmLBkpZOrAgMJimOy31S6vmF65VK1qdntjbe phe2u378FaD62YXoq5s3vwcavsxohDkfFxhrnThb2n20XzL.S6pk1Tj5y5r53HAPs0ECkR.GqLDy PIsdSD5VCF85Uh5WFi_L0J6ilFmSrniPvyZVw4PfFVjiZHk4yNQnOJgL1MgjYTs_gO6BlFdypygn 9pp4Ps4LzdKr3EXaeqUnqjTJmM163ZIDBG9mN9oIQfuLDtXxGMPkja4Y3gmhkVwX1wQe4l2S3g6v eOGGL2bZrz_8fG38SqvuDRgNNkRN5cNxUf2usTmx0BwTXNYp5iakenzpzNdgwkxBouVJ.2TeQtIu QWUt4LAk9RdHDUNBsLg_fWAb5SFa2OSmwzf9Fnh0j8ToNDMwElNKvNEBV9OP47n.jHMwHpSY8R6L wzXmYf9RAQm7eGuFRqxF1aR_Ob6mBBGpap0SyycO5iFrce7GO2FHzYbv0VW3ASCg4CWFpeQ4ve7c kG99f.A.8M0E.X0jDgpEXAKTygKXgBo.ksMCGNyZvcoXK9SWGT60IZ2E_USY37moVGt9mp35M6lk oF51lbLMI_SpszhrR84ms_R19QTsIUg23SOVu9Kk86vwK_kvtmQFwwZyCa8jrnu9om6BCtm3XmhV RHPSYSEKKoAzlaiSjgF4RcOiDyWu_GrZbQPPIzYAkLGccfnOVJWYYBjphTzJqnHNmu3vTPlmNN9D .Y3Ol_1d07wIDE9jVJ06Afyvjw5Qh63VCa.605XowIk0AlMALodpY8MwJarNNAEFY0vcuERCkGqD fRA-- X-Sonic-MF: X-Sonic-ID: f7bdf1f0-4f17-4259-b455-5e03658927b7 Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 01:03:33 +0000 Received: by hermes--production-gq1-6cf7749bc8-kkr9m (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID a4a676bf772f49a17ead82597b98981d; Thu, 23 Mar 2023 01:03:30 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Wed, 22 Mar 2023 18:03:19 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.84:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.84:from] X-Rspamd-Queue-Id: 4PhnCX0tctz46J3 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 22, 2023, at 16:17, Mark Millard wrote: > On Mar 22, 2023, at 15:39, Mark Millard wrote: >=20 >> On Mar 22, 2023, at 13:34, Mark Millard wrote: >>=20 >>> On Mar 22, 2023, at 12:40, George Mitchell = wrote: >>>=20 >>>> On 3/22/23 15:21, Mark Millard wrote: >>>>> George Mitchell wrote on >>>>> Date: Wed, 22 Mar 2023 17:36:39 UTC : >>>>> [...] >>>>>> Here are the very complicated instructions for reproducing the = problem: >>>>>> 1. Install and start misc/dnetc from ports. >>>>> Installing is likely easy, as likely would be building >>>>> with default options (if any). I know nothing about >>>>> starting misc/dnetc so that is research. (Possibly >>>>> trivial, although if it has alternatives to control >>>>> then I'd need to match that context too.) >>>>=20 >>>> service dnetc start >>>=20 >>> I built and installed misc/dnetc and got a binary >>> blob that clearly was not built in my environment: >>>=20 >>> # file /usr/local/distributed.net/dnetc >>> /usr/local/distributed.net/dnetc: ELF 64-bit LSB executable, x86-64, = version 1 (FreeBSD), statically linked, for FreeBSD 10.1 (1001515), = FreeBSD-style, stripped >>>=20 >>> Way older FreeBSD vintage than the locally available toolchains >>> would normally build. Some might be cautious about such a thing. >>>=20 >>> The man page reported that: >>>=20 >>> QUOTE >>> If you have never run the client before, it will initiate the = menu-driven >>> configuration. Save and quit when done, the configuration file = will be >>> saved in the same directory as the client. Now, simply restart the >>> client. =46rom that point on it will use the saved configuration. >>> END QUOTE >>>=20 >>> I've not seen what the configuration asks about yet. >>=20 >> I went through the configuration, basically just looking >> at it, other than providing an E-mail address. Then . . . >>=20 >> $ sudo service dnetc start >> Password: >> Cannot 'start' dnetc. Set dnetc_enable to YES in /etc/rc.conf or use = 'onestart' instead of 'start'. >>=20 >> $ sudo service dnetc onestart >>=20 >> I just let it run without any extra competing activity, other >> than I had my patched version of top running. It records and >> reports various maximum-observed (MaxObs) figures, here >> the load averages being relevant. >>=20 >> Top showed that dnetc started 32 processes, one per hardware >> thread. Mostly I saw: 100% nice and 0% idle. >>=20 >> Letting it run and then looking at the load averages (and >> their matching MaxObs figures) after something like 60+ min >> (not carefully timed: was doing other things) showed: >>=20 >> load averages: 31.97, 31.88, 31.66 MaxObs: 32.12, 31.97, 31.66 >>=20 >> (Note: The machine had been up for over 2.75 days before >> starting this and had not been building much of anything >> during that time.) >>=20 >> I've not yet experimented with having other, significant >> competing activity. >>=20 >>>>>> 2. Run "make buildworld". >>>>> So on the 32 hardware-thread (16 cores) amd64 machine that >>>>> I have access to, the test is to only have buildworld use >>>>> about one hardware thread, no matter what else is going on. >>>>> I never would have guessed that the steps would not involve >>>>> more like -j$(sysctl -n hw.ncpu) (so around -j32 in this >>>>> context). So it is good that you provided your note or >>>>> I'd not know if I'd done similarly or not when trying such. >>>>> [Note: -j1 and lack of -j are not strictly equivalent in >>>>> how make operates. As I remember, the distinction makes >>>>> a notable difference in the number of subprocesses created >>>>> directly by make (one per action "line" vs. one for the >>>>> whole block?). So even using -j1 might make a difference >>>>> vs. what you specified. I'd have to test to see.] >>>>=20 >>>> I am literally running "make buildworld" with no additional = options. >>>=20 >>> So required for repeating your results, but likely making >>> such results not be interesting relative to how I normally >>> deal with buildworld buildkernel and the likel, no matter >>> if there is other activity in an overlapping time frame or >>> not: my time preferences are too strong to wait for a single >>> hardware thread to do my normal builds, even with no >>> competing activity on the builder. >>>=20 >>>>>> Standard out conveniently reports how long it took (wall clock). >>>>> But nothing in your instructions indicate about how >>>>> to get an idea much progress dnetc made during the >>>>> various tests? [...] >>>>=20 >>>> Honestly, I've never worried about this part. But dnetc logs its >>>> progress in /usr/local/distributed.net/dnetc.txt, though not in = terms >>>> that are easy to relate to real-world progress. Oddly, when I run >>>> "make buildworld," I'm primarily interested in getting the world = built. >>>> Perhaps others feel differently. >>>=20 >>> Off topic for the specifics of the actual benchmark >>> that you run: >>>=20 >>> Then why not use of -jN ? In my context, any buildworld >>> using -j1 or no -j at all takes a huge amount of time >>> longer than letting it use all the hardware threads (or >>> so). (I've avoided having any I/O bound contexts for >>> such.) It does not take additional load on the system >>> for that to be true --including on the 4-core small arm >>> boards when I happen to buildworld on such (rare). >>>=20 >>>=20 >>>>> [...] >>>>> FYI: I've never built with and run the alternate >>>>> scheduler so if there is any appropriate background >>>>> for that that would not be obvious on finding basic >>>>> instructions, it would be appropriate to provide >>>>> such notes. >>>>> [...] >>>>=20 >>>> You have to build a new kernel, using a config file in which you = have >>>> replaced "options SCHED_ULE" with "options SCHED_4BSD". -- = George >>>=20 >>> Thanks for the notes. >>>=20 >>> I've not decided if I'll do anything with the binary >>> blob or not. >>=20 >=20 > FYI: >=20 > It is not your specific experiment, but I started my > "extra load" experimenst with . . . >=20 > I started a -j32 buildworld buildkernel with dnetc still > running. I'm generally seeing around 55% Active and 42% Note "Active": user, sorry. > nice, < 2% system (it was building libllvm at this point). > At that time: >=20 > load averages: 64.41, 60.52, 49.81 MaxObs: 64.47, 60.52, 49.81 >=20 Contrasting results for some obj-lib32 build activity: much more variety of User, nice, and system, including times with < 5% user, 90+% nice. But not typical overall. But lots of time roughly around 50%/50% or 35%/60%. There were times with 15+% system. Somewhat after buildkernel started: load averages: 69.15, 64.12, 58.72 MaxObs: 75.98, 64.12, 58.72 Harder to summarize, so overall timing reports from the buildworld and buildkernel stages. buildworld: -------------------------------------------------------------- ... World build completed on Wed Mar 22 16:37:57 PDT 2023 ... World built in 2615 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- buildkernel: -------------------------------------------------------------- ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 16:43:10 PDT = 2023 -------------------------------------------------------------- ... Kernel(s) GENERIC-NODBG built in 311 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- Afterwards: load averages: 36.08, 53.14, 55.79 MaxObs: 75.98, 65.77, 59.84 I then did (not all in the same window): $ sudo service dnetc onestop # rm -fr /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/ before another -j32 buildworld buildkernel (no dnetc). The reuslts for this were: buildworld: -------------------------------------------------------------- ... World build completed on Wed Mar 22 17:39:19 PDT 2023 ... World built in 1240 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- (compared to the 2615 for dnetc also in use) buildkernel: -------------------------------------------------------------- ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 17:41:17 PDT = 2023 -------------------------------------------------------------- ... Kernel(s) GENERIC-NODBG built in 118 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- (compared to the 311 for dnetc also in use) Experiments without -j32 will take a lot longer, even without dnetc in use. I'm not sure there will be such results today. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Mar 23 01:08:54 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhnKz45gHz40JHS for ; Thu, 23 Mar 2023 01:09:11 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-25.consmr.mail.gq1.yahoo.com (sonic312-25.consmr.mail.gq1.yahoo.com [98.137.69.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhnKy4xwBz47Hg for ; Thu, 23 Mar 2023 01:09:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=f8bdpUE1; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679533749; bh=Pe08wCyvKC0WAJi6IE/fDxKH85QuYHBRWyrJxazbIDc=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=f8bdpUE1dUTBB11W2inzWMWhimJ5cNa5v9sPBu6jLSfreLbJste0Ewz5HMzZenRzdYPcG6mb/siH1kk2w7LBgKsAydjjTsClLHjhap6SnsmczAXH6nzbjCNIhAMZJ0r9xrAS6uhs73GKHLpgBs2SinShpFo1FcdwWoSztUXG3cii2MS2tYB7bDOQOCH9Q992IAQSFHjlCGMk1KIYvvhkYhMqmpktlMP1e/a3iGE4iglA4y06evAZolFj+XBSxobPEAywR+aUGQyrTRhSZoVfuPZGfdlc590DlxORIDRnLM0x2RrzZN+R9IbRqqKLk/ZhXp8ieeudPaCxZq0FpAo51A== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679533749; bh=fVX4gmLVJdctRGbrtgXssZwqKIBUKFQheJEJwG3MlZf=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=M/Huo7zggk1n356i5SG7XlDAJ+ymvCZgyCAvD9tXyClbgts5Ow6FwC1OJvUzb5TXtKaH4Z0zyViO61l+s9PgbIjeCmydBZQz7ezWEvGTIVchJVEGm+MOKYlK6CnaN4emijOf2C904n3QdKnQOshLYaXqsk/iQN7qeNpQlva3Kcz/I+o0deiQSjN4D44m6C2562MLK+mMlzp5JWoEotDTkW2mFOhpP9RGf/iy3E1voyJxkFl8zqUbO/1hZNF/77nhzvMuTOuvUz9BmH+Nq/Qo+E6NTq1jTOntsSptNX92Ru2eDiQ2v+L8TVCOyPnkEuL/njgqpu/5693wS8Qvs4W9YA== X-YMail-OSG: ycR.xooVM1nPG4JheH1UnOC8Jrzmd57C.3.dBwFfIjp532Oo3Yj3.8FV29v3jhb UM_XDW1awwQ1IqiysOLNEZht5SJVTZRuhXlZjdQZRhsI6wgkZChDAnF2Exo3ylM9i_pO6NENU6iB gBK4zDrAH6AcLSmS7MGl63kycnlMPUtUCRO3WqefEv9MoWO.yKgptCswbSudYzfPnfrpdUfqZOSn a236.733IUNRjeeEZtSij7HLQWey.gDNMHqitQJniIhrYnSR8RzM12D01y.ksPSGItdUVdaSQ8zY ybSR7Y57zeCzbTuOJEdwXBcgFQm2WnnHGCF4kn5vi8m0yZJ6.omBJE2DO.JLPQ3C2zYJYldsfArE JTp_iTpCt_hfxYizukIpR6iSNXM1Ugsa0mY7UgBcPJ_U1kveNVaiEtRJfuhSF106aDhSM.euZXFo .kIVHltAs8PgmomNg_VqiV27wI3kJsXPwRqqDkOFKemWwhABJHZIdl0JA3Xdx9BrEQgqmT6I7UT2 oVMVuH8g62nhmytPzOztINW0S18YXL9s3wCJsxvOoZe7EG58rN7lpCvpo3uUUVOvBaz6mVjqPRb2 ozzdwtzhWqjco5Awd4T3KH9147cnGgmgIzEe6FnbYBdJnZZLmaq6wAueqrp5_8RbYaKLZt8GCN.J QMoVdarNIIJhMJFAUn7SqsNZCiJHO.xisZVGiJ4QCdHNq0tQwHlhsqbRu092YshIxQniBZILoMAg 4CSQSWakhw9zdRfI4rFmb_6TzaBEEsDNRXUcfo1xQ4TXfUbW2CKN62Pg1Ye_6G2VOx5yfCRCVtBp Kfcc7iTK7xT0Xqf9FqViolx2Fx02wr5tZiEIAz1XWMIgLd3wn50LGaaXTAs7Abr7uEq0NOk6YqVh W2bBiQHdKkQVYzJi5iumzrMzGfnpUTLZjE_UGuzGKHw6Uqbqj5lSJHBw9pjIOH5gPG_R9qaUXrKo MLPCIAsPC5m4FJYpcSTZdXroPD6k8ZSeRsGpRMp.ASFuCB3VSw68Dt6aqvK_UY4P8iS1L3Ejr493 xVttHOxyQUaIGKfYT6Fzta97JB0ExdNmx3dVYYRKOLUGQI9ggcN.YKwvZdM0N0TMdeXsz8ovLgr7 N0T3PmnibEQTtD.T833n4U_DTfWn33vmLmims6qNhap1TKbO_RTTUIMu3nuGT3fNajyO.zwqUR07 oITAGIEfz1BQ1hL5nSSyQFNVV_DMOKL5tMqIkrD5JXMdZ6CmhEr957NUIx15ZbIUDI.pyF.Lr0Eh UAwZlc7U6D3zcsx6RXWne2Ozg.Z3.kycFbzyfK2kY8DFlKqTJpGFfySYOC89xMvlwkp_pDUW_gLJ qPpLiAlrKjNTVNNccY4e2HHuq3x2cInemYv.3Dq67khSFIttWDEnp_C1Vq4AY1gyrkLd2gNg7nPu mabSLu5ZzO5RygObWvs9SDe_5g5ueET2JNeA0JTa5LjVVzdsgh_35106lSaYrT1pV0tjn16KbEQr 8DBDvxsk6OoZpLIlFrcdVG_qH.0H1U0APwCgMlf0y.wR.esZaRg75vBe4fNi6oLCGQajEOwqmUC_ fhWnzWYAbkdsBkHihgrTnRPvCesUVRPDjlj_sA6cN9zLbx_gMpi9znQrZbVv9cxME0KMb6Pryfdv TicaJnF.MkwukWpE.WLayiVW_s_NV3MM.0rPlBHM45UFOE58kOhZ2PoOFW82.b3yWDtf8vklnAnL ICEfp43byTiCsBz1XSgdgwfgtAU8mIlbBqrFyfdOzIJ6R3Jju741DsNq0JvWd8WwBcmNqBI9zz_F O.w5rUt4wj85BWSk_kpyCKvhNIji.8WwPldP4TeuVO02rtHsfs8EZRfiqJsaMyWBQ2wGskkgbRlF TNoIIMyjXt2czqC1aD4P0fCrIgtm_BRftcQnWRSYRCPhA6pX7jUZ8367aKhWW1VnJ6gUhjweoBsM 2rVsxZ2AcmkbYpZQFpZILdkDXqxYQo7xdb5Jjm4sNIGUNk0.QhYeD6DSzgEy6QHlmVHxIKzedYom tA9.2f3Q9aU58F_DDv3erOkROmsIKqqp_CZmEFEM7ReR70oy814YI17NILokORSU3xznpZgqt1Z. SiRBd639n4.NlPqbXj4k5MC_oWe.Wwq3WkigELTgZrI9yIFoJXXPug8QlRZJAoL5A02uWZ_LI7LG sk0sag0FuqmqJd0Le0Crcy9o0RTZAcUUgp9Rn9FMlxUsLe_6VarblLlQqRobagfjlWMhe4W1ppM5 RefMWUx7vAQJzFZ0if1nfkLjpBFgpYb4Iz0VH5BDkLKabXF.EJQasGyRf9s4tFcv456LSfpzTY1. W X-Sonic-MF: X-Sonic-ID: 8d934461-ad68-4e37-aa3d-13defac71845 Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 01:09:09 +0000 Received: by hermes--production-gq1-6cf7749bc8-c9j9s (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8994d531e2c676968f51d20f688215e9; Thu, 23 Mar 2023 01:09:04 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Wed, 22 Mar 2023 18:08:54 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <0196A767-5DC2-4DA3-BA5E-62A358CBCF64@yahoo.com> References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.206:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.206:from] X-Rspamd-Queue-Id: 4PhnKy4xwBz47Hg X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 22, 2023, at 18:03, Mark Millard wrote: > On Mar 22, 2023, at 16:17, Mark Millard wrote: >=20 >> On Mar 22, 2023, at 15:39, Mark Millard wrote: >>=20 >>> On Mar 22, 2023, at 13:34, Mark Millard wrote: >>>=20 >>>> On Mar 22, 2023, at 12:40, George Mitchell = wrote: >>>>=20 >>>>> On 3/22/23 15:21, Mark Millard wrote: >>>>>> George Mitchell wrote on >>>>>> Date: Wed, 22 Mar 2023 17:36:39 UTC : >>>>>> [...] >>>>>>> Here are the very complicated instructions for reproducing the = problem: >>>>>>> 1. Install and start misc/dnetc from ports. >>>>>> Installing is likely easy, as likely would be building >>>>>> with default options (if any). I know nothing about >>>>>> starting misc/dnetc so that is research. (Possibly >>>>>> trivial, although if it has alternatives to control >>>>>> then I'd need to match that context too.) >>>>>=20 >>>>> service dnetc start >>>>=20 >>>> I built and installed misc/dnetc and got a binary >>>> blob that clearly was not built in my environment: >>>>=20 >>>> # file /usr/local/distributed.net/dnetc >>>> /usr/local/distributed.net/dnetc: ELF 64-bit LSB executable, = x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 10.1 = (1001515), FreeBSD-style, stripped >>>>=20 >>>> Way older FreeBSD vintage than the locally available toolchains >>>> would normally build. Some might be cautious about such a thing. >>>>=20 >>>> The man page reported that: >>>>=20 >>>> QUOTE >>>> If you have never run the client before, it will initiate the = menu-driven >>>> configuration. Save and quit when done, the configuration file = will be >>>> saved in the same directory as the client. Now, simply restart the >>>> client. =46rom that point on it will use the saved configuration. >>>> END QUOTE >>>>=20 >>>> I've not seen what the configuration asks about yet. >>>=20 >>> I went through the configuration, basically just looking >>> at it, other than providing an E-mail address. Then . . . >>>=20 >>> $ sudo service dnetc start >>> Password: >>> Cannot 'start' dnetc. Set dnetc_enable to YES in /etc/rc.conf or use = 'onestart' instead of 'start'. >>>=20 >>> $ sudo service dnetc onestart >>>=20 >>> I just let it run without any extra competing activity, other >>> than I had my patched version of top running. It records and >>> reports various maximum-observed (MaxObs) figures, here >>> the load averages being relevant. >>>=20 >>> Top showed that dnetc started 32 processes, one per hardware >>> thread. Mostly I saw: 100% nice and 0% idle. >>>=20 >>> Letting it run and then looking at the load averages (and >>> their matching MaxObs figures) after something like 60+ min >>> (not carefully timed: was doing other things) showed: >>>=20 >>> load averages: 31.97, 31.88, 31.66 MaxObs: 32.12, 31.97, 31.66 >>>=20 >>> (Note: The machine had been up for over 2.75 days before >>> starting this and had not been building much of anything >>> during that time.) >>>=20 >>> I've not yet experimented with having other, significant >>> competing activity. >>>=20 >>>>>>> 2. Run "make buildworld". >>>>>> So on the 32 hardware-thread (16 cores) amd64 machine that >>>>>> I have access to, the test is to only have buildworld use >>>>>> about one hardware thread, no matter what else is going on. >>>>>> I never would have guessed that the steps would not involve >>>>>> more like -j$(sysctl -n hw.ncpu) (so around -j32 in this >>>>>> context). So it is good that you provided your note or >>>>>> I'd not know if I'd done similarly or not when trying such. >>>>>> [Note: -j1 and lack of -j are not strictly equivalent in >>>>>> how make operates. As I remember, the distinction makes >>>>>> a notable difference in the number of subprocesses created >>>>>> directly by make (one per action "line" vs. one for the >>>>>> whole block?). So even using -j1 might make a difference >>>>>> vs. what you specified. I'd have to test to see.] >>>>>=20 >>>>> I am literally running "make buildworld" with no additional = options. >>>>=20 >>>> So required for repeating your results, but likely making >>>> such results not be interesting relative to how I normally >>>> deal with buildworld buildkernel and the likel, no matter >>>> if there is other activity in an overlapping time frame or >>>> not: my time preferences are too strong to wait for a single >>>> hardware thread to do my normal builds, even with no >>>> competing activity on the builder. >>>>=20 >>>>>>> Standard out conveniently reports how long it took (wall clock). >>>>>> But nothing in your instructions indicate about how >>>>>> to get an idea much progress dnetc made during the >>>>>> various tests? [...] >>>>>=20 >>>>> Honestly, I've never worried about this part. But dnetc logs its >>>>> progress in /usr/local/distributed.net/dnetc.txt, though not in = terms >>>>> that are easy to relate to real-world progress. Oddly, when I run >>>>> "make buildworld," I'm primarily interested in getting the world = built. >>>>> Perhaps others feel differently. >>>>=20 >>>> Off topic for the specifics of the actual benchmark >>>> that you run: >>>>=20 >>>> Then why not use of -jN ? In my context, any buildworld >>>> using -j1 or no -j at all takes a huge amount of time >>>> longer than letting it use all the hardware threads (or >>>> so). (I've avoided having any I/O bound contexts for >>>> such.) It does not take additional load on the system >>>> for that to be true --including on the 4-core small arm >>>> boards when I happen to buildworld on such (rare). >>>>=20 >>>>=20 >>>>>> [...] >>>>>> FYI: I've never built with and run the alternate >>>>>> scheduler so if there is any appropriate background >>>>>> for that that would not be obvious on finding basic >>>>>> instructions, it would be appropriate to provide >>>>>> such notes. >>>>>> [...] >>>>>=20 >>>>> You have to build a new kernel, using a config file in which you = have >>>>> replaced "options SCHED_ULE" with "options SCHED_4BSD". -- = George >>>>=20 >>>> Thanks for the notes. >>>>=20 >>>> I've not decided if I'll do anything with the binary >>>> blob or not. >>>=20 >>=20 >> FYI: >>=20 >> It is not your specific experiment, but I started my >> "extra load" experimenst with . . . >>=20 >> I started a -j32 buildworld buildkernel with dnetc still >> running. I'm generally seeing around 55% Active and 42% >=20 > Note "Active": user, sorry. >=20 >> nice, < 2% system (it was building libllvm at this point). >> At that time: >>=20 >> load averages: 64.41, 60.52, 49.81 MaxObs: 64.47, 60.52, 49.81 >>=20 >=20 > Contrasting results for some obj-lib32 build activity: > much more variety of User, nice, and system, including > times with < 5% user, 90+% nice. But not typical overall. > But lots of time roughly around 50%/50% or 35%/60%. There > were times with 15+% system. >=20 > Somewhat after buildkernel started: >=20 > load averages: 69.15, 64.12, 58.72 MaxObs: 75.98, 64.12, 58.72 >=20 > Harder to summarize, so overall timing reports from the > buildworld and buildkernel stages. >=20 >=20 > buildworld: >=20 > -------------------------------------------------------------- > ... World build completed on Wed Mar 22 16:37:57 PDT 2023 > ... World built in 2615 seconds, ncpu: 32, make -j32 > -------------------------------------------------------------- >=20 >=20 > buildkernel: >=20 > -------------------------------------------------------------- > ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 16:43:10 = PDT 2023 > -------------------------------------------------------------- > ... Kernel(s) GENERIC-NODBG built in 311 seconds, ncpu: 32, make -j32 > -------------------------------------------------------------- >=20 > Afterwards: >=20 > load averages: 36.08, 53.14, 55.79 MaxObs: 75.98, 65.77, 59.84 >=20 >=20 > I then did (not all in the same window): >=20 > $ sudo service dnetc onestop > # rm -fr /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/ >=20 > before another -j32 buildworld buildkernel (no dnetc). The > reuslts for this were: >=20 >=20 > buildworld: >=20 > -------------------------------------------------------------- > ... World build completed on Wed Mar 22 17:39:19 PDT 2023 > ... World built in 1240 seconds, ncpu: 32, make -j32 > -------------------------------------------------------------- >=20 > (compared to the 2615 for dnetc also in use) >=20 >=20 > buildkernel: >=20 > -------------------------------------------------------------- > ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 17:41:17 = PDT 2023 > -------------------------------------------------------------- > ... Kernel(s) GENERIC-NODBG built in 118 seconds, ncpu: 32, make -j32 > -------------------------------------------------------------- >=20 > (compared to the 311 for dnetc also in use) I forgot to show the MaxObs load averages for the no-dnetc context: MaxObs: 39.77, 32.15, 25.75 > Experiments without -j32 will take a lot longer, even > without dnetc in use. I'm not sure there will be such > results today. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Mar 23 02:44:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhqS5754wz40PqX for ; Thu, 23 Mar 2023 02:44:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhqS45qQqz4Jp5 for ; Thu, 23 Mar 2023 02:44:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="Crknf/8b"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679539474; bh=JbEpmCgl8zYC032jZdGYKfLeTuYbtTuRj8c1NK61dvY=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Crknf/8bZsU04m3OcpoPapvFFYqgJsXlxY0JZvLydWR1mEe07WgMZIOVzKgTkUx0FPGaVK0AE2vrR+/XN5MMBSU6iVkbh0jxJSVcLdOhucCprLRqHciBdRhvFNy1SOvGKIS210N0HYYq7sxPbU+n3Pc9mCcZAVyW3HB5hlLszqfh7sejWRVthViAKowpL3UtFtDs2DRXd5CPBBk6YYs4AGC4LRxKddHCqdVqSwJ8g3Xlax2dbOGMG4x2qDAFf2ssV6E65TgzfGFaGuUPhYMyKeN6PyWS53rKb4PJHciKvSgnVLOkZwxtu076MvwaqmVh5g7QBBrqBmrkIj/T4wtWVQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679539474; bh=AhrrrEB9nzXzBTarAJAvNBl1q19KBdarSa0U1WvkCYs=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=uNU6ErwfNXJ4/0oeEBHgi+IPgwMPNjI3hiSP7qh/3Hy4El6Yhdn5uuLlMU2zXjf7OHagDFhJDrvJe1lrLoOts9H3okWxqbGdr6dww73Areh7ZzXluEcpmR115QMcx4tiy5OyTBvhvgoSYH7CqUYdAe+uEHYVmTENgfznE/lOx3vaKCy9wy2lpXZOjsVz44g+FmSTWsgxMZOSyj4bm8+g1E3e+3LIzTMG0gSpe6qBY16gE+onP8aIiZzqTH6CJwMpSwS0NaCVkWgOIIELd1tbdd7/Hlb8LV1U4AmEoX4KrH95W7b6VKdcDKQ44wkh0yr5Yig5goGtYucPFeznB6ibJw== X-YMail-OSG: sDEO51YVM1mA0PLPVyiGpj2Hn6Usod3C_6srxem57mIWMhxzX93t3.Uoyxg8O3P E6qBGLl8Tfj9jc8v0nezJYggt4Wqhiy7LIVIXIgmhDOf4JpZt2wbqphp9ckxm1rVxByUThmiYDS4 sezmmCxVh5d3ZIPKYtrDDCPxHl88KH5vKaaurP8pkw73HhBKNLM7D5n0aFwbpLYB0x9JtURF0LuH QfWDAKQR4Z3mvmzRdRn61jRbALAOouaa7FBgpEA9JShiD6GCUFdeimvBr_cV0Dy2.jbXUy_MMfJH C0z6xR55kCwWgLfNRIlOOh_9E_EJbWdqaO7Lx_E.bEy41GX9YO3SG8sB2o3y3t4D1GOR_wA2ENTR ev0Pk7TtZTRVR5LIItnw0_4QZWq4V5r7AzCbvcC_Wd1qdzTqIfwq7FbFJk2iml0oAmF9Td940rVp 5ylgzZV7w2UutmW_nlW1KgxU_M5IZ.lYj3q6YxlLCAIE78o5hZiwsu6bCM3qr8ukxiGypnoXWVhm 7hP9KsuCjAVpU.WofcVL_FR7zsasWsCuNEYSWO_iEQqqEkNRQvG6VIt2UA3aII1jNQeXiYFtiUeq 6WcF5s43DphHIodgjiibCB3o1omZs9Zm9dbIGdftrNP6ie4GjNXaC17g.ud4T3QD_0xoHcHKfOx0 ylQUmwWb0KUvtKkgxx6WMMxCsBcCE106E85GcPDTode2TKEB9C5jGZ4xO2PcAYx.dDKzn08AHAFo xy5rRxZJ0zwIeSFwxTHky_RUGTsYlLsJwIg6kZR8f0nrPsXUasBvLAd_pTJRgko4FR5Ey6z5w5x9 8Qodc7yZVOnHdBJ2m4sG6eDD_skpfEgs6w252hoL6oc4xTjLoB8VILlIN69qeF1rumB.z7INA0XN .lLX3h1fEW.avuK_RlpKeg.SQXL9ebLpWVKiZ9sIqGtX6TWgMbbaTVqEHWn6.rIEL4bfImAY.mbK jXGIbQX9teSlc1mYS32UDxuwpUAkAKBvqMEsbcmNls35e25DesLwcPpIlRJS8gGVZXuZ_nhzP.cr vYtcyrSlVHd_.1QjUGFLvtO2IQqll.HFHdXfWHOcdqjV6d5GATC_jGVXKStpb_fLV8d70K0pxA3h HH.v9bxeIUwUdeMxApOEyi29_9rQ9BKh49txsE.pjWLoG9Z1kIW_QJO3Bs5juZZH_I.LJKGM3QOc DX60ixROY3IZel4pLQ0WkYlgU_8fIu34_8rpeSEi4kAwbhsm2FBmxlrvMjUQjRtqmxAM6gQmD9fe XtucUrsZE1oLYhsI52D6z9_7.FqTeCSmvfRlvV9p7wRQ4G3P8945bA9uZgJnZT2N0VMlcrQGVjWN NlpL3fXyCrmEaHGoezpoVYbvgiVfA0sYXj7Pr8uHCVQKRBqRdQ2ruK_zj8bBWiM29Bxb8N6a4p_k Gl8bZMarFizIQcuT8z.YMmbtbVCX3At49HsbkHKk0QfrQhby7PfJqpczGgUHj9pt_6U1Opq8RC6G 9MetLP5va2DTrAsE6KTIu9mhqX5CVWHIR6O946cTkt0qu05B.obAXgC6qnrSDNqq4mqaaHKCoIEh gDI9pjzCeL71M4yyVmnrT4x0DcsKdObsJMbMPoFUP6MzDkoDqqly8xMLXchUYo.Azfb0_K_dsvNU thJdomKZeZ7XvD6OfCSZXylqo9SuYkuJSq_DUhzS6bmEFBhqUg.jnl05EBikNxMXaBjwM92WLNey EzYFfb9EGSZMsED6jgEoh8b93jVuHhuzidvSzSimrlwUTgIFVMU0a9VrR9Gz_bQIKB.0y_ys3YpU vcizEP3Mu1TiQLcS0oZnYWFDXv0AG1ppOJJUXyzkgdNdV2OVDmKT6Ku4d3KzbBpH14igzRo11wVW 2Nsng2IXtmTPQS2lUyTFnSiWntrNm5AfmTPrKnicbe193VFQ5mm5YBKiKoWOFWMrVg2mvop3VdKS EyGTkWqSWN4qaVFoqvjOAj42rTujJUvggSisOe6CjzM1MCdGLnwkZTTavhxQTgAsWuyT.AtYIVfA rBh2ZihYlp5dQo8DVWNBqtix2ZGerb9Oz7jMYhxeDsPQCErgc70I8UOhivKH8xoUZkXu.te31G0y .mNHyKFtvi5G9T9aZERoAQw_eulel4QbuZXyilt3gFuOUJatDibgQDUaOeJ95AW.WYt9dMl3bTak lDAvgLDlgkSkJY2mLb1ohL21GMQjOg.MEG4Ry_15uEiSBaMdNxa51_UlahcNvkcYiWywQc04zX68 3PCRyYnOj1p7htoPCoIJbNBGYlT3shC8OO9K77cZiAaF071cu.TONkJUzLmj5vNb3PCH5Ikrj0WB vnLE- X-Sonic-MF: X-Sonic-ID: 03bf1ad8-89b0-47d3-8c48-66c78c7ce8a6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 02:44:34 +0000 Received: by hermes--production-ne1-759c9b8c64-fztnz (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 5662090b4cb4eb6cb040a3096caeea55; Thu, 23 Mar 2023 02:44:31 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: <0196A767-5DC2-4DA3-BA5E-62A358CBCF64@yahoo.com> Date: Wed, 22 Mar 2023 19:44:19 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> <0196A767-5DC2-4DA3-BA5E-62A358CBCF64@yahoo.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.148:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.148:from] X-Rspamd-Queue-Id: 4PhqS45qQqz4Jp5 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 22, 2023, at 18:08, Mark Millard wrote: > On Mar 22, 2023, at 18:03, Mark Millard wrote: >=20 >> On Mar 22, 2023, at 16:17, Mark Millard wrote: >>=20 >>> On Mar 22, 2023, at 15:39, Mark Millard wrote: >>>=20 >>>> On Mar 22, 2023, at 13:34, Mark Millard wrote: >>>>=20 >>>>> On Mar 22, 2023, at 12:40, George Mitchell = wrote: >>>>>=20 >>>>>> On 3/22/23 15:21, Mark Millard wrote: >>>>>>> George Mitchell wrote on >>>>>>> Date: Wed, 22 Mar 2023 17:36:39 UTC : >>>>>>> [...] >>>>>>>> Here are the very complicated instructions for reproducing the = problem: >>>>>>>> 1. Install and start misc/dnetc from ports. >>>>>>> Installing is likely easy, as likely would be building >>>>>>> with default options (if any). I know nothing about >>>>>>> starting misc/dnetc so that is research. (Possibly >>>>>>> trivial, although if it has alternatives to control >>>>>>> then I'd need to match that context too.) >>>>>>=20 >>>>>> service dnetc start >>>>>=20 >>>>> I built and installed misc/dnetc and got a binary >>>>> blob that clearly was not built in my environment: >>>>>=20 >>>>> # file /usr/local/distributed.net/dnetc >>>>> /usr/local/distributed.net/dnetc: ELF 64-bit LSB executable, = x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 10.1 = (1001515), FreeBSD-style, stripped >>>>>=20 >>>>> Way older FreeBSD vintage than the locally available toolchains >>>>> would normally build. Some might be cautious about such a thing. >>>>>=20 >>>>> The man page reported that: >>>>>=20 >>>>> QUOTE >>>>> If you have never run the client before, it will initiate the = menu-driven >>>>> configuration. Save and quit when done, the configuration file = will be >>>>> saved in the same directory as the client. Now, simply restart the >>>>> client. =46rom that point on it will use the saved configuration. >>>>> END QUOTE >>>>>=20 >>>>> I've not seen what the configuration asks about yet. >>>>=20 >>>> I went through the configuration, basically just looking >>>> at it, other than providing an E-mail address. Then . . . >>>>=20 >>>> $ sudo service dnetc start >>>> Password: >>>> Cannot 'start' dnetc. Set dnetc_enable to YES in /etc/rc.conf or = use 'onestart' instead of 'start'. >>>>=20 >>>> $ sudo service dnetc onestart >>>>=20 >>>> I just let it run without any extra competing activity, other >>>> than I had my patched version of top running. It records and >>>> reports various maximum-observed (MaxObs) figures, here >>>> the load averages being relevant. >>>>=20 >>>> Top showed that dnetc started 32 processes, one per hardware >>>> thread. Mostly I saw: 100% nice and 0% idle. >>>>=20 >>>> Letting it run and then looking at the load averages (and >>>> their matching MaxObs figures) after something like 60+ min >>>> (not carefully timed: was doing other things) showed: >>>>=20 >>>> load averages: 31.97, 31.88, 31.66 MaxObs: 32.12, 31.97, = 31.66 >>>>=20 >>>> (Note: The machine had been up for over 2.75 days before >>>> starting this and had not been building much of anything >>>> during that time.) >>>>=20 >>>> I've not yet experimented with having other, significant >>>> competing activity. >>>>=20 >>>>>>>> 2. Run "make buildworld". >>>>>>> So on the 32 hardware-thread (16 cores) amd64 machine that >>>>>>> I have access to, the test is to only have buildworld use >>>>>>> about one hardware thread, no matter what else is going on. >>>>>>> I never would have guessed that the steps would not involve >>>>>>> more like -j$(sysctl -n hw.ncpu) (so around -j32 in this >>>>>>> context). So it is good that you provided your note or >>>>>>> I'd not know if I'd done similarly or not when trying such. >>>>>>> [Note: -j1 and lack of -j are not strictly equivalent in >>>>>>> how make operates. As I remember, the distinction makes >>>>>>> a notable difference in the number of subprocesses created >>>>>>> directly by make (one per action "line" vs. one for the >>>>>>> whole block?). So even using -j1 might make a difference >>>>>>> vs. what you specified. I'd have to test to see.] >>>>>>=20 >>>>>> I am literally running "make buildworld" with no additional = options. >>>>>=20 >>>>> So required for repeating your results, but likely making >>>>> such results not be interesting relative to how I normally >>>>> deal with buildworld buildkernel and the likel, no matter >>>>> if there is other activity in an overlapping time frame or >>>>> not: my time preferences are too strong to wait for a single >>>>> hardware thread to do my normal builds, even with no >>>>> competing activity on the builder. >>>>>=20 >>>>>>>> Standard out conveniently reports how long it took (wall = clock). >>>>>>> But nothing in your instructions indicate about how >>>>>>> to get an idea much progress dnetc made during the >>>>>>> various tests? [...] >>>>>>=20 >>>>>> Honestly, I've never worried about this part. But dnetc logs its >>>>>> progress in /usr/local/distributed.net/dnetc.txt, though not in = terms >>>>>> that are easy to relate to real-world progress. Oddly, when I = run >>>>>> "make buildworld," I'm primarily interested in getting the world = built. >>>>>> Perhaps others feel differently. >>>>>=20 >>>>> Off topic for the specifics of the actual benchmark >>>>> that you run: >>>>>=20 >>>>> Then why not use of -jN ? In my context, any buildworld >>>>> using -j1 or no -j at all takes a huge amount of time >>>>> longer than letting it use all the hardware threads (or >>>>> so). (I've avoided having any I/O bound contexts for >>>>> such.) It does not take additional load on the system >>>>> for that to be true --including on the 4-core small arm >>>>> boards when I happen to buildworld on such (rare). >>>>>=20 >>>>>=20 >>>>>>> [...] >>>>>>> FYI: I've never built with and run the alternate >>>>>>> scheduler so if there is any appropriate background >>>>>>> for that that would not be obvious on finding basic >>>>>>> instructions, it would be appropriate to provide >>>>>>> such notes. >>>>>>> [...] >>>>>>=20 >>>>>> You have to build a new kernel, using a config file in which you = have >>>>>> replaced "options SCHED_ULE" with "options SCHED_4BSD". -- = George >>>>>=20 >>>>> Thanks for the notes. >>>>>=20 >>>>> I've not decided if I'll do anything with the binary >>>>> blob or not. >>>>=20 >>>=20 >>> FYI: >>>=20 >>> It is not your specific experiment, but I started my >>> "extra load" experimenst with . . . >>>=20 >>> I started a -j32 buildworld buildkernel with dnetc still >>> running. I'm generally seeing around 55% Active and 42% >>=20 >> Note "Active": user, sorry. >>=20 >>> nice, < 2% system (it was building libllvm at this point). >>> At that time: >>>=20 >>> load averages: 64.41, 60.52, 49.81 MaxObs: 64.47, 60.52, 49.81 >>>=20 >>=20 >> Contrasting results for some obj-lib32 build activity: >> much more variety of User, nice, and system, including >> times with < 5% user, 90+% nice. But not typical overall. >> But lots of time roughly around 50%/50% or 35%/60%. There >> were times with 15+% system. >>=20 >> Somewhat after buildkernel started: >>=20 >> load averages: 69.15, 64.12, 58.72 MaxObs: 75.98, 64.12, 58.72 >>=20 >> Harder to summarize, so overall timing reports from the >> buildworld and buildkernel stages. >>=20 >>=20 >> buildworld: >>=20 >> -------------------------------------------------------------- >> ... World build completed on Wed Mar 22 16:37:57 PDT 2023 >> ... World built in 2615 seconds, ncpu: 32, make -j32 >> -------------------------------------------------------------- >>=20 >>=20 >> buildkernel: >>=20 >> -------------------------------------------------------------- >> ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 16:43:10 = PDT 2023 >> -------------------------------------------------------------- >> ... Kernel(s) GENERIC-NODBG built in 311 seconds, ncpu: 32, make = -j32 >> -------------------------------------------------------------- >>=20 >> Afterwards: >>=20 >> load averages: 36.08, 53.14, 55.79 MaxObs: 75.98, 65.77, 59.84 >>=20 >>=20 >> I then did (not all in the same window): >>=20 >> $ sudo service dnetc onestop >> # rm -fr /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/ >>=20 >> before another -j32 buildworld buildkernel (no dnetc). The >> reuslts for this were: >>=20 >>=20 >> buildworld: >>=20 >> -------------------------------------------------------------- >> ... World build completed on Wed Mar 22 17:39:19 PDT 2023 >> ... World built in 1240 seconds, ncpu: 32, make -j32 >> -------------------------------------------------------------- >>=20 >> (compared to the 2615 for dnetc also in use) >>=20 >>=20 >> buildkernel: >>=20 >> -------------------------------------------------------------- >> ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 17:41:17 = PDT 2023 >> -------------------------------------------------------------- >> ... Kernel(s) GENERIC-NODBG built in 118 seconds, ncpu: 32, make = -j32 >> -------------------------------------------------------------- >>=20 >> (compared to the 311 for dnetc also in use) >=20 > I forgot to show the MaxObs load averages for the no-dnetc > context: >=20 > MaxObs: 39.77, 32.15, 25.75 >=20 >> Experiments without -j32 will take a lot longer, even >> without dnetc in use. I'm not sure there will be such >> results today. >>=20 >=20 I decided to do some more of the less time consuming testing. SCHED_4BSD, no dnetc, -j32 buildworld buildkernel : buildworld: -------------------------------------------------------------- ... World build completed on Wed Mar 22 19:16:35 PDT 2023 ... World built in 1235 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- (compared to 1240 for SCHED_ULE) So: no significant difference. buildkernel (SCHED_4BSD building a SCHED_4BSD): -------------------------------------------------------------- ... Kernel build for GENERIC-NODBG-SCHED_4BSD completed on Wed Mar 22 = 19:18:34 PDT 2023 -------------------------------------------------------------- ... Kernel(s) GENERIC-NODBG-SCHED_4BSD built in 119 seconds, ncpu: 32, = make -j32 -------------------------------------------------------------- (compared to 118 for SCHED_ULE building a SCHED_ULE) So: no significant difference. I'll try it with dnetc also active. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Mar 23 03:15:21 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Phr7c351tz40S4W for ; Thu, 23 Mar 2023 03:15:24 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oa1-x29.google.com (mail-oa1-x29.google.com [IPv6:2001:4860:4864:20::29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Phr7b60fFz4Ml8; Thu, 23 Mar 2023 03:15:23 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=MbWOFzLs; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2001:4860:4864:20::29 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-oa1-x29.google.com with SMTP id 586e51a60fabf-17ab3a48158so21463378fac.1; Wed, 22 Mar 2023 20:15:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679541322; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=vxabSIv1feE/uHyXSxunztOBdeVrF7QhESji+QEHEyc=; b=MbWOFzLswxtXvN5cazQWu26ozSfuhFnYFx3WCKBWNakJl7MkgvhahB5S2vIsy6uzdi 2rbdQlnf9AxpPN8+NODN5qS5bwH8Zw8Xz7BRU8xkP4ollQ+fMTN+VXIHViERJXwtsM0j m58IoMoArfJo7niKU3VnBNn9WKSAFIk4XXLAWCRQnU3hOIDu0962n9WnviYyO7Kl6oHK pbnzkvruU+goUjkRTITc14HYVC7xHrfPsseh3a2YgfbPsb3hORv0GcB/3RDHC47MT55D jlqAWfVBo6yIfRwNf/m3BKjcLWvNKX3Yqmv/EFYhxTmwUQ/I3bEqqarshQ0ws5CxwwCq CHFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679541322; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vxabSIv1feE/uHyXSxunztOBdeVrF7QhESji+QEHEyc=; b=LflCFZBSamQxDOVYr4/Bdw8Ef8Jzaknp2xGjh4MnczuGXWjULaTRJZKvBqb6ORIrZU Hnyuuq4isMtlZNjqMadp1B6Xodki8zzwIkHJsghpdbwsPEL7YM9gePSm/kJRgUGvpSM1 7s/Hn5W0d5xo0eAJKfOPUtpmMSvph1PjGmTcawfWEgNlCh+QP5Z+ffTYcbi7mSvpWNtF LooPMG4txRfMpZIOWUD+yHbvytYMoRKzblPEpKPtRqtrWrliEXXfPUfhpWZdauRUiOiu OhpxblZYBHzVwyi42Vs9vh6q9Z42/+KPoHyj6wbtm9kNJvPWw4hyfGcoEbrDzb2wLfWi QMFw== X-Gm-Message-State: AAQBX9fOpdleb/wYpeOoSY4W2EqVmPcDNi9RJ2JpMOJ20OoiYdHBO3bS j7LA82UrzhBhvqpP87cZnYJa4pBNfHIJRwhenvMVWVVC X-Google-Smtp-Source: AK7set/bY9pXMqPfPqbWhKzrFdau6NLfxZTVhxzwNs0zpt5cl5a54YT6TkyEJHwcR6isFqXQPvf+0iVxVjmT8/eMO3g= X-Received: by 2002:a05:687c:2249:b0:177:b393:4009 with SMTP id yu9-20020a05687c224900b00177b3934009mr693855oab.4.1679541322077; Wed, 22 Mar 2023 20:15:22 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1911:0:b0:49c:b071:b1e3 with HTTP; Wed, 22 Mar 2023 20:15:21 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Thu, 23 Mar 2023 04:15:21 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: sgk@troutmask.apl.washington.edu Cc: Matthias Andree , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2001:4860:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2001:4860:4864:20::29:from]; ASN(0.00)[asn:15169, ipnet:2001:4860:4864::/48, country:US]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Phr7b60fFz4Ml8 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/22/23, Mateusz Guzik wrote: > On 3/22/23, Steve Kargl wrote: >> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >>> >>> Yes, there are reports that FreeBSD is not responsive by default - but >>> this >>> may make it get overall better throughput at the expense of >>> responsiveness, >>> because it might be doing fewer context switches. So just complaining >>> about >>> a longer buildworld without seeing how much dnetc did in the same >>> wallclock >>> time period is useless. Periodic rant's don't fix this lack of >>> information. >>> >> >> I reported the issue with ULE some 15 to 20 years ago. >> I gave up reporting the issue. The individuals with the >> requisite skills to hack on ULE did not; and yes, I lack >> those skills. The path of least resistance is to use >> 4BSD. >> >> % cat a.f90 >> ! >> ! Silly numerically intensive computation. >> ! >> program foo >> implicit none >> integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) >> integer i >> real(dp) x >> real(dp), allocatable :: a(:,:), b(:,:), c(:,:) >> call random_init(.true., .true.) >> allocate(a(n,n), b(n,n)) >> do i = 1, m >> call random_number(a) >> call random_number(b) >> c = matmul(a,b) >> x = sum(c) >> if (x < 0) stop 'Whoops' >> end do >> end program foo >> % gfortran11 -o z -O3 -march=native a.f90 >> % time ./z >> 42.16 real 42.04 user 0.09 sys >> % cat foo >> #! /bin/csh >> # >> # Launch NCPU+1 images with a 1 second delay >> # >> foreach i (1 2 3 4 5 6 7 8 9) >> ./z & >> sleep 1 >> end >> % ./foo >> >> In another xterm, you can watch the 9 images. >> >> % top >> st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 >> 11:43:01 >> 74 processes: 10 running, 64 sleeping >> CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle >> Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G >> Free >> Swap: 16G Total, 16G Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU >> COMMAND >> 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z >> 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z >> 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z >> 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z >> 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z >> 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z >> 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z >> 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z >> 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z >> >> With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's exit >> after 55-ish seconds. If you try this experiment on ULE, you'll get >> NCPU-1 >> ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as the >> two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, >> this was/is due to ULE's cpu affinity where processes never migrate to >> another cpu. Admittedly, this was several years ago. Maybe ULE has >> gotten better, but George's rant seems to suggest otherwise. >> > > While I have not tried openmpi yet, I can confirm there is a problem > here, but the situtation is not as clear cut as one might think. > > sched_4bsd *round robins* all workers across all CPUs, which comes at > a performance *hit* compared to ule if number of workers is <= CPU > count -- here ule maintains affinity, avoiding cache busting. If you > slap in $cpu_count + 1 workers, 4bsd keeps the round robin equally > penalizing everyone, while ule mostly penalizes a select victim. By > the end of it, you get lower total cpu time spent, but higher total > real time. See below for an example. > > Two massive problems with 4bsd, apart from mandatory round robin which > also happens to help by accident: > 1. it has one *global* lock, meaning the scheduler itself just does > not scale and this is visible at modest contemporary scales > 2. it does not understand topology -- no accounting done for ht nor > numa, but i concede the latter wont be a factor for most people > > That said, ULE definitely has performance bugs which need to be fixed. > At least for the case below, 4BSD just "lucked" into sucking less > simply because it is so primitive. > > I wrote a cpu burning program (memset 1 MB in a loop, with enough > iterations to take ~20 seconds on its own). > > I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 distinct > cores. > > The test runs *9* workers, here is a sample run: > > 4bsd: > 23.18 real 20.81 user 0.00 sys > 23.26 real 20.81 user 0.00 sys > 23.30 real 20.81 user 0.00 sys > 23.34 real 20.82 user 0.00 sys > 23.41 real 20.81 user 0.00 sys > 23.41 real 20.80 user 0.00 sys > 23.42 real 20.80 user 0.00 sys > 23.53 real 20.81 user 0.00 sys > 23.60 real 20.80 user 0.00 sys > 187.31s user 0.02s system 793% cpu 23.606 total > > ule: > 20.67 real 20.04 user 0.00 sys > 20.97 real 20.00 user 0.00 sys > 21.45 real 20.29 user 0.00 sys > 21.51 real 20.22 user 0.00 sys > 22.77 real 20.04 user 0.00 sys > 22.78 real 20.26 user 0.00 sys > 23.42 real 20.04 user 0.00 sys > 24.07 real 20.30 user 0.00 sys > 24.46 real 20.16 user 0.00 sys > 181.41s user 0.07s system 741% cpu 24.465 total > > It reliably uses 187s user time on 4BSD and 181s on ULE. At the same > time it also reliably has massive time imblance between > fastest/slowest in terms of total real time between workers *and* ULE > reliably uses more total real time to finish the entire thing. > > iow this is a tradeoff, but most likely a bad one > So I also ran the following setup: 8 core vm doing -j 8 buildkernel, while 8 nice -n 20 processes are cpu-bound. After the build ends workers report how many ops they did in that time. ule is way off the reservation here. unimpeded build takes: 441.39 real 3135.63 user 266.92, similar on both schedulers with cpu hoggers: 4bsd: 657.22 real 3152.02 user 253.30 sys [+49%] ule: 4427.69 real 3225.33 user 194.86 sys [+903%] ule spends less time in the kernel, but the time blows up -- over 10 x the base line. this is clearly a total non-starter. full stats: 4bsd: hogger pid/ops 58315: 5546013 58322: 5557294 58321: 5545052 58313: 5546347 58318: 5537874 58317: 5555303 58323: 5545116 58320: 5548530 runtimes: 657.23 real 230.02 user 0.02 sys 657.23 real 229.83 user 0.00 sys 657.23 real 230.50 user 0.00 sys 657.23 real 230.53 user 0.00 sys 657.23 real 230.14 user 0.01 sys 657.23 real 230.19 user 0.00 sys 657.23 real 230.09 user 0.00 sys 657.23 real 230.10 user 0.03 sys kernel build: 657.22 real 3152.02 user 253.30 sys ule: hogger pid/ops 77794: 95916836 77792: 95909794 77789: 96454064 77796: 95813778 77791: 96728121 77795: 96678291 77798: 97258060 77797: 96347984 4427.70 real 4001.94 user 0.10 sys 4427.70 real 3980.68 user 0.16 sys 4427.70 real 3973.96 user 0.10 sys 4427.70 real 3980.11 user 0.13 sys 4427.70 real 4012.32 user 0.07 sys 4427.70 real 4008.79 user 0.12 sys 4427.70 real 4034.77 user 0.09 sys 4427.70 real 3995.40 user 0.08 sys kernel build: 4427.69 real 3225.33 user 194.86 sys -- Mateusz Guzik From nobody Thu Mar 23 04:09:22 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhsLG3dJmz40VhH for ; Thu, 23 Mar 2023 04:09:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhsLF1Rwqz3Brq for ; Thu, 23 Mar 2023 04:09:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=IIu7GuDm; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679544578; bh=pB3PRf8WWhiNLcCu+Ix0rJK8MB9kjPIL4BWbyIxddQA=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=IIu7GuDmyeicPGWrkNQADuvmlM+fQmx1UkjDdiHu1zbTivLgRbY8P2wQBH7LIjyC+9bSlvp2UIMAX3aVbJNSVZwAao7QR3jcr/yYLQlWWcoYYUaHEw6a8U67n3xQgRr8YqyqD8rVk1KnVEqigCliNMrrAr1KS/AzowUgOXqiPxA61yp/NRo3fSG7T2QlDlzjCkolo/QXQkGUIE/6XuODIKqteuPvsTFAGsWDuYbDejhmgVX2hPKEUGel0fneLjjUbQGnkDhZSL/dR4vhAn0teBPQyHllthFkPEkw0A0gHuRObqq1IpalrYkz9/WYuCK3fxuKKIICk+ZLkvsnU+ZXgA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679544578; bh=c73l7JdaCsaM10SQ2TgnRM5llYog5DKajFB0Ozl6ad6=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=gsFt7Qr76wFo8ejjjkY/8NF+Pw6xrgt/3A+BWfiCruA5pRwU2dpi+z9UnIK/uLoT3aG4tUsJZNnOGp8u6IWJco3pYVnWGIO6Y9eXk8MKBMpDeOT5WxWbQK9mga4hfids3dRmHZCLDy1m63qYoQyY3Fc8slffZ16xQNDIoyBfRQXrZ6IRQUKQZNTdmqVTHm1gi+6vceZ4d52C+7TaP4yZvmSQQ4RvbklIjoJnw+RpT+RiSoDbbLlZ4u0COm+SKg6YassljCQjH0fFkVozVkHkl40QwAX8N2FVccYBlJh+wDGqhrBPiIOl9NH98FvcsNy2JYuuau/nH69rKKxrUOQrow== X-YMail-OSG: 3Hziu68VM1nqL7ueyD3vsI4TB2gv52BioDOWZbyT3KIZRECyJ3eN.n4QH7naEdp uKHF08I438xzYo66iiOgrBN_YKUrL7zV1calPTurcDNVkLNqsZxPFfrEI4wgFi78_nyCxdXq3_KB M4A5ujy_uWZrEYv4NbDbsFzIXQuhpr00rTrg.gSJ3csCzTqOpHpGUTPO3POVFUVqUgWeWgg9ln6Q RTx_odQmscaHswmZXnCD7F1LDnHODUl3JLwo5kRWOArQsSV7xCcC7hnTnG.97KI.kRJ8j8PNxPIl NjWf3TxPfyC3B3oNc67gdXQSxROVcVV8cTQ1fHsKW4izQKKm47Q.Sb4NGPZNNZsvWRrPAtQt26C2 IfdoYmAI1FcoyaxJtjsJzaDfN5F9pcAL4d2yh7FEftpXd5FhTkGCkB0UlYCM48pJAJ2Oyd1lTj6b FVNtQuD7xxG4WfR3pafhbStWn7525Jm5e_KdWzKAf0Do3vLRpekFKDWfTRCvIrCtveuoJf7BLtfZ if8C8.CM_4j9hTnaYfOqwc8Gu.JoFSven4Q52EvbT3rXon06xI1U46BqVorbVa2.fjmmcDInHgv3 aJnndpGjNrRr68nkfR717Jf4ik.yJqkj6Y2JBWaO0n_mHsbQEJsJaqtH7K0tA69lkqDKMjBH.H4U ni0s72t2cSSadiAHZwMhYugHOBNHN4NpCVEDi46XwOcCEHw3EDo0lZ0QT5FQ9Tw8D5jAzldfkDzM hzxcMHOXJqYIC1yuBp86ILOThYq4SNlEQXGSM_LQtnjZXNFuK14SCNs6DuwEinkJa.Ty4lT5pf3b xSjJD6A4cR7_ndobYSCqrJ_CA8M5FPeykzwtEnnvIyy6Jq3274iW5IgcIXvRL1QFn9FciTr3p2bk K4LwsJjkKQ8CW6fZhOMzgCa35siJKrSHt_UfLY3sCCP.zop.gm4kxBBazV2pELLDn6CCSAEhQ6aB ZKlK4Fg_4FYTfKTh4apMeYhSYvBVT1X3G5.Qy7C2BZ.DhJz.is3tLNV9o.zKx7GDpkxEU.fbxmLr QkN9yr6GsphSDQ5SEn.93By1rqVUrLMAQkBu0_LW9ojor4xB04x9Gc9K3FJR84bIhekXhljv8mbG vOWVOJWJe5ct7S46umkxRyhZgkZI_IRSi_zHr3ShBQSG3KSBRq2XvbIRTZ6yDjR3ZHH12nBG0jsQ hJWMu55Pim9ZY3vwsnTabYuk38IJOOea9MHts_tSoOu0bK773x00vRWH4NGGCv3NkEEzQ6zHCJQz hg0W4bnkqUJ9O7hO7y4YtH01USz8u.BTrEbBaQfsOrAGy5fawn.ZTqIZ111HqTfr.2ONlexDOSpP K19VH66gfdHRZl6vbmkI4iWLPJPHJ8KLznNRimGfCFkMxB71gZbyTfQARQPmgB9QDRmb8.lg5M_j Bp19oqSgzRIMnzekLID0pz2d8CmS2FLunfW54euz0ccw_l5ni8.scAqm17PF3TSgYEV.wqn6D0kH CqPdHHh.dHdrqbrsrOiXiHWVoHmK_FtiXVYgkYHxDmmbQ1J__dYAeYde_z0OfiLijKwTz_e2TM4P 5vQkrvWPPL0KCQD3t3.JJcscWzvg7LDhLRjtAi4GeRP23qa_7zJkV.URwnDMlrPHuUlQzwW30gxQ 0a21EiU6DTdwA0gjOJujGaqOOEoNQofPabZiK57.BYRckLi3Utx_LFfwTEym_3blhBJjaFDYdnJ_ fTo_loC56Hft4K2btBnn0Q42VAyvuFcQpClAMMLXWUDuaWmewxH9.Cs6QIsI_QRRsgAQQ5.XwjOP QUAW1b9Bg_48qd3roRerrEbSA.HYW23FQD_FOZwinmLp1WWv0KEjEoI0gr8XBhSgTu0Ib0rIW3wF cnmVBOf0NIqC29tQnRxI8.wX.UUDeoiQPZnamkkH2jnGU.LE_kNK6yED9t9RepuavsQgE4kPXSVn 03VctIqzhuRDdpXn3Ja0C4Oegkm4Jq5TzjOO7HzfqkErvGXxVqLgBISlBIvhYR0xvW7oc7_PRDnq mBO8OMs0EoxY90epE1Z5t3_sdzCyus9bT7x8W4isyw0WukKeZXUdZzZHDoVNmTPFVc8Bwu24FbQJ 3.KhXUvE5aV2Iz7ge6..mU.C8vIUbLV.XI.WpiyCl4LgpWNrrmgaC0Ce6E_c44_QGSsaS4Tr8krX 8KPlr_9IYi9l.NjE.VJIcvENc8ylSxOwLxP4GdIGLNIVlU96F7DB6qhFxWpW_mNOvgJmwsuzd8BC _zejEt929YoUALUX1hls4Ygk1ToVQUcPsimQtVw99WZlDoUSXDEWrw3kzre8kDaG4op5e4GNG_SB f X-Sonic-MF: X-Sonic-ID: 4e028b8f-853c-42ec-a813-dcc0da67ecb9 Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 04:09:38 +0000 Received: by hermes--production-bf1-777648578f-lk5ld (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID c140b7de6075f5be5081a3e6c61a88fe; Thu, 23 Mar 2023 04:09:34 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Wed, 22 Mar 2023 21:09:22 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <6822D3CE-789E-4FB0-BB62-BEEEA65DB72F@yahoo.com> References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> <0196A767-5DC2-4DA3-BA5E-62A358CBCF64@yahoo.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.84:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.84:from] X-Rspamd-Queue-Id: 4PhsLF1Rwqz3Brq X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N [I added a -j32 buildworld buildkernel with SCHED_4BSD and dnetc-in-use comparison, to the other ThreadRipper 1950X examples. SCHED_4BSD does take notably less time than SCHED_ULE when dnetc is also active: still a good match to the simple round-robin for this building activity. I will note that the 1950X UEFI/firmware is not configured present itself as NUMA but the FreeBSD kernels in use are NUMA capable as built.] On Mar 22, 2023, at 19:44, Mark Millard wrote: > On Mar 22, 2023, at 18:08, Mark Millard wrote: >=20 >> On Mar 22, 2023, at 18:03, Mark Millard wrote: >>=20 >>> On Mar 22, 2023, at 16:17, Mark Millard wrote: >>>=20 >>>> On Mar 22, 2023, at 15:39, Mark Millard wrote: >>>>=20 >>>>> On Mar 22, 2023, at 13:34, Mark Millard wrote: >>>>>=20 >>>>>> On Mar 22, 2023, at 12:40, George Mitchell = wrote: >>>>>>=20 >>>>>>> On 3/22/23 15:21, Mark Millard wrote: >>>>>>>> George Mitchell wrote on >>>>>>>> Date: Wed, 22 Mar 2023 17:36:39 UTC : >>>>>>>> [...] >>>>>>>>> Here are the very complicated instructions for reproducing the = problem: >>>>>>>>> 1. Install and start misc/dnetc from ports. >>>>>>>> Installing is likely easy, as likely would be building >>>>>>>> with default options (if any). I know nothing about >>>>>>>> starting misc/dnetc so that is research. (Possibly >>>>>>>> trivial, although if it has alternatives to control >>>>>>>> then I'd need to match that context too.) >>>>>>>=20 >>>>>>> service dnetc start >>>>>>=20 >>>>>> I built and installed misc/dnetc and got a binary >>>>>> blob that clearly was not built in my environment: >>>>>>=20 >>>>>> # file /usr/local/distributed.net/dnetc >>>>>> /usr/local/distributed.net/dnetc: ELF 64-bit LSB executable, = x86-64, version 1 (FreeBSD), statically linked, for FreeBSD 10.1 = (1001515), FreeBSD-style, stripped >>>>>>=20 >>>>>> Way older FreeBSD vintage than the locally available toolchains >>>>>> would normally build. Some might be cautious about such a thing. >>>>>>=20 >>>>>> The man page reported that: >>>>>>=20 >>>>>> QUOTE >>>>>> If you have never run the client before, it will initiate the = menu-driven >>>>>> configuration. Save and quit when done, the configuration file = will be >>>>>> saved in the same directory as the client. Now, simply restart = the >>>>>> client. =46rom that point on it will use the saved configuration. >>>>>> END QUOTE >>>>>>=20 >>>>>> I've not seen what the configuration asks about yet. >>>>>=20 >>>>> I went through the configuration, basically just looking >>>>> at it, other than providing an E-mail address. Then . . . >>>>>=20 >>>>> $ sudo service dnetc start >>>>> Password: >>>>> Cannot 'start' dnetc. Set dnetc_enable to YES in /etc/rc.conf or = use 'onestart' instead of 'start'. >>>>>=20 >>>>> $ sudo service dnetc onestart >>>>>=20 >>>>> I just let it run without any extra competing activity, other >>>>> than I had my patched version of top running. It records and >>>>> reports various maximum-observed (MaxObs) figures, here >>>>> the load averages being relevant. >>>>>=20 >>>>> Top showed that dnetc started 32 processes, one per hardware >>>>> thread. Mostly I saw: 100% nice and 0% idle. >>>>>=20 >>>>> Letting it run and then looking at the load averages (and >>>>> their matching MaxObs figures) after something like 60+ min >>>>> (not carefully timed: was doing other things) showed: >>>>>=20 >>>>> load averages: 31.97, 31.88, 31.66 MaxObs: 32.12, 31.97, = 31.66 >>>>>=20 >>>>> (Note: The machine had been up for over 2.75 days before >>>>> starting this and had not been building much of anything >>>>> during that time.) >>>>>=20 >>>>> I've not yet experimented with having other, significant >>>>> competing activity. >>>>>=20 >>>>>>>>> 2. Run "make buildworld". >>>>>>>> So on the 32 hardware-thread (16 cores) amd64 machine that >>>>>>>> I have access to, the test is to only have buildworld use >>>>>>>> about one hardware thread, no matter what else is going on. >>>>>>>> I never would have guessed that the steps would not involve >>>>>>>> more like -j$(sysctl -n hw.ncpu) (so around -j32 in this >>>>>>>> context). So it is good that you provided your note or >>>>>>>> I'd not know if I'd done similarly or not when trying such. >>>>>>>> [Note: -j1 and lack of -j are not strictly equivalent in >>>>>>>> how make operates. As I remember, the distinction makes >>>>>>>> a notable difference in the number of subprocesses created >>>>>>>> directly by make (one per action "line" vs. one for the >>>>>>>> whole block?). So even using -j1 might make a difference >>>>>>>> vs. what you specified. I'd have to test to see.] >>>>>>>=20 >>>>>>> I am literally running "make buildworld" with no additional = options. >>>>>>=20 >>>>>> So required for repeating your results, but likely making >>>>>> such results not be interesting relative to how I normally >>>>>> deal with buildworld buildkernel and the likel, no matter >>>>>> if there is other activity in an overlapping time frame or >>>>>> not: my time preferences are too strong to wait for a single >>>>>> hardware thread to do my normal builds, even with no >>>>>> competing activity on the builder. >>>>>>=20 >>>>>>>>> Standard out conveniently reports how long it took (wall = clock). >>>>>>>> But nothing in your instructions indicate about how >>>>>>>> to get an idea much progress dnetc made during the >>>>>>>> various tests? [...] >>>>>>>=20 >>>>>>> Honestly, I've never worried about this part. But dnetc logs = its >>>>>>> progress in /usr/local/distributed.net/dnetc.txt, though not in = terms >>>>>>> that are easy to relate to real-world progress. Oddly, when I = run >>>>>>> "make buildworld," I'm primarily interested in getting the world = built. >>>>>>> Perhaps others feel differently. >>>>>>=20 >>>>>> Off topic for the specifics of the actual benchmark >>>>>> that you run: >>>>>>=20 >>>>>> Then why not use of -jN ? In my context, any buildworld >>>>>> using -j1 or no -j at all takes a huge amount of time >>>>>> longer than letting it use all the hardware threads (or >>>>>> so). (I've avoided having any I/O bound contexts for >>>>>> such.) It does not take additional load on the system >>>>>> for that to be true --including on the 4-core small arm >>>>>> boards when I happen to buildworld on such (rare). >>>>>>=20 >>>>>>=20 >>>>>>>> [...] >>>>>>>> FYI: I've never built with and run the alternate >>>>>>>> scheduler so if there is any appropriate background >>>>>>>> for that that would not be obvious on finding basic >>>>>>>> instructions, it would be appropriate to provide >>>>>>>> such notes. >>>>>>>> [...] >>>>>>>=20 >>>>>>> You have to build a new kernel, using a config file in which you = have >>>>>>> replaced "options SCHED_ULE" with "options SCHED_4BSD". -- = George >>>>>>=20 >>>>>> Thanks for the notes. >>>>>>=20 >>>>>> I've not decided if I'll do anything with the binary >>>>>> blob or not. >>>>>=20 >>>>=20 >>>> FYI: >>>>=20 >>>> It is not your specific experiment, but I started my >>>> "extra load" experimenst with . . . >>>>=20 >>>> I started a -j32 buildworld buildkernel with dnetc still >>>> running. I'm generally seeing around 55% Active and 42% >>>=20 >>> Note "Active": user, sorry. >>>=20 >>>> nice, < 2% system (it was building libllvm at this point). >>>> At that time: >>>>=20 >>>> load averages: 64.41, 60.52, 49.81 MaxObs: 64.47, 60.52, = 49.81 >>>>=20 >>>=20 >>> Contrasting results for some obj-lib32 build activity: >>> much more variety of User, nice, and system, including >>> times with < 5% user, 90+% nice. But not typical overall. >>> But lots of time roughly around 50%/50% or 35%/60%. There >>> were times with 15+% system. >>>=20 >>> Somewhat after buildkernel started: >>>=20 >>> load averages: 69.15, 64.12, 58.72 MaxObs: 75.98, 64.12, 58.72 >>>=20 >>> Harder to summarize, so overall timing reports from the >>> buildworld and buildkernel stages. >>>=20 >>>=20 >>> buildworld: >>>=20 >>> -------------------------------------------------------------- >>> ... World build completed on Wed Mar 22 16:37:57 PDT 2023 >>> ... World built in 2615 seconds, ncpu: 32, make -j32 >>> -------------------------------------------------------------- >>>=20 >>>=20 >>> buildkernel: >>>=20 >>> -------------------------------------------------------------- >>> ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 16:43:10 = PDT 2023 >>> -------------------------------------------------------------- >>> ... Kernel(s) GENERIC-NODBG built in 311 seconds, ncpu: 32, make = -j32 >>> -------------------------------------------------------------- >>>=20 >>> Afterwards: >>>=20 >>> load averages: 36.08, 53.14, 55.79 MaxObs: 75.98, 65.77, 59.84 >>>=20 >>>=20 >>> I then did (not all in the same window): >>>=20 >>> $ sudo service dnetc onestop >>> # rm -fr /usr/obj/BUILDs/main-amd64-nodbg-clang-alt/usr/ >>>=20 >>> before another -j32 buildworld buildkernel (no dnetc). The >>> reuslts for this were: >>>=20 >>>=20 >>> buildworld: >>>=20 >>> -------------------------------------------------------------- >>> ... World build completed on Wed Mar 22 17:39:19 PDT 2023 >>> ... World built in 1240 seconds, ncpu: 32, make -j32 >>> -------------------------------------------------------------- >>>=20 >>> (compared to the 2615 for dnetc also in use) >>>=20 >>>=20 >>> buildkernel: >>>=20 >>> -------------------------------------------------------------- >>> ... Kernel build for GENERIC-NODBG completed on Wed Mar 22 17:41:17 = PDT 2023 >>> -------------------------------------------------------------- >>> ... Kernel(s) GENERIC-NODBG built in 118 seconds, ncpu: 32, make = -j32 >>> -------------------------------------------------------------- >>>=20 >>> (compared to the 311 for dnetc also in use) >>=20 >> I forgot to show the MaxObs load averages for the no-dnetc >> context: >>=20 >> MaxObs: 39.77, 32.15, 25.75 >>=20 >>> Experiments without -j32 will take a lot longer, even >>> without dnetc in use. I'm not sure there will be such >>> results today. >>>=20 >>=20 >=20 > I decided to do some more of the less time consuming > testing. SCHED_4BSD, no dnetc, -j32 buildworld buildkernel : >=20 >=20 > buildworld: >=20 > -------------------------------------------------------------- > ... World build completed on Wed Mar 22 19:16:35 PDT 2023 > ... World built in 1235 seconds, ncpu: 32, make -j32 > -------------------------------------------------------------- >=20 > (compared to 1240 for SCHED_ULE) >=20 > So: no significant difference. >=20 >=20 > buildkernel (SCHED_4BSD building a SCHED_4BSD): >=20 > -------------------------------------------------------------- > ... Kernel build for GENERIC-NODBG-SCHED_4BSD completed on Wed Mar 22 = 19:18:34 PDT 2023 > -------------------------------------------------------------- > ... Kernel(s) GENERIC-NODBG-SCHED_4BSD built in 119 seconds, ncpu: = 32, make -j32 > -------------------------------------------------------------- >=20 > (compared to 118 for SCHED_ULE building a SCHED_ULE) >=20 > So: no significant difference. I again forgot to show MaxObs load averages (for the above): MaxObs: 39.23, 31.58, 24.30 > I'll try it with dnetc also active. >=20 I still have no good indication of dnetc progress to allow comparison of the combination. So the below focuses on buildworld buildkernel . I expect that the comparative results suggest a buildworld/buildkernel vs. dnetc progress tradeoff, not that I can well quantify it. The below are with dnetc also active. load averages, MaxObs: 73.03, 65.48, 56.30 (I remembered this time!) buildworld: -------------------------------------------------------------- ... World build completed on Wed Mar 22 20:15:56 PDT 2023 ... World built in 1667 seconds, ncpu: 32, make -j32 -------------------------------------------------------------- (compared to 2615 for SCHED_ULE with dnetc and to 1240 or so for no dnetc) buildkernel: -------------------------------------------------------------- ... Kernel build for GENERIC-NODBG-SCHED_4BSD completed on Wed Mar 22 = 20:18:34 PDT 2023 -------------------------------------------------------------- ... Kernel(s) GENERIC-NODBG-SCHED_4BSD built in 158 seconds, ncpu: 32, = make -j32 -------------------------------------------------------------- (compared to 311 for SCHED_ULE with dnetc and to 118 or so for no dnetc) With dnetc active, it does not take being near -j1 (or no -j) for buildworld buildkernel to take noticably less time: -j32 (the number of hardware threads, 16 cores) also takes noticeably less time. buildworld buildkernel in this context seems to be a good match to SCHED_4BSD and its round-robin. (I make no general claim to SCHED_4BSD being better across a large range of contexts.) I've not decided if I'll try anything like a -j1 or no -j alternative. Without dnetc active, SCHED_ULE and SCHED_4BSD did not make much of a distinction. For how I use the builder machines, the scheduler choice is not suggested to be significant for my system-build activities. I've not tested port building in poudriere-devel for how I configure such. But nothing suggests to me to expect a significant distinction between the 2 schedulers for my way of working for building packages from ports. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Mar 23 06:42:18 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhwkN3V07z40wB0 for ; Thu, 23 Mar 2023 06:42:20 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhwkN33cKz3jKS; Thu, 23 Mar 2023 06:42:20 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679553740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BWHkPN8zbrQp6bM1NZcHJQiYkN3cBZxSMkMAk0DkvpA=; b=VkKvaSLt0XssCoVvOGwttvC1nRJE2VePmkKJR4PWiSdBhK1Cnc5m8guGGfyWTBtbspKGxh jx0NRVv1Ls+fz9zOQILZTdU8UJC9HTv4YnldtCRmY1rAtSjLzqhnIevoOs/dT9KM8aRAhO YhBn5F/gkYQjEf1lnXUe8UsiHCCKx9NEKF0UY3Wsn1vOA82UnDdBI92ytdCkzY7Dq7xJlQ hhszcgAb2vP8mL6RpLuV7nc7+GRx6DISfF8+l3vnkceXqy0tL4B/7QT6R16Hafnv7IB0N3 suf3LubeTUzxUsfPEBKkjPc6nQ9BVm7WMpauSICtZFJlrFXzCwcaeI25Q3LTyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679553740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=BWHkPN8zbrQp6bM1NZcHJQiYkN3cBZxSMkMAk0DkvpA=; b=vklUY35umyt1/o5ryYlcfs5/QttJSpYmJfLt+uMfCMN4C71SxMUyGUtFuRFNjHyW0S88Ay /sPuPnLu72f8Jb1ffOB4RhUH5Cat3XyZJ8RkbYuq/mWM9vAXtuJ+ZqAjwC5YKPfCunD8je sWFsLB71LF1xYxFBqOFDC40txCIyMYBj74n6BOeYGW3ecfjGq27N1uI5ow7xax1ycbPcVa UmB7dmB2NaNvUlslKuV1N1ZwUtb3DCo9snkMWwacEi61+JLHCba0uuIgeDub5djigmxoXV JbhTzIokTx7ERKdAJB+qTpDOzEWNDOahmJn5xezPXiyjGQnxZaQ79YksMB9slA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679553740; a=rsa-sha256; cv=none; b=iKvpMztNTUEvajIO5e+ll1ffyTwZBeJzqCaNELlLwDfjmBPRBtcPZM/fNrBuuT2+bm4aKC MotVzk6Px+1qdklUnRCJW6uaR/XYHJV2phn+rwgRS2BHwreQ8QUlVH+czKJgdqhFrcj6FT AtDYv4EORFfyg6KhRzekJPA/52CbPO1nsyd2nTIWUUb4sDDroKrlbK9uvTjRHYg703Aecg H6dVPI2gjqOXEnOGrqLzLeVeXB2J17Pgu0ivTQ0W6DS0mtjt51CysdpgTHh3n40GrVpRe1 to2Af+1AryWfUpuTHaNjmbDg7ZojnelM0XX83aoxwRRrO/42oYj6ZIqEvZIXaQ== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PhwkM73kxzRLK; Thu, 23 Mar 2023 06:42:19 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <961f9dc5-f73f-fc58-a368-26a44c60229a@freebsd.org> Date: Thu, 23 Mar 2023 06:42:18 +0000 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: ZFS and Encryption at dataset level Content-Language: en-US To: Dirk-Willem van Gulik Cc: freebsd-hackers@freebsd.org References: <8BAC3BC0-63B2-449D-BF0E-8E5A0A42F1E0@webweaving.org> <8BDBF2BE-77BB-481A-BFB0-2488A8FFA118@webweaving.org> From: Graham Perrin Organization: FreeBSD In-Reply-To: <8BDBF2BE-77BB-481A-BFB0-2488A8FFA118@webweaving.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------ccjEOJ00g0GW9yPY7f7qRrqq" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------ccjEOJ00g0GW9yPY7f7qRrqq Content-Type: multipart/mixed; boundary="------------nN8INcgAknyqSsVrOkjO2nBj"; protected-headers="v1" From: Graham Perrin To: Dirk-Willem van Gulik Cc: freebsd-hackers@freebsd.org Message-ID: <961f9dc5-f73f-fc58-a368-26a44c60229a@freebsd.org> Subject: Re: ZFS and Encryption at dataset level References: <8BAC3BC0-63B2-449D-BF0E-8E5A0A42F1E0@webweaving.org> <8BDBF2BE-77BB-481A-BFB0-2488A8FFA118@webweaving.org> In-Reply-To: <8BDBF2BE-77BB-481A-BFB0-2488A8FFA118@webweaving.org> --------------nN8INcgAknyqSsVrOkjO2nBj Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMTAvMDMvMjAyMyAxMjowOSwgRGlyay1XaWxsZW0gdmFuIEd1bGlrIHdyb3RlOg0KDQo+ IOKApiB3ZSBoYXZlIHNvbWUgc3Vkby1nbHVlIGluIHBsYWNlIGFzIGBsb2FkLWtleXMnIGlz IHVuZGVyIHpmczsgd2UgYmFzaWNhbGx5IG1hZGUgYSAnemZzLWxvYWQta2V5JyB0byBjdXJ0 YWlsICd6ZnMnIHBvd2VyLiDigKYNCg0KSSBkb24ndCB1bmRlcnN0YW5kIChwcm9iYWJseSBk b24ndCBuZWVkIHRvIHVuZGVyc3RhbmQpIHRoZSBjdXJ0YWlsbWVudCANCmFzcGVjdCwgYnV0 IHdpdGggMTMuMjogemZza2V5cyBzdXBwb3J0cyBhdXRvbG9hZGluZyBvZiBrZXlzIHN0b3Jl ZCBvbiBaRlMuDQoNClRoZSByZWxlYXNlIG5vdGUsIGF3YWl0aW5nIGFjY2VwdGFuY2U6IA0K PGh0dHBzOi8vcmV2aWV3cy5mcmVlYnNkLm9yZy9EMzkxNzI+Lg0KDQo= --------------nN8INcgAknyqSsVrOkjO2nBj-- --------------ccjEOJ00g0GW9yPY7f7qRrqq Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmQb9MoFAwAAAAAACgkQt2dIb0oY1Att Ew/+NC/h10rzWVc7FssErwZ6n8pmZ5BraBbfq4pliO5PnEGDcg09vTnc7rEj7fEyz/vPXe5160Bn Dxcybl4vPA9vILYBUtC+cIihPZFMe5skk2w10jYaNJON3kht/t1VrynScrewkQAuqrr+p9wwCdio /01FH0yPdL6z7i+XwDnfGyR8b1JzWQ6ca+qRr5QNy6T7bCg7RE+pfRD2gOcYNnmEoTbuR/tHM9WX QywlnjzoyYirGdZfvgoxFU68xjHuZH6wZ9+LzvWZHX4CMRzIRGvMnIrFxRAR3YCXTNkOASd6bED8 x8CKKpCcrKj7jMrN56Yxg6mau7UgOcLmNuv5qqk50f4k+t+7gm4EXmHVd4iFuKjDltmqUy8x/7X8 /+H00QLqmW0vceZtB3sTjjlAcokRf+f/hngN0T1a1sFAHB9Y8sAoG32wSGh9nLO95ZcJsSdtoTQJ /Lm3oOynsljHZCoVThTJ6lPw0/suMJq/ciR2KNwekGUVrLm27IP3qOjmjVXQ8+MKHRPPSAzYaQBm vma2uUaQJgnkLky3oU8nemFjypnMrN8UT9e3O5fpL3H4q0E5l7g2MuNC6fdvGYd0V7I0xPSvp18W naIZtyoqQ+fmHCTJjFr0DZs7Da2xdPD10HOjEAZLlci1jali0gF44H/g5Ua8Lk4GFXazIovmdRAv eUo= =pajr -----END PGP SIGNATURE----- --------------ccjEOJ00g0GW9yPY7f7qRrqq-- From nobody Thu Mar 23 07:57:45 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhyPj5j8tz4123x for ; Thu, 23 Mar 2023 07:58:01 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhyPj2Crrz3rqx for ; Thu, 23 Mar 2023 07:58:01 +0000 (UTC) (envelope-from Alexander@leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from outgoing.leidinger.net (p5b165130.dip0.t-ipconnect.de [91.22.81.48]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) client-signature ECDSA (P-256)) (Client CN "outgoing.leidinger.net", Issuer "R3" (verified OK)) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 6715424A29 for ; Thu, 23 Mar 2023 08:57:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1679558269; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=H7jHQznVTEq0tvDILTaMIGWqRSkWGJa63f2tIPr1A9I=; b=odTX7QzortXqe0lOV3hH4tWiGaI6wC774UEcc5gTGDJ5Fcyei/APPI9keTZ07A+BVxCli2 0ZGl6atleFp0hPdJqbYYkvvPF4NWNPj4f6GbL+2djeNG6AUSfhGpnEShqwrsX6JGv8ugWs eiAO1cPsJMiZro9yEKCQ59Ee1VeddrKtevWzGkGDOVf8Em46K3xmeDNZHZthUVNlmaajVk 2NdG+6Yzc05/rjrYWuvp2wU29mv0vV5rF9AR7qPO/wsVo485H5sjZZ4qJZ80V2bhT5EiQS aKfRkxvvaJ8oAo1lCU0QXgo2KhNKP3ZU/wMDuDeMM/XBZKKM0uTCvcIBeQa9Yg== Received: from webmail.leidinger.net (localhost [127.0.0.1]) by outgoing.leidinger.net (Postfix) with ESMTP id 0B9AA370D for ; Thu, 23 Mar 2023 08:57:46 +0100 (CET) Received: from www (uid 80) (envelope-from Alexander@leidinger.net) id ac50c by webmail.leidinger.net (DragonFly Mail Agent v0.13+ on webmail.leidinger.net); Thu, 23 Mar 2023 08:57:45 +0100 Date: Thu, 23 Mar 2023 08:57:45 +0100 Message-ID: <20230323085745.Horde.eeJqWekvVqCybD-psTGfIRE@webmail.leidinger.net> From: Alexander Leidinger To: Mateusz Guzik Cc: sgk@troutmask.apl.washington.edu, Matthias Andree , freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> In-Reply-To: Accept-Language: de,en Content-Type: multipart/signed; boundary="=_xdph0Tnv8P9I0E3lje7qzJ1"; protocol="application/pgp-signature"; micalg=pgp-sha256 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-Rspamd-Queue-Id: 4PhyPj2Crrz3rqx X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N This message is in MIME format and has been PGP signed. --=_xdph0Tnv8P9I0E3lje7qzJ1 Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting Mateusz Guzik (from Wed, 22 Mar 2023=20=20 20:23:42=20+0100): > sched_4bsd *round robins* all workers across all CPUs, which comes at > a performance *hit* compared to ule if number of workers is <=3D CPU > count -- here ule maintains affinity, avoiding cache busting. If you > slap in $cpu_count + 1 workers, 4bsd keeps the round robin equally > penalizing everyone, while ule mostly penalizes a select victim. By > the end of it, you get lower total cpu time spent, but higher total > real time. See below for an example. > > Two massive problems with 4bsd, apart from mandatory round robin which > also happens to help by accident: > 1. it has one *global* lock, meaning the scheduler itself just does > not scale and this is visible at modest contemporary scales > 2. it does not understand topology -- no accounting done for ht nor > numa, but i concede the latter wont be a factor for most people > > That said, ULE definitely has performance bugs which need to be fixed. > At least for the case below, 4BSD just "lucked" into sucking less > simply because it is so primitive. IIRC ULE also has a semantic difference compared with 4BSD in terms of=20= =20 handling=20of the "idle" (and realtime) priority. What I remember (it's=20= =20 been=20a while since ULE was introduced) is, that with 4BSD the=20=20 idprio(1)=20case could result in such processes getting no CPU time,=20=20 whereas=20with ULE it is quaranteed that such processes will get "some"=20= =20 (no=20idea what a more specific description of "some" would be) CPU=20=20 time.=20BTW: someone may want to review the BUGS part of idprio(1) if=20=20 the=20part about "preempted" is still correct. I remember there was a lengthy discussion about the handing of process=20= =20 priority=20back then, but a quick googling didn't seems to have had the=20= =20 right=20keywords to find it. What I got was this: Original ULE design/scientific paper: https://web.cs.ucdavis.edu/~roper/ecs150/ULE.pdf Laymans-terms description: https://people.freebsd.org/~jeff/FBSD_14_10ULE_aj.jm_cx10_09.pdf ULE 2.0: =20=20=20=20=20=20 https://lists.freebsd.org/pipermail/freebsd-current/2007-January/068404.htm= l Discussion=20about ULE/prio/sche_preempt_threads sysctl: =20=20=20=20=20=20 https://forums.freebsd.org/threads/what-is-sysctl-kern-sched-preempt_thresh= .85601/page-2 Bye, Alexander. --=20 http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_xdph0Tnv8P9I0E3lje7qzJ1 Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIzBAABCAAdFiEER9UlYXp1PSd08nWXEg2wmwP42IYFAmQcBnkACgkQEg2wmwP4 2IZO1BAAjzrfS3XZNUxEOWSBQuU8UvYUr3IYPHU4fqzXYprydAe/c05Fu85DUA+M 8iLGMDg073yTjOx+CzD9j56stBffr9UWiIuw0DOvkV8XtIRveiyZQw2DNK+ppnSZ 2C0HI0EbARipsrQd+g0eCHVc0i+9u8Ntr8EIMlzdiXu6BEtw4Pc1dzf4qyCVq7Tc gLEg0812v3ErvEIQcP4bKlN0uWQQlSjjADXh9bk2IPRwW1a8fIWiM1hujK7qrFrL gK6PcjbPCqUxwFUR8ogX4wkwLn8l8FtTlwjlq5ZRabuJeaJPG6cHbTyStZ3qgDMt GjzqEmsh3mHJrE5OtBn2sonOJvMaSJqigK9Tf+1Y021EvySmkaeiytFcWKnWAyHE NRQjJJWzrf5caQC4hTX/BhSE4TpWAuZ4J6hsdxNr3uhOKMc8gNhpOqUA6RljXLfJ 5L9k3SQvihlysK1JW+fxu75MiCYCF5YOlH43XqRULwMwQ03i47KKyDbeEkvZTsHX siowP7jDeq0TtW2USADSGRqlhG7QIgRWgNjnGtGBg7qOVghuQNQKcKdHFEyj4cuT o1WV+7wIMPIC1riOmKEIxWlDlp6yvURfTulcQM+6xNFkmMeoHEUI4CjxI97WDlsE xU2b2QY09DfJFoE9Hn3KNcAUIbNLBBP28SmYd0z7r7OpOgxjHZQ= =gkYZ -----END PGP SIGNATURE----- --=_xdph0Tnv8P9I0E3lje7qzJ1-- From nobody Thu Mar 23 10:03:36 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pj1Bf12zBz4185j for ; Thu, 23 Mar 2023 10:03:38 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pj1Bf0Tqbz41XD for ; Thu, 23 Mar 2023 10:03:38 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679565818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4TpZ2baSjKTJcQchmfuzXzk1bBgZI1veWzqkERgDZns=; b=auKPqwzn+X/vIFS+E9eFhf8FZSA5L7cdmpOpo+G8mgvMuiJ+1HUgXBoptVeL3OlGhkmJtk uJPP+hZukJNCO/5e9CyT8uQfkrY7gguv7fICW0iZe+6fg9DTKCG+kDNNnbnh5fSPSjeonD 4modk6vU7xmIGusM1/i8ko4hP9t9Xhf5zmLm22u97IVC9Xscm2mgpFJ4I1l/+EMxRV0uMf SfeIeabQp7awKKWtZwTFY6d7uNKcE3vahvIApfrFV7CmnuhAANbXMdBB0H9SpV3AZ6cx91 OxoSHOX5rjxwrAZdDNSd+qPzN8YoAXLP+ADr2nkZNmMC08sV0U5bpr5EBcIfAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679565818; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4TpZ2baSjKTJcQchmfuzXzk1bBgZI1veWzqkERgDZns=; b=shdT6TT/iQ+QkxljuaYsZ18eLNWDMc42idgneV5zd+NrZDVlj/mwK84+y8DfxsdnTWSzEJ XJ2GJUAtfGVjHjp5M+pirWb1fDrKfljZcMYvsXJkhzCcPoNLyvGdmXrGrCbW6V3o+R5Bwy 6JuMRZ8yquqtGb6UZc6JItnF5HAd6COSBJxcLHWnoCq3C4vgUaNcP53ZfEqhdgSUEXGAsy yNdDpBWThViG94Jcbe9Mp6qWR2nz/B5Z4hx2kdu8uMqLTV2quXaapBwYLvn2/QSiwbnfrz LIF5kF65gtadv4QA41fpkSRTpbeF3n3Y3ZB9OcZ2dIndSIUPcLeg08XUCdfiYQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679565818; a=rsa-sha256; cv=none; b=pKy+w60wdm0vFR6aZRuAW+uZpGj0mOpYM39aV8Q4PTyERLwmB/Mc/evMNX3Lqf3NP6hc+1 a+X5RhuNOBQnzkIBOUEp2Qj1WUTzveWXw/CXQzkIDUxdA59SNPssg6dBQFxEUt7O3qwG6O CmhRuzCMflg9uDr3eW6BLoUedOGq3NBzAMw8f5NVvOLKcdE4ZupoNyLb1Au+5YPZEQVaqP 6h6bwJIos4gFVHATY6x23RsJ2ILk/oNSDLFkcucwhVtMXbpQP05hl+k9YTnjZYAWRhEAQS weuwuP8mHTJKBZs32CTocUtVe1meevk8wQvDRiEGZk/mVN4MMkeZGvxsfZQRpQ== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pj1Bd6WhCzVlN for ; Thu, 23 Mar 2023 10:03:37 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from [192.168.1.202] (host86-160-181-152.range86-160.btcentralplus.com [86.160.181.152]) by smtp.theravensnest.org (Postfix) with ESMTPSA id 036E610571 for ; Thu, 23 Mar 2023 10:03:36 +0000 (GMT) Message-ID: Date: Thu, 23 Mar 2023 10:03:36 +0000 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-GB To: freebsd-hackers@freebsd.org References: <202303221710.32MHAhe9047582@gndrsh.dnsmgr.net> <27f46bc2-54f8-f5aa-79ca-184e86d185d8@m5p.com> From: David Chisnall In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-ThisMailContainsUnwantedMimeParts: N On 22/03/2023 18:03, George Mitchell wrote: > Rebuilding the kernel is the only way I know.  In an ideal world, the > scheduler would be a loadable kernel module (if that's even possible), Solaris supports multiple schedulers, as I believe does Linux, but I think in both cases it's a boot-time option. It's been too long since I looked at the early boot order to know if there's anything that handles linking the loader-provided modules that depends on any scheduler data structures. Doing that audit and ensuring that there aren't would be the first step. From there, it should be mostly build-system infrastructure to allow building the two schedulers as modules and switching between them at boot. Allowing switching schedulers after boot time is a much harder problem. I know some microkernels have done it, but even there (where the scheduler is a separate service) it's far from easy. David From nobody Thu Mar 23 10:07:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pj1Gv5sd2z4185t for ; Thu, 23 Mar 2023 10:07:19 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pj1Gv5KS4z42lD for ; Thu, 23 Mar 2023 10:07:19 +0000 (UTC) (envelope-from theraven@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679566039; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xjrLKWGLGWtUSr94njHW+GUuocc2jMBG4bc8Sxy65TI=; b=ZmZy3DqDF8P4jz9pAsWquFonKW/5g2b5th73CpAeb0yATE1rAjb5yKl6ZTqGo9gjmZaL23 xDI4/dMUwU+bI0AooVnh17OxliTI1weDU+H2mpOXRAuqLEybrvfY2/FHu1uFhfyQa3BKj3 rx1XN2oDSxSTdsiNqnTf+AAc9wXjLYOpguIwRhk1kvHoi4RBT+bLhNsx++9WuJN2uFbEyb d+fvpGV6rzwGUNGpCrHaqDBY/p53ih/1GPBEALZi897KNJAswtvkz3jRx1bGUKkh/nEW8S NLi0/YUOw06zaNgt2/sUCF94mloGfCdZrK53nvDGRVwQBU7uCxxToT5WGxyQhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679566039; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xjrLKWGLGWtUSr94njHW+GUuocc2jMBG4bc8Sxy65TI=; b=UI1Vcy0z1cPMDrb1fM7ABcL0C/p5jUTcaZ7YPvIbzD5sAL7XnP4NjGPKhI6RsOV9/zwuS0 1Lz2HAzgtfk7nZLHl8Q47Y1npi6endNm4ziSZ0DVeALkiBgPvprinmrK7BZJPDjnFI8tj5 ayQ+on1oFAwtg1KRS5l10VmKB/x1b8YwGz1lhbqtRh0YDMA4Z7pSccszTQRuyFBcWBcSu/ BbNBowMzdRyc/2V+hFinx5gZV0Wl6DISuCxA4HIzM7jn8b8NpLMbZGdeoJAA8Fum17AcQZ jIAzsXSjedDAj4znv/W1e0sGCxgDgpVdL1X6NBZb4r09ib6hQ5U3AJg5xV88FA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679566039; a=rsa-sha256; cv=none; b=hbhjB5q6+Cy7YjPJGBo+tHuMGEIQNBac6t5VH6x/am99UtR+9EGWaBRIXHe+hD9G1oFOh8 6cmfATRz7A0bz/V8pKPdIG3rqzVHX2bYlYLg0sLAd0s2CEFpF8LNoK3pTXA8Tr2jPAwCCT OKeyrv/hLliD7HftIY1SsmavBzqiJJi11MXycBpunaRwpyUmREzaiByTRQhMeTF+h/BInZ BZc5kE3uTm6EqI0KrHhd1CBSnDiJxeS3GkzVi+Gz0r8MwV27AlG+j81DFC6ETKHl6lEah0 ZWlmu/z7qPybFv7cTu24gq3koi8SSrymxzIJqnWwbqnubAXhTWlujAVr7RCsrw== Received: from smtp.theravensnest.org (smtp.theravensnest.org [45.77.103.195]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: theraven) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pj1Gv447yzVj2 for ; Thu, 23 Mar 2023 10:07:19 +0000 (UTC) (envelope-from theraven@FreeBSD.org) Received: from [192.168.1.202] (host86-160-181-152.range86-160.btcentralplus.com [86.160.181.152]) by smtp.theravensnest.org (Postfix) with ESMTPSA id 4681A10572 for ; Thu, 23 Mar 2023 10:07:19 +0000 (GMT) Message-ID: Date: Thu, 23 Mar 2023 10:07:19 +0000 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: Periodic rant about SCHED_ULE To: freebsd-hackers@freebsd.org References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> <7f26102c-7542-78f8-0c7b-ef3cdaa1a4a6@FreeBSD.org> Content-Language: en-GB From: David Chisnall In-Reply-To: <7f26102c-7542-78f8-0c7b-ef3cdaa1a4a6@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N On 22/03/2023 20:15, Stefan Esser wrote: > Better balancing of the load would probably make ULE take less real > time. The example of 9 identical tasks on 8 cores with 7 tasks getting > 100% of a core and the other 2 sharing a core and getting 50% each > could be resolved by moving a CPU bound process from the CPU with the > highest load to a random CPU (probably not the one with the lowest load > or limited to the same cluster or NUMA domain, since then it would stay > in a subset of the available cores). Two things have changed in CPUs since ULE was written that make the affinity less of a win and may make some low-frequency random rebalancing better: Snopping from another core's L1 is a lot cheaper (less true on multi-socket systems, but fortunately ULE is NUMA-aware and so can factor this in), which makes the cost of migrating a thread to another core much cheaper (there are still kernel synchronisation costs, but the cost of running on a core that doesn't have a warm cache is lower: the caches warm very quickly). CPUs now have a lot more power domains. If one core is doing a lot more work than others then there's a good chance that it will be thermally throttled but others may not if they're in a separate power / thermal domain. This means that keeping a compute-bound process on the same core is the worst thing that you can do if other cores are idle: that core may be throttled back to <2 GHz whereas a core on the other side of the chip may be able to run at >3 GHz. Evenly heating the entire CPU can have give much better performance if the number of active threads is less than the number of running cores and better fairness in other cases. Both ULE and 4BSD are unaware of the heterogeneity of modern CPUs, which often have 2-3 different kinds of core that run at different speeds and neither understands a concept of a power budget, so there's a lot of potential improvement here. Writing a bad (but working) scheduler is a fairly difficult task, writing a good one is much harder, so I'm not volunteering to do it, but if someone is interested then it would probably be a good candidate for Foundation funding. I've heard good things about the XNU scheduler recently, that might be a good source of inspiration. David From nobody Thu Mar 23 10:26:44 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pj1jW2m8Gz419MH for ; Thu, 23 Mar 2023 10:26:55 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pj1jW0Xtlz44d9; Thu, 23 Mar 2023 10:26:55 +0000 (UTC) (envelope-from mad@madpilot.net) Authentication-Results: mx1.freebsd.org; none Received: from mail (mail [192.168.254.3]) by mail.madpilot.net (Postfix) with ESMTP id 4Pj1jM0kdGz6dsH; Thu, 23 Mar 2023 11:26:47 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=madpilot.net; h= content-transfer-encoding:content-type:content-type:in-reply-to :from:from:content-language:references:subject:subject:date:date :message-id:received; s=bjowvop61wgh; t=1679567205; x= 1681381606; bh=+EXBD6k84ysDFiw2NKtYQ+YyVhiKal3QC6OcwFFaPfY=; b=O QG70QarAdOQ8X0gVmkpyG0aMJ78I79+RIGtNpPdSK+9gzt7t5IBS8ayO+WGvPZD6 FN0JMCegBMCBUjJUnNeIGlNKV3O2kvTjHlHuap5vT8eRWL0XTWUzAg8ZqT8gKntg 8Gyny5eHA5L1CRcADYiIX3nOoszmE7X9BHkit5KIlPnP/ZtKM0Gi9onIXFaIzAL+ 3CuOLnDTE8U+sf+GOBQR8r+s6K3mOoK9fH5G6c40Gz4N90rlGqcdYoixbljs4MGh hBIAtH4HaL3ILdfcnQx6CYpmcGaNfta/5Z5RKHs+CC5kJMf5MtcdO8Or/u5OsNiI 4F5dLCyzmhBR14cGVjsJg== Received: from mail.madpilot.net ([192.168.254.3]) by mail (mail.madpilot.net [192.168.254.3]) (amavisd-new, port 10026) with ESMTP id OsCbKGZW6YPg; Thu, 23 Mar 2023 11:26:45 +0100 (CET) Message-ID: <2de839ad-1af9-7930-c294-7523a2892649@madpilot.net> Date: Thu, 23 Mar 2023 11:26:44 +0100 Subject: Re: Periodic rant about SCHED_ULE To: David Chisnall , freebsd-hackers@freebsd.org References: <202303221710.32MHAhe9047582@gndrsh.dnsmgr.net> <27f46bc2-54f8-f5aa-79ca-184e86d185d8@m5p.com> Content-Language: en-US From: Guido Falsi In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4Pj1jW0Xtlz44d9 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org On 23/03/23 11:03, David Chisnall wrote: > On 22/03/2023 18:03, George Mitchell wrote: >> Rebuilding the kernel is the only way I know.  In an ideal world, the >> scheduler would be a loadable kernel module (if that's even possible), > > Solaris supports multiple schedulers, as I believe does Linux, but I > think in both cases it's a boot-time option.  It's been too long since I > looked at the early boot order to know if there's anything that handles > linking the loader-provided modules that depends on any scheduler data > structures.  Doing that audit and ensuring that there aren't would be > the first step.  From there, it should be mostly build-system > infrastructure to allow building the two schedulers as modules and > switching between them at boot. I guess this will be somewhat provided once we get pkgbase. At that point multiple kernel packages can be distributed and the user chooses which one gets loaded. -- Guido Falsi From nobody Thu Mar 23 11:01:44 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pj2Tq4JQgz41CTF for ; Thu, 23 Mar 2023 11:01:51 +0000 (UTC) (envelope-from rb@gid.co.uk) Received: from mx0.gid.co.uk (mx0.gid.co.uk [194.32.164.250]) by mx1.freebsd.org (Postfix) with ESMTP id 4Pj2Tq1gpyz47lZ; Thu, 23 Mar 2023 11:01:51 +0000 (UTC) (envelope-from rb@gid.co.uk) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (moriarty.gid.co.uk [194.32.164.17]) by mx0.gid.co.uk (8.14.2/8.14.2) with ESMTP id 32NB1iv6006830; Thu, 23 Mar 2023 11:01:44 GMT (envelope-from rb@gid.co.uk) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.2\)) Subject: Re: Periodic rant about SCHED_ULE From: Bob Bishop In-Reply-To: Date: Thu, 23 Mar 2023 11:01:44 +0000 Cc: "freebsd-hackers@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <5A2888BF-AB1F-40E0-AF42-ECE8D32AD908@gid.co.uk> References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> <7f26102c-7542-78f8-0c7b-ef3cdaa1a4a6@FreeBSD.org> To: David Chisnall X-Mailer: Apple Mail (2.3696.120.41.1.2) X-Rspamd-Queue-Id: 4Pj2Tq1gpyz47lZ X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:42831, ipnet:194.32.164.0/24, country:GB] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Hi, On 23 Mar 2023, at 10:07, David Chisnall wrote: >=20 > On 22/03/2023 20:15, Stefan Esser wrote: >> Better balancing of the load would probably make ULE take less real >> time. The example of 9 identical tasks on 8 cores with 7 tasks = getting >> 100% of a core and the other 2 sharing a core and getting 50% each >> could be resolved by moving a CPU bound process from the CPU with the >> highest load to a random CPU (probably not the one with the lowest = load >> or limited to the same cluster or NUMA domain, since then it would = stay >> in a subset of the available cores). >=20 > Two things have changed in CPUs since ULE was written that make the = affinity less of a win and may make some low-frequency random = rebalancing better: >=20 > Snopping from another core's L1 is a lot cheaper (less true on = multi-socket systems, but fortunately ULE is NUMA-aware and so can = factor this in), which makes the cost of migrating a thread to another = core much cheaper (there are still kernel synchronisation costs, but the = cost of running on a core that doesn't have a warm cache is lower: the = caches warm very quickly). >=20 > CPUs now have a lot more power domains. If one core is doing a lot = more work than others then there's a good chance that it will be = thermally throttled but others may not if they're in a separate power / = thermal domain. This means that keeping a compute-bound process on the = same core is the worst thing that you can do if other cores are idle: = that core may be throttled back to <2 GHz whereas a core on the other = side of the chip may be able to run at >3 GHz. Evenly heating the = entire CPU can have give much better performance if the number of active = threads is less than the number of running cores and better fairness in = other cases. >=20 > Both ULE and 4BSD are unaware of the heterogeneity of modern CPUs, = which often have 2-3 different kinds of core that run at different = speeds and neither understands a concept of a power budget, so there's a = lot of potential improvement here. Writing a bad (but working) = scheduler is a fairly difficult task, writing a good one is much harder, = so I'm not volunteering to do it, but if someone is interested then it = would probably be a good candidate for Foundation funding. I've heard = good things about the XNU scheduler recently, that might be a good = source of inspiration. >=20 > David >=20 This is spot on as a summary of the landscape. The MacOS scheduler = (based on XNU) [1] seems to do a pretty good job with heterogeneous = cores vs power management, and MacOS has APIs allowing applications to = take account of the thermal state of the total system[2]. But, I = haven=E2=80=99t seen any references to fine-grained thermal management = as outlined above. [1] = https://developer.apple.com/library/archive/documentation/Darwin/Conceptua= l/KernelProgramming/scheduler/scheduler.html [2] = https://developer.apple.com/library/archive/documentation/Performance/Conc= eptual/power_efficiency_guidelines_osx/RespondToThermalStateChanges.html -- Bob Bishop rb@gid.co.uk From nobody Thu Mar 23 11:54:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pj3fr6KL0z41Fp2 for ; Thu, 23 Mar 2023 11:54:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pj3fr3JNDz4FRH; Thu, 23 Mar 2023 11:54:44 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 32NBsUvg093853 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 23 Mar 2023 13:54:33 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 32NBsUvg093853 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 32NBsSKW093852; Thu, 23 Mar 2023 13:54:28 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 23 Mar 2023 13:54:28 +0200 From: Konstantin Belousov To: Mateusz Guzik Cc: sgk@troutmask.apl.washington.edu, Matthias Andree , freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4Pj3fr3JNDz4FRH X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Mar 23, 2023 at 04:15:21AM +0100, Mateusz Guzik wrote: > So I also ran the following setup: 8 core vm doing -j 8 buildkernel, > while 8 nice -n 20 processes are cpu-bound. After the build ends > workers report how many ops they did in that time. Why nice? Did you tried with the idle class instead? From nobody Thu Mar 23 16:42:22 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjB2m5kvkz41Y3v for ; Thu, 23 Mar 2023 16:42:24 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x32e.google.com (mail-ot1-x32e.google.com [IPv6:2607:f8b0:4864:20::32e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjB2m41x9z3Jdm; Thu, 23 Mar 2023 16:42:24 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ot1-x32e.google.com with SMTP id m20-20020a9d6094000000b0069caf591747so12448499otj.2; Thu, 23 Mar 2023 09:42:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679589743; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=+l7A8CIzEK2lOcqeu/FUIjnaTrFFE86jW7pYBHWQHms=; b=i7s/spi9IUwL2A/ZKnwst2Nx50RmUrE/dMS8uWJDjMx7AFRCLHe6/+SyIdYiJSyNRz mUUQJynpZYIaWNRzW7wAYden87bC90GCWaLrCaGjm5d1rgIhNhakyg9yXb129ARjVTrw o1azHaUZqf/eOlXVf6EghKOz5v7emm59Rx7TbGcdEy4/EnCLBh2jHe48s1unJI7lgoix C5u71dW5xeoPzOM+SuEmf9xpy8hjFqq3wXPXtsMA+FnJ+9HQkBvGrAuEqhIVLz8+byXF W2AKFak5e8dfrhcJuwGeJ1FqMVGYYs0LBRcpKOFUW6f3STjl+RRQJzAFr1dst5Zz3bCn 7u3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679589743; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+l7A8CIzEK2lOcqeu/FUIjnaTrFFE86jW7pYBHWQHms=; b=fTawMs6D31aSsTyNk3n/faZbybmwA3LtUZ7PHmvBObOsWyJjW5jiIBpOFw/WXHtNgx 1vZVf6qs3pKMj5zcLLaMO7HR3zM3lFFVo4S5QlgMGNqv+w2ftYBD6TYM2HrxWRZo1iPv OYkinqVd2DtrIQpGMuy8pC2i9fn+/0YRvHJ+BEzeIpiRQUJr2aOMRp4+KYs86DBtyGGS PwAuqcE80WSob1ExbClDN5MVLnhZBuJYdIbRo1ny4GPrVUucfVuTuBxnsBQ2ULzt6qAD KuT+rTQCTUt9wzoLFro+hMzMr3PEWdh0S7dLSZzk0OR9AN07nIuuXuUH59pcik0XeJcJ vFHg== X-Gm-Message-State: AO0yUKWASWIcFDGhpUB+zcTfR8o0ZgaEZKl+zYOGTXPAoicEZfmfhnul K0Wk3zyARF7vdtIVJliSJSe9anZAmT7y4JmRFSiHfKbA X-Google-Smtp-Source: AK7set+tk0p2VEinHdc2jJv5zry6Blko9ZdcSR9p3FDwVrQlZjpH0rMy+ArSlsOXybLYRd2yLG5dpsoYsTNNquAjjUg= X-Received: by 2002:a9d:63c5:0:b0:698:f988:7c30 with SMTP id e5-20020a9d63c5000000b00698f9887c30mr44854otl.2.1679589743425; Thu, 23 Mar 2023 09:42:23 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1911:0:b0:49c:b071:b1e3 with HTTP; Thu, 23 Mar 2023 09:42:22 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Thu, 23 Mar 2023 17:42:22 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Konstantin Belousov Cc: sgk@troutmask.apl.washington.edu, Matthias Andree , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4PjB2m41x9z3Jdm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 3/23/23, Konstantin Belousov wrote: > On Thu, Mar 23, 2023 at 04:15:21AM +0100, Mateusz Guzik wrote: >> So I also ran the following setup: 8 core vm doing -j 8 buildkernel, >> while 8 nice -n 20 processes are cpu-bound. After the build ends >> workers report how many ops they did in that time. > Why nice? Did you tried with the idle class instead? > The original message reported numbers with -n 20 and it is not particularly unusual. -- Mateusz Guzik From nobody Thu Mar 23 18:48:18 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjDrR1xlmz41g2d for ; Thu, 23 Mar 2023 18:48:39 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-54.consmr.mail.gq1.yahoo.com (sonic315-54.consmr.mail.gq1.yahoo.com [98.137.65.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjDrQ13Lkz3ktw for ; Thu, 23 Mar 2023 18:48:38 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="Qtm/Fw4B"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679597315; bh=jGRRcwakzE/RhLDxK+m9QSjge9z6haGhrTgfozRW3xw=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Qtm/Fw4Bn0+kfeUmonlVszlcq24T8wcIjgv4sdPYD3dfBuQNM84MseuPk5rD3MDPfmNVJlczQvapoehblZc2fA/Msa3X2xPROfuD1sfB8R9pIkx/oIhfSkbhTsogW4cOgYgIeTdb2J5IEH3UejC0VTGwxBbMdXvNdjJSWIDTV9sEnef07whH58Bxk/ux4GfYPiLkvz89UdDsYSp1BJe28wIGNr1ok8BUY8lj2RtGFM5Qlb/dSM7hruKxtkA5yd1V5bqGKS4bcgCW1ngK3aqMo+Tdq0er0qgDGsJJ8Zsb6CRNsmHDi/uKfpRpMzsV6SHrDyoZ1lEwNGIPteXhCIuGHQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679597315; bh=geCQLk/XmJtFXwo1uvXxNLmkdUCAjjIxLPgfI6w9+l2=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=c2Z1rfB9zX9tvQiXFGcQXZRrS7LyQycap7pe1ZBOdbx2UB0bMcgOyLIKN6XcRLoqUxNCF1unoiQ/MZ8wcK5+R0o72XQruuA2OqoxjZ6VZbzGKyR57QuemtyxOxdGMCTCkbZH8wErRECV+FF6UKrwl9LVLArK+palgcElKPFuNvzwmaCqFhfwi3ys5cnGbbFqwjTM9z5TUXyo5dcjJ1srVddZKcPOdsm3gv/zFdr53SeKumz7jejh/avZkPFVEjFb5JVvsbTes5x6E9f/1w8khEQljUVHifaAE1JCd5CGmIKkKNEo7dVIip8eRlDKgx85wpRfKbEW+A23Hu2sdWbulw== X-YMail-OSG: rk7TsnIVM1kOuOTFVv6w5tjEyTjRe9rXTw7NBO_nNyiGVSWTQ2JEjaSj3pEnp1r 1ccEg1GJyhDT_iH.HwrRiklmaOB5Cn1nh6nHDlfW75wChzmFmHUzYunL.8zNFz9rxIxu.1z4p5jF Zd_Clcs2yFeJ22udfupE0fKW5pCba7am878qfFQJo8ZoqNCNjkw1B4O3fqQkz4kl0bhTDPsoOH82 NjiPIwwgUOC0QYeBpqLHPkqxyQHXEUkRPoTGeHxHjQTpkUHE2IzuN4fZr.NdwqaxCpBXa9u4J3NZ vJjxtTw_zbkO.te0KfsNZY775p_MrYhUGmKBU.i6yKzKFvWskaFXBOZl8aQu7kgdqQbVlloY7mIA qe68UX7V2BFRRWKhGriPWlw1ONrMJSUngQ0OnznnDiyr73Qw0N_r.ontAKpbzzwmAC4W1PZoSAPZ aYO6kau8SgM94IMPXTYZ3cSKPUfRY.2E7DDUEWKZqRKV34Sw20RsZo4OfuG8VGYnU1eXFnowJujc bVST5PElERpqqfSGX7HFGQqOqtE.96CezwavFbltEogAOkHgfcmIYI18i3lH5RhRKJSi0G10n8TI _E6rpvPRkqehHOL1HEwYEZdLmuFi2JgfDP2XfnbICDSuu9bJju6spF0Fryx1vLcOn2a463KEO1jy dHT6aiXsktjnZE0d8De6zLBzIO93IfdDA8ikk7gTJvGdosD7gN78PcmlXEyaMyXWXfnQCFE5Q4WW 91.j4a9NZXmOO8zqj0yVC_d3y7dC6nKWUamWJigYG93il.GXMaNRLAlK7sLLeQ1vGfsYjY_z5d8b vsrgWYeJ5GUMeyr0_geBwWyVnFbFGhCQdbO4jYLlm5Jp1ODI5MkoXY154kz.X8FCVJpHxqrUl0zK 1CbMa0.Q3tuE.bUPX6ayDe5HWpIdRGvQSw4Z9Hje_bf2J1hnEDw4uX5w8XC8TI2AQlS62rKRbZPs j6TMKeT.JOylK6q2gBRTuzVONqQ7qqufphQKQM4fOh6v4KkTjuA4MCO0FucdTgsaRK0Oov9xSNee 0BJBNFJ85VUUXVBg8zm7rzEhYEhjZEA7EyBf7XH8m38YgWZKY0QoMJ.a.mUV0JaN3MSfiuNwpVk8 SFUtEvbHRI1PHOQE1APs5JTkJ1PA_tlgQaHr808SN.NrdF5Vn8NKpltdED9eod4z0TCwgJdi3f.5 0Yd8TQk82oQI2CRYIGhGYO02MhID_MMt.6jj1.fZGkPwBIf_BVPrChoogNTE3dtBgHYC8LwOfIHT srQa5RtXg_japf7Zsjpo5SJjQneUrG1R4eu0_6g2Y_J7K8h956OC7onIOrH9cuT77eNqA.aU1M.Y _HqLv7b8hYCK2_e7.RGuPBxZ33qfwbeQEY9F6D9s0Vw_.phujJSlQn8pS_knV6Wmf_PACXMi5ySE 2PcOnTiOrpdn1wtHbqIhNKOP00tXBGVDV4Pjzw78iljQC77HE3bQx1oEB8QuhSlmTsJvjkAUOzQU l.klqWwrqOp1dzPrbEwCIaWzZqAjh7DbNVI2GzxtnOZXcUAu6bnH1uwCUDQzRz5mVeWGwpn65dVy lv4PnQrxSEd5Rh4Ha6hLQy3a79TjRmmC3pUHrlDd7WU8SEZYi6Qc2wyiyx1dGbH9urEyQuHL7v5q teKwJvA1FbUP._5b2SuwKnnhPPXljDwzKEVIa7.UGcOgeZnnA7uvIvqzJ4JWjwat2IX2u9khqE1V FTapTuoSpX1fFL8UVNN1Xqxvm8wUPiFLD41McxbaxmuvwG4sUJ5MTVex6DfrHm023TdqHY1UZS0e qC3rzdJ70MWLZoMLdptcC2ineZcg2eguW7DsOTyyCP6NJJpx2y1nRCoRsRMzDB4s0NtSFYxWwEL9 TMlJkqaUiWRQBtWWiVJVq4TjVnMB2v7JXxFM0rbCZo.KH.JDDv2zemu46JmOI4TlKgg7Zi.J_kIv YS8jWVLvFglTT6D_D7YfWY6Vmhj7jHTqbZ62mYrYNkM6OBz_BpIIToBWEtnraZTmc1ubFbqQMKLD CdIdQWRkR7pblZB1bvilRX_GefcXaz5PbhnRvXpuCObNL8AgqHq.i1vSGz0YQ2NaBevmes_7yj3b FnJ7d5BJCwGnrlLSYiXtKMFNhzWfEClZkoi6cdL5Ddy.bOUYEwchTiCIgk6BjSeg.Ioo7A1dbP96 nMfa2uc1T.C5zIeIF9K3G7kCF6aMVJMhCxNG2056BTLe_kfm5_boj8MYOzmnHgL0r7KunzjNGJHR hm7DJYGEJjzBj17C4Ha0LuqdhpIxd5cd0epKi59S2_oO_Y.yWkBPUyxa.JLOGWzCwYrLJQX_.QtD 8 X-Sonic-MF: X-Sonic-ID: 34fec43c-8172-4797-8b24-75ddd89fd6ed Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 18:48:35 +0000 Received: by hermes--production-bf1-777648578f-7gmg8 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8f58610ad802a675c1326621b07b7803; Thu, 23 Mar 2023 18:48:30 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: <6822D3CE-789E-4FB0-BB62-BEEEA65DB72F@yahoo.com> Date: Thu, 23 Mar 2023 11:48:18 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <2C62B951-1884-4234-8F11-D324ED8C34BF@yahoo.com> References: <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1.ref@yahoo.com> <6BD317F2-7EDD-45C0-9DC9-5B94C1BBB8E1@yahoo.com> <952d9795-19dc-8ad1-bb75-5c556ca6795a@m5p.com> <78EF511D-BAF1-495F-BAC9-03AC1B8FD56A@yahoo.com> <0196A767-5DC2-4DA3-BA5E-62A358CBCF64@yahoo.com> <6822D3CE-789E-4FB0-BB62-BEEEA65DB72F@yahoo.com> To: George Mitchell X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.45 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.95)[-0.949]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TAGGED_RCPT(0.00)[freebsd]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.30:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PjDrQ13Lkz3ktw X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N This goes the direction of comparing SCHED_ULE vs. SCHED_4BSD for a poudriere bulk build sequence, no competing dnetc or the like involved. This is on the same systems as prior buildworld buildkernel and dnetc testing. The style of building allows large load averages compared to the 32 hardware threads (16 ThreadRipper 1950X cores). It is a root on ZFS context. Summary: SCHED_4BSD vs. SCHED_ULE makes little difference for this kind of context. SCHED_4BSD: # poudriere bulk -jmain-amd64-bulk_a -c -f ~/origins/amd64-origins.txt . . . [main-amd64-bulk_a-default] [2023-03-23_00h23m37s] [committing:] Queued: = 541 Built: 541 Failed: 0 Skipped: 0 Ignored: 0 Fetched: 0 = Tobuild: 0 Time: 03:32:46 So: a little over 3.5 hr. For maximum observed load averages, MaxObs: 197.37, 159.25, 110.71 It did use a little swap space: Swap: 491520Mi Total, . . ., 194184Ki MaxObsUsed, 111420Mi = MaxObs(Act+Lndry+SwapUsed) , 119490Mi MaxObs(Act+Wir+Lndry+SwapUsed) SCHED_ULE: # poudriere bulk -jmain-amd64-bulk_a -c -f ~/origins/amd64-origins.txt . . . [main-amd64-bulk_a-default] [2023-03-23_08h04m39s] [committing:] Queued: = 541 Built: 541 Failed: 0 Skipped: 0 Ignored: 0 Fetched: 0 = Tobuild: 0 Time: 03:28:00 So: a little under 3.5 hr. (Somewhat under 5 minutes less than for SCHED_4BSD.) For maximum observed load averages, MaxObs: 203.63, 156.13, 118.31 It did not use swap space (no MaxObsUsed reported): Swap: 491520Mi Total, . . ., 90569Mi MaxObs(Act+Lndry+SwapUsed) , 101048Mi MaxObs(Act+Wir+Lndry+SwapUsed) =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Mar 23 18:55:22 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjF0L4MZ8z41gjx for ; Thu, 23 Mar 2023 18:55:30 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjF0K6qqVz3mmv for ; Thu, 23 Mar 2023 18:55:29 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32NItMdK009403 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 14:55:28 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: <4831cdf2-57d7-a699-4ece-7f1f07b05845@m5p.com> Date: Thu, 23 Mar 2023 14:55:22 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE To: freebsd-hackers@freebsd.org References: <202303221710.32MHAhe9047582@gndrsh.dnsmgr.net> <27f46bc2-54f8-f5aa-79ca-184e86d185d8@m5p.com> Content-Language: en-US From: George Mitchell In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-3.22 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.99)[-0.987]; NEURAL_HAM_SHORT(-0.93)[-0.929]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; R_DKIM_NA(0.00)[]; TAGGED_FROM(0.00)[freebsd]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[m5p.com]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4PjF0K6qqVz3mmv X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/23/23 06:03, David Chisnall wrote: > On 22/03/2023 18:03, George Mitchell wrote: >> Rebuilding the kernel is the only way I know.  In an ideal world, the >> scheduler would be a loadable kernel module (if that's even possible), > > Solaris supports multiple schedulers, as I believe does Linux, but I > think in both cases it's a boot-time option.  It's been too long since I > looked at the early boot order to know if there's anything that handles > linking the loader-provided modules that depends on any scheduler data > structures.  Doing that audit and ensuring that there aren't would be > the first step.  From there, it should be mostly build-system > infrastructure to allow building the two schedulers as modules and > switching between them at boot. > > Allowing switching schedulers after boot time is a much harder problem. > I know some microkernels have done it, but even there (where the > scheduler is a separate service) it's far from easy. > > David I never expected to be able to switch schedulers after boot time, but I assume even switching at boot time would require the scheduler to be a kernel loadable module. -- George From nobody Thu Mar 23 20:45:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjHRt6vq9z40Jl4 for ; Thu, 23 Mar 2023 20:46:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-20.consmr.mail.gq1.yahoo.com (sonic306-20.consmr.mail.gq1.yahoo.com [98.137.68.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjHRs6N2Hz44ZS for ; Thu, 23 Mar 2023 20:46:01 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=c2taMCJ9; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679604359; bh=DMdBXo40IMs3dSk1xBfbCbc9IWUKwr+W3iHC54C8AGQ=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=c2taMCJ9M3aCzu3mdsYBMnY/i1LoYLMY1z7ZeRNBADUeLGB+wZk18bvJZXPPk7GBlClNIoO/3UCevMMNs92z49UZTcrqXyXLGQp+sAeNQ8W2u1SoNYsMF6GIOLo0bVWD2JeIvCBPmBNt/JnFCDXMu5wkMM1mHTHeKRKbFWix1QzdGZEDmNcBI4QHwg7fbGEXvkagN3pQUgE4eCClulEcG6f2LRyQTAY+0RSXNw72BC3O1F1eheT9paumRKqcI+GVwF4J1rY4HbkE03ZWCt9ToTpWrNy5gF7LAyyWtmj/ByBES6B/KGB8Z0iGJ9wthvzyH2yJcyGQYJb0PC/ap0wVWA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679604359; bh=MUIqOjGz+4lZv1IU6UQVBuVtVi9btmW1d4+9O66Lrmq=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=KpOPoCTNSwE+LNPDbb/8jlDncxWnZpDWuCitlGbV2Ue+gVJY3m/4vpUkO0ajLi3qiqWYUyyObHkRqfXW2nJmRqqsbXa52GWwlLbNzF7H96a4vGPRzN+xlrfOTsdDhZjptyaWE+HjJtkIrQTtZhk8P6HvZ3riMbFkCfJH20mbPKT7ILyp2gy7aqTRmw1M+30F4242W0zvkTNcYIut7ZqS+B6+bzdc1v91+txOlfG3BC0tTc3/SdzcBUZZfB3V6p+yFxyVcgC8Mn0V/qD7X/w9efRJzHQnvvX/SZDz1U3/fAqzIlfivWstCqlNTPmphgDohgfL/QEg3u70y8ZXGCpVyg== X-YMail-OSG: xYVs42QVM1m1h5_1DNX73BxLnbVH4Q.Pm2GMidk0hVpZQNILqN0tMajxQsk0Dci WfQ0FY31UfPs5ybiDvFkhi1G4.DKze5z2cykxqG1ZHVs0HIyHgvPYh0BuGA1U8gJe5d_lu0t3B0i L53uoUBeDc4KeiuL8Vjy0qBsGFdYqPjzZNuJgu.JIeLfWXOkTCNphNJV6TZ1lwdmF4c4ebbnTNqJ pF5b15SGmnujba1MgVO7XTm7mrZsMyK7xq6z5sRxbdgcTepH4HCP_ORNlGTrErkXbqmiVHfBQ3HC nJtc8NzMrEY7mKoHOMJXN0AyQZPvPDVQ749Ei7sQe95Vic6O8vF4yV7WzSTuK9GK1RaAKUdMJXq5 ZrDo3GhcxfoIEroSRBTNj6akLoOUp.fUMEOVc5j1cyARm6w3MwojxRNtfq3gdodSfrnV96P4.NiQ wri8N73IHm4aiE1jRG03HcK_oEFFRaIuXrA7.xuMIx0xeuBCFu9f44ejO_fVqkvCU.lO1Qzn8fYk rXQRV1Nc76wo06xuJu24QM4A_3.PeTvscWELPhyDwORNQbMcoXr9hoArnru95FG23HQ2e.3mP6Dy veYH8nKlacKdq0qGtK4fb0O_WY84JrL1lUX.LyjAf9oRHTXsmLgaKduB0XlucBSRY88YhAUW8UCW _lDFjg97j_HllwyL_yHxRBPyZw6L8.HY1DW5cHSCV7VeIGkcxfcaaV2iFKDsWD9hwB56mqTcmu0A g9pm109HuzgENUlMVkBsON38qutwtD6dzGkYWpUaLbDYVnFFf83kzHmXZaJhqn1T0hYhAKH5rzyB WnX7p4z8qee3NO1c2eCfKyNGcRe13iUVxWu8u9KImRcDxjGdQavCcREEWKuO57Lo8hGAeQg_BwUX BlKIdoKs0Af1oY_nrEHPzfZMby3Nwi6c.NRdXAZ03pm1fIxfDVo.VlVhdlqHNFWW_DO1ay1nDlIo bvC18qV4s_Vk4qlrZMubqoI8rZByI5ologD6DpyYv8_BUyJYdMqPY2Pse0e7.T50_ecHIJMsXVb0 o8GA44XPPMfrOLAonS.fgyLdLzuwPBMhoZHUnwt7T1p0G_gssyMZtnmlshGowCgxRZZZ9RP4RxRy 2yH9sVRVtBPAumtZfJuF.5dveglDj27jB1klYsk3i2hEeh8V3uE0hCb1HMW3H6Jq7szckrPmUIcE YAX8RPiQqo_pOf2d5wefsRi8ZVHZ7KSYQGoTPx6m1vunn1SxQKWmwOrXMezhOTNb6ZsfuTKE243X UiV1me20Lf97kSEYrKBqxaC7tWVHd5IseKNrr3iBRiKTb4Msl5YHVwPuCtEhylUCj9JRPtpSvqbC c1HFIKo505RdIKOtVhCT3ibf5GPq2N1e95cE0XBYoimNQoy.de2AWBoxvoEace_WyFRukjoqklfc x2Mv85TxCNfY0xLuKrPCkDVm9P5HpDDzpfDKhghrtcl7jtnR3BgXX6Y.IxgXLYDaEAzj6ynAAzbU SXwFRJXpQ_fMezVYWmpozcU7ocCAPG29eKBK_Dw2BuV68Iy5gZUx_r389ong5LejV4T4oThIU_fK FaazAmv73E8c0uBuBgEyMYHsW5F8nqUGyJgT1OcE75mD5xjDQAL6GA8IaK9E3p.F4gkUeFh2bk.k S6G1gd0XfUq.JzC6ULr6ckhsroSMWWj.Ix64zeyNFY.mbEJNb99hUh1equec41E8b38zeS_75KGc 9d0WGkPSunCybQGTnacuFjeaKmA9AuOEJuwVSTbbFWf_LaHegElNkC_7mAodckhzuL5J0SuceOny E2WC3CxYX9w2wD1mhgFkBFDnqmM6bmQBQj8GSZ91p0YFunY06vQKGAJiRzwZYL7LW54kCPZJtFg3 SN4khn8Z8DrqwEjWskd4K3GUJOmz.F0MVfOhojIXsz_1kp.zB__mfYS5dCCZ7qUyKZeo2VtW80zM J60lw6yFrbg4BJDgM2tHo4Uk.pgpnewFNdbv1Gdp_2kBB6w5ys5Ec0gyQnnP0vFum.Izo6He.tUf d3T4Q6hkMjcSSoQ1UJUson.KK47lqFjMiigAilpw3m_CXqyQVpMDxC45hrZJxhctkP1qqthf1U4I Arh9hJubUuHP3MgwbqamJg3s7rGsUcFl1lEw6cUvcYOs7HKKx2bCR3PhrHIlnA9u8ZUHD3nZh.vl GJQKSfNh_je5H34vocLqK2_t7hs3NoU6dNP0wLHUpaya9.SPia5KB3LE3cN9XvmL.180Fr0nNnGr 2waT_vH.Z52C7xhTnn2_TWfPV_VzBsK.Z5iCM129p1khJYy0jnB5Iy9dma8yxIkw1rk34Q6dveGK tKFQ- X-Sonic-MF: X-Sonic-ID: a73eb290-4d04-4ff5-99fa-c11bceab4094 Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 20:45:59 +0000 Received: by hermes--production-bf1-777648578f-gc4ts (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 84d095bbdeb4f8b54d883fcc3dd583d3; Thu, 23 Mar 2023 20:45:53 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE Message-Id: <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39@yahoo.com> Date: Thu, 23 Mar 2023 13:45:41 -0700 To: Warner Losh , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39.ref@yahoo.com> X-Spamd-Result: default: False [-3.26 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.76)[-0.765]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.83:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PjHRs6N2Hz44ZS X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Warner Losh wrote on Date: Wed, 22 Mar 2023 22:57:08 UTC : > On Wed, Mar 22, 2023 at 1:41=E2=80=AFPM George Mitchell = > wrote: >=20 > > service dnetc start > > I am literally running "make buildworld" with no additional options. > > > > > So what are the results for make buildworld -j $(sysctl -n hw.ncpu )? Note: My experiments have been in this -j $(sysctl -n hw.ncpu ) realm. > ULE scales much better, but when there's too little to do it can make = poor > choices. >=20 > ULE is better locked and don't fall over on high core count systems = like > BSD does at moderate load. (I'm presuming the above is not about the specifics of the effectively different interpretations of the likes of having extra "nice 20" activity by the two schedulers for the examples related to the original "rant", other than the -jN issue.) Any idea on what scale "high core count systems" need to be for what sort of "moderate load" to end up with signficant differences? What sort of context(s) show ULE scaling much better? On the 16 core ThreadRipper 1950X (32 hardware threads) I've really only demonstrated the "nice 20" distinction as significant between the schedulers so far. (I do not have acccess to anything with more hardware threads.) Note: I've not (yet?) been looking at having just a little more than the number of hardware threads active (no nice involvement). =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Mar 23 20:53:25 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjHcf25jcz40JrC for ; Thu, 23 Mar 2023 20:53:38 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjHcd5qsMz469g for ; Thu, 23 Mar 2023 20:53:37 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x534.google.com with SMTP id h8so212064ede.8 for ; Thu, 23 Mar 2023 13:53:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; t=1679604816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=1kmZuGnwDa/Bih5tEQG+siOUPEWXQHkR8OiWKbAoZ5s=; b=I5rMDDJwi5Ik/S2pW+9wYVHp0VgnGHtmRr1oSM5sTXWXdV5RjpQ3HOVv3WBj6DvjT1 R0gB4/Ylbsmg+xylwQ96nqy2u3S8H6DApZltMgudrEQbEKPN+Z61V5x3003ylCw4ejXZ tC51UUe4/46UMbaay3AxWk9gaGlF6OEDfG8cqKzjpy/B4KhPKiTaTq3OTHr1+akiUx+d npiIJ9QfQy+t0DY+PwInBQT/HP38JwMQ/qUM7DxfGzQMPdw+qY9f8RczXpxigg9PnbLo sM9Dbd2pS3EBRQvUIuYfbXdP6m7gYn6tDwi9Xe9ARkYQzQPw/gsqgnpEVVOC07fzqF0j TRPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679604816; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1kmZuGnwDa/Bih5tEQG+siOUPEWXQHkR8OiWKbAoZ5s=; b=EU3P1qRBVj+ly3yKJk/WtAP5VvJLGM0PeRXp/eJmR6Q1HvA0i+SyTVs499m1LFyLD0 QxoJxLkRS8llHFWc9ODMp7Ps1AIqxn71b9XDDV3/ZFTS77N2lHUIVNzQq8xzqlOAekke Bt/4Uh4KEUeEhad+xHOD33hkWgasGnQmqxucoJn+wi2jeboP8T8iQZjRdPvjrgk5z2ep HSTQ9Z3Z8Zig3ri6zCba7amjsaDVW4dGchGAnz48gkePTkl9T8gDk3bwwaBDBcdSeWKZ 3zI0MT3BB/CRJKDZA3iiaZlNxe2YJrl5FGhuQqqMx+j1gYMqCf9ASjYagTT4kGBlvHz6 KXUg== X-Gm-Message-State: AAQBX9eBFhtTxNZRnQbU31MiFG2HTddK0kmyF3Q8qJ2secj8netg+JhQ GL91i6yFRUzb0QaOFZLhJJR8js06UldGhGofTs2lag== X-Google-Smtp-Source: AKy350a52m77qhUHhFElRXVqZHCyO/2ZDIdhbBV7Drj4t5IkIfOAQtRNqVNvHoW8ocxYp+IupAKq58BqzorCxBIz4X4= X-Received: by 2002:a17:906:5fca:b0:930:310:abcf with SMTP id k10-20020a1709065fca00b009300310abcfmr207497ejv.2.1679604816121; Thu, 23 Mar 2023 13:53:36 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39.ref@yahoo.com> <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39@yahoo.com> In-Reply-To: <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39@yahoo.com> From: Warner Losh Date: Thu, 23 Mar 2023 21:53:25 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Mark Millard Cc: FreeBSD Hackers Content-Type: multipart/alternative; boundary="0000000000006da3d505f79778ca" X-Rspamd-Queue-Id: 4PjHcd5qsMz469g X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000006da3d505f79778ca Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Mar 23, 2023, 9:46 PM Mark Millard wrote: > Warner Losh wrote on > Date: Wed, 22 Mar 2023 22:57:08 UTC : > > > On Wed, Mar 22, 2023 at 1:41=E2=80=AFPM George Mitchell > > wrote: > > > > > service dnetc start > > > I am literally running "make buildworld" with no additional options. > > > > > > > > So what are the results for make buildworld -j $(sysctl -n hw.ncpu )? > > > Note: My experiments have been in this -j $(sysctl -n hw.ncpu ) > realm. > > > ULE scales much better, but when there's too little to do it can make > poor > > choices. > > > > ULE is better locked and don't fall over on high core count systems lik= e > > BSD does at moderate load. > > (I'm presuming the above is not about the specifics > of the effectively different interpretations of the > likes of having extra "nice 20" activity by the two > schedulers for the examples related to the original > "rant", other than the -jN issue.) > > Any idea on what scale "high core count systems" need to be > for what sort of "moderate load" to end up with signficant > differences? Sched_bsd is basically unusable on my 64 core 128 thread machine with make -j 150 (nice or no). With ULE I don't notice. That's not to say ule can't be better (me not noticing is hardly scientific), but I tried sched bsd when I got the thread ripper and found the machine too unresponsive when I was doing large builds... But I wasn't playing video on this box... so maybe I hit a local optimal point... Warner What sort of context(s) show ULE scaling much > better? On the 16 core ThreadRipper 1950X (32 hardware > threads) I've really only demonstrated the "nice 20" > distinction as significant between the schedulers so far. > (I do not have acccess to anything with more hardware threads.) > > Note: I've not (yet?) been looking at having just a little > more than the number of hardware threads active (no nice > involvement). > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --0000000000006da3d505f79778ca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


On Thu, Mar 23, 2023, 9:46 PM Mark Millard <marklmi@yahoo.com> wrote:
Warner Losh <imp_at_bsdimp.com= > wrote on
Date: Wed, 22 Mar 2023 22:57:08 UTC :

> On Wed, Mar 22, 2023 at 1:41=E2=80=AFPM George Mitchell <geor= ge+freebsd@m5p.com>
> wrote:
>
> > service dnetc start
> > I am literally running "make buildworld" with no additi= onal options.
> >
> >
> So what are the results for make buildworld -j $(sysctl -n hw.ncpu )?<= br>

Note: My experiments have been in this -j $(sysctl -n hw.ncpu )
realm.

> ULE scales much better, but when there's too little to do it can m= ake poor
> choices.
>
> ULE is better locked and don't fall over on high core count system= s like
> BSD does at moderate load.

(I'm presuming the above is not about the specifics
of the effectively different interpretations of the
likes of having extra "nice 20" activity by the two
schedulers for the examples related to the original
"rant", other than the -jN issue.)

Any idea on what scale "high core count systems" need to be
for what sort of "moderate load" to end up with signficant
differences?

Sched_bsd is basically unusable on my 64 core 128 thread machine = with make -j 150 (nice or no). With ULE I don't notice. That's not = to say ule can't be better (me not noticing is hardly scientific), but = I tried sched bsd when I got the thread ripper and found the machine too un= responsive when I was doing large builds...=C2=A0
But I wasn't playing video on this box... so = maybe I hit a local optimal point...

Warner

What sort of context= (s) show ULE scaling much
better? On the 16 core ThreadRipper 1950X (32 hardware
threads) I've really only demonstrated the "nice 20"
distinction as significant between the schedulers so far.
(I do not have acccess to anything with more hardware threads.)

Note: I've not (yet?) been looking at having just a little
more than the number of hardware threads active (no nice
involvement).

=3D=3D=3D
Mark Millard
marklmi at yahoo.com

--0000000000006da3d505f79778ca-- From nobody Thu Mar 23 20:54:07 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjHdN0Ngkz40JrH for ; Thu, 23 Mar 2023 20:54:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjHdM5j3Dz4765; Thu, 23 Mar 2023 20:54:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.17.1/8.17.1) with ESMTPS id 32NKs9Gw028423 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 23 Mar 2023 22:54:12 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua 32NKs9Gw028423 Received: (from kostik@localhost) by tom.home (8.17.1/8.17.1/Submit) id 32NKs75t028422; Thu, 23 Mar 2023 22:54:07 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 23 Mar 2023 22:54:07 +0200 From: Konstantin Belousov To: Mateusz Guzik Cc: sgk@troutmask.apl.washington.edu, Matthias Andree , freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on tom.home X-Rspamd-Queue-Id: 4PjHdM5j3Dz4765 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Mar 23, 2023 at 05:42:22PM +0100, Mateusz Guzik wrote: > On 3/23/23, Konstantin Belousov wrote: > > On Thu, Mar 23, 2023 at 04:15:21AM +0100, Mateusz Guzik wrote: > >> So I also ran the following setup: 8 core vm doing -j 8 buildkernel, > >> while 8 nice -n 20 processes are cpu-bound. After the build ends > >> workers report how many ops they did in that time. > > Why nice? Did you tried with the idle class instead? > > > > The original message reported numbers with -n 20 and it is not > particularly unusual. I do not think we should target this level of feature compatibility with the historic behavior. nice -20 was not guaranteed to behave the way it is requested in this thread ("only use CPU when no other threads are runnable"). Instead, it is the declared behavior of the idle class. If idle class is broken, then it is indeed should be fixed. But not nice -20, IMO. From nobody Thu Mar 23 20:50:35 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjHmk20DZz40KkN for ; Thu, 23 Mar 2023 21:00:38 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjHmj2wBSz47qC for ; Thu, 23 Mar 2023 21:00:37 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32NL07sG035191 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 22:00:07 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32NL07uN035190 for freebsd-hackers@freebsd.org; Thu, 23 Mar 2023 22:00:07 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32NKp2Q3069445 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK) for ; Thu, 23 Mar 2023 21:51:02 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32NKoZZJ049620 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 21:50:35 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32NKoZMR049619 for freebsd-hackers@freebsd.org; Thu, 23 Mar 2023 21:50:35 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Thu, 23 Mar 2023 21:50:35 +0100 From: Peter To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Thu, 23 Mar 2023 22:00:09 +0100 (CET) X-Spamd-Result: default: False [-2.18 / 15.00]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.88)[-0.881]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sub.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; HAS_XAW(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4PjHmj2wBSz47qC X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N > Yes, you've all heard it before, but I've just reverified on FreeBSD Yes, this is part of our culture: we need this periodic rant every once and again. > 13.1-RELEASE-p7 that SCHED_ULE gives terrible performance for "make > buildworld" in the presence of a totally compute-bound job (misc/dnetc Yes, this is well known, and works-as-designed. The reason is that a totally compute-bound job can only be preempted, it cannot be scheduled. A /mostly/ compute-bound job however does occasional system calls, and at that point it will be scheduled. Then, the design specifics of sched-ULE is that a preempted job will be re-positioned by the scheduler to it's former place, i.e. as said above, it will NOT be scheduled. In other words: it just continues to run until the quantum is empty. And that is the reason why your "totally compute-bound" job gets more compute than the competing one with i/o. You can actually ignore the responses with the naive questions here, because what you observe is what the code says that it has to happen. So, when I ran into this issue many years ago, I just fixed it, and my machines run fine since then. The only problem is: nobody wants the patch. (And no, I don't say that patch would solve all problems, but it did solve mine, and one could start from there.) From nobody Thu Mar 23 21:04:53 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjHsp0trdz40KcQ for ; Thu, 23 Mar 2023 21:05:02 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjHsn3fKfz4ChF for ; Thu, 23 Mar 2023 21:05:01 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32NL4rbV010011 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 17:04:59 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: Date: Thu, 23 Mar 2023 17:04:53 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE To: freebsd-hackers@freebsd.org References: Content-Language: en-US From: George Mitchell In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-3.27 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; NEURAL_HAM_LONG(-0.99)[-0.986]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[m5p.com]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; TAGGED_FROM(0.00)[freebsd] X-Rspamd-Queue-Id: 4PjHsn3fKfz4ChF X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/23/23 16:50, Peter wrote: >>[...] > Yes, this is part of our culture: we need this periodic rant every > once and again. > [...] > So, when I ran into this issue many years ago, I just fixed it, and my > machines run fine since then. > > The only problem is: nobody wants the patch. > > (And no, I don't say that patch would solve all problems, but it did > solve mine, and one could start from there.) I guess I wasn't paying attention, so this is the first I heard of the patch. Is it available somewhere? -- George From nobody Thu Mar 23 20:59:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjHz70qTtz40LGN for ; Thu, 23 Mar 2023 21:09:39 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjHz53cvbz4DJR for ; Thu, 23 Mar 2023 21:09:37 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32NL96QV041570 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 22:09:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32NL96Q8041569 for freebsd-hackers@freebsd.org; Thu, 23 Mar 2023 22:09:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32NL01tx074570 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK) for ; Thu, 23 Mar 2023 22:00:02 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32NKxSu4056103 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 21:59:28 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32NKxSRm056102 for freebsd-hackers@freebsd.org; Thu, 23 Mar 2023 21:59:28 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Thu, 23 Mar 2023 21:59:28 +0100 From: Peter To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Thu, 23 Mar 2023 22:09:08 +0100 (CET) X-Spamd-Result: default: False [2.92 / 15.00]; HFILTER_URL_ONELINE(2.50)[plain:0:1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FAKE_REPLY(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_SHORT(1.00)[0.998]; HFILTER_URL_ONLY(0.73)[0.33035714285714]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[sub.org]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; HAS_XAW(0.00)[]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1] X-Rspamd-Queue-Id: 4PjHz53cvbz4DJR X-Spamd-Bar: ++ X-ThisMailContainsUnwantedMimeParts: N Addendum: https://forums.freebsd.org/threads/kern-sched-quantum-creepy-sadistic-scheduler.65392/post-383621 From nobody Thu Mar 23 21:53:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjK5z5y0Tz40Nrw for ; Thu, 23 Mar 2023 22:00:39 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjK5y66Xqz4Kxf for ; Thu, 23 Mar 2023 22:00:38 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32NM06IY078282 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 23:00:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32NM06Ik078281 for freebsd-hackers@freebsd.org; Thu, 23 Mar 2023 23:00:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32NLs1Hr007707 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK) for ; Thu, 23 Mar 2023 22:54:02 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32NLr2IU058089 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 22:53:02 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32NLr2je058088 for freebsd-hackers@freebsd.org; Thu, 23 Mar 2023 22:53:02 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Thu, 23 Mar 2023 22:53:02 +0100 From: Peter To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Thu, 23 Mar 2023 23:00:08 +0100 (CET) X-Spamd-Result: default: False [-2.06 / 15.00]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.76)[-0.764]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[sub.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; HAS_XAW(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4PjK5y66Xqz4Kxf X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Quoting George Mitchell : > I guess I wasn't paying attention, so this is the first I heard of the > patch. Not Your fault. I'm not running -CURRENT, so I'm usually not here but in -stable, or in the forum. > Is it available somewhere? -- George https://forums.freebsd.org/threads/what-is-sysctl-kern-sched-preempt_thresh.85601/post-572775 From nobody Thu Mar 23 22:06:14 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjKDn3RG5z40PMX for ; Thu, 23 Mar 2023 22:06:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-55.consmr.mail.gq1.yahoo.com (sonic316-55.consmr.mail.gq1.yahoo.com [98.137.69.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjKDm6ZYTz4M5B for ; Thu, 23 Mar 2023 22:06:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679609190; bh=o9S3qdcS9aQat0rgRhxOPvSULl7LgGuDCSajQbuknXk=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=DSSofkhLB3KyXb7aLKnK5mkgG03E8TINsQgIrP1vbpUr9LoqzmqHv0j05UzbO4v1bRLNIe9mV2OTJtU+yB+vKS1vu0/wevCDZwmLV8XjoI3mBiwvNEpcrCO12Zwgb4Z+HnZTw4PS+0IuobJFYS22yfk6tt6X7bhv1EPxePMHLKMAYm96iM0yfMK3b9MPyIYT/S+lq51I12HADgewMVmu7F9mPxrD8u9UpyX33Jv5f+BP6idwY98QXtGTBjj2GjtmunH16dukCqmfyPaLICpNOXkNpsjOtFhcAU8iSmq75UvqT6XCA24T0xPfiG18oTlpuNLJFn+m8UIFcf1HrgkWAQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679609190; bh=4sUvc6GbfBlKbhkZuKiciRaZN47yiTbICKuzUaGfprR=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=lmXMuDBTkQxS8moRrPJJZNCAXSFoo75luPNkqbF+jV/5as8/hY/FnW40Q7e05vngje7adHQWcIpumcYAqbgzUZmPk/L4BYZE0+N19yHHxT66a3DNBD2OQFJ7ur/rmZNL1vIUK7YF4thBW7WOBXPcOkxTGylN0LfGdEe6W9F1lC/aSeMY0ejuh1Byf4zNtdyqfx0hlwqnetNsCS3Aic058uQmaeSjZkOFfOBGfpp0KLSw1gTrzU2kFLfbpoqxxaIiaEzQ3Dn+1xObIE3B75pxMITdyIv8WvFU+LmEfOLFEDtW7E+XFWXrQbsSQgQMdGs84MHtaxIpmQcpplX2U4p+0w== X-YMail-OSG: xMUEmI4VM1ko0QeUII3r2.2H1nNo4OK985S4ApmbHJqCqmk4mb63Q_agm_h5Jpk SNKNIS5URAGK1Wz14l9QcK9Nxtnzqbc86rq6KdBxtbmKu96YTK72xM1UGARr7EDUYzTs8rtsFyva JaAMOaPe8q43ynnzvhg3gVt1JqyC2qnAeqZdvlQz0kNx_IVEsabEXyHyZe4weXokFKAQ4mD8j9O8 uOKECghaC_Je.mTV4eOoHxKUnmRUb7zuJK.9PH1V.xrVtZWKBbnturYGQTWaE3kRZX9PM32MLGwj RSdJYiffzMjxWh1xKTXE_E8FBsKNIX8BSXpbni9jOpCeBQ3gcldrMlqbEvyGws6xRQxNktcz9MzO X1Cl7FqpZHhqeecX.WAhCqhYhYC8RjwQ3RcJ5Gpm3T3C03z7l3cyKrK6P5eSZvAZnPoVSw7E4Mke X9fAnpCB0aDw9FEX_CbX7QXBJQZjcolSi2gY7yxgyQxx6eKYXQt36pJYJ7SCRsifCMM_uUd5H.OC c.DbY_EFfixzXy0Bj4owVS9bP.0Am20i0rQY7ncWzt4dQ.k7jIWBOCXxQo3LqMCCACED0UXK7SjB Rgjax_c8tPLkIU78BdpD_UFeR13uts3Qm9Mc5vWVPlwu2UGXp1UkHgizIcAs50w7czwSc6Z3QnQe RcEBnuPC4zlaphKYx_MPxfAYd5HG8iYAHinfIMXQ.GswHhHrIAh34QhhVc6d6trrjBs.7dY_YKv2 qyvUc_2v5WYZaeTrpHxHwm.yrNFnGL3q3lYBXIQwWKIIbxLHso8lkv66qiAGVCib9Jipnh_y9UGK M2JoP0aKwrCHz48vlXfA7gv4onbWVJMZPFUKxa.eDYNxubwnwUYQ_j1dDiw5f2rXkoLUKQly553A p1LMD7JmG7rnfooBoo3Y6THQYgY2mAw0XVEQfzLujrhGAkf_.8GarUYOUTSsCTA.L5luQMZ_fev. ZqQ_pIgSjNPiGp_xCbUB1cGDoslmZHdgNfB2bi2wJOiMEOz2ah9ydPdwe7NOBBx6G8Jtgw.FlCPc 0kt7v6xf51b28FoHp17eQT17mdKSH7FGJFOKxrIosb7HfRF1RCDyxF6mkgrji0znEfC7sHz4ljhV yikv9Urz81foe7CBkngdBYIOv3kMWTe9ivg1en3hrZ9Irv5xK9OwdZykPf9S_pE..nZyeHe4GJkD VtLzwsNR__knY8bDVqiPfhrq7ke95azQpMrH4WEcyi4QI7l4fiTo0Dp9GVl3_RrZdDb0rMewDFuu soN7qWRhH4SJu0gAgVELYPeusNeZBydIyqA2bq7pUxqoXgbtn3bwiRiUg0qUEaKmCPoxoxyFbxzg wESchkCwbXtYUdQ.CDXXP1yI4SKY0OsBSN_miJbWS80H3nvYeV1jS3c73KUEM597pCCeCHMbEASO sA7bsBVfWITKB3rp6kaeKdntONAYzjhmNnKlr9HgLzNlJiNaM8GvzFcCkUt9_gYx1_J8HlxsNa78 ssFk2pAhGdD.CYf9PzI7ISnrIot7XSxnbEtX5NpHI33fKksG5ydKwzIJcYm4vrPJFpBIG8a9U5wX Tw3ik2fW9dnpTg.WE2aclkhTwAAZz6rcDOBVCpfdC6RGOxmmnO4vXomXR2Wwww8C_DEDHok6pCSP pQXdtB147UkIfkXRgm4pF.chp19QtvdZfuGjy9YBhdcL3Q31FiUCfY5MF8ehC.euNldtMouiOFre gdoaduPYwRo8a3V2mOIJqWsf.Wlmwb2sZv5t4op5LVmaPBkFVvOQOJfd_0kUYxkDy2WHCWRW87jc ELoYNQo8FG7xb2UHxkBFDI3eo6jiwa2nSrDqwtfAfVt9mjBy022rd5MIWz9CcU468R9bgi_rU9TF aSb5fbjTRKF6MdqldTU25YgORkK6psOIvJEwCkr3ybR.nJkqXCy2zsDm7QcBQJEWxeQd8rB5eyh. .sS8Hb.DSaipAVs82ccvLyBADpThGt_xqhpFGx1Nn7ExEJe5riyD9oieykXlsvyKheylsDv21kwM PwzVoPtQSHPlWINqaq2cJBFo.XSAeBk_08q25cpTHuEe0Rg72LiEMFqpuCyMew6shOD1rt9UfYx. J2QXbwxAoMC2ULq4xvrKAHRg6lY2iIvHoSN_aBNKLpUgqbd7o6v7.f2fBc_GDpJlOOvOGGATXv0S DnmERnvNpgG0bx2iSCBR4Gg0a8g03JHvya9SnsBBfoFJtKQUhAKXwCqgj4U7Nz9Gdurzil8Bd0Uu t7.TxjfNSu7FXZnh935KDajIt1670bMgPR7BCH8MrzXE20mHuf0nE5SOUZMWzDjHz6WmrEuzRwUJ d0EZf X-Sonic-MF: X-Sonic-ID: 54ad577e-67b6-4c3b-9df4-753a5a82d7a9 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 22:06:30 +0000 Received: by hermes--production-ne1-759c9b8c64-l4sxl (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID bfb0c18e226a0970b281309809bebda1; Thu, 23 Mar 2023 22:06:26 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Thu, 23 Mar 2023 15:06:14 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <907C8A16-D271-4C32-8DF2-A719CCC76FE0@yahoo.com> References: <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39.ref@yahoo.com> <24F6D88F-3F15-48FB-AA5A-AFD4B77A1D39@yahoo.com> To: Warner Losh X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4PjKDm6ZYTz4M5B X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mar 23, 2023, at 13:53, Warner Losh wrote: >=20 >> On Thu, Mar 23, 2023, 9:46 PM Mark Millard wrote: >> Warner Losh wrote on >> Date: Wed, 22 Mar 2023 22:57:08 UTC : >>=20 >> > On Wed, Mar 22, 2023 at 1:41=E2=80=AFPM George Mitchell = >> > wrote: >> >=20 >> > > service dnetc start >> > > I am literally running "make buildworld" with no additional = options. >> > > >> > > >> > So what are the results for make buildworld -j $(sysctl -n hw.ncpu = )? >>=20 >>=20 >> Note: My experiments have been in this -j $(sysctl -n hw.ncpu ) >> realm. >>=20 >> > ULE scales much better, but when there's too little to do it can = make poor >> > choices. >> >=20 >> > ULE is better locked and don't fall over on high core count systems = like >> > BSD does at moderate load. >>=20 >> (I'm presuming the above is not about the specifics >> of the effectively different interpretations of the >> likes of having extra "nice 20" activity by the two >> schedulers for the examples related to the original >> "rant", other than the -jN issue.) >>=20 >> Any idea on what scale "high core count systems" need to be >> for what sort of "moderate load" to end up with signficant >> differences? >=20 > Sched_bsd is basically unusable on my 64 core 128 thread machine with = make -j 150 (nice or no). So, well beyond what I've access to, even if 32 core 64 thread would also show such issues. > With ULE I don't notice. That's not to say ule can't be better (me not = noticing is hardly scientific), but I tried sched bsd when I got the = thread ripper and found the machine too unresponsive when I was doing = large builds...=20 So the classification is based on responsiveness. Good to know. (I assume you had avoided having interactive processes end up with their kernel stacks swapped out. That is a separate sort of issue.) > But I wasn't playing video on this box... so maybe I hit a local = optimal point... No X11 or such invovled in my context. Mostly ssh over EtherNet. I'm rarely at the (video) console (no serial console). I did not see responsiveness issues in the ssh sessions during my activities with the 2 schedulers. (So I'd not explicitly commented about such at the time.) Thanks for reporting the example. > Warner >=20 >> What sort of context(s) show ULE scaling much >> better? On the 16 core ThreadRipper 1950X (32 hardware >> threads) I've really only demonstrated the "nice 20" >> distinction as significant between the schedulers so far. >> (I do not have acccess to anything with more hardware threads.) >>=20 >> Note: I've not (yet?) been looking at having just a little >> more than the number of hardware threads active (no nice >> involvement). >=20 Side note: In order to be sure to avoid interactive processes ending up with their kernel threads swapped out, I use in /etc/sysctl.conf (which prevents far more from ending up that way): # # Together this pair avoids swapping out the process kernel stacks. # This avoids processes for interacting with the system from being # hung-up by such. vm.swap_enabled=3D0 vm.swap_idle_enabled=3D0 While I have such everywhere, only some machines and a rare type of use on them is actually likely need to above in my context. After adding the above, I've never had the loss of access problem again. But nothing I've done would indicate much about X11 use reasonableness, for example. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Mar 23 23:13:40 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjLkW4n3Hz40Sb8 for ; Thu, 23 Mar 2023 23:13:55 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjLkV2QW2z3CxT for ; Thu, 23 Mar 2023 23:13:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=OSWLOF1R; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679613232; bh=tDJdAUu2dnwxblQIIRMFI9dLe1uF8U+YEx47kIzRpvA=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=OSWLOF1RTE9KJ0QxSzep8YxSpjG6SyflwM78NBsUIDki3JlNc23dZHA82r0x7SrGneXW2G7LMpkEpIr+rlZs033HD/goVWR7y6s97QQQ+RswQduerFc+Y9VugLbt6C5JMwNVmhLkfzwQBkvJuBJILGlNJjLhc9ny3XKo9HnVCEAD8VDrx5E1LlwfWRiu4PJmBgMxzpj29K11qsTyKSCtVSbOxVNUtNn/uU487wvKuJz8mbQtV9zZHoYcU6Roy6+7Zx3JT7vKHV5WK6Foir1RTT3dhJCAzjmbyDFzJzCfeZrzuJLIR5M7vmczZTcSWYRuo1s3uq1yMFMsKsOsS5HXKw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679613232; bh=TB8FRiJPUpUfZL6l0YupYEcmQvbMo8pAXeawtlVt66j=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=sudwpkxShFy0p9gfhD4bV6AhV8oB8DUEqyKaWIbSgqSX4z8y+vGNArh4En1xBTs3OuoGnmIv16OEbhz4qjpL+7FS4Y/Qde9Kxq8H/fduw2McqaV9MGmBYrXwOZY5CwzsappP60SB6KvUEUVjeaSonQ0CIit+TsnwCyuc9sHEUxfTsWpvm3yq4xpBuNzSCoPC8I61lCTcsroVdRq43ThC01qCmoLKAF/xugCvtFHkQoDd9GJ8KG5wvFs6TsBjFipfqF0hdukJP8TTnE1pCQp8tZjKyCedz9nLfTDN9ygZZ2hsy5BGnsMwpplsyj/BqnutL7CTTF2CNER6uzELI9898Q== X-YMail-OSG: q295VLQVM1nsC9GwVWm_d9CzYIEYtAG5.4hjp2i9Sh4DpWzq3VONGz_mEUHBVPR htsyOFByO7O_pjzs.wwJrU..jm6yOLQPAWti8eDgCWP460.XEY6qQVR8gnRTJy4lrRLl3Ia.c8kX J1plU76yZuhv.pNwptt7FCpOQ12ClnuEWKgRV9QwJtgUFA0g3hcGUbLd2a8aXeUSZMnDVRFLVZkE pRkkDhpUKrF.uhJgzwYUa5Tyzc30q6XEonbjitnrwfqLinq3iOsngAgx66HHodu_PzxH.eRK4Fgd BgLkU4wgBsyaA2LCgCeJq_tb7rxqRSMDAuwtKPTXzH4NCV9kOys2bOQ22P2_meESQNuMFRfRVgww Intzm7fISQuF.6COrcFmw.gIQxJXsMO4acK0zSemY1REGS0cPReBUL1H2v._XGYJqMVHNqsPJ7Yx RdUlHp9e3ZxkIWjMYmLX.EnxYtv_pZdWwybaTM8Rj.NYeY2XgfrFhDSnN3ZR7KBLVkf.3lUr2Lcr NdRVKzVx1rJ8i1cyW5CbFW8ah7wb5SkL69yl3_eMG6uOY8APrPuTB4BJSKmOBoRaeRqmczs7ZibE wgxcW7fcTJSIlUMWr.oAjsFsgmIAz7byDcY0UWTWrm35t_2nYOwgJvALEpc0CklDdgK8dBabQb0G JJ6up8SVu.mO5TOkcymbydNNEiGCRaMgQ4YaQslWvBYdAmlUj7TIKQCuAkVWPbiKND4FaWHqpJdR 91nm9CTY5UgT6DU51DeGAygffIh7UjfHkYWdoWqBu73s686o_wy7THMozBUO4LrfBzyrUF7.QhH4 wBEgcea8bBS04FsXUJM6qEToAWcJkx7brrPmPkQH.NMJ558OancCn5xoZQ0S5t7mtbZ6V7dIhm.S b67hKDB8uz_Fed0iPZ.VyR.zOmMTK.i.VGqTxvgBvqpeGi.__z_WC59Z5FxwC6RlkVzZBpMmoW.J AhCOmb9F7vBOBkovojLgn69G85zG5VFLBj_qjvzLTAdpeqLUvjHxVZ9l16wX5PMf9zSxDEhJclaH 4moVHGCPGdd5SMCkFskBsi19ZldyFo35iyVfMHayc8CNRHZQz_7b9hHL4X5mSL_hMY33NvhF9vRK QdzuNhiFoRAK_HyjUIzP.T2590ZsOIGL8lDJU9rhTxf_4_dJmlRjoR9x2.Z5laKEQqE1aZ.BCTAj V8FcXwSC9p9IRdacmVixf6S1aZqVreEAVBr.OFTh9LQT0kRjPZPniE4Le.ps2tRYujcxjgz.0gBw xJcugy0HB3609W8XdGL8OyobpXboGpWTSZv.dJsy9tNJKlAxFefXNJQzwAvJELSzS9QL8Pt8yqEd cXkzfn2hoEqQpHmZWbAS7ED5QoLjTkvxfLGwBvo4ZILiYqP1JOJpGq6u.2.L9z0u42vwnVDi9QwC LhwPZJq8ApRIt_aMVgaD0MQ93xVzuUpPyM9QFZKOZUpVAtH_rXhOCKrHAmwpOzUgFoBjXCrkQglQ dtG14Y71HSBPzYGCD78RSzUK37FJi0Dqq5FaJmeCOJtwf9rrtLkk5nKhM4QQX6CP9OSj5TWbZAFq TZ4C4q9Ist06HPFX.Fa433LQxxKP5VFHuiT1p6MqY30qHA8FdCeKOtwd2OZzhbInQRriXd64Ojfk Zk8ukMWH58lvhWF78mFjVqh.gvCivjWXx4eCHpJ0B0FGQvUavZbC7AlDM5d8zcEaCoimYUPXenLo khnrc14fjL4Ri9zI_01Ni1xwMrBUImVF_MpMczRMcpRM.CmU_hZ_keuYZ5wxEXWzC_0Iol3V19kq gtZXdAJAwe8pdnITwoXohTovKz1r3oHfEsCrG_4x7WvYcsBSc1jLeQmXCcJOrDzBgU9RcZHaVcop 72WJKl64.gED3ltihNZHxav42EW3MRyXG1avyeYyg.tZRl2LBR0Muq4oxbAsGkd8KeowVDh7Vv9m 40qW0En6eB5_Gjqt8MDVtwuiTKL0l61rVgbm0vqkxZfV274sT28MpWAxnqGS7yeyeznE.ioe25Zf h46_Bc1JtSwH7xLZivk0yncViAps0DBYoItvYOJCZBfM0jaV9b5uKyqDFO7F2xm5ite.cmLTJhS3 MKkpMH.t8Wk8jArpFI.BEw9NCs8u.QJEHcEFBHxMNdkv3IDqYCgt7Wx_PDY1HIDwwZErPHiPd2M4 MrAhKwCGJyJh.ftPulBylpS9dIFETsAJSQodoV_kTvHFWY47Z1GSwi6Pn.fDUbo6sOH9s0MtpIvO vlRp2DjjCfTbHa9uZQSuTk0kFIV0mewN6gmLP3XeWnet9Hyo1kB965vo56YBg4IOZVD2QB0J.BYd WN_.DuQ-- X-Sonic-MF: X-Sonic-ID: 8101db68-9789-4273-931c-c4f5dd8fdecf Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Thu, 23 Mar 2023 23:13:52 +0000 Received: by hermes--production-gq1-6cf7749bc8-g5z7v (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7ef8b39bef906872d51cc99c7e39870f; Thu, 23 Mar 2023 23:13:51 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE Message-Id: Date: Thu, 23 Mar 2023 16:13:40 -0700 To: Konstantin Belousov , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.147:from]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.147:from] X-Rspamd-Queue-Id: 4PjLkV2QW2z3CxT X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Konstantin Belousov wrote on Date: Thu, 23 Mar 2023 20:54:07 UTC : > On Thu, Mar 23, 2023 at 05:42:22PM +0100, Mateusz Guzik wrote: > > On 3/23/23, Konstantin Belousov wrote: > > > On Thu, Mar 23, 2023 at 04:15:21AM +0100, Mateusz Guzik wrote: > > >> So I also ran the following setup: 8 core vm doing -j 8 buildkernel, > > >> while 8 nice -n 20 processes are cpu-bound. After the build ends > > >> workers report how many ops they did in that time. > > > Why nice? Did you tried with the idle class instead? > > > > > > > The original message reported numbers with -n 20 and it is not > > particularly unusual. > I do not think we should target this level of feature compatibility with > the historic behavior. nice -20 was not guaranteed to behave the way it > is requested in this thread ("only use CPU when no other threads are > runnable"). Instead, it is the declared behavior of the idle class. How common is the "idle class" in BSD's, Unix, Linux, etc.? Is it something rather FreeBSD specific for the way code must be set up? (I'm not claiming that the "only use CPU when no other threads are runnable" is the proper goal for "nice". But the sizable variability in the consequences of using nice in the 2 schedulers could of itself be an issue.) > If idle class is broken, then it is indeed should be fixed. But not > nice -20, IMO. I end up wondering how much this would mean ports and such would need more FreeBSD-specific changes. === Mark Millard marklmi at yahoo.com From nobody Fri Mar 24 00:00:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjMm40x6fz40WB0; Fri, 24 Mar 2023 00:00:20 +0000 (UTC) (envelope-from salvadore@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjMm36j2Yz3H80; Fri, 24 Mar 2023 00:00:19 +0000 (UTC) (envelope-from salvadore@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679616020; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=KQAxqWMqZrB0jEKZuxIumU+/mMEYEEoy0MRQtp125bE=; b=p1rzsxvDY2mGkxKxSK0OLwvwA20u+e6feLgYzDoVLvBR3JgAPP0n9E98im8h5CBccQBmWs RfGwzRjShl/tu9X5tcamFrjnCxzlsbEYUeziRDy1GVfzMlYNiKammf9KuPTQwV/T+STx6c 1wjuz2ZRnYLpTWA3JA4TjT0N3/Ncfw8myMJ877jUk2kWFabb8yTCMIbWt0OAOBWPiRvVqE seFXBVkxmInNDcDkmmxpIvA/4CWW1QgLsC+XKH9+yQi2VoDmVAqOdyDi1k3JkKwSnN08Oq 6+TLkbCbEb+ovCD6d7DWPKIjYTIdRb1mw9Hz/wl3Y11uwuvcYHe7T0jt5MDQzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679616020; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=KQAxqWMqZrB0jEKZuxIumU+/mMEYEEoy0MRQtp125bE=; b=TLUZ9gOZJgXGdr1nvRLc9n/ZMX83wUPgG4d2Wd2hXIum0T+YwJDv7zmTUGizO984j6ozxj A9gvPehGPGhGN7tEojQOhtFCP3zqfc2/v9RZXXy/6lq+6z6DoDc3jEA7Z+X0bKrAe3AF2o E9ys/LZW4SWVJOIH3cKALLeyA7vt9l0lNxvtjEqQ5IYF2vNI1Zcy4bhnWi3j1NxPnCKfCE JImq19y/vYVPbdHwlIDv9te753gsO7q4qnyEbnEc0hi66DCSzzv+z9R96XcIrDe9zXqU7x qgKHAQwF2MCuRVaLBblpSJXwKCir/4BBMacG1VYBYy1zAGmC2N9xaSzTi1dKKA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679616020; a=rsa-sha256; cv=none; b=U0TBAjrS8PCVKgCF8Kzj4OFZBUDI+FGFWEjwdtmNBlV1rm5u7r8tHaVTNcGfYaOOfAGwKP 3Ta5oTIpjBHQQnwcr218cZNa9rWKdPd40EwG82EaPLzOL114ceuvSoxSXqGlitvYiePRbz FuF+rUSql0Q8JxfZmQH8DzjZfb9i624Vp00n5kZ/Mttuta/2YcKUA2mq27jYvs93Q7PQuD 3PS1pDEY4ABq+hJ4fjibmkz/eKCSoqWUJuYcojG+f4KckzfMtM2mezT3xXiwnyA86FOwtP ErNNA8zgd4zc91EmDuyT3/geaODwa47usvRQKweVrzg3kposd5mBi87SVBk4PQ== Received: by freefall.freebsd.org (Postfix, from userid 1472) id CF6F112A47; Fri, 24 Mar 2023 00:00:19 +0000 (UTC) To: freebsd-status-calls@FreeBSD.org Subject: [LAST OFFICIAL REMINDER] Call for 2023Q1 status reports Cc: freebsd-current@FreeBSD.org,freebsd-hackers@FreeBSD.org,devsummit@FreeBSD.org,secretary@asiabsdcon.org Message-Id: <20230324000019.CF6F112A47@freefall.freebsd.org> Date: Fri, 24 Mar 2023 00:00:19 +0000 (UTC) From: Lorenzo Salvadore X-ThisMailContainsUnwantedMimeParts: N List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Dear FreeBSD Community, The deadline for the next FreeBSD Status Report update is March, 31st 2023 for work done since the last round of quarterly reports: January 2023 - March 2023. I would like to remind you that reports are published on a quarterly basis and are usually collected during the last month of each quarter, You are also welcome to submit them even earlier if you want, and the earlier you submit them, the more time we have for reviewing. Status report submissions do not need to be very long. They may be about anything happening in the FreeBSD project and community, and they provide a great way to inform FreeBSD users and developers about work that is underway or has been completed. Report submissions are not limited to committers; anyone doing anything interesting and FreeBSD related can -- and should -- write one! The following methods are available to submit your reports: * submit a review on Phabricator and add the group "status" to the reviewers list. You should put your reports in the directory doc/website/content/en/status/report-2023-01-2023-03/ (create it if it is missing); * submit a pull request at https://github.com/freebsd/freebsd-doc . You should put your reports in the directory doc/website/content/en/status/report-2023-01-2023-03/ (create it if it is missing); * send an email to status-submissions@FreeBSD.org including your report. An AsciiDoctor template is available at https://www.FreeBSD.org/status/report-sample.adoc . We look forward to seeing your 2023Q1 reports! Thanks, Lorenzo Salvadore (on behalf of status@) From nobody Fri Mar 24 00:02:14 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjMpR3Hr7z40mBL for ; Fri, 24 Mar 2023 00:02:23 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjMpQ611yz3KBs for ; Fri, 24 Mar 2023 00:02:22 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32O02FBr010943 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Thu, 23 Mar 2023 20:02:20 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: <5dc78659-4968-a871-ec69-0aa2f31efa25@m5p.com> Date: Thu, 23 Mar 2023 20:02:14 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE To: freebsd-hackers@freebsd.org References: Content-Language: en-US From: George Mitchell In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-1.52 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-0.97)[-0.966]; NEURAL_SPAM_SHORT(0.74)[0.741]; R_SPF_ALLOW(-0.20)[+a:c]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DMARC_NA(0.00)[m5p.com]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; TAGGED_FROM(0.00)[freebsd] X-Rspamd-Queue-Id: 4PjMpQ611yz3KBs X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N On 3/23/23 17:53, Peter wrote: > Quoting George Mitchell : > >> I guess I wasn't paying attention, so this is the first I heard of the >> patch. > > Not Your fault. I'm not running -CURRENT, so I'm usually not here > but in -stable, or in the forum. > >> Is it available somewhere? -- George > > https://forums.freebsd.org/threads/what-is-sysctl-kern-sched-preempt_thresh.85601/post-572775 > Thank you! -- George From nobody Fri Mar 24 01:13:09 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjPNB00V8z40qvY for ; Fri, 24 Mar 2023 01:13:14 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjPN86Fz5z3jtT for ; Fri, 24 Mar 2023 01:13:12 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b=UktEaKlG; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=BjH2zVtB; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.28 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 4D6BD5C00AC for ; Thu, 23 Mar 2023 21:13:12 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Thu, 23 Mar 2023 21:13:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:sender:subject:subject:to:to; s=fm2; t=1679620392; x=1679706792; bh=6M9r46pEMl7lE3n58Doi+7Y4o imKhcbEWGCHBGOh8P8=; b=UktEaKlGbe11Pqk4wplpQ9M630QqsnQct1X914ocs RLDncmvOSC3DT842GGKYfDI3yRB3hxMZ6TslRyF5T82zocfyNZA0uOvEFbqOhfih IvAeYR5cxCbOGxuVolvGyts9JiTxMkt8tbTERPLc5JpZb+W+ubi1AX2gomrggC/y W6q+CkuWgiu4mEE+1AvnomaVn/GbtDZZAU7lVVSduBIyh5DlNJLbglQ/WoEy4bIP kbz9Rtpc9RuOr6C79TGuHJhW7Ql9Cn43Zb9MtiGjb1lRXoP4I4hju4k5k6Ulb2XU jDfvB8alFfcUAdmjOx9DQOEscCa0MIlrWJls+e+4DAmXA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1679620392; x=1679706792; bh=6M9r46pEMl7lE3n58Doi+7Y4oimKhcbEWGC HBGOh8P8=; b=BjH2zVtBO5SXsjnedg4wrCwdnz6/T5HSXKy2BxXGJH2u4NHPnYJ +maN5Cdvg9agS8Hm0BISVeZLDBz4g4KA3/Xrm97g+DZGylM+lmhAs1SoJyNjDzqK G6CEcUIz0PrA5K5XTcRemMxrEYGdWVJ6DGEnb97/ZuVYjDDE71sPgadhD3YgZP2o n0pll/old19CctoPLQqMADUTboo0vFAlCqicq46i8wWyqIGGoOjI6hd6KAwP5wln lFdfEeB8hdiI0WwRenAqgqgXtboZ5CtJlIG5mBkNTFdJ8Q2z4G2ZSQfl5faic1Ds H76qiJnIjLhHlpXbYTh3mBp7JQaRziTyS/Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeghedgfeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkgggtugesthdtredttd dtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgrthht vghrnhepveduffeivdfffffghfegfeejfefftdeiteehteekfefhvdefgfettdeuheegff eunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepvhho ihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 23 Mar 2023 21:13:11 -0400 (EDT) Date: Fri, 24 Mar 2023 01:13:09 +0000 From: void To: freebsd-hackers@freebsd.org Subject: poudriere, zfs and atime Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Spamd-Result: default: False [-4.50 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.80)[-0.795]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.28:from]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4PjPN86Fz5z3jtT X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N Is it ok to set atime=off on filesystems used by things updating frequently, in this context it's git & poudriere on zfs? I know /var/mail needs it to be on, not sure about git & poudriere. thanks, -- From nobody Fri Mar 24 01:33:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjPqy1S4Gz40sJP for ; Fri, 24 Mar 2023 01:33:50 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjPqy0p4Yz3lsZ; Fri, 24 Mar 2023 01:33:50 +0000 (UTC) (envelope-from zlei@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679621630; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fGg4jS0xg0e9U6Vb2ikv315gQdLK/n0qzmxyc8FNtCs=; b=yoEr6l7vJdreOFotUT/hNyR6881UD4rrWduBpcemKILLdlLBcOwHrXbVb0MqAafIDZf0FB kdx8F6YjhqgpsT151WeZa4zwkJY2UyjXM7XSTHvokm+pdUeGRCGCYs36q8fgR2oyqpjxFN meKvLoULyQCV7G4AGmNvtOxl3zt1aJYeDRq0MDH6J5EQ5JVZTFRKPFUzw6nGKFIre5dGJp VUkje9Nky2/wTwcEWFocX62ZiGXA7dJVQOrGr5//jK3jz0XOMJ67uOpx6k3W+zkLGZaKlV xvHegxbzJSa8rEhXjbw3LMGFFreT6xlBGAURWJLqUlSTvOuztbl4i1oLJWfOMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679621630; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fGg4jS0xg0e9U6Vb2ikv315gQdLK/n0qzmxyc8FNtCs=; b=Gd8zAvOd78IWPk17rGXHrnY7HCmw+FdcfJw2udzMK7biz1gsy85zansH8dDiBDCJO/sSXO bBbCI3759wdNASbDhcJrsMELqrpJTLUsImPy/8ZW8s/ZsWA5gj/muIxQ3MpnHfQSwab/af qqPddfYYPZZv8eukD10FIbZJslUcp8Kj9/96yOc7LaN9zIJ2bNrKy5IbiTZIGGr1IaUN3n IDowe5UtGxO0+pkovbdkFLiXmXt59+GrlTnpbfs9SjmV49eJ/YuFUwHgA5DXjGqa7I6qN+ mhByLFd6/K1CSybsF47bm0gDv0wsqHxkctCa1A34V1mVad5qElwn4/nGj/v2zg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679621630; a=rsa-sha256; cv=none; b=FgCTxi9zE0fs0tvSUUUxoMeceiDmh664XRGhBfY8ALDHxA6jNWy7QCplQYu3++Ko48UC8X hQVEhLh/qgAW0iWiQhGLHPGfeOhQOHbmeYrrMVlNaRGppMoyCYF5l8d2K/JN0koJLtvpyV M/cnZu+E+S3s0NbJrC9biCr+yXhFksKHjb7NTgRYAGUcrj6iaelwR/RlfMfnPh29Tm3xQo CGfgEaczGNpmLuMEgJFIJ2mN5mLmBt9T+5jm6+DBOnja7dsYl4g2xDgvIhqVcMYW47HPkf uFg2eUeyAFT90wZJ+JxnZ5ScOYGVSRDRBu84+xncmSBiHcpiA1Cxj73tZschZg== Received: from smtpclient.apple (ns1.oxydns.net [45.32.91.63]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: zlei/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PjPqw4t0VzmtM; Fri, 24 Mar 2023 01:33:48 +0000 (UTC) (envelope-from zlei@FreeBSD.org) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.2\)) Subject: Re: poudriere, zfs and atime From: Zhenlei Huang In-Reply-To: Date: Fri, 24 Mar 2023 09:33:02 +0800 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: 7bit Message-Id: <4FC801C2-038A-4ACC-A1AD-F6B91D9C1246@FreeBSD.org> References: To: void X-Mailer: Apple Mail (2.3696.120.41.1.2) X-ThisMailContainsUnwantedMimeParts: N > On Mar 24, 2023, at 9:13 AM, void wrote: > > Is it ok to set atime=off on filesystems used by things updating > frequently, in this context it's git & poudriere on zfs? I know > /var/mail needs it to be on, not sure about git & poudriere. > thanks, > -- > `atime` on my build machine: zroot atime off local zroot/ROOT atime off inherited from zroot zroot/ROOT/default atime off inherited from zroot zroot/tmp atime off inherited from zroot zroot/usr atime off inherited from zroot zroot/usr/home atime off inherited from zroot zroot/usr/ports atime off inherited from zroot zroot/usr/src atime off inherited from zroot zroot/var atime off inherited from zroot zroot/var/audit atime off inherited from zroot zroot/var/crash atime off inherited from zroot zroot/var/log atime off inherited from zroot zroot/var/mail atime on local zroot/var/tmp atime off inherited from zroot I think is OK to turn off `atime` for git usage. I've never encountered any problem with if off during daily build (mainly kernel build). I'm not familiar with poudriere. Do you have any issues with it? Best regards, Zhenlei From nobody Fri Mar 24 01:51:12 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PjQD42l26z40sjf for ; Fri, 24 Mar 2023 01:51:16 +0000 (UTC) (envelope-from void@f-m.fm) Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PjQD34m3Zz3nZM for ; Fri, 24 Mar 2023 01:51:15 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm2 header.b="r48psl/e"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=mtvoVx6Y; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 66.111.4.28 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 52CF95C0129 for ; Thu, 23 Mar 2023 21:51:15 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Thu, 23 Mar 2023 21:51:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm2; t=1679622675; x=1679709075; bh=ph XH0/1cW4E15MqEYcVTzK4Sp3oQPZpwH5dBIGZ29tQ=; b=r48psl/eARM6LtTD6K XLNG7ag9ZcdYJsWTH4ROzlxhke2zlWe9lz0QECiQ8JRtv+2LPmCQMbHB3zO72ryP ADOUW7wCj9oa/B1tCjIpA3RX70u8MGLb1U+chrWwbbmZlKKl/6MN0A86EhoqKZ7q ++48VIQ5wAJc2BFbeAQZetG+uoz7WuIJwdxevibqoMhC9qovjZKR+eB/XgOUXtlV IjTlDGm4iZNM/nNgTdmty4DqTEbV7zYChpOFYvwjmY6Fuyb0oqmRcmF4qT6qaXZ4 zafYuWHn7b1oWFloWIvPoHcW/FPOocwiKPd5Fp4yeC2dr+k11m0B6MqwMbAv5DyB S59Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; t=1679622675; x=1679709075; bh=phXH0/1cW4E15 MqEYcVTzK4Sp3oQPZpwH5dBIGZ29tQ=; b=mtvoVx6YlITp9Lbe6j3IhYehgPsOj GQ56gm+IEnuPyhlbBma+ypeOnkpa9wH4zBang7pvnuTYkgNuphaBW+6QDEnwonar uo1wR7Aj91s9TPAflrDxe3JnA6wg7rHN7Lik+wbcaLxIXcRINZr9cHIRGvKht0jH 3dn+BaSLD3GPoZ4o3Q9Qc3K5cpuJH/+sx2rvxUkkMu58RKBYzCr8S//YZ1lifKBN G4FTFoDzB/GigygyLSxJwePOjEt3RsMuUn88OOy+ZF/axRaU1ySZY2FHQ7j25QuU 8cmz3N00nKdupPBb+2KsL51BF5jIR7v+zTCm5dPW5eTN6RkkRA4RPs1Qw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeghedggedtucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomhepvhhoihguuceovhhoihgusehfqdhmrdhfmheqnecuggftrfgr thhtvghrnhepkeeluddvlefhieelfefggffhffektdehleelgfdugfdvgeekjeejuddthe ehgfeunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhep vhhoihgusehfqdhmrdhfmh X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 23 Mar 2023 21:51:14 -0400 (EDT) Date: Fri, 24 Mar 2023 01:51:12 +0000 From: void To: freebsd-hackers@freebsd.org Subject: Re: poudriere, zfs and atime Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <4FC801C2-038A-4ACC-A1AD-F6B91D9C1246@FreeBSD.org> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <4FC801C2-038A-4ACC-A1AD-F6B91D9C1246@FreeBSD.org> X-Spamd-Result: default: False [-4.51 / 15.00]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.81)[-0.811]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm2,messagingengine.com:s=fm2]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.28:c]; MIME_GOOD(-0.10)[text/plain]; RWL_MAILSPIKE_GOOD(-0.10)[66.111.4.28:from]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.28:from]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4PjQD34m3Zz3nZM X-Spamd-Bar: ---- X-ThisMailContainsUnwantedMimeParts: N On Fri, Mar 24, 2023 at 09:33:02AM +0800, Zhenlei Huang wrote: >I think is OK to turn off `atime` for git usage. I've never encountered >any problem with if off during daily build (mainly kernel build). > >I'm not familiar with poudriere. Do you have any issues with it? No issues with it, it's just on an arch with limited resources and I'm trying to give it every advantage. I know that if atime is on a dataset, that zfs will be slower. I didn't know if git needed it or not. thanks, -- From nobody Fri Mar 24 19:41:15 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pjsyk6mdRz411my for ; Fri, 24 Mar 2023 19:41:18 +0000 (UTC) (envelope-from obiwac@gmail.com) Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pjsyk0PcKz3FPQ for ; Fri, 24 Mar 2023 19:41:18 +0000 (UTC) (envelope-from obiwac@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=lf19jlga; spf=pass (mx1.freebsd.org: domain of obiwac@gmail.com designates 2a00:1450:4864:20::534 as permitted sender) smtp.mailfrom=obiwac@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x534.google.com with SMTP id x3so11972242edb.10 for ; Fri, 24 Mar 2023 12:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679686876; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=kIaLjVRJyVJ+LlL8n0GA9DZw3bN56leUY6djDqkEKsM=; b=lf19jlgaH+SEtgbWJvOCx/4heRpfI39cOr0Z+jWKhaoVhqGuYxASfWVdShEzOoRvph itMqP153nzpBgqFDdWwta2xl4MhLd6lcQ5Bl6f6Arx5GefTSzQphyuCupbxmAb9tBv4Y f4L+Zrlq+keZo3ldlIBnXm0i7cT1GPcsuez4hsLnMC3mEHcIPu7wnHPqR5Wi5sedZvV4 fKHdCoQrwUN5YlWJk5rhuHx4YC/v5nKdAAKG7y5GA0YrTSjX2v/N5FGxBCEpbggrLaCA 72s3+HKEQye0sh+rWBygUvHrvCVhAdSLxmlWZaZgBJA8TTtx6bSJbCvnZXzGKBKJYA4/ QThA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679686876; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kIaLjVRJyVJ+LlL8n0GA9DZw3bN56leUY6djDqkEKsM=; b=jv4qsdbUSxjZWr01EAoFyA1THh8A2ALkqcQs5dqT7xpas35FmXwUi0buJb9sV+CJik BzDHiFzP0WirdhT8bYg+cxo7IMHbLKPxG4a8T41WE2xTaXM2carYuSuHjxMsPTQPQDQH hGBHuNDXDODpXRhGkSEPYkIoUe2amCLVSKaD2keYAUZ/WyKNh4CqnlBdZ2q/CeWzrPy7 CwpwKiJ6CgMh8V8Nv5W16tlJJclUXUKWsYdA/qiOtFR0BVkQdGfiRdtxNIDEG7Altw8D FWTIE8C8Jc1RCVyWEbR76/cwqqcdnCcuddEJ1tpDhOH8tZMvc5rmeDoBf9GG5oj927M7 IeEg== X-Gm-Message-State: AAQBX9dvbnXnD7i85y+Isv/DlBv886dpBFT3ZyVPDQ4YezyAP/5dW2XH AVhGKaRV0XzElg9Tx4fM7CuzYkURDfGj5sSMLvRWq3L+ X-Google-Smtp-Source: AKy350YCZ96zUSyEvQaF26rfltNFg3loDBUp8r5dW4slwptvs1v/qotex1bSZgHetxV4NYq4VRzcMGhPo7B0PkmavTg= X-Received: by 2002:a50:a444:0:b0:500:5463:35de with SMTP id v4-20020a50a444000000b00500546335demr2115875edb.8.1679686876381; Fri, 24 Mar 2023 12:41:16 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a17:907:d311:b0:8e7:7343:cad1 with HTTP; Fri, 24 Mar 2023 12:41:15 -0700 (PDT) From: obiwac Date: Fri, 24 Mar 2023 19:41:15 +0000 Message-ID: Subject: net80211 association failures in station mode To: freebsd-hackers Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.48 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.48)[-0.485]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::534:from]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; ARC_NA(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROMTLD(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Pjsyk0PcKz3FPQ X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Hey, Recently I've been getting a lot of association failures with wpa_supplicant(8). I looked into things a little and found that the issue comes when iterating over the sta scan table with mlmelookup (sys/net80211/ieee80211_ioctl.c) in setmlme_assoc_sta, which first checks if the MAC addresses are equal between the query and a scan entry, and then checks if the SSID's are equal. (Minor issue but the "Match mac address and *any* ssid." comment above is wrong, right?) Thing is that query used when calling setmlme_assoc_sta (in ieee80211_ioctl_setmlme) is the vap's SSID (vap->iv_des_ssid[0]), whereas the one used when calling setmlme_assoc_adhoc for adhoc networks is the SSID contained in the MLME request structure (mlme.im_ssid): static int ieee80211_ioctl_setmlme(struct ieee80211vap *vap, struct ieee80211req *ireq) { struct ieee80211req_mlme mlme; ... int error = copyin(ireq->i_data, &mlme, sizeof(mlme)); ... if (vap->iv_opmode == IEEE80211_M_STA && mlme.im_op == IEEE80211_MLME_ASSOC) return setmlme_assoc_sta(vap, mlme.im_macaddr, vap->iv_des_ssid[0].len, vap->iv_des_ssid[0].ssid); else if ((vap->iv_opmode == IEEE80211_M_IBSS || vap->iv_opmode == IEEE80211_M_AHDEMO) && mlme.im_op == IEEE80211_MLME_ASSOC) return setmlme_assoc_adhoc(vap, mlme.im_macaddr, mlme.im_ssid_len, mlme.im_ssid); ... } I would have expected these SSID arguments be the other way around, because e.g. when wpa_supplicant(8) tries to associate in station mode, it sets the wanted SSID in the MLME request structure and then calls IEEE802_IOC_MLME - in fact, when reversing these arguments, I can associate no problem and things operate as I'd expect again... But since this code has last been touched over a decade ago, I feel like there's something I'm missing/doing wrong here :P If this is indeed wrong I'll make a diff :) Thanks in advance, Aymeric From nobody Fri Mar 24 19:47:08 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pjt5s25dlz412r9 for ; Fri, 24 Mar 2023 19:47:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-55.consmr.mail.gq1.yahoo.com (sonic308-55.consmr.mail.gq1.yahoo.com [98.137.68.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pjt5r0MT9z3H11 for ; Fri, 24 Mar 2023 19:47:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=SBV6cF3s; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679687246; bh=DwXwyxxskJyWupENkNVo0mByS+/KjOwwHwEkZzWuCwM=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=SBV6cF3sYY2XcV8WIA/P5RxPHvxVx8tmEq+yXjLLwtZP4dI38xqOJbm9VjPc0x9tprx556pfzVAvYAuvK9iRIrFcRNyBnKW30sSOnaUXyYHDf0r+BrtiT1JRZEYFK7uYF2Fsk+0iIujxemY+1Blqeh3abEWeCixPJkeXlr6+WRrYxENiJHMGdlSmc0asXv6jb5b9dWNPHJ8hSwqXtop+AchFW7MTc3pywOqPeKjiIETMz0UJKIITjtSga/Ydi1PXiVv7xJZlFxxCDIo1JgBW3yO2FP4/zOEs04Dl1DaaiYjjkASUvx3HbhHIMMx8s9TqIZ5SCdYOidSQXPnIRldL5w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679687246; bh=D+uMj/4+Om5O4gLKi1BJJKqOTHejYSu6Iricmaligcu=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Tkeou6Q9P0VBsWkGCv/lBAvKw/FRzfY+NWlJNttzdHbB/gMnuHQ41EE2Osyzp1qce7LHczO4RQB1O3KUQW3tycIlCPuHpRMqXUub1rCmfFgPp720RPO6aflzoYhbVRA89ozz8QhbQtozqXDmnQCcNYE+6gq6vquxk1a5SI2iNwlCD/cX66ghtvatljnmVOq7gy3WVqDe3K1y2oJ57/g6txd4Y6bAYMcn5KYzB1ilhyWxpFWaeFn0R1j39MVNkHcgu06ySQKn2yHER53xMxZ9xsr07RzlXLQ6LusBFPs4NfzClVYxemJ+xAekpHzFP5XD3KY9qd8L4FeZ4NFed9jhgQ== X-YMail-OSG: iwTPm5YVM1n9GL7nHc5yzzyOBE1OgU43sWDEnEKaQXhTwh9CGlga.4tmsI231T9 1PVrR7qT348Iwta0FqGImPHsFI9vdBAOCmIQrM.nBqPZ5c6NKfasdI4au_nXBoCcbHKaDmxHmdCG YZI_Hbdy4.evAOpEa1Cx4pRiKO27THhYM4d.yJ9aJnp44jChfZt.CxlOBP7o.HR4ksb97BCTLmjE J4eZSzhBbdN0NX0xCACuS9PsIyRrrZxTrcpcNLvY6tl3DvEjbUgx2Hgfy6ZngvH4ojEVL4rMSFkG o_q.sONfXD4IRcpjAq_H9c_6a6vd5Squ0yzIWpK0qZiq2iexebbusc5uVZYW2kwEnp0rXKG1oMcX WjKC6MBYMUXy5HqL8QOUB8Wfe3nY.p60yRyMQWPmuIZW0tpDpNyBMpylyPVpw4JKzvHXztROOKCx 857hSZOiPcpuxfX3MvcyyEUUzorV4M.9WmoOse6JcjRQUJxxWTZYSMrWmaQxuDguoYbmhcH5eBu1 ajs_Zv7K1_Y9fA.dvaTpLVPyEtt.xGMRgszvjGk7aPU8_XycqfPv6cKwVEXhEnmqMgJcZd4IsRFb 7yySCGlELFNMPKUa5TJfUUGhdt95pzwy6xGfvr7AWWkKs15C0hpcR44NMhZmjI.JAt8d0uPl6PUf 12Oh1s3mcU43V9gRI2bkgFiBO.Y5pZ1Z8GhP6Uc26HTiPMryKKxmRFVPZliSMc.e_544pNNbKeCY 6vrfMxGUeZgQRbSRJ26_vmwlJIb6BLyyCNsBLYLziG2UycVPX.wEyzytiw26afazDa.YDSuPkhZ6 2CaXmXHKN6i35NiyMvR5DNI96IwgTLek19ALN_nUtSTDVHGyX3fLQqlttcxWzJn89BCEljG53dGc Uc1WeW2qULrFyS9ON054JywzJG8Iial9_AAYfq5INcBCdGk3B5kP5_VMqvSoEi6GFbsrkgBKJ6Zw D8S77RX50QT.ZiXcAJJeKZTBSgpz0ogH_JJFdtsx81LYXvBtTWpIB099GXBjdxqL8Cw1adwPFQYs JxuMf2rkumHPblZeKqfYOrYAWnK2laFceAYeY.wbHguoSKJNbcUeTqzkt.dtApmemTkBQIWIpo7c O5tciWu6yQscPP7W1NhCBXCDJ4MUwcekib.s87P4VeU5aTSpGeKiF41s.0OctqXUeYhhJtyJuSvL TjKGAUeyCJXrDIeA0diGtYDj.1P_Auio4TA07CWGDGAOaI0Po4EgHjTZV3db79ctWW7D7EkPePca DAtPadmnQqXYxvRNmeBlCXix2M26at.d5PD4CsgHVKF72N5ST28FxndF_zspL94Dnht2899rum7f GQYy_rYzMZJgWN5QLSe2BkdwPDHo2bE_g4M_8_ApBYyrvI.V7XAJX41FHBUnxPpH6II.5.bxjr6H hDqdhAqDIgVvtHlNfooU0V0BRKI_DHVaGJ7lXK1xF.eMNkETb9Et0M42ciq1g_ANfwVsdGX46f8O JfEji3i443yhiBCttzjxfwGXVHks1ZkUlqz24JhlaYr5RWd3rM_zD2GiPk1TR69lYaXpgOcIKH1a dK3TrKv_JI1a49400QEUuSyTPdXcXb5RwtnlkpvRxpFkU22l87rnsu6WJXMgS_DQaiAb7x1Y8i_8 8BY_KddsNo8ljtArcc.7G9pTmQMrO7vUYd5uBBgvbnB07ZDsx1KYH3Hm3Rrug_keNggzYvRCaSha EqYmmrFGVnCgmlFSfOTy571zUTOkGK9uuo.K8zN3WXPMQbuQpyohfJIoTwq6e8_drXQjUvWyvzVJ s9_TTdGfy9bRssX4Zxw3tFa6UrLumUocFahgsxfD4DEHJSgCMJiS9RJN1d_BFXMyFUJmOrwHDjPT CBj9W4epFOhBPBeRwGfyIwoVU9QQqn.M9TuGiueIKdWNGLbmu.exWdL72NhUTzqv2h2rj.sOf.K0 nSWyZ1NYOEBPtGDNGL0MO0hvobdo6XyJXD8HXsL.Sra98QW7rOhNJ.CCD.ZfGDvA9QYS0uafrMJn feMfGHukepUbvTZ0ewQ_o3e4JVTai9uOnHFYA9FCwv51V5mh2QmbqWjNluncaUZ_ZSvu957lKvKC NTww73CYx8tZquJcG3SMPl2.Yr0bHLIk1itbGFA2BcG5fnZdlJpn.yXNz2vgW2AO9_P17G7xpD_9 pxymV9KrbrqPSA_idL359TDNcoBwgxe3dkDFOJGLMryA5K_b7mzrJhiTgBeT25Wzi369t_CLPP0a nxgUGOZ6N5iIa4rhwA8qvW0GIl6.UT5pmmbGwJOqYlUMyKOvbo.z6yPITsMRQ62S_QZzCdGf9pwL vQQ-- X-Sonic-MF: X-Sonic-ID: ac16413d-dc9f-4a39-9535-7d2ade4938b6 Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Fri, 24 Mar 2023 19:47:26 +0000 Received: by hermes--production-bf1-777648578f-lk5ld (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 5dd19e69c26d88815e55cd8d8a6d623e; Fri, 24 Mar 2023 19:47:20 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE Message-Id: <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732@yahoo.com> Date: Fri, 24 Mar 2023 12:47:08 -0700 To: Steve Kargl , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732.ref@yahoo.com> X-Spamd-Result: default: False [-3.39 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.89)[-0.888]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.31:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Pjt5r0MT9z3H11 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Steve Kargl wrote on Date: Wed, 22 Mar 2023 19:04:06 UTC : > On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: > >=20 > > Yes, there are reports that FreeBSD is not responsive by default - = but this > > may make it get overall better throughput at the expense of = responsiveness, > > because it might be doing fewer context switches. So just = complaining about > > a longer buildworld without seeing how much dnetc did in the same = wallclock > > time period is useless. Periodic rant's don't fix this lack of = information. > >=20 >=20 > I reported the issue with ULE some 15 to 20 years ago. > I gave up reporting the issue. The individuals with the > requisite skills to hack on ULE did not; and yes, I lack > those skills. The path of least resistance is to use > 4BSD. >=20 > % cat a.f90 > ! > ! Silly numerically intensive computation. > ! > program foo > implicit none > integer, parameter :: m =3D 200, n =3D 1000, dp =3D kind(1.d0) > integer i > real(dp) x > real(dp), allocatable :: a(:,:), b(:,:), c(:,:) > call random_init(.true., .true.) > allocate(a(n,n), b(n,n)) > do i =3D 1, m > call random_number(a) > call random_number(b) > c =3D matmul(a,b) > x =3D sum(c) > if (x < 0) stop 'Whoops' > end do > end program foo > % gfortran11 -o z -O3 -march=3Dnative a.f90=20 > % time ./z > 42.16 real 42.04 user 0.09 sys > % cat foo > #! /bin/csh > # > # Launch NCPU+1 images with a 1 second delay > # > foreach i (1 2 3 4 5 6 7 8 9) > ./z & > sleep 1 > end > % ./foo >=20 > In another xterm, you can watch the 9 images. >=20 > % top > st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 11:43:01 > 74 processes: 10 running, 64 sleeping > CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle > Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G = Free > Swap: 16G Total, 16G Free >=20 > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z > 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z > 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z > 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z > 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z > 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z > 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z > 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z > 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z >=20 > With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's = exit > after 55-ish seconds. If you try this experiment on ULE, you'll get = NCPU-1 > ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as = the > two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, > this was/is due to ULE's cpu affinity where processes never migrate to > another cpu. Admittedly, this was several years ago. Maybe ULE has > gotten better, but George's rant seems to suggest otherwise. Note: I'm only beginning to explore your report/case. There is a significant difference in your report and George's report: his is tied to nice use (and I've replicated there being SCHED_4BSD vs. SCHED_ULE consequences in the same direction George reports but with much larger process counts involved). In those types of experiments, I without the nice use I did not find notable differences. But it is a rather different context than your examples. Thus the below as a start on separate experiments closer to what you report using. Not (yet) having a Fortran set up I did some simple expriments with stress --cpu N (N processes looping sqrt calculations) and top. In top I sorted by pid to make it obvious if a fixed process was getting a fixed CPU or WCPU. (I tried looking at both CPU and WCPU, varying the time between samples as well. I also varied stress's --backoff N . This was on a ThreadRipper 1950X (32 hardware threads, so 16 cores) that was running: # uname -apKU FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #95 = main-n261544-cee09bda03c8-dirty: Wed Mar 15 19:44:40 PDT 2023 = root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG amd64 amd64 1400082 1400082 (That is a SCHED_ULE kernel. "GENERIC-NODBG-SCHED_4BSD" would be listed for SCHED_4BSD if I end up doing experiments with it.) Trying things like --cpu 33 I never got a process that sustained a low CPU or WCPU figure. The low figures moved around across the processes as I watched. When I tried the likes of --cpu 48 all the CPU or WCPU figures were generally lower than 99% but also showed variable figures that moved around across the processes. It definitely did not produce a sustained, approximately uniform figure across the processes. For comparison/contrast: --cpu 32 does have all the 32 processes showing 99+% all the time. This seems at least suggestive that, in my context, the specific old behavior that you report does not show up, at least on the timescales that I was observing at. It still might not be something you would find appropriate, but its does appear to at least be different. There is the possibility that stress --cpu N leads to more being involved than I expect and that such is contributing to the behavior that I've observed. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Fri Mar 24 20:25:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pjtxf2HjVz415NW for ; Fri, 24 Mar 2023 20:25:26 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pjtxd6XS2z3Kr5 for ; Fri, 24 Mar 2023 20:25:25 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Authentication-Results: mx1.freebsd.org; none Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.17.1/8.17.1) with ESMTPS id 32OKPGJx071140 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 24 Mar 2023 13:25:16 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.17.1/8.17.1/Submit) id 32OKPGqL071139; Fri, 24 Mar 2023 13:25:16 -0700 (PDT) (envelope-from sgk) Date: Fri, 24 Mar 2023 13:25:16 -0700 From: Steve Kargl To: Mark Millard Cc: FreeBSD Hackers Subject: Re: Periodic rant about SCHED_ULE Message-ID: Reply-To: sgk@troutmask.apl.washington.edu References: <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732.ref@yahoo.com> <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732@yahoo.com> X-Rspamd-Queue-Id: 4Pjtxd6XS2z3Kr5 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Fri, Mar 24, 2023 at 12:47:08PM -0700, Mark Millard wrote: > Steve Kargl wrote on > Date: Wed, 22 Mar 2023 19:04:06 UTC : > > > I reported the issue with ULE some 15 to 20 years ago. > > I gave up reporting the issue. The individuals with the > > requisite skills to hack on ULE did not; and yes, I lack > > those skills. The path of least resistance is to use > > 4BSD. > > > > % cat a.f90 > > ! > > ! Silly numerically intensive computation. > > ! > > program foo > > implicit none > > integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) > > integer i > > real(dp) x > > real(dp), allocatable :: a(:,:), b(:,:), c(:,:) > > call random_init(.true., .true.) > > allocate(a(n,n), b(n,n)) > > do i = 1, m > > call random_number(a) > > call random_number(b) > > c = matmul(a,b) > > x = sum(c) > > if (x < 0) stop 'Whoops' > > end do > > end program foo > > % gfortran11 -o z -O3 -march=native a.f90 > > % time ./z > > 42.16 real 42.04 user 0.09 sys > > % cat foo > > #! /bin/csh > > # > > # Launch NCPU+1 images with a 1 second delay > > # > > foreach i (1 2 3 4 5 6 7 8 9) > > ./z & > > sleep 1 > > end > > % ./foo > > > > In another xterm, you can watch the 9 images. > > > > % top > > st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 11:43:01 > > 74 processes: 10 running, 64 sleeping > > CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle > > Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G Free > > Swap: 16G Total, 16G Free > > > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND > > 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z > > 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z > > 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z > > 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z > > 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z > > 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z > > 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z > > 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z > > 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z > > > > With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's exit > > after 55-ish seconds. If you try this experiment on ULE, you'll get NCPU-1 > > ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as the > > two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, > > this was/is due to ULE's cpu affinity where processes never migrate to > > another cpu. Admittedly, this was several years ago. Maybe ULE has > > gotten better, but George's rant seems to suggest otherwise. > > Note: I'm only beginning to explore your report/case. > > There is a significant difference in your report and > George's report: his is tied to nice use (and I've > replicated there being SCHED_4BSD vs. SCHED_ULE > consequences in the same direction George reports > but with much larger process counts involved). In > those types of experiments, I without the nice use > I did not find notable differences. But it is a > rather different context than your examples. Thus > the below as a start on separate experiments closer > to what you report using. Yes, I recognizes George's case is different. However, the common problem is ULE. My testcase shows/suggests that ULE is unsuitable for a HPC platform. > Not (yet) having a Fortran set up I did some simple > expriments with stress --cpu N (N processes looping > sqrt calculations) and top. In top I sorted by pid > to make it obvious if a fixed process was getting a > fixed CPU or WCPU. (I tried looking at both CPU and > WCPU, varying the time between samples as well. I > also varied stress's --backoff N . This was on a > ThreadRipper 1950X (32 hardware threads, so 16 cores) > that was running: You only need a numerically intensive program that runs for 30-45 seconds. I use Fortran everyday and wrote the silly example in 5 minutes. The matrix multiplication of two 1000x1000 double precision matrices has two benefits with this synthetic benchmark. It takes 40-ish seconds on my hardware (AMD FX-8350) and it blows out the cpu cache. > This seems at least suggestive that, in my context, the > specific old behavior that you report does not show up, > at least on the timescales that I was observing at. It > still might not be something you would find appropriate, > but its does appear to at least be different. > > There is the possibility that stress --cpu N leads to > more being involved than I expect and that such is > contributing to the behavior that I've observed. I can repeat the openmpi testing, but it will have to wait for a few weeks as I have a pressing deadline. The openmpi program is a classic boss-worker scenario (and an almost perfectly parallel application with litttle communication overhead). boss starts and initializes the environment and then launches numerical intensive workers. If boss+n workers > ncpu, you get a boss and a worker pinned to a cpu. If boss and worker ping-pong, it stalls the entire program. Admittedly, I last tested years ago. ULE may have had improvements. -- Steve From nobody Fri Mar 24 21:17:13 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pjw5j6vfJz417xr for ; Fri, 24 Mar 2023 21:17:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-21.consmr.mail.gq1.yahoo.com (sonic309-21.consmr.mail.gq1.yahoo.com [98.137.65.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pjw5j467Rz3jQq for ; Fri, 24 Mar 2023 21:17:29 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679692647; bh=5y7ZqkN/iiK581CEfO4QPbRJO6T+dyY0HuZ03C5y3Gk=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=sTnhpEXnOEt5ER1CqYDVTvVIrx2+aGgtI5i0k5Mc5wb03q3kliclNFDLlKSTl2VskneO4h1m+mS72YxcBMQiFdg2sP95IJHymWR/EoCbAMKEnS1VZcwkUlJC54ClLrEtAxnJ13EDOQuZU0rWIOWBsMR/ML1So6IffHv/GLZxdA5QOaZ/t0MuP5OoTIe/eXptIqVQN9P75zKDZOR4BHwD2Ii99n2OhIEyuU/dYr0BldjDEHNWC0FX4g721XoZuBBxRHqi2dGitxSwCyRzY17GgiMI0zQXxuZ52JDnmWSK3oYeSS2Vp56kkxL0ilP2yKe2iF/m0Yp6vUOMcIaqZqgMSg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679692647; bh=pzF5JGbaC2vCsM6z8mx7ZFTXvueyW+swoQ5CLUr+OZA=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=R6Q3dinfYgg0hpzIjKZayPIsF+LgAmamjYqEEMR8G2DCnFDLyRuL965GfF6s+RWwver3QzJb+m8OLw4wLgYhHsYmKEb/LADFLS9fa8SKhtUGO9R9nhIXF6MT4/SypeRLE/uv5Cphd16ZfCBkskZG+frYhgOv1SqRr6JACOPtzvLFi7VxnjtEs7vyB9NMv3LO9iYrfm10BzKnCuAqt4+I/MYTnM787HLgQkj8ajr0vUPScQFWdidEVK2tdVDO22a0PI2ucL1MgtGwYujWZECqZJX2/fLn5NLxHDHVhBxuyLaT9CHuEYs/0w0BNlUtVMfy+VO1Kc/b7/VJF0Qx2doWNg== X-YMail-OSG: L0gkl_YVM1mExvP.A3qTFpCaZKU11CmsLudAagh03T9_w_oCD2ohXD6XYLYJvp9 OMZrvat2PGAT2gbg8sw5o5o871Aa0WrLEbPEjFxUPoBK2IYb7o2wVq970z93J4AuqhFXMqWFJPqn C_WXc4PWqGhvWTSfTB9aR37i.i7ExPQAYu07N.Q51Z7X8k9eMJFdEOHV.t1Zwq5lTXXfFtL9mbky HSuZfJQllhkTn8TsiTN962q2kPdmvhzEgAAsXIff83kR_GiKvdBP.ME_aPKPm4wtH0F2ngclq8XB VqqqomR6i4bZTJYJSXQtNUbvwhYYG9fonA2FJIRBhqlBSgszK2owsxkjWfdsWcie5AMEJpIimejZ MtiylMrkTg30vVWJjFo_fvW5KFtmNUoaD6twQ5WaolOA4sHILjVp2vtbwIDn4KUdHFmzMDxeEWpm N68TBB9knvL3M1TclUWBw0EnxU7Ksdw0WTom56S4VnQ9EGkMvJf5mjMBwMhB8IwYBpP3UnYjWq7j B8G0aafYOWu9fw2BQ_y87upT99qri43.YIF0Fd_LvhygJ33lpT1pIbQDxR1ZDbmceIw.W0ZYe3yI bDeLeXgWNucW.0j7jbPYxW1QRzG8qyk_GyqOxuqWG_gw.jYVw.BAq.57iNoUeXDaxPiKDKMTuRaN 3B_PgBlO84wR.BNVngnn86.rSuUyiRZVO572jGjMg9s4C0PvM9nNvXr0jyWhLiQg8mzCT1Qn.vyO rvpONZkA4nLAtAw62Q4UX0DmDcPUNqqcL7hqKpYDCiE0L48t3km4C1HIyC1BPjCiHPMR0MXpAiwD qq_Ugg6GVKtodpF.8RzPHL37GPHXb3Ohw9JvTBENs8hLDxsMc7kTxlwPsCK.tHctDby2938Sn8u8 LB_rNiatRvQp8OAVBKTkd0z0PO5pOj7R92VQcadqKwL6nvEDFeI4F0fSTFVwL5cvmBdhub5qGnIP Njg1U988m.p_dpjTHj5JAazwmHH2n5TIVfa_M57jL0A2cvcrDw2dqTeBGgoxvdVwTWt0wXYFhAvU UhJGrMWYNPXhUCKuzNIE3tzk2ngqcB55b1tz.QNEmIQps7LmapXGz2BnwNf11l1WCOY2erukys4. 575zWMZBd.n0rbRBuSV5QMvMPz6lTSIMD9So1dgLYDEx2v5mEooQTxlOFLlUESLGsmMXcgOmon4Q 6SIzsUeekOuUjiptVIwBTqpmwTN4F5qWM9h5WiSF3bjTpGUvtF_TMwfT4EuVd3DSGOXFdqc0fNj4 _NK54eXXzG50FYsKvqM4dgva7Ge.OdqWOrhTS92DGhbdr1wYcz9O6D7Lj.D6hObxi1YU508UNmjJ ZjZD_VTpBIJZJQUAYfSDP1xHPN2fFmG4YkNk6X82264oiGKhdBJhR649kypxd3cv3VupMslOdCRi JIGWmEGuuO1eV3lE0vBWJFoxj3NtzUOJneYhPd05n0DTOZ49iuogpIeRSavyY.dw2WD9cs4wuuJ0 ODUkRIHyPBU3ZM9sTgHA4q9OcunEzsFi8cgS2hf_VO4M20lloSIAGWIJPsDOUVc3ukUdIVm_SZyX sOk1RZlMEFEY5GVmfmN6KkOWLS.vpMOrOpD4jIdcksDm.Uz9M60SUtp_cvPHNLsMwvhybm.B62Wk jgXReAXpCGntNB30C4Kz_7JMzwbjgmSGPBpBCRiLAXOdA8liTaIGZ1JfR.oHdhIA6A0_gUZqUZ5S _DphHW4pqfubemR56HGjPLx3yYVvUkKafPCfMTTl5vnIkCzj_r7NjBgjButE4mENzJOxTocHHK6p 3MqK4KV1gy4BVlbZRTMkjUxTyNJwckskJUVIqdOQeIEAseuOfRx9F0s2QwRX1Q5U9Y4nO2h_PH3C Tvys8agKjQePg2olwcvtk6axjtq_e_9dxYc1VV88smKgde1LEehY40qkNu26U_KJ.52rxz.p5HzW aaCxbK_InU1cWXIVT3prgiLDrwQ8mZFa2ViitoolbZZM5IVyE1NF1W.TkQvnwnuMswW74cBlgkL9 IUGdoPZCDvI3r4w1MNzx8wCeDPGAvigLIeigLHoAmdYegexu89P8Me0pRYG7pQy8OT1Y9S6MGJQA VURGXouMcM74kvpubnOlif2eHK7rZtgRtDrClAWnfJI4dnlpMKRli7gVLDmq7VxMWv0BT97k3m.x pDTU.SGfhOmT4YZJ3RbRqZ2144LuPBDggDrNZ59d3d8Q_6QaGnZjvuBOcuyK_se5ku35tkU4x7I1 Fb_YuPmRWW7dVZMDEigENn7cyz3BikuCXqwjDNM_kmDxwpSN1BhZOcnk0PqFRQd1vDqzXsMjwbzp raoU- X-Sonic-MF: X-Sonic-ID: a6f144cf-a360-4d4f-89e0-e9ca5cd0ec3d Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Fri, 24 Mar 2023 21:17:27 +0000 Received: by hermes--production-gq1-6cf7749bc8-kkr9m (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7f9549e00bc39a0fb6a0d458a969a94a; Fri, 24 Mar 2023 21:17:24 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Fri, 24 Mar 2023 14:17:13 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <374296F5-892E-48F4-858D-20E15B494AE6@yahoo.com> References: <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732.ref@yahoo.com> <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732@yahoo.com> To: sgk@troutmask.apl.washington.edu X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4Pjw5j467Rz3jQq X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mar 24, 2023, at 13:25, Steve Kargl = wrote: > On Fri, Mar 24, 2023 at 12:47:08PM -0700, Mark Millard wrote: >> Steve Kargl wrote on >> Date: Wed, 22 Mar 2023 19:04:06 UTC : >>=20 >>> I reported the issue with ULE some 15 to 20 years ago. >>> I gave up reporting the issue. The individuals with the >>> requisite skills to hack on ULE did not; and yes, I lack >>> those skills. The path of least resistance is to use >>> 4BSD. >>>=20 >>> % cat a.f90 >>> ! >>> ! Silly numerically intensive computation. >>> ! >>> program foo >>> implicit none >>> integer, parameter :: m =3D 200, n =3D 1000, dp =3D kind(1.d0) >>> integer i >>> real(dp) x >>> real(dp), allocatable :: a(:,:), b(:,:), c(:,:) >>> call random_init(.true., .true.) >>> allocate(a(n,n), b(n,n)) >>> do i =3D 1, m >>> call random_number(a) >>> call random_number(b) >>> c =3D matmul(a,b) >>> x =3D sum(c) >>> if (x < 0) stop 'Whoops' >>> end do >>> end program foo >>> % gfortran11 -o z -O3 -march=3Dnative a.f90=20 >>> % time ./z >>> 42.16 real 42.04 user 0.09 sys >>> % cat foo >>> #! /bin/csh >>> # >>> # Launch NCPU+1 images with a 1 second delay >>> # >>> foreach i (1 2 3 4 5 6 7 8 9) >>> ./z & >>> sleep 1 >>> end >>> % ./foo >>>=20 >>> In another xterm, you can watch the 9 images. >>>=20 >>> % top >>> st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 11:43:01 >>> 74 processes: 10 running, 64 sleeping >>> CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle >>> Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, = 14G Free >>> Swap: 16G Total, 16G Free >>>=20 >>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU COMMAND >>> 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z >>> 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z >>> 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z >>> 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z >>> 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z >>> 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z >>> 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z >>> 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z >>> 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z >>>=20 >>> With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's = exit >>> after 55-ish seconds. If you try this experiment on ULE, you'll get = NCPU-1 >>> ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as = the >>> two images ping-pong on one cpu. Back when I was testing ULE vs = 4BSD, >>> this was/is due to ULE's cpu affinity where processes never migrate = to >>> another cpu. Admittedly, this was several years ago. Maybe ULE has >>> gotten better, but George's rant seems to suggest otherwise. >>=20 >> Note: I'm only beginning to explore your report/case. >>=20 >> There is a significant difference in your report and >> George's report: his is tied to nice use (and I've >> replicated there being SCHED_4BSD vs. SCHED_ULE >> consequences in the same direction George reports >> but with much larger process counts involved). In >> those types of experiments, I without the nice use >> I did not find notable differences. But it is a >> rather different context than your examples. Thus >> the below as a start on separate experiments closer >> to what you report using. >=20 > Yes, I recognizes George's case is different. However, > the common problem is ULE. My testcase shows/suggests > that ULE is unsuitable for a HPC platform. >=20 >> Not (yet) having a Fortran set up I did some simple >> expriments with stress --cpu N (N processes looping >> sqrt calculations) and top. In top I sorted by pid >> to make it obvious if a fixed process was getting a >> fixed CPU or WCPU. (I tried looking at both CPU and >> WCPU, varying the time between samples as well. I >> also varied stress's --backoff N . This was on a >> ThreadRipper 1950X (32 hardware threads, so 16 cores) >> that was running: >=20 > You only need a numerically intensive program that runs > for 30-45 seconds. Well, with 32 hardware threads instead of 8, the time frames likely need to be longer proportionally: 33 processes created and run, with overlapping time needed. > I use Fortran everyday and wrote the > silly example in 5 minutes. The matrix multiplication > of two 1000x1000 double precision matrices has two > benefits with this synthetic benchmark. It takes 40-ish > seconds on my hardware (AMD FX-8350) and it blows out the > cpu cache. I've not checked on the caching issue for what I've done below. Let me know if you expect it is important to check. >> This seems at least suggestive that, in my context, the >> specific old behavior that you report does not show up, >> at least on the timescales that I was observing at. It >> still might not be something you would find appropriate, >> but its does appear to at least be different. >>=20 >> There is the possibility that stress --cpu N leads to >> more being involved than I expect and that such is >> contributing to the behavior that I've observed. >=20 > I can repeat the openmpi testing, but it will have to=20 > wait for a few weeks as I have a pressing deadline. I'll be curious to learn what you then find. > The openmpi program is a classic boss-worker scenario > (and an almost perfectly parallel application with litttle > communication overhead). boss starts and initializes the > environment and then launches numerical intensive=20 > workers. If boss+n workers > ncpu, you get a boss and > a worker pinned to a cpu. If boss and worker ping-pong, > it stalls the entire program. =46rom what I've seen, boss+1worker doing ping-pong at times would not be prevented from happening sometimes for a while but would not be sustained indefinitely. > Admittedly, I last tested years ago. ULE may have had > improvements. Actually I do have a fortran: gfortran12 (automatically). (My original search had a typo.) I'll have to adjust the parameters for your example: # gfortran12 -o z -O3 -march=3Dnative a.f90 # time ./z 27.91 real 27.85 user 0.06 sys but I've 32 hardware threads, so the loop waiting for 1 sec between for 33 examples would have the first ones exit before the last ones start. Looks like n=3D2000 would be sufficient: # gfortran12 -o z -O3 -march=3Dnative a.f90 # time ./z 211.25 real 211.06 user 0.18 sys For 33 processes, things are as I described when I look with the likes of: # top -a -opid -s5 Varying the time scale to shorter allows seeing process WCPU figures move around more between the processes more. Longer shows less of the WCPU variability across the processes. (As I remember, -s defaults to 3 seconds and has a minimum of 1 second.) Given the 32 hardware threads, I used 33 processes via: # more runz #! /bin/csh # # Launch NCPU+1 images with a 1 second delay # foreach d (1 2 3) foreach i (1 2 3 4 5 6 7 8 9 10) ./z & sleep 1 end end foreach j (1 2 3) ./z & sleep 1 end My guess is that if you end up seeing what you originally described, some environmental difference would be involved in why I see different behavior, something to then be tracked down for what is different in the 2 contexts. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sat Mar 25 01:46:43 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pk24S514hz41S24 for ; Sat, 25 Mar 2023 01:46:48 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x32c.google.com (mail-ot1-x32c.google.com [IPv6:2607:f8b0:4864:20::32c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pk24R03rLz46qc; Sat, 25 Mar 2023 01:46:47 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=d1bnIATG; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::32c as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-x32c.google.com with SMTP id i16-20020a9d6110000000b0069fb468226aso1836803otj.3; Fri, 24 Mar 2023 18:46:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679708804; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Wcn3NqTFR0W1dzgm7msYse/0q+q7QNadQKjFp/X3cYQ=; b=d1bnIATGQi8mP5i77Tm6QGYWkQhOSvGA4+U2+do6U8W3Wdbtm3hU+yHBXNTYyhEbKF WLEsp9DqK44Rd/Zu8oAAPGEJZ2c6+NBUbhW/UOyZpoxt0MXh5Wx9lYV8Zg9QltK6+U42 8aIWLxvjknOqrZGnfuFBfJnADg+Ut7Yx+HBPCMxTeY7XVETpingq7U3C08NckE8aLu1w hUOELWuvnhRRstDsoUWEEQF3mcfSXpVw8UF4RTjQprOmZ4yhyXWv7ClPi4m9IXotbhRx fKeU+bdIa5alzBaB1hAYXEXs/QqOJVvZPiizsidKHb5pwTRsJgw6Rnso7vbLf0fbL3W8 oHqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679708804; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Wcn3NqTFR0W1dzgm7msYse/0q+q7QNadQKjFp/X3cYQ=; b=OVNvTSitQXRqSN8nLwB4kGqdbr2c1eEaqM1GeyyAXqEO+2RtpFc+xS74mBq2j9nKgG /pqDcBQpIV8fnNVyDDr4uRwLYN/xjG+UiA0cbLsSK1viev+IIcuS2BYm0FWhX3a55GnO heCwwyxylp/ZZ/jG8LTcgR6YyGOCCvMF+0he314NrHzE2p8zHs7vrAhwXTXAQTn/HHZ4 2reBKFScNDbtVDq/NlUCnLmMps7/5VHzvFC/43z5NOJF6DIiK1jITmmvCdh4fSJi1NuC kFdSpbuepQ5w2BFfhhMP7kRbsyl1BZNOiCuhljU069Q6tGTFCMApzIANhmRg1HVMgzui OizQ== X-Gm-Message-State: AO0yUKWb5Z9+4qMKcaLmd7s2BXR7qgrKIH+LckHHK9snkYgR0QHH7orW 7WR/bvnyjTcIAg62zBTF4M2HvS4gs23uoHJm9R8jgsE9 X-Google-Smtp-Source: AK7set8rL+5H6Dd1PtEImnOMfZIiXejj+JIwR2vt/ey4hSmdT1fnwGgWKvoeiP3J+LtqOdLEsKJWXl4Hb6fttyd60+E= X-Received: by 2002:a9d:67d8:0:b0:69f:229:ce72 with SMTP id c24-20020a9d67d8000000b0069f0229ce72mr1689541otn.2.1679708803744; Fri, 24 Mar 2023 18:46:43 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:ac9:5705:0:b0:49c:b071:b1e3 with HTTP; Fri, 24 Mar 2023 18:46:43 -0700 (PDT) In-Reply-To: References: <8173cc7e-e934-dd5c-312a-1dfa886941aa@FreeBSD.org> <8cfdb951-9b1f-ecd3-2291-7a528e1b042c@m5p.com> From: Mateusz Guzik Date: Sat, 25 Mar 2023 02:46:43 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: sgk@troutmask.apl.washington.edu Cc: Matthias Andree , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.90 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.90)[-0.900]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32c:from]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4Pk24R03rLz46qc X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/23/23, Mateusz Guzik wrote: > On 3/22/23, Mateusz Guzik wrote: >> On 3/22/23, Steve Kargl wrote: >>> On Wed, Mar 22, 2023 at 07:31:57PM +0100, Matthias Andree wrote: >>>> >>>> Yes, there are reports that FreeBSD is not responsive by default - but >>>> this >>>> may make it get overall better throughput at the expense of >>>> responsiveness, >>>> because it might be doing fewer context switches. So just complaining >>>> about >>>> a longer buildworld without seeing how much dnetc did in the same >>>> wallclock >>>> time period is useless. Periodic rant's don't fix this lack of >>>> information. >>>> >>> >>> I reported the issue with ULE some 15 to 20 years ago. >>> I gave up reporting the issue. The individuals with the >>> requisite skills to hack on ULE did not; and yes, I lack >>> those skills. The path of least resistance is to use >>> 4BSD. >>> >>> % cat a.f90 >>> ! >>> ! Silly numerically intensive computation. >>> ! >>> program foo >>> implicit none >>> integer, parameter :: m = 200, n = 1000, dp = kind(1.d0) >>> integer i >>> real(dp) x >>> real(dp), allocatable :: a(:,:), b(:,:), c(:,:) >>> call random_init(.true., .true.) >>> allocate(a(n,n), b(n,n)) >>> do i = 1, m >>> call random_number(a) >>> call random_number(b) >>> c = matmul(a,b) >>> x = sum(c) >>> if (x < 0) stop 'Whoops' >>> end do >>> end program foo >>> % gfortran11 -o z -O3 -march=native a.f90 >>> % time ./z >>> 42.16 real 42.04 user 0.09 sys >>> % cat foo >>> #! /bin/csh >>> # >>> # Launch NCPU+1 images with a 1 second delay >>> # >>> foreach i (1 2 3 4 5 6 7 8 9) >>> ./z & >>> sleep 1 >>> end >>> % ./foo >>> >>> In another xterm, you can watch the 9 images. >>> >>> % top >>> st pid: 1709; load averages: 4.90, 1.61, 0.79 up 0+00:56:46 >>> 11:43:01 >>> 74 processes: 10 running, 64 sleeping >>> CPU: 99.9% user, 0.0% nice, 0.1% system, 0.0% interrupt, 0.0% idle >>> Mem: 369M Active, 187M Inact, 240K Laundry, 889M Wired, 546M Buf, 14G >>> Free >>> Swap: 16G Total, 16G Free >>> >>> PID USERNAME THR PRI NICE SIZE RES STATE C TIME CPU >>> COMMAND >>> 1699 kargl 1 56 0 68M 35M RUN 3 0:41 92.60% z >>> 1701 kargl 1 56 0 68M 35M RUN 0 0:41 92.33% z >>> 1689 kargl 1 56 0 68M 35M CPU5 5 0:47 91.63% z >>> 1691 kargl 1 56 0 68M 35M CPU0 0 0:45 89.91% z >>> 1695 kargl 1 56 0 68M 35M CPU2 2 0:43 88.56% z >>> 1697 kargl 1 56 0 68M 35M CPU6 6 0:42 88.48% z >>> 1705 kargl 1 55 0 68M 35M CPU1 1 0:39 88.12% z >>> 1703 kargl 1 56 0 68M 35M CPU4 4 0:39 87.86% z >>> 1693 kargl 1 56 0 68M 35M CPU7 7 0:45 78.12% z >>> >>> With 4BSD, you see the ./z's with 80% or greater CPU. All the ./z's >>> exit >>> after 55-ish seconds. If you try this experiment on ULE, you'll get >>> NCPU-1 >>> ./z's with nearly 99% CPU and 2 ./z's with something like 45-ish% as the >>> two images ping-pong on one cpu. Back when I was testing ULE vs 4BSD, >>> this was/is due to ULE's cpu affinity where processes never migrate to >>> another cpu. Admittedly, this was several years ago. Maybe ULE has >>> gotten better, but George's rant seems to suggest otherwise. >>> >> >> While I have not tried openmpi yet, I can confirm there is a problem >> here, but the situtation is not as clear cut as one might think. >> >> sched_4bsd *round robins* all workers across all CPUs, which comes at >> a performance *hit* compared to ule if number of workers is <= CPU >> count -- here ule maintains affinity, avoiding cache busting. If you >> slap in $cpu_count + 1 workers, 4bsd keeps the round robin equally >> penalizing everyone, while ule mostly penalizes a select victim. By >> the end of it, you get lower total cpu time spent, but higher total >> real time. See below for an example. >> >> Two massive problems with 4bsd, apart from mandatory round robin which >> also happens to help by accident: >> 1. it has one *global* lock, meaning the scheduler itself just does >> not scale and this is visible at modest contemporary scales >> 2. it does not understand topology -- no accounting done for ht nor >> numa, but i concede the latter wont be a factor for most people >> >> That said, ULE definitely has performance bugs which need to be fixed. >> At least for the case below, 4BSD just "lucked" into sucking less >> simply because it is so primitive. >> >> I wrote a cpu burning program (memset 1 MB in a loop, with enough >> iterations to take ~20 seconds on its own). >> >> I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 distinct >> cores. >> >> The test runs *9* workers, here is a sample run: >> >> 4bsd: >> 23.18 real 20.81 user 0.00 sys >> 23.26 real 20.81 user 0.00 sys >> 23.30 real 20.81 user 0.00 sys >> 23.34 real 20.82 user 0.00 sys >> 23.41 real 20.81 user 0.00 sys >> 23.41 real 20.80 user 0.00 sys >> 23.42 real 20.80 user 0.00 sys >> 23.53 real 20.81 user 0.00 sys >> 23.60 real 20.80 user 0.00 sys >> 187.31s user 0.02s system 793% cpu 23.606 total >> >> ule: >> 20.67 real 20.04 user 0.00 sys >> 20.97 real 20.00 user 0.00 sys >> 21.45 real 20.29 user 0.00 sys >> 21.51 real 20.22 user 0.00 sys >> 22.77 real 20.04 user 0.00 sys >> 22.78 real 20.26 user 0.00 sys >> 23.42 real 20.04 user 0.00 sys >> 24.07 real 20.30 user 0.00 sys >> 24.46 real 20.16 user 0.00 sys >> 181.41s user 0.07s system 741% cpu 24.465 total >> >> It reliably uses 187s user time on 4BSD and 181s on ULE. At the same >> time it also reliably has massive time imblance between >> fastest/slowest in terms of total real time between workers *and* ULE >> reliably uses more total real time to finish the entire thing. >> >> iow this is a tradeoff, but most likely a bad one >> > > So I also ran the following setup: 8 core vm doing -j 8 buildkernel, > while 8 nice -n 20 processes are cpu-bound. After the build ends > workers report how many ops they did in that time. > > ule is way off the reservation here. > > unimpeded build takes: 441.39 real 3135.63 user 266.92, similar on > both schedulers > > with cpu hoggers: > 4bsd: 657.22 real 3152.02 user 253.30 sys [+49%] > ule: 4427.69 real 3225.33 user 194.86 sys [+903%] > > ule spends less time in the kernel, but the time blows up -- over 10 x > the base line. > this is clearly a total non-starter. > > full stats: > > 4bsd: > hogger pid/ops > 58315: 5546013 > 58322: 5557294 > 58321: 5545052 > 58313: 5546347 > 58318: 5537874 > 58317: 5555303 > 58323: 5545116 > 58320: 5548530 > > runtimes: > > 657.23 real 230.02 user 0.02 sys > 657.23 real 229.83 user 0.00 sys > 657.23 real 230.50 user 0.00 sys > 657.23 real 230.53 user 0.00 sys > 657.23 real 230.14 user 0.01 sys > 657.23 real 230.19 user 0.00 sys > 657.23 real 230.09 user 0.00 sys > 657.23 real 230.10 user 0.03 sys > > kernel build: > 657.22 real 3152.02 user 253.30 sys > > ule: > hogger pid/ops > 77794: 95916836 > 77792: 95909794 > 77789: 96454064 > 77796: 95813778 > 77791: 96728121 > 77795: 96678291 > 77798: 97258060 > 77797: 96347984 > > 4427.70 real 4001.94 user 0.10 sys > 4427.70 real 3980.68 user 0.16 sys > 4427.70 real 3973.96 user 0.10 sys > 4427.70 real 3980.11 user 0.13 sys > 4427.70 real 4012.32 user 0.07 sys > 4427.70 real 4008.79 user 0.12 sys > 4427.70 real 4034.77 user 0.09 sys > 4427.70 real 3995.40 user 0.08 sys > > kernel build: > 4427.69 real 3225.33 user 194.86 sys > added a probe to runq_add*: diff --git a/sys/kern/kern_switch.c b/sys/kern/kern_switch.c index aee0c9cbd1ae..db73226547c5 100644 --- a/sys/kern/kern_switch.c +++ b/sys/kern/kern_switch.c @@ -357,6 +357,9 @@ runq_setbit(struct runq *rq, int pri) rqb->rqb_bits[RQB_WORD(pri)] |= RQB_BIT(pri); } +SDT_PROVIDER_DECLARE(sched); +SDT_PROBE_DEFINE3(sched, , , runq_add, "struct thread *", "int", "int"); + /* * Add the thread to the queue specified by its priority, and set the * corresponding status bit. @@ -378,6 +381,7 @@ runq_add(struct runq *rq, struct thread *td, int flags) } else { TAILQ_INSERT_TAIL(rqh, td, td_runq); } + SDT_PROBE3(sched, , , runq_add, td, flags, pri); } void @@ -396,6 +400,7 @@ runq_add_pri(struct runq *rq, struct thread *td, u_char pri, int flags) } else { TAILQ_INSERT_TAIL(rqh, td, td_runq); } + SDT_PROBE3(sched, , , runq_add, td, flags, pri); } /* * Return true if there are runnable processes of any priority on the run and used this one-liner: dtrace -b 16M -x aggsize=32M -x dynvarsize=32M -n 'sched:::runq_add /args[0]->td_ucred->cr_uid == 2000/ { self->runq_t = timestamp; } sched:::on-cpu /self->runq_t/ { @runqlat["runq", execname == "cpuburner-long" ? "cpuburner" : "rest"] = quantize(timestamp - self->runq_t); self->runq_t = 0; } sched:::on-cpu /curthread->td_ucred->cr_uid == 2000/ { self->oncpu_t = timestamp; } sched:::off-cpu /self->oncpu_t/ { @oncpu["oncpu", execname == "cpuburner-long" ? "cpuburner" : "rest"] = quantize(timestamp - self->oncpu_t); } tick-300s { exit(0); } caped at 5 minutes because dtrace starts running into aggregation drops Key takeaway: 4bsd let's the cpu hog stay on cpu for longer than ule, but when it is taken off, it waits a long time to get back. in contrast, it gets back on very quick with ule and it is everyone else waiting big time. times in nanoseconds 4bsd: runq cpuburner value ------------- Distribution ------------- count 2048 | 0 4096 | 2 8192 |@@@@@@@@@@ 4343 16384 |@@@@@ 2159 32768 |@ 363 65536 |@@@ 1359 131072 | 215 262144 | 129 524288 | 101 1048576 | 132 2097152 | 185 4194304 |@ 390 8388608 |@ 318 16777216 |@ 398 33554432 |@@ 838 67108864 |@@@@@@@ 3025 134217728 |@@@@@ 2474 268435456 |@@@ 1552 536870912 | 153 1073741824 | 0 runq rest value ------------- Distribution ------------- count 2048 | 0 4096 | 637 8192 |@@@@@@@@@@@@@@@@@@@@@@@@ 364832 16384 |@@@ 52982 32768 |@ 11613 65536 |@@ 34286 131072 |@@@ 39734 262144 |@@ 23261 524288 |@ 21985 1048576 |@ 18999 2097152 |@ 10789 4194304 | 6239 8388608 | 4725 16777216 | 4598 33554432 | 4050 67108864 | 5857 134217728 | 3979 268435456 | 2024 536870912 | 2011 1073741824 | 1105 2147483648 | 841 4294967296 | 519 8589934592 | 372 17179869184 | 133 34359738368 | 37 68719476736 | 30 137438953472 | 1 274877906944 | 0 oncpu cpuburner value ------------- Distribution ------------- count 2048 | 0 4096 | 20 8192 | 10 16384 | 19 32768 | 161 65536 | 137 131072 | 77 262144 | 104 524288 | 147 1048576 |@ 301 2097152 |@ 482 4194304 |@@@ 1178 8388608 |@@@@@@@@@@@@@@@ 6971 16777216 |@ 474 33554432 |@ 669 67108864 |@@@@@@@@@@@@@@@@ 7299 134217728 | 14 268435456 | 38 536870912 | 38 1073741824 | 2 2147483648 | 2 4294967296 | 0 oncpu rest value ------------- Distribution ------------- count 512 | 0 1024 | 102 2048 |@@@@@@@@@@@ 146555 4096 |@@ 23373 8192 |@@@ 45165 16384 |@ 14531 32768 |@@@@@ 67664 65536 |@@@@@@ 80155 131072 |@@@@@ 64609 262144 |@@ 21509 524288 |@@ 23393 1048576 |@@ 21058 2097152 |@ 7854 4194304 |@ 8788 8388608 |@ 20242 16777216 | 2352 33554432 | 2084 67108864 | 4388 134217728 | 1123 268435456 | 755 536870912 | 135 1073741824 | 2 2147483648 | 0 ule: runq cpuburner value ------------- Distribution ------------- count 2048 | 0 4096 | 2 8192 |@@@@@@@@@@@@@@@@@@@@@@@@ 37193 16384 |@@@@@ 8612 32768 |@ 1481 65536 |@@@@@ 8430 131072 |@@ 3102 262144 |@ 938 524288 | 457 1048576 |@ 2063 2097152 | 257 4194304 | 41 8388608 | 428 16777216 | 12 33554432 | 1 67108864 | 2 134217728 | 0 runq rest value ------------- Distribution ------------- count 4096 | 0 8192 |@ 649 16384 |@@@@ 1953 es 32768 |@ 539 65536 |@@@@@ 2369 131072 |@@@@ 1904 262144 |@ 471 524288 | 131 1048576 |@ 458 2097152 |@ 443 4194304 |@ 547 8388608 |@ 632 16777216 |@@ 984 33554432 |@@@ 1606 67108864 |@@@@@@@@ 3894 134217728 |@ 511 268435456 |@ 489 536870912 |@ 445 1073741824 |@ 475 2147483648 |@ 314 4294967296 | 188 8589934592 | 136 17179869184 | 100 34359738368 | 81 68719476736 | 39 137438953472 | 3 274877906944 | 0 oncpu cpuburner value ------------- Distribution ------------- count 2048 | 0 4096 | 13 8192 | 311 16384 |@@ 2369 32768 |@ 853 65536 |@ 2302 131072 |@ 1302 262144 | 518 524288 |@ 872 1048576 |@ 1172 2097152 |@ 1435 4194304 |@@ 2706 8388608 |@@@@@@@@@@@@@@@@@@@ 29669 16777216 |@@ 2733 33554432 |@@ 3910 67108864 |@@@@@@ 9991 134217728 |@ 1800 268435456 |@ 840 536870912 | 200 1073741824 | 15 2147483648 | 7 4294967296 | 1 8589934592 | 0 oncpu rest value ------------- Distribution ------------- count 512 | 0 1024 |@ 419 2048 |@@@@@@@@@ 5730 4096 |@ 778 8192 |@@@@@ 3233 16384 |@@ 1400 32768 |@@@@ 2739 65536 |@@@@@@@@@@@@ 7550 131072 |@@@ 1720 262144 | 151 524288 | 112 1048576 |@@@ 1688 2097152 | 174 4194304 | 86 8388608 |@ 411 16777216 | 7 33554432 | 3 67108864 | 0 -- Mateusz Guzik From nobody Sat Mar 25 15:47:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkNtm2hCzz41M17 for ; Sat, 25 Mar 2023 15:54:40 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkNtk5368z49p6 for ; Sat, 25 Mar 2023 15:54:38 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of pmc@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=pmc@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32PFs6ej053110 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sat, 25 Mar 2023 16:54:07 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32PFs65f053109 for freebsd-hackers@freebsd.org; Sat, 25 Mar 2023 16:54:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32PFncAu079184 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK) for ; Sat, 25 Mar 2023 16:49:38 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32PFlgk4004538 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sat, 25 Mar 2023 16:47:44 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32PFlgdE004537 for freebsd-hackers@freebsd.org; Sat, 25 Mar 2023 16:47:42 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sat, 25 Mar 2023 16:47:42 +0100 From: Peter To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sat, 25 Mar 2023 16:54:09 +0100 (CET) X-Spamd-Result: default: False [-2.27 / 15.00]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.969]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; BLOCKLISTDE_FAIL(0.00)[2a0b:f840::12:server fail]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[sub.org]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; HAS_XAW(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4PkNtk5368z49p6 X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Quoting George Mitchell : >> https://forums.freebsd.org/threads/what-is-sysctl-kern-sched-preempt_thresh.85 >> >Thank you! -- George You're welcome. Can I get a success/failure report? --------------------------------------------------------------------- >> On 3/22/23, Steve Kargl wrote: >>> >>> I reported the issue with ULE some 15 to 20 years ago. Can I get the PR number, please? --------------------------------------------------------------------- Test usecase: ============= Create two compute tasks competing for the same -otherwise unused- core, one without, one with syscalls: # cpuset -l 13 sh -c "while true; do :; done" & # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null Within a few seconds the two task are balanced, running at nearly the same PRI and using each 50% of the core: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 5166 root 1 88 0 13M 3264K RUN 13 9:23 51.65% sh 10675 root 1 87 0 13M 3740K CPU13 13 1:30 48.57% gzip This changes when the tar reaches /usr/include with it's many small files. Now smaller blocks are delivered to gzip, it does more syscalls, and things get ugly: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 5166 root 1 94 0 13M 3264K RUN 13 18:07 95.10% sh 19028 root 1 81 0 13M 3740K CPU13 13 1:23 4.87% gzip This does not happen because tar would be slow in moving data to gzip: tar reads from SSD, or more likely from ARC, and this is always faster than gzip-9. The imbalance is made by the scheduler. From nobody Sat Mar 25 17:54:17 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkRYG0Kfjz41Vdr for ; Sat, 25 Mar 2023 17:54:42 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkRYF4gvgz4PPb for ; Sat, 25 Mar 2023 17:54:41 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Authentication-Results: mx1.freebsd.org; none Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.17.1/8.17.1) with ESMTPS id 32PHsI9V096755 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 25 Mar 2023 10:54:18 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.17.1/8.17.1/Submit) id 32PHsHX5096754; Sat, 25 Mar 2023 10:54:17 -0700 (PDT) (envelope-from sgk) Date: Sat, 25 Mar 2023 10:54:17 -0700 From: Steve Kargl To: Peter Cc: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: Reply-To: sgk@troutmask.apl.washington.edu References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4PkRYF4gvgz4PPb X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sat, Mar 25, 2023 at 04:47:42PM +0100, Peter wrote: > --------------------------------------------------------------------- > >> On 3/22/23, Steve Kargl wrote: > >>> > >>> I reported the issue with ULE some 15 to 20 years ago. > > Can I get the PR number, please? > Don't remember if I ever files a bug report. There's one here that was simply closed. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=85820 Search the current hackers mailing list archive where I know I've reported the issue a few times. Here's a few examples https://lists.freebsd.org/pipermail/freebsd-hackers/2008-October/026375.html https://lists.freebsd.org/pipermail/freebsd-current/2009-October/012599.html https://lists.freebsd.org/pipermail/freebsd-current/2010-January/014943.html -- Steve From nobody Sat Mar 25 18:14:11 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkS084ThXz41XRr for ; Sat, 25 Mar 2023 18:14:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-20.consmr.mail.gq1.yahoo.com (sonic305-20.consmr.mail.gq1.yahoo.com [98.137.64.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkS072GpMz3Drx for ; Sat, 25 Mar 2023 18:14:31 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=akgJgXDa; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679768068; bh=sBysVb4XFx4ipjzfGmlmR2au+qgUbz+Uo4p6lRKjUhs=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=akgJgXDa5ajtR+rsPaAbuZI+4qig8K23wU8gjkv74Ea8RTxMKR+fEYvu6ThGSz8fPERCZc3Y/LDvjH34onmRPyTJ0bQZvO/FFxB+V50vOl9MR13pi/x02Mdks5F/dJaCHQmcJDcHhT6hBmluzcsjwCuEYfctZUooAduxMs3l5fKhqtEwJTa4+tufSn7WwPhAP9SdyCE1HoFXoTY2v+LVpWU18UIovqMGb3Nyo7omb6trwAjO7y1ZzLVWOI3gjdxpg+BJ0WyhBm+D+0uNPZnA+uC4w03fNvNcNPvaGGzDCc/h1uku4mkdn2fBlbTUoUJVU5OxjhkGujsgptagiDGnuw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679768068; bh=iCzwFUbEqxOHlZAlW/YL1GVWgUoVEh0j8X9f06iaS6P=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=kwwZBwh0omu0IkVMqsvLKqSlQ2RTY0V8oAchmaCfNaMWbEhzXfYQ0mNqTO167QGHSCxtIBFKbxZzSnnml6WzVvHrN3Th8byEUfE38tSiV2zCOshvUWy/Lkk7NRrtPntpAlcQ2ljBDBJ1J2Whydd18W1/ywYHT5KsXYufi0CjJ4927YRUeZPAq8OdU54+u3YpU7SgnilaaSaxJATF+D30yXwXLfzrX5zAWjKlBUm8LgqkxjaLUvESUo1Yv6L880ysjsZ4mDxJJ4wn5THs6A2sTiCT+JMJLpeOA6/O5An9yHnXWN9djwJeV2JFVJVU6bySNHEoV8QmEi1L6GcenpgJIQ== X-YMail-OSG: YfNc3MEVM1mswA5ngZzxzngtQAJc.ZReSLTXN.PaDBJpwrKO0Gi6XeIpaGKgXLa SkSv0LbA4qeFdAqsM8gXH_BY8TjW.Mb8X78DcMm1f_kegQ7k_Wgrl0Pwj8O5VI5WdTAgMNt6uqq7 ERZm6jYIsXJiMSSg0wVk4Bs4CG2NlwhwKKwVtat5eG5Z9AwyA.WdJ2sYKn_G3WWVMKovgwnJrGAr jq0heKEfId37gjkeOsugFOyR5bPw96_.Kc80lZs4DPaSAL2h9xgVmjxtsqRLV4eqZPmtWTL64bs4 tiFOJDGUGO2g72GbaTo4rnGSZtyX3Bvxmn2bzSSVDp_dNOERZR2jNkLUKO25Rbgc7s3a6S8O4mUj 7LspP6B23ncFOwQrVLkYi2oBSK1jtnkTuIAcu12Ot1XFjzflaIx2Ht1EDS32H_jaSeXB60cCkL6a zLDrgbTJxIbNy3ruGi4kM3s6vX2y9Ouz8ujn6WkiWREJnTVJdztAdwXeyrXpv9K1TSpCN_jo9Ytq AtnxxvK4wtmnGUAbHzzwejC8ix.VgL2uAILr.vP1DZEHjAZzWnK_MHl4BUp7nQju8wC_4CG_Km3r JeSfFUMS3hYRJEQ3xkqjBzOfxuKvFY9HUIwBM9_NQlm1HHcvABfUPHT9cqPaYp9Lrq.apYDUME_F pe3Di.vL6fJxp22eP1SujdeL7tkfPZirqXec.ftE8mOV9dy4zjoJRflzHHujvev7_BhMlKlI0GFg f_DQz.lZmyuk1gPrBZixY1ISDm7JzBscMGvlquF6.qJf6cwp4Ly8sfoH.Q9JNDG7cLMbOezsc3hB 3mCO2QuPyiKM5FaqZF6I_qEC0iXcjoGVqfLQCB.Cyn951rYt0MH8Ix4Em55jDFaGk8LysONU45o6 Wt5cwHEwK2LOBgGwVxLrGKqLabQHMGV.BcPsBaR0e4NjaTjXvEppQ7pqhp2YyCSzswMz_r8yDiD3 MnGcmz3oCE3djrM9B.JJq8Y8.RURK9.yIA40Vn4pvWz7N7EX55IZZyVxyyl_Yl4nWryYcHJTlKL9 8ZFEaHnJAgI9TUG5N1xLRv2jIprbi0sPVJiUidjY_hedN4NM9GXZTNVRTE0IxZwkg6F65J4x6Hhn zHsb1XvR_NJJwN5Kin0ACgCJUFq5S4J04miJr.71tzsu_vqlwqL66EZl.yWaTnDRp6S5KG_Btf2A 7pX4y_5jpPgypgNDc5gJcVToWrPPoIn6jV6CUCdiDe3hwIVFi1Gy4VIOHKGl4t9CpkAPaaGu_MZd Az2b5Qmag7VNAm5Tz3Woz2YgYU0rA99eY0knW9dS4ey77BwUWvfKiCxO1KE.P6JK95ZXa.m2qamc FumRICG0BOO9I6fA_68CnZaMLSDFkCxLrNzPt0xIp9F2ScWGJsvhqsUk0j4zG744Bsdze8ZyV7SN u11P4CmeEUDxdVYeHqiTETbjytDIIRPYfAOkW6fgDpr_QC4yktNZk1MkrnHykOHAdCaOPTUtjG3t QHFGmmN3ASIH5VHa5DSwBcUpcjtVFUB5uuVjE_mjsFdH5UZrExFX5aZCYooo6HGEbom8rUCJGMJe tDwiR..JfTgXHnXnqjgbwehUBK4Sib9ENbYr6itO2NskbkSH7rR0wt1P4tModazbYI0qadLoAVWo Fb0UW.cOnzhvrOLVkMwZgHQj4gkaOzd2FJvMzGpEEmY2wHtFwuXSNJhbsKqwCyBAFNf9K18A7jiq jE1UOP8WkybOp8HfuBMhKzyWk1Zo2i.CvhQ29x7aHE050CUyTcg0VUsi_J2Gy47mycULeZK3jPYk z_pEItB4NvRWxZHMPLCmMqmX3swtyYp2RN1us60chhLjjQkkgFB5j_ENoZyYPh1gl5.bWDM6sy5h 9vlqajcx3q0E8RBkSALzGaTPC2hLADR1JZQCKnCERTc8oa6gr8O_BRpy2tCAwCN7mIMtv0wDM8GJ 6JyDcBGkHTB2ys6lZS.Jhd8ENMzAfKlTOpYvSFURYcWkBRdO5OvwCt8UqKIDRWDG3C275y0FqEUF EXZhDa.XrljvuH.i30vtK7LXhJYxgoNzg6T8Bmah74ksFfwYBhUy9LcYqbD3n_Zfl4UdXqhsZaXd pXjPcjf_Vaylbt.YsPIz0pubCFS2Y5h1QmSG6RxOUsOTeq8JD.XVWjEWOsMNVdUFQLJLMhDPbQ5u .EN8Aoq9BIBIRYYQ8LnReXeoS2jOnIpqr79gwxHXrxlxIcipqyi_Ifl5kYf4Gv4BTFiHC1IBPm0K cyMS_iOB93P4KaDDwkC1UUkB6s_942bAavsNFglOVQFre.EIb42yeyiZds1Qj4ot61uPEDe.Wjq7 9pw-- X-Sonic-MF: X-Sonic-ID: 53c96eb3-27d6-45c5-a63d-74ffd1ef389a Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Sat, 25 Mar 2023 18:14:28 +0000 Received: by hermes--production-bf1-777648578f-75chz (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 3af61c8aca561d0e34e3ab64f6f8428d; Sat, 25 Mar 2023 18:14:23 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE Message-Id: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> Date: Sat, 25 Mar 2023 11:14:11 -0700 To: Peter , FreeBSD Hackers X-Mailer: Apple Mail (2.3731.400.51.1.1) References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.80)[-0.796]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.83:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PkS072GpMz3Drx X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Peter wrote on Date: Sat, 25 Mar 2023 15:47:42 UTC : > Quoting George Mitchell : >=20 > >> = https://forums.freebsd.org/threads/what-is-sysctl-kern-sched-preempt_thres= h.85 > >> > >Thank you! -- George >=20 > You're welcome. Can I get a success/failure report? >=20 >=20 > --------------------------------------------------------------------- > >> On 3/22/23, Steve Kargl wrote: > >>> > >>> I reported the issue with ULE some 15 to 20 years ago. >=20 > Can I get the PR number, please? >=20 >=20 > --------------------------------------------------------------------- > Test usecase: > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > Create two compute tasks competing for the same -otherwise unused- = core,=20 > one without, one with syscalls:=20 >=20 > # cpuset -l 13 sh -c "while true; do :; done" &=20 > # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null=20 >=20 > Within a few seconds the two task are balanced, running at nearly the=20= > same PRI and using each 50% of the core:=20 >=20 > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND=20 > 5166 root 1 88 0 13M 3264K RUN 13 9:23 51.65% sh=20 > 10675 root 1 87 0 13M 3740K CPU13 13 1:30 48.57% gzip=20 >=20 > This changes when the tar reaches /usr/include with it's many small=20 > files. Now smaller blocks are delivered to gzip, it does more=20 > syscalls, and things get ugly:=20 >=20 > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND=20 > 5166 root 1 94 0 13M 3264K RUN 13 18:07 95.10% sh=20 > 19028 root 1 81 0 13M 3740K CPU13 13 1:23 4.87% gzip=20 Why did PID 10675 change to 19028? > This does not happen because tar would be slow in moving data to=20 > gzip: tar reads from SSD, or more likely from ARC, and this is=20 > always faster than gzip-9. The imbalance is made by the scheduler. When I tried that tar line, I get lots of output to stderr: # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null tar: Removing leading '/' from member names a . a root a wrkdirs a bin a usr . . . Was that an intentional part of the test? To avoid this I used: # tar cvf - / 2>/dev/null | cpuset -l 13 gzip -9 2>&1 > /dev/null At which point I get the likes of: 17129 root 1 68 0 14192Ki 3628Ki RUN 13 0:20 = 3.95% gzip -9 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 0:00 = 0.27% tar cvf - / (bsdtar) 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 8:05 = 95.93% sh -c while true; do :; done up front. For reference, I also see the likes of the following from "gstat -spod" (it is a root on ZFS context with PCIe Optane media): dT: 1.063s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name . . . 0 68 68 14 937 0.0 0 0 0 0.0 = 0 0 0 0.0 0 0.0 0.1| nvd2 . . . =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sat Mar 25 18:23:04 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkSBK5hz0z41XKC for ; Sat, 25 Mar 2023 18:23:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-25.consmr.mail.gq1.yahoo.com (sonic303-25.consmr.mail.gq1.yahoo.com [98.137.64.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkSBJ4VHtz3GZ6 for ; Sat, 25 Mar 2023 18:23:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=D+6zf49O; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679768598; bh=A8T2cNzKjrsZDGbrEGIYM7bIWpUg0nzqkgILXEUJhXA=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=D+6zf49OomzPJHKKg4UVxApKyMJ2lZEQRSusDAa+6BiQDGbMzOXZHCxrvD18GyZaNSri02Phj4D8+ZYxU2W1Qrk2r00g22hUINKFGoIp3zXqFIaqV4qqkERtrtWdFpKV25Nbx75dnKYtG1Lwn4VTUYNGB+ljkL96uLv4zUrOMeA2G94eQkGtuer58kqFilLxk/2SPAP03Ed1a3Hd177QM8zGslV2B9aDh3VrHO+/r0L6PhVbFY5W77pbgcSs3uuKHXHisGjwZ929O0Txw7h05Zj1fIeNzvLT4HRn1nyUglulkOi/sOG72cJtFb+N7RrOrHyFINFk9CK0cLtbNAjJ5Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679768598; bh=f5NJerh2kbt178W7xkr2xE6JR6DKZ1oTjiuRF+KdG5g=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=kq74UdPZC1F7HbYanvxKSLEnZ1J/OFPv8HXVGbfE5Z7A1M7CWjZ9K5bWQkT75YAIWvzTRrkIXByWF17qB8MM0EHIaFsXwZGqVNUPFVy3+IJXaZEbHZpQmPuzVB/c1wKmFc1xmoaQutm38LvrKyDeI2BgDoYUdOW0NZFflko2eLoY5A6Hpx4TL/4unho1KvQKm4AnKRBZWUbrv22buUvLAcUMW8I6cisv7aJC8tq3gjzBT9Vsm3bWHjvlgBQIoMNfVrKu7m21tcARJLyp70Pw2SxpoFCW+2mLu6H4DS09EEGeS7md9mZBMtwhpaT6w8iLMMGW0KjR3IGJZWgK8jaGJg== X-YMail-OSG: HA5zqMwVM1kA68Pe9RS_1wymrDWc0eW0pHhX4RHXLM4X1K0zScwS9NK1u33g.Ir dR3Bnd0ilxYJ.k6Ko4.l5I0o_HF5IEJdgAebjYNsi0CyTnkqFzlmOhHGbnchhk9sYwmYbF.C8ewc AHcrgZjnHijLNi_TWzu0pwOAnMB4juy_mt8aNuEL9GQVERJ_fVBTOC.Fp2MIGQGg4OXpE7svZ_Yc bOoelHyIt3.f_XoCsJ3Y7_RI_3inWDDG0tzYFoOPDijrnod0c7EmgcObWWhkt9adw3t0yms1JDOD dWMIYXFld3NcHwgBsiOStujWuCNhrDj6jK4ThSH6eYXPWlY3nv4bFG6.YSRd_WuIKr7mfVP4AOfG stj8L8ki8VJNnBaB4cCoj2.ZQl16rhW3g.aS4hjr3wUtuX11W6SKWdPEcSGXsnkFPHTHFsnRtoYc MpFawOXLRAm8agv8fGWMKIt2REk1FXf4iZqeqNgRBKnXsRJ_tm99Eq6I.co3bcmBTzFUQPlRhzMK BoIXgEOUGUdXqDEhMuOEjtngJnDLqVA1UCKOITPqRrdwII.Yln9n97UnITqRU21nqtpBU.dCQQlt s2IknreWIMKlVL8kC6Y6v60D.uVXlaRupSYrsUqolpxbWXUhjj81lx9ljlJw7mzhZ5YhoaCAvrwv Tbn.s5D83htwyynELRX99dyv0wiKeTP1fc4oCBFoJvaVCv3CCET7mKrI2FbIDFXDdiWNPG7LxEc8 HluDJFnM7VX8RM7HrkDJuJiTcxxAQmR.azPJyQLuxQqq908_hwHfwadBMfECqn88yuuNQBwrvJeP ESKQq17sueTJkSvj2kQHM0.dkGHcJNSL.mHc2wGaIu_WcgAj7nYbe4_yq3zFHPzYnvXj0FoUuOAJ 91ZZ5BzlrQqRtWDFsq0nlrcOUMckmVh_1HNZRXIFEVDZ6cRJUhrx41YrhWP..8EGLJcGOnRhnZA8 BcQQOvo.0NuXqrXwQpno4dXpa5FGCWj20NNMJ4WAnkPEqpjFMl1dHLp6TK.9_WOlldNCjjlFPLRU n9AoK6K3MrzmsvDCtGgRQTbZrUu.82LV.qoVgtvYi8wTAAgyGtHwi9ex6XB05rggmJdIcAkF0BcR oVQrN2Fy4yKImHQ9xv5btOMUzbBaaW7oAUtErNhjrcohGtP2DdoDTK.6EbbcugBegwefdhNc5Mdo 5xQW9a8SFUNn3nfTQ87GI4Qkv_ZKhPiARP6HuDuBnLthbsk.oYQRUGtvsdsoqjbR1fMQX3aw4JFz uzFL_o2p7fwuW.x5_gt8kOi9TA3kBn4aYu5TAgxyh7jaf.EzGYQT3I6_NJ1uRkK7eYG7ZLqDnMC6 ypDI63C6BOl19gFeYoiMobSNCva2CEJLlyXDPZ2WMA3POAgwVbiHsmuEewRUs7C1FNhL4mxwNGEu 0ToZPnrEcFgtlmkKZIxyVjBTzyJyX2caiiSF2P4Ck1gMHlCJdFkH22auRDo3XbFeDoUhU09XvTW2 JpsqEpZhEDxZulI0kRpUQqBaz.26vCG1tNNWVzX_uxFQUJYO6XF.HKi1uHMdsiZyJEfP_nAR.B3V S6QmgPfelsuF82lTvXxhnoOnWMb6XoevKrh8WsqImDJnQKJeGDeav_t0n5w_RbhLX0VbBsOZm9Rd UWmfdMNA552zU4QIKVKGWokB3mZoYANNas89f_3A8krrW_fYaxX4V0qYvQ.vZmzw6T66fJW4RaZv AlFcNb5rvAHOvWJ2Cb0mOyVUodxFinm3CkUI6nQNeR9fF7hC1Ytqot6.nNtd6ouVF1Sywv1XOVr2 PCAqQ2Q7gdmUFd1KgjGbGDYpn7sqw5MMl37KXpK3f_qu0IJNn2nHpyHQrS9KPXcGHqBnubfV_RIe uMwGANEq114js4sXp4OQl89r4b_3R6ks4on00_jn38HS8UDJNVDKSjC2xZzJHKD53tgcnzNdPlhW pptGxqUMpMOceRlN6_E7wrjh1Uzgr8VyvSyGWu_MuNWWWnTxylGdNj8rEF8bzeuxvyrSp5XXso9N KN2i9D6fuvW7XL0WzL._twx6zLy8cI005cVWz2tcRvi8UF4zDerTGRof3wcZGZyD_JurvjumynLH vz1vwaHPPP__AE3lFseXT91j2NzrytoAWZUxLH7MhyuEVgGRTPHe28JnYtI5qr7pynIMF8LyNpja YaTEAGlgJYXPnXz8UGKmF2i4k8.dDNwJS8VAB7twB7RkHkrELr1p.6UUadp.ATTZnJwZVOu2iSXI 1qk.mpMHAbAkOzKIYS7qUxvdYnE8WzEWR7fnooEL8rnvtFiyjM2r0xxDly3ps2yELKCN8oLRhxC1 JAQ-- X-Sonic-MF: X-Sonic-ID: 9d834da9-91bd-4f56-bbdb-2a4069b030c4 Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Sat, 25 Mar 2023 18:23:18 +0000 Received: by hermes--production-gq1-6cf7749bc8-q7lrl (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID d2f9f0dd8de28b81e8a773019631e00d; Sat, 25 Mar 2023 18:23:15 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE Date: Sat, 25 Mar 2023 11:23:04 -0700 References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> To: Peter , FreeBSD Hackers In-Reply-To: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> Message-Id: <76DAACBB-C865-4779-A340-D66C35D610B4@yahoo.com> X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.25 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.75)[-0.752]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.206:from]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PkSBJ4VHtz3GZ6 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Mar 25, 2023, at 11:14, Mark Millard wrote: > Peter wrote on > Date: Sat, 25 Mar 2023 15:47:42 UTC : >=20 >> Quoting George Mitchell : >>=20 >>>> = https://forums.freebsd.org/threads/what-is-sysctl-kern-sched-preempt_thres= h.85 >>>>=20 >>> Thank you! -- George >>=20 >> You're welcome. Can I get a success/failure report? >>=20 >>=20 >> --------------------------------------------------------------------- >>>> On 3/22/23, Steve Kargl wrote: >>>>>=20 >>>>> I reported the issue with ULE some 15 to 20 years ago. >>=20 >> Can I get the PR number, please? >>=20 >>=20 >> --------------------------------------------------------------------- >> Test usecase: >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>=20 >> Create two compute tasks competing for the same -otherwise unused- = core,=20 >> one without, one with syscalls:=20 >>=20 >> # cpuset -l 13 sh -c "while true; do :; done" &=20 >> # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null=20 >>=20 >> Within a few seconds the two task are balanced, running at nearly the=20= >> same PRI and using each 50% of the core:=20 >>=20 >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND=20 >> 5166 root 1 88 0 13M 3264K RUN 13 9:23 51.65% sh=20 >> 10675 root 1 87 0 13M 3740K CPU13 13 1:30 48.57% gzip=20 >>=20 >> This changes when the tar reaches /usr/include with it's many small=20= >> files. Now smaller blocks are delivered to gzip, it does more=20 >> syscalls, and things get ugly:=20 >>=20 >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND=20 >> 5166 root 1 94 0 13M 3264K RUN 13 18:07 95.10% sh=20 >> 19028 root 1 81 0 13M 3740K CPU13 13 1:23 4.87% gzip=20 >=20 > Why did PID 10675 change to 19028? >=20 >> This does not happen because tar would be slow in moving data to=20 >> gzip: tar reads from SSD, or more likely from ARC, and this is=20 >> always faster than gzip-9. The imbalance is made by the scheduler. >=20 >=20 > When I tried that tar line, I get lots of output to stderr: >=20 > # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null > tar: Removing leading '/' from member names > a . > a root > a wrkdirs > a bin > a usr > . . . >=20 > Was that an intentional part of the test? >=20 > To avoid this I used: >=20 > # tar cvf - / 2>/dev/null | cpuset -l 13 gzip -9 2>&1 > /dev/null >=20 > At which point I get the likes of: >=20 > 17129 root 1 68 0 14192Ki 3628Ki RUN 13 0:20 = 3.95% gzip -9 > 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 0:00 = 0.27% tar cvf - / (bsdtar) > 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 8:05 = 95.93% sh -c while true; do :; done >=20 > up front. >=20 > For reference, I also see the likes of the following from > "gstat -spod" (it is a root on ZFS context with PCIe Optane media): >=20 > dT: 1.063s w: 1.000s > L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name > . . . > 0 68 68 14 937 0.0 0 0 0 0.0 = 0 0 0 0.0 0 0.0 0.1| nvd2 > . . . >=20 >=20 I left it running and I'm now seeing: 17129 root 1 107 0 14192Ki 3628Ki CPU13 13 3:01 = 48.10% gzip -9 17128 root 1 21 0 58300Ki 15428Ki pipdwt 20 0:04 = 2.02% tar cvf - / (bsdtar) 17097 root 1 115 0 13364Ki 3060Ki RUN 13 16:30 = 51.77% sh -c while true; do :; done Also examples of the likes of: dT: 1.063s w: 1.000s L(q) ops/s r/s kB kBps ms/r w/s kB kBps ms/w = d/s kB kBps ms/d o/s ms/o %busy Name . . . 0 1213 1213 5 6456 0.0 0 0 0 0.0 = 0 0 0 0.0 0 0.0 1.2| nvd2 . . . FYI: ThreadRipper 1950X context. Looks like what I'll see is very dependent on when I look at what it is doing: the details involved matter. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sat Mar 25 19:02:43 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkT3x0ywgz41Zyg for ; Sat, 25 Mar 2023 19:02:53 +0000 (UTC) (envelope-from thierry@pompo.net) Received: from erza.lautre.net (erza.lautre.net [80.67.160.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "lautre.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkT3w06cMz3LYf for ; Sat, 25 Mar 2023 19:02:51 +0000 (UTC) (envelope-from thierry@pompo.net) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of thierry@pompo.net designates 80.67.160.89 as permitted sender) smtp.mailfrom=thierry@pompo.net; dmarc=none Received: from graf.pompo.net (graf.pompo.net [82.66.0.218]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by erza.lautre.net (Postfix) with ESMTPSA id 919A7E7EB8 for ; Sat, 25 Mar 2023 20:02:43 +0100 (CET) Received: by graf.pompo.net (Postfix, from userid 1001) id 27BE45D2126; Sat, 25 Mar 2023 20:02:43 +0100 (CET) Date: Sat, 25 Mar 2023 20:02:43 +0100 From: Thierry Thomas To: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732.ref@yahoo.com> <6A29E7ED-0A1E-49F9-9224-AC3D5B0E0732@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="nt9i0JGeA+lHJaUC" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 13.1-STABLE amd64 Organization: Kabbale Eros X-Face: (hRbQnK~Pt7$ct`!fupO(`y_WL4^-Iwn4@ly-.,[4xC4xc;y=\ipKMNm<1J>lv@PP~7Z<.tKjAnXLs: X-PGP: 0xF1C516B3C8359753 X-Spamd-Result: default: False [-2.80 / 15.00]; RBL_VIRUSFREE_BOTNET(2.00)[80.67.160.89:from]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_SHORT(-1.00)[-0.998]; FORGED_SENDER(0.30)[thierry@freebsd.org,thierry@pompo.net]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; BAD_REP_POLICIES(0.10)[]; DMARC_NA(0.00)[freebsd.org]; FREEFALL_USER(0.00)[thierry]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DOM_EQ_FROM_DOM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[thierry@freebsd.org,thierry@pompo.net]; HAS_ORG_HEADER(0.00)[]; TO_DN_NONE(0.00)[]; R_SPF_ALLOW(0.00)[+mx]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:20766, ipnet:80.67.160.0/19, country:FR]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4PkT3w06cMz3LYf X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N --nt9i0JGeA+lHJaUC Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Le ven. 24 mars 23 =C3=A0 21:25:16 +0100, Steve Kargl =C3=A9crivait=C2=A0: > I can repeat the openmpi testing, but it will have to=20 > wait for a few weeks as I have a pressing deadline. Just a note: Intel MPI Benchmark is suitable for this kind of tests. It is in our ports tree as benchmarks/imb, but very outdated and not maintained. Newer releases are now on GitHub: . It has been on my TODO-list to upgrade this port for quiet a long time, but I=E2=80=99ve been busy on other side=E2=80=A6 If someone has some cycle= s, do not hesitate! --=20 Th. Thomas. --nt9i0JGeA+lHJaUC Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQJ8BAEBCgBmBQJkH0VSXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFNTM2QkU4NTM4NTM5OUQwMEI2RkFBNzZG MUM1MTZCM0M4MzU5NzUzAAoJEPHFFrPINZdT+VMP/jVPdEP64A93TNWV3bF3/3s+ A/eFWg9Lj/84QKYroazv9tEmWuCRYv0rRJhMiU1wKH6GfLNJv1WrxL+aMpMmEm79 Me5TepA96t4o5/ZKQvWyF0mwn6V6sX5pP9sqzvTgjAAEL7so1I22jeEpmdwKRrzb wzrKz2rIiDuNKSwfCdPiXXdXmqAU7yEdL5qyWudITbiS8Dd9fruWZvzVVo7R4g5M RJjWW0CTkSqvMDReCLsMWDMbRMCbrbZnNrV/mBrorgN263DJnx0sV4z6MKw8cHTy duU/m+w3C+SxBD6j6/Jr32j0UpA2So9Xy050uEf/y91KkLYSL/LFyDS/Gy1LQuT6 LO6JQ4/eQ6as+JnKlNGXGQ/oaUjD42hrUhvOMgH5Y+vB3ReZ34I9ytS9xFPR0a6v N3hb0idADPhJCnCL1T/Jwk0I646ZJf/4hOX2jFTopH0qy6apT0KN1AQ2U/BiU/d3 torXDzcfFYUKcLBmHiLLfo70llSQReZrSwumEJq7bR0CYdzGl6I8GHpTWb6o/dxZ ESnbg73UP5LYOlzWdRiUajvwNj4oltqUqaLHGUaXeP8FWou1Up1hR3iFBb1D84x4 HNIBaICGAHiFHBy7q9R2wR9D1iRWB9ShvR/obOneuZyVySzbUVJnY20nsJoTPSY0 8DvCx9js4oUGfwOyVNwO =EbWA -----END PGP SIGNATURE----- --nt9i0JGeA+lHJaUC-- From nobody Sat Mar 25 19:05:56 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkT7s3Nkkz41Zqk for ; Sat, 25 Mar 2023 19:06:17 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Received: from mailhost.m5p.com (mailhost.m5p.com [74.104.188.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "m5p.com", Issuer "R3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkT7q4Y6lz3MpZ for ; Sat, 25 Mar 2023 19:06:15 +0000 (UTC) (envelope-from george+freebsd@m5p.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of george+freebsd@m5p.com designates 74.104.188.4 as permitted sender) smtp.mailfrom=george+freebsd@m5p.com; dmarc=none Received: from [IPV6:2001:470:1f07:15ff::26] (court.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:26]) (authenticated bits=0) by mailhost.m5p.com (8.16.1/8.15.2) with ESMTPSA id 32PJ5u1u023375 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sat, 25 Mar 2023 15:06:03 -0400 (EDT) (envelope-from george+freebsd@m5p.com) Message-ID: Date: Sat, 25 Mar 2023 15:05:56 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.4.0 Subject: Re: Periodic rant about SCHED_ULE Content-Language: en-US To: freebsd-hackers@freebsd.org References: From: George Mitchell In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.0 required=10.0 tests=HELO_NO_DOMAIN,NICE_REPLY_A autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on mattapan.m5p.com X-Spamd-Result: default: False [-3.28 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.99)[-0.994]; NEURAL_HAM_SHORT(-0.99)[-0.991]; R_SPF_ALLOW(-0.20)[+a]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[m5p.com]; ASN(0.00)[asn:701, ipnet:74.104.0.0/16, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TAGGED_FROM(0.00)[freebsd] X-Rspamd-Queue-Id: 4PkT7q4Y6lz3MpZ X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 3/25/23 11:47, Peter wrote: > Quoting George Mitchell : > >>> https://forums.freebsd.org/threads/what-is-sysctl-kern-sched-preempt_thresh.85 >>> >> Thank you! -- George > > You're welcome. Can I get a success/failure report? > > > [...] Thanks for your help. Regretfully, I have to report that the patch did not really help in my case (make buildworld while running dnetc). But I have belatedly realized that it's a really good idea to use make -j12 to build kernels and the world. With 4BSD and dnetc running, I can now buildworld in 3512 seconds (down from 20477), and with ULE in 14193 seconds (down from 50290). Peter's patch got it down to 13755 seconds. Also, I have to belatedly admit that my snarky March 23 message to Mark Millard (claiming that all you had to do to run dnetc was to type "service dnetc start") omitted the necessity of doing some first run configuration. I apologize. -- George From nobody Sat Mar 25 18:58:55 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkTCk6m0vz41b2k for ; Sat, 25 Mar 2023 19:09:38 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkTCk2PBdz3Nhh for ; Sat, 25 Mar 2023 19:09:38 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32PJ97Bu097145 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 25 Mar 2023 20:09:07 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32PJ97Uc097144; Sat, 25 Mar 2023 20:09:07 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32PJ1cCF077148 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sat, 25 Mar 2023 20:01:39 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32PIwtBB006163 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 25 Mar 2023 19:58:55 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32PIwt3j006162; Sat, 25 Mar 2023 19:58:55 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sat, 25 Mar 2023 19:58:55 +0100 From: Peter To: Mark Millard Cc: FreeBSD Hackers Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sat, 25 Mar 2023 20:09:09 +0100 (CET) X-Rspamd-Queue-Id: 4PkTCk2PBdz3Nhh X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sat, Mar 25, 2023 at 11:14:11AM -0700, Mark Millard wrote: ! Why did PID 10675 change to 19028? Because it went into some NFS share, and it would still be there if I hadn't restartet it a bit differently. ! When I tried that tar line, I get lots of output to stderr: ! ! # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null ! tar: Removing leading '/' from member names ! a . ! a root ! a wrkdirs ! a bin ! a usr ! . . . ! ! Was that an intentional part of the test? Yes. So you can see what it is currently feeding to gzip (small or big files - or some NFS share, where the operation becomes pointless). ! # tar cvf - / 2>/dev/null | cpuset -l 13 gzip -9 2>&1 > /dev/null ! ! At which point I get the likes of: ! ! 17129 root 1 68 0 14192Ki 3628Ki RUN 13 0:20 3.95% gzip -9 ! 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 0:00 0.27% tar cvf - / (bsdtar) ! 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 8:05 95.93% sh -c while true; do :; done ! ! up front. Ah. So? To me this doesn't look good. If both jobs are runnable, they should each get ~50%. ! For reference, I also see the likes of the following from ! "gstat -spod" (it is a root on ZFS context with PCIe Optane media): So we might assume that indeed both jobs are runable, and the only significant difference is that one does system calls while the other doesn't. The point of this all is: identify the malfunction with the most simple usecase. (And for me here is a malfunction.) And then, obviousely, fix it. From nobody Sat Mar 25 19:08:11 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkTQ4646Kz41bPv for ; Sat, 25 Mar 2023 19:18:36 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkTQ44hggz3Q5P for ; Sat, 25 Mar 2023 19:18:36 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32PJI54G004002 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 25 Mar 2023 20:18:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32PJI5Md004001; Sat, 25 Mar 2023 20:18:05 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32PJAcic081805 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sat, 25 Mar 2023 20:10:40 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32PJ8Bt1006254 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 25 Mar 2023 20:08:11 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32PJ8BUC006253; Sat, 25 Mar 2023 20:08:11 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sat, 25 Mar 2023 20:08:11 +0100 From: Peter To: Steve Kargl Cc: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sat, 25 Mar 2023 20:18:08 +0100 (CET) X-Rspamd-Queue-Id: 4PkTQ44hggz3Q5P X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sat, Mar 25, 2023 at 10:54:17AM -0700, Steve Kargl wrote: ! On Sat, Mar 25, 2023 at 04:47:42PM +0100, Peter wrote: ! > --------------------------------------------------------------------- ! > >> On 3/22/23, Steve Kargl wrote: ! > >>> ! > >>> I reported the issue with ULE some 15 to 20 years ago. ! > ! > Can I get the PR number, please? ! > ! ! Don't remember if I ever files a bug report. Too bad. A bugreport has the advantage that one can attach fixes to it, and that things get a lot more focused. ! There's one ! here that was simply closed. ! ! https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=85820 Hm, that is not really suitable. From nobody Sat Mar 25 19:44:54 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkV0Z2nkNz41cnf for ; Sat, 25 Mar 2023 19:45:02 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkV0Z16mmz3hqH; Sat, 25 Mar 2023 19:45:02 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679773502; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=q9H2fOFnf9Eff42T7sjG8XmL9IMt+KMX17ikU8kT1pQ=; b=Z2OhAu85mdy7slzAHmyK9dKh+F1G9gNDVeQwDp5+mvtEa3Acp5SSKEnivvhJlzNieC2emh TYvHn/hyhXV2VD1O1A5YpxZG1uIfJ3eSHC2qzmWTYyJe40VLK/NvXYNFAc1BgzNMFX61Pp t+Xd9mtAxPyT6E9y2NhBnO9u9AI0qxVCLxEOchIfkUTvQAYCwLRpPiAwOcPqGyWd60Abai ovQvpLzWpJobhtVbZjfZIPgq+xlgHN5fYbcXPyz7SLlqkd7LROyB6hVloF5iCYf0MR92+h De796V61x5Vh4gBE7Tj1U8hj7olKgN1TO9gYGlZHuwt9P2l1p96YFHqd1iAG8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679773502; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=q9H2fOFnf9Eff42T7sjG8XmL9IMt+KMX17ikU8kT1pQ=; b=LGPneIq6xilYzJz4iqqBDEor8TMqSEmpq1XbQxYOftXd8rROHPggZ9IGAB0hpILfO4XsT9 TaNRlsBD+/u1Gaql+zht6vJ04LBJWQoeJrLXAzy7qeOuurPOVap+CAPqW9yvaBmprCv6SD bZEDt4jM2epimUrUCZE0wXur3W2GA9EwHfsieJIt8QyaBXi1HUfz3yyRWCgDB144KTELP6 rG65eGBAusnqbxhtouI372RAjj9/vkn+w0T62LUXMT3kTVeZTImp7SSEoddiklBdaepfod H8WFptZKgUYOBu/atIWNaop6rZ8Pguni9c+wBlJp9lUxQrWIXaNw5lF9tK5o9Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679773502; a=rsa-sha256; cv=none; b=CN7WwlsuFfDQHZH+QGSrILA82MIkj7fBJN7F56ohFk7qcBT4o0Rk2wq6wxr9dzJnyR7/9f ueJqm+L4SjoPPYIhgcSEvNs+GVudGdIoDC5/vu5SjQOk8uypJkw2TcSwjW62K1W4TYEEf+ WhrY/aQDjII3oNf8nq5OUQPU2IAE6nhgFhq70NWeH+FXJvZMQTeT6v2PUZmGrz950ulCBg z3v1qTJhwLs/5abQvkW1YackOYcHVrkJJw8nS7TrUVa2Q2vo8eztwTTYnASJk3fge7+40q +5zRKNPZ0YiNBZV+/4ZMf71flDAbizkgYCPXtqKa7sla7enQNAGfcBS8AMkqJA== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PkV0Y0CtKzcmk; Sat, 25 Mar 2023 19:45:00 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <670cff73-a3e6-8501-b33c-df358149adb7@freebsd.org> Date: Sat, 25 Mar 2023 19:44:54 +0000 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 To: freebsd-hackers@freebsd.org From: Graham Perrin Subject: git-apply(1): error: content/en/ports/_index.adoc: No such file or directory Content-Language: en-GB Organization: FreeBSD Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------4V470iQYdf65d2vtPbCd6lC9" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------4V470iQYdf65d2vtPbCd6lC9 Content-Type: multipart/mixed; boundary="------------AB6Ems4lLu0g4YQ0f4TjcAoV"; protected-headers="v1" From: Graham Perrin To: freebsd-hackers@freebsd.org Message-ID: <670cff73-a3e6-8501-b33c-df358149adb7@freebsd.org> Subject: git-apply(1): error: content/en/ports/_index.adoc: No such file or directory --------------AB6Ems4lLu0g4YQ0f4TjcAoV Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 Q2FuIGFueW9uZSBleHBsYWluIHdoYXQncyBiZWxvdz8NCg0KVGhlIC5kaWZmIGlzIGRvd25s b2FkZWQgZnJvbSA8aHR0cHM6Ly9yZXZpZXdzLmZyZWVic2Qub3JnL0QzODUxMD4uDQoNCg0K JSBmaWxlIC91c3IvZG9jL3dlYnNpdGUvY29udGVudC9lbi9wb3J0cy9faW5kZXguYWRvYw0K L3Vzci9kb2Mvd2Vic2l0ZS9jb250ZW50L2VuL3BvcnRzL19pbmRleC5hZG9jOiBBU0NJSSB0 ZXh0DQolIGdpdCAtQyAvdXNyL2RvYyBhcHBseSAtdiAvdG1wL0QzODUxMC5kaWZmDQpDaGVj a2luZyBwYXRjaCBjb250ZW50L2VuL3BvcnRzL19pbmRleC5hZG9jLi4uDQplcnJvcjogY29u dGVudC9lbi9wb3J0cy9faW5kZXguYWRvYzogTm8gc3VjaCBmaWxlIG9yIGRpcmVjdG9yeQ0K JSBoZWFkIC1uIDggL3RtcC9EMzg1MTAuZGlmZg0KSW5kZXg6IHdlYnNpdGUvY29udGVudC9l bi9wb3J0cy9faW5kZXguYWRvYw0KPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQ0KLS0tIHdlYnNpdGUvY29udGVu dC9lbi9wb3J0cy9faW5kZXguYWRvYw0KKysrIHdlYnNpdGUvY29udGVudC9lbi9wb3J0cy9f aW5kZXguYWRvYw0KQEAgLTExLDIxICsxMSwyMCBAQA0KDQonJycnJw0KDQolDQoNCg== --------------AB6Ems4lLu0g4YQ0f4TjcAoV-- --------------4V470iQYdf65d2vtPbCd6lC9 Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmQfTzgFAwAAAAAACgkQt2dIb0oY1Asq /A//ZPdsl12Z5m7463UkEk1Dpn67lcHDpVjnBH2zh788hz72C0HqsPzobsCQcV979j1gItgZ19e4 ZFO3rbMB9/Be5f5eO/uu0017RBakJ0hx1CEUUcViq06AgrB8plk72N+h9lLa3OKsDs2p8RABanXT T+E41CSOw6JvuLNd5sBBJqpmzqP3atxTLwGKRehP3D64vnVvHtvHuF5KhYElq5Dfxtov8nzAA7dG rJgIKxDBh60S3nXQK8GEDAZ8u4rMoluFxPDGKNHSVHWhAHqi46SZAMMIRGz49ZiLLcBU6/h0r2+4 kWVOldC310piUuuwj39kUfIGtgsULF/7/cri7rcnCPH90gjZKH903+/Rf1hcv6+/R95IQel2DX7N LWp05yWmciqaNA3UDyg4XNtyc8n5Ba52l0pa4jJ0+pG2VNJSgCKm4cQEnDKIERMkdv0OEvwD2eY6 4JuJwLDxKcaMU2/u+cOpms3WMDbzUEsDVmdBxQ+RCQ7JxuKYwXUaeZcW33cjcHlz2Yhk7UjoEuWg T3SsLO8Xg+8dtoO/FlPuuaG7eMb46t0Z1suLrqxGqnXPdUhNoe5GZcM+DQMGaWxjRIiXY33OGSyH PXl+QFdtqcHWACcz3kaGLKkEdoUmqnQ5mD0huqn8QVrjO3jTe9kgDsjTKYYB42vHxJ1TikTBioif ImI= =kQCS -----END PGP SIGNATURE----- --------------4V470iQYdf65d2vtPbCd6lC9-- From nobody Sat Mar 25 19:55:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkVF512M4z41d7l for ; Sat, 25 Mar 2023 19:55:53 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkVF45YY9z3kQ4 for ; Sat, 25 Mar 2023 19:55:52 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Authentication-Results: mx1.freebsd.org; none Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.17.1/8.17.1) with ESMTPS id 32PJtS03005247 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 25 Mar 2023 12:55:28 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.17.1/8.17.1/Submit) id 32PJtSxF005246; Sat, 25 Mar 2023 12:55:28 -0700 (PDT) (envelope-from sgk) Date: Sat, 25 Mar 2023 12:55:28 -0700 From: Steve Kargl To: Peter Cc: freebsd-hackers@freebsd.org Subject: Re: Periodic rant about SCHED_ULE Message-ID: Reply-To: sgk@troutmask.apl.washington.edu References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4PkVF45YY9z3kQ4 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sat, Mar 25, 2023 at 08:08:11PM +0100, Peter wrote: > On Sat, Mar 25, 2023 at 10:54:17AM -0700, Steve Kargl wrote: > ! On Sat, Mar 25, 2023 at 04:47:42PM +0100, Peter wrote: > ! > --------------------------------------------------------------------- > ! > >> On 3/22/23, Steve Kargl wrote: > ! > >>> > ! > >>> I reported the issue with ULE some 15 to 20 years ago. > ! > > ! > Can I get the PR number, please? > ! > > ! > ! Don't remember if I ever files a bug report. > > Too bad. A bugreport has the advantage that one can attach fixes > to it, and that things get a lot more focused. If you follow the URLs I posted to the mailing lists and read the threads associated with those URLs, you'll find I was in direct contact with the individual who designed and implemented ULE. A bug report in 2008 would have sat for years unless I closed it, which I have done with other bug reports I've submitted over the years. > ! There's one > ! here that was simply closed. > ! > ! https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=85820 > > Hm, that is not really suitable. You can re-open a bug report and enshrine your patch. Or, you can create your own bug report with your analysis of the problem, and then enshrine your patch. -- Steve From nobody Sat Mar 25 20:34:50 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkW661hZ8z41gjM for ; Sat, 25 Mar 2023 20:34:54 +0000 (UTC) (envelope-from yuri@aetern.org) Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkW656sqqz3nl9; Sat, 25 Mar 2023 20:34:53 +0000 (UTC) (envelope-from yuri@aetern.org) Authentication-Results: mx1.freebsd.org; none Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 105D05C00E8; Sat, 25 Mar 2023 16:34:53 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Sat, 25 Mar 2023 16:34:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aetern.org; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm2; t= 1679776493; x=1679862893; bh=5DXbLDLElUO+355/HfuDeQeq81lNnW+W5tR zfN9lwCM=; b=Ek8qOL4HiJ/tAGy6OOvQ5gwflzjSbUVCzU0VfzrRi63I0JDPDcC alD0jlfIj1wjPGcgqyeo9nbjBByJEeMmiR+h9FaEChb7K0KPp4odI+i89hM5KKlP rbn8tQly+zvg3fugu1jK8B9poIijxmQQwKUwWkzOrTt67x8WJ11ff59WP4EHuYSx au1y4lDyxr+b7nNd8LsXxViXXgRNsSNRxYpiojJRvHH8r1Pvbf4d65u3V3G0ixAQ FbebtV+nuOMq4y2fjFbbuS4y8hPiCK3oRtSHjFYS0GzaFKFRFMwEYKyRCQ84epx0 FXqODsIfZsYRy8CTU0d2DTbhmO29E+pv4ng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t=1679776493; x= 1679862893; bh=5DXbLDLElUO+355/HfuDeQeq81lNnW+W5tRzfN9lwCM=; b=E epmVFzm7+CcA8ZBspPhAIUjWwlxNhFWIsETIYE+f62M2H6yAta+Zjoj8t62lTrV2 i2t4p9MjWuuZUrCc1up6cOFsOWMv2BoI/7pJc0N5Aa4VmJl6And3Zv8gaxEyj6hP ENW5CmtQka4it4bU1FpESYzGemdDwBo5Y1UiMKdHbqqbZFYi4z/a85VHA7bdZcuW bE8QE1ItIkeqAYX/EHBtBpLH/yg4tYnt0ZycFzmHfls+TJs5n0qvUBrnf/bn1Y14 Vn8uCHz7N9/WkLrLOPqkx1CPR2T9atFZAMOd57iZWVbtTGowaUTQ1kioRedLu0mQ maWWRITH2f2PkbyEyIHcw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdegkedgudegvdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefkffggfgfuvfhfhfgjtgfgsehtjeertddtfeejnecuhfhrohhmpegjuhhr ihcuoeihuhhrihesrggvthgvrhhnrdhorhhgqeenucggtffrrghtthgvrhhnpeegheelve euhefhgffgteetffeifffhudetudetuedvhedvhfffvdethfffjeejvdenucffohhmrghi nhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmh epmhgrihhlfhhrohhmpeihuhhrihesrggvthgvrhhnrdhorhhg X-ME-Proxy: Feedback-ID: i0d79475b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 25 Mar 2023 16:34:52 -0400 (EDT) Message-ID: <634f894e-02a7-6a03-c28b-c1cc5f4a1402@aetern.org> Date: Sat, 25 Mar 2023 21:34:50 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: git-apply(1): error: content/en/ports/_index.adoc: No such file or directory Content-Language: en-US To: Graham Perrin , freebsd-hackers@freebsd.org References: <670cff73-a3e6-8501-b33c-df358149adb7@freebsd.org> From: Yuri In-Reply-To: <670cff73-a3e6-8501-b33c-df358149adb7@freebsd.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4PkW656sqqz3nl9 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:19151, ipnet:66.111.4.0/24, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N Graham Perrin wrote: > Can anyone explain what's below? > > The .diff is downloaded from . > > > % file /usr/doc/website/content/en/ports/_index.adoc > /usr/doc/website/content/en/ports/_index.adoc: ASCII text > % git -C /usr/doc apply -v /tmp/D38510.diff > Checking patch content/en/ports/_index.adoc... > error: content/en/ports/_index.adoc: No such file or directory > % head -n 8 /tmp/D38510.diff > Index: website/content/en/ports/_index.adoc > =================================================================== > --- website/content/en/ports/_index.adoc > +++ website/content/en/ports/_index.adoc > @@ -11,21 +11,20 @@ > > ''''' > > % > The default value for stripping leading path components is 1, as usually the diff looks like: --- a/path/to/file +++ b/path/to/file ...and it's not there in the diff you are trying to apply. IOW, try specifying -p0. From nobody Sat Mar 25 20:41:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkWFr1t7Pz41hJg for ; Sat, 25 Mar 2023 20:41:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkWFq6Krtz3qZ4 for ; Sat, 25 Mar 2023 20:41:35 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679776893; bh=uEH1zw6LRuVjmd+gt0EkDjjGDuwAIRcYbZr9NR+e8HM=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=qDVocR48XPCveQVLxDoVpXYaKYWyNCEj0d5PFh/x/716jEoxH0OZNG/EYJP8Ow3XfCs4TvLepvLJMd17ZlfE1zxt5iIo2fWxGTtVUzLCCaSROgDmDIRgG/RiKmffrRQxbMxbs0J70jrbUjMQt0/XfRX7Z2dEFhLPO3QNnSyaIGLniKjo6waCGGCvM/mHECuAK+KENZbN4lwpFy1VelUamr7UPqWNXAPs3TaPsF2q035abKivwfWFT3y/ufbfAq3duj/QnDOuWn1B7xkRFBXXflrynmXQxxpjJajsBCKtwUQeo2KH3FMrWU/Jge8bVtj7BOK4MsDCan6vhedkZAQ+0w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1679776893; bh=yTKGpJl/0mvkD9LC5CCfwNk6CkM1BRc8quk/C5pLEzm=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=AjaGJsFEHgj/T70oLvVMYiEkmKwMSlM55bTdsYx6qQfQudMYdLKtzgw3YKD+J5gzpTOlfRlp+aFdTejfIpiLScUPQNI7tREcWXM3OlC6WOg0cS+TY56lBss17kKj9caVS0Vm05ljxy2w6wabR4mEgSeZKN1FH56DwgX7zNN9dyvDlIHs/c5KkCRJTSqJelyzhKon/ya02+YJLmmtlnEEkRiSrM4V9JuSgvG3bgpUrcS+6p7euQht5lm3C/DWvsr2blx95iUjQHHpL8ItJq4rsVYVu6zlmTZrl2Lc7CTiGEKVvI7asbJhiRmRz7t5nUO+jAZIbVzUO5+zBnaM2PzwNw== X-YMail-OSG: 9k22ctsVM1nBxhI7aoe.HTdxaK8.dvtlPPos28P6O_btScsG52c0fvm0BJDfGsJ 74ePMgm_bpj1eKc6I7NHTwqXsY0hcjqu21cdIb2DcFMH.1IJo7nuht938DNMq_km_sxlkBMBkFf_ Lt6Z6IBRXkXwDzbzIhhdePp5cpqp1SIeiPLI5KI1WuqB4iG79UytjEuIAHHViYK6cZi_ykvhEJTE Nao2jb6EjrwM1UO9Y3A7OAaS5vTyd3tv0OEjREmyzbovCNPSDillDds7cgqjp31tz3R6jUQQqk4l 2wEhfDYvjVo3HzZWyMrWtNv3D.RCgBs_.EBPHi0ET0Fl2LnEe13pizKH8cHdbZoiVxsd_3hp31Nf 92mZ._pUvetFzsXX9RpEYuK0dNaNvIG2nxf4Svs8a3k_zbbieV_xsbgsbqq4SYyfoBPGTucgy0B0 v_HhXJIxPO1QFHwEvbvMy1eN7ipr4AhPBD75fl6HcfMvPahQLXdbnRcdZx9y6dsJaNKYRsNa8Sh8 PUjgSpJL8mUqmwmNFMRCsQ5yWZhltMT0KF1PNiQ1WtvoRYOTU6.6Bc8efs9IdvqbH6wUMNjAzRTo pLTRuyIHmgP0Twy6R52UHratsJW0EBPFk0hFKu0Aw6BgG6rVfAtnaT19xIIuhUbRBekfrFFqmNH. lpOEkbdPFnOafJJIWU9tZKefGyNMXlXnWudnnDvUoI9Dc5r.TqIsZwMZvZg3dd6KTgNTxXw_lHRP MrckpHqkJgNdqFgvpTlQFSBqDrBWNXD8wkK1u2xiDW8OfaDYyaJdyJRNsEHjzKHdSmz23zenLlu7 aJmOMd.fnbcJhw6dPH.fALNRLJjnTnSKHg6_s6OcAs3Z7zUnT2bl7VdoVbEFp_L6hnCryFCYegP3 tl.D940P9pLNvcBdLsZrbnJGmhzESgwrI7BYDkvc4dggBi6MvjEv40xEMiv9nKSy735cSY6pykN_ 7adCFhi3NxSaYvRl_xp00BV3MAkt80UYiLzY77SMh2tWig3jerpWol156seFL.4Lxcjzwkqx2aoO 1dJNvjjAjOg8PZFKRsYP6GHiU1YEpfhr3zuTuu2ShEqkQVqeg3PIwDyZ7ZewzJnwwKsqKY.avhxb uh35aW9IBBMTJ9u_TNhpopKUamWywCKQfewg3XXRqEiKeRoA8LV5vGhQcMiHpNhWI8e6a7a89xBi MEuGb3u9roxmluEnJHevofXneiW_jhhLRz.kCE2JnbQ9FSZoTaYdw.ltEO78z1soayroTdNQK9xl b3TBGimoG3ndg0IapaPs_WQzc8TRMvopJjdjDT137MJNV7BnYzg3h59aZ3XbCXLxiLeauPBymsU2 hozwJphC9VqrHx5aRB_yMq.YfMRtPdUNLtrENyNaWbOG4CjvKQqSN10DEdlVnEBCWwqCtNY51hja bWm9JybpFQaqTEJEsQd0ikUXwHRJIOvCjgF5NSzck7kg7PbG3Vq5v.cGGgzEUwxWSmM4wQ.KeHUL i2MCtuTv6TbWbXNsbbPr4dLKbjy1yJJb8NrnyWZ6keKOBvM_pg_009Jqgvjd7_vhlbf4b18lYLlv yTd9uVQVPoQUyfBdWQvtf.eyGGRV8rDexhQnxmwXvRAoLsqD1bBfgl_RwLzPQbROKIzycSz0AR_. FI5HmzTubwWtB8ocYxyCQyhZucUUsii1AkWN_hVHOTbnn6hucGNBUsjSMHFFgi4RwBrLBZXAeMtr YdkEUhiWGylcJ5epNC8XJoHp6pg._mTC3.NhrGVutZMB.QDsWZnaJRrWKNmkW4UAIbAvoC4jyC_f Yd369SGr9kpX9QCnOVo6NEqYsvvIYSxrxeJKsBugIx7JegY6yWIkvHq5aM0LeX7ZAgHz5g7E4MbX neMAwXzR97EP9zqZ9X7BbqqmY1XsENeP6bRpW72vuQXGPb6NjkCPb_BwBB53RSoOwrNuooBuKR4t Dht0IQ64I.b6pUjzE9CBVhoK9uKOlRbJ3GmQn2_zGkcEMKZdvEBqZ9p46IvF3_wAzRWx.FeSJViE vFUcfA_lEk1Y6W_QVE_cwySU5I8eZMWnMfqSnx8kb7SJgSjZK2dszOXiWcSydzj.yuvTRYfprjBW _ib25Jo5a3ayHj1o1N.5geBTABjEz7KGTdzoWsit6clR5IHnUJcIU2oXuoPfU2g89gsabRMbB9ys eOglfVwMSjyHIMIvboweJ2hj53fySifPjHiMlIbfggNmhyU.5.BWaRhGx7lF92_HbYm5dItULkkK ywbFuOorETFcZelqjxeHQg8drX_ea2xtAqaiK1D8Fp4mNrFRmJtqyAPqK._frKcAOnQwuyuycdIg Exg-- X-Sonic-MF: X-Sonic-ID: 81d9cae1-b872-4745-95ca-ef6023ee7d82 Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Sat, 25 Mar 2023 20:41:33 +0000 Received: by hermes--production-ne1-759c9b8c64-f7wvp (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID fd861c660fd6ae1e34f6a95bba34e908; Sat, 25 Mar 2023 20:41:28 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Periodic rant about SCHED_ULE From: Mark Millard In-Reply-To: Date: Sat, 25 Mar 2023 13:41:16 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> To: Peter X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4PkWFq6Krtz3qZ4 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mar 25, 2023, at 11:58, Peter wrote: > On Sat, Mar 25, 2023 at 11:14:11AM -0700, Mark Millard wrote: >=20 > ! Why did PID 10675 change to 19028? >=20 > Because it went into some NFS share, and it would still be there if I > hadn't restartet it a bit differently. >=20 > ! When I tried that tar line, I get lots of output to stderr: > !=20 > ! # tar cvf - / | cpuset -l 13 gzip -9 > /dev/null > ! tar: Removing leading '/' from member names > ! a . > ! a root > ! a wrkdirs > ! a bin > ! a usr > ! . . . > !=20 > ! Was that an intentional part of the test? >=20 > Yes. So you can see what it is currently feeding to gzip (small or > big files - or some NFS share, where the operation becomes pointless). >=20 > ! # tar cvf - / 2>/dev/null | cpuset -l 13 gzip -9 2>&1 > /dev/null I should have had at the the order >/dev/null 2>&1 Sorry. > !=20 > ! At which point I get the likes of: > !=20 > ! 17129 root 1 68 0 14192Ki 3628Ki RUN 13 0:20 = 3.95% gzip -9 > ! 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 0:00 = 0.27% tar cvf - / (bsdtar) > ! 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 8:05 = 95.93% sh -c while true; do :; done > !=20 > ! up front. >=20 > Ah. So? To me this doesn't look good. If both jobs are runnable, they > should each get ~50%. >=20 > ! For reference, I also see the likes of the following from > ! "gstat -spod" (it is a root on ZFS context with PCIe Optane media): >=20 > So we might assume that indeed both jobs are runable, and the only > significant difference is that one does system calls while the other > doesn't. >=20 > The point of this all is: identify the malfunction with the most > simple usecase. (And for me here is a malfunction.) > And then, obviousely, fix it. I tried the following that still involves pipe-io but avoids file system I/O (so: simplifying even more): cat /dev/random | cpuset -l 13 gzip -9 >/dev/null 2>&1 mixed with: cpuset -l 13 sh -c "while true; do :; done" & So far what I've observed is just the likes of: 17736 root 1 112 0 13364Ki 3048Ki RUN 13 2:03 = 53.15% sh -c while true; do :; done 17735 root 1 111 0 14192Ki 3676Ki CPU13 13 2:20 = 46.84% gzip -9 17734 root 1 23 0 12704Ki 2364Ki pipewr 24 0:14 = 4.81% cat /dev/random Simplifying this much seems to get a different result. Pipe I/O of itself does not appear to lead to the behavior you are worried about. Trying cat /dev/zero instead ends up similar: 17778 root 1 111 0 14192Ki 3672Ki CPU13 13 0:20 = 51.11% gzip -9 17777 root 1 24 0 12704Ki 2364Ki pipewr 30 0:02 = 5.77% cat /dev/zero 17736 root 1 112 0 13364Ki 3048Ki RUN 13 6:36 = 48.89% sh -c while true; do :; done It seems that, compared to using tar and a file system, there is some significant difference in context that leads to the behavioral difference. It would probably be of interest to know what the distinction(s) are in order to have a clue how to interpret the results. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Sat Mar 25 21:25:38 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkXDq2ST4z41kFQ for ; Sat, 25 Mar 2023 21:25:47 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkXDq1n8Nz3vyL; Sat, 25 Mar 2023 21:25:47 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679779547; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=vnsFVY0D4QI6RjEymbeulUePKPrw7tnh2iiCaqY1q/0=; b=HS3PuU74PCJIL9TLV+DKz+YMOiQ6GKhUhs6BtyGqSro9+8tdqQWrlQ8zhqC677GDj09Me+ 2jkmHG9D/Nt1KEvRkjH7bs0pevEksk1D6FqbOxrphdfYT7Xxl9JrPeK56IpQ7yHKCbHvmi hAzAEllAH4WHYp84XpvWkmLqrOjiWkkUiTMwBPh8xW0ssNKbN9WowXmHPJQiRrAjWUi+AB RdQ2PsdX3cxRrLdvIq0/kRdqnTqDd1jFhUqG0dpcEyB1zZjXoS8v/AZDthjlbi3jbqnlk/ sdpgqigDxql1LTMfYraHJ9DsTtOuTfd0F+r8G1QVL99991fDyQXt26KGoNRXzQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679779547; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=vnsFVY0D4QI6RjEymbeulUePKPrw7tnh2iiCaqY1q/0=; b=n25798OUAVhEWA5den1KQEyTbicRhI8g+oD6B6JfE1pJZ+sqvW+NSug41cxqnxl4RmvPOX M8m8mhbIKegcmsQs7ZMfFhwZGKZZ/cxz9a8uTEOBWMDUW4iMdpq9KgipjLT10Yln6/iXGs J2gljb/3bmO2TH4Jfq816An59zBXGCc+n77FY+LWKnHu/xXVdUMBk5hBUFgfCmaDEOAUYe LchgkT4v6uP6/g5bbS9WNwexFyxeYULKL759XujOFVyDZl96uhSWBS0vVerQaddQR3zSFp Dj/w+pRNmMZgfh2oBl0KR/XIoJpnRHl0IahYlDNHPbTng+rvnEjEGYVs8UOEjQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679779547; a=rsa-sha256; cv=none; b=VypfnIQYeoksiV0WLeQh9Vx6GDguK++jyaYYb0c+Zbf8VhKdP8MJ9PMpp1IIyMQrgnc+xU wJkqULMBRJ1cKpHnK4pM8e7tk6Acf0MHEAr+Yu27WHbVgG3Dxgpy4jDenWKbID+mGq3QIG gp2KhvAfNeKQANv3Tg8D4cY/KyEwMs6oOUn4B+V6TL578VXgQgemfe0MYv2wRaexL5ulw7 MFQS+P8IsIYx1OM2c5lPWDB6bSGS1tViHxSD/5qod/Mf24MiC6d16hI4ByjtC1fhWhqDeM raB1Ytyf/Lc6QRo8sUb0yvYibFA2hERJAjXJy2uaCy2PzKNQPviKLMga4iE99Q== Received: from [IPV6:2001:470:1f1c:a0::2] (tunnel642390-pt.tunnel.tserv1.lon2.ipv6.he.net [IPv6:2001:470:1f1c:a0::2]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: grahamperrin) by smtp.freebsd.org (Postfix) with ESMTPSA id 4PkXDm3LrzzfS4; Sat, 25 Mar 2023 21:25:43 +0000 (UTC) (envelope-from grahamperrin@freebsd.org) Message-ID: <954dfd7a-cedc-448e-2f6f-73c73f437704@freebsd.org> Date: Sat, 25 Mar 2023 21:25:38 +0000 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: git-apply(1): error: content/en/ports/_index.adoc: No such file or directory Content-Language: en-US, en-GB To: freebsd-hackers@freebsd.org, Pau Amma References: <670cff73-a3e6-8501-b33c-df358149adb7@freebsd.org> <634f894e-02a7-6a03-c28b-c1cc5f4a1402@aetern.org> From: Graham Perrin Organization: FreeBSD In-Reply-To: <634f894e-02a7-6a03-c28b-c1cc5f4a1402@aetern.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------hJkjSY4oZma74ntNvq8L1iFz" X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------hJkjSY4oZma74ntNvq8L1iFz Content-Type: multipart/mixed; boundary="------------xu5bser8vlI0ofWDBniXeeN3"; protected-headers="v1" From: Graham Perrin To: freebsd-hackers@freebsd.org, Pau Amma Message-ID: <954dfd7a-cedc-448e-2f6f-73c73f437704@freebsd.org> Subject: Re: git-apply(1): error: content/en/ports/_index.adoc: No such file or directory References: <670cff73-a3e6-8501-b33c-df358149adb7@freebsd.org> <634f894e-02a7-6a03-c28b-c1cc5f4a1402@aetern.org> In-Reply-To: <634f894e-02a7-6a03-c28b-c1cc5f4a1402@aetern.org> --------------xu5bser8vlI0ofWDBniXeeN3 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64 T24gMjUvMDMvMjAyMyAyMDozNCwgWXVyaSB3cm90ZToNCj4gR3JhaGFtIFBlcnJpbiB3cm90 ZToNCj4+IOKApg0KPiBUaGUgZGVmYXVsdCB2YWx1ZSBmb3Igc3RyaXBwaW5nIGxlYWRpbmcg cGF0aCBjb21wb25lbnRzIGlzIDEsIGFzIHVzdWFsbHkNCj4gdGhlIGRpZmYgbG9va3MgbGlr ZToNCj4NCj4gLS0tIGEvcGF0aC90by9maWxlDQo+ICsrKyBiL3BhdGgvdG8vZmlsZQ0KPg0K PiAuLi5hbmQgaXQncyBub3QgdGhlcmUgaW4gdGhlIGRpZmYgeW91IGFyZSB0cnlpbmcgdG8g YXBwbHkuDQo+DQo+IElPVywgdHJ5IHNwZWNpZnlpbmcgLXAwLg0KDQoNClRoYW5rcyEgUGF1 IEFtbWEgb2ZmZXJlZCB0aGUgc2FtZSBoaW50LCB3aXRoIGEgcmVsZXZhbnQgcXVvdGUgZnJv bSB0aGUgDQptYW51YWwgcGFnZS4NCg0KQXBwbGllZCwgY29tbWl0dGVkOiANCjxodHRwczov L2dpdGh1Yi5jb20vZnJlZWJzZC9mcmVlYnNkLWRvYy9jb21taXQvN2EyODEzMDk5YmE1Njhi NDIwMTRjMTRjMDU1NGQxMTUzZTY4NWNhNT4NCg0K --------------xu5bser8vlI0ofWDBniXeeN3-- --------------hJkjSY4oZma74ntNvq8L1iFz Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsF5BAABCAAjFiEEWT/lssMHB+28ly8Kt2dIb0oY1AsFAmQfZtIFAwAAAAAACgkQt2dIb0oY1Aug dhAAi300bSBMt73pFiz+j7UbkdGOJGRXroVIWkHJT3Zo/IyGU8Rp0ra6rsxNzZTKPvX/oXLzvEXZ 4OiNQtdp+C3fkUt3GB2mlDkqX8h5EJhMdvakiHwHuBR2lxC9zF28hMAeHkCVOdqN9FhHV+NLpFiB jA1sqr+SQtLfvQL8QrIOsv8pFiVIwr/oHVXeEFsUeJxGSbWgdSF6EyFhy0TdrV3/h2DP4jm+JwLn E35w/zGN+ECJ5iW3ALB7pGiLo/OzTtlQKytW7WZySKS6xV+eB77cZSlWQYygsL74q0d97WCcsaLw Y/nvbuVhcx4EKEvezLo2f7c0LrDnyaE7lzoVtBYYPS7/zYVFlQV4PhpdHnxz2GJZSbp2qlJPXy0Z qsKBwuO1XxZyTi1l4NX8/HTJYQuwkPRX51VU8nMMmiH1W4upGqMjv8K2OxEsIG4V5FrHBrja9rVd VzVjyYOHft4088a1qqZzYJnc/eGUcUBdg6ywF+k5gXpRjpHoCKQ4DS5YEjP8zlriUDuNpQJV6Oeq g2dyVwI5riztpPz9eJj8tuEVE/bjQuWZWYugOSMWS6G+GigmSj4NhduwVQynHI66Y2bRydCl8hLy l6sbWZaRSs9pFJrz/MUhyDURemLw6rssiqUhz+IMdrj/RVrtwXgL1HP/uVL6XAWIJHgUHUpYg7wZ 0e4= =V3Fd -----END PGP SIGNATURE----- --------------hJkjSY4oZma74ntNvq8L1iFz-- From nobody Sat Mar 25 21:51:49 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkXt93M03z41mRQ for ; Sat, 25 Mar 2023 21:54:41 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkXt90K8xz40Nm for ; Sat, 25 Mar 2023 21:54:40 +0000 (UTC) (envelope-from pmc@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 32PLs6mH016236 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 25 Mar 2023 22:54:07 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 32PLs699016235; Sat, 25 Mar 2023 22:54:06 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (disp-e.intra.daemon.contact [IPv6:fd00:0:0:0:0:0:0:112]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32PLqcsv075208 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=OK); Sat, 25 Mar 2023 22:52:40 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: from disp.intra.daemon.contact (localhost [127.0.0.1]) by disp.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 32PLpn8O013551 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 25 Mar 2023 22:51:50 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) Received: (from pmc@localhost) by disp.intra.daemon.contact (8.17.1/8.17.1/Submit) id 32PLpnEZ013550; Sat, 25 Mar 2023 22:51:49 +0100 (CET) (envelope-from pmc@citylink.dinoex.sub.org) X-Authentication-Warning: disp.intra.daemon.contact: pmc set sender to pmc@citylink.dinoex.sub.org using -f Date: Sat, 25 Mar 2023 22:51:49 +0100 From: Peter To: Mark Millard Cc: FreeBSD Hackers Subject: Re: Periodic rant about SCHED_ULE Message-ID: References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Sat, 25 Mar 2023 22:54:09 +0100 (CET) X-Rspamd-Queue-Id: 4PkXt90K8xz40Nm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Sat, Mar 25, 2023 at 01:41:16PM -0700, Mark Millard wrote: ! On Mar 25, 2023, at 11:58, Peter wrote: ! > ! ! > ! At which point I get the likes of: ! > ! ! > ! 17129 root 1 68 0 14192Ki 3628Ki RUN 13 0:20 3.95% gzip -9 ! > ! 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 0:00 0.27% tar cvf - / (bsdtar) ! > ! 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 8:05 95.93% sh -c while true; do :; done ! > ! ! > ! up front. ! > ! > Ah. So? To me this doesn't look good. If both jobs are runnable, they ! > should each get ~50%. ! > ! > ! For reference, I also see the likes of the following from ! > ! "gstat -spod" (it is a root on ZFS context with PCIe Optane media): ! > ! > So we might assume that indeed both jobs are runable, and the only ! > significant difference is that one does system calls while the other ! > doesn't. ! > ! > The point of this all is: identify the malfunction with the most ! > simple usecase. (And for me here is a malfunction.) ! > And then, obviousely, fix it. ! ! I tried the following that still involves pipe-io but avoids ! file system I/O (so: simplifying even more): ! ! cat /dev/random | cpuset -l 13 gzip -9 >/dev/null 2>&1 ! ! mixed with: ! ! cpuset -l 13 sh -c "while true; do :; done" & ! ! So far what I've observed is just the likes of: ! ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 2:03 53.15% sh -c while true; do :; done ! 17735 root 1 111 0 14192Ki 3676Ki CPU13 13 2:20 46.84% gzip -9 ! 17734 root 1 23 0 12704Ki 2364Ki pipewr 24 0:14 4.81% cat /dev/random ! ! Simplifying this much seems to get a different result. Okay, then you have simplified too much and the malfunction is not visible anymore. ! Pipe I/O of itself does not appear to lead to the ! behavior you are worried about. How many bytes does /dev/random deliver in a single read() ? ! Trying cat /dev/zero instead ends up similar: ! ! 17778 root 1 111 0 14192Ki 3672Ki CPU13 13 0:20 51.11% gzip -9 ! 17777 root 1 24 0 12704Ki 2364Ki pipewr 30 0:02 5.77% cat /dev/zero ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 6:36 48.89% sh -c while true; do :; done ! ! It seems that, compared to using tar and a file system, there ! is some significant difference in context that leads to the ! behavioral difference. It would probably be of interest to know ! what the distinction(s) are in order to have a clue how to ! interpret the results. I can tell you: With tar, tar can likely not output data from more than one input file in a single output write(). So, when reading big files, we get probably 16k or more per system call over the pipe. But if the files are significantly smaller than that (e.g. in /usr/include), then we get gzip doing more system calls per time unit. And that makes a difference, because a system call goes into the scheduler and reschedules the thread. This 95% vs. 5% imbalance is the actual problem that has to be addressed, because this is not suitable for me, I cannot wait for my tasks starving along at a tenth of the expected compute only because some number crunching does also run on the core. Now, reading from /dev/random cannot reproduce it. Reading from tar can reproduce it under certain conditions - and that is all that is needed. From nobody Sat Mar 25 22:09:56 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PkYCr1cj6z41mgQ for ; Sat, 25 Mar 2023 22:10:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oi1-x236.google.com (mail-oi1-x236.google.com [IPv6:2607:f8b0:4864:20::236]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PkYCq2kbGz448C for ; Sat, 25 Mar 2023 22:09:59 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oi1-x236.google.com with SMTP id b19so3797757oib.7 for ; Sat, 25 Mar 2023 15:09:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679782197; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=o1Qm3bfyKU9LaUK/9woJ5bCbJN05+hgE2DuCm+/kj10=; b=Y3+lOsOkmdtpNBfaiQIh3OBzzSwNXQuwCtkzft0sm8YgXjwGavvrjpaVoDNuGhL+VJ GYpmRYjQdrVDIQ8YERh/G2vkJdIkXw000ad9Xr44/c4+Nx4ULnI1u/nVcM55gUOIQtME xVEv9rZsx/TL4e814Jc4+ts16GuvGKl3bTwH+Qc1CQI/bS6QWO4WDZd2A96X6OKeM3WE BRfJRzj9HDoUWR7cbyf2w73KzpMGZgJtvsQXGKw8H3/ppH/LDaBZJilpBFR4acOOl3Tj d3RLcOXrBZsjlzeenB0WUFLHd1S2HKLBpGFA9AqWu9WUOBDBSH1LaB1bVDMfXPDzZ3bR 4ofw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679782197; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=o1Qm3bfyKU9LaUK/9woJ5bCbJN05+hgE2DuCm+/kj10=; b=7gIhnw9KP5qER8XR78XxdgYxola+3SGShyK+3g2Kq/KKKteh/TKjW7yMeAy6gcppj0 Sfz2IG6t4EZtrIOnbWLJwp+MLnGr3WjdJ3asaIm2Q5bxmRALzNWssp1ZmUPqPg5WITSe qZ2jUBHDTxEuUSzJ9ubhdJiJ45T7PW+/yglUEvw5c2l42LDKwcYsoHzIKnySNfOIqh2M Eipn7dVXMwaWe6ELDgJIMlltDYOYyKCkTu6klF6ckGSAF/IsApo94j61xUlPMcHnE9Id uVrB7XwQb0oxpP0fpQDzkGcEfvSasU8f+hQh2/lxRkEGfl1JQQrsAeZT4tWqI3WoTp35 b7xg== X-Gm-Message-State: AO0yUKXlX/HY2TXLqIJJpRJUixhI/+BW5iSMUuGTSTsFVBinQKTKO9oR 9Nt1TZOq70CnxrWVSVRIBmwiLNhAVR5QOWRZ+ZI= X-Google-Smtp-Source: AK7set+8aeTTkjpDHTwysBwDezjGvMzJcp5DKh65SYONBziO/yDOvP2p6Qck78PLtI/9BSgi01pfSP5uErg0LBxn+hY= X-Received: by 2002:a54:4108:0:b0:384:3518:df33 with SMTP id l8-20020a544108000000b003843518df33mr2086016oic.4.1679782197265; Sat, 25 Mar 2023 15:09:57 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:12c2:0:b0:49c:b071:b1e3 with HTTP; Sat, 25 Mar 2023 15:09:56 -0700 (PDT) In-Reply-To: References: <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA.ref@yahoo.com> <5AF26266-5B4C-4A7F-8784-4C6308B6C5CA@yahoo.com> From: Mateusz Guzik Date: Sat, 25 Mar 2023 23:09:56 +0100 Message-ID: Subject: Re: Periodic rant about SCHED_ULE To: Peter Cc: Mark Millard , FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4PkYCq2kbGz448C X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On 3/25/23, Peter wrote: > On Sat, Mar 25, 2023 at 01:41:16PM -0700, Mark Millard wrote: > ! On Mar 25, 2023, at 11:58, Peter wrote: > > ! > ! > ! > ! At which point I get the likes of: > ! > ! > ! > ! 17129 root 1 68 0 14192Ki 3628Ki RUN 13 0:20 > 3.95% gzip -9 > ! > ! 17128 root 1 20 0 58300Ki 13880Ki pipdwt 18 0:00 > 0.27% tar cvf - / (bsdtar) > ! > ! 17097 root 1 133 0 13364Ki 3060Ki CPU13 13 8:05 > 95.93% sh -c while true; do :; done > ! > ! > ! > ! up front. > ! > > ! > Ah. So? To me this doesn't look good. If both jobs are runnable, they > ! > should each get ~50%. > ! > > ! > ! For reference, I also see the likes of the following from > ! > ! "gstat -spod" (it is a root on ZFS context with PCIe Optane media): > ! > > ! > So we might assume that indeed both jobs are runable, and the only > ! > significant difference is that one does system calls while the other > ! > doesn't. > ! > > ! > The point of this all is: identify the malfunction with the most > ! > simple usecase. (And for me here is a malfunction.) > ! > And then, obviousely, fix it. > ! > ! I tried the following that still involves pipe-io but avoids > ! file system I/O (so: simplifying even more): > ! > ! cat /dev/random | cpuset -l 13 gzip -9 >/dev/null 2>&1 > ! > ! mixed with: > ! > ! cpuset -l 13 sh -c "while true; do :; done" & > ! > ! So far what I've observed is just the likes of: > ! > ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 2:03 > 53.15% sh -c while true; do :; done > ! 17735 root 1 111 0 14192Ki 3676Ki CPU13 13 2:20 > 46.84% gzip -9 > ! 17734 root 1 23 0 12704Ki 2364Ki pipewr 24 0:14 > 4.81% cat /dev/random > ! > ! Simplifying this much seems to get a different result. > > Okay, then you have simplified too much and the malfunction is not > visible anymore. > > ! Pipe I/O of itself does not appear to lead to the > ! behavior you are worried about. > > How many bytes does /dev/random deliver in a single read() ? > > ! Trying cat /dev/zero instead ends up similar: > ! > ! 17778 root 1 111 0 14192Ki 3672Ki CPU13 13 0:20 > 51.11% gzip -9 > ! 17777 root 1 24 0 12704Ki 2364Ki pipewr 30 0:02 > 5.77% cat /dev/zero > ! 17736 root 1 112 0 13364Ki 3048Ki RUN 13 6:36 > 48.89% sh -c while true; do :; done > ! > ! It seems that, compared to using tar and a file system, there > ! is some significant difference in context that leads to the > ! behavioral difference. It would probably be of interest to know > ! what the distinction(s) are in order to have a clue how to > ! interpret the results. > > I can tell you: > With tar, tar can likely not output data from more than one input > file in a single output write(). So, when reading big files, we > get probably 16k or more per system call over the pipe. But if the > files are significantly smaller than that (e.g. in /usr/include), > then we get gzip doing more system calls per time unit. And that > makes a difference, because a system call goes into the scheduler > and reschedules the thread. > > This 95% vs. 5% imbalance is the actual problem that has to be > addressed, because this is not suitable for me, I cannot wait for my > tasks starving along at a tenth of the expected compute only because > some number crunching does also run on the core. > > Now, reading from /dev/random cannot reproduce it. Reading from > tar can reproduce it under certain conditions - and that is all that > is needed. > > So far it's look like it's not syscall usage per se, but "voluntary" time spent off cpu. Any time a thread goes off in a manner different than preemption it adds to counters which ULE interprets to mean the thread is "less interactive" and threads which do stay on cpu on their own should be preferred -- this is how syscall-less hog is considered important. The mechanism lacks distinction between sleeps on purpose, waitpid, yield and similar vs going off cpu due to lock contention or initiating i/o. Even then, I don't know if the mechanism works at all. That said, it may be just whacking all that "interactivity scoring" will happen to resolve this bug. No to be confused with making things optimal. -- Mateusz Guzik