From nobody Fri Mar 31 21:33:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PpDBj2x1dz43HX2 for ; Fri, 31 Mar 2023 21:36:45 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PpDBh3PmMz3mxF for ; Fri, 31 Mar 2023 21:36:44 +0000 (UTC) (envelope-from jroberson@jroberson.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=jroberson-net.20210112.gappssmtp.com header.s=20210112 header.b=nhlv0j+a; spf=none (mx1.freebsd.org: domain of jroberson@jroberson.net has no SPF policy when checking 2607:f8b0:4864:20::102e) smtp.mailfrom=jroberson@jroberson.net; dmarc=none Received: by mail-pj1-x102e.google.com with SMTP id j13so21842255pjd.1 for ; Fri, 31 Mar 2023 14:36:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jroberson-net.20210112.gappssmtp.com; s=20210112; t=1680298603; h=mime-version:references:message-id:in-reply-to:subject:to:from:date :from:to:cc:subject:date:message-id:reply-to; bh=GkvLA4J7/67CS6q1egDijef6oduiJIB+874ml0PnuTY=; b=nhlv0j+adZkrDon5a7lK4BiSW4b/MXTJkWmAKH0gAYzSCukdOraYqcHU1sm4NmBlP4 enTtio70LtWpKtf4qo1P5DwBZ3LJw8XRrsUiFMkIxNeh1OZxGYj3rj9kJjtTF+sDf7S9 +5gg7o09qsbthBfTPH4rLUIzO+zExqooO9jJ8qek8pD+AalfSUyCErS+RKm7aX4TZKfe Hyn1eHXPA9wu9hz1M2HvgoiWCUa0RxBbYwQ/dVnxwx7KgYhjxauUH1K+jReX3Un+ozMK M6MK7Hsxmyf1rqHQjzVQ90G+VQtZo3Quop8wPjgTS858bl9Iqm+p1i6K7irSLVX6wAUH ralg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680298603; h=mime-version:references:message-id:in-reply-to:subject:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GkvLA4J7/67CS6q1egDijef6oduiJIB+874ml0PnuTY=; b=pGvwPZvjyhoCJpruq69iNrD1rIIppNYb9aLsPHrYa2maRmcj+rXnvrwdRwBBIDGO64 4ZjHInUGHS/m0SI+iTIfDuYTgSb6XyUvQQzPjtD/avxRq5M6NlgXgPP13DLxF8HYhQ2w 4K1+KKKx5i+q4srppnf08WG99WP9iNoh2cv6twFqoat6wSYwVZYi1zOlpAqMAnd7CFyZ O2gCMo0gTuED96pMfG1RBVwx6IvXy485WAXNYrKzesHFOIpGS/lo+IHeOFk6nRV2uDPf dc6KJUN1iXpgzmAAc2uv1zbt2QiizQmT5qEHtdLNWE61o+9b0D0tcuwoEIIJTndWYXye GyXQ== X-Gm-Message-State: AAQBX9cRU7boik/jJFME7BGWT98vJN3PnuJiGH2vJg5p+fCVli3+vYpk 6t8PTiY7V4cnQtiPPWuE0acuana80BMwTWnsvac= X-Google-Smtp-Source: AKy350Y1ysbafr60vhkYXjxdWiL3s4Jlc5cz5Uj1DLuBg4uzGFnxzYDMGa7KXwip/wgWrkAvkqBisw== X-Received: by 2002:a17:90b:1b12:b0:23f:46a5:248e with SMTP id nu18-20020a17090b1b1200b0023f46a5248emr31017530pjb.44.1680298602903; Fri, 31 Mar 2023 14:36:42 -0700 (PDT) Received: from [192.168.0.31] (c-98-246-66-2.hsd1.wa.comcast.net. [98.246.66.2]) by smtp.gmail.com with ESMTPSA id x3-20020a17090a8a8300b00234465cd2a7sm1881713pjn.56.2023.03.31.14.36.41 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 31 Mar 2023 14:36:42 -0700 (PDT) Date: Fri, 31 Mar 2023 14:33:16 -0700 (PDT) From: Jeff Roberson To: freebsd-hackers@freebsd.org Subject: Re: ULE process to resolution In-Reply-To: Message-ID: <11380305-6261-6c08-fd15-299e695fa342@jroberson.net> References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[jroberson-net.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; R_SPF_NA(0.00)[no SPF record]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102e:from]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[jroberson-net.20210112.gappssmtp.com:+]; RCVD_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[jroberson.net]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Queue-Id: 4PpDBh3PmMz3mxF X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N I found an old patch of mine that addresses some of the issues with rapid sleeping/waking batch processes here: https://reviews.freebsd.org/D15985 Seems there are some bits relevant to behavior described earlier on hackers@. I was not subscribed to this list so I can't reply to the specific message. Jeff On Fri, 31 Mar 2023, Jeff Roberson wrote: > Hi Folks, > > For those who don't know, I am the original author of ULE. I have not had > much time for FreeBSD in recent years but this thread was forwarded to me and > I am dishearetened at the state of things. I will give my perspective and > propose a path to resolve this systematically. > > The fundamental benefit of ULE is also the fundamental challenge, That is: N > cpu local decisions need to add up to a reasonable approximation of a correct > global decision. This is necessary to scale to large core counts, large > thread counts, and preserve some affinity. You could permute 4BSD further > towards these goals but I posit that you would simply have to work through > the same bugs. > > As I read these threads I can state with a high degree of confidence that > many of these tests worked with superior results with ULE at one time. It may > be that tradeoffs have changed or exposed weaknesses, it may also be that > it's simply been broken over time. I see a large number of commits intended > to address point issues and wonder whether we adequately explored the > consquences. Indeed I see solutions involving tunables proposed here that > will definitively break other cases. > > I know that CPU tradeoffs have changed. ULE was written in a way that the > topology could be annotated and cost of migration can be specified. It is > adaptable to this but someone has to put in the effort. The cost function > was written in ticks which does not scale down properly and accurate cpu tick > counters could now be used for more precise time-keeping for more specific > affinity. Over time people have also added additional searches to pickcpu > which don't scale well to very high core count systems. NUMA and > heterogeneous CPUs are also possible in the graph framework but need further > investment. > > The other thing that has changed over time is the ability of the > interactivity score to correctly detect truely interactive applications. When > I wrote it you could do a buildworld on a single core or small multi-core > system and play mp3s and browse the web without a hiccup. However, web > browsers have evolved to be significantly more resource intensive. I'm not > sure a heuristic can or should catch this case. We're probably long overdue > to add x window focus hints as most other operating systems do. I don't > think tossing the interactivity score is really going to produce the desired > results. Linux CFS disagrees with me but I have always been able to achieve > superior responsiveness with ULE. My intuition is that with an x window focus > hint we could dial back the interactive threshold and have better tradeoffs > with the soft real-time score. > > schedgraph is also no longer adequate for modern systems. In my professional > life I have taken the same types of data sources and built text based > processes on top because graphical representations just can't scale to the > number of events and cores for full system scheduling. For complex > scheduling issues you need detailed introspection. You're not going to tweak > variables and run buildworlds to arrive at success by supposition with any > kind of reasonable velocity. > > The first step to resolving this is to come up with a list of regression > tests and catalog how they behave compared to 4BSD. When I wrote the > scheduler I also wrote a simple fixed duty cycle program that could be run > with different scheduling parameters and report on its cpu usage and latency. > Combining many copies of this program you can simulate various kinds of > interactions. It is available at people.freebsd.org/~jeff/late.tgz. I know > there is also a linux scheduler benchmark that may be worth porting. > > If someone would start making regression tests I am happy to fix bugs or > review bug fixes. Personally I would start from fairness given different > nice values on a single CPU, and then multi-cpu. Evaluate allocation with > variation on load to core count ratios. It should not take a few hours to > iterate through the interesting cases here before going on to more complex > questions about buildworld or firefox etc. This would need to be something > we carried forward in the source tree and ask people to re-run as part of > scheduler CRs or we're just going to find ourselves back in this spot again. > > I also have a backlog of improvements for large multi-core systems from work > I did years ago that have not made it into the tree. And I have an old > review for patches to improve the reliability of priority in causing > scheduling events that may be germane. If we can collaborate on a testing > framework I could trickle these in. > > Thanks, > Jeff > From nobody Mon Apr 3 08:20:38 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PqkNq6hXVz43rJF for ; Mon, 3 Apr 2023 08:20:43 +0000 (UTC) (envelope-from yuri@aetern.org) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PqkNq2Jkrz3FRr for ; Mon, 3 Apr 2023 08:20:43 +0000 (UTC) (envelope-from yuri@aetern.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=aetern.org header.s=fm2 header.b=nGX0BoIv; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=NCgISY9z; spf=pass (mx1.freebsd.org: domain of yuri@aetern.org designates 64.147.123.24 as permitted sender) smtp.mailfrom=yuri@aetern.org Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id C87973200906 for ; Mon, 3 Apr 2023 04:20:41 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Mon, 03 Apr 2023 04:20:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aetern.org; h=cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:message-id:mime-version:reply-to:sender :subject:subject:to:to; s=fm2; t=1680510041; x=1680596441; bh=nU 5KDuc7P2GqzGSuUVs//ip3GbTS8RwIfI0GMuhNpT0=; b=nGX0BoIvzWSNliRGI4 NEW39Yj4uLAl/rH2GQlodVhZqb1CGsvGD1dew8cI5su9vyRW/DhGP5sjBN92+iLV tpS/vaVgAngcOofauowFeOpYXrANStOIgc4TLNdA5sCsA9FBNrUT9mNJ24IQ9M0b gKjrAMWI40jDHj6yBquISsnHSDTvuRmvnIHaEEJV/uQZyDT6BnEB6MyNdwjFSUL1 oeN9XRZ6iYJl18WUV1IB6N9Zbkh7rVdo1sIQuLro7txgSgqV463CjtNWS1dJyEus ixl2QR0edLLhmL8gAj0ZfGm3i4uNsPeXWC9qZZ3/tNbgbzSw13fxBUV4nAAVF19j 6hSw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:message-id:mime-version:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; t=1680510041; x=1680596441; bh=nU5KDuc7P2Gqz GSuUVs//ip3GbTS8RwIfI0GMuhNpT0=; b=NCgISY9zS743cZd97iVUVQizxpOhz Ph2BfF4sWnEJy9j7WcqMwtVCEESUGAeJ/KzZVbdMkggGpfwcYncnHS1U/flLPJ9G qMmRLkf0BvnZB4pDA98UfrF0vzP4ETQEteWe6nwOx1JCqDFgfyMbZj9l21joRJ/v S3LsjA6zwMSOloEc6AZI0nGEC8gNdTsIFuhomavvKBtJP4MfS9aCIwROQf3c7//6 ZT1KNvAdzt8OjKY3tbZrIyfotmEnvstSYeTr6l6n3E1zAjRVHIvnEzFnv55Dvycy vFXfMtkRWkCpe3Yf78bLgXA4FWssHlGjqxka7UKiImu+m3lXaNv6mib1w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrvdeijedgtddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefkffggfgfvhffutgfgsehtjeertd dtfeejnecuhfhrohhmpegjuhhrihcuoeihuhhrihesrggvthgvrhhnrdhorhhgqeenucgg tffrrghtthgvrhhnpeekgfdtffeghfehgeelvdeuveekvdekjeetuefgveehiedtgfekhf evleehvdeljeenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpeihuhhrihesrggvthgvrhhnrd horhhg X-ME-Proxy: Feedback-ID: i0d79475b:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Mon, 3 Apr 2023 04:20:40 -0400 (EDT) Message-ID: Date: Mon, 3 Apr 2023 10:20:38 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 Content-Language: en-US To: hackers@freebsd.org From: Yuri Subject: "dirty" callback in vt drivers Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4PqkNq2Jkrz3FRr X-Spamd-Bar: / X-Spamd-Result: default: False [-0.40 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; R_DKIM_ALLOW(-0.20)[aetern.org:s=fm2,messagingengine.com:s=fm2]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; DKIM_TRACE(0.00)[aetern.org:+,messagingengine.com:+]; PREVIOUSLY_DELIVERED(0.00)[hackers@freebsd.org]; local_wl_from(0.00)[yuri@aetern.org] X-Rspamd-Pre-Result: action=no action; module=multimap; Matched map: local_wl_from X-ThisMailContainsUnwantedMimeParts: N Hi, I have implemented a simple hyper-v framebuffer driver (based on linux hyperv_fb one) acting as a glue between hypervisor and vt_fb: https://reviews.freebsd.org/D39395 All seems to be good, except that I am missing any sane way to know of the "dirty" updates that should be sent to hypervisor, there do not seem to be any callbacks in vt_fb. At the moment, as a PoC, there's a loop at the end of the attach callback that sends the "dirty the entire buffer" message to hypervisor once per 1/20 second, and that works and does not seem to put any real load on the host/guest, but still feels wrong. Is there a better way for this glue driver to be notified of the updates? From nobody Tue Apr 4 00:09:34 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pr7Rw5CTFz43XS1 for ; Tue, 4 Apr 2023 00:09:48 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pr7Rw0Dz6z3lph for ; Tue, 4 Apr 2023 00:09:47 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=iitbombay-org.20210112.gappssmtp.com header.s=20210112 header.b=PL0ajZZj; spf=pass (mx1.freebsd.org: domain of bakul@iitbombay.org designates 2607:f8b0:4864:20::1032 as permitted sender) smtp.mailfrom=bakul@iitbombay.org; dmarc=none Received: by mail-pj1-x1032.google.com with SMTP id qe8-20020a17090b4f8800b0023f07253a2cso32214206pjb.3 for ; Mon, 03 Apr 2023 17:09:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20210112.gappssmtp.com; s=20210112; t=1680566986; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:from:to:cc:subject:date:message-id:reply-to; bh=is/VO53tn0QJoQ7q60ZVPMFRchNzLZm4uIY2Z/cs/us=; b=PL0ajZZjNv7GMTAYaM2ZhnAajw8ca0cfOPNKdFHg6eOdmEvEWKP19DXrXQ6Qe/NFrr ibm2f2u1NV/kn0C8WyGcswJ8u3MRclBJ7jbQYWSHASwF6dLcTmWKOvP102+ytha26yt8 FwxpqBcOWzBxPreiSpW6/mLYuXZio3kzuIB3QRy2b9RCxDiklwCn/NnqiylfeELWhaZs YQ0JwR83iYszpVc6mukFcv/sm0+/0u4uHTary2evZqivNzTtftig+m4rpxjhfVci01I7 YqdZHL/QPo9EepxcAW2kv9HJdwhFHjPfGUAhZ3Sz0GKdPEz4IlxxVYBb2eqK3MD2z6JW zIfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680566986; h=to:date:message-id:subject:mime-version:content-transfer-encoding :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=is/VO53tn0QJoQ7q60ZVPMFRchNzLZm4uIY2Z/cs/us=; b=SAaDJ+O5JhbWmq7HoLGmDNdQFzKpLj6XD+KY4wKdnno4LnSijOz8w+4loVl/AGb0Js 7QBP2BhnPKmnXKJINVWqv6MZwuHNnC3rpiRtC/IJphidbdNDTaRY7cWD7qmccy+ArOtc EaadLq6FivfT12OFYqzJHQMwo67/NF633LLFseAqVespSJBn5++fXgG0a0XbFmA4PH8T lUP18tAWmQVqaW/sFrxguEsyvZ4G1NNOc36Zu06xzSAy8KUhOLZ3ed1MLJjS1hi6L5Ma qDHQoR8YmBtcQGUI8rGU2yDA25+CqSVxrhsPWVYXs047EU1gosc+TF3z5Kw06NyCGnLH TbSw== X-Gm-Message-State: AAQBX9eyq72cpOYE8cpU+SPiEe/voPlQOckbFh7vzF4vpbbiJcGlfJl1 mWHeDNUR0X4Nm0qy6xXqJ1ijK+S0DldRDKKv18IOrg== X-Google-Smtp-Source: AKy350b/ktL9I61832ja63bNBiDqBbTb3GKeZ2LHW+yYiOa0Ut/5nQMB62AS/1gCUHZ/L1Ug4SGfpg== X-Received: by 2002:a17:90a:398c:b0:234:1621:3792 with SMTP id z12-20020a17090a398c00b0023416213792mr846112pjb.4.1680566985906; Mon, 03 Apr 2023 17:09:45 -0700 (PDT) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id kd14-20020a17090b348e00b00233ebab3770sm6704232pjb.23.2023.04.03.17.09.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Apr 2023 17:09:45 -0700 (PDT) From: Bakul Shah Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: ccache Message-Id: <671864AD-11F0-487B-9597-ACF28D24591B@iitbombay.org> Date: Mon, 3 Apr 2023 17:09:34 -0700 To: freebsd-hackers X-Mailer: Apple Mail (2.3731.500.231) X-Spamd-Result: default: False [-2.48 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-0.98)[-0.978]; SUBJECT_ENDS_SPACES(0.50)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[iitbombay-org.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; FREEFALL_USER(0.00)[bakul]; TO_MATCH_ENVRCPT_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[iitbombay.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[iitbombay-org.20210112.gappssmtp.com:+]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1032:from]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Pr7Rw0Dz6z3lph X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Am I use ccache incorrectly or is there a bug? # CCACHE_DIR=/usr/obj/ccache ccache -s cache directory /usr/obj/ccache primary config /usr/obj/ccache/ccache.conf secondary config (readonly) /usr/local/etc/ccache.conf stats updated Mon Apr 3 16:54:31 2023 ... cleanups performed 507 files in cache 2795230 cache size 2.9 GB <=== max cache size 15.0 GB # find /usr/obj/ccache -type f |wc 2795395 2795395 142222206 # du -sh /usr/obj/ccache 83G /usr/obj/ccache <=== I have WITH_CCACHE_BUILD=yes CCACHE_DIR=/usr/obj/ccache in /etc/make.conf From nobody Tue Apr 4 01:48:30 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pr9dv25L4z43fFd for ; Tue, 4 Apr 2023 01:48:35 +0000 (UTC) (envelope-from leres@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pr9dv1WGVz3w4P; Tue, 4 Apr 2023 01:48:35 +0000 (UTC) (envelope-from leres@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1680572915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jy6El8lwX7wvpzAnQ/l9k1uzkeC4ATVagA5bFP6/5cI=; b=qhx5q/2MxvyPH9iKYdktfSDWQ1wxMMjNfJIIB7Ri1Os4fEN5zxua40kCEmWRabCfhyNV1X DZ3n+WRKPERj/GiE7MIdN+aXlwtKfzoEip8yRnvSjvNyh2/zBn8ZRs3OqnoAaUaRFC9Kiu fmMD2NSd49SE4kuPj1Md+LzsD99qBGQNatyquHXsd1mx+/J/NwOEEJkP3TiiaENic49FIh piVlYgNIfc+dOWCgXKlCeAyGInG8KyDJBU45lTl+u8lJ9emfNmyJzOL40NvWmR5tIj0AcV n4DbrvbDskF8RB+tXjAu5olbO/UQmCGHJRlPlOUfuKqlwXUs2j6qDbYr8BWUOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1680572915; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Jy6El8lwX7wvpzAnQ/l9k1uzkeC4ATVagA5bFP6/5cI=; b=UsqHIBQptZO0cbEgidfuXhGnc/qw1P5MmtF7vY4+bIBwLHEtL1o/nytQW0/pMzEjg5LX2r utvasdX8ScYte5eg9Lfpg3cGlnz2Wa2M5BzoHj8BoxJ0TyvLXWC/XNcw6+kWhEtMb1LvaZ snBOwLRqI8sK0pFeS52I5Zg/TAOt25GL0RqyqGiiAAZ1+CuLbMxJfTT7xPNW+hzOc3e6rl GM102SbKoLn3cFurODEd7os7CVuIJUFAYi5zA9gtWVrITx9G9RDgluX4oDgUxPIEBFBEYv u8rItd5NE2rRG8ppOKDSpbC+EGy8UGRU2H0WcNiFrNMmnS+7Okd3+xP5BfYjBA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1680572915; a=rsa-sha256; cv=none; b=baeciYyT/+LIch8xOqZwVy4TXCLka22kGClUQNEYH/R+dZqmL4Nw/bjtM+eFR2acm9rwXm +oQ0+tk1l+aqXCdEYIKINrRigaWzFNZPT1PIfft8AxgKeUu4617+uMFBNoavw4VtasOm+S LlIs3d8QZVGMFwQzccyIL+CTWh+FJD0LcxIEn5wiIgAWAgAx2VeNbE2WudJnu2Sf1mUYEx g0bTAhRj/BhYYxG7ztoP97WQNJ4vha/pYuTUIe99pXllnJ8gu7ROrxW+DSSUXc4bPyIE3B WbkHcC3NqYQ1JHVH2o08g1jwF3hx9hw/+chc8Ir52ttOKiyWU9jG+IcmWZoUSQ== Received: from [IPV6:2620:83:8000:102::cb] (hot.ee.lbl.gov [IPv6:2620:83:8000:102::cb]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: leres) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Pr9dt5tL4zhYF; Tue, 4 Apr 2023 01:48:34 +0000 (UTC) (envelope-from leres@freebsd.org) Message-ID: <92bc05d9-14fc-0928-4f36-4b55815303fe@freebsd.org> Date: Mon, 3 Apr 2023 18:48:30 -0700 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 Subject: Re: ccache Content-Language: en-US To: Bakul Shah , freebsd-hackers References: <671864AD-11F0-487B-9597-ACF28D24591B@iitbombay.org> From: Craig Leres In-Reply-To: <671864AD-11F0-487B-9597-ACF28D24591B@iitbombay.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ThisMailContainsUnwantedMimeParts: N On 4/3/23 17:09, Bakul Shah wrote: > Am I use ccache incorrectly or is there a bug? > > # CCACHE_DIR=/usr/obj/ccache ccache -s > cache directory /usr/obj/ccache > primary config /usr/obj/ccache/ccache.conf > secondary config (readonly) /usr/local/etc/ccache.conf > stats updated Mon Apr 3 16:54:31 2023 > ... > cleanups performed 507 > files in cache 2795230 > cache size 2.9 GB <=== > max cache size 15.0 GB > # find /usr/obj/ccache -type f |wc > 2795395 2795395 142222206 > # du -sh /usr/obj/ccache > 83G /usr/obj/ccache <=== > > I have > > WITH_CCACHE_BUILD=yes > CCACHE_DIR=/usr/obj/ccache > > in /etc/make.conf Thank you for bringing this up; I have the same issue and have never figured it out. But I think I've found another piece or two of the puzzle. Using my favorite ktrace trick we can see that by default it tries to open /usr/local/etc/ccache.conf: zinc 32 % cd /tmp && ktrace -di ccache -s > /dev/null zinc 33 % kdump | fgrep NAMI | fgrep ccache.conf 17418 ccache NAMI "/usr/local/etc/ccache.conf" 17418 ccache NAMI "/home/zinc/u0/leres/.ccache/ccache.conf" I suspect the trick here is when ccache runs inside a poudriere jail, ccache.conf is not present and/or not in the right location. On my build server I have /var/cache/ccache/ccache.conf which I believe works for some things but I still have 46 GB in /var/cache/ccache and /var/cache/ccache/ccache.conf is trying to limit use to 8 GB. If I start up a poudriere jail and look around I find /root/.ccache/ccache.conf is a copy of /var/cache/ccache/ccache.conf. So I don't get why it doesn't work. (Now I'll sit back and wait for something who knows more chimes in...) Craig From nobody Tue Apr 4 02:41:47 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PrBqW54vPz43jmD for ; Tue, 4 Apr 2023 02:41:59 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PrBqW3P4Bz44QM for ; Tue, 4 Apr 2023 02:41:59 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-pl1-x62b.google.com with SMTP id w4so29952453plg.9 for ; Mon, 03 Apr 2023 19:41:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20210112.gappssmtp.com; s=20210112; t=1680576118; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=FO3FSzhOvcfExnz1kXLjfRmQUiRBGs+nubjQ/V8PjXM=; b=jOgEenDHDfxEVttO8eMkGpFIqR0zvUvg4t5H0552/VsYSnV5j3s3ljbcsnnubvUF6R JQU4jC5pjQlEjnXgN3pNXCxZThEfyniC2bj+W204wDMWbgkKf5dXvaLrM0ABpqTFy54a be6bJUrXkxzIUTQj5MLqU3Xjbrz7sDRaAWHUDZ0QtyzUTzSHV9Kmhq9fF8ItAIsKd7M4 lSaek6ZThee9rFCm9F5m+HDsHBOEfVaPrvAb3jZjDOgiaHS4LEUgh+QIZPA40C+Fzm5m zbtOiU0jXBapc2F7y/tkeiLfCoj9rLP3Bp/ZyYKn/8M7CW4goxNSnNmCteRrsnJjlQG7 4nww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680576118; h=to:references:message-id:content-transfer-encoding:cc:date :in-reply-to:from:subject:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FO3FSzhOvcfExnz1kXLjfRmQUiRBGs+nubjQ/V8PjXM=; b=r8QwwZPUPTmnBOtrg5/wjoMeY9JGmmuwSIHSvDsGY8M63rVrnQ4qbxutfKyNRV7/ad ajzOG04s901wgUofGzdQqN3ioUfNo4vrYEmWPxMoLUNZRE+Nq1ob9frkceB2Y701J7SP EBz+tiKgQgjjyu3nErU3+qMekK8QpM/kBK3b8JD1hZ2pljC/ykOHFH+/yMG26cMjcupZ Wi/dksLnIK9bzcOJdoEha70Q90w6Ku2JwPKPpxdhm/Q5x9i7bb2nW5PWin0vA71wiMVf lAxVlg/C+Hn3KhgP+vzRlnmghXU+zhleG+FMK+cKcoRBAVjCfcH3jEkmh1GBi4CSbpGZ RzBg== X-Gm-Message-State: AAQBX9cstPleMv1V5qjTsYGrKVbGIbsaF/yuvtP8+TSVOFyITzh3Vn+e SyA/H9njohW7nk8djqKLlbAREQG9PY186Mx/VjpMhw== X-Google-Smtp-Source: AKy350aCPT08PbdC8vr14Wt8P7fwJlT2U9+MINwQL2nxiVtbZk0ebmUBC1rrsVsd4dToSGOH86NMaw== X-Received: by 2002:a05:6a20:6687:b0:db:6a5a:3ce7 with SMTP id o7-20020a056a20668700b000db6a5a3ce7mr824659pzh.12.1680576118255; Mon, 03 Apr 2023 19:41:58 -0700 (PDT) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id m3-20020aa79003000000b006260645f2a7sm7904036pfo.17.2023.04.03.19.41.57 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Apr 2023 19:41:58 -0700 (PDT) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.500.231\)) Subject: Re: ccache From: Bakul Shah In-Reply-To: <92bc05d9-14fc-0928-4f36-4b55815303fe@freebsd.org> Date: Mon, 3 Apr 2023 19:41:47 -0700 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <671864AD-11F0-487B-9597-ACF28D24591B@iitbombay.org> <92bc05d9-14fc-0928-4f36-4b55815303fe@freebsd.org> To: Craig Leres X-Mailer: Apple Mail (2.3731.500.231) X-Rspamd-Queue-Id: 4PrBqW3P4Bz44QM X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On Apr 3, 2023, at 6:48 PM, Craig Leres wrote: >=20 > On 4/3/23 17:09, Bakul Shah wrote: >> Am I use ccache incorrectly or is there a bug? >> # CCACHE_DIR=3D/usr/obj/ccache ccache -s >> cache directory /usr/obj/ccache >> primary config /usr/obj/ccache/ccache.conf >> secondary config (readonly) /usr/local/etc/ccache.conf >> stats updated Mon Apr 3 16:54:31 2023 >> ... >> cleanups performed 507 >> files in cache 2795230 >> cache size 2.9 GB <=3D=3D=3D >> max cache size 15.0 GB >> # find /usr/obj/ccache -type f |wc >> 2795395 2795395 142222206 >> # du -sh /usr/obj/ccache >> 83G /usr/obj/ccache <=3D=3D=3D >> I have >> WITH_CCACHE_BUILD=3Dyes >> CCACHE_DIR=3D/usr/obj/ccache >> in /etc/make.conf >=20 > Thank you for bringing this up; I have the same issue and have never = figured it out. But I think I've found another piece or two of the = puzzle. >=20 > Using my favorite ktrace trick we can see that by default it tries to = open /usr/local/etc/ccache.conf: >=20 > zinc 32 % cd /tmp && ktrace -di ccache -s > /dev/null > zinc 33 % kdump | fgrep NAMI | fgrep ccache.conf > 17418 ccache NAMI "/usr/local/etc/ccache.conf" > 17418 ccache NAMI "/home/zinc/u0/leres/.ccache/ccache.conf" >=20 > I suspect the trick here is when ccache runs inside a poudriere jail, = ccache.conf is not present and/or not in the right location. I checked that it uses the correct ccache.conf and ccache dir. >=20 > On my build server I have /var/cache/ccache/ccache.conf which I = believe works for some things but I still have 46 GB in = /var/cache/ccache and /var/cache/ccache/ccache.conf is trying to limit = use to 8 GB. >=20 > If I start up a poudriere jail and look around I find = /root/.ccache/ccache.conf is a copy of /var/cache/ccache/ccache.conf. So = I don't get why it doesn't work. >=20 > (Now I'll sit back and wait for something who knows more chimes in...) >=20 > Craig I did the same thing this time as before -- I manually ran "ccache -c". Even now the number it reports is smaller by a few G than what du -sh reveals, and it says there are many more files in the cache than find reveals. I suspect it either doesn't do proper bookkeeping or gets confused if you ^Ced a build at the wrong time. It does seem to update $CCACHE_DIR/?/stats files. ccache -c took a very long time. ktrace shows it does the equivalent of=20 for each file f to be deleted rename $f $f.ccache.rm.tmp unlink $f.ccache.rm.tmp Not sure why the rename is needed. From nobody Tue Apr 4 13:42:48 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PrTVF6S8Xz43bNV for ; Tue, 4 Apr 2023 13:43:01 +0000 (UTC) (envelope-from raghavself28@gmail.com) Received: from mail-il1-x135.google.com (mail-il1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PrTVD6DgFz3G41 for ; Tue, 4 Apr 2023 13:43:00 +0000 (UTC) (envelope-from raghavself28@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=fQeiobuo; spf=pass (mx1.freebsd.org: domain of raghavself28@gmail.com designates 2607:f8b0:4864:20::135 as permitted sender) smtp.mailfrom=raghavself28@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-il1-x135.google.com with SMTP id u8so17064086ilb.2 for ; Tue, 04 Apr 2023 06:43:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680615780; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=2tbjDVtHA6F6S29WQeCxrRbmuQmiYzcdgdxKCqqa+5k=; b=fQeiobuopCB6FWEDZVq1MZ+1lfPYabyIcC+nq0M3PjMt+zspSqJW3TAk9Qeys1yjFJ DT1nlqysGsB46RKnUpGoqMHF31woLrkT8r3xVfIQFaqI5ttC5AqNlopA3Maei1llBkv7 hT2Qcu8k7XuE1JGD4gUTNGt/7Uz4z6a0udpgH8k8xrz5aDKRMjaIhrf1XWq3hC5sX0TM WZhHkS5TMSNJ9pMUYOHnIe6/paZzkrEANpAkWXcaKgwLZC0iWzf506YDIJeV7Qtj/xON pRZNSpJBAj8J/ma4Nf0XAxdyoJgV+z89ogm7tLA070HjeYM5Z7j0G5j7WShv1aOv4pBc 8oVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680615780; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=2tbjDVtHA6F6S29WQeCxrRbmuQmiYzcdgdxKCqqa+5k=; b=WYVHdslx/fCyLiATUZEjn1B17v9c2XbfA33MthX8wNW70L7edSR+Tgl8dlSkAbugRH m/FIqyToWm2l3BWZ8DX6ds3F5qtPjnkkHoF/elBw0+PByLg6C6X0cgEtVmGrAckL/8A0 J/piT6iSgWytym1VixNseQ/XFg8/V0Ug4mmdHNDeSxII7BV0/sUJcRkJ5bsuWYbnPL4r RR+yY2cevGZUFz+1c/EYvgLT81LpIqdk7a/hMqqgnEYzqn9ATMspfDz1HJcvxMXEo2Br +Az9EIIwQBQgR+7qOvis8TbECpqTWM4Ji0UjCTw2LVgzahgbAhLdRB0rFeJ7KGLXK6U9 5oew== X-Gm-Message-State: AAQBX9fxB0W1D4na7zQcfWRncEVNUTsUW7O4dvZqUO2V4h1Vjln7/ME9 py5xnRdH85HD7VqfAtk5d4hMHV40glCT7zjvLs19gPhCP0/aOA== X-Google-Smtp-Source: AKy350a1i+/B4GtF3/rlPuP6oSWCOzTnVforptnU5s15w7uytoGGwSKotq5DH9s4a5+rcSd/OJAOtPy56TPKijcGhHI= X-Received: by 2002:a92:1811:0:b0:310:a298:1c95 with SMTP id 17-20020a921811000000b00310a2981c95mr1532756ily.6.1680615779726; Tue, 04 Apr 2023 06:42:59 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: Raghav Sharma Date: Tue, 4 Apr 2023 19:12:48 +0530 Message-ID: Subject: [GSoC] Search for Mentor for SquashFuse port project To: hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::135:from]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4PrTVD6DgFz3G41 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N Hello! I am Interested in working on SquashFuse port to FreeBSD kernel as a GSoC project. I had a small conversation with @jrm and he directed me here to search for a co-mentor willing to mentor me this summer. I already submitted a proposal for the project. So If anyone is interested in mentoring/co - mentoring me, we can then discuss my proposal. Thanks Raghav From nobody Tue Apr 4 14:46:34 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PrW4q5F6Yz43hN7 for ; Tue, 4 Apr 2023 14:54:35 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PrW4p07Bmz3Q1X for ; Tue, 4 Apr 2023 14:54:33 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of li-fbsd@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=li-fbsd@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 334Es76f003480 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Tue, 4 Apr 2023 16:54:07 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 334Es72r003479 for freebsd-hackers@freebsd.org; Tue, 4 Apr 2023 16:54:07 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from admn.intra.daemon.contact (localhost [127.0.0.1]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 334EkcoA096039 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Tue, 4 Apr 2023 16:46:39 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from intra.daemon.contact (news@localhost) by admn.intra.daemon.contact (8.17.1/8.17.1/Submit) with NNTP id 334EkYHH096016 for freebsd-hackers@freebsd.org; Tue, 4 Apr 2023 16:46:34 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-Authentication-Warning: admn.intra.daemon.contact: news set sender to li-fbsd@citylink.dinoex.sub.org using -f From: "Peter 'PMc' Much" X-Newsgroups: m2n.fbsd.hackers Subject: low TCP speed, wrong rtt measurement Date: Tue, 4 Apr 2023 14:46:34 -0000 (UTC) Message-ID: Injection-Date: Tue, 4 Apr 2023 14:46:34 -0000 (UTC) Injection-Info: admn.intra.daemon.contact; logging-data="90721"; mail-complaints-to="usenet@citylink.dinoex.sub.org" User-Agent: slrn/1.0.3 (FreeBSD) To: freebsd-hackers@freebsd.org X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Tue, 04 Apr 2023 16:54:10 +0200 (CEST) X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; FORGED_SENDER(0.30)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; R_DKIM_NA(0.00)[]; DMARC_NA(0.00)[sub.org]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_NEQ_ENVFROM(0.00)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org] X-Rspamd-Queue-Id: 4PrW4p07Bmz3Q1X X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org ** maybe this should rather go the -net list, but then ** there are only bug messages Hi, I'm trying to transfer backup data via WAN; the link bandwidth is only ~2 Mbit, but this can well run for days and just saturate the spare bandwidth. The problem is, it doesn't saturate the bandwidth. I found that the backup application opens the socket in this way: if ((fd = socket(ipaddr->GetFamily(), SOCK_STREAM, 0)) < 0) { Apparently that doesn't work well. So I patched the application to do it this way: - if ((fd = socket(ipaddr->GetFamily(), SOCK_STREAM, 0)) < 0) { + if ((fd = socket(ipaddr->GetFamily(), SOCK_STREAM, IPPROTO_TCP)) < 0) { The result, observed with tcpdump, was now noticeably different, but rather worse than better. I tried various cc algorithms, all behaved very bad with the exception of cc_vegas. Vegas, after tuning the alpha and beta, gave satisfying results with less than 1% tradeoff. But only for a time. After transferring for a couple of hours the throughput went bad again: # netstat -aC Proto Recv-Q Send-Q Local Address Foreign Address (state) CC cwin ssthresh MSS ECN tcp6 0 57351 edge-jo.26996 pole-n.22 ESTABLISHED vegas 22203 10392 1311 off tcp4 0 106305 edge-e.62275 pole-n.bacula-sd ESTABLISHED vegas 11943 5276 1331 off The first connection is freshly created. The second one runs for a day already , and it is obviousely hosed - it doesn't recover. # sysctl net.inet.tcp.cc.vegas net.inet.tcp.cc.vegas.beta: 14 net.inet.tcp.cc.vegas.alpha: 8 8 (alpha) x 1331 (mss) = 10648 The cwin is adjusted to precisely one tick above the alpha, and doesn't rise further. (Increasing the alpha further does solve the issue for this connection - but that is not how things are supposed to work.) Now I tried to look into the data that vegas would use for it's decisions, and found this: # dtrace -n 'fbt:kernel:vegas_ack_received:entry { printf("%s %u %d %d %d %d", execname,\ (*((struct tcpcb **)(arg0+24)))->snd_cwnd,\ ((struct ertt *)((*((struct tcpcb **)(arg0+24)))->osd->osd_slots[0]))->minrtt,\ ((struct ertt *)((*((struct tcpcb **)(arg0+24)))->osd->osd_slots[0]))->marked_snd_cwnd,\ ((struct ertt *)((*((struct tcpcb **)(arg0+24)))->osd->osd_slots[0]))->bytes_tx_in_marked_rtt,\ ((struct ertt *)((*((struct tcpcb **)(arg0+24)))->osd->osd_slots[0]))->markedpkt_rtt);\ }' CPU ID FUNCTION:NAME 6 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 131 17 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 17 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 3 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 131 5 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 17 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 131 11 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 106 15 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 13 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 16 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 106 3 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 One can see that the "minrtt" value for the freshly created connection is 56 (which is very plausible). But the old and hosed connection shows minrtt = 1, which explains the observed cwin. The minrtt gets calculated in sys/netinet/khelp/h_ertt.c: e_t->rtt = tcp_ts_getticks() - txsi->tx_ts + 1; There is a "+1", so this was apparently zero. But source and destination are at least 1000 km apart. So either we have had one of the rare occasions of hyperspace tunnelling, or something is going wrong in the ertt measurement code. For now this is a one-time observation, but it might also explain why the other cc algorithms behaved badly. These algorithms are widely in use and should work - the ertt measurement however is the same for all of them. From nobody Tue Apr 4 19:24:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Prd4Y3vK3z443k4 for ; Tue, 4 Apr 2023 19:24:45 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-ot1-x332.google.com (mail-ot1-x332.google.com [IPv6:2607:f8b0:4864:20::332]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Prd4X1Ylqz4G7C for ; Tue, 4 Apr 2023 19:24:44 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=D60mKLSX; spf=pass (mx1.freebsd.org: domain of mjguzik@gmail.com designates 2607:f8b0:4864:20::332 as permitted sender) smtp.mailfrom=mjguzik@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ot1-x332.google.com with SMTP id q23-20020a05683031b700b006a1370e214aso15040380ots.11 for ; Tue, 04 Apr 2023 12:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680636283; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=vZYmiIGjFU9D4OWMBFQlzWiu/kVg8cmtPHvKdME/lv8=; b=D60mKLSXpepKbhjPDZXs7izfJ7XuTay6s8kXPfaYXphK0n/S39gEZRoZUBB0e0wWOb bzTZt/LeIT1J/HTtqWkAGj8bFhi9OQl9Na6CbM+ZPcphPA0HxbmADXSgMvFXDRQaYAGr HA//6syL/kqdQ7x0d4gE3HXKDFfSBvt4o+JxPeYWBAt40HMx9G94H5GdbT5SSkBu0F7l KYuleVk1lj1wWKnUmwwhQpDmtvqrBKcXSEd+/0bx3gpspkIFQ3q33yWPF07MLhl6z/K1 /IzhdUTR42a03diBZJ4kvh5hQeTDTakVyDyfRAd9NG70pm1S26o6pGrDvcZjF5r321Ir X4xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680636283; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vZYmiIGjFU9D4OWMBFQlzWiu/kVg8cmtPHvKdME/lv8=; b=myHWfe4bphr928oeMi8N5YnMsLLhzbpIshXLfr2ybtdXxm/nAG/bqPcSgW0mv1Frc9 XTFk2qViR7ZIUYZQbLznHiiH2twzu653XenWitTiniLIeib27CZUVN3xmMYg807DVUdc t4I6O+7xS7BtfVPbho0hYeETOehuuQeJSNo6CckQJb4RyLTZTs6y3JSWyH9F8Nly5rlt NFjQFPGw8TzDlH6CZDid9m3OjjkOD3q/vqt2hN1PWYumlI44K+AAIZvyyMP3p/MTUFfP 3vnvp5XgNq+jmZkve/r0/YR981jX0Xo4MgzhWPrbhUNU3BdqoCdQi0KeCZROvUVnO5Kn rhJw== X-Gm-Message-State: AAQBX9cCvm2gaMsNL+9IHFWlMd5u3kdjORxAT0Hoe8CRaqlxEXHC9eJF pk5+zxsOf7taAI9pjmugcjgU6FtQympAMJkrRp8dVlGB X-Google-Smtp-Source: AKy350Ya3LtZEPHkXTim4kakEwyXHpL0hofpnPWRBmvLsF7BKkdFLJEjsk06+bhHpM+ptTRa3Ct0CWMIYqC+Rt2kCjY= X-Received: by 2002:a9d:7456:0:b0:6a3:8d70:b404 with SMTP id p22-20020a9d7456000000b006a38d70b404mr1187735otk.2.1680636283015; Tue, 04 Apr 2023 12:24:43 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:791:0:b0:49c:b071:b1e3 with HTTP; Tue, 4 Apr 2023 12:24:42 -0700 (PDT) In-Reply-To: References: From: Mateusz Guzik Date: Tue, 4 Apr 2023 21:24:42 +0200 Message-ID: Subject: Re: ULE process to resolution To: Jeff Roberson Cc: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Spamd-Result: default: False [-3.00 / 15.00]; URI_HIDDEN_PATH(1.00)[https://people.freebsd.org/~mjg/.junk/cpuburner1.c]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::332:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4Prd4X1Ylqz4G7C X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N Hello, On 3/31/23, Jeff Roberson wrote: > As I read these threads I can state with a high degree of confidence that > many of these tests worked with superior results with ULE at one time. > It may be that tradeoffs have changed or exposed weaknesses, it may also > be that it's simply been broken over time. I see a large number of > commits intended to address point issues and wonder whether we adequately > explored the consquences. Indeed I see solutions involving tunables > proposed here that will definitively break other cases. > One of the reporters claims the bug they complain about was there since early days. This made me curious how many problems reproduce on something like 7.1 (dated 2009), to that end I created an 8 core vm which I ran of bunch of tests on in addition to main. All 3 problems reported below reproduced there, no X testing though :) Bugs (one not reported in the other thread): 1. threads walking around the machine when spending little time off cpu, all while the machine is otherwise idle The problem with this on bare metal is that the victim cpu may be partially powered off, so now there is latency stemming from poking it back up, whatever other migration cost aside. I noticed this few years back when looking at postgres -- both the server and pgbench would walk around everywhere, reducing perf. I checked this reproduces on fresh main. The box at hand as 2 sockets * 10 cores * 2 threads. I *suspect* this is adequately modeled with a microbenchmark https://github.com/antonblanchard/will-it-scale/ named context_switch1_processes -- it too experiences all-machine walk unless explicitly bound (pass -n to *not* bind it). I verified they walk all around on 7.1 as well, but I don't know if postgres also would. how to bench: su - postgres /usr/local/bin/pg_ctl -D /var/db/postgres/data15 -l logfile start pgbench -i -s 10 pgbench -M prepared -S -T 800000 -c 1 -j 1 -P1 postgres ... and you are in. 2. unfairness when oversubscribing with cpu hogs Steve Kargl claims he reported this one numerous times since the early days of ULE, I confirmed it was a problem on 7.1 and is a problem today. Say an 8 core vm (with making sure these are cores pinned on the host) I'm going to copy paste my other message here: I wrote a cpu burning program (memset 1 MB in a loop, with enough iterations to take ~20 seconds on its own). I booted an 8 core bhyve vm, where I made sure to cpuset is to 8 distinct cores. The test runs *9* workers, here is a sample run: [copy] 4bsd: 23.18 real 20.81 user 0.00 sys 23.26 real 20.81 user 0.00 sys 23.30 real 20.81 user 0.00 sys 23.34 real 20.82 user 0.00 sys 23.41 real 20.81 user 0.00 sys 23.41 real 20.80 user 0.00 sys 23.42 real 20.80 user 0.00 sys 23.53 real 20.81 user 0.00 sys 23.60 real 20.80 user 0.00 sys 187.31s user 0.02s system 793% cpu 23.606 total ule: 20.67 real 20.04 user 0.00 sys 20.97 real 20.00 user 0.00 sys 21.45 real 20.29 user 0.00 sys 21.51 real 20.22 user 0.00 sys 22.77 real 20.04 user 0.00 sys 22.78 real 20.26 user 0.00 sys 23.42 real 20.04 user 0.00 sys 24.07 real 20.30 user 0.00 sys 24.46 real 20.16 user 0.00 sys 181.41s user 0.07s system 741% cpu 24.465 total [/paste] While ule spends fewer *cycles*, it spends more real time and it is *probably* bad. you can repro with: https://people.freebsd.org/~mjg/.junk/cpuburner1.c cc -O0 -o cpuburner1 cpuburner1.c and a magic script: #!/bin/sh ins=$1 shift while [ $ins -ne 0 ]; do time ./cpuburner1 $1 $2 & ins=$((ins-1)) done wait run like this, pick the second number to take 20-ish seconds on your cpu: sh burn.sh 1048576 500000 3. threads struggling to get back on cpu against nice -n 20 higs This acutely affects buildkernel. I once more played around, the bug was already there in 7.1, extending total time from ~4 minutes to 30. The problem is introduced with the machinery to attempt to provide fairness for pri <= PRI_MAX_BATCH. I verified that with straight up removing all of it. Then buildikernel managed to finish in sensible time, but the cpu hogs were overly negatively affected -- little cpu time and very unfairly distributed between them. Key point though that this *can* stick to close to base time. I had seen the patch from https://reviews.freebsd.org/D15985 , it does not fix the problem but it does alleviate it to some extent. It is weirdly hacky and seems to be targeting just the testcase you had instead of the more general problem. I applied it to a 2018-ish tree so that there are no woes from rebasing. stock: 290.95 real 2048.22 user 247.967 sys stock+hogs: 883.81 real 2111.34 user 189.42 sys patched+hogs: 460.84 real 2055.63 user 232.00 sys Interestingly stock kernel from that period is less affected by the general problem, but it is still pretty bad. With the patch things improve markedly, but there is still ~50% increase in real time which is way too much for being paired against -n 20. https://people.freebsd.org/~mjg/.junk/cpuburner2.c magic script: #!/bin/sh workers=$1 n=$2 size=$3 bkw=$4 echo workers $workers nice $n buildkernel $bkw shift while [ $workers -ne 0 ]; do time nice -n $n ./cpuburner $size & workers=$((workers-1)) done time make -C /usr/src -ssss -j $bkw buildkernel > /dev/null # XXX webdev-style pkill cpuburner wait sample use: time sh burn+bk.sh 8 20 1048576 8 I figured there would be a regression test suite available, with tests checking what happens for known cases with possibly contradictory requirements. Got nothing, instead I found people use hackbench (:S) or just a workload. All that said, I'm buggering off the subject. My interest in it was limited to the nice problem, since I have pretty good reasons to suspect this is what is causing pathological total real time instances for package builds. Have fun, -- Mateusz Guzik From nobody Tue Apr 4 22:34:12 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PrjKx3qDXz43HmX for ; Tue, 4 Apr 2023 22:36:37 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "uucp.dinoex.sub.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PrjKw22XRz4cvr for ; Tue, 4 Apr 2023 22:36:36 +0000 (UTC) (envelope-from li-fbsd@citylink.dinoex.sub.org) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of li-fbsd@citylink.dinoex.sub.org designates 2a0b:f840::12 as permitted sender) smtp.mailfrom=li-fbsd@citylink.dinoex.sub.org; dmarc=none Received: from uucp.dinoex.org (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]) by uucp.dinoex.org (8.17.1/8.17.1) with ESMTPS id 334Ma60L044511 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 5 Apr 2023 00:36:06 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-MDaemon-Deliver-To: Received: (from uucp@localhost) by uucp.dinoex.org (8.17.1/8.17.1/Submit) with UUCP id 334Ma6ai044510 for freebsd-hackers@freebsd.org; Wed, 5 Apr 2023 00:36:06 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from admn.intra.daemon.contact (localhost [127.0.0.1]) by admn.intra.daemon.contact (8.17.1/8.17.1) with ESMTPS id 334MYCV3056154 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Wed, 5 Apr 2023 00:34:13 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) Received: from intra.daemon.contact (news@localhost) by admn.intra.daemon.contact (8.17.1/8.17.1/Submit) with NNTP id 334MYCZA056145 for freebsd-hackers@freebsd.org; Wed, 5 Apr 2023 00:34:12 +0200 (CEST) (envelope-from li-fbsd@citylink.dinoex.sub.org) X-Authentication-Warning: admn.intra.daemon.contact: news set sender to li-fbsd@citylink.dinoex.sub.org using -f From: "Peter 'PMc' Much" X-Newsgroups: m2n.fbsd.hackers Subject: Re: ULE process to resolution Date: Tue, 4 Apr 2023 22:34:12 -0000 (UTC) Message-ID: References: Injection-Date: Tue, 4 Apr 2023 22:34:12 -0000 (UTC) Injection-Info: admn.intra.daemon.contact; logging-data="46431"; mail-complaints-to="usenet@citylink.dinoex.sub.org" User-Agent: slrn/1.0.3 (FreeBSD) To: freebsd-hackers@freebsd.org X-Milter: Spamilter (Reciever: uucp.dinoex.org; Sender-ip: 0:0:2a0b:f840::; Sender-helo: uucp.dinoex.org;) X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (uucp.dinoex.org [IPv6:2a0b:f840:0:0:0:0:0:12]); Wed, 05 Apr 2023 00:36:09 +0200 (CEST) X-Spamd-Result: default: False [-2.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; FORGED_SENDER(0.30)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; R_DKIM_NA(0.00)[]; DMARC_NA(0.00)[sub.org]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:205376, ipnet:2a0b:f840::/32, country:DE]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_XAW(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; FROM_NEQ_ENVFROM(0.00)[pmc@citylink.dinoex.sub.org,li-fbsd@citylink.dinoex.sub.org] X-Rspamd-Queue-Id: 4PrjKw22XRz4cvr X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org On 2023-04-04, Mateusz Guzik wrote: > All that said, I'm buggering off the subject. My interest in it was > limited to the nice problem, since I have pretty good reasons to > suspect this is what is causing pathological total real time instances > for package builds. > > Have fun, Wow. So, that was Groundhog Day, again. From nobody Sat Apr 8 22:46:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Pv9My3DStz44yBb for ; Sat, 8 Apr 2023 22:46:54 +0000 (UTC) (envelope-from rpp@ci.com.au) Received: from mippet.ci.com.au (mippet.ci.com.au [192.65.182.30]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Pv9Mv1R2Hz3jfw for ; Sat, 8 Apr 2023 22:46:50 +0000 (UTC) (envelope-from rpp@ci.com.au) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ci.com.au header.s=jun2016 header.b=mmb5WZeZ; spf=pass (mx1.freebsd.org: domain of rpp@ci.com.au designates 192.65.182.30 as permitted sender) smtp.mailfrom=rpp@ci.com.au; dmarc=pass (policy=none) header.from=ci.com.au Received: from mippet-dkim.ci.com.au (mippet-dkim.ci.com.au [192.168.1.244]) by mippet-dkim.ci.com.au (8.16.1/8.16.1/CE050417) with ESMTP id 338Mke88052330 for ; Sun, 9 Apr 2023 08:46:40 +1000 (AEST) (envelope-from rpp@ci.com.au) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ci.com.au; s=jun2016; t=1680994000; bh=StSXIYHa9C4ldr3RwyOQfe2SXSRsSyuHYskuj/EGAao=; h=Date:From:To:Subject:References:In-Reply-To; b=mmb5WZeZ2NAp4ZcQgd3+yZ6zDcYLDx/r5wqF2TjSAb1jXJtkQQmCBNFz5kZT9tqYu K0TOk0SepTkr5hsEysjK8Hwyzg6eq45pXhhnhxi8WjW1diBt48JHeXfTS20/6wVPI5 DyRTNFQ7S64oCbuF99rZYhbA1yGhq7nu6yuo8Fa8xo1dTyxhsmh4UkTX8fb88bEEBX Fsj2PZdSfs5eA5q+0HJaYZPJu6ENRleZFapuJtyULI9JszQy+SdyNskI/YqsTAM+rm w/6EjrCpMs49Q+akgW9JakZSWnNcL6Y9jMWukIaYE0pqiby4OcYQ9zcFFswVZ5gw6z Er+lc8Yv6j50A== Received: from jodi.ci.com.au (jodi.ci.com.au [192.168.1.21]) by mippet.ci.com.au (8.16.1/8.16.1/CE120917) with ESMTPS id 338MkeW0052327 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sun, 9 Apr 2023 08:46:40 +1000 (AEST) (envelope-from rpp@ci.com.au) Received: from jodi.ci.com.au (jodi.ci.com.au [192.168.1.21]) by jodi.ci.com.au (8.17.1/8.17.1) with SMTP id 338MkdYk008892 for ; Sun, 9 Apr 2023 08:46:39 +1000 (AEST) (envelope-from rpp@ci.com.au) Date: Sun, 9 Apr 2023 08:46:39 +1000 From: Richard Perini To: freebsd-hackers@freebsd.org Subject: Re: low TCP speed, wrong rtt measurement Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[ci.com.au,none]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[ci.com.au:s=jun2016]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[ci.com.au:+]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_THREE(0.00)[4]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_NONE(0.00)[]; ASN(0.00)[asn:9792, ipnet:192.65.182.0/24, country:AU] X-Rspamd-Queue-Id: 4Pv9Mv1R2Hz3jfw X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On Tue, Apr 04, 2023 at 02:46:34PM -0000, Peter 'PMc' Much wrote: > ** maybe this should rather go the -net list, but then > ** there are only bug messages > > Hi, > I'm trying to transfer backup data via WAN; the link bandwidth is > only ~2 Mbit, but this can well run for days and just saturate the spare > bandwidth. > > The problem is, it doesn't saturate the bandwidth. > > I found that the backup application opens the socket in this way: > if ((fd = socket(ipaddr->GetFamily(), SOCK_STREAM, 0)) < 0) { > > Apparently that doesn't work well. So I patched the application to do > it this way: > - if ((fd = socket(ipaddr->GetFamily(), SOCK_STREAM, 0)) < 0) { > + if ((fd = socket(ipaddr->GetFamily(), SOCK_STREAM, IPPROTO_TCP)) < 0) { > > The result, observed with tcpdump, was now noticeably different, but > rather worse than better. > > I tried various cc algorithms, all behaved very bad with the exception > of cc_vegas. Vegas, after tuning the alpha and beta, gave satisfying > results with less than 1% tradeoff. > > But only for a time. After transferring for a couple of hours the > throughput went bad again: > > # netstat -aC > Proto Recv-Q Send-Q Local Address Foreign Address (state) CC cwin ssthresh MSS ECN > tcp6 0 57351 edge-jo.26996 pole-n.22 ESTABLISHED vegas 22203 10392 1311 off > tcp4 0 106305 edge-e.62275 pole-n.bacula-sd ESTABLISHED vegas 11943 5276 1331 off > > The first connection is freshly created. The second one runs for a day > already , and it is obviousely hosed - it doesn't recover. > > # sysctl net.inet.tcp.cc.vegas > net.inet.tcp.cc.vegas.beta: 14 > net.inet.tcp.cc.vegas.alpha: 8 > > 8 (alpha) x 1331 (mss) = 10648 > > The cwin is adjusted to precisely one tick above the alpha, and > doesn't rise further. (Increasing the alpha further does solve the > issue for this connection - but that is not how things are supposed to > work.) > > Now I tried to look into the data that vegas would use for it's > decisions, and found this: > > # dtrace -n 'fbt:kernel:vegas_ack_received:entry { printf("%s %u %d %d %d %d", execname,\ > (*((struct tcpcb **)(arg0+24)))->snd_cwnd,\ > ((struct ertt *)((*((struct tcpcb **)(arg0+24)))->osd->osd_slots[0]))->minrtt,\ > ((struct ertt *)((*((struct tcpcb **)(arg0+24)))->osd->osd_slots[0]))->marked_snd_cwnd,\ > ((struct ertt *)((*((struct tcpcb **)(arg0+24)))->osd->osd_slots[0]))->bytes_tx_in_marked_rtt,\ > ((struct ertt *)((*((struct tcpcb **)(arg0+24)))->osd->osd_slots[0]))->markedpkt_rtt);\ > }' > CPU ID FUNCTION:NAME > 6 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 131 > 17 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 > 17 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 > 3 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 131 > 5 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 > 17 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 131 > 11 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 106 > 15 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 > 13 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 > 16 17478 vegas_ack_received:entry ng_queue 11943 1 11943 10552 106 > 3 17478 vegas_ack_received:entry ng_queue 22203 56 22203 20784 261 > > One can see that the "minrtt" value for the freshly created connection > is 56 (which is very plausible). > But the old and hosed connection shows minrtt = 1, which explains the > observed cwin. > > The minrtt gets calculated in sys/netinet/khelp/h_ertt.c: > e_t->rtt = tcp_ts_getticks() - txsi->tx_ts + 1; > There is a "+1", so this was apparently zero. > > But source and destination are at least 1000 km apart. So either we > have had one of the rare occasions of hyperspace tunnelling, or > something is going wrong in the ertt measurement code. > > For now this is a one-time observation, but it might also explain why > the other cc algorithms behaved badly. These algorithms are widely in > use and should work - the ertt measurement however is the same for all of > them. I can confirm I am seeing similar problems transferring files to our various production sites around Australia. Various types/sizes of links and bandwidths. I can saturate the nearby links, but the link utilisation/saturation decreases with distance. I've tried various transfer protocols: ftp, scp, rcp, http: results are similar for all. Ping times for the closest WAN link is 2.3ms, furthest is 60ms. On the furthest link, we get around 15% utilisation. Transfer between 2 Windows hosts on the furthest link yields ~80% utilisation. FreeBSD versions involved are 12.1 and 12.2. -- Richard Perini Ramico Australia Pty Ltd Sydney, Australia rpp@ci.com.au +61 2 9552 5500 ----------------------------------------------------------------------------- "The difference between theory and practice is that in theory there is no difference, but in practice there is"