From nobody Sat Jan 1 18:44:04 2022 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3F2A7191BE7B; Sat, 1 Jan 2022 18:44:24 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JR9sM05Twz4X1r; Sat, 1 Jan 2022 18:44:23 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: by mail-pf1-x433.google.com with SMTP id u20so26017641pfi.12; Sat, 01 Jan 2022 10:44:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=jr2PDLGqCk4ULqMmc0+XMQFSOsaE48vQGp3vxVsgSzs=; b=mykzB9epAA6hOcCbOZ86gsZ1r/EzN81uYugsyJNTnomT4cveW/9MdtYKqXOn2DqBOu wKpoMTr5fxITww0YWKXvedPdMQtFpvSnc+V6sWEU1bd5T8+zUh4vp36H3Bj8+NINkKUN hv1BxUjz6w6mWVt6Iw6wUkpGaiQ6HLL+RDBNfYywfpvc7JRWBQ9HWs72aUEXfkRfH7qv Ec8l2TROXK01ow39ZJRufRCNrWrSzyO8A6BtoQnM1FcwB6yvEKuAizXOWAmudVqTuqS5 UJTm5dJ/uBvBY0X7Eu0JFRYzg56BB5op7nkQhpLpoRR+M0+ygoMpUe/PGWXCFGXIJNbJ /AQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=jr2PDLGqCk4ULqMmc0+XMQFSOsaE48vQGp3vxVsgSzs=; b=ZIfgnpuVO/fTJraaFio+zaG5f0ltcfnTTtJ75ndtjJGhHb6HAXreryWoWY5G4hYp7P CzBPUOQG/8lqI1EliqL2VeVbmld/mfYETGu6k6Mlu8bBHC28a7IDzRSjxDPcnv60qsE0 BUy+Smv1DIyMPMZqxqPSTNyasLm6SGsSb3Tn9qjWp+C8I6tF3pyxVNH+pSTQxiE9HzOL Avlen6QXvxp0qEbaM7obYcKSJ2eaQ4goVEvRC5x6oxcWQcx3Hljq0e4VDr8mvxMR3DYg wK09HV++ZT/QI6I/6YVli2BP+O9zs4ZpeJOyAoBoLi7n9EEx8QId79kH3CF3lH6/25MQ VeJg== X-Gm-Message-State: AOAM531M7HLdhghCe5Q4EZWQxQCkMARTdL+W44efTErbGqssFEd+bId/ XeHrCD7z/rxmuphm2eqFA7Qu0S3ySychOGvL8i/I4hZZ X-Google-Smtp-Source: ABdhPJz28XRsjRMdRBSgzCXnBu6j2tTQnAPrZ7sjcjyJvoBfuPvTUYFwTJ+fNfNW08gJ89mHmudJoG9wwkjvt/q0wbc= X-Received: by 2002:aa7:88ce:0:b0:4ba:efec:39e0 with SMTP id k14-20020aa788ce000000b004baefec39e0mr40355891pff.80.1641062656263; Sat, 01 Jan 2022 10:44:16 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <02d9ca2a-2fd7-942b-f411-be007ef88327@FreeBSD.org> In-Reply-To: <02d9ca2a-2fd7-942b-f411-be007ef88327@FreeBSD.org> From: Ryan Stone Date: Sat, 1 Jan 2022 13:44:04 -0500 Message-ID: Subject: Re: schedgraph.d experience, per-CPU buffers, pipes To: Andriy Gapon Cc: FreeBSD Current , FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4JR9sM05Twz4X1r X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=mykzB9ep; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rysto32@gmail.com designates 2607:f8b0:4864:20::433 as permitted sender) smtp.mailfrom=rysto32@gmail.com X-Spamd-Result: default: False [-2.08 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_SPAM_MEDIUM(0.92)[0.924]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::433:from]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N I've definitely experienced the issue about different buffers rolling over faster than others and producing confusing schedgraph data. I'm away from home this week, but I believe that I have a script that tries to chop off the schedgraph data at the point where the most recent CPU to roll over has no more data. If I think of it I'll try to pass it along. Another issue that I remember encountering is that there is a limitation on the total amount of space that you can allocate to dtrace buffers and it does not scale to the number of CPUs in the system, so the more CPUs that you have, the less buffer you can allocate per CPU. I seem to recall that the limit being very small compared to the amount of memory on a modern large core count system. On Fri, Dec 24, 2021 at 8:08 AM Andriy Gapon wrote: > > > I would like to share some experience or maybe rather a warning about using > DTrace for tracing scheduling events. Unlike KTR which has a global circular > buffer, DTrace with bufpolicy=ring uses per-CPU circular buffers. So, if there > is an asymmetry in processor load, the buffers will fill and wrap-around at > different speeds. In the end, they might have approximately equal numbers of > events but those may cover very different time intervals. So, some additional > post-processing is required to find the latest event among first ones of each > per-CPU buffer. Any traces from before that would have information gaps > ("missing" processors) and would be very confusing. > > Also, I noticed that processes passing a lot of data through pipes produce a lot > of scheduling events as they seem to get blocked and unlocked every few > microseconds (on a modern performant system with the default pipe sizing > configuration). That contributes to a quick wrap-around of circular buffers. > > -- > Andriy Gapon >