From nobody Tue Aug 17 01:19:35 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5FC17174E035 for ; Tue, 17 Aug 2021 01:19:44 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GpY9D1yjVz4YwQ; Tue, 17 Aug 2021 01:19:44 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: by mail-lj1-x22b.google.com with SMTP id f2so8484953ljn.1; Mon, 16 Aug 2021 18:19:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=6BulJIAFN2yoT2qc3P1XkMyuZiDq6zHdDPP7d7x0BXM=; b=MJy9XX6/O5736N/NNN/vjczqvDuKSSyCxqr0Ej0crwPZuDAZAum546d1ZMflhVjf1S lVnXa7CuFHfB88/1DFDqQVK5V/cirhf24KMrOKlPnkWB7XDsGy0bAq1wYdzU9l4I8NVK zjMLSNCLixiHuC/0CAv6/KBKtNufTkIb6x1b/QYx+Jcn8CkLMFcYY3jpe9vLzCe5PpG1 kDMIz9iyG3k0sRl892Imt2W/GcU/SBrp2S5j/wY8WSKa/edZ7bORh/F9AC4P0BN5IydP b7Qt+sS7RMD0Cc43UVNzvWnTvkH92IdKsmVTyV30bTnlb3MVY8LXtUCJP4hBJkfTDvaK oesQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=6BulJIAFN2yoT2qc3P1XkMyuZiDq6zHdDPP7d7x0BXM=; b=mPQrFvpfXsFGshqNfJ7cBYJy+6xH4BxnywDQPtiF4fYw6B93t2Z5mXjIIw1yEPBkVd 4h/q/LfNTFVi60g/zKclR5kDgcGUYpLLDDOyz9cGNN+COVoEvcNRc1voBksjn+aEr222 wmg3Dwd21MKruVXW+pJJOThxphZD+4wn2+enUZM802RSSqi+pgp0OJ1REI04N0wM5cVY rEjp+Zg824/wA5kxjRQBqh8AChm/WocQdETJuo5k1dluQQ5/HNYwQ+tI/k562DXsepJ1 FPFgF6aClbFIAxi4KOKW6URFmNbz9uj95/cUuhMxNmb6slzBENdMsso8rwCV5Bpx2DVW SvaA== X-Gm-Message-State: AOAM533rmvfs6AeavfTc5co+NBexc7BppnFcNA4K2yjfHwmEQec7ixpY 31YyBFtV0XoPM/MMZNkz01hHe2fImwvub1F8BgoYiuoo X-Google-Smtp-Source: ABdhPJziD5IxXeocbz8X4NdQwWyzU1gB07v2eaQAJmV2twbRrPoQO4bflJN0WZibHSsq7B438drSP9XARTTePVf7jGs= X-Received: by 2002:a2e:bd85:: with SMTP id o5mr811474ljq.313.1629163176649; Mon, 16 Aug 2021 18:19:36 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a2e:6f04:0:0:0:0:0 with HTTP; Mon, 16 Aug 2021 18:19:35 -0700 (PDT) In-Reply-To: References: From: Mateusz Guzik Date: Tue, 17 Aug 2021 03:19:35 +0200 Message-ID: Subject: Re: sysctl is too slow To: Alan Somers Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4GpY9D1yjVz4YwQ X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N On 8/16/21, Alan Somers wrote: > Yes, I see what you're talking about now. There are a bunch of linked > lists in sysctl_find_oid etc. Good point. > -Alan > You still want to get a flamegraph, chances are most of the problem is in zfs. > On Mon, Aug 16, 2021 at 1:30 PM Mateusz Guzik wrote: > >> Last time I checked lookup of a sysctl was very bad with linear scans all >> over. >> >> Short of complete revamp of the entire thing I would start with >> replacing the scans with a RB tree at each level. As is if you indeed >> have 5000 datasets, you are doing increasingly longer walks. >> >> On 8/16/21, Alan Somers wrote: >> > ztop feels very sluggish on a server with 5000 ZFS datasets. Dtrace >> shows >> > that almost all of its time is spent in sys_sysctl. ktrace shows that >> both >> > ztop and sysctl(8) call sys_sysctl a total of five times for each >> > sysctl >> > they care about: >> > >> > 1) To get the next oid >> > 2) To get the sysctl's name >> > 3) To get the oidfmt >> > 4) To get the size of the value >> > 5) To get the value itself. >> > >> > Each of these steps takes about equal time, and together all five take >> > about 100us. If the time per call is mostly syscall overhead, then the >> > process could be sped up by 80% by combining all of these things into a >> > single syscall: return the next oid, its name, its format, the size of >> its >> > value, and optimistically the value itself, assuming the user passed a >> > sufficiently large buffer. >> > >> > Am I missing something? Is there any other reason why sysctl is so >> > slow? >> > Or should I forget about it, and try to export ZFS's dataset stats >> through >> > devstat instead? >> > -Alan >> > >> >> >> -- >> Mateusz Guzik >> > -- Mateusz Guzik