From nobody Mon Aug 16 21:13:11 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9E53817654A4 for ; Mon, 16 Aug 2021 21:13:29 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oo1-f48.google.com (mail-oo1-f48.google.com [209.85.161.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GpRj53x4Fz3h8x for ; Mon, 16 Aug 2021 21:13:29 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-oo1-f48.google.com with SMTP id m11-20020a056820034b00b0028bb60b551fso1176262ooe.5 for ; Mon, 16 Aug 2021 14:13:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Jqo4SKACkIl/X3UniObK5Nd1ZXiN055iAWP4MonNqP0=; b=BBF/5OoC19Qwz/tt7CeM9CWKFK9OX2raSt9nImRB62aICtDbNDXzO024XXnefAyKKL 5zc42Gqr3IJZxEFqhaqcN7tl+fkmNBlwAnk3rj4CZClr1Y0h3Tpn8tA0YW06XAUzJ9To uaGLqHq1MWEjbICm3gid11skxW6tUwfd3JgvZ2KC8TpTKEfR6c7kmXWpLoeErTkS5ydD Jzw12nDP4cjuQSgzeHRst8OSQ++cZIzmsC1N+UUhkT8LfjOoroKg5oiQ7lB7crFdSFoc qsRMXs4lS8UPKJIYaP8bWy352KtnawwTacGcFgaL03YNjkDYVevTh5lyg+6QIpOp+bOO wNYg== X-Gm-Message-State: AOAM533+GcAmJTWmL42QqRYXUf202qcv3oXAm93EX9bx8Ij8R88gbLfN V/wEiaWQv1xguk4npnBvVjwJptCvCzTG4y8OYuo= X-Google-Smtp-Source: ABdhPJwwB3BtRvswaTP0tBp93lGfyt6kPrNhzLM16kZj4z457vEK8102fP0nXj3j3Dc9dyH8rxkD70jCr5q6lqtKe30= X-Received: by 2002:a4a:b481:: with SMTP id b1mr193595ooo.79.1629148402810; Mon, 16 Aug 2021 14:13:22 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Mon, 16 Aug 2021 15:13:11 -0600 Message-ID: Subject: Re: sysctl is too slow To: Mateusz Guzik Cc: FreeBSD Hackers Content-Type: multipart/alternative; boundary="000000000000d6013605c9b3ab76" X-Rspamd-Queue-Id: 4GpRj53x4Fz3h8x X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: Y --000000000000d6013605c9b3ab76 Content-Type: text/plain; charset="UTF-8" Yes, I see what you're talking about now. There are a bunch of linked lists in sysctl_find_oid etc. Good point. -Alan On Mon, Aug 16, 2021 at 1:30 PM Mateusz Guzik wrote: > Last time I checked lookup of a sysctl was very bad with linear scans all > over. > > Short of complete revamp of the entire thing I would start with > replacing the scans with a RB tree at each level. As is if you indeed > have 5000 datasets, you are doing increasingly longer walks. > > On 8/16/21, Alan Somers wrote: > > ztop feels very sluggish on a server with 5000 ZFS datasets. Dtrace > shows > > that almost all of its time is spent in sys_sysctl. ktrace shows that > both > > ztop and sysctl(8) call sys_sysctl a total of five times for each sysctl > > they care about: > > > > 1) To get the next oid > > 2) To get the sysctl's name > > 3) To get the oidfmt > > 4) To get the size of the value > > 5) To get the value itself. > > > > Each of these steps takes about equal time, and together all five take > > about 100us. If the time per call is mostly syscall overhead, then the > > process could be sped up by 80% by combining all of these things into a > > single syscall: return the next oid, its name, its format, the size of > its > > value, and optimistically the value itself, assuming the user passed a > > sufficiently large buffer. > > > > Am I missing something? Is there any other reason why sysctl is so slow? > > Or should I forget about it, and try to export ZFS's dataset stats > through > > devstat instead? > > -Alan > > > > > -- > Mateusz Guzik > --000000000000d6013605c9b3ab76--