From owner-freebsd-hackers@freebsd.org Thu Jun 11 08:30:29 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C1B6034F18A for ; Thu, 11 Jun 2020 08:30:29 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49jH9c5nSXz4F6C for ; Thu, 11 Jun 2020 08:30:28 +0000 (UTC) (envelope-from shrikanth07@gmail.com) Received: by mail-wm1-x344.google.com with SMTP id u13so4160436wml.1 for ; Thu, 11 Jun 2020 01:30:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=dQ+kYZJ5KorOOFBosBbFXoFQttZJl6rhwqJPWKd/fyU=; b=pDmTtZXbf99ggmsM9mq/9psAiAyL6kEzycTl7wuGYom0UxkeOkE8ayynMO8xvptzFX E6WmNsauCyzHZiGKHNGYOSytwogEQeKeleUoLr8riWr5zkPNXW4XPq9sZISoJKxSK1vk ZrB4xX2ppO2Ne+VWhISGEYylpGy1pBfvMPOELlkK+WEMOggMHXYEg/K3pVOWVjPmPSML bu2hivDcF97QfDHhfyomaBD/tvmZsGEX2YDlPZIt+lA8eo10SXsy7NwYAWj+Ov/Cbxz8 awVxfUTvSj24AONTRsnw7E4UFYLPh1cHZ/Ssx4nBSKvGHFaEqifxi31+bkSSJ2G7bPx7 Nr6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=dQ+kYZJ5KorOOFBosBbFXoFQttZJl6rhwqJPWKd/fyU=; b=D6RhXT16A8qoz0S8VNDlhRVO5sxgHva/Rq3c/+ILIrlONC7l1im5WnYEiWvu0HYaGA dhZcDfck/KbwsiW5Pw6ZmJzBI86pSBuqG2z9H4SKqxMHnCqaqmpT6jyP/B1paXLQnsjj oshetYZ87Xfqd928MwT3s5uWSpYd/JKFm3x7wdXgLunwTTBIBberaDmCiY3V/6RYyrC/ jgZpTO5NpqDbflxBwbaoZllUcSOisml47G8sLC6voNNR+gsLtGU8qJTl92LWINW3TNhe X8acVJvFzMfY9UcCvKJPxHSnGXBOqVx1VGG1pJfAITC654lQT6/3YTNzIJGwtjXLOxr7 Qkog== X-Gm-Message-State: AOAM532ZDvutwhAxm9Y1yGMrWxhqMTAKPVImIp5h409LRzVs3CfcJcuc Wt+l5zR7RrdK6Rd7IPAU03Bx7gLegHG9yPL3mCOuYuZ4 X-Google-Smtp-Source: ABdhPJwybQ23H1r/FQmWSkhQmwRqyd3CZBufqtqEfMvQleOSuT3Uy49HBJuU5p7bh+/CpoT6qjndxg4cCjrx95f414M= X-Received: by 2002:a1c:b443:: with SMTP id d64mr6670230wmf.157.1591864226101; Thu, 11 Jun 2020 01:30:26 -0700 (PDT) MIME-Version: 1.0 From: Shrikanth Kamath Date: Thu, 11 Jun 2020 01:30:14 -0700 Message-ID: Subject: Xentimer/Lapic differences when used as a eventtimer source To: freebsd-hackers@freebsd.org X-Rspamd-Queue-Id: 49jH9c5nSXz4F6C X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=pDmTtZXb; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shrikanth07@gmail.com designates 2a00:1450:4864:20::344 as permitted sender) smtp.mailfrom=shrikanth07@gmail.com X-Spamd-Result: default: False [-2.68 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.028]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::344:from]; NEURAL_SPAM_SHORT(0.35)[0.346]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jun 2020 08:30:29 -0000 Trying to understand the implementation differences for xentimer v/s lapic when used as eventtimer sources. I have 2 setups where there is FreeBSD 11.0 based Junos running in a virtual environment one on a xen based hypervisor and another kvm based. The xen version has xentimer as the eventtimer source v/s lapic for the kvm version. For an application that is doing nanosleep with same resolution of 100 ns and running on these two systems the rate of timer interrupts is order of magnitude different and hogs the cpu in case of lapic but for xentimer it is not as significant. Below I have listed some readings from dtrace probes I ran to understand, For xen setup using xentimer as eventtimer # sysctl kern.eventtimer kern.eventtimer.periodic: 0 kern.eventtimer.timer: XENTIMER kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 2 kern.eventtimer.choice: XENTIMER(950) LAPIC(100) i8254(100) RTC(0) kern.eventtimer.et.XENTIMER.quality: 950 kern.eventtimer.et.XENTIMER.frequency: 1000000000 kern.eventtimer.et.XENTIMER.flags: 6 For kvm setup using lapic as eventtimer # sysctl kern.eventtimer kern.eventtimer.periodic: 0 kern.eventtimer.timer: LAPIC kern.eventtimer.idletick: 0 kern.eventtimer.singlemul: 2 kern.eventtimer.choice: LAPIC(600) RTC(0) kern.eventtimer.et.LAPIC.quality: 600 kern.eventtimer.et.LAPIC.frequency: 3000050812 kern.eventtimer.et.LAPIC.flags: 7 Running fbt::et_start:entry with conditional for execname to be the application and using an aggregation to count() calls, shows ~6K calls per second for xentimer v/s ~95K calls for lapic Trying to count number of timer interrupts on each with dtrace -n 'fbt:kernel-xen:xentimer_intr:entry /cpu != 0/ { @cnt[cpu] = count();} tick-1s { printa(@cnt); clear(@cnt);}' gives --> 6083 interrupts per second v/s dtrace -n 'fbt:kernel:lapic_handle_timer:entry /cpu != 0/ { @cnt[cpu] = count();} tick-1s { printa(@cnt); clear(@cnt);}' gives --> 99063 interrupts per second For xentimer #vmstat -i | grep :xen interrupt total rate cpu1:xen 3642296440 6084 v/s lapic # vmstat -i interrupt total rate cpu1:timer 61458150612 102715 Also does lapic source have additional overhead of VM entry/VM exit? -- Shrikanth R K