From owner-freebsd-virtualization@freebsd.org Tue Jul 23 11:45:46 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 52F05A731E for ; Tue, 23 Jul 2019 11:45:46 +0000 (UTC) (envelope-from dennis.noordsij@alumni.helsinki.fi) Received: from mail.iletsel.nl (mail.iletsel.nl [94.103.156.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B6FF18CF82 for ; Tue, 23 Jul 2019 11:45:41 +0000 (UTC) (envelope-from dennis.noordsij@alumni.helsinki.fi) To: freebsd-virtualization@freebsd.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=iletsel.nl; s=ILETSEL; t=1563882332; bh=3y1yF6j2bjLGjvnlSdNU0kA/ORtfWArLS09IkZogsos=; h=To:From:Subject:Date; b=xDpgxF0qX9JL0YMelJQvu2AKIOhUWLnow3Y5WuX032/RarvcPVJi3nAmF08p0MAga yGIq6a+77TppXs15WMbq7tMyI+bOS9CDgQSOS3dxe3FSCdK/imqodCSGnXuOJTOIc0 Qc7Zp7ql6bQpDAdXhm2FeUXtalahYLADmSmMcI+o= From: Dennis Noordsij Subject: Timecounter problem as a guest Message-ID: <641a92ec-81ad-2e26-a3b8-3a7238736757@iletsel.nl> Date: Tue, 23 Jul 2019 13:45:40 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.7.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: B6FF18CF82 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=iletsel.nl header.s=ILETSEL header.b=xDpgxF0q X-Spamd-Result: default: False [-1.61 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.60)[-0.598,0]; R_DKIM_ALLOW(-0.20)[iletsel.nl:s=ILETSEL]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.99)[-0.990,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[helsinki.fi]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_SHORT(0.76)[0.761,0]; DKIM_TRACE(0.00)[iletsel.nl:+]; RCVD_IN_DNSWL_MED(-0.20)[67.156.103.94.list.dnswl.org : 127.0.4.2]; MX_GOOD(-0.01)[mail.it.helsinki.fi,send2.it.helsinki.fi,post2.it.helsinki.fi]; R_SPF_NA(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(0.22)[asn: 25459(1.10), country: NL(0.01)]; ASN(0.00)[asn:25459, ipnet:94.103.144.0/20, country:NL]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jul 2019 11:45:46 -0000 Hi, Recently I moved some FreeBSD systems to VPS provider TransIP, who use Linux-KVM based virtualization. Initial performance was surprisingly bad, and the CPU graphs very were spikey with as much system time being spent as user time. Via PostgreSQL I ended up trying out pg_test_timing which reported the following for the default timecounter (HPET): (note choices are: kern.timecounter.choice: i8254(0) ACPI-fast(900) HPET(950) TSC-low(-100) dummy(-1000000)) Testing timing overhead for 3 seconds. Per loop time including overhead: 6481.08 ns Histogram of timing durations:   < us   % of total      count      1      0.00000          0      2      0.00000          0      4      0.00000          0      8     88.79165     411005     16      9.53451      44134     32      1.03848       4807     64      0.49796       2305    128      0.10370        480    256      0.02981        138    512      0.00259         12   1024      0.00086          4   2048      0.00022          1   4096      0.00000          0   8192      0.00000          0  16384      0.00022          1 With the other timecounter choices of i8245 and ACPI-fast the result look like the above, no results under 4us. Only with TSC-low does it look like: Testing timing overhead for 3 seconds. Per loop time including overhead: 41.22 ns Histogram of timing durations:   < us   % of total      count      1     95.97088   69846421      2      4.02214    2927264      4      0.00136        988      8      0.00288       2096     16      0.00132        958     32      0.00074        542     64      0.00047        345    128      0.00016        114    256      0.00004         29    512      0.00000          3   1024      0.00000          2   2048      0.00000          3 and indeed the CPU graphs cleaned up completely with much lower CPU averages and no excessive system CPU time after switching to TSC-low. Webserver and database response times dropped as well (at least according to their own reporting). To rule out this being just a symptom of timekeeping: the providers own CPU graphs (so from the outside of the VPS as a whole) also show this VPS to consume roughly half the CPU it does with TSC-low compared to the other options, and you can tell the difference right away when changing the kern.timecounter.hardware sysctl. The main problem however is that the system clock now keeps time atrociously badly. Chrony with the most aggressive settings barely manages to keep the time and the CPU graphs now show regular gaps where the system time jumped because of a correction. It looks very sloppy to the users if the recorded times of their actions/files are not correct. This is all on a 6 core system with lots of threads and churn and short lived apps coming and going. A 4-core database system, with a stable number of threads and processes, running in the same virtualization environment, doesn't really have either of these problems, that is, CPU usage wasn't that spikey or system CPU usage that high even with HPET, and the time doesn't drift as much either with TSC-low. I figured this is a virtualization question as these kinds of symptoms are probably generic. What is the host doing? Additional information from within the guest: hw.machine: amd64 hw.model: Westmere E56xx/L56xx/X56xx (Nehalem-C) hw.ncpu: 6 hw.hv_vendor: KVMKVMKVM hw.clockrate: 2593 (has 24GB memory) (They do perform live migrations so I don't know what the real underlying hardware is but probably similar, it's pretty stale at this point) I wonder if anyone could talk a bit about what might be going on. Thank you, Dennis