From owner-freebsd-current@freebsd.org Tue Dec 10 16:29:44 2019 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9E2951DDD09 for ; Tue, 10 Dec 2019 16:29:44 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47XQWW5xWXz3Ktf for ; Tue, 10 Dec 2019 16:29:43 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x72b.google.com with SMTP id l124so4966904qkf.8 for ; Tue, 10 Dec 2019 08:29:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=qKBHCViZ843UEqO7pcagAazT2GgsTWVHZLrhVa078w0=; b=FAYUWK1ybX5FylAglog+FXtXovMifj+TQOoxvHcMmLOxDeHWkM9rRzuLyLolIIGXs1 CVEhvcmnWmG/vaHYz6u4r6gjXv3ZN8bRfrlYJ9hkWiNusLwApi5C4phmWLiHrrwz2Ku7 JfBjoKEUu+ECD4N11X4doFzdzr9n4fdsb3dxAuFhpgCSHpU7QQmHi1ErX94v0RYLZaj9 tZDff8o/DJ5+d1fVDdzo7N6ZUE6DHK0nu0pOS/0wyXb9HFRYSDJhHrmg8HVnr6kIq1t1 FqmgkfUDcRlv7+r9wlByvugHBIFzopkOigkSR0IrCnYLjYa93oCBlkVLpi1IoWGVsLCU Aidg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=qKBHCViZ843UEqO7pcagAazT2GgsTWVHZLrhVa078w0=; b=kFrl3zosnJ5agdDla0XUN52MujnSVB/cYTTT6Rj4n1MbkC4rQxC2k9ektc/jh0cQBh 3X8mhr/wgrdm46yf+dbYwTY1U6fARrMK2L+T61tDLAxPiSACYIzFWm6Xauv+CbVdIUnn lCFTtimtVCBSjp7vB4blcfCfdo/5FhXhJwGD/5ZxZpeK4qHwntgdf9sOazIS/3qa+aOP vf9Krv1R9wk8V5dN9eCI+ifUSj7ff+hM5Sf7Lv5dvuIB9KDitv8IFFJAnKAtaGl2BJIB yMSoOOupgmY240zSh0t0bqRuXGuYA2ejw+421h7K7bWIlnVCwfkFsc/mSomXOU6muRVQ FVKA== X-Gm-Message-State: APjAAAWh0I5qMJBgGocIn8EZRvZelWwTP5M6MOEnlJro8UD126rGkmsk Onxn07EjqJVTX0esfxcOhdM= X-Google-Smtp-Source: APXvYqwIfaEb/Gg9/XwAjNyR8LYWN3gpnSy6B9KsBM8IsqVnRU17+E5tT6s0W1j+bgC2XWVZQOQlmA== X-Received: by 2002:a05:620a:1030:: with SMTP id a16mr20651547qkk.1.1575995382834; Tue, 10 Dec 2019 08:29:42 -0800 (PST) Received: from raichu (toroon0560w-lp130-05-69-158-183-252.dsl.bell.ca. [69.158.183.252]) by smtp.gmail.com with ESMTPSA id y91sm1274859qtd.28.2019.12.10.08.29.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 10 Dec 2019 08:29:41 -0800 (PST) Sender: Mark Johnston Date: Tue, 10 Dec 2019 11:29:36 -0500 From: Mark Johnston To: Ryan Stone Cc: FreeBSD Current Subject: Re: dtrace not working on bhyve VM without invariant_tsc Message-ID: <20191210162936.GA8996@raichu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Rspamd-Queue-Id: 47XQWW5xWXz3Ktf X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=FAYUWK1y; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::72b as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-4.42 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[b.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-2.72)[ip: (-9.40), ipnet: 2607:f8b0::/32(-2.22), asn: 15169(-1.92), country: US(-0.05)]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Dec 2019 16:29:44 -0000 On Mon, Dec 09, 2019 at 09:27:01PM -0500, Ryan Stone wrote: > I have a bhyve VM guest on my laptop where dtrace just constantly > aborts whenever I try to use it: > > [rstone@ebpf dtrace]sudo dtrace -s fdcopy.d > Assertion failed: (buf->dtbd_timestamp >= first_timestamp), file > /usr/home/rstone/git/bsd-worktree/ebpf-import/cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c, > line 3026. > Abort trap > > I believe that the problem is caused by dtrace unconditionally using > rdtsc() to implement dtrace_gethrtime(), assuming that the values will > be stable for a given CPU. The VM's vcpus seem to be getting migrated > frequently. > > Should dtrace instead be using the system timecounter? That should > stand a much better chance of being monotonically increasing. One complication here is that the timecounter must be readable without calling any symbols in the kernel, to avoid probe recursion. That is, dtrace_gethrtime() cannot unconditionally use the system timecounter. It might be possible to use the timecounter specifically for dtbd_timestamp, but I suspect that will just shift the problem elsewhere: dtrace uses the TSC to provide a global order for individual records. As Eric noted you can pin the vCPUs, but I'm surprised that laptop does not have an invariant TSC. DTrace used to attempt to synchronize TSCs, but I disabled this for guests in r345359.