From owner-freebsd-virtualization@freebsd.org Thu Dec 8 00:05:41 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85ACCC6C750 for ; Thu, 8 Dec 2016 00:05:41 +0000 (UTC) (envelope-from rajil.s@gmail.com) Received: from mail-pg0-x242.google.com (mail-pg0-x242.google.com [IPv6:2607:f8b0:400e:c05::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E4B0366 for ; Thu, 8 Dec 2016 00:05:41 +0000 (UTC) (envelope-from rajil.s@gmail.com) Received: by mail-pg0-x242.google.com with SMTP id 3so24931578pgd.0 for ; Wed, 07 Dec 2016 16:05:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:references:from:to:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding; bh=cy3a3cJFLCbORx8LwqqMLhc/k0c6AJ1s9PiQqaB0k5E=; b=qoMV1UNWtUCl3x3X/+j9Kz5XEf77IrrNsLNh9OVSOANC+7/o8YQa0wnrrr33VYmArv tNgLxSTIeaNojkjVenrB+kBTJMYtp5uW1qhnJU06Q30gt6na3hQJn3RrU7OzNoNGhATV mWjk4nDejxyT6V9HrGdxvnviByHoPWwi/G62CEP9gK5sDpNKSgoMjDYCFrF7fox6wJVd Ke6JnysmCVEJhX6U2oZdfUQMQprh9I50ljtn5holCKNgx8+vwS1s6y2F6TdgFwUWyntO hvElzpmFrnVgB7Jql8KzOCrJJTn0ymNyphIOn26CWnra65L4Ubh/Mf5/bqBrlkxCWsaL VrOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:references:from:to:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding; bh=cy3a3cJFLCbORx8LwqqMLhc/k0c6AJ1s9PiQqaB0k5E=; b=NWXIq+wIIJWhucw2Ug0gZhAHXGxJ4xDV/on5RMyVMncPf7cY6QBavsgr56anUB0QID WrR08hpBHqrxMo4ZUBeWZqCoQof0XMF2pdulKAtV9TsPckvxGYo6bPkNEqyeHmZV9IiZ uru9FS6RHlbe489AKZ9VgGyh0oL0mV3XSyYCe/LiN3ogVc5Nku3rZ0AP5A2eNwFnPyCz YAp+Fx2wVQ2+zBrjHFfWrKf3lAlyryWsDREYzYzIKgJr4NggylzQakXXo0jWCOUFgXy2 0q77GLBxKH/ey5qTGypFq77T8v0fkccRCRC3fGFHHsNev7JjPGOAZWxhR1meGVok20Ry rzhg== X-Gm-Message-State: AKaTC03+gkdWy2VOKE4Y0qaeaWaJxIcm2X6z0Ycv98QYs3apj5fdu7FCuopxWwwuz4KvcQ== X-Received: by 10.84.128.46 with SMTP id 43mr153392750pla.100.1481155540633; Wed, 07 Dec 2016 16:05:40 -0800 (PST) Received: from [172.16.1.28] (c-73-155-87-64.hsd1.tx.comcast.net. [73.155.87.64]) by smtp.gmail.com with ESMTPSA id o126sm45249481pga.34.2016.12.07.16.05.38 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 07 Dec 2016 16:05:40 -0800 (PST) Subject: Re: Debian 8 CPU stall References: <9c9e83a5-16c6-0ab5-0ac4-af0a54430706@freebsd.org> From: Rajil Saraswat To: freebsd-virtualization@FreeBSD.org Message-ID: <6cff5bf2-9654-8627-83c4-6ab48ee763b5@gmail.com> Date: Wed, 7 Dec 2016 18:05:06 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: <9c9e83a5-16c6-0ab5-0ac4-af0a54430706@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2016 00:05:41 -0000 On 12/06/2016 10:50 PM, Peter Grehan wrote: > Hi Rajil, > >> I get these messages in Debian 8 VM running in bhyve FreeBSD-11 >> release. Any idea what could be the issue: >> >> INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, >> t=11047 jiffies, g=1038939, c=1038938, q=77) >> INFO: Stall ended before state dump start > > That's a sign that a vCPU wasn't able to run for an amount of time. > > Is the system oversubscribed ? i.e. more vCPUs than physical CPUs ? > Or, is the guest performing a lot of i/o ? > > later, > > Peter. > No the system is not oversubscribed. I have a 11 vCPU (1 on debian and 1 on ubuntu) on a 24 core machine. The debian jail is running x2go and an ssh server for remote access, so the I/O shouldnt be an issue. The ubuntu jail doesnt give out any warning messages though. Following is the latest error i received in the debian vm: [152444.353007] INFO: rcu_sched self-detected stall on CPU { 0} (t=6809 jiffies g=2685598 c=2685597 q=6) [152444.354261] sending NMI to all CPUs: [152444.354270] NMI backtrace for cpu 0 [152444.354274] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.36-1+deb8u2 [152444.354275] Hardware name: BHYVE, BIOS 1.00 03/14/2014 [152444.354277] task: ffffffff8181a460 ti: ffffffff81800000 task.ti: ffffffff81800000 [152444.354278] RIP: 0010:[] [] default_send_IPI_mask_sequence_phys+0xad/0xe0 [152444.354293] RSP: 0018:ffff88007fc03e18 EFLAGS: 00000046 [152444.354294] RAX: 0000000000000400 RBX: 000000000000a0ea RCX: 0000000000000000 [152444.354296] RDX: 0000000000000000 RSI: 0000000000000200 RDI: 0000000000000300 [152444.354297] RBP: ffffffff818e29c0 R08: ffffffff818e29c0 R09: 00000000000001bb [152444.354298] R10: 0000000000000000 R11: ffff88007fc03b96 R12: 0000000000000400 [152444.354299] R13: 0000000000000096 R14: 0000000000000002 R15: 0000000000000000 [152444.354301] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [152444.354303] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [152444.354303] CR2: 00000000019a3240 CR3: 00000000797ca000 CR4: 00000000000406f0 [152444.354305] Stack: [152444.354306] 0000000000000000 ffff88007fc0d6a0 ffffffff81853800 0000000000000000 [152444.354308] ffffffff818e2f20 0000000000000006 ffffffff81853800 ffffffff81047cf3 [152444.354309] ffff88007fc0d6a0 ffffffff810c73ea ffffffff00000007 ffffffff810c8f35 [152444.354311] Call Trace: [152444.354313] [152444.354318] [] ? arch_trigger_all_cpu_backtrace+0xc3/0x140 [152444.354327] [] ? rcu_check_callbacks+0x42a/0x670 [152444.354331] [] ? timekeeping_update.constprop.9+0x35/0x70 [152444.354335] [] ? tick_sched_handle.isra.16+0x60/0x60 [152444.354343] [] ? update_process_times+0x40/0x70 [152444.354345] [] ? tick_sched_handle.isra.16+0x20/0x60 [152444.354347] [] ? tick_sched_timer+0x3c/0x60 [152444.354351] [] ? __run_hrtimer+0x67/0x210 [152444.354353] [] ? hrtimer_interrupt+0xe9/0x220 [152444.354359] [] ? smp_apic_timer_interrupt+0x3b/0x50 [152444.354365] [] ? apic_timer_interrupt+0x6d/0x80 [152444.354366] [152444.354374] [] ? mwait_idle+0xa0/0xa0 [152444.354381] [] ? native_safe_halt+0x2/0x10 [152444.354384] [] ? default_idle+0x19/0xd0 [152444.354389] [] ? cpu_startup_entry+0x374/0x470 [152444.354392] [] ? start_kernel+0x497/0x4a2 [152444.354394] [] ? set_init_arg+0x4e/0x4e [152444.354396] [] ? early_idt_handler_array+0x120/0x120 [152444.354398] [] ? x86_64_start_kernel+0x14d/0x15c [152444.354399] Code: 8b 0c 25 00 53 5f ff 80 e5 10 75 f2 44 89 f8 c1 e0 18 89 04 25 10 53 5f ff 41 83 fe 02 44 89 e0 41 0f 45 c6 89 04 25 00 53 5f ff 91 4c 89 ef 57 9d 0f 1f 44 00 00 48 83 c4 08 5b 5d 41 5c 41 I use vm-bhyve for managing the jails which looks like this for debian8: guest="linux" loader="grub" cpu=1 memory=2048M network0_type="virtio-net" network0_switch="lannetwork" disk0_type="virtio-blk" disk0_name="/dev/zvol/vmpool/os2" disk0_dev="custom" passthru0="2/0/0" passthru1="2/0/1" Thanks, Rajil