From nobody Thu Apr 27 00:23:35 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Q6GgR4FJkz47mPj; Thu, 27 Apr 2023 00:23:47 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Q6GgR3nn3z3mDj; Thu, 27 Apr 2023 00:23:47 +0000 (UTC) (envelope-from kevans@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682555027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CELBTP1Vowz9rgjig3G+arPLz4D+dL/oIRT6s49ECJ0=; b=B0361guy6SeCRH1ueMpzaJIQEq93mu8I8eOXOeIO3N7jzyiWNWZVbSaHe4A/OtgzjIHuL4 LsA4GNBH5bJCnn4JHUc/JYRKhz70ZUZiI7cBq2NskrKsFxmcKekWcSjq0h9hO5sOIqpR9P mhWn/IMjbG/dRs37+N6KxsuhK5tuz2+r29Vne0a7GnSa8wc/uoVBtEtgrZGiaFmCFPNkCg sJ+dTqbQnzzktnFN25/ABYa8t5R8o/TPdgofnECCCiBP3KIK3lqOsnXMR0+dnfGCmZ8Q1c TsvnU5iiZheLBYSe5M5UkAxK076JLEJEgI36VMDVML8buBtD4RvuVoqyWlAfKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1682555027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CELBTP1Vowz9rgjig3G+arPLz4D+dL/oIRT6s49ECJ0=; b=cXvhYCyBCWQN2rIq0CVmbPJW3s+6Kj1n4Iwe4BLeN0fBYAAfu5qv4jWrdouwEZXEYpwmls aZl09K7mJVHXRQnbbutZ7zIU0Yb0ijqIMj+weMPXC3SR7h5M5KFbKctiLwLLlF5QDtLxSW wl7hYI59SGAg/SixV2keXVrM15epQEQYuu6sKMJerog5jxrmTB+FnnBFvqfS7dcR/Ax8ww +8OclA4B+7vjgStONZeY55+TDN2fdkbbXR/6+vWX1CTnTMLUCh/MfZEebw6bEwQvmZrVIp AQF0nucDoqKHSPlEnQEeNRJpVDV2ancG1wSmZr+787DbNtl0txRhWaBTpuKD+A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1682555027; a=rsa-sha256; cv=none; b=BPyGblC33YljYHU9b16yG7piUCGBHKeVyXfXEL0IEk6qwijTjBYYCumHlKoWSgRnQdSF7h myoO7Xz7y8nZQq17UVLK+iWHl36rK2nhk+5eckagEJAc4Xw6FCi0ndxsFR1GcK9UlR5ZuM 0M0yUSU7/K2SVk8c1aBan1WbAZ6+DsZ4oU68YvhHT4UOt5Jt3LckfAmnUSEW/T8WeMXpVk qq1rYKwWhN6Dfth5sMUMYrIRXWRc5mBzwzPe1k/UFXRf7iVF5424kMTYN2J4ttoe0mXlgf zLRLNdNt9d3cVYQ0NavYw94djcS1+6dFrf1NtQnztpc8fUeiRCVSZGDSul7WbQ== Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: kevans) by smtp.freebsd.org (Postfix) with ESMTPSA id 4Q6GgR2knhzpVj; Thu, 27 Apr 2023 00:23:47 +0000 (UTC) (envelope-from kevans@freebsd.org) Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-5ef7c475379so34912086d6.3; Wed, 26 Apr 2023 17:23:47 -0700 (PDT) X-Gm-Message-State: AAQBX9fP72X2A9j1p4z1Kn1IjK+0xJSZeD5TnuQRDgdrVXYiPFV7e2pB Ak64HnU3wQ3hOeRU8nzZxlVJz8IY6mss7Mf+pOE= X-Google-Smtp-Source: AKy350YVK4ClVL7f+jqvTMkT2xGYCR7dKszNDLmcyF9pZK15GWR6JnYp2uFdUIcP+BEmYtQaYLdEj1IlfTpCIB/+Or8= X-Received: by 2002:a05:6214:529a:b0:5c5:c835:c8f1 with SMTP id kj26-20020a056214529a00b005c5c835c8f1mr35353234qvb.22.1682555026509; Wed, 26 Apr 2023 17:23:46 -0700 (PDT) List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 References: <202210271354.29RDsUoH077155@gitrepo.freebsd.org> In-Reply-To: From: Kyle Evans Date: Wed, 26 Apr 2023 19:23:35 -0500 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [EXTERNAL] Re: git: 9729f076e4d9 - main - arm64: Hyper-V: enablement for ARM64 in Hyper-V (Part 3, final) To: Souradeep Chakrabarti Cc: Wei Hu , "src-committers@freebsd.org" , "dev-commits-src-all@freebsd.org" , "dev-commits-src-main@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-ThisMailContainsUnwantedMimeParts: N On Wed, Apr 26, 2023 at 4:37=E2=80=AFPM Souradeep Chakrabarti wrote: > > > > > >-----Original Message----- > >From: Souradeep Chakrabarti > >Sent: Thursday, April 27, 2023 2:01 AM > >To: 'Kyle Evans' > >Cc: 'Wei Hu' ; 'src-committers@freebsd.org' >committers@freebsd.org>; 'dev-commits-src-all@freebsd.org' >all@freebsd.org>; 'dev-commits-src-main@freebsd.org' >main@freebsd.org> > >Subject: RE: [EXTERNAL] Re: git: 9729f076e4d9 - main - arm64: Hyper-V: > >enablement for ARM64 in Hyper-V (Part 3, final) > > > > > > > > > >>-----Original Message----- > >>From: Souradeep Chakrabarti > >>Sent: Wednesday, April 26, 2023 7:26 PM > >>To: Kyle Evans > >>Cc: Wei Hu ; src-committers@freebsd.org; > >>dev-commits-src- all@freebsd.org; dev-commits-src-main@freebsd.org > >>Subject: RE: [EXTERNAL] Re: git: 9729f076e4d9 - main - arm64: Hyper-V: > >>enablement for ARM64 in Hyper-V (Part 3, final) > >> > >> > >> > >> > >>>-----Original Message----- > >>>From: Kyle Evans > >>>Sent: Wednesday, April 26, 2023 3:39 AM > >>>To: Souradeep Chakrabarti > >>>Cc: Kyle Evans ; Wei Hu ; src- > >>>committers@freebsd.org; dev-commits-src-all@freebsd.org; > >>>dev-commits-src- main@freebsd.org > >>>Subject: Re: [EXTERNAL] Re: git: 9729f076e4d9 - main - arm64: Hyper-V: > >>>enablement for ARM64 in Hyper-V (Part 3, final) > [... snip ...] > >>>Hi, > >>> > >>>That seems odd. What happens if you bump the SYSINIT up to SI_SUB_SMP > >>>+ 1, SI_ORDER_FIRST? We don't know for a fact that all APs are ready > >>>for scheduling until after smp_after_idle_runnable(), which is also at > >>>SI_ORDER_ANY > >>>-- maybe there's just something going horribly wrong. > >>>That would perhaps explain why it's fine on a single processor system, > >>>which won't do anything useful (at least in later parts of SI_SUB_SMP)= . > >>[Souradeep] > >>In ARM64 SMP(VM with two cpu), storvsc attach is happening two times > >>for single scsi controller. > >>But for intel similar VM (two cpu), it is happening once. > >>For the dummy/fake storvsc in arm64, we are getting stuck at device_att= ach. > >> > >>Details: > >> > >>vmbus_scan_done(), not getting invoked because vmbus_add_child() is not > >>complete for a channel 15, because of which vmbus_devtq is having one > >>task pending. > >> > >>Now > >>By passing NMI in the hung system, after examining all threads: > >> > >>sched_switch() at sched_switch+0x4dc > >>mi_switch() at mi_switch+0x194 > >>sleepq_switch() at sleepq_switch+0xfc > >>_cv_wait() at _cv_wait+0x160 > >>_sema_wait() at _sema_wait+0x50 > >>storvsc_attach() at storvsc_attach+0x610 > >>device_attach() at device_attach+0x3f8 > >>device_probe_and_attach() at device_probe_and_attach+0x7c > >>vmbus_add_child() at vmbus_add_child+0x64 > >> > >>Now , > >> > >>It is stuck at waiting on sema_wait() on request->synch_sema in > >>hv_storvsc_channel_init() because > >>sema_post() on request->synch_sema is not getting invoked. Which unlock= s it. > >>This is because we are waiting on sema_wait on synch_sema > >>hv_storvsc_channel_init(), for storvsc1 , but there is no storvsc1 > >>device. So not getting a callback called for storvsc1. > >> > >>From ARM64 debug log: > >>If you see at line 545 again SCI device got detected. > >> > >> Line 370: storvsc0: Enlightened SCSI device detected > >> Line 371: storvsc0: on vmbus0 > >> Line 406: (probe0:storvsc0:0:0:0): storvsc scsi_status =3D 2, sr= b_status =3D 6 > >> Line 421: Fixed Direct Access SPC-3 SCSI= device > >> Line 436: da0: Fixed Direct Access SPC-3= SCSI device > >> Line 443: pass1: Removable CD-ROM SPC= -3 > >>SCSI device > >> Line 447: cd0: Removable CD-ROM SPC-3 > >SCSI > >>device > >> Line 545: storvsc1: Enlightened SCSI device detected > >> Line 547: storvsc1: Enlightened SCSI device detected > >> Line 549: storvsc1: hv_storvsc_on_channel_callback= is > >>called > >> > >>From Log: > >> > >>unknown: device_add_child for chan15 > >>storvsc1: Enlightened SCSI device detected > >>storvsc1: Enlightened SCSI device detected > >>storvsc1: on vmbus0 > >>storvsc ringbuffer size: 262144, max_io: 512 > >>storvsc1: chan15 assigned to cpu1 [vcpu1] > >>hn0: link state changed to UP > >>vmbus0: vmbus_chanmsg_handle type 0xa > >>storvsc1: gpadl_conn(chan15) succeeded > >>vmbus0: vmbus_chanmsg_handle type 0x6 > >>storvsc1: chan15 opened > >>waiting on sema wait synch_sema hv_storvsc_channel_init > >[Souradeep] The fix is working, the test bed had an issue, after fixing = that the fix is > >working. > >I will share the fix by this week. > [Souradeep] Small update, the problem happens only if there is an extra > SCSI controller on the system. Then it fails to attach storvsc for that S= CSI. > Excellent! Knowing now what configuration causes it; does it reproduce on x86 as well with an extra SCSI controller? I'd expect so, but maybe not- Thanks, Kyle Evans