From owner-freebsd-current@freebsd.org Sun Mar 10 15:14:21 2019 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCC651536A07 for ; Sun, 10 Mar 2019 15:14:20 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pf1-x42b.google.com (mail-pf1-x42b.google.com [IPv6:2607:f8b0:4864:20::42b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BFF53814E8 for ; Sun, 10 Mar 2019 15:14:19 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mail-pf1-x42b.google.com with SMTP id n125so1798589pfn.5 for ; Sun, 10 Mar 2019 08:14:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=xC4/GyEx9kHUWzcvvSw0VMwWyRkdll5e1YFF98Vwt4A=; b=jMaLszXozd9SNMWsQzyPvS6mxmtPqAsF+tGBvSZqWgXIqcdtdSOyiPpPfEP5WnhgUp cQQLTmIRyqlMfvaa29pa347jnRTecBXgdIQaNBNdDRlqZ7fzHTvtCiZkNXlMRoluiLsx MZ6+zkZ/1OJz1L/BsGdZSfopr7g9fSmwcIOb+LqxcSFNabpD29CEBCPnIZNw/MB6rhJG xd4DK9EJeiduB7rCyfB4/ZzXNhrfnJKZHi8ByQay6xqsMZ09RPb1QbhAPGSzWfq8HFvR bejRxxq+FIpwUzx080KVSRuvbMHXVZjfFar98iXsmAULV92LhzcJSeQHOFhKNGqHpgCW Wjbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=xC4/GyEx9kHUWzcvvSw0VMwWyRkdll5e1YFF98Vwt4A=; b=l4MlRmPW3Erm8wJH0q18dYsu+i8f0+g5kzwrkO3YIbgqPPYPPGEdwF0ltwfUqC4Ey0 4JReFMwR5NQmriADIJRoUpn7PWSbZWLNPpy7lfij4g8pI/5ZC2ua82AAjT3h10r3LBBd Xbu32+K/i9ZY2GetgH+pgCputAhB9dy1uaTJKhQ1dtXbwWWTtHM3m5IPXMthXANxpGra QxbUgWalSX6eeA9YIreJIolSYp5/YJKfG7ikbWHrB4B1b4/+RU+4JXI2g7/wkzOX00NH mPm8MU6N4mMweu+nnRFctH8VYBhEtRjOfWt9RNnL6LlGaLPzSFJO9aJ5W9+0LWiOyF+4 Ld9A== X-Gm-Message-State: APjAAAWIDcyHvajqDWPyPLe5uM5t9ICyXpS8meSrP9LZVdpucx1sCoUg A3g/D7FIT0YI2GQgiuj8FFtbVbsU X-Google-Smtp-Source: APXvYqx6DQ2uI0wA6EHVj455X0wfBEUS5+mJD0yCnc0AW7FNXKdYuNVTgMPsr673lCEXytlmkVpXwg== X-Received: by 2002:aa7:9143:: with SMTP id 3mr28665295pfi.238.1552230858055; Sun, 10 Mar 2019 08:14:18 -0700 (PDT) Received: from [192.168.20.22] (c-73-19-52-228.hsd1.wa.comcast.net. [73.19.52.228]) by smtp.gmail.com with ESMTPSA id f65sm5313757pff.21.2019.03.10.08.14.17 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 10 Mar 2019 08:14:17 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (1.0) Subject: Re: PMC Sierra SPC SAS-SATA Card + ESXi From: Enji Cooper X-Mailer: iPhone Mail (16D57) In-Reply-To: Date: Sun, 10 Mar 2019 08:14:16 -0700 Cc: freebsd-current@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <8A07386F-9C19-4438-B659-EDC8193043AF@gmail.com> References: To: Evgheni Melman X-Rspamd-Queue-Id: BFF53814E8 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=jMaLszXo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of yaneurabeya@gmail.com designates 2607:f8b0:4864:20::42b as permitted sender) smtp.mailfrom=yaneurabeya@gmail.com X-Spamd-Result: default: False [-6.28 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.98)[-0.978,0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[b.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.79)[ip: (-9.08), ipnet: 2607:f8b0::/32(-2.73), asn: 15169(-2.06), country: US(-0.07)] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2019 15:14:21 -0000 > On Mar 9, 2019, at 18:50, Evgheni Melman wrote: >=20 > So I got this weird setup, where I need to passthrough a PMC-Sierra PM8003= > card (same as PM8001, but with external QSFP afaik) to FreeBSD 11-STABLE > running in ESXi. > The card works fine in baremetal FreeBSD 11. Boot is ok, all disks > detected. When done in ESXi with passthrough though: > 1. First try, host crashed with PSOD. After tweaking passthrough.map in > esxi to do d3d0 reset, host does not psod anymore. > 2. FreeBSD kernel panics at pciAttach. After analyzing the driver > (sys\dev\pms), turns out it only works in MSI-X mode. Disabling > msi_blacklist in loader.conf makes FreeBSD not panic. > 3. FreeBSD deadlocks on boot. After analyzing the driver more (turning on > all debug features, and adding a few extra), I got this log (pastebin link= : > https://pastebin.com/cEGwFZZ2). The code tries to initialize SGPIO, but fo= r > whatever reason does not get an interrupt reply when run on ESXi which > makes it loop for infinity waiting for said interrupt. >=20 > The deadlock happens in the >=20 > while(!sgpioResponseSet) > { > tiCOMDelayedInterruptHandler(tiRoot, 0, 1, tiNonInterruptContext); > } >=20 > section of SendSgpioRequest in > sys\dev\pms\RefTisa\tisa\sassata\common\tdport.c >=20 > Sgpi seems not-so-critical feature to me, so I just limited the while loop= > to 100 iterations, which lets SendSgpioRequest return an error that is not= > considered fatal, and lets the code execute further which can be seen in > the log: https://pastebin.com/Rk6MD5ci (disregard the final fatal trap, I > think it is caused by me and/or my debugging code, same modifications > without debugging features do not trap and I get same trap on baremetal > with my debug kernel). >=20 > With those modifications, pciAttach is successful, I finally get a boot, > the card exists in lspci, BUT no drives connected to it are detected. >=20 > At this stage, I am out of ideas, looking for suggestions. If you=E2=80=99re trying to get the host production ready, I recommend using= an LSI/Avago based card (or emulate it in ESXi), or avoid FreeBSD if you ca= n. The driver in FreeBSD is based on a Linux driver that=E2=80=99s over a de= cade old and had a number of stability bugs the last time I used it [2 years= ago]. -Enji=