From owner-freebsd-current@freebsd.org Sun Mar 10 02:50:40 2019 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A75FF154413C for ; Sun, 10 Mar 2019 02:50:40 +0000 (UTC) (envelope-from mevgheni@gmail.com) Received: from mail-pg1-x52c.google.com (mail-pg1-x52c.google.com [IPv6:2607:f8b0:4864:20::52c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 32A9588127 for ; Sun, 10 Mar 2019 02:50:39 +0000 (UTC) (envelope-from mevgheni@gmail.com) Received: by mail-pg1-x52c.google.com with SMTP id q206so1284897pgq.4 for ; Sat, 09 Mar 2019 18:50:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=lbstRbPpfJTam2EqQVk652yRpgcjTXQZoakxFYRM+0o=; b=rSOlboNFgoGoABasOdfyO4zCD2R/To5ebBhn0863gF8zHkjjp3eGXxVq7r4eXUykw2 LEIFb3Bbosqnbe8jcl4ibXt9m3KqIM22oJhpAgJjEe1AH3sXFMDm7KTvpdnk86IEV1/y UlLbbnFvj/VQZZXjPBAERVxyu8v31zcRwCRQr+1dV+ZDS2VfhK3PmVV8CdRtqDJ4CbQU AVtkoUfiAmCsvS2AT1Ipap2QPKeOnX3rBRheaFu5B2PESQlmF299S7hfSc0tSCIR7Yec Ksw2DNCu3Y4kCLh1isXFMu84SrTPOhmJ/+RkdpyMe+2APjHudbg5TNukwRIh1BsKp16q r6jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=lbstRbPpfJTam2EqQVk652yRpgcjTXQZoakxFYRM+0o=; b=SEUdQWuJVyjYrawIOJRNXEZqN0ERdPv+dWqapDa3X2ovm+usMip3OwqbE3E/DfKVkf qovjOjJA63LZRKudooHCjvvMRO1tVDsPFdwCUGd0y4fwSoLLVMxO+cZyrclQwKOnam/6 g2JZZuj6el0lPnOnVaRazlhqvShp6wrMArEJu1Cr5AMHLtmvWLOp131lYrFD0GOg/32/ pagDmeUAGCnTr8m8kHxcYrIcBiM+5zI8wfJLbDmltxSO31J+rXnNCXy9alzx+zyD5IND +kLb5Mmr8OUsH649qf6zhtPxmUAq3NZ2SVDGsKxzdAQ+upfTjKHM9kHuCnmKxifOITqF Ww/g== X-Gm-Message-State: APjAAAU/FfunSh47bTFvXTrIAv13/ycUJ9SkJKmzms3IcqNV3T76dQ78 fh7U7tqesS9rP4yIn7yjuiCXUdNq+LxywO7A2nxEZLQY X-Google-Smtp-Source: APXvYqzXhYwRa6JJuEtPNk8T9ySSnIlbFpUzt2sTU5Z1Ag0LdCih7fQBSajV3lbGC83B2rMGn9zuA7nqMRfy7+JE6mI= X-Received: by 2002:a17:902:9f94:: with SMTP id g20mr27411245plq.0.1552186237768; Sat, 09 Mar 2019 18:50:37 -0800 (PST) MIME-Version: 1.0 From: Evgheni Melman Date: Sat, 9 Mar 2019 21:50:29 -0500 Message-ID: Subject: PMC Sierra SPC SAS-SATA Card + ESXi To: freebsd-current@freebsd.org X-Rspamd-Queue-Id: 32A9588127 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=rSOlboNF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mevgheni@gmail.com designates 2607:f8b0:4864:20::52c as permitted sender) smtp.mailfrom=mevgheni@gmail.com X-Spamd-Result: default: False [-5.58 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; URI_COUNT_ODD(1.00)[5]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.84)[-0.839,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.73)[ip: (-8.80), ipnet: 2607:f8b0::/32(-2.74), asn: 15169(-2.06), country: US(-0.07)]; RCVD_IN_DNSWL_NONE(0.00)[c.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2019 02:50:40 -0000 So I got this weird setup, where I need to passthrough a PMC-Sierra PM8003 card (same as PM8001, but with external QSFP afaik) to FreeBSD 11-STABLE running in ESXi. The card works fine in baremetal FreeBSD 11. Boot is ok, all disks detected. When done in ESXi with passthrough though: 1. First try, host crashed with PSOD. After tweaking passthrough.map in esxi to do d3d0 reset, host does not psod anymore. 2. FreeBSD kernel panics at pciAttach. After analyzing the driver (sys\dev\pms), turns out it only works in MSI-X mode. Disabling msi_blacklist in loader.conf makes FreeBSD not panic. 3. FreeBSD deadlocks on boot. After analyzing the driver more (turning on all debug features, and adding a few extra), I got this log (pastebin link: https://pastebin.com/cEGwFZZ2). The code tries to initialize SGPIO, but for whatever reason does not get an interrupt reply when run on ESXi which makes it loop for infinity waiting for said interrupt. The deadlock happens in the while(!sgpioResponseSet) { tiCOMDelayedInterruptHandler(tiRoot, 0, 1, tiNonInterruptContext); } section of SendSgpioRequest in sys\dev\pms\RefTisa\tisa\sassata\common\tdport.c Sgpi seems not-so-critical feature to me, so I just limited the while loop to 100 iterations, which lets SendSgpioRequest return an error that is not considered fatal, and lets the code execute further which can be seen in the log: https://pastebin.com/Rk6MD5ci (disregard the final fatal trap, I think it is caused by me and/or my debugging code, same modifications without debugging features do not trap and I get same trap on baremetal with my debug kernel). With those modifications, pciAttach is successful, I finally get a boot, the card exists in lspci, BUT no drives connected to it are detected. At this stage, I am out of ideas, looking for suggestions.