From owner-freebsd-stable@freebsd.org Sun Nov 15 22:29:35 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E87EA30344 for ; Sun, 15 Nov 2015 22:29:35 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id 8428518F0 for ; Sun, 15 Nov 2015 22:29:35 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id DB0DD26E for ; Sun, 15 Nov 2015 22:29:35 +0000 (UTC) Date: Sun, 15 Nov 2015 22:29:35 +0000 (GMT) From: jenkins-admin@FreeBSD.org To: freebsd-stable@freebsd.org Message-ID: <1623488894.15.1447626575436.JavaMail.jenkins@jenkins-9.freebsd.org> In-Reply-To: <1675482905.12.1447616897417.JavaMail.jenkins@jenkins-9.freebsd.org> References: <1675482905.12.1447616897417.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: Jenkins build is back to normal : Build-UFS-image #2745 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkKKb2VAfYQKfu1t7qk4nR5qzUBEI+UqT4BPec4qHVhqUy0FFdq50sMH+3y9bCDNOufctov6VqTNffZ3YXArnZK95YF0OX97fh+E9txYOUX1adc+TikcKjuYpHmL5dE62eaZTI+4A5jnRonskQ1PaoIFz0Kbu4mWzkFsmdiXTraGzomXq4cHUCATA2+K4eDYgjXEQI30z3GOMmmZ4t/+6QGk1cMb/BqMWHbn80AsRCb4tU7Hpd72XLDpsuO7YRP1Q0CjmNAuBOTj+sFiiOe6U9HpqOlQN+iFUvBdZo/ybuy5Kh71cAaYQNL68cYdZJ6binH/DkG3KY/fS7DFYAeuwjwIDAQAB X-Jenkins-Job: Build-UFS-image X-Jenkins-Result: SUCCESS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Nov 2015 22:29:35 -0000 See From owner-freebsd-stable@freebsd.org Mon Nov 16 01:32:47 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F6A8A2F7FC for ; Mon, 16 Nov 2015 01:32:47 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id 8E0101DEE for ; Mon, 16 Nov 2015 01:32:47 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id 97B912EA for ; Mon, 16 Nov 2015 01:32:47 +0000 (UTC) Date: Mon, 16 Nov 2015 01:32:47 +0000 (GMT) From: jenkins-admin@FreeBSD.org To: freebsd-stable@freebsd.org Message-ID: <1659341858.17.1447637567590.JavaMail.jenkins@jenkins-9.freebsd.org> In-Reply-To: <684051696.16.1447627894171.JavaMail.jenkins@jenkins-9.freebsd.org> References: <684051696.16.1447627894171.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: Jenkins build is back to normal : Build-UFS-image #2747 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAkKKb2VAfYQKfu1t7qk4nR5qzUBEI+UqT4BPec4qHVhqUy0FFdq50sMH+3y9bCDNOufctov6VqTNffZ3YXArnZK95YF0OX97fh+E9txYOUX1adc+TikcKjuYpHmL5dE62eaZTI+4A5jnRonskQ1PaoIFz0Kbu4mWzkFsmdiXTraGzomXq4cHUCATA2+K4eDYgjXEQI30z3GOMmmZ4t/+6QGk1cMb/BqMWHbn80AsRCb4tU7Hpd72XLDpsuO7YRP1Q0CjmNAuBOTj+sFiiOe6U9HpqOlQN+iFUvBdZo/ybuy5Kh71cAaYQNL68cYdZJ6binH/DkG3KY/fS7DFYAeuwjwIDAQAB X-Jenkins-Job: Build-UFS-image X-Jenkins-Result: SUCCESS X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 01:32:47 -0000 See From owner-freebsd-stable@freebsd.org Mon Nov 16 09:09:52 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03640A3096B; Mon, 16 Nov 2015 09:09:52 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu01176b.smtpx.saremail.com (cu01176b.smtpx.saremail.com [195.16.151.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AE4A01E05; Mon, 16 Nov 2015 09:09:51 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.2.2] (izaro.sarenet.es [192.148.167.11]) by proxypop01.sare.net (Postfix) with ESMTPSA id 422FC9DCA79; Mon, 16 Nov 2015 10:00:35 +0100 (CET) Subject: Re: LSI SAS2008 mps driver preferred firmware version Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20151114143104.GA41119@in-addr.com> Date: Mon, 16 Nov 2015 10:00:32 +0100 Cc: Kai Gallasch , freebsd-scsi@freebsd.org, Royce Williams , freebsd-stable Content-Transfer-Encoding: quoted-printable Message-Id: <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> To: Gary Palmer X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 09:09:52 -0000 On Nov 14, 2015, at 3:31 PM, Gary Palmer wrote: > You can do thinks in /boot/loader.conf to hard code bus and drive > assignments. =20 >=20 > e.g. >=20 > hint.da.0.at=3D"scbus0" > hint.da.0.target=3D"19" > hint.da.0.unit=3D"0" > hint.da.1.at=3D"scbus0" > hint.da.1.target=3D"18" > hint.da.1.unit=3D"0" Beware, the targer number assignment is not predictable. There's no = guarantee especially if you replace a disk. Borja. From owner-freebsd-stable@freebsd.org Mon Nov 16 19:36:22 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CCF8A3023A; Mon, 16 Nov 2015 19:36:22 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ob0-x232.google.com (mail-ob0-x232.google.com [IPv6:2607:f8b0:4003:c01::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FA87132A; Mon, 16 Nov 2015 19:36:22 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by obdgf3 with SMTP id gf3so130932275obd.3; Mon, 16 Nov 2015 11:36:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=NCqONzSCvI/Il6YjpEVTiDVTeyfq96vcXjBL6GdnQmc=; b=dIDYvnbT8HSTBfiA7HrnqBVXlupkND9CQYjlQaduPnQ8UOHYP5/VhkIIAxYMRnHTPl uEy7no/gIDK4MTT6UlvlmLOuXzSUnITc6TUUDzg9rQiqpv5nrcmQc9IcC6tEFGsmXsxa C0a0Jo2C57NparL4V/kdAl/3YwuoToCl8atlGlosrxTxI0+pXZNiPjmGu1JI4L5Gq5fk 2Y0vyV4n3Q4c/0giCpxbnKIDEKutHVpqCa/mvGCPJ7+qkZCKosUM5PBroBlnmmO94e7g weGkzUsAVWQREPJkaOHsh9/x9U44rIjshk8TBtw+xFSAtscbNTxxll1qiOKUtRRSJ2o5 DGiA== MIME-Version: 1.0 X-Received: by 10.182.111.196 with SMTP id ik4mr24217131obb.60.1447702581324; Mon, 16 Nov 2015 11:36:21 -0800 (PST) Sender: kob6558@gmail.com Received: by 10.202.98.131 with HTTP; Mon, 16 Nov 2015 11:36:21 -0800 (PST) In-Reply-To: <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> Date: Mon, 16 Nov 2015 11:36:21 -0800 X-Google-Sender-Auth: DnSvn7GLpkMPhfkWQsJBPxzeGgs Message-ID: Subject: Re: LSI SAS2008 mps driver preferred firmware version From: Kevin Oberman To: Borja Marcos Cc: Gary Palmer , freebsd-scsi@freebsd.org, Royce Williams , freebsd-stable , Kai Gallasch Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 19:36:22 -0000 On Mon, Nov 16, 2015 at 1:00 AM, Borja Marcos wrote: > > On Nov 14, 2015, at 3:31 PM, Gary Palmer wrote: > > > You can do thinks in /boot/loader.conf to hard code bus and drive > > assignments. > > > > e.g. > > > > hint.da.0.at="scbus0" > > hint.da.0.target="19" > > hint.da.0.unit="0" > > hint.da.1.at="scbus0" > > hint.da.1.target="18" > > hint.da.1.unit="0" > > Beware, the target number assignment is not predictable. There's no > guarantee especially if you replace > a disk. > > > > > > Borja. > As already mentioned, unless you are using zfs, use gpart to label you file systems/disks. Then use the /dev/gpt/LABEL as the mount device in fstab. -- Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 From owner-freebsd-stable@freebsd.org Mon Nov 16 19:40:14 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 025B3A3037E; Mon, 16 Nov 2015 19:40:14 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-ob0-x244.google.com (mail-ob0-x244.google.com [IPv6:2607:f8b0:4003:c01::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B9A5F1770; Mon, 16 Nov 2015 19:40:13 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by obbbj7 with SMTP id bj7so13219112obb.3; Mon, 16 Nov 2015 11:40:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=kdG2KizSVUPL+J7KLCanem6IYZHMCJCm25+YsXbqAcE=; b=VNjPofSo2aKJgaRvyW/SBDRctXFiqyBqh3XtusrjynkfOXK7pzC6Nw+xBnWHKofci4 APBO91bad4985i3smghf6fGbOF8Ek1uYAowzmEHrOU6MV2aBIiviGuAnxab8CxjQlr4v OO5ewcUyqfHPxkvPgHgtumUgwGz3xicjVJXnOU8tE8RzbxFPwDUIsHeFAMe7YL9zf+kP JfEQuZBX1SIYbBh9RDzCeg7c7WNgu8wYOstVCZAovhY4rvUuZx9IGPWt7Nm3g3M5U6Qv XJiS5qJbeiseqlOGUiW2hTGsWdsZ9QUdgIEQqm5u+EnPnRHDIHMrSvKUWnbfh6oto30r O3GA== MIME-Version: 1.0 X-Received: by 10.182.91.6 with SMTP id ca6mr2519502obb.24.1447702813038; Mon, 16 Nov 2015 11:40:13 -0800 (PST) Received: by 10.76.29.74 with HTTP; Mon, 16 Nov 2015 11:40:12 -0800 (PST) In-Reply-To: References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> Date: Mon, 16 Nov 2015 11:40:12 -0800 Message-ID: Subject: Re: LSI SAS2008 mps driver preferred firmware version From: Freddie Cash To: Kevin Oberman Cc: Borja Marcos , freebsd-scsi@freebsd.org, Royce Williams , freebsd-stable , Kai Gallasch Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 19:40:14 -0000 On Mon, Nov 16, 2015 at 11:36 AM, Kevin Oberman wrote= : > On Mon, Nov 16, 2015 at 1:00 AM, Borja Marcos wrote: > > > > > On Nov 14, 2015, at 3:31 PM, Gary Palmer wrote: > > > > > You can do thinks in /boot/loader.conf to hard code bus and drive > > > assignments. > > > > > > e.g. > > > > > > hint.da.0.at=3D"scbus0" > > > hint.da.0.target=3D"19" > > > hint.da.0.unit=3D"0" > > > hint.da.1.at=3D"scbus0" > > > hint.da.1.target=3D"18" > > > hint.da.1.unit=3D"0" > > > > Beware, the target number assignment is not predictable. There's no > > guarantee especially if you replace > > a disk. > > > > > > > > > > > > Borja. > > > > As already mentioned, unless you are using zfs, use gpart to label you fi= le > systems/disks. Then use the /dev/gpt/LABEL as the mount device in fstab. > =E2=80=8BEven if you are using ZFS, labelling the drives with the location = of the disk in the system (enclosure, column, row, whatever) makes things so much easier to work with when there are disk-related issues. Just create a single partition that covers the whole disk, label it, and use the label to create the vdevs in the pool.=E2=80=8B --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-stable@freebsd.org Mon Nov 16 20:57:41 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8A8EA30828; Mon, 16 Nov 2015 20:57:41 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 703B91812; Mon, 16 Nov 2015 20:57:41 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1ZyQqE-000Obi-ST; Mon, 16 Nov 2015 23:57:34 +0300 Date: Mon, 16 Nov 2015 23:57:34 +0300 From: Slawa Olhovchenkov To: Freddie Cash Cc: Kevin Oberman , freebsd-scsi@freebsd.org, Royce Williams , freebsd-stable , Borja Marcos , Kai Gallasch Subject: Re: LSI SAS2008 mps driver preferred firmware version Message-ID: <20151116205734.GM48728@zxy.spb.ru> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 20:57:41 -0000 On Mon, Nov 16, 2015 at 11:40:12AM -0800, Freddie Cash wrote: > On Mon, Nov 16, 2015 at 11:36 AM, Kevin Oberman wrote: > > > On Mon, Nov 16, 2015 at 1:00 AM, Borja Marcos wrote: > > > > > > > > On Nov 14, 2015, at 3:31 PM, Gary Palmer wrote: > > > > > > > You can do thinks in /boot/loader.conf to hard code bus and drive > > > > assignments. > > > > > > > > e.g. > > > > > > > > hint.da.0.at="scbus0" > > > > hint.da.0.target="19" > > > > hint.da.0.unit="0" > > > > hint.da.1.at="scbus0" > > > > hint.da.1.target="18" > > > > hint.da.1.unit="0" > > > > > > Beware, the target number assignment is not predictable. There's no > > > guarantee especially if you replace > > > a disk. > > > > > > > > > > > > > > > > > > Borja. > > > > > > > As already mentioned, unless you are using zfs, use gpart to label you file > > systems/disks. Then use the /dev/gpt/LABEL as the mount device in fstab. > > > > ​Even if you are using ZFS, labelling the drives with the location of the > disk in the system (enclosure, column, row, whatever) makes things so much > easier to work with when there are disk-related issues. > > Just create a single partition that covers the whole disk, label it, and > use the label to create the vdevs in the pool.​ Bad idea. Re-placed disk in different bay don't relabel automaticly. Other issuse where disk placed in bay some remotely hands in data center -- I am relay don't know how disk distributed by bays. Best way for identify disk -- uses enclouse services. I have many sites with ZFS on whole disk and some sites with ZFS on GPT partition. ZFS on GPT more heavy for administration. From owner-freebsd-stable@freebsd.org Mon Nov 16 21:19:57 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A7F4A30EA9; Mon, 16 Nov 2015 21:19:57 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-ob0-x235.google.com (mail-ob0-x235.google.com [IPv6:2607:f8b0:4003:c01::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10595189B; Mon, 16 Nov 2015 21:19:57 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by obdgf3 with SMTP id gf3so133127158obd.3; Mon, 16 Nov 2015 13:19:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=DbhmXduNBQvMEXo+tgkVysvcxT4EF9WC4VYxdBB5FYk=; b=YzFMQCxcuVNE6leuw7bTbw2Coni/5NaeRJTbc82XHjv48I6KLwiuzT3jnT6TspMMTG Xn8rFqSQghYDN4onDzQ58/GN69w/5fpN7aFpdedIdN1F7hIzUchVufmE6DEEyX4W6IcB cnvqqQglaiA2CexMzl/9Z6Jj0SuPlAcQQHELrYALN4c0Rs6Uj2lST48VUm7gWR6TM+Ag xT2EWjDoGs9yzwOdx2X+lmvctRCGw6iV5SKFx/CGR8zi5GLj8qJ+GE67OFrDqkRs5Hvg DtqMCvtio1cjaYpWpIBgh0xcVgBxwzXZ/5HPBvrMpOn6uN3e89biO8jX0kT9c+Ph1SPR JVFQ== MIME-Version: 1.0 X-Received: by 10.60.76.42 with SMTP id h10mr22713812oew.13.1447708796286; Mon, 16 Nov 2015 13:19:56 -0800 (PST) Received: by 10.76.29.74 with HTTP; Mon, 16 Nov 2015 13:19:55 -0800 (PST) In-Reply-To: <20151116205734.GM48728@zxy.spb.ru> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> Date: Mon, 16 Nov 2015 13:19:55 -0800 Message-ID: Subject: Re: LSI SAS2008 mps driver preferred firmware version From: Freddie Cash To: Slawa Olhovchenkov Cc: Kevin Oberman , freebsd-scsi@freebsd.org, Royce Williams , freebsd-stable , Borja Marcos , Kai Gallasch Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Nov 2015 21:19:57 -0000 On Mon, Nov 16, 2015 at 12:57 PM, Slawa Olhovchenkov wrote= : > On Mon, Nov 16, 2015 at 11:40:12AM -0800, Freddie Cash wrote: > > > On Mon, Nov 16, 2015 at 11:36 AM, Kevin Oberman > wrote: > > > As already mentioned, unless you are using zfs, use gpart to label yo= u > file > > > systems/disks. Then use the /dev/gpt/LABEL as the mount device in > fstab. > > > > > > > =E2=80=8BEven if you are using ZFS, labelling the drives with the locat= ion of the > > disk in the system (enclosure, column, row, whatever) makes things so > much > > easier to work with when there are disk-related issues. > > > > Just create a single partition that covers the whole disk, label it, an= d > > use the label to create the vdevs in the pool.=E2=80=8B > > Bad idea. > Re-placed disk in different bay don't relabel automaticly. > =E2=80=8BDid the original disk get labelled automatically? No, you had to = do that when you first started using it. So, why would you expect a replaced disk to get labelled automatically? Offline the dead/dying disk. Physically remove the disk. Insert the new disk. Partition / label the new disk. "zfs replace" using the new label to get it into the pool.=E2=80=8B > Other issuse where disk placed in bay some remotely hands in data > center -- I am relay don't know how disk distributed by bays. > =E2=80=8BYou label the disks as they are added to the system the first time= . That way, you always know where each disk is located, and you only deal with the labels. Then, when you need to replace a disk (or ask someone in a remote location to replace it) it's a simple matter: the label on the disk itself tells you where the disk is physically located. And it doesn't change if the controller decides to change the direction it enumerates devices. Which is easier to tell someone in a remote location: Replace disk enc0a6 (meaning enclosure 0, column A, row 6)? or Replace the disk called da36?=E2=80=8B =E2=80=8Bor Find the disk with serial number XXXXXXXX? or Replace the disk where the light is (hopefully) flashing (but I can't tell you which enclosure, front or back, or anything else like that)? The first one lets you know exactly where the disk is located physically. The second one just tells you the name of the device as determined by the OS, but doesn't tell you anything about where it is located. And it can change with a kernel update, driver update, or firmware update! The third requires you to pull every disk in turn to read the serial number off the drive itself. In order for the second or third option to work, you'd have to write down the device names and/or serial numbers and stick that onto the drive bay itself.=E2=80=8B > Best way for identify disk -- uses enclouse services. > =E2=80=8BOnly if your enclosure services are actually working (or even enab= led). I've yet to work on a box where that actually works (we custom-build our storage boxes using OTS hardware). Best way, IMO, is to use the physical location of the device as the actual device name itself. That way, there's never any ambiguity at the physical layer, the driver layer, the OS layer, or the ZFS pool layer.=E2=80=8B > I have many sites with ZFS on whole disk and some sites with ZFS on > GPT partition. ZFS on GPT more heavy for administration. > =E2=80=8BIt's 1 extra step: partition the drive, supplying the location of= the drive as the label for the partition. Everything else works exactly the same. I used to do everything with whole drives and no labels. Did that for about a month, until 2 separate drives on separate controllers died (in a 24-bay setup) and I couldn't figure out where they were located as a BIOS upgrade changed which controller loaded first. And then I had to work on a server that someone else configured with direct-attach bays (24 cables) that were connected almost at random. Then I used glabel(8) to label the entire disk, and things were much better. But that didn't always play well with 4K drives, and replacing drives that were the same size didn't always work as the number of sectors in each disk was different (ZFS plays better with this now). Then I started to GPT partition things, and life has been so much simpler. All the partitions are aligned to 1 MB, and I can manually set the size of the partition to work around different physical sector counts. All the partitions are labelled using the physical location of the disk (originally just row/column naming like a spreadsheet, but now I'm adding enclosure name as well as we expand to multiple enclosures per system). It's so much simpler now, ESPECIALLY when I have to get someone to do something remotely. :) =E2=80=8BEveryone has their own way to manage things. I just haven't seen = any better setup than labelling the drives themselves using their physical location.=E2=80=8B --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-stable@freebsd.org Tue Nov 17 08:08:08 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A0F8A319F4; Tue, 17 Nov 2015 08:08:08 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C6CAA1051; Tue, 17 Nov 2015 08:08:07 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id tAH87v6K076985; Tue, 17 Nov 2015 09:07:57 +0100 (CET) Received: from [217.29.44.157] ([217.29.44.157]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id tAH87vtO031155; Tue, 17 Nov 2015 09:07:57 +0100 (CET) (envelope-from hausen@punkt.de) Subject: ZFS on labelled partitions (was: Re: LSI SAS2008 mps driver preferred firmware version) Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Content-Type: multipart/signed; boundary="Apple-Mail=_1B2B703B-9A6A-4088-B7E0-8CC3E339EA61"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.6b2 From: "Patrick M. Hausen" In-Reply-To: Date: Tue, 17 Nov 2015 09:08:12 +0100 Cc: freebsd-stable , freebsd-scsi@freebsd.org Message-Id: <878D84A9-4553-413B-ACA4-5ABF499F28C1@punkt.de> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> To: Freddie Cash X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 08:08:08 -0000 --Apple-Mail=_1B2B703B-9A6A-4088-B7E0-8CC3E339EA61 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, all, > Am 16.11.2015 um 22:19 schrieb Freddie Cash : >=20 > =E2=80=8BYou label the disks as they are added to the system the first = time. That > way, you always know where each disk is located, and you only deal = with the > labels. we do the same for obvious reasons. But I always wonder about the = possible downsides, because ZFS documentation explicitly states: ZFS operates on raw devices, so it is possible to create a = storage pool comprised of logical volumes, either software or hardware. This configuration is not = recommended, as ZFS works best when it uses raw physical devices. Using logical volumes = might sacrifice performance, reliability, or both, and should be avoided. (from http://docs.oracle.com/cd/E19253-01/819-5461/gbcik/index.html) Can anyone shed some lght on why not using raw devices might sacrifice performance or reliability? Or is this just outdated folklore? Thanks, Patrick -- punkt.de GmbH * Kaiserallee 13a * 76133 Karlsruhe Tel. 0721 9109 0 * Fax 0721 9109 100 info@punkt.de http://www.punkt.de Gf: J=C3=BCrgen Egeling AG Mannheim 108285 --Apple-Mail=_1B2B703B-9A6A-4088-B7E0-8CC3E339EA61 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJWSuBsAAoJEJBvLuLt2olcScMH/1VvysKjTcZqVOIltP1FyZWn LzaS3uGKVPNJKYKmL6fl0oluynUhFdupvPqLT8qI5ViMe6lpA6ay9dCZ2uhdcOhS mu2qT8orRTSNWr4DZmpuH0lxulYypEcBGN0oxwokcTuv3imVE1iMSBFnUoNWS1NT cf/jBREsixk3YFr2Az7cKG80Jcnp/T2sPrTMQV1UWvkz4qe1JEUCU5VflUhPEH56 CJpp5U1647nrlvqmsv5XZgENtA3QGN2WrhFei56C1wGRkrmqDGy81WQzQAaVsPbD t6JgaOJaFKB+m4S1T3Ar0Qf96rp6/mm+/f96fCaurzK4oBuHSuuaJPs5cHUPke4= =MTIy -----END PGP SIGNATURE----- --Apple-Mail=_1B2B703B-9A6A-4088-B7E0-8CC3E339EA61-- From owner-freebsd-stable@freebsd.org Tue Nov 17 08:23:07 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6876AA31DC0; Tue, 17 Nov 2015 08:23:07 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 247DA1A2D; Tue, 17 Nov 2015 08:23:06 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id DBF7A28431; Tue, 17 Nov 2015 09:22:57 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id C13682842B; Tue, 17 Nov 2015 09:22:56 +0100 (CET) Message-ID: <564AE3E0.9060209@quip.cz> Date: Tue, 17 Nov 2015 09:22:56 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: "Patrick M. Hausen" , Freddie Cash CC: freebsd-scsi@freebsd.org, freebsd-stable Subject: Re: ZFS on labelled partitions References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> <878D84A9-4553-413B-ACA4-5ABF499F28C1@punkt.de> In-Reply-To: <878D84A9-4553-413B-ACA4-5ABF499F28C1@punkt.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 08:23:07 -0000 Patrick M. Hausen wrote on 11/17/2015 09:08: > Hi, all, > >> Am 16.11.2015 um 22:19 schrieb Freddie Cash : >> >> ​You label the disks as they are added to the system the first time. That >> way, you always know where each disk is located, and you only deal with the >> labels. > > we do the same for obvious reasons. But I always wonder about the possible > downsides, because ZFS documentation explicitly states: > > ZFS operates on raw devices, so it is possible to create a storage pool comprised of logical > volumes, either software or hardware. This configuration is not recommended, as ZFS works > best when it uses raw physical devices. Using logical volumes might sacrifice performance, > reliability, or both, and should be avoided. > > (from http://docs.oracle.com/cd/E19253-01/819-5461/gbcik/index.html) > > Can anyone shed some lght on why not using raw devices might sacrifice > performance or reliability? Or is this just outdated folklore? It was on Solaris but not on FreeBSD. If you were using partitions on Solaris the drive cache was disabled (or something like that, I am not 100% sure) Miroslav Lachman From owner-freebsd-stable@freebsd.org Tue Nov 17 08:23:09 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9AE0A31DDF; Tue, 17 Nov 2015 08:23:09 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x22d.google.com (mail-wm0-x22d.google.com [IPv6:2a00:1450:400c:c09::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 473261A44; Tue, 17 Nov 2015 08:23:09 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by wmww144 with SMTP id w144so14247000wmw.0; Tue, 17 Nov 2015 00:23:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=gDedhX817bOHm9pQoLUg0E3hYQEtgvcFTGwPzmhQOyA=; b=yCl0BKOE8rNrACU7UxTzWcIvXvC0Xi/C3jso+rhQJiAUV2XIHFaqd3UkuzcnCydSQS udUS9V86QVUDJXRkdWHJy2ecE1UKOm4UoG+tVix5ylT9Wn9/QGFi7xuQLTqPFn8yHk0U u3bS1UlvbJa9nnOjpjXag/jqDqAvcEauZ0eIutQS7FDY6kkJIxE/LUjBCmJ4ROpqR0LW k1CCwmdekUXYrBCEYSdyY4Fbosms0kmzrtRchhvOPy/7VdOD6Wx8xV/TVF6pFBFPTCY1 xdlZ0QOloHmP+pFeSYWQvuca5923NcHmKDJQpLShpoJGkUM5XFbulrb5wZxYPLo52GZd wjag== MIME-Version: 1.0 X-Received: by 10.194.239.104 with SMTP id vr8mr40013392wjc.64.1447748586845; Tue, 17 Nov 2015 00:23:06 -0800 (PST) Received: by 10.28.181.213 with HTTP; Tue, 17 Nov 2015 00:23:06 -0800 (PST) In-Reply-To: <878D84A9-4553-413B-ACA4-5ABF499F28C1@punkt.de> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> <878D84A9-4553-413B-ACA4-5ABF499F28C1@punkt.de> Date: Tue, 17 Nov 2015 08:23:06 +0000 Message-ID: Subject: Re: ZFS on labelled partitions (was: Re: LSI SAS2008 mps driver preferred firmware version) From: krad To: "Patrick M. Hausen" Cc: Freddie Cash , freebsd-scsi@freebsd.org, freebsd-stable Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 08:23:09 -0000 >From what i remember its a control thing. If you have another layer below zfs, be it software based or hardware based, zfs cant be sure what is going on, therefore cant guarantee anything. This is quite a big thing when it comes to data integrity which is a big reason to use zfs. I remember having to be very careful with some external caching arrays and making sure that they flushed correctly as often they ignore the scsi flush commands. This is one reason why I would always use the IT based firmware rather then the RAID one, as its less likely to lead to issues. On 17 November 2015 at 08:08, Patrick M. Hausen wrote: > Hi, all, > > > Am 16.11.2015 um 22:19 schrieb Freddie Cash : > > > > =E2=80=8BYou label the disks as they are added to the system the first = time. > That > > way, you always know where each disk is located, and you only deal with > the > > labels. > > we do the same for obvious reasons. But I always wonder about the possibl= e > downsides, because ZFS documentation explicitly states: > > ZFS operates on raw devices, so it is possible to create a storag= e > pool comprised of logical > volumes, either software or hardware. This configuration is not > recommended, as ZFS works > best when it uses raw physical devices. Using logical volumes > might sacrifice performance, > reliability, or both, and should be avoided. > > (from http://docs.oracle.com/cd/E19253-01/819-5461/gbcik/index.html) > > Can anyone shed some lght on why not using raw devices might sacrifice > performance or reliability? Or is this just outdated folklore? > > Thanks, > Patrick > -- > punkt.de GmbH * Kaiserallee 13a * 76133 Karlsruhe > Tel. 0721 9109 0 * Fax 0721 9109 100 > info@punkt.de http://www.punkt.de > Gf: J=C3=BCrgen Egeling AG Mannheim 108285 > > From owner-freebsd-stable@freebsd.org Tue Nov 17 08:32:36 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECE5DA2E0F4; Tue, 17 Nov 2015 08:32:35 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com [IPv6:2a00:1450:400c:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 81F271061; Tue, 17 Nov 2015 08:32:35 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by wmvv187 with SMTP id v187so214865460wmv.1; Tue, 17 Nov 2015 00:32:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=js8gPnZ617uyojxHsa6RMFJvhn/HmboCYLYIvFMcRew=; b=EWzuCFqaoIVxWIOW9WQ/MnPyMeSg9HUFedBjygjgyJo8CaJZVQW+5W/XOoGgO5vOs4 cVv9cw1jtsSfsMdMdK4HT5+9mCIuD+ajn3Ez3OvH3MvWeHtuNLFZc1fl54lIRnMF40uL WL/VuoEIyirmTYP+V5618viV5vgEYK/zDB9ncTqT+F1qJUxhIFKuuahnFgsnTXqc83/4 wxtwTm47RQQdURMatb24zumkQGVGKiy9+RcdqIoY5FJAPuehyFPcZCc/1l2MUrcMLPx6 YkH+q9p/yG77FSeoMdayqefvGDKB21a6SpRw9NLetqF61jC81uFS1triw8QMJDtPt73V t6mQ== MIME-Version: 1.0 X-Received: by 10.194.185.173 with SMTP id fd13mr41841837wjc.54.1447749153937; Tue, 17 Nov 2015 00:32:33 -0800 (PST) Received: by 10.28.181.213 with HTTP; Tue, 17 Nov 2015 00:32:33 -0800 (PST) In-Reply-To: <564AE3E0.9060209@quip.cz> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> <878D84A9-4553-413B-ACA4-5ABF499F28C1@punkt.de> <564AE3E0.9060209@quip.cz> Date: Tue, 17 Nov 2015 08:32:33 +0000 Message-ID: Subject: Re: ZFS on labelled partitions From: krad To: Miroslav Lachman <000.fbsd@quip.cz> Cc: "Patrick M. Hausen" , Freddie Cash , freebsd-scsi@freebsd.org, freebsd-stable Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 08:32:36 -0000 It was a control thing again, if you were using a partition another application could be using the drive on another partition, therefore zfs couldn't guarantee exclusive use of the disk so had to be more careful in the way it operated the drive. I think this meant I went into write through mode like you say. On 17 November 2015 at 08:22, Miroslav Lachman <000.fbsd@quip.cz> wrote: > Patrick M. Hausen wrote on 11/17/2015 09:08: > >> Hi, all, >> >> Am 16.11.2015 um 22:19 schrieb Freddie Cash : >>> >>> =E2=80=8BYou label the disks as they are added to the system the first = time. >>> That >>> way, you always know where each disk is located, and you only deal with >>> the >>> labels. >>> >> >> we do the same for obvious reasons. But I always wonder about the possib= le >> downsides, because ZFS documentation explicitly states: >> >> ZFS operates on raw devices, so it is possible to create a >> storage pool comprised of logical >> volumes, either software or hardware. This configuration is not >> recommended, as ZFS works >> best when it uses raw physical devices. Using logical volumes >> might sacrifice performance, >> reliability, or both, and should be avoided. >> >> (from http://docs.oracle.com/cd/E19253-01/819-5461/gbcik/index.html) >> >> Can anyone shed some lght on why not using raw devices might sacrifice >> performance or reliability? Or is this just outdated folklore? >> > > It was on Solaris but not on FreeBSD. If you were using partitions on > Solaris the drive cache was disabled (or something like that, I am not 10= 0% > sure) > > Miroslav Lachman > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Tue Nov 17 08:37:16 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 952C1A2E2B1; Tue, 17 Nov 2015 08:37:16 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: from mail-wm0-x242.google.com (mail-wm0-x242.google.com [IPv6:2a00:1450:400c:c09::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 23AA5124E; Tue, 17 Nov 2015 08:37:16 +0000 (UTC) (envelope-from kraduk@gmail.com) Received: by wmec201 with SMTP id c201so2973252wme.3; Tue, 17 Nov 2015 00:37:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=UPVJFo7Ljh4RXtH263iyf1LDRVM3hGJbihvLXbBwdwo=; b=dyPWJB0EUUPdg9+ZK2MZ9fzJrVmgKLIhxTiD9yrI9UViFNrVq22gJoT6Hj3EEFg39h URIURBDKUQ27kpEql59bghXQXzkhcWolFfHOMyBpxYAk7tXGcASgKhtQRx/+5//oPFJI 7WU2vJX1q1tmnj9tk81E7fXUsu5IvN3NSeg3+FRzK16z48v1AUrQTa+xLD2HdrL9mc0V dQ4Z8xZnfbMo3CWiz1Xbm3KzMRTwJaQGjSQEfg+FtpYKAu0ZK9kfpyjogwwQSpZcNBjc B0uVwuaJT7cahEw2fweOhsN4dI5P9wdo94N9zQBwMeDVodjHDW1HLAVwWvK5tNOCwo2l oojg== MIME-Version: 1.0 X-Received: by 10.194.134.3 with SMTP id pg3mr46399390wjb.63.1447749434500; Tue, 17 Nov 2015 00:37:14 -0800 (PST) Received: by 10.28.181.213 with HTTP; Tue, 17 Nov 2015 00:37:14 -0800 (PST) In-Reply-To: References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> Date: Tue, 17 Nov 2015 08:37:14 +0000 Message-ID: Subject: Re: LSI SAS2008 mps driver preferred firmware version From: krad To: Freddie Cash Cc: Slawa Olhovchenkov , Royce Williams , freebsd-stable , Borja Marcos , Kai Gallasch , Kevin Oberman , freebsd-scsi@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 08:37:16 -0000 I disagree, get the remote hands to copy the serial number to an easily visible location on the drive when its in the enclosure. Then label the drives with the serial number (or a compatible version of it). That way the label is tied to the drive, and you dont have to rely on the remote hands 100%. Better still do the physical labelling yourself On 16 November 2015 at 21:19, Freddie Cash wrote: > On Mon, Nov 16, 2015 at 12:57 PM, Slawa Olhovchenkov > wrote: > > > On Mon, Nov 16, 2015 at 11:40:12AM -0800, Freddie Cash wrote: > > > > > On Mon, Nov 16, 2015 at 11:36 AM, Kevin Oberman > > wrote: > > > > As already mentioned, unless you are using zfs, use gpart to label > you > > file > > > > systems/disks. Then use the /dev/gpt/LABEL as the mount device in > > fstab. > > > > > > > > > > =E2=80=8BEven if you are using ZFS, labelling the drives with the loc= ation of > the > > > disk in the system (enclosure, column, row, whatever) makes things so > > much > > > easier to work with when there are disk-related issues. > > > > > > Just create a single partition that covers the whole disk, label it, > and > > > use the label to create the vdevs in the pool.=E2=80=8B > > > > Bad idea. > > Re-placed disk in different bay don't relabel automaticly. > > > > =E2=80=8BDid the original disk get labelled automatically? No, you had t= o do that > when you first started using it. So, why would you expect a replaced dis= k > to get labelled automatically? > > Offline the dead/dying disk. > Physically remove the disk. > Insert the new disk. > Partition / label the new disk. > "zfs replace" using the new label to get it into the pool.=E2=80=8B > > > > Other issuse where disk placed in bay some remotely hands in data > > center -- I am relay don't know how disk distributed by bays. > > > > =E2=80=8BYou label the disks as they are added to the system the first ti= me. That > way, you always know where each disk is located, and you only deal with t= he > labels. > > Then, when you need to replace a disk (or ask someone in a remote locatio= n > to replace it) it's a simple matter: the label on the disk itself tells > you where the disk is physically located. And it doesn't change if the > controller decides to change the direction it enumerates devices. > > Which is easier to tell someone in a remote location: > Replace disk enc0a6 (meaning enclosure 0, column A, row 6)? > or > Replace the disk called da36?=E2=80=8B > =E2=80=8Bor > Find the disk with serial number XXXXXXXX? > or > Replace the disk where the light is (hopefully) flashing (but I can't > tell you which enclosure, front or back, or anything else like that)? > > The first one lets you know exactly where the disk is located physically. > > The second one just tells you the name of the device as determined by the > OS, but doesn't tell you anything about where it is located. And it can > change with a kernel update, driver update, or firmware update! > > The third requires you to pull every disk in turn to read the serial numb= er > off the drive itself. > > In order for the second or third option to work, you'd have to write down > the device names and/or serial numbers and stick that onto the drive bay > itself.=E2=80=8B > > > > Best way for identify disk -- uses enclouse services. > > > > =E2=80=8BOnly if your enclosure services are actually working (or even en= abled). > I've yet to work on a box where that actually works (we custom-build our > storage boxes using OTS hardware). > > Best way, IMO, is to use the physical location of the device as the actua= l > device name itself. That way, there's never any ambiguity at the physica= l > layer, the driver layer, the OS layer, or the ZFS pool layer.=E2=80=8B > > > > I have many sites with ZFS on whole disk and some sites with ZFS on > > GPT partition. ZFS on GPT more heavy for administration. > > > > =E2=80=8BIt's 1 extra step: partition the drive, supplying the location = of the > drive as the label for the partition. > > Everything else works exactly the same. > > I used to do everything with whole drives and no labels. Did that for > about a month, until 2 separate drives on separate controllers died (in a > 24-bay setup) and I couldn't figure out where they were located as a BIOS > upgrade changed which controller loaded first. And then I had to work on= a > server that someone else configured with direct-attach bays (24 cables) > that were connected almost at random. > > Then I used glabel(8) to label the entire disk, and things were much > better. But that didn't always play well with 4K drives, and replacing > drives that were the same size didn't always work as the number of sector= s > in each disk was different (ZFS plays better with this now). > > Then I started to GPT partition things, and life has been so much simpler= . > All the partitions are aligned to 1 MB, and I can manually set the size o= f > the partition to work around different physical sector counts. All the > partitions are labelled using the physical location of the disk (original= ly > just row/column naming like a spreadsheet, but now I'm adding enclosure > name as well as we expand to multiple enclosures per system). It's so mu= ch > simpler now, ESPECIALLY when I have to get someone to do something > remotely. :) > > =E2=80=8BEveryone has their own way to manage things. I just haven't see= n any > better setup than labelling the drives themselves using their physical > location.=E2=80=8B > > -- > Freddie Cash > fjwcash@gmail.com > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Tue Nov 17 15:26:49 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E308EA3059A for ; Tue, 17 Nov 2015 15:26:49 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7196015CC for ; Tue, 17 Nov 2015 15:26:49 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by wmvv187 with SMTP id v187so233189958wmv.1 for ; Tue, 17 Nov 2015 07:26:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=to:from:subject:message-id:date:user-agent:mime-version :content-type; bh=NPx8M80i6m8yoq6RtE2Q0DTnUOrIHMFcXU2gu+ZaeOY=; b=ARjDiCDCLaK3Ji9W2Zg0AKG2JDCtk5GL1fQZDJ0YAVo5+0RiIhpnYA94W4lFfKCgeQ ShEh9Lc9Yqv9hoX89b0aa/BkP8H1shJz9mwTD8GYGv+ynLwj85TBk/jRBD/b/1o5Oj1s Oag84QB2bGRK5kWMYc5V43OGIDSLQIThfveYcempERzTLLXU+YGA9owRW3yNRWga0wRJ 7gSSIpY3drR9UMtAbQJfc0jlBjZhhRvgpoEi0dYmHiqCmVO0BvS2i/KoWRW0X8+DDMvd 9o3YEv0WOOqSXMXZBxqRr4bQVCpN6bVxHqvwKR7+GmGKP409U5XA1baTG5MANhQndjml ilXQ== X-Received: by 10.28.52.213 with SMTP id b204mr3601136wma.32.1447774008001; Tue, 17 Nov 2015 07:26:48 -0800 (PST) Received: from Johans-MacBook-Air.local (92-70-102-130.glasvezel.netexpo.nl. [92.70.102.130]) by smtp.googlemail.com with ESMTPSA id lx4sm32137897wjb.5.2015.11.17.07.26.47 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 17 Nov 2015 07:26:47 -0800 (PST) To: freebsd-stable@freebsd.org From: Johan Hendriks Subject: LACP with 3 interfaces. X-Enigmail-Draft-Status: N1110 Message-ID: <564B4736.3000100@gmail.com> Date: Tue, 17 Nov 2015 16:26:46 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 15:26:50 -0000 Hello all We have a NFS server witch has three network ports. We have bonded these interfaces as a lagg interface, but when we use the server it looks like only two interfaces are used. This is our rc.conf file ifconfig_igb0="up" ifconfig_igb1="up" ifconfig_igb2="up" cloned_interfaces="lagg0" ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 192.168.100.222 netmask 255.255.255.0" ifconfig tell us the following. lagg0: flags=8843 metric 0 mtu 1500 options=403bb ether a0:36:9f:7d:fc:2f inet 192.168.100.222 netmask 0xffffff00 broadcast 192.168.100.255 nd6 options=29 media: Ethernet autoselect status: active laggproto lacp lagghash l2,l3,l4 laggport: igb1 flags=1c laggport: igb2 flags=1c laggport: igb3 flags=1c So all looks fine But with 4 machines putting a 1GB file on the server one is always using full wirespeed, the rest is around 30 / 40 MB. It never uses all three intefaces but two at max. So we are topped at 200MB/s where we were expecting around 300MB/s #systat -if shows this also. It shows two interfaces at work where igb1 is sitting and doing nothing? /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 Load Average | Interface Traffic Peak Total lagg1 in 0.253 KB/s 0.683 KB/s 459.781 KB out 0.000 KB/s 0.000 KB/s 0.000 KB lagg0 in 0.289 KB/s 215.439 MB/s 4.113 GB out 0.091 KB/s 114.269 MB/s 2.061 GB lo0 in 0.000 KB/s 0.068 KB/s 0.770 KB out 0.000 KB/s 0.068 KB/s 0.770 KB igb2 in 0.011 KB/s *98.401 MB/s* 1.039 GB out 0.022 KB/s 1.474 MB/s 27.311 MB igb1 in 0.143 KB/s *0.466 KB/s* 192.422 KB out 0.022 KB/s 1.959 MB/s 27.066 MB igb0 in 0.135 KB/s *117.340 MB/s * 3.074 GB out 0.114 KB/s 112.679 MB/s 2.007 GB Is there something we can do to make sure lagg0 uses all the interfaces. regards Johan From owner-freebsd-stable@freebsd.org Tue Nov 17 15:55:04 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41CC6A30BE8 for ; Tue, 17 Nov 2015 15:55:04 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0659B1367 for ; Tue, 17 Nov 2015 15:55:04 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by oige206 with SMTP id e206so7252769oig.2 for ; Tue, 17 Nov 2015 07:55:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=mJ68nrBUKNfosztn0AcUx0FAfJzqe1qJydYTah61+Sk=; b=OvSzacWfxUnY0LbwDpTfxbo2HZ0ceq643Doz++CWPEXSaV9CXI64Bj+C1BSXPhhTSm MhWfcC4absXPVlW+X6Mri4QkzCzwDJwvB+xzDsnqR9Y/VnNgMRX23tqLq7ZlnQQiE099 dCOF0jK5VTng6kFAIfCLqPe3H1/I2C90nB02hHX07Wh6gppcGacZHZD2Vm/mlWOhVYdi /g2643kWE5KPr4u61wIOIjJNFspwRcGQgf6jiE3AzJV493NYnJnuaGs+fIAXZrPWhLyg 99lSnNcV8ERhEWifaHZ9tbfVXEqXtJ5MpEnuUK6Vvi5okOEmOfcPUinqLxVHlkp8Ztsk SNNQ== MIME-Version: 1.0 X-Received: by 10.202.171.139 with SMTP id u133mr24510968oie.107.1447775703318; Tue, 17 Nov 2015 07:55:03 -0800 (PST) Sender: asomers@gmail.com Received: by 10.202.0.7 with HTTP; Tue, 17 Nov 2015 07:55:03 -0800 (PST) In-Reply-To: <564B4736.3000100@gmail.com> References: <564B4736.3000100@gmail.com> Date: Tue, 17 Nov 2015 08:55:03 -0700 X-Google-Sender-Auth: JLcyW-f3Mbg2RmKpsbbD9-GVO24 Message-ID: Subject: Re: LACP with 3 interfaces. From: Alan Somers To: Johan Hendriks Cc: FreeBSD Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 15:55:04 -0000 On Tue, Nov 17, 2015 at 8:26 AM, Johan Hendriks wrote: > Is there something we can do to make sure lagg0 uses all the interfaces. Nope. LACP doesn't actively load balance its interfaces. Each flow gets assigned to a single interface based on a hash of the source and destination MACs, IP addresses, and TCP/UDP ports. With many clients, all interfaces will probably be used, but with few clients, there's a lot of luck involved. If you want more bandwidth, you can try fiddling with IP addresses and port numbers to influence the hash function, but even if you get it to distribute the way you want, all your work may be undone by a reboot. The best option is to buy a 10Gbps NIC for the server. They aren't too expensive, anymore, though the switches are still pricey. A cheaper option, if you'll only ever have 4 clients, is to discard the lagg and assign a separate IP address to each igb port, then manually distribute those addresses amongst your clients. If you do this, you unfortunately won't gain the reliability features of LACP. -Alan From owner-freebsd-stable@freebsd.org Tue Nov 17 15:55:41 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07A70A30C88 for ; Tue, 17 Nov 2015 15:55:41 +0000 (UTC) (envelope-from lists@opsec.eu) Received: from home.opsec.eu (home.opsec.eu [IPv6:2001:14f8:200::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B6F3814E5 for ; Tue, 17 Nov 2015 15:55:40 +0000 (UTC) (envelope-from lists@opsec.eu) Received: from pi by home.opsec.eu with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1Zyibc-000Fnz-7K; Tue, 17 Nov 2015 16:55:40 +0100 Date: Tue, 17 Nov 2015 16:55:40 +0100 From: Kurt Jaeger To: Johan Hendriks Cc: freebsd-stable@freebsd.org Subject: Re: LACP with 3 interfaces. Message-ID: <20151117155540.GG35480@home.opsec.eu> References: <564B4736.3000100@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <564B4736.3000100@gmail.com> X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 15:55:41 -0000 Hi! > We have a NFS server witch has three network ports. > > We have bonded these interfaces as a lagg interface, but when we use the > server it looks like only two interfaces are used. > > This is our rc.conf file > > ifconfig_igb0="up" > ifconfig_igb1="up" > ifconfig_igb2="up" > cloned_interfaces="lagg0" > ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 > 192.168.100.222 netmask 255.255.255.0" This says you are lagg'in igb0 to igb2. > ifconfig tell us the following. [...] > laggport: igb1 flags=1c > laggport: igb2 flags=1c > laggport: igb3 flags=1c This says it's lagg'in igb1 to igb3 ? Why the difference ? -- pi@opsec.eu +49 171 3101372 5 years to go ! From owner-freebsd-stable@freebsd.org Tue Nov 17 16:07:49 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1FCDA31067; Tue, 17 Nov 2015 16:07:48 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-oi0-x232.google.com (mail-oi0-x232.google.com [IPv6:2607:f8b0:4003:c06::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B23F41BE1; Tue, 17 Nov 2015 16:07:48 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by oige206 with SMTP id e206so7523395oig.2; Tue, 17 Nov 2015 08:07:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=jGimzqf72d8yTZww+w/pVpMX50a7lefVSCwrOX4Vv8Y=; b=WsU42+dyVzDatxadD1Cin2n+8iwHV75oAlezbagLLloRNqvjegSxiFzw07EbynV+Cg 3/5wBBgz8QeMcYqZ8Bzsdjg9RF2UYXxhuxO31+iVvxHYlza/6QjFYtWGp7BNoUqUNpDW PjgAFB0n5v6a3S4SLHsLB/u+2e6H4LSo9IybKfRQoufHmOvF7erv8Z2ZhdXzgbEppYfj CUAvtaj2veZ9TZGGn4llNKsZw4fKYN5jHXJVmNMiBxQnSD/fEGbQnr0b9Q6YwjukCP6E jTOuuYXL83Be/d3Nb9riq8upOfYpSSzwt4QNCTiiK2KDELpNUrhKWWG6S4nijuHuTKkU Osnw== MIME-Version: 1.0 X-Received: by 10.202.207.12 with SMTP id f12mr24682300oig.101.1447776467368; Tue, 17 Nov 2015 08:07:47 -0800 (PST) Received: by 10.76.29.74 with HTTP; Tue, 17 Nov 2015 08:07:47 -0800 (PST) In-Reply-To: <878D84A9-4553-413B-ACA4-5ABF499F28C1@punkt.de> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> <878D84A9-4553-413B-ACA4-5ABF499F28C1@punkt.de> Date: Tue, 17 Nov 2015 08:07:47 -0800 Message-ID: Subject: Re: ZFS on labelled partitions (was: Re: LSI SAS2008 mps driver preferred firmware version) From: Freddie Cash To: "Patrick M. Hausen" Cc: freebsd-stable , freebsd-scsi@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 16:07:49 -0000 On Tue, Nov 17, 2015 at 12:08 AM, Patrick M. Hausen wrote= : > Hi, all, > > > Am 16.11.2015 um 22:19 schrieb Freddie Cash : > > > > =E2=80=8BYou label the disks as they are added to the system the first = time. > That > > way, you always know where each disk is located, and you only deal with > the > > labels. > > we do the same for obvious reasons. But I always wonder about the possibl= e > downsides, because ZFS documentation explicitly states: > > ZFS operates on raw devices, so it is possible to create a storag= e > pool comprised of logical > volumes, either software or hardware. This configuration is not > recommended, as ZFS works > best when it uses raw physical devices. Using logical volumes > might sacrifice performance, > reliability, or both, and should be avoided. > > (from http://docs.oracle.com/cd/E19253-01/819-5461/gbcik/index.html) > > Can anyone shed some lght on why not using raw devices might sacrifice > performance or reliability? Or is this just outdated folklore? > =E2=80=8BOn Solaris, using raw devices allows ZFS to enable the caches on t= he disks themselves, while using any kind of partitioning on the disk forces the caches to be disabled. This is not an issue on FreeBSD due to the way GEOM works. Caches on disks are enabled regardless of how the disk is accessed (raw, dd-partitioned, MBR-partitioned, GPT-partitioned, gnop, geli, whatever). This is a common misconception and FAQ with ZFS on FreeBSD and one reason to not take any Sun/Oracle documentation at face value, as it doesn't always apply to FreeBSD. There were several posts from pjd@ about this back in the 7.x days when ZFS was first imported to FreeBSD. --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-stable@freebsd.org Tue Nov 17 16:15:09 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DA015A31303 for ; Tue, 17 Nov 2015 16:15:08 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com [IPv6:2a00:1450:400c:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 739C713B3 for ; Tue, 17 Nov 2015 16:15:08 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wmec201 with SMTP id c201so235862532wme.0 for ; Tue, 17 Nov 2015 08:15:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-type:content-transfer-encoding; bh=xbe48kIVNPtiS8nd6RnMIG7Mql7x8fD/+3OOy8d5V28=; b=ApBXJE/SPokiGcnLqc/qqi/0yzdT+rvXcR0lHTcG48X7iH4qSRZyuowJ9JGXA/s3kw 1IFEOFNOnL2i5hAPxXu4R1m2KoKxvyeIxz3lGXQBG0RLpFTy2mScDaq76zub/vNZV4mP pVuHM3bAP84iWFpFfTXaPvPuVwGK8RYFhMllF0hUOgKRPqzB7DWsYtf0WPMvnGPSc4PO /WbGBvwJOXsM2HkwUBFnPdvzHK3KNyg1sAKoPxwQGhb0OBz8HC+Rd+is0kvyMg27ohf3 qA9jSQIiiCIuaSgDpZm55uOXjiH23IEW8U2MyffwfPH6g/QauJTDyRafgI3t33jjUZv1 475w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=xbe48kIVNPtiS8nd6RnMIG7Mql7x8fD/+3OOy8d5V28=; b=f8B3Q22Aiy31uhnaEsW9fDCRyA0EPUuNstdSw2JKocWQ5vXON/6eSVAraL9Pdc0TqZ 2y3rqJq1EG6UsDfn5o2YlnCZY4/swxn3L0o/84niE3qPYRf25hfdY3KC4AdTOrmCGvuH 2q36NMgaCN7kXDu8WRcrYb3u03QX2N1nwtNAN/a5ADEM2CmvsPL9iOtPWaCtchZZcE7E XPVKSLYbeRUdZWm9J5oaizRK9fBGl2wrmihBe1dafbD/M2D2btjyt2xsg4/Irho5xRR2 /aXqMKwFVGM4UNkyATrL7t68BtoStzzB66JHWaqpfUqIjC3wpEqLSqFxDLyYnzldzAPr ZE6A== X-Gm-Message-State: ALoCoQnDykCkHECpLdwnbFxkeeF49pNBTagGQ87hQ32pXcM/Sbm4JHNhKrm/Pv5T0iqnbIR4sRfp X-Received: by 10.28.64.11 with SMTP id n11mr3496562wma.37.1447776906105; Tue, 17 Nov 2015 08:15:06 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id m11sm24658919wma.5.2015.11.17.08.15.04 for (version=TLSv1/SSLv3 cipher=OTHER); Tue, 17 Nov 2015 08:15:04 -0800 (PST) Subject: Re: LACP with 3 interfaces. To: freebsd-stable@freebsd.org References: <564B4736.3000100@gmail.com> From: Steven Hartland Message-ID: <564B528F.1070008@multiplay.co.uk> Date: Tue, 17 Nov 2015 16:15:11 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <564B4736.3000100@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 16:15:09 -0000 On 17/11/2015 15:26, Johan Hendriks wrote: > Hello all > > We have a NFS server witch has three network ports. > > We have bonded these interfaces as a lagg interface, but when we use the > server it looks like only two interfaces are used. > > This is our rc.conf file > > ifconfig_igb0="up" > ifconfig_igb1="up" > ifconfig_igb2="up" > cloned_interfaces="lagg0" > ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 > 192.168.100.222 netmask 255.255.255.0" > > ifconfig tell us the following. > > lagg0: flags=8843 metric 0 mtu 1500 > > options=403bb > ether a0:36:9f:7d:fc:2f > inet 192.168.100.222 netmask 0xffffff00 broadcast 192.168.100.255 > nd6 options=29 > media: Ethernet autoselect > status: active > laggproto lacp lagghash l2,l3,l4 > laggport: igb1 flags=1c > laggport: igb2 flags=1c > laggport: igb3 flags=1c > This shows that your server is using l2,l3,l4 hashing for lacp but what options have you configured on the switch? From owner-freebsd-stable@freebsd.org Tue Nov 17 16:16:26 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8039A3136C for ; Tue, 17 Nov 2015 16:16:26 +0000 (UTC) (envelope-from lists@opsec.eu) Received: from home.opsec.eu (home.opsec.eu [IPv6:2001:14f8:200::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 841321567; Tue, 17 Nov 2015 16:16:26 +0000 (UTC) (envelope-from lists@opsec.eu) Received: from pi by home.opsec.eu with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1Zyivj-000Jlt-GF; Tue, 17 Nov 2015 17:16:27 +0100 Date: Tue, 17 Nov 2015 17:16:27 +0100 From: Kurt Jaeger To: Alan Somers Cc: Johan Hendriks , FreeBSD Subject: Re: LACP with 3 interfaces. Message-ID: <20151117161627.GH35480@home.opsec.eu> References: <564B4736.3000100@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 16:16:26 -0000 Hi! > > Is there something we can do to make sure lagg0 uses all the interfaces. > > Nope. LACP doesn't actively load balance its interfaces. On FreeBSD 11 man lagg(4) says: The driver currently supports the aggregation protocols failover (the default), lacp, loadbalance, roundrobin, broadcast, and none. with roundrobin Distributes outgoing traffic using a round-robin scheduler through all active ports and accepts incoming traffic from any active port. If the three ports are needed for sending, shouldn't this work ? -- pi@opsec.eu +49 171 3101372 5 years to go ! From owner-freebsd-stable@freebsd.org Tue Nov 17 16:20:15 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 73F91A31487 for ; Tue, 17 Nov 2015 16:20:15 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3BA5B1A0D for ; Tue, 17 Nov 2015 16:20:15 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by obbnk6 with SMTP id nk6so10452022obb.2 for ; Tue, 17 Nov 2015 08:20:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=B0RLH68yKp6Omz2zo8VG14lzpwFdBme4ulFdDBPZFdA=; b=LP85GbiJAUr4k3P6rqZ1yHdwih7ZRaL/ednSMrMfN7YgCDsQdf+I6IJdN9el8JN/8m CGnVwXwIsMtdS337EdaPUv60pv4eGtvueZHfAd37PKdjd1Dtn0Nu62OkieP2ZAzmxw7A N7bWghFF3i1O6CW2LBsRRAlPopDeSyAIB4s7Ot4hQqzvmf8wwd02WiqFbT+M01IZ1EBd TEJfUaoQTa2MBU/tgQPhg+GGZn2aqO1yvMgzWYAFhwpTvV/g3DavxXAEyobl08YGBd1H bI7PRgLFfeefoMal3a7+ebue5BXI093UZPUshx6LtgH0YgSGdQQsVbYpgHcfYeP3cGcE mabA== MIME-Version: 1.0 X-Received: by 10.182.165.131 with SMTP id yy3mr28380518obb.49.1447777214649; Tue, 17 Nov 2015 08:20:14 -0800 (PST) Sender: asomers@gmail.com Received: by 10.202.0.7 with HTTP; Tue, 17 Nov 2015 08:20:14 -0800 (PST) In-Reply-To: <20151117161627.GH35480@home.opsec.eu> References: <564B4736.3000100@gmail.com> <20151117161627.GH35480@home.opsec.eu> Date: Tue, 17 Nov 2015 09:20:14 -0700 X-Google-Sender-Auth: G1M-lrlEGciaRfC-IPX1tu8-HsQ Message-ID: Subject: Re: LACP with 3 interfaces. From: Alan Somers To: Kurt Jaeger Cc: Johan Hendriks , FreeBSD Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 16:20:15 -0000 On Tue, Nov 17, 2015 at 9:16 AM, Kurt Jaeger wrote: > Hi! > >> > Is there something we can do to make sure lagg0 uses all the interfaces. >> >> Nope. LACP doesn't actively load balance its interfaces. > > On FreeBSD 11 > > man lagg(4) > > says: > > The driver currently supports the aggregation protocols failover (the > default), lacp, loadbalance, roundrobin, broadcast, and none. > > with > > roundrobin Distributes outgoing traffic using a round-robin scheduler > through all active ports and accepts incoming traffic from > any active port. > > If the three ports are needed for sending, shouldn't this work ? > Be careful with roundrobin or loadbalance. Both of them will distribute outbound traffic across all ports, but at the expense of causing your NFS clients to receive out-of-order TCP packets. This increases their CPU load. You may find that performance with roundrobin is actually worse than with LACP because of the out-of-order issue. Also, neither roundrobin nor loadbalance will help distribute inbound traffic. -Alan From owner-freebsd-stable@freebsd.org Tue Nov 17 19:43:48 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1D06A313BA for ; Tue, 17 Nov 2015 19:43:48 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-wm0-x230.google.com (mail-wm0-x230.google.com [IPv6:2a00:1450:400c:c09::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BDF11E36 for ; Tue, 17 Nov 2015 19:43:48 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by wmec201 with SMTP id c201so42236210wme.1 for ; Tue, 17 Nov 2015 11:43:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:to:references:from:cc:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=Go5ExD0dqCV/RzhLgjs7GI/5I69yhlNzvPSOqjV9xxg=; b=cKVfbJduyAHphcQF7uqtFrnEjYpSYq9JdpyFg/eol4WPLLNFbjpmA/y/Wsu/gQEGnw ZCE8UW0m6ULT81Y01/Qyxip1qnQ3bwM9ffloVlSdYh+OZLiCz1D2jmhmhmcmkPlu7j/6 LiQkA/XzN5dXQWUYXzi3Q1eIp/kALUYa7dK9uiAUHqBIrxvpI2DcZ+ibseFYhL3ZxtS8 kbrMceoUb9u8dFSzvpygMdmihyIaJN/7fcOQ66CJRkWSj1COdAG862ZlW2GqkJBvatoJ lksHi/kq7XgC0NAggx2/t6wVSqSCwtt80X7aeXGo92Sl7Y8OvtyEWfvTMU+S2kxlqE84 wuVw== X-Received: by 10.28.104.197 with SMTP id d188mr4935272wmc.55.1447789426521; Tue, 17 Nov 2015 11:43:46 -0800 (PST) Received: from Johans-MacBook-Air.local (92-70-102-130.glasvezel.netexpo.nl. [92.70.102.130]) by smtp.googlemail.com with ESMTPSA id t194sm25467326wmt.11.2015.11.17.11.43.45 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 17 Nov 2015 11:43:45 -0800 (PST) Subject: Re: LACP with 3 interfaces. To: Steven Hartland References: <564B4736.3000100@gmail.com> <564B528F.1070008@multiplay.co.uk> From: Johan Hendriks X-Enigmail-Draft-Status: N1110 Cc: freebsd-stable@freebsd.org Message-ID: <564B8371.6070001@gmail.com> Date: Tue, 17 Nov 2015 20:43:45 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <564B528F.1070008@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 19:43:48 -0000 Op 17/11/15 om 17:15 schreef Steven Hartland: > > > On 17/11/2015 15:26, Johan Hendriks wrote: >> Hello all >> >> We have a NFS server witch has three network ports. >> >> We have bonded these interfaces as a lagg interface, but when we use the >> server it looks like only two interfaces are used. >> >> This is our rc.conf file >> >> ifconfig_igb0="up" >> ifconfig_igb1="up" >> ifconfig_igb2="up" >> cloned_interfaces="lagg0" >> ifconfig_lagg0="laggproto lacp laggport igb0 laggport igb1 laggport igb2 >> 192.168.100.222 netmask 255.255.255.0" >> >> ifconfig tell us the following. >> >> lagg0: flags=8843 metric 0 >> mtu 1500 >> >> options=403bb >> ether a0:36:9f:7d:fc:2f >> inet 192.168.100.222 netmask 0xffffff00 broadcast 192.168.100.255 >> nd6 options=29 >> media: Ethernet autoselect >> status: active >> laggproto lacp lagghash l2,l3,l4 >> laggport: igb1 flags=1c >> laggport: igb2 flags=1c >> laggport: igb3 flags=1c >> > This shows that your server is using l2,l3,l4 hashing for lacp but > what options have you configured on the switch? > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" The switch is an old 3com switch for testing, and it is not very configureable. So It just says LACP and nothing else, I can choose between static and dynamic. The interfaces are copy paste, but the machine has a lot more interfaces, and first I started to switch interfaces. So the rc.conf is before I switched to other interfaces. That is why the interfaces from ifconfig and the rc.conf file does not match. Thank you all for your time. I now know that it will use the inteface and that the hashing is the reason it sticks to only two interfaces. If it uses the mac adres, or ip adres to hash it can explain the choice it makes. Thank you all for your answers and time. regards Johan From owner-freebsd-stable@freebsd.org Tue Nov 17 22:08:35 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 039DFA31768; Tue, 17 Nov 2015 22:08:35 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BC5341313; Tue, 17 Nov 2015 22:08:34 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by ioir85 with SMTP id r85so34205812ioi.1; Tue, 17 Nov 2015 14:08:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=c6bzUkfOgqEN3rAxugbiP+LAQGCwAaGiN+dfbLw7TV4=; b=U3KyJnatjOltFVD5CjmNYqasV2uRZCO5Bix1uDHgfNMHbtD+3dQeKrSqK3gQ3vEMxn d+D9+QSEytQthLriWZ9f9yq2usuDk8pXKNnRtmw8fmja1s6oExSnnbj9DENKKsRMMfdz 46moPD8ZpYOGIqd6y++HPsYfC8+M9fgsyu0diLW3eRBtmTYhiKCiufoZ9tW6pJv9ePYz ph93c+WTufmmkiAW+Oogk2zYtvpdFvFztX7CqsOeiY2c4P2aB5QIn/xhr9Ea1DWO7SIR bdtMFnK8ZXg/hCnPLdrNEJvQEqkFZNi6LD61iEaTZrxt8Q8iPnivYEOaxzVpMqjpgJC3 a6fQ== MIME-Version: 1.0 X-Received: by 10.107.155.149 with SMTP id d143mr41223533ioe.145.1447798114108; Tue, 17 Nov 2015 14:08:34 -0800 (PST) Received: by 10.36.159.67 with HTTP; Tue, 17 Nov 2015 14:08:34 -0800 (PST) Date: Tue, 17 Nov 2015 18:08:34 -0400 Message-ID: Subject: Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target From: Christopher Forgeron To: FreeBSD stable , FreeBSD Filesystems Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 22:08:35 -0000 I just submitted this as a bug: ( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204641 ) ..but I thought I should bring it to the list's attention for more exposure - If that's a no-no, let me know, as I have a few others that are related to this that I'd like to discuss. - - - - Consider this scenario: Virtual FreeBSD Machine, with a zpool created out of iSCSI disks. Physical FreeBSD Machine, with a zpool holding a sparse file that is the target for the iSCSI disk. This setup works in an environment with all 10.1 machines, doesn't with all 10.2 machines. - The 10.2 Machines are 10.2-p7 RELEASE, updated via freebsd-update, no custom. - The 10.1 Machine are 10.1-p24 RELEASE, updated via freebsd-update, no custom. - iSCSI is all CAM iSCSI, not the old istgt platform. - The iSCSI Target is a sparse file, stored on a zpool (not a vdev Target) The target machine is the same physical machine, with the same zpools - I either boot 10.1 or 10.2 for testing, and use the same zpool/disks to ensure nothing is changing. If I have a 10.2 iSCSI Initiator (client) connected to a 10.2 iSCSI Target, TRIM doesn't work (shows as NONE below). If I have a 10.2 iSCSI Initiator (client) connected to a 10.1 iSCSI Target, TRIM does work. (There is another bug with that last scenario as well, but I will open it separately) ...for clarity, a 10.1 iSCSI Initiator connected to a 10.1 iSCSI Target also works perfectly. I have ~20 of these in the field. On the 10.1 / 10.2 Targets, the ctl.conf file is identical. Zpools are identical, because they are shared between reboots of the same iSCSI target machine. On the 10.2 initiator machine, connected to a 10.2 Target machine: # sysctl -a | grep cam.da kern.cam.da.2.minimum_cmd_size: 6 kern.cam.da.2.delete_max: 131072 kern.cam.da.2.delete_method: NONE kern.cam.da.1.error_inject: 0 kern.cam.da.1.sort_io_queue: 0 kern.cam.da.1.minimum_cmd_size: 6 kern.cam.da.1.delete_max: 131072 kern.cam.da.1.delete_method: NONE kern.cam.da.0.error_inject: 0 kern.cam.da.0.sort_io_queue: -1 kern.cam.da.0.minimum_cmd_size: 6 kern.cam.da.0.delete_max: 131072 kern.cam.da.0.delete_method: NONE Note the delete_method is NONE # sysctl -a | grep trim vfs.zfs.trim.max_interval: 1 vfs.zfs.trim.timeout: 30 vfs.zfs.trim.txg_delay: 32 vfs.zfs.trim.enabled: 1 vfs.zfs.vdev.trim_max_pending: 10000 vfs.zfs.vdev.trim_max_active: 64 vfs.zfs.vdev.trim_min_active: 1 vfs.zfs.vdev.trim_on_init: 1 kstat.zfs.misc.zio_trim.failed: 0 kstat.zfs.misc.zio_trim.unsupported: 181 kstat.zfs.misc.zio_trim.success: 0 kstat.zfs.misc.zio_trim.bytes: 0 Note no trimmed bytes. On the target machine, 10.1 and 10.2 share the same config file: /etc/ctl.conf portal-group pg0 { discovery-auth-group no-authentication listen 0.0.0.0 listen [::] } lun 0 { path /pool92/iscsi/iscsi.zvol blocksize 4K size 5T option unmap "on" option scsiname "pool92" option vendor "pool92" option insecure_tpc "on" } } target iqn.iscsi1.zvol { auth-group no-authentication portal-group pg0 lun 0 { path /pool92_1/iscsi/iscsi.zvol blocksize 4K size 5T option unmap "on" option scsiname "pool92_1" option vendor "pool92_1" option insecure_tpc "on" } } When I boot a 10.1 Target server, the 10.2 initiator connects, and we do see proper UNMAP ability: kern.cam.da.2.minimum_cmd_size: 6 kern.cam.da.2.delete_max: 5497558138880 kern.cam.da.2.delete_method: UNMAP kern.cam.da.1.error_inject: 0 kern.cam.da.1.sort_io_queue: 0 kern.cam.da.1.minimum_cmd_size: 6 kern.cam.da.1.delete_max: 5497558138880 kern.cam.da.1.delete_method: UNMAP kern.cam.da.0.error_inject: 0 kern.cam.da.0.sort_io_queue: -1 kern.cam.da.0.minimum_cmd_size: 6 kern.cam.da.0.delete_max: 131072 kern.cam.da.0.delete_method: NONE Please let me know what you'd like to know next. Thanks. From owner-freebsd-stable@freebsd.org Tue Nov 17 22:16:50 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E86B8A31B25; Tue, 17 Nov 2015 22:16:50 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: from mail-ig0-x232.google.com (mail-ig0-x232.google.com [IPv6:2607:f8b0:4001:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4B5A1AC1; Tue, 17 Nov 2015 22:16:50 +0000 (UTC) (envelope-from csforgeron@gmail.com) Received: by igl9 with SMTP id 9so89456405igl.0; Tue, 17 Nov 2015 14:16:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZqYhpOA4p8OqAB18n4A7qXMvcv4PiZYCMTTjb5p5y9c=; b=B67t4ECi11+EzQYrQu+V2v3h2pJa3pfdnJeRuOHs/C1c7zAhEoicBJH4eNGnh13dyu oYfotPEIeHjzaz0M1xIDNvMf+4HuflAsm0Raxdfq/yIZpvxOgFXMDcEkRZRDnZ/tGh74 qCGm7srNNQJAHpX8jirrBPERlD6fNFWtDRQ5oxGq089JYlvxh591YbVHQ5H2rBbUjDb/ kYzenvDy9l50jtuwCPlc7LWI+SIpOz3/MHIfPJ/idBSmYXHc2un22I6CHjh5cNiCvItz ExH6cwLOzKfZrJRuXvMGUB1QEEMcTGbK3Eq1tvZ+nE7hK8BaOs7semnhWqOku2Ii4FOF FQ2w== MIME-Version: 1.0 X-Received: by 10.50.56.114 with SMTP id z18mr4474564igp.62.1447798610172; Tue, 17 Nov 2015 14:16:50 -0800 (PST) Received: by 10.36.159.67 with HTTP; Tue, 17 Nov 2015 14:16:50 -0800 (PST) In-Reply-To: <21685.40094.453028.585630@hergotha.csail.mit.edu> References: <21685.40094.453028.585630@hergotha.csail.mit.edu> Date: Tue, 17 Nov 2015 18:16:50 -0400 Message-ID: Subject: Re: Some filesystem performance numbers From: Christopher Forgeron To: Garrett Wollman Cc: FreeBSD Filesystems , FreeBSD stable Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 22:16:51 -0000 Sounds interesting. I'd love to see your results when you're ready to share, or even the 'work in progress' if you want to share privately . On Tue, Jan 13, 2015 at 6:30 PM, Garrett Wollman wrote: > I recently bought a copy of the SPECsfs2014 benchmark, and I've been > using it to test out our NFS server platform. One scenario of > interest to me is identifying where the limits are in terms of the > local CAM/storage/filesystem implementation versus bottlenecks unique > to the NFS server, and to that end I've been running the benchmark > suite directly on the server's local disk. (This is of course also > the way you'd benchmark for shared-nothing container-based > virtualization.) > > I have found a few interesting results on my test platform: > > 1) I can quantify the cost of using SHA256 vs. fletcher4 as the ZFS > checksum algorithm. On the VDA workload (essentially a simulated > video streaming/recording application), my server can do about half as > many "streams" with SHA256 as it can with fletcher4. > > 2) Both L2ARC and separate ZIL have small but measurable performance > impacts. I haven't examined the differences closely. > > 3) LZ4 compression also makes a small performance impact, but as > advertised, less than LZJB for mostly-incompressible data. > > I hope to be able to present the actual benchmark results at some > point, as well as some results for the other three workloads. > > -GAWollman > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Tue Nov 17 23:15:03 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 62872A316BD for ; Tue, 17 Nov 2015 23:15:03 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from mail.ultra-secure.de (mail.ultra-secure.de [88.198.178.88]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B4CD81343 for ; Tue, 17 Nov 2015 23:15:02 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: (qmail 32485 invoked by uid 89); 17 Nov 2015 23:13:02 -0000 Received: from unknown (HELO ?192.168.1.200?) (rainer@ultra-secure.de@217.71.83.52) by mail.ultra-secure.de with ESMTPA; 17 Nov 2015 23:13:02 -0000 From: Rainer Duffner Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: After BIOS-Upgrade, I can't (UEFI-) boot anymore Date: Wed, 18 Nov 2015 00:13:01 +0100 Message-Id: <9A4A45D3-AED0-4312-AA05-47F24BE9D24F@ultra-secure.de> Cc: freebsd-proliant@freebsd.org To: freebsd-stable Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Nov 2015 23:15:03 -0000 Hi, I have a HP DL380 G9, that I boot from an internal SmartArray Controller = provided RAID1 into FreeBSD 10.1 amd64. I have upgraded the BIOS to the 2015-10 release and on reboot, I now get = a message that /boot/loader.efi=20 can=E2=80=99t be found. I can legacy boot it into mfsbsd and the file is there. How can I fix this? Or how can I debug this and why is this failing in the first place? Rainer From owner-freebsd-stable@freebsd.org Wed Nov 18 00:28:52 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E937BA32684 for ; Wed, 18 Nov 2015 00:28:51 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-wm0-x231.google.com (mail-wm0-x231.google.com [IPv6:2a00:1450:400c:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7F0711F7C for ; Wed, 18 Nov 2015 00:28:51 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by wmvv187 with SMTP id v187so254166211wmv.1 for ; Tue, 17 Nov 2015 16:28:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay_co_uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:cc:message-id:date:user-agent :mime-version:in-reply-to:content-type:content-transfer-encoding; bh=WrfcV9A5j8hoTxH7Ej4tM+CK8mNQw48+UwsY+frvIIk=; b=AhgLLBViXd11l9UFeIG80QfjfJhrhOaWNlckWz9nR+ZVCu1ddcyO8x7ZmwbZadSSSQ fCmPyeb215zj4wVOpl4TRBdtWdSMbLLuKivT/4wcrnYvnTIoHkjk+mb6lfxoTRWMtW3M 5fIcyM9Tj+5kEOkZS2ZCPv+hZ58YVNE3/F2LGLX/kkvA14VzNuPEm83UCvxZcv0CcxkW ejYlFvqv0UJe6LeqXcs44QFGOPa+A9q03L1FjyvWgvrD4iOQC9W2JAa6jEww988EOHT3 s8QvBXQMJqqFY1xNOlRzrBSLGuMObKk76WzDblaic/apA7nt4Epil1cGsI53mCl4DeZh HVyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:from:cc:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=WrfcV9A5j8hoTxH7Ej4tM+CK8mNQw48+UwsY+frvIIk=; b=TEFQen1hRd4CPD6GG0TJTuyFKTiSBBCmaxDX97W4fxaVV+bTwVawAr7DAzN5GPziWE nF/vw4Ww38YcpNEwlHNrVJ970kNrETIIvcBIsHNy/6ONKZn0QzUwMCLlK8/8J6UNF59M x26osJ2LeoZIwST8fVfEbvhyjZC1i/HwseSMsMIg9rcc8nrgL+Odbd0c5X/dPQ9XzU88 INjy3tEb6pxUmvBMy9FWss9UfBKDmjDtZ+dNEwnvoIO5KCz8L2H3XAjBsOYx+yUaCAeb mcHDMTJjnKuJMWFMI7rt1V5l2LnVNuY44bycpugzw9t3DiuzI74RJeGlUbe4X64SaoVt KIeQ== X-Gm-Message-State: ALoCoQmCvjDYnMOpt5r60y1LWz5FZeEW8ugRz/owJ/RYUqtCIpPQg7Rw9rXjbkCo8eUZIL8FX68C X-Received: by 10.194.189.133 with SMTP id gi5mr46563691wjc.13.1447806528683; Tue, 17 Nov 2015 16:28:48 -0800 (PST) Received: from [10.10.1.58] (liv3d.labs.multiplay.co.uk. [82.69.141.171]) by smtp.gmail.com with ESMTPSA id v192sm26418214wmv.5.2015.11.17.16.28.47 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 17 Nov 2015 16:28:47 -0800 (PST) Subject: Re: Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target To: freebsd-stable@freebsd.org References: From: Steven Hartland Cc: Alexander Motin Message-ID: <564BC646.5070708@multiplay.co.uk> Date: Wed, 18 Nov 2015 00:28:54 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 00:28:52 -0000 On 17/11/2015 22:08, Christopher Forgeron wrote: > I just submitted this as a bug: > > ( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204641 ) > > ..but I thought I should bring it to the list's attention for more exposure > - If that's a no-no, let me know, as I have a few others that are related > to this that I'd like to discuss. > > - - - - > > > Consider this scenario: > > Virtual FreeBSD Machine, with a zpool created out of iSCSI disks. > Physical FreeBSD Machine, with a zpool holding a sparse file that is the > target for the iSCSI disk. > > This setup works in an environment with all 10.1 machines, doesn't with all > 10.2 machines. > > - The 10.2 Machines are 10.2-p7 RELEASE, updated via freebsd-update, no > custom. > - The 10.1 Machine are 10.1-p24 RELEASE, updated via freebsd-update, no > custom. > - iSCSI is all CAM iSCSI, not the old istgt platform. > - The iSCSI Target is a sparse file, stored on a zpool (not a vdev Target) > > The target machine is the same physical machine, with the same zpools - I > either boot 10.1 or 10.2 for testing, and use the same zpool/disks > > to ensure nothing is changing. > > If I have a 10.2 iSCSI Initiator (client) connected to a 10.2 iSCSI Target, > TRIM doesn't work (shows as NONE below). > If I have a 10.2 iSCSI Initiator (client) connected to a 10.1 iSCSI Target, > TRIM does work. > > (There is another bug with that last scenario as well, but I will open it > separately) > > ...for clarity, a 10.1 iSCSI Initiator connected to a 10.1 iSCSI Target > also works perfectly. I have ~20 of these in the field. > > On the 10.1 / 10.2 Targets, the ctl.conf file is identical. Zpools are > identical, because they are shared between reboots of the same iSCSI > > target machine. > > > > On the 10.2 initiator machine, connected to a 10.2 Target machine: > > # sysctl -a | grep cam.da > > kern.cam.da.2.minimum_cmd_size: 6 > kern.cam.da.2.delete_max: 131072 > kern.cam.da.2.delete_method: NONE > kern.cam.da.1.error_inject: 0 > kern.cam.da.1.sort_io_queue: 0 > kern.cam.da.1.minimum_cmd_size: 6 > kern.cam.da.1.delete_max: 131072 > kern.cam.da.1.delete_method: NONE > kern.cam.da.0.error_inject: 0 > kern.cam.da.0.sort_io_queue: -1 > kern.cam.da.0.minimum_cmd_size: 6 > kern.cam.da.0.delete_max: 131072 > kern.cam.da.0.delete_method: NONE > > Note the delete_method is NONE > > > # sysctl -a | grep trim > vfs.zfs.trim.max_interval: 1 > vfs.zfs.trim.timeout: 30 > vfs.zfs.trim.txg_delay: 32 > vfs.zfs.trim.enabled: 1 > vfs.zfs.vdev.trim_max_pending: 10000 > vfs.zfs.vdev.trim_max_active: 64 > vfs.zfs.vdev.trim_min_active: 1 > vfs.zfs.vdev.trim_on_init: 1 > kstat.zfs.misc.zio_trim.failed: 0 > kstat.zfs.misc.zio_trim.unsupported: 181 > kstat.zfs.misc.zio_trim.success: 0 > kstat.zfs.misc.zio_trim.bytes: 0 > > Note no trimmed bytes. > > > On the target machine, 10.1 and 10.2 share the same config file: > /etc/ctl.conf > > portal-group pg0 { > discovery-auth-group no-authentication > listen 0.0.0.0 > listen [::] > } > > lun 0 { > path /pool92/iscsi/iscsi.zvol > blocksize 4K > size 5T > option unmap "on" > option scsiname "pool92" > option vendor "pool92" > option insecure_tpc "on" > } > } > > > target iqn.iscsi1.zvol { > auth-group no-authentication > portal-group pg0 > > lun 0 { > path /pool92_1/iscsi/iscsi.zvol > blocksize 4K > size 5T > option unmap "on" > option scsiname "pool92_1" > option vendor "pool92_1" > option insecure_tpc "on" > } > } > > > When I boot a 10.1 Target server, the 10.2 initiator connects, and we do > see proper UNMAP ability: > > > kern.cam.da.2.minimum_cmd_size: 6 > kern.cam.da.2.delete_max: 5497558138880 > kern.cam.da.2.delete_method: UNMAP > kern.cam.da.1.error_inject: 0 > kern.cam.da.1.sort_io_queue: 0 > kern.cam.da.1.minimum_cmd_size: 6 > kern.cam.da.1.delete_max: 5497558138880 > kern.cam.da.1.delete_method: UNMAP > kern.cam.da.0.error_inject: 0 > kern.cam.da.0.sort_io_queue: -1 > kern.cam.da.0.minimum_cmd_size: 6 > kern.cam.da.0.delete_max: 131072 > kern.cam.da.0.delete_method: NONE > > > Please let me know what you'd like to know next. > Having a quick flick through the code it looks like umap is now only supported on dev backed and not file backed. I believe the following commit is the cause: https://svnweb.freebsd.org/base?view=revision&revision=279005 This was an MFC of: https://svnweb.freebsd.org/base?view=revision&revision=278672 I'm guessing this was an unintentional side effect mav? Regards Steve From owner-freebsd-stable@freebsd.org Wed Nov 18 08:19:16 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21760A32F35 for ; Wed, 18 Nov 2015 08:19:16 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-lb0-x22f.google.com (mail-lb0-x22f.google.com [IPv6:2a00:1450:4010:c04::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 948491832 for ; Wed, 18 Nov 2015 08:19:15 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by lbblt2 with SMTP id lt2so19970758lbb.3 for ; Wed, 18 Nov 2015 00:19:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=MgzNnsY2Y5MoK4Ut4JkTfXiUmybzDvkhQwXxH6x6+uA=; b=aEwikSq9iDcq37PIY3koQCY3qR+sz7d8SR/Ab4HRpMomn2cyZOILJXQkdUzbXwBktH ecAcCXMfwuQmnGIuT1vFM6SBXQx9mBNUkrVV6iMmyavNNdEf16SzbtmO7DS3YHuv/DC1 OA+xzb5PDkahpoKZpxHOPp3duGrXZ4LB9hbm4EwPdGc9cKgMiRq0oWOGS/YQ6ffDFVg6 reaEMkk63125EhigD00kDzWneMZaem/nwUewNuwv32KD1HpGY3Z5HDokdA4/yiIQSHGs c1V6kKMC/BTRTFPK8PCKTKlzXciLg7qOfGSZeHOH6lBkm1brGga6qndKMbWFOYqJ7et8 KAVg== X-Received: by 10.112.235.65 with SMTP id uk1mr102414lbc.118.1447834753486; Wed, 18 Nov 2015 00:19:13 -0800 (PST) Received: from mavbook.mavhome.dp.ua ([2a01:d0:c0a9:3:c685:8ff:fe11:1aa2]) by smtp.googlemail.com with ESMTPSA id h37sm255318lfi.8.2015.11.18.00.19.11 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 18 Nov 2015 00:19:12 -0800 (PST) Sender: Alexander Motin Message-ID: <564C347E.3030705@FreeBSD.org> Date: Wed, 18 Nov 2015 10:19:10 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Steven Hartland , freebsd-stable@freebsd.org Subject: Re: Bug 204641 - 10.2 UNMAP/TRIM not available on a zfs zpool that uses iSCSI disks, backed on a zpool file target References: <564BC646.5070708@multiplay.co.uk> In-Reply-To: <564BC646.5070708@multiplay.co.uk> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 08:19:16 -0000 On 18.11.2015 02:28, Steven Hartland wrote: > On 17/11/2015 22:08, Christopher Forgeron wrote: >> I just submitted this as a bug: >> >> ( https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=204641 ) >> >> ..but I thought I should bring it to the list's attention for more >> exposure >> - If that's a no-no, let me know, as I have a few others that are related >> to this that I'd like to discuss. > Having a quick flick through the code it looks like umap is now only > supported on dev backed and not file backed. > > I believe the following commit is the cause: > https://svnweb.freebsd.org/base?view=revision&revision=279005 > > This was an MFC of: > https://svnweb.freebsd.org/base?view=revision&revision=278672 > > I'm guessing this was an unintentional side effect mav? As I have replied on the ticket: CTL never supported UNMAP on file-backed LUNs due to lack of respective API for hole punching on FreeBSD. At this time UNMAP works for ZVOLs in both device and file modes and raw devices. -- Alexander Motin From owner-freebsd-stable@freebsd.org Wed Nov 18 10:07:14 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C161A320C5; Wed, 18 Nov 2015 10:07:14 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B9EDA113C; Wed, 18 Nov 2015 10:07:13 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1Zyzdt-0001PA-LE; Wed, 18 Nov 2015 13:07:09 +0300 Date: Wed, 18 Nov 2015 13:07:09 +0300 From: Slawa Olhovchenkov To: Garrett Wollman Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Some filesystem performance numbers Message-ID: <20151118100709.GN48728@zxy.spb.ru> References: <21685.40094.453028.585630@hergotha.csail.mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <21685.40094.453028.585630@hergotha.csail.mit.edu> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 10:07:14 -0000 On Tue, Jan 13, 2015 at 05:30:54PM -0500, Garrett Wollman wrote: > I recently bought a copy of the SPECsfs2014 benchmark, and I've been > using it to test out our NFS server platform. One scenario of > interest to me is identifying where the limits are in terms of the > local CAM/storage/filesystem implementation versus bottlenecks unique > to the NFS server, and to that end I've been running the benchmark > suite directly on the server's local disk. (This is of course also > the way you'd benchmark for shared-nothing container-based > virtualization.) > > I have found a few interesting results on my test platform: > > 1) I can quantify the cost of using SHA256 vs. fletcher4 as the ZFS > checksum algorithm. On the VDA workload (essentially a simulated > video streaming/recording application), my server can do about half as > many "streams" with SHA256 as it can with fletcher4. For VDA recordsize=1M (or more) can give performance impcat in case saturated HDD by IOPS. > 2) Both L2ARC and separate ZIL have small but measurable performance > impacts. I haven't examined the differences closely. This is depend of fractions hot/warm/cold content. > 3) LZ4 compression also makes a small performance impact, but as > advertised, less than LZJB for mostly-incompressible data. > > I hope to be able to present the actual benchmark results at some > point, as well as some results for the other three workloads. > > -GAWollman > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Wed Nov 18 10:25:08 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7152BA325EC; Wed, 18 Nov 2015 10:25:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27EB7113F; Wed, 18 Nov 2015 10:25:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1ZyzvC-0002G2-HL; Wed, 18 Nov 2015 13:25:02 +0300 Date: Wed, 18 Nov 2015 13:25:02 +0300 From: Slawa Olhovchenkov To: Freddie Cash Cc: Royce Williams , freebsd-stable , Borja Marcos , Kai Gallasch , Kevin Oberman , freebsd-scsi@freebsd.org Subject: Re: LSI SAS2008 mps driver preferred firmware version Message-ID: <20151118102502.GO48728@zxy.spb.ru> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 10:25:08 -0000 On Mon, Nov 16, 2015 at 01:19:55PM -0800, Freddie Cash wrote: > On Mon, Nov 16, 2015 at 12:57 PM, Slawa Olhovchenkov wrote: > > > On Mon, Nov 16, 2015 at 11:40:12AM -0800, Freddie Cash wrote: > > > > > On Mon, Nov 16, 2015 at 11:36 AM, Kevin Oberman > > wrote: > > > > As already mentioned, unless you are using zfs, use gpart to label you > > file > > > > systems/disks. Then use the /dev/gpt/LABEL as the mount device in > > fstab. > > > > > > > > > > ​Even if you are using ZFS, labelling the drives with the location of the > > > disk in the system (enclosure, column, row, whatever) makes things so > > much > > > easier to work with when there are disk-related issues. > > > > > > Just create a single partition that covers the whole disk, label it, and > > > use the label to create the vdevs in the pool.​ > > > > Bad idea. > > Re-placed disk in different bay don't relabel automaticly. > > > > ​Did the original disk get labelled automatically? No, you had to do that > when you first started using it. So, why would you expect a > replaced disk Initial labeling is problem too. For new chassis with 36 identical disk (already installed) -- what is simple way to labeling disks? > to get labelled automatically? Consistency keeping is another problem. > Offline the dead/dying disk. > Physically remove the disk. > Insert the new disk. > Partition / label the new disk. > "zfs replace" using the new label to get it into the pool.​ New disk can be inserted in free bay. This is may be done by remote hand. And I can be miss information where disk is placed. > > Other issuse where disk placed in bay some remotely hands in data > > center -- I am relay don't know how disk distributed by bays. > > > > ​You label the disks as they are added to the system the first time. That > way, you always know where each disk is located, and you only deal with the > labels. > > Then, when you need to replace a disk (or ask someone in a remote location > to replace it) it's a simple matter: the label on the disk itself tells > you where the disk is physically located. And it doesn't change if the > controller decides to change the direction it enumerates devices. > > Which is easier to tell someone in a remote location: "Replace disk in bay with blinked led" Author: bapt Date: Sat Sep 5 00:06:01 2015 New Revision: 287473 URL: https://svnweb.freebsd.org/changeset/base/287473 Log: Add a new sesutil(8) utility This is an utility for managing SCSI Enclosure Services (SES) device. For now only one command is supported "locate" which will change the test of the external LED associated to a given disk. Usage if the following: sesutil locate disk [on|off] Disk can be a device name: "da12" or a special keyword: "all". > Replace disk enc0a6 (meaning enclosure 0, column A, row 6)? > or > Replace the disk called da36?​ > ​or > Find the disk with serial number XXXXXXXX? > or > Replace the disk where the light is (hopefully) flashing (but I can't > tell you which enclosure, front or back, or anything else like that)? > > The first one lets you know exactly where the disk is located physically. > > The second one just tells you the name of the device as determined by the > OS, but doesn't tell you anything about where it is located. And it can > change with a kernel update, driver update, or firmware update! > > The third requires you to pull every disk in turn to read the serial number > off the drive itself. Usaly serial number can be read w/o pull disk (for SuperMicro cases this is true, remote hand replaced disk by S/N for me w/o pull every disk). > In order for the second or third option to work, you'd have to write down > the device names and/or serial numbers and stick that onto the drive bay > itself.​ > > > > Best way for identify disk -- uses enclouse services. > > > > ​Only if your enclosure services are actually working (or even enabled). > I've yet to work on a box where that actually works (we custom-build our > storage boxes using OTS hardware). > > Best way, IMO, is to use the physical location of the device as the actual > device name itself. That way, there's never any ambiguity at the physical > layer, the driver layer, the OS layer, or the ZFS pool layer.​ > > > > I have many sites with ZFS on whole disk and some sites with ZFS on > > GPT partition. ZFS on GPT more heavy for administration. > > > > ​It's 1 extra step: partition the drive, supplying the location of the > drive as the label for the partition. > > Everything else works exactly the same. > > I used to do everything with whole drives and no labels. Did that for > about a month, until 2 separate drives on separate controllers died (in a > 24-bay setup) and I couldn't figure out where they were located as a BIOS > upgrade changed which controller loaded first. And then I had to work on a > server that someone else configured with direct-attach bays (24 cables) > that were connected almost at random. All currently used by me servers have some randoms in detecting and reporting controllers and HDDs. No problem for ZFS and/or replacing by remote hands (by S/N). From owner-freebsd-stable@freebsd.org Wed Nov 18 13:52:55 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0BCF6A30C7C; Wed, 18 Nov 2015 13:52:55 +0000 (UTC) (envelope-from KevinO'Connor@merseyfire.gov.uk) Received: from mfrsmail1.merseyfire.gov.uk (mfrsmail1.merseyfire.gov.uk [91.220.52.132]) by mx1.freebsd.org (Postfix) with ESMTP id B598C14EE; Wed, 18 Nov 2015 13:52:54 +0000 (UTC) (envelope-from KevinO'Connor@merseyfire.gov.uk) Received: from MFRSEXCH-TDA.merseyfire.gov.uk (unknown [10.10.0.47]) by Websense Email Security Gateway with ESMTPS id 03649729734D1; Wed, 18 Nov 2015 13:37:08 +0000 (GMT) Received: from MFRSEXCH-HQ.merseyfire.gov.uk ([fe80::7cc7:fd90:39aa:5ea9]) by MFRSEXCH-TDA.merseyfire.gov.uk ([fe80::1c:dd4e:96cc:9837%17]) with mapi id 14.03.0248.002; Wed, 18 Nov 2015 13:37:09 +0000 From: "O'Connor, Kevin" To: Rainer Duffner , freebsd-stable CC: "freebsd-proliant@freebsd.org" Subject: RE: After BIOS-Upgrade, I can't (UEFI-) boot anymore Thread-Topic: After BIOS-Upgrade, I can't (UEFI-) boot anymore Thread-Index: AQHRIY0p2JzIjhj9+E+WHwk8uLU0q56hxe+w Date: Wed, 18 Nov 2015 13:33:16 +0000 Message-ID: <81C9CF72068BAA4C85C4FA6FF062C7B367809725@MFRSEXCH-HQ.merseyfire.gov.uk> References: <9A4A45D3-AED0-4312-AA05-47F24BE9D24F@ultra-secure.de> In-Reply-To: <9A4A45D3-AED0-4312-AA05-47F24BE9D24F@ultra-secure.de> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.10.0.7] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 13:52:55 -0000 DQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogb3duZXItZnJlZWJzZC1w cm9saWFudEBmcmVlYnNkLm9yZyBbbWFpbHRvOm93bmVyLWZyZWVic2QtDQo+IHByb2xpYW50QGZy ZWVic2Qub3JnXSBPbiBCZWhhbGYgT2YgUmFpbmVyIER1ZmZuZXINCj4gU2VudDogMTcgTm92ZW1i ZXIgMjAxNSAyMzoxMw0KPiBUbzogZnJlZWJzZC1zdGFibGUgPGZyZWVic2Qtc3RhYmxlQGZyZWVi c2Qub3JnPg0KPiBDYzogZnJlZWJzZC1wcm9saWFudEBmcmVlYnNkLm9yZw0KPiBTdWJqZWN0OiBB ZnRlciBCSU9TLVVwZ3JhZGUsIEkgY2FuJ3QgKFVFRkktKSBib290IGFueW1vcmUNCj4gDQo+IEhp LA0KPiANCj4gSSBoYXZlIGEgSFAgREwzODAgRzksIHRoYXQgSSBib290IGZyb20gYW4gaW50ZXJu YWwgU21hcnRBcnJheSBDb250cm9sbGVyDQo+IHByb3ZpZGVkIFJBSUQxIGludG8gRnJlZUJTRCAx MC4xIGFtZDY0Lg0KPiANCj4gSSBoYXZlIHVwZ3JhZGVkIHRoZSBCSU9TIHRvIHRoZSAyMDE1LTEw IHJlbGVhc2UgYW5kIG9uIHJlYm9vdCwgSSBub3cgZ2V0IGENCj4gbWVzc2FnZSB0aGF0DQo+IA0K PiAvYm9vdC9sb2FkZXIuZWZpDQo+IA0KPiBjYW7igJl0IGJlIGZvdW5kLg0KPiANCj4gSSBjYW4g bGVnYWN5IGJvb3QgaXQgaW50byBtZnNic2QgYW5kIHRoZSBmaWxlIGlzIHRoZXJlLg0KPiANCj4g SG93IGNhbiBJIGZpeCB0aGlzPw0KPiBPciBob3cgY2FuIEkgZGVidWcgdGhpcyBhbmQgd2h5IGlz IHRoaXMgZmFpbGluZyBpbiB0aGUgZmlyc3QgcGxhY2U/DQoNCg0KRnJvbSB0aGUgd2lraQ0KDQpU aGUgYm9vdCBwcm9jZXNzIHByb2NlZWRzIGFzIGZvbGxvd3M6DQoNCiAgICBVRUZJIGZpcm13YXJl IHJ1bnMgYXQgcG93ZXIgdXAgYW5kIHNlYXJjaGVzIGZvciBhbiBPUyBsb2FkZXIgaW4gdGhlIEVG SSBzeXN0ZW0gcGFydGl0aW9uLiBUaGUgcGF0aCB0byB0aGUgbG9hZGVyIG1heSBiZSBzZXQgYnkg YW4gRUZJIGVudmlyb25tZW50IHZhcmlhYmxlLCB3aXRoIGEgZGVmYXVsdCBvZiAvRUZJL0JPT1Qv Qk9PVFg2NC5FRkkuDQoNCiAgICAgICAgRm9yIEZyZWVCU0QsIGJvb3QxLmVmaSBpcyBpbnN0YWxs ZWQgYXMgL0VGSS9CT09UL0JPT1RYNjQuRUZJLg0KICAgICAgICBib290MS5lZmlmYXQgaXMgYW4g aW1hZ2Ugb2Ygc3VjaCBhIEZBVCBmaWxlc3lzdGVtIGZvciB1c2UgYnkgYnNkaW5zdGFsbCANCiAg ICBib290MS5lZmkgbG9jYXRlcyB0aGUgZmlyc3QgcGFydGl0aW9uIHdpdGggYSB0eXBlIG9mIGZy ZWVic2QtdWZzLCBhbmQgZnJvbSBpdCBsb2FkcyBsb2FkZXIuZWZpLiAoVGhpcyBtYXkgYmUgYSBk aWZmZXJlbnQgZGlzayB0aGFuIHRoZSBvbmUgaG9sZGluZyB0aGUgRUZJIHN5c3RlbSBwYXJ0aXRp b24uKQ0KICAgIGxvYWRlci5lZmkgbG9hZHMgYW5kIGJvb3RzIHRoZSBrZXJuZWwsIGFzIGRlc2Ny aWJlZCBpbiBsb2FkZXIoOCkuDQoNClNvIG15IGJlc3QgZ3Vlc3MgaXMgdGhhdCBzb21ldGhpbmcg aGFzIGJlZW4gY2hhbmdlZCBieSB0aGUgdXBncmFkZXMgYW5kIGJvb3QxLmVmaSAgbm8gbG9uZ2Vy IGtub3dzIHRoZSBjb3JyZWN0IGxvY2F0aW9uIG9mICAvYm9vdC9sb2FkZXIuZWZpDQoNCllvdSds bCBoYXZlIHRvIGdvIGRpZ2dpbmcgaW4gdGhlIEVGSSBzeXN0ZW0gcGFydGl0aW9uIHRvIHdvcmsg b3V0IHdoYXQgaGFzIGNoYW5nZWQuIChJIGFzc3VtZSB5b3UgaGF2ZSBkb25lIGFuIGF1dG9tYXRl ZCBpbnN0YWxsIG9mIHRoZSBIUCBzdXBwb3J0IERWRCBhbmQgdXBncmFkZWQgdGhlIGFycmF5IGNv bnRyb2xsZXIgYW5kIHRoZSBIREQgbWljcm9jb2RlIGV0Yy4pDQoNCktldmluDQoNCj4gDQo+IA0K PiBSYWluZXINCj4gDQo+IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fDQo+IGZyZWVic2QtcHJvbGlhbnRAZnJlZWJzZC5vcmcgbWFpbGluZyBsaXN0DQo+IGh0 dHBzOi8vbGlzdHMuZnJlZWJzZC5vcmcvbWFpbG1hbi9saXN0aW5mby9mcmVlYnNkLXByb2xpYW50 DQo+IFRvIHVuc3Vic2NyaWJlLCBzZW5kIGFueSBtYWlsIHRvICJmcmVlYnNkLXByb2xpYW50LQ0K PiB1bnN1YnNjcmliZUBmcmVlYnNkLm9yZyINClRoaXMgZS1tYWlsIGFuZCBhbnkgZmlsZXMgdHJh bnNtaXR0ZWQgd2l0aCBpdCBhcmUgY29uZmlkZW50aWFsIGFuZCBpbnRlbmRlZCBzb2xlbHkgZm9y IHRoZSB1c2Ugb2YgdGhlIGluZGl2aWR1YWwgb3IgZW50aXR5IHRvIHdob20gdGhleSBhcmUgYWRk cmVzc2VkLg0KSWYgeW91IGhhdmUgcmVjZWl2ZWQgdGhpcyBlLW1haWwgaW4gZXJyb3IgcGxlYXNl IG5vdGlmeSB0aGUgb3JpZ2luYXRvciBvZiB0aGUgbWVzc2FnZS4gDQoNCkFueSB2aWV3cyBleHBy ZXNzZWQgaW4gdGhpcyBtZXNzYWdlIGFyZSB0aG9zZSBvZiB0aGUgaW5kaXZpZHVhbCBzZW5kZXIs IGV4Y2VwdCB3aGVyZSB0aGUgc2VuZGVyIHNwZWNpZmllcyBhbmQgd2l0aCBhdXRob3JpdHksIHN0 YXRlcyB0aGVtIHRvIGJlIHRoZSB2aWV3cyBvZiBNZXJzZXlzaWRlIEZpcmUgJiBSZXNjdWUgU2Vy dmljZSwgKE1GUlMpLg0KDQpJbmNvbWluZyBhbmQgb3V0Z29pbmcgZW1haWxzIG1heSBiZSBtb25p dG9yZWQgaW4gbGluZSB3aXRoIGN1cnJlbnQgbGVnaXNsYXRpb24uDQoNClN0ZXBzIGhhdmUgYmVl biB0YWtlbiB0byBlbnN1cmUgdGhhdCB0aGlzIGVtYWlsIGFuZCBhdHRhY2htZW50cyBhcmUgZnJl ZSBmcm9tIGFueSB2aXJ1cy4gIEluIGtlZXBpbmcgd2l0aCBnb29kIGNvbXB1dGluZyBwcmFjdGlj ZSB0aGUgcmVjaXBpZW50IHNob3VsZCBlbnN1cmUgdGhleSBhcmUgYWN0dWFsbHkgdmlydXMgZnJl ZS4NCg0KaHR0cDovL3d3dy5tZXJzZXlmaXJlLmdvdi51ay8NCg0K From owner-freebsd-stable@freebsd.org Wed Nov 18 16:15:17 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2CEC3A3206D; Wed, 18 Nov 2015 16:15:17 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-ob0-x234.google.com (mail-ob0-x234.google.com [IPv6:2607:f8b0:4003:c01::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DEC7C1164; Wed, 18 Nov 2015 16:15:16 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by obdgf3 with SMTP id gf3so37409354obd.3; Wed, 18 Nov 2015 08:15:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2KE4uk0bTK3DXRF/8MdHwvPzUzVV7xcqo/KJVoAxPqo=; b=AfKhaiHyQpJTWXhcegPHNC2Du0OkmLxCKPe5/N7wCvNttmWkAAL/lYDYCW4N4tg45D y0SN9+Y4dFRxEQMJK9V/hXoW7fhpZEZCXNZn9z3BQHw4plspSk+lH2k29XRSt4ic79PX zsR7MpTMWg/dubkMN5MrKmzQH+Hn2Tctna0+vlKKlRzLEXnrdD3Mub/x91enEm06PGFu pCr7t+ljNqnwWDzkPujB5Cj9Mpl6vXLlvR0UxSdU1qogtNPKXEFhOUmUoNofii+nblbk YfVfceUr8Zt+uHkZi/BY7VDPGALdUA7TDtzhS4rWzEL/VYu04lmta2Ruz5d49DUhq3AE uJPA== MIME-Version: 1.0 X-Received: by 10.60.76.42 with SMTP id h10mr1543574oew.13.1447863316098; Wed, 18 Nov 2015 08:15:16 -0800 (PST) Received: by 10.76.80.229 with HTTP; Wed, 18 Nov 2015 08:15:15 -0800 (PST) In-Reply-To: <20151118102502.GO48728@zxy.spb.ru> References: <5644FF09.9090200@free.de> <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> <20151118102502.GO48728@zxy.spb.ru> Date: Wed, 18 Nov 2015 08:15:15 -0800 Message-ID: Subject: Re: LSI SAS2008 mps driver preferred firmware version From: Freddie Cash To: Slawa Olhovchenkov Cc: freebsd-stable , freebsd-scsi@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 16:15:17 -0000 On Wed, Nov 18, 2015 at 2:25 AM, Slawa Olhovchenkov wrote: > On Mon, Nov 16, 2015 at 01:19:55PM -0800, Freddie Cash wrote: > > > On Mon, Nov 16, 2015 at 12:57 PM, Slawa Olhovchenkov > wrote: > > =E2=80=8BDid the original disk get labelled automatically? No, you had= to do > that > > when you first started using it. So, why would you expect a > > replaced disk > > Initial labeling is problem too. > For new chassis with 36 identical disk (already installed) -- what is > simple way to labeling disks? > =E2=80=8BThat's the easy part. Boot with all the drives pulled out a bit, = so they aren't connected/detected. Insert first disk, wait for it to be detected and get a /dev node, then partition/label it. Repeat for each disk. Takes about 5 minutes to label a 45-bay JBOD chassis. No different than how you would get the serial number off each disk before inserting them into the chassis, so you'd know for sure which slot they're in. "Replace disk in bay with blinked led" > > Author: bapt > Date: Sat Sep 5 00:06:01 2015 > =E2=80=8BAnd, how did you manage to do that before Sep 5, 2015?=E2=80=8B Usaly serial number can be read w/o pull disk (for SuperMicro cases > this is true, remote hand replaced disk by S/N for me w/o pull every disk= ). > =E2=80=8BHow? We have all SuperMicro storage chassis (SC2xx, SC8xx, and JB= ODs) and server chassis in our data centre here. None of them allow you to read the serial number off the physical disk without pulling the disk out completely.=E2=80=8B You'd have to manually label each bay with the serial= number before inserting the disk into the chassis ... which is no different from labelling the device in the OS. Except it's much faster to find a 3D co-ordinate (enc0a6) than to scan every bay looking for a specific serial number. But, to each their own. :) Everyone has their "perfect" system that works for them. :D --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-stable@freebsd.org Wed Nov 18 16:54:20 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55711A32A46; Wed, 18 Nov 2015 16:54:20 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B47B1D9F; Wed, 18 Nov 2015 16:54:20 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1Zz5zs-000Ahq-TV; Wed, 18 Nov 2015 19:54:16 +0300 Date: Wed, 18 Nov 2015 19:54:16 +0300 From: Slawa Olhovchenkov To: Freddie Cash Cc: freebsd-stable , freebsd-scsi@freebsd.org Subject: Re: LSI SAS2008 mps driver preferred firmware version Message-ID: <20151118165416.GS31314@zxy.spb.ru> References: <56472686.5030301@free.de> <20151114143104.GA41119@in-addr.com> <7710CBCC-E68F-4454-9E29-E50ED1C6B511@sarenet.es> <20151116205734.GM48728@zxy.spb.ru> <20151118102502.GO48728@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 16:54:20 -0000 On Wed, Nov 18, 2015 at 08:15:15AM -0800, Freddie Cash wrote: > On Wed, Nov 18, 2015 at 2:25 AM, Slawa Olhovchenkov wrote: > > > On Mon, Nov 16, 2015 at 01:19:55PM -0800, Freddie Cash wrote: > > > > > On Mon, Nov 16, 2015 at 12:57 PM, Slawa Olhovchenkov > > wrote: > > > ​Did the original disk get labelled automatically? No, you had to do > > that > > > when you first started using it. So, why would you expect a > > > replaced disk > > > > Initial labeling is problem too. > > For new chassis with 36 identical disk (already installed) -- what is > > simple way to labeling disks? > > > > ​That's the easy part. Boot with all the drives pulled out a bit, so they > aren't connected/detected. > > Insert first disk, wait for it to be detected and get a /dev node, then > partition/label it. Repeat for each disk. Takes about 5 minutes to label > a 45-bay JBOD chassis. Hmm, from me to server more then 1700km, how I can do this? > No different than how you would get the serial number off each disk before > inserting them into the chassis, so you'd know for sure which slot they're > in. This is do by manufacturer. Or in DC after service ordering. I am don't assemble servers, in general. And I am don't see servers and don't know how they look. > "Replace disk in bay with blinked led" > > > > Author: bapt > > Date: Sat Sep 5 00:06:01 2015 > > > > ​And, how did you manage to do that before Sep 5, 2015?​ Deteched disk don't blink activity LED. > Usaly serial number can be read w/o pull disk (for SuperMicro cases > > this is true, remote hand replaced disk by S/N for me w/o pull every disk). > > > > ​How? We have all SuperMicro storage chassis (SC2xx, SC8xx, and JBODs) and > server chassis in our data centre here. None of them allow you to read the > serial number off the physical disk without pulling the disk out > completely.​ You'd have to manually label each bay with the serial number > before inserting the disk into the chassis ... which is no different from > labelling the device in the OS. Except it's much faster to find a 3D > co-ordinate (enc0a6) than to scan every bay looking for a specific serial > number. For SC847A this do for me in NL DC (as I understand -- through holes at an angle). > But, to each their own. :) Everyone has their "perfect" system that works > for them. :D > > -- > Freddie Cash > fjwcash@gmail.com From owner-freebsd-stable@freebsd.org Wed Nov 18 19:14:08 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37D3DA328BB for ; Wed, 18 Nov 2015 19:14:08 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: from mail.ultra-secure.de (mail.ultra-secure.de [88.198.178.88]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A0E6D13E0 for ; Wed, 18 Nov 2015 19:14:07 +0000 (UTC) (envelope-from rainer@ultra-secure.de) Received: (qmail 61621 invoked by uid 89); 18 Nov 2015 19:10:31 -0000 Received: from unknown (HELO ?192.168.1.200?) (rainer@ultra-secure.de@217.71.83.52) by mail.ultra-secure.de with ESMTPA; 18 Nov 2015 19:10:31 -0000 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.1 \(3096.5\)) Subject: Re: After BIOS-Upgrade, I can't (UEFI-) boot anymore From: Rainer Duffner In-Reply-To: <81C9CF72068BAA4C85C4FA6FF062C7B367809725@MFRSEXCH-HQ.merseyfire.gov.uk> Date: Wed, 18 Nov 2015 20:10:30 +0100 Cc: freebsd-stable , "freebsd-proliant@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <9A4A45D3-AED0-4312-AA05-47F24BE9D24F@ultra-secure.de> <81C9CF72068BAA4C85C4FA6FF062C7B367809725@MFRSEXCH-HQ.merseyfire.gov.uk> To: "O'Connor, Kevin" X-Mailer: Apple Mail (2.3096.5) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Nov 2015 19:14:08 -0000 > Am 18.11.2015 um 14:33 schrieb O'Connor, Kevin = : >=20 >=20 >=20 > =46rom the wiki >=20 > The boot process proceeds as follows: >=20 > UEFI firmware runs at power up and searches for an OS loader in the = EFI system partition. The path to the loader may be set by an EFI = environment variable, with a default of /EFI/BOOT/BOOTX64.EFI. >=20 > For FreeBSD, boot1.efi is installed as /EFI/BOOT/BOOTX64.EFI. > boot1.efifat is an image of such a FAT filesystem for use by = bsdinstall=20 > boot1.efi locates the first partition with a type of freebsd-ufs, = and from it loads loader.efi. (This may be a different disk than the one = holding the EFI system partition.) > loader.efi loads and boots the kernel, as described in loader(8). >=20 > So my best guess is that something has been changed by the upgrades = and boot1.efi no longer knows the correct location of /boot/loader.efi >=20 > You'll have to go digging in the EFI system partition to work out what = has changed. (I assume you have done an automated install of the HP = support DVD and upgraded the array controller and the HDD microcode = etc.) >=20 > Kevin >=20 I=E2=80=99ve figured it out already (after sleeping a few hours and = looking at it all morning. The system contains an additional controller (H240, in JBOD mode) that = hosts another 8 disks. The first of these disks previously (and briefly) housed another FreeBSD = installation, with the GPT etc. that comes with it. Even though it was now part of a zpool, the labels etc. persisted. I had = forgotten about this... Upon the BIOS upgrade, the system suddenly started looking at this disk, = too and tried to boot from it. I had to offline the disk, remove the partitions and the GPT and online = the disk again - and then it would boot again. From owner-freebsd-stable@freebsd.org Sat Nov 21 12:31:50 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A116A32A8B for ; Sat, 21 Nov 2015 12:31:50 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3B4371773 for ; Sat, 21 Nov 2015 12:31:49 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 6CB72284B8 for ; Sat, 21 Nov 2015 13:31:40 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id BF51028430 for ; Sat, 21 Nov 2015 13:31:39 +0100 (CET) Message-ID: <5650642B.5030707@quip.cz> Date: Sat, 21 Nov 2015 13:31:39 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: freebsd-stable Stable Subject: su on 10.2: TERM: Undefined variable Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2015 12:31:50 -0000 I noticed some weird behavior of "su" command in shell scripts running from cron after upgrade from FreeBSD 8.4 to 10.2. If I have this in script su -m www -c 'ls -l' then I get "TERM: Undefined variable" on the stderr if this script is run from cron. It works fine on FreeBSD 8.4 Is it intentional behavior? Miroslav Lachman From owner-freebsd-stable@freebsd.org Sat Nov 21 18:10:12 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EC5F7A346E8; Sat, 21 Nov 2015 18:10:12 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (jenkins-9.freebsd.org [8.8.178.209]) by mx1.freebsd.org (Postfix) with ESMTP id D0C4313D0; Sat, 21 Nov 2015 18:10:12 +0000 (UTC) (envelope-from jenkins-admin@FreeBSD.org) Received: from jenkins-9.freebsd.org (localhost [127.0.0.1]) by jenkins-9.freebsd.org (Postfix) with ESMTP id 9648819C6; Sat, 21 Nov 2015 18:10:12 +0000 (UTC) Date: Sat, 21 Nov 2015 18:10:10 +0000 (GMT) From: jenkins-admin@FreeBSD.org To: kib@FreeBSD.org, jenkins-admin@FreeBSD.org, freebsd-stable@FreeBSD.org, freebsd-i386@FreeBSD.org Message-ID: <1229899027.93.1448129412100.JavaMail.jenkins@jenkins-9.freebsd.org> Subject: FreeBSD_STABLE_10-i386 - Build #657 - Failure MIME-Version: 1.0 X-Jenkins-Job: FreeBSD_STABLE_10-i386 X-Jenkins-Result: FAILURE Precedence: bulk Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2015 18:10:13 -0000 FreeBSD_STABLE_10-i386 - Build #657 - Failure: Build information: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/6= 57/ Full change log: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/657= /changes Full build log: https://jenkins.FreeBSD.org/job/FreeBSD_STABLE_10-i386/657/= console Change summaries: 291134 by kib: MFC r290492: Move intmax_t and uintmax_t type declarations to sys/_stdint.h. The end of the build log: [...truncated 101003 lines...] --- lib.all__D --- --- localzone.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/services/localzone.c -= o localzone.po --- sbin.all__D --- --- main.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_ffs -I/usr/src/sbin/fsck_ffs/../mount = -std=3Dgnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror= -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -c /usr= /src/sbin/fsck_ffs/main.c -o main.o --- all_subdir_fsck_msdosfs --- =3D=3D=3D> sbin/fsck_msdosfs (all) --- main.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_msdosfs/../fsck -std=3Dgnu99 -Qunused-= arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k= -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointe= r-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunuse= d-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredu= ndant-decls -Wold-style-definition -Wmissing-variable-declarations -Wno-poi= nter-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -= c /usr/src/sbin/fsck_msdosfs/main.c -o main.o --- secure.all__D --- --- a_octet.po --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/a= sn1/a_octet.c -o a_octet.po --- lib.all__D --- --- locks.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/locks.c -o locks.= po --- sbin.all__D --- --- check.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_msdosfs/../fsck -std=3Dgnu99 -Qunused-= arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k= -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointe= r-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunuse= d-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredu= ndant-decls -Wold-style-definition -Wmissing-variable-declarations -Wno-poi= nter-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -= c /usr/src/sbin/fsck_msdosfs/check.c -o check.o --- secure.all__D --- --- a_print.po --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../..--- lib.all__D --- --- log.po --- --- secure.all__D --- /crypto/openssl/crypto/asn1/a_print.c -o a_print.po --- lib.all__D --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/log.c -o log.po --- sbin.all__D --- --- boot.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_msdosfs/../fsck -std=3Dgnu99 -Qunused-= arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k= -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointe= r-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunuse= d-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredu= ndant-decls -Wold-style-definition -Wmissing-variable-declarations -Wno-poi= nter-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -= c /usr/src/sbin/fsck_msdosfs/boot.c -o boot.o --- all_subdir_fsck_ffs --- --- pass1.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_ffs -I/usr/src/sbin/fsck_ffs/../mount = -std=3Dgnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror= -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -c /usr= /src/sbin/fsck_ffs/pass1.c -o pass1.o --- lib.all__D --- --- lookup3.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/storage/lookup3.c= -o lookup3.po --- secure.all__D --- --- a_set.po --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/a= sn1/a_set.c -o a_set.po --- lib.all__D --- --- lruhash.po --- --- sbin.all__D --- --- all_subdir_fsck_msdosfs --- --- fat.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_msdosfs/../fsck -std=3Dgnu99 -Qunused-= arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k= -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointe= r-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunuse= d-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredu= ndant-decls -Wold-style-definition -Wmissing-variable-declarations -Wno-poi= nter-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -= c /usr/src/sbin/fsck_msdosfs/fat.c -o fat.o --- lib.all__D --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/storage/lruhash.c= -o lruhash.po --- secure.all__D --- --- a_sign.po --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/a= sn1/a_sign.c -o a_sign.po --- lib.all__D --- --- mesh.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/services/mesh.c -o mes= h.po --- sbin.all__D --- --- dir.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_msdosfs/../fsck -std=3Dgnu99 -Qunused-= arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k= -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointe= r-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunuse= d-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredu= ndant-decls -Wold-style-definition -Wmissing-variable-declarations -Wno-poi= nter-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -= c /usr/src/sbin/fsck_msdosfs/dir.c -o dir.o --- all_subdir_fsck_ffs --- --- pass1b.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_ffs -I/usr/src/sbin/fsck_ffs/../mount = -std=3Dgnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror= -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -c /usr= /src/sbin/fsck_ffs/pass1b.c -o pass1b.o --- secure.all__D --- --- a_strex.po --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../..--- lib.all__D --- --- mini_event.po --- --- secure.all__D --- /crypto/openssl/crypto/asn1/a_strex.c -o a_strex.po --- lib.all__D --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/mini_event.c -o m= ini_event.po --- sbin.all__D --- --- pass2.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_ffs -I/usr/src/sbin/fsck_ffs/../mount = -std=3Dgnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror= -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -c /usr= /src/sbin/fsck_ffs/pass2.c -o pass2.o --- lib.all__D --- --- modstack.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/services/modstack.c -o= modstack.po --- module.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/module.c -o modul= e.po --- secure.all__D --- --- a_strnid.po --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/a= sn1/a_strnid.c -o a_strnid.po --- lib.all__D --- --- msgencode.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/data/msgencode.c = -o msgencode.po --- sbin.all__D --- --- pass3.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_ffs -I/usr/src/sbin/fsck_ffs/../mount = -std=3Dgnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror= -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -c /usr= /src/sbin/fsck_ffs/pass3.c -o pass3.o --- secure.all__D --- --- a_time.po --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/a= sn1/a_time.c -o a_time.po --- lib.all__D --- --- msgparse.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/data/msgparse.c -= o msgparse.po --- sbin.all__D --- --- pass4.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_ffs -I/usr/src/sbin/fsck_ffs/../mount = -std=3Dgnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror= -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -c /usr= /src/sbin/fsck_ffs/pass4.c -o pass4.o --- all_subdir_fsck_msdosfs --- --- fsutil.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_msdosfs/../fsck -std=3Dgnu99 -Qunused-= arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k= -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointe= r-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunuse= d-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredu= ndant-decls -Wold-style-definition -Wmissing-variable-declarations -Wno-poi= nter-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable -= c /usr/src/sbin/fsck_msdosfs/../fsck/fsutil.c -o fsutil.o --- secure.all__D --- --- a_type.po --- --- sbin.all__D --- --- all_subdir_fsck_ffs --- --- pass5.o --- cc -O2 -pipe -I/usr/src/sbin/fsck_ffs -I/usr/src/sbin/fsck_ffs/../mount = -std=3Dgnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror= -Wall -Wno-format-y2k -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -c /usr= /src/sbin/fsck_ffs/pass5.c -o pass5.o --- secure.all__D --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/a= sn1/a_type.c -o a_type.po --- lib.all__D --- --- msgreply.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/data/msgreply.c -= o msgreply.po --- sbin.all__D --- --- all_subdir_fsck_msdosfs --- --- fsck_msdosfs.8.gz --- gzip -cn /usr/src/sbin/fsck_msdosfs/fsck_msdosfs.8 > fsck_msdosfs.8.gz --- fsck_msdosfs --- cc -O2 -pipe -I/usr/src/sbin/fsck_msdosfs/../fsck -std=3Dgnu99 -Qunused-= arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k= -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointe= r-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunuse= d-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredu= ndant-decls -Wold-style-definition -Wmissing-variable-declarations -Wno-poi= nter-sign -Wno-empty-body -Wno-string-plus-int -Wno-unused-const-variable = -o fsck_msdosfs main.o check.o boot.o fat.o dir.o fsutil.o=20 --- secure.all__D --- --- a_utctm.po --- cc -pg -O2 -pipe -DTERMIOS -DANSI_SOURCE -I/usr/src/secure/lib/libcrypt= o/../../../crypto/openssl -I/usr/src/secure/lib/libcrypto/../../../crypto/o= penssl/crypto -I/usr/obj/usr/src/secure/lib/libcrypto -DOPENSSL_THREADS -DD= SO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DOPENSSL_IA32_SSE2 -DAES_ASM -DVPAES_AS= M -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -= DMD5_ASM -DGHASH_ASM -DRMD160_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DWH= IRLPOOL_ASM -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/= asn1 -I/usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/evp -I/= usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/modes -std=3Dgn= u89 -Qunused-arguments -fstack-protector -Wno-pointer-sign -Wno-empty-body= -Wno-string-plus-int -Wno-unused-const-variable -Wno-tautological-compare = -Wno-unused-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-= conversion -Wno-switch -Wno-switch-enum -Wno-knr-promoted-parameter -Wno-pa= rentheses -c /usr/src/secure/lib/libcrypto/../../../crypto/openssl/crypto/a= sn1/a_utctm.c -o a_utctm.po --- share.all__D --- =3D=3D=3D> share (all) --- lib.all__D --- --- net_help.po --- cc -pg -I/usr/src/lib/libunbound/../../contrib/unbound -I/usr/src/lib/lib= unbound/../../contrib/ldns -I/usr/obj/usr/src/lib/libunbound -std=3Dgnu99 -= Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-fo= rmat-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes = -Wpointer-arith -Wno-uninitialized -Wno-pointer-sign -Wno-empty-body -Wno-s= tring-plus-int -Wno-unused-const-variable -Wno-tautological-compare -Wno-un= used-value -Wno-parentheses-equality -Wno-unused-function -Wno-enum-convers= ion -c /usr/src/lib/libunbound/../../contrib/unbound/util/net_help.c -o net= _help.po --- share.all__D --- --- _sub.all --- =3D=3D=3D> share/colldef (all) --- bg_BG.CP1251.out --- colldef -I /usr/src/share/colldef -o bg_BG.CP1251.out /usr/src/share/collde= f/bg_BG.CP1251.src sh: colldef: not found *** [bg_BG.CP1251.out] Error code 127 make[4]: stopped in /usr/src/share/colldef 1 error make[4]: stopped in /usr/src/share/colldef *** [_sub.all] Error code 2 make[3]: stopped in /usr/src/share 1 error make[3]: stopped in /usr/src/share *** [share.all__D] Error code 2 make[2]: stopped in /usr/src --- secure.all__D --- A failure has been detected in another branch of the parallel make make[5]: stopped in /usr/src/secure/lib/libcrypto *** [_sub.all] Error code 2 make[4]: stopped in /usr/src/secure/lib 1 error make[4]: stopped in /usr/src/secure/lib *** [_sub.all] Error code 2 make[3]: stopped in /usr/src/secure 1 error make[3]: stopped in /usr/src/secure *** [secure.all__D] Error code 2 make[2]: stopped in /usr/src --- sbin.all__D --- --- all_subdir_fsck_ffs --- A failure has been detected in another branch of the parallel make make[4]: stopped in /usr/src/sbin/fsck_ffs *** [all_subdir_fsck_ffs] Error code 2 make[3]: stopped in /usr/src/sbin 1 error make[3]: stopped in /usr/src/sbin *** [sbin.all__D] Error code 2 make[2]: stopped in /usr/src --- lib.all__D --- A failure has been detected in another branch of the parallel make make[4]: stopped in /usr/src/lib/libunbound *** [all_subdir_libunbound] Error code 2 make[3]: stopped in /usr/src/lib 1 error make[3]: stopped in /usr/src/lib *** [lib.all__D] Error code 2 make[2]: stopped in /usr/src 4 errors make[2]: stopped in /usr/src *** [everything] Error code 2 make[1]: stopped in /usr/src 1 error make[1]: stopped in /usr/src *** [buildworld] Error code 2 make: stopped in /usr/src 1 error make: stopped in /usr/src Build step 'Execute shell' marked build as failure [WARNINGS] Skipping publisher since build result is FAILURE [PostBuildScript] - Execution post build scripts. [FreeBSD_STABLE_10-i386] $ /bin/sh -xe /tmp/hudson8542043854554102297.sh + export 'PATH=3D/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/b= in' + export 'jname=3DFreeBSD_STABLE_10-i386' + echo 'clean up jail FreeBSD_STABLE_10-i386' clean up jail FreeBSD_STABLE_10-i386 + sudo jail -r FreeBSD_STABLE_10-i386 + sudo ifconfig igb0 inet6 2610:1c1:1:607c::105:1 -alias + sudo umount FreeBSD_STABLE_10-i386/usr/src + sudo umount FreeBSD_STABLE_10-i386/dev + sudo rm -fr FreeBSD_STABLE_10-i386 rm: FreeBSD_STABLE_10-i386/sbin/init: Operation not permitted rm: FreeBSD_STABLE_10-i386/sbin: Directory not empty rm: FreeBSD_STABLE_10-i386/var/empty: Operation not permitted rm: FreeBSD_STABLE_10-i386/var: Directory not empty rm: FreeBSD_STABLE_10-i386/usr/bin/chpass: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/crontab: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/opiepasswd: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/ypchpass: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/login: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/ypchfn: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/su: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/ypchsh: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/yppasswd: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/chfn: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/opieinfo: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/chsh: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin/passwd: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/bin: Directory not empty rm: FreeBSD_STABLE_10-i386/usr/lib/librt.so.1: Operation not permitted rm: FreeBSD_STABLE_10-i386/usr/lib: Directory not empty rm: FreeBSD_STABLE_10-i386/usr: Directory not empty rm: FreeBSD_STABLE_10-i386/lib/libthr.so.3: Operation not permitted rm: FreeBSD_STABLE_10-i386/lib/libc.so.7: Operation not permitted rm: FreeBSD_STABLE_10-i386/lib/libcrypt.so.5: Operation not permitted rm: FreeBSD_STABLE_10-i386/lib: Directory not empty rm: FreeBSD_STABLE_10-i386/libexec/ld-elf.so.1: Operation not permitted rm: FreeBSD_STABLE_10-i386/libexec: Directory not empty rm: FreeBSD_STABLE_10-i386: Directory not empty + true + sudo chflags -R noschg FreeBSD_STABLE_10-i386 + sudo rm -fr FreeBSD_STABLE_10-i386 Email was triggered for: Failure - Any Sending email for trigger: Failure - Any From owner-freebsd-stable@freebsd.org Sat Nov 21 19:12:03 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A511A333BE for ; Sat, 21 Nov 2015 19:12:03 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1C5611346 for ; Sat, 21 Nov 2015 19:12:03 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1a0DZg-0000VJ-8Z; Sat, 21 Nov 2015 22:11:52 +0300 Date: Sat, 21 Nov 2015 22:11:52 +0300 From: Slawa Olhovchenkov To: Luigi Rizzo Cc: Stable Stable Subject: Re: NETMAP and off-by-one? Message-ID: <20151121191152.GV31314@zxy.spb.ru> References: <20151113213039.GK48728@zxy.spb.ru> <20151114084658.GZ31314@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151114084658.GZ31314@zxy.spb.ru> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2015 19:12:03 -0000 On Sat, Nov 14, 2015 at 11:46:58AM +0300, Slawa Olhovchenkov wrote: > On Fri, Nov 13, 2015 at 03:40:04PM -0800, Luigi Rizzo wrote: > > > On Fri, Nov 13, 2015 at 1:30 PM, Slawa Olhovchenkov wrote: > > > I am see strange things: like NETMAP stop transmit after `head` and `cur` > > > touch `tail`. > > > > > > But: > > > > > > /* > > > * check if space is available in the ring. > > > */ > > > static inline int > > > nm_ring_empty(struct netmap_ring *ring) > > > { > > > return (ring->cur == ring->tail); > > > } > > > > > > i.e. if cur == (tail-1) mod ring_size -- space is available in the > > > ring and I can put packet in output buffer. > > > > The design requires to leave at least one empty slot in the buffer. > > The name of the function is correct, the comment is probably not, > > unless a bug has creeped in recently the code was very careful > > in not using the free slot that separates the two regions. > > Please, do some clarification, in case for transmit patch: > > - can I put in txring in case cur == (tail-1) mod ring_size? > - can I use poll with only `events |= POLLIN` for transmiting? > > in case for receive patch: > > - can I detect input overflow? I am asks this questions becaus in may case don't see automatic txsync on poll: device open as np->nr_ringid = NETMAP_DO_RX_POLL; np->nr_flags = NR_REG_ALL_NIC; rc = ioctl(balancer.inside.fd, NIOCREGIF, np); poll do as pollfd[1].fd = thr->inside.fd; pollfd[1].events |= POLLIN; poll(pollfd, nfd, 1000); and I don't see output sysncing w/o pollfd[1].events |= POLLIN | POLLOUT; Only one round packet outputs I see and no slot updates. From owner-freebsd-stable@freebsd.org Sat Nov 21 20:42:17 2015 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B058A347C6 for ; Sat, 21 Nov 2015 20:42:17 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from mx1.eichornenterprises.com (mx1.eichornenterprises.com [104.236.13.122]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.eichornenterprises.com", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id DA032116B for ; Sat, 21 Nov 2015 20:42:16 +0000 (UTC) (envelope-from ike@michaeleichorn.com) Received: from mail.eichornenterprises.com (cpe-184-59-147-149.neo.res.rr.com [184.59.147.149]) by mx1.eichornenterprises.com (OpenSMTPD) with ESMTP id 20725efb; Sat, 21 Nov 2015 15:42:06 -0500 (EST) Received: by mail.eichornenterprises.com (OpenSMTPD) with ESMTPSA id 882ebaf9 TLS version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO; Sat, 21 Nov 2015 15:42:06 -0500 (EST) Message-ID: <1448138525.10590.18.camel@michaeleichorn.com> Subject: Re: su on 10.2: TERM: Undefined variable From: "Michael B. Eichorn" To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-stable Stable Date: Sat, 21 Nov 2015 15:42:05 -0500 In-Reply-To: <5650642B.5030707@quip.cz> References: <5650642B.5030707@quip.cz> Content-Type: multipart/signed; micalg="sha-256"; protocol="application/x-pkcs7-signature"; boundary="=-TYk7xwhDuZnQHsJ1cOl5" X-Mailer: Evolution 3.18.1 Mime-Version: 1.0 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2015 20:42:17 -0000 --=-TYk7xwhDuZnQHsJ1cOl5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, 2015-11-21 at 13:31 +0100, Miroslav Lachman wrote: > I noticed some weird behavior of "su" command in shell scripts > running=20 > from cron after upgrade from FreeBSD 8.4 to 10.2. >=20 > If I have this in script >=20 > su -m www -c 'ls -l' >=20 > then I get "TERM: Undefined variable" on the stderr if this script is > run from cron. > It works fine on FreeBSD 8.4 >=20 > Is it intentional behavior? >=20 > Miroslav Lachman I cannot reproduce your problem. I used the following script: #!/bin/sh echo "BEGIN TEST" echo $TERM su -m www -c 'ls -l' echo $TERM echo "END TEST" crontab is: * * * * * /root/test.sh and the result is: BEGIN TEST total 520765 -rw-r--r--=C2=A0=C2=A0 1 root=C2=A0 wheel=C2=A0 724002816 Aug 12 11:45 Free= BSD-10.2- RELEASE-amd64-disc1.iso -rwxr-xr-x=C2=A0=C2=A0 1 root=C2=A0 wheel=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 90 Nov 21 15:33 test.sh END TEST # freebsd-version 10.2-RELEASE-p7 --=-TYk7xwhDuZnQHsJ1cOl5 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Disposition: attachment; filename="smime.p7s" Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCEqAw ggYwMIIFGKADAgECAgMOXcYwDQYJKoZIhvcNAQELBQAwgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQK Ew1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWdu aW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVu dCBDQTAeFw0xNTA2MTMyMDI0NDZaFw0xNjA2MTQwMDM1NTBaMEgxHzAdBgNVBAMMFmlrZUBtaWNo YWVsZWljaG9ybi5jb20xJTAjBgkqhkiG9w0BCQEWFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDJVdWALPz5h2s5zUQGIJYl6Vp8FPtZNko8q/3s crCsxXJLprMaDdpnqTsmkbmEfKvsqPQE6HVOpGxVRTl/tCm+VvouW9eY9ITMigb1OnHdU13CKO0j drgeU1nHst0qxwsIofRD7nC4dakT6exnrVndlBmLrf/bLPh2qOM8YK5qKK6m33fE7AyYrwiYAWFT 3fERI7LakjaabrIoS/Y1rCdL5FaCTMOlRbZyduc8HkrgjT2JW+i4fVcKyGL5gExBJWfS3q1uGFaB ie6pYtl8lZPtvN0JSfibP003RBoLgzqHJKW91RL0qNeDjKZi/5nrlU398l9UoVvLLO3KxoPBXKCx AgMBAAGjggLcMIIC2DAJBgNVHRMEAjAAMAsGA1UdDwQEAwIEsDAdBgNVHSUEFjAUBggrBgEFBQcD AgYIKwYBBQUHAwQwHQYDVR0OBBYEFJZqarc6CcrOs6eAwOgrMznk5ZWWMB8GA1UdIwQYMBaAFFNy 7ZKc4NrLAVx8fpY1TvLUuFGCMCEGA1UdEQQaMBiBFmlrZUBtaWNoYWVsZWljaG9ybi5jb20wggFM BgNVHSAEggFDMIIBPzCCATsGCysGAQQBgbU3AQIDMIIBKjAuBggrBgEFBQcCARYiaHR0cDovL3d3 dy5zdGFydHNzbC5jb20vcG9saWN5LnBkZjCB9wYIKwYBBQUHAgIwgeowJxYgU3RhcnRDb20gQ2Vy dGlmaWNhdGlvbiBBdXRob3JpdHkwAwIBARqBvlRoaXMgY2VydGlmaWNhdGUgd2FzIGlzc3VlZCBh Y2NvcmRpbmcgdG8gdGhlIENsYXNzIDEgVmFsaWRhdGlvbiByZXF1aXJlbWVudHMgb2YgdGhlIFN0 YXJ0Q29tIENBIHBvbGljeSwgcmVsaWFuY2Ugb25seSBmb3IgdGhlIGludGVuZGVkIHB1cnBvc2Ug aW4gY29tcGxpYW5jZSBvZiB0aGUgcmVseWluZyBwYXJ0eSBvYmxpZ2F0aW9ucy4wNgYDVR0fBC8w LTAroCmgJ4YlaHR0cDovL2NybC5zdGFydHNzbC5jb20vY3J0dTEtY3JsLmNybDCBjgYIKwYBBQUH AQEEgYEwfzA5BggrBgEFBQcwAYYtaHR0cDovL29jc3Auc3RhcnRzc2wuY29tL3N1Yi9jbGFzczEv Y2xpZW50L2NhMEIGCCsGAQUFBzAChjZodHRwOi8vYWlhLnN0YXJ0c3NsLmNvbS9jZXJ0cy9zdWIu Y2xhc3MxLmNsaWVudC5jYS5jcnQwIwYDVR0SBBwwGoYYaHR0cDovL3d3dy5zdGFydHNzbC5jb20v MA0GCSqGSIb3DQEBCwUAA4IBAQB4K8iQw+0FRn3xEnB3vIIu2Vi4C3ZGnOMWP90FFXLrZ6uAu9AK xVCjXUVP6nAEsOopTMu769vVecdBvg0KO2i5aTDTdTLX4g9d020g4OLWW1NiynAkX8oKqJLqZ53q vHK4zP4KWPS3bSqDWVCosTMfI+H6tkg+6G3gS0HHoHTLKZhIT3z6PQZAfeofM7ed6NOdAcj0J2lP ODHzzz7Y9x4wMwYJdidorzUDVYkNIkim8ak7hK9F60NadA5w/BirFATSlzRyV0h1tl6oNisEaQcq tGvy6UoCTDhzaJ7pQValfDXJ/A47P0hNj/CX/PmkY1wQHsEJz2pbh5lqteP/fO0rMIIGMDCCBRig AwIBAgIDDl3GMA0GCSqGSIb3DQEBCwUAMIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRD b20gTHRkLjErMCkGA1UECxMiU2VjdXJlIERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYG A1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQcmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0EwHhcN MTUwNjEzMjAyNDQ2WhcNMTYwNjE0MDAzNTUwWjBIMR8wHQYDVQQDDBZpa2VAbWljaGFlbGVpY2hv cm4uY29tMSUwIwYJKoZIhvcNAQkBFhZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAyVXVgCz8+YdrOc1EBiCWJelafBT7WTZKPKv97HKwrMVyS6az Gg3aZ6k7JpG5hHyr7Kj0BOh1TqRsVUU5f7Qpvlb6LlvXmPSEzIoG9Tpx3VNdwijtI3a4HlNZx7Ld KscLCKH0Q+5wuHWpE+nsZ61Z3ZQZi63/2yz4dqjjPGCuaiiupt93xOwMmK8ImAFhU93xESOy2pI2 mm6yKEv2NawnS+RWgkzDpUW2cnbnPB5K4I09iVvouH1XCshi+YBMQSVn0t6tbhhWgYnuqWLZfJWT 7bzdCUn4mz9NN0QaC4M6hySlvdUS9KjXg4ymYv+Z65VN/fJfVKFbyyztysaDwVygsQIDAQABo4IC 3DCCAtgwCQYDVR0TBAIwADALBgNVHQ8EBAMCBLAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUF BwMEMB0GA1UdDgQWBBSWamq3OgnKzrOngMDoKzM55OWVljAfBgNVHSMEGDAWgBRTcu2SnODaywFc fH6WNU7y1LhRgjAhBgNVHREEGjAYgRZpa2VAbWljaGFlbGVpY2hvcm4uY29tMIIBTAYDVR0gBIIB QzCCAT8wggE7BgsrBgEEAYG1NwECAzCCASowLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3BvbGljeS5wZGYwgfcGCCsGAQUFBwICMIHqMCcWIFN0YXJ0Q29tIENlcnRpZmljYXRp b24gQXV0aG9yaXR5MAMCAQEagb5UaGlzIGNlcnRpZmljYXRlIHdhcyBpc3N1ZWQgYWNjb3JkaW5n IHRvIHRoZSBDbGFzcyAxIFZhbGlkYXRpb24gcmVxdWlyZW1lbnRzIG9mIHRoZSBTdGFydENvbSBD QSBwb2xpY3ksIHJlbGlhbmNlIG9ubHkgZm9yIHRoZSBpbnRlbmRlZCBwdXJwb3NlIGluIGNvbXBs aWFuY2Ugb2YgdGhlIHJlbHlpbmcgcGFydHkgb2JsaWdhdGlvbnMuMDYGA1UdHwQvMC0wK6ApoCeG JWh0dHA6Ly9jcmwuc3RhcnRzc2wuY29tL2NydHUxLWNybC5jcmwwgY4GCCsGAQUFBwEBBIGBMH8w OQYIKwYBBQUHMAGGLWh0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9zdWIvY2xhc3MxL2NsaWVudC9j YTBCBggrBgEFBQcwAoY2aHR0cDovL2FpYS5zdGFydHNzbC5jb20vY2VydHMvc3ViLmNsYXNzMS5j bGllbnQuY2EuY3J0MCMGA1UdEgQcMBqGGGh0dHA6Ly93d3cuc3RhcnRzc2wuY29tLzANBgkqhkiG 9w0BAQsFAAOCAQEAeCvIkMPtBUZ98RJwd7yCLtlYuAt2RpzjFj/dBRVy62ergLvQCsVQo11FT+pw BLDqKUzLu+vb1XnHQb4NCjtouWkw03Uy1+IPXdNtIODi1ltTYspwJF/KCqiS6med6rxyuMz+Clj0 t20qg1lQqLEzHyPh+rZIPuht4EtBx6B0yymYSE98+j0GQH3qHzO3nejTnQHI9CdpTzgx888+2Pce MDMGCXYnaK81A1WJDSJIpvGpO4SvRetDWnQOcPwYqxQE0pc0cldIdbZeqDYrBGkHKrRr8ulKAkw4 c2ie6UFWpXw1yfwOOz9ITY/wl/z5pGNcEB7BCc9qW4eZarXj/3ztKzCCBjQwggQcoAMCAQICAR4w DQYJKoZIhvcNAQEFBQAwfTELMAkGA1UEBhMCSUwxFjAUBgNVBAoTDVN0YXJ0Q29tIEx0ZC4xKzAp BgNVBAsTIlNlY3VyZSBEaWdpdGFsIENlcnRpZmljYXRlIFNpZ25pbmcxKTAnBgNVBAMTIFN0YXJ0 Q29tIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MB4XDTA3MTAyNDIxMDE1NVoXDTE3MTAyNDIxMDE1 NVowgYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1 cmUgRGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAx IFByaW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC AQoCggEBAMcJg8zOLdgasSmkLhOrlr6KMoOMpohBllVHrdRvEg/q6r8jR+EK75xCGhR8ToREoqe7 zM9/UnC6TS2y9UKTpT1v7RSMzR0t6ndl0TWBuUr/UXBhPk+Kmy7bI4yW4urC+y7P3/1/X7U8ocb8 VpH/Clt+4iq7nirMcNh6qJR+xjOhV+VHzQMALuGYn5KZmc1NbJQYclsGkDxDz2UbFqE2+6vIZoL+ jb9x4Pa5gNf1TwSDkOkikZB1xtB4ZqtXThaABSONdfmv/Z1pua3FYxnCFmdr/+N2JLKutIxMYqQO Jebr/f/h5t95m4JgrM3Y/w7YX9d7YAL9jvN4SydHsU6n65cCAwEAAaOCAa0wggGpMA8GA1UdEwEB /wQFMAMBAf8wDgYDVR0PAQH/BAQDAgEGMB0GA1UdDgQWBBRTcu2SnODaywFcfH6WNU7y1LhRgjAf BgNVHSMEGDAWgBROC+8apEBbpRdphzDKNGhD0EGu8jBmBggrBgEFBQcBAQRaMFgwJwYIKwYBBQUH MAGGG2h0dHA6Ly9vY3NwLnN0YXJ0c3NsLmNvbS9jYTAtBggrBgEFBQcwAoYhaHR0cDovL3d3dy5z dGFydHNzbC5jb20vc2ZzY2EuY3J0MFsGA1UdHwRUMFIwJ6AloCOGIWh0dHA6Ly93d3cuc3RhcnRz c2wuY29tL3Nmc2NhLmNybDAnoCWgI4YhaHR0cDovL2NybC5zdGFydHNzbC5jb20vc2ZzY2EuY3Js MIGABgNVHSAEeTB3MHUGCysGAQQBgbU3AQIBMGYwLgYIKwYBBQUHAgEWImh0dHA6Ly93d3cuc3Rh cnRzc2wuY29tL3BvbGljeS5wZGYwNAYIKwYBBQUHAgEWKGh0dHA6Ly93d3cuc3RhcnRzc2wuY29t L2ludGVybWVkaWF0ZS5wZGYwDQYJKoZIhvcNAQEFBQADggIBAAqDCH14qywGXLhjjF6uHLkjd02h cdh9hrw+VUsv+q1eeQWB21jWj3kJ96AUlPCoEGZ/ynJNScWy6QMVQjbbMXltUfO4n4bGGdKo3awP Wp61tjAFgraLJgDk+DsSvUD6EowjMTNx25GQgyYJ5RPIzKKR9tQW8gGK+2+RHxkUCTbYFnL6kl8C h507rUdPPipJ9CgJFws3kDS3gOS5WFMxcjO5DwKfKSETEPrHh7p5shuuNktvsv6hxHTLhiMKX893 gxdT3XLS9OKmCv87vkINQcNEcIIoFWbP9HORz9v3vQwR4e3ksLc2JZOAFK+ssS5XMEoznzpihEP0 PLc4dCBYjbvSD7kxgDwZ+Aj8Q9PkbvE9sIPP7ON0fz095HdThKjiVJe6vofq+n6b1NBc8XdrQvBm unwxD5nvtTW4vtN6VY7mUCmxsCieuoBJ9OlqmsVWQvifIYf40dJPZkk9YgGTzWLpXDSfLSplbY2L L9C9U0ptvjcDjefLTvqSFc7tw1sEhF0n/qpA2r0GpvkLRDmcSwVyPvmjFBGqUp/pNy8ZuPGQmHwF i2/14+xeSUDG2bwnsYJQG2EdJCB6luQ57GEnTA/yKZSTKI8dDQa8Sd3zfXb19mOgSF0bBdXbuKhE puP9wirslFe6fQ1t5j5R0xi72MZ8ikMu1RQZKCyDbMwazlHiMYIDfzCCA3sCAQEwgZQwgYwxCzAJ BgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUgRGlnaXRh bCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFByaW1hcnkg SW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCWCGSAFlAwQCAQUAoIIBuzAYBgkqhkiG9w0B CQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNTExMjEyMDQyMDVaMC8GCSqGSIb3DQEJ BDEiBCCqZWQ9J0Wpgh9JgUDIDe+SOlI8b26GgNVipHqlj5HJrjCBpQYJKwYBBAGCNxAEMYGXMIGU MIGMMQswCQYDVQQGEwJJTDEWMBQGA1UEChMNU3RhcnRDb20gTHRkLjErMCkGA1UECxMiU2VjdXJl IERpZ2l0YWwgQ2VydGlmaWNhdGUgU2lnbmluZzE4MDYGA1UEAxMvU3RhcnRDb20gQ2xhc3MgMSBQ cmltYXJ5IEludGVybWVkaWF0ZSBDbGllbnQgQ0ECAw5dxjCBpwYLKoZIhvcNAQkQAgsxgZeggZQw gYwxCzAJBgNVBAYTAklMMRYwFAYDVQQKEw1TdGFydENvbSBMdGQuMSswKQYDVQQLEyJTZWN1cmUg RGlnaXRhbCBDZXJ0aWZpY2F0ZSBTaWduaW5nMTgwNgYDVQQDEy9TdGFydENvbSBDbGFzcyAxIFBy aW1hcnkgSW50ZXJtZWRpYXRlIENsaWVudCBDQQIDDl3GMA0GCSqGSIb3DQEBAQUABIIBAMEEjPY6 oVzla90HfbIjfHbpLXWERXhkPWSB181pt4ogw2+N65fh6cUWoZYhR9yNPwbjL9Mcjl2apZjkBkFL BlKkYNRr4DrHR0BXVDqX2lEK0k6zQoVIYLLiaHpKncdOp8JxsQBfSkGB0afRS+LyqcqySiuejHoq yNHBQj9zuMYDE5JfKno2kxdB27glWerU+Vdbn3t3vkCrnj/yJy+pSYp1mRz29xknMeH8sbqsAl+q HA4hEmQZch8yKdK6MCkGw1jsLaAVxnHe6ObNU3TsaYzNXT/VmMJBV626ghK01zkyJRj/jsuMIqzk 86JCdzJZgLmp6bpsHUtVx27K/5h940IAAAAAAAA= --=-TYk7xwhDuZnQHsJ1cOl5--