Date: Tue, 30 Apr 2019 12:07:55 -0400 (EDT) From: Walter Cramer <wfc@mintsol.com> To: Michelle Sullivan <michelle@sorbs.net> Cc: Karl Denninger <karl@denninger.net>, freebsd-stable@freebsd.org Subject: Re: ZFS... Message-ID: <20190430102024.E84286@mulder.mintsol.com> In-Reply-To: <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net> References: <30506b3d-64fb-b327-94ae-d9da522f3a48@sorbs.net> <CAOtMX2gf3AZr1-QOX_6yYQoqE-H%2B8MjOWc=eK1tcwt5M3dCzdw@mail.gmail.com> <56833732-2945-4BD3-95A6-7AF55AB87674@sorbs.net> <3d0f6436-f3d7-6fee-ed81-a24d44223f2f@netfence.it> <17B373DA-4AFC-4D25-B776-0D0DED98B320@sorbs.net> <70fac2fe3f23f85dd442d93ffea368e1@ultra-secure.de> <70C87D93-D1F9-458E-9723-19F9777E6F12@sorbs.net> <CAGMYy3tYqvrKgk2c==WTwrH03uTN1xQifPRNxXccMsRE1spaRA@mail.gmail.com> <5ED8BADE-7B2C-4B73-93BC-70739911C5E3@sorbs.net> <d0118f7e-7cfc-8bf1-308c-823bce088039@denninger.net> <2e4941bf-999a-7f16-f4fe-1a520f2187c0@sorbs.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Brief "Old Man" summary/perspective here... Computers and hard drives are complex, sensitive physical things. They,=20 or the data on them, can be lost to fire, flood, lightning strikes, theft,= =20 transportation screw-ups, and more. Mass data corruption by faulty=20 hardware or software is mostly rare, but does happen. Then there's the=20 users - authorized or not - who are inept or malicious. You can spent a fortune to make loss of the "live" data in your home=20 server / server room / data center very unlikely. Is that worth the time= =20 and money? Depends on the business case. At any scale, it's best to have= =20 a manager - who understands both computers and the bottom line - keep a=20 close eye on this. "Real" protection from data loss means multiple off-site and generally=20 off-line backups. You could spend a fortune on that, too...but for your=20 use case (~21TB in an array that could hold ~39TB, and what sounds like a= =20 "home power user" budget), I'd say to put together two "backup servers" -= =20 cheap little (aka transportable) FreeBSD systems with, say 7x6GB HD's,=20 raidz1. With even a 1Gbit ethernet connection to your main system, savvy= =20 use of (say) rsync (net/rsync in Ports), and the sort of "know your data /= =20 divide & conquer" tactics that Karl mentions, you should be able to=20 complete initial backups (on both backup servers) in <1 month. After that= =20 - rsync can generally do incremental backups far, far faster. How often=20 you gently haul the backup servers to/from your off-site location(s)=20 depends on a bunch of factors - backup frequency, cost of bandwidth, etc. Never skimp on power supplies. -Walter [Credits: Nothing above is original. Others have already made most of my= =20 points in this thread. It's pretty much all decades-old computer wisdom=20 in any case.] On Tue, 30 Apr 2019, Michelle Sullivan wrote: > Karl Denninger wrote: >> On 4/30/2019 05:14, Michelle Sullivan wrote: >>>> On 30 Apr 2019, at 19:50, Xin LI <delphij@gmail.com> wrote: >>>>> On Tue, Apr 30, 2019 at 5:08 PM Michelle Sullivan <michelle@sorbs.net= >=20 > wrote: >>>>> but in my recent experience 2 issues colliding at the same time resul= ts=20 > in disaster >>>> Do we know exactly what kind of corruption happen to your pool? If yo= u=20 > see it twice in a row, it might suggest a software bug that should be=20 > investigated. >>>> >>>> All I know is it=E2=80=99s a checksum error on a meta slab (122) and f= rom what=20 > I can gather it=E2=80=99s the spacemap that is corrupt... but I am no exp= ert. I=20 > don=E2=80=99t believe it=E2=80=99s a software fault as such, because this= was cause by a=20 > hard outage (damaged UPSes) whilst resilvering a single (but completely= =20 > failed) drive. ...and after the first outage a second occurred (same as = the=20 > first but more damaging to the power hardware)... the host itself was not= =20 > damaged nor were the drives or controller. >> ..... >>>> Note that ZFS stores multiple copies of its essential metadata, and in= my=20 > experience with my old, consumer grade crappy hardware (non-ECC RAM, with= =20 > several faulty, single hard drive pool: bad enough to crash almost monthl= y=20 > and damages my data from time to time), >>> This was a top end consumer grade mb with non ecc ram that had been=20 > running for 8+ years without fault (except for hard drive platter failure= s.).=20 > Uptime would have been years if it wasn=E2=80=99t for patching. >> Yuck. >> >> I'm sorry, but that may well be what nailed you. >> >> ECC is not just about the random cosmic ray. It also saves your bacon >> when there are power glitches. > > No. Sorry no. If the data is only half to disk, ECC isn't going to save= =20 > you at all... it's all about power on the drives to complete the write. >> >> Unfortunately however there is also cache memory on most modern hard >> drives, most of the time (unless you explicitly shut it off) it's on for >> write caching, and it'll nail you too. Oh, and it's never, in my >> experience, ECC. > > No comment on that - you're right in the first part, I can't comment if= =20 > there are drives with ECC. > >> >> In addition, however, and this is something I learned a LONG time ago >> (think Z-80 processors!) is that as in so many very important things >> "two is one and one is none." >> >> In other words without a backup you WILL lose data eventually, and it >> WILL be important. >> >> Raidz2 is very nice, but as the name implies it you have two >> redundancies. If you take three errors, or if, God forbid, you *write* >> a block that has a bad checksum in it because it got scrambled while in >> RAM, you're dead if that happens in the wrong place. > > Or in my case you write part data therefore invalidating the checksum... >> >>> Yeah.. unlike UFS that has to get really really hosed to restore from= =20 > backup with nothing recoverable it seems ZFS can get hosed where issues o= ccur=20 > in just the wrong bit... but mostly it is recoverable (and my experience = has=20 > been some nasty shit that always ended up being recoverable.) >>> >>> Michelle >> Oh that is definitely NOT true.... again, from hard experience, >> including (but not limited to) on FreeBSD. >> >> My experience is that ZFS is materially more-resilient but there is no >> such thing as "can never be corrupted by any set of events." > > The latter part is true - and my blog and my current situation is not=20 > limited to or aimed at FreeBSD specifically, FreeBSD is my experience.= =20 > The former part... it has been very resilient, but I think (based on=20 > this certain set of events) it is easily corruptible and I have just=20 > been lucky. You just have to hit a certain write to activate the issue,= =20 > and whilst that write and issue might be very very difficult (read: hit= =20 > and miss) to hit in normal every day scenarios it can and will=20 > eventually happen. > >> Backup >> strategies for moderately large (e.g. many Terabytes) to very large >> (e.g. Petabytes and beyond) get quite complex but they're also very >> necessary. >> > and there in lies the problem. If you don't have a many 10's of=20 > thousands of dollars backup solutions, you're either: > > 1/ down for a looooong time. > 2/ losing all data and starting again... > > ..and that's the problem... ufs you can recover most (in most=20 > situations) and providing the *data* is there uncorrupted by the fault=20 > you can get it all off with various tools even if it is a complete=20 > mess.... here I am with the data that is apparently ok, but the=20 > metadata is corrupt (and note: as I had stopped writing to the drive=20 > when it started resilvering the data - all of it - should be intact...=20 > even if a mess.) > > Michelle > > --=20 > Michelle Sullivan > http://www.mhix.org/ > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Tue Apr 30 16:13:22 2019 Return-Path: <owner-freebsd-stable@freebsd.org> Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BCB2815969AF for <freebsd-stable@mailman.ysv.freebsd.org>; Tue, 30 Apr 2019 16:13:21 +0000 (UTC) (envelope-from cindy.allan@webbasedata.com) Received: from n1nlsmtp03.shr.prod.ams1.secureserver.net (n1nlsmtp03.shr.prod.ams1.secureserver.net [188.121.43.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "relay-hosting.secureserver.net", Issuer "Starfield Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A735881500 for <freebsd-stable@freebsd.org>; Tue, 30 Apr 2019 16:13:20 +0000 (UTC) (envelope-from cindy.allan@webbasedata.com) Received: from n3plcpnl0275.prod.ams3.secureserver.net ([160.153.155.31]) by : HOSTING RELAY : with ESMTP id LV4IhPWnzRzlGLV4IhFv4D; Tue, 30 Apr 2019 08:53:18 -0700 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=webbasedata.com; s=default; h=Content-Type:MIME-Version:Message-ID:Date: Subject:To:From:Reply-To:Sender:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=9nJhgVjHEX9Ovot1f30bCkj+WZl+Mt7N8eH0nSNTUOg=; b=xg8Q61Jc4ExHXYlnAScgrEf2Wr hR6i4sI+RLvq6EeI1z5+bCq+kqFzrkm/Blqq/uNSYd+4y9aHET04+wYid/PWLdcbrXPKM4BR87B5Q dT/RR/ZXBk8ZPqO9OFYvcQdo8iW6r7+g0JQwqCV4xSqqz7bx7tTxUqSLMQ4pT1KPskcQRlaLATaG9 h29DVT39ZUjYZk233HZpKz7BnHGI+5/d5c62q19+wfuewCuSrvbDnsKXveanWnAFaSLEJbEuM9Aoe hGnoIcSnrEocg0GyF2Z8xAMKTOxX7tOmmQl8HER0m108MsJ5xMPKbX+0ZHO+6BzWVy+MIW73sNhV8 Xcmynm3w==; Received: from [139.99.99.151] (port=50205 helo=WS40) by n3plcpnl0275.prod.ams3.secureserver.net with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-SHA384:256) (Exim 4.91) (envelope-from <cindy.allan@webbasedata.com>) id 1hLTyT-00GXrI-A5 for freebsd-stable@freebsd.org; Tue, 30 Apr 2019 07:43:14 -0700 Reply-To: <cindy.allan@webbasedata.com> From: "Cindy Allan" <cindy.allan@webbasedata.com> To: <freebsd-stable@freebsd.org> Subject: Nginx Users List. Date: Tue, 30 Apr 2019 20:12:08 +0530 Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAAMnCNLY+8m5DqJeNa2mWxprCgAAAEAAAAPCyBdjlMIZOhXcZm9+rg8kBAAAAAA==@webbasedata.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 15.0 Thread-Index: AdT/YuKTCdF7Wfz5R1uOQGEbfY4MSg== Content-Language: en-us X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - n3plcpnl0275.prod.ams3.secureserver.net X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - webbasedata.com X-Get-Message-Sender-Via: n3plcpnl0275.prod.ams3.secureserver.net: authenticated_id: cindy.allan@webbasedata.com X-Authenticated-Sender: n3plcpnl0275.prod.ams3.secureserver.net: cindy.allan@webbasedata.com X-Source: X-Source-Args: X-Source-Dir: X-CMAE-Envelope: MS4wfPf4yWhLhGF3dyQ1D2nD8MjRCdYfI3ZRAjmse0YDRvpDFqfNLDNJpLIeLXhhr1bKdw6NxZf1lCEwjS6t3jn6WoTT55WcIpO9mlAu2KsFdlir36LnQO2q TTOz6/zY78c6VMV9PMdt1KvPTxCWzd6uI3lXfH6yf52BfZDQYurX/FCiCSL/7hPSPrCcmdk62iHZAyUvkFY/6mvgkMZz++l5x+usK4y1PcCoelFdGdR23UlG X-Rspamd-Queue-Id: A735881500 X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none (invalid DKIM record) header.d=webbasedata.com header.s=default header.b=xg8Q61Jc X-Spamd-Result: default: False [2.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_REPLYTO(0.00)[cindy.allan@webbasedata.com]; MX_INVALID(0.50)[cached]; HAS_X_SOURCE(0.00)[]; REPLYTO_ADDR_EQ_FROM(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[webbasedata.com:~]; HAS_X_ANTIABUSE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; IP_SCORE(0.37)[ip: (1.21), ipnet: 188.121.40.0/22(0.49), asn: 26496(0.22), country: US(-0.06)]; ASN(0.00)[asn:26496, ipnet:188.121.40.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; HAS_X_AS(0.00)[cindy.allan@webbasedata.com]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.17)[0.171,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[webbasedata.com]; NEURAL_SPAM_MEDIUM(0.35)[0.352,0]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_LONG(0.91)[0.907,0]; RCVD_IN_DNSWL_NONE(0.00)[193.43.121.188.list.dnswl.org : 127.0.5.0]; R_DKIM_PERMFAIL(0.00)[webbasedata.com:s=default]; R_SPF_NA(0.00)[]; HAS_X_GMSV(0.00)[cindy.allan@webbasedata.com]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>; List-Post: <mailto:freebsd-stable@freebsd.org> List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>, <mailto:freebsd-stable-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 30 Apr 2019 16:13:22 -0000 Hi, I would like to know if you would be interested in acquiring Nginx Users List. Information fields: First Name, Last Name, Title, Email Address, Phone Number, Company Name, Company URL, Company physical address, SIC Code, Industry, Company Size (Revenue and Employee). We can also help you with: Apache HTTP Server, Docker, Gunicorn, HAProxy, Redis, Squid (software), Apache Tomcat, Internet Information Services, uWSGI, lighttpd, Node.js, PHP, PostgreSQL, MySQL, Jenkins (software), CentOS, Ubuntu, Hypertext Transfer Protocol, Django, OpenSSL, MariaDB, WordPress, Laravel and many more. For further information on Cost and Counts let me know your Target Geography: ___________. Note: We would provide any industry, any job title of contact lists as per your requirements. Kind Regards, Cindy Allan Sr. Marketing Manager To quit accepting future messages, it would be ideal if you answer with "Withdraw" in the Subject Line.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190430102024.E84286>