From owner-freebsd-stable@freebsd.org Mon May 30 19:46:53 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0408B55876 for ; Mon, 30 May 2016 19:46:53 +0000 (UTC) (envelope-from richard.huber@computeroxy.com) Received: from cloudserver071065.home.net.pl (cloudserver071065.home.net.pl [79.96.79.199]) by mx1.freebsd.org (Postfix) with SMTP id DD0A51864 for ; Mon, 30 May 2016 19:46:52 +0000 (UTC) (envelope-from richard.huber@computeroxy.com) Received: from cloudserver071065.home.net.pl (79.96.79.199) (HELO www.computeroxy.com) by serwer1571831.home.pl (79.96.79.199) with SMTP (IdeaSmtpServer v0.80.2) id 8a079fab73e72c57; Mon, 30 May 2016 21:40:09 +0200 Date: Mon, 30 May 2016 21:40:09 +0200 To: freebsd-stable@freebsd.org From: Richard Huber Subject: Recently posted academic job vacancies at Computeroxy Message-ID: X-Mailer: PHPMailer 5.2.14 (https://github.com/PHPMailer/PHPMailer) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 May 2016 19:46:53 -0000 Computeroxy =20 Dear Colleague, We are pleased to present you our specialised newsletter including your=0Aa= cademic job vacancies in schools of computer, electrical and=0Amathematical= sciences and engineering recently posted=0Aat=C2=A0Computeroxy.com=C2= =A0worldwide. Austria Professorship "Foundations of Data Science with a focus on Big Data=0AManag= ement" PRIORITY! Graz University of Technology PhD Position in Machine Learning=20 AIT Austrian Institute of Technology, Department of Technology PhD Student in Computer Science=20 University of Innsbruck, Department of Computer Science Belgium Postdoc Position in Human-Computer Interaction=20 KU Leuven, Department of Computer Science Canada Nanotechnology, Tenure-track Assistant Professor of Chemistry PRIORITY! University of Waterloo Tenure-track position in Security of Cyber-Physical Systems PRIORITY! Concordia University, Faculty of Engineering and Computer Science PhD in Process Automation Technology; Instrumentation and Control,=0AElectr= ical Engineering, Mechatronics PRIORITY! McMaster University, School of Engineering Technology China Post-Doctoral Fellows in Simulation and Application of Software=0ADepartmen= t PRIORITY! Peking University, Beijing Innovation Center for Engineering Science and=0A= Advanced Technology Post-Doctoral Fellows in Energy Technology Department PRIORITY! Peking University, Beijing Innovation Center for Engineering Science and=0A= Advanced Technology Faculty Positions in Robotics PRIORITY! Peking University, Department of Mechanics and Engineering Science=0AColleg= e of Engineering Faculty Positions in Computational mechanics PRIORITY! Peking University, Department of Mechanics and Engineering Science=0AColleg= e of Engineering Postdoctoral and Senior Research Fellow Positions PRIORITY! Tsinghua University, Department of Electronic Engineering Research Fellow in Microsystems=20 Shanghai Institute of Microsystem and Information Technology Professor of Mathematics and Statistics=20 Zhejiang University, School of Mathematical Sciences Professor of Electronic Engineering=20 Zhejiang University, College of Information Science and Electronic=0AEngine= ering Associate Professor of Computer Science=20 University of Nottingham in Ningbo China, School of Computer Science Professor of Marine Robotics=20 Zhejiang University, Ocean College Associate Professor of Financial Mathematics=20 Xi'an Jiaotong, Department of Mathematical Sciences Estonia ERA Chair holder in Cognitive Electronics (Cognitronics) PRIORITY! Tallinn University of Technology, Department of Electronics, Faculty of=0AI= nformation Technology Fiji Associate Professor of Computing Science=20 University of the South Pacific, Department of Computer Science Finland Early Stage Researcher in Technology=20 University of Eastern Finland , Department of IT France Head of Networking and Cybersecurity Department PRIORITY! Telecom Bretagne PhD Position in Computer Science=20 CEA Tech, Department of Computer Science PhD Position in Computer Science=20 CEA Tech, Department of Computer Science Germany PhD Positions in Mathematical Sciences PRIORITY! Bielefeld University PhD Positions in Control and Optimization of Complex Systems=20 Chemnitz University of Technology Research Associate in Wireless Bio-robotic System Integration =20 Ovesco Endoscopy AG Postdoctoral Fellow in IT=20 Professorship of Dependable Software Systems Professorship of Computer Science=20 University of Applied Sciences Zwickau=20 Post-Doc in Hearing-Impaired Signal Processing=20 Carl von Ossietzky University of Oldenburg PhD Position in Computer Science=20 University of Siegen, Department of Computer Science PhD Positions in Control and Optimization of Complex Systems=20 Chemnitz University of Technology, Department of Computer Science Hungary Head of Data Science and Engineering Department PRIORITY! Faculty of Informatics Faculty Position in Network=20 Holografika, Department of Computer Research Iraq Assistant Professor of Electronics=20 University of Kurdistan Hewl=C3=AAr, Department of Engineering Assistant Professor of Computer Sciences=20 University of Kurdistan Hewl=C3=AAr, Department of Engineering Italy Junior Professors of Information Processing Systems=20 Free University of Bozen, Department of IT Research Assistant in Information Processing Systems=20 Free University of Bozen, Department of IT Japan Faculty Position in Artificial Intelligence=20 Keio University, Faculty of Environment and Information Studies Faculty Position in Electrical Engineering=20 Kyushu University, Department of Engineering Associate Professor of Mathematics=20 Okayama University, Department of Mathematics Faculty Position in Cyber Security=20 Keio University, Faculty of Environment and Information Studies Faculty Position in Computer Science=20 Keio University, Faculty of Environment and Information Studies Faculty Position in Data Science=20 Keio University Associate Professor of Technology=20 Yamaguchi University, Graduate School of Sciences and Technology for=0AInno= vation Luxembourg PhD Position in Context-Sensitive Security=20 University of Luxembourg, Department of Computer Science Research Associate in Wireless Communications & Networking=20 University of Luxembourg, Department of Computer Science Research Associates in Computer Vision=20 University of Luxembourg, Department of Computer Science PhD Candidates in Computer Vision=20 University of Luxembourg, Department of Computer Science Research Scientist in Software Engineering =20 University of Luxembourg, Department of Computer Science Research Scientist in Software Engineering =20 University of Luxembourg, Department of Computer Science Mexico Full Time Professor of Mathematics=20 Universidad de Monterrey, Department of Mathematics Netherlands Two PhD positions in the area of software engineering/software evolution=0A= PRIORITY! Eindhoven University of Technology PhD Studentship in Nano-Optomechanics=20 Eindhoven University of Technology, Department of Nanotechnology PhD Studentship in Nano-Optomechanics=20 Eindhoven University of Technology, Department of Nanotechnology New Zealand Professor in Sustainable Energy Systems PRIORITY! Victoria University Faculty of Engineering Professor in Cyber Security PRIORITY! Victoria University of Wellington, Faculty of Engineering Professor of Information System=20 Auckland University of Technology Lecturer/Senior Lecturer in Information Technology=20 Massey University, School of Engineering and Advanced Technology Lecturer in Mathematics =20 University of Auckland, Faculty of Science Norway Postdoctoral Fellowship in applied and Computational Mathematics=20 University of Bergen, Department of Mathematics=20 PhD Position in Parameterized / Multivariate Algorithmics =20 University of Bergen, Department of Informatics=20 Saudi Arabia Post-Doctoral Fellow in Electrical Engineering PRIORITY! King Abdullah University of Science and Technology (KAUST) Assistant, Associate, and Full Professor positions in Industrial=0AEngineer= ing, Operations Research, and Control & Instrumentation =0APRIORITY! King Fahd University of Petroleum & Minerals South Africa Senior Lecturer in Computer Science=20 University of KwaZulu-Natal, College of Agriculture, Engineering and=0AScie= nce Associate Lecturer in Actuarial Science=20 University of the Witwatersrand, Department of Mathematics Sweden PhD Student in Informatics=20 University of Sk=C3=B6vde, School of Informatics Senior Lecturer in Computer Science=20 Linnaeus University, Department of Computer Science Senior Lecturer in Computer Science With Specialisation in Software=0AEngi= neering=20 Linnaeus University, Department of Computer Science Professor of Software Technology=20 KTH Royal Institute of Technology, School of Technology Senior Lecturer in Computer Science With Specialisation in Embedded=0ASyst= ems=20 Linnaeus University, Department of Computer Science Switzerland Research or Postdoctoral Fellow in modeling 2G to 5G radio communication=0A= systems PRIORITY! HEFR Switzerland, La Haute =C3=A9cole d=E2=80=99ing=C3=A9nierie et d= =E2=80=99architecture de=0AFribourg (HEIA-FR) Dozentin/Dozenten fuer Embedded Software Engineering PRIORITY! HSR Hochschule fuer Technik Rapperswil Responsable des fili=C3=A8res Informatique et T=C3=A9l=C3=A9communications = et=0Aprofesseur-e dans le domaine TIC PRIORITY! La Haute =C3=A9cole d=E2=80=99ing=C3=A9nierie et d=E2=80=99architecture de = Fribourg=0A(HEIA-FR) Postdoctoral Fellow in IT=20 School of Engineering and Architecture of Fribourg=20 United Arab Emirates Faculty Position in Information Technology=20 Abu Dhabi University, Department of Computer Engineering Chair in Computer Information Science=20 Higher Colleges of Technology, Department of Computer Science United Kingdom Lecturers/Senior Lecturers in Computer Networks and/or Information=0ASystem= s Security PRIORITY! Sheffield Hallam University Lecturers/Senior Lecturers in Computing PRIORITY! Sheffield Hallam University Reader in Informatics and Security PRIORITY! Sheffield Hallam University Lecturer/Senior Lecturer in Ethical Hacking and Cybersecurity=20 Coventry University, Faculty of Engineering Research Fellow in Computer Science=20 University of St Andrews, School of Computer Science Research Fellow in Automated Software Engineering=20 University of Birmingham, School of Computer Science Faculty Position in Computer Science=20 University of Reading, National Centre for Atmospheric Science Research Associate in Software=20 University of Liverpool, Faculty of Science and Engineering Lecturer in Data Science=20 Lancaster University, School of Computing and Communications Lecturer / Senior Lecturer in Information Systems =20 Birmingham City University, Digital Media Technology Research Fellow in Computer Vision=20 University of Surrey, Department of Electrical & Electronic Engineering Lecturer in Enterprise Systems Engineering=20 Birmingham City University, Department of Enterprise Systems Postdoctoral Research Assistant in Electronic Engineering=20 Queen Mary University of London, Department of Electronic Engineering &=0AC= omputer Science Research Associates in Computer Science=20 University of Manchester, Faculty of Engineering & Physical Sciences=20 Postdoctoral Research Assistant in Computer Vision=20 University of Oxford, Department of Engineering Science Chair in Trusted Systems and Software Engineering=20 University of Leeds, School of Computing Research Associate in Spoken Dialogue Systems=20 Heriot-Watt University, School of Mathematical and Computer Sciences Research Assistant/Associate in Cyber Risk=20 University of Cambridge, Judge Business School Lecturer in Combinatorics=20 University of Strathclyde, Department of Computer and Information=0AScience= s Associate Professor of Computer Games Development=20 University Campus Suffolk, Department of Science and Technology PhD Student Position in IoT Security and Privacy=20 University of Surrey, Department of Computer Science Lecturer/Senior Lecturer in Computer Science=20 University of Birmingham, School of Computer Science Lecturer in Mathematics =20 University of Dundee,Department of Mathematics Lecturer in Statistics=20 University of Kent, School of Mathematics, Statistics and Actuarial=0AScien= ce Lecturer in Statistics=20 Coventry University, Department of Mathematics Research Associate in Wireless Security of Connected Autonomous Vehicles=0A University of Bristol, Department of Electrical & Electronic Engineering Postdoctoral Research Assistant in Quantum Effects in Electronic=0ANanodevi= ces=20 University of Oxford, Department of Materials Lecturer / Senior Lecturer in Computer Networks =20 Sheffield Hallam University, Department of Computing Lecturers / Senior Lecturers in Computing=20 Sheffield Hallam University, Department of Computing PhD Position in Mathematics=20 University of Kent, Department of Mathematics United States Senior Communication Systems Researcher PRIORITY! Sharp Laboratories of America (SLA) Senior Java Engineer PRIORITY! Sharp Laboratories of America (SLA) Senior Video Coding Researcher PRIORITY! Sharp Laboratories of America (SLA) Assistant or Associate Professor of Biostatistics and Bioinformatics =0APRI= ORITY! University of Mississippi Medical Center, Center of Biostatistics and=0ABio= informatics Bioinformatics Faculty Leader PRIORITY! University of Mississippi Medical Center, Center of Biostatistics and=0ABio= informatics Jackson Heart Study Associate Director of Data Management PRIORITY! University of Mississippi Medical Center, Center of Biostatistics and=0ABio= informatics Visiting Assistant Professor of Maths=20 Widener University, Department of Mathematics Associate Teaching Professor of Computer Engineering=20 Drexel University, Department of Computer Engineering Adjunct Faculty Position of Information Technologies=20 Marion Technical College, Department of IT Associate Professor of Technology=20 Stevens Institute of Technology PhD Scholarships in Computer Science=20 IMT Institute of Advanced Studies, Department of Computer Science Visiting Professor of Computer Science=20 Cleveland State University, Department of Computer Science Instructor in Software Engineering=20 Keiser University, Department of Software Engineering Assistant Professor of Electrical Engineering=20 Keiser University, Department of Software Engineering Non-Tenure Track Position in Software Engineering =20 Pennsylvania State University, Department of Software Engineering Instructor in Computer & Information Science=20 Walters State Community College, Department of Computer Engineering Faculty Full Time Position in Information and Engineering Technology =20 Prince George's Community College, Department of Engineering Technology Assistant Professor of Power Systems and Power Electronics=20 Southern Illinois University Carbondale, Department of Electrical and=0ACom= puter Engineering=20 Adjunct Instructor in Electrical Engineering Technology=20 Gateway Technical College, Department of Electrical Engineering=0ATechnolog= y Instructor in Electronics Engineering Technology=20 Craven Community College Research Assistant Professor of Statistical Signal Processing=20 Barron Associates Assistant/Associate Professor of Electrical Engineering=20 Arkansas Tech University, Department of Electrical Engineering Faculty Assistant professor of Electrical Engineering=20 University of New Haven, Department of Electrical & Computer Engineering Adjunct Professor of Electrical Engineering =20 Erie Community College, Department of Electrical Engineering=20 Non-Tenure Track Faculty Position in Electrical Engineering =20 Pennsylvania State University, Department of Electrical Engineering Postdoctoral Scholar in Electrical Engineering=20 University of California, Department of Electrical Engineering Instructor in Mathematics=20 Keiser University, Department of Mathematics Adjunct Faculty Position in Mathematics=20 Iona College, Department of Mathematics Assistant/Associate Professor of Mathematics=20 North Greenville University, Department of Mathematics Professor of Mathematics=20 Arizona Western College, Department of Mathematics Instructor in Mathematics=20 Keiser University, Department of Mathematics Visiting Assistant Professor of Mathematics=20 Champlain College, Department of Mathematics Adjunct Assistant Professor of Mathematics=20 Johnson County Community College, Department of Mathematics Adjunct Instructor in Math=20 Newbury College, Department of Mathematics Instructor in Mathematics=20 Northern Marianas College, Department of Mathematics Adjunct Instructor in Maths=20 Marymount California University, Division of Mathematics Full Time Faculty Position in Mathematics=20 Massasoit Community College, Division of Mathematics Head of Electronics Engineering=20 Savannah Technical College Research Assistant Professor of Computer Science=20 University of Houston, Department of Computer Science Visiting Assistant Professor of Computer Science=20 Eastern Kentucky University, Department of Computer Science Professor of Computer Science=20 Texas A&M University, Department of Computer Science Instructor in Computer Science=20 Garden City Community College, Department of Computer Science Visiting Asst Professor/Instructor in Computer Science=20 Miami University, Department of Computer Science Instructor in Computer Science=20 Texas Southmost College, Department of Computer Science Lecturer in Information Systems=20 University of Nevada, Department of Computer Science Faculty Position in Computer Science=20 Texas Southmost College, Department of Computer Science Postdoctoral Research Associate in Computer Science=20 Northeastern University, Department of Computer Science Faculty Position in Computer Programming=20 University of Houston, Department of Computer Science Lecturer in Computer Science=20 Princeton University, Department of Computer Science Assistant Professor of Computer Science=20 Marywood University, Depatment of Computer Science Instructor in Computer Science=20 University of Memphis, Depatment of Computer Science Chair in Software Engineering=20 Iowa State University=20 and many more at Computeroxy.com To learn more about these and other job vacancies, we invite you to=0Avisit= our website www.computeroxy.com and/or to "register"=C2=A0and/or to=0A"con= tact us". Want to post a job vacancy? Attract the attention of our academic=0Aaudienc= e of more than 320.000 professors, lecturers, researchers and=0Aacademic ma= nagers who are at present employed in the=0Ahighest-ranking=C2=A0schools of= computer, electrical and mathematical=0Asciences and engineering worldwide= and who receive our specialised=0Anewsletters twice a month "post a job va= cancy on Computeroxy"=C2=A0and/or to=0A"contact us". Yours faithfully Richard Huber PhD =09COMPUTEROXY Your Academic Vacancies in Schools of Computer, Electrical and=0AMathematic= al Sciences and Engineering Europe - Asia - The Americas - Oceania - The Middle East - Africa computeroxy@computeroxy.com www.computeroxy.comIf you do not wish to receive Computeroxy newsletters=0A= or have received it in error, you may unsubscribe by replying to=0AComputer= oxy with the word "unsubscribe" in the subject line. Thank you=0Avery much. From owner-freebsd-stable@freebsd.org Tue May 31 17:27:05 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3D0D0B58693 for ; Tue, 31 May 2016 17:27:05 +0000 (UTC) (envelope-from andrejj-krivonogov0@rambler.ru) Received: from huan2.mail.rambler.ru (huan2.mail.rambler.ru [81.19.66.15]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF284172D for ; Tue, 31 May 2016 17:27:04 +0000 (UTC) (envelope-from andrejj-krivonogov0@rambler.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=rambler.ru; s=mail; h=Content-Type:MIME-Version:Date:Subject:To:From:Reply-To:Message-ID; bh=AK+jhdSaKA1WVGjkFV8jDtlLW4KBovbiCx4bkPiQqEQ=; b=ZfVh8mfZ0XaGAo1iRlGiYnyUcqujMpOhmw6A3EPUmmKwy8hbtoubzwHQr9tO1qz0W9SG2Vbkhubin8gU9AyQqeQFPcEsEEiKpfk8GV9ILvvckl/5JhU5rsGkM1FIrCEKn6YwXvXJQXVx1HTwDoTIPP38X7ecInTGoWRpsIcQUBE=; Received: from [UNAVAILABLE] ([218.22.135.190]:43046 helo=t.su) by huan2.mail.rambler.ru with esmtpa (Exim 4.76) (envelope-from ) id 1b7nRV-0007vk-ED for freebsd-stable@freebsd.org; Tue, 31 May 2016 20:27:03 +0300 Message-ID: <06FCB0F2E6321CAF73763F81D552AB66@rambler.ru> Reply-To: =?koi8-u?Q?=E9=CE=C7=C1?= From: =?koi8-u?Q?=E9=CE=C7=C1?= To: Subject: =?koi8-u?Q?=FA=C4=D2a=D7=D3=D4=D7=D5=CA=D4e?= Date: Tue, 31 May 2016 21:28:31 +0400 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Windows Live Mail 14.0.8117.416 X-MimeOLE: Produced By Microsoft MimeOLE V14.0.8117.416 X-Rambler-User: andrejj-krivonogov0@rambler.ru/218.22.135.190 Content-Type: text/plain; charset="koi8-u" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 17:27:05 -0000 From owner-freebsd-stable@freebsd.org Tue May 31 18:02:17 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9ED78B562F1 for ; Tue, 31 May 2016 18:02:17 +0000 (UTC) (envelope-from smack@saraforlife.com) Received: from server5.ofiplus.com (server5.ofiplus.com [50.28.1.166]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 83D97146B for ; Tue, 31 May 2016 18:02:17 +0000 (UTC) (envelope-from smack@saraforlife.com) Received: from [127.0.0.1] (port=59540 helo=efbrrsoh) by server5.ofiplus.com with esmtp (Exim 4.87) (envelope-from ) id 1b7nzQ-000900-GQ for freebsd-stable@freebsd.org; Tue, 31 May 2016 12:02:04 -0600 Message-ID: 009901c4804d$f9a1cd55$917d37bc@smack@saraforlife.com From: "Amoxicillin Best online shop" To: freebsd-stable@freebsd.org Subject: Elizabeth Trazodone Price: USD0.55 Date: Tue, 31 May 2016 12:02:04 -0600 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - server5.ofiplus.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - saraforlife.com X-Get-Message-Sender-Via: server5.ofiplus.com: acl_c_authenticated_local_user: rhmanage X-Authenticated-Sender: server5.ofiplus.com: rhmanage X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 18:02:17 -0000 http://kingdomclover.ru/Fazal.php Elizabeth! Trazodone Price: USD0.55 Trazodone is a tetracyclic antidepressant medication. It is=20 thought to increase the activity of one of the brain chemicals=20 (serotonin) which may become unbalanced and cause depression. It=20 may also be used for relief of anxiety disorders (eg,=20 sleeplessness, tension) and chronic pain. Amoxicillin Best online shop From owner-freebsd-stable@freebsd.org Tue May 31 18:18:35 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 287B7B5AF2D for ; Tue, 31 May 2016 18:18:35 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 177641E05 for ; Tue, 31 May 2016 18:18:35 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 16C62B5AF27; Tue, 31 May 2016 18:18:35 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1661DB5AF26 for ; Tue, 31 May 2016 18:18:35 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (unknown [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 8266B1E00 for ; Tue, 31 May 2016 18:18:34 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [IPv6:2a02:2698:27:6a14:5c89:f90c:3f00:ecf6] ([IPv6:2a02:2698:27:6a14:5c89:f90c:3f00:ecf6]) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id u4VIIUF9025416 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Tue, 31 May 2016 23:18:31 +0500 (YEKT) (envelope-from emz@norma.perm.ru) To: stable@freebsd.org From: "Eugene M. Zheganin" Subject: HAST, zfs and local mirroring Message-ID: Date: Tue, 31 May 2016 23:18:32 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 18:18:35 -0000 Hi. I wat to start using HAST, I have two nodes and a pair of disk on each node. So I want to use HASt in an environment where each HAST resource would be mirrored. What is the preferred approach if I want to use ZFS on an end-device to avoid exsessive fscking, and, in the same time, I want to have some redundancy on a block level ? I see two possibility: HAST on a zvol of a mirrored pool, and a ZFS on a hast. But recently I heard that nested zfs (like zfs on zvol) is clamed unsupported. Futhermore, I have zfs on a geli on a zvol, and this solution proved itself to be very affected to livelocking - when disk i/o on a such fs is above some treshold, system is locking, and the only way out is to reset it. Should I chose geom_mirror to provide a device for HAST and the build ZFS on it ? Thanks. Eugene. From owner-freebsd-stable@freebsd.org Tue May 31 21:49:31 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A402B56BDF for ; Tue, 31 May 2016 21:49:31 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 0B85E12EA for ; Tue, 31 May 2016 21:49:31 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 0AEA0B56BDE; Tue, 31 May 2016 21:49:31 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A8F7B56BDD for ; Tue, 31 May 2016 21:49:31 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-it0-x22a.google.com (mail-it0-x22a.google.com [IPv6:2607:f8b0:4001:c0b::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CA4B812E9 for ; Tue, 31 May 2016 21:49:30 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-it0-x22a.google.com with SMTP id e62so72947499ita.1 for ; Tue, 31 May 2016 14:49:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=aJFoncY94aRdmzwmIIRBq9vElUn/q+jLUX/pk4JUb6o=; b=S7QlpJvyf4LCL7sSSf9e0waaG9mGZ5rSNKYFXyEqhsd/COK2HT1tLLePwmsRoytAQJ gO6d6gBRVv5yJX+IKqWQVBq038Ty+iVJwWjIjaTcHh+mm8NUZBQ/eGiYMuFcy/6laXDj Ro2Rcpj1CeLPRwoRBwRSXzJe8I8pmdVOlIAWHerxra3lY0G+CWwIpn3gFBlu9HhL+4Do lL9EOlw6QnbE6lIl0p41QCfIaxbrgtrHMef3vnHmPHKtr7DofPJ+YU43konFoPgYQLw/ TEtIG+523vuenmmwyGEWBZNRXTKDs45xVRFzvvulzs4ctLABscNAinbYCVILJOmaFQU7 josg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=aJFoncY94aRdmzwmIIRBq9vElUn/q+jLUX/pk4JUb6o=; b=dMNcf/kR6CkXoICNZi8SmrEGTudHTRdiylZv/cuNgHXZSaprJ6O2uewzZN6QI5ryXl v1vi9crHgPCPMd/oQ/OoPE3baSQSyuGfyKBmiW1lXbO6qt1/n7/TBx9N06oMZDTxMxCV QGGjz8MlOT2Dq+HJZyKVhHnNi7PsO6YMpxdOOK9siR3kE43+e1gxFH3B/GFRQblszlmf AHqSC9Yo3C//+P5A34AkNXFhTWuvoshQmGlKjlMS0C6RlHdOldQtolCk2s27/+K56YHd tdgBX5fku3KcoBie5NncGfWPbFG5bt9dyjaq/D8QsrOHxoagFjoHmMjJ6EYA1r1Wk+AI 5ioQ== X-Gm-Message-State: ALyK8tLhNEzEIyMvGSzGAUiQqGC80IEAhhICaQ9Uc1UlRhrIRUH+v9L1y9ANUk74tNrnAE0f9cKCG83xsdxWqw== MIME-Version: 1.0 X-Received: by 10.36.89.4 with SMTP id p4mr958320itb.44.1464731370208; Tue, 31 May 2016 14:49:30 -0700 (PDT) Received: by 10.107.182.214 with HTTP; Tue, 31 May 2016 14:49:30 -0700 (PDT) In-Reply-To: References: Date: Tue, 31 May 2016 14:49:30 -0700 Message-ID: Subject: Re: HAST, zfs and local mirroring From: Freddie Cash To: "Eugene M. Zheganin" Cc: stable@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 21:49:31 -0000 On Tue, May 31, 2016 at 11:18 AM, Eugene M. Zheganin wrote: > I wat to start using HAST, I have two nodes and a pair of disk on each > node. So I want to use HASt in an environment where each HAST resource > would be mirrored. What is the preferred approach if I want to use ZFS on > an end-device to avoid exsessive fscking, and, in the same time, I want t= o > have some redundancy on a block level ? I see two possibility: HAST on a > zvol of a mirrored pool, and a ZFS on a hast. But recently I heard that > nested zfs (like zfs on zvol) is clamed unsupported. Futhermore, I have z= fs > on a geli on a zvol, and this solution proved itself to be very affected = to > livelocking - when disk i/o on a such fs is above some treshold, system i= s > locking, and the only way out is to reset it. Should I chose geom_mirror = to > provide a device for HAST and the build ZFS on it ? > =E2=80=8BThe generally recommend way to do this is to create a HAST resourc= e out of 1 disk from each system, and then build the ZFS pool using the HAST resources as the "disks". That way, your ZFS pool is made up of 2 HAST devices in a mirror vdev. And each of the two HAST devices uses one disk from each server (total of four disks). =E2=80=8B --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-stable@freebsd.org Wed Jun 1 07:21:18 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26F80B60389 for ; Wed, 1 Jun 2016 07:21:18 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 123681065 for ; Wed, 1 Jun 2016 07:21:18 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 0DE30B60388; Wed, 1 Jun 2016 07:21:18 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D8A9B60387 for ; Wed, 1 Jun 2016 07:21:18 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (unknown [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 75FB51062 for ; Wed, 1 Jun 2016 07:21:17 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (pc846408.norma.com [IPv6:fd00::73d] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id u517LE01014882 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Wed, 1 Jun 2016 12:21:14 +0500 (YEKT) (envelope-from emz@norma.perm.ru) Subject: Re: HAST, zfs and local mirroring To: stable@freebsd.org References: From: "Eugene M. Zheganin" Message-ID: <574E8CEA.1010908@norma.perm.ru> Date: Wed, 1 Jun 2016 12:21:14 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2016 07:21:18 -0000 Hi. On 01.06.16 02:49, Freddie Cash wrote: > On Tue, May 31, 2016 at 11:18 AM, Eugene M. Zheganin > >wrote: > > I wat to start using HAST, I have two nodes and a pair of disk on > each node. So I want to use HASt in an environment where each HAST > resource would be mirrored. What is the preferred approach if I > want to use ZFS on an end-device to avoid exsessive fscking, and, > in the same time, I want to have some redundancy on a block level > ? I see two possibility: HAST on a zvol of a mirrored pool, and a > ZFS on a hast. But recently I heard that nested zfs (like zfs on > zvol) is clamed unsupported. Futhermore, I have zfs on a geli on a > zvol, and this solution proved itself to be very affected to > livelocking - when disk i/o on a such fs is above some treshold, > system is locking, and the only way out is to reset it. Should I > chose geom_mirror to provide a device for HAST and the build ZFS > on it ? > > > ​The generally recommend way to do this is to create a HAST resource > out of 1 disk from each system, and then build the ZFS pool using the > HAST resources as the "disks". > > That way, your ZFS pool is made up of 2 HAST devices in a mirror vdev. > > And each of the two HAST devices uses one disk from each server (total > of four disks). > ​ > > > But I don't need two HAST devices, I need one, but redundant on each server. Eugene. From owner-freebsd-stable@freebsd.org Wed Jun 1 13:23:08 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D030FB60F93 for ; Wed, 1 Jun 2016 13:23:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id BF3FF1E48 for ; Wed, 1 Jun 2016 13:23:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id BE9F0B60F92; Wed, 1 Jun 2016 13:23:08 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE4D2B60F91 for ; Wed, 1 Jun 2016 13:23:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 85D951E47 for ; Wed, 1 Jun 2016 13:23:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1b866u-00064c-DD; Wed, 01 Jun 2016 16:23:00 +0300 Date: Wed, 1 Jun 2016 16:23:00 +0300 From: Slawa Olhovchenkov To: "Eugene M. Zheganin" Cc: stable@freebsd.org Subject: Re: HAST, zfs and local mirroring Message-ID: <20160601132300.GA75625@zxy.spb.ru> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Jun 2016 13:23:08 -0000 On Tue, May 31, 2016 at 11:18:32PM +0500, Eugene M. Zheganin wrote: > Hi. > > I wat to start using HAST, I have two nodes and a pair of disk on each > node. So I want to use HASt in an environment where each HAST resource > would be mirrored. What is the preferred approach if I want to use ZFS > on an end-device to avoid exsessive fscking, and, in the same time, I > want to have some redundancy on a block level ? I see two possibility: > HAST on a zvol of a mirrored pool, and a ZFS on a hast. But recently I > heard that nested zfs (like zfs on zvol) is clamed unsupported. > Futhermore, I have zfs on a geli on a zvol, and this solution proved > itself to be very affected to livelocking - when disk i/o on a such fs > is above some treshold, system is locking, and the only way out is to > reset it. Should I chose geom_mirror to provide a device for HAST and > the build ZFS on it ? Only FS support changed data bypass FS layer is Files-11 ODS-2 level, may be hardware support required. Can you use ZFS mirror with one vdev local and other vdev by iSCSI? Every node using separate ZFS pool in this case. From owner-freebsd-stable@freebsd.org Thu Jun 2 07:12:42 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99066B63AAE for ; Thu, 2 Jun 2016 07:12:42 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 876661E94 for ; Thu, 2 Jun 2016 07:12:42 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 86C47B63AAC; Thu, 2 Jun 2016 07:12:42 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86663B63AA9 for ; Thu, 2 Jun 2016 07:12:42 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (unknown [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0802B1E93 for ; Thu, 2 Jun 2016 07:12:41 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (pc846408.norma.com [IPv6:fd00::73d] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id u527CZcq056404 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Thu, 2 Jun 2016 12:12:38 +0500 (YEKT) (envelope-from emz@norma.perm.ru) Subject: Re: HAST, zfs and local mirroring To: stable@freebsd.org References: <20160601132300.GA75625@zxy.spb.ru> From: "Eugene M. Zheganin" Message-ID: <574FDC63.6090100@norma.perm.ru> Date: Thu, 2 Jun 2016 12:12:35 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <20160601132300.GA75625@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 07:12:42 -0000 Hi. On 01.06.16 18:23, Slawa Olhovchenkov wrote: > > Only FS support changed data bypass FS layer is Files-11 ODS-2 level, > may be hardware support required. > > Can you use ZFS mirror with one vdev local and other vdev by iSCSI? > Every node using separate ZFS pool in this case. If you mean that I should distribute the HAST one-disk device via iSCSI and then use it as a half of zfs mirrored pool on each node, then I should ask how the pool will decide which half is more recent - local or iSCSI, after I will import it and the kernel will found that it's vdevs differs a lot ? And why should I prefere this overcomplicated scheme over the geom_mirror, which seems rather simple when comparing. Seems like I can point HAST to /dev/mirror/whatever device, right ? Thanks. Eugene. From owner-freebsd-stable@freebsd.org Thu Jun 2 12:27:31 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6490EB6586A for ; Thu, 2 Jun 2016 12:27:31 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 559AC14EE for ; Thu, 2 Jun 2016 12:27:31 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 51446B65869; Thu, 2 Jun 2016 12:27:31 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 50D5DB65868; Thu, 2 Jun 2016 12:27:31 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 190F714ED; Thu, 2 Jun 2016 12:27:31 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1b8Rih-000F0I-Vv; Thu, 02 Jun 2016 15:27:27 +0300 Date: Thu, 2 Jun 2016 15:27:27 +0300 From: Slawa Olhovchenkov To: stable@freebsd.org Cc: freebsd-security@freebsd.org Subject: unbound and ntp issuse Message-ID: <20160602122727.GB75625@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 12:27:31 -0000 Default install with local_unbound and ntpd can't be functional with incorrect date/time in BIOS: Unbound requred correct time for DNSSEC check and refuseing queries ("Jul 1 20:17:29 yellowrat unbound: [3444:0] info: failed to prime trust anchor -- DNSKEY rrset is not secure . DNSKEY IN") ntpd don't have any numeric IP of ntp servers in ntp.conf -- only symbolic names like 0.freebsd.pool.ntp.org, as result -- can't resolve (see above, about DNSKEY). IMHO, ntp.conf need to include some numeric IP of public ntp servers. # date Tue Jul 1 20:36:31 MSD 2008 From owner-freebsd-stable@freebsd.org Thu Jun 2 14:50:36 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 495ADB6398B for ; Thu, 2 Jun 2016 14:50:36 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 387F512DD for ; Thu, 2 Jun 2016 14:50:36 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 37E71B63989; Thu, 2 Jun 2016 14:50:36 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37900B63988 for ; Thu, 2 Jun 2016 14:50:36 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F1D8A12DC for ; Thu, 2 Jun 2016 14:50:35 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1b8Tx6-000IfD-3P; Thu, 02 Jun 2016 17:50:28 +0300 Date: Thu, 2 Jun 2016 17:50:28 +0300 From: Slawa Olhovchenkov To: "Eugene M. Zheganin" Cc: stable@freebsd.org Subject: Re: HAST, zfs and local mirroring Message-ID: <20160602145028.GC75625@zxy.spb.ru> References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <574FDC63.6090100@norma.perm.ru> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 14:50:36 -0000 On Thu, Jun 02, 2016 at 12:12:35PM +0500, Eugene M. Zheganin wrote: > Hi. > > On 01.06.16 18:23, Slawa Olhovchenkov wrote: > > > > Only FS support changed data bypass FS layer is Files-11 ODS-2 level, > > may be hardware support required. > > > > Can you use ZFS mirror with one vdev local and other vdev by iSCSI? > > Every node using separate ZFS pool in this case. > If you mean that I should distribute the HAST one-disk device via iSCSI > and then use it as a half of zfs mirrored pool on each node, then I > should ask how the pool will decide which half is more recent - local or > iSCSI, after I will import it and the kernel will found that it's vdevs > differs a lot ? And why should I prefere this overcomplicated scheme > over the geom_mirror, which seems rather simple when comparing. Seems > like I can point HAST to /dev/mirror/whatever device, right ? I am suggesting next setup: node0: own pool zroot0: mirror-0: local_disk0 remote-iscsi_disk1/1 local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1 node1: own pool zroot1: mirror-0: local_disk0 remote-iscsi_disk0/1 local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0 No HAST. Disks synced by ZFS over iSCSI. From owner-freebsd-stable@freebsd.org Thu Jun 2 07:38:46 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9DB21B648FB for ; Thu, 2 Jun 2016 07:38:46 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8D22F1D06 for ; Thu, 2 Jun 2016 07:38:46 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 88C8FB648FA; Thu, 2 Jun 2016 07:38:46 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85E50B648F9 for ; Thu, 2 Jun 2016 07:38:46 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (unknown [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F22411D05 for ; Thu, 2 Jun 2016 07:38:45 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (pc846408.norma.com [IPv6:fd00::73d] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id u527caoQ059058 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Thu, 2 Jun 2016 12:38:40 +0500 (YEKT) (envelope-from emz@norma.perm.ru) To: stable@freebsd.org From: "Eugene M. Zheganin" Subject: vt(4) and HP iLO Message-ID: <574FE27C.5080106@norma.perm.ru> Date: Thu, 2 Jun 2016 12:38:36 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 07:38:46 -0000 Hi. Recently I've updates some of my HP DL g6 servers to vt(4) console driver. These servers are equipped with lo100i IPMI management device (basically it mimics IPMI). After some time I connected to it's web-interface and opened the Virtual KVM tab, Then I saw this: http://static.enaza.ru/userupload/gyazo/88ff4d116877f5ec2f529a3485c8.png It's not a BMC freeze, it's live, It actually transmits changes. One can see the mouse cursor and the text updating. It's even possible to log in, only it's not readable. These artifacts start to appear as soon as the kernel starts to boot, - when in initial bootstrap/zfsloader - everything is okay, until the actual point of kernel booting. Yeah, this can be solved by setting the hw.vga.textmode=1 in the loader.conf, but in the same time this lo100i is able to display graphics, and it successfully does it. For example it shows: - it's own POST graphic splashscreen - Linux fb, when centos is booting - Solaris GUI and text console, which is using graphics sinec Solaris 11.0. - Windows GUI You can say that probably it's HP problem, and I can partially agree: everything is fine in graphic mode when using vt(4) on a IBM RSA or Supermicro IPMI. But this logic is transitive, so HP guys can easily say that it's FreeBSD problem, since their device is able to show other OS graphics. I'm writing this post just in case this fact isn't known. I personally love FreeBSd and it's new vt driver, but I think it probably needs some improvements. Thanks. Eugene. From owner-freebsd-stable@freebsd.org Thu Jun 2 19:49:20 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82508B65FC8 for ; Thu, 2 Jun 2016 19:49:20 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 69BF51B67 for ; Thu, 2 Jun 2016 19:49:20 +0000 (UTC) (envelope-from peter@rulingia.com) Received: by mailman.ysv.freebsd.org (Postfix) id 6558CB65FC7; Thu, 2 Jun 2016 19:49:20 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64F8EB65FC6 for ; Thu, 2 Jun 2016 19:49:20 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (vps.rulingia.com [103.243.244.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F35AF1B66 for ; Thu, 2 Jun 2016 19:49:19 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp59-167-167-3.static.internode.on.net [59.167.167.3]) by vps.rulingia.com (8.15.2/8.15.2) with ESMTPS id u52Jn3ra033157 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 3 Jun 2016 05:49:09 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id u52JmwQp084799 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 3 Jun 2016 05:48:58 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id u52JmuuX084798; Fri, 3 Jun 2016 05:48:56 +1000 (AEST) (envelope-from peter) Date: Fri, 3 Jun 2016 05:48:56 +1000 From: Peter Jeremy To: "Eugene M. Zheganin" Cc: stable@freebsd.org Subject: Re: HAST, zfs and local mirroring Message-ID: <20160602194856.GA84753@server.rulingia.com> References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="45Z9DzgjV8m4Oswq" Content-Disposition: inline In-Reply-To: <574FDC63.6090100@norma.perm.ru> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Jun 2016 19:49:20 -0000 --45Z9DzgjV8m4Oswq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2016-Jun-02 12:12:35 +0500, "Eugene M. Zheganin" wro= te: >differs a lot ? And why should I prefere this overcomplicated scheme >over the geom_mirror, which seems rather simple when comparing. Seems >like I can point HAST to /dev/mirror/whatever device, right ? Because using RAID of any sort under ZFS defeats a lot of the smarts in ZFS. In particular, you can no longer rely on scrub verifying that all your media content is valid. I've also had bad experiences with gmirror volumes silently getting out of sync on a crash. --=20 Peter Jeremy --45Z9DzgjV8m4Oswq Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJXUI2oXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIyN0Ux NkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs0N5oP/jBlrR3tlalCBVuA0VB5Z6M6 FgIUVsaPYZtTKjLXe9jDE1+bAkSc7W41pLaM7HgEerpshSI7BWoY08R06d3mtRgJ pmjObRATeQn/QGz2l+j5GDW+WSdZ1+ZHf8UUUlusBXxprWW7d4dyOK3iwwJLq3qC ack30bXRf4HSeAnuF7xWi7XJZBfYr8tBaFenzQsMn6/w/5CLdi1NaKZrcBX4tgel qDkcyra68grHErV2QOVdge2Zwj2SaWvylKl2cv9KH6zfX+vxI4oWljJxJpP6tYnt 76q/Wnsx29ZakOAGTTXixykLEnIJrY3AlqIAN/+wpUzmz/XKh5UqWo/YLssQWii3 F3a6GF2gWMu5ol6h1g6CaM8ripGyvQdf2RQw8Gy47BmEbggO9dBvYps8wiFeQgi5 J3Ui2TEw6BRFlm3nTNxmmTMXdD7AyJ4AAvOyZakypoNn0JgotAadNZY6SmeLNtUu yYSXNIGvzS+pUBCVcmumkiHmTUiLIWZ6B0WAJ/EOzFIn/ejh9GwvoSH90DnmZMdn 9BjBofnCxaGDaJgN+FT2RWmQSMHcR63p+JGCT4982xRn9u8S+2X77latGEYIMRs0 JL2hLheyJKMUWn3UbeJgBYZWElY++cD+CNoY+HGfEExLg/0JPcP+T7ddO3LXRiL4 eWCwqrW5IvUbvn3KSyUD =uZEu -----END PGP SIGNATURE----- --45Z9DzgjV8m4Oswq-- From owner-freebsd-stable@freebsd.org Fri Jun 3 08:40:58 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E04B7B6504A for ; Fri, 3 Jun 2016 08:40:58 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id CC45A11B9 for ; Fri, 3 Jun 2016 08:40:58 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: by mailman.ysv.freebsd.org (Postfix) id C7E60B65049; Fri, 3 Jun 2016 08:40:58 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C7899B65048 for ; Fri, 3 Jun 2016 08:40:58 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (unknown [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 42AFF11B5 for ; Fri, 3 Jun 2016 08:40:57 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (pc846408.norma.com [IPv6:fd00::73d] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id u538esNo012897 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Fri, 3 Jun 2016 13:40:54 +0500 (YEKT) (envelope-from emz@norma.perm.ru) Subject: Re: HAST, zfs and local mirroring To: stable@freebsd.org References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> <20160602145028.GC75625@zxy.spb.ru> From: "Eugene M. Zheganin" Message-ID: <57514296.7090906@norma.perm.ru> Date: Fri, 3 Jun 2016 13:40:54 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <20160602145028.GC75625@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 08:40:59 -0000 Hi. On 02.06.16 19:50, Slawa Olhovchenkov wrote: > I am suggesting next setup: > > node0: > own pool zroot0: mirror-0: local_disk0 > remote-iscsi_disk1/1 > local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1 > > node1: > own pool zroot1: mirror-0: local_disk0 > remote-iscsi_disk0/1 > local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0 > > > No HAST. > Disks synced by ZFS over iSCSI. But this way I will get two independent zfs pools (or I still didn't get it), one half of each will be stored on another machine. And I need something different - a continuously replicated disk resource, that would be available on both machines in case either will crash. Cluster filesystem would be fine, but as far as I know there's no such thing on FreeBSD, so I accept it will be unavailable on slave while mounted as read-write on the master. HAST looks like a thing thats fits my requirement, only that I wank it to be redundant inside each host too, and all the documentation shows examples without local redundancy. Thanks. Eugene. From owner-freebsd-stable@freebsd.org Fri Jun 3 09:05:11 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 922B4B6841B for ; Fri, 3 Jun 2016 09:05:11 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 80008124D for ; Fri, 3 Jun 2016 09:05:11 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 7BD37B6841A; Fri, 3 Jun 2016 09:05:11 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B76CB68418 for ; Fri, 3 Jun 2016 09:05:11 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (unknown [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E4926124C for ; Fri, 3 Jun 2016 09:05:10 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from bsdrookie.norma.com. (pc846408.norma.com [IPv6:fd00::73d] (may be forged)) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPS id u53957SH016213 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Fri, 3 Jun 2016 14:05:07 +0500 (YEKT) (envelope-from emz@norma.perm.ru) Subject: Re: HAST, zfs and local mirroring References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> <20160602145028.GC75625@zxy.spb.ru> To: stable@freebsd.org From: "Eugene M. Zheganin" Message-ID: <57514843.6010003@norma.perm.ru> Date: Fri, 3 Jun 2016 14:05:07 +0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.7.0 MIME-Version: 1.0 In-Reply-To: <20160602145028.GC75625@zxy.spb.ru> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 09:05:11 -0000 Hi, On 02.06.16 19:50, Slawa Olhovchenkov wrote: > > I am suggesting next setup: > > node0: > own pool zroot0: mirror-0: local_disk0 > remote-iscsi_disk1/1 > local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1 > > node1: > own pool zroot1: mirror-0: local_disk0 > remote-iscsi_disk0/1 > local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0 You message lead me to a though that I could use iSCSI to replicate the zfs pool from node1 to both iSCSI-provided disk on a node2 in a 4-way mirror, right ? Are there any obvious obstacles to this, that I don't see, considering the bandwith will be enough ? Thanks. Eugene. From owner-freebsd-stable@freebsd.org Fri Jun 3 10:17:09 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00144B66FF5 for ; Fri, 3 Jun 2016 10:17:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E238F113E for ; Fri, 3 Jun 2016 10:17:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id DDD18B66FF4; Fri, 3 Jun 2016 10:17:08 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD760B66FF3 for ; Fri, 3 Jun 2016 10:17:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A317B113D for ; Fri, 3 Jun 2016 10:17:08 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1b8mA5-000NAg-Jc; Fri, 03 Jun 2016 13:17:05 +0300 Date: Fri, 3 Jun 2016 13:17:05 +0300 From: Slawa Olhovchenkov To: "Eugene M. Zheganin" Cc: stable@freebsd.org Subject: Re: HAST, zfs and local mirroring Message-ID: <20160603101705.GD75625@zxy.spb.ru> References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> <20160602145028.GC75625@zxy.spb.ru> <57514296.7090906@norma.perm.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <57514296.7090906@norma.perm.ru> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 10:17:09 -0000 On Fri, Jun 03, 2016 at 01:40:54PM +0500, Eugene M. Zheganin wrote: > Hi. > > On 02.06.16 19:50, Slawa Olhovchenkov wrote: > > I am suggesting next setup: > > > > node0: > > own pool zroot0: mirror-0: local_disk0 > > remote-iscsi_disk1/1 > > local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1 > > > > node1: > > own pool zroot1: mirror-0: local_disk0 > > remote-iscsi_disk0/1 > > local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0 > > > > > > No HAST. > > Disks synced by ZFS over iSCSI. > But this way I will get two independent zfs pools (or I still didn't get > it), one half of each will be stored on another machine. Yes. > And I need > something different - a continuously replicated disk resource, that > would be available on both machines in case either will crash. I think this is imposible. All modern FS (cluster also) believe exclusive modifivation of on-disk data. Cluster FS just acts as services (but still believe exclusive access to HDD) > Cluster > filesystem would be fine, but as far as I know there's no such thing on > FreeBSD, so I accept it will be unavailable on slave while mounted as > read-write on the master. HAST looks like a thing thats fits my > requirement, only that I wank it to be redundant inside each host too, > and all the documentation shows examples without local redundancy. You can't touch local HDD HASTed to remote. From owner-freebsd-stable@freebsd.org Fri Jun 3 10:36:28 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A46AAB686A8 for ; Fri, 3 Jun 2016 10:36:28 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 92887103B for ; Fri, 3 Jun 2016 10:36:28 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 91DC3B686A7; Fri, 3 Jun 2016 10:36:28 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 91872B686A6 for ; Fri, 3 Jun 2016 10:36:28 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 551231039 for ; Fri, 3 Jun 2016 10:36:28 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1b8mSo-000Njj-HZ; Fri, 03 Jun 2016 13:36:26 +0300 Date: Fri, 3 Jun 2016 13:36:26 +0300 From: Slawa Olhovchenkov To: "Eugene M. Zheganin" Cc: stable@freebsd.org Subject: Re: HAST, zfs and local mirroring Message-ID: <20160603103626.GE75625@zxy.spb.ru> References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> <20160602145028.GC75625@zxy.spb.ru> <57514843.6010003@norma.perm.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <57514843.6010003@norma.perm.ru> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 10:36:28 -0000 On Fri, Jun 03, 2016 at 02:05:07PM +0500, Eugene M. Zheganin wrote: > Hi, > > On 02.06.16 19:50, Slawa Olhovchenkov wrote: > > > > I am suggesting next setup: > > > > node0: > > own pool zroot0: mirror-0: local_disk0 > > remote-iscsi_disk1/1 > > local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1 > > > > node1: > > own pool zroot1: mirror-0: local_disk0 > > remote-iscsi_disk0/1 > > local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0 > You message lead me to a though that I could use iSCSI to replicate the > zfs pool from node1 to both iSCSI-provided disk on a node2 in a 4-way > mirror, right ? Are there any obvious obstacles to this, that I don't > see, considering the bandwith will be enough ? No, just regular mirror. And yes, two independed pool. 10G link prefered. 1G link acts like old HDD, about 70MB/s transfer. Every transit switch degrade performance. From owner-freebsd-stable@freebsd.org Fri Jun 3 10:40:17 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D298FB68839 for ; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C0F4F120E for ; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: by mailman.ysv.freebsd.org (Postfix) id BC9C6B68838; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC442B68837 for ; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (constantine.ingresso.co.uk [IPv6:2a02:b90:3002:e550::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48098120C for ; Fri, 3 Jun 2016 10:40:17 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from dilbert.london-internal.ingresso.co.uk ([10.64.50.6] helo=dilbert.ingresso.co.uk) by constantine.ingresso.co.uk with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.86_2 (FreeBSD)) (envelope-from ) id 1b8mWR-000Ee0-5j for stable@freebsd.org; Fri, 03 Jun 2016 10:40:11 +0000 Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.87 (FreeBSD)) (envelope-from ) id 1b8mWR-0000uA-3H for stable@freebsd.org; Fri, 03 Jun 2016 11:40:11 +0100 To: stable@freebsd.org Subject: Re: HAST, zfs and local mirroring In-Reply-To: <57514843.6010003@norma.perm.ru> Message-Id: From: Pete French Date: Fri, 03 Jun 2016 11:40:11 +0100 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 10:40:17 -0000 > You message lead me to a though that I could use iSCSI to replicate the > zfs pool from node1 to both iSCSI-provided disk on a node2 in a 4-way > mirror, right ? Are there any obvious obstacles to this, that I don't > see, considering the bandwith will be enough ? I have spent a long time doing this kind of thing - always needing a pair of machines, 2 discs in each and 4-way mirroring. over the last decade weve used ggated/ggatec + gmirrors, iscsi + gmirror, iscsi + zfs and finally hast + zfs. Out of all that I have to say that hnast has been a godsend - its is by far the best of all the solutions. so my recommentation would be to hast the drives into two hast resources and zpool on top of that. yes, you have two hast devices, but why is that an issue ? what we also do is to add separate ether cards into the machines just dedicated to the hast traffic between them, so that is not a bottlneck. the reason hwy hast is so good compared to all the other network block evel solutions we have tried is that it behaves ell in the case when things fail. Theres nothing wrong with either ggatec or iscsi in the normal case, but in the past I had a lot of issues with the mirroring layer not understanding properly when the unerlying devices vanish due to crashes or other unexpected events. As the purpose of all of this is, preseumably, fault tolerance, then thats unfortunate. this may have chnaged, I havent tried these technologies out in a while, but with hast you dont ever see a device simply vanish rom underneath your zpool, and the whole thing continues to work reliably and smoothly until the remote node comes back up, whereupon it resyncs perfectly. really, go with hast + zpool unless you have a good reason not to... -pete. From owner-freebsd-stable@freebsd.org Fri Jun 3 15:13:23 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D1F46B69434 for ; Fri, 3 Jun 2016 15:13:23 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id BA3651E6A for ; Fri, 3 Jun 2016 15:13:23 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: by mailman.ysv.freebsd.org (Postfix) id B965EB69433; Fri, 3 Jun 2016 15:13:23 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B911AB69432 for ; Fri, 3 Jun 2016 15:13:23 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 53B4D1E67 for ; Fri, 3 Jun 2016 15:13:22 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221]) by hz.grosbein.net (8.14.9/8.14.9) with ESMTP id u53FD969033596 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 3 Jun 2016 17:13:11 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: peter@rulingia.com Received: from [10.58.0.10] (dadvw [10.58.0.10]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id u53FCxGB072386 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 3 Jun 2016 22:12:59 +0700 (KRAT) (envelope-from eugen@grosbein.net) Subject: Re: HAST, zfs and local mirroring To: Peter Jeremy References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> <20160602194856.GA84753@server.rulingia.com> Cc: stable@freebsd.org From: Eugene Grosbein Message-ID: <57519E77.7060507@grosbein.net> Date: Fri, 3 Jun 2016 22:12:55 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <20160602194856.GA84753@server.rulingia.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM autolearn=no version=3.3.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on hz.grosbein.net X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 15:13:23 -0000 03.06.2016 2:48, Peter Jeremy пишет: > On 2016-Jun-02 12:12:35 +0500, "Eugene M. Zheganin" wrote: >> differs a lot ? And why should I prefere this overcomplicated scheme >> over the geom_mirror, which seems rather simple when comparing. Seems >> like I can point HAST to /dev/mirror/whatever device, right ? > > Because using RAID of any sort under ZFS defeats a lot of the smarts > in ZFS. In particular, you can no longer rely on scrub verifying that > all your media content is valid. I've also had bad experiences with > gmirror volumes silently getting out of sync on a crash. gmirror or (gmirror+gjournal) ones? From owner-freebsd-stable@freebsd.org Fri Jun 3 16:38:10 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 236ACB68B9C for ; Fri, 3 Jun 2016 16:38:10 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 034A011EF for ; Fri, 3 Jun 2016 16:38:10 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 0297FB68B9A; Fri, 3 Jun 2016 16:38:10 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0239FB68B99 for ; Fri, 3 Jun 2016 16:38:10 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-qg0-x233.google.com (mail-qg0-x233.google.com [IPv6:2607:f8b0:400d:c04::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B1C0511EE for ; Fri, 3 Jun 2016 16:38:09 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-qg0-x233.google.com with SMTP id 93so14352469qgx.2 for ; Fri, 03 Jun 2016 09:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=vfOhTqyPc8/PpsI4QfWODiqXk7xuJzq6Yhw3YdLjmTY=; b=ipQeABY4h68ecFBdNc9LX/DMRnpAkxQYhQqREpG79HZMBzJp3shfdrR2OvUyOxdARD 8KxYvB8UwAE3CweXSHKqbwlDFoclGaUjITB8MO2ZFVhOfEUOGIytcmGvTkeaQMDTSBBq RnpG2Iih6MsHcftOEN5ZoayDgDZWMbzPLLc/hOfkWI0dcjLDhoyRkJYgZuZ+0JlKjj0u BDNuJfdaOWKsHdHhuRQEj6kxDOBUGdi0qPGC1Z5zRJn0cmMNHHDTkg8M8+yVACip2xEz 1iLXdPbWg4Nozr+zqpUY+lKHtV8NIhTTE7w1QMtNcwqATbttniZoSf+QMoH9FD2sjq+w 0bnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=vfOhTqyPc8/PpsI4QfWODiqXk7xuJzq6Yhw3YdLjmTY=; b=MyoXLQKS/0pk2+q5j4lgF1fYP68+mMMwOsZrisK4LBIhhlnZfLduN1/pvf6sGAF0RD 7nt0g/C9MsQloGmDNu1RSqQvOF7jE3tfbb3UFteoShRxnw+5Tm8W4QfleCDRKBzgz+lH MRBxRXXtfDzY7XnJS/KUg/M7IyvNPdlIh1KJs+qQuZUPXRiXMdxeqsRunOgvfZEC84Zm ZaVoYhyAot7lGjsizd5vA/Y3CYUy6yuuhZ6rRVaDOC8MbXu3jVq0E/J4ohMEfn2AalVO 7B6nQXZyKFpPoC6xDogo8Z/i63LLvmJo7FMdwlnRVA4O/p74SQ/Nk7LY79rdMQ8h6o39 pAvg== X-Gm-Message-State: ALyK8tIhmIdulE/7tXhXAws/uBA8uW7aNQyLvErsJhLw+srEBF4IoPncuwVsBVsXP3vkwFx+EiWRdlQBCvbwMA== X-Received: by 10.140.91.53 with SMTP id y50mr3866947qgd.104.1464971888817; Fri, 03 Jun 2016 09:38:08 -0700 (PDT) MIME-Version: 1.0 Received: by 10.237.38.5 with HTTP; Fri, 3 Jun 2016 09:38:08 -0700 (PDT) In-Reply-To: <57514296.7090906@norma.perm.ru> References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> <20160602145028.GC75625@zxy.spb.ru> <57514296.7090906@norma.perm.ru> From: Freddie Cash Date: Fri, 3 Jun 2016 09:38:08 -0700 Message-ID: Subject: Re: HAST, zfs and local mirroring To: "Eugene M. Zheganin" Cc: stable@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 16:38:10 -0000 On Fri, Jun 3, 2016 at 1:40 AM, Eugene M. Zheganin wrote: > Hi. > > On 02.06.16 19:50, Slawa Olhovchenkov wrote: > > I am suggesting next setup: > > > > node0: > > own pool zroot0: mirror-0: local_disk0 > > remote-iscsi_disk1/1 > > local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1 > > > > node1: > > own pool zroot1: mirror-0: local_disk0 > > remote-iscsi_disk0/1 > > local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0 > > > > > > No HAST. > > Disks synced by ZFS over iSCSI. > But this way I will get two independent zfs pools (or I still didn't get > it), one half of each will be stored on another machine. And I need > something different - a continuously replicated disk resource, that > would be available on both machines in case either will crash. Cluster > filesystem would be fine, but as far as I know there's no such thing on > FreeBSD, so I accept it will be unavailable on slave while mounted as > read-write on the master. HAST looks like a thing thats fits my > requirement, only that I wank it to be redundant inside each host too, > and all the documentation shows examples without local redundancy. > =E2=80=8BWhich is exactly the setup that I showed you. Host1: diskA diskB Host2: diskC diskD You create two HAST resources: hast1: using diskA and diskC hast2: using diskB and diskD Then create a ZFS pool using hast1 and hast2 as a mirror vdev. That way, if you lose diskA, then diskC takes over, and ZFS never even notices. If you lose both diskA and diskC, you still have half the ZFS mirror vdev running, so the pool is intact. If you lose all of Host1, Host2 takes over, imports the pool, and carries on. HAST + CARP (to manage the fail-over and pool export/import) + ZFS gives you everything (well, mostly) that you want. There will be a smidgeon of downtime during the host-to-host fail-over that may or may not be workable in your setup. The other option is to investigate the Ceph clustered filesystem. I believe there's been work ongoing this year to get it working on top of ZFS on FreeBSD. There's also GlusterFS, which has seen a bunch of work to get it working on top of ZFS on FreeBSD. No idea how well those last two work. --=20 Freddie Cash fjwcash@gmail.com From owner-freebsd-stable@freebsd.org Fri Jun 3 18:34:27 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E768B691FC for ; Fri, 3 Jun 2016 18:34:27 +0000 (UTC) (envelope-from freebsd-security-local@be-well.ilk.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 2E7631DA4 for ; Fri, 3 Jun 2016 18:34:27 +0000 (UTC) (envelope-from freebsd-security-local@be-well.ilk.org) Received: by mailman.ysv.freebsd.org (Postfix) id 2DDBEB691FB; Fri, 3 Jun 2016 18:34:27 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D65AB691FA; Fri, 3 Jun 2016 18:34:27 +0000 (UTC) (envelope-from freebsd-security-local@be-well.ilk.org) Received: from be-well.ilk.org (be-well.ilk.org [23.30.133.173]) by mx1.freebsd.org (Postfix) with ESMTP id 0D01D1DA3; Fri, 3 Jun 2016 18:34:26 +0000 (UTC) (envelope-from freebsd-security-local@be-well.ilk.org) Received: from lowell-desk.lan (router.lan [172.30.250.2]) by be-well.ilk.org (Postfix) with ESMTP id D1C4D33C1E; Fri, 3 Jun 2016 14:34:19 -0400 (EDT) Received: by lowell-desk.lan (Postfix, from userid 1147) id 4873B39828; Fri, 3 Jun 2016 14:34:19 -0400 (EDT) From: Lowell Gilbert To: Slawa Olhovchenkov Cc: stable@freebsd.org, freebsd-security@freebsd.org Subject: Re: unbound and ntp issuse References: <20160602122727.GB75625@zxy.spb.ru> Date: Fri, 03 Jun 2016 14:34:18 -0400 In-Reply-To: <20160602122727.GB75625@zxy.spb.ru> (Slawa Olhovchenkov's message of "Thu, 2 Jun 2016 15:27:27 +0300") Message-ID: <44lh2mi0k5.fsf@lowell-desk.lan> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 18:34:27 -0000 Slawa Olhovchenkov writes: > Default install with local_unbound and ntpd can't be functional with > incorrect date/time in BIOS: > > Unbound requred correct time for DNSSEC check and refuseing queries > ("Jul 1 20:17:29 yellowrat unbound: [3444:0] info: failed to prime > trust anchor -- DNSKEY rrset is not secure . DNSKEY IN") > > ntpd don't have any numeric IP of ntp servers in ntp.conf -- only > symbolic names like 0.freebsd.pool.ntp.org, as result -- can't > resolve (see above, about DNSKEY). I can't see how this would happen. DNSSEC doesn't seem to be required in a regular install as far as I can see. Certainly I don't have any problem on any of my systems, and I've never configured an anchor on the internal systems. > IMHO, ntp.conf need to include some numeric IP of public ntp servers. Ouch; that's a terrible idea, for several different reasons. From owner-freebsd-stable@freebsd.org Fri Jun 3 19:15:27 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 02F07B690CE for ; Fri, 3 Jun 2016 19:15:27 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id E6AE01EC5 for ; Fri, 3 Jun 2016 19:15:26 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id E6187B690CD; Fri, 3 Jun 2016 19:15:26 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5A5DB690CC; Fri, 3 Jun 2016 19:15:26 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ACE351EC4; Fri, 3 Jun 2016 19:15:26 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1b8uZ2-000Aje-2Z; Fri, 03 Jun 2016 22:15:24 +0300 Date: Fri, 3 Jun 2016 22:15:24 +0300 From: Slawa Olhovchenkov To: Lowell Gilbert Cc: stable@freebsd.org, freebsd-security@freebsd.org Subject: Re: unbound and ntp issuse Message-ID: <20160603191523.GE75630@zxy.spb.ru> References: <20160602122727.GB75625@zxy.spb.ru> <44lh2mi0k5.fsf@lowell-desk.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <44lh2mi0k5.fsf@lowell-desk.lan> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 19:15:27 -0000 On Fri, Jun 03, 2016 at 02:34:18PM -0400, Lowell Gilbert wrote: > Slawa Olhovchenkov writes: > > > Default install with local_unbound and ntpd can't be functional with > > incorrect date/time in BIOS: > > > > Unbound requred correct time for DNSSEC check and refuseing queries > > ("Jul 1 20:17:29 yellowrat unbound: [3444:0] info: failed to prime > > trust anchor -- DNSKEY rrset is not secure . DNSKEY IN") > > > > ntpd don't have any numeric IP of ntp servers in ntp.conf -- only > > symbolic names like 0.freebsd.pool.ntp.org, as result -- can't > > resolve (see above, about DNSKEY). > > I can't see how this would happen. DNSSEC doesn't seem to be required in > a regular install as far as I can see. Certainly I don't have any I don't know reasson for enforcing DNSSEC in regular install. I am just select `local_unbound` at setup time and enter `127.0.0.1` as nameserver address. > problem on any of my systems, and I've never configured an anchor on the > internal systems. > > > IMHO, ntp.conf need to include some numeric IP of public ntp servers. > > Ouch; that's a terrible idea, for several different reasons. What else? From owner-freebsd-stable@freebsd.org Fri Jun 3 20:11:22 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74033B69E5A for ; Fri, 3 Jun 2016 20:11:22 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 5AEB7174A for ; Fri, 3 Jun 2016 20:11:22 +0000 (UTC) (envelope-from peter@rulingia.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5A3EBB69E59; Fri, 3 Jun 2016 20:11:22 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59E6EB69E58 for ; Fri, 3 Jun 2016 20:11:22 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (vps.rulingia.com [103.243.244.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E82D01747 for ; Fri, 3 Jun 2016 20:11:21 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp59-167-167-3.static.internode.on.net [59.167.167.3]) by vps.rulingia.com (8.15.2/8.15.2) with ESMTPS id u53KB52h042389 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 4 Jun 2016 06:11:12 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id u53KAxXE078728 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 4 Jun 2016 06:10:59 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id u53KAux8078727; Sat, 4 Jun 2016 06:10:56 +1000 (AEST) (envelope-from peter) Date: Sat, 4 Jun 2016 06:10:56 +1000 From: Peter Jeremy To: Eugene Grosbein Cc: stable@freebsd.org Subject: Re: HAST, zfs and local mirroring Message-ID: <20160603201056.GA85345@server.rulingia.com> References: <20160601132300.GA75625@zxy.spb.ru> <574FDC63.6090100@norma.perm.ru> <20160602194856.GA84753@server.rulingia.com> <57519E77.7060507@grosbein.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="h31gzZEtNLTqOjlF" Content-Disposition: inline In-Reply-To: <57519E77.7060507@grosbein.net> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.6.1 (2016-04-27) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Jun 2016 20:11:22 -0000 --h31gzZEtNLTqOjlF Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2016-Jun-03 22:12:55 +0700, Eugene Grosbein wrote: >> all your media content is valid. I've also had bad experiences with >> gmirror volumes silently getting out of sync on a crash. > > >gmirror or (gmirror+gjournal) ones? Plain gmirror. --=20 Peter Jeremy --h31gzZEtNLTqOjlF Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJXUeRPXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFRUIyOTg2QzMwNjcxRTc0RTY1QzIyN0Ux NkE1OTdBMEU0QTIwQjM0AAoJEBall6Dkogs0TQgP/3ZNgK8U5b+F0eWvrBYJJB9J KlnSbjLpKGfom2ux9d0b5rhh4fNPlYYJdhFBm2plcPY/WGr3E6djP6NZ4BIBnDYw PivQ3s9gnaCZQ/hlJhWt4TdhXkwh5m9vQyJHrwU6FPheMnl3/n7YZLXH45ci/XRL 4qEKUJAdCs6Npf8jf+XkWhz4oHcMPOvV60wTP+NE9mO8a2d0Vp7B8WUYQjBjOgWN JVHnKNcsWKYvlh/bsWptRcsjdzCgk/kZbKu0az1iGlwe9rDQKIfTxwuasIeb50Bm XAs/cRl3k0oFIe1Bj8DDIRslKbWAp2TN+LzWmdV5JWeetD/viGBgagQJjX1gbWp1 H0NUVbZiWx7E0i1cH0yufP3vXmLmUpfEUJ4l7TB1OsqC6IsyLpc3E4iio3BFlynf +QErPlix1bYb0F/+b1P0pwTn6JwD/VZL8KrZY39Sp4cG3wpv1XuS7beH9Get244u zoc4A+2NLjHzfcAjV7obqDF2aLKGSu6IB2eknA5L9IWFVDqX1la9erUTIz6GyDPB 0tE5FriKOrWTqrX2ev6V3ND18EgL07dcHSpCu7Js9ifS9wQd4ibSWW0LqVJEykVM F55bhH+yT5ouX3kRQJhMzPRqWCuvD/W+AKyctfFO/Fq43m0XIUKpK1gieR1L+UCS thIG7ZfpAEU+i41U+SOE =7OFZ -----END PGP SIGNATURE----- --h31gzZEtNLTqOjlF--