From owner-freebsd-cluster@FreeBSD.ORG Sun Jul 31 03:53:26 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 13E0116A420 for ; Sun, 31 Jul 2005 03:53:25 +0000 (GMT) (envelope-from debug@MIT.EDU) Received: from south-station-annex.mit.edu (SOUTH-STATION-ANNEX.MIT.EDU [18.72.1.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 80A4E43D45 for ; Sun, 31 Jul 2005 03:53:24 +0000 (GMT) (envelope-from debug@MIT.EDU) Received: from grand-central-station.mit.edu (GRAND-CENTRAL-STATION.MIT.EDU [18.7.21.82]) by south-station-annex.mit.edu (8.12.4/8.9.2) with ESMTP id j6V3rDSj011920; Sat, 30 Jul 2005 23:53:13 -0400 (EDT) Received: from outgoing-legacy.mit.edu (OUTGOING-LEGACY.MIT.EDU [18.7.22.104]) by grand-central-station.mit.edu (8.12.4/8.9.2) with ESMTP id j6V3rAW5000357; Sat, 30 Jul 2005 23:53:11 -0400 (EDT) Received: from SDC.mit.edu (c-24-128-127-106.hsd1.ma.comcast.net [24.128.127.106]) ) by outgoing-legacy.mit.edu (8.12.4/8.12.4) with ESMTP id j6V3qv0w018561; Sat, 30 Jul 2005 23:52:57 -0400 (EDT) Message-Id: <5.2.1.1.2.20050730234326.042a7a20@hesiod> X-Sender: kurt@hesiod (Unverified) X-Mailer: QUALCOMM Windows Eudora Version 5.2.1 Date: Sat, 30 Jul 2005 23:52:58 -0400 To: beowulf@beowulf.org From: Cluster 2005 Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Spam-Score: 3.116 X-Spam-Level: *** (3.116) X-Spam-Flag: NO X-Scanned-By: MIMEDefang 2.42 X-Mailman-Approved-At: Sun, 31 Jul 2005 13:11:21 +0000 Cc: Subject: IEEE Cluster 2005 Online Registration now open X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Jul 2005 03:53:26 -0000 see http://www.cluster2005.org for info. From owner-freebsd-cluster@FreeBSD.ORG Mon Aug 1 13:58:12 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4543916A41F for ; Mon, 1 Aug 2005 13:58:12 +0000 (GMT) (envelope-from tbyte@OTEL.net) Received: from BASE.OTEL.net (BASE.OTEL.net [212.36.8.132]) by mx1.FreeBSD.org (Postfix) with ESMTP id D83DA43D4C for ; Mon, 1 Aug 2005 13:58:11 +0000 (GMT) (envelope-from tbyte@OTEL.net) Received: from dragon.otel.net ([212.36.8.135]) by BASE.OTEL.net with esmtp (Exim 4.30; FreeBSD) id 1Dzanx-000KxF-Mq; Mon, 01 Aug 2005 16:58:09 +0300 Message-ID: <42EE2A3E.4080009@OTEL.net> Date: Mon, 01 Aug 2005 16:57:18 +0300 From: Iasen Kostov User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050524 X-Accept-Language: bg, en-us, en MIME-Version: 1.0 To: Nick I References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-cluster@freebsd.org Subject: Re: Cluster links X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2005 13:58:12 -0000 Nick I wrote: >Hi, > >I work on a Web site at www.clusterbuilder.org. The site highlights a >broad spectrum of high performance computing related technologies. It >has links to reviews, comparisons, pricing and other information >related to the many HPC solutions available. The site also allows for >you to complete one request-for-quote form and have multiple hardware >vendors of your choice provide pricing quotes to you. > >We need knowledgeable cluster users to submit sites that have proved a >valuable resources to them. One particular area that we want to make >sure is complete is the Operating Systems section, which Debian is >listed under. We want to find information that helps HPC >users using Yellow Dog Linux Please take a minute and submit >additional links associated with Debian, operating systems or other >areas that will benefit the HPC community. (To submit content for the >site, click on the Submit Content section on www.clusterbuilder.org). > >Thank you for your help. > >Nick >_______________________________________________ >freebsd-cluster@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-cluster >To unsubscribe, send any mail to "freebsd-cluster-unsubscribe@freebsd.org" > > > FreeBSD is *NOT* "other Linux" (its more true to put MacOS X in "Other FreeBSD" :) ) as You stated on that site and it would be good to fix it ASAP because it show a lack of understanding Unices. Despite of this the site could prove useful someday. From owner-freebsd-cluster@FreeBSD.ORG Mon Aug 1 17:02:29 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 76F9E16A41F for ; Mon, 1 Aug 2005 17:02:29 +0000 (GMT) (envelope-from anton@nikiforov.ru) Received: from mail.newlines.ru (mail.newlines.ru [195.54.211.132]) by mx1.FreeBSD.org (Postfix) with ESMTP id B995843D48 for ; Mon, 1 Aug 2005 17:02:27 +0000 (GMT) (envelope-from anton@nikiforov.ru) Received: from mail.newlines.ru (root@localhost) by mail.newlines.ru (8.12.10/8.12.9) with SMTP id j71H2P08037180 for ; Mon, 1 Aug 2005 17:02:25 GMT (envelope-from anton@nikiforov.ru) Received: from [192.168.80.100] (81.13.10.229.florent-mashin.rmt.ru [81.13.10.229] (may be forged)) (authenticated bits=0) by mail.newlines.ru (8.12.10/8.12.9) with ESMTP id j71H2OVr037160 for ; Mon, 1 Aug 2005 17:02:25 GMT (envelope-from anton@nikiforov.ru) Message-ID: <42EE559A.3080000@nikiforov.ru> Date: Mon, 01 Aug 2005 21:02:18 +0400 From: Anton Nikiforov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041217 X-Accept-Language: ru, en-us, en MIME-Version: 1.0 To: freebsd-cluster@freebsd.org Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms060906000909090707000303" Subject: DRBD-like something for FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: anton@nikiforov.ru List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2005 17:02:29 -0000 This is a cryptographically signed message in MIME format. --------------ms060906000909090707000303 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Dear ALL, I'm running a few clusters with Linux, but do not like it ;) and would like to move to FreeBSD. I know about vrrpd and pfsync and other solutions, i know how to run Oracle or Postfix or PostgreSQL under such a configuration. but the only thing i cannot find is something like DRBD device that is working pretty cool and stable under linux (it is mirroring raw partitions from one node to other/s). Could someone please direct me to the link or list where i can read something about? Best regards, Anton Nikiforov --------------ms060906000909090707000303 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGKDCC AuEwggJKoAMCAQICAw6AYzANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwNDE2MTEwMzExWhcNMDYwNDE2MTEwMzEx WjBEMR8wHQYDVQQDExZUaGF3dGUgRnJlZW1haWwgTWVtYmVyMSEwHwYJKoZIhvcNAQkBFhJh bnRvbkBuaWtpZm9yb3YucnUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDp45jI HORB4RVcbIIDMttb19fW9gb9dIX4CVBbCZSPmX+gpsYUCswB/wcqnF98LoSTIYxiY/hUrENH 5lObA+oEazWWmZQR5MQIgZViQ3H7vJ1KpaQ8tR7atUsCMudFb9Wu4jYgvFbjhYLO0cOFOfhr n99ucG5qpsXwriIbDYLT20xFvsbCk/zlMwPsIfxell+EM87MglUH5uym8LtcWVvfZgzYuNN0 1lJXF4Qs17X3y3XELuuRowdQGZQ6nNM2StTePuOL6J3piVERqhscLIpM9rjfH6nV8HM1+BW3 trgt5rWqzSfFlvxk6MF6cvz06xnE6Yw4FV63lrRzaiwm914/AgMBAAGjPzA9MA4GA1UdDwEB /wQEAwIHgDAdBgNVHREEFjAUgRJhbnRvbkBuaWtpZm9yb3YucnUwDAYDVR0TAQH/BAIwADAN BgkqhkiG9w0BAQQFAAOBgQASvk2h3CLH4S3NOw9yRfEdpHruWxov2mQvsV8qZKwjG8/661ze FmsQhAS18+6hCgK84qNrCINydH06Y5jsAGmwS8r9m+xOPxDKiehmOSsOpSVShzIfWdRx5Ni1 uFvPwH9L6czsOlw0PAQnYEv0jVbel6SA5MUWHwJ8liIGxkhi3jCCAz8wggKooAMCAQICAQ0w DQYJKoZIhvcNAQEFBQAwgdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUx EjAQBgNVBAcTCUNhcGUgVG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNV BAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQ ZXJzb25hbCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxA dGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYT AlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNU aGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOB jQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B1OVQ CjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79AgAJk 16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8ECDAG AQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVQ ZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgG A1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNm rGcPf6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0 niVQlGLH2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx 0x1G/11fZU8xggJEMIICQAIBATBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUg Q29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1h aWwgSXNzdWluZyBDQQIDDoBjMAkGBSsOAwIaBQCggbEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMDUwODAxMTcwMjE4WjAjBgkqhkiG9w0BCQQxFgQU/iiT d9nKRvQI3q2mUJLn/s9VREMwUgYJKoZIhvcNAQkPMUUwQzAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwDQYJKoZI hvcNAQEBBQAEggEAbhq7a0HF8a8teQsaBJAjqi52gr8M0cVV/MYBqzNHoOCP56ryuCsWWKdh J2kY71kZZ/66WoIS0QnfjYrMRWVtQp5nGEH2GzlN0x1shLiRdX77j9wlbiLpwWex/ot3wLJu jjfBCVTaUYNiL4wp0q6cbYABfDk8I90IDd50sDIucq7GBVwVjm7/xpCyrFiQ8EJfgLq6Am7l nCq3urnAdkKP8v0khwwzssZ3S9KOWMxmDbcBQCphIPH/ZR0xQN990W2Erfs85GvhdUziolKi 6OOTRtQzt7F9lQa5aySeZvWR9MFAxpV5gehXCG2tzJLkj9b7J/lPQ2MJKCW/GQxM9lixfQAA AAAAAA== --------------ms060906000909090707000303-- From owner-freebsd-cluster@FreeBSD.ORG Mon Aug 1 19:29:34 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 53BEC16A41F for ; Mon, 1 Aug 2005 19:29:34 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh1.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id DE7F843D46 for ; Mon, 1 Aug 2005 19:29:33 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [10.177.171.220] (neutrino.centtech.com [10.177.171.220]) by mh1.centtech.com (8.13.1/8.13.1) with ESMTP id j71JTVev095076; Mon, 1 Aug 2005 14:29:32 -0500 (CDT) (envelope-from anderson@centtech.com) Message-ID: <42EE7813.8020902@centtech.com> Date: Mon, 01 Aug 2005 14:29:23 -0500 From: Eric Anderson User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.8) Gecko/20050603 X-Accept-Language: en-us, en MIME-Version: 1.0 To: anton@nikiforov.ru References: <42EE559A.3080000@nikiforov.ru> In-Reply-To: <42EE559A.3080000@nikiforov.ru> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1000/Sun Jul 31 14:28:06 2005 on mh1.centtech.com X-Virus-Status: Clean Cc: freebsd-cluster@freebsd.org Subject: Re: DRBD-like something for FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Aug 2005 19:29:34 -0000 Anton Nikiforov wrote: > Dear ALL, > I'm running a few clusters with Linux, but do not like it ;) and would > like to move to FreeBSD. > I know about vrrpd and pfsync and other solutions, i know how to run > Oracle or Postfix or PostgreSQL under such a configuration. but the only > thing i cannot find is something like DRBD device that is working pretty > cool and stable under linux (it is mirroring raw partitions from one > node to other/s). > Could someone please direct me to the link or list where i can read > something about? GEOM Gate and GEOM mirror are probably what you want. Check out ggated, ggatec and geom_mirror. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------ From owner-freebsd-cluster@FreeBSD.ORG Wed Aug 3 18:43:53 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 53E3616A41F for ; Wed, 3 Aug 2005 18:43:53 +0000 (GMT) (envelope-from anton@nikiforov.ru) Received: from mail.newlines.ru (mail.newlines.ru [195.54.211.132]) by mx1.FreeBSD.org (Postfix) with ESMTP id 76D0943D46 for ; Wed, 3 Aug 2005 18:43:51 +0000 (GMT) (envelope-from anton@nikiforov.ru) Received: from mail.newlines.ru (root@localhost) by mail.newlines.ru (8.12.10/8.12.9) with SMTP id j73IhncA044387 for ; Wed, 3 Aug 2005 18:43:49 GMT (envelope-from anton@nikiforov.ru) Received: from [192.168.80.100] (81.13.10.229.florent-mashin.rmt.ru [81.13.10.229] (may be forged)) (authenticated bits=0) by mail.newlines.ru (8.12.10/8.12.9) with ESMTP id j73IhmVr044365 for ; Wed, 3 Aug 2005 18:43:48 GMT (envelope-from anton@nikiforov.ru) Message-ID: <42F1105B.5040507@nikiforov.ru> Date: Wed, 03 Aug 2005 22:43:39 +0400 From: Anton Nikiforov User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.5) Gecko/20041217 X-Accept-Language: ru, en-us, en MIME-Version: 1.0 To: freebsd-cluster@freebsd.org References: <42EE559A.3080000@nikiforov.ru> <42EE7813.8020902@centtech.com> In-Reply-To: <42EE7813.8020902@centtech.com> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms080906050304020905040404" Subject: Re: DRBD-like something for FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: anton@nikiforov.ru List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Aug 2005 18:43:53 -0000 This is a cryptographically signed message in MIME format. --------------ms080906050304020905040404 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Eric Anderson wrote: > Anton Nikiforov wrote: > >> Dear ALL, >> I'm running a few clusters with Linux, but do not like it ;) and would >> like to move to FreeBSD. >> I know about vrrpd and pfsync and other solutions, i know how to run >> Oracle or Postfix or PostgreSQL under such a configuration. but the >> only thing i cannot find is something like DRBD device that is working >> pretty cool and stable under linux (it is mirroring raw partitions >> from one node to other/s). >> Could someone please direct me to the link or list where i can read >> something about? > > > GEOM Gate and GEOM mirror are probably what you want. Check out ggated, > ggatec and geom_mirror. > > Eric Thanks for the replay, Eric, Please correct if i'm wrong. You suggest to make two devices one is local daX device and the other is ggated/gagatec device and build a mirror set on them using geom_mirror module and gmirror utility? That is pretty cool idea, but what i should do in case main (ggated) system crash? I'm still reading and investigating this issue, and will test it as soon as i'll get to PCs/Servers to test, but as for now i do not see the way to promote second node as prymary in case of real prymary crashed. Or i should test it with vrrpd (for example) and restart secondary as ggated instead of being ggatec? Did not clear yet. If you have the solution or an idea - could you please drop me a line? algorythm looks the following for me: We have two computers: stage 0 (up and running claster) 1. da0 is exported via ggated over the network. This computer runing postgresql (for example) that storing data on da0. 2. da1 is being imported over network via ggatec as da1. Local da0 and remote da1 are mirrored via ggated/geom_mirror.ko and there is no service activity on this device. just a mirroring of data. stage 1 (slave failed) 1. working as before 2. down Requests to update remote data failed. There should be no problem. stage 2 (slave is coming up) 1. working as before 2. booting and starting to mirror date from the master in case secondary goes down and up again there are no troubles. it just boots itself in the stage0 config. that is it stage 3 (master is down) 1. down 2. should consider master is down and stop ggatec, then start ggated and export the da0. Then mount da0 and start PostgreSQL. stage 4 (master coming up) 1. Should understand it is not the master anymore and start ggatec and mirror remote data to the local drive. 2. running as a master. with services moved/started at stage 3 Here troubles begins. When bouth become synchronized the service that was running on this disk shoul be moved from 2 to 1. to do that i'll have to restart services in stage 0. But how? using vrrpd? or some other utility? How can i understand that the system is ready to get the Postgresql back to 1 and stop it on 2? stage 5 (bouth down, master coming up first) 1. Starting as master, exactly as it did before stage 0 2. Starting as in stage 1 stage 6 (bouth down, slave is coming up first) 1. Have to determine which was doun first and in case 1 was - start as the slave and vise versa - in case it was down last - should start as master. 2. Have to determine which was doun first and in case 2 was - start as the slave and vise versa - in case it was down last - should start as master. This is main confuse. Maybe i did not read mutch, but it is not clear for me, how to inderstand which drive contain the last data? Best regards, Anton Nikiforov > > > > --------------ms080906050304020905040404 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGKDCC AuEwggJKoAMCAQICAw6AYzANBgkqhkiG9w0BAQQFADBiMQswCQYDVQQGEwJaQTElMCMGA1UE ChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNv bmFsIEZyZWVtYWlsIElzc3VpbmcgQ0EwHhcNMDUwNDE2MTEwMzExWhcNMDYwNDE2MTEwMzEx WjBEMR8wHQYDVQQDExZUaGF3dGUgRnJlZW1haWwgTWVtYmVyMSEwHwYJKoZIhvcNAQkBFhJh bnRvbkBuaWtpZm9yb3YucnUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDp45jI HORB4RVcbIIDMttb19fW9gb9dIX4CVBbCZSPmX+gpsYUCswB/wcqnF98LoSTIYxiY/hUrENH 5lObA+oEazWWmZQR5MQIgZViQ3H7vJ1KpaQ8tR7atUsCMudFb9Wu4jYgvFbjhYLO0cOFOfhr n99ucG5qpsXwriIbDYLT20xFvsbCk/zlMwPsIfxell+EM87MglUH5uym8LtcWVvfZgzYuNN0 1lJXF4Qs17X3y3XELuuRowdQGZQ6nNM2StTePuOL6J3piVERqhscLIpM9rjfH6nV8HM1+BW3 trgt5rWqzSfFlvxk6MF6cvz06xnE6Yw4FV63lrRzaiwm914/AgMBAAGjPzA9MA4GA1UdDwEB /wQEAwIHgDAdBgNVHREEFjAUgRJhbnRvbkBuaWtpZm9yb3YucnUwDAYDVR0TAQH/BAIwADAN BgkqhkiG9w0BAQQFAAOBgQASvk2h3CLH4S3NOw9yRfEdpHruWxov2mQvsV8qZKwjG8/661ze FmsQhAS18+6hCgK84qNrCINydH06Y5jsAGmwS8r9m+xOPxDKiehmOSsOpSVShzIfWdRx5Ni1 uFvPwH9L6czsOlw0PAQnYEv0jVbel6SA5MUWHwJ8liIGxkhi3jCCAz8wggKooAMCAQICAQ0w DQYJKoZIhvcNAQEFBQAwgdExCzAJBgNVBAYTAlpBMRUwEwYDVQQIEwxXZXN0ZXJuIENhcGUx EjAQBgNVBAcTCUNhcGUgVG93bjEaMBgGA1UEChMRVGhhd3RlIENvbnN1bHRpbmcxKDAmBgNV BAsTH0NlcnRpZmljYXRpb24gU2VydmljZXMgRGl2aXNpb24xJDAiBgNVBAMTG1RoYXd0ZSBQ ZXJzb25hbCBGcmVlbWFpbCBDQTErMCkGCSqGSIb3DQEJARYccGVyc29uYWwtZnJlZW1haWxA dGhhd3RlLmNvbTAeFw0wMzA3MTcwMDAwMDBaFw0xMzA3MTYyMzU5NTlaMGIxCzAJBgNVBAYT AlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNU aGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBDQTCBnzANBgkqhkiG9w0BAQEFAAOB jQAwgYkCgYEAxKY8VXNV+065yplaHmjAdQRwnd/p/6Me7L3N9VvyGna9fww6YfK/Uc4B1OVQ CjDXAmNaLIkVcI7dyfArhVqqP3FWy688Cwfn8R+RNiQqE88r1fOCdz0Dviv+uxg+B79AgAJk 16emu59l0cUqVIUPSAR/p7bRPGEEQB5kGXJgt/sCAwEAAaOBlDCBkTASBgNVHRMBAf8ECDAG AQH/AgEAMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwudGhhd3RlLmNvbS9UaGF3dGVQ ZXJzb25hbEZyZWVtYWlsQ0EuY3JsMAsGA1UdDwQEAwIBBjApBgNVHREEIjAgpB4wHDEaMBgG A1UEAxMRUHJpdmF0ZUxhYmVsMi0xMzgwDQYJKoZIhvcNAQEFBQADgYEASIzRUIPqCy7MDaNm rGcPf6+svsIXoUOWlJ1/TCG4+DYfqi2fNi/A9BxQIJNwPP2t4WFiw9k6GX6EsZkbAMUaC4J0 niVQlGLH2ydxVyWN3amcOY6MIE9lX5Xa9/eH1sYITq726jTlEBpbNU1341YheILcIRk13iSx 0x1G/11fZU8xggJEMIICQAIBATBpMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUg Q29uc3VsdGluZyAoUHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1h aWwgSXNzdWluZyBDQQIDDoBjMAkGBSsOAwIaBQCggbEwGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMDUwODAzMTg0MzM5WjAjBgkqhkiG9w0BCQQxFgQUcc+m 13VDe+2HSpghdmZcJAkJZukwUgYJKoZIhvcNAQkPMUUwQzAKBggqhkiG9w0DBzAOBggqhkiG 9w0DAgICAIAwDQYIKoZIhvcNAwICAUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwDQYJKoZI hvcNAQEBBQAEggEA1XLtfjHZMguOS0pkae+b89JkjDnPjR17Dw2WlbftapXb95dI4RabbhBz hLDf04ldfvYDgTTU/KWcZ+VQP7iDyuGkbvjqZSOBKYvycuEmo2pkS3EfYvuM/zpu6k7RQDaH 1g3z6u8D4HcUBi/7FDkjO5Jihik+xpmSMUZauTF7R+lN2DxSAjRPnu8+TWiUYK83Q+oJLOsx blMzqmjSGI82rMVVNTmKRiYfHDjCuRkUQ97XCe6tzogdSk4rUkFvvyVwpSBuauAxB7YDCVXV 6kUXKXDd8RS33aG6dmznh1ltw3YrEW1zhv4kawuDBRqNirwEjqERLSPl6lVSzdrwbCUvkgAA AAAAAA== --------------ms080906050304020905040404-- From owner-freebsd-cluster@FreeBSD.ORG Thu Aug 4 07:41:48 2005 Return-Path: X-Original-To: freebsd-cluster@freebsd.org Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9BCAA16A41F for ; Thu, 4 Aug 2005 07:41:48 +0000 (GMT) (envelope-from dimma@torch.higis.ru) Received: from torch.higis.ru (gate.higis.ru [81.195.168.122]) by mx1.FreeBSD.org (Postfix) with ESMTP id 73C3143D48 for ; Thu, 4 Aug 2005 07:41:47 +0000 (GMT) (envelope-from dimma@torch.higis.ru) Received: from localhost (localhost [127.0.0.1]) by torch.higis.ru (8.12.10/8.12.10) with ESMTP id j747fjKn069339; Thu, 4 Aug 2005 11:41:45 +0400 (MSD) (envelope-from dimma@torch.higis.ru) Received: from torch.higis.ru ([127.0.0.1]) by localhost (torch.higis.ru [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 69035-07; Thu, 4 Aug 2005 11:41:37 +0400 (MSD) Received: from torch.higis.ru (localhost [127.0.0.1]) by torch.higis.ru (8.12.10/8.12.10) with ESMTP id j747fWeu069327; Thu, 4 Aug 2005 11:41:32 +0400 (MSD) (envelope-from dimma@torch.higis.ru) Received: (from dimma@localhost) by torch.higis.ru (8.12.10/8.12.10/Submit) id j747fWQw069323; Thu, 4 Aug 2005 11:41:32 +0400 (MSD) (envelope-from dimma) Date: Thu, 4 Aug 2005 11:41:31 +0400 From: Dmitriy Kirhlarov To: Anton Nikiforov Message-ID: <20050804074117.GG46130@torch.higis.ru> References: <42EE559A.3080000@nikiforov.ru> <42EE7813.8020902@centtech.com> <42F1105B.5040507@nikiforov.ru> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="4zI0WCX1RcnW9Hbu" Content-Disposition: inline In-Reply-To: <42F1105B.5040507@nikiforov.ru> User-Agent: Mutt/1.5.6i X-Virus-Scanned: by amavisd-new at higis.ru Cc: freebsd-cluster@freebsd.org Subject: Re: DRBD-like something for FreeBSD X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2005 07:41:48 -0000 --4zI0WCX1RcnW9Hbu Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi Anton! On Wed, 03 Aug 2005, Anton Nikiforov wrote: > not see the way to promote second node as prymary in case of real=20 > prymary crashed. Or i should test it with vrrpd (for example) and=20 > restart secondary as ggated instead of being ggatec? Did not clear yet.= =20 > If you have the solution or an idea - could you please drop me a line? Yes. /usr/ports/net/freevrrpd > algorythm looks the following for me: > We have two computers: > stage 0 (up and running claster) > 1. da0 is exported via ggated over the network. > This computer runing postgresql (for example) that storing data on da0. > 2. da1 is being imported over network via ggatec as da1. Local da0 and=20 > remote da1 are mirrored via ggated/geom_mirror.ko > and there is no service activity on this device. just a mirroring of data. No. When you work with ggate{c,d} you get devices ggate{0,1,2,...}. Not with da= 1. > stage 1 (slave failed) > 1. working as before > 2. down > Requests to update remote data failed. There should be no problem. >=20 > stage 2 (slave is coming up) > 1. working as before > 2. booting and starting to mirror date from the master > in case secondary goes down and up again there are no troubles. it just= =20 > boots itself in the stage0 config. that is it When slave is start you can rebuild mirror on the fly. But it get many time (in my case) -- 54Gb slice rebuilded slow :( > stage 3 (master is down) > 1. down > 2. should consider master is down and stop ggatec, then start ggated and= =20 > export the da0. Then mount da0 and start PostgreSQL. when master down, then freevrrpd can run script on slave machine. > stage 4 (master coming up) > 1. Should understand it is not the master anymore and start ggatec and=20 > mirror remote data to the local drive. > 2. running as a master. with services moved/started at stage 3 > Here troubles begins. When bouth become synchronized the service that=20 > was running on this disk shoul be moved from 2 to 1. to do that i'll=20 > have to restart services in stage 0. But how? using vrrpd? or some other= =20 > utility? How can i understand that the system is ready to get the=20 > Postgresql back to 1 and stop it on 2? Yes. vrrp -- your friend. But moving from node2 -> node1, when node1 coming= up bad idea. Service must continue to run on node2. > stage 5 (bouth down, master coming up first) >=20 > stage 6 (bouth down, slave is coming up first) My algoritm. No difference between master and slave in start time. We have: /etc/rc.conf -- this file identical on both nodes. Many parametes in it and= 2 includes: /etc/rc.clh -- specific addreses and specific params for every node. /etc/rc.cluster -- simlink dinamically switching between: /etc/rc.custer.master or /etc/rc.cluster.slave If we can ping shared IP, when starting -- we start as slave. In not -- mas= ter. We start freevrrpd with master or slave config (see /etc/rc.d/freevrrpd). /etc/rc.d/freevrrpd run after network start, but before all daemons. It work in RELENG_5* with new style rc-scripts. /boot/loader.conf have different autoboot_delay on different nodes (bad gua= rantee what important start-ping not run in one time on both machines) > This is main confuse. Maybe i did not read mutch, but it is not clear=20 > for me, how to inderstand which drive contain the last data? Yes. It's a problem in one case: reboot node0 and reboot node1, when mirror not fully synchronize. See attached scripts -- this my experiments. It have some known and, my be = unknown issues: - in /usr/local/ifstated/bin/ggatec-gmirror.sh script I can't get unit nam= e with one line: ggate=3D`/sbin/ggatec ${ggatec_flags}` - when I restart /etc/rc.d/freevrrpd many time I get many ggate devices. - parametert for ggate{c,d} may be not ideal, but it work on gigabit inter= connect. - I can't undestand what I must open on firewall for freevrrpd. By. Dmitriy=20 --4zI0WCX1RcnW9Hbu Content-Type: application/octet-stream Content-Disposition: attachment; filename="cluster.tbz2" Content-Transfer-Encoding: base64 QlpoOTFBWSZTWaJchtEADN1/5ey0AKB/////P///7/////4AgIEAAAEBQgAIYBH99W1d2fe9 PYAvHK27sc9d3jhrq60dnvbXT1iegBo2zyaBVK7sAcRtmEkRNRoaNTNU8p4TSekn6mppsKfq T9FG02p6keoGTCBoP1IyeofqgZBKEBMiaNGSaZE0mynij1B4o0GmgAAaBoAB6gAONDQ0NANA YgaAyAADTQANAMgAAAYSEiEammUbSTyBNlNNGTTI0GRpoA00BoBpoAAAEUqaNGJqm9U/T1TK bU0Gmg9QGgaAAANAABkA0ABIkCBNAExAQamETNE09JpoyGQAxGmQAA0A/73hIeXyKDOb9nhh 3/llpLvvtcYjyEDWjSzMSJDQ2mJDMkhAoaElxENJHf5i7yXz81Il5enasHejG+RcEiUxptjb QVMXlLYR4FWvBsaqdCJWtKTrWIm7opOtgSOoSGhJMvAYYS9O8qYY8F/7f09ZWU8dmTF6I492 ZoXLYzMxkGgw4x9w9U5CK2tcCfIeWegpg1jsSotYUC2YfvqlmH3BgP5IHIRB/VE2WrbaKWag 7KzjScsBEkObGKyhson4ZTJlWhdZRYdgiixj0vsvzdvHYv+VjLZn4r7oKyJL5hj46ucjEGSt MjvxgFozSLeAS7QVwGkml9RHb2kqcr8uxXqixHFrQ2FRzGUj+3OFMOYyVOW0y1jvetFr3OtK MlrhvjpNteXiY8lyrb5iQ2GULZVSGboxkQcVGK5U23jZvg5JwGs3rl5U4grkFIBy68RlbYxv HjNiLXZdk3JXOnAiYZwrIekQcFqZbsry6U2/f0aPATq+UAyM9RcV3gs7QwJPfLuQEdmsLeTR qNZD8S1TnE5TgHlt+tdHQdNDsVLniLx5G7JSYbuYwyzO7uNiSvTpyapJpmaCD8CfgQmgbdGw bVYRjaR9VyCwJRHvGSBQgyEYdc98bBAOF9zIaGpW+aa5MPmX6IPQzU7muYfIbTMJHqo6bNbD HTFX5sWa2o3qxkvHUfQep1yuHdO1RrR5IiWswcy8Zf5fg6jBRWUWJ1B4KVsIBYvEo7tdVmFl lrTvuanVCr0wqquc/XUn2zThI1D+G58shmaCtiesDyO7OHDnxU0EPHUavRoynUzp15a+2hzc +d4QGsyzqXETZaPXff2mdu06hazgRbUMVakXOG0ysLH1KUM3B4k9k+Sn1JQyxmStzOmLcpt0 UXDiNvAddtKPy4XlXqkVO5f0Fs3Wb05lKR254Q0MP09ciM9IvIKZuVMk4ROMnq0Ks99TsuPP F2Vdd4taWVgTxg6w2P7cJ7u7R/pXT4PMaJvyY6LJ861vjkycLsc9U4HwfFYZrbZPwxd4Xzmq jA8RRRS9ahvNBOG7iczuMPm9Z95AiiJJKBobUmHwl8TmRATbiEQGIaYhtiaEtd9LfQEdwDdY f2wMksBoII7rLEiQ0ymNowYqdhUhESP1PdLUjDM6RoVMYFmrDaGMY6gzWRde9kjU3IJSS8vm dE6JjjO3xUIkdDIwjAsoTZ3PvMTKRjld8ZeD5QYVqfK2+MxgO8HfbjcKsdDNUKc0LqfA7csb aTWohc4SWTSO2MpM7ZgdVUYNtWnOQ3ME1yQwzqMjTAVOQrc8Kehu1DtI093mZ+XPFuwevymw 8h4zMqeUwXI9vJRmEJQDlMQR8YGf1jgvGN++cXL9rogdwYaoQBoGcbOiEGB8XlWOuNCGvl5X GDollwjd5iRgmAq4FMxCi8znOlMV8BWFBoWSsnRZmIReaDfEwQBg7cjNepM5rQZTBk0HCd1L PdR9Atm6BsOT1BdZsx22u7EZBSNpYkibQ3UGlAYMOm1Cr8oxhm2iDTVxM4MEkr31XJUIUOha WIXpDAWCMCDM9G58NROugpgKyyFjWoGBIkb3fQJkiGxDCTZJtybQt/f7MeQEjnx5CRngizR5 guu85Z3M63IUmgU1VAIBS59fn1+FsdCb33rUe/56tajY8nxiuuksEG14AkNa5iQ5eTnCCs6B wmkcqog95BXxam4kxXzJScrz18plN2DHErGYM/KSpQqgFMHTCL7umrVJuhv0GFkPsxb+QVRW MjG/42XrBKeJW0c2YEQuA0GSJ5sDPK7FCirYWj/CiMS187o4SbRXSuhHzUnbMDVQfZNkAkiX 3QYICIOls+8HxpZK2g8IkYPflnHuzWe25p33iD1QHsyRGqxKYhEGIpFeOBCWtYhFtBJ3MxQY 0pStJTU7CR614FpzTRazCL0F82kq1Yy07XllYbQioJF94WdaIhlBjQTSRcE7gZl74RZ3XEri cyYRcpsnATuSutuC9b8L1JmCaJAReXyTJNV0wZLkUYYuaZZZiQ1krgkNIyM0UdammMDn7hRK QdYY02fk9LbeVEvRpfwnsfE0ms8gzBlwXdYegN4+ScykLZV5CI9qhklVI8Pxu9EenceC4WBH Tt6sbYFRs4o5POVWRCcHxskxLiOAac2Q43QUW60dySMVCqKvUUaDemPBxuWCNc5++bQ6vs5/ xTd4q9F7ri+pIM6K9dBaydiVT9wEsTHsS79a+vm9SRHOxrsLw9Yr/XSradbZyYZXVXMKi4VF u3c1yMRCwsK5AhthibEWJ75aVvGv3mtZuy2Eo1LFZhmnhRQsin61cU0FXIALGRVYbDr6oFSt VFvdLKkjA21Yx6VbUq2VYlIssUCRJb4QGWw0nmIgahhw8p85BB8weM1ETeO4dg5uU5n8/d0/ CxcUzBzOtLeJpmNWrvQO07sMT5iZY3pZuYwxGh88hGWtBFG5UKBsUmozgzMiHiWZJGBL3lJY KosMLF14UsqTL3F5pjBlPxcx/K0/CJG0EjdcvZtfBJKn0BGjQFkKjg0rdGFJyAOpAdKlBtfa lGeBgQT8F6BfXYfZ7W1ADMmD7VRI5jEuTnxBphQUE3XuUMUxFxGJrEpq5AxpjFCWWoSGLeiz hemGayq4Yy+xEQraxXhWwDeTyuqGG0cBKFxUvWNASE0i+Lr1sA4KRoY5bn4On4qht4nnv6yF bjhE01zHYvEtNqGxjBsYoI4GIF4Dw7CUlf4+IGBibZL4fTB0prpQC5NaE2gAaCRpuQBBhlmv PrtIQrUZKyDYFGTIQLr3ndcWQsARvzTJIuMB6fp9n+f9/fckef/X9rHsZe3qoVwL9Tq8vTIY dkBzskqh50pkhs9fuD6Oa6g6o7EiuUBucQsIPWc/qY3KZuGRSsrLsHSLcZa7GdeFVCzVZSuj ZwWwvBGnAhYnJE0VEg2FfmaV1VUwg28rbum902aazZRQ7DeToos2NCsSJvJNbd6XroRI8+HS PejntLjslLRPiIsDLEvEJH/hIjTYCEavqcZrBi4dM5M2MmrEImPbS6nC3O2vcxFB5sIUWT2c hUZvk6RidGLGsR41CG4YZKDbKSCEkVk1Gc4ATCegeY4nFbxZgSEYmPoDgdZIsXWUh6PaHLYZ FEj3WrNFLhjSS0J6M2a5nQBS1cW1l1cUNe+NCRskaBFAL8/TMIOeUG0io0jJldMFzEJNDmd8 e2iqRQkRs3AjjPBHKiCp8OgYRISwhaO4EBMD8+KmIw6j5+SErTAz4IlkS6t6Jnd5drA9TlRt C1ZZd2aqlydgVFso1Ad51rskwA5sXmlNRC5F0XyjJ1HJeeQ/SRRSRDcX1V+3W8IBraJooql0 jHUKl8zdeiikClIjcOaWs1pCwcyRfuurIxbbI1SUtTIPnfncOtcpFVbXCdKptOmZzX0hzMDp JAa2LoIs0NiOcbiIMfBVkKnVVzAgRoGeUIH5uEH35jHYDUGz4NtMPND3DsSzVI8vmqdNCAit j4Mmw2cheMRgzgkNQNEEjwm0qCReL1kIqUMUdaXQLK2zU33d/LqIp0PYXEtE1UitHARLd4sP iVDlQEySKirx47cDVjL7QsMSL435g2mfP5yV4Ro4AykGt/bBI4jNaeGHIc4Za9KroAlkk5MG x48ToQIwDMQsShrA0B1WiYdKWKG2hsQhjE8zAAzT8BUJHaMDXmLoVxJkZyrBirKQpwgLQFQd nXkzE6ZUU21cLpzC0aktoELZKHHBJmuNxqy1t2AtYiGhptLJDRZdom0XNBxKgkU1pBU5oOHG RDd5JGqVDT4RlkuyhzFkUMsqX/JOpObTpRyUyxE6iN4LBGkMYDTLJCVumtRup6pOgGokHQvn xzSbBhiBBjBAQsgeiNLRoawAYoQUBKO2SmjIoUQpXHOSwAxqGMc/MsTLCwF5ZKV6CackonF1 sm8kTspGtHeDZ6CGM7SRAoYtbWvX0IhRCAGu4n1mdCh6p2d953b++wbCaQcMTKJXaikhJMJ0 SuU/BfpSXRTelKaPROGxtJjEOw/A9S4Ja+KDRr8OHa9TKJQxZeCwb1r4llwb+Zgc7UV5MJRJ zeGFDY1wjgxjX8qwdbWsx0QUS5QEBRjLosSL4SRrFHpvKrQRsNFEY2VIUSqVVBhTCIaUkIbV hZoKLyG08D5ZGsw21nfqC+Qut08OJkGjg0n8kIwOgKm/WtoOWwnhGrkmoGyQkMIaGxl7jabS 0DbJbepFMBoxiCwxESSopSSSJJUhQpKBjGDhtU8BicOG4ELD0mYtwKlK2mKXFcjyTScSCJKU 1Hg9cQHwpQoIan2hpwXL6XtCkdHC256D3ciyzq7J0WsLhtUHURZpI1NQ0EMXi4bd4Lia4IWa jScjFG8m0cyQXeJpWICFDCbOBu2bWVlDGrhsRyYiJ9FsFI8JXQbl7CXMKiMWgrSBF7zoMMtp tGDRjxrMRUUw5KQ67wodl+4v9T2eJwlzzz0JSNoXi6K6gvtI9hXIvGMGBR78JaS0PnRLC5B9 XE1DCgFl3271clNs7CD8odPNvDcIC0sTXQ8BUYwns5yEi4zbLHsSIEzahlRSi6yTGxjExjTQ 0kDZeEjwDQwpyOgmYkLmSXLbD7BYdPUdWmNs9g0Se5BKKhMmojtn07q1dYpIuZYtLS8C4M4i 6DBNDKTl41V+jJdY7gHOJTADtDSswDMsCA2hkaynBsZZgYNKMFmwRwXyQILuMljMe9QDRkMN d76FdGelh2dFc0t+YZrQhGcklr0JEbWGrVNYi+xlvoX452oZQYo3XrL1V4uaGBGGk4HF9Wk9 3MD5Wku3WFCUAGKD0CU3zaOpMoED6rBjLh7QX9YgLDWh0hTIAuB4cMlL1GgJMZCqqUJYM9l+ AijQlQcyypoxgDqRBmx8XIHhxPvCxDvvFxXYVGdZfaCkGp8gRfms6qWkmF6odA8tjfuCA+lE zuB4WMhkOFh4h+en2vCT5hRcKz7ZykXF2YzaoK/8XckU4UJCiXIbRA== --4zI0WCX1RcnW9Hbu--