From owner-freebsd-cluster@FreeBSD.ORG Mon Apr 6 18:13:40 2009 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3BD9A1065670 for ; Mon, 6 Apr 2009 18:13:40 +0000 (UTC) (envelope-from lavalamp@spiritual-machines.org) Received: from mx04.pub.collaborativefusion.com (mx04.pub.collaborativefusion.com [206.210.72.84]) by mx1.freebsd.org (Postfix) with ESMTP id 069A48FC1C for ; Mon, 6 Apr 2009 18:13:39 +0000 (UTC) (envelope-from lavalamp@spiritual-machines.org) Received: from [192.168.2.161] ([206.210.89.202]) by mx04.pub.collaborativefusion.com (StrongMail Enterprise 4.1.1.4(4.1.1.4-47689)); Mon, 06 Apr 2009 13:58:32 -0400 X-VirtualServerGroup: Default X-MailingID: 00000::00000::00000::00000::::32 X-SMHeaderMap: mid="X-MailingID" X-Destination-ID: freebsd-cluster@freebsd.org X-SMFBL: ZnJlZWJzZC1jbHVzdGVyQGZyZWVic2Qub3Jn From: "Brian A. Seklecki" To: Sebastiaan van Erk In-Reply-To: <49D71864.3010403@sebster.com> References: <49D71864.3010403@sebster.com> Content-Type: text/plain Date: Mon, 06 Apr 2009 13:58:31 -0400 Message-Id: <1239040711.16390.2195.camel@soundwave.ws.pitbpa0.priv.collaborativefusion.com> Mime-Version: 1.0 X-Mailer: Evolution 2.24.5 (2.24.5-1.fc10) Content-Transfer-Encoding: 7bit Cc: freebsd-cluster@freebsd.org Subject: Re: CARP and NIC-teaming on ESXi X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Apr 2009 18:13:40 -0000 On Sat, 2009-04-04 at 10:20 +0200, Sebastiaan van Erk wrote: > I see something like this (with carp > advertisements approximately every second): > > 10:08:11.241213 00:00:5e:00:01:4a There are a handful of settings buried deep in ESXi related to virtual switch promiscuous mode, multicast, and MAC spoofing (transmitting from the VRRP/HSRP/CARP address). You have to configure them in about 3-4 places. ~BAS From owner-freebsd-cluster@FreeBSD.ORG Mon Apr 6 21:07:34 2009 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D56B1065B20 for ; Mon, 6 Apr 2009 21:07:34 +0000 (UTC) (envelope-from sebster@sebster.com) Received: from mail.sebster.com (mail.sebster.com [193.46.80.82]) by mx1.freebsd.org (Postfix) with SMTP id C1B0F8FC1F for ; Mon, 6 Apr 2009 21:07:33 +0000 (UTC) (envelope-from sebster@sebster.com) Received: (qmail 29625 invoked from network); 6 Apr 2009 21:07:32 -0000 Received: from unknown (HELO ?192.168.1.33?) (sebster@195.240.254.51) by 10.0.98.3 with SMTP; 6 Apr 2009 21:07:31 -0000 Message-ID: <49DA6F13.8000908@sebster.com> Date: Mon, 06 Apr 2009 23:07:31 +0200 From: Sebastiaan van Erk User-Agent: Thunderbird 2.0.0.21 (X11/20090318) MIME-Version: 1.0 To: "Brian A. Seklecki" References: <49D71864.3010403@sebster.com> <1239040711.16390.2195.camel@soundwave.ws.pitbpa0.priv.collaborativefusion.com> In-Reply-To: <1239040711.16390.2195.camel@soundwave.ws.pitbpa0.priv.collaborativefusion.com> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms030008040002090406030807" Cc: freebsd-cluster@freebsd.org Subject: Re: CARP and NIC-teaming on ESXi X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Apr 2009 21:07:35 -0000 This is a cryptographically signed message in MIME format. --------------ms030008040002090406030807 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, Thanks for the reply. I had already found the promiscuous mode of the virtual switch that the boxes are on, otherwise CARP does not work at all. CARP itself works (I have it set up with 2 real ESXi machines each with a virtual FreeBSD machine which is doing CARP, so that if either physical machine fails, the CARP kicks in). The only problem is that together with NIC teaming on ESXi it breaks; even when I just set the second NIC to standby mode. Do you have any idea where I can find the settings and which ones to tune? Regards, Sebastiaan Brian A. Seklecki wrote: > On Sat, 2009-04-04 at 10:20 +0200, Sebastiaan van Erk wrote: >> I see something like this (with carp >> advertisements approximately every second): >> >> 10:08:11.241213 00:00:5e:00:01:4a > > There are a handful of settings buried deep in ESXi related to virtual > switch promiscuous mode, multicast, and MAC spoofing (transmitting from > the VRRP/HSRP/CARP address). You have to configure them in about 3-4 > places. ~BAS > > > > --------------ms030008040002090406030807 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJUTCC AwMwggJsoAMCAQICEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDYzMDEzNTE1N1oX DTA5MDYzMDEzNTE1N1owaDEQMA4GA1UEBBMHdmFuIEVyazETMBEGA1UEKhMKU2ViYXN0aWFh bjEbMBkGA1UEAxMSU2ViYXN0aWFhbiB2YW4gRXJrMSIwIAYJKoZIhvcNAQkBFhNzZWJzdGVy QHNlYnN0ZXIuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsJDDAeYHVmH/ GVxi+bhFx27dmg++9BdhPJfk8k041sqEqq7oXnR2GT54quY3Ac7A1BuOM2JvoICraGmjud4y b3EanRnqGIK6iH+VAhhTlV/Owrb2Qm1e13DLxwLp1SocSQl4IrEbF9Y5H3ASdIrE0iFqkpju nPiiHeNhz3LaI5ipjiluKYoH+F6gPx8njHoaDxPePCkSLg4r0IA0afLM74LVZxCRBZEfyRZS J6VVUJefKlz91dWSzR/3xSw/rO4u9Ds/Zh7VBUKy3K+YFryHxRpUek0gSepE1b70Q39L9Sqd M/NZqMvFpwrqgW2Zh2Nh8nqRge90maR4ypBzz3GzLwIDAQABozAwLjAeBgNVHREEFzAVgRNz ZWJzdGVyQHNlYnN0ZXIuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAS1Sk NMgDVzb0ktO9tPPacV0KdKhTYOHcICVmuDEe2sFHOkjLAI1iAKp640pqJEVqvRnfRcCFJ9hK koPjjVZ+ui2rVmJWBG6FSloLRS/YYED4tUAw6DQhK61UOpjkpQxjCdm+5bHG/2ZgJAda1j0x uiN822+xFkcaW/5PQgxSRxcwggMDMIICbKADAgECAhBTfA2qzDbriiQxLX7NFGqlMA0GCSqG SIb3DQEBBQUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD QTAeFw0wODA2MzAxMzUxNTdaFw0wOTA2MzAxMzUxNTdaMGgxEDAOBgNVBAQTB3ZhbiBFcmsx EzARBgNVBCoTClNlYmFzdGlhYW4xGzAZBgNVBAMTElNlYmFzdGlhYW4gdmFuIEVyazEiMCAG CSqGSIb3DQEJARYTc2Vic3RlckBzZWJzdGVyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP ADCCAQoCggEBALCQwwHmB1Zh/xlcYvm4Rcdu3ZoPvvQXYTyX5PJNONbKhKqu6F50dhk+eKrm NwHOwNQbjjNib6CAq2hpo7neMm9xGp0Z6hiCuoh/lQIYU5VfzsK29kJtXtdwy8cC6dUqHEkJ eCKxGxfWOR9wEnSKxNIhapKY7pz4oh3jYc9y2iOYqY4pbimKB/heoD8fJ4x6Gg8T3jwpEi4O K9CANGnyzO+C1WcQkQWRH8kWUielVVCXnypc/dXVks0f98UsP6zuLvQ7P2Ye1QVCstyvmBa8 h8UaVHpNIEnqRNW+9EN/S/UqnTPzWajLxacK6oFtmYdjYfJ6kYHvdJmkeMqQc89xsy8CAwEA AaMwMC4wHgYDVR0RBBcwFYETc2Vic3RlckBzZWJzdGVyLmNvbTAMBgNVHRMBAf8EAjAAMA0G CSqGSIb3DQEBBQUAA4GBAEtUpDTIA1c29JLTvbTz2nFdCnSoU2Dh3CAlZrgxHtrBRzpIywCN YgCqeuNKaiRFar0Z30XAhSfYSpKD441Wfrotq1ZiVgRuhUpaC0Uv2GBA+LVAMOg0ISutVDqY 5KUMYwnZvuWxxv9mYCQHWtY9MbojfNtvsRZHGlv+T0IMUkcXMIIDPzCCAqigAwIBAgIBDTAN BgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTES MBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UE CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBl cnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0 aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMC WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1Ro YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GN ADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTX p6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYB Af8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBl cnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYD VQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2as Zw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSe JVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHT HUb/XV9lTzGCA3EwggNtAgEBMHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBD b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp bCBJc3N1aW5nIENBAhBTfA2qzDbriiQxLX7NFGqlMAkGBSsOAwIaBQCgggHQMBgGCSqGSIb3 DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA5MDQwNjIxMDczMVowIwYJKoZI hvcNAQkEMRYEFPQ6iby5GOMdo7UaGutZXkTeSnpqMF8GCSqGSIb3DQEJDzFSMFAwCwYJYIZI AWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUr DgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTEl MCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3Rl IFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwgYcGCyqG SIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRp bmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vp bmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEBBQAEggEAXTj2MLQ6tUYZUJQk awMELGjObwUdUzRuw4JL389dayDKF3YY625mOxpazHzbnlXX1mwYBY6s07PK7XVRqGMnRUja Wetals480ZDxCFRahYyYpNL/UeuZ9tvuD75TfrHhbFQ82ugrjymCs+b5U+hyN1n27phlB/CI fBbJwypx5DNRQaJE0R1AjQJru4jt69eIH1dUqn6NxXcHjrl26XLKmaWazcQhQMDrZ0h6pLEV O6kZikneCCrk2HhfWqI/bxro9wkTfo15qvxMDRlLnNLshRaYmWV5UcoJQgQSEGobf50DZ6h6 K8LkukJsm70RBp8S3iJJUxwdigPFgg4JHbwQxgAAAAAAAA== --------------ms030008040002090406030807-- From owner-freebsd-cluster@FreeBSD.ORG Wed Apr 8 10:29:45 2009 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A769106564A for ; Wed, 8 Apr 2009 10:29:45 +0000 (UTC) (envelope-from sebster@sebster.com) Received: from mail.sebster.com (mail.sebster.com [193.46.80.82]) by mx1.freebsd.org (Postfix) with SMTP id 5ADB58FC17 for ; Wed, 8 Apr 2009 10:29:44 +0000 (UTC) (envelope-from sebster@sebster.com) Received: (qmail 54726 invoked from network); 8 Apr 2009 10:29:42 -0000 Received: from unknown (HELO ?10.10.1.5?) (sebster@212.83.238.46) by 10.0.98.3 with SMTP; 8 Apr 2009 10:29:42 -0000 Message-ID: <49DC7C96.2050203@sebster.com> Date: Wed, 08 Apr 2009 12:29:42 +0200 From: Sebastiaan van Erk User-Agent: Thunderbird 2.0.0.21 (X11/20090318) MIME-Version: 1.0 To: freebsd-cluster@freebsd.org Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms000905000207020407030606" Subject: CARP, openvpn in bridged mode, and ping X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Apr 2009 10:29:45 -0000 This is a cryptographically signed message in MIME format. --------------ms000905000207020407030606 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, I have the following setup: two servers with a virtual LAN IP address shared with CARP (the hosts are 10.0.80.77 and 10.0.80.76 and the virtual IP address is 10.0.80.1). When I ping the VIP from any host on the LAN, it works fine. Next I have some openvpn clients (both 10.0.80.77 and 10.0.80.76 have openvpn servers on their external IPs). The client IPs are on the LAN using a bridge and are 10.0.80.150 (linux client) and 10.0.80.6 (freebsd client). From linux I can ping the VIP (10.0.80.1) just fine, but when I do arping I see (with tcpdump) that the the ARP requests are received by the carp master on the tap0 device, but that it does not reply. From a FreeBSD VPN client I cannot ping the VIP (10.0.80.1), because it does the ARP requests indefinitely and gets no answer. Both machines ping to the other hosts on the LAN just fine (e.g., all of them can ping 10.0.80.77 just fine). Is there any way to get ARP to work (and thereby, ping to work) in this configuration? Regards, Sebastiaan PS: the relevant ifconfig info is: 10.0.80.77 (carp master and vpn server): em1: flags=8943 metric 0 mtu 1500 options=98 ether 00:0c:29:61:2a:55 inet 10.0.80.77 netmask 0xffffff00 broadcast 10.0.80.255 media: Ethernet autoselect (1000baseTX ) status: active bridge0: flags=8843 metric 0 mtu 1500 ether 12:d8:09:8d:1b:88 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: tap0 flags=143 ifmaxaddr 0 port 9 priority 128 path cost 2000000 member: em1 flags=143 ifmaxaddr 0 port 2 priority 128 path cost 20000 carp1: flags=49 metric 0 mtu 1500 inet 10.0.80.1 netmask 0xffffff00 carp: MASTER vhid 174 advbase 1 advskew 0 tap0: flags=8943 metric 0 mtu 1500 ether 00:bd:c0:02:00:00 Opened by PID 1199 10.0.80.150 (the linux openvpn client): tap0 Link encap:Ethernet HWaddr 46:c2:27:c9:36:e3 inet addr:10.0.80.150 Bcast:10.0.80.255 Mask:255.255.255.0 inet6 addr: fe80::44c2:27ff:fec9:36e3/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:34336 errors:0 dropped:0 overruns:0 frame:0 TX packets:12951 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:11939564 (11.9 MB) TX bytes:1191746 (1.1 MB) 10.0.80.6 (the freebsd openvpn client): tap0: flags=8843 metric 0 mtu 1500 ether 00:bd:bf:f6:08:00 inet 10.0.80.6 netmask 0xffffff00 broadcast 10.0.80.255 Opened by PID 71953 --------------ms000905000207020407030606 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJUTCC AwMwggJsoAMCAQICEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDYzMDEzNTE1N1oX DTA5MDYzMDEzNTE1N1owaDEQMA4GA1UEBBMHdmFuIEVyazETMBEGA1UEKhMKU2ViYXN0aWFh bjEbMBkGA1UEAxMSU2ViYXN0aWFhbiB2YW4gRXJrMSIwIAYJKoZIhvcNAQkBFhNzZWJzdGVy QHNlYnN0ZXIuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsJDDAeYHVmH/ GVxi+bhFx27dmg++9BdhPJfk8k041sqEqq7oXnR2GT54quY3Ac7A1BuOM2JvoICraGmjud4y b3EanRnqGIK6iH+VAhhTlV/Owrb2Qm1e13DLxwLp1SocSQl4IrEbF9Y5H3ASdIrE0iFqkpju nPiiHeNhz3LaI5ipjiluKYoH+F6gPx8njHoaDxPePCkSLg4r0IA0afLM74LVZxCRBZEfyRZS J6VVUJefKlz91dWSzR/3xSw/rO4u9Ds/Zh7VBUKy3K+YFryHxRpUek0gSepE1b70Q39L9Sqd M/NZqMvFpwrqgW2Zh2Nh8nqRge90maR4ypBzz3GzLwIDAQABozAwLjAeBgNVHREEFzAVgRNz ZWJzdGVyQHNlYnN0ZXIuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAS1Sk NMgDVzb0ktO9tPPacV0KdKhTYOHcICVmuDEe2sFHOkjLAI1iAKp640pqJEVqvRnfRcCFJ9hK koPjjVZ+ui2rVmJWBG6FSloLRS/YYED4tUAw6DQhK61UOpjkpQxjCdm+5bHG/2ZgJAda1j0x uiN822+xFkcaW/5PQgxSRxcwggMDMIICbKADAgECAhBTfA2qzDbriiQxLX7NFGqlMA0GCSqG SIb3DQEBBQUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD QTAeFw0wODA2MzAxMzUxNTdaFw0wOTA2MzAxMzUxNTdaMGgxEDAOBgNVBAQTB3ZhbiBFcmsx EzARBgNVBCoTClNlYmFzdGlhYW4xGzAZBgNVBAMTElNlYmFzdGlhYW4gdmFuIEVyazEiMCAG CSqGSIb3DQEJARYTc2Vic3RlckBzZWJzdGVyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP ADCCAQoCggEBALCQwwHmB1Zh/xlcYvm4Rcdu3ZoPvvQXYTyX5PJNONbKhKqu6F50dhk+eKrm NwHOwNQbjjNib6CAq2hpo7neMm9xGp0Z6hiCuoh/lQIYU5VfzsK29kJtXtdwy8cC6dUqHEkJ eCKxGxfWOR9wEnSKxNIhapKY7pz4oh3jYc9y2iOYqY4pbimKB/heoD8fJ4x6Gg8T3jwpEi4O K9CANGnyzO+C1WcQkQWRH8kWUielVVCXnypc/dXVks0f98UsP6zuLvQ7P2Ye1QVCstyvmBa8 h8UaVHpNIEnqRNW+9EN/S/UqnTPzWajLxacK6oFtmYdjYfJ6kYHvdJmkeMqQc89xsy8CAwEA AaMwMC4wHgYDVR0RBBcwFYETc2Vic3RlckBzZWJzdGVyLmNvbTAMBgNVHRMBAf8EAjAAMA0G CSqGSIb3DQEBBQUAA4GBAEtUpDTIA1c29JLTvbTz2nFdCnSoU2Dh3CAlZrgxHtrBRzpIywCN YgCqeuNKaiRFar0Z30XAhSfYSpKD441Wfrotq1ZiVgRuhUpaC0Uv2GBA+LVAMOg0ISutVDqY 5KUMYwnZvuWxxv9mYCQHWtY9MbojfNtvsRZHGlv+T0IMUkcXMIIDPzCCAqigAwIBAgIBDTAN BgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTES MBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UE CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBl cnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0 aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMC WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1Ro YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GN ADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTX p6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYB Af8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBl cnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYD VQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2as Zw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSe JVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHT HUb/XV9lTzGCA3EwggNtAgEBMHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBD b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp bCBJc3N1aW5nIENBAhBTfA2qzDbriiQxLX7NFGqlMAkGBSsOAwIaBQCgggHQMBgGCSqGSIb3 DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA5MDQwODEwMjk0MlowIwYJKoZI hvcNAQkEMRYEFEYU1P9qLbtcD+UHx2+5jFrUSNwzMF8GCSqGSIb3DQEJDzFSMFAwCwYJYIZI AWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUr DgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTEl MCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3Rl IFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwgYcGCyqG SIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRp bmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vp bmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEBBQAEggEAo2p1k5g1hzbX+2x4 hdKS1rFmtMOeRWUMXE6i+HWrIOSScAG9BnzpngJ9nDFXkJu5id6HNmRePFM+aqhNWebw7ECN KkxS2eJ0d1k0J+onqGbEYjKupzOWHeSHvMzAmU9980+5Vwiq+la2ZRZT7465tvaWypEehwbC eupJTJCaEJT0VTnY8jTxlD+GiDkZOKkhyFaYkuhRpL97pU/sdcVxdixrV1eFFQIocA6J8mpo FitmXPYtuyzF4UMD4uM7rZbIENMt2eLy/kOtIWeq1ouKQeALmDx/K+gfWUzaWrb410pJUpO5 l+mfZQcScNiYH4iyscSic5/EXwg2lCXUghXaGwAAAAAAAA== --------------ms000905000207020407030606-- From owner-freebsd-cluster@FreeBSD.ORG Wed Apr 8 14:49:42 2009 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DB79106566C for ; Wed, 8 Apr 2009 14:49:42 +0000 (UTC) (envelope-from sebster@sebster.com) Received: from mail.sebster.com (mail.sebster.com [193.46.80.82]) by mx1.freebsd.org (Postfix) with SMTP id 8E1258FC0A for ; Wed, 8 Apr 2009 14:49:41 +0000 (UTC) (envelope-from sebster@sebster.com) Received: (qmail 58184 invoked from network); 8 Apr 2009 14:49:40 -0000 Received: from unknown (HELO ?10.10.1.5?) (sebster@212.83.238.46) by 10.0.98.3 with SMTP; 8 Apr 2009 14:49:39 -0000 Message-ID: <49DCB983.2070700@sebster.com> Date: Wed, 08 Apr 2009 16:49:39 +0200 From: Sebastiaan van Erk User-Agent: Thunderbird 2.0.0.21 (X11/20090318) MIME-Version: 1.0 To: freebsd-cluster@freebsd.org References: <49DC7C96.2050203@sebster.com> In-Reply-To: <49DC7C96.2050203@sebster.com> Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms030009050200030306040909" Subject: Re: CARP, openvpn in bridged mode, and ping X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Apr 2009 14:49:42 -0000 This is a cryptographically signed message in MIME format. --------------ms030009050200030306040909 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, just to explain further, ARP does not seem to work in the following situation: client (tap0, 10.0.80.6) -> server (tap0 -> bridge0 -> em1, 10.0.80.77 -> carp1, 10.0.80.1) On the server I see on the em1 interface with tcpdump: 16:45:56.992297 00:bd:bf:f6:08:00 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 10.0.80.1 tell 10.0.80.6 but the server does not reply. If the client is directly on the em1 interface (another machine which is directly on the LAN and not via the bridge), the server DOES reply: 16:46:47.502250 00:0c:29:e2:46:c8 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: arp who-has 10.0.80.1 tell 10.0.80.3 16:46:47.502283 00:0c:29:61:2a:55 > 00:0c:29:e2:46:c8, ethertype ARP (0x0806), length 42: arp reply 10.0.80.1 is-at 00:00:5e:00:01:02 Regards, Sebastiaan Sebastiaan van Erk wrote: > Hi, > > I have the following setup: two servers with a virtual LAN IP address > shared with CARP (the hosts are 10.0.80.77 and 10.0.80.76 and the > virtual IP address is 10.0.80.1). > > When I ping the VIP from any host on the LAN, it works fine. > > Next I have some openvpn clients (both 10.0.80.77 and 10.0.80.76 have > openvpn servers on their external IPs). The client IPs are on the LAN > using a bridge and are 10.0.80.150 (linux client) and 10.0.80.6 (freebsd > client). > > From linux I can ping the VIP (10.0.80.1) just fine, but when I do > arping I see (with tcpdump) that the the ARP requests are received by > the carp master on the tap0 device, but that it does not reply. > > From a FreeBSD VPN client I cannot ping the VIP (10.0.80.1), because it > does the ARP requests indefinitely and gets no answer. > > Both machines ping to the other hosts on the LAN just fine (e.g., all of > them can ping 10.0.80.77 just fine). > > Is there any way to get ARP to work (and thereby, ping to work) in this > configuration? > > Regards, > Sebastiaan > > PS: the relevant ifconfig info is: > > 10.0.80.77 (carp master and vpn server): > em1: flags=8943 metric 0 > mtu 1500 > options=98 > ether 00:0c:29:61:2a:55 > inet 10.0.80.77 netmask 0xffffff00 broadcast 10.0.80.255 > media: Ethernet autoselect (1000baseTX ) > status: active > bridge0: flags=8843 metric 0 mtu > 1500 > ether 12:d8:09:8d:1b:88 > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap0 flags=143 > ifmaxaddr 0 port 9 priority 128 path cost 2000000 > member: em1 flags=143 > ifmaxaddr 0 port 2 priority 128 path cost 20000 > carp1: flags=49 metric 0 mtu 1500 > inet 10.0.80.1 netmask 0xffffff00 > carp: MASTER vhid 174 advbase 1 advskew 0 > tap0: flags=8943 metric > 0 mtu 1500 > ether 00:bd:c0:02:00:00 > Opened by PID 1199 > > 10.0.80.150 (the linux openvpn client): > tap0 Link encap:Ethernet HWaddr 46:c2:27:c9:36:e3 > inet addr:10.0.80.150 Bcast:10.0.80.255 Mask:255.255.255.0 > inet6 addr: fe80::44c2:27ff:fec9:36e3/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:34336 errors:0 dropped:0 overruns:0 frame:0 > TX packets:12951 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:100 > RX bytes:11939564 (11.9 MB) TX bytes:1191746 (1.1 MB) > > 10.0.80.6 (the freebsd openvpn client): > tap0: flags=8843 metric 0 mtu 1500 > ether 00:bd:bf:f6:08:00 > inet 10.0.80.6 netmask 0xffffff00 broadcast 10.0.80.255 > Opened by PID 71953 > > --------------ms030009050200030306040909 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJUTCC AwMwggJsoAMCAQICEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDYzMDEzNTE1N1oX DTA5MDYzMDEzNTE1N1owaDEQMA4GA1UEBBMHdmFuIEVyazETMBEGA1UEKhMKU2ViYXN0aWFh bjEbMBkGA1UEAxMSU2ViYXN0aWFhbiB2YW4gRXJrMSIwIAYJKoZIhvcNAQkBFhNzZWJzdGVy QHNlYnN0ZXIuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsJDDAeYHVmH/ GVxi+bhFx27dmg++9BdhPJfk8k041sqEqq7oXnR2GT54quY3Ac7A1BuOM2JvoICraGmjud4y b3EanRnqGIK6iH+VAhhTlV/Owrb2Qm1e13DLxwLp1SocSQl4IrEbF9Y5H3ASdIrE0iFqkpju nPiiHeNhz3LaI5ipjiluKYoH+F6gPx8njHoaDxPePCkSLg4r0IA0afLM74LVZxCRBZEfyRZS J6VVUJefKlz91dWSzR/3xSw/rO4u9Ds/Zh7VBUKy3K+YFryHxRpUek0gSepE1b70Q39L9Sqd M/NZqMvFpwrqgW2Zh2Nh8nqRge90maR4ypBzz3GzLwIDAQABozAwLjAeBgNVHREEFzAVgRNz ZWJzdGVyQHNlYnN0ZXIuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAS1Sk NMgDVzb0ktO9tPPacV0KdKhTYOHcICVmuDEe2sFHOkjLAI1iAKp640pqJEVqvRnfRcCFJ9hK koPjjVZ+ui2rVmJWBG6FSloLRS/YYED4tUAw6DQhK61UOpjkpQxjCdm+5bHG/2ZgJAda1j0x uiN822+xFkcaW/5PQgxSRxcwggMDMIICbKADAgECAhBTfA2qzDbriiQxLX7NFGqlMA0GCSqG SIb3DQEBBQUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD QTAeFw0wODA2MzAxMzUxNTdaFw0wOTA2MzAxMzUxNTdaMGgxEDAOBgNVBAQTB3ZhbiBFcmsx EzARBgNVBCoTClNlYmFzdGlhYW4xGzAZBgNVBAMTElNlYmFzdGlhYW4gdmFuIEVyazEiMCAG CSqGSIb3DQEJARYTc2Vic3RlckBzZWJzdGVyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP ADCCAQoCggEBALCQwwHmB1Zh/xlcYvm4Rcdu3ZoPvvQXYTyX5PJNONbKhKqu6F50dhk+eKrm NwHOwNQbjjNib6CAq2hpo7neMm9xGp0Z6hiCuoh/lQIYU5VfzsK29kJtXtdwy8cC6dUqHEkJ eCKxGxfWOR9wEnSKxNIhapKY7pz4oh3jYc9y2iOYqY4pbimKB/heoD8fJ4x6Gg8T3jwpEi4O K9CANGnyzO+C1WcQkQWRH8kWUielVVCXnypc/dXVks0f98UsP6zuLvQ7P2Ye1QVCstyvmBa8 h8UaVHpNIEnqRNW+9EN/S/UqnTPzWajLxacK6oFtmYdjYfJ6kYHvdJmkeMqQc89xsy8CAwEA AaMwMC4wHgYDVR0RBBcwFYETc2Vic3RlckBzZWJzdGVyLmNvbTAMBgNVHRMBAf8EAjAAMA0G CSqGSIb3DQEBBQUAA4GBAEtUpDTIA1c29JLTvbTz2nFdCnSoU2Dh3CAlZrgxHtrBRzpIywCN YgCqeuNKaiRFar0Z30XAhSfYSpKD441Wfrotq1ZiVgRuhUpaC0Uv2GBA+LVAMOg0ISutVDqY 5KUMYwnZvuWxxv9mYCQHWtY9MbojfNtvsRZHGlv+T0IMUkcXMIIDPzCCAqigAwIBAgIBDTAN BgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTES MBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UE CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBl cnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0 aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMC WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1Ro YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GN ADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTX p6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYB Af8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBl cnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYD VQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2as Zw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSe JVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHT HUb/XV9lTzGCA3EwggNtAgEBMHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBD b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp bCBJc3N1aW5nIENBAhBTfA2qzDbriiQxLX7NFGqlMAkGBSsOAwIaBQCgggHQMBgGCSqGSIb3 DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA5MDQwODE0NDkzOVowIwYJKoZI hvcNAQkEMRYEFILD+1yUl5IKYh8H3BIrdm7ogyXgMF8GCSqGSIb3DQEJDzFSMFAwCwYJYIZI AWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUr DgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTEl MCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3Rl IFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwgYcGCyqG SIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRp bmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vp bmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEBBQAEggEAIiLluJkk5vKErNYM q/AuqhQBw8q85oFYdwbDySCTTx6l3I2KLOBMAONatuYh6QnGXJKjaF3fEzHy0UeKJkTdDhEm kRueLW3rvmmWrgrhgKGNx07cZcBv+oWmTUpoYSoh9ee5hItQ4qMtZZZHFBWePjm88tJs5n7g 6wUH6vQHmgVz8cUoZY8sCLhWbqpqF99r4KJrfcuScvTKelc42yCtF6NXOkzhtRKc9f62lgqF 9lFG08uL3Z41XIKwks8zqpRyH9LwHWHuklHpVuwzQ8tEdCKHmaMZYvktBNmoRcU0D39TFBr/ PaYtWigc9+svzFdWTGdikEZ4FVyJlGeTIKfaBgAAAAAAAA== --------------ms030009050200030306040909-- From owner-freebsd-cluster@FreeBSD.ORG Thu Apr 9 07:43:56 2009 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1A381065716 for ; Thu, 9 Apr 2009 07:43:56 +0000 (UTC) (envelope-from bounces+305227.46043618.562566@icpbounce.com) Received: from smtp2.icpbounce.com (smtp2.icpbounce.com [216.27.93.124]) by mx1.freebsd.org (Postfix) with ESMTP id 6A15E8FC1C for ; Thu, 9 Apr 2009 07:43:56 +0000 (UTC) (envelope-from bounces+305227.46043618.562566@icpbounce.com) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp2.icpbounce.com (Postfix) with ESMTP id 4B746F877B for ; Thu, 9 Apr 2009 03:22:35 -0400 (EDT) Date: Thu, 9 Apr 2009 03:22:35 -0400 To: freebsd-cluster@freebsd.org From: Global Access Travel Message-ID: <5ac96c3f266b730658a62652e94e26cf@localhost.localdomain> X-Priority: 3 X-Mailer: PHPMailer [version 1.72] Errors-To: bounces+305227.46043618.562566@icpbounce.com X-List-Unsubscribe: X-Unsubscribe-Web: X-ICPINFO: X-Return-Path-Hint: bounces+305227.46043618.562566@icpbounce.com MIME-Version: 1.0 Content-Type: text/plain; charset = "utf-8" Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Private Shore Excursions-Turkey X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Apr 2009 07:43:59 -0000 [http://www.turkeycalling.us] PRIVATE SHORE EXCURSIONS- TURKEY Your cruise clients will make the best of their time in Turkey on a private shore excursion! Istanbul Kusadasi & Ephesus [mailto:incoming@gaturkey.com?subject=Private Shore Excursions- Turkey] **************************************************************************** Yasal Uyarı; Bu e-posta, sadece adreste belirtilen kisi veya kurulusun kullanimini hedeflemekte olup,mesajda yer alan bilgiler kisiye ozel ve gizli olabilir, yasalar ya da anlasmalar geregi ücüncü kisiler ile paylasilmasi mümkün olmayabilir.Mesaji alan kisi, mesajin gönderilmek istendigi kisi veya kurulus degilse,bu mesaji yaymak,dagitmak veya kopyalamak yasaktir Mesaj tarafiniza yanlislikla ulasmissa lütfen mesaji geri gönderiniz ve sisteminizden siliniz. Global Turizm Hizmetleri Anonim Sirketi bu mesajin icerigi ile ilgili olarak hicbir hukuksal sorumlulugu kabul etmez. **************************************************************************** Disclaimer; This e-mail communication is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and that may not be made public by law or agreement. If the recipient of this message is not the intended recipient or entity, you are hereby notified that any further dissemination, distribution or copying of this information is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete it from your system. The Global Turizm Hizmetleri Anonim Sirketi does not accept legal responsibility for the contents of this message. *********************************************************************************************** Yasal Uyarı; Bu e-posta, sadece adreste belirtilen kisi veya kurulusun kullanimini hedeflemekte olup,mesajda yer alan bilgiler kisiye ozel ve gizli olabilir, yasalar ya da anlasmalar geregi ücüncü kisiler ile paylasilmasi mümkün olmayabilir.Mesaji alan kisi, mesajin gönderilmek istendigi kisi veya kurulus degilse,bu mesaji yaymak,dagitmak veya kopyalamak yasaktir Mesaj tarafiniza yanlislikla ulasmissa lütfen mesaji geri gönderiniz ve sisteminizden siliniz. Global Turizm Hizmetleri Anonim Sirketi bu mesajin icerigi ile ilgili olarak hicbir hukuksal sorumlulugu kabul etmez. ********************************************************************************************** Disclaimer; This e-mail communication is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential and that may not be made public by law or agreement. If the recipient of this message is not the intended recipient or entity, you are hereby notified that any further dissemination, distribution or copying of this information is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete it from your system. The Global Turizm Hizmetleri Anonim Sirketi does not accept legal responsibility for the contents of this message. This message was sent by: Global Access Incoming, Nuzhetiye cad, istanbul, besiktas 34357, Turkey Powered by iContact: http://freetrial.icontact.com To be removed click here: http://app.icontact.com/icp/mmail-mprofile.pl?r=46043618&l=82228&s=EL1A&m=562566&c=305227 Forward to a friend: http://app.icontact.com/icp/sub/forward?m=562566&s=46043618&c=EL1A&cid=305227