From owner-freebsd-xen@freebsd.org Wed Aug 5 10:25:56 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 497E39B351F for ; Wed, 5 Aug 2015 10:25:56 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 31D0C119 for ; Wed, 5 Aug 2015 10:25:56 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: by mailman.ysv.freebsd.org (Postfix) id 2EA1A9B351E; Wed, 5 Aug 2015 10:25:56 +0000 (UTC) Delivered-To: xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D4079B351C for ; Wed, 5 Aug 2015 10:25:56 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.krpservers.com", Issuer "COMODO RSA Domain Validation Secure Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 92F02118 for ; Wed, 5 Aug 2015 10:25:52 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by smtp.krpservers.com (8.14.9/8.14.9) with ESMTP id t75APl9F034452 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Aug 2015 11:25:48 +0100 (BST) (envelope-from kpielorz_lst@tdx.co.uk) Date: Wed, 05 Aug 2015 11:25:41 +0100 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , xen@FreeBSD.org Subject: Re: Boot FreeBSD 10 in HVM mode under Xen? Message-ID: <3544DE5D0E56BBBE05C573B4@[10.12.30.106]> In-Reply-To: <55BB8DB2.6040705@citrix.com> References: <55BB8392.4050000@citrix.com> <55BB8DB2.6040705@citrix.com> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Aug 2015 10:25:56 -0000 --On 31 July 2015 17:01 +0200 Roger Pau Monn=C3=A9 = wrote: > Yes, this is certainly possible. IIRC I posted a patch to the > freebsd-xen mailing list in order to do that. I will try to find/refresh > it and post it again so you can try it. It might have to wait until > Monday however, since I'm quite busy today. Hi, I realise you're busy - but did you get a chance to dig out that patch? -=20 We're just installing hotfixes here [which reminded me - as we have an=20 number of non-agile HVM machines to move around! :(] Thanks, -Karl From owner-freebsd-xen@freebsd.org Thu Aug 6 11:47:12 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 770069B3270 for ; Thu, 6 Aug 2015 11:47:12 +0000 (UTC) (envelope-from prvs=65356f562=roger.pau@citrix.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 60CD61C56 for ; Thu, 6 Aug 2015 11:47:12 +0000 (UTC) (envelope-from prvs=65356f562=roger.pau@citrix.com) Received: by mailman.ysv.freebsd.org (Postfix) id 5DF129B326F; Thu, 6 Aug 2015 11:47:12 +0000 (UTC) Delivered-To: xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C8DC9B326D for ; Thu, 6 Aug 2015 11:47:12 +0000 (UTC) (envelope-from prvs=65356f562=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 152471C55 for ; Thu, 6 Aug 2015 11:47:11 +0000 (UTC) (envelope-from prvs=65356f562=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.15,622,1432598400"; d="scan'208";a="291990088" Message-ID: <55C34932.10307@citrix.com> Date: Thu, 6 Aug 2015 13:46:58 +0200 From: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Karl Pielorz , Subject: Re: Boot FreeBSD 10 in HVM mode under Xen? References: <55BB8392.4050000@citrix.com> <55BB8DB2.6040705@citrix.com> <3544DE5D0E56BBBE05C573B4@[10.12.30.106]> In-Reply-To: <3544DE5D0E56BBBE05C573B4@[10.12.30.106]> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 11:47:12 -0000 El 05/08/15 a les 12.25, Karl Pielorz ha escrit: > > > --On 31 July 2015 17:01 +0200 Roger Pau Monné wrote: > >> Yes, this is certainly possible. IIRC I posted a patch to the >> freebsd-xen mailing list in order to do that. I will try to find/refresh >> it and post it again so you can try it. It might have to wait until >> Monday however, since I'm quite busy today. > > Hi, > > I realise you're busy - but did you get a chance to dig out that patch? > - We're just installing hotfixes here [which reminded me - as we have an > number of non-agile HVM machines to move around! :(] Hello, Sorry for the delay, as usual I've been busy with a bunch of other stuff. You can find the patch against HEAD here: https://people.freebsd.org/~royger/0001-xen-allow-disabling-PV-disks-and-nics.patch The description is self-explanatory I think. I've not tried to apply it to stable-10, but I don't think there have been many changes in this area in general. If you need help backporting it ping me and I will provide a suitable version. Roger. From owner-freebsd-xen@freebsd.org Thu Aug 6 13:06:40 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92EB19B4FEF for ; Thu, 6 Aug 2015 13:06:40 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 7A0059C0 for ; Thu, 6 Aug 2015 13:06:40 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: by mailman.ysv.freebsd.org (Postfix) id 76FF99B4FEE; Thu, 6 Aug 2015 13:06:40 +0000 (UTC) Delivered-To: xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7697D9B4FED for ; Thu, 6 Aug 2015 13:06:40 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from smtp.krpservers.com (smtp.krpservers.com [62.13.128.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.krpservers.com", Issuer "COMODO RSA Domain Validation Secure Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2AAA59BF for ; Thu, 6 Aug 2015 13:06:35 +0000 (UTC) (envelope-from kpielorz_lst@tdx.co.uk) Received: from [10.12.30.106] (vpn01-01.tdx.co.uk [62.13.130.213]) (authenticated bits=0) by smtp.krpservers.com (8.14.9/8.14.9) with ESMTP id t76D6AXB042126 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 6 Aug 2015 14:06:11 +0100 (BST) (envelope-from kpielorz_lst@tdx.co.uk) Date: Thu, 06 Aug 2015 14:06:07 +0100 From: Karl Pielorz To: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= , xen@FreeBSD.org Subject: Re: Boot FreeBSD 10 in HVM mode under Xen? Message-ID: In-Reply-To: <55C34932.10307@citrix.com> References: <55BB8392.4050000@citrix.com> <55BB8DB2.6040705@citrix.com> <3544DE5D0E56BBBE05C573B4@[10.12.30.106]> <55C34932.10307@citrix.com> X-Mailer: Mulberry/4.0.8 (Win32) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 13:06:40 -0000 --On 06 August 2015 13:46 +0200 Roger Pau Monn=C3=A9 = wrote: > Hello, > > Sorry for the delay, as usual I've been busy with a bunch of other > stuff. You can find the patch against HEAD here: > > https://people.freebsd.org/~royger/0001-xen-allow-disabling-PV-disks-and- > nics.patch > > The description is self-explanatory I think. I've not tried to apply it > to stable-10, but I don't think there have been many changes in this > area in general. If you need help backporting it ping me and I will > provide a suitable version. Hi, Thanks for the patch! - I couldn't apply it automatically (not entirely=20 sure what to do with a git diff like that :) - and, the hvm.c part of patch = wouldn't apply cleanly anyway (too much difference). *Having said that* - I applied it by hand to my 10.1-R system, and it seems = to work perfectly. I've put the XENHVM and xenpci back in the kernel,=20 re-enabled veridian for that VM, and put 'hw.xen.disable_pv_nics=3D1' in = boot=20 loader. I now have a PV disk, but HVM NIC (re0) based machine - that's agile, and=20 works with OpenVPN and inter-VM routing. I can post the standard 'diff -u' patchset against 10.1 back to the list if = needed (I know other people had PV NIC issues and were using the same=20 non-agile "boot as HVM" fix for it). Cheers, -Karl From owner-freebsd-xen@freebsd.org Thu Aug 6 13:21:51 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E1E7E9B4415 for ; Thu, 6 Aug 2015 13:21:51 +0000 (UTC) (envelope-from prvs=65356f562=roger.pau@citrix.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CAC6FEF9 for ; Thu, 6 Aug 2015 13:21:51 +0000 (UTC) (envelope-from prvs=65356f562=roger.pau@citrix.com) Received: by mailman.ysv.freebsd.org (Postfix) id C99BC9B4410; Thu, 6 Aug 2015 13:21:51 +0000 (UTC) Delivered-To: xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C92BC9B440F for ; Thu, 6 Aug 2015 13:21:51 +0000 (UTC) (envelope-from prvs=65356f562=roger.pau@citrix.com) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Verizon Public SureServer CA G14-SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 73912EF8 for ; Thu, 6 Aug 2015 13:21:51 +0000 (UTC) (envelope-from prvs=65356f562=roger.pau@citrix.com) X-IronPort-AV: E=Sophos;i="5.15,622,1432598400"; d="scan'208";a="292021795" Message-ID: <55C35F30.8020309@citrix.com> Date: Thu, 6 Aug 2015 15:20:48 +0200 From: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Karl Pielorz , Subject: Re: Boot FreeBSD 10 in HVM mode under Xen? References: <55BB8392.4050000@citrix.com> <55BB8DB2.6040705@citrix.com> <3544DE5D0E56BBBE05C573B4@[10.12.30.106]> <55C34932.10307@citrix.com> In-Reply-To: Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2015 13:21:52 -0000 El 06/08/15 a les 15.06, Karl Pielorz ha escrit: > > --On 06 August 2015 13:46 +0200 Roger Pau Monné > wrote: > >> Hello, >> >> Sorry for the delay, as usual I've been busy with a bunch of other >> stuff. You can find the patch against HEAD here: >> >> https://people.freebsd.org/~royger/0001-xen-allow-disabling-PV-disks-and- >> nics.patch >> >> The description is self-explanatory I think. I've not tried to apply it >> to stable-10, but I don't think there have been many changes in this >> area in general. If you need help backporting it ping me and I will >> provide a suitable version. > > Hi, > > Thanks for the patch! - I couldn't apply it automatically (not entirely > sure what to do with a git diff like that :) - and, the hvm.c part of > patch wouldn't apply cleanly anyway (too much difference). > > *Having said that* - I applied it by hand to my 10.1-R system, and it > seems to work perfectly. I've put the XENHVM and xenpci back in the > kernel, re-enabled veridian for that VM, and put > 'hw.xen.disable_pv_nics=1' in boot loader. > > I now have a PV disk, but HVM NIC (re0) based machine - that's agile, > and works with OpenVPN and inter-VM routing. > > I can post the standard 'diff -u' patchset against 10.1 back to the list > if needed (I know other people had PV NIC issues and were using the same > non-agile "boot as HVM" fix for it). I'm going to let the patch sit here for a while so people can test/comment. Since I'm going on vacations next week, if there are no complains I'm planning to apply it when I'm back on the 17th. Roger. From owner-freebsd-xen@freebsd.org Fri Aug 7 09:02:38 2015 Return-Path: Delivered-To: freebsd-xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0ECFB9B5104 for ; Fri, 7 Aug 2015 09:02:38 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id E06731FE0 for ; Fri, 7 Aug 2015 09:02:37 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id DD15F9B5103; Fri, 7 Aug 2015 09:02:37 +0000 (UTC) Delivered-To: xen@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2CE29B5102 for ; Fri, 7 Aug 2015 09:02:37 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: from mail-wi0-x233.google.com (mail-wi0-x233.google.com [IPv6:2a00:1450:400c:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 51D481FDD for ; Fri, 7 Aug 2015 09:02:37 +0000 (UTC) (envelope-from seanrees@gmail.com) Received: by wicne3 with SMTP id ne3so52337923wic.1 for ; Fri, 07 Aug 2015 02:02:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=UjU05XL8yFOtNCiDfReXPfdk5SZ4w7fJWwhAm7oUX64=; b=nuETuVHokxFR3vaAm5ks0T1R591+pQRaeeTbcGGc0MhYv6h1y5TV68pDs5HtDygiuh omEcG55PWJ3AnM0cn9yWK9VG7MtEUFqOPX/VwTbxvtzgxYoS1mdEENLq964PxY/Hqa90 B4k2B/kBD9MvV34FCUC+DH408B+qiCqDNlBu73QZvnpVOWgf1hikrl5Za94Vu1cc/Xf2 QXWGNZB7qQD3WTARkYbatMU3k7cZK05el7Ki/QZqZyXVEJDyfhlyEqRqDKurX8YmxaCg Gh6ogV5KyY6e0ZxUjvCza7h2PSJwxMEHiK+xJs4rJnd+7YWlFMBZiWh0eCMmnBFrcXFn wUFw== MIME-Version: 1.0 X-Received: by 10.180.86.163 with SMTP id q3mr5121504wiz.75.1438938155722; Fri, 07 Aug 2015 02:02:35 -0700 (PDT) Received: by 10.28.99.193 with HTTP; Fri, 7 Aug 2015 02:02:35 -0700 (PDT) In-Reply-To: References: <40E060AB-F0B9-48F3-9309-947CAB5C1939@sarenet.es> <5A7F36D35FF28CD4B9D2DE92@10.12.30.106> Date: Fri, 7 Aug 2015 10:02:35 +0100 Message-ID: Subject: Re: Poor network performance with Xen + OpenVPN? From: "seanrees@gmail.com" To: Egoitz Aurrekoetxea Cc: Karl Pielorz , "freebsd-xen@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 07 Aug 2015 09:02:38 -0000 I've not tried running OpenVPN against itself in a test bed. I no longer have the option as I had my provider downgrade my problematic VM from 4.4 to 3.4, which made the problem disappear. The OpenVPN config's I use are pretty standard. Server: port XXYYZ dev tun ca CA cert CERT key KEY dh DH server A.B.C.D 255.255.255.0 ifconfig-pool-persist ipp.txt keepalive 2 8 comp-lzo user nobody group nobody persist-key persist-tun status openvpn-status.log verb 3 push "dhcp-option DNS 8.8.8.8" push "dhcp-option DNS 8.8.4.4" Client: client dev tun proto udp remote A XXYYZ remote B XXYYZ resolv-retry infinite nobind user nobody group nobody persist-key persist-tun ca CA cert CERT key KEY ns-cert-type server verb 3 ping 2 ping-restart 5 On Sat, Aug 1, 2015 at 12:45 PM, Egoitz Aurrekoetxea wrote: > Could you paste your onpenvpn=E2=80=99s server and client config files?. = Which OS > the client runs? > > Have you tested this openvpn connectivity without any network devices > between them?. With for example an ip in the > loopback interface of the openvpn server? > > Best regards, > > > [image: sarenet] > *Egoitz Aurrekoetxea* > Departamento de sistemas > 944 209 470 > Parque Tecnol=C3=B3gico. Edificio 103 > 48170 Zamudio (Bizkaia) > egoitz@sarenet.es > www.sarenet.es > > Antes de imprimir este correo electr=C3=B3nico piense si es necesario hac= erlo. > > El 30/7/2015, a las 11:41, seanrees@gmail.com escribi=C3=B3: > > I have the same issue. I managed to track it down to a difference between > somewhere between Xen 3.4 and 4.4 (my provider uses both). > > 3.4 works fine (I can sustain a few mbps through OpenVPN). 4.4 I get > ~0.05mbps max. I wasn't able to track it down. > > Both with the virtual drivers (if_xn) with all combinations of tso, lro, > rxcsum and txcsum on/off tested. > > Sean > > On Wed, Jul 29, 2015 at 2:58 PM, Karl Pielorz > wrote: > >> >> >> --On 29 July 2015 14:28 +0200 Egoitz Aurrekoetxea >> wrote: >> >> Hi! >>> >>> Have you disabled tso, lro and friends?. >>> >>> >> If by that you mean, >> >> ifconfig xn0 -rxcsum -txcsum -tso4 -lro >> >> Yes - I've tried that on the virtual host running OpenVPN, it didn't see= m >> to make any difference :( >> >> I'll hopefully get some more time later to look at it again. >> >> Regards, >> >> -Karl >> _______________________________________________ >> freebsd-xen@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-xen >> To unsubscribe, send any mail to "freebsd-xen-unsubscribe@freebsd.org" >> > > >