From owner-freebsd-net@freebsd.org Sun Dec 17 07:12:02 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F552E80E16 for ; Sun, 17 Dec 2017 07:12:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 881FC66570 for ; Sun, 17 Dec 2017 07:12:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vBH7C29c017350 for ; Sun, 17 Dec 2017 07:12:02 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 193246] Bug in IPv6 multicast join(), uncovered by Jenkins Date: Sun, 17 Dec 2017 07:12:02 +0000 X-Bugzilla-Reason: AssignedTo CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: tablosazi.farahan@gmail.com X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Dec 2017 07:12:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D193246 vali gholami changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |tablosazi.farahan@gmail.com --- Comment #14 from vali gholami --- http://00014.ir http://1-flymusic.ir http://8y8.ir http://abanshargh.ir http://akstoaks.ir http://alibaba-fans.ir http://alvand-ads.ir http://amingames.ir http://amlake-pasargad.ir http://androidsystem.ir http://arax24.ir http://arax724.ir http://armanhardware.ir http://arsisgame= .ir http://asanban.ir http://asanbaran.ir http://asgas.ir http://ashanews.ir http://asl-ic.ir http://astakala.ir http://atromarket.ir http://azarpajang.= ir http://bahartent.ir http://bartarresins.ir http://bazigaranesahne.ir http://bermudasystem.ir http://buyclockfantasy.ir http://cs8.ir http://d77.= ir http://dresskade.ir http://drhesabisch.ir http://editexpert.ir http://eemenshop.ir http://ehsa30.ir http://elameharighearjmand.ir http://e-larestan.ir http://elsku.ir http://emadcenter.ir http://eta90.ir http://far30sms.ir http://fixpost.ir http://fsbigroup.ir http://gemgem.ir http://glutenfree.ir http://group-software.ir http://harim-pak.ir http://hdserial.ir http://healthplanner.ir http://homana-nikooei.ir http://honardorcheh.ir http://honarshiraz.ir http://hsplaser.ir http://insa= t.ir http://iranitb.ir http://iranvmag.ir http://irboiler.ir http://isuntrade.ir http://ithandmade.ir http://jahantest1.ir http://karevanhayeqadir.ir http://kashmarsalam.ir http://kconf.ir http://kore2iran.ir http://kosar-kal= a.ir http://mahdidevotees.ir http://marjaehamayesh.ir http://mgolden.ir http://mhosein.ir http://mohsenmirzazadeh.ir http://myalibabamusic.ir http://nama94.ir http://nettrick.ir http://niaze-rooz.ir http://noaradecor.= ir http://nod32-pass.ir http://novindpfile.ir http://n-vasegh.ir http://parlpd= .ir http://photoselfi.ir http://pooyawood.ir http://roofbam.ir http://saadatedu= .ir http://saba-gostar.ir http://sadafwood.ir http://sadcover.ir http://s-amini= .ir http://sarebanekavir.ir http://sazmansokhan.ir http://shafaghgostaran.ir http://shandizasansor.ir http://sh-iranshahr.ir http://sms7000.ir http://sogandmusic.ir http://steelfood.ir http://sticker1.ir http://technoguard.ir http://tegolestan.ir http://telegramup.ir http://torbat24.ir http://trustech.ir http://turkmenili.ir http://vray4max.= ir http://winsoftware.ir http://www.3hf.ir http://www.4drupal.ir http://www.arazinet.ir http://www.arianagame.ir http://www.clickbartar.ir http://www.maxstoreco.ir http://www.daryabchat.ir http://www.eset-ir.ir http://www.marketstudies.ir http://www.raadmehr.ir http://www.rcmb.ir http://www.shahinmag.ir http://www.starfam.ir http://www.steel-industrial.ir http://www.suqr.ir http://www.tajervenizi.ir http://www.zarrindesign.ir http://yahoo-shop.ir ------------------------------------------------------------ http://appsray= .ir http://rayit.ir http://webonyan.ir http://raypress.ir http://webbonyan.ir http://gameray.ir http://egameweb.ir http://webiweb.ir http://raystore.ir http://raysecurity.ir http://webegame.ir http://rayananews.ir http://webitt= .ir http://alsafir.ir http://www.fun30t.ir http://www.adrh.ir http://2funsara.ir http://www.saesamane.ir http://themefars.ir http://irhip.ir http://www.sharj25.ir http://maraghehee.ir http://mathpub.ir http://seminar-learning.ir http://shahrekord-ads.ir http://mobchat.ir http://www.nativeiran.ir http://www.water-iut.ir http://www.irancnco.ir http://itlovers.ir http://www.shamimemahneshan.ir http://kootool.ir http://www.waghe-e.ir http://www.ncema.ir http://hamayeshmehr.ir http://baranclip.ir http://darab-ads.ir http://nooreravand.ir http://www.bonarazadegantaziyeh.ir http://www.sorkhaabi.ir http://janjalynews.ir http://fastcamp.ir http://www.sekeh337.ir http://www.iqc13.ir http://www.aliarm.ir http://www.malakbanoo.ir http://www.pt30.ir http://www.rianco.ir http://www.tex-one.ir http://www.facialmask.ir http://takestan-ads.ir http://www.shervinpardaz.ir http://www.weecharge.ir http://hamedan-music.ir http://www.2pmusic.ir http://www.memarydownload.ir http://www.sayeh66.ir http://www.tobuy.ir http://www.isfahancycling.ir http://tehran-investment.ir http://neginemahvelat.ir http://www.iran-night.ir http://soft2015.ir http://2roos.ir http://elschool.ir http://www.chahkor.ir http://naubahar.ir http://majidshakeri.ir http://www.smsjadid.ir http://www.mountainguides.ir http://nice-sms.ir http://aeoiconf.ir http://www.mathrde.ir http://bizir.ir http://nafis-co.ir http://rbt-pishvaz.ir http://hashtagco.ir http://www.mdcorp.ir http://mubonit.ir http://storeha.ir http://www.faraghat-kh.ir http://www.ilamph.ir http://fatemyeh-ravand.ir http://www.alifr.ir http://oxindata.ir http://biofa.ir http://sinababaei.ir http://zarinfa.ir http://www.hyppercom.ir http://www.dlall.ir http://www.arse3.ir http://www.polkanhaml.ir http://sportingshop.ir http://abrebaran.ir http://ava-code.ir http://modarooz.ir http://www.cgmaster.ir http://www.rafeeds.ir http://isf-26.ir http://mesbahrc.ir http://garch98.ir http://www.magtour.ir http://rasalinux= .ir http://photo4u.ir http://parscamp.ir http://www.bazare-amlak.ir http://afathi.ir http://zyaoddin.ir http://www.chbchess.ir http://www.miladlabs.ir http://nimaadvertise.ir http://sms-inbox.ir http://jo0k.ir http://rankalexa1.ir http://trmaw.ir http://ig114.ir http://foxtarh.ir http://dailex.ir http://graphicworld.ir http://www.jccac.= ir http://aazam9.ir http://farhangishu.ir http://rohabmusic.ir http://petroleum-eng.ir http://www.redcircle.ir http://www.webhostsearch.ir http://www.besharate-no.ir http://sadebegir.ir http://sadrdanesh.ir http://joks.ir http://98patoghi.ir http://iranturkmenleri.ir http://bamboblog.ir http://amarm.ir http://www.drmotamednejad.ir http://www.adskg.ir http://akschin.ir http://logosaz3d.ir http://ifmee.ir http://khaf-tavan.ir http://www.asalbeheshti.ir http://www.tannazghorbani.ir http://aroosbahar.ir http://hismusic.ir http://abdarya.ir http://hirubsungharchak.ir http://neskaenergy.ir http://alborzbano.ir http://www.hamyari-lr.ir http://10cec.ir http://www.mpcms.ir http://parmisseed.ir http://itraveltour.ir http://maghalateisi.ir http://payamak21.ir http://farsipaper.ir http://www.f3000.ir http://www.aminmoshaver.ir http://btc-info.ir http://ayoubsys.ir http://dr-seo.ir http://salehmedia.ir http://resaneh1.ir http://3rdnci-asna-ipp.ir http://alishateri.ir http://gis98.ir http://www.darolfeiz.ir http://chaijan.ir http://zarasang.ir http://imenhazrati.ir http://aboutmore.ir http://smscontest.ir http://sazefelezipaydar.ir http://www.iltayco.ir http://aradcover.ir http://www.12313.ir http://www.featdesign.ir http://www.golestannjf.ir http://www.istgahniyaz.ir http://ctmusic.ir http://www.farsi-download.ir http://thermowoods.ir http://www.nshgm1.ir http://sepehrportal.ir http://globalchat.ir http://downloadzzz.ir http://majid-phone1.ir http://my-channels.ir http://firoozehcarpet.ir http://footballlist.ir http://sepidarandesign.ir http://shc1.ir http://aabrothers.ir http://skyhop= e.ir http://www.archcontest.ir http://irmarket24.ir http://persian-tourism.ir http://mehranplastic.ir http://vozaramobl.ir http://golbargdairy.ir http://www.altinlift.ir http://drmoodi.ir http://nexclip.ir http://ghasemy5= 0.ir http://www.avizhehsaz.ir http://mavvaj.ir http://overs.ir http://www.pay4sell.ir http://21dl.ir http://www.tasisatpardis.ir http://ostovatours.ir http://downloadpluse.ir http://buxnet.ir http://arad-volleyball.ir http://zanjanli.ir http://cc30shop.ir http://sms-contest.ir http://www.medodarman.ir http://hafezdriving.ir http://talarhashtbehesht.ir http://monomarket.ir http://ptpportal.ir http://nickelodeon.ir http://dlsoftwares.ir http://english4all.ir http://hafez-news.ir http://gamecheats.ir http://payacpc.ir http://www.pazhohe.ir http://qomikala.ir http://hompa.ir http://nasir-wood.= ir http://entezarsoft.ir http://www.pnu-soal.ir http://kiofood.ir http://hotelconference.ir http://frotel47.ir http://ks-co.ir http://es-k.ir http://3venshop.ir http://vectoria.ir http://farskids1.ir http://persian-gt= a.ir http://mamanha3.ir http://refquiz.ir http://www.agcom.ir http://tandistak.ir http://kimiafam.ir http://farzandan3.ir http://daneshgu.ir http://aeua.ir http://www.m3dia.ir http://www.game123.ir http://www.nirvanaweb.ir http://www.kfmisagh.ir http://zabax.ir http://www.sobhangen.ir http://www.yyar.ir http://www.hsqom.ir http://www.parandstudio.ir http://www.cpu64.ir http://ps-music.ir http://www.tfarahan.ir http://www.zarinfair.ir http://www.mvcportal.ir http://healthcarecenter.ir http://careonline.ir http://turkinc.ir http://healthcareonline.ir http://khoyweb.ir http://raycomarket.ir http://raycosoftware.ir http://raycostore.ir http://raycotec.ir http://rayct.ir http://raysoft.ir http://raytc.ir http://webkhoy.ir http://melkaras.ir http://market-aras.ir http://i-aras.ir http://e-aras.ir http://earas.ir http://clinictabriz.ir http://aras-market.ir http://tejartaras.ir http://tebtabriz.ir http://tebsonatitabriz.ir http://tabrizdoctor.ir http://shop-aras.ir http://khoytokhmeh.ir http://hypertabriz.ir http://hostturk.ir http://goldtabriz.ir http://faratabriz.ir http://calltabriz.ir http://bazartokhmeh.ir http://turkhost.ir http://tokhmeshop.ir http://tokhmehshop.ir http://tokhmehcenter.ir http://tokhmecenter.ir http://tabrizyab.ir http://tabriztv.ir http://tabrizno.ir http://tabrizgame= .ir http://tabrizengine.ir http://tabrizcall.ir http://santtabriz.ir http://hotel-aras.ir http://onlinik.ir/ http://cloudonlinenic.ir/ http://madraseonline.ir/ http://madrasehonline.ir/ http://madaresonline.ir/ http://khoybazari.ir/ http://bazarionline.ir/ http://urmiabazari.ir/ http://salmasbazari.ir/ http://onlinnic.ir/ http://nomrehonline.ir/ http://nomaratonline.ir/ http://webmastertabrizurmiabazari.ir/ http://webmastertabriz.ir/ http://tabrizcad.ir/ http://sanattabriz.ir/ http://khoyhyper.ir/ http://khoyhony.ir/ http://khoyfarsh.ir/ http://khoyasal.ir/ http://hyperaras.ir/ http://honykhoy.ir/ http://farshkhoy.ir/ http://datacentertabriz.ir/ http://asalkhoy.ir/ http://arasshopcenter.ir/ http://7namehkhoy.ir/ http://turkshopcenter.ir/ http://tabriztools.ir/ http://tabrizstudio.ir/ http://tabrizscan.ir/ http://tabrizpizza.ir/ http://tabrizpark.ir/ http://tabrizmodir.ir/ http://tabrizmarktabrizmaster.ir/ http://tabrizmark.ir/ http://tabrizkadeh.= ir/ http://tabrizkade.ir/ http://tabrizhelp.ir/ http://tabrizgram.ir/ http://tabrizwork.ir/ http://tabrizmoble.ir/ http://tabrizhouse.ir/ http://tabrizcar.ir/ ------------------------------------ http://www.amar36= 5.ir http://09123498298.ir http://pamar.ir http://arazproje.ir http://signsaras.= ir http://signsaraz.ir http://signsfarahan.ir http://suleforosh.ir http://tabloosazi.ir http://zaminforosh.ir http://9125879258.ir http://9375883058.ir http://www.lbfarahan.ir http://www.ghfarahan.ir http://pix-land.blogfa.com http://projeamar.blog.ir http://projectstatis.blogfa.com http://projectstatis.rozblog.com http://projectstatistics.blog.ir http://projectstatistics.niloblog.com http://projectstats.avablog.ir http://projectstats.blog.ir http://projectstats.javanblog.ir http://projectstats.samenblog.com http://projectstats1.blogfa.com http://projectstatus.avablog.ir http://prozhe-amar.blogfa.com http://spam.blogfa.com http://stats09375883058.blogfa.com http://projeamari.blogfa.com http://projectanalysis.mihanblog.com http://statisticsproject.blogfa.com http://tahghighstan.blogfa.com http://www.amar101.blogfa.com http://saheldarya.7blog.ir http://pezeshkyar.arisfa.com http://Shabebarfi.armanblog.ir http://varzeshsara.avablog.ir http://Azadweb.azadblog.com http://Batoo.b88.ir http://dostan.bestblog.ir http://Toristi.bigsite.ir http://Tarane18.blog.ir http://Hambaazi.blogfa.com http://estekhdami.blogia.ir http://behtarinha.blogiran.net http://Musicnaab.blognovin.com http://baharnarenj.blogparsi.com http://Molodi.blogpart.ir http://Bestgirl.blograz.ir http://razesalamati1.blogsky.com http://Alghameh.blogtarin.com http://deklameh.blogtez.com http://taranoom18.blogveb.com http://dabirestani.deyblog.ir http://amozeshyar.eklablog.com http://donyayenet.epage.ir http://mamnoo.famblog.ir http://tabasom.farazblog.com http://daneshjoei.fardblog.com http://Nemonehsoal.farsiblog.com http://tabestoon.geblog.ir http://maghaleh.iran.sc http://funkadeh.iranblag.com http://fotoax.jahanblog.net http://shabgard.jasaz.com http://jazadkadeh.javanblog.ir http://khabarjadid.limooblog.com http://tanzkadeh.loxblog.com http://tahghigh18.mihanblog.com http://niazmandiha.mojblog.ir http://niazsara.monoblog.ir http://divari.nedablog.ir http://Darham.niazblog.ir http://dabestan.niloblog.com http://salamati.novinblog.net http://dehati.parsablog.com http://Baharnews.parsiblog.com http://tanhatarin.parsunit.com http://doghalb.persianblog.ir http://yadgari.ratablog.com http://baharestan.roomfa.com http://modkadeh.royablog.ir http://sargarmi724.rozblog.com http://webmasteri.samenblog.com http://mosaferati.shblog.ir http://gardeshgari.sitearia.ir http://Khandani.smu.ir http://ahangjadid.takblog.net http://filmjadid.tarlog.com http://mashinbaz.tibablog.ir http://lebasmajlesi.tinablog.ir http://backlink724.titrblog.ir http://bazigaran.toonblog.ir http://pishwaz.twoblog.ir http://madahi.wblog.xyz http://hotelyar.yektablog= .net http://digibacklink.ir/ http://dgbacklink.ir/ http://royalbacklink.ir/ http://highbacklink.ir/ http://1zekr.com/forums/member.php?u=3D5634 http://1zekr.yaalee.com/forums/member.php?u=3D5634 http://220volt.ir/forum/threads/637 http://2l-f.com/vb/member.php?u=3D935 http://2ndvarp.net/member.php?716-tablooaraz http://3dpe.ir/forum/threads/%D8%B7%D8%B1%D8%A7%D8%AD%DB%8C-%D8%B3%D8%A7%DB= %8C%D8%AA-%D8%B5%D8%B1%D8%A7%D9%81%DB%8C.75/ http://3ss3.com/member.php?u=3D21209 http://4glte.vn/member.php?u=3D13460 http://4r.mu-viet.vn/showthread.php?1323-H%C6%B0%E1%BB%9Bng-d%E1%BA%ABn-ch%= E1%BB%A9c-n%C4%83ng-m%E1%BB%9Bi-Reset-B%C3%B9-v%C3%A0-Reset-H%E1%BB%99&p=3D= 14999#post14999 http://4um.overclocking.cz/member.php?u=3D34728 http://aces.ir/blogs/tablooaraz/699-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%A9-%= D9%84%DB%8C%D9%86%DA%A9-%D8%AD%D8%B1%D9%81%D9%87-%D8%A7%DB%8C.html http://aces.ir/members/tablooaraz.html http://adminzone.ir/topic/161-%D8%B3%DB%8C%D8%B3%D8%AA%D9%85-%D9%85%D8%AF%D= B%8C%D8%B1%DB%8C%D8%AA-%D9%85%D8%AD%D8%AA%D9%88%D8%A7-cms-%DA%86%DB%8C%D8%B= 3%D8%AA/ http://afroforum.ru/member.php?u=3D1790 http://alhoush.com/forums/member.php?121997-tablooaraz http://alivahedidiz.com/forum/showthread.php?tid=3D3108 http://alphacs.ro/member.php?34156-tablooaraz http://alqsid.com/vb/member.php?u=3D397 http://animeleague.net/forum/member.php?104953-tablooaraz http://anjoman.parsfootball.com/member.php?u=3D34586 http://anjoman.roochi.ir/showthread.php?tid=3D2337 http://arabiforall.com/Forum/Post/66/ http://arshadeomran.ir/forum/thread1030-97.html http://artificial.ir/intelligence/member.php?u=3D134589 http://asemaniha.net/forum/member.php/549-tablooaraz http://asl8.ir/social/thread-101.html http://asrenevisandegi.com/index.php?/topic/595-%D8%B7%D8%B1%D8%A7%D8%AD%DB= %8C-%D8%B3%D8%A7%DB%8C%D8%AA-%D8%B4%DB%8C%D8%B1%D8%A7%D8%B2/ http://aswaqbayte.com/member.php?u=3D2797 http://ayask.com/forum/viewtopic.php?f=3D21&t=3D993 http://a-zgsm.com/mobile/member.php?4695-tablooaraz http://azpnu.ir/User-tablooaraz--28367 http://baharsite.com/forum/viewtopic.php?f=3D150&t=3D735&p=3D60813 http://baharsite.com/forum/viewtopic.php?f=3D150&t=3D735&p=3D60813#p60813 http://baras.ir/forum/thread-637-post-32099.html http://bda.lovebongda.com/member.php?u=3D15310 http://bdc-forum.it/member.php?u=3D109212 http://behdari.com/Thread-%D8%AE%D9%88%D8%A7%D8%B5-%D8%AE%D8%A7%D8%B1-%D9%8= 5%D8%B1%D9%8A%D9%85-%D8%AF%D9%88%D8%B3%D8%AA-%D8%A8%D8%A7-%D9%88%D9%81%D8%A= 7%D9%8A-%D9%83%D8%A8%D8%AF http://beltronicsradarforum.com/forums/member.php?u=3D2591 http://beporsbedoon.com/showthread.php?7717-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8= %DA%A9-%D9%84%DB%8C%D9%86%DA%A9&p=3D13967#post13967 http://beyamooz.com/forum/members/1026-tablooaraz http://bia2skin.ir/forum/member.php?u=3D4963 http://biketrials.ru/live/member.php?u=3D49421 http://blogandforum.fairfaxcryobank.com/forum/member.php?51418-tablooaraz http://blogandforum.fairfaxcryobank.com/forum/member.php?51418-tablooaraz&t= ab=3Dvisitor_messaging&page=3D2#visitor_messaging http://bodogame.com/forum/members/1422.html http://bodyclass.ir/Thread-%D8%AA%D9%81%D8%A7%D9%88%D8%AA-%D8%AA%D8%A8%D9%8= 4%DB%8C%D8%BA%D8%A7%D8%AA-%D8%B7%D8%B1%D8%A7%D8%AD%DB%8C-%D8%B3%D8%A7%DB%8C= %D8%AA http://bpums.ac.ir/forums/yaf_postsm2204_khsht-mw-dr-khnm-h.aspx http://bulletin.thewho.com/member.php?6535-tablooaraz http://bulletin.thewho.com/member.php?6535-tablooaraz&tab=3Dvisitor_messagi= ng&page=3D2#visitor_messaging http://bulug.ir/forum/User-roozfreight http://cafemohinh.vn/member.php?u=3D= 2065 http://camcom.cc/vb/member.php?u=3D1109 http://carina-club.ru/member.php?u= =3D30625 http://cfps.pl/forum/member.php?u=3D13385350 http://cgworkshop.org/forum/member.php?41864-tablooaraz http://clancommunity.ru/member.php?tab=3Dvisitor_messaging&u=3D13745&page= =3D2#visitor_messaging http://clancommunity.ru/member.php?u=3D13745 http://clan-dfm.org/forums/member.php?u=3D2318 http://cmmnty.mthmtcs.ir/index.php?threads/%D8%B3%D9%88%D8%A7%D9%84%D8%A7%D= 8%AA-%D8%A2%D8%B2%D9%85%D9%88%D9%86-%DA%A9%D8%A7%D8%B1%D8%B4%D9%86%D8%A7%D8= %B3%DB%8C-%D8%A7%D8%B1%D8%B4%D8%AF-94.78/ http://code-igniter.ir/showthread.php?tid=3D41 http://codfans.ir/member.php?19398-tablooaraz http://community.arab.sh/member.php?u=3D5964 http://community.kingofflips.com/member.php?120625-tablooaraz http://community.mybbiran.com/thread-20268-post-145255.html http://csko.cz/forum/member.php?133346-tablooaraz http://cuddetalk.com/member.php?6901-tablooaraz http://d4mb.ir/Forum/Post/160/Page/1 http://dalghak.net/forum/Thread-%D8%A7%D8%AC%D8%A7%D8%B1%D9%87-%D8%AE%D9%88= %D8%AF%D8%B1%D9%88-%D8%A2%D9%85%D8%B1%DB%8C%DA%A9%D8%A7%DB%8C%DB%8C http://dangtocvietnam.com/diendan/member.php?u=3D882 http://dark-age.com/forum/member.php?9618-tablooaraz http://darknet.su/member.php?37058-tablooaraz http://dashtiha.com/Thread-%D8%AC%D8%A7%D8%B0%D8%A8%D9%87-%D9%87%D8%A7%DB%8= C-%DA%AF%D8%B1%D8%AF%D8%B4%DA%AF%D8%B1%DB%8C-%DA%A9%D8%A7%D9%86%D8%A7%D8%AF= %D8%A7 http://dd.muvn2.net/member.php?9875-tablooaraz http://diendan.cnttcd8.edu.vn/member.php?18260-tablooaraz http://diendan.dienchan.com/member.php?22741-tablooaraz http://diendan.huyenthoaimu.net/member.php?1782-tablooaraz http://diendan.kiemthebavuong.com/member.php?2574-tablooaraz http://diendan.muhaokiet.com/member.php?11798-tablooaraz http://diendan.muhuyenthoai.com/member.php?12629-tablooaraz http://diendan.mu-korea.net/member.php?7539-tablooaraz http://diendan.muphilong.com/member.php?9306-tablooaraz http://diendan.muphuchung.com/member.php?9398-tablooaraz http://diendan.musaigon.vn/member.php?114384-tablooaraz http://diendan.muthaiduong.vn/member.php?5411-tablooaraz http://diendan.muthienviet.com/member.php?11804-tablooaraz http://diendan.mutuyetdinh.com/member.php?8693-tablooaraz http://diendan.muviet.com/member.php?43673-tablooaraz http://diendan.muvietss6.vn/member.php?3707-tablooaraz http://dmctalk.org/member.php?88260-tablooaraz http://dtphorum.com/pr4/member.php?59406-tablooaraz http://egyhunt.net/member.php?u=3D19709 http://elderscrolls-online.tiscali.cz/forums/member.php?2631-tablooaraz http://elmas.ir/forum/member.php?3301-tablooaraz http://elmas.ir/forum/showthread.php?322-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA= %A9-%D9%84%DB%8C%D9%86%DA%A9 http://en.blitzkrieg.com/forum/member.php?u=3D71284 http://enad199.com/vb/member.php?u=3D9076 http://eye4you.ir/forum/member.php?3422-tablooaraz http://f.havajanah.ir/showthread.php?tid=3D248 http://fanzine.ir/memberlist.php?mode=3Dviewprofile&u=3D2391 http://farspatogh.ir/entries/21727/ http://fcbavaria.ir/forum/members/tablooaraz/ http://fclan.ru/4room/member.php?u=3D5224 http://ffnet.net/forum/member.php?u=3D9474 http://filforex.com/vb/member.php?u=3D3719 http://forsakenforum.de/member.php?41270-tablooaraz http://forum.abfa-khj.ir/showthread.php?2-hi-to-all-peolle http://forum.abfa-khj.ir/showthread.php?63-%D8%AC%D8%B4%D9%86-%D9%85%D8%A7%= D9%87-%DA%A9%D8%A7%D9%85%D9%84 http://forum.acgih.ir/members/tablooaraz.html http://forum.activerayan.ir/member.php?45-tablooaraz http://forum.allgsmunlock.com/member.php?270159-tablooaraz http://forum.androidsos.ir/showthread.php?tid=3D130706 http://forum.aratta.ir/member.php?228-tablooaraz http://forum.aratta.ir/member.php?228-tablooaraz http://forum.arduino.ir/4/15/1777.html http://forum.arvandplak.ir/member.php?u=3D2233 http://forum.arvandplak.ir/showthread.php?t=3D630 http://forum.atomfish.ru/member.php?u=3D51439 http://forum.bafghsalam.ir/showthread.php?tid=3D2777 http://forum.banianbehboodi.ir/showthread.php?tid=3D7417 http://forum.barato.ir/Thread-%D8%B1%D8%A7-%D9%87%D8%A7%DB%8C-%D8%A7%D9%81%= D8%B2%D8%A7%DB%8C%D8%B4-%D8%A8%D8%A7%D8%B2%D8%AF%DB%8C%D8%AF-%D8%B3%D8%A7%D= B%8C%D8%AA http://forum.bargtools.ir/showthread.php?tid=3D5716 http://forum.barname.org/showthread.php?t=3D1840 http://forum.bazaarbargh.ir/entry.php?10-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA= %A9-%D9%84%DB%8C%D9%86%DA%A9-%D8%AD%D8%B1%D9%81%D9%87-%D8%A7%DB%8C-%D9%88-%= D8%A7%D8%B1%D8%B2%D8%A7%D9%86 http://forum.boi-velos.com/member.php?u=3D3354 http://forum.cafedexign.com/members/47854.html http://forum.cakephp.ir/thread-1578-post-8812.html http://forum.cinemacenter.ir/member33875.html http://forum.codecorona.com/showthread.php?tid=3D7152 http://forum.cytco.net/member.php?u=3D118 http://forum.dejkoob.ir/member.php?u=3D3211 http://forum.diangucmu.com/member.php?4518-tablooaraz http://forum.dotabaz.com/member6948.html http://forum.download.ir/member/900-tablooaraz http://forum.download.ir/member/900-tablooaraz http://forum.drasgharian.ir/entry.php?12-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA= %A9-%D9%84%DB%8C%D9%86%DA%A9-5 http://forum.dubbedclub.com/member.php?u=3D1775 http://forum.dubbedclub.com/showthread.php?t=3D1893 http://forum.duragarages.com/member.php?150468-tablooaraz http://forum.eastmans.com/member.php/13669-tablooaraz http://forum.edcoan.ir/entry.php?21-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%A9-%= D9%84%DB%8C%D9%86%DA%A9-5 http://forum.entitygaming.com/member.php?10852-tablooaraz http://forum.eslamteb.com/Post/1373/ http://forum.exceliran.com/member.php/48002-tablooaraz http://forum.faosclass.com/Thread-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%A9-%D9= %84%DB%8C%D9%86%DA%A9 http://forum.farsdownload.net/member.php?84-tablooaraz http://forum.firooz.com/viewtopic.php?f=3D75&t=3D6999 http://forum.freedownload.ir/thread-400598.html http://forum.gamgos.net/member/86804-tablooaraz http://forum.gharchsanat.ir/showthread.php?tid=3D126 http://forum.giamsat.org/member.php?27386-tablooaraz http://forum.goaudio.su/member.php?77201-tablooaraz http://forum.graphiran.com/members/tablooaraz.html http://forum.gsm-developers.com/profile/23630-tablooaraz/ http://forum.gtr-masters.hu/member.php?7690-tablooaraz http://forum.hadafdownload.com/member.php?2162-tablooaraz http://forum.hadafdownload.com/showthread.php?2648-%D8%B4%D8%B1%D8%A7%DB%8C= %D8%B7-%D9%88-%D9%85%D9%88%D9%84%D9%81%D9%87-%D9%87%D8%A7%DB%8C-%D9%87%D8%A= 7%D8%B3%D8%AA-%D8%AE%D9%88%D8%A8-%D8%AF%D8%B1-%D8%B7%D8%B1%D8%A7%D8%AD%DB%8= C-%D8%B3%D8%A7%DB%8C%D8%AA http://forum.himym.cz/viewtopic.php?f=3D12&t=3D32067 http://forum.hl.ua/member.php?u=3D57448 http://forum.hnkvz.hr/member.php?u= =3D2362 http://forum.hogwartz.ir/member.php?2330-tablooaraz http://forum.hpaba.com/member.php?481-tablooaraz http://forum.illpumpyouup.com/member.php?u=3D12690 http://forum.imanvfx.com/thread3285.html http://forum.irani-dl.com/thread453.html http://forum.iran-mavad.com/member.php?u=3D9361 http://forum.ir-click.com/index.php?topic=3D788.0 http://forum.jdmstyletuning.com/member.php?42962-tablooaraz http://forum.kafesahel.com/member/21-tablooaraz http://forum.kalimdor.ir/member.php?628-tablooaraz http://forum.kalout.ir/blogs/tablooaraz/%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%= A9-%D9%84%DB%8C%D9%86%DA%A9-20/ http://forum.kara365.ir/entry.php?2651-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%A= 9-%D9%84%DB%8C%D9%86%DA%A9-%D8%A7%D8%B1%D8%B2%D8%A7%D9%86-%D9%88-%D8%AD%D8%= B1%D9%81%D9%87-%D8%A7%DB%8C http://forum.kavoshteam.com/members/tablooaraz/25-1.html http://forum.kgb-hosting.ro/member.php?408-tablooaraz http://forum.korepix.ir/Post/969 http://forum.leagueoflegunds.ir/member.php?995-tablooaraz http://forum.legionisci.com/member.php?u=3D152091 http://forum.legion-noire.com/member.php/734-tablooaraz http://forum.mehad.ir/showthread.php?t=3D106 http://forum.mimsoft.ir/member.php?714-tablooaraz http://forum.misawa.de/member.php/2305-tablooaraz http://forum.mkcenter.ir/member.php?u=3D21693 http://forum.mobile4persian.net/member113358.html http://forum.monji12.com/threads/25198/ http://forum.motarjemonline.com/member/9829-tablooaraz http://forum.motarjemonline.com/member/9829-tablooaraz http://forum.mspsoft.com/members/tablooaraz.html http://forum.mu4viet.com/member.php?u=3D5070 http://forum.muanhhung.com/member.php?8003-tablooaraz http://forum.muanviet.com/member.php?5578-tablooaraz http://forum.mubachkim.com/member.php?8407-tablooaraz http://forum.mubachu.com/member.php?7995-tablooaraz http://forum.mubacviet.com/member.php?6215-tablooaraz http://forum.mubalong.net/member.php?6617-tablooaraz http://forum.mubaochau.com/member.php?7614-tablooaraz http://forum.mubaoviet.net/member.php?u=3D4138 http://forum.mubavuong.net/member.php?u=3D3854 http://forum.mubomtan.com/member.php?u=3D3345 http://forum.mubz.bg/member.php?u=3D2738 http://forum.muchaulong.com/member.php?10995-tablooaraz http://forum.muchuate.com/member.php?9118-tablooaraz http://forum.mudainam.net/member.php?5436-tablooaraz http://forum.mudaiviet.com/member.php?7960-tablooaraz http://forum.mudangcap.vn/member.php?3107-tablooaraz http://forum.mudenhat.com/member.php?8343-tablooaraz http://forum.mudinhcao.net/member.php?u=3D3231 http://forum.mudinhmenh.com/member.php?5550-tablooaraz http://forum.mufptvn.com/member.php?u=3D3351 http://forum.mugiangho.com/member.php?8438-tablooaraz http://forum.muhacvuong.com/member.php?5447-tablooaraz http://forum.muhaiphong.vn/member.php?8577-tablooaraz http://forum.muhanoiss6.com/member.php?u=3D3371 http://forum.muhiepkhach.com/member.php?8125-tablooaraz http://forum.muhoangdao.com/member.php?6354-tablooaraz http://forum.muhoanglong.net/member.php?u=3D4061 http://forum.muhoanhao.com/member.php?9089-tablooaraz http://forum.muhoanmy.com/member.php?9326-tablooaraz http://forum.muhoisinh.com/member.php?7983-tablooaraz http://forum.muhoisinh.net/member.php?u=3D3426 http://forum.muhoisinh.net/member.php?u=3D3426 http://forum.muhoiuc.net/member.php?u=3D3239 http://forum.muhuyenthoaiss6.com/member.php?u=3D3814 http://forum.muhuyetlong.net/member.php?u=3D3642 http://forum.mukhaihoan.com/member.php?u=3D3896 http://forum.mukhatvong.net/member.php?u=3D4203 http://forum.mukinhdo.com/member.php?6255-tablooaraz http://forum.mulacviet.com/member.php?11620-tablooaraz http://forum.mulienviet.com/member.php?8555-tablooaraz http://forum.mulucdia.com/member.php?u=3D4689 http://forum.mumienbac.com/member.php?u=3D3918 http://forum.munamquoc.com/member.php?10188-tablooaraz http://forum.munuthan.com/member.php?10731-tablooaraz http://forum.munuvuong.com/member.php?10646-tablooaraz http://forum.muphucsinh.com/member.php?11601-tablooaraz http://forum.muquocte.com/member.php?u=3D3458 http://forum.muquyenluc.com/member.php?8428-tablooaraz http://forum.muquyvuong.com/member.php?8243-tablooaraz http://forum.musaigonss6.com/member.php?u=3D3618 http://forum.musieuviet.com/member.php?8566-tablooaraz http://forum.muss2.vn/member.php?45618-tablooaraz http://forum.muss7.com/member.php?u=3D4415 http://forum.muss8.net/member.php?u=3D3836 http://forum.mustatir.com/showthread.php?tid=3D43 http://forum.musv.net/member.php?8073-tablooaraz http://forum.mutaisinh.net/member.php?u=3D3333 http://forum.muthanhcong.com/member.php?8307-tablooaraz http://forum.muthanhdia.com/member.php?8263-tablooaraz http://forum.muthanvu.com/member.php?7803-tablooaraz http://forum.mutheky.com/member.php?8030-tablooaraz http://forum.muthienkim.vn/member.php?3839-tablooaraz http://forum.muthienmenh.net/member.php?u=3D3519 http://forum.muthienmenh.net/member.php?u=3D3519 http://forum.muthienquoc.com/member.php?5448-tablooaraz http://forum.muthientu.net/member.php?1488-tablooaraz http://forum.muthientu.vn/member.php?29992-tablooaraz http://forum.muthoidai.com/member.php?8377-tablooaraz http://forum.muthoigian.net/member.php?99-tablooaraz http://forum.mutienphong.com/member.php?5428-tablooaraz http://forum.mutoanquoc.com/member.php?u=3D3874 http://forum.mutranhba.net/member.php?u=3D3950 http://forum.mutranhba.net/member.php?u=3D3950 http://forum.mutranhba.net/member.php?u=3D3950 http://forum.mutruyenkyss6.com/member.php?u=3D3787 http://forum.muviet.net.vn/member.php?16315-tablooaraz http://forum.muvietnamss6.com/member.php?u=3D4280 http://forum.muvietss7.com/member.php?u=3D4415 http://forum.muvietvuong.com/member.php?8164-tablooaraz http://forum.muvinhcuu.com/member.php?6382-tablooaraz http://forum.muxungba.com/member.php?u=3D3369 http://forum.naghsh-negar.ir/showthread.php?7141-%D9%87%D8%AA%D9%84-%D9%BE%= D8%A7%D8%B1%D9%85%DB%8C%D8%B3-%DA%A9%DB%8C%D8%B4 http://forum.najrobotics.com/showthread.php?tid=3D1496 http://forum.nis.singidunum.ac.rs/member.php?708-tablooaraz http://forum.oghyanos.ir/thread1032.html http://forum.p30download.com/member.php?332122-tablooaraz http://forum.p30talk.com/index.php?threads/%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%= DA%A9-%D9%84%DB%8C%D9%86%DA%A9-5.627/ http://forum.p30world.com/member.php?u=3D1335477 http://forum.p313.ir/member.php?action=3Dprofile&uid=3D1574 http://forum.parsbtc.com/member.php?927-tablooaraz http://forum.parsiking.com/member.php?u=3D59156 http://forum.persianrc.com/showthread.php?p=3D122730 http://forum.persianseven.ir/t266943.html http://forum.pesholland.nl/member.php?1444-tablooaraz http://forum.pioneer-life.ir/member7451.html http://forum.pnu4u.com/showthread.php?tid=3D3625 http://forum.poemse.com/showthread.php?pid=3D2627 http://forum.pop-music.ir/threads/%D8%AF%D8%B4%D8%AA-%D8%AE%D8%B4%DA%A9%DB%= 8C%D8%AF%D9%88-%D8%B2%D9%85%DB%8C%D9%86-%D8%B3%D9%88%D8%AE%D8%AA-%D8%B4%D8%= B9%D8%B1.11678/ http://forum.pstrophy.ir/member.php?684-tablooaraz http://forum.qeshmvoltage.com/showthread.php?tid=3D887 http://forum.razhanmobile.com/index.php?threads/%D8%A8%D9%87%DB%8C%D9%86%D9= %87-%D8%B3%D8%A7%D8%B2%DB%8C-%D8%B3%D8%A7%DB%8C%D8%AA-%D8%B4%D8%B1%DA%A9%D8= %AA-%D9%87%D8%A7%DB%8C-%D8%B5%D9%86%D8%B9%D8%AA%DB%8C.24820/ http://forum.razhanmobile.com/index.php?threads/=D8=A8=D9=87=DB=8C=D9=86=D9= =87-=D8=B3=D8=A7=D8=B2=DB=8C-=D8=B3=D8=A7=DB=8C=D8=AA-=D8=B4=D8=B1=DA=A9=D8= =AA-=D9=87=D8=A7=DB=8C-=D8=B5=D9=86=D8=B9=D8=AA=DB=8C.24820 http://forum.roq.ir/u3700 http://forum.saabturboclub.com/member.php/103983-tablooaraz http://forum.safarnaame.ir/entry.php?1-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%A= 9-%D9%84%DB%8C%D9%86%DA%A9 http://forum.scriptcamp.ir/Forum/Post/633 http://forum.semeng.ir/showthread.php?tid=3D27166 http://forum.shopkeeper.ir/showthread.php?t=3D106353 http://forum.skhorasaniec.ir/member.php?28090-tablooaraz http://forum.skhorasaniec.ir/member.php?28090-tablooaraz http://forum.smartbeen.com/showthread.php?tid=3D709 http://forum.springdl.ir/thread-31744-post-71071.html http://forum.talarearoos.com/thread-%D8%B7%D8%B1%D8%A7%D8%AD%DB%8C-%D8%A8%D= 8%B3%DB%8C%D8%A7%D8%B1-%D8%B2%DB%8C%D8%A8%D8%A7%DB%8C-%D9%81%D8%B6%D8%A7%DB= %8C-%D8%AD%DB%8C%D8%A7%D8%B7--1348 http://forum.tambura.com.hr/member.php?u=3D8427 http://forum.tebpress.com/threads/19-%D8%A7%D8%B5%D9%84-%D9%85%D9%87%D9%85-= %D8%B2%D9%86%D8%AF%DA%AF%DB%8C.719/ http://forum.techno-electro.com/%D9%85%D9%88%D8%B6%D9%88%D8%B9-%D8%A8%D9%87= %D8%AA%D8%B1%DB%8C%D9%86-%D8%B1%D8%A7%D9%87-%D9%87%D8%A7%DB%8C-%D8%A8%D9%87= %D8%A8%D9%88%D8%AF-%D8%B1%D8%AA%D8%A8%D9%87-%D8%B3%D8%A6%D9%88-%D8%AF%D8%B1= -%DA%AF%D9%88%DA%AF%D9%84 http://forum.tehran-gaming.com/members/tablooaraz.html http://forum.tenisnews.com.br/member.php?u=3D31939 http://forum.tenpointcrossbows.com/member.php?16615-tablooaraz http://forum.tiguans.ru/member.php?u=3D49528 http://forum.troll98.ir/entry.php?10-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%A9-= %D9%84%DB%8C%D9%86%DA%A9-5 http://forum.ukuleleunderground.com/member.php?146273-tablooaraz http://forum.vegankind.ir/members/8524.html http://forum.vkontakte.dj/member.php?10018-tablooaraz http://forum.vmusic.ir/member.php?u=3D6597 http://forum.volleyworld.ir/Thread-%D8%A7%D9%87%D9%85%DB%8C%D8%AA-%D9%87%D8= %AF%D8%A7%DB%8C%D8%A7%DB%8C-%D8%AA%D8%A8%D9%84%DB%8C%D8%BA%D8%A7%D8%AA%DB%8= C-%D8%AF%D8%B1-%DA%A9%D8%B3%D8%A8-%D9%88 http://forum.walterfootball.com/member/209936-tablooaraz http://forum.wepage.ir/thread-28152-post-92811.html http://forum.wikigamers.ir/entry.php?9-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%A= 9-%D9%84%DB%8C%D9%86%DA%A9 http://forum.windowsclub.net/member.php/25481-tablooaraz http://forumindo.com/member.php?66634-tablooaraz http://forums.arlongpark.net/member.php?u=3D81621 http://forums.bizna.ir/showthread.php?tid=3D4965 http://forums.bizna.ir/showthread.php?tid=3D632 http://forums.boursy.com/member.php?u=3D14711 http://forums.boursy.com/member.php?u=3D14711 http://forums.computershopper.com/member.php?u=3D45891 http://forums.directhosting.ca/member.php?23665-tablooaraz http://forums.episodeinteractive.com/member/82274-tablooaraz http://forums.forestbear.net/member.php?4633-tablooaraz http://forums.gingerscraps.net/member.php?21296-tablooaraz http://forums.mtbr.com/member.php?u=3D830836 http://forums.nodiatis.com/forums/member.php?u=3D59407 http://forums.nodoubt.com/member.php?35960-tablooaraz http://forums.p30day.com/member/60979-tablooaraz http://forums.p30day.com/member/60979-tablooaraz http://forums.parsjoom.ir/thread-20768-post-100351.html http://forums.pcpowerplay.com.au/member.php/133136-tablooaraz http://forums.persianinoz.com/members/tablooaraz.html http://forums.prosportsdaily.com/member.php?396679-tablooaraz http://forums.show-time.top/member.php?u=3D29230 http://forums.tehranpatogh1.ir/member.php?action=3Dprofile&uid=3D167 http://forums.tppc.info/member.php?u=3D30780 http://forums.warchest.com/member.php/111729-tablooaraz http://forumunik.com/member.php?4532-tablooaraz http://forzainterforums.com/member.php?9527-tablooaraz http://freetalk.takbb.ir/showthread.php?tid=3D3278 http://game.kuban.ru/forum/member.php?u=3D231 http://gentra-club.com.ua/member.php/527-tablooaraz http://gfrc.vn/forum/member.php?u=3D28804 http://ghanunejazb.ir/forum/showthread.php?tid=3D3368 http://glxforum.ir/blogs/tablooaraz/11-backlink.html http://glxforum.ir/entry.php?b=3D4 http://glxforum.ir/entry.php?b=3D5 http://glxforum.ir/entry.php?b=3D6 http://glxforum.ir/entry.php?b=3D6 http://goldendict.org/forum/viewtopic.php?f=3D4&t=3D23616 http://goldendict.org/forum/viewtopic.php?f=3D4&t=3D23616&start=3D10 http://golestane.net/showthread.php?t=3D88897 http://gomatlab.com/forum/User-tablooaraz http://groobyforum.com/member.php?26630-tablooaraz http://haftkhat.ir/forum/members/tablooaraz/ http://hangcu.vn/member.php?u=3D104400 http://hbportal.ir/showthread.php?ti= d=3D68 http://help.shoguto.com/member.php?2349-tablooaraz http://highboostforum.com/forum/member.php/19992-tablooaraz http://hotshotsgolf.com/blog.php?3436-tablooaraz http://iibimsolutions.ir/forum/member.php?u=3D295 http://ilammoallem.ir/forum/showthread.php?tid=3D71 http://ipcamsoft.com/support/member.php?97733-tablooaraz http://iranbookforum.com/thread-3249-post-12554.html http://iranclubs.org/forums/member.php?u=3D40182 http://iranclubs.org/forums/member.php?u=3D40182 http://irandelphi.ir/member.php?u=3D2151 http://iranwebadmin.com/threads/6-=D8=A7=D8=B4=D8=AA=D8=A8=D8=A7=D9=87-=D9= =85=D9=87=D9=84=DA=A9-=DA=A9=D9=87-=D9=87=D8=B1-=D8=B7=D8=B1=D8=A7=D8=AD-= =D8=B3=D8=A7=DB=8C=D8=AA=DB=8C-=D9=85=D9=85=DA=A9=D9=86=D9=87-=D9=85=D8=B1= =D8=AA=DA=A9=D8=A8-=D8=A8=D8=B4=D9=87.598/ http://irserv.ir/showthread.php?tid=3D3992 http://justih.org/Binder-Bench/member.php?u=3D16891 http://ketnoi123.com/member.php?u=3D41250 http://khoisukinhdoanh.net/member.php?108912-tablooaraz http://kpnu-csc.ir/forum/members/tablooaraz/72-a.html http://krkan-muzika.org/forum/member.php?25547-tablooaraz http://ladytalk.ir/member.php?action=3Dprofile&uid=3D716 http://lalaei.ir/Forum/User-tablooaraz http://landcruisergroup.com/vb/member.php?u=3D1161 http://limbine.com/thread-5236.html http://limbine.com/thread-5797-post-15748.html http://mafiaia.ir/entry.php?693-%D8%AE%D8%B1%DB%8C%D8%AF-%D8%A8%DA%A9-%D9%8= 4%DB%8C%D9%86%DA%A9 http://mamisite.com/forum/t1017-3.html http://medsmart.ir/forum/member.php?u=3D375 http://moshaverfa.com/forum/threads/17096/ http://my-bb.ir/Thread-%D9%81%D8%B1%D9%88%D8%B4-%D8%B3%D8%A7%DB%8C%D8%AA-%D= 8%AF%D9%84%D9%82%DA%A9 http://nabzezendegi.ir/Thread-%D8%A8%D9%87%D8%AA%D8%B1%DB%8C%D9%86-%D8%AF%D= B%8C%D8%AF%D9%86%DB%8C%E2%80%8C%D9%87%D8%A7%DB%8C-%D9%85%D8%A7%D9%84%D8%B2%= DB%8C--51240 http://naghdefarsi.com/forum.html?view=3Dtopic&catid=3D38&id=3D95076 http://net2web.ir/blogs/u784-e170/ http://noplus.ir/showthread.php?t=3D437 http://otageabi.ir/showthread.php?t=3D776 http://palarnet.ir/forum/User-tablooaraz http://palarnet.ir/forum/User-tablooaraz#comments/1 http://pamm-trade.com/forum/member/37-tablooaraz http://pasargadtabac.com/forum/showthread.php?tid=3D3223 http://pccamp.ir/forums/blogs/tablooaraz/a-109/ http://persianmotor.net/showthread.php?t=3D16651 http://persianpet.org/forum/member54343.html http://persian-roleplay.ir/member.php?120-tablooaraz http://phudeviet.org/forum/member.php?28149-tablooaraz.html http://pilotairline.ir/forums/entry.php?b=3D3 http://pishroforum.ir/member.php?action=3Dprofile&uid=3D5227 http://progforum.ir/threads/=D8=B3=D9=88=D8=A7=D9=84=D8=A7=D8=AA-=D8=B1=D8= =A7=DB=8C=D8=AC-=D8=B3=D8=A6=D9=88-=D8=B3=D8=A7=DB=8C=D8=AA.4978/ http://promakers.ir/member.php?action=3Dprofile&uid=3D20716 http://promakers.ir/member.php?action=3Dprofile&uid=3D20716#comments/1 http://psypooya.ir/showthread.php?tid=3D180 http://rahpouyan.ir/tablooaraz http://ravanbonyan.com/forum/member.php?u=3D9051 http://rayatolmahdi.com/entry.php?2-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%DA%A9-%= D9%84%DB%8C%D9%86%DA%A9-%D8%AD%D8%B1%D9%81%D9%87-%D8%A7%DB%8C http://rebelconquer.net/member.php/3318-tablooaraz http://sadeghsalamy.com/member.php?249-tablooaraz http://sahelanime.com/Forum/entry.php?b=3D13 http://salamjavaneh.ir/thread-3923-post-205214.html http://sescoshop.com/forums/showthread.php?tid=3D21 http://sfk.ibk.se/forum/member.php?50081-tablooaraz http://skyforum.ir/showthread.php?tid=3D6 http://spfga.org/forum/member.php?2705-tablooaraz http://springwoodslasher.com/forum/member.php/4625-tablooaraz http://tadbiraneh.com/member.php?3108-tablooaraz http://technotop.ir/forum/members/tablooaraz.htm http://tonel.org/thread15828.html http://topostudio.ir/Thread-=DA=A9=D8=A7=D8=BA=D8=B0-=D8=AF=DB=8C=D9=88=D8= =A7=D8=B1=DB=8C-=D8=B3=D9=87-=D8=A8=D8=B9=D8=AF=DB=8C http://trade.takbb.ir/showthread.php?tid=3D3322 http://trilianforum.ir/entry.php?3-%DD%D1%E6%D4-%C8%98-%E1%26%231740%3B%E4%= 98p?98-Pl-maloy83 http://trollfun.ir/forum/index.php?threads/%D8%A7%D8%AA%D9%88%D8%A8%D8%A7%D= 8%B1-%D8%B1%D9%88%D8%B2-%D9%85%D8%B9%D8%AA%D8%A8%D8%B1-%D8%AF%D8%B1-%D8%A7%= D9%85%D8%B1-%D8%AD%D9%85%D9%84-%D8%A8%D8%A7%D8%B1-%D8%AF%D8%B1-%D8%AA%D9%87= %D8%B1%D8%A7%D9%86.24778/ http://trollfun.ir/forum/index.php?threads/5%D9%86%DA%A9%D8%AA%D9%87-%D8%A8= %D8%B1%D8%A7%DB%8C-%D9%85%D8%AD%D8%A7%D9%81%D8%B8%D8%AA-%D8%A7%D8%B2-%D9%87= %D8%A7%D8%B3%D8%AA-%D8%B3%DB%8C-%D9%BE%D9%86%D9%84.24396/ http://vls.jums.ac.ir/forum/index.php?members/tablooaraz.374/ http://vnsharing.site/forum/member.php?u=3D307792 http://voipforum.ir/member108.html http://westeros.ir/forum/viewtopic.php?f=3D20&t=3D14&p=3D23881 http://wfa.org.ir/threads/%D9%85%D8%B9%D8%B1%D9%81%DB%8C-%D8%B3%D8%A7%DB%8C= %D8%AA-%D9%87%D8%A7%DB%8C-%D8%AF%D8%B1%D8%AC-%D8%A7%DA%AF%D9%87%DB%8C-%D8%B= 1%D8%A7%DB%8C%DA%AF%D8%A7%D9%86.130/ http://www.3dbuzz.com/forum/members/456289-tablooaraz http://www.aerocenter.ir/forum/blogs/tablooaraz/1230-backlink.html http://www.aerospacetalk.ir/vb/member.php?u=3D70948 http://www.airsoftgear.ca/member.php?1033-tablooaraz http://www.aramgroup.ir/Forum/members/tablooaraz/11-a.html http://www.ara-research.com/forum/showthread.php?tid=3D16 http://www.asapardazesh.ir/forum/entry.php?b=3D18 http://www.askquran.ir/member.php?u=3D88945 http://www.automationforum.vn/members/2332-tablooaraz http://www.ayehayeentezar.com/member10780.html http://www.beattips2.com/vb3/member.php?58844-tablooaraz http://www.bodoh.ir/showthread.php?t=3D7355 http://www.bourselar.ir/Forum/Post/807 http://www.buxnama.com/forum/thread-1611-post-5651.html http://www.cciran.ir/forum/showthread.php?1436-%D9%81%D8%B1%D9%88%D8%B4-%D8= %A8%DA%A9-%D9%84%DB%8C%D9%86%DA%A9 http://www.classichorsemanship.com/member.php?u=3D3949 http://www.d2messageboard.com/member.php?u=3D51266 http://www.d3scene.com/forum/members/tablooaraz.html http://www.daneshju.ir/forum/member.php?u=3D382476 http://www.derby.ir/member.php?u=3D67288 http://www.developercenter.ir/forum/member.php?u=3D188358 http://www.developercenter.ir/forum/showthread.php?t=3D36580 http://www.drumsmoking.com/members/2496.html http://www.emdadgar.com/forum/showthread.php?pid=3D30174 http://www.esksfans.com/forum/member.php?21870-tablooaraz http://www.expeditioncolorado.net/forum/member/93-tablooaraz http://www.fafan.ir/topic/29-%D8%B7%D8%B1%D8%A7%D8%AD%DB%8C-%D8%B3%D8%A7%DB= %8C%D8%AA-%DA%A9%D8%AA%D8%A7%D8%A8%D8%AE%D8%A7%D9%86%D9%87/ http://www.fifa-infinity.com/forums/member.php?u=3D13792 http://www.firemans.ir/Thread-%DA%A9%D8%AF%D8%A7%D9%85-%D8%AC%D8%B2%DB%8C%D= 8%B2%D9%87-%D8%B9%D8%B1%D9%88%D8%B3-%D8%B3%D9%88%D8%A7%D8%AD%D9%84-%D9%BE%D= 9%88%DA%A9%D8%AA-%D8%A7%D8%B3%D8%AA%D8%9F http://www.flashkhor.com/forum/showthread.php?tid=3D265184 http://www.fortress-forever.com/forums/member.php?u=3D20577 http://www.forum.canary98.ir/entry.php?b=3D14 http://www.forum.hostsaz.ir/thread-37-post-48.html http://www.forum.mybirds.ir/members/tablooaraz.html http://www.forum.parsbtc.com/showthread.php?2517-=D8=A7=D9=86=D9=88=D8=A7= =D8=B9-=D8=A8=DA=A9-=D9=84=DB=8C=D9=86=DA=A9-=D8=A8=D8=A7-=DA=A9=DB=8C=D9= =81=DB=8C=D8=AA http://www.forum.zibatan.ir/members/tablooaraz-22185/ http://www.forums.downloadkral.com/entry.php?32621-%D9%81%D8%B1%D9%88%D8%B4= -%D8%A8%DA%A9-%D9%84%DB%8C%D9%86%DA%A9-1 http://www.forums.mihandownload.com/member375804.html http://www.forzanocerina.it/forum/member.php?u=3D5592 http://www.gharian.ir/showthread.php?s=3D53f46c5a&tid=3D7211 http://www.hachiroku.com.au/forums/member.php?7638-tablooaraz http://www.heroes-networks.com/member/7808-tablooaraz http://www.highlander-community.com/forum/member/249-tablooaraz http://www.himt.ir/thread-4832-post-37578.html http://www.hobbyshop.ir/member.php?1136-tablooaraz http://www.iecloob.ir/showthread.php?tid=3D1514 http://www.iniuria.us/forum/member.php?138376-tablooaraz http://www.interbasket.net/forums/member.php?113314-tablooaraz http://www.i-phone.ir/forums/thread99980227.html http://www.iranamerica.com/forum/showthread.php?t=3D44456 http://www.iranamerica.com/forum/showthread.php?t=3D44499 http://www.iran-eng.ir/member.php/930500-tablooaraz http://www.iran-eng.ir/member.php/930500-tablooaraz http://www.iranianelite.com/members/tablooaraz-6197/ http://www.iranjoman.com/member-68538/ http://www.iranmicro.ir/forum/members/tablooaraz/ http://www.iranshahrsaz.com/member.php?action=3Dprofile&uid=3D109156 http://www.iranufc.biz/member5501.html http://www.iranufc.biz/member5501.ht= ml http://www.ir-eng.ir/f164/%D8%AA%D8%AE%D8%AA%D9%87-%D9%86%D8%B1%D8%AF-%D8%A= 8%D8%A7%D8%B2%DB%8C-%D8%A7%D8%B5%DB%8C%D9%84-%D8%A7%DB%8C%D8%B1%D8%A7%D9%86= %DB%8C-11126.html http://www.irosclass.ir/tablooaraz/ http://www.ir-tci.org/forum/member.php?u=3D1275273 http://www.joomi.ir/forum/members/tablooaraz/ http://www.joomi.ir/forum/members/tablooaraz/ http://www.justih.org/Binder-Bench/member.php?u=3D17166 http://www.keshavarzirooz.ir/Forum/Post/21/ http://www.khatesefid.com/forum/showthread.php?530-%D9%81%D8%B1%D9%88%D8%B4= -%D8%A8%DA%A9-%D9%84%DB%8C%D9%86%DA%A9 http://www.lawwiki.ir/forum/showthread.php?tid=3D3002 http://www.leatherneck.com/forums/member.php?123138-tablooaraz http://www.maidireborsa.it/member.php?462790-tablooaraz http://www.mbartar.ir/Forum/yaf_postsm20616_hmyt-hdyy-tblyGty-dr-khsb-w-khr= -shm.aspx http://www.mellifera.ir/Forum/post30878.html http://www.migranthelp.com/phpbbfa/viewtopic.php?p=3D315559 http://www.mihangame.com/forum/member.php?u=3D3808 http://www.mihanwebhosting.com/forum/f44/15-%D8%A7%D8%B4%D8%AA%D8%A8%D8%A7%= D9%87-%D8%AF%D8%B1-%D8%B7%D8%B1%D8%A7%D8%AD%DB%8C-%D8%B3%D8%A7%DB%8C%D8%AA-= 844/ http://www.my-fish.ir/Forum/Post/199 http://www.myl.ir/member.php?10-tabloo= araz http://www.ncee2013.ir/Forum/Post/85/ http://www.northwestmagic.com/forums/member.php?u=3D10151 http://www.oil-eng.ir/member/335-tablooaraz http://www.oople.com/forums/member.php?u=3D25726 http://www.p30data.com/forum/showthread.php?66187-%D9%81%D8%B1%D9%88%D8%B4-= %D8%A8%DA%A9-%D9%84%DB%8C%D9%86%DA%A9 http://www.pcbsupplies.co.uk/member.php/17993-tablooaraz http://www.pelletsmoking.com/blogs/tablooaraz-12236/backlink-4/ http://www.phorum.ir/thread-245-post-246.html http://www.phorum.ir/thread-474.html http://www.pmsclan.com/member/86143-tablooaraz http://www.raazebaghaa.org/forum/member.php?action=3Dprofile&uid=3D922 http://www.racechat.co.nz/horse/member.php?5001-tablooaraz http://www.roboyar.ir/forum/index.php/topic,91.0.html http://www.roozmenu.com/showthread.php?p=3D811249 http://www.runuo.net/forum/member/1542-tablooaraz http://www.sadafak.ir/f-33/%D8%AA%D9%81%D8%A7%D9%88%D8%AA-%D9%82%D8%A7%D8%B= 1%DA%86-%D9%86%D8%A7%D8%AE%D9%86-%D9%88-%D9%BE%D8%B3%D9%88%D8%B1%DB%8C%D8%A= 7%D8%B2%DB%8C%D8%B3-%D9%86%D8%A7%D8%AE%D9%86-324%20/ http://www.sanjeshy.org/forum/showthread.php?tid=3D21 http://www.sc2mafia.com/forum/member.php/30346-tablooaraz http://www.shadowera.com/member.php?98560-tablooaraz http://www.shahrsakhtafzar.com/forum/member.php?u=3D249204 http://www.slickdawg.com/member.php?951-tablooaraz http://www.support-wp.ir/f23/%D8%A7%D8%B1%D8%B3%D8%A7%D9%84-%D9%86%D9%88%D8= %AA%DB%8C%D9%81%DB%8C%DA%A9%DB%8C%D8%B4%D9%86-4598/ http://www.tajerbank.com/forum/members/18848.html http://www.thermolife.com/forum/members/tablooaraz/ http://www.tollyclub.com/forums/member.php?2220-tablooaraz http://www.tollyclub.com/forums/member.php?2220-tablooaraz http://www.topforum.ir/threads/259669 http://www.urbangamers.net/forum/member.php?11900-tablooaraz http://www.vbiran.ir/members/tablooaraz/ http://www.vbiran.ir/weblogha/tablooaraz/1619-backlink.html http://www.vbiran.ir/weblogha/tablooaraz/1619-backlink.html http://www.visitorguard.com/forum/member/123-tablooaraz http://www.volvoxc.com/forums/member.php?61290-tablooaraz http://www.warofrightsforum.com/member.php?6043-tablooaraz http://www.webhostingtalk.af/member/44-tablooaraz http://www.weldeng.net/MyBB/Thread-=D8=B7=D8=B1=D8=A7=D8=AD=DB=8C-=D8=B3=D8= =A7=DB=8C=D8=AA-=D9=81=D8=B1=D9=88=D8=B4-=D9=81=D8=A7=DB=8C=D9=84 http://www.wikivb.ir/member.php?u=3D13728 http://www.xebushanoi.com/forum/member.php?u=3D117908 http://www.yaalee.com/forums/member.php?u=3D4438 http://www.yekta-core.ir/members/tablooaraz.html http://yaran.us/member.php?148-tablooaraz http://yareghaeb.com/entry.php?b= =3D36 http://yareghaeb.com/entry.php?b=3D37 http://yareghaeb.com/entry.php?b=3D38 http://yareghaeb.com/entry.php?b=3D39 http://yasbooks.com/Forum/post54.html https://arafarin.com/forum/topic/77-%D9%86%D8%B1%D9%85-%D8%A7%D9%81%D8%B2%D= 8%A7%D8%B1%D9%87%D8%A7%DB%8C-%D9%85%D9%88%D8%B1%D8%AF-%D9%86%DB%8C%D8%A7%D8= %B2-%D8%A8%D8%B1%D8%A7%DB%8C-%D8%B7%D8%B1%D8%A7%D8%AD%DB%8C-%D8%B3%D8%A7%DB= %8C%D8%AA/ https://ariadl.ir/member.php?u=3D139128 https://ariadl.ir/member.php?u=3D13= 9128 https://aw.my.com/en/forum/member.php?525219-u_18271821 https://barnamenevisan.org/Forums/Post7652.html https://digielectric.com/forum/showthread.php?tid=3D15229 https://dotastage.ir/forum/members/110.html https://ekahroba.com/forum/members/821.html https://fishsniffer.com/forums/member.php?u=3D40342 https://forum.aev-conversions.com/member/35150-tablooaraz https://forum.azaronline.com/%D9%87%D8%A7%D8%B3%D8%AA-%D9%84%DB%8C%D9%86%D9= %88%DA%A9%D8%B3/1308-%D9%88%DB%8C%DA%98%DA%AF%DB%8C-%D9%87%D8%A7%DB%8C-%D9%= 87%D8%A7%D8%B3%D8%AA-%D8%AE%D9%88%D8%A8.html https://forum.azaronline.com/members/tablooaraz.html https://forum.azaronline.com/members/tablooaraz.html https://forum.carx-tech.com/member/2762-tablooaraz https://forum.opencart.ir/member9473.html https://forum.p30web.org/members/12139 https://forum.parsiandownload.net/thread-16427.html https://forum.persianbax.ir/threads/%D8%B3%D8%A7%D9%84%D9%86-%D8%B2%DB%8C%D= 8%A8%D8%A7%DB%8C%DB%8C-%D8%B9%D8%B1%D9%88%D8%B3%DB%8C-%D8%AE%D9%88%D8%A8-%D= B%8C%D8%A7-%D8%A8%D8%AF%D8%9F.36827/ https://forum.persiantools.com/threads/=D8=A8=DB=8C=D9=86=DA=AF-=D9=88=D8= =A8=D9=85=D8=B3=D8=AA=D8=B1-=DA=86=DB=8C=D8=B3=D8=AA-=D8=9F.893695 https://forum.projectgorgon.com/member.php?1886-tablooaraz https://forum.scopely.com/member.php?56082-tablooaraz https://forum.slingshotroadsters.com/member/1767-tablooaraz https://forum.talarebourse.com/threads/%D9%87%D8%B2%DB%8C%D9%86%D9%87-%D8%A= 7%D8%B1%D8%AA%D9%88%D8%AF%D9%86%D8%B3%DB%8C.217988/ https://forum.zenstudios.com/member.php?282963-tablooaraz https://forums.animparadise.com/member.php?u=3D16561 https://forums.eagle.ru/member.php?u=3D130385 https://forums.sakhtafzarmag.com/members/27325-tablooaraz https://forums.sneakycrab.com/member/272-tablooaraz https://forums.viivo.com/member.php?6268-tablooaraz https://iranchip.ir/member/270-tablooaraz https://iripb.com/forums/topic/1583-%D9%85%D8%AD%D8%AA%D9%88%D8%A7%DB%8C-%D= 8%A7%D9%82%D8%AA%D8%A8%D8%A7%D8%B3%DB%8C-%DA%86%DB%8C%D8%B3%D8%AA/ https://japanbase.net/forum/member.php?1662-tablooaraz https://kleefa.ir/forums/showthread.php?tid=3D1419 https://kleefa.ir/forums/showthread.php?tid=3D1749 https://mamasite.ir/fa/forum/516985/%D8%A7%D9%86%D9%88%D8%A7%D8%B9-%D8%B7%D= 8%B1%D8%AD-%D8%B5%D9%86%D8%B9%D8%AA%DB%8C https://manesht.ir/forum/user-140615.html https://mihanscript.ir/forum/entry.php?22-%D9%81%D8%B1%D9%88%D8%B4-%D8%A8%D= A%A9-%D9%84%DB%8C%D9%86%DA%A9-3 https://minitooti.ir/member.php?881-tablooaraz https://mmatorch.com/forum/member.php?2560-tablooaraz https://sefserver.com/forum/member.php?385-tablooaraz https://talk.patoghu.com/members/tablooaraz-u590824.html https://www.1forexforum.com/member.php?4545-baeveasuasda8070 https://www.buy-steroids.com.au/forum/member/3911-tablooaraz https://www.darksphere.co.uk/forum/member/57-tablooaraz https://www.downloadc3.com/forums/member.php?33109-tablooaraz https://www.gumbug.com/forum/member/237-tablooaraz https://www.happydays.ir/user-1442.html https://www.happydays.ir/user-1442.html#comments/1 https://www.hianzali.com/members/1296.html https://www.isarms.com/forums/members/tablooaraz.html https://www.nowaroos.com/Thread-%D8%AA%D8%B4%D8%B1%DB%8C%D9%81%D8%A7%D8%AA-= %D8%B1%D9%86%D8%B3%D8%A7%D9%86%D8%B3-%DB%8C%D8%A7%D9%88%D8%B1%DB%8C--3372?p= id=3D7455692#pid7455692 https://www.parsipet.ir/member.php?u=3D22918 https://www.parstuning.com/members/13666.html https://www.parstuning.com/members/13666.html https://www.pokecommunity.com/member.php?u=3D694922 https://www.roadnav.com/member/11362-tablooaraz https://www.tabletennisdaily.com/forum/member.php?42425-tablooaraz https://www.tarfandestan.com/forum/blogs/566595/blog15900.html https://www.tarfandestan.com/forum/member566595.html https://www.tarfandestan.com/forum/member566595.html --=20 You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.= From owner-freebsd-net@freebsd.org Sun Dec 17 07:52:17 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4280CE8202F; Sun, 17 Dec 2017 07:52:17 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-it0-x229.google.com (mail-it0-x229.google.com [IPv6:2607:f8b0:4001:c0b::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 01A3467BFE; Sun, 17 Dec 2017 07:52:17 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-it0-x229.google.com with SMTP id u62so25110887ita.2; Sat, 16 Dec 2017 23:52:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=qLtKkYsJxbBYZ1R8Ck+GbDEbiQ/WBfB0LwNqLxKzl4U=; b=oVq9lkcO7dJYLxUDS3Iy1iQ6p1OISPLT2ezO0l2lNbj8a2vQPQWObV98SlPc3pm9hk 9FjPVwgwP7Au/BxR2NrYMl1twDeiW4ZJBwn0n1cL/cJCxGPmlUqb+xAfnQppN+iWijBU mYneXeJVDE8VhPmaLK/2oYAv5wJX8QmJT9BRjA3Il6L0oq7aLaz9KGVE1NyDY0ApD/V2 y8dJdjwBWQ2n/ZlYcRmMdxIRXNaPvi/MO+KC7xnYkpwcxB/+GJPDv8T8ieUibWAXFbdf 8frz3Eb6gas/I9wdZBxNXFcaex1vtp+FOh4YQhzSw1mVu8Ve22LKKau7IDEhoYIaJq9s gaqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=qLtKkYsJxbBYZ1R8Ck+GbDEbiQ/WBfB0LwNqLxKzl4U=; b=AMe3b0dLDYLoivV8wZLhDNe/jdbMwH336y8nBbpLpRZPAwRqy9kZwmFAURKOGFwMWZ jdSf8CtLR/XRy7TBaD5rGQxJ+Nzx4NreiVuxCGSK4jxJLooPmeSQbH5djLlKHVM5KVOO mpyimkLRtmXgJpBNUOBRzZctZmquox4u0c9FP6NOQ2zmHl8zD/1cNAQkxD1/RYT5xVro zcJriVERu8524eHoRZBMKj/ejDcgNyiUiwXtQv+aDGYgndyB6Qsw0ERsM6/In9Z80bny 8hwmJyKUAVhxQmUz7cbKnsxYHVJatdPF7mMiUv8SbX3ETeNU0tYFMxUwPpxqMFWDEvAz e3OQ== X-Gm-Message-State: AKGB3mLeDk22nEIygmODhd0LmsWf/eO/QfTelypQ4YbAWhH3Q5fqdRhJ 5QZUM0zg+oVrFfg7/jlH7Ba9iDX39eJZAxqgrAY= X-Google-Smtp-Source: ACJfBot5Uj/kR8IBsCYu85nXRqUpT2DImRtruLCt8iwyVIs8HoFjCsYw2YuJM7cX24FmA1mKENX49L6U+z+JSZFQUaE= X-Received: by 10.36.131.200 with SMTP id d191mr15476180ite.97.1513497136118; Sat, 16 Dec 2017 23:52:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.2.144.71 with HTTP; Sat, 16 Dec 2017 23:52:15 -0800 (PST) Received: by 10.2.144.71 with HTTP; Sat, 16 Dec 2017 23:52:15 -0800 (PST) In-Reply-To: References: From: Sami Halabi Date: Sun, 17 Dec 2017 09:52:15 +0200 Message-ID: Subject: need help using ng_patch to modify src/dst packets or alternative way To: freebsd-net@freebsd.org, FreeBSD Current Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Dec 2017 07:52:17 -0000 hi, Can you help in my situation? My goal is so Box in my lan 10.1.1.2 to talk to 10.1.1.1 and actually it would be talking to X.X.X.X outside ip using one of my public IPs say 1.1.1.1. I'm trying to modify packets to passthrough to a local IP. I have a box that a specific IP is routed to it.. say 1.1.1.1 in my bce0 i don't have that ip configured but i have my public IP that say 2.2.2.2 that 1.1.1.1 is routed to it. i configured 10.1.1.1/24 in bce0, my target box is 10.1.1.2/24. i tried the following inside ngctl: mkpeer ipfw: patch 300 in name ipfw:300 src_dst_chg msg src_dst_chg: setconfig { count=2 csum_flags=1 ops=[ { mode=1 value=0x0a010101 length=4 offset=3 } { mode=1 value=0x0a010102 length=4 offset=4 } ] } in my box(10.1.1.1) i did: sysctl net.inet.ip.fw.one_pass=0 /sbin/ipfw add 50 netgraph 300 ip from any to any to 1.1.1.1 then i do simple ping from outside box i see the packets arrive on my 160 rule but never leaves the box.. I would at least see packeta flow one direction to 10.1.1.2 and then that need another ipfw and netgraph opposite rule. If you have alternative way I'm happy to try... Help much appreciated... Sami From owner-freebsd-net@freebsd.org Sun Dec 17 09:29:34 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 723E7E841DA; Sun, 17 Dec 2017 09:29:34 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D51736A15F; Sun, 17 Dec 2017 09:29:33 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBH9TNoh065840 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 17 Dec 2017 10:29:24 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: sodynet1@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBH9TFZn012275 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sun, 17 Dec 2017 16:29:15 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: need help using ng_patch to modify src/dst packets or alternative way To: Sami Halabi , freebsd-net@freebsd.org, FreeBSD Current References: From: Eugene Grosbein Message-ID: <5A3638E5.6090308@grosbein.net> Date: Sun, 17 Dec 2017 16:29:09 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Dec 2017 09:29:34 -0000 17.12.2017 14:52, Sami Halabi пишет: > hi, > > Can you help in my situation? My goal is so Box in my lan 10.1.1.2 to talk > to 10.1.1.1 and actually it would be talking to X.X.X.X outside ip using > one of my public IPs say 1.1.1.1. If you need this just for single or several tcp ports, easiest way is to use any of port forwarders/bouncers like this: pkg install bounce bounce -a 10.1.1.1 -b 1.1.1.1 -p 443 X.X.X.X 443 From owner-freebsd-net@freebsd.org Sun Dec 17 10:59:27 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39B24E85B19; Sun, 17 Dec 2017 10:59:27 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: from mail-it0-x230.google.com (mail-it0-x230.google.com [IPv6:2607:f8b0:4001:c0b::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DDDA16C4CB; Sun, 17 Dec 2017 10:59:26 +0000 (UTC) (envelope-from sodynet1@gmail.com) Received: by mail-it0-x230.google.com with SMTP id d137so24631392itc.2; Sun, 17 Dec 2017 02:59:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=IsRX8m3q8HkLj0DYOG7YFnXmJSWDwlRV1L+Sex6URxE=; b=nO7QYSMBqma3wRuvtc5YMGE7EGh+E+MPjdr2RGEHjlJPIYnjXOa4QxOJcK8STBg2Ut 1ToQgsmUxtCV8MJH9V2anCHGzZ0lqvgei+lq1/zsl8R/3Ha5ddwVlQ1YHkKo94JuIaH+ v8FHYhGbGBgSmmgVSNyKfFseYLlNgWbzDhzgr1nBb+henutsSqIZJVogj0sfz6f+DTFR 2K7wjHbHPzRfXX3b1VAcNiPsUa7lLOJyqtehvTJDNS6BBC8esEhlZHt28JsPjPww6ygP 4QulfNgrGegds4rYpNf08bbD7qtFJYUVF9YGqryUTUFr/Gd5CV3e2J15voGyFnESmIN4 QIeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=IsRX8m3q8HkLj0DYOG7YFnXmJSWDwlRV1L+Sex6URxE=; b=KaFCW2O2QvTBk+rt/BnK0uh79EoG+BkMsttGLb/TjKSlD/eunVGQR2ovSqyG1Gdmh4 ejqxsOMAKW0BQCeBoJRZ/LVvBuahyIyrVMzZU7ElNx9IA/qBzQ9DuorJBrgon1zPWbpV 2Ir7uwuKHWuO4oW+sZWsKVc8xJa6L+qGxdWE6XAx+dJ2rIyZeQYRlBQ+4tYUeJxaMkG9 eYKcBeKZmxqXZECDsG0OPk6cqXvlgoV43sFnRBCFQBaYRxHhA3ReRz8JcJzwiAfzC4bs FJrkumJNiYxcmz+6HqroQcyKBaUr20v3BSt3BqPWA2Si8NWI1i7xl8xzvmZgMECyT5YB X+Lw== X-Gm-Message-State: AKGB3mI84VvyPb1OtD9eCNtcQrF9U2gxkJ5NeAR7NuyIQm4YhP1UtQYZ QyqBhZ+pkMawuLCvrUTTFMDQi4zaIaLkXw7CxxM= X-Google-Smtp-Source: ACJfBov5O64XilvXpoQ3tVbY/x0tdGU6VavAyjzl2r2h4dX9l3/j5IA4E2GKX/lvpTZs4Z0LZmAaywhRnFh4phEDKLQ= X-Received: by 10.36.133.135 with SMTP id r129mr16758271itd.69.1513508366204; Sun, 17 Dec 2017 02:59:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.2.144.71 with HTTP; Sun, 17 Dec 2017 02:59:25 -0800 (PST) Received: by 10.2.144.71 with HTTP; Sun, 17 Dec 2017 02:59:25 -0800 (PST) In-Reply-To: <5A3638E5.6090308@grosbein.net> References: <5A3638E5.6090308@grosbein.net> From: Sami Halabi Date: Sun, 17 Dec 2017 12:59:25 +0200 Message-ID: Subject: Re: need help using ng_patch to modify src/dst packets or alternative way To: Eugene Grosbein Cc: freebsd-net@freebsd.org, FreeBSD Current Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Dec 2017 10:59:27 -0000 Hi Eugene, I'm looking for a solution for IP traffic. in linux iptables its possible but I couldn't find freebsd way yet. bkuncr soulution works for tcp only. Thanks for the hint though, Sami =D7=91=D7=AA=D7=90=D7=A8=D7=99=D7=9A 17 =D7=91=D7=93=D7=A6=D7=9E=D7=B3 2017= 11:29 AM,=E2=80=8F "Eugene Grosbein" =D7=9B=D7=AA=D7= =91: > 17.12.2017 14:52, Sami Halabi =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > hi, > > > > Can you help in my situation? My goal is so Box in my lan 10.1.1.2 to > talk > > to 10.1.1.1 and actually it would be talking to X.X.X.X outside ip usin= g > > one of my public IPs say 1.1.1.1. > > If you need this just for single or several tcp ports, easiest way > is to use any of port forwarders/bouncers like this: > > pkg install bounce > bounce -a 10.1.1.1 -b 1.1.1.1 -p 443 X.X.X.X 443 > > > From owner-freebsd-net@freebsd.org Sun Dec 17 18:42:49 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48296E91C23; Sun, 17 Dec 2017 18:42:49 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id B04AB79B9D; Sun, 17 Dec 2017 18:42:47 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBHIgXNO069873 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 17 Dec 2017 19:42:33 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: sodynet1@gmail.com Received: from [10.58.0.4] (dadv@[10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBHIgOV7039505 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Mon, 18 Dec 2017 01:42:25 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: need help using ng_patch to modify src/dst packets or alternative way To: Sami Halabi References: <5A3638E5.6090308@grosbein.net> Cc: freebsd-net@freebsd.org, FreeBSD Current From: Eugene Grosbein Message-ID: <5A36BA90.8020302@grosbein.net> Date: Mon, 18 Dec 2017 01:42:24 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Dec 2017 18:42:49 -0000 17.12.2017 17:59, Sami Halabi wrote: > Hi Eugene, > I'm looking for a solution for IP traffic. in linux iptables its possible but I couldn't find freebsd way yet. > bkuncr soulution works for tcp only. Then, you need to realize that for every packet, you need to change (translate) both of source IP address from 10.1.1.2 to 1.1.1.1 and destination IP address from 10.1.1.1 to X.X.X.X. This is called network address translation and, in fact, you need NAT. But not ordinary "simple" NAT that translates only source address in outgoing packets (and destination in incoming replies) but double or "binat" to translate destination address in outgoing packets too (and source address in corresponding replies). This is possible to do with two instances of "ipfw nat" (or natd) for single external destination but not for arbitrary number of external destinations. They say, "pf(4)" packet filter can perform "binat" properly. I have not tried that. You should start reading its documentation. From owner-freebsd-net@freebsd.org Mon Dec 18 14:17:56 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B652E8DE9E for ; Mon, 18 Dec 2017 14:17:56 +0000 (UTC) (envelope-from johnllyon@gmail.com) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AFF117C676 for ; Mon, 18 Dec 2017 14:17:55 +0000 (UTC) (envelope-from johnllyon@gmail.com) Received: by mail-wm0-x22b.google.com with SMTP id f9so29830042wmh.0 for ; Mon, 18 Dec 2017 06:17:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=zOgyS6eJH17rbxaFx75wcbG6OGlfoExoutLs0WcLiI4=; b=dqIADQnhvN3bWBYex8xDyXtUQRpGQBVZgoC0y+abUxBMDt9oSa/lC1jJonUl1b793U ns0+ttFcNv5crc5J6PV4f1QGCP+Bbk3RrphYMGypmAmuNrFiiITw4WHS8Bqk2mOiu4ek y14vMF1O3FjJhODpl0EGV9VAKAcyo+uiBhPXLTwLAzXdRtpe01sk5zvUaAGGN2fDs6C+ 0mjoc0wFNGB3X5RDuPLsqSgkJQwvrLNP9tFUInnd7Tsx9OpGw6oA37tp47OR9i6n1VFJ nWTPtSDwupCkkRIK6d1ajbBMX3OWUFYIuGouhxvBV3T5VqofvPcMRxNvuE+vhyKpR3J5 aVzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=zOgyS6eJH17rbxaFx75wcbG6OGlfoExoutLs0WcLiI4=; b=lOGSCkI0OdKFJaQ7ppt3lUSg1RUnXaRHInylMB977tgETujL6l1R7gixOq2Rd+9Ui8 mGWZBoAWrWHe226nnzYq9TWCMxUVc2akeGEG3TVtAomHAcveddogxT6thRMZRJgY9k1Z 0Z3XYUUwJMy1C4Wm53loB6MUsX9bK0lK2MVHHJ2bXZH0Z2Knguaayg12ancf7TxxUgq+ WSKSnyVWSyz4t3sQGIg/Y8a8Xanl+9IuF070Nd0tjCRukoeelVvPO3W9FQZV/6f870E4 +1exgC23S5R0t1WneBGC6otNRGho9BoRuSly/nAPbk3CCYvbWeg52jZH5yHHi8ihzC8c 3FjQ== X-Gm-Message-State: AKGB3mKUbH44qpBASIxa0hiP3oaYKb6WFthlL7LSePizvmZWHA+GqXxf 58rImn32KglM2QAQbXdFDOcRLzLac0odlsextw0= X-Google-Smtp-Source: ACJfBovf6bD7MXwHdi2L8Emo5Z5jKwZ21shloC9G2mvTqVIAPlHaEc/rngbAlhUG9vlox73O+igsrUAdXCmI15wJ6dI= X-Received: by 10.80.241.19 with SMTP id w19mr13499edl.123.1513606674110; Mon, 18 Dec 2017 06:17:54 -0800 (PST) MIME-Version: 1.0 Received: by 10.80.211.20 with HTTP; Mon, 18 Dec 2017 06:17:33 -0800 (PST) In-Reply-To: <5A34E7CF.2000104@omnilan.de> References: <5A3225BF.6020205@omnilan.de> <5A32F63E.8010205@grosbein.net> <5A338C5A.20300@omnilan.de> <5A34E7CF.2000104@omnilan.de> From: John Lyon Date: Mon, 18 Dec 2017 09:17:33 -0500 Message-ID: Subject: Re: Need Netgraph Help To: Harry Schmalzbauer Cc: freebsd-net@freebsd.org, Eugene Grosbein Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Dec 2017 14:17:56 -0000 Harry, Thanks for the help. I haven't had the chance to sit down and tinker over the weekend, but I hope I may get sometime in the next day or two. I will see what happens when I try to connect my filter as you suggested. My intention is essentially to use NetGraph in order to add some simple layer 2 firewalling capabilities to my PFSense router (FreeBSD + pf + pretty GUI for other functions) on my network. Unfortunately, pf on FreeBSD only appears to filter at layers 3 and 4. I need to also filter and redirect layer 2 traffic. I'm aware that IPFW can probably do what I want (filter and redirect based on MAC address and ethernet frame type). However, I prefer the pretty GUI of PFSense for convenience and time saving (I could duplicate all of the rules and functionality in the command line, but the GUI makes administration a lot easier). However, I don't want to hack together a solution that involves two firewalls running on the same box when I was hoping to use Netgraph to filter at layer 2 before passing other traffic up to pf for layer 3 and 4 filtering. However, this may be the route I have to go (PFSense does use both PF and IPFW when the "captive portal" functionality is enabled, so it is technically possible). Thanks for the link to the NetBSD presentation. I'm already aware of it, it was one of the first things I found when I googled about netgraph trying to sort through this whole mess. :-) -------------------------------- John L. Lyon PGP Key Available At: https://www.dropbox.com/s/skmedtscs0tgex7/02150BFE.asc On Sat, Dec 16, 2017 at 4:30 AM, Harry Schmalzbauer wrote: > Bez=C3=BCglich John Lyon's Nachricht vom 15.12.2017 19:59 (localtime): > > Harry and Eugene (and others), > > > > I appreciate all of your help. It's been really insightful. Although = I > > feel like I'm getting much closer to the solution, I don't think my > problem > > has been diagnosed. I've outlined my thought process below. Can you > > please tell me if I am misunderstanding something? Admittedly, I am no= t > a > > kernel developer and my C language skills have atrophied the last few > > years. However, I've reviewed my script and I looked in the code for > > ng_etf.c and I don't think I am violating any of the requirements for > > linking a hook for no match. > > > > As Eugene stated: > > > >>> 1) referenced "matchook" exists and you should not use "indirect name= " > > here, > >>> only hook own name, or else you get error ENOENT (No such file or > > directory); > > > > This does not seem to be a problem as the upper and lower hooks for the > em1 > > already exist (I can confirm this). > > > >>> 2) referenced "matchook" is *not* downstream hook, or else you get > error > >>> EINVAL (Invalid argument); > > > > I read the ng_etf.c file in the source tree and found this little > snippet: > > > > /* and is not the downstream hook */ > > if (hook =3D=3D etfp->downstream_hook.hook) { > > error =3D EINVAL; > > break; > > } > > > > This appears to be an error check to make sure you are not creating a > cycle > > in the graph by referencing the ETF node's own downstream hook (i.e. > > filtering incoming traffic and circularly feeding non-matching frames > back > > into the ETF's own filter). I'm not doing this. I am feeding > non-matching > > packets into the *lower* hook of another ether node and not back into t= he > > *downstream* hook of the etf node I am creating. As a result, my > netgraph > > Ah, sorry, I was reading your setup too quickly and missed that em0|em1 > detail. > Since I'm no netgraph expert and also no kernel hacker due to C skills, > and on top I don't have any ng_etf experience, I'm out at this point > unfortunately. I just remembered the shell quoting issue I had once > myself and thougth this would be an easy one ;-) > > I _think_ it's not possible to redierct the packets that way with > ng_etf. You'd need at least to add the third hook to ng_etf. In the > manpage, it's a user land hook. > Have you tried if > ngctl connect em1: lan_filter: lower mydrain > works? > If so, your "setfilter" message might also work. > I think the missing third hook is the key to your solution =E2=80=93 whil= e I > don't know your intention, but I guess you want to get specific > type-tagged frames beeing transmitted on a dedicated interface. > > Pleas see > http://www.netbsd.org/gallery/presentations/ast/2012_ > AsiaBSDCon/Tutorial_NETGRAPH.pdf > on page 32+33. That example corresponds to the man page. > > Hope that helps, > > -harry > From owner-freebsd-net@freebsd.org Tue Dec 19 06:33:04 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D49FE8CE88 for ; Tue, 19 Dec 2017 06:33:04 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from frv191.fwdcdn.com (frv191.fwdcdn.com [212.42.77.191]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 12AEF68525 for ; Tue, 19 Dec 2017 06:33:03 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from [10.10.80.11] (helo=frv197.fwdcdn.com) by frv191.fwdcdn.com with esmtp ID 1eRBCA-0000pT-Jv for freebsd-net@freebsd.org; Tue, 19 Dec 2017 08:16:06 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:Message-Id:To: Subject:From:Date:Sender:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=lEQRQ5tdYioFYrEmPHM+WHMJhzSRUo+kqbdl+A8MYSY=; b=qA12F1p/nVWB5UzDnxYpX2LeH1 C4tfjKDJKk8wU91STN/UxCVtTTJK1PDA06Xq5tb/FZoNiSpS4pCNUFxa1FsHWBJvtXBk7ND7pi0Zi sLpfbgNm2H+Aswubrex6fS7ISTEGGYXuQ1xnBt0xRY3CB6fNpDY+rtwbQPyZhdyK6vc4=; Received: from [10.10.10.52] (helo=frv52.fwdcdn.com) by frv197.fwdcdn.com with smtp ID 1eRBC2-000D4x-89 for freebsd-net@freebsd.org; Tue, 19 Dec 2017 08:15:58 +0200 Date: Tue, 19 Dec 2017 08:15:58 +0200 From: wishmaster Subject: ng_patch and swap_pager_getswapspace error To: freebsd-net@freebsd.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1513663683.700534911.voagagit@frv52.fwdcdn.com> Received: from artemrts@ukr.net by frv52.fwdcdn.com; Tue, 19 Dec 2017 08:15:58 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Dec 2017 06:33:04 -0000 Hi, after I have applied ng_patch for setting TTL for outgoing packets with below rules kldload ng_ipfw 2>/dev/null kldload ng_patch 2>/dev/null /usr/sbin/ngctl -f- <<-SEQ mkpeer ipfw: patch 100 in name ipfw:100 ttl_set msg ttl_set: setconfig { count=1 csum_flags=1 ops=[ \ { mode=1 value=128 length=1 offset=8 } ] } SEQ /sbin/ipfw add 15002 netgraph 100 ip from me to not me recv "*" I see this error messages in the log-file: Dec 18 02:36:56 xxx kernel: swap_pager_getswapspace(32): failed Dec 18 02:37:00 xxx last message repeated 719 times Dec 18 02:37:00 xxx kernel: swap_pager_getswapspace(24): failed Dec 18 02:37:00 xxx kernel: swap_pager_getswapspace(32): failed Dec 18 02:37:00 xxx kernel: swap_pager_getswapspace(24): failed Dec 18 02:37:00 xxx kernel: swap_pager_getswapspace(32): failed ...... Dec 18 08:14:12 xxx last message repeated 15 times Dec 18 08:14:42 xxx last message repeated 12 times Dec 18 08:14:44 xxx kernel: pid 16634 (nginx), uid 80, was killed: out of swap space Dec 18 08:14:44 xxx kernel: Dec 18 08:14:44 ukrnasos kernel: pid 16634 (nginx), uid 80, was killed: out of swap space Dec 18 08:14:46 xxx kernel: swap_pager_getswapspace(32): failed Dec 18 08:15:12 xxx last message repeated 10 times This server contains VNET'ed jails for www, db, php. unaeme -a FreeBSD xxx 11.1-STABLE FreeBSD 11.1-STABLE #0 r325503: Tue Nov 7 13:38:44 EET 2017 wishmaster@xxx:/usr/obj/usr/src/sys/MY amd64 What is this? From owner-freebsd-net@freebsd.org Tue Dec 19 13:33:32 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6444EE83120 for ; Tue, 19 Dec 2017 13:33:32 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id EA85A78C72 for ; Tue, 19 Dec 2017 13:33:30 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBJDXHnB090052 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 Dec 2017 14:33:17 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: artemrts@ukr.net Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id vBJDXDet060997; Tue, 19 Dec 2017 20:33:13 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: ng_patch and swap_pager_getswapspace error To: wishmaster , freebsd-net@freebsd.org References: <1513663683.700534911.voagagit@frv52.fwdcdn.com> From: Eugene Grosbein X-Enigmail-Draft-Status: N1110 Message-ID: <5A391519.8040707@grosbein.net> Date: Tue, 19 Dec 2017 20:33:13 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <1513663683.700534911.voagagit@frv52.fwdcdn.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE, T_DATE_IN_FUTURE_Q_PLUS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * 0.0 T_DATE_IN_FUTURE_Q_PLUS Date: is over 4 months after Received: date * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-Spam-Level: ** X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Dec 2017 13:33:32 -0000 On 19.12.2017 13:15, wishmaster wrote: > Hi, > > after I have applied ng_patch for setting TTL for outgoing packets with below rules > > kldload ng_ipfw 2>/dev/null > kldload ng_patch 2>/dev/null > > /usr/sbin/ngctl -f- <<-SEQ > mkpeer ipfw: patch 100 in > name ipfw:100 ttl_set > msg ttl_set: setconfig { count=1 csum_flags=1 ops=[ \ > { mode=1 value=128 length=1 offset=8 } ] } > SEQ > > /sbin/ipfw add 15002 netgraph 100 ip from me to not me recv "*" Why do you have incoming ip packets sourced from your IP? This generally means source was spoofed and such incoming packets should be silently dropped. > What is this? "After" does not mean "because of". Your processes ate all memory and swap and ng_patch cannot be responsible for that. From owner-freebsd-net@freebsd.org Tue Dec 19 14:46:41 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8549E87B6C for ; Tue, 19 Dec 2017 14:46:41 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from frv190.fwdcdn.com (frv190.fwdcdn.com [212.42.77.190]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A9E337B966 for ; Tue, 19 Dec 2017 14:46:41 +0000 (UTC) (envelope-from artemrts@ukr.net) Received: from frv198.fwdcdn.com ([212.42.77.198]) by frv190.fwdcdn.com with esmtp ID 1eRJA9-0006Sf-Ju for freebsd-net@freebsd.org; Tue, 19 Dec 2017 16:46:33 +0200 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-Id:To:Subject:From:Date:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=elo2TxutjyN1F+CIeatp2TeUgKmcS+i0Wadlc2SyaF8=; b=qVt88I7vWNT//rRk2a2Y/1qB9j 4YGFTx8T3BU1untbpcwKiXLXTPaA+1l4yrfFix+u47dmx+syaLysi89RCWR77dUtLUwd/pP6tOvAE NdizKu6ax9XNUet6OYF2zZWfdktXjfBT8KJ4SFZ5Yf/dM7kK+0A8NZT61utDHm+9qJqs=; Received: from [10.10.10.52] (helo=frv52.fwdcdn.com) by frv198.fwdcdn.com with smtp ID 1eRJA0-0006G2-SF for freebsd-net@freebsd.org; Tue, 19 Dec 2017 16:46:24 +0200 Date: Tue, 19 Dec 2017 16:46:24 +0200 From: wishmaster Subject: Re[2]: ng_patch and swap_pager_getswapspace error To: freebsd-net@freebsd.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1513694407.556184943.ya3sdvt4@frv52.fwdcdn.com> In-Reply-To: <5A391519.8040707@grosbein.net> References: <1513663683.700534911.voagagit@frv52.fwdcdn.com> <5A391519.8040707@grosbein.net> X-Reply-Action: reply Received: from artemrts@ukr.net by frv52.fwdcdn.com; Tue, 19 Dec 2017 16:46:24 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Dec 2017 14:46:42 -0000 --- Original message --- From: "Eugene Grosbein" Date: 19 December 2017, 15:33:42 > On 19.12.2017 13:15, wishmaster wrote: > > Hi, > > > > after I have applied ng_patch for setting TTL for outgoing packets with below rules > > > > kldload ng_ipfw 2>/dev/null > > kldload ng_patch 2>/dev/null > > > > /usr/sbin/ngctl -f- <<-SEQ > > mkpeer ipfw: patch 100 in > > name ipfw:100 ttl_set > > msg ttl_set: setconfig { count=1 csum_flags=1 ops=[ \ > > { mode=1 value=128 length=1 offset=8 } ] } > > SEQ > > > > /sbin/ipfw add 15002 netgraph 100 ip from me to not me recv "*" > > Why do you have incoming ip packets sourced from your IP? It's ok. I use per-interface ACL. # out ipfw -fq table tbl_OUT_IF flush ... ipfw table tbl_OUT_IF add tun1 15000 # ... $cmd 100 skipto tablearg log all from any to any in recv "table(tbl_IN_IF)" $cmd 110 skipto tablearg log all from any to any out xmit "table(tbl_OUT_IF)" ### OUT ext_if tun0 $cmd 15000 nat 1 log all from not me to not me recv "*" # LAN traffic # !!! 15002 here $cmd 15020 allow log all from me to not me recv "*" # LAN traffic $cmd 15150 allow log all from me to any $ks :nts # Router traffic $cmd 15499 deny log all from any to any From owner-freebsd-net@freebsd.org Wed Dec 20 08:48:15 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39BD7E9FEB5 for ; Wed, 20 Dec 2017 08:48:15 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C4450633E5 for ; Wed, 20 Dec 2017 08:48:13 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBK8m5kG098708 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 20 Dec 2017 09:48:06 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: artemrts@ukr.net Received: from eg.sd.rdtc.ru (eugen@localhost [127.0.0.1]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTP id vBK8m2mR070678; Wed, 20 Dec 2017 15:48:02 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: ng_patch and swap_pager_getswapspace error To: wishmaster , freebsd-net@freebsd.org References: <1513663683.700534911.voagagit@frv52.fwdcdn.com> <5A391519.8040707@grosbein.net> <1513694407.556184943.ya3sdvt4@frv52.fwdcdn.com> From: Eugene Grosbein Message-ID: <5A3A23C2.2030707@grosbein.net> Date: Wed, 20 Dec 2017 15:48:02 +0700 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <1513694407.556184943.ya3sdvt4@frv52.fwdcdn.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE, T_DATE_IN_FUTURE_Q_PLUS autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * 0.0 T_DATE_IN_FUTURE_Q_PLUS Date: is over 4 months after Received: date * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 2.6 LOCAL_FROM From my domains * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-Spam-Level: ** X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Dec 2017 08:48:15 -0000 On 19.12.2017 21:46, wishmaster wrote: >>> /sbin/ipfw add 15002 netgraph 100 ip from me to not me recv "*" >> >> Why do you have incoming ip packets sourced from your IP? > > It's ok. I use per-interface ACL. > > # out > ipfw -fq table tbl_OUT_IF flush > ... > ipfw table tbl_OUT_IF add tun1 15000 # > ... > > > $cmd 100 skipto tablearg log all from any to any in recv "table(tbl_IN_IF)" > $cmd 110 skipto tablearg log all from any to any out xmit "table(tbl_OUT_IF)" > > > ### OUT ext_if tun0 > $cmd 15000 nat 1 log all from not me to not me recv "*" # LAN traffic > # !!! 15002 here > $cmd 15020 allow log all from me to not me recv "*" # LAN traffic It is not OK. It does not make any sense: "from me ... recv" is NOT any kind of normal LAN traffic. This expression describes spoofed traffic. From owner-freebsd-net@freebsd.org Thu Dec 21 13:14:24 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69646E961A7 for ; Thu, 21 Dec 2017 13:14:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5779665C69 for ; Thu, 21 Dec 2017 13:14:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vBLDEOW8076583 for ; Thu, 21 Dec 2017 13:14:24 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211062] [ixv] sr-iov virtual function driver fails to attach Date: Thu, 21 Dec 2017 13:14:24 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: IntelNetworking, needs-patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: piotr.pietruszewski@intel.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 13:14:24 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211062 Piotr Pietruszewski changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |piotr.pietruszewski@intel.c | |om --- Comment #9 from Piotr Pietruszewski --- (In reply to Richard Gallamore from comment #8) It is likely that this bug is fixed in newest driver provided by Intel on https://downloadcenter.intel.com/download/14688/Intel-Network-Adapters-Driv= er-for-PCIe-10-Gigabit-Network-Connections-Under-FreeBSD- . Feedback about problem resolution would be greatly appreciated. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Thu Dec 21 20:25:17 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 157F3E881FE; Thu, 21 Dec 2017 20:25:17 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:302:1100::7:9a96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B528F76B5A; Thu, 21 Dec 2017 20:25:16 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2jmc3pxtzDl2; Thu, 21 Dec 2017 21:24:48 +0100 (CET) From: Michael Grimm Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: performance issue within VNET jail Message-Id: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> Date: Thu, 21 Dec 2017 21:24:47 +0100 To: freebsd-jail@FreeBSD.org, freebsd-net@freebsd.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 20:25:17 -0000 Hi [ I did recently migrate my servers from bare metal to cloud instances = (OpenStack at OVH) ] [ FreeBSD 11.1-STABLE #0 r327055 = ] My setup is as follows and didn't change for the last couple of years: extIF0/pf/NAT <=E2=80=94> epairXa (bridge0) epairXb <-> jail Downloading a file (by wget) at the host is around 30 MB/s, and an = example tcpdump at extIF0 looks as follows: 19:32:10.711769 IP (tos 0x20, ttl 56, id 37539, offset 0, flags [DF], = proto TCP (6), length 8680) remote.http > myhost.14367: Flags [.], cksum 0x64ed (incorrect -> = 0x3223), seq 5753:14381, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 8628: HTTP 19:32:10.713851 IP (tos 0x20, ttl 56, id 37545, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x8d1e), seq 14381:15819, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713899 IP (tos 0x20, ttl 56, id 37546, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x6ade), seq 15819:17257, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713934 IP (tos 0x20, ttl 56, id 37547, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x1173), seq 17257:18695, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713962 IP (tos 0x20, ttl 56, id 37548, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0xcf7a), seq 18695:20133, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP When downloading the very same file within a VIMAGE jail the performance = drops to around 80 KB/s, quite a dramatic loss. An example tcpdump at = exitIF0 looks as follows: 19:34:36.284175 IP (tos 0x0, ttl 56, id 28618, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0x4478), seq 1449:4345, ack 146, win 235, options [nop,nop,TS val = 1007182125 ecr 4141429], length 2896: HTTP 19:34:36.481904 IP (tos 0x0, ttl 56, id 28620, offset 0, flags [DF], = proto TCP (6), length 1500) remote.http > myhost.63382: Flags [.], cksum 0xd11d (correct), seq = 1449:2897, ack 146, win 235, options [nop,nop,TS val 1007182175 ecr = 4141429], length 1448: HTTP 19:34:36.484109 IP (tos 0x0, ttl 56, id 28621, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0x2e5b), seq 15929:18825, ack 146, win 235, options [nop,nop,TS val = 1007182175 ecr 4141629], length 2896: HTTP 19:34:36.682006 IP (tos 0x0, ttl 56, id 28623, offset 0, flags [DF], = proto TCP (6), length 1500) remote.http > myhost.63382: Flags [.], cksum 0x4ab6 (correct), seq = 2897:4345, ack 146, win 235, options [nop,nop,TS val 1007182225 ecr = 4141629], length 1448: HTTP 19:34:36.684159 IP (tos 0x0, ttl 56, id 28624, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0xd7db), seq 18825:21721, ack 146, win 235, options [nop,nop,TS val = 1007182225 ecr 4141829], length 2896: HTTP A tcpdump at epairXa looks comparable. I did reduce all MTU settings at the involved interfaces from their = initial settings (1490) to an experimental setting of 1400, just to be = on the save side, to no avail. (FYI: I did have to reduce from 1500 to = 1490 to please IPSec after migration from bare metal to cloud = infrastructure.) Then, I did test the following settings found in the Net, to no avail = either: sysctl net.inet.tcp.tso=3D0 sysctl net.link.bridge.pfil_onlyip=3D0 sysctl net.link.bridge.pfil_bridge=3D0 sysctl net.link.bridge.pfil_member=3D0 sysctl net.add_addr_allfibs=3D0 I do have to admit that I am lost here, and that I cannot think about = what is going wrong. The last download I did try at my old severs has = been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I = did move my infrastructure from bare metal to cloud, thus I cannot test = anymore if my old servers would have shown that performance issue in the = meantime. Thus any feedback is highly recommended! Thanks in advance and regards, Michael From owner-freebsd-net@freebsd.org Thu Dec 21 20:39:53 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B7EDE892FE; Thu, 21 Dec 2017 20:39:53 +0000 (UTC) (envelope-from srs0=bbyf=dr=sigsegv.be=kristof@codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A754677654; Thu, 21 Dec 2017 20:39:52 +0000 (UTC) (envelope-from srs0=bbyf=dr=sigsegv.be=kristof@codepro.be) Received: from [192.168.228.1] (ptr-8ripyyhtmq100mi0yvj.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:2419:4e02:e137:9435:e3a1:1a8f]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id B482237C13; Thu, 21 Dec 2017 21:39:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sigsegv.be; s=mail; t=1513888789; bh=6hL79YC2NK2PptCdyIOzYzmY3O5K/hx2/NTMQ/Rse0Y=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=OWnNGkyVj3idYVW9FX7ICcU03qHT2ys39emr96EaxRUb8uvQuw7fPK6abXz+EExqd l/JoJ/MU1u9SP2wfTfSbNUxfAjmUIEwKeN30lZ+jDvucb5CeA1HeFvyp+jaBkaUEdV CZjWBonk0/5qn/N9Feb8N2bv2oYlQfo2pm/tGcWE= From: "Kristof Provost" To: "Michael Grimm" Cc: freebsd-jail@FreeBSD.org, freebsd-net@freebsd.org Subject: Re: performance issue within VNET jail Date: Thu, 21 Dec 2017 21:39:48 +0100 X-Mailer: MailMate (2.0BETAr6102) Message-ID: In-Reply-To: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 20:39:53 -0000 On 21 Dec 2017, at 21:24, Michael Grimm wrote: > > I do have to admit that I am lost here, and that I cannot think about > what is going wrong. The last download I did try at my old severs has > been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and > I did move my infrastructure from bare metal to cloud, thus I cannot > test anymore if my old servers would have shown that performance issue > in the meantime. > > Thus any feedback is highly recommended! > Can you try turning off TSO? (`ifconfig $ifname -tso`) There have been issues with pf and TSO checksums, which looked a lot like this (i.e. bad TCP performance). Those problems should be fixed, but this is easy to test. Regards, Kristof From owner-freebsd-net@freebsd.org Thu Dec 21 20:50:28 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D87B5E89DF4; Thu, 21 Dec 2017 20:50:28 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A4D8777E0C; Thu, 21 Dec 2017 20:50:28 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2kL2420czDnb; Thu, 21 Dec 2017 21:50:18 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Thu, 21 Dec 2017 21:50:17 +0100 Cc: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> To: Kristof Provost X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 20:50:28 -0000 Kristof Provost wrote: >=20 > On 21 Dec 2017, at 21:24, Michael Grimm wrote: >> I do have to admit that I am lost here, and that I cannot think about = what is going wrong. The last download I did try at my old severs has = been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I = did move my infrastructure from bare metal to cloud, thus I cannot test = anymore if my old servers would have shown that performance issue in the = meantime. >>=20 >> Thus any feedback is highly recommended! > Can you try turning off TSO? (`ifconfig $ifname -tso`) >=20 > There have been issues with pf and TSO checksums, which looked a lot = like this (i.e. bad TCP performance). Those problems should be fixed, = but this is easy to test. >=20 I did try it, but without success.=20 This only worked for the external interface, though. Both epairX = interfaces didn't accept that command: ifconfig: -tso: Invalid argument I did mention that I previously tried "sysctl net.inet.tcp.tso=3D0". = That shoukld do the same, right? Thanks and regards, Michael From owner-freebsd-net@freebsd.org Thu Dec 21 21:03:35 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D4EFCE8AA49; Thu, 21 Dec 2017 21:03:35 +0000 (UTC) (envelope-from srs0=bbyf=dr=sigsegv.be=kristof@codepro.be) Received: from venus.codepro.be (venus.codepro.be [IPv6:2a01:4f8:162:1127::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.codepro.be", Issuer "Gandi Standard SSL CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D494789AD; Thu, 21 Dec 2017 21:03:35 +0000 (UTC) (envelope-from srs0=bbyf=dr=sigsegv.be=kristof@codepro.be) Received: from [192.168.228.1] (ptr-8ripyyhtmq100mi0yvj.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:2419:4e02:e137:9435:e3a1:1a8f]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id A78FC37C85; Thu, 21 Dec 2017 22:03:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sigsegv.be; s=mail; t=1513890213; bh=xvo9zMCVIMPojS8zyhAPkTetnm0MPO1558zfbZsWy0s=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=lNNwIGEDS0F1GHY6Bj5lZ9DTjQ7vgyQUNL2ZnugnkGnzXUc8vAHXo6RzgybgZAqZu enje2OtN10mwPmcEdN6QKYriRUSkHi/1FuHX2eFdv8ETu3UomOyQwuhWJu0IDbA/Rs z8eebIPX0dB9cqVCtS9ipiTBYAbt9lxdew8BCqKE= From: "Kristof Provost" To: "Michael Grimm" Cc: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org Subject: Re: performance issue within VNET jail Date: Thu, 21 Dec 2017 22:03:32 +0100 X-Mailer: MailMate (2.0BETAr6102) Message-ID: In-Reply-To: <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:03:35 -0000 On 21 Dec 2017, at 21:50, Michael Grimm wrote: > Kristof Provost wrote: >> >> On 21 Dec 2017, at 21:24, Michael Grimm wrote: > >>> I do have to admit that I am lost here, and that I cannot think >>> about what is going wrong. The last download I did try at my old >>> severs has been some weeks ago. Ever since I did upgrade FreeBSD >>> 11.1-STABLE, and I did move my infrastructure from bare metal to >>> cloud, thus I cannot test anymore if my old servers would have shown >>> that performance issue in the meantime. >>> >>> Thus any feedback is highly recommended! > >> Can you try turning off TSO? (`ifconfig $ifname -tso`) >> >> There have been issues with pf and TSO checksums, which looked a lot >> like this (i.e. bad TCP performance). Those problems should be fixed, >> but this is easy to test. >> > > I did try it, but without success. > Hmm. I’ve got no ideas at the moment. I run a very similar setup (although on CURRENT), and see no performance issues from my jails. Can you test a performance test without pf? Perhaps from the local LAN for example? That should help narrow it down a bit, at least. Regards, Kristof From owner-freebsd-net@freebsd.org Thu Dec 21 21:42:44 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E11BDE8CF93; Thu, 21 Dec 2017 21:42:44 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx2.enfer-du-nord.net (mx2.enfer-du-nord.net [IPv6:2001:41d0:401:2100::5:8a0e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA5847A13F; Thu, 21 Dec 2017 21:42:44 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx2.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2lV844m7z5t; Thu, 21 Dec 2017 22:42:24 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Thu, 21 Dec 2017 22:42:22 +0100 Cc: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> To: Kristof Provost X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:42:45 -0000 Kristof Provost wrote > On 21 Dec 2017, at 21:50, Michael Grimm wrote: >> Kristof Provost wrote: >>> Can you try turning off TSO? (`ifconfig $ifname -tso`) >>>=20 >>> There have been issues with pf and TSO checksums, which looked a lot = like this (i.e. bad TCP performance). Those problems should be fixed, = but this is easy to test. >> I did try it, but without success. > Hmm. I=E2=80=99ve got no ideas at the moment. I run a very similar = setup (although on CURRENT), and see no performance issues from my = jails. > Can you test a performance test without pf? Perhaps from the local LAN = for example? That should help narrow it down a bit, at least. Well I prepared on of my webservers running at hostB/jailX to serve a = sample file for local downloading tests: 1) hostA wget from hostB/jailX sample file: about 30 MB/s 2) hostA/jailY wget from hostB/jailX sample file: about 30 MB/s 3) hostB wget from hostB/jailX sample file: about 190 MB/s 4) hostB/jailY wget from hostB/jailX sample file: about 190 MB/s Hmm. At least tests 3) and 4) omit the pf firewall. Tests 1) qnd 2) = include passing two firewalls, one at each host. BUT: Both hosts are = connected via an IPSec tunnel, and that's esp not tcp. Can anyone draw conclusions from this test?=20 I cannot ;-) Thanks and regards, Michael From owner-freebsd-net@freebsd.org Thu Dec 21 21:49:15 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3545AE8D5BF; Thu, 21 Dec 2017 21:49:15 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BC6837A447; Thu, 21 Dec 2017 21:49:14 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBLLmveh015114 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 21 Dec 2017 22:48:57 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: trashcan@ellael.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBLLmrlS003447 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 22 Dec 2017 04:48:53 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: performance issue within VNET jail To: Michael Grimm , Kristof Provost References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> Cc: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org From: Eugene Grosbein Message-ID: <5A3C2C42.6060904@grosbein.net> Date: Fri, 22 Dec 2017 04:48:50 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:49:15 -0000 22.12.2017 4:42, Michael Grimm wrote: > Well I prepared on of my webservers running at hostB/jailX to serve a sample file for local downloading tests: > > 1) hostA wget from hostB/jailX sample file: about 30 MB/s > 2) hostA/jailY wget from hostB/jailX sample file: about 30 MB/s > 3) hostB wget from hostB/jailX sample file: about 190 MB/s > 4) hostB/jailY wget from hostB/jailX sample file: about 190 MB/s > > Hmm. At least tests 3) and 4) omit the pf firewall. Tests 1) qnd 2) include passing two firewalls, one at each host. BUT: Both hosts are connected via an IPSec tunnel, and that's esp not tcp. > > Can anyone draw conclusions from this test? > I cannot ;-) Make sure and double check that your ESP packets do not get fragmented. From owner-freebsd-net@freebsd.org Thu Dec 21 21:59:49 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 13CE4E8E0B3; Thu, 21 Dec 2017 21:59:49 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BCBAD7ABAB; Thu, 21 Dec 2017 21:59:48 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2lt21D2MzDsY; Thu, 21 Dec 2017 22:59:38 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: <5A3C2C42.6060904@grosbein.net> Date: Thu, 21 Dec 2017 22:59:37 +0100 Cc: Kristof Provost , freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: <5DAD8B80-FE3C-49D2-A645-EE144474D5FE@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <5A3C2C42.6060904@grosbein.net> To: Eugene Grosbein X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 21:59:49 -0000 > On 21. Dec 2017, at 22:48, Eugene Grosbein wrote: >=20 > 22.12.2017 4:42, Michael Grimm wrote: >=20 >> Well I prepared on of my webservers running at hostB/jailX to serve a = sample file for local downloading tests: >>=20 >> 1) hostA wget from hostB/jailX sample file: about 30 MB/s >> 2) hostA/jailY wget from hostB/jailX sample file: about 30 = MB/s >> 3) hostB wget from hostB/jailX sample file: about 190 MB/s >> 4) hostB/jailY wget from hostB/jailX sample file: about 190 = MB/s >>=20 >> Hmm. At least tests 3) and 4) omit the pf firewall. Tests 1) qnd 2) = include passing two firewalls, one at each host. BUT: Both hosts are = connected via an IPSec tunnel, and that's esp not tcp. >>=20 >> Can anyone draw conclusions from this test?=20 >> I cannot ;-) >=20 > Make sure and double check that your ESP packets do not get = fragmented. Hmm, I do not know how to achieve that. May the following tcpdump = excerpts answer your question, or do you want me to look somewhere else? At hostA while downloading from hostB/jailX and "tcpdump -i extIF esp = -vv" 22:52:42.341023 IP (tos 0x0, ttl 64, id 40481, offset 0, flags [none], = proto ESP (50), length 140) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5fe699), length 120 22:52:42.341079 IP (tos 0x0, ttl 53, id 64310, offset 1480, flags = [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 22:52:42.341151 IP (tos 0x0, ttl 64, id 40483, offset 0, flags [none], = proto ESP (50), length 140) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5fe69a), length 120 22:52:42.341169 IP (tos 0x0, ttl 53, id 64312, offset 1480, flags = [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 22:52:42.341238 IP (tos 0x0, ttl 53, id 64314, offset 1480, flags = [none], proto ESP (50), length 100) hostB > hostA: ip-proto-50 At hostB the same dump looks like: 22:52:42.463511 IP (tos 0x0, ttl 53, id 41153, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaa8), length 104 22:52:42.463518 IP (tos 0x0, ttl 53, id 41155, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaa9), length 104 22:52:42.463593 IP (tos 0x0, ttl 53, id 41157, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaaa), length 104 22:52:42.463601 IP (tos 0x0, ttl 53, id 41159, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaab), length 104 22:52:42.463673 IP (tos 0x0, ttl 53, id 41161, offset 0, flags [none], = proto ESP (50), length 124) hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5feaac), length 104 Thanks and regards, Michael >=20 >=20 > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@freebsd.org Thu Dec 21 22:21:06 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 76B50E8F1F1; Thu, 21 Dec 2017 22:21:06 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0CB317B59D; Thu, 21 Dec 2017 22:21:05 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBLMKr5Z015312 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 21 Dec 2017 23:20:54 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: trashcan@ellael.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBLMKooX012573 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Fri, 22 Dec 2017 05:20:50 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: performance issue within VNET jail To: Michael Grimm References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <5A3C2C42.6060904@grosbein.net> <5DAD8B80-FE3C-49D2-A645-EE144474D5FE@ellael.org> Cc: Kristof Provost , freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org From: Eugene Grosbein Message-ID: <5A3C33BF.9050902@grosbein.net> Date: Fri, 22 Dec 2017 05:20:47 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <5DAD8B80-FE3C-49D2-A645-EE144474D5FE@ellael.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 22:21:06 -0000 22.12.2017 4:59, Michael Grimm wrote: >> Make sure and double check that your ESP packets do not get fragmented. > > > Hmm, I do not know how to achieve that. May the following tcpdump excerpts answer your question, or do you want me to look somewhere else? > > At hostA while downloading from hostB/jailX and "tcpdump -i extIF esp -vv" > > 22:52:42.341023 IP (tos 0x0, ttl 64, id 40481, offset 0, flags [none], proto ESP (50), length 140) > hostA > hostB: ESP(spi=0x01d9ec34,seq=0x5fe699), length 120 > 22:52:42.341079 IP (tos 0x0, ttl 53, id 64310, offset 1480, flags [none], proto ESP (50), length 100) > hostB > hostA: ip-proto-50 It shows non-zero offsets, so your ESP packets *are* fragmented. I guess, this is the reason of your problems as fragmented ESP packets are known to cause problems due to different reasons. Simpliest way to avoid such issues is to decrease MTU of IPSEC tunnel and/or TCP MSS so that incapsulated ESP packets do not get fragmented. From owner-freebsd-net@freebsd.org Thu Dec 21 22:35:05 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDF5DE8FF42; Thu, 21 Dec 2017 22:35:05 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx2.enfer-du-nord.net (mx2.enfer-du-nord.net [87.98.149.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9129C7BD4B; Thu, 21 Dec 2017 22:35:04 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx2.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2mfn4J5GzD0; Thu, 21 Dec 2017 23:34:57 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: <5A3C33BF.9050902@grosbein.net> Date: Thu, 21 Dec 2017 23:34:56 +0100 Cc: freebsd-jail@FreeBSD.org, freebsd-net@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <998F52B1-F07C-4A2D-ABB5-3F86D7D4BD09@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <5A3C2C42.6060904@grosbein.net> <5DAD8B80-FE3C-49D2-A645-EE144474D5FE@ellael.org> <5A3C33BF.9050902@grosbein.net> To: Eugene Grosbein X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 22:35:06 -0000 Eugene Grosbein wrote: > 22.12.2017 4:59, Michael Grimm wrote: >>> Make sure and double check that your ESP packets do not get = fragmented. >>=20 >>=20 >> Hmm, I do not know how to achieve that. May the following tcpdump = excerpts answer your question, or do you want me to look somewhere else? >>=20 >> At hostA while downloading from hostB/jailX and "tcpdump -i extIF esp = -vv" >>=20 >> 22:52:42.341023 IP (tos 0x0, ttl 64, id 40481, offset 0, flags = [none], proto ESP (50), length 140) >> hostA > hostB: ESP(spi=3D0x01d9ec34,seq=3D0x5fe699), length 120 >> 22:52:42.341079 IP (tos 0x0, ttl 53, id 64310, offset 1480, flags = [none], proto ESP (50), length 100) >> hostB > hostA: ip-proto-50 >=20 > It shows non-zero offsets, so your ESP packets *are* fragmented. > I guess, this is the reason of your problems as fragmented ESP packets = are known to cause problems > due to different reasons. Simpliest way to avoid such issues is to = decrease MTU of IPSEC tunnel > and/or TCP MSS so that incapsulated ESP packets do not get fragmented. Well, you already helped me out with IPSEC very recently, and I already = did decrease my MTU from 1500 to 1490. That increased my tunnel = performance dramatically, already. Thanks, I will decrease MTU further. BUT: In this thread I did report that I already had decreased MTU for = testing purposes on all involved interfaces down to 1400 to no avail, = and that my performance issue is regarding downloads within VNET jails = using TCP, not ESP. The very same external interfaces do not show a = performance drop if connected via ESP tunnel, but when trying to = download files from the internet, and only when the download is started = within a VNET jail. At the host downloads are only limited by the = bandwidth provided by the hosting company. BUT: It might well be that I did completely misunderstood your reply = instead ;-) Thanks and regards, Michael From owner-freebsd-net@freebsd.org Fri Dec 22 02:13:14 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65A91E9D132 for ; Fri, 22 Dec 2017 02:13:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 52DA42A4B for ; Fri, 22 Dec 2017 02:13:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vBM2DDwE076213 for ; Fri, 22 Dec 2017 02:13:14 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-net@FreeBSD.org Subject: [Bug 211062] [ixv] sr-iov virtual function driver fails to attach Date: Fri, 22 Dec 2017 02:13:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: IntelNetworking, needs-patch X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: ultima@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-net@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 02:13:14 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211062 --- Comment #10 from Richard Gallamore --- (In reply to Piotr Pietruszewski from comment #9) Hello Piotr, Thank you very much for the link. After compiling and installing the driver everything appears to work though I have noticed an errors, i'm not sure if= it is a false positive and also I think the vf is short by one. I have done ba= sic testing (pinging) on the vfs and seems to work fine. Setting hw.ix.num_queues seems to no longer matter so I removed it from /boot/loader.conf. I'm not sure if this was intentional. One error I found = so far: Dec 21 15:50:42 S1 kernel: ix1: CRITICAL: ECC ERROR!! Please Reboot!! Not sure if its a false positive. Seems to happen after the 3rd invoke of iovctl after reboot, but not entirely sure of the trigger. 64 vfs per port should be available, However, vfs <=3D 63 everything appear= ed normal other than the occasional error previously mentioned. vfs >=3D 64 will return error: iovctl: Failed to configure SR-IOV: No space left on device. 64 is supposed to be the max vfs or am I mistaken? or is the first interface (ix1) count as one of the vfs? FreeBSD S1 12.0-CURRENT FreeBSD 12.0-CURRENT #2 r327068: Thu Dec 21 13:00:34 PST 2017 # cat /boot/loader.conf if_ix_load=3D"YES" # cat /etc/iovctl.conf PF { device : ix1; num_vfs : 32; } DEFAULT { passthrough : true; } VF-0 { passthrough : false; } VF-1 { passthrough : false; } It would be nice to get this in head but after seeing r327031[1], I don't s= ee this happening anytime soon. Also want to mention, tested head driver(4.0.0-k) before 3.2.17 and sr-iov = is returning the similar/same errors mentioned in earlier posts on this thread. [1] https://svnweb.freebsd.org/base?view=3Drevision&revision=3D327031 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-net@freebsd.org Fri Dec 22 19:12:11 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D37EE8407E; Fri, 22 Dec 2017 19:12:11 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E81B63F3E; Fri, 22 Dec 2017 19:12:10 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e03:dc01:c8f8:8a2b:f09d:2a5a] (p2003008C2E03DC01C8F88A2BF09D2A5A.dip0.t-ipconnect.de [IPv6:2003:8c:2e03:dc01:c8f8:8a2b:f09d:2a5a]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z3J6B0y89z3DS; Fri, 22 Dec 2017 20:12:02 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Fri, 22 Dec 2017 20:11:59 +0100 Cc: Kristof Provost , Eugene Grosbein Content-Transfer-Encoding: quoted-printable Message-Id: <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> To: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 19:12:11 -0000 Kristof Provost wrote: > I run a very similar setup (although on CURRENT), and see no = performance issues from my jails. In utter despair I did upgrade one server to CURRENT (#327076) today, = but that hasn't been successful :-( Ok, right now I do know: (#) there is *no* performance loss (TCP) when: (-) fetching files from outside through PF/extIF to host (-) fetching files from partner server host via IPSEC tunnel = bound to extIF (ESP) to host (-) fetching files from partner server host via IPSEC tunnel = bound to extIF (ESP) to jail via bridge (-) fetching files from partner server jail via bridge and then = via IPSEC tunnel bound to extIF (ESP) to host (-) fetching files from partner server jail via bridge and then = via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail (#) there is a *dramatic* performance loss (TCP) when: (-) fetching files from outside through PF/extIF via bridge to = jail (#) I did try to tweak the following settings *without* success: (-) sysctl net.inet.tcp.tso=3D0=20 (-) sysctl net.link.bridge.pfil_onlyip=3D0 (-) sysctl net.link.bridge.pfil_bridge=3D0 (-) sysctl net.link.bridge.pfil_member=3D0=20 (-) reducing mtu to 1400 (1490 before) on all interfaces extIF, = bridge, epairXs (-) deactivating "scrub in all" and "scrub out on $extIF all = random-id" in /etc/pf.conf (-) setting "set require-order yes" and "set require-order no" = in /etc/pf.conf [1] [1] I do see more a lot of out-of-order packages within a jail "netstat = -s -p tcp" after those slow downloads, but not after downloads via IPSEC = tunnel from partner host. That leads me to the conclusions: (#) the bridge is not to blame (#) it's either the PF/NATing or something else, right? Thanks for your suggestions so far, but I am lost here. Any ideas? Regards, Michael From owner-freebsd-net@freebsd.org Fri Dec 22 20:16:07 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F2C7E871C5; Fri, 22 Dec 2017 20:16:07 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B57B650B2; Fri, 22 Dec 2017 20:16:06 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBMKFr99024657 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 22 Dec 2017 21:15:53 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: trashcan@ellael.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBMKFgPO076905 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sat, 23 Dec 2017 03:15:42 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: performance issue within VNET jail To: Michael Grimm , freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> Cc: Kristof Provost From: Eugene Grosbein Message-ID: <5A3D67EC.6010907@grosbein.net> Date: Sat, 23 Dec 2017 03:15:40 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 20:16:07 -0000 23.12.2017 2:11, Michael Grimm wrote: > Kristof Provost wrote: > >> I run a very similar setup (although on CURRENT), and see no performance issues from my jails. > > In utter despair I did upgrade one server to CURRENT (#327076) today, but that hasn't been successful :-( > > Ok, right now I do know: > > (#) there is *no* performance loss (TCP) when: > > (-) fetching files from outside through PF/extIF to host > (-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to host > (-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to jail via bridge > (-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) to host > (-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail > > (#) there is a *dramatic* performance loss (TCP) when: > > (-) fetching files from outside through PF/extIF via bridge to jail > > (#) I did try to tweak the following settings *without* success: > > (-) sysctl net.inet.tcp.tso=0 > (-) sysctl net.link.bridge.pfil_onlyip=0 > (-) sysctl net.link.bridge.pfil_bridge=0 > (-) sysctl net.link.bridge.pfil_member=0 > (-) reducing mtu to 1400 (1490 before) on all interfaces extIF, bridge, epairXs > (-) deactivating "scrub in all" and "scrub out on $extIF all random-id" in /etc/pf.conf > (-) setting "set require-order yes" and "set require-order no" in /etc/pf.conf [1] > > [1] I do see more a lot of out-of-order packages within a jail "netstat -s -p tcp" after those slow downloads, but not after downloads via IPSEC tunnel from partner host. > > That leads me to the conclusions: > > (#) the bridge is not to blame > (#) it's either the PF/NATing or something else, right? > > Thanks for your suggestions so far, but I am lost here. Any ideas? It seems to me some kind of bug in the PF. I personally never tried it, I use ipfw and it works just fine. Maybe, you should try to switch to it too, at least for a test. From owner-freebsd-net@freebsd.org Fri Dec 22 20:30:45 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03872E87D9C; Fri, 22 Dec 2017 20:30:45 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx2.enfer-du-nord.net (mx2.enfer-du-nord.net [87.98.149.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BCEFF65759; Fri, 22 Dec 2017 20:30:41 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e03:dc01:c8f8:8a2b:f09d:2a5a] (p2003008C2E03DC01C8F88A2BF09D2A5A.dip0.t-ipconnect.de [IPv6:2003:8c:2e03:dc01:c8f8:8a2b:f09d:2a5a]) by mx2.enfer-du-nord.net (Postfix) with ESMTPSA id 3z3Krs1746z3Dv; Fri, 22 Dec 2017 21:30:37 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: <5A3D67EC.6010907@grosbein.net> Date: Fri, 22 Dec 2017 21:30:35 +0100 Cc: Eugene Grosbein Content-Transfer-Encoding: quoted-printable Message-Id: <53687746-C487-4712-AA52-DE86CE70FDEF@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> <5A3D67EC.6010907@grosbein.net> To: freebsd-net@freebsd.org, freebsd-pf@FreeBSD.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 20:30:45 -0000 Hi =E2=80=94 [ I am including freebsd-pf@FreeBSD.org now and removing = freebsd-jail@FreeBSD.org ] [ Thread starts at = https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html = ] Eugene Grosbein wrote: > Michael Grimm wrote: >> Kristof Provost wrote: >>> I run a very similar setup (although on CURRENT), and see no = performance issues from my jails. >>=20 >> In utter despair I did upgrade one server to CURRENT (#327076) today, = but that hasn't been successful :-( >>=20 >> Ok, right now I do know: >>=20 >> (#) there is *no* performance loss (TCP) when: >>=20 >> (-) fetching files from outside through PF/extIF to host >> (-) fetching files from partner server host via IPSEC tunnel = bound to extIF (ESP) to host >> (-) fetching files from partner server host via IPSEC tunnel = bound to extIF (ESP) to jail via bridge >> (-) fetching files from partner server jail via bridge and then = via IPSEC tunnel bound to extIF (ESP) to host >> (-) fetching files from partner server jail via bridge and then = via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail >>=20 >> (#) there is a *dramatic* performance loss (TCP) when: >>=20 >> (-) fetching files from outside through PF/extIF via bridge to = jail >>=20 >> (#) I did try to tweak the following settings *without* success: >>=20 >> (-) sysctl net.inet.tcp.tso=3D0=20 >> (-) sysctl net.link.bridge.pfil_onlyip=3D0 >> (-) sysctl net.link.bridge.pfil_bridge=3D0 >> (-) sysctl net.link.bridge.pfil_member=3D0=20 >> (-) reducing mtu to 1400 (1490 before) on all interfaces extIF, = bridge, epairXs >> (-) deactivating "scrub in all" and "scrub out on $extIF all = random-id" in /etc/pf.conf >> (-) setting "set require-order yes" and "set require-order no" = in /etc/pf.conf [1] >>=20 >> [1] I do see more a lot of out-of-order packages within a jail = "netstat -s -p tcp" after those slow downloads, but not after downloads = via IPSEC tunnel from partner host. >>=20 >> That leads me to the conclusions: >>=20 >> (#) the bridge is not to blame >> (#) it's either the PF/NATing or something else, right? >>=20 >> Thanks for your suggestions so far, but I am lost here. Any ideas? >=20 > It seems to me some kind of bug in the PF. > I personally never tried it, I use ipfw and it works just fine. Before testing IPFW (which I have never used before) I'd like to ask the = experts in freebsd-pf@FreeBSD.org about possible tests/tweaks regarding = PF. Thanks to all involved so far and regards, Michael From owner-freebsd-net@freebsd.org Sat Dec 23 13:12:03 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2B08EEA56F1; Sat, 23 Dec 2017 13:12:03 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (bird.sbone.de [46.4.1.90]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CBF103AA3; Sat, 23 Dec 2017 13:12:02 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 7072025D3A6F; Sat, 23 Dec 2017 13:11:54 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id A36C7D1F86C; Sat, 23 Dec 2017 13:11:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id o_N6HaD3ToTQ; Sat, 23 Dec 2017 13:11:51 +0000 (UTC) Received: from [192.168.1.224] (unknown [IPv6:fde9:577b:c1a9:f001::2]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id 1949BD1F7F6; Sat, 23 Dec 2017 13:11:49 +0000 (UTC) From: "Bjoern A. Zeeb" To: "Michael Grimm" Cc: freebsd-net@freebsd.org, freebsd-pf@FreeBSD.org Subject: Re: performance issue within VNET jail Date: Sat, 23 Dec 2017 13:11:40 +0000 X-Mailer: MailMate (2.0BETAr6102) Message-ID: In-Reply-To: <53687746-C487-4712-AA52-DE86CE70FDEF@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> <5A3D67EC.6010907@grosbein.net> <53687746-C487-4712-AA52-DE86CE70FDEF@ellael.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Dec 2017 13:12:03 -0000 On 22 Dec 2017, at 20:30, Michael Grimm wrote: > Hi — > > [ I am including freebsd-pf@FreeBSD.org now and removing > freebsd-jail@FreeBSD.org ] > [ Thread starts at > https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html > ] >>> >>> (#) there is a *dramatic* performance loss (TCP) when: >>> >>> (-) fetching files from outside through PF/extIF via bridge to jail … >>> >>> Thanks for your suggestions so far, but I am lost here. Any ideas? >> >> It seems to me some kind of bug in the PF. >> I personally never tried it, I use ipfw and it works just fine. > > Before testing IPFW (which I have never used before) I'd like to ask > the experts in freebsd-pf@FreeBSD.org about possible tests/tweaks > regarding PF. OK, too complicated setups; I am not getting it fully. Can you please just describe the one case that doesn’t work well in all detail and ignore all the others for a moment? (a) what’s the external host interface? (b) pf runs on the base system? (c) you are bridging into a VNET-jail? How exactly? Are you bridging to epairs? (d) where exactly are you NATing? (e) why are you bridging and NATing? That makes little sense to me. Couldn’t you NAT and forward or just bridge? (f) what’s inside the VNET jail? Another pf or anything? (g) out of curiosity, does dmesg on the base system indicate anything? To understand your performance problem better: (1) you are doing a fetch of a rather large file to test from within the VNET jail? Or what are you fetching? Are you using fetch? (2) if you fetch from within the same VNET jail does that perform? (3) if you fetch something to the VNET jail from the base system just going through your internal setup but not leaving the machine, does that still perform? (4) if you fetch something to the VNET jail from the same LAN (if possible to test) does that perform? (5) if you fetch something to the VNET jail from a close by location does that make a difference to something on the other side of the planet? /bz From owner-freebsd-net@freebsd.org Sat Dec 23 14:06:16 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E8A0EA67FC; Sat, 23 Dec 2017 14:06:16 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:302:1100::7:9a96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D59B9644D8; Sat, 23 Dec 2017 14:06:15 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e30:b801:3cc5:4d29:36db:5290] (p2003008C2E30B8013CC54D2936DB5290.dip0.t-ipconnect.de [IPv6:2003:8c:2e30:b801:3cc5:4d29:36db:5290]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z3nGr5lqNz3Cq; Sat, 23 Dec 2017 15:06:12 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: [SOLVED] performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Sat, 23 Dec 2017 15:06:10 +0100 Cc: freebsd-net@freebsd.org, freebsd-pf@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: <5FD6CE98-601B-46B7-B598-83BE5A31200A@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> <5A3D67EC.6010907@grosbein.net> <53687746-C487-4712-AA52-DE86CE70FDEF@ellael.org> To: "Bjoern A. Zeeb" X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Dec 2017 14:06:16 -0000 Bjoern A. Zeeb wrote: >=20 > On 22 Dec 2017, at 20:30, Michael Grimm wrote: >> Hi =E2=80=94 >>=20 >> [ I am including freebsd-pf@FreeBSD.org now and removing = freebsd-jail@FreeBSD.org ] >> [ Thread starts at = https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html = ] >>>>=20 >>>> (#) there is a *dramatic* performance loss (TCP) when: >>>>=20 >>>> (-) fetching files from outside through PF/extIF via bridge to = jail > =E2=80=A6 >>>>=20 >>>> Thanks for your suggestions so far, but I am lost here. Any ideas? >>>=20 >>> It seems to me some kind of bug in the PF. >>> I personally never tried it, I use ipfw and it works just fine. >>=20 >> Before testing IPFW (which I have never used before) I'd like to ask = the experts in freebsd-pf@FreeBSD.org about possible tests/tweaks = regarding PF. >=20 > OK, too complicated setups; I am not getting it fully. ;-) > Can you please just describe the one case that doesn=E2=80=99t work = well in all detail and ignore all the others for a moment? >=20 > (a) what=E2=80=99s the external host interface? vtnet > (b) pf runs on the base system? yes > (c) you are bridging into a VNET-jail? How exactly? Are you bridging = to epairs? yes, I am bridging epairs > (d) where exactly are you NATing? I am NATing IPv4 and IPv6 at the host' PF ffirewall > (e) why are you bridging and NATing? That makes little sense to me. = Couldn=E2=80=99t you NAT and forward or just bridge? hmm, that has been developed by myself over the years. I do "consider" = my jails as jails with their own network stack, like isolated "VM".=20 > (f) what=E2=80=99s inside the VNET jail? Another pf or anything? no more firewall, my jails are merely service jails (dns, mail, web, = =E2=80=A6) > (g) out of curiosity, does dmesg on the base system indicate anything? No. > To understand your performance problem better: >=20 > (1) you are doing a fetch of a rather large file to test from within = the VNET jail? Or what are you fetching? Are you using fetch? yes, I do something like the following with the jail: wget = https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.1/FreeBSD-11.1-REL= EASE-amd64-bootonly.iso -O /dev/null > (2) if you fetch from within the same VNET jail does that perform? > (3) if you fetch something to the VNET jail from the base system just = going through your internal setup but not leaving the machine, does that = still perform? > (4) if you fetch something to the VNET jail from the same LAN (if = possible to test) does that perform? > (5) if you fetch something to the VNET jail from a close by location = does that make a difference to something on the other side of the = planet? I will skip these questions for the time being, because I did solve my = issue 15 minutes before your mail ;-) And I feel sorry for all your now = "wasted" efforts in trying to help me. As I am using vtnet interface in a cloud environment (Public Cloud by = OVH) I did read the vtnet(4) man pages and stumbled about "LOADER = TUNABLES" like: hw.vtnet.lro_disable hw.vtnet.X.lro_disable This tunable disables LRO. The default value is 0. Well, without knowing and understanding the implications of those loader = tunables I did disabled them step by step, and bingo, setting =E2=80=A6 hw.vtnet.lro_disable=3D"1" =E2=80=A6 in /boot/loader.conf" and performance is back from KB/s to = MB/s. I really do not understand what I have done and why it is working and = whether that will have negative implications for my servers. Perhaps = someone of you experts could help me understand it. Because I am leaving in some hours for Xmas vacations, I won't be able = to come back to this issue for some days now.=20 I'd like to thank all of you for your patience and help, and: Merry Christmas and with kind regards, Michael From owner-freebsd-net@freebsd.org Sat Dec 23 14:41:11 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 885F4EA7309; Sat, 23 Dec 2017 14:41:11 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4118C65661; Sat, 23 Dec 2017 14:41:11 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 5092025D3888; Sat, 23 Dec 2017 14:41:08 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 45198D1F86B; Sat, 23 Dec 2017 14:41:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id kzLfhmU8yNcl; Sat, 23 Dec 2017 14:41:06 +0000 (UTC) Received: from [10.248.105.126] (unknown [IPv6:fde9:577b:c1a9:f001::2]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id BBADFD1F7F6; Sat, 23 Dec 2017 14:41:05 +0000 (UTC) From: "Bjoern A. Zeeb" To: "Michael Grimm" Cc: freebsd-net@freebsd.org, freebsd-pf@FreeBSD.org Subject: Re: [SOLVED] performance issue within VNET jail Date: Sat, 23 Dec 2017 14:41:11 +0000 X-Mailer: MailMate (2.0BETAr6102) Message-ID: In-Reply-To: <5FD6CE98-601B-46B7-B598-83BE5A31200A@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> <5A3D67EC.6010907@grosbein.net> <53687746-C487-4712-AA52-DE86CE70FDEF@ellael.org> <5FD6CE98-601B-46B7-B598-83BE5A31200A@ellael.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Dec 2017 14:41:11 -0000 On 23 Dec 2017, at 14:06, Michael Grimm wrote: > I will skip these questions for the time being, because I did solve my > issue 15 minutes before your mail ;-) And I feel sorry for all your > now "wasted" efforts in trying to help me. That’s OK. You solved the issue; that’s what’s important! > Because I am leaving in some hours for Xmas vacations, I won't be able > to come back to this issue for some days now. > > I'd like to thank all of you for your patience and help, and: > > Merry Christmas and with kind regards, And to you! /bz