From owner-freebsd-questions@freebsd.org Fri May 4 06:09:47 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA219FC8F0C for ; Fri, 4 May 2018 06:09:47 +0000 (UTC) (envelope-from Anand.Khoje@cavium.com) Received: from NAM03-CO1-obe.outbound.protection.outlook.com (mail-co1nam03on0082.outbound.protection.outlook.com [104.47.40.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "Microsoft IT TLS CA 4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0272F7AB37 for ; Fri, 4 May 2018 06:09:46 +0000 (UTC) (envelope-from Anand.Khoje@cavium.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=HVwkOIvaj32HL3qpaaTlksYlBTO94HALS4ihjFMvfK4=; b=gLHLolBt+/F9UPofFj+tzwaZOch2TnZiOP5QmI9PcnAYI+XiVNwOAwWj3+von8phBwMf7HjboG7cyMF5U7tejFGZeQxcmRcA+0nw2BAGrJZMB6YrCS0WLl4UJay30rA4doRUvn/Q/RTQHV1lcNvmD7eLF19VVLNBv5XiQYAXzvE= Received: from BYAPR07MB4872.namprd07.prod.outlook.com (52.135.205.147) by BYAPR07MB4567.namprd07.prod.outlook.com (52.135.226.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.715.22; Fri, 4 May 2018 06:09:44 +0000 Received: from BYAPR07MB4872.namprd07.prod.outlook.com ([fe80::3932:37e9:3d07:ba1c]) by BYAPR07MB4872.namprd07.prod.outlook.com ([fe80::3932:37e9:3d07:ba1c%2]) with mapi id 15.20.0715.024; Fri, 4 May 2018 06:09:44 +0000 From: "Khoje, Anand" To: "freebsd-questions@freebsd.org" Subject: Krping server port not getting released after completion of krping session Thread-Topic: Krping server port not getting released after completion of krping session Thread-Index: AdPjbNBq0zCey0xeQJeFkLnXYRLMyg== Date: Fri, 4 May 2018 06:09:44 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=Anand.Khoje@cavium.com; x-originating-ip: [103.9.74.64] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; BYAPR07MB4567; 7:mlUuJb7QWGpmyrvVWWaArORXrYyBTdJM5IlGukVZCJrLYMvBFThJNJF6/J8hHcpUqm/aMD7rktL6han8VIh/OzFiR36G88ugKXIpePnXbh+nV24mdK0eqWU6C9DLbUprfpBMnn0IFwj8ESr21kqnkMpeEkAYEJ3ktHWPjzvop3kCy5/bZ+t4DbiAUYZIa8FjMcxWk0VgtOfB0QWH3KkfE9IvSiT5NsIEVOyTqEKXMnNpwN9PpVfkD0jLN8q4O/HV x-ms-exchange-antispam-srfa-diagnostics: SOS; x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:BYAPR07MB4567; x-ms-traffictypediagnostic: BYAPR07MB4567: x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(158342451672863)(21748063052155)(21532816269658); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3231254)(944501410)(52105095)(3002001)(93006095)(93001095)(10201501046)(6041310)(20161123564045)(20161123562045)(20161123558120)(20161123560045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:BYAPR07MB4567; BCL:0; PCL:0; RULEID:; SRVR:BYAPR07MB4567; x-forefront-prvs: 06628F7CA4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(366004)(396003)(376002)(346002)(39380400002)(39860400002)(199004)(189003)(5630700001)(6506007)(106356001)(25786009)(72206003)(8936002)(68736007)(105586002)(478600001)(33656002)(2906002)(3280700002)(9686003)(6916009)(99286004)(2900100001)(6306002)(2351001)(476003)(5250100002)(54896002)(26005)(6436002)(790700001)(81156014)(486006)(8676002)(86362001)(66066001)(3846002)(74316002)(14454004)(7736002)(102836004)(2501003)(81166006)(7696005)(53936002)(5640700003)(55016002)(97736004)(3660700001)(9326002)(316002)(186003)(6116002)(5660300001); DIR:OUT; SFP:1101; SCL:1; SRVR:BYAPR07MB4567; H:BYAPR07MB4872.namprd07.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: OHL/HkFiINd7EnieLEDl2s3Y/v5oiUlTVG2GIpSTYyXgYXhs0SD8SO1kzGQiUH9/CFlybHqxjHvlg/lOka8CcTgYenezUjMzuacy0CiFuD1mCIhUyrllarGhYLPookD9snNSQSlnJf9NFTRURhqTT03N2PCUvGhHOhFi5sVdkedFz0hpkkqXeo9PN9TmRfyi spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 98c72872-3178-4215-7725-08d5b185a1d1 X-OriginatorOrg: cavium.com X-MS-Exchange-CrossTenant-Network-Message-Id: 98c72872-3178-4215-7725-08d5b185a1d1 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 May 2018 06:09:44.3137 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR07MB4567 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 May 2018 06:09:48 -0000 Hi, I am currently working on RoCE v1 support for Cavium BigBear adapters. Alth= ough, RoCE traffic seems to be working now, I have noticed that once we est= ablish krping server at one end, using command : echo "server,addr=3D192.168.11.11,port=3D9991,verbose" > /dev/krping and run krping client from the peer system, using command : echo "client,count=3D10,addr=3D192.168.11.11,port=3D9991,verbose" > /dev/kr= ping The session ends after sending 10 RDMA packets. Here are the logs : server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghi= jklmnopqr server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghij= klmnopqrs server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghijk= lmnopqrst server ping data: rdma-ping-3: DEFGHIJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghijkl= mnopqrstu server ping data: rdma-ping-4: EFGHIJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghijklm= nopqrstuv server ping data: rdma-ping-5: FGHIJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghijklmn= opqrstuvw server ping data: rdma-ping-6: GHIJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghijklmno= pqrstuvwx server ping data: rdma-ping-7: HIJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghijklmnop= qrstuvwxy server ping data: rdma-ping-8: IJKLMNOPQRSTUVWXYZ[\134]^_`abcdefghijklmnopq= rstuvwxyz server ping data: rdma-ping-9: JKLMNOPQRSTUVWXYZ[\134]^_`abcdefghijklmnopqr= stuvwxyzA DISCONNECT EVENT... wait for RDMA_READ_ADV state 10 cq completion in ERROR state But, if we attempt to re-establish the krping server in the same system wit= h the same port number (i.e. 9991), we get an error saying : rdma_bind_addr error -48 (address already in use?) Meanwhile, I checked with Mellanox Connect X3 adapters, I see the same issu= e. My question is, is this behavior normal? Or is there some issue with my set= up? Is it due to some ibcore state machine being stuck in some state waitin= g for an event which has not been taken care of by the driver? P.S: Here is the output of netstat when a krping server is established : # netstat Active Internet connections Proto Recv-Q Send-Q Local Address Foreign Address = (state) tcp4 0 0 192.168.11.11.9991 *.* = CLOSED This remains the same throughout, even after completion of krping session. = I cannot see this information on the krping client system though. Thanks, Anand A. K.