From owner-freebsd-infiniband@freebsd.org Wed Mar 24 04:58:25 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2B30C5A921E for ; Wed, 24 Mar 2021 04:58:25 +0000 (UTC) (envelope-from matsuo.hiroshi.39@gmail.com) Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F4www1Ypmz3GCj for ; Wed, 24 Mar 2021 04:58:24 +0000 (UTC) (envelope-from matsuo.hiroshi.39@gmail.com) Received: by mail-ed1-x52e.google.com with SMTP id e7so26035828edu.10 for ; Tue, 23 Mar 2021 21:58:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=SXrTxv7V5vIq6tKyVCjpn/ZUayJljIqDvZINDDn6zg4=; b=AR/6f6u9wLwBHnRYhMya49pO92Vbi+dmsV9cLNDOqLwUo2BkJT/yXGC8jnocTucWCs V8abI08C+p3TKChQ5guqM1DR+RVYCXEoshwZTUtUh3P+IIG46Kasjhq6FxhgphorOW4N x5V/dGW+DxvUCYKtBZXtoByLatNge09bElML4pfEtp6/6sERRAFDQ5bJQlF2KM6ZywYW cvT6jtcHEB0E5Tg8LRecKF2JLoUrX+3IAUSVRNYbyrVfbUfy4grvaqhM6Sb23K5ufvjc OigUOIk4PsdtEvV7gHLCF5ZxqEwpf6VoSRsqVTTGCBdvXXTNnsREpe+nFzC9u+LwYpmi LYxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=SXrTxv7V5vIq6tKyVCjpn/ZUayJljIqDvZINDDn6zg4=; b=ao0KvgdYz5M2uxyyFPPC5DKOGXcXLwJtiAHqyLI6SeGPCxpnSthQRlyiUIUxbHKWeH 6VKUwr6ifA8lowzVtI/3TvuFp8EsT39gPR8TykgxCzE5aozVTSejsfdgnCNVxMNJmvwj 4vT+9q8Lo2lp1CaGOy7LP7HQS0JDtq12EgX12f0PZSYB7WHXxaZ/ubw0pApQ2gJ5BwHq pcZZ3mn+0tgTQjRRF3Fy2vbeNLLczD1FclUP8TG7UXFFANLH+2qaXHCsjGUMcdkG4fcL LVtuZx/x8UjYiyN02nRIPvWXlT9/BQP8quGfexTCO0INEy2sfgcX18uAav+gQ+hn985d L/hA== X-Gm-Message-State: AOAM531arABrnW8BaXa74STgLdvG1fjoSePMHn5MtNc1RPeCKxIXZXmM Fq2OHhANX1t5mtFJMyKMefYop7oKCRPXzTdDMGYhlbtMxXrugQ== X-Google-Smtp-Source: ABdhPJxsPjucyCQs1VACPL3MW9bDGDiAHH5Hf7XwiiwKWsi0cfuGnPgWEHmM1C99TAhWD49rZQghD9PaxwXBGYTJMsY= X-Received: by 2002:aa7:ce16:: with SMTP id d22mr1355902edv.95.1616561902817; Tue, 23 Mar 2021 21:58:22 -0700 (PDT) MIME-Version: 1.0 From: hiroshi matsuo Date: Wed, 24 Mar 2021 13:57:47 +0900 Message-ID: Subject: Data corruption via IPoIB in connected mode between Linux and FreeBSD To: freebsd-infiniband@freebsd.org X-Rspamd-Queue-Id: 4F4www1Ypmz3GCj X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=AR/6f6u9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of matsuohiroshi39@gmail.com designates 2a00:1450:4864:20::52e as permitted sender) smtp.mailfrom=matsuohiroshi39@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::52e:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; SH_EMAIL_DBL_DONT_QUERY_IPS(0.00)[10.0.1.1:email]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::52e:from:127.0.2.255]; DBL_PROHIBIT(0.00)[10.0.1.1:email]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52e:from]; HTTP_TO_IP(1.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-Mailman-Approved-At: Wed, 24 Mar 2021 06:24:49 +0000 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2021 04:58:25 -0000 Dear, I'm trying IPoIB between Linux and FreeBSD with Mellanox ConnectX-2 cards. Now I have a strange problem above my knowledge. CentOS-7: IP address=1.0.1.2/24 (attached to ipoib device), transport mode=connected MTU=65520 (following RedHat document) FreeBSD-12.2: IP address=1.0.1.1/24 (attached to ipoib device), transport mode=connected (I built with IPOIB_CM options) MTU=4092 (default? I want this set 65520 to be the same as CentOS, but I can not. mlx4_core0: 65520 is invalid IBTA mtu dmesg shows. Why?) FreeBSD box has a 2TB ZFS pool and there are about 4,000,000 files in it. A few days ago I copied all files from FreeBSD to CentOS by rsync like this: centos$ rsync -av -e ssh matsuo@10.0.1.1:/tank/data/ ~/data At one time I found a corrupted file accidentally, however rsync finished with no error message. I have looked into all files and compared between copies and originals. At last I understand that: 1. There are 24 corrupted files (MD5 value is different from original) i.e. 0.0006% failure, 99.9994% success 2. Every corrupted file has just one byte which is different from original and the position of the error byte seems random. So not a burst error. I doubt whether CM is established but I don't know the way to inspect it deeply. Please point out to me what is the root cause what is wrong about my setup document worth reading first and so on. In addition I had iperf tests. 16Gbps (in CentOS-CentOS case) 4Gbps (in CentOS-FreeBSD case) So I think My FreeBSD server does not work properly and something wrong. Thank you. Hiroshi Matsuo From owner-freebsd-infiniband@freebsd.org Wed Mar 24 07:53:49 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 563215AD7D5 for ; Wed, 24 Mar 2021 07:53:49 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F50qJ0FCBz3Q8m for ; Wed, 24 Mar 2021 07:53:47 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2020.home.selasky.org (unknown [178.17.145.105]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 9F51126007C; Wed, 24 Mar 2021 08:53:40 +0100 (CET) Subject: Re: Data corruption via IPoIB in connected mode between Linux and FreeBSD To: hiroshi matsuo , freebsd-infiniband@freebsd.org References: From: Hans Petter Selasky Message-ID: <68a1d236-45d9-5343-5ad6-9fee877f5ff1@selasky.org> Date: Wed, 24 Mar 2021 08:53:23 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4F50qJ0FCBz3Q8m X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hps@selasky.org designates 88.99.82.50 as permitted sender) smtp.mailfrom=hps@selasky.org X-Spamd-Result: default: False [-1.30 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.turbocat.net]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[gmail.com,freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[88.99.82.50:from]; ASN(0.00)[asn:24940, ipnet:88.99.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; SH_EMAIL_DBL_DONT_QUERY_IPS(0.00)[10.0.1.1:email]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[selasky.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; SPAMHAUS_ZRD(0.00)[88.99.82.50:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DBL_PROHIBIT(0.00)[10.0.1.1:email]; R_DKIM_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2021 07:53:49 -0000 Hi Hiroshi, On 3/24/21 5:57 AM, hiroshi matsuo wrote: > Dear, > > I'm trying IPoIB between Linux and FreeBSD with Mellanox ConnectX-2 cards. > Now I have a strange problem above my knowledge. > > CentOS-7: > IP address=1.0.1.2/24 (attached to ipoib device), > transport mode=connected > MTU=65520 (following RedHat document) > > FreeBSD-12.2: > IP address=1.0.1.1/24 (attached to ipoib device), > transport mode=connected (I built with IPOIB_CM options) > MTU=4092 (default? I want this set 65520 to be the same as CentOS, but I > can not. > mlx4_core0: 65520 > is invalid IBTA mtu > dmesg shows. Why?) Did you apply my patch to FreeBSD 12.2 for connected mode? Try to subtract the size of the infiniband address, 20 bytes. Try setting the MTU to 9000 instead. Using such a large MTU with FreeBSD doesn't make sense. Are you using an infiniband router between? If yes, has this been configured to handle this big MTU? > > FreeBSD box has a 2TB ZFS pool and there are about 4,000,000 files in it. > A few days ago I copied all files from FreeBSD to CentOS by rsync > like this: > centos$ rsync -av -e ssh matsuo@10.0.1.1:/tank/data/ ~/data > > At one time I found a corrupted file accidentally, however rsync finished > with no error message. > I have looked into all files and compared between copies and originals. At > last > I understand that: > 1. There are 24 corrupted files (MD5 value is different from original) > i.e. 0.0006% failure, 99.9994% success > 2. Every corrupted file has just one byte which is different from original > and the position of the error byte seems random. So not a burst error. > > I doubt whether CM is established but I don't know the way to inspect it > deeply. If you have IPOIB_CM set, you should be good. > > Please point out to me > what is the root cause > what is wrong about my setup > document worth reading first > and so on. Like said, there is a bug in IPoIB CM mode, that is not fixed unless you apply the patch I sent, which Mellanox will upstream later. > > In addition I had iperf tests. > 16Gbps (in CentOS-CentOS case) > 4Gbps (in CentOS-FreeBSD case) > So I think My FreeBSD server does not work properly and something wrong. --HPS From owner-freebsd-infiniband@freebsd.org Wed Mar 24 14:22:22 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C9CAE5B73F9 for ; Wed, 24 Mar 2021 14:22:22 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F59Rd5x1tz4YBm for ; Wed, 24 Mar 2021 14:22:21 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: by mail-qk1-x72b.google.com with SMTP id q26so12885234qkm.6 for ; Wed, 24 Mar 2021 07:22:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-transfer-encoding:content-language; bh=k/r7foV0GRj81tJa+qa22ZkJQI/vVq1/8DCbP2Y3Mhk=; b=nLPvQdaGR9uyPF5yeeMa6Tw6VUEVAzKxJvUoUdf9DjZc7i+LwUaikY2SD+ipy1TeSF kZ1w4MoV/hbNB/a5ZMlnDbyWFxQTJMA47G7ns71TdoFRRORdoIjQaJ4h6qZ5wpNGEWkQ /sbW7NJ4phvFo2oxcNNvQ8FOlHYSKwjO2XwWcC4z0uHmpBJ/wUhUnjfSZXldcJdRdyFf 5nL6uXnkpPi1aoFd34wrXmKTy17MDNHr7r9h4mHVR7u4qF++VPmdospmTB4enebotV3V T429xQO/SiVZTJg2Y5empU5Evnokv+i6Sj8b6a/S/mz6nvE/spP5ILMhLxDD2Yb/tOJt 8thA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=k/r7foV0GRj81tJa+qa22ZkJQI/vVq1/8DCbP2Y3Mhk=; b=kpQAXc+ikTwRjAf4evDK2b2WligLL65uRkQpDMjEsV5DjOTufS1IQ8ncrGnp+WV0jV rr1LY5r08RhKdMQBinjyvXGGb0CicMxFg0/UwyCl8Z6PSGFbbUD8dPP1RwCs6cLq5+vq 3hw3SymwiOCdBHjMY+TkGKwcBUy8tAyrlA9ljoHVJ2IogeQnlj2IcuJ45JIlT0yYQLAT PmboXsHjBHaa9wQt4AntwBUJE1okzLVcOY451SjrD2k09SvT9lrUNie1rWZvtgeEiivd vqqJ+LDhvyKwOu0no6FciyB0aKFG5jU2zGUQX+4SPUh3ty7v26FLlmi3pCtcK/SA9+FS M50Q== X-Gm-Message-State: AOAM532S5TbeLhojSzjA7F3j+B8v+Gj/yibfJOHw7McbpGvk2In07Zog AGKaNDm6uJ3Fh2jUX+kSyGhV+2LZGd0= X-Google-Smtp-Source: ABdhPJyVYfaD9Vs6niR+aJiWuAhBDQOxdsePR6NxmGyLpKiHNHhGVP+u1X/43XdyOuzPkmpkFMoHaQ== X-Received: by 2002:a05:620a:641:: with SMTP id a1mr3182836qka.257.1616595740699; Wed, 24 Mar 2021 07:22:20 -0700 (PDT) Received: from coral.acadix.biz (2603-6000-a401-3a00-0223-24ff-fe37-c4d7.res6.spectrum.com. [2603:6000:a401:3a00:223:24ff:fe37:c4d7]) by smtp.gmail.com with ESMTPSA id i17sm1546982qtr.33.2021.03.24.07.22.19 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 24 Mar 2021 07:22:20 -0700 (PDT) Subject: Re: Data corruption via IPoIB in connected mode between Linux and FreeBSD To: freebsd-infiniband@freebsd.org References: <68a1d236-45d9-5343-5ad6-9fee877f5ff1@selasky.org> From: Jason Bacon Message-ID: <64c041f8-4e75-7e61-6689-c37fc72235f4@gmail.com> Date: Wed, 24 Mar 2021 09:21:05 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <68a1d236-45d9-5343-5ad6-9fee877f5ff1@selasky.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 4F59Rd5x1tz4YBm X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=nLPvQdaG; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::72b as permitted sender) smtp.mailfrom=bacon4000@gmail.com X-Spamd-Result: default: False [-0.23 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::72b:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::72b:from:127.0.2.255]; NEURAL_SPAM_SHORT(0.77)[0.773]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72b:from]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2021 14:22:22 -0000 On 3/24/21 2:53 AM, Hans Petter Selasky wrote: > Try setting the MTU to 9000 instead. Using such a large MTU with > FreeBSD doesn't make sense. In my experience a while back, large MTUs caused temporary hangups. 64k was extremely unstable, 32k somewhat unstable, and 16k seemed OK. -- Earth is a beta site. From owner-freebsd-infiniband@freebsd.org Wed Mar 24 18:48:01 2021 Return-Path: Delivered-To: freebsd-infiniband@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C47FE5BCF85 for ; Wed, 24 Mar 2021 18:48:01 +0000 (UTC) (envelope-from matsuo.hiroshi.39@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F5HL90R5Dz4qwF for ; Wed, 24 Mar 2021 18:48:00 +0000 (UTC) (envelope-from matsuo.hiroshi.39@gmail.com) Received: by mail-ej1-x636.google.com with SMTP id kt15so25138987ejb.12 for ; Wed, 24 Mar 2021 11:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=DpyppNYAm4DEAx2Q4nu6c3bH2hvKQPvS+D/86/SmPtw=; b=CuEZWjLrjOzd3ye6JoKT37I3Zmik8lz+qvAIkSv6Wy2gjf6JIfq0pL7zukY8UTRP2T EM0XvKdl0oMro88ZgCZcd4Ea5ydmvdAJemlhIXsMViBEHTwz1PKQMTzz3wq67h/laML8 l6JavFW6qB/+ErNOMtJdgDQ4kenowDWnVEercDEdV1NxZRcKty5ddsm2x88Fubh/O2xA fGx993tMrDA1qITzw+Cd5UQIhR6fDtC7C5nGcfl3HF2Y0rccIQZGywPzBjjNHdu0+xWs 2h2K9Jmzbc1x35Vziq+Vt8dXVHNBSSvWlWTTe7Wv6mCJBwMtHZ3S+pXEy82DYHB1zIJc exJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=DpyppNYAm4DEAx2Q4nu6c3bH2hvKQPvS+D/86/SmPtw=; b=GaXysN1PDWoYbW3f9+mOgnvsQMUJogmAn4uq/jspoeiWjAwldGEd0MSqe42evovR5h 1UN0ew7Wi3SUCSbHiTAXkouyLtGAmjKbJ4xzVFJoVCkJYyXU4L7XvVJvofXPdC/DzguE HycF1p2S9wmiqHnDDcrVPBtA79ie+by1BO3dj2YWuNfbpifEs6Tb6pzOL7gJ0VakGovp f4QnXj04as7m54rf0aRltFkHgoHlIhvUcBpzGUlAxU3E2et9dk8LRGLyXD7FNeshcpAL 9BQ2Bg69t9916keXrV/h1OJUeXXDNvg6LPx+EsPUU4+ypd2McvPzjl40Jkk2oXICg3za 568Q== X-Gm-Message-State: AOAM530F7RezRIKOvxrai1iHe8nQVk8MUeFj7Q4c/qu775gpj0/4TP6U ySbHUfvjuehgTbzoU6haEFZ5GEOIsH9mVOUdlFwSR2OuZcC1bg== X-Google-Smtp-Source: ABdhPJy4d5Ir+c1dL+KXDlm3LZxL0m6/6BZD26DyVqRQYksO2GOt8xwiSIqRPxRCgZCS9IaOAVc6Ax9qAPAIa5Ja8IA= X-Received: by 2002:a17:906:4705:: with SMTP id y5mr5397580ejq.119.1616611679159; Wed, 24 Mar 2021 11:47:59 -0700 (PDT) MIME-Version: 1.0 From: hiroshi matsuo Date: Thu, 25 Mar 2021 03:47:23 +0900 Message-ID: Subject: Re: Data corruption via IPoIB in connected mode between Linux and FreeBSD To: freebsd-infiniband@freebsd.org X-Rspamd-Queue-Id: 4F5HL90R5Dz4qwF X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=CuEZWjLr; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of matsuohiroshi39@gmail.com designates 2a00:1450:4864:20::636 as permitted sender) smtp.mailfrom=matsuohiroshi39@gmail.com X-Spamd-Result: default: False [-0.36 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.36)[-0.358]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::636:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; FAKE_REPLY(1.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-infiniband@freebsd.org]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::636:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::636:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-infiniband] X-Mailman-Approved-At: Wed, 24 Mar 2021 21:32:09 +0000 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-infiniband@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Infiniband on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Mar 2021 18:48:01 -0000 Hi Hans, Thank you, very good information for me. I will try it this weekend. > Did you apply my patch to FreeBSD 12.2 for connected mode? Maybe not. Is it included in FreeBSD-12.2-p4? > Are you using an infiniband router between? No. I connect two servers directly with one cable. Under this condition, I think bigger mtu is simply better as far as the peer (CentOS) permits. Is this wrong?