From owner-freebsd-fs@freebsd.org Mon Oct 29 03:14:09 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4825C10EDF20; Mon, 29 Oct 2018 03:14:09 +0000 (UTC) (envelope-from avv314@gmail.com) Received: from mail-lf1-x136.google.com (mail-lf1-x136.google.com [IPv6:2a00:1450:4864:20::136]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A59C57B45F; Mon, 29 Oct 2018 03:14:08 +0000 (UTC) (envelope-from avv314@gmail.com) Received: by mail-lf1-x136.google.com with SMTP id c24-v6so4867904lfi.12; Sun, 28 Oct 2018 20:14:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=ggJzXfVQYDNIJENuH0D/T6zuBu68RltBSOiZnJETWm8=; b=Iy+aGVQfIi6TqC92jmsn79DhIHCWXeRYcjdSO1nx3Pp0kfNQ0IG6mp40+JuTyRMAhk SEGL6hh8WZ8WY9KK8CgjCq5B2XsSTs7Pre48GQBkzSjePY9guQGF++nkiZIyUuwi/G8U LzckYaxwekKYEtoUIq0zf5vLBi69Kqli4ZeMBouV+/hx0grH41f24KusbLEojV1jC3St XG8CUAmgZTPnNFHYLUNs5uyIW0Jb7b+ouQQDqpcUEBwURLL5DnVv9wE/vKuTuZ8TEhmT o/R4QpHGWsfQM9I7AM+gw/1wFQrRPvz7h4KCqwI+76Pzcb1o8Pdg3FMGOt/uMxWELpRT RD8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=ggJzXfVQYDNIJENuH0D/T6zuBu68RltBSOiZnJETWm8=; b=ksam3zqF8dYVG0Mf0Lg2FvIITXR9jVenfQJdbbxEz9VnD7jO0viNQ8HlV2emhE27y6 Jzlu7WGM6rrebytG3h5PAwn4ScQHffHBNFU1ylbzVAgimOXYZE5XMlbfvnvBKVitGqm7 5jlJVrT6Q+vlvNkpfu3Us8iU3oJR/usZ5jYYoz8IyIK5/rSSKte6BsDkRVtOb+hKaWaT i4k7TAisimYyddXTL9Zx8q/RY5rd1yeBN7TRb8tPmlZdQ8OIYlJkcxAlywE73UMFie6I K3vi4mnR+kodWjX8w7GSB0UGvikn5gyaeQA/kZ+Vp3q6gCWL/xPHdRZsk9HouFQ+bOLi UUAw== X-Gm-Message-State: AGRZ1gIX1svJBh+U0P9wyrnq8bfbWDyzsEP1Rp6a01F/8mGCU3a2FO7E IN6fxKjOYraCIdUYkXC2Q8ZoVotcsxr/n+W56XLd7ZlZ X-Google-Smtp-Source: AJdET5d6t3DptH6g1eM5Z0kMzKQh2b8B5O0H3UHDoAhIEdxIQPmkKQajYQYvj3ztcV5F7xPTg/tYDWeqJbKdtxoCDY4= X-Received: by 2002:a19:d824:: with SMTP id p36-v6mr6836298lfg.23.1540782846693; Sun, 28 Oct 2018 20:14:06 -0700 (PDT) MIME-Version: 1.0 From: Andrew Vylegzhanin Date: Mon, 29 Oct 2018 06:13:55 +0300 Message-ID: Subject: NFS + Infiniband problem To: freebsd-infiniband@freebsd.org, freebsd-fs@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 03:14:09 -0000 Hello everyone, I have a several FreeBSD machines connected via Infiniband netwok ( FDR switch Mellanox SW3036 + ConnectX-3 VPI cards ). One of them is a NAS-server with multiply ZFS pools. All kernels (11.2-RELEASE on clients and 12.0-BETA1 (11.2 also tried) on server) are with infiniband connected mode (option IPOIB_CM, option SDM) and world with OFED stack support. (WITH_OFED='yes'). File transfers via FTP or SSH between server and clients works almost flawless ( ~ 12 Gbit/s ). But when I try to copy in/out some significant data via NFS share mounted on clients, NFS i/o hangs at all or got extremely slow (couple kB/s) transfer speed after uncertain amount of copied data. For example, on the one node I can copy 1GB file, and after NFS hang on file with size 30 kb. Some details: [root@node4 ~]# mount_nfs -o wsize=30000 -o proto=tcp 10.0.2.1:/zdata2 /mnt [root@node4 ~]# dd if=/dev/zero of=/mnt/N1 bs=1m count=1024 Ctrl-T for "hang" dd load: 0.01 cmd: dd 1061 [bo_wwait] 70.95r 0.00u 0.00s 0% 2112k load: 0.01 cmd: dd 1061 [bo_wwait] 72.89r 0.00u 0.00s 0% 2112k for "slow" dd load: 0.00 cmd: dd 2254 [nfsaio] 224.18r 0.00u 0.13s 0% 3132k load: 0.00 cmd: dd 2254 [nfsaio] 225.94r 0.00u 0.13s 0% 3132k I've tried mount with different wsize option with same result. Any help would be greatly appreciated. -- Andrew