From owner-freebsd-net@freebsd.org Mon Sep 3 21:06:54 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 488EDFF6ADF for ; Mon, 3 Sep 2018 21:06:54 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BCB9376C8F for ; Mon, 3 Sep 2018 21:06:53 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by mail-wm0-x241.google.com with SMTP id s12-v6so2324660wmc.0 for ; Mon, 03 Sep 2018 14:06:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding:content-language; bh=CFoDGqfyALSK13ClVAs+DCpD8EqgWAuXjdL+K5NbgbM=; b=ipNdSCNxxnjGHD1/KogFtSRITOYoE2YNyQnng1DLQh4/OHyMS6qRzFzVn/Lkj5HvZa EPJfWTRe4HEnc+5+fu1MP8gmzQRYyJgxprH7VtW+v1Ye9OMPRXLq0trkDmkS5iuRMxWs 59FRu4nr8Edfo4XvuygAfPggqChd+iyDHNUDZ8WX4JYBAQqftQvaBTHYEOE7IlwNS63c gjiY++4ajRg9jf8ky4L/MTCd9Hk6AIc2hGDmzG/7RsVqbDm8gqy7pPuukP+yQhqZKufi VUvUBk/jWEUmmpO0E8a4+s/SBqFm1lgkTD2IM8l+Ua0MDB03QKK/vCTK4vEY8YpHVuXm 5E7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=CFoDGqfyALSK13ClVAs+DCpD8EqgWAuXjdL+K5NbgbM=; b=PlTKKTCkNBrcJjx2QvOOkJQfUQSWmvLoqQVz8eh+I6IydhIxuL+tK8tbOni9w1lvRz vx6K1vnWAcy78t5Mn2hFeY7RXeneMvZxZFAHVfyMnwZGaSBKfxvea+Gu34eTx+1oRa/u uQXSvb8Dn7QL8RTyZWK30VKIYzGuUtJa0aH7rQlwRqIwf6sgZ5e+NFfInMecTy2muoFR kFwfWGEE4f/KAsUxxBS24ft32HUFgDtwgwvSSoSDs1lMBbdiXAYXQO6dadfrf4+HDolr eNek/G4mbYSOYpqNAam3XqvhnaT5k9/WcEPQG1SzAD413A3tMB97XvlwmUNgCDkJ/4eX dkXA== X-Gm-Message-State: APzg51D3HIOa3R8GtMcAR8+6623/ig+wloO9UBQFQnwiv5/jmqt5Caxr ZP3/TTeVqFHrd7mv//Um2M+MySvE X-Google-Smtp-Source: ANB0VdYe5YShRXhq4Is5A+0DGQIEsnfGtqytSGFf4y+VVB2D9iUypTGxOAXXfsU/39OxbZW3XS7NTw== X-Received: by 2002:a1c:c283:: with SMTP id s125-v6mr5679859wmf.158.1536008812297; Mon, 03 Sep 2018 14:06:52 -0700 (PDT) Received: from Sting-Ray.optiplex-networks.com (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.gmail.com with ESMTPSA id w3-v6sm13557003wrn.16.2018.09.03.14.06.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Sep 2018 14:06:51 -0700 (PDT) To: freebsd-net@freebsd.org From: Kaya Saman Subject: iSCSI issues after upgrading to 11.2 x64 RELEASE Message-ID: <541494c3-d275-dee2-ff5e-8b276ef8d9d6@gmail.com> Date: Mon, 3 Sep 2018 22:06:50 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US-large X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Sep 2018 21:06:54 -0000 Hi, I'm not sure what's going on but after running an upgrade from 11.1 to 11.2 I'm getting many iscsi errors. It actually gets so severe that it freezes the system. Dmesg is flooded with lines like this: (da25:iscsi2:0:0:1): WRITE(10). CDB: 2a 00 28 76 02 50 00 00 08 00 (da25:iscsi2:0:0:1): CAM status: SCSI Status Error (da25:iscsi2:0:0:1): SCSI status: Check Condition (da25:iscsi2:0:0:1): SCSI sense: UNIT ATTENTION asc:29,7 (I_T nexus loss occurred) (da25:iscsi2:0:0:1): Retrying command (per sense data) (da27:iscsi1:0:0:0): READ(16). CDB: 88 00 00 00 00 04 41 df 82 78 00 00 01 00 00 00 (da27:iscsi1:0:0:0): CAM status: SCSI Status Error (da27:iscsi1:0:0:0): SCSI status: Check Condition (da27:iscsi1:0:0:0): SCSI sense: UNIT ATTENTION asc:29,7 (I_T nexus loss occurred) (da27:iscsi1:0:0:0): Retrying command (per sense data) The upgrade actually turned into a fresh install since my zroot got corrupted after one of my mirrored disks died and I replaced the remaining original with a new one. The system originally was on 11.1 so the 'clean' reinstall went to 11.2. There is also quite high cpu interrupt I'm noticing too at a steady 25% though can go up to 75% with system load averages getting to above 40 from 'top' output. A little bit of digging around I came across this: https://lists.freebsd.org/pipermail/freebsd-net/2017-June/048293.html however, as mentioned below the 'loader' variables cause a complete system lockup. In 11.1 I had these variables set in /boot/loader.conf: #net.isr.numthreads=4 #net.isr.maxthreads=4 #net.isr.bindthreads=1 #net.isr.dispatch=deferred #hw.igb.max_interrupt_rate=64000 #net.inet.tcp.tcbhashsize=32000 #kern.ipc.nmbjumbo9=1280000 #kern.ipc.nmbjumbo16=1280000 #kern.ipc.nmbclusters=2000000 #kern.ipc.nmbjumbop=128000 after going to 11.2 the system will freeze and become totally unusable after a few hours of boot if I enable these values....?? Another thing I noticed in the dmesg a few times was this error: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219866 "ctl_datamove: tag 0x174e97 on (28:34:0) aborted" It has been reported though I am not sure if a fix has been pushed yet? My setup is as follows: routed network (LAN) -> lagg0 (lacp 4gb) -> server -> lagg1 (lacp 2gb) -> iscsi target 1                                                   |-> iscsi target 2 The NIC's are Intel based using igb kernel driver: igb0: flags=8843 metric 0 mtu 9000 options=6403bb The iscsi network is running on a separate isolated vlan which is not routed. Can anyone suggest anything to stop my system from completely locking up and becoming unresponsive? At the moment I'm not sure if switching to 'Stable' or 'Current' branches is a good solution? Regards, Kaya