From owner-svn-src-stable@freebsd.org Fri Nov 27 15:26:20 2015 Return-Path: Delivered-To: svn-src-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CE83A3AD59; Fri, 27 Nov 2015 15:26:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 66B8618B1; Fri, 27 Nov 2015 15:26:20 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id tARFQJus070085; Fri, 27 Nov 2015 15:26:19 GMT (envelope-from mav@FreeBSD.org) Received: (from mav@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id tARFQJGE070081; Fri, 27 Nov 2015 15:26:19 GMT (envelope-from mav@FreeBSD.org) Message-Id: <201511271526.tARFQJGE070081@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mav set sender to mav@FreeBSD.org using -f From: Alexander Motin Date: Fri, 27 Nov 2015 15:26:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-10@freebsd.org Subject: svn commit: r291388 - stable/10/sys/cam/ctl X-SVN-Group: stable-10 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SVN commit messages for all the -stable branches of the src tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Nov 2015 15:26:20 -0000 Author: mav Date: Fri Nov 27 15:26:19 2015 New Revision: 291388 URL: https://svnweb.freebsd.org/changeset/base/291388 Log: MFC r290670: Modify target port groups logic in CTL. - Introduce "ha_shared" port option, which being set to "on" moves the port into separate port group, shared between HA nodes. This allows to better handle cases when iSCSI portals are bound to CARP address that can dynamically move between nodes. Some initiators (at least VMware) don't detect that after iSCSI reconnect they've attached to different SCSI port from different port group, that totally breakes ALUA status parsing. In theory, I believe, it should be enough to have different iSCSI portal group tags on different nodes to make initiators detect this condition, but it seems like VMware ignores those values, and even full LUN retaste forced by UA does not help. - Make CTL report up to three port groups: 1 -- non-HA mode or ports with "ha_shared" option set, 2 -- HA node 1, 3 -- HA node 2. - Report Transitioning state for all port groups when HA interlink is connected, but neither of nodes is primary for the LUN. Modified: stable/10/sys/cam/ctl/ctl.c stable/10/sys/cam/ctl/ctl_frontend.c stable/10/sys/cam/ctl/ctl_frontend.h stable/10/sys/cam/ctl/ctl_private.h Directory Properties: stable/10/ (props changed) Modified: stable/10/sys/cam/ctl/ctl.c ============================================================================== --- stable/10/sys/cam/ctl/ctl.c Fri Nov 27 15:19:36 2015 (r291387) +++ stable/10/sys/cam/ctl/ctl.c Fri Nov 27 15:26:19 2015 (r291388) @@ -1821,13 +1821,13 @@ ctl_init(void) SYSCTL_ADD_INT(&softc->sysctl_ctx, SYSCTL_CHILDREN(softc->sysctl_tree), OID_AUTO, "ha_id", CTLFLAG_RDTUN, &softc->ha_id, 0, "HA head ID (0 - no HA)"); - if (softc->ha_id == 0 || softc->ha_id > NUM_TARGET_PORT_GROUPS) { + if (softc->ha_id == 0 || softc->ha_id > NUM_HA_SHELVES) { softc->flags |= CTL_FLAG_ACTIVE_SHELF; softc->is_single = 1; softc->port_cnt = CTL_MAX_PORTS; softc->port_min = 0; } else { - softc->port_cnt = CTL_MAX_PORTS / NUM_TARGET_PORT_GROUPS; + softc->port_cnt = CTL_MAX_PORTS / NUM_HA_SHELVES; softc->port_min = (softc->ha_id - 1) * softc->port_cnt; } softc->port_max = softc->port_min + softc->port_cnt; @@ -7139,8 +7139,8 @@ ctl_report_tagret_port_groups(struct ctl { struct scsi_maintenance_in *cdb; int retval; - int alloc_len, ext, total_len = 0, g, pc, pg, gs, os; - int num_target_port_groups, num_target_ports; + int alloc_len, ext, total_len = 0, g, pc, pg, ts, os; + int num_ha_groups, num_target_ports, shared_group; struct ctl_lun *lun; struct ctl_softc *softc; struct ctl_port *port; @@ -7174,11 +7174,8 @@ ctl_report_tagret_port_groups(struct ctl return(retval); } - if (softc->is_single) - num_target_port_groups = 1; - else - num_target_port_groups = NUM_TARGET_PORT_GROUPS; num_target_ports = 0; + shared_group = (softc->is_single != 0); mtx_lock(&softc->ctl_lock); STAILQ_FOREACH(port, &softc->port_list, links) { if ((port->status & CTL_PORT_STATUS_ONLINE) == 0) @@ -7186,15 +7183,18 @@ ctl_report_tagret_port_groups(struct ctl if (ctl_lun_map_to_port(port, lun->lun) >= CTL_MAX_LUNS) continue; num_target_ports++; + if (port->status & CTL_PORT_STATUS_HA_SHARED) + shared_group = 1; } mtx_unlock(&softc->ctl_lock); + num_ha_groups = (softc->is_single) ? 0 : NUM_HA_SHELVES; if (ext) total_len = sizeof(struct scsi_target_group_data_extended); else total_len = sizeof(struct scsi_target_group_data); total_len += sizeof(struct scsi_target_port_group_descriptor) * - num_target_port_groups + + (shared_group + num_ha_groups) + sizeof(struct scsi_target_port_descriptor) * num_target_ports; alloc_len = scsi_4btoul(cdb->length); @@ -7231,24 +7231,62 @@ ctl_report_tagret_port_groups(struct ctl mtx_lock(&softc->ctl_lock); pg = softc->port_min / softc->port_cnt; - if (softc->ha_link == CTL_HA_LINK_OFFLINE) - gs = TPG_ASYMMETRIC_ACCESS_UNAVAILABLE; - else if (softc->ha_link == CTL_HA_LINK_UNKNOWN) - gs = TPG_ASYMMETRIC_ACCESS_TRANSITIONING; - else if (softc->ha_mode == CTL_HA_MODE_ACT_STBY) - gs = TPG_ASYMMETRIC_ACCESS_STANDBY; - else - gs = TPG_ASYMMETRIC_ACCESS_NONOPTIMIZED; - if (lun->flags & CTL_LUN_PRIMARY_SC) { - os = gs; - gs = TPG_ASYMMETRIC_ACCESS_OPTIMIZED; - } else - os = TPG_ASYMMETRIC_ACCESS_OPTIMIZED; - for (g = 0; g < num_target_port_groups; g++) { - tpg_desc->pref_state = (g == pg) ? gs : os; + if (lun->flags & (CTL_LUN_PRIMARY_SC | CTL_LUN_PEER_SC_PRIMARY)) { + /* Some shelf is known to be primary. */ + if (softc->ha_link == CTL_HA_LINK_OFFLINE) + os = TPG_ASYMMETRIC_ACCESS_UNAVAILABLE; + else if (softc->ha_link == CTL_HA_LINK_UNKNOWN) + os = TPG_ASYMMETRIC_ACCESS_TRANSITIONING; + else if (softc->ha_mode == CTL_HA_MODE_ACT_STBY) + os = TPG_ASYMMETRIC_ACCESS_STANDBY; + else + os = TPG_ASYMMETRIC_ACCESS_NONOPTIMIZED; + if (lun->flags & CTL_LUN_PRIMARY_SC) { + ts = TPG_ASYMMETRIC_ACCESS_OPTIMIZED; + } else { + ts = os; + os = TPG_ASYMMETRIC_ACCESS_OPTIMIZED; + } + } else { + /* No known primary shelf. */ + if (softc->ha_link == CTL_HA_LINK_OFFLINE) { + ts = TPG_ASYMMETRIC_ACCESS_UNAVAILABLE; + os = TPG_ASYMMETRIC_ACCESS_OPTIMIZED; + } else if (softc->ha_link == CTL_HA_LINK_UNKNOWN) { + ts = TPG_ASYMMETRIC_ACCESS_TRANSITIONING; + os = TPG_ASYMMETRIC_ACCESS_OPTIMIZED; + } else { + ts = os = TPG_ASYMMETRIC_ACCESS_TRANSITIONING; + } + } + if (shared_group) { + tpg_desc->pref_state = ts; tpg_desc->support = TPG_AO_SUP | TPG_AN_SUP | TPG_S_SUP | TPG_U_SUP | TPG_T_SUP; - scsi_ulto2b(g + 1, tpg_desc->target_port_group); + scsi_ulto2b(1, tpg_desc->target_port_group); + tpg_desc->status = TPG_IMPLICIT; + pc = 0; + STAILQ_FOREACH(port, &softc->port_list, links) { + if ((port->status & CTL_PORT_STATUS_ONLINE) == 0) + continue; + if (!softc->is_single && + (port->status & CTL_PORT_STATUS_HA_SHARED) == 0) + continue; + if (ctl_lun_map_to_port(port, lun->lun) >= CTL_MAX_LUNS) + continue; + scsi_ulto2b(port->targ_port, tpg_desc->descriptors[pc]. + relative_target_port_identifier); + pc++; + } + tpg_desc->target_port_count = pc; + tpg_desc = (struct scsi_target_port_group_descriptor *) + &tpg_desc->descriptors[pc]; + } + for (g = 0; g < num_ha_groups; g++) { + tpg_desc->pref_state = (g == pg) ? ts : os; + tpg_desc->support = TPG_AO_SUP | TPG_AN_SUP | TPG_S_SUP | + TPG_U_SUP | TPG_T_SUP; + scsi_ulto2b(2 + g, tpg_desc->target_port_group); tpg_desc->status = TPG_IMPLICIT; pc = 0; STAILQ_FOREACH(port, &softc->port_list, links) { @@ -7257,6 +7295,8 @@ ctl_report_tagret_port_groups(struct ctl continue; if ((port->status & CTL_PORT_STATUS_ONLINE) == 0) continue; + if (port->status & CTL_PORT_STATUS_HA_SHARED) + continue; if (ctl_lun_map_to_port(port, lun->lun) >= CTL_MAX_LUNS) continue; scsi_ulto2b(port->targ_port, tpg_desc->descriptors[pc]. @@ -9628,7 +9668,7 @@ ctl_inquiry_evpd_devid(struct ctl_scsiio struct ctl_softc *softc; struct ctl_lun *lun; struct ctl_port *port; - int data_len; + int data_len, g; uint8_t proto; softc = control_softc; @@ -9722,8 +9762,11 @@ ctl_inquiry_evpd_devid(struct ctl_scsiio desc->id_type = SVPD_ID_PIV | SVPD_ID_ASSOC_PORT | SVPD_ID_TYPE_TPORTGRP; desc->length = 4; - scsi_ulto2b(ctsio->io_hdr.nexus.targ_port / softc->port_cnt + 1, - &desc->identifier[2]); + if (softc->is_single || port->status & CTL_PORT_STATUS_HA_SHARED) + g = 1; + else + g = 2 + ctsio->io_hdr.nexus.targ_port / softc->port_cnt; + scsi_ulto2b(g, &desc->identifier[2]); desc = (struct scsi_vpd_id_descriptor *)(&desc->identifier[0] + sizeof(struct scsi_vpd_id_trgt_port_grp_id)); Modified: stable/10/sys/cam/ctl/ctl_frontend.c ============================================================================== --- stable/10/sys/cam/ctl/ctl_frontend.c Fri Nov 27 15:19:36 2015 (r291387) +++ stable/10/sys/cam/ctl/ctl_frontend.c Fri Nov 27 15:26:19 2015 (r291388) @@ -310,6 +310,7 @@ ctl_port_online(struct ctl_port *port) { struct ctl_softc *softc = port->ctl_softc; struct ctl_lun *lun; + const char *value; uint32_t l; if (port->lun_enable != NULL) { @@ -328,6 +329,13 @@ ctl_port_online(struct ctl_port *port) if (port->port_online != NULL) port->port_online(port->onoff_arg); mtx_lock(&softc->ctl_lock); + if (softc->is_single == 0) { + value = ctl_get_opt(&port->options, "ha_shared"); + if (value != NULL && strcmp(value, "on") == 0) + port->status |= CTL_PORT_STATUS_HA_SHARED; + else + port->status &= ~CTL_PORT_STATUS_HA_SHARED; + } port->status |= CTL_PORT_STATUS_ONLINE; STAILQ_FOREACH(lun, &softc->lun_list, links) { if (ctl_lun_map_to_port(port, lun->lun) >= CTL_MAX_LUNS) Modified: stable/10/sys/cam/ctl/ctl_frontend.h ============================================================================== --- stable/10/sys/cam/ctl/ctl_frontend.h Fri Nov 27 15:19:36 2015 (r291387) +++ stable/10/sys/cam/ctl/ctl_frontend.h Fri Nov 27 15:26:19 2015 (r291388) @@ -41,7 +41,8 @@ typedef enum { CTL_PORT_STATUS_NONE = 0x00, - CTL_PORT_STATUS_ONLINE = 0x01 + CTL_PORT_STATUS_ONLINE = 0x01, + CTL_PORT_STATUS_HA_SHARED = 0x02 } ctl_port_status; typedef int (*fe_init_t)(void); Modified: stable/10/sys/cam/ctl/ctl_private.h ============================================================================== --- stable/10/sys/cam/ctl/ctl_private.h Fri Nov 27 15:19:36 2015 (r291387) +++ stable/10/sys/cam/ctl/ctl_private.h Fri Nov 27 15:26:19 2015 (r291388) @@ -356,10 +356,7 @@ struct ctl_devid { uint8_t data[]; }; -/* - * For report target port groups. - */ -#define NUM_TARGET_PORT_GROUPS 2 +#define NUM_HA_SHELVES 2 #define CTL_WRITE_BUFFER_SIZE 262144