From owner-freebsd-scsi@FreeBSD.ORG Fri Dec 4 23:05:30 2009 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 18EF31065670 for ; Fri, 4 Dec 2009 23:05:30 +0000 (UTC) (envelope-from pisymbol@gmail.com) Received: from mail-yx0-f171.google.com (mail-yx0-f171.google.com [209.85.210.171]) by mx1.freebsd.org (Postfix) with ESMTP id C18EA8FC0C for ; Fri, 4 Dec 2009 23:05:29 +0000 (UTC) Received: by yxe1 with SMTP id 1so2611333yxe.3 for ; Fri, 04 Dec 2009 15:05:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=Zj7xDlnu4zsVjeoNsOo4MlfLu5YWaoCggczWVjIMcv0=; b=NomGlt1xap+fpASL0MNDd4XCGt77RvoGsxTNgVtz/e0VG3NhtzGEgPHiCJfJRrUjp0 WzyHPZHGnATG3NT/wxyeEUI8QcyCoTD5jMTLHl9iGx/dolvxcK41ARMc5M7JAoAqqKJr 9tlbojFLoEtFh9ozG021HEUX19iWq6Mt4N3IQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=xPNflSjCO/kJhKmrTxL4iOSfexAy9vuA5QucV6Pi9awwrLslNNDLh0RTVUqGE+gTkM H7b9v5ksGepjuWhlMWtH5DhngUKWezUshBJt5pUN+2mYagUCqqAnMmZMp3zZ1Wy7DIbp rFDWUASxYTHX58AAtdWskbN6GPiiz2RfLer9o= MIME-Version: 1.0 Received: by 10.101.165.14 with SMTP id s14mr4819731ano.186.1259966556753; Fri, 04 Dec 2009 14:42:36 -0800 (PST) Date: Fri, 4 Dec 2009 17:42:36 -0500 Message-ID: <3c0b01820912041442q27c6f28fpbf53a6523945d77d@mail.gmail.com> From: Alexander Sack To: freebsd-scsi@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Adaptec aac(4) FIB starvation issues on BUS scan X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Dec 2009 23:05:30 -0000 Hey Everyone: I am running into several issues with the Adaptec aac(4) driver and FIB starvation during the probe PATH on xpt_bus_scan(). I am on right on a 6.1-amd64 machine with two Adaptec 5085s (6 channel) controllers connected to external JBODs as well as an MFI controller for the OS itself. Though this is legacy 6.1, I feel (have to test!) this might also be indicative on HEAD, RELENG_7-8, etc. Let me explain: aac_cam_action() (aac_cam.c): 250 case XPT_PATH_INQ: 251 { 252 struct ccb_pathinq *cpi = &ccb->cpi; 253 254 cpi->version_num = 1; 255 cpi->hba_inquiry = PI_WIDE_16; 256 cpi->target_sprt = 0; 257 258 /* Resetting via the passthrough causes problems. */ 259 cpi->hba_misc = PIM_NOBUSRESET; 260 cpi->hba_eng_cnt = 0; 261 cpi->max_target = camsc->inf->TargetsPerBus; The number of FIBs allocated to this card is 512 (older cards are 256). The max_target per bus is 287 (0x11F). On a six channel controller with a BUS scan done in parallel I see a lot of this: ... (probe501:aacp1:0:214:0): Request Requeued (probe501:aacp1:0:214:0): Retrying Command (probe520:aacp1:0:233:0): Request Requeued (probe520:aacp1:0:233:0): Retrying Command (probe528:aacp1:0:241:0): Request Requeued (probe528:aacp1:0:241:0): Retrying Command (probe540:aacp1:0:253:0): Request Requeued (probe540:aacp1:0:253:0): Retrying Command (probe541:aacp1:0:254:0): Request Requeued (probe541:aacp1:0:254:0): Retrying Command .... Now, this occurs because during the normal XPT_SCSI_IO request path during INQUIRY's (I didn't check but its out of the cam_periph_alloc()/probeXXX() stuff, I'm still debugging it). It runs out of FIBS as per: 311 /* Async ops that require communcation with the controller */ 312 313 mtx_lock(&sc->aac_io_lock); 314 if (aac_alloc_command(sc, &cm)) { WE ARE HERE, aac_alloc_command returned EBUSY 315 struct aac_event *event; 316 317 xpt_freeze_simq(sim, 1); 318 ccb->ccb_h.status = CAM_REQUEUE_REQ; 319 xpt_done(ccb); 320 event = malloc(sizeof(struct aac_event), M_AACCAM, 321 M_NOWAIT | M_ZERO); 322 if (event == NULL) { 323 device_printf(sc->aac_dev, 324 "Warning, out of memory for event\n"); 325 /* XXX Yuck, what to do here? */ 326 mtx_unlock(&sc->aac_io_lock); 327 return; 328 } 329 event->ev_callback = aac_cam_event; 330 event->ev_arg = camsc; 331 event->ev_type = AAC_EVENT_CMFREE; 332 aac_add_event(sc, event); 333 mtx_unlock(&sc->aac_io_lock); 334 return; 335 } The aac_alloc_command() function tries to grab a free FIB that has been released back to the driver but none are available (aac_dequeue_free returned NULL). So what it does is attempt to wakeup its internal working ktrhead, aac_command_thread(), to go allocate more FIBs. In the meantime, it freezes the SIMQ and sets CAM_REQUEUE_REQ (unconditionally force CAM to retry the request) and then allocate an internal event which will free the simq provided it can allocate a fresh batch of FIBs (iff under the maximum allowed by the controller). Questions: 1) What is the downside of this change: 259 cpi->hba_misc = PIM_NOBUSRESET | PIM_SEQSCAN; This makes the BUS scan an ORDER OF MAGNITUDE faster with no forced retries. I mean it. Instead of waiting for many many seconds, I wait for less than a second for targets to come back. 2) Why is CAM_REQUEUE_REQ appropriate? Isn't the driver banking on the fact that more FIBs will be allocated when at this point you have hit a resource starvation issue and something like CAM_RESRC_UNAVAIL to throttle jobs and give time for the either FIBs to be released back and or the command threads to allocate it. I am going to test this change anyway. 3) In the a similar topic of question 2, FIBs are indeed preallocated: 579 while (sc->total_fibs < AAC_PREALLOCATE_FIBS) { 580 if (aac_alloc_commands(sc) != 0) 581 break; 582 } AAC_PREALLOCATE_FIBS is set to 128. Any reason why not to preallocate all of them or at least HALF of them (256 of 512)? :D FIBS are 2k in size so 256 of them, 512k, is not THAT much to ask the kernel these days on most modern systems. By this change does not solve problem one (still too many requests). I want to point out that we've seen hangs during probe in our internal labs but its very hard to reproduce (I'm pretty sure now its due to commands eternally being retried by CAM due to FIB resource starvation but I am still trying to prove it). Any feedback would be most welcomed, -aps