From owner-freebsd-current@FreeBSD.ORG Fri Apr 29 17:36:25 2005 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8628416A4CE; Fri, 29 Apr 2005 17:36:25 +0000 (GMT) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 873D443D46; Fri, 29 Apr 2005 17:36:22 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (junior-wifi.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j3THe5im032060; Fri, 29 Apr 2005 11:40:06 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <42727078.8090307@samsco.org> Date: Fri, 29 Apr 2005 11:35:52 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Vinod Kashyap References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.8 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org cc: freebsd-current@FreeBSD.org cc: "Bjoern A. Zeeb" Subject: Re: Problem with twa in HEAD X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2005 17:36:25 -0000 Vinod Kashyap wrote: > >>-----Original Message----- >>From: Scott Long [mailto:scottl@samsco.org] >>Sent: Thursday, April 28, 2005 11:57 PM >>To: Vinod Kashyap >>Cc: Bjoern A. Zeeb; freebsd-current@FreeBSD.org >>Subject: Re: Problem with twa in HEAD >> >> >>Vinod Kashyap wrote: >> >>>>-----Original Message----- >>>>From: Bjoern A. Zeeb [mailto:bz@FreeBSD.org] >>>>Sent: Tuesday, April 26, 2005 3:26 AM >>>>To: Vinod Kashyap >>>>Subject: RE: Problem with twa in HEAD >>>> >>>> >>>>On Mon, 25 Apr 2005, Vinod Kashyap wrote: >>>> >>>>Hi, >>>> >>>> >>>> >>>>>>-----Original Message----- >>>>>>From: Bjoern A. Zeeb [mailto:bz@FreeBSD.org] >>>>>>Sent: Monday, April 25, 2005 6:45 AM >>>>>>To: Vinod Kashyap >>>>>>Subject: Re: Problem with twa in HEAD >>>>>> >>>>>> >>>>>>On Fri, 22 Apr 2005, Bjoern A. Zeeb wrote: >>>>>> >>>>>>Hi, >>>>>> >>>>>> >>>>>> >>>>>>>scottl redirected me to you. >>>>>>> >>>>>>>I am currently debugging "hangs" on reboot and shutdown on a >>>>>>>SMP machine with 12 discs at a >>>>>>> >>>>>>>3ware device driver for 9000 series storage controllers, >>>>>> >>>>>>version: 3.60.00.016 >>>>>> >>>>>> >>>>>>>twa0: <3ware 9000 series Storage Controller> port >>>>>> >>>>>>0x9800-0x98ff mem 0xfe8ffc00-0xfe8ffcff,0xfb800000-0xfbffffff >>>>>>irq 28 at device 6.0 on pci3 >>>>>> >>>>>> >>>>>>>twa0: [FAST] >>>>>>>twa0: INFO: (0x15: 0x1300): Controller details:: 12 ports, >>>>>> >>>>>>Firmware FE9X 2.06.00.009, BIOS BE9X 2.03.01.051 >>>>>> >>>>>> >>>>>>>What I know so far is that Giant is held by sync. >>>>>>> >>>>>>>Things a "spinning" in cam/cam_xpt.c around: >>>>>>> >>>>>>>--- cam_xpt.c 31 Mar 2005 21:42:49 -0000 1.152 >>>>>>>+++ cam_xpt.c 22 Apr 2005 18:42:43 -0000 >>>>>>>@@ -3643,6 +3643,7 @@ xpt_polled_action(union ccb *start_ccb) >>>>>>> != CAM_REQ_INPROG) >>>>>>> break; >>>>>>> DELAY(1000); >>>>>>> printf("XXX status=%02x\n", >>>>>> >>>>>>start_ccb->ccb_h.status); >>>>>> >>>>>> >>>>>>> } >>>>>>> if (timeout == 0) { >>>>>>> /* >>>>>>> >>>>>>> >>>>>>>with status being 0x200. >>>>>>> >>>>>>>Seems the twa has a command stuck in it. >>>>>>> >>>>>>>I have seen the comment in dev/twa/tw_osl_cam.c ~ line 253 about >>>>>>>queuing and CAM_SIM_QUEUED but I don't know enough about cam. >>>>>>>I seems no all patchs out of this functions seem to >>>> >>>>clear that from >>>> >>>> >>>>>>>status? >>>>>>> >>>>>>>Any help apreaciated ;) I can try patches; as long as I >>>> >>>>can break >>>> >>>> >>>>>>>to db> to reboot. >>>>>> >>>>>>further debugging shows that is seems to be spinning in twa_poll. >>>>>>see debug output from TWA_DEBUG 3. The problem is that at >>>> >>>>this point >>>> >>>> >>>>>>I am no longer able to break to debugger. >>>>>> >>>>>>twa0: tw_osli_execute_scsi: XPT_SCSI_IO: Single virtual address! >>>>>>twa0: tw_osli_execute_scsi: XPT_SCSI_IO: Single virtual address! >>>>>>unmount of /dev failed (BUSY) >>>>>>twa0: tw_osli_execute_scsi: XPT_SCSI_IO: Single virtual address! >>>>>>twa0: tw_osli_execute_scsi: XPT_SCSI_IO: Single virtual address! >>>>>>Uptime: 2m57s >>>>>>twa0: tw_osli_execute_scsi: XPT_SCSI_IO: Single virtual address! >>>>>>twa0: twa_poll: entering; sc = 0xc57bb200 >>>>>>twa0: twa_poll: exiting; sc = 0xc57bb200 >>>>>>twa0: twa_poll: entering; sc = 0xc57bb200 >>>>>>twa0: twa_poll: exiting; sc = 0xc57bb200 >>>>>>twa0: twa_poll: entering; sc = 0xc57bb200 >>>>>>twa0: twa_poll: exiting; sc = 0xc57bb200 >>>>>>twa0: twa_poll: entering; sc = 0xc57bb200 >>>>>>twa0: twa_poll: exiting; sc = 0xc57bb200 >>>>>>twa0: twa_poll: entering; sc = 0xc57bb200 >>>>>>twa0: twa_poll: exiting; sc = 0xc57bb200 >>>>>>twa0: twa_poll: entering; sc = 0xc57bb200 >>>>>>twa0: twa_poll: exiting; sc = 0xc57bb200 >>>>>>twa0: twa_poll: entering; sc = 0xc57bb200 >>>>>>twa0: twa_poll: exiting; sc = 0xc57bb200 >>>>>>... >>>>>> >>>>> >>>>>I am in the middle of an office move right now. >>>>>I will get back to you once I have some time to look into this. >>>> >>>> >>>>thanks for the information; I'll be able to test at least >> >>until end of >> >>>>this week and hopefully next week too. >>>> >>> >>> >>>I looked into this, and this is what is happening: >>>On reboot/halt, the following function calling sequence happens: >>>... --> dashutdown --> xpt_polled_action --> twa_poll. >>>But, the interrupt handler in twa is still active at this time, >>>since twa_detach/twa_shutdown hasn't been called yet. Before >>>twa_poll can fetch the response for the posted command, the ISR >>>gets called when the firmware posts the response. The ISR clears >>>the interrupt bit on the controller, registers a taskqueue >> >>handler like >> >>>it always does, and exits. Meanwhile, xpt_polled_action continues >>>to call twa_poll, which cannot determine that the command >> >>has completed, >> >>>since the interrupt bit on the controller is already cleared. So, >>>we get into a (near) never-ending loop (the timeout for >> >>scsi_synchronize_cache, >> >>>which is what is being tried here, is, for whatever reason, >> >>60 minutes, >> >>>and so, the system is as good as hung). >>> >>>Now, does anyone know why xpt_polled_action is being called from >>>dashutdown, even before the ISR has been unregistered (via >> >>twa_detach)? >> >>>Bjoern, this patch should work-around your problem, >> >>although it's not >> >>>the fix. Also, it still leaves a window for the race >> >>condition described >> >>>above. >>> >> >>xpt_polled_action() expects that it can simulate interrupts by calling >>the driver poll vector, and that by calling it enough times the driver >>will eventually complete all the outstanding I/O it has. As you note, >>it'll repeat this for a very long time. So the question is >>then why the >>twa driver isn't completing the outstanding I/O. If I were you I'd >>remove the call to tw_cl_interrupt() in twa_poll() and just >>unconditionally call tw_cl_deferred_interrupt() and have it check >>everything. > > > The call to tw_cl_interrupt cannot be removed as that's where the interrupt > is cleared. However, the call to tw_cl_deferred_interrupt can be made > not conditional to the return value from tw_cl_interrupt. > Whatever that is, my question is why polling is being resorted to, when > interrupts are available? > > >> The locking here (and in twa_pci_intr()) is >>flawed anyways, >>you have a race between when tw_cl_interrupt() drops its lock right >>before return and when you check it's return value. I'd like say that >>it's harmless, except that you expect to pass state from one >>function to >>the next, so the race is a real one. It's likely why this case is > > > If the state passed to the second function is invalid when the second > function executes, it just does nothing. So, this is pretty harmless. > > >>failing. An ideal FAST handler should only clear the >>hardware interrupt >>register and launch the appropriate handlers, it shouldn't try to pass >>state to the handlers. Look at aac for an example here, but >>also please >>recall that I've already discouraged you from using a a fast handler >>plus taskqueue for this driver. If your taskqueue handlers need state >>from when the interrupt was cleared, then they simply aren't a good >>candidate for this model. >> > > > In twa, state will need to be passed to the taskqueue handlers since the > ISR clears the state on the controller. > > Have it your way then, I'm frankly very weary of arguing with you. Scott