Date: Thu, 12 Jan 2012 00:34:33 +0000 (UTC) From: "Kenneth D. Merry" <ken@FreeBSD.org> To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r229997 - in head: share/examples/scsi_target sys/amd64/conf sys/cam/ctl sys/cam/scsi sys/conf sys/dev/ata sys/dev/ciss sys/i386/conf sys/ia64/conf sys/sparc64/conf usr.bin usr.bin/ctls... Message-ID: <201201120034.q0C0YXrq084182@svn.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: ken Date: Thu Jan 12 00:34:33 2012 New Revision: 229997 URL: http://svn.freebsd.org/changeset/base/229997 Log: Add the CAM Target Layer (CTL). CTL is a disk and processor device emulation subsystem originally written for Copan Systems under Linux starting in 2003. It has been shipping in Copan (now SGI) products since 2005. It was ported to FreeBSD in 2008, and thanks to an agreement between SGI (who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is available under a BSD-style license. The intent behind the agreement was that Spectra would work to get CTL into the FreeBSD tree. Some CTL features: - Disk and processor device emulation. - Tagged queueing - SCSI task attribute support (ordered, head of queue, simple tags) - SCSI implicit command ordering support. (e.g. if a read follows a mode select, the read will be blocked until the mode select completes.) - Full task management support (abort, LUN reset, target reset, etc.) - Support for multiple ports - Support for multiple simultaneous initiators - Support for multiple simultaneous backing stores - Persistent reservation support - Mode sense/select support - Error injection support - High Availability support (1) - All I/O handled in-kernel, no userland context switch overhead. (1) HA Support is just an API stub, and needs much more to be fully functional. ctl.c: The core of CTL. Command handlers and processing, character driver, and HA support are here. ctl.h: Basic function declarations and data structures. ctl_backend.c, ctl_backend.h: The basic CTL backend API. ctl_backend_block.c, ctl_backend_block.h: The block and file backend. This allows for using a disk or a file as the backing store for a LUN. Multiple threads are started to do I/O to the backing device, primarily because the VFS API requires that to get any concurrency. ctl_backend_ramdisk.c: A "fake" ramdisk backend. It only allocates a small amount of memory to act as a source and sink for reads and writes from an initiator. Therefore it cannot be used for any real data, but it can be used to test for throughput. It can also be used to test initiators' support for extremely large LUNs. ctl_cmd_table.c: This is a table with all 256 possible SCSI opcodes, and command handler functions defined for supported opcodes. ctl_debug.h: Debugging support. ctl_error.c, ctl_error.h: CTL-specific wrappers around the CAM sense building functions. ctl_frontend.c, ctl_frontend.h: These files define the basic CTL frontend port API. ctl_frontend_cam_sim.c: This is a CTL frontend port that is also a CAM SIM. This frontend allows for using CTL without any target-capable hardware. So any LUNs you create in CTL are visible in CAM via this port. ctl_frontend_internal.c, ctl_frontend_internal.h: This is a frontend port written for Copan to do some system-specific tasks that required sending commands into CTL from inside the kernel. This isn't entirely relevant to FreeBSD in general, but can perhaps be repurposed. ctl_ha.h: This is a stubbed-out High Availability API. Much more is needed for full HA support. See the comments in the header and the description of what is needed in the README.ctl.txt file for more details. ctl_io.h: This defines most of the core CTL I/O structures. union ctl_io is conceptually very similar to CAM's union ccb. ctl_ioctl.h: This defines all ioctls available through the CTL character device, and the data structures needed for those ioctls. ctl_mem_pool.c, ctl_mem_pool.h: Generic memory pool implementation used by the internal frontend. ctl_private.h: Private data structres (e.g. CTL softc) and function prototypes. This also includes the SCSI vendor and product names used by CTL. ctl_scsi_all.c, ctl_scsi_all.h: CTL wrappers around CAM sense printing functions. ctl_ser_table.c: Command serialization table. This defines what happens when one type of command is followed by another type of command. ctl_util.c, ctl_util.h: CTL utility functions, primarily designed to be used from userland. See ctladm for the primary consumer of these functions. These include CDB building functions. scsi_ctl.c: CAM target peripheral driver and CTL frontend port. This is the path into CTL for commands from target-capable hardware/SIMs. README.ctl.txt: CTL code features, roadmap, to-do list. usr.sbin/Makefile: Add ctladm. ctladm/Makefile, ctladm/ctladm.8, ctladm/ctladm.c, ctladm/ctladm.h, ctladm/util.c: ctladm(8) is the CTL management utility. It fills a role similar to camcontrol(8). It allow configuring LUNs, issuing commands, injecting errors and various other control functions. usr.bin/Makefile: Add ctlstat. ctlstat/Makefile ctlstat/ctlstat.8, ctlstat/ctlstat.c: ctlstat(8) fills a role similar to iostat(8). It reports I/O statistics for CTL. sys/conf/files: Add CTL files. sys/conf/NOTES: Add device ctl. sys/cam/scsi_all.h: To conform to more recent specs, the inquiry CDB length field is now 2 bytes long. Add several mode page definitions for CTL. sys/cam/scsi_all.c: Handle the new 2 byte inquiry length. sys/dev/ciss/ciss.c, sys/dev/ata/atapi-cam.c, sys/cam/scsi/scsi_targ_bh.c, scsi_target/scsi_cmds.c, mlxcontrol/interface.c: Update for 2 byte inquiry length field. scsi_da.h: Add versions of the format and rigid disk pages that are in a more reasonable format for CTL. amd64/conf/GENERIC, i386/conf/GENERIC, ia64/conf/GENERIC, sparc64/conf/GENERIC: Add device ctl. i386/conf/PAE: The CTL frontend SIM at least does not compile cleanly on PAE. Sponsored by: Copan Systems, SGI and Spectra Logic MFC after: 1 month Added: head/sys/cam/ctl/ head/sys/cam/ctl/README.ctl.txt (contents, props changed) head/sys/cam/ctl/ctl.c (contents, props changed) head/sys/cam/ctl/ctl.h (contents, props changed) head/sys/cam/ctl/ctl_backend.c (contents, props changed) head/sys/cam/ctl/ctl_backend.h (contents, props changed) head/sys/cam/ctl/ctl_backend_block.c (contents, props changed) head/sys/cam/ctl/ctl_backend_block.h (contents, props changed) head/sys/cam/ctl/ctl_backend_ramdisk.c (contents, props changed) head/sys/cam/ctl/ctl_cmd_table.c (contents, props changed) head/sys/cam/ctl/ctl_debug.h (contents, props changed) head/sys/cam/ctl/ctl_error.c (contents, props changed) head/sys/cam/ctl/ctl_error.h (contents, props changed) head/sys/cam/ctl/ctl_frontend.c (contents, props changed) head/sys/cam/ctl/ctl_frontend.h (contents, props changed) head/sys/cam/ctl/ctl_frontend_cam_sim.c (contents, props changed) head/sys/cam/ctl/ctl_frontend_internal.c (contents, props changed) head/sys/cam/ctl/ctl_frontend_internal.h (contents, props changed) head/sys/cam/ctl/ctl_ha.h (contents, props changed) head/sys/cam/ctl/ctl_io.h (contents, props changed) head/sys/cam/ctl/ctl_ioctl.h (contents, props changed) head/sys/cam/ctl/ctl_mem_pool.c (contents, props changed) head/sys/cam/ctl/ctl_mem_pool.h (contents, props changed) head/sys/cam/ctl/ctl_private.h (contents, props changed) head/sys/cam/ctl/ctl_scsi_all.c (contents, props changed) head/sys/cam/ctl/ctl_scsi_all.h (contents, props changed) head/sys/cam/ctl/ctl_ser_table.c (contents, props changed) head/sys/cam/ctl/ctl_util.c (contents, props changed) head/sys/cam/ctl/ctl_util.h (contents, props changed) head/sys/cam/ctl/scsi_ctl.c (contents, props changed) head/usr.bin/ctlstat/ head/usr.bin/ctlstat/Makefile (contents, props changed) head/usr.bin/ctlstat/ctlstat.8 (contents, props changed) head/usr.bin/ctlstat/ctlstat.c (contents, props changed) head/usr.sbin/ctladm/ head/usr.sbin/ctladm/Makefile (contents, props changed) head/usr.sbin/ctladm/ctladm.8 (contents, props changed) head/usr.sbin/ctladm/ctladm.c (contents, props changed) head/usr.sbin/ctladm/ctladm.h (contents, props changed) head/usr.sbin/ctladm/util.c (contents, props changed) Modified: head/share/examples/scsi_target/scsi_cmds.c head/sys/amd64/conf/GENERIC head/sys/cam/scsi/scsi_all.c head/sys/cam/scsi/scsi_all.h head/sys/cam/scsi/scsi_da.h head/sys/cam/scsi/scsi_targ_bh.c head/sys/conf/NOTES head/sys/conf/files head/sys/dev/ata/atapi-cam.c head/sys/dev/ciss/ciss.c head/sys/i386/conf/GENERIC head/sys/i386/conf/PAE head/sys/ia64/conf/GENERIC head/sys/sparc64/conf/GENERIC head/usr.bin/Makefile head/usr.sbin/Makefile head/usr.sbin/mlxcontrol/interface.c Modified: head/share/examples/scsi_target/scsi_cmds.c ============================================================================== --- head/share/examples/scsi_target/scsi_cmds.c Thu Jan 12 00:05:35 2012 (r229996) +++ head/share/examples/scsi_target/scsi_cmds.c Thu Jan 12 00:34:33 2012 (r229997) @@ -328,7 +328,7 @@ tcmd_inquiry(struct ccb_accept_tio *atio bcopy(&inq_data, ctio->data_ptr, sizeof(inq_data)); ctio->dxfer_len = inq_data.additional_length + 4; ctio->dxfer_len = min(ctio->dxfer_len, - SCSI_CDB6_LEN(inq->length)); + scsi_2btoul(inq->length)); ctio->ccb_h.flags |= CAM_DIR_IN | CAM_SEND_STATUS; ctio->scsi_status = SCSI_STATUS_OK; } Modified: head/sys/amd64/conf/GENERIC ============================================================================== --- head/sys/amd64/conf/GENERIC Thu Jan 12 00:05:35 2012 (r229996) +++ head/sys/amd64/conf/GENERIC Thu Jan 12 00:34:33 2012 (r229997) @@ -131,7 +131,8 @@ device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct ATA/SCSI access) -device ses # SCSI Environmental Services (and SAF-TE) +device ses # Enclosure Services (SES and SAF-TE) +device ctl # CAM Target Layer # RAID controllers interfaced to the SCSI subsystem device amr # AMI MegaRAID Added: head/sys/cam/ctl/README.ctl.txt ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/cam/ctl/README.ctl.txt Thu Jan 12 00:34:33 2012 (r229997) @@ -0,0 +1,449 @@ +/* $FreeBSD$ */ + +CTL - CAM Target Layer Description + +Revision 1.4 (December 29th, 2011) +Ken Merry <ken@FreeBSD.org> + +Table of Contents: +================= + +Introduction +Features +Configuring and Running CTL +Revision 1.N Changes +To Do List +Code Roadmap +Userland Commands + +Introduction: +============ + +CTL is a disk and processor device emulation subsystem originally written +for Copan Systems under Linux starting in 2003. It has been shipping in +Copan (now SGI) products since 2005. + +It was ported to FreeBSD in 2008, and thanks to an agreement between SGI +(who acquired Copan's assets in 2010) and Spectra Logic in 2010, CTL is +available under a BSD-style license. The intent behind the agreement was +that Spectra would work to get CTL into the FreeBSD tree. + +Features: +======== + + - Disk and processor device emulation. + - Tagged queueing + - SCSI task attribute support (ordered, head of queue, simple tags) + - SCSI implicit command ordering support. (e.g. if a read follows a mode + select, the read will be blocked until the mode select completes.) + - Full task management support (abort, LUN reset, target reset, etc.) + - Support for multiple ports + - Support for multiple simultaneous initiators + - Support for multiple simultaneous backing stores + - Persistent reservation support + - Mode sense/select support + - Error injection support + - High Availability support (1) + - All I/O handled in-kernel, no userland context switch overhead. + +(1) HA Support is just an API stub, and needs much more to be fully + functional. See the to-do list below. + +Configuring and Running CTL: +=========================== + + - After applying the CTL patchset to your tree, build world and install it + on your target system. + + - Add 'device ctl' to your kernel configuration file. + + - If you're running with a 8Gb or 4Gb Qlogic FC board, add + 'options ISP_TARGET_MODE' to your kernel config file. Keep in mind that + the isp(4) driver can run in target or initiator mode, but not both on + the same machine. 'device ispfw' or loading the ispfw module is also + recommended. + + - Rebuild and install a new kernel. + + - Reboot with the new kernel. + + - To add a LUN with the RAM disk backend: + + ctladm create -b ramdisk -s 10485760000000000000 + ctladm port -o on + + - You should now see the CTL disk LUN through camcontrol devlist: + +scbus6 on ctl2cam0 bus 0: +<FREEBSD CTLDISK 0001> at scbus6 target 1 lun 0 (da24,pass32) +<> at scbus6 target -1 lun -1 () + + This is visible through the CTL CAM SIM. This allows using CTL without + any physical hardware. You should be able to issue any normal SCSI + commands to the device via the pass(4)/da(4) devices. + + If any target-capable HBAs are in the system (e.g. isp(4)), and have + target mode enabled, you should now also be able to see the CTL LUNs via + that target interface. + + Note that all CTL LUNs are presented to all frontends. There is no + LUN masking, or separate, per-port configuration. + + - Note that the ramdisk backend is a "fake" ramdisk. That is, it is + backed by a small amount of RAM that is used for all I/O requests. This + is useful for performance testing, but not for any data integrity tests. + + - To add a LUN with the block/file backend: + + truncate -s +1T myfile + ctladm create -b block -o file=myfile + ctladm port -o on + + - You can also see a list of LUNs and their backends like this: + +# ctladm devlist +LUN Backend Size (Blocks) BS Serial Number Device ID + 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0 + 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1 + 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2 + 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3 + 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4 + 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5 + 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6 + 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7 + 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8 + 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9 + 10 block 2147483648 512 MYSERIAL 10 MYDEVID 10 + 11 block 2147483648 512 MYSERIAL 11 MYDEVID 11 + + - You can see the LUN type and backing store for block/file backend LUNs + like this: + +# ctladm devlist -v +LUN Backend Size (Blocks) BS Serial Number Device ID + 0 block 2147483648 512 MYSERIAL 0 MYDEVID 0 + lun_type=0 + num_threads=14 + file=testdisk0 + 1 block 2147483648 512 MYSERIAL 1 MYDEVID 1 + lun_type=0 + num_threads=14 + file=testdisk1 + 2 block 2147483648 512 MYSERIAL 2 MYDEVID 2 + lun_type=0 + num_threads=14 + file=testdisk2 + 3 block 2147483648 512 MYSERIAL 3 MYDEVID 3 + lun_type=0 + num_threads=14 + file=testdisk3 + 4 block 2147483648 512 MYSERIAL 4 MYDEVID 4 + lun_type=0 + num_threads=14 + file=testdisk4 + 5 block 2147483648 512 MYSERIAL 5 MYDEVID 5 + lun_type=0 + num_threads=14 + file=testdisk5 + 6 block 2147483648 512 MYSERIAL 6 MYDEVID 6 + lun_type=0 + num_threads=14 + file=testdisk6 + 7 block 2147483648 512 MYSERIAL 7 MYDEVID 7 + lun_type=0 + num_threads=14 + file=testdisk7 + 8 block 2147483648 512 MYSERIAL 8 MYDEVID 8 + lun_type=0 + num_threads=14 + file=testdisk8 + 9 block 2147483648 512 MYSERIAL 9 MYDEVID 9 + lun_type=0 + num_threads=14 + file=testdisk9 + 10 ramdisk 0 0 MYSERIAL 0 MYDEVID 0 + lun_type=3 + 11 ramdisk 204800000000000 512 MYSERIAL 1 MYDEVID 1 + lun_type=0 + + +Revision 1.4 Changes +==================== + - Added in the second HA mode (where CTL does the data transfers instead + of having data transfers done below CTL), and abstracted out the Copan + HA API. + + - Fixed the phantom device problem in the CTL CAM SIM and improved the + CAM SIM to automatically trigger a rescan when the port is enabled and + disabled. + + - Made the number of threads in the block backend configurable via sysctl, + loader tunable and the ctladm command line. (You can now specify + -o num_threads=4 when creating a LUN with ctladm create.) + + - Fixed some LUN selection issues in ctlstat(8) and allowed for selection + of LUN numbers up to 1023. + + - General cleanup. + + - This version intended for public release. + +Revision 1.3 Changes +==================== + - Added descriptor sense support to CTL. It can be enabled through the + control mode page (10), but is disabled by default. + + - Improved error injection support. The number of errors that can be + injected with 'ctladm inject' has been increased, and any arbitrary + sense data may now be injected as well. + + - The port infrastructure has been revamped. Individual ports and types + of ports may now be enabled and disabled from the command line. ctladm + now has the ability to set the WWNN and WWPN for each port. + + - The block backend can now send multiple I/Os to backing files. Multiple + writes are only allowed for ZFS, but multiple readers are allowed for + any filesystem. + + - The block and ramdisk backends now support setting the LUN blocksize. + There are some restrictions when the backing device is a block device, + but otherwise the blocksize may be set to anything. + +Revision 1.2 Changes +==================== + + - CTL initialization process has been revamped. Instead of using an + ad-hoc method, it is now sequenced through SYSINIT() calls. + + - A block/file backend has been added. This allows using arbitrary files + or block devices as a backing store. + + - The userland LUN configuration interface has been completely rewritten. + Configuration is now done out of band. + + - The ctladm(8) command line interface has been revamped, and is now + similar to camcontrol(8). + +To Do List: +========== + + - Make CTL buildable as a module. Work needs to be done on initialization, + and on freeing resources and LUNs when it is built as a module. + + - Use devstat(9) for CTL's statistics collection. CTL uses a home-grown + statistics collection system that is similar to devstat(9). ctlstat + should be retired in favor of iostat, etc., once aggregation modes are + available in iostat to match the behavior of ctlstat -t and dump modes + are available to match the behavior of ctlstat -d/ctlstat -J. + + - ZFS ARC backend for CTL. Since ZFS copies all I/O into the ARC + (Adaptive Replacement Cache), running the block/file backend on top of a + ZFS-backed zdev or file will involve an extra set of copies. The + optimal solution for backing targets served by CTL with ZFS would be to + allocate buffers out of the ARC directly, and DMA to/from them directly. + That would eliminate an extra data buffer allocation and copy. + + - Switch CTL over to using CAM CCBs instead of its own union ctl_io. This + will likely require a significant amount of work, but will eliminate + another data structure in the stack, more memory allocations, etc. This + will also require changes to the CAM CCB structure to support CTL. + + - Full-featured High Availability support. The HA API that is in ctl_ha.h + is essentially a renamed version of Copan's HA API. There is no + substance to it, but it remains in CTL to show what needs to be done to + implement active/active HA from a CTL standpoint. The things that would + need to be done include: + - A kernel level software API for message passing as well as DMA + between at least two nodes. + - Hardware support and drivers for inter-node communication. This + could be as simples as ethernet hardware and drivers. + - A "supervisor", or startup framework to control and coordinate + HA startup, failover (going from active/active to single mode), + and failback (going from single mode to active/active). + - HA support in other components of the stack. The goal behind HA + is that one node can fail and another node can seamlessly take + over handling I/O requests. This requires support from pretty + much every component in the storage stack, from top to bottom. + CTL is one piece of it, but you also need support in the RAID + stack/filesystem/backing store. You also need full configuration + mirroring, and all peer nodes need to be able to talk to the + underlying storage hardware. + +Code Roadmap: +============ + +CTL has the concept of pluggable frontend ports and backends. All +frontends and backends can be active at the same time. You can have a +ramdisk-backed LUN present along side a file backed LUN. + +ctl.c: +----- + +This is the core of CTL, where all of the command handlers and a lot of +other things live. Yes, it is large. It started off small and grew to its +current size over time. Perhaps it can be split into more files at some +point. + +Here is a roadmap of some of the primary functions in ctl.c. Starting here +and following the various leaf functions will show the command flow. + +ctl_queue() This is where commands from the frontend ports come + in. + +ctl_queue_sense() This is only used for non-packetized SCSI. i.e. + parallel SCSI prior to U320 and perhaps U160. + +ctl_work_thread() This is the primary work thread, and everything gets + executed from there. + +ctl_scsiio_precheck() This where all of the initial checks are done, and I/O + is either queued for execution or blocked. + +ctl_scsiio() This is where the command handler is actually + executed. (See ctl_cmd_table.c for the mapping of + SCSI opcode to command handler function.) + +ctl_done() This is the routine called (or ctl_done_lock()) to + initiate the command completion process. + +ctl_process_done() This is where command completion actually happens. + +ctl.h: +----- + +Basic function declarations and data structures. + +ctl_backend.c, +ctl_backend.h: +------------- + +These files define the basic CTL backend API. The comments in the header +explain the API. + +ctl_backend_block.c +ctl_backend_block.h: +------------------- + +The block and file backend. This allows for using a disk or a file as the +backing store for a LUN. Multiple threads are started to do I/O to the +backing device, primarily because the VFS API requires that to get any +concurrency. + +ctl_backend_ramdisk.c: +--------------------- + +A "fake" ramdisk backend. It only allocates a small amount of memory to +act as a source and sink for reads and writes from an initiator. Therefore +it cannot be used for any real data, but it can be used to test for +throughput. It can also be used to test initiators' support for extremely +large LUNs. + +ctl_cmd_table.c: +--------------- + +This is a table with all 256 possible SCSI opcodes, and command handler +functions defined for supported opcodes. It is included in ctl.c. + +ctl_debug.h: +----------- + +Simplistic debugging support. + +ctl_error.c, +ctl_error.h: +----------- + +CTL-specific wrappers around the CAM sense building functions. + +ctl_frontend.c, +ctl_frontend.h: +-------------- + +These files define the basic CTL frontend port API. The comments in the +header explain the API. + +ctl_frontend_cam_sim.c: +---------------------- + +This is a CTL frontend port that is also a CAM SIM. The idea is that this +frontend allows for using CTL without any target-capable hardware. So any +LUNs you create in CTL are visible via this port. + + +ctl_frontend_internal.c +ctl_frontend_internal.h: +----------------------- + +This is a frontend port written for Copan to do some system-specific tasks +that required sending commands into CTL from inside the kernel. This isn't +entirely relevant to FreeBSD in general, but can perhaps be repurposed or +removed later. + +ctl_ha.h: +-------- + +This is a stubbed-out High Availability API. See the comments in the +header and the description of what is needed as far as HA support above. + +ctl_io.h: +-------- + +This defines most of the core CTL I/O structures. union ctl_io is +conceptually very similar to CAM's union ccb. + +ctl_ioctl.h: +----------- + +This defines all ioctls available through the CTL character device, and +the data structures needed for those ioctls. + +ctl_mem_pool.c +ctl_mem_pool.h: +-------------- + +Generic memory pool implementation. This is currently only used by the +internal frontend. The internal frontend can probably be rewritten to use +UMA zones and this can be removed. + +ctl_private.h: +------------- + +Private data structres (e.g. CTL softc) and function prototypes. This also +includes the SCSI vendor and product names used by CTL. + +ctl_scsi_all.c +ctl_scsi_all.h: +-------------- + +CTL wrappers around CAM sense printing functions. + +ctl_ser_table.c: +--------------- + +Command serialization table. This defines what happens when one type of +command is followed by another type of command. e.g., what do you do when +you have a mode select followed by a write? You block the write until the +mode select is complete. That is defined in this table. + +ctl_util.c +ctl_util.h: +---------- + +CTL utility functions, primarily designed to be used from userland. See +ctladm for the primary consumer of these functions. These include CDB +building functions. + +scsi_ctl.c: +---------- + +CAM target peripheral driver and CTL frontend port. This is the path into +CTL for commands from target-capable hardware/SIMs. + +Userland Commands: +================= + +ctladm(8) fills a role similar to camcontrol(8). It allow configuring LUNs, +issuing commands, injecting errors and various other control functions. + +ctlstat(8) fills a role similar to iostat(8). It reports I/O statistics +for CTL. Added: head/sys/cam/ctl/ctl.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sys/cam/ctl/ctl.c Thu Jan 12 00:34:33 2012 (r229997) @@ -0,0 +1,13082 @@ +/*- + * Copyright (c) 2003-2009 Silicon Graphics International Corp. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions, and the following disclaimer, + * without modification. + * 2. Redistributions in binary form must reproduce at minimum a disclaimer + * substantially similar to the "NO WARRANTY" disclaimer below + * ("Disclaimer") and any redistribution must be conditioned upon + * including a substantially similar Disclaimer requirement for further + * binary redistribution. + * + * NO WARRANTY + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTIBILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * HOLDERS OR CONTRIBUTORS BE LIABLE FOR SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, + * STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING + * IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGES. + * + * $Id: //depot/users/kenm/FreeBSD-test2/sys/cam/ctl/ctl.c#8 $ + */ +/* + * CAM Target Layer, a SCSI device emulation subsystem. + * + * Author: Ken Merry <ken@FreeBSD.org> + */ + +#define _CTL_C + +#include <sys/cdefs.h> +__FBSDID("$FreeBSD$"); + +#include <sys/param.h> +#include <sys/systm.h> +#include <sys/kernel.h> +#include <sys/types.h> +#include <sys/kthread.h> +#include <sys/bio.h> +#include <sys/fcntl.h> +#include <sys/lock.h> +#include <sys/mutex.h> +#include <sys/condvar.h> +#include <sys/malloc.h> +#include <sys/conf.h> +#include <sys/ioccom.h> +#include <sys/queue.h> +#include <sys/sbuf.h> +#include <sys/endian.h> +#include <sys/sysctl.h> + +#include <cam/cam.h> +#include <cam/scsi/scsi_all.h> +#include <cam/scsi/scsi_da.h> +#include <cam/ctl/ctl_io.h> +#include <cam/ctl/ctl.h> +#include <cam/ctl/ctl_frontend.h> +#include <cam/ctl/ctl_frontend_internal.h> +#include <cam/ctl/ctl_util.h> +#include <cam/ctl/ctl_backend.h> +#include <cam/ctl/ctl_ioctl.h> +#include <cam/ctl/ctl_ha.h> +#include <cam/ctl/ctl_private.h> +#include <cam/ctl/ctl_debug.h> +#include <cam/ctl/ctl_scsi_all.h> +#include <cam/ctl/ctl_error.h> + +struct ctl_softc *control_softc = NULL; + +/* + * The default is to run with CTL_DONE_THREAD turned on. Completed + * transactions are queued for processing by the CTL work thread. When + * CTL_DONE_THREAD is not defined, completed transactions are processed in + * the caller's context. + */ +#define CTL_DONE_THREAD + +/* + * * Use the serial number and device ID provided by the backend, rather than + * * making up our own. + * */ +#define CTL_USE_BACKEND_SN + +/* + * Size and alignment macros needed for Copan-specific HA hardware. These + * can go away when the HA code is re-written, and uses busdma for any + * hardware. + */ +#define CTL_ALIGN_8B(target, source, type) \ + if (((uint32_t)source & 0x7) != 0) \ + target = (type)(source + (0x8 - ((uint32_t)source & 0x7)));\ + else \ + target = (type)source; + +#define CTL_SIZE_8B(target, size) \ + if ((size & 0x7) != 0) \ + target = size + (0x8 - (size & 0x7)); \ + else \ + target = size; + +#define CTL_ALIGN_8B_MARGIN 16 + +/* + * Template mode pages. + */ + +/* + * Note that these are default values only. The actual values will be + * filled in when the user does a mode sense. + */ +static struct copan_power_subpage power_page_default = { + /*page_code*/ PWR_PAGE_CODE | SMPH_SPF, + /*subpage*/ PWR_SUBPAGE_CODE, + /*page_length*/ {(sizeof(struct copan_power_subpage) - 4) & 0xff00, + (sizeof(struct copan_power_subpage) - 4) & 0x00ff}, + /*page_version*/ PWR_VERSION, + /* total_luns */ 26, + /* max_active_luns*/ PWR_DFLT_MAX_LUNS, + /*reserved*/ {0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0} +}; + +static struct copan_power_subpage power_page_changeable = { + /*page_code*/ PWR_PAGE_CODE | SMPH_SPF, + /*subpage*/ PWR_SUBPAGE_CODE, + /*page_length*/ {(sizeof(struct copan_power_subpage) - 4) & 0xff00, + (sizeof(struct copan_power_subpage) - 4) & 0x00ff}, + /*page_version*/ 0, + /* total_luns */ 0, + /* max_active_luns*/ 0, + /*reserved*/ {0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0} +}; + +static struct copan_aps_subpage aps_page_default = { + APS_PAGE_CODE | SMPH_SPF, //page_code + APS_SUBPAGE_CODE, //subpage + {(sizeof(struct copan_aps_subpage) - 4) & 0xff00, + (sizeof(struct copan_aps_subpage) - 4) & 0x00ff}, //page_length + APS_VERSION, //page_version + 0, //lock_active + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0} //reserved +}; + +static struct copan_aps_subpage aps_page_changeable = { + APS_PAGE_CODE | SMPH_SPF, //page_code + APS_SUBPAGE_CODE, //subpage + {(sizeof(struct copan_aps_subpage) - 4) & 0xff00, + (sizeof(struct copan_aps_subpage) - 4) & 0x00ff}, //page_length + 0, //page_version + 0, //lock_active + {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0} //reserved +}; + +static struct copan_debugconf_subpage debugconf_page_default = { + DBGCNF_PAGE_CODE | SMPH_SPF, /* page_code */ + DBGCNF_SUBPAGE_CODE, /* subpage */ + {(sizeof(struct copan_debugconf_subpage) - 4) >> 8, + (sizeof(struct copan_debugconf_subpage) - 4) >> 0}, /* page_length */ + DBGCNF_VERSION, /* page_version */ + {CTL_TIME_IO_DEFAULT_SECS>>8, + CTL_TIME_IO_DEFAULT_SECS>>0}, /* ctl_time_io_secs */ +}; + +static struct copan_debugconf_subpage debugconf_page_changeable = { + DBGCNF_PAGE_CODE | SMPH_SPF, /* page_code */ + DBGCNF_SUBPAGE_CODE, /* subpage */ + {(sizeof(struct copan_debugconf_subpage) - 4) >> 8, + (sizeof(struct copan_debugconf_subpage) - 4) >> 0}, /* page_length */ + 0, /* page_version */ + {0xff,0xff}, /* ctl_time_io_secs */ +}; + +static struct scsi_format_page format_page_default = { + /*page_code*/SMS_FORMAT_DEVICE_PAGE, + /*page_length*/sizeof(struct scsi_format_page) - 2, + /*tracks_per_zone*/ {0, 0}, + /*alt_sectors_per_zone*/ {0, 0}, + /*alt_tracks_per_zone*/ {0, 0}, + /*alt_tracks_per_lun*/ {0, 0}, + /*sectors_per_track*/ {(CTL_DEFAULT_SECTORS_PER_TRACK >> 8) & 0xff, + CTL_DEFAULT_SECTORS_PER_TRACK & 0xff}, + /*bytes_per_sector*/ {0, 0}, + /*interleave*/ {0, 0}, + /*track_skew*/ {0, 0}, + /*cylinder_skew*/ {0, 0}, + /*flags*/ SFP_HSEC, + /*reserved*/ {0, 0, 0} +}; + +static struct scsi_format_page format_page_changeable = { + /*page_code*/SMS_FORMAT_DEVICE_PAGE, + /*page_length*/sizeof(struct scsi_format_page) - 2, + /*tracks_per_zone*/ {0, 0}, + /*alt_sectors_per_zone*/ {0, 0}, + /*alt_tracks_per_zone*/ {0, 0}, + /*alt_tracks_per_lun*/ {0, 0}, + /*sectors_per_track*/ {0, 0}, + /*bytes_per_sector*/ {0, 0}, + /*interleave*/ {0, 0}, + /*track_skew*/ {0, 0}, + /*cylinder_skew*/ {0, 0}, + /*flags*/ 0, + /*reserved*/ {0, 0, 0} +}; + +static struct scsi_rigid_disk_page rigid_disk_page_default = { + /*page_code*/SMS_RIGID_DISK_PAGE, + /*page_length*/sizeof(struct scsi_rigid_disk_page) - 2, + /*cylinders*/ {0, 0, 0}, + /*heads*/ CTL_DEFAULT_HEADS, + /*start_write_precomp*/ {0, 0, 0}, + /*start_reduced_current*/ {0, 0, 0}, + /*step_rate*/ {0, 0}, + /*landing_zone_cylinder*/ {0, 0, 0}, + /*rpl*/ SRDP_RPL_DISABLED, + /*rotational_offset*/ 0, + /*reserved1*/ 0, + /*rotation_rate*/ {(CTL_DEFAULT_ROTATION_RATE >> 8) & 0xff, + CTL_DEFAULT_ROTATION_RATE & 0xff}, + /*reserved2*/ {0, 0} +}; + +static struct scsi_rigid_disk_page rigid_disk_page_changeable = { + /*page_code*/SMS_RIGID_DISK_PAGE, + /*page_length*/sizeof(struct scsi_rigid_disk_page) - 2, + /*cylinders*/ {0, 0, 0}, + /*heads*/ 0, + /*start_write_precomp*/ {0, 0, 0}, + /*start_reduced_current*/ {0, 0, 0}, + /*step_rate*/ {0, 0}, + /*landing_zone_cylinder*/ {0, 0, 0}, + /*rpl*/ 0, + /*rotational_offset*/ 0, + /*reserved1*/ 0, + /*rotation_rate*/ {0, 0}, + /*reserved2*/ {0, 0} +}; + +static struct scsi_caching_page caching_page_default = { + /*page_code*/SMS_CACHING_PAGE, + /*page_length*/sizeof(struct scsi_caching_page) - 2, + /*flags1*/ SCP_DISC | SCP_WCE, + /*ret_priority*/ 0, + /*disable_pf_transfer_len*/ {0xff, 0xff}, + /*min_prefetch*/ {0, 0}, + /*max_prefetch*/ {0xff, 0xff}, + /*max_pf_ceiling*/ {0xff, 0xff}, + /*flags2*/ 0, + /*cache_segments*/ 0, + /*cache_seg_size*/ {0, 0}, + /*reserved*/ 0, + /*non_cache_seg_size*/ {0, 0, 0} +}; + +static struct scsi_caching_page caching_page_changeable = { + /*page_code*/SMS_CACHING_PAGE, + /*page_length*/sizeof(struct scsi_caching_page) - 2, + /*flags1*/ 0, + /*ret_priority*/ 0, + /*disable_pf_transfer_len*/ {0, 0}, + /*min_prefetch*/ {0, 0}, + /*max_prefetch*/ {0, 0}, + /*max_pf_ceiling*/ {0, 0}, + /*flags2*/ 0, + /*cache_segments*/ 0, + /*cache_seg_size*/ {0, 0}, + /*reserved*/ 0, + /*non_cache_seg_size*/ {0, 0, 0} +}; + +static struct scsi_control_page control_page_default = { + /*page_code*/SMS_CONTROL_MODE_PAGE, + /*page_length*/sizeof(struct scsi_control_page) - 2, + /*rlec*/0, + /*queue_flags*/0, + /*eca_and_aen*/0, + /*reserved*/0, + /*aen_holdoff_period*/{0, 0} +}; + +static struct scsi_control_page control_page_changeable = { + /*page_code*/SMS_CONTROL_MODE_PAGE, + /*page_length*/sizeof(struct scsi_control_page) - 2, + /*rlec*/SCP_DSENSE, + /*queue_flags*/0, + /*eca_and_aen*/0, + /*reserved*/0, + /*aen_holdoff_period*/{0, 0} +}; + +SYSCTL_NODE(_kern_cam, OID_AUTO, ctl, CTLFLAG_RD, 0, "CAM Target Layer"); + +/* + * XXX KDM move these into the softc. + */ +static int rcv_sync_msg; +static int persis_offset; +static uint8_t ctl_pause_rtr; +static int ctl_is_single; +static int index_to_aps_page; + + +/* + * Serial number (0x80), device id (0x83), and supported pages (0x00) + */ +#define SCSI_EVPD_NUM_SUPPORTED_PAGES 3 + +static void ctl_isc_event_handler(ctl_ha_channel chanel, ctl_ha_event event, + int param); +static void ctl_copy_sense_data(union ctl_ha_msg *src, union ctl_io *dest); +static void ctl_init(void); +void ctl_shutdown(void); +static int ctl_open(struct cdev *dev, int flags, int fmt, struct thread *td); +static int ctl_close(struct cdev *dev, int flags, int fmt, struct thread *td); +static void ctl_ioctl_online(void *arg); +static void ctl_ioctl_offline(void *arg); +static int ctl_ioctl_targ_enable(void *arg, struct ctl_id targ_id); +static int ctl_ioctl_targ_disable(void *arg, struct ctl_id targ_id); +static int ctl_ioctl_lun_enable(void *arg, struct ctl_id targ_id, int lun_id); +static int ctl_ioctl_lun_disable(void *arg, struct ctl_id targ_id, int lun_id); +static int ctl_ioctl_do_datamove(struct ctl_scsiio *ctsio); +static int ctl_serialize_other_sc_cmd(struct ctl_scsiio *ctsio, int have_lock); +static int ctl_ioctl_submit_wait(union ctl_io *io); +static void ctl_ioctl_datamove(union ctl_io *io); +static void ctl_ioctl_done(union ctl_io *io); +static void ctl_ioctl_hard_startstop_callback(void *arg, + struct cfi_metatask *metatask); +static void ctl_ioctl_bbrread_callback(void *arg,struct cfi_metatask *metatask); +static int ctl_ioctl_fill_ooa(struct ctl_lun *lun, uint32_t *cur_fill_num, + struct ctl_ooa *ooa_hdr); +static int ctl_ioctl(struct cdev *dev, u_long cmd, caddr_t addr, int flag, + struct thread *td); +uint32_t ctl_get_resindex(struct ctl_nexus *nexus); +uint32_t ctl_port_idx(int port_num); +#ifdef unused +static union ctl_io *ctl_malloc_io(ctl_io_type io_type, uint32_t targ_port, + uint32_t targ_target, uint32_t targ_lun, + int can_wait); +static void ctl_kfree_io(union ctl_io *io); +#endif /* unused */ +static void ctl_free_io_internal(union ctl_io *io, int have_lock); +static int ctl_alloc_lun(struct ctl_softc *ctl_softc, struct ctl_lun *lun, + struct ctl_be_lun *be_lun, struct ctl_id target_id); +static int ctl_free_lun(struct ctl_lun *lun); +static void ctl_create_lun(struct ctl_be_lun *be_lun); +/** +static void ctl_failover_change_pages(struct ctl_softc *softc, + struct ctl_scsiio *ctsio, int master); +**/ + +static int ctl_do_mode_select(union ctl_io *io); +static int ctl_pro_preempt(struct ctl_softc *softc, struct ctl_lun *lun, + uint64_t res_key, uint64_t sa_res_key, + uint8_t type, uint32_t residx, + struct ctl_scsiio *ctsio, + struct scsi_per_res_out *cdb, + struct scsi_per_res_out_parms* param); +static void ctl_pro_preempt_other(struct ctl_lun *lun, + union ctl_ha_msg *msg); +static void ctl_hndl_per_res_out_on_other_sc(union ctl_ha_msg *msg); +static int ctl_inquiry_evpd_supported(struct ctl_scsiio *ctsio, int alloc_len); +static int ctl_inquiry_evpd_serial(struct ctl_scsiio *ctsio, int alloc_len); +static int ctl_inquiry_evpd_devid(struct ctl_scsiio *ctsio, int alloc_len); +static int ctl_inquiry_evpd(struct ctl_scsiio *ctsio); +static int ctl_inquiry_std(struct ctl_scsiio *ctsio); +static int ctl_get_lba_len(union ctl_io *io, uint64_t *lba, uint32_t *len); +static ctl_action ctl_extent_check(union ctl_io *io1, union ctl_io *io2); +static ctl_action ctl_check_for_blockage(union ctl_io *pending_io, + union ctl_io *ooa_io); +static ctl_action ctl_check_ooa(struct ctl_lun *lun, union ctl_io *pending_io, + union ctl_io *starting_io); +static int ctl_check_blocked(struct ctl_lun *lun); +static int ctl_scsiio_lun_check(struct ctl_softc *ctl_softc, + struct ctl_lun *lun, + struct ctl_cmd_entry *entry, + struct ctl_scsiio *ctsio); +//static int ctl_check_rtr(union ctl_io *pending_io, struct ctl_softc *softc); +static void ctl_failover(void); +static int ctl_scsiio_precheck(struct ctl_softc *ctl_softc, + struct ctl_scsiio *ctsio); +static int ctl_scsiio(struct ctl_scsiio *ctsio); + +static int ctl_bus_reset(struct ctl_softc *ctl_softc, union ctl_io *io); +static int ctl_target_reset(struct ctl_softc *ctl_softc, union ctl_io *io, + ctl_ua_type ua_type); +static int ctl_lun_reset(struct ctl_lun *lun, union ctl_io *io, + ctl_ua_type ua_type); +static int ctl_abort_task(union ctl_io *io); +static void ctl_run_task_queue(struct ctl_softc *ctl_softc); +#ifdef CTL_IO_DELAY +static void ctl_datamove_timer_wakeup(void *arg); +static void ctl_done_timer_wakeup(void *arg); +#endif /* CTL_IO_DELAY */ + +static void ctl_send_datamove_done(union ctl_io *io, int have_lock); +static void ctl_datamove_remote_write_cb(struct ctl_ha_dt_req *rq); +static int ctl_datamove_remote_dm_write_cb(union ctl_io *io); +static void ctl_datamove_remote_write(union ctl_io *io); +static int ctl_datamove_remote_dm_read_cb(union ctl_io *io); +static void ctl_datamove_remote_read_cb(struct ctl_ha_dt_req *rq); +static int ctl_datamove_remote_sgl_setup(union ctl_io *io); +static int ctl_datamove_remote_xfer(union ctl_io *io, unsigned command, + ctl_ha_dt_cb callback); +static void ctl_datamove_remote_read(union ctl_io *io); +static void ctl_datamove_remote(union ctl_io *io); +static int ctl_process_done(union ctl_io *io, int have_lock); +static void ctl_work_thread(void *arg); + +/* + * Load the serialization table. This isn't very pretty, but is probably + * the easiest way to do it. + */ +#include "ctl_ser_table.c" + +/* + * We only need to define open, close and ioctl routines for this driver. + */ +static struct cdevsw ctl_cdevsw = { + .d_version = D_VERSION, + .d_flags = 0, + .d_open = ctl_open, + .d_close = ctl_close, + .d_ioctl = ctl_ioctl, + .d_name = "ctl", +}; + + +MALLOC_DEFINE(M_CTL, "ctlmem", "Memory used for CTL"); + +/* + * If we have the CAM SIM, we may or may not have another SIM that will + * cause CTL to get initialized. If not, we need to initialize it. + */ +SYSINIT(ctl_init, SI_SUB_CONFIGURE, SI_ORDER_THIRD, ctl_init, NULL); + +static void +ctl_isc_handler_finish_xfer(struct ctl_softc *ctl_softc, + union ctl_ha_msg *msg_info) +{ + struct ctl_scsiio *ctsio; + + if (msg_info->hdr.original_sc == NULL) { + printf("%s: original_sc == NULL!\n", __func__); + /* XXX KDM now what? */ + return; + } + + ctsio = &msg_info->hdr.original_sc->scsiio; + ctsio->io_hdr.flags |= CTL_FLAG_IO_ACTIVE; + ctsio->io_hdr.msg_type = CTL_MSG_FINISH_IO; + ctsio->io_hdr.status = msg_info->hdr.status; + ctsio->scsi_status = msg_info->scsi.scsi_status; + ctsio->sense_len = msg_info->scsi.sense_len; + ctsio->sense_residual = msg_info->scsi.sense_residual; + ctsio->residual = msg_info->scsi.residual; + memcpy(&ctsio->sense_data, &msg_info->scsi.sense_data, + sizeof(ctsio->sense_data)); + memcpy(&ctsio->io_hdr.ctl_private[CTL_PRIV_LBA_LEN].bytes, + &msg_info->scsi.lbalen, sizeof(msg_info->scsi.lbalen));; + STAILQ_INSERT_TAIL(&ctl_softc->isc_queue, &ctsio->io_hdr, links); + ctl_wakeup_thread(); +} + +static void +ctl_isc_handler_finish_ser_only(struct ctl_softc *ctl_softc, + union ctl_ha_msg *msg_info) +{ + struct ctl_scsiio *ctsio; + + if (msg_info->hdr.serializing_sc == NULL) { + printf("%s: serializing_sc == NULL!\n", __func__); + /* XXX KDM now what? */ + return; + } + + ctsio = &msg_info->hdr.serializing_sc->scsiio; +#if 0 + /* + * Attempt to catch the situation where an I/O has + * been freed, and we're using it again. + */ + if (ctsio->io_hdr.io_type == 0xff) { + union ctl_io *tmp_io; + tmp_io = (union ctl_io *)ctsio; + printf("%s: %p use after free!\n", __func__, + ctsio); + printf("%s: type %d msg %d cdb %x iptl: " + "%d:%d:%d:%d tag 0x%04x " + "flag %#x status %x\n", + __func__, + tmp_io->io_hdr.io_type, + tmp_io->io_hdr.msg_type, + tmp_io->scsiio.cdb[0], + tmp_io->io_hdr.nexus.initid.id, + tmp_io->io_hdr.nexus.targ_port, + tmp_io->io_hdr.nexus.targ_target.id, + tmp_io->io_hdr.nexus.targ_lun, + (tmp_io->io_hdr.io_type == + CTL_IO_TASK) ? + tmp_io->taskio.tag_num : + tmp_io->scsiio.tag_num, + tmp_io->io_hdr.flags, + tmp_io->io_hdr.status); + } +#endif + ctsio->io_hdr.msg_type = CTL_MSG_FINISH_IO; *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201201120034.q0C0YXrq084182>