Date: Fri, 18 Jun 2021 15:08:40 GMT From: Warner Losh <imp@FreeBSD.org> To: doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org Subject: git: 8e3f48400a - main - arch-handbook: reformat scsi chapter one sentence per line before editing. Message-ID: <202106181508.15IF8efl037946@gitrepo.freebsd.org>
next in thread | raw e-mail | index | archive | help
The branch main has been updated by imp: URL: https://cgit.FreeBSD.org/doc/commit/?id=8e3f48400a61a87e4df4e3cdc0c9f3528b788e9d commit 8e3f48400a61a87e4df4e3cdc0c9f3528b788e9d Author: Warner Losh <imp@FreeBSD.org> AuthorDate: 2021-06-18 15:07:40 +0000 Commit: Warner Losh <imp@FreeBSD.org> CommitDate: 2021-06-18 15:07:54 +0000 arch-handbook: reformat scsi chapter one sentence per line before editing. --- .../en/books/arch-handbook/scsi/_index.adoc | 301 ++++++++++++++++----- 1 file changed, 227 insertions(+), 74 deletions(-) diff --git a/documentation/content/en/books/arch-handbook/scsi/_index.adoc b/documentation/content/en/books/arch-handbook/scsi/_index.adoc index 684ecd184c..c9a889df66 100644 --- a/documentation/content/en/books/arch-handbook/scsi/_index.adoc +++ b/documentation/content/en/books/arch-handbook/scsi/_index.adoc @@ -34,25 +34,37 @@ toc::[] [[scsi-synopsis]] == Synopsis -This document assumes that the reader has a general understanding of device drivers in FreeBSD and of the SCSI protocol. Much of the information in this document was extracted from the drivers: +This document assumes that the reader has a general understanding of device drivers in FreeBSD and of the SCSI protocol. +Much of the information in this document was extracted from the drivers: * ncr ([.filename]#/sys/pci/ncr.c#) by Wolfgang Stanglmeier and Stefan Esser * sym ([.filename]#/sys/dev/sym/sym_hipd.c#) by Gerard Roudier * aic7xxx ([.filename]#/sys/dev/aic7xxx/aic7xxx.c#) by Justin T. Gibbs -and from the CAM code itself (by Justin T. Gibbs, see [.filename]#/sys/cam/*#). When some solution looked the most logical and was essentially verbatim extracted from the code by Justin T. Gibbs, I marked it as "recommended". +and from the CAM code itself (by Justin T. Gibbs, see [.filename]#/sys/cam/*#). +When some solution looked the most logical and was essentially verbatim extracted from the code by Justin T. Gibbs, I marked it as "recommended". -The document is illustrated with examples in pseudo-code. Although sometimes the examples have many details and look like real code, it is still pseudo-code. It was written to demonstrate the concepts in an understandable way. For a real driver other approaches may be more modular and efficient. It also abstracts from the hardware details, as well as issues that would cloud the demonstrated concepts or that are supposed to be described in the other chapters of the developers handbook. Such details are commonly shown as calls to functions with descriptive names, comments or pseudo-statements. Fortunately real life full-size examples with all the details can be found in the real drivers. +The document is illustrated with examples in pseudo-code. +Although sometimes the examples have many details and look like real code, it is still pseudo-code. +It was written to demonstrate the concepts in an understandable way. +For a real driver other approaches may be more modular and efficient. +It also abstracts from the hardware details, as well as issues that would cloud the demonstrated concepts or that are supposed to be described in the other chapters of the developers handbook. +Such details are commonly shown as calls to functions with descriptive names, comments or pseudo-statements. +Fortunately real life full-size examples with all the details can be found in the real drivers. [[scsi-general]] == General Architecture -CAM stands for Common Access Method. It is a generic way to address the I/O buses in a SCSI-like way. This allows a separation of the generic device drivers from the drivers controlling the I/O bus: for example the disk driver becomes able to control disks on both SCSI, IDE, and/or any other bus so the disk driver portion does not have to be rewritten (or copied and modified) for every new I/O bus. Thus the two most important active entities are: +CAM stands for Common Access Method. +It is a generic way to address the I/O buses in a SCSI-like way. +This allows a separation of the generic device drivers from the drivers controlling the I/O bus: for example the disk driver becomes able to control disks on both SCSI, IDE, and/or any other bus so the disk driver portion does not have to be rewritten (or copied and modified) for every new I/O bus. +Thus the two most important active entities are: * _Peripheral Modules_ - a driver for peripheral devices (disk, tape, CD-ROM, etc.) * _SCSI Interface Modules_ (SIM) - a Host Bus Adapter drivers for connecting to an I/O bus such as SCSI or IDE. -A peripheral driver receives requests from the OS, converts them to a sequence of SCSI commands and passes these SCSI commands to a SCSI Interface Module. The SCSI Interface Module is responsible for passing these commands to the actual hardware (or if the actual hardware is not SCSI but, for example, IDE then also converting the SCSI commands to the native commands of the hardware). +A peripheral driver receives requests from the OS, converts them to a sequence of SCSI commands and passes these SCSI commands to a SCSI Interface Module. +The SCSI Interface Module is responsible for passing these commands to the actual hardware (or if the actual hardware is not SCSI but, for example, IDE then also converting the SCSI commands to the native commands of the hardware). As we are interested in writing a SCSI adapter driver here, from this point on we will consider everything from the SIM standpoint. @@ -68,7 +80,9 @@ A typical SIM driver needs to include the following CAM-related header files: #include <cam/scsi/scsi_all.h> .... -The first thing each SIM driver must do is register itself with the CAM subsystem. This is done during the driver's `xxx_attach()` function (here and further xxx_ is used to denote the unique driver name prefix). The `xxx_attach()` function itself is called by the system bus auto-configuration code which we do not describe here. +The first thing each SIM driver must do is register itself with the CAM subsystem. +This is done during the driver's `xxx_attach()` function (here and further xxx_ is used to denote the unique driver name prefix). +The `xxx_attach()` function itself is called by the system bus auto-configuration code which we do not describe here. This is achieved in multiple steps: first it is necessary to allocate the queue of requests associated with this SIM: @@ -81,7 +95,9 @@ This is achieved in multiple steps: first it is necessary to allocate the queue } .... -Here `SIZE` is the size of the queue to be allocated, maximal number of requests it could contain. It is the number of requests that the SIM driver can handle in parallel on one SCSI card. Commonly it can be calculated as: +Here `SIZE` is the size of the queue to be allocated, maximal number of requests it could contain. +It is the number of requests that the SIM driver can handle in parallel on one SCSI card. +Commonly it can be calculated as: [.programlisting] .... @@ -106,11 +122,12 @@ Note that if we are not able to create a SIM descriptor we free the `devq` also If a SCSI card has multiple SCSI buses on it then each bus requires its own `cam_sim` structure. -An interesting question is what to do if a SCSI card has more than one SCSI bus, do we need one `devq` structure per card or per SCSI bus? The answer given in the comments to the CAM code is: either way, as the driver's author prefers. +An interesting question is what to do if a SCSI card has more than one SCSI bus, do we need one `devq` structure per card or per SCSI bus? +The answer given in the comments to the CAM code is: either way, as the driver's author prefers. The arguments are: -* `action_func` - pointer to the driver's `xxx_action` function. +* `action_func` - pointer to the driver's `xxx_action` function. + [source,c] ---- @@ -127,11 +144,21 @@ static void (); ---- * driver_name - the name of the actual driver, such as "ncr" or "wds". -* `softc` - pointer to the driver's internal descriptor for this SCSI card. This pointer will be used by the driver in future to get private data. +* `softc` - pointer to the driver's internal descriptor for this SCSI card. +This pointer will be used by the driver in future to get private data. * unit - the controller unit number, for example for controller "mps0" this number will be 0 -* mtx - Lock associated with this SIM. For SIMs that don't know about locking, pass in Giant. For SIMs that do, pass in the lock used to guard this SIM's data structures. This lock will be held when xxx_action and xxx_poll are called. -* max_dev_transactions - maximal number of simultaneous transactions per SCSI target in the non-tagged mode. This value will be almost universally equal to 1, with possible exceptions only for the non-SCSI cards. Also the drivers that hope to take advantage by preparing one transaction while another one is executed may set it to 2 but this does not seem to be worth the complexity. -* max_tagged_dev_transactions - the same thing, but in the tagged mode. Tags are the SCSI way to initiate multiple transactions on a device: each transaction is assigned a unique tag and the transaction is sent to the device. When the device completes some transaction it sends back the result together with the tag so that the SCSI adapter (and the driver) can tell which transaction was completed. This argument is also known as the maximal tag depth. It depends on the abilities of the SCSI adapter. +* mtx - Lock associated with this SIM. +For SIMs that don't know about locking, pass in Giant. +For SIMs that do, pass in the lock used to guard this SIM's data structures. +This lock will be held when xxx_action and xxx_poll are called. +* max_dev_transactions - maximal number of simultaneous transactions per SCSI target in the non-tagged mode. +This value will be almost universally equal to 1, with possible exceptions only for the non-SCSI cards. +Also the drivers that hope to take advantage by preparing one transaction while another one is executed may set it to 2 but this does not seem to be worth the complexity. +* max_tagged_dev_transactions - the same thing, but in the tagged mode. +Tags are the SCSI way to initiate multiple transactions on a device: each transaction is assigned a unique tag and the transaction is sent to the device. +When the device completes some transaction it sends back the result together with the tag so that the SCSI adapter (and the driver) can tell which transaction was completed. +This argument is also known as the maximal tag depth. +It depends on the abilities of the SCSI adapter. Finally we register the SCSI buses associated with our SCSI adapter: @@ -143,13 +170,23 @@ Finally we register the SCSI buses associated with our SCSI adapter: } .... -If there is one `devq` structure per SCSI bus (i.e., we consider a card with multiple buses as multiple cards with one bus each) then the bus number will always be 0, otherwise each bus on the SCSI card should be get a distinct number. Each bus needs its own separate structure cam_sim. +If there is one `devq` structure per SCSI bus (i.e., we consider a card with multiple buses as multiple cards with one bus each) then the bus number will always be 0, otherwise each bus on the SCSI card should be get a distinct number. +Each bus needs its own separate structure cam_sim. -After that our controller is completely hooked to the CAM system. The value of `devq` can be discarded now: sim will be passed as an argument in all further calls from CAM and devq can be derived from it. +After that our controller is completely hooked to the CAM system. +The value of `devq` can be discarded now: sim will be passed as an argument in all further calls from CAM and devq can be derived from it. -CAM provides the framework for such asynchronous events. Some events originate from the lower levels (the SIM drivers), some events originate from the peripheral drivers, some events originate from the CAM subsystem itself. Any driver can register callbacks for some types of the asynchronous events, so that it would be notified if these events occur. +CAM provides the framework for such asynchronous events. +Some events originate from the lower levels (the SIM drivers), some events originate from the peripheral drivers, some events originate from the CAM subsystem itself. +Any driver can register callbacks for some types of the asynchronous events, so that it would be notified if these events occur. -A typical example of such an event is a device reset. Each transaction and event identifies the devices to which it applies by the means of "path". The target-specific events normally occur during a transaction with this device. So the path from that transaction may be re-used to report this event (this is safe because the event path is copied in the event reporting routine but not deallocated nor passed anywhere further). Also it is safe to allocate paths dynamically at any time including the interrupt routines, although that incurs certain overhead, and a possible problem with this approach is that there may be no free memory at that time. For a bus reset event we need to define a wildcard path including all devices on the bus. So we can create the path for the future bus reset events in advance and avoid problems with the future memory shortage: +A typical example of such an event is a device reset. +Each transaction and event identifies the devices to which it applies by the means of "path". +The target-specific events normally occur during a transaction with this device. +So the path from that transaction may be re-used to report this event (this is safe because the event path is copied in the event reporting routine but not deallocated nor passed anywhere further). +Also it is safe to allocate paths dynamically at any time including the interrupt routines, although that incurs certain overhead, and a possible problem with this approach is that there may be no free memory at that time. +For a bus reset event we need to define a wildcard path including all devices on the bus. +So we can create the path for the future bus reset events in advance and avoid problems with the future memory shortage: [.programlisting] .... @@ -176,11 +213,16 @@ As you can see the path includes: If the driver can not allocate this path it will not be able to work normally, so in that case we dismantle that SCSI bus. -And we save the path pointer in the `softc` structure for future use. After that we save the value of sim (or we can also discard it on the exit from `xxx_probe()` if we wish). +And we save the path pointer in the `softc` structure for future use. +After that we save the value of sim (or we can also discard it on the exit from `xxx_probe()` if we wish). -That is all for a minimalistic initialization. To do things right there is one more issue left. +That is all for a minimalistic initialization. +To do things right there is one more issue left. -For a SIM driver there is one particularly interesting event: when a target device is considered lost. In this case resetting the SCSI negotiations with this device may be a good idea. So we register a callback for this event with CAM. The request is passed to CAM by requesting CAM action on a CAM control block for this type of request: +For a SIM driver there is one particularly interesting event: when a target device is considered lost. +In this case resetting the SCSI negotiations with this device may be a good idea. +So we register a callback for this event with CAM. +The request is passed to CAM by requesting CAM action on a CAM control block for this type of request: [.programlisting] .... @@ -203,9 +245,14 @@ static void (); ---- -Do some action on request of the CAM subsystem. Sim describes the SIM for the request, CCB is the request itself. CCB stands for "CAM Control Block". It is a union of many specific instances, each describing arguments for some type of transactions. All of these instances share the CCB header where the common part of arguments is stored. +Do some action on request of the CAM subsystem. +Sim describes the SIM for the request, CCB is the request itself. +CCB stands for "CAM Control Block". +It is a union of many specific instances, each describing arguments for some type of transactions. +All of these instances share the CCB header where the common part of arguments is stored. -CAM supports the SCSI controllers working in both initiator ("normal") mode and target (simulating a SCSI device) mode. Here we only consider the part relevant to the initiator mode. +CAM supports the SCSI controllers working in both initiator ("normal") mode and target (simulating a SCSI device) mode. +Here we only consider the part relevant to the initiator mode. There are a few function and macros (in other words, methods) defined to access the public data in the struct sim: @@ -217,7 +264,8 @@ There are a few function and macros (in other words, methods) defined to access To identify the device, `xxx_action()` can get the unit number and pointer to its structure softc using these functions. -The type of request is stored in `ccb->ccb_h.func_code`. So generally `xxx_action()` consists of a big switch: +The type of request is stored in `ccb->ccb_h.func_code`. +So generally `xxx_action()` consists of a big switch: [.programlisting] .... @@ -238,17 +286,30 @@ The type of request is stored in `ccb->ccb_h.func_code`. So generally `xxx_actio As can be seen from the default case (if an unknown command was received) the return code of the command is set into `ccb->ccb_h.status` and the completed CCB is returned back to CAM by calling `xpt_done(ccb)`. -`xpt_done()` does not have to be called from `xxx_action()`: For example an I/O request may be enqueued inside the SIM driver and/or its SCSI controller. Then when the device would post an interrupt signaling that the processing of this request is complete `xpt_done()` may be called from the interrupt handling routine. - -Actually, the CCB status is not only assigned as a return code but a CCB has some status all the time. Before CCB is passed to the `xxx_action()` routine it gets the status CCB_REQ_INPROG meaning that it is in progress. There are a surprising number of status values defined in [.filename]#/sys/cam/cam.h# which should be able to represent the status of a request in great detail. More interesting yet, the status is in fact a "bitwise or" of an enumerated status value (the lower 6 bits) and possible additional flag-like bits (the upper bits). The enumerated values will be discussed later in more detail. The summary of them can be found in the Errors Summary section. The possible status flags are: - -* _CAM_DEV_QFRZN_ - if the SIM driver gets a serious error (for example, the device does not respond to the selection or breaks the SCSI protocol) when processing a CCB it should freeze the request queue by calling `xpt_freeze_simq()`, return the other enqueued but not processed yet CCBs for this device back to the CAM queue, then set this flag for the troublesome CCB and call `xpt_done()`. This flag causes the CAM subsystem to unfreeze the queue after it handles the error. -* _CAM_AUTOSNS_VALID_ - if the device returned an error condition and the flag CAM_DIS_AUTOSENSE is not set in CCB the SIM driver must execute the REQUEST SENSE command automatically to extract the sense (extended error information) data from the device. If this attempt was successful the sense data should be saved in the CCB and this flag set. -* _CAM_RELEASE_SIMQ_ - like CAM_DEV_QFRZN but used in case there is some problem (or resource shortage) with the SCSI controller itself. Then all the future requests to the controller should be stopped by `xpt_freeze_simq()`. The controller queue will be restarted after the SIM driver overcomes the shortage and informs CAM by returning some CCB with this flag set. -* _CAM_SIM_QUEUED_ - when SIM puts a CCB into its request queue this flag should be set (and removed when this CCB gets dequeued before being returned back to CAM). This flag is not used anywhere in the CAM code now, so its purpose is purely diagnostic. +`xpt_done()` does not have to be called from `xxx_action()`: For example an I/O request may be enqueued inside the SIM driver and/or its SCSI controller. +Then when the device would post an interrupt signaling that the processing of this request is complete `xpt_done()` may be called from the interrupt handling routine. + +Actually, the CCB status is not only assigned as a return code but a CCB has some status all the time. +Before CCB is passed to the `xxx_action()` routine it gets the status CCB_REQ_INPROG meaning that it is in progress. +There are a surprising number of status values defined in [.filename]#/sys/cam/cam.h# which should be able to represent the status of a request in great detail. +More interesting yet, the status is in fact a "bitwise or" of an enumerated status value (the lower 6 bits) and possible additional flag-like bits (the upper bits). +The enumerated values will be discussed later in more detail. +The summary of them can be found in the Errors Summary section. +The possible status flags are: + +* _CAM_DEV_QFRZN_ - if the SIM driver gets a serious error (for example, the device does not respond to the selection or breaks the SCSI protocol) when processing a CCB it should freeze the request queue by calling `xpt_freeze_simq()`, return the other enqueued but not processed yet CCBs for this device back to the CAM queue, then set this flag for the troublesome CCB and call `xpt_done()`. +This flag causes the CAM subsystem to unfreeze the queue after it handles the error. +* _CAM_AUTOSNS_VALID_ - if the device returned an error condition and the flag CAM_DIS_AUTOSENSE is not set in CCB the SIM driver must execute the REQUEST SENSE command automatically to extract the sense (extended error information) data from the device. +If this attempt was successful the sense data should be saved in the CCB and this flag set. +* _CAM_RELEASE_SIMQ_ - like CAM_DEV_QFRZN but used in case there is some problem (or resource shortage) with the SCSI controller itself. +Then all the future requests to the controller should be stopped by `xpt_freeze_simq()`. +The controller queue will be restarted after the SIM driver overcomes the shortage and informs CAM by returning some CCB with this flag set. +* _CAM_SIM_QUEUED_ - when SIM puts a CCB into its request queue this flag should be set (and removed when this CCB gets dequeued before being returned back to CAM). +This flag is not used anywhere in the CAM code now, so its purpose is purely diagnostic. * _CAM_QOS_VALID_ - The QOS data is now valid. -The function `xxx_action()` is not allowed to sleep, so all the synchronization for resource access must be done using SIM or device queue freezing. Besides the aforementioned flags the CAM subsystem provides functions `xpt_release_simq()` and `xpt_release_devq()` to unfreeze the queues directly, without passing a CCB to CAM. +The function `xxx_action()` is not allowed to sleep, so all the synchronization for resource access must be done using SIM or device queue freezing. +Besides the aforementioned flags the CAM subsystem provides functions `xpt_release_simq()` and `xpt_release_devq()` to unfreeze the queues directly, without passing a CCB to CAM. The CCB header contains the following fields: @@ -271,7 +332,8 @@ The most common initiator mode requests are: * _XPT_SCSI_IO_ - execute an I/O transaction + -The instance "struct ccb_scsiio csio" of the union ccb is used to transfer the arguments. They are: +The instance "struct ccb_scsiio csio" of the union ccb is used to transfer the arguments. +They are: ** _cdb_io_ - pointer to the SCSI command buffer or the buffer itself ** _cdb_len_ - SCSI command length @@ -280,7 +342,9 @@ The instance "struct ccb_scsiio csio" of the union ccb is used to transfer the a ** _sglist_cnt_ - counter of the scatter/gather segments ** _scsi_status_ - place to return the SCSI status ** _sense_data_ - buffer for the SCSI sense information if the command returns an error (the SIM driver is supposed to run the REQUEST SENSE command automatically in this case if the CCB flag CAM_DIS_AUTOSENSE is not set) -** _sense_len_ - the length of that buffer (if it happens to be higher than size of sense_data the SIM driver must silently assume the smaller value) resid, sense_resid - if the transfer of data or SCSI sense returned an error these are the returned counters of the residual (not transferred) data. They do not seem to be especially meaningful, so in a case when they are difficult to compute (say, counting bytes in the SCSI controller's FIFO buffer) an approximate value will do as well. For a successfully completed transfer they must be set to zero. +** _sense_len_ - the length of that buffer (if it happens to be higher than size of sense_data the SIM driver must silently assume the smaller value) resid, sense_resid - if the transfer of data or SCSI sense returned an error these are the returned counters of the residual (not transferred) data. +They do not seem to be especially meaningful, so in a case when they are difficult to compute (say, counting bytes in the SCSI controller's FIFO buffer) an approximate value will do as well. +For a successfully completed transfer they must be set to zero. ** _tag_action_ - the kind of tag to use: *** CAM_TAG_ACTION_NONE - do not use tags for this transaction @@ -317,7 +381,10 @@ Also we check that the device is supported at all by our controller: } .... + -Then allocate whatever data structures (such as card-dependent hardware control block) we need to process this request. If we can not then freeze the SIM queue and remember that we have a pending operation, return the CCB back and ask CAM to re-queue it. Later when the resources become available the SIM queue must be unfrozen by returning a ccb with the `CAM_SIMQ_RELEASE` bit set in its status. Otherwise, if all went well, link the CCB with the hardware control block (HCB) and mark it as queued. +Then allocate whatever data structures (such as card-dependent hardware control block) we need to process this request. +If we can not then freeze the SIM queue and remember that we have a pending operation, return the CCB back and ask CAM to re-queue it. +Later when the resources become available the SIM queue must be unfrozen by returning a ccb with the `CAM_SIMQ_RELEASE` bit set in its status. +Otherwise, if all went well, link the CCB with the hardware control block (HCB) and mark it as queued. + [.programlisting] .... @@ -335,7 +402,9 @@ Then allocate whatever data structures (such as card-dependent hardware control ccb_h->status |= CAM_SIM_QUEUED; .... + -Extract the target data from CCB into the hardware control block. Check if we are asked to assign a tag and if yes then generate an unique tag and build the SCSI tag messages. The SIM driver is also responsible for negotiations with the devices to set the maximal mutually supported bus width, synchronous rate and offset. +Extract the target data from CCB into the hardware control block. +Check if we are asked to assign a tag and if yes then generate an unique tag and build the SCSI tag messages. +The SIM driver is also responsible for negotiations with the devices to set the maximal mutually supported bus width, synchronous rate and offset. + [.programlisting] .... @@ -347,9 +416,13 @@ Extract the target data from CCB into the hardware control block. Check if we ar generate_negotiation_messages(hcb); .... + -Then set up the SCSI command. The command storage may be specified in the CCB in many interesting ways, specified by the CCB flags. The command buffer can be contained in CCB or pointed to, in the latter case the pointer may be physical or virtual. Since the hardware commonly needs physical address we always convert the address to the physical one, typically using the busdma API. +Then set up the SCSI command. +The command storage may be specified in the CCB in many interesting ways, specified by the CCB flags. +The command buffer can be contained in CCB or pointed to, in the latter case the pointer may be physical or virtual. +Since the hardware commonly needs physical address we always convert the address to the physical one, typically using the busdma API. + -In case if a physical address is requested it is OK to return the CCB with the status `CAM_REQ_INVALID`, the current drivers do that. If necessary a physical address can be also converted or mapped back to a virtual address but with big pain, so we do not do that. +In case if a physical address is requested it is OK to return the CCB with the status `CAM_REQ_INVALID`, the current drivers do that. +If necessary a physical address can be also converted or mapped back to a virtual address but with big pain, so we do not do that. + [.programlisting] .... @@ -369,7 +442,10 @@ In case if a physical address is requested it is OK to return the CCB with the s hcb->cmdlen = csio->cdb_len; .... + -Now it is time to set up the data. Again, the data storage may be specified in the CCB in many interesting ways, specified by the CCB flags. First we get the direction of the data transfer. The simplest case is if there is no data to transfer: +Now it is time to set up the data. +Again, the data storage may be specified in the CCB in many interesting ways, specified by the CCB flags. +First we get the direction of the data transfer. +The simplest case is if there is no data to transfer: + [.programlisting] .... @@ -379,7 +455,15 @@ Now it is time to set up the data. Again, the data storage may be specified in t goto end_data; .... + -Then we check if the data is in one chunk or in a scatter-gather list, and the addresses are physical or virtual. The SCSI controller may be able to handle only a limited number of chunks of limited length. If the request hits this limitation we return an error. We use a special function to return the CCB to handle in one place the HCB resource shortages. The functions to add chunks are driver-dependent, and here we leave them without detailed implementation. See description of the SCSI command (CDB) handling for the details on the address-translation issues. If some variation is too difficult or impossible to implement with a particular card it is OK to return the status `CAM_REQ_INVALID`. Actually, it seems like the scatter-gather ability is not used anywhere in the CAM code now. But at least the case for a single non-scattered virtual buffer must be implemented, it is actively used by CAM. +Then we check if the data is in one chunk or in a scatter-gather list, and the addresses are physical or virtual. +The SCSI controller may be able to handle only a limited number of chunks of limited length. +If the request hits this limitation we return an error. +We use a special function to return the CCB to handle in one place the HCB resource shortages. +The functions to add chunks are driver-dependent, and here we leave them without detailed implementation. +See description of the SCSI command (CDB) handling for the details on the address-translation issues. +If some variation is too difficult or impossible to implement with a particular card it is OK to return the status `CAM_REQ_INVALID`. +Actually, it seems like the scatter-gather ability is not used anywhere in the CAM code now. +But at least the case for a single non-scattered virtual buffer must be implemented, it is actively used by CAM. + [.programlisting] .... @@ -480,16 +564,20 @@ And here is a possible implementation of the function returning CCB: * _XPT_RESET_DEV_ - send the SCSI "BUS DEVICE RESET" message to a device + -There is no data transferred in CCB except the header and the most interesting argument of it is target_id. Depending on the controller hardware a hardware control block just like for the XPT_SCSI_IO request may be constructed (see XPT_SCSI_IO request description) and sent to the controller or the SCSI controller may be immediately programmed to send this RESET message to the device or this request may be just not supported (and return the status `CAM_REQ_INVALID`). Also on completion of the request all the disconnected transactions for this target must be aborted (probably in the interrupt routine). +There is no data transferred in CCB except the header and the most interesting argument of it is target_id. +Depending on the controller hardware a hardware control block just like for the XPT_SCSI_IO request may be constructed (see XPT_SCSI_IO request description) and sent to the controller or the SCSI controller may be immediately programmed to send this RESET message to the device or this request may be just not supported (and return the status `CAM_REQ_INVALID`). +Also on completion of the request all the disconnected transactions for this target must be aborted (probably in the interrupt routine). + -Also all the current negotiations for the target are lost on reset, so they might be cleaned too. Or they clearing may be deferred, because anyway the target would request re-negotiation on the next transaction. +Also all the current negotiations for the target are lost on reset, so they might be cleaned too. +Or they clearing may be deferred, because anyway the target would request re-negotiation on the next transaction. * _XPT_RESET_BUS_ - send the RESET signal to the SCSI bus + No arguments are passed in the CCB, the only interesting argument is the SCSI bus indicated by the struct sim pointer. + A minimalistic implementation would forget the SCSI negotiations for all the devices on the bus and return the status CAM_REQ_CMP. + -The proper implementation would in addition actually reset the SCSI bus (possible also reset the SCSI controller) and mark all the CCBs being processed, both those in the hardware queue and those being disconnected, as done with the status CAM_SCSI_BUS_RESET. Like: +The proper implementation would in addition actually reset the SCSI bus (possible also reset the SCSI controller) and mark all the CCBs being processed, both those in the hardware queue and those being disconnected, as done with the status CAM_SCSI_BUS_RESET. +Like: + [.programlisting] .... @@ -546,13 +634,16 @@ The proper implementation would in addition actually reset the SCSI bus (possibl Implementing the SCSI bus reset as a function may be a good idea because it would be re-used by the timeout function as a last resort if the things go wrong. * _XPT_ABORT_ - abort the specified CCB + -The arguments are transferred in the instance "struct ccb_abort cab" of the union ccb. The only argument field in it is: +The arguments are transferred in the instance "struct ccb_abort cab" of the union ccb. +The only argument field in it is: + _abort_ccb_ - pointer to the CCB to be aborted + -If the abort is not supported just return the status CAM_UA_ABORT. This is also the easy way to minimally implement this call, return CAM_UA_ABORT in any case. +If the abort is not supported just return the status CAM_UA_ABORT. +This is also the easy way to minimally implement this call, return CAM_UA_ABORT in any case. + -The hard way is to implement this request honestly. First check that abort applies to a SCSI transaction: +The hard way is to implement this request honestly. +First check that abort applies to a SCSI transaction: + [.programlisting] .... @@ -567,7 +658,8 @@ The hard way is to implement this request honestly. First check that abort appli .... + -Then it is necessary to find this CCB in our queue. This can be done by walking the list of all our hardware control blocks in search for one associated with this CCB: +Then it is necessary to find this CCB in our queue. +This can be done by walking the list of all our hardware control blocks in search for one associated with this CCB: + [.programlisting] .... @@ -596,7 +688,9 @@ Then it is necessary to find this CCB in our queue. This can be done by walking hcb=found_hcb; .... + -Now we look at the current processing status of the HCB. It may be either sitting in the queue waiting to be sent to the SCSI bus, being transferred right now, or disconnected and waiting for the result of the command, or actually completed by hardware but not yet marked as done by software. To make sure that we do not get in any races with hardware we mark the HCB as being aborted, so that if this HCB is about to be sent to the SCSI bus the SCSI controller will see this flag and skip it. +Now we look at the current processing status of the HCB. +It may be either sitting in the queue waiting to be sent to the SCSI bus, being transferred right now, or disconnected and waiting for the result of the command, or actually completed by hardware but not yet marked as done by software. +To make sure that we do not get in any races with hardware we mark the HCB as being aborted, so that if this HCB is about to be sent to the SCSI bus the SCSI controller will see this flag and skip it. + [.programlisting] .... @@ -620,7 +714,11 @@ Now we look at the current processing status of the HCB. It may be either sittin break; .... + -If the CCB is being transferred right now we would like to signal to the SCSI controller in some hardware-dependent way that we want to abort the current transfer. The SCSI controller would set the SCSI ATTENTION signal and when the target responds to it send an ABORT message. We also reset the timeout to make sure that the target is not sleeping forever. If the command would not get aborted in some reasonable time like 10 seconds the timeout routine would go ahead and reset the whole SCSI bus. Since the command will be aborted in some reasonable time we can just return the abort request now as successfully completed, and mark the aborted CCB as aborted (but not mark it as done yet). +If the CCB is being transferred right now we would like to signal to the SCSI controller in some hardware-dependent way that we want to abort the current transfer. +The SCSI controller would set the SCSI ATTENTION signal and when the target responds to it send an ABORT message. +We also reset the timeout to make sure that the target is not sleeping forever. +If the command would not get aborted in some reasonable time like 10 seconds the timeout routine would go ahead and reset the whole SCSI bus. +Since the command will be aborted in some reasonable time we can just return the abort request now as successfully completed, and mark the aborted CCB as aborted (but not mark it as done yet). + [.programlisting] .... @@ -641,7 +739,8 @@ If the CCB is being transferred right now we would like to signal to the SCSI co break; .... + -If the CCB is in the list of disconnected then set it up as an abort request and re-queue it at the front of hardware queue. Reset the timeout and report the abort request to be completed. +If the CCB is in the list of disconnected then set it up as an abort request and re-queue it at the front of hardware queue. +Reset the timeout and report the abort request to be completed. + [.programlisting] .... @@ -658,9 +757,13 @@ If the CCB is in the list of disconnected then set it up as an abort request and return; .... + -That is all for the ABORT request, although there is one more issue. As the ABORT message cleans all the ongoing transactions on a LUN we have to mark all the other active transactions on this LUN as aborted. That should be done in the interrupt routine, after the transaction gets aborted. +That is all for the ABORT request, although there is one more issue. +As the ABORT message cleans all the ongoing transactions on a LUN we have to mark all the other active transactions on this LUN as aborted. +That should be done in the interrupt routine, after the transaction gets aborted. + -Implementing the CCB abort as a function may be quite a good idea, this function can be re-used if an I/O transaction times out. The only difference would be that the timed out transaction would return the status CAM_CMD_TIMEOUT for the timed out request. Then the case XPT_ABORT would be small, like that: +Implementing the CCB abort as a function may be quite a good idea, this function can be re-used if an I/O transaction times out. +The only difference would be that the timed out transaction would return the status CAM_CMD_TIMEOUT for the timed out request. +Then the case XPT_ABORT would be small, like that: + [.programlisting] .... @@ -692,7 +795,8 @@ The arguments are transferred in the instance "struct ccb_trans_setting cts" of ** _CCB_TRANS_BUS_WIDTH_VALID_ - bus width ** _CCB_TRANS_DISC_VALID_ - set enable/disable disconnection ** _CCB_TRANS_TQ_VALID_ - set enable/disable tagged queuing -** _flags_ - consists of two parts, binary arguments and identification of sub-operations. The binary arguments are: +** _flags_ - consists of two parts, binary arguments and identification of sub-operations. +The binary arguments are: *** _CCB_TRANS_DISC_ENB_ - enable disconnection *** _CCB_TRANS_TAG_ENB_ - enable tagged queuing @@ -702,9 +806,15 @@ The arguments are transferred in the instance "struct ccb_trans_setting cts" of *** _CCB_TRANS_CURRENT_SETTINGS_ - change the current negotiations *** _CCB_TRANS_USER_SETTINGS_ - remember the desired user values sync_period, sync_offset - self-explanatory, if sync_offset==0 then the asynchronous mode is requested bus_width - bus width, in bits (not bytes) + -Two sets of negotiated parameters are supported, the user settings and the current settings. The user settings are not really used much in the SIM drivers, this is mostly just a piece of memory where the upper levels can store (and later recall) its ideas about the parameters. Setting the user parameters does not cause re-negotiation of the transfer rates. But when the SCSI controller does a negotiation it must never set the values higher than the user parameters, so it is essentially the top boundary. +Two sets of negotiated parameters are supported, the user settings and the current settings. +The user settings are not really used much in the SIM drivers, this is mostly just a piece of memory where the upper levels can store (and later recall) its ideas about the parameters. +Setting the user parameters does not cause re-negotiation of the transfer rates. +But when the SCSI controller does a negotiation it must never set the values higher than the user parameters, so it is essentially the top boundary. + -The current settings are, as the name says, current. Changing them means that the parameters must be re-negotiated on the next transfer. Again, these "new current settings" are not supposed to be forced on the device, just they are used as the initial step of negotiations. Also they must be limited by actual capabilities of the SCSI controller: for example, if the SCSI controller has 8-bit bus and the request asks to set 16-bit wide transfers this parameter must be silently truncated to 8-bit transfers before sending it to the device. +The current settings are, as the name says, current. +Changing them means that the parameters must be re-negotiated on the next transfer. +Again, these "new current settings" are not supposed to be forced on the device, just they are used as the initial step of negotiations. +Also they must be limited by actual capabilities of the SCSI controller: for example, if the SCSI controller has 8-bit bus and the request asks to set 16-bit wide transfers this parameter must be silently truncated to 8-bit transfers before sending it to the device. + One caveat is that the bus width and synchronous parameters are per target while the disconnection and tag enabling parameters are per lun. + @@ -767,7 +877,8 @@ The code looks like: return; .... + -Then when the next I/O request will be processed it will check if it has to re-negotiate, for example by calling the function target_negotiated(hcb). It can be implemented like this: +Then when the next I/O request will be processed it will check if it has to re-negotiate, for example by calling the function target_negotiated(hcb). +It can be implemented like this: + [.programlisting] .... @@ -786,11 +897,14 @@ Then when the next I/O request will be processed it will check if it has to re-n } .... + -After the values are re-negotiated the resulting values must be assigned to both current and goal parameters, so for future I/O transactions the current and goal parameters would be the same and `target_negotiated()` would return TRUE. When the card is initialized (in `xxx_attach()`) the current negotiation values must be initialized to narrow asynchronous mode, the goal and current values must be initialized to the maximal values supported by controller. +After the values are re-negotiated the resulting values must be assigned to both current and goal parameters, so for future I/O transactions the current and goal parameters would be the same and `target_negotiated()` would return TRUE. +When the card is initialized (in `xxx_attach()`) the current negotiation values must be initialized to narrow asynchronous mode, the goal and current values must be initialized to the maximal values supported by controller. + _XPT_GET_TRAN_SETTINGS_ - get values of SCSI transfer settings + -This operations is the reverse of XPT_SET_TRAN_SETTINGS. Fill up the CCB instance "struct ccb_trans_setting cts" with data as requested by the flags CCB_TRANS_CURRENT_SETTINGS or CCB_TRANS_USER_SETTINGS (if both are set then the existing drivers return the current settings). Set all the bits in the valid field. +This operations is the reverse of XPT_SET_TRAN_SETTINGS. +Fill up the CCB instance "struct ccb_trans_setting cts" with data as requested by the flags CCB_TRANS_CURRENT_SETTINGS or CCB_TRANS_USER_SETTINGS (if both are set then the existing drivers return the current settings). +Set all the bits in the valid field. + _XPT_CALC_GEOMETRY_ - calculate logical (BIOS) geometry of the disk + @@ -802,7 +916,8 @@ The arguments are transferred in the instance "struct ccb_calc_geometry ccg" of ** _heads_ - output, logical heads ** _secs_per_track_ - output, logical sectors per track + -If the returned geometry differs much enough from what the SCSI controller BIOS thinks and a disk on this SCSI controller is used as bootable the system may not be able to boot. The typical calculation example taken from the aic7xxx driver is: +If the returned geometry differs much enough from what the SCSI controller BIOS thinks and a disk on this SCSI controller is used as bootable the system may not be able to boot. +The typical calculation example taken from the aic7xxx driver is: + [.programlisting] .... @@ -830,7 +945,9 @@ If the returned geometry differs much enough from what the SCSI controller BIOS return; .... + -This gives the general idea, the exact calculation depends on the quirks of the particular BIOS. If BIOS provides no way set the "extended translation" flag in EEPROM this flag should normally be assumed equal to 1. Other popular geometries are: +This gives the general idea, the exact calculation depends on the quirks of the particular BIOS. +If BIOS provides no way set the "extended translation" flag in EEPROM this flag should normally be assumed equal to 1. +Other popular geometries are: + [.programlisting] .... @@ -891,7 +1008,13 @@ static void (); ---- -The poll function is used to simulate the interrupts when the interrupt subsystem is not functioning (for example, when the system has crashed and is creating the system dump). The CAM subsystem sets the proper interrupt level before calling the poll routine. So all it needs to do is to call the interrupt routine (or the other way around, the poll routine may be doing the real action and the interrupt routine would just call the poll routine). Why bother about a separate function then? This has to do with different calling conventions. The `xxx_poll` routine gets the struct cam_sim pointer as its argument while the PCI interrupt routine by common convention gets pointer to the struct `xxx_softc` and the ISA interrupt routine gets just the device unit number. So the poll routine would normally look as: +The poll function is used to simulate the interrupts when the interrupt subsystem is not functioning (for example, when the system has crashed and is creating the system dump). +The CAM subsystem sets the proper interrupt level before calling the poll routine. +So all it needs to do is to call the interrupt routine (or the other way around, the poll routine may be doing the real action and the interrupt routine would just call the poll routine). +Why bother about a separate function then? +This has to do with different calling conventions. +The `xxx_poll` routine gets the struct cam_sim pointer as its argument while the PCI interrupt routine by common convention gets pointer to the struct `xxx_softc` and the ISA interrupt routine gets just the device unit number. +So the poll routine would normally look as: [.programlisting] .... @@ -963,7 +1086,12 @@ Implementation for a single type of event, AC_LOST_DEVICE, looks like: The exact type of the interrupt routine depends on the type of the peripheral bus (PCI, ISA and so on) to which the SCSI controller is connected. -The interrupt routines of the SIM drivers run at the interrupt level splcam. So `splcam()` should be used in the driver to synchronize activity between the interrupt routine and the rest of the driver (for a multiprocessor-aware driver things get yet more interesting but we ignore this case here). The pseudo-code in this document happily ignores the problems of synchronization. The real code must not ignore them. A simple-minded approach is to set `splcam()` on the entry to the other routines and reset it on return thus protecting them by one big critical section. To make sure that the interrupt level will be always restored a wrapper function can be defined, like: +The interrupt routines of the SIM drivers run at the interrupt level splcam. +So `splcam()` should be used in the driver to synchronize activity between the interrupt routine and the rest of the driver (for a multiprocessor-aware driver things get yet more interesting but we ignore this case here). +The pseudo-code in this document happily ignores the problems of synchronization. +The real code must not ignore them. +A simple-minded approach is to set `splcam()` on the entry to the other routines and reset it on return thus protecting them by one big critical section. +To make sure that the interrupt level will be always restored a wrapper function can be defined, like: [.programlisting] .... @@ -983,11 +1111,16 @@ The interrupt routines of the SIM drivers run at the interrupt level splcam. So } .... -This approach is simple and robust but the problem with it is that interrupts may get blocked for a relatively long time and this would negatively affect the system's performance. On the other hand the functions of the `spl()` family have rather high overhead, so vast amount of tiny critical sections may not be good either. +This approach is simple and robust but the problem with it is that interrupts may get blocked for a relatively long time and this would negatively affect the system's performance. +On the other hand the functions of the `spl()` family have rather high overhead, so vast amount of tiny critical sections may not be good either. -The conditions handled by the interrupt routine and the details depend very much on the hardware. We consider the set of "typical" conditions. +The conditions handled by the interrupt routine and the details depend very much on the hardware. +We consider the set of "typical" conditions. -First, we check if a SCSI reset was encountered on the bus (probably caused by another SCSI controller on the same SCSI bus). If so we drop all the enqueued and disconnected requests, report the events and re-initialize our SCSI controller. It is important that during this initialization the controller will not issue another reset or else two controllers on the same SCSI bus could ping-pong resets forever. The case of fatal controller error/hang could be handled in the same place, but it will probably need also sending RESET signal to the SCSI bus to reset the status of the connections with the SCSI devices. +First, we check if a SCSI reset was encountered on the bus (probably caused by another SCSI controller on the same SCSI bus). +If so we drop all the enqueued and disconnected requests, report the events and re-initialize our SCSI controller. +It is important that during this initialization the controller will not issue another reset or else two controllers on the same SCSI bus could ping-pong resets forever. +The case of fatal controller error/hang could be handled in the same place, but it will probably need also sending RESET signal to the SCSI bus to reset the status of the connections with the SCSI devices. [.programlisting] .... @@ -1052,7 +1185,9 @@ First, we check if a SCSI reset was encountered on the bus (probably caused by a } .... -If interrupt is not caused by a controller-wide condition then probably something has happened to the current hardware control block. Depending on the hardware there may be other non-HCB-related events, we just do not consider them here. Then we analyze what happened to this HCB: +If interrupt is not caused by a controller-wide condition then probably something has happened to the current hardware control block. +Depending on the hardware there may be other non-HCB-related events, we just do not consider them here. +Then we analyze what happened to this HCB: [.programlisting] .... @@ -1100,7 +1235,8 @@ Then look if this status is related to the REQUEST SENSE command and if so handl } .... -Else the command itself has completed, pay more attention to details. If auto-sense is not disabled for this CCB and the command has failed with sense data then run REQUEST SENSE command to receive that data. +Else the command itself has completed, pay more attention to details. +If auto-sense is not disabled for this CCB and the command has failed with sense data then run REQUEST SENSE command to receive that data. [.programlisting] .... @@ -1182,7 +1318,8 @@ One typical thing would be negotiation events: negotiation messages received fro } .... -Then we handle any errors that could have happened during auto-sense in the same simple-minded way as before. Otherwise we look closer at the details again. +Then we handle any errors that could have happened during auto-sense in the same simple-minded way as before. +Otherwise we look closer at the details again. [.programlisting] .... @@ -1192,7 +1329,8 @@ Then we handle any errors that could have happened during auto-sense in the same switch(hcb_status) { .... -The next event we consider is unexpected disconnect. Which is considered normal after an ABORT or BUS DEVICE RESET message and abnormal in other cases. +The next event we consider is unexpected disconnect. +Which is considered normal after an ABORT or BUS DEVICE RESET message and abnormal in other cases. [.programlisting] .... @@ -1308,9 +1446,13 @@ This concludes the generic interrupt handling although specific controllers may [[scsi-errors]] == Errors Summary -When executing an I/O request many things may go wrong. The reason of error can be reported in the CCB status with great detail. Examples of use are spread throughout this document. For completeness here is the summary of recommended responses for the typical error conditions: +When executing an I/O request many things may go wrong. +The reason of error can be reported in the CCB status with great detail. +Examples of use are spread throughout this document. +For completeness here is the summary of recommended responses for the typical error conditions: -* _CAM_RESRC_UNAVAIL_ - some resource is temporarily unavailable and the SIM driver cannot generate an event when it will become available. An example of this resource would be some intra-controller hardware resource for which the controller does not generate an interrupt when it becomes available. +* _CAM_RESRC_UNAVAIL_ - some resource is temporarily unavailable and the SIM driver cannot generate an event when it will become available. +An example of this resource would be some intra-controller hardware resource for which the controller does not generate an interrupt when it becomes available. * _CAM_UNCOR_PARITY_ - unrecovered parity error occurred * _CAM_DATA_RUN_ERR_ - data overrun or unexpected data phase (going in other direction than specified in CAM_DIR_MASK) or odd transfer length for wide transfer * _CAM_SEL_TIMEOUT_ - selection timeout occurred (target does not respond) @@ -1324,14 +1466,22 @@ When executing an I/O request many things may go wrong. The reason of error can * _CAM_BDR_SENT_ - BUS DEVICE RESET message was sent to the target * _CAM_UNREC_HBA_ERROR_ - unrecoverable Host Bus Adapter Error * _CAM_REQ_TOO_BIG_ - the request was too large for this controller -* _CAM_REQUEUE_REQ_ - this request should be re-queued to preserve transaction ordering. This typically occurs when the SIM recognizes an error that should freeze the queue and must place other queued requests for the target at the sim level back into the XPT queue. Typical cases of such errors are selection timeouts, command timeouts and other like conditions. In such cases the troublesome command returns the status indicating the error, the and the other commands which have not be sent to the bus yet get re-queued. +* _CAM_REQUEUE_REQ_ - this request should be re-queued to preserve transaction ordering. +This typically occurs when the SIM recognizes an error that should freeze the queue and must place other queued requests for the target at the sim level back into the XPT queue. +Typical cases of such errors are selection timeouts, command timeouts and other like conditions. +In such cases the troublesome command returns the status indicating the error, the and the other commands which have not be sent to the bus yet get re-queued. * _CAM_LUN_INVALID_ - the LUN ID in the request is not supported by the SCSI controller * _CAM_TID_INVALID_ - the target ID in the request is not supported by the SCSI controller [[scsi-timeout]] == Timeout Handling -When the timeout for an HCB expires that request should be aborted, just like with an XPT_ABORT request. The only difference is that the returned status of aborted request should be CAM_CMD_TIMEOUT instead of CAM_REQ_ABORTED (that is why implementation of the abort better be done as a function). But there is one more possible problem: what if the abort request itself will get stuck? In this case the SCSI bus should be reset, just like with an XPT_RESET_BUS request (and the idea about implementing it as a function called from both places applies here too). Also we should reset the whole SCSI bus if a device reset request got stuck. So after all the timeout function would look like: +When the timeout for an HCB expires that request should be aborted, just like with an XPT_ABORT request. +The only difference is that the returned status of aborted request should be CAM_CMD_TIMEOUT instead of CAM_REQ_ABORTED (that is why implementation of the abort better be done as a function). +But there is one more possible problem: what if the abort request itself will get stuck? +In this case the SCSI bus should be reset, just like with an XPT_RESET_BUS request (and the idea about implementing it as a function called from both places applies here too). +Also we should reset the whole SCSI bus if a device reset request got stuck. +So after all the timeout function would look like: [.programlisting] .... @@ -1354,4 +1504,7 @@ xxx_timeout(void *arg) } .... -When we abort a request all the other disconnected requests to the same target/LUN get aborted too. So there appears a question, should we return them with status CAM_REQ_ABORTED or CAM_CMD_TIMEOUT? The current drivers use CAM_CMD_TIMEOUT. This seems logical because if one request got timed out then probably something really bad is happening to the device, so if they would not be disturbed they would time out by themselves. +When we abort a request all the other disconnected requests to the same target/LUN get aborted too. +So there appears a question, should we return them with status CAM_REQ_ABORTED or CAM_CMD_TIMEOUT? +The current drivers use CAM_CMD_TIMEOUT. +This seems logical because if one request got timed out then probably something really bad is happening to the device, so if they would not be disturbed they would time out by themselves.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?202106181508.15IF8efl037946>