Date: Sun, 03 Mar 2002 12:17:27 -0800 From: Julian Elischer <julian@elischer.org> To: Maksim Yevmenkin <e.max@verizon.net> Cc: freebsd-current@freebsd.org, archie@freebsd.org Subject: Re: Netgraph, device drivers and mutexes Message-ID: <3C8284D7.C5A8A893@elischer.org> References: <3C827021.A7FA73A4@verizon.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Maksim Yevmenkin wrote:
>
> Hackers,
Ah here we have teh one part of -curretn netgraph that is not
yet fully implemented..
Netgraph has it's own locking, using a spinlock in the worst case
but trying like crazy to avoid havong to use it. It implememts
a reader-writer (shared/exclusive) locking scheme where a failure
to gain the lock does not result in a blocking action but rather,
results an in element being queued for later processing.
All "message" type requests are by default trying to get "write"
(exclusive) locks, and data is by default trying to get "read"
(shared) locks. (These default behaviours can be over-ridden by the node
if it knows for example that it's data operations need to
change some state that means that they must have exclusive access to
the node state or someo other resource).
This is all build in to the system so if you are communicating with the
node only by using messages (the suggested method) then there is no
more locking that you need to do.
The problem come however when external code wants ot interact with
netgraph nodes. In this situation you should use the following
technique.
You specify a function that will do the work that you require.
It should be of the form
typedef void ng_item_fn(node_p node, hook_p hook, void *arg1, int arg2);
(though you shouldn't use the typedef)
external code can call
int ng_send_fn(node_p node, hook_p hook, ng_item_fn *fn,
void *arg1, int arg2);
If it can gain the lock it will immediatly run function, but if it can
NOT gain the lock, it will queue it to be run as soon as the lock is released.
If you wish to use a timeout then your actual timeout code should call this
function to do the work you wish to be done.
Note there is a little more to it..
While the timeout is valid you MUST hold a reference on the node
using NG_NODE_REF() and IF you supply a hook argument (it's optional)
you MUST hold a reference on that too using NG_HOOK_REF().
When your timeout function is called, you can drop these references
using NG_NODE_UNREF() and NG_HOOK_UNREF() AFTER you have called
ng_send_fn(). This is to ensure that the timeout code doesnot try to
access a node that has been freed. (or a hook). In the case where
ng_send_fn() needs to queue the request it will take out its own
references so you cna then drop yours. (if however you are about to
set another timeout() you may just save time and keep them, howeer if you do
you need to check the validity of the node using NG_NODE_IS_VALID()
and the same for the hook. This is to ensure that you are not seting
a timeout on a node that has been shut down and is just waiting for
your last reference to be released before being freed.
This is actually the way that any code that was called from outside
the netgraph framework should interract with netgraph nodes.
E.g. device drivers that are run from interrupt context should do this
to pass the data to their own netgraph parts, and the ng_socket node
should do this to pass the data into the netgraph part of the module
in a SMP situation. This has not yet been done, so you will find only
a small number of usages of ng_send_fn() in the code as it stands but
my next netgraph 'push' will be to do this..
I will also implement a couple of helper functions to allow this to be
dome more conveniently.
I hope that this helps you!
in 4.x of course the whole thing is under the BGL so you can ignore it
entirely.
Please let me know immediatly if you have problems.
Also be aware that there is an Ng_device node on the horizon that
creates a device in the devfs. You may want to use this to
provide an interface to the bluetooth devices that you can expose to
userland. Packets receieved from netgraph are buffered and availabel for
'read()' and data passed by write() is bufferedup and passed out through
the netgraph hooks.
Mark Santcroos <marks@ripe.net> is writing it if you want an early version
to play with.
My first use of it will be to make a 'pipe' device using
a tcp ksocket and ng_device on two differnt machines, so that anything that
is written to /dev/xyz on one machine is readable by 'cat'
on teh other machine :-)
>
> i have some (probably stupid) questions about Netgraph,
> device drivers and mutexes. i'm using -current as of this
> weekend.
>
> i have written draft version of the driver for 3com/HP
> Bluetooth Card (PC-Card). the driver is a pure Netgraph
> node, i.e. no device nor network interface registered at
> all. the only interface is Netgraph.
>
> the dirver is very simple - it detects and attaches the
> card, allocates resources, registers interrupt service
> routine (for now at NET level, but it probably shouldn't)
> and creates Negraph node. it sort of works, but i'm trying
> to figure out what kind of locking i need (if i need any).
>
> the same locking questions goes for the other Netgraph
> nodes that connected to the driver node. i want very
> simple locks to do the following:
>
> 1) handle timeouts with timeout(9)/untimeout(9) -
> my _biggest_ concern. all i could find in -current is
> everyone use splXXX/splx :( is it broken on SMP?
yes see above... I assume you are running on -current?
>
> 2) modify node's internal data structure in a safe way.
> i'm talking about things like linked lists, queues
> etc.
if you are going to modify a node you need to take
out a write lock on the device.
There are various ways to do this:
1/ the node (or someone else) uses
NG_NODE_FORCE_WRITER(node);
this means that ALL incoming requests to this node are
assumed to change state, even if they are just data,
and so they will always get exclusive locks.
2/use NG_HOOK_FORCE_WRITER(hook)
on a particular hook.
Any daa arriving via this hook will insist on getting
an exclusive lock.
3/ ng_send_fn() will always assume it has to get an exclusive lock.
So making changes via a queued function is always safe.
4/ arriveing control messages will assume they need an exclusive lock.
So making changes via a control message is always safe.
5/ any arriving item with the NGQF_WRITER bit set in item->el_flags
will be treated as a writer, regardless of whether it is data or control
or function. Actually it is a negative bit.
item->el_flags &= ~NGQF_READER; is how that is achieved.
(maybe I should fix/change that)
>
> since splXXX(9) functions are no longer relevant in
> -current (please correct me if i wrong), i was looking
> at mutex(9). i have noticed several device drivers that
> also use Netgraph (if_ar, if_sr, if_lmc and udbp), and
> they use MTX_DEF mutexes and splXXX(9) functions. is that
> OK with Netgraph? the man page says that MTX_DEF mutexes
> _will_ context switch when they are already held.
try not use these. they shoould work but might stall the
netgraph system.
>
> can/should i use MTX_SPIN mutexes? i have tried to use
> them, but WITNESS code gets very upset (panic) unless i
> modify "order_lists" in kern/subr_wintess.c. i would
> rather not do it, since i'm not fully aware of what's
> going on.
No, use the built in locking.
>
> any other ways to handle that? i had crazy idea to write
> a Netgraph "timeout" node, that does nothing but accepts
> requests to set/remove timeout and send message back when
> timeout has expired. it won't solve timeout problem, but
> put it into one place.
Use the ng_send_fn() call.
ng_timeout(node, args...)
{
NG_NODE_REF(node);
NG_NODE_PRIVATE(node)->priv_callout_handle =
timeout(my_timeout_fn, node, time);
}
void
my_timeout(void *nd)
{
struct ng_node *node = nd;
if (NG_NODE_IS_VALID(node)
ng_send_fn(node, NULL, &my_real_timeout, NULL, 0);
NG_NODE_UNREF(node);
}
void
my_real_timeout(node_p node, hook_p hook, void *arg1, int arg2)
{
do_timeout_things(node);
}
This is GUARANTEED to be run with an exclusive writer lock on that node.
Please let me know how this all works out.
p.s. Netgraph question shouldalso be sent to -net
>
> thanks,
> max
>
> To Unsubscribe: send mail to majordomo@FreeBSD.org
> with "unsubscribe freebsd-current" in the body of the message
--
+------------------------------------+ ______ _ __
| __--_|\ Julian Elischer | \ U \/ / hard at work in
| / \ julian@elischer.org +------>x USA \ a very strange
| ( OZ ) \___ ___ | country !
+- X_.---._/ presently in San Francisco \_/ \\
v
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3C8284D7.C5A8A893>