Date: Mon, 18 Feb 2002 13:42:32 +0100 (MET) From: Andy Sporner <sporner@nentec.de> To: Terry Lambert <tlambert2@mindspring.com> Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: Porting a device driver from NetBSD to FreeBSD Message-ID: <XFMail.020218134232.sporner@nentec.de> In-Reply-To: <3C70EE45.74A43103@mindspring.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Terry (and other snoopers--readers :-))
>
> I guess I should ask you if this is an SMP system and/or
> is the kernel compiled for SMP, or ist it UP (this is a
> two by two marix, so the correct answer is not "yes/no" 8-)).
Monoproc machine. Standard (OUT of box config file -- GENERIC).
Sorry, I have no access to the machine now to tell you direct, but
I didn't change anything in this regard in the GENERIC config file.
>> > It might just be that you are setting PCI_COMMAND_MASTER_ENABLE,
>> > rather than setting:
>> >
>> > command |= PCI_COMMAND_IO_ENABLE | PCI_COMMAND_MEM_ENABLE |
>> > PCI_COMMAND_MASTER_ENABLE;
>>
>> I can't use the IO mode because the registers of the crossbar are
>> not available there. The only register that is there is also in MEM
>> access mode too.
>
> The reason I pointed this out is that you had _neither_ of
> these set in the configuration write, and you will need to
> set *at least* one of them for things to be happy.
>
For clarity I will set it direct, but it is already set on the read of
the value.
>
>> o There is no blocking to read the cause register of the
>> crossbar. If nothing is set, we just leave immediately.
>
> Hm. You may want to read this register via the I/O space in
> order to cause a bus stall (annoying, but maybe more accurate).
Hmmm. I don't think I can get to these registers this way (not available
in the IO Map) The documentation (though I know sometimes how good it
can be (NOT!)) seems to say that only one register is available through
IO reads and the rest (including this special one) are available through
MEM read.
>
>> o As to the much faster. It seems a different case.
>>
>> During initialization we read and write memory
>> on the switch devices. This requires a PCI data
>> transfer to send the command to the switch (to
>> read the memory) and a response. We read every
>> 64 words (16MB Ram) and it takes 3-4 minutes on
>> NetBSD. The same code in the initalization routine
>> on FreeBSD is so quick it almost cannot be counted.
>
> This is concerning. Perhaps it is failing because of the
> lack of the initialization flag for memory commands. Basically,
> 16MB at 132MB/S (32 bit PCI @ 33 MHz) should be that fast, if
> all you are doing is a read of that much linear data across the
> bus. If you are chunking it up with register tweaking and a lot
> of little reads, then it should take longer.
>
We are doing pattern write/reads and this passes the test herein lies the
example:
start1 = 0x234000;
end1 = sw->dram_size ? 0x3DFFFF : 0x24FFFF;
start2 = sw->dram_size ? 0x520000 : 0x390000;
end2 = sw->dram_size ? 0x7FFFFF : 0x3FFFFF;
for (addr = start1; addr <= end2 - TEST_GAP; addr += TEST_GAP) {
reg_data = TEST_PATTERN;
dev_write_dram(sc, glink, addr, CONTROL_DRAM, ®_data, 1);
reg_data = 0x0;
dev_read_dram(sc, glink, addr, CONTROL_DRAM, ®_data, 1);
if (reg_data != TEST_PATTERN) {
printf("Failed...\n");
return (-1);
} /* if */
reg_data = 0x0;
dev_write_dram(sc, glink, addr, CONTROL_DRAM, ®_data, 1);
which eventually is reduced to the following for reads and writes:
/* data is the value to be written to the switch device */
/* addr is the address on the switch device */
for (;;) {
r = dev_cb_read_4(sc, CB_STATUS);
if ((r & BIT_8) == 0) {
break;
} /* if */
} /* for */
dev_cb_write_4(sc, CB_PCI_DATA_WORD_0, data);
r = ((addr >> 14) & 0xffff); /* bit15 equal to 0 write */
dev_cb_write_4(sc, CB_PCI_DATA_WORD_1, r);
r = (sw->dev << 22)|(region << 16)|(addr & 0x3fff);
dev_cb_write_4(sc, CB_PCI_ADDRESS_WORD, r);
Conversly Reads are similar from the switch device. That is to say we wait for
a bit to be set and then read some PCI communications registers on the crossbar
to see the results.
We use these abstractions to write to the crossbar device:
UINT32 dev_cb_write_4(galnet_softc_t *sc, UINT32 ofs, UINT32 val)
{
bus_space_write_4(sc->sc_st, sc->sc_sh, ofs, val);
return (0);
} /* dev_cb_write_4() */
UINT32 dev_cb_read_4(galnet_softc_t *sc, UINT32 ofs)
{
return (bus_space_read_4(sc->sc_st, sc->sc_sh, ofs));
} /* dev_cb_read_4() */
which is using data initialized during the attach function.
> Either way, there's something weird here with the difference
> between FreeBSD and NetBSD being so large.
>
>> We also see this as an impact in establishing new
>> connections, since establishing a new connection
>> requires 16 words to be transferred to the switch
>> memory.
>
> Heh. I can see the attraction to FreeBSD, then...
Here is the logic:
1. Memory test on FreeBSD is significantly faster than the
the NetBSD counterpart (also boots MUCH faster!) on the
exact same hardware (just swapped the boot harddisks).
2. The Memory test transfers high bulks of data using the
same methods as the setting of entries to handle hardware
forwarding of data.
3. Thus if Memory test is faster this follows that the setting
of hardware forwarding entries must also be faster.
> You probably just didn't get to the right list or the right
> people.
I was writting to tech-kern. I would wait weeks for responses.
>
> Being unable to drop details of your hardware can also mean
> that people will have to turn their diagnostic process over
> in the email message, and a lot of people are going to be a
> bit reluctant to do that, if they go by instincts, or if they
> consider it "secret". Other people might not respond on the
> principle that the driver will end up being closed. We have
> a lot of kooks. ;^).
:-) Agreed. BTW: As a reward for people who actually READ
posts through completely. I am making an preview type anouncement
of my clustering software (Hi-AV) availability at:
http://www.sporner.com/bsdclusters. I will (hopefully!!!) be
making some more improvements in the next week or so to the
documentation. I have to put the stuff though another test round
to, but in principle it works. It has to be a lower priority
than my regular work so there is no interference.
This is more or less an experiment to see how many people read
complete messages. I will make a more formal announcment the
right way in a week or so.
Hopefully I get better luck announcing it this way. I have tried
many times to get those TURKEYs at www.bsdtoday.com to put something
on their clustering page about it and I have yet to see or hear
anything from them. It gets rather frustrating!
>
> In any case, good luck with your project...
>
Thanks for your help...
Andy
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?XFMail.020218134232.sporner>