[vortex] What are the issues related to increasing Rx ring size ?

[Donald Becker]

On Thu, 23 Sep 2004, Amitava Biswas wrote:

> I have gone through the source code of 3c905 driver.
> I want to use this driver under a situation when packets arrive in
> bursts (few hundered to say one thousand small (64Byte) packets in one
> burst). I want to avoid loss of packets, so I want to increase the
> receive ring size. I have the following questions- 
> 
> 1) Assuming that I have lots of RAM and take care of the protocol and
> socket processing layers (increase the socket buffer sizes there too,
> by making changes in kernel and libraries),

Mostly the kernel.  And keep in mind that the queue layer -- the code
run just after the hardware driver layer -- must have the source code
modified to allow a larger number of packets to be queued.

> I also assume that lots of "sk_buff"s have already been preallocated
> so that the interrupt portion of code does not get performance hits
> due to on the fly alloc,

You'll still encounter this problem.  The driver will try to refill the
Rx ring in every loop.  Slow skbuff allocation code will always bite you.

> Can I increase the Rx ring size to 1024 ? If not why ? What is the limit ? 

This driver doesn't have a specific limit.  Some hardware does have
hardware limitations on the ring size.  For my drivers, the limitation
is noted in both the Principles of Operation notes at the top of the
file and where the Rx ring size is defined.

> 2) What are the other issues which may be the hurdles or constraints
> to the driver operation with large Rx ring size ? These may be both -
> system / resource constraints and performance constraints. 

An issue to consider: if the kernel version you are using implements
per-device or whole-network-stack limits on memory usage.  When you fill
a huge Rx queue, the effect is the same as if the networking code is hit
with a denial-of-service attack.

When I wrote the drivers I _actually_ _did_ measure many aspects of
their operation to select appropriate parameters.  I instrumented the
ring usage to select minimum appropriate number for heavy use, frequently
for cluster-oriented workloads.   While system capabilities have changed
over time (e.g. typical 16MB RAM to 512+MB RAM), there has been little
change in packet size distribution and interrupt latency that sets the
queue parameters.
IMNSHO, the only real reason to change these numbers is if you really do
have an environment that generates bursts of minimum length packets as
the typical traffic.

-- 
Donald Becker				becker at scyld.com
Scyld Software	 			Scyld Beowulf cluster systems
914 Bay Ridge Road, Suite 220		www.scyld.com
Annapolis MD 21403			410-990-9993