cpu load balancing problem on smp

[Marc Donner <donner@dbd-breitband.de>]

Hi @all

we have detected some problems on our live systems and so i have build a test 
setup in our lab as follow:

3 Core 2 duo servers, each with 2 CPUs,  with GE interfaces. 2 of them are 
only for generating network traffic. the 3rd server is the one i want to 
test. it is connected over two GE links to the other servers. the testserver 
is configured as an ip router. running kernel 2.6.20. 

now if i  let traffic flow over the box, about 600Mbit/s and about 120k 
packets/s all seems to be ok. the load is balanced over all cpus. if now 
insert some iptables rules, about 500, the softirq load increases, but all 
seems to be ok. now i insert some rules more, and suddenly 1 CPU is 100% 
loaded and the other ones are 99% idle. the load toggles now between the cpus 
in intervals.

i have made some graphs http://www.poweraudio.de/linux/cpu.html  

this seems to me to be not normal.

any suggestions?

regards
Marc