Re: Configure MSI-X vectors to target different CPUs

Re: Configure MSI-X vectors to target different CPUs

[caiying]

Thanks, Robert. My device does support multiple vectors.

When looking into functions called by pci_enable_msix(), I found msi_compose_msg() in arch/i386/kernel/io_apic.c. It tries to get destination CPU (TARGET_CPUS) and set this information to msg->address_lo. My question is about TARGET_CPUS. Under the asm-i386/mach-default, it is the cpu_online_map. Under asm-i386/mach-bigsmp, it is the cpumask_of_cpu(cpu), where cpu is a single one. I would guess if a single CPU is set as destination, only that CPU will be interrupted. But what will happen when the cpu_online_map is set as destination? Any CPU can be interrupted then? Or depending on affinity of the corresponding irq?

Please 
CC'ed 
me 
(caiying@yahoo.com) 
answers/comments  
in 
response 
to 
this 
posting. 

Thanks,
Ying

----- Original Message ----
From: Robert Hancock <hancockr@shaw.ca>
To: caiying@yahoo.com
Cc: linux-kernel@vger.kernel.org
Sent: Thursday, February 21, 2008 7:59:14 PM
Subject: Re: Configure MSI-X vectors to target different CPUs


caiying@yahoo.com 
wrote:
> 
Hi,
> 
> 
In 
MSI-HOWTO, 
it's 
said:
> 
> 
"Using 
MSI 
enables 
the 
device 
functions 
to 
support 
two 
or 
more 
vectors, 
which 
can 
be 
configured 
to 
target 
different 
CPUs 
to 
increase 
scalability."
> 
> 
So 
how 
can 
I 
set 
up 
MSI-X 
vectors 
to 
target 
different 
CPUs? 
I 
want 
to 
allocate 
the 
same 
number 
of 
MSI-X 
vectors 
as 
CPUs, 
and 
equally 
distribute 
them 
to 
every 
CPU.
> 
> 
Is 
it 
automatically 
done 
by 
Linux 
when 
I 
call 
pci_enable_msix()? 
If 
yes, 
how? 
If 
not, 
what 
should 
I 
do? 
My 
guess 
is 
to 
set 
the 
affinity 
of 
the 
interrupts 
manually. 
Am 
I 
right?
> 
> 
Please 
CC'ed 
me 
(caiying@yahoo.com) 
answers/comments  
in 
response 
to 
this 
posting. 
> 
> 
Thanks,
> 
Ying

If 
the 
device 
actually 
supports 
multiple 
vectors 
(not 
all 
do), 
I 
think 
they 
should 
show 
up 
as 
separate 
interrupts 
in 
/proc/interrupts 
and 
you 
can 
either 
set 
the 
affinity 
manually, 
or 
maybe 
irqbalance 
is 
smart 
enough 
for 
this.

Careful, 
though, 
as 
in 
some 
cases 
this 
may 
reduce 
performance 
due 
to 
causing 
more 
cache 
line 
bouncing 
between 
CPUs.

Re: Configure MSI-X vectors to target different CPUs

[Robert Hancock]

caiying@yahoo.com wrote:
> Hi,
> 
> In MSI-HOWTO, it's said:
> 
> "Using MSI enables the device functions to support two or more vectors, which can be configured to target different CPUs to increase scalability."
> 
> So how can I set up MSI-X vectors to target different CPUs? I want to allocate the same number of MSI-X vectors as CPUs, and equally distribute them to every CPU.
> 
> Is it automatically done by Linux when I call pci_enable_msix()? If yes, how? If not, what should I do? My guess is to set the affinity of the interrupts manually. Am I right?
> 
> Please CC'ed me (caiying@yahoo.com) answers/comments  in response to this posting. 
> 
> Thanks,
> Ying

If the device actually supports multiple vectors (not all do), I think 
they should show up as separate interrupts in /proc/interrupts and you 
can either set the affinity manually, or maybe irqbalance is smart 
enough for this.

Careful, though, as in some cases this may reduce performance due to 
causing more cache line bouncing between CPUs.

Configure MSI-X vectors to target different CPUs

[caiying]

Hi,

In MSI-HOWTO, it's said:

"Using MSI enables the device functions to support two or more vectors, which can be configured to target different CPUs to increase scalability."

So how can I set up MSI-X vectors to target different CPUs? I want to allocate the same number of MSI-X vectors as CPUs, and equally distribute them to every CPU.

Is it automatically done by Linux when I call pci_enable_msix()? If yes, how? If not, what should I do? My guess is to set the affinity of the interrupts manually. Am I right?

Please CC'ed me (caiying@yahoo.com) answers/comments  in response to this posting. 

Thanks,
Ying