[PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Halil Pasic]

While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
it may not always be, and we must not rely on this.

Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
that code like
for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
                vcpu = kvm_get_vcpu(kvm, vcpu_id);
		do_stuff(vcpu);
}
is not legit. The trouble is, we do actually use kvm->arch.idle_mask
like this. To fix this problem we have two options. Either use
kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
or switch to indexing via vcpu_idx. The latter is preferable for obvious
reasons.

Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to
indexing it by vcpu_idx.  To keep gisa_int.kicked_mask indexed by the
same index as idle_mask lets make the same change for it as well.

Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
Fixes: 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
Cc: <stable@vger.kernel.org> # 3.15+
---
 arch/s390/include/asm/kvm_host.h |  1 +
 arch/s390/kvm/interrupt.c        | 12 ++++++------
 arch/s390/kvm/kvm-s390.c         |  2 +-
 arch/s390/kvm/kvm-s390.h         |  2 +-
 4 files changed, 9 insertions(+), 8 deletions(-)

diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index 161a9e12bfb8..630eab0fa176 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -957,6 +957,7 @@ struct kvm_arch{
 	atomic64_t cmma_dirty_pages;
 	/* subset of available cpu features enabled by user space */
 	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
+	/* indexed by vcpu_idx */
 	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
 	struct kvm_s390_gisa_interrupt gisa_int;
 	struct kvm_s390_pv pv;
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index d548d60caed2..16256e17a544 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
 static void __set_cpu_idle(struct kvm_vcpu *vcpu)
 {
 	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
-	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
 {
 	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
@@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
 
 static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
 {
-	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
+	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
 	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
 	struct kvm_vcpu *vcpu;
 
-	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
-		vcpu = kvm_get_vcpu(kvm, vcpu_id);
+	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
+		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
 		if (psw_ioint_disabled(vcpu))
 			continue;
 		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
 		if (deliverable_mask) {
 			/* lately kicked but not yet running */
-			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
+			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
 				return;
 			kvm_s390_vcpu_wakeup(vcpu);
 			return;
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index 4527ac7b5961..8580543c5bc3 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
 		kvm_s390_patch_guest_per_regs(vcpu);
 	}
 
-	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
+	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
 
 	vcpu->arch.sie_block->icptcode = 0;
 	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
index 9fad25109b0d..ecd741ee3276 100644
--- a/arch/s390/kvm/kvm-s390.h
+++ b/arch/s390/kvm/kvm-s390.h
@@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
 
 static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
 {
-	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
+	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
 }
 
 static inline int kvm_is_ucontrol(struct kvm *kvm)

base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba
-- 
2.25.1

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Christian Borntraeger]

On 27.08.21 14:54, Halil Pasic wrote:
> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
> it may not always be, and we must not rely on this.
> 
> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
> that code like
> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
>                  vcpu = kvm_get_vcpu(kvm, vcpu_id);
> 		do_stuff(vcpu);
> }
> is not legit. The trouble is, we do actually use kvm->arch.idle_mask
> like this. To fix this problem we have two options. Either use
> kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
> or switch to indexing via vcpu_idx. The latter is preferable for obvious
> reasons.
> 
> Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to
> indexing it by vcpu_idx.  To keep gisa_int.kicked_mask indexed by the
> same index as idle_mask lets make the same change for it as well.
> 
> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> Fixes: 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
> Cc: <stable@vger.kernel.org> # 3.15+

Reviewed-by: Christian Bornträger <borntraeger@de.ibm.com>
Will apply and wait for the nightly run.

> ---
>   arch/s390/include/asm/kvm_host.h |  1 +
>   arch/s390/kvm/interrupt.c        | 12 ++++++------
>   arch/s390/kvm/kvm-s390.c         |  2 +-
>   arch/s390/kvm/kvm-s390.h         |  2 +-
>   4 files changed, 9 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 161a9e12bfb8..630eab0fa176 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -957,6 +957,7 @@ struct kvm_arch{
>   	atomic64_t cmma_dirty_pages;
>   	/* subset of available cpu features enabled by user space */
>   	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
> +	/* indexed by vcpu_idx */
>   	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
>   	struct kvm_s390_gisa_interrupt gisa_int;
>   	struct kvm_s390_pv pv;
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index d548d60caed2..16256e17a544 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>   static void __set_cpu_idle(struct kvm_vcpu *vcpu)
>   {
>   	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
> -	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
>   {
>   	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
> @@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
>   
>   static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
>   {
> -	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
> +	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
>   	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
>   	struct kvm_vcpu *vcpu;
>   
> -	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> -		vcpu = kvm_get_vcpu(kvm, vcpu_id);
> +	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
> +		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
>   		if (psw_ioint_disabled(vcpu))
>   			continue;
>   		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
>   		if (deliverable_mask) {
>   			/* lately kicked but not yet running */
> -			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
> +			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
>   				return;
>   			kvm_s390_vcpu_wakeup(vcpu);
>   			return;
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 4527ac7b5961..8580543c5bc3 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
>   		kvm_s390_patch_guest_per_regs(vcpu);
>   	}
>   
> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
>   
>   	vcpu->arch.sie_block->icptcode = 0;
>   	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> index 9fad25109b0d..ecd741ee3276 100644
> --- a/arch/s390/kvm/kvm-s390.h
> +++ b/arch/s390/kvm/kvm-s390.h
> @@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
>   
>   static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
>   {
> -	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static inline int kvm_is_ucontrol(struct kvm *kvm)
> 
> base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba
> 

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Claudio Imbrenda]

On Fri, 27 Aug 2021 14:54:29 +0200
Halil Pasic <pasic@linux.ibm.com> wrote:

> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
> it may not always be, and we must not rely on this.

why?

maybe add a simple explanation of why vcpu_idx and vcpu_id can be
different, namely:
KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two
might not match 

> 
> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
> that code like
> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
>                 vcpu = kvm_get_vcpu(kvm, vcpu_id);

you can also add a sentence to clarify that kvm_get_vcpu expects an
vcpu_idx, not an vcpu_id.

> 		do_stuff(vcpu);
> }
> is not legit. The trouble is, we do actually use kvm->arch.idle_mask
> like this. To fix this problem we have two options. Either use
> kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
> or switch to indexing via vcpu_idx. The latter is preferable for obvious
> reasons.
> 
> Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to
> indexing it by vcpu_idx.  To keep gisa_int.kicked_mask indexed by the
> same index as idle_mask lets make the same change for it as well.
> 
> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> Fixes: 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")

otherwise looks good to me.

Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>

> Cc: <stable@vger.kernel.org> # 3.15+
> ---
>  arch/s390/include/asm/kvm_host.h |  1 +
>  arch/s390/kvm/interrupt.c        | 12 ++++++------
>  arch/s390/kvm/kvm-s390.c         |  2 +-
>  arch/s390/kvm/kvm-s390.h         |  2 +-
>  4 files changed, 9 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 161a9e12bfb8..630eab0fa176 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -957,6 +957,7 @@ struct kvm_arch{
>  	atomic64_t cmma_dirty_pages;
>  	/* subset of available cpu features enabled by user space */
>  	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
> +	/* indexed by vcpu_idx */
>  	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
>  	struct kvm_s390_gisa_interrupt gisa_int;
>  	struct kvm_s390_pv pv;
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index d548d60caed2..16256e17a544 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>  static void __set_cpu_idle(struct kvm_vcpu *vcpu)
>  {
>  	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
> -	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>  }
>  
>  static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
>  {
>  	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>  }
>  
>  static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
> @@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
>  
>  static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
>  {
> -	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
> +	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
>  	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
>  	struct kvm_vcpu *vcpu;
>  
> -	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> -		vcpu = kvm_get_vcpu(kvm, vcpu_id);
> +	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
> +		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
>  		if (psw_ioint_disabled(vcpu))
>  			continue;
>  		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
>  		if (deliverable_mask) {
>  			/* lately kicked but not yet running */
> -			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
> +			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
>  				return;
>  			kvm_s390_vcpu_wakeup(vcpu);
>  			return;
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 4527ac7b5961..8580543c5bc3 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
>  		kvm_s390_patch_guest_per_regs(vcpu);
>  	}
>  
> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
>  
>  	vcpu->arch.sie_block->icptcode = 0;
>  	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> index 9fad25109b0d..ecd741ee3276 100644
> --- a/arch/s390/kvm/kvm-s390.h
> +++ b/arch/s390/kvm/kvm-s390.h
> @@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
>  
>  static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
>  {
> -	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>  }
>  
>  static inline int kvm_is_ucontrol(struct kvm *kvm)
> 
> base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Christian Borntraeger]

On 27.08.21 16:06, Claudio Imbrenda wrote:
> On Fri, 27 Aug 2021 14:54:29 +0200
> Halil Pasic <pasic@linux.ibm.com> wrote:
> 
>> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
>> it may not always be, and we must not rely on this.
> 
> why?
> 
> maybe add a simple explanation of why vcpu_idx and vcpu_id can be
> different, namely:
> KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two
> might not match
> 
>>
>> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
>> that code like
>> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
>>                  vcpu = kvm_get_vcpu(kvm, vcpu_id);
> 
> you can also add a sentence to clarify that kvm_get_vcpu expects an
> vcpu_idx, not an vcpu_id.
> 
>> 		do_stuff(vcpu);

I will modify the patch description accordingly before sending to Paolo.
Thanks for noticing.


>> }
>> is not legit. The trouble is, we do actually use kvm->arch.idle_mask
>> like this. To fix this problem we have two options. Either use
>> kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
>> or switch to indexing via vcpu_idx. The latter is preferable for obvious
>> reasons.
>>
>> Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to
>> indexing it by vcpu_idx.  To keep gisa_int.kicked_mask indexed by the
>> same index as idle_mask lets make the same change for it as well.
>>
>> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
>> Fixes: 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
> 
> otherwise looks good to me.
> 
> Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
> 
>> Cc: <stable@vger.kernel.org> # 3.15+
>> ---
>>   arch/s390/include/asm/kvm_host.h |  1 +
>>   arch/s390/kvm/interrupt.c        | 12 ++++++------
>>   arch/s390/kvm/kvm-s390.c         |  2 +-
>>   arch/s390/kvm/kvm-s390.h         |  2 +-
>>   4 files changed, 9 insertions(+), 8 deletions(-)
>>
>> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
>> index 161a9e12bfb8..630eab0fa176 100644
>> --- a/arch/s390/include/asm/kvm_host.h
>> +++ b/arch/s390/include/asm/kvm_host.h
>> @@ -957,6 +957,7 @@ struct kvm_arch{
>>   	atomic64_t cmma_dirty_pages;
>>   	/* subset of available cpu features enabled by user space */
>>   	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
>> +	/* indexed by vcpu_idx */
>>   	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
>>   	struct kvm_s390_gisa_interrupt gisa_int;
>>   	struct kvm_s390_pv pv;
>> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
>> index d548d60caed2..16256e17a544 100644
>> --- a/arch/s390/kvm/interrupt.c
>> +++ b/arch/s390/kvm/interrupt.c
>> @@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>>   static void __set_cpu_idle(struct kvm_vcpu *vcpu)
>>   {
>>   	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
>> -	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
>> +	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>>   }
>>   
>>   static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
>>   {
>>   	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
>> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
>> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>>   }
>>   
>>   static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
>> @@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
>>   
>>   static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
>>   {
>> -	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
>> +	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
>>   	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
>>   	struct kvm_vcpu *vcpu;
>>   
>> -	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
>> -		vcpu = kvm_get_vcpu(kvm, vcpu_id);
>> +	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
>> +		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
>>   		if (psw_ioint_disabled(vcpu))
>>   			continue;
>>   		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
>>   		if (deliverable_mask) {
>>   			/* lately kicked but not yet running */
>> -			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
>> +			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
>>   				return;
>>   			kvm_s390_vcpu_wakeup(vcpu);
>>   			return;
>> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
>> index 4527ac7b5961..8580543c5bc3 100644
>> --- a/arch/s390/kvm/kvm-s390.c
>> +++ b/arch/s390/kvm/kvm-s390.c
>> @@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
>>   		kvm_s390_patch_guest_per_regs(vcpu);
>>   	}
>>   
>> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
>> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
>>   
>>   	vcpu->arch.sie_block->icptcode = 0;
>>   	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
>> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
>> index 9fad25109b0d..ecd741ee3276 100644
>> --- a/arch/s390/kvm/kvm-s390.h
>> +++ b/arch/s390/kvm/kvm-s390.h
>> @@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
>>   
>>   static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
>>   {
>> -	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
>> +	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>>   }
>>   
>>   static inline int kvm_is_ucontrol(struct kvm *kvm)
>>
>> base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba
> 

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Halil Pasic]

On Fri, 27 Aug 2021 16:06:16 +0200
Claudio Imbrenda <imbrenda@linux.ibm.com> wrote:

> On Fri, 27 Aug 2021 14:54:29 +0200
> Halil Pasic <pasic@linux.ibm.com> wrote:
> 
> > While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,

s/vcp_id/vcpu_id/

> > it may not always be, and we must not rely on this.  
> 
> why?
> 
> maybe add a simple explanation of why vcpu_idx and vcpu_id can be
> different, namely:
> KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two
> might not match 

Not sure that is a good explanation. A quote from
Documentation/virt/kvm/api.rst:

"""
4.7 KVM_CREATE_VCPU
-------------------

:Capability: basic
:Architectures: all
:Type: vm ioctl
:Parameters: vcpu id (apic id on x86)
:Returns: vcpu fd on success, -1 on error
                
This API adds a vcpu to a virtual machine. No more than max_vcpus may be added.
The vcpu id is an integer in the range [0, max_vcpu_id).
        
The recommended max_vcpus value can be retrieved using the KVM_CAP_NR_VCPUS of
the KVM_CHECK_EXTENSION ioctl() at run-time.
The maximum possible value for max_vcpus can be retrieved using the
KVM_CAP_MAX_VCPUS of the KVM_CHECK_EXTENSION ioctl() at run-time.
"""

Based on that and a quick look at the code, it looks to me like the
set of valid vcpu_id values are a subset of the range of vcpu_idx-es,
i.e. that kvm could decide to choose vcpu_id for the value of vcpu_idx.
I don't think it should, but it could. Were the set of valid vcpu_id
values not a subset of the set of supported vcpu_idx values, then one
could argue that this is why.

I didn't want to get into explaining the why, I just wanted to state the
fact.

> 
> > 
> > Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
> > that code like
> > for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> >                 vcpu = kvm_get_vcpu(kvm, vcpu_id);  
> 
> you can also add a sentence to clarify that kvm_get_vcpu expects an
> vcpu_idx, not an vcpu_id.

maybe ...

> 
> > 		do_stuff(vcpu);
> > }
> > is not legit. The trouble is, we do actually use kvm->arch.idle_mask

... s/legit\./legit, because kvm_get_vcpu() expects a vcpu_idx and not a
vcpu_id.

But I agree kvm_get_vcpu(kvm, vcpu_id); does not scream BUG at me while
kvm_get_vcpu_by_idx(kvm, vcpu_id) would look much more suspicious.

[..]
> 
> otherwise looks good to me.
> 
> Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>

Thanks for your reveiew!

Halil

[..]

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Halil Pasic]

On Fri, 27 Aug 2021 18:36:48 +0200
Christian Borntraeger <borntraeger@de.ibm.com> wrote:

> On 27.08.21 16:06, Claudio Imbrenda wrote:
> > On Fri, 27 Aug 2021 14:54:29 +0200
> > Halil Pasic <pasic@linux.ibm.com> wrote:
> >   
> >> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,

s/vcp_id/vcpu_id/

> >> it may not always be, and we must not rely on this.  
> > 
> > why?
> > 
> > maybe add a simple explanation of why vcpu_idx and vcpu_id can be
> > different, namely:
> > KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two
> > might not match
> >   
> >>
> >> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
> >> that code like
> >> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> >>                  vcpu = kvm_get_vcpu(kvm, vcpu_id);  
> > 
> > you can also add a sentence to clarify that kvm_get_vcpu expects an
> > vcpu_idx, not an vcpu_id.
> >   
> >> 		do_stuff(vcpu);  
> 
> I will modify the patch description accordingly before sending to Paolo.
> Thanks for noticing.

Can you also please fix the typo I pointed out above (in the first line
of the long description).

Thanks!
Halil

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Michael Mueller]

On 27.08.21 14:54, Halil Pasic wrote:
> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
> it may not always be, and we must not rely on this.
> 
> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
> that code like
> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
>                  vcpu = kvm_get_vcpu(kvm, vcpu_id);
> 		do_stuff(vcpu);
> }
> is not legit. The trouble is, we do actually use kvm->arch.idle_mask
> like this. To fix this problem we have two options. Either use
> kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
> or switch to indexing via vcpu_idx. The latter is preferable for obvious
> reasons.
> 
> Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to
> indexing it by vcpu_idx.  To keep gisa_int.kicked_mask indexed by the
> same index as idle_mask lets make the same change for it as well.
> 
> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> Fixes: 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
> Cc: <stable@vger.kernel.org> # 3.15+
> ---
>   arch/s390/include/asm/kvm_host.h |  1 +
>   arch/s390/kvm/interrupt.c        | 12 ++++++------
>   arch/s390/kvm/kvm-s390.c         |  2 +-
>   arch/s390/kvm/kvm-s390.h         |  2 +-
>   4 files changed, 9 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 161a9e12bfb8..630eab0fa176 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -957,6 +957,7 @@ struct kvm_arch{
>   	atomic64_t cmma_dirty_pages;
>   	/* subset of available cpu features enabled by user space */
>   	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
> +	/* indexed by vcpu_idx */
>   	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
>   	struct kvm_s390_gisa_interrupt gisa_int;
>   	struct kvm_s390_pv pv;
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index d548d60caed2..16256e17a544 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>   static void __set_cpu_idle(struct kvm_vcpu *vcpu)
>   {
>   	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
> -	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
>   {
>   	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
> @@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
>   
>   static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
>   {
> -	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
> +	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
>   	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
>   	struct kvm_vcpu *vcpu;
>   
> -	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> -		vcpu = kvm_get_vcpu(kvm, vcpu_id);
> +	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
> +		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
>   		if (psw_ioint_disabled(vcpu))
>   			continue;
>   		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
>   		if (deliverable_mask) {
>   			/* lately kicked but not yet running */
> -			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
> +			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
>   				return;
>   			kvm_s390_vcpu_wakeup(vcpu);
>   			return;
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 4527ac7b5961..8580543c5bc3 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
>   		kvm_s390_patch_guest_per_regs(vcpu);
>   	}
>   
> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
>   
>   	vcpu->arch.sie_block->icptcode = 0;
>   	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> index 9fad25109b0d..ecd741ee3276 100644
> --- a/arch/s390/kvm/kvm-s390.h
> +++ b/arch/s390/kvm/kvm-s390.h
> @@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
>   
>   static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
>   {
> -	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static inline int kvm_is_ucontrol(struct kvm *kvm)
> 
> base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

Reviewed-by: Michael Mueller <mimu@linux.ibm.com>

I did also run some tests and have prepared a qemu that will choose
vcpu_id's that are not equal to the index in the kernel.

Michael

> 

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Christian Borntraeger]

On 27.08.21 23:23, Halil Pasic wrote:
> On Fri, 27 Aug 2021 18:36:48 +0200
> Christian Borntraeger <borntraeger@de.ibm.com> wrote:
> 
>> On 27.08.21 16:06, Claudio Imbrenda wrote:
>>> On Fri, 27 Aug 2021 14:54:29 +0200
>>> Halil Pasic <pasic@linux.ibm.com> wrote:
>>>    
>>>> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
> 
> s/vcp_id/vcpu_id/
> 
>>>> it may not always be, and we must not rely on this.
>>>
>>> why?
>>>
>>> maybe add a simple explanation of why vcpu_idx and vcpu_id can be
>>> different, namely:
>>> KVM decides the vcpu_idx, userspace decides the vcpu_id, thus the two
>>> might not match
>>>    
>>>>
>>>> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
>>>> that code like
>>>> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
>>>>                   vcpu = kvm_get_vcpu(kvm, vcpu_id);
>>>
>>> you can also add a sentence to clarify that kvm_get_vcpu expects an
>>> vcpu_idx, not an vcpu_id.
>>>    
>>>> 		do_stuff(vcpu);
>>
>> I will modify the patch description accordingly before sending to Paolo.
>> Thanks for noticing.
> 
> Can you also please fix the typo I pointed out above (in the first line
> of the long description).

I already queued, but I think this is not a big deal.

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Petr Tesařík]

Hi Halil,

Dne 27. 08. 21 v 14:54 Halil Pasic napsal(a):
> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
> it may not always be, and we must not rely on this.
> 
> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
> that code like
> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
>                  vcpu = kvm_get_vcpu(kvm, vcpu_id);
> 		do_stuff(vcpu);
> }
> is not legit. The trouble is, we do actually use kvm->arch.idle_mask
> like this. To fix this problem we have two options. Either use
> kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
> or switch to indexing via vcpu_idx. The latter is preferable for obvious
> reasons.

I'm just backporting this fix to SLES 12 SP5, and I've noticed that 
there is still this code in __floating_irq_kick():

	/* find idle VCPUs first, then round robin */
	sigcpu = find_first_bit(fi->idle_mask, online_vcpus);
/* ... round robin loop removed ...
	dst_vcpu = kvm_get_vcpu(kvm, sigcpu);

It seems to me that this does exactly the thing that is not legit, but 
I'm no expert. Did I miss something?

Petr T

> Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to
> indexing it by vcpu_idx.  To keep gisa_int.kicked_mask indexed by the
> same index as idle_mask lets make the same change for it as well.
> 
> Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> Fixes: 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
> Cc: <stable@vger.kernel.org> # 3.15+
> ---
>   arch/s390/include/asm/kvm_host.h |  1 +
>   arch/s390/kvm/interrupt.c        | 12 ++++++------
>   arch/s390/kvm/kvm-s390.c         |  2 +-
>   arch/s390/kvm/kvm-s390.h         |  2 +-
>   4 files changed, 9 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> index 161a9e12bfb8..630eab0fa176 100644
> --- a/arch/s390/include/asm/kvm_host.h
> +++ b/arch/s390/include/asm/kvm_host.h
> @@ -957,6 +957,7 @@ struct kvm_arch{
>   	atomic64_t cmma_dirty_pages;
>   	/* subset of available cpu features enabled by user space */
>   	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
> +	/* indexed by vcpu_idx */
>   	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
>   	struct kvm_s390_gisa_interrupt gisa_int;
>   	struct kvm_s390_pv pv;
> diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> index d548d60caed2..16256e17a544 100644
> --- a/arch/s390/kvm/interrupt.c
> +++ b/arch/s390/kvm/interrupt.c
> @@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
>   static void __set_cpu_idle(struct kvm_vcpu *vcpu)
>   {
>   	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
> -	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
>   {
>   	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
> @@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
>   
>   static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
>   {
> -	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
> +	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
>   	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
>   	struct kvm_vcpu *vcpu;
>   
> -	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> -		vcpu = kvm_get_vcpu(kvm, vcpu_id);
> +	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
> +		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
>   		if (psw_ioint_disabled(vcpu))
>   			continue;
>   		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
>   		if (deliverable_mask) {
>   			/* lately kicked but not yet running */
> -			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
> +			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
>   				return;
>   			kvm_s390_vcpu_wakeup(vcpu);
>   			return;
> diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> index 4527ac7b5961..8580543c5bc3 100644
> --- a/arch/s390/kvm/kvm-s390.c
> +++ b/arch/s390/kvm/kvm-s390.c
> @@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
>   		kvm_s390_patch_guest_per_regs(vcpu);
>   	}
>   
> -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
> +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
>   
>   	vcpu->arch.sie_block->icptcode = 0;
>   	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
> diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> index 9fad25109b0d..ecd741ee3276 100644
> --- a/arch/s390/kvm/kvm-s390.h
> +++ b/arch/s390/kvm/kvm-s390.h
> @@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
>   
>   static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
>   {
> -	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> +	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
>   }
>   
>   static inline int kvm_is_ucontrol(struct kvm *kvm)
> 
> base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Halil Pasic]

On Mon, 31 Jan 2022 11:13:18 +0100
Petr Tesařík <ptesarik@suse.cz> wrote:

> Hi Halil,
> 
> Dne 27. 08. 21 v 14:54 Halil Pasic napsal(a):
> > While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
> > it may not always be, and we must not rely on this.
> > 
> > Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
> > that code like
> > for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> >                  vcpu = kvm_get_vcpu(kvm, vcpu_id);
> > 		do_stuff(vcpu);
> > }
> > is not legit. The trouble is, we do actually use kvm->arch.idle_mask
> > like this. To fix this problem we have two options. Either use
> > kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
> > or switch to indexing via vcpu_idx. The latter is preferable for obvious
> > reasons.  
> 
> I'm just backporting this fix to SLES 12 SP5, and I've noticed that 
> there is still this code in __floating_irq_kick():
> 
> 	/* find idle VCPUs first, then round robin */
> 	sigcpu = find_first_bit(fi->idle_mask, online_vcpus);
> /* ... round robin loop removed ...
> 	dst_vcpu = kvm_get_vcpu(kvm, sigcpu);
> 
> It seems to me that this does exactly the thing that is not legit, but 
> I'm no expert. Did I miss something?
> 

We made that legit by making the N-th bit in idle_mask correspond to the
vcpu whose vcpu_idx == N. The second argument of kvm_get_vcpu() is the
vcpu_idx. IMHO that ain't super-intuitive because it ain't spelled out.

So before this was a mismatch (with a vcpu_id based bitmap we would have
to use kvm_get_vcpu_by_id()), and with this patch applied this code
becomes legit because both idle_mask and kvm_get_vcpu() operate with
vcpu_idx.

Does that make sense?

I'm sorry the commit message did not convey this clearly enough...

Regards,
Halil


> Petr T
> 
> > Let us make switch from indexing kvm->arch.idle_mask by vcpu_id to
> > indexing it by vcpu_idx.  To keep gisa_int.kicked_mask indexed by the
> > same index as idle_mask lets make the same change for it as well.
> > 
> > Signed-off-by: Halil Pasic <pasic@linux.ibm.com>
> > Fixes: 1ee0bc559dc3 ("KVM: s390: get rid of local_int array")
> > Cc: <stable@vger.kernel.org> # 3.15+
> > ---
> >   arch/s390/include/asm/kvm_host.h |  1 +
> >   arch/s390/kvm/interrupt.c        | 12 ++++++------
> >   arch/s390/kvm/kvm-s390.c         |  2 +-
> >   arch/s390/kvm/kvm-s390.h         |  2 +-
> >   4 files changed, 9 insertions(+), 8 deletions(-)
> > 
> > diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
> > index 161a9e12bfb8..630eab0fa176 100644
> > --- a/arch/s390/include/asm/kvm_host.h
> > +++ b/arch/s390/include/asm/kvm_host.h
> > @@ -957,6 +957,7 @@ struct kvm_arch{
> >   	atomic64_t cmma_dirty_pages;
> >   	/* subset of available cpu features enabled by user space */
> >   	DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS);
> > +	/* indexed by vcpu_idx */
> >   	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
> >   	struct kvm_s390_gisa_interrupt gisa_int;
> >   	struct kvm_s390_pv pv;
> > diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
> > index d548d60caed2..16256e17a544 100644
> > --- a/arch/s390/kvm/interrupt.c
> > +++ b/arch/s390/kvm/interrupt.c
> > @@ -419,13 +419,13 @@ static unsigned long deliverable_irqs(struct kvm_vcpu *vcpu)
> >   static void __set_cpu_idle(struct kvm_vcpu *vcpu)
> >   {
> >   	kvm_s390_set_cpuflags(vcpu, CPUSTAT_WAIT);
> > -	set_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> > +	set_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
> >   }
> >   
> >   static void __unset_cpu_idle(struct kvm_vcpu *vcpu)
> >   {
> >   	kvm_s390_clear_cpuflags(vcpu, CPUSTAT_WAIT);
> > -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> > +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
> >   }
> >   
> >   static void __reset_intercept_indicators(struct kvm_vcpu *vcpu)
> > @@ -3050,18 +3050,18 @@ int kvm_s390_get_irq_state(struct kvm_vcpu *vcpu, __u8 __user *buf, int len)
> >   
> >   static void __airqs_kick_single_vcpu(struct kvm *kvm, u8 deliverable_mask)
> >   {
> > -	int vcpu_id, online_vcpus = atomic_read(&kvm->online_vcpus);
> > +	int vcpu_idx, online_vcpus = atomic_read(&kvm->online_vcpus);
> >   	struct kvm_s390_gisa_interrupt *gi = &kvm->arch.gisa_int;
> >   	struct kvm_vcpu *vcpu;
> >   
> > -	for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> > -		vcpu = kvm_get_vcpu(kvm, vcpu_id);
> > +	for_each_set_bit(vcpu_idx, kvm->arch.idle_mask, online_vcpus) {
> > +		vcpu = kvm_get_vcpu(kvm, vcpu_idx);
> >   		if (psw_ioint_disabled(vcpu))
> >   			continue;
> >   		deliverable_mask &= (u8)(vcpu->arch.sie_block->gcr[6] >> 24);
> >   		if (deliverable_mask) {
> >   			/* lately kicked but not yet running */
> > -			if (test_and_set_bit(vcpu_id, gi->kicked_mask))
> > +			if (test_and_set_bit(vcpu_idx, gi->kicked_mask))
> >   				return;
> >   			kvm_s390_vcpu_wakeup(vcpu);
> >   			return;
> > diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
> > index 4527ac7b5961..8580543c5bc3 100644
> > --- a/arch/s390/kvm/kvm-s390.c
> > +++ b/arch/s390/kvm/kvm-s390.c
> > @@ -4044,7 +4044,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)
> >   		kvm_s390_patch_guest_per_regs(vcpu);
> >   	}
> >   
> > -	clear_bit(vcpu->vcpu_id, vcpu->kvm->arch.gisa_int.kicked_mask);
> > +	clear_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.gisa_int.kicked_mask);
> >   
> >   	vcpu->arch.sie_block->icptcode = 0;
> >   	cpuflags = atomic_read(&vcpu->arch.sie_block->cpuflags);
> > diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/kvm-s390.h
> > index 9fad25109b0d..ecd741ee3276 100644
> > --- a/arch/s390/kvm/kvm-s390.h
> > +++ b/arch/s390/kvm/kvm-s390.h
> > @@ -79,7 +79,7 @@ static inline int is_vcpu_stopped(struct kvm_vcpu *vcpu)
> >   
> >   static inline int is_vcpu_idle(struct kvm_vcpu *vcpu)
> >   {
> > -	return test_bit(vcpu->vcpu_id, vcpu->kvm->arch.idle_mask);
> > +	return test_bit(kvm_vcpu_get_idx(vcpu), vcpu->kvm->arch.idle_mask);
> >   }
> >   
> >   static inline int kvm_is_ucontrol(struct kvm *kvm)
> > 
> > base-commit: 77dd11439b86e3f7990e4c0c9e0b67dca82750ba  

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Petr Tesařík]

Hi Halil,

Dne 31. 01. 22 v 12:53 Halil Pasic napsal(a):
> On Mon, 31 Jan 2022 11:13:18 +0100
> Petr Tesařík <ptesarik@suse.cz> wrote:
> 
>> Hi Halil,
>>
>> Dne 27. 08. 21 v 14:54 Halil Pasic napsal(a):
>>> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
>>> it may not always be, and we must not rely on this.
>>>
>>> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
>>> that code like
>>> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
>>>                   vcpu = kvm_get_vcpu(kvm, vcpu_id);
>>> 		do_stuff(vcpu);
>>> }
>>> is not legit. The trouble is, we do actually use kvm->arch.idle_mask
>>> like this. To fix this problem we have two options. Either use
>>> kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
>>> or switch to indexing via vcpu_idx. The latter is preferable for obvious
>>> reasons.
>>
>> I'm just backporting this fix to SLES 12 SP5, and I've noticed that
>> there is still this code in __floating_irq_kick():
>>
>> 	/* find idle VCPUs first, then round robin */
>> 	sigcpu = find_first_bit(fi->idle_mask, online_vcpus);
>> /* ... round robin loop removed ...
>> 	dst_vcpu = kvm_get_vcpu(kvm, sigcpu);
>>
>> It seems to me that this does exactly the thing that is not legit, but
>> I'm no expert. Did I miss something?
>>
> 
> We made that legit by making the N-th bit in idle_mask correspond to the
> vcpu whose vcpu_idx == N. The second argument of kvm_get_vcpu() is the
> vcpu_idx. IMHO that ain't super-intuitive because it ain't spelled out.
> 
> So before this was a mismatch (with a vcpu_id based bitmap we would have
> to use kvm_get_vcpu_by_id()), and with this patch applied this code
> becomes legit because both idle_mask and kvm_get_vcpu() operate with
> vcpu_idx.
> 
> Does that make sense?

Yes!

> I'm sorry the commit message did not convey this clearly enough...

No, it's not your fault. I didn't pay enough attention to the change, 
and with vcpu_id and vcpu_idx being so similar I got confused.

In short, there's no bug now, indeed. Thanks for your patience.

Petr T

Re: [PATCH 1/1] KVM: s390: index kvm->arch.idle_mask by vcpu_idx

[Halil Pasic]

On Mon, 31 Jan 2022 13:09:26 +0100
Petr Tesařík <ptesarik@suse.cz> wrote:

> Hi Halil,
> 
> Dne 31. 01. 22 v 12:53 Halil Pasic napsal(a):
> > On Mon, 31 Jan 2022 11:13:18 +0100
> > Petr Tesařík <ptesarik@suse.cz> wrote:
> >   
> >> Hi Halil,
> >>
> >> Dne 27. 08. 21 v 14:54 Halil Pasic napsal(a):  
> >>> While in practice vcpu->vcpu_idx ==  vcpu->vcp_id is often true,
> >>> it may not always be, and we must not rely on this.
> >>>
> >>> Currently kvm->arch.idle_mask is indexed by vcpu_id, which implies
> >>> that code like
> >>> for_each_set_bit(vcpu_id, kvm->arch.idle_mask, online_vcpus) {
> >>>                   vcpu = kvm_get_vcpu(kvm, vcpu_id);
> >>> 		do_stuff(vcpu);
> >>> }
> >>> is not legit. The trouble is, we do actually use kvm->arch.idle_mask
> >>> like this. To fix this problem we have two options. Either use
> >>> kvm_get_vcpu_by_id(vcpu_id), which would loop to find the right vcpu_id,
> >>> or switch to indexing via vcpu_idx. The latter is preferable for obvious
> >>> reasons.  
> >>
> >> I'm just backporting this fix to SLES 12 SP5, and I've noticed that
> >> there is still this code in __floating_irq_kick():
> >>
> >> 	/* find idle VCPUs first, then round robin */
> >> 	sigcpu = find_first_bit(fi->idle_mask, online_vcpus);
> >> /* ... round robin loop removed ...
> >> 	dst_vcpu = kvm_get_vcpu(kvm, sigcpu);
> >>
> >> It seems to me that this does exactly the thing that is not legit, but
> >> I'm no expert. Did I miss something?
> >>  
> > 
> > We made that legit by making the N-th bit in idle_mask correspond to the
> > vcpu whose vcpu_idx == N. The second argument of kvm_get_vcpu() is the
> > vcpu_idx. IMHO that ain't super-intuitive because it ain't spelled out.
> > 
> > So before this was a mismatch (with a vcpu_id based bitmap we would have
> > to use kvm_get_vcpu_by_id()), and with this patch applied this code
> > becomes legit because both idle_mask and kvm_get_vcpu() operate with
> > vcpu_idx.
> > 
> > Does that make sense?  
> 
> Yes!
> 
> > I'm sorry the commit message did not convey this clearly enough...  
> 
> No, it's not your fault. I didn't pay enough attention to the change, 
> and with vcpu_id and vcpu_idx being so similar I got confused.

No problem at all!

> 
> In short, there's no bug now, indeed. Thanks for your patience.
> 

Thank you for being mindful when backporting!

Regards,
Halil