[RFC PATCH 0/4] [RESEND] Change default MSGMNI tunable to scale with lowmem

[RFC PATCH 0/4] [RESEND] Change default MSGMNI tunable to scale with lowmem

[Nadia.Derbey]

Resending the set of patches after Andrew's remark about the fact that we
should take the lowmem into account, rather than the total ram.

Also, these patches have been enhanced with memory hotplug management.

Note: they now have to applied to 2.6.24-rc7.

-------------------

On large systems we'd like to allow a larger number of message queues.
In some cases up to 32K. However simply setting MSGMNI to a larger value may
cause problems for smaller systems.

The first patch of this series introduces a default maximum number of message
queue ids that scales with the amount of lowmem.

Since msgmni is per namespace and there is no amount of memory dedicated to
each namespace so far, the second patch of this series scales msgmni to
the number of ipc namespaces.

In the last patch, a notifier block is added to the ipc namespace structure to
manage memory hotplug. The callback routine is activated upon memory
add/remove and it recomputes msgmni. One callback routine is added to the
memory notifier chain each time an ipc namespace is allocated. It is removed
when the coresponding ipc namespace is freed.

I still have 1 issue that I'll try to solve next:
  . use the notification mechanism to recompute all the msg_ctlmni each time
    an ipc namespace is created / removed.
    

These patches should be applied to 2.6.24-rc7, in the following order:

[PATCH 1/4]: ipc_scale_msgmni_with_lowmem.patch
[PATCH 2/4]: ipc_scale_msgmni_with_namespaces.patch
[PATCH 3/4]: ipc_slab_memory_callback_prio_to_const.patch
[PATCH 4/4]: ipc_recompute_msgmni_on_memory_hotplug.patch


Regards,
Nadia

--

[RFC PATCH 1/4] [RESEND] Scaling msgmni to the amount of lowmem

[Nadia.Derbey]

[-- Attachment #1: ipc_scale_msgmni_with_lowmem.patch --]
[-- Type: text/plain, Size: 4161 bytes --]

[PATCH 01/04]

This patch computes msg_ctlmni to make it scale with the amount of lowmem.
msg_ctlmni is now set to make the message queues occupy 1/32 of the available
lowmem.

Some cleaning has also been done in the MSGXXX constants:
  . MSGPOOL: the msgctl man page says it's not used, but it also defines it as
             a size in bytes (the code expresses it in Kbytes).
  . MSGSEG definition has been removed since it used only once in msgctl().

Signed-off-by: Nadia Derbey <Nadia.Derbey@bull.net>

---
 include/linux/msg.h |   14 +++++++++++---
 ipc/msg.c           |   44 ++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 53 insertions(+), 5 deletions(-)

Index: linux-2.6.24-rc7/include/linux/msg.h
===================================================================
--- linux-2.6.24-rc7.orig/include/linux/msg.h	2008-01-11 14:08:41.000000000 +0100
+++ linux-2.6.24-rc7/include/linux/msg.h	2008-01-11 14:50:47.000000000 +0100
@@ -49,17 +49,25 @@ struct msginfo {
 	unsigned short  msgseg; 
 };
 
+/*
+ * Scaling factor to compute msgmni:
+ * the memory dedicated to msg queues (msgmni * msgmnb) should occupy
+ * at most 1/MSG_MEM_SCALE of the lowmem (see the formula in ipc/msg.c):
+ * up to 8MB       : msgmni = 16 (MSGMNI)
+ * 4 GB            : msgmni = 8K
+ * more than 16 GB : msgmni = 32K (IPCMNI)
+ */
+#define MSG_MEM_SCALE 32
+
 #define MSGMNI    16   /* <= IPCMNI */     /* max # of msg queue identifiers */
 #define MSGMAX  8192   /* <= INT_MAX */   /* max size of message (bytes) */
 #define MSGMNB 16384   /* <= INT_MAX */   /* default max size of a message queue */
 
 /* unused */
-#define MSGPOOL (MSGMNI*MSGMNB/1024)  /* size in kilobytes of message pool */
+#define MSGPOOL (MSGMNI * MSGMNB) /* size in bytes of message pool */
 #define MSGTQL  MSGMNB            /* number of system message headers */
 #define MSGMAP  MSGMNB            /* number of entries in message map */
 #define MSGSSZ  16                /* message segment size */
-#define __MSGSEG ((MSGPOOL*1024)/ MSGSSZ) /* max no. of segments */
-#define MSGSEG (__MSGSEG <= 0xffff ? __MSGSEG : 0xffff)
 
 #ifdef __KERNEL__
 #include <linux/list.h>
Index: linux-2.6.24-rc7/ipc/msg.c
===================================================================
--- linux-2.6.24-rc7.orig/ipc/msg.c	2008-01-11 14:08:48.000000000 +0100
+++ linux-2.6.24-rc7/ipc/msg.c	2008-01-11 15:51:00.000000000 +0100
@@ -27,6 +27,7 @@
 #include <linux/msg.h>
 #include <linux/spinlock.h>
 #include <linux/init.h>
+#include <linux/mm.h>
 #include <linux/proc_fs.h>
 #include <linux/list.h>
 #include <linux/security.h>
@@ -79,12 +80,51 @@ static int newque(struct ipc_namespace *
 static int sysvipc_msg_proc_show(struct seq_file *s, void *it);
 #endif
 
+/*
+ * Scale msgmni with the available lowmem size: the memory dedicated to msg
+ * queues should occupy at most 1/MSG_MEM_SCALE of lowmem.
+ */
+static void recompute_msgmni(struct ipc_namespace *ns)
+{
+	struct sysinfo i;
+	unsigned long allowed;
+
+	si_meminfo(&i);
+	allowed = (((i.totalram - i.totalhigh) / MSG_MEM_SCALE) * i.mem_unit)
+		/ MSGMNB;
+
+	/*
+	 * Never fall under the current minimum value
+	 */
+	if (allowed < MSGMNI) {
+		ns->msg_ctlmni = MSGMNI;
+		goto out_callback;
+	}
+
+	/*
+	 * Never go over the current max value
+	 */
+	if (allowed > IPCMNI) {
+		ns->msg_ctlmni = IPCMNI;
+		goto out_callback;
+	}
+
+	ns->msg_ctlmni = allowed;
+
+out_callback:
+
+	printk(KERN_INFO "msgmni has been set to %d for ipc namespace %p\n",
+		ns->msg_ctlmni, ns);
+}
+
 static void __msg_init_ns(struct ipc_namespace *ns, struct ipc_ids *ids)
 {
 	ns->ids[IPC_MSG_IDS] = ids;
 	ns->msg_ctlmax = MSGMAX;
 	ns->msg_ctlmnb = MSGMNB;
-	ns->msg_ctlmni = MSGMNI;
+
+	recompute_msgmni(ns);
+
 	atomic_set(&ns->msg_bytes, 0);
 	atomic_set(&ns->msg_hdrs, 0);
 	ipc_init_ids(ids);
@@ -458,7 +498,7 @@ asmlinkage long sys_msgctl(int msqid, in
 		msginfo.msgmax = ns->msg_ctlmax;
 		msginfo.msgmnb = ns->msg_ctlmnb;
 		msginfo.msgssz = MSGSSZ;
-		msginfo.msgseg = MSGSEG;
+		msginfo.msgseg = min(MSGPOOL / MSGSSZ, 0xffff);
 		down_read(&msg_ids(ns).rw_mutex);
 		if (cmd == MSG_INFO) {
 			msginfo.msgpool = msg_ids(ns).in_use;

--

[RFC PATCH 2/4] [RESEND] Scaling msgmni to the number of ipc namespaces

[Nadia.Derbey]

[-- Attachment #1: ipc_scale_msgmni_with_namespaces.patch --]
[-- Type: text/plain, Size: 2875 bytes --]

[PATCH 02/04]

Since all the namespaces see the same amount of memory (the total one)
this patch introduces a new variable that counts the ipc namespaces and
divides msg_ctlmni by this counter.

Signed-off-by: Nadia Derbey <Nadia.Derbey@bull.net>

---
 include/linux/ipc.h |    1 +
 ipc/msg.c           |    8 ++++++--
 ipc/util.c          |    7 +++++++
 3 files changed, 14 insertions(+), 2 deletions(-)

Index: linux-2.6.24-rc7/include/linux/ipc.h
===================================================================
--- linux-2.6.24-rc7.orig/include/linux/ipc.h	2008-01-11 15:50:21.000000000 +0100
+++ linux-2.6.24-rc7/include/linux/ipc.h	2008-01-11 16:00:50.000000000 +0100
@@ -121,6 +121,7 @@ struct ipc_namespace {
 };
 
 extern struct ipc_namespace init_ipc_ns;
+extern atomic_t nr_ipc_ns;
 
 #ifdef CONFIG_SYSVIPC
 #define INIT_IPC_NS(ns)		.ns		= &init_ipc_ns,
Index: linux-2.6.24-rc7/ipc/util.c
===================================================================
--- linux-2.6.24-rc7.orig/ipc/util.c	2008-01-11 15:50:21.000000000 +0100
+++ linux-2.6.24-rc7/ipc/util.c	2008-01-11 16:12:38.000000000 +0100
@@ -51,6 +51,9 @@ struct ipc_namespace init_ipc_ns = {
 	},
 };
 
+atomic_t nr_ipc_ns = ATOMIC_INIT(1);
+
+
 static struct ipc_namespace *clone_ipc_ns(struct ipc_namespace *old_ns)
 {
 	int err;
@@ -61,6 +64,8 @@ static struct ipc_namespace *clone_ipc_n
 	if (ns == NULL)
 		goto err_mem;
 
+	atomic_inc(&nr_ipc_ns);
+
 	err = sem_init_ns(ns);
 	if (err)
 		goto err_sem;
@@ -80,6 +85,7 @@ err_msg:
 	sem_exit_ns(ns);
 err_sem:
 	kfree(ns);
+	atomic_dec(&nr_ipc_ns);
 err_mem:
 	return ERR_PTR(err);
 }
@@ -109,6 +115,7 @@ void free_ipc_ns(struct kref *kref)
 	msg_exit_ns(ns);
 	shm_exit_ns(ns);
 	kfree(ns);
+	atomic_dec(&nr_ipc_ns);
 }
 
 /**
Index: linux-2.6.24-rc7/ipc/msg.c
===================================================================
--- linux-2.6.24-rc7.orig/ipc/msg.c	2008-01-11 15:51:00.000000000 +0100
+++ linux-2.6.24-rc7/ipc/msg.c	2008-01-11 16:08:32.000000000 +0100
@@ -83,15 +83,19 @@ static int sysvipc_msg_proc_show(struct 
 /*
  * Scale msgmni with the available lowmem size: the memory dedicated to msg
  * queues should occupy at most 1/MSG_MEM_SCALE of lowmem.
+ * Also take into account the number of nsproxies created so far.
  */
 static void recompute_msgmni(struct ipc_namespace *ns)
 {
 	struct sysinfo i;
 	unsigned long allowed;
+	int nb_ns;
 
 	si_meminfo(&i);
 	allowed = (((i.totalram - i.totalhigh) / MSG_MEM_SCALE) * i.mem_unit)
 		/ MSGMNB;
+	nb_ns = atomic_read(&nr_ipc_ns);
+	allowed /= nb_ns;
 
 	/*
 	 * Never fall under the current minimum value
@@ -104,8 +108,8 @@ static void recompute_msgmni(struct ipc_
 	/*
 	 * Never go over the current max value
 	 */
-	if (allowed > IPCMNI) {
-		ns->msg_ctlmni = IPCMNI;
+	if (allowed > IPCMNI / nb_ns) {
+		ns->msg_ctlmni = IPCMNI / nb_ns;
 		goto out_callback;
 	}
 

--

[RFC PATCH 3/4] [RESEND] Defining the slab_memory_callback priority as a constant

[Nadia.Derbey]

[-- Attachment #1: ipc_slab_memory_callback_prio_to_const.patch --]
[-- Type: text/plain, Size: 1492 bytes --]

[PATCH 03/04]

This is a trivial patch that defines the priority of slab_memory_callback in
the callback chain as a constant.
This is to prepare for next patch in the series.

Signed-off-by: Nadia Derbey <Nadia.Derbey@bull.net>

---
 include/linux/memory.h |    6 ++++++
 mm/slub.c              |    2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

Index: linux-2.6.24-rc7/include/linux/memory.h
===================================================================
--- linux-2.6.24-rc7.orig/include/linux/memory.h	2008-01-11 14:08:41.000000000 +0100
+++ linux-2.6.24-rc7/include/linux/memory.h	2008-01-11 16:26:03.000000000 +0100
@@ -54,6 +54,12 @@ struct memory_notify {
 struct notifier_block;
 struct mem_section;
 
+/*
+ * Priorities for the hotplug memory callback routines (stored in decreasing
+ * order in the callback chain)
+ */
+#define SLAB_CALLBACK_PRI       1
+
 #ifndef CONFIG_MEMORY_HOTPLUG_SPARSE
 static inline int memory_dev_init(void)
 {
Index: linux-2.6.24-rc7/mm/slub.c
===================================================================
--- linux-2.6.24-rc7.orig/mm/slub.c	2008-01-11 14:08:45.000000000 +0100
+++ linux-2.6.24-rc7/mm/slub.c	2008-01-11 16:27:11.000000000 +0100
@@ -2816,7 +2816,7 @@ void __init kmem_cache_init(void)
 	kmalloc_caches[0].refcount = -1;
 	caches++;
 
-	hotplug_memory_notifier(slab_memory_callback, 1);
+	hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI);
 #endif
 
 	/* Able to allocate the per node structures */

--

[RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

[Nadia.Derbey]

[-- Attachment #1: ipc_recompute_msgmni_on_memory_hotplug.patch --]
[-- Type: text/plain, Size: 6754 bytes --]

[PATCH 04/04]

This patch introduces the registration of a callback routine that recomputes
msg_ctlmni upon memory add / remove.

A notifier block structure has been added to the ipc namespace structure.

Each time a new ipc namespace is allocated, this notifier block callback
routine is registered into the memory hotplug notifier chain (it is
unregistered when the ipc namespace is freed). That routine is then activated
when memory is added or removed, in order to recompute msgmni for that
namespace.  


Signed-off-by: Nadia Derbey <Nadia.Derbey@bull.net>

---
 include/linux/ipc.h    |   25 +++++++++++++++++++++
 include/linux/memory.h |    1 
 ipc/msg.c              |    2 -
 ipc/util.c             |   57 +++++++++++++++++++++++++++++++++++++++++++++++++
 ipc/util.h             |    2 +
 5 files changed, 86 insertions(+), 1 deletion(-)

Index: linux-2.6.24-rc7/include/linux/ipc.h
===================================================================
--- linux-2.6.24-rc7.orig/include/linux/ipc.h	2008-01-11 16:00:50.000000000 +0100
+++ linux-2.6.24-rc7/include/linux/ipc.h	2008-01-11 16:52:04.000000000 +0100
@@ -81,6 +81,10 @@ struct ipc_kludge {
 
 #include <linux/kref.h>
 #include <linux/spinlock.h>
+#ifdef CONFIG_MEMORY_HOTPLUG
+#include <linux/notifier.h>
+#include <linux/memory.h>
+#endif /* CONFIG_MEMORY_HOTPLUG */
 
 #define IPCMNI 32768  /* <= MAX_INT limit for ipc arrays (including sysctl changes) */
 
@@ -118,6 +122,10 @@ struct ipc_namespace {
 	size_t		shm_ctlall;
 	int		shm_ctlmni;
 	int		shm_tot;
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+	struct notifier_block ipc_memory_hotplug;
+#endif
 };
 
 extern struct ipc_namespace init_ipc_ns;
@@ -153,6 +161,23 @@ static inline void put_ipc_ns(struct ipc
 #endif
 }
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+
+extern void register_ipc_hotplug_notifier(struct ipc_namespace *);
+
+static inline void unregister_ipc_hotplug_notifier(struct ipc_namespace *ns)
+{
+	unregister_memory_notifier(&ns->ipc_memory_hotplug);
+}
+
+#else /* CONFIG_MEMORY_HOTPLUG */
+
+#define register_ipc_hotplug_notifier(ns)   do { } while (0)
+#define unregister_ipc_hotplug_notifier(ns)  do { } while (0)
+
+#endif /* CONFIG_MEMORY_HOTPLUG */
+
+
 #endif /* __KERNEL__ */
 
 #endif /* _LINUX_IPC_H */
Index: linux-2.6.24-rc7/include/linux/memory.h
===================================================================
--- linux-2.6.24-rc7.orig/include/linux/memory.h	2008-01-11 16:26:03.000000000 +0100
+++ linux-2.6.24-rc7/include/linux/memory.h	2008-01-11 16:53:10.000000000 +0100
@@ -59,6 +59,7 @@ struct mem_section;
  * order in the callback chain)
  */
 #define SLAB_CALLBACK_PRI       1
+#define IPC_CALLBACK_PRI        10
 
 #ifndef CONFIG_MEMORY_HOTPLUG_SPARSE
 static inline int memory_dev_init(void)
Index: linux-2.6.24-rc7/ipc/util.h
===================================================================
--- linux-2.6.24-rc7.orig/ipc/util.h	2008-01-11 14:08:48.000000000 +0100
+++ linux-2.6.24-rc7/ipc/util.h	2008-01-11 16:56:23.000000000 +0100
@@ -134,6 +134,8 @@ extern int ipcget_new(struct ipc_namespa
 extern int ipcget_public(struct ipc_namespace *, struct ipc_ids *,
 			struct ipc_ops *, struct ipc_params *);
 
+extern void recompute_msgmni(struct ipc_namespace *);
+
 static inline int ipc_buildid(int id, int seq)
 {
 	return SEQ_MULTIPLIER * seq + id;
Index: linux-2.6.24-rc7/ipc/util.c
===================================================================
--- linux-2.6.24-rc7.orig/ipc/util.c	2008-01-11 16:12:38.000000000 +0100
+++ linux-2.6.24-rc7/ipc/util.c	2008-01-11 16:58:29.000000000 +0100
@@ -33,6 +33,7 @@
 #include <linux/audit.h>
 #include <linux/nsproxy.h>
 #include <linux/rwsem.h>
+#include <linux/memory.h>
 
 #include <asm/unistd.h>
 
@@ -54,6 +55,50 @@ struct ipc_namespace init_ipc_ns = {
 atomic_t nr_ipc_ns = ATOMIC_INIT(1);
 
 
+#ifdef CONFIG_MEMORY_HOTPLUG
+
+static int ipc_memory_callback(struct notifier_block *self,
+				unsigned long action, void *arg)
+{
+	struct ipc_namespace *ns;
+
+	switch (action) {
+	case MEM_ONLINE:    /* memory successfully brought online */
+	case MEM_OFFLINE:   /* or offline: it's time to recompute xxxmni */
+		ns = container_of(self, struct ipc_namespace,
+				ipc_memory_hotplug);
+		/*
+		 * No need to get a reference on the ns: the 1st job of
+		 * free_ipc_ns() is to unregister the callback routine.
+		 * blocking_notifier_chain_unregister takes the wr lock to do
+		 * it.
+		 * When this callback routine is called the rd lock is held by
+		 * blocking_notifier_call_chain.
+		 * So the ipc ns cannot be freed while we are here.
+		 */
+		recompute_msgmni(ns);
+		break;
+	case MEM_GOING_ONLINE:
+	case MEM_GOING_OFFLINE:
+	case MEM_CANCEL_ONLINE:
+	case MEM_CANCEL_OFFLINE:
+	default:
+		break;
+	}
+
+	return NOTIFY_OK;
+}
+
+void register_ipc_hotplug_notifier(struct ipc_namespace *ns)
+{
+	memset(&ns->ipc_memory_hotplug, 0, sizeof(ns->ipc_memory_hotplug));
+	ns->ipc_memory_hotplug.notifier_call = ipc_memory_callback;
+	ns->ipc_memory_hotplug.priority = IPC_CALLBACK_PRI;
+	register_memory_notifier(&ns->ipc_memory_hotplug);
+}
+
+#endif /* CONFIG_MEMORY_HOTPLUG */
+
 static struct ipc_namespace *clone_ipc_ns(struct ipc_namespace *old_ns)
 {
 	int err;
@@ -76,6 +121,8 @@ static struct ipc_namespace *clone_ipc_n
 	if (err)
 		goto err_shm;
 
+	register_ipc_hotplug_notifier(ns);
+
 	kref_init(&ns->kref);
 	return ns;
 
@@ -111,6 +158,15 @@ void free_ipc_ns(struct kref *kref)
 	struct ipc_namespace *ns;
 
 	ns = container_of(kref, struct ipc_namespace, kref);
+	/*
+	 * Unregistering the hotplug notifier at the beginning guarantees
+	 * that the ipc namespace won't be freed while we are inside the
+	 * the callback routine. Since the blocking_notifier_chain_XXX
+	 * routines hold a rw lock on the notifier list,
+	 * unregister_ipc_hotplug_notifier() won't take the rw lock before
+	 * blocking_notifier_call_chain() has released the rd lock.
+	 */
+	unregister_ipc_hotplug_notifier(ns);
 	sem_exit_ns(ns);
 	msg_exit_ns(ns);
 	shm_exit_ns(ns);
@@ -130,6 +186,7 @@ static int __init ipc_init(void)
 	sem_init();
 	msg_init();
 	shm_init();
+	register_ipc_hotplug_notifier(&init_ipc_ns);
 	return 0;
 }
 __initcall(ipc_init);
Index: linux-2.6.24-rc7/ipc/msg.c
===================================================================
--- linux-2.6.24-rc7.orig/ipc/msg.c	2008-01-11 16:08:32.000000000 +0100
+++ linux-2.6.24-rc7/ipc/msg.c	2008-01-11 17:03:31.000000000 +0100
@@ -85,7 +85,7 @@ static int sysvipc_msg_proc_show(struct 
  * queues should occupy at most 1/MSG_MEM_SCALE of lowmem.
  * Also take into account the number of nsproxies created so far.
  */
-static void recompute_msgmni(struct ipc_namespace *ns)
+void recompute_msgmni(struct ipc_namespace *ns)
 {
 	struct sysinfo i;
 	unsigned long allowed;

--

Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

[Yasunori Goto]

Hello Nadia-san.

> @@ -118,6 +122,10 @@ struct ipc_namespace {
>  	size_t		shm_ctlall;
>  	int		shm_ctlmni;
>  	int		shm_tot;
> +
> +#ifdef CONFIG_MEMORY_HOTPLUG
> +	struct notifier_block ipc_memory_hotplug;
> +#endif
>  };

I'm sorry, but I don't see why each ipc namespace must have each callbacks
of memory hotplug.
I prefer only one callback for each subsystem, not for each namespace.
In addition, the recompute_msgmni() calculation looks very similar for
all ipc namespace.
Or do you wish each ipc namespace have different callback for the future?



BTW, have you ever tested this patch? If you don't have any test environment
for memory hotplug code, then I'll check it. :-)

Bye.

-- 
Yasunori Goto 

Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

[Nadia Derbey]

Yasunori Goto wrote:
> Hello Nadia-san.
> 
> 
>>@@ -118,6 +122,10 @@ struct ipc_namespace {
>> 	size_t		shm_ctlall;
>> 	int		shm_ctlmni;
>> 	int		shm_tot;
>>+
>>+#ifdef CONFIG_MEMORY_HOTPLUG
>>+	struct notifier_block ipc_memory_hotplug;
>>+#endif
>> };
> 
> 
> I'm sorry, but I don't see why each ipc namespace must have each callbacks
> of memory hotplug.
> I prefer only one callback for each subsystem, not for each namespace.
> In addition, the recompute_msgmni() calculation looks very similar for
> all ipc namespace.
> Or do you wish each ipc namespace have different callback for the future?
> 

Actually, this is what I wanted to do at the very beginning: have a 
single callback that would recompute the msgmni for each ipc namespace. 
But the issue here is that the namespaces are not linked to each other, 
so I had no simple way to go through all the namespaces.
I solved the issue by having a callback for any single ipc namespace and 
make it recompute the msgmni value for itslef.

> 
> 
> BTW, have you ever tested this patch? If you don't have any test environment
> for memory hotplug code, then I'll check it. :-)

Well, I tested it but not in "real configuration": what I did is that I 
changed the status by hand under sysfs to offline. I also changed 
remove_memory() in mm/memory_hotplug.c in the following way (instead of 
returninf EINVAL):
1) decrease the total_ram pages
2) call memory_notify(MEM_OFFLINE, NULL)

and checked that the msgmni was recomputed.

But sure, if you are candidate to test it, that would be great!



Regards,
Nadia

Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

[Yasunori Goto]

> Yasunori Goto wrote:
> > Hello Nadia-san.
> > 
> > 
> >>@@ -118,6 +122,10 @@ struct ipc_namespace {
> >> 	size_t		shm_ctlall;
> >> 	int		shm_ctlmni;
> >> 	int		shm_tot;
> >>+
> >>+#ifdef CONFIG_MEMORY_HOTPLUG
> >>+	struct notifier_block ipc_memory_hotplug;
> >>+#endif
> >> };
> > 
> > 
> > I'm sorry, but I don't see why each ipc namespace must have each callbacks
> > of memory hotplug.
> > I prefer only one callback for each subsystem, not for each namespace.
> > In addition, the recompute_msgmni() calculation looks very similar for
> > all ipc namespace.
> > Or do you wish each ipc namespace have different callback for the future?
> > 
> 
> Actually, this is what I wanted to do at the very beginning: have a 
> single callback that would recompute the msgmni for each ipc namespace. 
> But the issue here is that the namespaces are not linked to each other, 
> so I had no simple way to go through all the namespaces.
> I solved the issue by having a callback for any single ipc namespace and 
> make it recompute the msgmni value for itslef.

The recompute_msg() must be called when new ipc_namespace is created/removed
as you mentioned. I think namespaces should be linked each other for it
in the end....



> > 
> > BTW, have you ever tested this patch? If you don't have any test environment
> > for memory hotplug code, then I'll check it. :-)
> 
> Well, I tested it but not in "real configuration": what I did is that I 
> changed the status by hand under sysfs to offline. I also changed 
> remove_memory() in mm/memory_hotplug.c in the following way (instead of 
> returninf EINVAL):
> 1) decrease the total_ram pages
> 2) call memory_notify(MEM_OFFLINE, NULL)
> 
> and checked that the msgmni was recomputed.

You can also online again after offline by writing sysfs.

> But sure, if you are candidate to test it, that would be great!

Ok. I'll check it too.
Bye.

-- 
Yasunori Goto 

Re: [RFC PATCH 4/4] [RESEND] Recomputing msgmni on memory add / remove

[Nadia Derbey]

Yasunori Goto wrote:
>>Yasunori Goto wrote:
>>
>>>Hello Nadia-san.
>>>
>>>
>>>
>>>>@@ -118,6 +122,10 @@ struct ipc_namespace {
>>>>	size_t		shm_ctlall;
>>>>	int		shm_ctlmni;
>>>>	int		shm_tot;
>>>>+
>>>>+#ifdef CONFIG_MEMORY_HOTPLUG
>>>>+	struct notifier_block ipc_memory_hotplug;
>>>>+#endif
>>>>};
>>>
>>>
>>>I'm sorry, but I don't see why each ipc namespace must have each callbacks
>>>of memory hotplug.
>>>I prefer only one callback for each subsystem, not for each namespace.
>>>In addition, the recompute_msgmni() calculation looks very similar for
>>>all ipc namespace.
>>>Or do you wish each ipc namespace have different callback for the future?
>>>
>>
>>Actually, this is what I wanted to do at the very beginning: have a 
>>single callback that would recompute the msgmni for each ipc namespace. 
>>But the issue here is that the namespaces are not linked to each other, 
>>so I had no simple way to go through all the namespaces.
>>I solved the issue by having a callback for any single ipc namespace and 
>>make it recompute the msgmni value for itslef.
> 
> 
> The recompute_msg() must be called when new ipc_namespace is created/removed
> as you mentioned. I think namespaces should be linked each other for it
> in the end....

Not if I do it the same way as for memory hotplug:
1) definee a "namespace event notifier"
2) insert another notifier block into the ipc namespace.
3) The callback routines in the notifier chain would be activated upon 
namespace creation / removal.

But I'm still thinking about it: Matt Helsley suggested that we might 
just copy the old namespace's value when creating a new namespace.

There's also the issue of an msgmni that has been changed via procfs: 
may be we should not unconditionally recompute it upon namespace 
creation / removal, so unregister the callback for the owning namespace 
as soon as msgmni has been changed from userspace.


Regards,
Nadia

Re: [RFC PATCH 0/4] [RESEND] Change default MSGMNI tunable to scale with lowmem

[Nadia Derbey]

I'm realizing that I wrote Andrew's address wrong. So please if ever 
you're answering in this thread correct his address by hand.
Sorry for that!
Now if you find it more convenient that I start a new thread with the 
right address, please tell me and I'll do it.

Regards,
Nadia

Re: [RFC PATCH 0/4] [RESEND] Change default MSGMNI tunable to scale with lowmem

[Matt Helsley]

On Tue, 2008-01-15 at 10:06 +0100, Nadia Derbey wrote:
> I'm realizing that I wrote Andrew's address wrong. So please if ever 
> you're answering in this thread correct his address by hand.
> Sorry for that!
> Now if you find it more convenient that I start a new thread with the 
> right address, please tell me and I'll do it.
> 
> Regards,
> Nadia

Not a big deal. If I do reply please double check that I haven't
forgotten. It's easy to miss -- I only just noticed this :)

Cheers,
	-Matt

[RFC PATCH 0/4] [RESEND] Change default MSGMNI tunable to scale with lowmem

[Nadia Derbey]

Hi,

Yesterday, I posted to lkml a series of patches that make the ipc 
tunable msgmni scale with lowmem (see thread 
http://lkml.org/lkml/2008/1/14/196).

Since these patches watch for memory hotplug notifications, I thought 
that comments from linux-mm people would be interesting.

Also, since these patches change the ipc_namespace structure, comments 
from people subscribed at the containers mailing list would be welcome too.

Notes:
1) please, Cc me since I'm not subscribed to linux-mm.
2) sorry for lkml subsribers who are receiving this mail for the 2nd 
time, but I wanted all the comments to be shared.


Here is patch 0, the complete series can be found in the thread 
mentioned above.

-----------------

On large systems we'd like to allow a larger number of message queues. 
In some cases up to 32K. However simply setting MSGMNI to a larger value 
may cause problems for smaller systems.

The first patch of this series introduces a default maximum number of 
message queue ids that scales with the amount of lowmem.

Since msgmni is per namespace and there is no amount of memory dedicated 
to each namespace so far, the second patch of this series scales msgmni 
to the number of ipc namespaces.

In the last patch, a notifier block is added to the ipc namespace 
structure to manage memory hotplug. The callback routine is activated 
upon memory add/remove and it recomputes msgmni. One callback routine is 
added to the memory notifier chain each time an ipc namespace is 
allocated. It is removed when the coresponding ipc namespace is freed.

I still have 1 issue that I'll try to solve next:
   . use the notification mechanism to recompute all the msg_ctlmni each
     time an ipc namespace is created / removed.


These patches should be applied to 2.6.24-rc7, in the following order:

[PATCH 1/4]: ipc_scale_msgmni_with_lowmem.patch
[PATCH 2/4]: ipc_scale_msgmni_with_namespaces.patch
[PATCH 3/4]: ipc_slab_memory_callback_prio_to_const.patch
[PATCH 4/4]: ipc_recompute_msgmni_on_memory_hotplug.patch


Regards,
Nadia

Re: [RFC PATCH 0/4] [RESEND] Change default MSGMNI tunable to scale with lowmem

[Nadia Derbey]

Resending with the right address for the containers mailing list.
So sorry...


Nadia Derbey wrote:
> Hi,
> 
> Yesterday, I posted to lkml a series of patches that make the ipc 
> tunable msgmni scale with lowmem (see thread 
> http://lkml.org/lkml/2008/1/14/196).
> 
> Since these patches watch for memory hotplug notifications, I thought 
> that comments from linux-mm people would be interesting.
> 
> Also, since these patches change the ipc_namespace structure, comments 
> from people subscribed at the containers mailing list would be welcome too.
> 
> Notes:
> 1) please, Cc me since I'm not subscribed to linux-mm.
> 2) sorry for lkml subsribers who are receiving this mail for the 2nd 
> time, but I wanted all the comments to be shared.
> 
> 
> Here is patch 0, the complete series can be found in the thread 
> mentioned above.
> 
> -----------------
> 
> On large systems we'd like to allow a larger number of message queues. 
> In some cases up to 32K. However simply setting MSGMNI to a larger value 
> may cause problems for smaller systems.
> 
> The first patch of this series introduces a default maximum number of 
> message queue ids that scales with the amount of lowmem.
> 
> Since msgmni is per namespace and there is no amount of memory dedicated 
> to each namespace so far, the second patch of this series scales msgmni 
> to the number of ipc namespaces.
> 
> In the last patch, a notifier block is added to the ipc namespace 
> structure to manage memory hotplug. The callback routine is activated 
> upon memory add/remove and it recomputes msgmni. One callback routine is 
> added to the memory notifier chain each time an ipc namespace is 
> allocated. It is removed when the coresponding ipc namespace is freed.
> 
> I still have 1 issue that I'll try to solve next:
>   . use the notification mechanism to recompute all the msg_ctlmni each
>     time an ipc namespace is created / removed.
> 
> 
> These patches should be applied to 2.6.24-rc7, in the following order:
> 
> [PATCH 1/4]: ipc_scale_msgmni_with_lowmem.patch
> [PATCH 2/4]: ipc_scale_msgmni_with_namespaces.patch
> [PATCH 3/4]: ipc_slab_memory_callback_prio_to_const.patch
> [PATCH 4/4]: ipc_recompute_msgmni_on_memory_hotplug.patch
> 
> 
> Regards,
> Nadia
>