LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* [SLUB 0/2] SLUB: The unqueued slab allocator V6
@ 2007-03-31 19:30 Christoph Lameter
  2007-03-31 19:31 ` [SLUB 1/2] SLUB core Christoph Lameter
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Christoph Lameter @ 2007-03-31 19:30 UTC (permalink / raw)
  To: akpm; +Cc: linux-mm, linux-kernel, Christoph Lameter, mpm

[PATCH] SLUB The unqueued slab allocator v6

Note that the definition of the return type of ksize() is currently
different between mm and Linus' tree. Patch is conforming to mm.
This patch also needs sprint_symbol() support from mm.

V5->V6:

- Straighten out various coding issues u.a. to make the hot path clearer
  in slab_alloc and slab_free. This adds more gotos. sigh.

- Detailed alloc / free tracking including pid, cpu, time of alloc / free
  if SLAB_STORE_USER is enabled or slub_debug=U specified on boot.

- sysfs support via /sys/slab. Drop /proc/slubinfo support.
  Include slabinfo tool that produces an output similar to what
  /proc/slabinfo does. Tool needs to be made more sophisticated
  to allow control of various slub options at runtime. Currently
  reports total slab sizes, slab fragmentation and slab effectiveness
  (actual object use vs. slab space use).

- Runtime debug option changes per slab via /sys/slab/<slabcache>.
  All slab debug options can be configured via sysfs provided that
  no objects have been allocated yet.

- Deal with i386 use of slab page structs. Main patch disables
  slub for i386 (CONFIG_ARCH_USES_SLAB_PAGE_STRUCT). Then a special
  patch removes the page sized slabs and removes that setting.
  See the caveats in that patch for further details.

V4->V5:

- Single object slabs only for slabs > slub_max_order otherwise generate
  sufficient objects to avoid frequent use of the page allocator. This is
  necessary to compensate for fragmentation caused by frequent uses of
  the page allocator. We expect slabs of PAGE_SIZE from this rule since
  multi object slabs require uses of fields that are in use on i386 and
  x86_64. See the quicklist patchset for a way to fix that issue
  and a patch to get rid of the PAGE_SIZE special casing.

- Drop pass through to page allocator due to page allocator fragmenting
  memory. The buffering through large order allocations is done in SLUB.
  Infrequent larger order allocations cause less fragmentation
  than frequent small order allocations.

- We need to update object sizes when merging slabs otherwise kzalloc
  will not initialize the full object (this caused the failure on
  various platforms).

- Padding checks before redzone checks so that we get messages about
  the corruption of whole slab and not about a single object.

V3->V4
- Rename /proc/slabinfo to /proc/slubinfo. We have a different format after
  all.
- More bug fixes and stabilization of diagnostic functions. This seems
  to be finally something that works wherever we test it.
- Serialize kmem_cache_create and kmem_cache_destroy via slub_lock (Adrian's
  idea)
- Add two new modifications (separate patches) to guarantee
  a mininum number of objects per slab and to pass through large
  allocations.

V2->V3
- Debugging and diagnostic support. This is runtime enabled and not compile
  time enabled. Runtime debugging can be controlled via kernel boot options
  on an individual slab cache basis or globally.
- Slab Trace support (For individual slab caches).
- Resiliency support: If basic sanity checks are enabled (via F f.e.)
  (boot option) then SLUB will do the best to perform diagnostics and
  then continue (i.e. mark corrupted objects as used).
- Fix up numerous issues including clash of SLUBs use of page
  flags with i386 arch use for pmd and pgds (which are managed
  as slab caches, sigh).
- Dynamic per CPU array sizing.
- Explain SLUB slabcache flags

V1->V2
- Fix up various issues. Tested on i386 UP, X86_64 SMP, ia64 NUMA.
- Provide NUMA support by splitting partial lists per node.
- Better Slab cache merge support (now at around 50% of slabs)
- List slab cache aliases if slab caches are merged.
- Updated descriptions /proc/slabinfo output

This is a new slab allocator which was motivated by the complexity of the
existing code in mm/slab.c. It attempts to address a variety of concerns
with the existing implementation.

A. Management of object queues

   A particular concern was the complex management of the numerous object
   queues in SLAB. SLUB has no such queues. Instead we dedicate a slab for
   each allocating CPU and use objects from a slab directly instead of
   queueing them up.

B. Storage overhead of object queues

   SLAB Object queues exist per node, per CPU. The alien cache queue even
   has a queue array that contain a queue for each processor on each
   node. For very large systems the number of queues and the number of
   objects that may be caught in those queues grows exponentially. On our
   systems with 1k nodes / processors we have several gigabytes just tied up
   for storing references to objects for those queues  This does not include
   the objects that could be on those queues. One fears that the whole
   memory of the machine could one day be consumed by those queues.

C. SLAB meta data overhead

   SLAB has overhead at the beginning of each slab. This means that data
   cannot be naturally aligned at the beginning of a slab block. SLUB keeps
   all meta data in the corresponding page_struct. Objects can be naturally
   aligned in the slab. F.e. a 128 byte object will be aligned at 128 byte
   boundaries and can fit tightly into a 4k page with no bytes left over.
   SLAB cannot do this.

D. SLAB has a complex cache reaper

   SLUB does not need a cache reaper for UP systems. On SMP systems
   the per CPU slab may be pushed back into partial list but that
   operation is simple and does not require an iteration over a list
   of objects. SLAB expires per CPU, shared and alien object queues
   during cache reaping which may cause strange hold offs.

E. SLAB has complex NUMA policy layer support

   SLUB pushes NUMA policy handling into the page allocator. This means that
   allocation is coarser (SLUB does interleave on a page level) but that
   situation was also present before 2.6.13. SLABs application of
   policies to individual slab objects allocated in SLAB is
   certainly a performance concern due to the frequent references to
   memory policies which may lead a sequence of objects to come from
   one node after another. SLUB will get a slab full of objects
   from one node and then will switch to the next.

F. Reduction of the size of partial slab lists

   SLAB has per node partial lists. This means that over time a large
   number of partial slabs may accumulate on those lists. These can
   only be reused if allocator occur on specific nodes. SLUB has a global
   pool of partial slabs and will consume slabs from that pool to
   decrease fragmentation.

G. Tunables

   SLAB has sophisticated tuning abilities for each slab cache. One can
   manipulate the queue sizes in detail. However, filling the queues still
   requires the uses of the spin lock to check out slabs. SLUB has a global
   parameter (min_slab_order) for tuning. Increasing the minimum slab
   order can decrease the locking overhead. The bigger the slab order the
   less motions of pages between per CPU and partial lists occur and the
   better SLUB will be scaling.

G. Slab merging

   We often have slab caches with similar parameters. SLUB detects those
   on boot up and merges them into the corresponding general caches. This
   leads to more effective memory use. About 50% of all caches can
   be eliminated through slab merging. This will also decrease
   slab fragmentation because partial allocated slabs can be filled
   up again. Slab merging can be switched off by specifying
   slub_nomerge on boot up.

   Note that merging can expose heretofore unknown bugs in the kernel
   because corrupted objects may now be placed differently and corrupt
   differing neighboring objects. Enable sanity checks to find those.

H. Diagnostics

   The current slab diagnostics are difficult to use and require a
   recompilation of the kernel. SLUB contains debugging code that
   is always available (but is kept out of the hot code paths).
   SLUB diagnostics can be enabled via the "slab_debug" option.
   Parameters can be specified to select a single or a group of
   slab caches for diagnostics. This means that the system is running
   with the usual performance and it is much more likely that
   race conditions can be reproduced.

I. Resiliency

   If basic sanity checks are on then SLUB is capable of detecting
   common error conditions and recover as best as possible to allow the
   system to continue.

J. Tracing

   Tracing can be enabled via the slab_debug=T,<slabcache> option
   during boot. SLUB will then protocol all actions on that slabcache
   and dump the object contents on free.

K. On demand DMA cache creation.

   Generally DMA caches are not needed. If a kmalloc is used with
   __GFP_DMA then just create this single slabcache that is needed.
   For systems that have no ZONE_DMA requirement the support is
   completely eliminated.

L. Performance increase

   Some benchmarks have shown speed improvements on kernbench in the
   range of 5-10%. The locking overhead of slub is based on the
   underlying base allocation size. If we can reliably allocate
   larger order pages then it is possible to increase slub
   performance much further. The anti-fragmentation patches may
   enable further performance increases.

Tested on:
i386 UP + SMP, x86_64 UP + SMP + NUMA emulation, IA64 NUMA + Simulator

SLUB Boot options

slub_nomerge		Disable merging of slabs
slub_min_order=x	Require a minimum order for slab caches. This
			increases the managed chunk size and therefore
			reduces meta data and locking overhead.
slub_min_objects=x	Mininum objects per slab. Default is 8.
slub_max_order=x	Avoid generating slabs larger than order specified.
slub_debug		Enable all diagnostics for all caches
slub_debug=<options>	Enable selective options for all caches
slub_debug=<o>,<cache>	Enable selective options for a certain set of
			caches

Available Debug options
F		Double Free checking, sanity and resiliency
R		Red zoning
P		Object / padding poisoning
U		Track last free / alloc
T		Trace all allocs / frees (only use for individual slabs).

To use SLUB: Apply this patch and then select SLUB as the default slab
allocator.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [SLUB 1/2] SLUB core
  2007-03-31 19:30 [SLUB 0/2] SLUB: The unqueued slab allocator V6 Christoph Lameter
@ 2007-03-31 19:31 ` Christoph Lameter
  2007-04-03  7:29   ` Nick Piggin
  2007-03-31 19:31 ` [SLUB 2/2] i386 arch page size slab fixes Christoph Lameter
  2007-03-31 19:31 ` [SLUB tool] slabinfo: Display slab statistics Christoph Lameter
  2 siblings, 1 reply; 10+ messages in thread
From: Christoph Lameter @ 2007-03-31 19:31 UTC (permalink / raw)
  To: akpm; +Cc: linux-mm, mpm, linux-kernel, Christoph Lameter

SLUB Core patch V6

This provides basic SLUB functionality and allows a choice of
slab allocators during kernel configuration. The default is still
slab. SLUB has been tested in various configurations but I think we
can be quite sure that there are still remaining issues.

SLUB is not selectable on platforms that modify the page structs of
slab memory (i386, FRV) but see the next patch for a i386 fix.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.21-rc5-mm3/include/linux/mm_types.h
===================================================================
--- linux-2.6.21-rc5-mm3.orig/include/linux/mm_types.h	2007-03-30 21:50:18.000000000 -0700
+++ linux-2.6.21-rc5-mm3/include/linux/mm_types.h	2007-03-30 21:50:42.000000000 -0700
@@ -19,10 +19,16 @@ struct page {
 	unsigned long flags;		/* Atomic flags, some possibly
 					 * updated asynchronously */
 	atomic_t _count;		/* Usage count, see below. */
-	atomic_t _mapcount;		/* Count of ptes mapped in mms,
+	union {
+		atomic_t _mapcount;	/* Count of ptes mapped in mms,
 					 * to show when page is mapped
 					 * & limit reverse map searches.
 					 */
+		struct {	/* SLUB uses */
+			short unsigned int inuse;
+			short unsigned int offset;
+		};
+	};
 	union {
 	    struct {
 		unsigned long private;		/* Mapping-private opaque data:
@@ -43,8 +49,15 @@ struct page {
 #if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS
 	    spinlock_t ptl;
 #endif
+	    struct {			/* SLUB uses */
+		struct page *first_page;	/* Compound pages */
+		struct kmem_cache *slab;	/* Pointer to slab */
+	    };
+	};
+	union {
+		pgoff_t index;		/* Our offset within mapping. */
+		void *freelist;		/* SLUB: pointer to free object */
 	};
-	pgoff_t index;			/* Our offset within mapping. */
 	struct list_head lru;		/* Pageout list, eg. active_list
 					 * protected by zone->lru_lock !
 					 */
Index: linux-2.6.21-rc5-mm3/include/linux/slab.h
===================================================================
--- linux-2.6.21-rc5-mm3.orig/include/linux/slab.h	2007-03-30 21:50:18.000000000 -0700
+++ linux-2.6.21-rc5-mm3/include/linux/slab.h	2007-03-30 21:50:42.000000000 -0700
@@ -32,6 +32,7 @@ typedef struct kmem_cache kmem_cache_t _
 #define SLAB_PANIC		0x00040000UL	/* Panic if kmem_cache_create() fails */
 #define SLAB_DESTROY_BY_RCU	0x00080000UL	/* Defer freeing slabs to RCU */
 #define SLAB_MEM_SPREAD		0x00100000UL	/* Spread some memory over cpuset */
+#define SLAB_TRACE		0x00200000UL	/* Trace allocations and frees */
 
 /* Flags passed to a constructor functions */
 #define SLAB_CTOR_CONSTRUCTOR	0x001UL		/* If not set, then deconstructor */
@@ -42,7 +43,7 @@ typedef struct kmem_cache kmem_cache_t _
  * struct kmem_cache related prototypes
  */
 void __init kmem_cache_init(void);
-extern int slab_is_available(void);
+int slab_is_available(void);
 
 struct kmem_cache *kmem_cache_create(const char *, size_t, size_t,
 			unsigned long,
@@ -95,9 +96,14 @@ static inline void *kcalloc(size_t n, si
  * the appropriate general cache at compile time.
  */
 
-#ifdef CONFIG_SLAB
+#if defined(CONFIG_SLAB) || defined(CONFIG_SLUB)
+#ifdef CONFIG_SLUB
+#include <linux/slub_def.h>
+#else
 #include <linux/slab_def.h>
+#endif /* !CONFIG_SLUB */
 #else
+
 /*
  * Fallback definitions for an allocator not wanting to provide
  * its own optimized kmalloc definitions (like SLOB).
@@ -184,7 +190,7 @@ static inline void *__kmalloc_node(size_
  * allocator where we care about the real place the memory allocation
  * request comes from.
  */
-#ifdef CONFIG_DEBUG_SLAB
+#if defined(CONFIG_DEBUG_SLAB) || defined(CONFIG_SLUB)
 extern void *__kmalloc_track_caller(size_t, gfp_t, void*);
 #define kmalloc_track_caller(size, flags) \
 	__kmalloc_track_caller(size, flags, __builtin_return_address(0))
@@ -202,7 +208,7 @@ extern void *__kmalloc_track_caller(size
  * standard allocator where we care about the real place the memory
  * allocation request comes from.
  */
-#ifdef CONFIG_DEBUG_SLAB
+#if defined(CONFIG_DEBUG_SLAB) || defined(CONFIG_SLUB)
 extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, void *);
 #define kmalloc_node_track_caller(size, flags, node) \
 	__kmalloc_node_track_caller(size, flags, node, \
Index: linux-2.6.21-rc5-mm3/include/linux/slub_def.h
===================================================================
--- /dev/null	1970-01-01 00:00:00.000000000 +0000
+++ linux-2.6.21-rc5-mm3/include/linux/slub_def.h	2007-03-30 22:31:02.000000000 -0700
@@ -0,0 +1,206 @@
+#ifndef _LINUX_SLUB_DEF_H
+#define _LINUX_SLUB_DEF_H
+
+/*
+ * SLUB : A Slab allocator without object queues.
+ *
+ * (C) 2007 SGI, Christoph Lameter <clameter@sgi.com>
+ */
+#include <linux/types.h>
+#include <linux/gfp.h>
+#include <linux/workqueue.h>
+#include <linux/kobject.h>
+
+struct kmem_cache_node {
+	spinlock_t list_lock;	/* Protect partial list and nr_partial */
+	unsigned long nr_partial;
+	atomic_long_t nr_slabs;
+	struct list_head partial;
+};
+
+/*
+ * Slab cache management.
+ */
+struct kmem_cache {
+	/* Used for retriving partial slabs etc */
+	unsigned long flags;
+	int size;		/* The size of an object including meta data */
+	int objsize;		/* The size of an object without meta data */
+	int offset;		/* Free pointer offset. */
+	atomic_t cpu_slabs;	/* != 0 -> flusher scheduled. */
+	int defrag_ratio;
+	unsigned int order;
+
+	/*
+	 * Avoid an extra cache line for UP, SMP and for the node local to
+	 * struct kmem_cache.
+	 */
+	struct kmem_cache_node local_node;
+
+	/* Allocation and freeing of slabs */
+	int objects;		/* Number of objects in slab */
+	int refcount;		/* Refcount for slab cache destroy */
+	void (*ctor)(void *, struct kmem_cache *, unsigned long);
+	void (*dtor)(void *, struct kmem_cache *, unsigned long);
+	int inuse;		/* Offset to metadata */
+	int align;		/* Alignment */
+	const char *name;	/* Name (only for display!) */
+	struct list_head list;	/* List of slab caches */
+	struct kobject kobj;	/* For sysfs */
+
+#ifdef CONFIG_SMP
+	struct delayed_work flush;
+	struct mutex flushing;
+#endif
+#ifdef CONFIG_NUMA
+	struct kmem_cache_node *node[MAX_NUMNODES];
+#endif
+	struct page *cpu_slab[NR_CPUS];
+};
+
+/*
+ * Kmalloc subsystem.
+ */
+#define KMALLOC_SHIFT_LOW 3
+
+#ifdef CONFIG_LARGE_ALLOCS
+#define KMALLOC_SHIFT_HIGH 25
+#else
+#if !defined(CONFIG_MMU) || NR_CPUS > 512 || MAX_NUMNODES > 256
+#define KMALLOC_SHIFT_HIGH 20
+#else
+#define KMALLOC_SHIFT_HIGH 18
+#endif
+#endif
+
+/*
+ * We keep the general caches in an array of slab caches that are used for
+ * 2^x bytes of allocations.
+ */
+extern struct kmem_cache kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
+
+/*
+ * Sorry that the following has to be that ugly but some versions of GCC
+ * have trouble with constant propagation and loops.
+ */
+static inline int kmalloc_index(int size)
+{
+	if (size == 0)
+		return 0;
+	if (size > 64 && size <= 96)
+		return 1;
+	if (size > 128 && size <= 192)
+		return 2;
+	if (size <=          8) return 3;
+	if (size <=         16) return 4;
+	if (size <=         32) return 5;
+	if (size <=         64) return 6;
+	if (size <=        128) return 7;
+	if (size <=        256) return 8;
+	if (size <=        512) return 9;
+	if (size <=       1024) return 10;
+	if (size <=   2 * 1024) return 11;
+	if (size <=   4 * 1024) return 12;
+	if (size <=   8 * 1024) return 13;
+	if (size <=  16 * 1024) return 14;
+	if (size <=  32 * 1024) return 15;
+	if (size <=  64 * 1024) return 16;
+	if (size <= 128 * 1024) return 17;
+	if (size <= 256 * 1024) return 18;
+#if KMALLOC_SHIFT_HIGH > 18
+	if (size <=  512 * 1024) return 19;
+	if (size <= 1024 * 1024) return 20;
+#endif
+#if KMALLOC_SHIFT_HIGH > 20
+	if (size <=  2 * 1024 * 1024) return 21;
+	if (size <=  4 * 1024 * 1024) return 22;
+	if (size <=  8 * 1024 * 1024) return 23;
+	if (size <= 16 * 1024 * 1024) return 24;
+	if (size <= 32 * 1024 * 1024) return 25;
+#endif
+	return -1;
+
+/*
+ * What we really wanted to do and cannot do because of compiler issues is:
+ *	int i;
+ *	for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
+ *		if (size <= (1 << i))
+ *			return i;
+ */
+}
+
+/*
+ * Find the slab cache for a given combination of allocation flags and size.
+ *
+ * This ought to end up with a global pointer to the right cache
+ * in kmalloc_caches.
+ */
+static inline struct kmem_cache *kmalloc_slab(size_t size)
+{
+	int index = kmalloc_index(size);
+
+	if (index == 0)
+		return NULL;
+
+	if (index < 0) {
+		/*
+		 * Generate a link failure. Would be great if we could
+		 * do something to stop the compile here.
+		 */
+		extern void __kmalloc_size_too_large(void);
+		__kmalloc_size_too_large();
+	}
+	return &kmalloc_caches[index];
+}
+
+#ifdef CONFIG_ZONE_DMA
+#define SLUB_DMA __GFP_DMA
+#else
+/* Disable DMA functionality */
+#define SLUB_DMA 0
+#endif
+
+static inline void *kmalloc(size_t size, gfp_t flags)
+{
+	if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
+		struct kmem_cache *s = kmalloc_slab(size);
+
+		if (!s)
+			return NULL;
+
+		return kmem_cache_alloc(s, flags);
+	} else
+		return __kmalloc(size, flags);
+}
+
+static inline void *kzalloc(size_t size, gfp_t flags)
+{
+	if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
+		struct kmem_cache *s = kmalloc_slab(size);
+
+		if (!s)
+			return NULL;
+
+		return kmem_cache_zalloc(s, flags);
+	} else
+		return __kzalloc(size, flags);
+}
+
+#ifdef CONFIG_NUMA
+extern void *__kmalloc_node(size_t size, gfp_t flags, int node);
+
+static inline void *kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	if (__builtin_constant_p(size) && !(flags & SLUB_DMA)) {
+		struct kmem_cache *s = kmalloc_slab(size);
+
+		if (!s)
+			return NULL;
+
+		return kmem_cache_alloc_node(s, flags, node);
+	} else
+		return __kmalloc_node(size, flags, node);
+}
+#endif
+
+#endif /* _LINUX_SLUB_DEF_H */
Index: linux-2.6.21-rc5-mm3/init/Kconfig
===================================================================
--- linux-2.6.21-rc5-mm3.orig/init/Kconfig	2007-03-30 21:50:19.000000000 -0700
+++ linux-2.6.21-rc5-mm3/init/Kconfig	2007-03-30 21:50:42.000000000 -0700
@@ -496,15 +496,6 @@ config SHMEM
 	  option replaces shmem and tmpfs with the much simpler ramfs code,
 	  which may be appropriate on small systems without swap.
 
-config SLAB
-	default y
-	bool "Use full SLAB allocator" if (EMBEDDED && !SMP && !SPARSEMEM)
-	help
-	  Disabling this replaces the advanced SLAB allocator and
-	  kmalloc support with the drastically simpler SLOB allocator.
-	  SLOB is more space efficient but does not scale well and is
-	  more susceptible to fragmentation.
-
 config VM_EVENT_COUNTERS
 	default y
 	bool "Enable VM event counters for /proc/vmstat" if EMBEDDED
@@ -514,6 +505,46 @@ config VM_EVENT_COUNTERS
 	  on EMBEDDED systems.  /proc/vmstat will only show page counts
 	  if VM event counters are disabled.
 
+choice
+	prompt "Choose SLAB allocator"
+	default SLAB
+	help
+	   This option allows to select a slab allocator.
+
+config SLAB
+	bool "SLAB"
+	help
+	  The regular slab allocator that is established and known to work
+	  well in all environments. It organizes chache hot objects in
+	  per cpu and per node queues. SLAB is the default choice for
+	  slab allocator.
+
+config SLUB
+	depends on EXPERIMENTAL && !ARCH_USES_SLAB_PAGE_STRUCT
+	bool "SLUB (Unqueued Allocator)"
+	help
+	   SLUB is a slab allocator that minimizes cache line usage
+	   instead of managing queues of cached objects (SLAB approach).
+	   Per cpu caching is realized using slabs of objects instead
+	   of queues of objects. SLUB can use memory efficiently
+	   way and has enhanced diagnostics.
+
+config SLOB
+#
+#	SLOB cannot support SMP because SLAB_DESTROY_BY_RCU does not work
+#	properly.
+#
+	depends on EMBEDDED && !SMP && !SPARSEMEM
+	bool "SLOB (Simple Allocator)"
+	help
+	   SLOB replaces the SLAB allocator with a drastically simpler
+	   allocator.  SLOB is more space efficient that SLAB but does not
+	   scale well (single lock for all operations) and is more susceptible
+	   to fragmentation. SLOB it is a great choice to reduce
+	   memory usage and code size for embedded systems.
+
+endchoice
+
 endmenu		# General setup
 
 config RT_MUTEXES
@@ -529,10 +560,6 @@ config BASE_SMALL
 	default 0 if BASE_FULL
 	default 1 if !BASE_FULL
 
-config SLOB
-	default !SLAB
-	bool
-
 config PAGE_GROUP_BY_MOBILITY
 	bool "Group pages based on their mobility in the page allocator"
 	def_bool y
Index: linux-2.6.21-rc5-mm3/mm/Makefile
===================================================================
--- linux-2.6.21-rc5-mm3.orig/mm/Makefile	2007-03-30 21:50:19.000000000 -0700
+++ linux-2.6.21-rc5-mm3/mm/Makefile	2007-03-30 21:50:42.000000000 -0700
@@ -26,6 +26,7 @@ obj-$(CONFIG_TMPFS_POSIX_ACL) += shmem_a
 obj-$(CONFIG_TINY_SHMEM) += tiny-shmem.o
 obj-$(CONFIG_SLOB) += slob.o
 obj-$(CONFIG_SLAB) += slab.o
+obj-$(CONFIG_SLUB) += slub.o
 obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
Index: linux-2.6.21-rc5-mm3/mm/slub.c
===================================================================
--- /dev/null	1970-01-01 00:00:00.000000000 +0000
+++ linux-2.6.21-rc5-mm3/mm/slub.c	2007-03-30 22:31:02.000000000 -0700
@@ -0,0 +1,2831 @@
+/*
+ * SLUB: A slab allocator that limits cache line use instead of queuing
+ * objects in per cpu and per node lists.
+ *
+ * The allocator synchronizes using per slab locks and only
+ * uses a centralized lock to manage a pool of partial slabs.
+ *
+ * (C) 2007 SGI, Christoph Lameter <clameter@sgi.com>
+ */
+
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/bit_spinlock.h>
+#include <linux/interrupt.h>
+#include <linux/bitops.h>
+#include <linux/slab.h>
+#include <linux/seq_file.h>
+#include <linux/cpu.h>
+#include <linux/cpuset.h>
+#include <linux/mempolicy.h>
+#include <linux/ctype.h>
+#include <linux/kallsyms.h>
+
+/*
+ * Lock order:
+ *   1. slab_lock(page)
+ *   2. slab->list_lock
+ *
+ * SLUB assigns one slab for allocation to each processor.
+ * Allocations only occur from these slabs called cpu slabs.
+ *
+ * If a cpu slab exists then a workqueue thread checks every 10
+ * seconds if the cpu slab is still in use. The cpu slab is pushed back
+ * to the list if inactive [only needed for SMP].
+ *
+ * Slabs with free elements are kept on a partial list.
+ * There is no list for full slabs. If an object in a full slab is
+ * freed then the slab will show up again on the partial lists.
+ * Otherwise there is no need to track full slabs (but we keep a counter).
+ *
+ * Slabs are freed when they become empty. Teardown and setup is
+ * minimal so we rely on the page allocators per cpu caches for
+ * fast frees and allocs.
+ *
+ * Overloading of page flags that are otherwise used for LRU management.
+ *
+ * PageActive 		The slab is used as a cpu cache. Allocations
+ * 			may be performed from the slab. The slab is not
+ * 			on a partial list.
+ *
+ * PageReferenced	The per cpu slab was used recently. This is used
+ * 			to push back per cpu slabs if they are unused
+ * 			for a longer time period.
+ *
+ * PageError		Slab requires special handling due to debug
+ * 			options set or a single page slab. This moves
+ * 			slab handling out of the fast path.
+ */
+
+/*
+ * Issues still to be resolved:
+ *
+ * - The per cpu array is updated for each new slab and and is a remote
+ *   cacheline for most nodes. This could become a bouncing cacheline given
+ *   enough frequent updates. There are 16 pointers in a cacheline.so at
+ *   max 16 cpus could compete. Likely okay.
+ *
+ * - Support PAGE_ALLOC_DEBUG. Should be easy to do.
+ *
+ * - Support DEBUG_SLAB_LEAK. Trouble is we do not know where the full
+ *   slabs are in SLUB.
+ *
+ * - SLAB_DEBUG_INITIAL is not supported but I have never seen a use of
+ *   it.
+ *
+ * - Variable sizing of the per node arrays
+ */
+
+/*
+ * Flags from the regular SLAB that SLUB does not support:
+ */
+#define SLUB_UNIMPLEMENTED (SLAB_DEBUG_INITIAL)
+
+#define DEBUG_DEFAULT_FLAGS (SLAB_DEBUG_FREE | SLAB_RED_ZONE | \
+				SLAB_POISON | SLAB_STORE_USER)
+/*
+ * Set of flags that will prevent slab merging
+ */
+#define SLUB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
+		SLAB_TRACE | SLAB_DESTROY_BY_RCU)
+
+#define SLUB_MERGE_SAME (SLAB_DEBUG_FREE | SLAB_RECLAIM_ACCOUNT | \
+		SLAB_CACHE_DMA)
+
+#ifndef ARCH_KMALLOC_MINALIGN
+#define ARCH_KMALLOC_MINALIGN sizeof(void *)
+#endif
+
+#ifndef ARCH_SLAB_MINALIGN
+#define ARCH_SLAB_MINALIGN sizeof(void *)
+#endif
+
+/* Internal SLUB flags */
+#define __OBJECT_POISON 0x80000000	/* Poison object */
+
+static int kmem_size = sizeof(struct kmem_cache);
+
+#ifdef CONFIG_SMP
+static struct notifier_block slab_notifier;
+#endif
+
+static enum {
+	DOWN,		/* No slab functionality available */
+	PARTIAL,	/* kmem_cache_open() works but kmalloc does not */
+	UP,		/* Everything works */
+	SYSFS		/* Sysfs up */
+} slab_state = DOWN;
+
+int slab_is_available(void) {
+	return slab_state >= UP;
+}
+
+/* A list of all slab caches on the system */
+static DECLARE_RWSEM(slub_lock);
+LIST_HEAD(slab_caches);
+
+#ifdef CONFIG_SYSFS
+static int sysfs_slab_add(struct kmem_cache *);
+static int sysfs_slab_alias(struct kmem_cache *, const char *);
+static void sysfs_slab_remove(struct kmem_cache *);
+#else
+static int sysfs_slab_add(struct kmem_cache *s) { return 0; }
+static int sysfs_slab_alias(struct kmem_cache *s, const char *p) { return 0; }
+static void sysfs_slab_remove(struct kmem_cache *s) {}
+#endif
+
+/********************************************************************
+ * 			Core slab cache functions
+ *******************************************************************/
+
+struct kmem_cache_node *get_node(struct kmem_cache *s, int node)
+{
+#ifdef CONFIG_NUMA
+	return s->node[node];
+#else
+	return &s->local_node;
+#endif
+}
+
+/*
+ * Object debugging
+ */
+static void print_section(char *text, u8 *addr, unsigned int length)
+{
+	int i, offset;
+	int newline = 1;
+	char ascii[17];
+
+	if (length > 128)
+		length = 128;
+	ascii[16] = 0;
+
+	for (i = 0; i < length; i++) {
+		if (newline) {
+			printk(KERN_ERR "%10s 0x%p: ", text, addr + i);
+			newline = 0;
+		}
+		printk(" %02x", addr[i]);
+		offset = i % 16;
+		ascii[offset] = isgraph(addr[i]) ? addr[i] : '.';
+		if (offset == 15) {
+			printk(" %s\n",ascii);
+			newline = 1;
+		}
+	}
+	if (!newline) {
+		i %= 16;
+		while (i < 16) {
+			printk("   ");
+			ascii[i] = ' ';
+			i++;
+		}
+		printk(" %s\n", ascii);
+	}
+}
+
+/*
+ * Slow version of get and set free pointer.
+ *
+ * This requires touching the cache lines of kmem_cache.
+ * The offset can also be obtained from the page. In that
+ * case it is in the cacheline that we already need to touch.
+ */
+static void *get_freepointer(struct kmem_cache *s, void *object)
+{
+	return *(void **)(object + s->offset);
+}
+
+static void set_freepointer(struct kmem_cache *s, void *object, void *fp)
+{
+	*(void **)(object + s->offset) = fp;
+}
+
+/*
+ * Tracking user of a slab.
+ */
+struct track {
+	void *addr;		/* Called from address */
+	int cpu;		/* Was running on cpu */
+	int pid;		/* Pid context */
+	unsigned long when;	/* When did the operation occur */
+};
+
+struct track *get_track(struct kmem_cache *s, void *object, int alloc)
+{
+	struct track *p;
+
+	if (s->offset)
+		p = object + s->offset + sizeof(void *);
+	else
+		p = object + s->inuse;
+
+	return p + alloc;
+}
+
+static void set_track(struct kmem_cache *s, void *object,
+				int alloc, void *addr)
+{
+	struct track *p;
+
+	if (s->offset)
+		p = object + s->offset + sizeof(void *);
+	else
+		p = object + s->inuse;
+
+	p += alloc;
+	if (addr) {
+		p->addr = addr;
+		p->cpu = smp_processor_id();
+		p->pid = current ? current->pid : -1;
+		p->when = jiffies;
+	} else
+		memset(p, 0, sizeof(struct track));
+}
+
+#define set_tracking(__s, __o, __a) set_track(__s, __o, __a, \
+			__builtin_return_address(0))
+
+static void init_tracking(struct kmem_cache *s, void *object)
+{
+	if (s->flags & SLAB_STORE_USER) {
+		set_track(s, object, 0, NULL);
+		set_track(s, object, 1, NULL);
+	}
+}
+
+static void print_track(const char *s, struct track *t)
+{
+#ifdef CONFIG_KALLSYMS
+	char *modname;
+	const char *name;
+	unsigned long offset, size;
+	char namebuf[KSYM_NAME_LEN + 1];
+#endif
+
+	if (!t->addr)
+		return;
+
+#ifdef CONFIG_KALLSYMS
+	name = kallsyms_lookup((unsigned long)t->addr, &size, &offset,
+		&modname, namebuf);
+
+	if (name) {
+		printk(KERN_ERR "%s: %s+%#lx/%#lx", s, name, offset, size);
+		if (modname)
+			printk(" [%s]", modname);
+	} else
+#endif
+		printk(KERN_ERR "%s: 0x%p", s, t->addr);
+	printk(" jiffies since=%lu cpu=%u pid=%d\n", jiffies - t->when, t->cpu, t->pid);
+}
+
+static void print_trailer(struct kmem_cache *s, u8 *p)
+{
+	unsigned int off;	/* Offset of last byte */
+
+	if (s->flags & SLAB_RED_ZONE)
+		print_section("Redzone", p + s->objsize,
+			s->inuse - s->objsize);
+
+	printk(KERN_ERR "FreePointer 0x%p -> 0x%p\n",
+			p + s->offset,
+			get_freepointer(s, p));
+
+	if (s->offset)
+		off = s->offset + sizeof(void *);
+	else
+		off = s->inuse;
+
+	if (s->flags & SLAB_STORE_USER) {
+		print_track("Last alloc", get_track(s, p, 0));
+		print_track("Last free ", get_track(s, p, 1));
+		off += 2 * sizeof(struct track);
+	}
+
+	if (off != s->size)
+		/* Beginning of the filler is the free pointer */
+		print_section("Filler", p + off, s->size - off);
+}
+
+static void object_err(struct kmem_cache *s, struct page *page,
+			u8 *object, char *reason)
+{
+	u8 *addr = page_address(page);
+
+	printk(KERN_ERR "*** SLUB: %s in %s@0x%p Slab 0x%p\n",
+			reason, s->name, object, page);
+	printk(KERN_ERR "    offset=%u flags=0x%04lx inuse=%u freelist=0x%p\n",
+		(int)(object - addr), page->flags, page->inuse,
+		page->freelist);
+	if (object > addr + 16)
+		print_section("Bytes b4", object - 16, 16);
+	print_section("Object", object, s->objsize);
+	print_trailer(s, object);
+	dump_stack();
+}
+
+static void init_object(struct kmem_cache *s, void *object, int active)
+{
+	u8 *p = object;
+
+	if (s->flags & __OBJECT_POISON) {
+		memset(p, POISON_FREE, s->objsize - 1);
+		p[s->objsize -1] = POISON_END;
+	}
+
+	if (s->flags & SLAB_RED_ZONE)
+		memset(p + s->objsize,
+			active ? SLUB_RED_ACTIVE : SLUB_RED_INACTIVE,
+			s->inuse - s->objsize);
+}
+
+static int check_bytes(u8 *start, unsigned int value, unsigned int bytes)
+{
+	while (bytes) {
+		if (*start != (u8)value)
+			return 0;
+		start++;
+		bytes--;
+	}
+	return 1;
+}
+
+
+static int check_valid_pointer(struct kmem_cache *s, struct page *page,
+					 void *object)
+{
+	void *base;
+
+	if (!object)
+		return 1;
+
+	base = page_address(page);
+	if (object < base || object >= base + s->objects * s->size ||
+		(object - base) % s->size) {
+		return 0;
+	}
+
+	return 1;
+}
+
+/*
+ * Object layout:
+ *
+ * object address
+ * 	Bytes of the object to be managed.
+ * 	If the freepointer may overlay the object then the free
+ * 	pointer is the first word of the object.
+ * 	Poisoning uses 0x6b (POISON_FREE) and the last byte is
+ * 	0xa5 (POISON_END)
+ *
+ * object + s->objsize
+ * 	Padding to reach word boundary. This is also used for Redzoning.
+ * 	Padding is extended to word size if Redzoning is enabled
+ * 	and objsize == inuse.
+ * 	We fill with 0xbb (RED_INACTIVE) for inactive objects and with
+ * 	0xcc (RED_ACTIVE) for objects in use.
+ *
+ * object + s->inuse
+ * 	A. Free pointer (if we cannot overwrite object on free)
+ * 	B. Tracking data for SLAB_STORE_USER
+ * 	C. Padding to reach required alignment boundary
+ * 		Padding is done using 0x5a (POISON_INUSE)
+ *
+ * object + s->size
+ *
+ * If slabcaches are merged then the objsize and inuse boundaries are to
+ * be ignored. And therefore no slab options that rely on these boundaries
+ * may be used with merged slabcaches.
+ */
+
+static int check_pad_bytes(struct kmem_cache *s, struct page *page, u8 *p)
+{
+	unsigned long off = s->inuse;	/* The end of info */
+
+	if (s->offset)
+		/* Freepointer is placed after the object. */
+		off += sizeof(void *);
+
+	if (s->flags & SLAB_STORE_USER)
+		/* We also have user information there */
+		off += 2 * sizeof(struct track);
+
+	if (s->size == off)
+		return 1;
+
+	if (check_bytes(p + off, POISON_INUSE, s->size - off))
+		return 1;
+
+	object_err(s, page, p, "Object padding check fails");
+	return 0;
+}
+
+static int slab_pad_check(struct kmem_cache *s, struct page *page)
+{
+	u8 *p;
+	int length, remainder;
+
+	if (!(s->flags & SLAB_POISON))
+		return 1;
+
+	p = page_address(page);
+	length = s->objects * s->size;
+	remainder = (PAGE_SIZE << s->order) - length;
+	if (!remainder)
+		return 1;
+
+	if (!check_bytes(p + length, POISON_INUSE, remainder)) {
+		printk(KERN_ERR "SLUB: %s slab 0x%p: Padding fails check\n",
+			s->name, p);
+		print_section("Slab Pad", p + length, remainder);
+		return 0;
+	}
+	return 1;
+}
+
+static int check_object(struct kmem_cache *s, struct page *page,
+					void *object, int active)
+{
+	u8 *p = object;
+	u8 *endobject = object + s->objsize;
+
+	if (s->flags & SLAB_RED_ZONE) {
+		if (!check_bytes(endobject,
+			active ? SLUB_RED_ACTIVE : SLUB_RED_INACTIVE,
+			s->inuse - s->objsize)) {
+				object_err(s, page, object,
+				active ? "Redzone Active check fails" :
+					"Redzone Inactive check fails");
+				return 0;
+		}
+	} else
+	if ((s->flags & SLAB_POISON) && s->objsize < s->inuse &&
+	    !check_bytes(endobject, POISON_INUSE, s->inuse - s->objsize))
+		object_err(s, page, p, "Alignment padding check fails");
+
+	if (s->flags & SLAB_POISON) {
+		if (!active && (s->flags & __OBJECT_POISON)
+			&& (!check_bytes(p, POISON_FREE, s->objsize - 1) ||
+				p[s->objsize -1] != POISON_END)) {
+			object_err(s, page, p, "Poison check failed");
+			return 0;
+		}
+		if (!check_pad_bytes(s, page, p))
+			return 0;
+	}
+
+	if (!s->offset && active)
+		/*
+		 * Object and freepointer overlap. Cannot check
+		 * freepointer while object is allocated.
+		 */
+		return 1;
+
+	/* Check free pointer validity */
+	if (!check_valid_pointer(s, page, get_freepointer(s, p))) {
+			object_err(s, page, p, "Freepointer corrupt");
+			/*
+			 * No choice but to zap it. This may cause
+			 * another error because the object count
+			 * is now wrong.
+			 */
+			set_freepointer(s, p, NULL);
+			return 0;
+	}
+	return 1;
+}
+
+static int check_slab(struct kmem_cache *s, struct page *page)
+{
+	if (!PageSlab(page)) {
+		printk(KERN_CRIT "SLUB: %s Not a valid slab page @0x%p "
+			"flags=%lx mapping=0x%p count=%d \n",
+			s->name, page, page->flags, page->mapping,
+			page_count(page));
+		return 0;
+	}
+	if (page->offset * sizeof(void *) != s->offset) {
+		printk(KERN_CRIT "SLUB: %s Corrupted offset %lu in slab @0x%p"
+			" flags=0x%lx mapping=0x%p count=%d\n",
+			s->name,
+			(unsigned long)(page->offset * sizeof(void *)),
+			page,
+			page->flags,
+			page->mapping,
+			page_count(page));
+		return 0;
+	}
+	if (page->inuse > s->objects) {
+		printk(KERN_CRIT "SLUB: %s Inuse %u > max %u in slab "
+			"page @0x%p flags=%lx mapping=0x%p count=%d\n",
+			s->name, page->inuse, s->objects, page, page->flags,
+			page->mapping, page_count(page));
+		return 0;
+	}
+	return slab_pad_check(s, page);
+}
+
+/*
+ * Determine if a certain object on a page is on the freelist and
+ * therefore free. Must hold the slab lock for cpu slabs to
+ * guarantee that the chains are consistent.
+ */
+static int on_freelist(struct kmem_cache *s, struct page *page, void *search)
+{
+	int nr = 0;
+	void *fp = page->freelist;
+	void *object = NULL;
+
+	while (fp && nr <= s->objects) {
+		if (fp == search)
+			return 1;
+		if (!check_valid_pointer(s, page, fp)) {
+			if (object) {
+				object_err(s, page, object,
+					"Freechain corrupt");
+				set_freepointer(s, object, NULL);
+				break;
+			} else {
+				printk(KERN_ERR "SLUB: %s slab 0x%p "
+					"freepointer 0x%p corrupted.\n",
+					s->name, page, fp);
+				dump_stack();
+				page->freelist = NULL;
+				page->inuse = s->objects;
+				return 0;
+			}
+			break;
+		}
+		object = fp;
+		fp = get_freepointer(s, object);
+		nr++;
+	}
+
+	if (page->inuse != s->objects - nr) {
+		printk(KERN_CRIT "slab %s: page 0x%p wrong object count."
+			" counter is %d but counted were %d\n",
+			s->name, page, page->inuse,
+			s->objects - nr);
+		page->inuse = s->objects - nr;
+	}
+	return 0;
+}
+
+static int alloc_object_checks(struct kmem_cache *s, struct page *page,
+							void *object)
+{
+	if (!check_slab(s, page))
+		goto bad;
+
+	if (object && !on_freelist(s, page, object)) {
+		printk(KERN_ERR "SLAB: %s Object 0x%p@0x%p "
+			"already allocated.\n",
+			s->name, object, page);
+		goto dump;
+	}
+
+	if (!check_valid_pointer(s, page, object)) {
+		object_err(s, page, object, "Freelist Pointer check fails");
+		goto dump;
+	}
+
+	if (!object)
+		return 1;
+
+	if (!check_object(s, page, object, 0))
+		goto bad;
+	init_object(s, object, 1);
+
+	if (s->flags & SLAB_TRACE) {
+		printk("TRACE %s alloc 0x%p inuse=%d fp=0x%p\n",
+			s->name, object, page->inuse,
+			page->freelist);
+		dump_stack();
+	}
+	return 1;
+dump:
+	dump_stack();
+bad:
+	/* Mark slab full */
+	page->inuse = s->objects;
+	page->freelist = NULL;
+	return 0;
+}
+
+static int free_object_checks(struct kmem_cache *s, struct page *page,
+							void *object)
+{
+	if (!check_slab(s, page)) {
+		goto fail;
+	}
+
+	if (!check_valid_pointer(s, page, object)) {
+		printk(KERN_ERR "SLUB: %s slab 0x%p invalid "
+			"object pointer 0x%p\n",
+			s->name, page, object);
+		goto fail;
+	}
+
+	if (on_freelist(s, page, object)) {
+		printk(KERN_CRIT "SLUB: %s slab 0x%p object "
+			"0x%p already free.\n", s->name, page, object);
+		goto fail;
+	}
+
+	if (!check_object(s, page, object, 1))
+		return 0;
+
+	if (unlikely(s != page->slab)) {
+		if (!PageSlab(page))
+			printk(KERN_CRIT "slab_free %s size %d: attempt to"
+				"free object(0x%p) outside of slab.\n",
+				s->name, s->size, object);
+		else
+		if (!page->slab)
+			printk(KERN_CRIT
+				"slab_free : no slab(NULL) for object 0x%p.\n",
+						object);
+		else
+		printk(KERN_CRIT "slab_free %s(%d): object at 0x%p"
+				" belongs to slab %s(%d)\n",
+				s->name, s->size, object,
+				page->slab->name, page->slab->size);
+		goto fail;
+	}
+	if (s->flags & SLAB_TRACE) {
+		printk("TRACE %s free 0x%p inuse=%d fp=0x%p\n",
+			s->name, object, page->inuse,
+			page->freelist);
+		print_section("Object", object, s->objsize);
+		dump_stack();
+	}
+	init_object(s, object, 0);
+	return 1;
+fail:
+	dump_stack();
+	return 0;
+}
+
+/*
+ * Slab allocation and freeing
+ */
+static struct page *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
+{
+	struct page * page;
+	int pages = 1 << s->order;
+
+	if (s->order)
+		flags |= __GFP_COMP;
+
+	if (s->flags & SLUB_DMA)
+		flags |= GFP_DMA;
+
+	if (node == -1)
+		page = alloc_pages(flags, s->order);
+	else
+		page = alloc_pages_node(node, flags, s->order);
+
+	if (!page)
+		return NULL;
+
+	mod_zone_page_state(page_zone(page),
+		(s->flags & SLAB_RECLAIM_ACCOUNT) ?
+		NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE,
+		pages);
+
+	return page;
+}
+
+static void setup_object(struct kmem_cache *s, struct page *page,
+				void *object)
+{
+	if (PageError(page)) {
+		init_object(s, object, 0);
+		init_tracking(s, object);
+	}
+
+	if (unlikely(s->ctor)) {
+		int mode = SLAB_CTOR_CONSTRUCTOR;
+
+		if (!(s->flags & __GFP_WAIT))
+			mode |= SLAB_CTOR_ATOMIC;
+
+		s->ctor(object, s, mode);
+	}
+}
+
+static struct page *new_slab(struct kmem_cache *s, gfp_t flags, int node)
+{
+	struct page *page;
+	struct kmem_cache_node *n;
+	void *start;
+	void *end;
+	void *last;
+	void *p;
+
+	if (flags & __GFP_NO_GROW)
+		return NULL;
+
+	BUG_ON(flags & ~(GFP_DMA | GFP_LEVEL_MASK));
+
+	if (flags & __GFP_WAIT)
+		local_irq_enable();
+
+	page = allocate_slab(s, flags & GFP_LEVEL_MASK, node);
+	if (!page)
+		goto out;
+
+	n = get_node(s, page_to_nid(page));
+	if (n)
+		atomic_long_inc(&n->nr_slabs);
+	page->offset = s->offset / sizeof(void *);
+	page->slab = s;
+	page->flags |= 1 << PG_slab;
+	if (s->flags & (SLAB_DEBUG_FREE | SLAB_RED_ZONE | SLAB_POISON |
+			SLAB_STORE_USER | SLAB_TRACE))
+		page->flags |= 1 << PG_error;
+
+	start = page_address(page);
+	end = start + s->objects * s->size;
+
+	if (unlikely(s->flags & SLAB_POISON))
+		memset(start, POISON_INUSE, PAGE_SIZE << s->order);
+
+	last = start;
+	for(p = start + s->size; p < end; p += s->size) {
+		setup_object(s, page, last);
+		set_freepointer(s, last, p);
+		last = p;
+	}
+	setup_object(s, page, last);
+	set_freepointer(s, last, NULL);
+
+	page->freelist = start;
+	page->inuse = 0;
+out:
+	if (flags & __GFP_WAIT)
+		local_irq_disable();
+	return page;
+}
+
+static void __free_slab(struct kmem_cache *s, struct page *page)
+{
+	int pages = 1 << s->order;
+
+	if (unlikely(PageError(page) || s->dtor)) {
+		void *start = page_address(page);
+		void *end = start + (pages << PAGE_SHIFT);
+		void *p;
+
+		slab_pad_check(s, page);
+		for (p = start; p <= end - s->size; p += s->size) {
+			if (s->dtor)
+				s->dtor(p, s, 0);
+			check_object(s, page, p, 0);
+		}
+	}
+
+	mod_zone_page_state(page_zone(page),
+		(s->flags & SLAB_RECLAIM_ACCOUNT) ?
+		NR_SLAB_RECLAIMABLE : NR_SLAB_UNRECLAIMABLE,
+		- pages);
+
+	page->mapping = NULL;
+	__free_pages(page, s->order);
+}
+
+static void rcu_free_slab(struct rcu_head *h)
+{
+	struct page *page;
+
+	page = container_of((struct list_head *)h, struct page, lru);
+	__free_slab(page->slab, page);
+}
+
+static void free_slab(struct kmem_cache *s, struct page *page)
+{
+	if (unlikely(s->flags & SLAB_DESTROY_BY_RCU)) {
+		/*
+		 * RCU free overloads the RCU head over the LRU
+		 */
+		struct rcu_head *head = (void *)&page->lru;
+
+		call_rcu(head, rcu_free_slab);
+	} else
+		__free_slab(s, page);
+}
+
+static void discard_slab(struct kmem_cache *s, struct page *page)
+{
+	struct kmem_cache_node *n = get_node(s, page_to_nid(page));
+
+	atomic_long_dec(&n->nr_slabs);
+	reset_page_mapcount(page);
+	page->flags &= ~(1 << PG_slab | 1 << PG_error);
+	free_slab(s, page);
+}
+
+/*
+ * Per slab locking using the pagelock
+ */
+static __always_inline void slab_lock(struct page *page)
+{
+#ifdef CONFIG_SMP
+	bit_spin_lock(PG_locked, &page->flags);
+#endif
+}
+
+static __always_inline void slab_unlock(struct page *page)
+{
+#ifdef CONFIG_SMP
+	bit_spin_unlock(PG_locked, &page->flags);
+#endif
+}
+
+static __always_inline int slab_trylock(struct page *page)
+{
+	int rc = 1;
+
+#ifdef CONFIG_SMP
+	rc = bit_spin_trylock(PG_locked, &page->flags);
+#endif
+	return rc;
+}
+
+/*
+ * Management of partially allocated slabs
+ */
+static void add_partial(struct kmem_cache *s, struct page *page)
+{
+	struct kmem_cache_node *n = get_node(s, page_to_nid(page));
+
+	spin_lock(&n->list_lock);
+	n->nr_partial++;
+	list_add_tail(&page->lru, &n->partial);
+	spin_unlock(&n->list_lock);
+}
+
+static void remove_partial(struct kmem_cache *s,
+						struct page *page)
+{
+	struct kmem_cache_node *n = get_node(s, page_to_nid(page));
+
+	spin_lock(&n->list_lock);
+	list_del(&page->lru);
+	n->nr_partial--;
+	spin_unlock(&n->list_lock);
+}
+
+/*
+ * Lock page and remove it from the partial list
+ *
+ * Must hold list_lock
+ */
+static int lock_and_del_slab(struct kmem_cache_node *n, struct page *page)
+{
+	if (slab_trylock(page)) {
+		list_del(&page->lru);
+		n->nr_partial--;
+		return 1;
+	}
+	return 0;
+}
+
+/*
+ * Try to get a partial slab from a specific node
+ */
+static struct page *get_partial_node(struct kmem_cache_node *n)
+{
+	struct page *page;
+
+	/*
+	 * Racy check. If we mistakenly see no partial slabs then we
+	 * just allocate an empty slab. If we mistakenly try to get a
+	 * partial slab then get_partials() will return NULL.
+	 */
+	if (!n || !n->nr_partial)
+		return NULL;
+
+	spin_lock(&n->list_lock);
+	list_for_each_entry(page, &n->partial, lru)
+		if (lock_and_del_slab(n, page))
+			goto out;
+	page = NULL;
+out:
+	spin_unlock(&n->list_lock);
+	return page;
+}
+
+/*
+ * Get a page from somewhere. Search in increasing NUMA
+ * distances.
+ */
+static struct page *get_any_partial(struct kmem_cache *s, gfp_t flags)
+{
+#ifdef CONFIG_NUMA
+	struct zonelist *zonelist;
+	struct zone **z;
+	struct page *page;
+
+	/*
+	 * The defrag ratio allows to configure the tradeoffs between
+	 * inter node defragmentation and node local allocations.
+	 * A lower defrag_ratio increases the tendency to do local
+	 * allocations instead of scanning throught the partial
+	 * lists on other nodes.
+	 *
+	 * If defrag_ratio is set to 0 then kmalloc() always
+	 * returns node local objects. If its higher then kmalloc()
+	 * may return off node objects in order to avoid fragmentation.
+	 *
+	 * A higher ratio means slabs may be taken from other nodes
+	 * thus reducing the number of partial slabs on those nodes.
+	 */
+	if (!s->defrag_ratio || get_cycles() % 1024 > s->defrag_ratio)
+		return NULL;
+
+	zonelist = &NODE_DATA(slab_node(current->mempolicy))
+					->node_zonelists[gfp_zone(flags)];
+	for (z = zonelist->zones; *z; z++) {
+		struct kmem_cache_node *n;
+
+		n = get_node(s, zone_to_nid(*z));
+
+		if (n && cpuset_zone_allowed_hardwall(*z, flags) &&
+				n->nr_partial > 2) {
+			page = get_partial_node(n);
+			if (page)
+				return page;
+		}
+	}
+#endif
+	return NULL;
+}
+
+/*
+ * Get a partial page, lock it and return it.
+ */
+static struct page *get_partial(struct kmem_cache *s, gfp_t flags, int node)
+{
+	struct page *page;
+	int searchnode = (node == -1) ? numa_node_id() : node;
+
+	page = get_partial_node(get_node(s, searchnode));
+	if (page || (flags & __GFP_THISNODE))
+		return page;
+
+	return get_any_partial(s, flags);
+}
+
+/*
+ * Move a page back to the lists.
+ *
+ * Must be called with the slab lock held.
+ *
+ * On exit the slab lock will have been dropped.
+ */
+static void putback_slab(struct kmem_cache *s, struct page *page)
+{
+	if (page->inuse) {
+		if (page->freelist)
+			add_partial(s, page);
+		slab_unlock(page);
+	} else {
+		slab_unlock(page);
+		discard_slab(s, page);
+	}
+}
+
+/*
+ * Remove the cpu slab
+ */
+static void deactivate_slab(struct kmem_cache *s, struct page *page, int cpu)
+{
+	s->cpu_slab[cpu] = NULL;
+	ClearPageActive(page);
+	ClearPageReferenced(page);
+
+	putback_slab(s, page);
+}
+
+static void flush_slab(struct kmem_cache *s, struct page *page, int cpu)
+{
+	slab_lock(page);
+	deactivate_slab(s, page, cpu);
+}
+
+/*
+ * Flush cpu slab.
+ * Called from IPI handler with interrupts disabled.
+ */
+static void __flush_cpu_slab(struct kmem_cache *s, int cpu)
+{
+	struct page *page = s->cpu_slab[cpu];
+
+	if (likely(page))
+		flush_slab(s, page, cpu);
+}
+
+static void flush_cpu_slab(void *d)
+{
+	struct kmem_cache *s = d;
+	int cpu = smp_processor_id();
+
+	__flush_cpu_slab(s, cpu);
+}
+
+#ifdef CONFIG_SMP
+/*
+ * Called from IPI to check and flush cpu slabs.
+ */
+static void check_flush_cpu_slab(void *private)
+{
+	struct kmem_cache *s = private;
+	int cpu = smp_processor_id();
+	struct page *page = s->cpu_slab[cpu];
+
+	if (page) {
+		if (!TestClearPageReferenced(page))
+			return;
+		flush_slab(s, page, cpu);
+	}
+	atomic_dec(&s->cpu_slabs);
+}
+
+/*
+ * Called from eventd
+ */
+static void flusher(struct work_struct *w)
+{
+	struct kmem_cache *s = container_of(w, struct kmem_cache, flush.work);
+
+	if (!mutex_trylock(&s->flushing))
+		return;
+
+	atomic_set(&s->cpu_slabs, num_online_cpus());
+	on_each_cpu(check_flush_cpu_slab, s, 1, 1);
+	if (atomic_read(&s->cpu_slabs))
+		schedule_delayed_work(&s->flush, 30 * HZ);
+	mutex_unlock(&s->flushing);
+}
+
+static void flush_all(struct kmem_cache *s)
+{
+	if (atomic_read(&s->cpu_slabs)) {
+		mutex_lock(&s->flushing);
+		cancel_delayed_work(&s->flush);
+		atomic_set(&s->cpu_slabs, 0);
+		on_each_cpu(flush_cpu_slab, s, 1, 1);
+		mutex_unlock(&s->flushing);
+	}
+}
+#else
+static void flush_all(struct kmem_cache *s)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	flush_cpu_slab(s);
+	local_irq_restore(flags);
+}
+#endif
+
+/*
+ * slab_alloc is optimized to only modify two cachelines on the fast path
+ * (aside from the stack):
+ *
+ * 1. The page struct
+ * 2. The first cacheline of the object to be allocated.
+ *
+ * The only cache lines that are read (apart from code) is the
+ * per cpu array in the kmem_cache struct.
+ *
+ * Fastpath is not possible if we need to get a new slab or have
+ * debugging enabled (which means all slabs are marked with PageError)
+ */
+static __always_inline void *slab_alloc(struct kmem_cache *s,
+					gfp_t gfpflags, int node)
+{
+	struct page *page;
+	void **object;
+	unsigned long flags;
+	int cpu;
+
+	local_irq_save(flags);
+	cpu = smp_processor_id();
+	page = s->cpu_slab[cpu];
+	if (!page)
+		goto new_slab;
+
+	slab_lock(page);
+	if (unlikely(node != -1 && page_to_nid(page) != node))
+		goto another_slab;
+redo:
+	object = page->freelist;
+	if (unlikely(!object))
+		goto another_slab;
+	if (unlikely(PageError(page)))
+		goto debug;
+
+have_object:
+	page->inuse++;
+	page->freelist = object[page->offset];
+	SetPageReferenced(page);
+	slab_unlock(page);
+	local_irq_restore(flags);
+	return object;
+
+another_slab:
+	deactivate_slab(s, page, cpu);
+
+new_slab:
+	page = get_partial(s, gfpflags, node);
+	if (likely(page)) {
+have_slab:
+		s->cpu_slab[cpu] = page;
+		SetPageActive(page);
+
+#ifdef CONFIG_SMP
+		if (!atomic_read(&s->cpu_slabs) && keventd_up()) {
+			atomic_inc(&s->cpu_slabs);
+			schedule_delayed_work(&s->flush, 30 * HZ);
+		}
+#endif
+		goto redo;
+	}
+
+	page = new_slab(s, gfpflags, node);
+	if (page) {
+		if (s->cpu_slab[cpu]) {
+			/*
+			 * Someone else populated the cpu_slab while
+			 * we enabled interrupts. The page may not
+			 * be on the requested node.
+			 */
+			if (node == -1 ||
+				page_to_nid(s->cpu_slab[cpu]) == node) {
+				/*
+				 * Current cpuslab is acceptable and we
+				 * want the current one since its cache hot
+				 */
+				discard_slab(s, page);
+				page = s->cpu_slab[cpu];
+				slab_lock(page);
+				goto redo;
+			}
+			/* Dump the current slab */
+			flush_slab(s, s->cpu_slab[cpu], cpu);
+		}
+		slab_lock(page);
+		goto have_slab;
+	}
+	local_irq_restore(flags);
+	return NULL;
+debug:
+	if (!alloc_object_checks(s, page, object))
+		goto another_slab;
+	if (s->flags & SLAB_STORE_USER)
+		set_tracking(s, object, 0);
+	goto have_object;
+}
+
+void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
+{
+	return slab_alloc(s, gfpflags, -1);
+}
+EXPORT_SYMBOL(kmem_cache_alloc);
+
+#ifdef CONFIG_NUMA
+void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node)
+{
+	return slab_alloc(s, gfpflags, node);
+}
+EXPORT_SYMBOL(kmem_cache_alloc_node);
+#endif
+
+/*
+ * The fastpath only writes the cacheline of the page struct and the first
+ * cacheline of the object.
+ *
+ * No special cachelines need to be read
+ */
+static void slab_free(struct kmem_cache *s, struct page *page, void *x)
+{
+	void *prior;
+	void **object = (void *)x;
+	unsigned long flags;
+
+	local_irq_save(flags);
+	slab_lock(page);
+
+	if (unlikely(PageError(page)))
+		goto debug;
+checks_ok:
+	prior = object[page->offset] = page->freelist;
+	page->freelist = object;
+	page->inuse--;
+
+	if (unlikely(PageActive(page)))
+		/*
+		 * Cpu slabs are never on partial lists and are
+		 * never freed.
+		 */
+		goto out_unlock;
+
+	if (unlikely(!page->inuse))
+		goto slab_empty;
+
+	/*
+	 * Objects left in the slab. If it
+	 * was not on the partial list before
+	 * then add it.
+	 */
+	if (unlikely(!prior))
+		add_partial(s, page);
+
+out_unlock:
+	slab_unlock(page);
+	local_irq_restore(flags);
+	return;
+
+slab_empty:
+	if (prior)
+		/*
+		 * Partially used slab that is on the partial list.
+		 */
+		remove_partial(s, page);
+
+	slab_unlock(page);
+	discard_slab(s, page);
+	local_irq_restore(flags);
+	return;
+
+debug:
+	if (free_object_checks(s, page, x))
+		goto checks_ok;
+	goto out_unlock;
+}
+
+void kmem_cache_free(struct kmem_cache *s, void *x)
+{
+	struct page * page;
+
+	page = virt_to_page(x);
+
+	if (unlikely(PageCompound(page)))
+		page = page->first_page;
+
+
+	if (unlikely(PageError(page) && (s->flags & SLAB_STORE_USER)))
+		set_tracking(s, x, 1);
+	slab_free(s, page, x);
+}
+EXPORT_SYMBOL(kmem_cache_free);
+
+/* Figure out on which slab object the object resides */
+static struct page *get_object_page(const void *x)
+{
+	struct page *page = virt_to_page(x);
+
+	if (unlikely(PageCompound(page)))
+		page = page->first_page;
+
+	if (!PageSlab(page))
+		return NULL;
+
+	return page;
+}
+
+/*
+ * kmem_cache_open produces objects aligned at "size" and the first object
+ * is placed at offset 0 in the slab (We have no metainformation on the
+ * slab, all slabs are in essence "off slab").
+ *
+ * In order to get the desired alignment one just needs to align the
+ * size.
+ *
+ * Notice that the allocation order determines the sizes of the per cpu
+ * caches. Each processor has always one slab available for allocations.
+ * Increasing the allocation order reduces the number of times that slabs
+ * must be moved on and off the partial lists and therefore may influence
+ * locking overhead.
+ *
+ * The offset is used to relocate the free list link in each object. It is
+ * therefore possible to move the free list link behind the object. This
+ * is necessary for RCU to work properly and also useful for debugging.
+ */
+
+/*
+ * Mininum / Maximum order of slab pages. This influences locking overhead
+ * and slab fragmentation. A higher order reduces the number of partial slabs
+ * and increases the number of allocations possible without having to
+ * take the list_lock.
+ */
+static int slub_min_order = 0;
+static int slub_max_order = 4;
+
+/*
+ * Minimum number of objects per slab. This is necessary in order to
+ * reduce locking overhead. Similar to the queue size in SLAB.
+ */
+static int slub_min_objects = 8;
+
+/*
+ * Merge control. If this is set then no merging of slab caches will occur.
+ */
+static int slub_nomerge = 0;
+
+/*
+ * Debug settings:
+ */
+static int slub_debug = 0;
+
+static char *slub_debug_slabs = NULL;
+
+static int calculate_order(int size)
+{
+	int order;
+	int rem;
+
+	for (order = max(slub_min_order, fls(size - 1) - PAGE_SHIFT);
+			order < MAX_ORDER; order++) {
+		unsigned long slab_size = PAGE_SIZE << order;
+
+		if (slub_max_order > order &&
+				slab_size < slub_min_objects * size)
+			continue;
+
+		if (slab_size < size)
+			continue;
+
+		rem = slab_size % size;
+
+		if (rem <= (PAGE_SIZE << order) / 8)
+			break;
+
+	}
+	if (order >= MAX_ORDER)
+		return -E2BIG;
+	return order;
+}
+
+static unsigned long calculate_alignment(unsigned long flags,
+		unsigned long align)
+{
+	if (flags & SLAB_HWCACHE_ALIGN)
+		return L1_CACHE_BYTES;
+
+	if (flags & SLAB_MUST_HWCACHE_ALIGN)
+		return max_t(unsigned long, align, L1_CACHE_BYTES);
+
+	if (align < ARCH_SLAB_MINALIGN)
+		return ARCH_SLAB_MINALIGN;
+
+	return ALIGN(align, sizeof(void *));
+}
+
+
+static void init_kmem_cache_node(struct kmem_cache_node *n)
+{
+	memset(n, 0, sizeof(struct kmem_cache_node));
+	atomic_long_set(&n->nr_slabs, 0);
+	spin_lock_init(&n->list_lock);
+	INIT_LIST_HEAD(&n->partial);
+}
+
+static void free_kmem_cache_nodes(struct kmem_cache *s)
+{
+#ifdef CONFIG_NUMA
+	int node;
+
+	for_each_online_node(node) {
+		struct kmem_cache_node *n = s->node[node];
+		if (n && n != &s->local_node)
+			kmem_cache_free(kmalloc_caches, n);
+		s->node[node] = NULL;
+	}
+#endif
+}
+
+static int init_kmem_cache_nodes(struct kmem_cache *s, gfp_t gfpflags)
+{
+#ifdef CONFIG_NUMA
+	int node;
+	int local_node;
+
+	if (slab_state >= UP)
+		local_node = page_to_nid(virt_to_page(s));
+	else
+		local_node = 0;
+
+	for_each_online_node(node) {
+		struct kmem_cache_node *n;
+
+		if (local_node == node)
+			n = &s->local_node;
+		else
+		if (slab_state == DOWN) {
+			/*
+			 * No kmalloc_node yet so do it by hand.
+			 * We know that this is the first slab on the
+			 * node for this slabcache. There are no concurrent
+			 * accesses possible.
+			 */
+			struct page *page;
+
+			BUG_ON(s->size < sizeof(struct kmem_cache_node));
+			page = new_slab(kmalloc_caches, gfpflags, node);
+
+			BUG_ON(!page);
+			n = page->freelist;
+			page->freelist = get_freepointer(kmalloc_caches, n);
+			page->inuse++;
+		} else
+			n = kmem_cache_alloc_node(kmalloc_caches,
+							gfpflags, node);
+
+		if (!n) {
+			free_kmem_cache_nodes(s);
+			return 0;
+		}
+
+		s->node[node] = n;
+		init_kmem_cache_node(n);
+
+		if (slab_state == DOWN)
+			atomic_long_inc(&n->nr_slabs);
+	}
+#else
+	init_kmem_cache_node(&s->local_node);
+#endif
+	return 1;
+}
+
+int calculate_sizes(struct kmem_cache *s)
+{
+	unsigned long flags = s->flags;
+	unsigned long size = s->objsize;
+	unsigned long align = s->align;
+
+	if ((flags & SLAB_POISON) && !(flags & SLAB_DESTROY_BY_RCU) &&
+			!s->ctor && !s->dtor)
+		flags |= __OBJECT_POISON;
+	else
+		flags &= ~__OBJECT_POISON;
+
+	size = ALIGN(size, sizeof(void *));
+
+	/*
+	 * If we redzone then check if we have space through above
+	 * alignment. If not then add an additional word, so
+	 * that we have a guard value to check for overwrites.
+	 */
+	if ((flags & SLAB_RED_ZONE) && size == s->objsize)
+		size += sizeof(void *);
+
+	s->inuse = size;
+
+	if (((flags & (SLAB_DESTROY_BY_RCU | SLAB_POISON)) ||
+		s->ctor || s->dtor)) {
+		/*
+		 * Relocate free pointer after the object if it is not
+		 * permitted to overwrite the first word of the object on
+		 * kmem_cache_free.
+		 *
+		 * This is the case if we do RCU, have a constructor or
+		 * destructor or are poisoning the objects.
+		 */
+		s->offset = size;
+		size += sizeof(void *);
+	}
+
+	if (flags & SLAB_STORE_USER)
+		size += 2 * sizeof(struct track);
+
+	align = calculate_alignment(flags, align);
+
+	size = ALIGN(size, align);
+	s->size = size;
+
+	s->order = calculate_order(size);
+	if (s->order < 0)
+		return 0;
+
+	s->objects = (PAGE_SIZE << s->order) / size;
+	if (!s->objects || s->objects > 65535)
+		return 0;
+	return 1;
+
+}
+
+static int kmem_cache_open(struct kmem_cache *s, gfp_t gfpflags,
+		const char *name, size_t size,
+		size_t align, unsigned long flags,
+		void (*ctor)(void *, struct kmem_cache *, unsigned long),
+		void (*dtor)(void *, struct kmem_cache *, unsigned long))
+{
+	memset(s, 0, kmem_size);
+	s->name = name;
+	s->ctor = ctor;
+	s->dtor = dtor;
+	s->objsize = size;
+	s->flags = flags;
+	s->align = align;
+
+	BUG_ON(flags & SLUB_UNIMPLEMENTED);
+
+	if (s->size >= 65535 * sizeof(void *))
+		BUG_ON(flags & (SLAB_RED_ZONE | SLAB_POISON |
+				SLAB_STORE_USER | SLAB_DESTROY_BY_RCU));
+	else
+		/*
+		 * Enable debugging if selected on the kernel commandline.
+		 */
+		if (slub_debug && (!slub_debug_slabs ||
+		    strncmp(slub_debug_slabs, name,
+		    	strlen(slub_debug_slabs)) == 0))
+				s->flags |= slub_debug;
+
+	if (!calculate_sizes(s))
+		goto error;
+
+	s->refcount = 1;
+#ifdef CONFIG_NUMA
+	s->defrag_ratio = 100;
+#endif
+
+#ifdef CONFIG_SMP
+	mutex_init(&s->flushing);
+	atomic_set(&s->cpu_slabs, 0);
+	INIT_DELAYED_WORK(&s->flush, flusher);
+#endif
+	if (init_kmem_cache_nodes(s, gfpflags & ~SLUB_DMA))
+		return 1;
+error:
+	if (flags & SLAB_PANIC)
+		panic("Cannot create slab %s size=%lu realsize=%u "
+			"order=%u offset=%u flags=%lx\n",
+			s->name, (unsigned long)size, s->size, s->order,
+			s->offset, flags);
+	return 0;
+}
+EXPORT_SYMBOL(kmem_cache_open);
+
+/*
+ * Check if a given pointer is valid
+ */
+int kmem_ptr_validate(struct kmem_cache *s, const void *object)
+{
+	struct page * page;
+	void *addr;
+
+	page = get_object_page(object);
+
+	if (!page || s != page->slab)
+		/* No slab or wrong slab */
+		return 0;
+
+	addr = page_address(page);
+	if (object < addr || object >= addr + s->objects * s->size)
+		/* Out of bounds */
+		return 0;
+
+	if ((object - addr) % s->size)
+		/* Improperly aligned */
+		return 0;
+
+	/*
+	 * We could also check if the object is on the slabs freelist.
+	 * But this would be too expensive and it seems that the main
+	 * purpose of kmem_ptr_valid is to check if the object belongs
+	 * to a certain slab.
+	 */
+	return 1;
+}
+EXPORT_SYMBOL(kmem_ptr_validate);
+
+/*
+ * Determine the size of a slab object
+ */
+unsigned int kmem_cache_size(struct kmem_cache *s)
+{
+	return s->objsize;
+}
+EXPORT_SYMBOL(kmem_cache_size);
+
+const char *kmem_cache_name(struct kmem_cache *s)
+{
+	return s->name;
+}
+EXPORT_SYMBOL(kmem_cache_name);
+
+/*
+ * Attempt to free all slabs on a node
+ */
+static int free_list(struct kmem_cache *s, struct kmem_cache_node *n,
+			struct list_head *list)
+{
+	int slabs_inuse = 0;
+	unsigned long flags;
+	struct page *page, *h;
+
+	spin_lock_irqsave(&n->list_lock, flags);
+	list_for_each_entry_safe(page, h, list, lru)
+		if (!page->inuse) {
+			list_del(&page->lru);
+			discard_slab(s, page);
+		} else
+			slabs_inuse++;
+	spin_unlock_irqrestore(&n->list_lock, flags);
+	return slabs_inuse;
+}
+
+/*
+ * Release all resources used by slab cache
+ */
+static int kmem_cache_close(struct kmem_cache *s)
+{
+	int node;
+
+	flush_all(s);
+
+	/* Attempt to free all objects */
+	for_each_online_node(node) {
+		struct kmem_cache_node *n = get_node(s, node);
+
+		free_list(s, n, &n->partial);
+		if (atomic_long_read(&n->nr_slabs))
+			return 1;
+	}
+	free_kmem_cache_nodes(s);
+	return 0;
+}
+EXPORT_SYMBOL(kmem_cache_close);
+
+/*
+ * Close a cache and release the kmem_cache structure
+ * (must be used for caches created using kmem_cache_create)
+ */
+void kmem_cache_destroy(struct kmem_cache *s)
+{
+	down_write(&slub_lock);
+	if (s->refcount)
+		s->refcount--;
+	else {
+		list_del(&s->list);
+		WARN_ON(kmem_cache_close(s));
+		sysfs_slab_remove(s);
+		kfree(s);
+	}
+	up_write(&slub_lock);
+}
+EXPORT_SYMBOL(kmem_cache_destroy);
+
+/********************************************************************
+ *		Kmalloc subsystem
+ *******************************************************************/
+
+struct kmem_cache kmalloc_caches[KMALLOC_SHIFT_HIGH + 1] __cacheline_aligned;
+EXPORT_SYMBOL(kmalloc_caches);
+
+#ifdef CONFIG_ZONE_DMA
+static struct kmem_cache *kmalloc_caches_dma[KMALLOC_SHIFT_HIGH + 1];
+#endif
+
+static int __init setup_slub_min_order(char *str)
+{
+	get_option (&str, &slub_min_order);
+
+	return 1;
+}
+
+__setup("slub_min_order=", setup_slub_min_order);
+
+static int __init setup_slub_max_order(char *str)
+{
+	get_option (&str, &slub_max_order);
+
+	return 1;
+}
+
+__setup("slub_max_order=", setup_slub_max_order);
+
+static int __init setup_slub_min_objects(char *str)
+{
+	get_option (&str, &slub_min_objects);
+
+	return 1;
+}
+
+__setup("slub_min_objects=", setup_slub_min_objects);
+
+static int __init setup_slub_nomerge(char *str)
+{
+	slub_nomerge = 1;
+	return 1;
+}
+
+__setup("slub_nomerge", setup_slub_nomerge);
+
+static int __init setup_slub_debug(char *str)
+{
+	if (!str || *str != '=')
+		slub_debug = DEBUG_DEFAULT_FLAGS;
+	else {
+		str++;
+		if (*str == 0 || *str == ',')
+			slub_debug = DEBUG_DEFAULT_FLAGS;
+		else
+		for( ;*str && *str != ','; str++)
+			switch (*str) {
+			case 'f' : case 'F' :
+				slub_debug |= SLAB_DEBUG_FREE;
+				break;
+			case 'z' : case 'Z' :
+				slub_debug |= SLAB_RED_ZONE;
+				break;
+			case 'p' : case 'P' :
+				slub_debug |= SLAB_POISON;
+				break;
+			case 'u' : case 'U' :
+				slub_debug |= SLAB_STORE_USER;
+				break;
+			case 't' : case 'T' :
+				slub_debug |= SLAB_TRACE;
+				break;
+			default:
+				printk(KERN_CRIT "slub_debug option '%c' "
+					"unknown. skipped\n",*str);
+			}
+	}
+
+	if (*str == ',')
+		slub_debug_slabs = str + 1;
+	return 1;
+}
+
+__setup("slub_debug", setup_slub_debug);
+
+static struct kmem_cache *create_kmalloc_cache(struct kmem_cache *s,
+		const char *name, int size, gfp_t gfp_flags)
+{
+	unsigned int flags = 0;
+
+	if (gfp_flags & SLUB_DMA)
+		flags = SLAB_CACHE_DMA;
+
+	down_write(&slub_lock);
+	if (!kmem_cache_open(s, gfp_flags, name, size, ARCH_KMALLOC_MINALIGN,
+			flags, NULL, NULL))
+		goto panic;
+
+	list_add(&s->list, &slab_caches);
+	up_write(&slub_lock);
+	if (sysfs_slab_add(s))
+		goto panic;
+	return s;
+
+panic:
+	panic("Creation of kmalloc slab %s size=%d failed.\n",
+			name, size);
+}
+
+static struct kmem_cache *get_slab(size_t size, gfp_t flags)
+{
+	int index = kmalloc_index(size);
+
+	if (!size)
+		return NULL;
+
+	/* Allocation too large? */
+	BUG_ON(index < 0);
+
+#ifdef CONFIG_ZONE_DMA
+	if ((flags & SLUB_DMA)) {
+		struct kmem_cache *s;
+		struct kmem_cache *x;
+		char *text;
+		size_t realsize;
+
+		s = kmalloc_caches_dma[index];
+		if (s)
+			return s;
+
+		/* Dynamically create dma cache */
+		x = kmalloc(kmem_size, flags & ~SLUB_DMA);
+		if (!x)
+			panic("Unable to allocate memory for dma cache\n");
+
+		if (index <= KMALLOC_SHIFT_HIGH)
+			realsize = 1 << index;
+		else {
+			if (index == 1)
+				realsize = 96;
+			else
+				realsize = 192;
+		}
+
+		text = kasprintf(flags & ~SLUB_DMA, "kmalloc_dma-%d",
+				(unsigned int)realsize);
+		s = create_kmalloc_cache(x, text, realsize, flags);
+		kmalloc_caches_dma[index] = s;
+		return s;
+	}
+#endif
+	return &kmalloc_caches[index];
+}
+
+void *__kmalloc(size_t size, gfp_t flags)
+{
+	struct kmem_cache *s = get_slab(size, flags);
+
+	if (s)
+		return kmem_cache_alloc(s, flags);
+	return NULL;
+}
+EXPORT_SYMBOL(__kmalloc);
+
+#ifdef CONFIG_NUMA
+void *__kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	struct kmem_cache *s = get_slab(size, flags);
+
+	if (s)
+		return kmem_cache_alloc_node(s, flags, node);
+	return NULL;
+}
+EXPORT_SYMBOL(__kmalloc_node);
+#endif
+
+size_t ksize(const void *object)
+{
+	struct page *page = get_object_page(object);
+	struct kmem_cache *s;
+
+	BUG_ON(!page);
+	s = page->slab;
+	BUG_ON(!s);
+
+	/*
+	 * Debugging requires use of the padding between object
+	 * and whatever may come after it.
+	 */
+	if (s->flags & (SLAB_RED_ZONE | SLAB_POISON))
+		return s->objsize;
+
+	/*
+	 * If we have the need to store the freelist pointer
+	 * back there or track user information then we can
+	 * only use the space before that information.
+	 */
+	if (s->flags & (SLAB_DESTROY_BY_RCU | SLAB_STORE_USER))
+		return s->inuse;
+
+	/*
+	 * Else we can use all the padding etc for the allocation
+	 */
+	return s->size;
+}
+EXPORT_SYMBOL(ksize);
+
+void kfree(const void *x)
+{
+	struct kmem_cache *s;
+	struct page * page;
+
+	if (!x)
+		return;
+
+	page = virt_to_page(x);
+
+	if (unlikely(PageCompound(page)))
+		page = page->first_page;
+
+	s = page->slab;
+
+	if (unlikely(PageError(page) && (s->flags & SLAB_STORE_USER)))
+		set_tracking(s, (void *)x, 1);
+	slab_free(s, page, (void *)x);
+}
+EXPORT_SYMBOL(kfree);
+
+/**
+ * krealloc - reallocate memory. The contents will remain unchanged.
+ *
+ * @p: object to reallocate memory for.
+ * @new_size: how many bytes of memory are required.
+ * @flags: the type of memory to allocate.
+ *
+ * The contents of the object pointed to are preserved up to the
+ * lesser of the new and old sizes.  If @p is %NULL, krealloc()
+ * behaves exactly like kmalloc().  If @size is 0 and @p is not a
+ * %NULL pointer, the object pointed to is freed.
+ */
+void *krealloc(const void *p, size_t new_size, gfp_t flags)
+{
+	struct kmem_cache *new_cache;
+	void *ret;
+	struct page *page;
+
+	if (unlikely(!p))
+		return kmalloc(new_size, flags);
+
+	if (unlikely(!new_size)) {
+		kfree(p);
+		return NULL;
+	}
+
+	page = virt_to_page(p);
+
+	if (unlikely(PageCompound(page)))
+		page = page->first_page;
+
+	new_cache = get_slab(new_size, flags);
+
+	/*
+ 	 * If new size fits in the current cache, bail out.
+ 	 */
+	if (likely(page->slab == new_cache))
+		return (void *)p;
+
+	ret = kmalloc(new_size, flags);
+	if (ret) {
+		memcpy(ret, p, min(new_size, ksize(p)));
+		kfree(p);
+	}
+	return ret;
+}
+EXPORT_SYMBOL(krealloc);
+
+/********************************************************************
+ *			Basic setup of slabs
+ *******************************************************************/
+
+void __init kmem_cache_init(void)
+{
+	int i;
+
+#ifdef CONFIG_NUMA
+	/*
+	 * Must first have the slab cache available for the allocations of the
+	 * struct kmalloc_cache_node's. There is special bootstrap code in
+	 * kmem_cache_open for slab_state == DOWN.
+	 */
+	create_kmalloc_cache(&kmalloc_caches[0], "kmem_cache_node",
+		sizeof(struct kmem_cache_node), GFP_KERNEL);
+#endif
+
+	/* Able to allocate the per node structures */
+	slab_state = PARTIAL;
+
+	/* Caches that are not of the two-to-the-power-of size */
+	create_kmalloc_cache(&kmalloc_caches[1],
+				"kmalloc-96", 96, GFP_KERNEL);
+	create_kmalloc_cache(&kmalloc_caches[2],
+				"kmalloc-192", 192, GFP_KERNEL);
+
+	for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
+		create_kmalloc_cache(&kmalloc_caches[i],
+			"kmalloc", 1 << i, GFP_KERNEL);
+
+	slab_state = UP;
+
+	/* Provide the correct kmalloc names now that the caches are up */
+	for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
+		kmalloc_caches[i]. name =
+			kasprintf(GFP_KERNEL, "kmalloc-%d", 1 << i);
+
+#ifdef CONFIG_SMP
+	register_cpu_notifier(&slab_notifier);
+#endif
+
+	if (nr_cpu_ids)	/* Remove when nr_cpu_ids is fixed upstream ! */
+		kmem_size = offsetof(struct kmem_cache, cpu_slab)
+			 + nr_cpu_ids * sizeof(struct page *);
+
+	printk(KERN_INFO "SLUB V6: General Slabs=%d, HW alignment=%d, "
+		"Processors=%d, Nodes=%d\n",
+		KMALLOC_SHIFT_HIGH, L1_CACHE_BYTES,
+		nr_cpu_ids, nr_node_ids);
+}
+
+/*
+ * Find a mergeable slab cache
+ */
+static struct kmem_cache *find_mergeable(size_t size,
+		size_t align, unsigned long flags,
+		void (*ctor)(void *, struct kmem_cache *, unsigned long),
+		void (*dtor)(void *, struct kmem_cache *, unsigned long))
+{
+	struct list_head *h;
+
+	if (slub_nomerge || (flags & SLUB_NEVER_MERGE))
+		return NULL;
+
+	if (ctor || dtor)
+		return NULL;
+
+	size = ALIGN(size, sizeof(void *));
+	align = calculate_alignment(flags, align);
+	size = ALIGN(size, align);
+
+	list_for_each(h, &slab_caches) {
+		struct kmem_cache *s =
+			container_of(h, struct kmem_cache, list);
+
+		if (size > s->size)
+			continue;
+
+		if (s->flags & SLUB_NEVER_MERGE)
+			continue;
+
+		if (s->dtor || s->ctor)
+			continue;
+
+		if (((flags | slub_debug) & SLUB_MERGE_SAME) !=
+			(s->flags & SLUB_MERGE_SAME))
+				continue;
+		/*
+		 * Check if alignment is compatible.
+		 * Courtesy of Adrian Drzewiecki
+		 */
+		if ((s->size & ~(align -1)) != s->size)
+			continue;
+
+		if (s->size - size >= sizeof(void *))
+			continue;
+
+		return s;
+	}
+	return NULL;
+}
+
+struct kmem_cache *kmem_cache_create(const char *name, size_t size,
+		size_t align, unsigned long flags,
+		void (*ctor)(void *, struct kmem_cache *, unsigned long),
+		void (*dtor)(void *, struct kmem_cache *, unsigned long))
+{
+	struct kmem_cache *s;
+
+	down_write(&slub_lock);
+	s = find_mergeable(size, align, flags, dtor, ctor);
+	if (s) {
+		s->refcount++;
+		/*
+		 * Adjust the object sizes so that we clear
+		 * the complete object on kzalloc.
+		 */
+		s->objsize = max(s->objsize, (int)size);
+		s->inuse = max_t(int, s->inuse, ALIGN(size, sizeof(void *)));
+		if (sysfs_slab_alias(s, name))
+			goto err;
+	} else {
+		s = kmalloc(kmem_size, GFP_KERNEL);
+		if (s && kmem_cache_open(s, GFP_KERNEL, name,
+				size, align, flags, ctor, dtor)) {
+			if (sysfs_slab_add(s)) {
+				kfree(s);
+				goto err;
+			}
+			list_add(&s->list, &slab_caches);
+		} else
+			kfree(s);
+	}
+	up_write(&slub_lock);
+	return s;
+
+err:
+	up_write(&slub_lock);
+	if (flags & SLAB_PANIC)
+		panic("Cannot create slabcache %s\n", name);
+	else
+		s = NULL;
+	return s;
+}
+EXPORT_SYMBOL(kmem_cache_create);
+
+void *kmem_cache_zalloc(struct kmem_cache *s, gfp_t flags)
+{
+	void *x;
+
+	x = kmem_cache_alloc(s, flags);
+	if (x)
+		memset(x, 0, s->objsize);
+	return x;
+}
+EXPORT_SYMBOL(kmem_cache_zalloc);
+
+#ifdef CONFIG_SMP
+static void for_all_slabs(void (*func)(struct kmem_cache *, int), int cpu)
+{
+	struct list_head *h;
+
+	down_read(&slub_lock);
+	list_for_each(h, &slab_caches) {
+		struct kmem_cache *s =
+			container_of(h, struct kmem_cache, list);
+
+		func(s, cpu);
+	}
+	up_read(&slub_lock);
+}
+
+/*
+ * Use the cpu notifier to insure that the slab are flushed
+ * when necessary.
+ */
+static int __cpuinit slab_cpuup_callback(struct notifier_block *nfb,
+		unsigned long action, void *hcpu)
+{
+	long cpu = (long)hcpu;
+
+	switch (action) {
+	case CPU_UP_CANCELED:
+	case CPU_DEAD:
+		for_all_slabs(__flush_cpu_slab, cpu);
+		break;
+	default:
+		break;
+	}
+	return NOTIFY_OK;
+}
+
+static struct notifier_block __cpuinitdata slab_notifier =
+	{ &slab_cpuup_callback, NULL, 0 };
+
+#endif
+
+/***************************************************************
+ *	Compatiblility definitions
+ **************************************************************/
+
+int kmem_cache_shrink(struct kmem_cache *s)
+{
+	flush_all(s);
+	return 0;
+}
+EXPORT_SYMBOL(kmem_cache_shrink);
+
+#ifdef CONFIG_NUMA
+
+/*****************************************************************
+ * Generic reaper used to support the page allocator
+ * (the cpu slabs are reaped by a per slab workqueue).
+ *
+ * Maybe move this to the page allocator?
+ ****************************************************************/
+
+static DEFINE_PER_CPU(unsigned long, reap_node);
+
+static void init_reap_node(int cpu)
+{
+	int node;
+
+	node = next_node(cpu_to_node(cpu), node_online_map);
+	if (node == MAX_NUMNODES)
+		node = first_node(node_online_map);
+
+	__get_cpu_var(reap_node) = node;
+}
+
+static void next_reap_node(void)
+{
+	int node = __get_cpu_var(reap_node);
+
+	/*
+	 * Also drain per cpu pages on remote zones
+	 */
+	if (node != numa_node_id())
+		drain_node_pages(node);
+
+	node = next_node(node, node_online_map);
+	if (unlikely(node >= MAX_NUMNODES))
+		node = first_node(node_online_map);
+	__get_cpu_var(reap_node) = node;
+}
+#else
+#define init_reap_node(cpu) do { } while (0)
+#define next_reap_node(void) do { } while (0)
+#endif
+
+#define REAPTIMEOUT_CPUC	(2*HZ)
+
+#ifdef CONFIG_SMP
+static DEFINE_PER_CPU(struct delayed_work, reap_work);
+
+static void cache_reap(struct work_struct *unused)
+{
+	next_reap_node();
+	refresh_cpu_vm_stats(smp_processor_id());
+	schedule_delayed_work(&__get_cpu_var(reap_work),
+				      REAPTIMEOUT_CPUC);
+}
+
+static void __devinit start_cpu_timer(int cpu)
+{
+	struct delayed_work *reap_work = &per_cpu(reap_work, cpu);
+
+	/*
+	 * When this gets called from do_initcalls via cpucache_init(),
+	 * init_workqueues() has already run, so keventd will be setup
+	 * at that time.
+	 */
+	if (keventd_up() && reap_work->work.func == NULL) {
+		init_reap_node(cpu);
+		INIT_DELAYED_WORK(reap_work, cache_reap);
+		schedule_delayed_work_on(cpu, reap_work, HZ + 3 * cpu);
+	}
+}
+
+static int __init cpucache_init(void)
+{
+	int cpu;
+
+	/*
+	 * Register the timers that drain pcp pages and update vm statistics
+	 */
+	for_each_online_cpu(cpu)
+		start_cpu_timer(cpu);
+	return 0;
+}
+__initcall(cpucache_init);
+#endif
+
+/*
+ * These are not as efficient as kmalloc for the non debug case.
+ * We do not have the page struct available so we have to touch one
+ * cacheline in struct kmem_cache to check slab flags.
+ */
+void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, void *caller)
+{
+	struct kmem_cache *s = get_slab(size, gfpflags);
+	void *object;
+
+	if (!s)
+		return NULL;
+
+	object = kmem_cache_alloc(s, gfpflags);
+
+	if (object && (s->flags & SLAB_STORE_USER))
+		set_track(s, object, 0, caller);
+
+	return object;
+}
+
+void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
+					int node, void *caller)
+{
+	struct kmem_cache *s = get_slab(size, gfpflags);
+	void *object;
+
+	if (!s)
+		return NULL;
+
+	object = kmem_cache_alloc_node(s, gfpflags, node);
+
+	if (object && (s->flags & SLAB_STORE_USER))
+		set_track(s, object, 0, caller);
+
+	return object;
+}
+
+#ifdef CONFIG_SYSFS
+
+static unsigned long count_partial(struct kmem_cache_node *n)
+{
+	unsigned long flags;
+	unsigned long x = 0;
+	struct page *page;
+
+	spin_lock_irqsave(&n->list_lock, flags);
+	list_for_each_entry(page, &n->partial, lru)
+		x += page->inuse;
+	spin_unlock_irqrestore(&n->list_lock, flags);
+	return x;
+}
+
+enum slab_stat_type {
+	SL_FULL,
+	SL_PARTIAL,
+	SL_CPU,
+	SL_OBJECTS
+};
+
+#define SO_FULL		(1 << SL_FULL)
+#define SO_PARTIAL	(1 << SL_PARTIAL)
+#define SO_CPU		(1 << SL_CPU)
+#define SO_OBJECTS	(1 << SL_OBJECTS)
+
+static unsigned long slab_objects(struct kmem_cache *s,
+			char *buf, unsigned long flags)
+{
+	unsigned long total = 0;
+	int cpu;
+	int node;
+	int x;
+	unsigned long *nodes;
+
+	nodes = kmalloc(sizeof(unsigned long) * nr_node_ids, GFP_KERNEL);
+
+	for_each_online_node(node) {
+		struct kmem_cache_node *n = get_node(s, node);
+
+		nodes[node] = 0;
+
+		if (flags & SO_FULL) {
+			if (flags & SO_OBJECTS)
+				x = atomic_read(&n->nr_slabs)
+						* s->objects;
+			else
+				x = atomic_read(&n->nr_slabs);
+			total += x;
+			nodes[node] += x;
+		}
+		if (flags & SO_PARTIAL) {
+			if (flags & SO_OBJECTS)
+				x = count_partial(n);
+			else
+				x = n->nr_partial;
+			total += x;
+			nodes[node] += x;
+		}
+	}
+
+	if (flags & SO_CPU)
+		for_each_possible_cpu(cpu) {
+			struct page *page = s->cpu_slab[cpu];
+
+			if (page) {
+				int x = 0;
+				int node = page_to_nid(page);
+
+				if (flags & SO_OBJECTS)
+					x = page->inuse;
+				else
+					x = 1;
+				total += x;
+				nodes[node] += x;
+			}
+		}
+
+	x = sprintf(buf, "%lu", total);
+#ifdef CONFIG_NUMA
+	for_each_online_node(node)
+		if (nodes[node])
+			x += sprintf(buf + x, " N%d=%lu",
+					node, nodes[node]);
+#endif
+	kfree(nodes);
+	return x + sprintf(buf + x, "\n");
+}
+
+static int any_slab_objects(struct kmem_cache *s)
+{
+	int node;
+	int cpu;
+
+	for_each_possible_cpu(cpu)
+		if (s->cpu_slab[cpu])
+			return 1;
+
+	for_each_node(node) {
+		struct kmem_cache_node *n = get_node(s, node);
+
+		if (n->nr_partial || atomic_read(&n->nr_slabs))
+			return 1;
+	}
+	return 0;
+}
+
+#define to_slab_attr(n) container_of(n, struct slab_attribute, attr)
+#define to_slab(n) container_of(n, struct kmem_cache, kobj);
+
+struct slab_attribute {
+	struct attribute attr;
+	ssize_t (*show)(struct kmem_cache *s, char *buf);
+	ssize_t (*store)(struct kmem_cache *s, const char *x, size_t count);
+};
+
+#define SLAB_ATTR_RO(_name) \
+	static struct slab_attribute _name##_attr = __ATTR_RO(_name)
+
+#define SLAB_ATTR(_name) \
+	static struct slab_attribute _name##_attr =  \
+	__ATTR(_name, 0644, _name##_show, _name##_store)
+
+
+static ssize_t slab_size_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", s->size);
+}
+SLAB_ATTR_RO(slab_size);
+
+static ssize_t align_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", s->align);
+}
+SLAB_ATTR_RO(align);
+
+static ssize_t object_size_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", s->objsize);
+}
+SLAB_ATTR_RO(object_size);
+
+static ssize_t objs_per_slab_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", s->objects);
+}
+SLAB_ATTR_RO(objs_per_slab);
+
+static ssize_t order_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", s->order);
+}
+SLAB_ATTR_RO(order);
+
+static ssize_t ctor_show(struct kmem_cache *s, char *buf)
+{
+	if (s->ctor) {
+		int n = sprint_symbol(buf, (unsigned long)s->ctor);
+
+		return n + sprintf(buf + n, "\n");
+	}
+	return 0;
+}
+SLAB_ATTR_RO(ctor);
+
+static ssize_t dtor_show(struct kmem_cache *s, char *buf)
+{
+	if (s->dtor) {
+		int n = sprint_symbol(buf, (unsigned long)s->dtor);
+
+		return n + sprintf(buf + n, "\n");
+	}
+	return 0;
+}
+SLAB_ATTR_RO(dtor);
+
+static ssize_t aliases_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", s->refcount - 1);
+}
+SLAB_ATTR_RO(aliases);
+
+static ssize_t slabs_show(struct kmem_cache *s, char *buf)
+{
+	return slab_objects(s, buf, SO_FULL|SO_PARTIAL|SO_CPU);
+}
+SLAB_ATTR_RO(slabs);
+
+static ssize_t partial_show(struct kmem_cache *s, char *buf)
+{
+	return slab_objects(s, buf, SO_PARTIAL);
+}
+SLAB_ATTR_RO(partial);
+
+static ssize_t cpu_slabs_show(struct kmem_cache *s, char *buf)
+{
+	return slab_objects(s, buf, SO_CPU);
+}
+SLAB_ATTR_RO(cpu_slabs);
+
+static ssize_t objects_show(struct kmem_cache *s, char *buf)
+{
+	return slab_objects(s, buf, SO_FULL|SO_PARTIAL|SO_CPU|SO_OBJECTS);
+}
+SLAB_ATTR_RO(objects);
+
+static ssize_t sanity_checks_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags & SLAB_DEBUG_FREE));
+}
+
+static ssize_t sanity_checks_store(struct kmem_cache *s,
+				const char *buf, size_t length)
+{
+	s->flags &= ~SLAB_DEBUG_FREE;
+	if (buf[0] == '1')
+		s->flags |= SLAB_DEBUG_FREE;
+	return length;
+}
+SLAB_ATTR(sanity_checks);
+
+static ssize_t trace_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags & SLAB_TRACE));
+}
+
+static ssize_t trace_store(struct kmem_cache *s, const char *buf,
+							size_t length)
+{
+	s->flags &= ~SLAB_TRACE;
+	printk("_trace_store = %s\n", buf);
+	if (buf[0] == '1')
+		s->flags |= SLAB_TRACE;
+	return length;
+}
+SLAB_ATTR(trace);
+
+static ssize_t reclaim_account_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags & SLAB_RECLAIM_ACCOUNT));
+}
+
+static ssize_t reclaim_account_store(struct kmem_cache *s,
+				const char *buf, size_t length)
+{
+	s->flags &= ~SLAB_RECLAIM_ACCOUNT;
+	if (buf[0] == '1')
+		s->flags |= SLAB_RECLAIM_ACCOUNT;
+	return length;
+}
+SLAB_ATTR(reclaim_account);
+
+static ssize_t hwcache_align_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags &
+		(SLAB_HWCACHE_ALIGN|SLAB_MUST_HWCACHE_ALIGN)));
+}
+SLAB_ATTR_RO(hwcache_align);
+
+#ifdef CONFIG_ZONE_DMA
+static ssize_t cache_dma_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags & SLAB_CACHE_DMA));
+}
+SLAB_ATTR_RO(cache_dma);
+#endif
+
+static ssize_t destroy_by_rcu_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags & SLAB_DESTROY_BY_RCU));
+}
+SLAB_ATTR_RO(destroy_by_rcu);
+
+static ssize_t red_zone_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags & SLAB_RED_ZONE));
+}
+
+static ssize_t red_zone_store(struct kmem_cache *s,
+				const char *buf, size_t length)
+{
+	if (any_slab_objects(s))
+		return -EBUSY;
+
+	s->flags &= ~SLAB_RED_ZONE;
+	if (buf[0] == '1')
+		s->flags |= SLAB_RED_ZONE;
+	calculate_sizes(s);
+	return length;
+}
+SLAB_ATTR(red_zone);
+
+static ssize_t poison_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags & SLAB_POISON));
+}
+
+static ssize_t poison_store(struct kmem_cache *s,
+				const char *buf, size_t length)
+{
+	if (any_slab_objects(s))
+		return -EBUSY;
+
+	s->flags &= ~SLAB_POISON;
+	if (buf[0] == '1')
+		s->flags |= SLAB_POISON;
+	calculate_sizes(s);
+	return length;
+}
+SLAB_ATTR(poison);
+
+static ssize_t store_user_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", !!(s->flags & SLAB_STORE_USER));
+}
+
+static ssize_t store_user_store(struct kmem_cache *s,
+				const char *buf, size_t length)
+{
+	if (any_slab_objects(s))
+		return -EBUSY;
+
+	s->flags &= ~SLAB_STORE_USER;
+	if (buf[0] == '1')
+		s->flags |= SLAB_STORE_USER;
+	calculate_sizes(s);
+	return length;
+}
+SLAB_ATTR(store_user);
+
+#ifdef CONFIG_NUMA
+static ssize_t defrag_ratio_show(struct kmem_cache *s, char *buf)
+{
+	return sprintf(buf, "%d\n", s->defrag_ratio / 10);
+}
+
+static ssize_t defrag_ratio_store(struct kmem_cache *s,
+				const char *buf, size_t length)
+{
+	int n = simple_strtoul(buf, NULL, 10);
+
+	if (n < 100)
+		s->defrag_ratio = n * 10;
+	return length;
+}
+SLAB_ATTR(defrag_ratio);
+#endif
+
+static struct attribute * slab_attrs[] = {
+	&slab_size_attr.attr,
+	&object_size_attr.attr,
+	&objs_per_slab_attr.attr,
+	&order_attr.attr,
+	&objects_attr.attr,
+	&slabs_attr.attr,
+	&partial_attr.attr,
+	&cpu_slabs_attr.attr,
+	&ctor_attr.attr,
+	&dtor_attr.attr,
+	&aliases_attr.attr,
+	&align_attr.attr,
+	&sanity_checks_attr.attr,
+	&trace_attr.attr,
+	&hwcache_align_attr.attr,
+	&reclaim_account_attr.attr,
+	&destroy_by_rcu_attr.attr,
+	&red_zone_attr.attr,
+	&poison_attr.attr,
+	&store_user_attr.attr,
+#ifdef CONFIG_ZONE_DMA
+	&cache_dma_attr.attr,
+#endif
+#ifdef CONFIG_NUMA
+	&defrag_ratio_attr.attr,
+#endif
+	NULL
+};
+
+static struct attribute_group slab_attr_group = {
+	.attrs = slab_attrs,
+};
+
+static ssize_t slab_attr_show(struct kobject *kobj,
+				struct attribute *attr,
+				char *buf)
+{
+	struct slab_attribute *attribute;
+	struct kmem_cache *s;
+	int err;
+
+	attribute = to_slab_attr(attr);
+	s = to_slab(kobj);
+
+	if (!attribute->show)
+		return -EIO;
+
+	err = attribute->show(s, buf);
+
+	return err;
+}
+
+static ssize_t slab_attr_store(struct kobject *kobj,
+				struct attribute *attr,
+				const char *buf, size_t len)
+{
+	struct slab_attribute *attribute;
+	struct kmem_cache *s;
+	int err;
+
+	attribute = to_slab_attr(attr);
+	s = to_slab(kobj);
+
+	if (!attribute->store)
+		return -EIO;
+
+	err = attribute->store(s, buf, len);
+
+	return err;
+}
+
+static struct sysfs_ops slab_sysfs_ops = {
+	.show = slab_attr_show,
+	.store = slab_attr_store,
+};
+
+static struct kobj_type slab_ktype = {
+	.sysfs_ops = &slab_sysfs_ops,
+};
+
+static int uevent_filter(struct kset *kset, struct kobject *kobj)
+{
+	struct kobj_type *ktype = get_ktype(kobj);
+
+	if (ktype == &slab_ktype)
+		return 1;
+	return 0;
+}
+
+static struct kset_uevent_ops slab_uevent_ops = {
+	.filter = uevent_filter,
+};
+
+decl_subsys(slab, &slab_ktype, &slab_uevent_ops);
+
+static int sysfs_slab_add(struct kmem_cache *s)
+{
+	int err;
+
+	if (slab_state < SYSFS)
+		/* Defer until later */
+		return 0;
+
+	kobj_set_kset_s(s, slab_subsys);
+	kobject_set_name(&s->kobj, s->name);
+	kobject_init(&s->kobj);
+	err = kobject_add(&s->kobj);
+	if (err)
+		return err;
+
+	err = sysfs_create_group(&s->kobj, &slab_attr_group);
+	if (err)
+		return err;
+	kobject_uevent(&s->kobj, KOBJ_ADD);
+	return 0;
+}
+
+static void sysfs_slab_remove(struct kmem_cache *s)
+{
+	kobject_uevent(&s->kobj, KOBJ_REMOVE);
+	kobject_del(&s->kobj);
+}
+
+/*
+ * Need to buffer aliases during bootup until sysfs becomes
+ * available lest we loose that information.
+ */
+struct saved_alias {
+	struct kmem_cache *s;
+	const char *name;
+	struct saved_alias *next;
+};
+
+struct saved_alias *alias_list;
+
+int sysfs_slab_alias(struct kmem_cache *s, const char *name)
+{
+	struct saved_alias *al;
+
+	if (slab_state == SYSFS)
+		return sysfs_create_link(&slab_subsys.kset.kobj,
+						&s->kobj, name);
+
+	al = kmalloc(sizeof(struct saved_alias), GFP_KERNEL);
+	if (!al)
+		return -ENOMEM;
+
+	al->s = s;
+	al->name = name;
+	al->next = alias_list;
+	alias_list = al;
+	return 0;
+}
+
+int __init slab_sysfs_init(void)
+{
+	int err;
+	struct list_head *h;
+
+	err = subsystem_register(&slab_subsys);
+	if (err) {
+		printk(KERN_ERR "Cannot register slab subsystem.\n");
+		return -ENOSYS;
+	}
+
+	slab_state = SYSFS;
+
+	list_for_each(h, &slab_caches) {
+		struct kmem_cache *s =
+			container_of(h, struct kmem_cache, list);
+
+		err = sysfs_slab_add(s);
+		BUG_ON(err);
+	}
+
+	while (alias_list) {
+		struct saved_alias *al = alias_list;
+
+		alias_list = alias_list->next;
+		err = sysfs_slab_alias(al->s, al->name);
+		BUG_ON(err);
+		kfree(al);
+	}
+
+	return 0;
+}
+
+__initcall(slab_sysfs_init);
+#endif
Index: linux-2.6.21-rc5-mm3/include/linux/poison.h
===================================================================
--- linux-2.6.21-rc5-mm3.orig/include/linux/poison.h	2007-03-30 21:50:18.000000000 -0700
+++ linux-2.6.21-rc5-mm3/include/linux/poison.h	2007-03-30 21:50:42.000000000 -0700
@@ -18,6 +18,9 @@
 #define	RED_INACTIVE	0x5A2CF071UL	/* when obj is inactive */
 #define	RED_ACTIVE	0x170FC2A5UL	/* when obj is active */
 
+#define SLUB_RED_INACTIVE	0xbb
+#define SLUB_RED_ACTIVE		0xcc
+
 /* ...and for poisoning */
 #define	POISON_INUSE	0x5a	/* for use-uninitialised poisoning */
 #define POISON_FREE	0x6b	/* for use-after-free poisoning */
Index: linux-2.6.21-rc5-mm3/arch/frv/Kconfig
===================================================================
--- linux-2.6.21-rc5-mm3.orig/arch/frv/Kconfig	2007-03-30 21:50:02.000000000 -0700
+++ linux-2.6.21-rc5-mm3/arch/frv/Kconfig	2007-03-30 21:50:42.000000000 -0700
@@ -53,6 +53,10 @@ config ARCH_HAS_ILOG2_U64
 	bool
 	default y
 
+config ARCH_USES_SLAB_PAGE_STRUCT
+	bool
+	default y
+
 mainmenu "Fujitsu FR-V Kernel Configuration"
 
 source "init/Kconfig"
Index: linux-2.6.21-rc5-mm3/arch/i386/Kconfig
===================================================================
--- linux-2.6.21-rc5-mm3.orig/arch/i386/Kconfig	2007-03-30 21:50:02.000000000 -0700
+++ linux-2.6.21-rc5-mm3/arch/i386/Kconfig	2007-03-30 22:30:53.000000000 -0700
@@ -79,6 +79,10 @@ config ARCH_MAY_HAVE_PC_FDC
 	bool
 	default y
 
+config ARCH_USES_SLAB_PAGE_STRUCT
+	bool
+	default y
+
 config DMI
 	bool
 	default y

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [SLUB 2/2] i386 arch page size slab fixes
  2007-03-31 19:30 [SLUB 0/2] SLUB: The unqueued slab allocator V6 Christoph Lameter
  2007-03-31 19:31 ` [SLUB 1/2] SLUB core Christoph Lameter
@ 2007-03-31 19:31 ` Christoph Lameter
  2007-03-31 19:55   ` Andrew Morton
  2007-04-02 23:00   ` William Lee Irwin III
  2007-03-31 19:31 ` [SLUB tool] slabinfo: Display slab statistics Christoph Lameter
  2 siblings, 2 replies; 10+ messages in thread
From: Christoph Lameter @ 2007-03-31 19:31 UTC (permalink / raw)
  To: akpm; +Cc: linux-mm, linux-kernel, Christoph Lameter, mpm

Fixup i386 arch for SLUB support

i386 arch code currently uses the page struct of slabs for various purposes.
This interferes with slub and so SLUB has been disabled for i386 by setting
ARCH_USES_SLAB_PAGE_STRUCT.

This patch removes the use of page sized slabs for maintaining pgds and pmds.

Patch by William Irwin with only very minor modifications by me which are

1. Removal of HIGHMEM64G slab caches. It seems that virtualization hosts
   require a a full pgd page.

2. Add missing virtualization hook. Seems that we need a new way
   of serializing paravirt_alloc(). It may need to do its own serialization.

3. Remove ARCH_USES_SLAB_PAGE_STRUCT

Note that this makes things work without debugging on.
The arch still fails to boot properly if full SLUB debugging is on with
a cryptic message:

CPU: AMD Athlon(tm) 64 Processor 3000+ stepping 00
Checking 'hlt' instruction... OK.
ACPI: Core revision 20070126
ACPI: setting ELCR to 0200 (from 1ca0)
BUG: at kernel/sched.c:3417 sub_preempt_count()
 [<c0342d43>] _spin_unlock_irq+0x13/0x30
 [<c01160e6>] schedule_tail+0x36/0xd0
 [<c0102df8>] __switch_to+0x28/0x180
 [<c0103f9a>] ret_from_fork+0x6/0x1c
 [<c012acf0>] kthread+0x0/0xe0

This may have a coule of reasons:

1. SLUB breakage. kmalloc caches have been initialized but maybe debugging
   uses a facility that is not available that early (can find nothing).

2. SLAB does not enable full debugging for page order slabs. SLUB does.
   So we were so far unable to verify that the code is clean for these
   slabs. There could be some unsolved slab issues. i386 fails to boot
   if any of the debug options that require additional metadata at the
   end of an object or poisoning is enabled. Boot will work with sanity
   checks only.

Signed-off-by: Christoph Lameter <clameter@sgi.com>
Signed-off-by: William Lee Irwin III <wli@holomorphy.com>

Index: linux-2.6.21-rc5-mm3/arch/i386/mm/init.c
===================================================================
--- linux-2.6.21-rc5-mm3.orig/arch/i386/mm/init.c	2007-03-30 18:26:11.000000000 -0700
+++ linux-2.6.21-rc5-mm3/arch/i386/mm/init.c	2007-03-30 18:28:04.000000000 -0700
@@ -696,31 +696,6 @@ int remove_memory(u64 start, u64 size)
 EXPORT_SYMBOL_GPL(remove_memory);
 #endif
 
-struct kmem_cache *pgd_cache;
-struct kmem_cache *pmd_cache;
-
-void __init pgtable_cache_init(void)
-{
-	if (PTRS_PER_PMD > 1) {
-		pmd_cache = kmem_cache_create("pmd",
-					PTRS_PER_PMD*sizeof(pmd_t),
-					PTRS_PER_PMD*sizeof(pmd_t),
-					0,
-					pmd_ctor,
-					NULL);
-		if (!pmd_cache)
-			panic("pgtable_cache_init(): cannot create pmd cache");
-	}
-	pgd_cache = kmem_cache_create("pgd",
-				PTRS_PER_PGD*sizeof(pgd_t),
-				PTRS_PER_PGD*sizeof(pgd_t),
-				0,
-				pgd_ctor,
-				PTRS_PER_PMD == 1 ? pgd_dtor : NULL);
-	if (!pgd_cache)
-		panic("pgtable_cache_init(): Cannot create pgd cache");
-}
-
 /*
  * This function cannot be __init, since exceptions don't work in that
  * section.  Put this after the callers, so that it cannot be inlined.
Index: linux-2.6.21-rc5-mm3/arch/i386/mm/pageattr.c
===================================================================
--- linux-2.6.21-rc5-mm3.orig/arch/i386/mm/pageattr.c	2007-03-25 15:56:23.000000000 -0700
+++ linux-2.6.21-rc5-mm3/arch/i386/mm/pageattr.c	2007-03-30 18:28:04.000000000 -0700
@@ -87,24 +87,23 @@ static void flush_kernel_map(void *arg)
 
 static void set_pmd_pte(pte_t *kpte, unsigned long address, pte_t pte) 
 { 
-	struct page *page;
-	unsigned long flags;
+	struct mm_struct *mm;
 
 	set_pte_atomic(kpte, pte); 	/* change init_mm */
 	if (PTRS_PER_PMD > 1)
 		return;
 
-	spin_lock_irqsave(&pgd_lock, flags);
-	for (page = pgd_list; page; page = (struct page *)page->index) {
-		pgd_t *pgd;
+	spin_lock(&mmlist_lock);
+	list_for_each_entry(mm, &init_mm.mmlist, mmlist) {
+		pgd_t *pgd = mm->pgd;
 		pud_t *pud;
 		pmd_t *pmd;
-		pgd = (pgd_t *)page_address(page) + pgd_index(address);
+
 		pud = pud_offset(pgd, address);
 		pmd = pmd_offset(pud, address);
 		set_pte_atomic((pte_t *)pmd, pte);
 	}
-	spin_unlock_irqrestore(&pgd_lock, flags);
+	spin_unlock(&mmlist_lock);
 }
 
 /* 
Index: linux-2.6.21-rc5-mm3/arch/i386/mm/pgtable.c
===================================================================
--- linux-2.6.21-rc5-mm3.orig/arch/i386/mm/pgtable.c	2007-03-25 15:56:23.000000000 -0700
+++ linux-2.6.21-rc5-mm3/arch/i386/mm/pgtable.c	2007-03-30 18:28:04.000000000 -0700
@@ -181,109 +181,30 @@ void reserve_top_address(unsigned long r
 #endif
 }
 
-pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
-{
-	return (pte_t *)__get_free_page(GFP_KERNEL|__GFP_REPEAT|__GFP_ZERO);
-}
-
-struct page *pte_alloc_one(struct mm_struct *mm, unsigned long address)
-{
-	struct page *pte;
-
-#ifdef CONFIG_HIGHPTE
-	pte = alloc_pages(GFP_KERNEL|__GFP_HIGHMEM|__GFP_REPEAT|__GFP_ZERO, 0);
-#else
-	pte = alloc_pages(GFP_KERNEL|__GFP_REPEAT|__GFP_ZERO, 0);
-#endif
-	return pte;
-}
-
-void pmd_ctor(void *pmd, struct kmem_cache *cache, unsigned long flags)
-{
-	memset(pmd, 0, PTRS_PER_PMD*sizeof(pmd_t));
-}
-
-/*
- * List of all pgd's needed for non-PAE so it can invalidate entries
- * in both cached and uncached pgd's; not needed for PAE since the
- * kernel pmd is shared. If PAE were not to share the pmd a similar
- * tactic would be needed. This is essentially codepath-based locking
- * against pageattr.c; it is the unique case in which a valid change
- * of kernel pagetables can't be lazily synchronized by vmalloc faults.
- * vmalloc faults work because attached pagetables are never freed.
- * The locking scheme was chosen on the basis of manfred's
- * recommendations and having no core impact whatsoever.
- * -- wli
- */
-DEFINE_SPINLOCK(pgd_lock);
-struct page *pgd_list;
-
-static inline void pgd_list_add(pgd_t *pgd)
-{
-	struct page *page = virt_to_page(pgd);
-	page->index = (unsigned long)pgd_list;
-	if (pgd_list)
-		set_page_private(pgd_list, (unsigned long)&page->index);
-	pgd_list = page;
-	set_page_private(page, (unsigned long)&pgd_list);
-}
-
-static inline void pgd_list_del(pgd_t *pgd)
-{
-	struct page *next, **pprev, *page = virt_to_page(pgd);
-	next = (struct page *)page->index;
-	pprev = (struct page **)page_private(page);
-	*pprev = next;
-	if (next)
-		set_page_private(next, (unsigned long)pprev);
-}
-
-void pgd_ctor(void *pgd, struct kmem_cache *cache, unsigned long unused)
-{
-	unsigned long flags;
-
-	if (PTRS_PER_PMD == 1) {
-		memset(pgd, 0, USER_PTRS_PER_PGD*sizeof(pgd_t));
-		spin_lock_irqsave(&pgd_lock, flags);
-	}
-
-	clone_pgd_range((pgd_t *)pgd + USER_PTRS_PER_PGD,
-			swapper_pg_dir + USER_PTRS_PER_PGD,
-			KERNEL_PGD_PTRS);
-
-	if (PTRS_PER_PMD > 1)
-		return;
-
-	/* must happen under lock */
-	paravirt_alloc_pd_clone(__pa(pgd) >> PAGE_SHIFT,
-			__pa(swapper_pg_dir) >> PAGE_SHIFT,
-			USER_PTRS_PER_PGD, PTRS_PER_PGD - USER_PTRS_PER_PGD);
-
-	pgd_list_add(pgd);
-	spin_unlock_irqrestore(&pgd_lock, flags);
-}
-
-/* never called when PTRS_PER_PMD > 1 */
-void pgd_dtor(void *pgd, struct kmem_cache *cache, unsigned long unused)
-{
-	unsigned long flags; /* can be called from interrupt context */
-
-	paravirt_release_pd(__pa(pgd) >> PAGE_SHIFT);
-	spin_lock_irqsave(&pgd_lock, flags);
-	pgd_list_del(pgd);
-	spin_unlock_irqrestore(&pgd_lock, flags);
-}
+#define __pgd_alloc()	((pgd_t *)get_zeroed_page(GFP_KERNEL|__GFP_REPEAT))
+#define __pgd_free(pgd)	free_page((unsigned long)(pgd))
 
 pgd_t *pgd_alloc(struct mm_struct *mm)
 {
 	int i;
-	pgd_t *pgd = kmem_cache_alloc(pgd_cache, GFP_KERNEL);
+	pgd_t *pgd = __pgd_alloc();
 
-	if (PTRS_PER_PMD == 1 || !pgd)
+	if (!pgd)
+		return NULL;
+	clone_pgd_range((pgd_t *)pgd + USER_PTRS_PER_PGD,
+		swapper_pg_dir + USER_PTRS_PER_PGD, KERNEL_PGD_PTRS);
+	if (PTRS_PER_PMD == 1)
 		return pgd;
+	/*
+	 * Beware. We do not have the pgd_lock for serialization anymore.
+	 * paravirt_alloc_pd_clone needs to have its own serialization?
+	 */
+	 paravirt_alloc_pd_clone(__pa(pgd) >> PAGE_SHIFT,
+		__pa(swapper_pg_dir) >> PAGE_SHIFT,
+		USER_PTRS_PER_PGD, PTRS_PER_PGD - USER_PTRS_PER_PGD);
 
 	for (i = 0; i < USER_PTRS_PER_PGD; ++i) {
-		pmd_t *pmd = kmem_cache_alloc(pmd_cache, GFP_KERNEL);
+		pmd_t *pmd = (pmd_t *)get_zeroed_page(GFP_KERNEL|__GFP_REPEAT);
 		if (!pmd)
 			goto out_oom;
 		paravirt_alloc_pd(__pa(pmd) >> PAGE_SHIFT);
@@ -296,9 +217,9 @@ out_oom:
 		pgd_t pgdent = pgd[i];
 		void* pmd = (void *)__va(pgd_val(pgdent)-1);
 		paravirt_release_pd(__pa(pmd) >> PAGE_SHIFT);
-		kmem_cache_free(pmd_cache, pmd);
+		free_page((unsigned long)pmd);
 	}
-	kmem_cache_free(pgd_cache, pgd);
+	__pgd_free(pgd);
 	return NULL;
 }
 
@@ -312,8 +233,8 @@ void pgd_free(pgd_t *pgd)
 			pgd_t pgdent = pgd[i];
 			void* pmd = (void *)__va(pgd_val(pgdent)-1);
 			paravirt_release_pd(__pa(pmd) >> PAGE_SHIFT);
-			kmem_cache_free(pmd_cache, pmd);
+			free_page((unsigned long)pmd);
 		}
 	/* in the non-PAE case, free_pgtables() clears user pgd entries */
-	kmem_cache_free(pgd_cache, pgd);
+	__pgd_free(pgd);
 }
Index: linux-2.6.21-rc5-mm3/include/asm-i386/pgalloc.h
===================================================================
--- linux-2.6.21-rc5-mm3.orig/include/asm-i386/pgalloc.h	2007-03-30 18:26:15.000000000 -0700
+++ linux-2.6.21-rc5-mm3/include/asm-i386/pgalloc.h	2007-03-30 18:28:04.000000000 -0700
@@ -35,8 +35,22 @@ do {								\
 extern pgd_t *pgd_alloc(struct mm_struct *);
 extern void pgd_free(pgd_t *pgd);
 
-extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
-extern struct page *pte_alloc_one(struct mm_struct *, unsigned long);
+static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long uvaddr)
+{
+	return (pte_t *)get_zeroed_page(GFP_KERNEL|__GFP_REPEAT);
+}
+
+#ifdef CONFIG_HIGHPTE
+static inline struct page *pte_alloc_one(struct mm_struct *mm, unsigned long uvaddr)
+{
+	return alloc_page(GFP_KERNEL|__GFP_HIGHMEM|__GFP_REPEAT|__GFP_ZERO);
+}
+#else /* !CONFIG_HIGHPTE */
+static inline struct page *pte_alloc_one(struct mm_struct *mm, unsigned long uvaddr)
+{
+	return alloc_page(GFP_KERNEL|__GFP_REPEAT|__GFP_ZERO);
+}
+#endif /* !CONFIG_HIGHPTE */
 
 static inline void pte_free_kernel(pte_t *pte)
 {
Index: linux-2.6.21-rc5-mm3/include/asm-i386/pgtable.h
===================================================================
--- linux-2.6.21-rc5-mm3.orig/include/asm-i386/pgtable.h	2007-03-30 18:26:15.000000000 -0700
+++ linux-2.6.21-rc5-mm3/include/asm-i386/pgtable.h	2007-03-30 18:28:04.000000000 -0700
@@ -35,15 +35,6 @@ struct vm_area_struct;
 #define ZERO_PAGE(vaddr) (virt_to_page(empty_zero_page))
 extern unsigned long empty_zero_page[1024];
 extern pgd_t swapper_pg_dir[1024];
-extern struct kmem_cache *pgd_cache;
-extern struct kmem_cache *pmd_cache;
-extern spinlock_t pgd_lock;
-extern struct page *pgd_list;
-
-void pmd_ctor(void *, struct kmem_cache *, unsigned long);
-void pgd_ctor(void *, struct kmem_cache *, unsigned long);
-void pgd_dtor(void *, struct kmem_cache *, unsigned long);
-void pgtable_cache_init(void);
 void paging_init(void);
 
 /*
Index: linux-2.6.21-rc5-mm3/include/asm-i386/pgtable-2level.h
===================================================================
--- linux-2.6.21-rc5-mm3.orig/include/asm-i386/pgtable-2level.h	2007-03-25 15:56:23.000000000 -0700
+++ linux-2.6.21-rc5-mm3/include/asm-i386/pgtable-2level.h	2007-03-30 18:28:04.000000000 -0700
@@ -67,5 +67,6 @@ static inline int pte_exec_kernel(pte_t 
 #define __swp_entry_to_pte(x)		((pte_t) { (x).val })
 
 void vmalloc_sync_all(void);
+#define pgtable_cache_init()		do { } while (0)
 
 #endif /* _I386_PGTABLE_2LEVEL_H */
Index: linux-2.6.21-rc5-mm3/include/asm-i386/pgtable-3level.h
===================================================================
--- linux-2.6.21-rc5-mm3.orig/include/asm-i386/pgtable-3level.h	2007-03-25 15:56:23.000000000 -0700
+++ linux-2.6.21-rc5-mm3/include/asm-i386/pgtable-3level.h	2007-03-30 18:28:04.000000000 -0700
@@ -188,5 +188,6 @@ static inline pmd_t pfn_pmd(unsigned lon
 #define __pmd_free_tlb(tlb, x)		do { } while (0)
 
 #define vmalloc_sync_all() ((void)0)
+void pgtable_cache_init(void);
 
 #endif /* _I386_PGTABLE_3LEVEL_H */
Index: linux-2.6.21-rc5-mm3/arch/i386/mm/fault.c
===================================================================
--- linux-2.6.21-rc5-mm3.orig/arch/i386/mm/fault.c	2007-03-30 18:26:11.000000000 -0700
+++ linux-2.6.21-rc5-mm3/arch/i386/mm/fault.c	2007-03-30 18:28:04.000000000 -0700
@@ -618,19 +618,19 @@ void vmalloc_sync_all(void)
 	BUILD_BUG_ON(TASK_SIZE & ~PGDIR_MASK);
 	for (address = start; address >= TASK_SIZE; address += PGDIR_SIZE) {
 		if (!test_bit(pgd_index(address), insync)) {
-			unsigned long flags;
-			struct page *page;
+			struct mm_struct *mm;
+			int broken = 0;
 
-			spin_lock_irqsave(&pgd_lock, flags);
-			for (page = pgd_list; page; page =
-					(struct page *)page->index)
-				if (!vmalloc_sync_one(page_address(page),
-								address)) {
-					BUG_ON(page != pgd_list);
-					break;
-				}
-			spin_unlock_irqrestore(&pgd_lock, flags);
-			if (!page)
+			spin_lock(&mmlist_lock);
+			list_for_each_entry(mm, &init_mm.mmlist, mmlist) {
+				if (vmalloc_sync_one(mm->pgd, address))
+					continue;
+				BUG_ON(mm->mmlist.prev != &init_mm.mmlist);
+				broken = 1;
+				break;
+			}
+			spin_unlock(&mmlist_lock);
+			if (!broken)
 				set_bit(pgd_index(address), insync);
 		}
 		if (address == start && test_bit(pgd_index(address), insync))
Index: linux-2.6.21-rc5-mm3/arch/i386/Kconfig
===================================================================
--- linux-2.6.21-rc5-mm3.orig/arch/i386/Kconfig	2007-03-30 18:27:03.000000000 -0700
+++ linux-2.6.21-rc5-mm3/arch/i386/Kconfig	2007-03-30 18:28:04.000000000 -0700
@@ -79,10 +79,6 @@ config ARCH_MAY_HAVE_PC_FDC
 	bool
 	default y
 
-config ARCH_USES_SLAB_PAGE_STRUCT
-	bool
-	default y
-
 config DMI
 	bool
 	default y

^ permalink raw reply	[flat|nested] 10+ messages in thread

* [SLUB tool] slabinfo: Display slab statistics
  2007-03-31 19:30 [SLUB 0/2] SLUB: The unqueued slab allocator V6 Christoph Lameter
  2007-03-31 19:31 ` [SLUB 1/2] SLUB core Christoph Lameter
  2007-03-31 19:31 ` [SLUB 2/2] i386 arch page size slab fixes Christoph Lameter
@ 2007-03-31 19:31 ` Christoph Lameter
  2 siblings, 0 replies; 10+ messages in thread
From: Christoph Lameter @ 2007-03-31 19:31 UTC (permalink / raw)
  To: akpm; +Cc: linux-mm, mpm, linux-kernel, Christoph Lameter

/*
 * Slabinfo: Tool to get reports about slabs
 *
 * (C) 2007 sgi, Christoph Lameter <clameter@sgi.com>
 *
 * Compile by doing:
 *
 * gcc -o slabinfo slabinfo.c
 */
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <dirent.h>
#include <string.h>
#include <unistd.h>
#include <stdarg.h>

char buffer[200];

int show_alias = 0;
int show_slab = 1;
int show_parameter = 0;
int skip_zero = 1;

int page_size;

void fatal(const char *x, ...)
{
	va_list ap;

	va_start(ap, x);
	vfprintf(stderr, x, ap);
	va_end(ap);
	exit(1);
}

/*
 * Get the contents of an attribute
 */
unsigned long get_obj(char *name)
{
	FILE *f = fopen(name, "r");
	unsigned long result = 0;

	if (!f) {
		getcwd(buffer, sizeof(buffer));
		fatal("Cannot open file '%s/%s'\n", buffer, name);
	}

	if (fgets(buffer,sizeof(buffer), f))
		result = atol(buffer);
	fclose(f);
	return result;
}

/*
 * Put a size string together
 */
int store_size(char *buffer, unsigned long value)
{
	unsigned long divisor = 1;
	char trailer = 0;
	int n;

	if (value > 1000000000UL) {
		divisor = 100000000UL;
		trailer = 'G';
	} else if (value > 1000000UL) {
		divisor = 100000UL;
		trailer = 'M';
	} else if (value > 1000UL) {
		divisor = 100;
		trailer = 'K';
	}

	value /= divisor;
	n = sprintf(buffer, "%ld",value);
	if (trailer) {
		buffer[n] = trailer;
		n++;
		buffer[n] = 0;
	}
	if (divisor != 1) {
		memmove(buffer + n - 2, buffer + n - 3, 4);
		buffer[n-2] = '.';
		n++;
	}
	return n;
}

void alias(const char *name)
{
	char *target;

	if (!show_alias)
		return;
	/* Read link target */
	printf("%20s -> %s", name, target);
}

int line = 0;

void first_line(void)
{
	printf("Name                Objects   Objsize    Space "
		"Slabs/Part/Cpu O/S O %%Fr %%Ef Flg\n");
}

void slab(const char *name)
{
	unsigned long aliases, align, cache_dma, cpu_slabs, destroy_by_rcu;
	unsigned long hwcache_align, object_size, objects, objs_per_slab;
	unsigned long order, partial, poison, reclaim_account, red_zone;
	unsigned long sanity_checks, slab_size, slabs, store_user, trace;
	char size_str[20];
	char dist_str[40];
	char flags[20];
	char *p = flags;

	if (!show_slab)
		return;

	if (chdir(name))
		fatal("Unable to access slab %s\n", name);

	aliases = get_obj("aliases");
	align = get_obj("align");
	cache_dma = get_obj("cache_dma");
	cpu_slabs = get_obj("cpu_slabs");
	destroy_by_rcu = get_obj("destroy_by_rcu");
	hwcache_align = get_obj("hwcache_align");
	object_size = get_obj("object_size");
	objects = get_obj("objects");
	objs_per_slab = get_obj("objs_per_slab");
	order = get_obj("order");
	partial = get_obj("partial");
	poison = get_obj("poison");
	reclaim_account = get_obj("reclaim_account");
	red_zone = get_obj("red_zone");
	sanity_checks = get_obj("sanity_checks");
	slab_size = get_obj("slab_size");
	slabs = get_obj("slabs");
	store_user = get_obj("store_user");
	trace = get_obj("trace");

	if (skip_zero && !slabs)
		goto out;

	store_size(size_str, slabs * page_size);
	sprintf(dist_str,"%lu/%lu/%lu", slabs, partial, cpu_slabs);

	if (!line++)
		first_line();

	if (aliases)
		*p++ = '*';
	if (cache_dma)
		*p++ = 'd';
	if (hwcache_align)
		*p++ = 'A';
	if (poison)
		*p++ = 'P';
	if (reclaim_account)
		*p++ = 'a';
	if (red_zone)
		*p++ = 'Z';
	if (sanity_checks)
		*p++ = 'F';
	if (store_user)
		*p++ = 'U';
	if (trace)
		*p++ = 'T';

	*p = 0;
	printf("%-20s %8ld %7d %8s %14s %3ld %1ld %3d %3d %s\n",
			name, objects, object_size, size_str, dist_str,
			objs_per_slab, order,
			slabs ? (partial * 100) / slabs : 100,
			slabs ? (objects * object_size * 100) / (slabs * (page_size << order)) : 100,
			flags);
out:
	chdir("..");
}

void parameter(const char *name)
{
	if (!show_parameter)
		return;
}

int main(int argc, char *argv[])
{
	DIR *dir;
	struct dirent *de;

	page_size = getpagesize();
	if (chdir("/sys/slab"))
		fatal("This kernel does not have SLUB support.\n");

	dir = opendir(".");
	while ((de = readdir(dir))) {
		if (de->d_name[0] == '.')
			continue;
		switch (de->d_type) {
		   case DT_LNK:
			alias(de->d_name);
			break;
		   case DT_DIR:
			slab(de->d_name);
			break;
		   case DT_REG:
			parameter(de->d_name);
			break;
		   default :
			fatal("Unknown file type %lx\n", de->d_type);
		}
	}
	closedir(dir);
	return 0;
}


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [SLUB 2/2] i386 arch page size slab fixes
  2007-03-31 19:31 ` [SLUB 2/2] i386 arch page size slab fixes Christoph Lameter
@ 2007-03-31 19:55   ` Andrew Morton
  2007-03-31 20:00     ` Christoph Lameter
  2007-04-02 18:30     ` Christoph Lameter
  2007-04-02 23:00   ` William Lee Irwin III
  1 sibling, 2 replies; 10+ messages in thread
From: Andrew Morton @ 2007-03-31 19:55 UTC (permalink / raw)
  To: Christoph Lameter; +Cc: linux-mm, linux-kernel, mpm

On Sat, 31 Mar 2007 11:31:07 -0800 (PST) Christoph Lameter <clameter@sgi.com> wrote:

> Fixup i386 arch for SLUB support
> 
> i386 arch code currently uses the page struct of slabs for various purposes.
> This interferes with slub and so SLUB has been disabled for i386 by setting
> ARCH_USES_SLAB_PAGE_STRUCT.
> 
> This patch removes the use of page sized slabs for maintaining pgds and pmds.
> 
> Patch by William Irwin with only very minor modifications by me which are
> 
> 1. Removal of HIGHMEM64G slab caches. It seems that virtualization hosts
>    require a a full pgd page.
> 
> 2. Add missing virtualization hook. Seems that we need a new way
>    of serializing paravirt_alloc(). It may need to do its own serialization.
> 
> 3. Remove ARCH_USES_SLAB_PAGE_STRUCT
> 
> Note that this makes things work without debugging on.
> The arch still fails to boot properly if full SLUB debugging is on with
> a cryptic message:
> 
> CPU: AMD Athlon(tm) 64 Processor 3000+ stepping 00
> Checking 'hlt' instruction... OK.
> ACPI: Core revision 20070126
> ACPI: setting ELCR to 0200 (from 1ca0)
> BUG: at kernel/sched.c:3417 sub_preempt_count()
>  [<c0342d43>] _spin_unlock_irq+0x13/0x30
>  [<c01160e6>] schedule_tail+0x36/0xd0
>  [<c0102df8>] __switch_to+0x28/0x180
>  [<c0103f9a>] ret_from_fork+0x6/0x1c
>  [<c012acf0>] kthread+0x0/0xe0

This all has the potential to make my inbox hurt.

Can we disable SLUB on i386 in Kconfig until it gets sorted out?

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [SLUB 2/2] i386 arch page size slab fixes
  2007-03-31 19:55   ` Andrew Morton
@ 2007-03-31 20:00     ` Christoph Lameter
  2007-04-02 18:30     ` Christoph Lameter
  1 sibling, 0 replies; 10+ messages in thread
From: Christoph Lameter @ 2007-03-31 20:00 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-mm, linux-kernel, mpm

On Sat, 31 Mar 2007, Andrew Morton wrote:

> Can we disable SLUB on i386 in Kconfig until it gets sorted out?

Yes just do not apply this patch. The first one disables SLUB on i386.


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [SLUB 2/2] i386 arch page size slab fixes
  2007-03-31 19:55   ` Andrew Morton
  2007-03-31 20:00     ` Christoph Lameter
@ 2007-04-02 18:30     ` Christoph Lameter
  1 sibling, 0 replies; 10+ messages in thread
From: Christoph Lameter @ 2007-04-02 18:30 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-mm, linux-kernel, mpm

On Sat, 31 Mar 2007, Andrew Morton wrote:

> Can we disable SLUB on i386 in Kconfig until it gets sorted out?

Found it. Another issue with relying on page size special casing in the 
SLAB.


i386: Another case of i386 exploiting page sized slab cache special casing

i386 uses kmalloc to allocate the thread structure assuming that the 
allocation results in a page size aligned allocation. That has worked so 
far because SLAB exempts page sized slabs from debugging and aligns them 
in special ways that gobeyond the restrictions imposed by 
KMALLOC_ARCH_MINALIGN (which are valid for all other kmalloc caches).

SLUB also works fine since the page sized allocations neatly align at page
boundaries. However, if debugging is switched on then SLUB have to add 
additional debugging information after the object increasing the slab 
size. The total object only be aligned following the requirements imposed 
by KMALLOC_ARCH_MINALIGN. There is nothing there that would require
a page size alignment.

The solution here is to replace the calls to kmalloc with calls into 
the page allocator. An alternate solution may be to create a custom slab 
cache and setting the alignment is to PAGE_SIZE (works with SLUB. 
There is no problem here with page_struct modifications). That would allow slub 
debugging to be used on the threadinfo structure.

But this is a page sized allocation after all and for efficiencies sake we 
should call directly into the page allocator and not use slab.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.21-rc5-mm3/include/asm-i386/thread_info.h
===================================================================
--- linux-2.6.21-rc5-mm3.orig/include/asm-i386/thread_info.h	2007-04-02 05:57:31.000000000 +0000
+++ linux-2.6.21-rc5-mm3/include/asm-i386/thread_info.h	2007-04-02 06:03:09.000000000 +0000
@@ -95,12 +95,14 @@
 
 /* thread information allocation */
 #ifdef CONFIG_DEBUG_STACK_USAGE
-#define alloc_thread_info(tsk) kzalloc(THREAD_SIZE, GFP_KERNEL)
+#define alloc_thread_info(tsk) ((struct thread_info *) \
+	__get_free_pages(GFP_KERNEL| __GFP_ZERO, get_order(THREAD_SIZE)))
 #else
-#define alloc_thread_info(tsk) kmalloc(THREAD_SIZE, GFP_KERNEL)
+#define alloc_thread_info(tsk) ((struct thread_info *) \
+	__get_free_pages(GFP_KERNEL, get_order(THREAD_SIZE)))
 #endif
 
-#define free_thread_info(info)	kfree(info)
+#define free_thread_info(info)	free_pages((unsigned long)(info), get_order(THREAD_SIZE))
 
 #else /* !__ASSEMBLY__ */
 

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [SLUB 2/2] i386 arch page size slab fixes
  2007-03-31 19:31 ` [SLUB 2/2] i386 arch page size slab fixes Christoph Lameter
  2007-03-31 19:55   ` Andrew Morton
@ 2007-04-02 23:00   ` William Lee Irwin III
  2007-04-02 23:02     ` Christoph Lameter
  1 sibling, 1 reply; 10+ messages in thread
From: William Lee Irwin III @ 2007-04-02 23:00 UTC (permalink / raw)
  To: Christoph Lameter; +Cc: akpm, linux-mm, linux-kernel, mpm

On Sat, Mar 31, 2007 at 11:31:07AM -0800, Christoph Lameter wrote:
> Patch by William Irwin with only very minor modifications by me which are
> 1. Removal of HIGHMEM64G slab caches. It seems that virtualization hosts
>    require a a full pgd page.

The HIGHMEM64G slab allocations are meaningfully performant vs.
page-sized allocations where virtualization is absent. I would
personally rather whip Xen into shape enough to be able to handle the
minimal pgd allocations than retain the oversized pgd allocations even
in only the Xen case. Also, the entire unshared kernel pmd shenanigan
in Xen is an artifact of its recursive pagetable affair, which can also
be done away with a SMOP.


On Sat, Mar 31, 2007 at 11:31:07AM -0800, Christoph Lameter wrote:
> 2. Add missing virtualization hook. Seems that we need a new way
>    of serializing paravirt_alloc(). It may need to do its own serialization.
> 3. Remove ARCH_USES_SLAB_PAGE_STRUCT

This doesn't quite cover all bases. The changes to pageattr.c and
fault.c are dubious and need verification at the very least. They were
largely slapped together to get the files past the compiler for the
performance comparisons that were never properly done.


-- wli

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [SLUB 2/2] i386 arch page size slab fixes
  2007-04-02 23:00   ` William Lee Irwin III
@ 2007-04-02 23:02     ` Christoph Lameter
  0 siblings, 0 replies; 10+ messages in thread
From: Christoph Lameter @ 2007-04-02 23:02 UTC (permalink / raw)
  To: William Lee Irwin III; +Cc: akpm, linux-mm, linux-kernel, mpm

On Mon, 2 Apr 2007, William Lee Irwin III wrote:

> This doesn't quite cover all bases. The changes to pageattr.c and
> fault.c are dubious and need verification at the very least. They were
> largely slapped together to get the files past the compiler for the
> performance comparisons that were never properly done.

I looked through them but then I am no i386 specialist though. Looked 
fine.



^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [SLUB 1/2] SLUB core
  2007-03-31 19:31 ` [SLUB 1/2] SLUB core Christoph Lameter
@ 2007-04-03  7:29   ` Nick Piggin
  0 siblings, 0 replies; 10+ messages in thread
From: Nick Piggin @ 2007-04-03  7:29 UTC (permalink / raw)
  To: Christoph Lameter; +Cc: akpm, linux-mm, mpm, linux-kernel, Christoph Hellwig

[-- Attachment #1: Type: text/plain, Size: 2532 bytes --]

Christoph Lameter wrote:
> SLUB Core patch V6
> 
> This provides basic SLUB functionality and allows a choice of
> slab allocators during kernel configuration. The default is still
> slab. SLUB has been tested in various configurations but I think we
> can be quite sure that there are still remaining issues.
> 
> SLUB is not selectable on platforms that modify the page structs of
> slab memory (i386, FRV) but see the next patch for a i386 fix.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>
> 
> Index: linux-2.6.21-rc5-mm3/include/linux/mm_types.h
> ===================================================================
> --- linux-2.6.21-rc5-mm3.orig/include/linux/mm_types.h	2007-03-30 21:50:18.000000000 -0700
> +++ linux-2.6.21-rc5-mm3/include/linux/mm_types.h	2007-03-30 21:50:42.000000000 -0700
> @@ -19,10 +19,16 @@ struct page {
>  	unsigned long flags;		/* Atomic flags, some possibly
>  					 * updated asynchronously */
>  	atomic_t _count;		/* Usage count, see below. */
> -	atomic_t _mapcount;		/* Count of ptes mapped in mms,
> +	union {
> +		atomic_t _mapcount;	/* Count of ptes mapped in mms,
>  					 * to show when page is mapped
>  					 * & limit reverse map searches.
>  					 */
> +		struct {	/* SLUB uses */
> +			short unsigned int inuse;
> +			short unsigned int offset;
> +		};
> +	};
>  	union {
>  	    struct {
>  		unsigned long private;		/* Mapping-private opaque data:
> @@ -43,8 +49,15 @@ struct page {
>  #if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS
>  	    spinlock_t ptl;
>  #endif
> +	    struct {			/* SLUB uses */
> +		struct page *first_page;	/* Compound pages */
> +		struct kmem_cache *slab;	/* Pointer to slab */
> +	    };
> +	};
> +	union {
> +		pgoff_t index;		/* Our offset within mapping. */
> +		void *freelist;		/* SLUB: pointer to free object */
>  	};
> -	pgoff_t index;			/* Our offset within mapping. */
>  	struct list_head lru;		/* Pageout list, eg. active_list
>  					 * protected by zone->lru_lock !
>  					 */

I have an alternate suggestion for the struct page overloading issue.

I think anonymous unions are probably better than nothing, because they
make the .c code more readable, but they clutter up the header badly
and don't provide so strong type checking as we could implement.

This is just an incomplete RFC patch, because it actually requires a
fair amount of types in .c code to be changed if we are to implement it
(which I didn't want to do if nobody likes the idea).

(sorry it's an attachment, this mailer is broken)

-- 
SUSE Labs, Novell Inc.

[-- Attachment #2: mm-multi-page.patch --]
[-- Type: text/plain, Size: 3338 bytes --]

Index: linux-2.6/include/linux/mm_types.h
===================================================================
--- linux-2.6.orig/include/linux/mm_types.h
+++ linux-2.6/include/linux/mm_types.h
@@ -23,27 +23,20 @@ struct page {
 					 * to show when page is mapped
 					 * & limit reverse map searches.
 					 */
-	union {
-	    struct {
-		unsigned long private;		/* Mapping-private opaque data:
-					 	 * usually used for buffer_heads
-						 * if PagePrivate set; used for
-						 * swp_entry_t if PageSwapCache;
-						 * indicates order in the buddy
-						 * system if PG_buddy is set.
-						 */
-		struct address_space *mapping;	/* If low bit clear, points to
-						 * inode address_space, or NULL.
-						 * If page mapped as anonymous
-						 * memory, low bit is set, and
-						 * it points to anon_vma object:
-						 * see PAGE_MAPPING_ANON below.
-						 */
-	    };
-#if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS
-	    spinlock_t ptl;
-#endif
-	};
+	unsigned long private;		/* Mapping-private opaque data:
+				 	 * usually used for buffer_heads
+					 * if PagePrivate set; used for
+					 * swp_entry_t if PageSwapCache;
+					 * indicates order in the buddy
+					 * system if PG_buddy is set.
+					 */
+	struct address_space *mapping;	/* If low bit clear, points to
+					 * inode address_space, or NULL.
+					 * If page mapped as anonymous
+					 * memory, low bit is set, and
+					 * it points to anon_vma object:
+					 * see PAGE_MAPPING_ANON below.
+					 */
 	pgoff_t index;			/* Our offset within mapping. */
 	struct list_head lru;		/* Pageout list, eg. active_list
 					 * protected by zone->lru_lock !
@@ -64,4 +57,45 @@ struct page {
 #endif /* WANT_PAGE_VIRTUAL */
 };
 
+/*
+ * We can provide alternate delcarations for 'struct page', instead of
+ * overloading fields that we would like to use for non-pagecache (eg.
+ * page tables or slab pages). However flags and _count must always be
+ * in the correct position, and not overloaded, because they might be
+ * accessed by thing without a direct reference (eg. buddy allocator,
+ * swsusp). So flags and _count get defined for us, and our struct is
+ * automatically padded out to the size of struct page. A compile time
+ * check ensures that this size isn't exceeded.
+ */
+#define DEFINE_PAGE_STRUCT(name)	\
+struct name {				\
+	union {				\
+		struct {		\
+			unsigned long flags; \
+			atomic_t _count; \
+			struct
+
+#define END_PAGE_STRUCT(name)		\
+			;		\
+		};			\
+		struct page ___page;	\
+	};				\
+};					\
+					\
+static inline void struct_##name##_build_bug_on(void) \
+{					\
+	BUILD_BUG_ON(sizeof(struct name) > sizeof(struct page)); \
+}
+
+DEFINE_PAGE_STRUCT(pt_page) {
+#if NR_CPUS >= CONFIG_SPLIT_PTLOCK_CPUS
+	spinlock_t ptl;
+#endif
+} END_PAGE_STRUCT(pt_page)
+
+DEFINE_PAGE_STRUCT(buddy_page) {
+	unsigned int order;
+	struct list_head list;
+} END_PAGE_STRUCT(buddy_page)
+
 #endif /* _LINUX_MM_TYPES_H */
Index: linux-2.6/mm/slab.c
===================================================================
--- linux-2.6.orig/mm/slab.c
+++ linux-2.6/mm/slab.c
@@ -446,6 +446,11 @@ struct kmem_cache {
 #endif
 };
 
+DEFINE_PAGE_STRUCT(slab_page) {
+	struct list_head *cache;
+	struct list_head *slab;
+} END_PAGE_STRUCT(slab_page)
+
 #define CFLGS_OFF_SLAB		(0x80000000UL)
 #define	OFF_SLAB(x)	((x)->flags & CFLGS_OFF_SLAB)
 

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2007-04-03  7:29 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-03-31 19:30 [SLUB 0/2] SLUB: The unqueued slab allocator V6 Christoph Lameter
2007-03-31 19:31 ` [SLUB 1/2] SLUB core Christoph Lameter
2007-04-03  7:29   ` Nick Piggin
2007-03-31 19:31 ` [SLUB 2/2] i386 arch page size slab fixes Christoph Lameter
2007-03-31 19:55   ` Andrew Morton
2007-03-31 20:00     ` Christoph Lameter
2007-04-02 18:30     ` Christoph Lameter
2007-04-02 23:00   ` William Lee Irwin III
2007-04-02 23:02     ` Christoph Lameter
2007-03-31 19:31 ` [SLUB tool] slabinfo: Display slab statistics Christoph Lameter

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).