LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Paul Jackson <pj@sgi.com>
To: balbir@linux.vnet.ibm.com
Cc: menage@google.com, vatsa@in.ibm.com,
	ckrm-tech@lists.sourceforge.net, balbir@in.ibm.com,
	haveblue@us.ibm.com, xemul@sw.ru, dev@sw.ru,
	containers@lists.osdl.org, devel@openvz.org,
	ebiederm@xmission.com, mbligh@google.com, rohitseth@google.com,
	serue@us.ibm.com, akpm@linux-foundation.org,
	svaidy@linux.vnet.ibm.com, linux-kernel@vger.kernel.org
Subject: Re: [ckrm-tech] [PATCH 1/9] Containers (V9): Basic container framework
Date: Tue, 1 May 2007 23:12:54 -0700	[thread overview]
Message-ID: <20070501231254.4267777e.pj@sgi.com> (raw)
In-Reply-To: <46380901.2050001@linux.vnet.ibm.com>

[-- Attachment #1: Type: text/plain, Size: 3153 bytes --]

Balbir wrote:
> Would it be possible to extract those test cases and integrate them
> with a testing framework like LTP? Do you have any regression test
> suite for cpusets that can be made available publicly so that
> any changes to cpusets can be validated?

There are essentially two sorts of cpuset regression tests of interest.

I have one such test, and the batch scheduler developers have various
tests of their batch schedulers.

1) Testing batch schedulers against cpusets:

    I doubt that the batch scheduler developers would be able to
    extract a cpuset test from their tests, or be able to share it if
    they did.  Their tests tend to be large tests of batch schedulers,
    and only incidentally test cpusets -- if we break cpusets,
    in sometimes even subtle ways that they happen to depend on,
    we break them.

    Sometimes there is no way to guess exactly what sorts of changes
    will break their code; we'll just have to schedule at least one
    run through one or more of them that rely heavily on cpusets
    before a change as big as rebasing cpusets on containers is
    reasonably safe.  This test cycle won't be all that easy, so I'd
    wait until we are pretty close to what we think should be taken
    into the mainline kernel.

    I suppose I will have to be the one co-ordinating this test,
    as I am the only one I know with a presence in both camps.

    Once this test is done, from then forward, if we break them,
    we'll just have to deal with it as we do now, when the breakage
    shows up well down stream from the main kernel tree, at the point
    that a major batch scheduler release runs into a major distribution
    release containing the breakage.  There is no practical way that I
    can see, as an ongoing basis, to continue testing for such breakage
    with every minor change to cpuset related code in the kernel.  Any
    breakage found this way is dealt with by changes in user level code.

    Once again, I have bcc'd one or more developers of batch schedulers,
    so they can see what nonsense I am spouting about them now ;).

2) Testing cpusets with a specific test.

    There I can do better.  Attached is the cpuset regression test I
    use.  It requires at least 4 cpus and 2 memory nodes to do anything
    useful.  It is copyright by SGI, released under GPL license.

    This regression test is the primary cpuset test upon which I
    relied during the development of cpusets, and continue to rely.
    Except for one subtle race condition in the test itself, it has
    not changed in the last two to three years.

    This test requires no user level code not found in an ordinary
    distro.  It does require the taskset and numactl commands,
    for the purposes of testing certain interactions with them.
    It assumes that there are not other cpusets currently setup in
    the system that happen to conflict with the ones it creates.

    See further comments within the test script itself.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.925.600.0401

[-- Attachment #2: cpuset_test --]
[-- Type: application/octet-stream, Size: 8168 bytes --]

#!/bin/bash -e
# 
# cpuset_test - regression test for Linux kernel cpuset support.
#
# Copyright (c) 2005 Silicon Graphic, Inc.  All rights reserved.
#
###
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#
###
#
# This is a regression test for cpusets.  It tests the various
# functions, displays successes and failures to stdout and stderr.
# 
# If successful, it exits with a value of 0, and the last line
# of output is:
# 
# 	cpuset test success !!!
# 
# If failed, it exits with a value of 1, and the last line of
# output is:
# 
# 	cpuset test failed ;););)
# 
# It needs to be run with root permissions, and it requires
# being run on a system with at least 4 CPUs to actually test
# anything.
#
# If run on a system without cpuset support in the kernel, it
# will fail with an exit status of 1, and output of:
# 
# 	mount: fs type cpuset not supported by kernel
# 	Failed: mount -t cpuset cpuset /dev/cpuset at LINE 46
# 	cpuset test failed ;););)
# 
# If run on a system with 3 or fewer CPUs, it does nothing much,
# successfully, and exits with a value of 0.
# 
# All success and fail lines for specific tests provide the
# LINE number in this shell script of the test command, for
# convenient analysis.
# 
# Paul Jackson
# pj@sgi.com
# 26 Jan 2005


exitval=1
trap '
    test $exitval -eq 0 && r="success !!!" || r="failed ;););)"
    echo cpuset test $r
    trap 0
    exit $exitval
' 0 1 2 3 15

fail()
{
	echo Failed: $*
	exitval=1
	exit
}

succeed()
{
	echo Succeeded: $*
	exitval=0
	exit
}

verbose()
{
	line=$1
	shift
	$* || fail $* at LINE $line
	echo $* at LINE $line ok
}

do_echo()
{
	/bin/echo $1 > $2 &&
		echo echo $1 '>' $2 at LINE $3 ok ||
		fail echo $1 '>' $2 at LINE $3
}

test -d /dev/cpuset &&
	echo directory /dev/cpuset exists at LINE $LINENO ||
	verbose $LINENO mkdir /dev/cpuset

mount | grep '/dev/cpuset type cpuset' >/dev/null &&
	echo /dev/cpuset mounted at LINE $LINENO ||
	verbose $LINENO mount -t cpuset cpuset /dev/cpuset

# If run on a system with < 4 CPUs, do nothing, successfully.
test $(grep -c processor /proc/cpuinfo) -ge 4 ||
	succeed cpuset testing skipped for lack of at least 4 CPUs

# Do all testing in our own cpuset subtree 'cpuset_test_tree'
test -d /dev/cpuset/cpuset_test_tree &&
	echo cpuset cpuset_test_tree exists at LINE $LINENO ||
	verbose $LINENO mkdir /dev/cpuset/cpuset_test_tree

verbose $LINENO cd /dev/cpuset/cpuset_test_tree

rmdir ? 2>&- || true

do_echo 0-3 cpus $LINENO
do_echo 0-1 mems $LINENO
do_echo 0 cpu_exclusive $LINENO
do_echo 0 mem_exclusive $LINENO
do_echo $$ tasks $LINENO

test $(</proc/self/cpuset) = /cpuset_test_tree ||
	fail bind self to /cpuset_test_tree at LINE $LINENO
echo bind self to /cpuset_test_tree at LINE $LINENO ok

# Set up 4 child cpusets, named 0, 1, 2, and 3, containing
# CPUs 0, 1, 2, and 3 respectively, on Memory Nodes 0, 0, 1,
# and 1.  Bind this task to each cpuset, in order.
for i in 0 1 2 3
do
    test -d $i && rmdir $i
    mkdir $i
    do_echo $i $i/cpus $LINENO
    do_echo $(expr $i / 2) $i/mems $LINENO
    do_echo $$ $i/tasks $LINENO
    j=$(awk < /proc/self/stat '{print $39}')
    test $i -eq $j ||
	fail bind to CPU $i failed - on CPU $j instead LINE $LINENO
    echo bind to CPU $i at LINE $LINENO ok
done

# Bind this task to the parent /cpuset_test_tree cpuset.
do_echo $$ tasks $LINENO

test $(</proc/self/cpuset) = /cpuset_test_tree ||
        fail bind self to /cpuset_test_tree

# Try to make child cpuset 0 exclusive for CPUs.
# This should fail since its parent isn't exclusive.
/bin/echo 1 > 0/cpu_exclusive 2>&- &&
	fail nested cpu_exclusive at LINE $LINENO
echo nested cpu_exclusive at LINE $LINENO ok

# Now make the parent cpuset exclusive.
do_echo 1 cpu_exclusive $LINENO

# Now making the child cpuset 0 exclusive should work.
/bin/echo 1 > 0/cpu_exclusive 2>&- ||
	fail nested cpu_exclusive at LINE $LINENO
echo nested cpu_exclusive at LINE $LINENO ok

# As above, for Memory exclusive instead of CPU exclusive.
/bin/echo 1 > 0/mem_exclusive 2>&- &&
	fail nested mem_exclusive at LINE $LINENO
echo nested mem_exclusive at LINE $LINENO ok

do_echo 1 mem_exclusive $LINENO

# Even with parent mem_exclusive set, cpuset 0
# should refuse to set mem_exclusive, since it
# shares the same mems as child cpuset 1.
/bin/echo 1 > 0/mem_exclusive 2>&- &&
	fail sibling mem_exclusive at LINE $LINENO
echo sibling mem_exclusive at LINE $LINENO ok

# Remove child cpuset "1".
verbose $LINENO rmdir 1

# Now finally should be able to make cpuset 0 mems_exclusive.
/bin/echo 1 > 0/mem_exclusive 2>&- ||
	fail sibling mem_exclusive at LINE $LINENO
echo sibling mem_exclusive at LINE $LINENO ok

# Now lets chdir to 0, and work in that cpuset a while.
verbose $LINENO cd /dev/cpuset/cpuset_test_tree/0

# Attach current task to cpuset 0, verifying attachment.
do_echo $$ tasks $LINENO

j=$(awk < /proc/self/stat '{print $39}')
test 0 -eq $j ||
    fail bind to CPU 0 failed - on CPU $j instead LINE $LINENO
echo bind to CPU 0 at LINE $LINENO ok

# Should be unable to change cpus to 0-2, because we're
# still cpu_exclusive (from above) and 0-2 would overlap
# our sibling cpuset ../1.
/bin/echo 0-2 > cpus 2>&- &&
	fail exclusive cpus growth at LINE $LINENO
echo exclusive cpus growth at LINE $LINENO ok

# Remove our cpu_exclusive.
do_echo 0 cpu_exclusive $LINENO

# Now we should be able to change CPUs to 0-2.
/bin/echo 0-2 > cpus 2>&- ||
	fail exclusive cpus growth at LINE $LINENO
echo exclusive cpus growth at LINE $LINENO ok

# And we should be able to use taskset (sched_setaffinity)
# to run a task on each of the CPUs in our cpuset, 0-2.
for i in 0 1 2
do
    verbose $LINENO taskset -c $i /bin/true
done

# But we should be unable to taskset a task onto CPU 3.
taskset -c 3 echo testing taskset on CPU 3 2>&- &&
	fail testing taskset on invalid CPU 3 at LINE $LINENO
echo testing taskset on invalid CPU 3 at LINE $LINENO ok

# Here lies information on which CPUs are on which nodes.
sys=/sys/devices/system/node

# Test numactl (set_mempolicy) on each of the first 4 nodes,
# verifying we can only use numactl to run on a Memory Node
# or CPU if it is in our current cpuset.
for node in 0 1 2 3
do

    # Skip non-existant nodes
    test -d $sys/node$node || continue

    # Only nodes that have at least one of cpus 0, 1 or 2
    # (cpus in our current cpuset) can numactl --cpubind.
    shouldwork=false
    for cpu in 0 1 2
    do
	    test -L $sys/node$node/cpu$cpu && shouldwork=true
    done

    if $shouldwork
    then
	    verbose $LINENO numactl --cpubind=$node /bin/true
    else
	    numactl --cpubind=$node /bin/true 2>&- &&
		    fail test numactl on cpubind $node at LINE $LINENO
	    echo test numactl on invalid cpubind $node at LINE $LINENO ok
    fi

    # Only node 0 (the mems in our current cpuset) can numactl --membind
    if test $node -eq 0
    then
	    verbose $LINENO numactl --membind=$node /bin/true
    else
	    numactl --membind=$node /bin/true 2>&- &&
		    fail test numactl on membind $node at LINE $LINENO
	    echo test numactl on membind $node at LINE $LINENO ok
    fi
done

# Back to top of our cpuset test tree
verbose $LINENO cd /dev/cpuset/cpuset_test_tree

do_echo $$ ../tasks $LINENO
#  Mark all test cpusets to be removed on release.
echo 1 | tee $(find . -name notify_on_release) >/dev/null

verbose $LINENO cd /dev/cpuset
rmdir cpuset_test_tree/? ||
	fail rmdir sub cpusets at LINE $LINENO
sleep 3		# time for /sbin/cpuset_release_agent to rmdir
test -d cpuset_test_tree &&
	fail notify_on_release at LINE $LINENO

exitval=0

  reply	other threads:[~2007-05-02  6:13 UTC|newest]

Thread overview: 36+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2007-04-27 10:46 [PATCH 0/9] Containers (V9): Generic Process Containers menage
2007-04-27 10:46 ` [PATCH 1/9] Containers (V9): Basic container framework menage
2007-04-29  3:12   ` [ckrm-tech] " Paul Jackson
2007-05-01 17:40   ` Balbir Singh
2007-05-01 17:46     ` Paul Menage
2007-05-01 18:40     ` [ckrm-tech] " Paul Jackson
2007-05-02  3:44       ` Balbir Singh
2007-05-02  6:12         ` Paul Jackson [this message]
2007-05-10  4:09           ` Balbir Singh
2007-05-10  4:47             ` Paul Jackson
2007-05-10  4:49               ` Balbir Singh
2007-04-27 10:46 ` [PATCH 2/9] Containers (V9): Example CPU accounting subsystem menage
2007-05-01 17:52   ` Balbir Singh
2007-04-27 10:46 ` [PATCH 3/9] Containers (V9): Add tasks file interface menage
2007-05-01 18:12   ` Balbir Singh
2007-05-01 20:37     ` [ckrm-tech] " Paul Menage
2007-05-02  3:25       ` Srivatsa Vaddagiri
2007-05-02  3:25         ` Paul Menage
2007-05-02  3:46           ` Srivatsa Vaddagiri
2007-05-08 14:51           ` Balbir Singh
2007-05-10 21:21             ` Paul Menage
2007-05-11  2:31               ` Balbir Singh
2007-05-02  3:58       ` Balbir Singh
2007-04-27 10:46 ` [PATCH 4/9] Containers (V9): Add fork/exit hooks menage
2007-04-27 10:46 ` [PATCH 5/9] Containers (V9): Add container_clone() interface menage
2007-04-27 10:46 ` [PATCH 6/9] Containers (V9): Add procfs interface menage
2007-04-27 10:46 ` [PATCH 7/9] Containers (V9): Make cpusets a client of containers menage
2007-04-27 10:46 ` [PATCH 8/9] Containers (V9): Share css_group arrays between tasks with same container memberships menage
2007-04-27 10:46 ` [PATCH 9/9] Containers (V9): Simple debug info subsystem menage
2007-04-29  1:47 ` [PATCH 0/9] Containers (V9): Generic Process Containers Paul Jackson
2007-04-29  9:37 ` Paul Jackson
2007-04-30 17:12   ` Srivatsa Vaddagiri
2007-04-30 17:09     ` Paul Menage
2007-04-30 18:06       ` Srivatsa Vaddagiri
2007-04-30 17:23     ` Christoph Hellwig
2007-04-30 18:16       ` [ckrm-tech] " Srivatsa Vaddagiri

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20070501231254.4267777e.pj@sgi.com \
    --to=pj@sgi.com \
    --cc=akpm@linux-foundation.org \
    --cc=balbir@in.ibm.com \
    --cc=balbir@linux.vnet.ibm.com \
    --cc=ckrm-tech@lists.sourceforge.net \
    --cc=containers@lists.osdl.org \
    --cc=dev@sw.ru \
    --cc=devel@openvz.org \
    --cc=ebiederm@xmission.com \
    --cc=haveblue@us.ibm.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mbligh@google.com \
    --cc=menage@google.com \
    --cc=rohitseth@google.com \
    --cc=serue@us.ibm.com \
    --cc=svaidy@linux.vnet.ibm.com \
    --cc=vatsa@in.ibm.com \
    --cc=xemul@sw.ru \
    --subject='Re: [ckrm-tech] [PATCH 1/9] Containers (V9): Basic container framework' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).