LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
* DOS by unprivileged user
@ 2018-04-19 19:13 Ferry Toth
  2018-04-20  4:46 ` Mike Galbraith
                   ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Ferry Toth @ 2018-04-19 19:13 UTC (permalink / raw)
  To: linux-kernel

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo 
(SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
+ hyperthreading. But I will be easy enough to get the same effect with more 
RAM, other fs etc.

In this case gitk allocates more and more memory (until my system freezes 
6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files 
(can't investigate as there is no time until it freezes) and the io wait 
goes to 100% on all cores. At this point it is impossible to login from 
remote and local keyboard and mouse are frozen. Hard reset is the only way 
out at this point.

IMHO there is something wrong in how the kernel hands out resources, in this 
case memory, CPU time and disk accesses. It should be easily measurable that 
a single application is allocating all memory and consequently all CPU time 
is spent in io wait. I'm pretty sure that if the kernel would stop or reduce 
allocating CPU time to the hogging (user) process causing excessive io wait, 
the machine would continue just fine. And probably a single process should 
not even be allowed to allocate this amount of memory. Or at least should 
not force other applications to get swapped out.

In effect gitk should have beeb forced to run slow or be killed off (which 
doesn't happen because probably the kernel itself doesn't receive enough CPU 
time).

As a positive side effect maybe someone would dig into gitk and make it less 
memory hungry (qgit is half as hungry, still a lot).

-- 
Ferry Toth

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-19 19:13 DOS by unprivileged user Ferry Toth
@ 2018-04-20  4:46 ` Mike Galbraith
  2018-04-20  8:39   ` Ferry Toth
  2018-04-22 10:16 ` Pavel Machek
       [not found] ` <4285098.DEWjdbWF2X@delfion>
  2 siblings, 1 reply; 14+ messages in thread
From: Mike Galbraith @ 2018-04-20  4:46 UTC (permalink / raw)
  To: Ferry Toth; +Cc: lkml

On Thu, 2018-04-19 at 21:13 +0200, Ferry Toth wrote:
> It appears any ordinary user can easily create a DOS on linux.
> 
> One sure way to reproduce this is to open gitk on the linux kernel repo 
> (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> + hyperthreading. But I will be easy enough to get the same effect with more 
> RAM, other fs etc.
> 
> In this case gitk allocates more and more memory (until my system freezes 
> 6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files 
> (can't investigate as there is no time until it freezes) and the io wait 
> goes to 100% on all cores. At this point it is impossible to login from 
> remote and local keyboard and mouse are frozen. Hard reset is the only way 
> out at this point.

datapoint: my i4790/ext4 box running master.yesterday booted mem=8G
became highly unpleasant to use, but I retained control, and the all
cores going to 100% thing did not happen at any time.

I didn't try constraining on the gitk user, just turned it loose a few
times to see if it managed to render box effectively dead.  It failed
to kill my box, but (expectedly) did make it suck rocks.

	-Mike

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-20  4:46 ` Mike Galbraith
@ 2018-04-20  8:39   ` Ferry Toth
  2018-04-20 12:37     ` Mike Galbraith
  0 siblings, 1 reply; 14+ messages in thread
From: Ferry Toth @ 2018-04-20  8:39 UTC (permalink / raw)
  To: Mike Galbraith; +Cc: lkml

Op vrijdag 20 april 2018 06:46:58 CEST schreef Mike Galbraith:
> On Thu, 2018-04-19 at 21:13 +0200, Ferry Toth wrote:
> > It appears any ordinary user can easily create a DOS on linux.
> > 
> > One sure way to reproduce this is to open gitk on the linux kernel repo 
> > (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> > + hyperthreading. But I will be easy enough to get the same effect with more 
> > RAM, other fs etc.
> > 
> > In this case gitk allocates more and more memory (until my system freezes 
> > 6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files 
> > (can't investigate as there is no time until it freezes) and the io wait 
> > goes to 100% on all cores. At this point it is impossible to login from 
> > remote and local keyboard and mouse are frozen. Hard reset is the only way 
> > out at this point.
> 
> datapoint: my i4790/ext4 box running master.yesterday booted mem=8G
> became highly unpleasant to use, but I retained control, and the all
> cores going to 100% thing did not happen at any time.
> 
> I didn't try constraining on the gitk user, just turned it loose a few
> times to see if it managed to render box effectively dead.  It failed
> to kill my box, but (expectedly) did make it suck rocks.
> 
> 	-Mike
> 

Yes, might be less dramatic with ext4 than with btrfs (COW icw fsync on hdd'\0s destroys performance 
of things like virtualbox, databases, dpkg).

Nevertheless I feel one process should not be allowed to harm other processes by denying them resources. Even if when btrfs makes it easy abuse I think the scheduler should have throttled gitk.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-20  8:39   ` Ferry Toth
@ 2018-04-20 12:37     ` Mike Galbraith
  0 siblings, 0 replies; 14+ messages in thread
From: Mike Galbraith @ 2018-04-20 12:37 UTC (permalink / raw)
  To: Ferry Toth; +Cc: lkml

On Fri, 2018-04-20 at 10:39 +0200, Ferry Toth wrote:
> 
> Nevertheless I feel one process should not be allowed to harm other
> processes by denying them resources. Even if when btrfs makes it easy
> abuse I think the scheduler should have throttled gitk.

Memory management is not in it's job description.

	-Mike

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-19 19:13 DOS by unprivileged user Ferry Toth
  2018-04-20  4:46 ` Mike Galbraith
@ 2018-04-22 10:16 ` Pavel Machek
  2018-04-22 17:43   ` vcaputo
       [not found] ` <4285098.DEWjdbWF2X@delfion>
  2 siblings, 1 reply; 14+ messages in thread
From: Pavel Machek @ 2018-04-22 10:16 UTC (permalink / raw)
  To: Ferry Toth; +Cc: linux-kernel

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Thu 2018-04-19 21:13:35, Ferry Toth wrote:
> It appears any ordinary user can easily create a DOS on linux.
> 
> One sure way to reproduce this is to open gitk on the linux kernel repo 
> (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> + hyperthreading. But I will be easy enough to get the same effect with more 
> RAM, other fs etc.

You may want to disable swap.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-22 10:16 ` Pavel Machek
@ 2018-04-22 17:43   ` vcaputo
  2018-04-23  0:27     ` Michal Hocko
  0 siblings, 1 reply; 14+ messages in thread
From: vcaputo @ 2018-04-22 17:43 UTC (permalink / raw)
  To: Pavel Machek; +Cc: Ferry Toth, linux-kernel

On Sun, Apr 22, 2018 at 12:16:54PM +0200, Pavel Machek wrote:
> On Thu 2018-04-19 21:13:35, Ferry Toth wrote:
> > It appears any ordinary user can easily create a DOS on linux.
> > 
> > One sure way to reproduce this is to open gitk on the linux kernel repo 
> > (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> > + hyperthreading. But I will be easy enough to get the same effect with more 
> > RAM, other fs etc.
> 
> You may want to disable swap.
> 

I run without swap on my laptops, and still observe long periods of
thrashing on the road towards OOM.  What seems to occur is the active
file-backed mappings of executables/libraries become a sort of swap
area, repeatedly being discarded and faulted back in as the context
switches occur.

If there's any good way to prevent this, I'd like to know.

Regards,
Vito Caputo

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-22 17:43   ` vcaputo
@ 2018-04-23  0:27     ` Michal Hocko
  2018-04-23  7:13       ` Pavel Machek
  0 siblings, 1 reply; 14+ messages in thread
From: Michal Hocko @ 2018-04-23  0:27 UTC (permalink / raw)
  To: vcaputo; +Cc: Pavel Machek, Ferry Toth, linux-kernel

On Sun 22-04-18 10:43:00, vcaputo@pengaru.com wrote:
> On Sun, Apr 22, 2018 at 12:16:54PM +0200, Pavel Machek wrote:
> > On Thu 2018-04-19 21:13:35, Ferry Toth wrote:
> > > It appears any ordinary user can easily create a DOS on linux.
> > > 
> > > One sure way to reproduce this is to open gitk on the linux kernel repo 
> > > (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> > > + hyperthreading. But I will be easy enough to get the same effect with more 
> > > RAM, other fs etc.
> > 
> > You may want to disable swap.
> > 
> 
> I run without swap on my laptops, and still observe long periods of
> thrashing on the road towards OOM.  What seems to occur is the active
> file-backed mappings of executables/libraries become a sort of swap
> area, repeatedly being discarded and faulted back in as the context
> switches occur.
> 
> If there's any good way to prevent this, I'd like to know.

I am afraid there is none yet. Johannes had some ground work for
page cache trashing detection https://marc.info/?i=20170727153010.23347-1-hannes%40cmpxchg.org
but there was no version of the patchseries for quite some time and
there was no integration into the oom detection which would be
non-trivial as well.

I realize this sucks. But the reality is that this is far from trivial
to resolve without introducing pre-mature OOM killer invocations.
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-23  0:27     ` Michal Hocko
@ 2018-04-23  7:13       ` Pavel Machek
  0 siblings, 0 replies; 14+ messages in thread
From: Pavel Machek @ 2018-04-23  7:13 UTC (permalink / raw)
  To: Michal Hocko; +Cc: vcaputo, Ferry Toth, linux-kernel

[-- Attachment #1: Type: text/plain, Size: 1980 bytes --]

On Sun 2018-04-22 18:27:38, Michal Hocko wrote:
> On Sun 22-04-18 10:43:00, vcaputo@pengaru.com wrote:
> > On Sun, Apr 22, 2018 at 12:16:54PM +0200, Pavel Machek wrote:
> > > On Thu 2018-04-19 21:13:35, Ferry Toth wrote:
> > > > It appears any ordinary user can easily create a DOS on linux.
> > > > 
> > > > One sure way to reproduce this is to open gitk on the linux kernel repo 
> > > > (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core 
> > > > + hyperthreading. But I will be easy enough to get the same effect with more 
> > > > RAM, other fs etc.
> > > 
> > > You may want to disable swap.
> > > 
> > 
> > I run without swap on my laptops, and still observe long periods of
> > thrashing on the road towards OOM.  What seems to occur is the active
> > file-backed mappings of executables/libraries become a sort of swap
> > area, repeatedly being discarded and faulted back in as the context
> > switches occur.
> > 
> > If there's any good way to prevent this, I'd like to know.
> 
> I am afraid there is none yet. Johannes had some ground work for
> page cache trashing detection https://marc.info/?i=20170727153010.23347-1-hannes%40cmpxchg.org
> but there was no version of the patchseries for quite some time and
> there was no integration into the oom detection which would be
> non-trivial as well.
> 
> I realize this sucks. But the reality is that this is far from trivial
> to resolve without introducing pre-mature OOM killer invocations.

Another problem is that what "unusable machine" in X/web browser
situation may be normal load for build server...

I guess one way would be "hey, this is my X server; if it is waiting
for disk for more than 10 seconds, you probably want to OOM kill
someone. Ouch and same goes for my window manager". 

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
       [not found]     ` <6057755.ozdVOybsI6@delfion>
@ 2018-04-23  8:04       ` Mike Galbraith
  2018-04-25 14:54         ` Alan Cox
  0 siblings, 1 reply; 14+ messages in thread
From: Mike Galbraith @ 2018-04-23  8:04 UTC (permalink / raw)
  To: Ferry Toth; +Cc: linux-kernel

On Sun, 2018-04-22 at 21:37 +0200, Ferry Toth wrote:
> > Yes your memory hog scenario thoroughly wrecks the user experience, but
> > the process scheduler in not the source of that wreckage, it's a memory
> > management issue.  With no constraints in place, anybody can just keep
> > on allocating until the entire system starts grinding itself to dust.
> > 
> > 	-Mike
>  
> That is exactly the issue I think. It is not just a user experience,
> they is no distinction between crashing the kernel and grinding it to
> dust. The effect we have is: any user on a multi user system can
> crash the system.
>  
> Memory management / constraints can't solve the problem either: it
> might be intentional to alloc such amounts of memory.

Constraints definitely can solve this particular problem instance. 
Plain old ulimit can stop the memory hog (wish) dead in its tracks, or
you can use memory cgroup controller to constrain it in a more friendly
manner.  I started gitk in a 4G constrained cgroup, which redirected
its greedy fingers to the swap bin for the remainder of its needs.

But yes, there is currently no wonderful one size fits all fully
automatic solution to running low on memory that doesn't involve
running to the store.

> I think memory allocation and io waits can't be decoupled from
> scheduling as they are now.

The scheduler is not decoupled from either, it is intimately involved
in both.  However, none of the decision making smarts for either reside
in the scheduler, nor should they.

	-Mike

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-23  8:04       ` Mike Galbraith
@ 2018-04-25 14:54         ` Alan Cox
  2018-04-25 16:21           ` Mike Galbraith
                             ` (2 more replies)
  0 siblings, 3 replies; 14+ messages in thread
From: Alan Cox @ 2018-04-25 14:54 UTC (permalink / raw)
  To: Mike Galbraith; +Cc: Ferry Toth, linux-kernel

> > I think memory allocation and io waits can't be decoupled from
> > scheduling as they are now.  
> 
> The scheduler is not decoupled from either, it is intimately involved
> in both.  However, none of the decision making smarts for either reside
> in the scheduler, nor should they.

It belongs in both.

Classical Unix systems never had this problem because they respond to
thrashing by ensuring that all processes consumed CPU and made some
progress. Linux handles it by thrashing itself to dealth while BSD always
handled it by moving from paging more towards swapping and behaving like
a swap bound batch machine.

Linux thrashes itself to death, the classic BSD algorithn instead throws
fairness out of the window under extreme load to prevent it. It might take
a few seconds but at least you will get your prompt back.

Alan

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-25 14:54         ` Alan Cox
@ 2018-04-25 16:21           ` Mike Galbraith
  2018-04-25 16:50           ` Mike Galbraith
  2018-04-30 10:00           ` Ferry Toth
  2 siblings, 0 replies; 14+ messages in thread
From: Mike Galbraith @ 2018-04-25 16:21 UTC (permalink / raw)
  To: Alan Cox; +Cc: Ferry Toth, linux-kernel

On Wed, 2018-04-25 at 15:54 +0100, Alan Cox wrote:
> > > I think memory allocation and io waits can't be decoupled from
> > > scheduling as they are now.  
> > 
> > The scheduler is not decoupled from either, it is intimately involved
> > in both.  However, none of the decision making smarts for either reside
> > in the scheduler, nor should they.
> 
> It belongs in both.

If mm decision making belongs within the process scheduler, it follows
that IO requests, dirty page writeback etc. do as well.  Nope, I don't
think we want to create a squid-uler, with tentacles extending all over
the dang kernel.

The thrashing problem could use some attention, but we'll have to agree
to disagree about the scheduler growing mm, io (etc) smarts.

	-Mike

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-25 14:54         ` Alan Cox
  2018-04-25 16:21           ` Mike Galbraith
@ 2018-04-25 16:50           ` Mike Galbraith
  2018-04-30 10:00           ` Ferry Toth
  2 siblings, 0 replies; 14+ messages in thread
From: Mike Galbraith @ 2018-04-25 16:50 UTC (permalink / raw)
  To: Alan Cox; +Cc: Ferry Toth, linux-kernel

On Wed, 2018-04-25 at 15:54 +0100, Alan Cox wrote:
> 
> Classical Unix systems never had this problem because they respond to
> thrashing by ensuring that all processes consumed CPU and made some
> progress. Linux handles it by thrashing itself to dealth while BSD always
> handled it by moving from paging more towards swapping and behaving like
> a swap bound batch machine.

Memcg constrained the gitk hog nicely, forcing it do dig into swap
after hitting its limit.  Dunno if there are any userspace bits that
can use it wisely though.

	-Mike

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-25 14:54         ` Alan Cox
  2018-04-25 16:21           ` Mike Galbraith
  2018-04-25 16:50           ` Mike Galbraith
@ 2018-04-30 10:00           ` Ferry Toth
  2018-04-30 10:35             ` Miguel Ojeda
  2 siblings, 1 reply; 14+ messages in thread
From: Ferry Toth @ 2018-04-30 10:00 UTC (permalink / raw)
  To: Alan Cox; +Cc: Mike Galbraith, linux-kernel

Op woensdag 25 april 2018 16:54:59 CEST schreef Alan Cox:
> > > I think memory allocation and io waits can't be decoupled from
> > > scheduling as they are now.  
> > 
> > The scheduler is not decoupled from either, it is intimately involved
> > in both.  However, none of the decision making smarts for either reside
> > in the scheduler, nor should they.
> 
> It belongs in both.
> 
> Classical Unix systems never had this problem because they respond to
> thrashing by ensuring that all processes consumed CPU and made some
> progress. Linux handles it by thrashing itself to dealth while BSD always
> handled it by moving from paging more towards swapping and behaving like
> a swap bound batch machine.
> 
> Linux thrashes itself to death, the classic BSD algorithn instead throws
> fairness out of the window under extreme load to prevent it. It might take
> a few seconds but at least you will get your prompt back.
> 
> Alan
> 
I haven\0t tried BSD.

But when I was young I allocated 10MB on a HP9000 (UX) with 1MB of RAM. People wanted to launch me out of the window (18th floor).

I did not want to say Unix was better, only with so much emphasis on security I'\0m surprised how easy it is for a regular user to bring linux to on it\0s knees.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: DOS by unprivileged user
  2018-04-30 10:00           ` Ferry Toth
@ 2018-04-30 10:35             ` Miguel Ojeda
  0 siblings, 0 replies; 14+ messages in thread
From: Miguel Ojeda @ 2018-04-30 10:35 UTC (permalink / raw)
  To: Ferry Toth; +Cc: Alan Cox, Mike Galbraith, linux-kernel

On Mon, Apr 30, 2018 at 12:00 PM, Ferry Toth <ftoth@telfort.nl> wrote:
> Op woensdag 25 april 2018 16:54:59 CEST schreef Alan Cox:
>> > > I think memory allocation and io waits can't be decoupled from
>> > > scheduling as they are now.
>> >
>> > The scheduler is not decoupled from either, it is intimately involved
>> > in both.  However, none of the decision making smarts for either reside
>> > in the scheduler, nor should they.
>>
>> It belongs in both.
>>
>> Classical Unix systems never had this problem because they respond to
>> thrashing by ensuring that all processes consumed CPU and made some
>> progress. Linux handles it by thrashing itself to dealth while BSD always
>> handled it by moving from paging more towards swapping and behaving like
>> a swap bound batch machine.
>>
>> Linux thrashes itself to death, the classic BSD algorithn instead throws
>> fairness out of the window under extreme load to prevent it. It might take
>> a few seconds but at least you will get your prompt back.
>>
>> Alan
>>
> I haven t tried BSD.
>
> But when I was young I allocated 10MB on a HP9000 (UX) with 1MB of RAM. People wanted to launch me out of the window (18th floor).
>
> I did not want to say Unix was better, only with so much emphasis on security I' m surprised how easy it is for a regular user to bring linux to on it s knees.

While it is true that things can be improved/tweaked for typical
desktop/single user usage; this isn't really a security issue. For
shared systems, there are a few ways to soft/hard limit resources:
nice, *limit, cgroups, systemd limits, containers/VMs...

Cheers,
Miguel

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2018-04-30 10:35 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-04-19 19:13 DOS by unprivileged user Ferry Toth
2018-04-20  4:46 ` Mike Galbraith
2018-04-20  8:39   ` Ferry Toth
2018-04-20 12:37     ` Mike Galbraith
2018-04-22 10:16 ` Pavel Machek
2018-04-22 17:43   ` vcaputo
2018-04-23  0:27     ` Michal Hocko
2018-04-23  7:13       ` Pavel Machek
     [not found] ` <4285098.DEWjdbWF2X@delfion>
     [not found]   ` <1524325275.8078.2.camel@gmx.de>
     [not found]     ` <6057755.ozdVOybsI6@delfion>
2018-04-23  8:04       ` Mike Galbraith
2018-04-25 14:54         ` Alan Cox
2018-04-25 16:21           ` Mike Galbraith
2018-04-25 16:50           ` Mike Galbraith
2018-04-30 10:00           ` Ferry Toth
2018-04-30 10:35             ` Miguel Ojeda

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).