LKML Archive on lore.kernel.org
help / color / mirror / Atom feed
From: Chao Yu <yuchao0@huawei.com>
To: =?UTF-8?Q?megi@xff.cz_
Cc: Jaegeuk Kim <jaegeuk@kernel.org>, <linux-kernel@vger.kernel.org>,
	<linux-f2fs-devel@lists.sourceforge.net>
Subject: Re: [f2fs-dev] Writes stoped working on f2fs after the compression support was added
Date: Tue, 25 Feb 2020 19:32:38 +0800	[thread overview]
Message-ID: <a47c85b1-8a0a-a1d4-453b-73562f0aca57@huawei.com> (raw)
In-Reply-To: <39712bf4-210b-d7b6-cbb1-eb57585d991a@huawei.com>

On 2020/2/25 19:24, Chao Yu wrote:
> On 2020/2/24 22:31, Ondřej Jirman wrote:
>> On Mon, Feb 24, 2020 at 03:03:49PM +0100, megi xff wrote:
>>> On Mon, Feb 24, 2020 at 02:58:37PM +0100, megi xff wrote:
>>>> Hello,
>>>>
>>>> On Mon, Feb 24, 2020 at 06:41:03PM +0800, Chao Yu wrote:
>>>>> On 2020/2/24 18:37, Chao Yu wrote:
>>>>>> Hi,
>>>>>>
>>>>>> Thanks for the report.
>>>>>>
>>>>>> Could you dump all other task stack info via "echo "t" > /proc/sysrq-trigger"?
>>>>>>
>>>>>>>
>>>>>>> [  246.758021] INFO: task kworker/u16:1:58 blocked for more than 122 seconds.
>>>>>>> [  246.758040]       Not tainted 5.6.0-rc2-00590-g9983bdae4974e #11
>>>>>>> [  246.758044] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>>>> [  246.758052] kworker/u16:1   D    0    58      2 0x00000000
>>>>>>> [  246.758090] Workqueue: writeback wb_workfn (flush-179:0)
>>>>>>> [  246.758099] Backtrace:
>>>>>>> [  246.758121] [<c0912b90>] (__schedule) from [<c0913234>] (schedule+0x78/0xf4)
>>>>>>> [  246.758130]  r10:da644000 r9:00000000 r8:da645a60 r7:da283e10 r6:00000002 r5:da644000
>>>>>>> [  246.758132]  r4:da4d3600
>>>>>>> [  246.758148] [<c09131bc>] (schedule) from [<c017ec74>] (rwsem_down_write_slowpath+0x24c/0x4c0)
>>>>>>> [  246.758152]  r5:00000001 r4:da283e00
>>>>>>> [  246.758161] [<c017ea28>] (rwsem_down_write_slowpath) from [<c0915f2c>] (down_write+0x6c/0x70)
>>>>>>> [  246.758167]  r10:da283e00 r9:da645d80 r8:d9ed0000 r7:00000001 r6:00000000 r5:eff213b0
>>>>>>> [  246.758169]  r4:da283e00
>>>>>>> [  246.758187] [<c0915ec0>] (down_write) from [<c0435b80>] (f2fs_write_single_data_page+0x608/0x7ac)
>>>>>>
>>>>>> I'm not sure what is this semaphore, I suspect this is F2FS_I(inode)->i_sem, in order to make
>>>>>> sure of this, can you help to add below function, and use them to replace
>>>>>> all {down,up}_{write,read}(&.i_sem) invoking? then reproduce this issue and catch the log.
>>>>>
>>>>> Sorry, just forgot attaching below function.
>>>>>
>>>>> void inode_down_write(struct inode *inode)
>>>>> {
>>>>> 	printk("%s from %pS\n", __func__, __builtin_return_address(0));
>>>>> 	down_write(&F2FS_I(inode)->i_sem);
>>>>> }
>>>>>
>>>>> void inode_up_write(struct inode *inode)
>>>>> {
>>>>> 	up_write(&F2FS_I(inode)->i_sem);
>>>>> 	printk("%s from %pS\n", __func__, __builtin_return_address(0));
>>>>> }
>>>>>
>>>>> void inode_down_read(struct inode *inode)
>>>>> {
>>>>> 	printk("%s from %pS\n", __func__, __builtin_return_address(0));
>>>>> 	down_read(&F2FS_I(inode)->i_sem);
>>>>> }
>>>>>
>>>>> void inode_up_read(struct inode *inode)
>>>>> {
>>>>> 	up_read(&F2FS_I(inode)->i_sem);
>>>>> 	printk("%s from %pS\n", __func__, __builtin_return_address(0));
>>>>> }
>>>>>
>>>>
>>>> Here's the log and vmlinux file that may help mapping the code addresses back to
>>>> code, hope it helps:
>>>>
>>>> https://megous.com/dl/tmp/f2fs-dmesg-log
>>>> https://megous.com/dl/tmp/f2fs-log-build-artifacts.tar.gz
>>>
>>> Just by a looks of it:
>>>
>>> root@tbs2[/proc/sys/kernel] # dmesg | grep up_write | wc -l
>>> 324
>>> root@tbs2[/proc/sys/kernel] # dmesg | grep down_write | wc -l
>>> 347
>>>
>>> there seems to be a mismatch of lock/unlock counts.
>>  
>> Sorry, a wrong grep expression.
>>
>> root@tbs2[~] # dmesg | grep inode_down_write | wc -l
>> 357
>> root@tbs2[~] # dmesg | grep inode_up_write | wc -l
>> 357
>> root@tbs2[~] # dmesg | grep inode_up_read | wc -l
>> 16
>> root@tbs2[~] # dmesg | grep inode_down_read | wc -l
>> 16
> 
> I don't know why we have consistent down/up pair, but through disassembled
> code, I doubt it's the f2fs_inode->i_sem.
> 
> c0435d7c:       ebf54af8        bl      c0188964 <printk>
> c0435d80:       e1a00006        mov     r0, r6
> c0435d84:       eb138135        bl      c0916260 <down_write>
> 
> inode_down_write()
> 
> c0435d88:       e284ce1d        add     ip, r4, #464    ; 0x1d0
> 
> We are stuck here.
> 
> [  430.675754] [<c0916260>] (down_write) from [<c0435d88>] (f2fs_write_single_data_page+0x600/0x7d8)
>                                                 ^^^^^^^^^
> [  430.675764] [<c0435788>] (f2fs_write_single_data_page) from [<c0436214>] (f2fs_write_cache_pages+0x2b4/0x7c4)
> 
> 
> c0435d8c:       e14b0ad4        ldrd    r0, [fp, #-164] ; 0xffffff5c
> c0435d90:       e1cc20d0        ldrd    r2, [ip]
> c0435d94:       e1520000        cmp     r2, r0
> c0435d98:       e0d33001        sbcs    r3, r3, r1
> c0435d9c:       b1cc00f0        strdlt  r0, [ip]
> c0435da0:       e1a00006        mov     r0, r6
> c0435da4:       ebf52227        bl      c017e648 <up_write>
> c0435da8:       e51b2098        ldr     r2, [fp, #-152] ; 0xffffff68
> c0435dac:       e30c0730        movw    r0, #50992      ; 0xc730
> c0435db0:       e59f11a4        ldr     r1, [pc, #420]  ; c0435f5c <f2fs_write_single_data_page+0x7d4>
> c0435db4:       e34c00b6        movt    r0, #49334      ; 0xc0b6
> c0435db8:       ebf54ae9        bl      c0188964 <printk>
> 
> inode_up_write()

Can we have a try with below diff to avoid call down_write under page lock?

Not sure it can solve this issue.

diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index cb41260ca941..f145c1ea977d 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -2650,11 +2650,6 @@ int f2fs_write_single_data_page(struct page *page, int *submitted,

 	if (err) {
 		file_set_keep_isize(inode);
-	} else {
-		down_write(&F2FS_I(inode)->i_sem);
-		if (F2FS_I(inode)->last_disk_size < psize)
-			F2FS_I(inode)->last_disk_size = psize;
-		up_write(&F2FS_I(inode)->i_sem);
 	}

 done:
@@ -2675,6 +2670,14 @@ int f2fs_write_single_data_page(struct page *page, int *submitted,
 		submitted = NULL;
 	}
 	unlock_page(page);
+
+	if (!err) {
+		down_write(&F2FS_I(inode)->i_sem);
+		if (F2FS_I(inode)->last_disk_size < psize)
+			F2FS_I(inode)->last_disk_size = psize;
+		up_write(&F2FS_I(inode)->i_sem);
+	}
+
 	if (!S_ISDIR(inode->i_mode) && !IS_NOQUOTA(inode) &&
 					!F2FS_I(inode)->cp_task)
 		f2fs_balance_fs(sbi, need_balance_fs);

Thanks,

> 
> Thanks,
> 
>>
>> So it's probably not inode locking.
>>
>>> root@tbs2[/proc/sys/kernel] # dmesg | grep down_read | wc -l
>>> 16
>>> root@tbs2[/proc/sys/kernel] # dmesg | grep up_read | wc -l
>>> 16
>>>
>>> regards,
>>> 	o.
>>>
>>>> thank you,
>>>> 	o.
>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>> [  246.758190]  r5:eff213b0 r4:da283c60
>>>>>>> [  246.758198] [<c0435578>] (f2fs_write_single_data_page) from [<c0435fd8>] (f2fs_write_cache_pages+0x2b4/0x7c4)
>>>>>>> [  246.758204]  r10:da645c28 r9:da283d60 r8:da283c60 r7:0000000f r6:da645d80 r5:00000001
>>>>>>> [  246.758206]  r4:eff213b0
>>>>>>> [  246.758214] [<c0435d24>] (f2fs_write_cache_pages) from [<c043682c>] (f2fs_write_data_pages+0x344/0x35c)
>>>>>>> [  246.758220]  r10:00000000 r9:d9ed002c r8:d9ed0000 r7:00000004 r6:da283d60 r5:da283c60
>>>>>>> [  246.758223]  r4:da645d80
>>>>>>> [  246.758238] [<c04364e8>] (f2fs_write_data_pages) from [<c0267ee8>] (do_writepages+0x3c/0xd4)
>>>>>>> [  246.758244]  r10:0000000a r9:c0e03d00 r8:00000c00 r7:c0264ddc r6:da645d80 r5:da283d60
>>>>>>> [  246.758246]  r4:da283c60
>>>>>>> [  246.758254] [<c0267eac>] (do_writepages) from [<c0310cbc>] (__writeback_single_inode+0x44/0x454)
>>>>>>> [  246.758259]  r7:da283d60 r6:da645eac r5:da645d80 r4:da283c60
>>>>>>> [  246.758266] [<c0310c78>] (__writeback_single_inode) from [<c03112d0>] (writeback_sb_inodes+0x204/0x4b0)
>>>>>>> [  246.758272]  r10:0000000a r9:c0e03d00 r8:da283cc8 r7:da283c60 r6:da645eac r5:da283d08
>>>>>>> [  246.758274]  r4:d9dc9848
>>>>>>> [  246.758281] [<c03110cc>] (writeback_sb_inodes) from [<c03115cc>] (__writeback_inodes_wb+0x50/0xe4)
>>>>>>> [  246.758287]  r10:da3797a8 r9:c0e03d00 r8:d9dc985c r7:da645eac r6:00000000 r5:d9dc9848
>>>>>>> [  246.758289]  r4:da5a8800
>>>>>>> [  246.758296] [<c031157c>] (__writeback_inodes_wb) from [<c03118f4>] (wb_writeback+0x294/0x338)
>>>>>>> [  246.758302]  r10:fffbf200 r9:da644000 r8:c0e04e64 r7:d9dc9848 r6:d9dc9874 r5:da645eac
>>>>>>> [  246.758305]  r4:d9dc9848
>>>>>>> [  246.758312] [<c0311660>] (wb_writeback) from [<c0312dac>] (wb_workfn+0x35c/0x54c)
>>>>>>> [  246.758318]  r10:da5f2005 r9:d9dc984c r8:d9dc9948 r7:d9dc9848 r6:00000000 r5:d9dc9954
>>>>>>> [  246.758321]  r4:000031e6
>>>>>>> [  246.758334] [<c0312a50>] (wb_workfn) from [<c014f2b8>] (process_one_work+0x214/0x544)
>>>>>>> [  246.758340]  r10:da5f2005 r9:00000200 r8:00000000 r7:da5f2000 r6:ef044400 r5:da5eb000
>>>>>>> [  246.758343]  r4:d9dc9954
>>>>>>> [  246.758350] [<c014f0a4>] (process_one_work) from [<c014f634>] (worker_thread+0x4c/0x574)
>>>>>>> [  246.758357]  r10:ef044400 r9:c0e03d00 r8:ef044418 r7:00000088 r6:ef044400 r5:da5eb014
>>>>>>> [  246.758359]  r4:da5eb000
>>>>>>> [  246.758368] [<c014f5e8>] (worker_thread) from [<c01564fc>] (kthread+0x144/0x170)
>>>>>>> [  246.758374]  r10:ec9e5e90 r9:dabf325c r8:da5eb000 r7:da644000 r6:00000000 r5:da5fe000
>>>>>>> [  246.758377]  r4:dabf3240
>>>>>>> [  246.758386] [<c01563b8>] (kthread) from [<c01010e8>] (ret_from_fork+0x14/0x2c)
>>>>>>> [  246.758391] Exception stack(0xda645fb0 to 0xda645ff8)
>>>>>>> [  246.758397] 5fa0:                                     00000000 00000000 00000000 00000000
>>>>>>> [  246.758402] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
>>>>>>> [  246.758407] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000
>>>>>>> [  246.758413]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c01563b8
>>>>>>> [  246.758416]  r4:da5fe000
>>>>>>> .
>>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Linux-f2fs-devel mailing list
>>>>>> Linux-f2fs-devel@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
>>>>>>
>> .
>>
> 
> 
> _______________________________________________
> Linux-f2fs-devel mailing list
> Linux-f2fs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
> 

  reply	other threads:[~2020-02-25 11:32 UTC|newest]

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-09 22:23 [PATCH 1/6] f2fs: call f2fs_balance_fs outside of locked page Jaegeuk Kim
2019-12-09 22:23 ` [PATCH 2/6] f2fs: declare nested quota_sem and remove unnecessary sems Jaegeuk Kim
2019-12-10  6:20   ` [f2fs-dev] " Chao Yu
2019-12-09 22:23 ` [PATCH 3/6] f2fs: keep quota data on write_begin failure Jaegeuk Kim
2019-12-10  6:22   ` [f2fs-dev] " Chao Yu
2019-12-09 22:23 ` [PATCH 4/6] f2fs: should avoid recursive filesystem ops Jaegeuk Kim
2019-12-10  6:22   ` [f2fs-dev] " Chao Yu
2019-12-09 22:23 ` [PATCH 5/6] f2fs: set GFP_NOFS when moving inline dentries Jaegeuk Kim
2019-12-10  6:22   ` [f2fs-dev] " Chao Yu
2019-12-09 22:23 ` [PATCH 6/6] f2fs: set I_LINKABLE early to avoid wrong access by vfs Jaegeuk Kim
2019-12-10  6:37   ` [f2fs-dev] " Chao Yu
2019-12-11  1:21     ` Jaegeuk Kim
2019-12-11  1:23       ` Chao Yu
2019-12-11  1:31         ` Jaegeuk Kim
2019-12-11  1:42           ` Chao Yu
2019-12-12 16:55             ` Jaegeuk Kim
2019-12-10  2:09 ` [f2fs-dev] [PATCH 1/6] f2fs: call f2fs_balance_fs outside of locked page Chao Yu
2020-02-22  4:46 ` Ondřej Jirman
2020-02-22 18:17   ` Writes stoped working on f2fs after the compression support was added Ondřej Jirman
2020-02-24 10:37     ` Chao Yu
2020-02-24 10:41       ` [f2fs-dev] " Chao Yu
2020-02-24 13:58         ` Ondřej Jirman
2020-02-24 14:03           ` Ondřej Jirman
2020-02-24 14:31             ` Ondřej Jirman
2020-02-25 11:24               ` Chao Yu
2020-02-25 11:32                 ` Chao Yu [this message]
2020-02-25 12:08                 ` Ondřej Jirman
2020-02-25 12:27                   ` Ondřej Jirman
2020-02-26  1:58                     ` Chao Yu
2020-02-26 12:11                       ` Ondřej Jirman
2020-02-26 18:05                         ` Ondřej Jirman
2020-02-27  2:01                           ` Chao Yu
2020-03-06 12:02                             ` Ondřej Jirman
2020-03-06 12:43                               ` Ondřej Jirman
2020-03-11  9:02                               ` Chao Yu
2020-03-11 10:33                                 ` Ondřej Jirman
2020-03-11 10:51                                   ` Chao Yu
2020-03-11 11:01                                     ` Ondřej Jirman
2020-03-11 17:01                               ` Jaegeuk Kim
2020-03-22 10:15                                 ` Chao Yu
2020-02-24 14:20         ` Ondřej Jirman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a47c85b1-8a0a-a1d4-453b-73562f0aca57@huawei.com \
    --to=yuchao0@huawei.com \
    --cc==?UTF-8?Q?megi@xff.cz_ \
    --cc=jaegeuk@kernel.org \
    --cc=linux-f2fs-devel@lists.sourceforge.net \
    --cc=linux-kernel@vger.kernel.org \
    --subject='Re: [f2fs-dev] Writes stoped working on f2fs after the compression support was added' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).