LU-4090 kernel: a kernel patch for jbd2 hung
patch get from linux vanilla kernel commit
0ef54180e0187117062939202b96faf04c8673bc (v3.10-rc2)
jbd2:drop checkpoint mutex when waiting in __jbd2_log_wait_for_space()
While trying to debug an an issue under extreme I/O loading
on preempt-rt kernels, the following backtrace was observed
via SysRQ output:
rm D
ffff8802203afbc0 4600 4878 4748 0x00000000
ffff8802217bfb78 0000000000000082 ffff88021fc2bb80 ffff88021fc2bb80
ffff88021fc2bb80 ffff8802217bffd8 ffff8802217bffd8 ffff8802217bffd8
ffff88021f1d4c80 ffff88021fc2bb80 ffff8802217bfb88 ffff88022437b000
Call Trace:
[<
ffffffff8172dc34>] schedule+0x24/0x70
[<
ffffffff81225b5d>] jbd2_log_wait_commit+0xbd/0x140
[<
ffffffff81060390>] ? __init_waitqueue_head+0x50/0x50
[<
ffffffff81223635>] jbd2_log_do_checkpoint+0xf5/0x520
[<
ffffffff81223b09>] __jbd2_log_wait_for_space+0xa9/0x1f0
[<
ffffffff8121dc40>] start_this_handle.isra.10+0x2e0/0x530
[<
ffffffff81060390>] ? __init_waitqueue_head+0x50/0x50
[<
ffffffff8121e0a3>] jbd2__journal_start+0xc3/0x110
[<
ffffffff811de7ce>] ? ext4_rmdir+0x6e/0x230
[<
ffffffff8121e0fe>] jbd2_journal_start+0xe/0x10
[<
ffffffff811f308b>] ext4_journal_start_sb+0x5b/0x160
[<
ffffffff811de7ce>] ext4_rmdir+0x6e/0x230
[<
ffffffff811435c5>] vfs_rmdir+0xd5/0x140
[<
ffffffff8114370f>] do_rmdir+0xdf/0x120
[<
ffffffff8105c6b4>] ? task_work_run+0x44/0x80
[<
ffffffff81002889>] ? do_notify_resume+0x89/0x100
[<
ffffffff817361ae>] ? int_signal+0x12/0x17
[<
ffffffff81145d85>] sys_unlinkat+0x25/0x40
[<
ffffffff81735f22>] system_call_fastpath+0x16/0x1b
What is interesting here, is that we call log_wait_commit, from
within wait_for_space, but we are still holding the checkpoint_mutex
as it surrounds mostly the whole of wait_for_space. And then, as we
are waiting, journal_commit_transaction can run, and if the
JBD2_FLUSHED bit is set, then we will also try to take the same
checkpoint_mutex.
It seems that we need to drop the checkpoint_mutex while sitting in
jbd2_log_wait_commit, if we want to guarantee that progress can be
made by jbd2_journal_commit_transaction(). There does not seem to be
anything preempt-rt specific about this, other then perhaps increasing
the odds of it happening.
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Li Xi <lixi@ddn.com>
Change-Id: Ibd28d221118061cf1e4e7729fda095e84313b459
Reviewed-on: http://review.whamcloud.com/11041
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>