From: Li Xi Date: Thu, 10 Jul 2014 04:44:20 +0000 (+0800) Subject: LU-4090 kernel: a kernel patch for jbd2 hung X-Git-Tag: 2.6.52~53 X-Git-Url: https://git.whamcloud.com/?p=fs%2Flustre-release.git;a=commitdiff_plain;h=f8e80f01d6883c3e55841528ac16811f6588e260 LU-4090 kernel: a kernel patch for jbd2 hung patch get from linux vanilla kernel commit 0ef54180e0187117062939202b96faf04c8673bc (v3.10-rc2) jbd2:drop checkpoint mutex when waiting in __jbd2_log_wait_for_space() While trying to debug an an issue under extreme I/O loading on preempt-rt kernels, the following backtrace was observed via SysRQ output: rm D ffff8802203afbc0 4600 4878 4748 0x00000000 ffff8802217bfb78 0000000000000082 ffff88021fc2bb80 ffff88021fc2bb80 ffff88021fc2bb80 ffff8802217bffd8 ffff8802217bffd8 ffff8802217bffd8 ffff88021f1d4c80 ffff88021fc2bb80 ffff8802217bfb88 ffff88022437b000 Call Trace: [] schedule+0x24/0x70 [] jbd2_log_wait_commit+0xbd/0x140 [] ? __init_waitqueue_head+0x50/0x50 [] jbd2_log_do_checkpoint+0xf5/0x520 [] __jbd2_log_wait_for_space+0xa9/0x1f0 [] start_this_handle.isra.10+0x2e0/0x530 [] ? __init_waitqueue_head+0x50/0x50 [] jbd2__journal_start+0xc3/0x110 [] ? ext4_rmdir+0x6e/0x230 [] jbd2_journal_start+0xe/0x10 [] ext4_journal_start_sb+0x5b/0x160 [] ext4_rmdir+0x6e/0x230 [] vfs_rmdir+0xd5/0x140 [] do_rmdir+0xdf/0x120 [] ? task_work_run+0x44/0x80 [] ? do_notify_resume+0x89/0x100 [] ? int_signal+0x12/0x17 [] sys_unlinkat+0x25/0x40 [] system_call_fastpath+0x16/0x1b What is interesting here, is that we call log_wait_commit, from within wait_for_space, but we are still holding the checkpoint_mutex as it surrounds mostly the whole of wait_for_space. And then, as we are waiting, journal_commit_transaction can run, and if the JBD2_FLUSHED bit is set, then we will also try to take the same checkpoint_mutex. It seems that we need to drop the checkpoint_mutex while sitting in jbd2_log_wait_commit, if we want to guarantee that progress can be made by jbd2_journal_commit_transaction(). There does not seem to be anything preempt-rt specific about this, other then perhaps increasing the odds of it happening. Signed-off-by: Bobi Jam Signed-off-by: Li Xi Change-Id: Ibd28d221118061cf1e4e7729fda095e84313b459 Reviewed-on: http://review.whamcloud.com/11041 Tested-by: Jenkins Tested-by: Maloo Reviewed-by: Bobi Jam Reviewed-by: Bob Glossman Reviewed-by: James Simmons Reviewed-by: Alex Zhuravlev Reviewed-by: Oleg Drokin --- diff --git a/lustre/kernel_patches/patches/jbd2-log_wait_for_space-2.6-rhel6.patch b/lustre/kernel_patches/patches/jbd2-log_wait_for_space-2.6-rhel6.patch new file mode 100644 index 0000000..585068c --- /dev/null +++ b/lustre/kernel_patches/patches/jbd2-log_wait_for_space-2.6-rhel6.patch @@ -0,0 +1,69 @@ +patch get from linux vanilla kernel commit +0ef54180e0187117062939202b96faf04c8673bc (v3.10-rc2) + +jbd2: drop checkpoint mutex when waiting in __jbd2_log_wait_for_space() + +While trying to debug an an issue under extreme I/O loading +on preempt-rt kernels, the following backtrace was observed +via SysRQ output: + +rm D ffff8802203afbc0 4600 4878 4748 0x00000000 +ffff8802217bfb78 0000000000000082 ffff88021fc2bb80 ffff88021fc2bb80 +ffff88021fc2bb80 ffff8802217bffd8 ffff8802217bffd8 ffff8802217bffd8 +ffff88021f1d4c80 ffff88021fc2bb80 ffff8802217bfb88 ffff88022437b000 +Call Trace: +[] schedule+0x24/0x70 +[] jbd2_log_wait_commit+0xbd/0x140 +[] ? __init_waitqueue_head+0x50/0x50 +[] jbd2_log_do_checkpoint+0xf5/0x520 +[] __jbd2_log_wait_for_space+0xa9/0x1f0 +[] start_this_handle.isra.10+0x2e0/0x530 +[] ? __init_waitqueue_head+0x50/0x50 +[] jbd2__journal_start+0xc3/0x110 +[] ? ext4_rmdir+0x6e/0x230 +[] jbd2_journal_start+0xe/0x10 +[] ext4_journal_start_sb+0x5b/0x160 +[] ext4_rmdir+0x6e/0x230 +[] vfs_rmdir+0xd5/0x140 +[] do_rmdir+0xdf/0x120 +[] ? task_work_run+0x44/0x80 +[] ? do_notify_resume+0x89/0x100 +[] ? int_signal+0x12/0x17 +[] sys_unlinkat+0x25/0x40 +[] system_call_fastpath+0x16/0x1b + +What is interesting here, is that we call log_wait_commit, from +within wait_for_space, but we are still holding the checkpoint_mutex +as it surrounds mostly the whole of wait_for_space. And then, as we +are waiting, journal_commit_transaction can run, and if the JBD2_FLUSHED +bit is set, then we will also try to take the same checkpoint_mutex. + +It seems that we need to drop the checkpoint_mutex while sitting in +jbd2_log_wait_commit, if we want to guarantee that progress can be made +by jbd2_journal_commit_transaction(). There does not seem to be +anything preempt-rt specific about this, other then perhaps increasing +the odds of it happening. + +Signed-off-by: Paul Gortmaker +Signed-off-by: "Theodore Ts'o" + +Index: linux-2.6.32-358.18.1.el6-master/fs/jbd2/checkpoint.c +=================================================================== +--- linux-2.6.32-358.18.1.el6-master.orig/fs/jbd2/checkpoint.c ++++ linux-2.6.32-358.18.1.el6-master/fs/jbd2/checkpoint.c +@@ -156,7 +156,15 @@ void __jbd2_log_wait_for_space(journal_t + /* We were able to recover space; yay! */ + ; + } else if (tid) { ++ /* ++ * jbd2_journal_commit_transaction() may want ++ * to take the checkpoint_mutex if JBD2_FLUSHED ++ * is set. So we need to temporarily drop it. ++ */ ++ mutex_unlock(&journal->j_checkpoint_mutex); + jbd2_log_wait_commit(journal, tid); ++ spin_lock(&journal->j_state_lock); ++ continue; + } else { + printk(KERN_ERR "%s: needed %d blocks and " + "only had %d space available\n", diff --git a/lustre/kernel_patches/series/2.6-rhel6.series b/lustre/kernel_patches/series/2.6-rhel6.series index 517aff6..006ce73 100644 --- a/lustre/kernel_patches/series/2.6-rhel6.series +++ b/lustre/kernel_patches/series/2.6-rhel6.series @@ -5,3 +5,4 @@ blkdev_tunables-2.6-rhel6.patch bh_lru_size_config.patch quota-replace-dqptr-sem.patch quota-avoid-dqget-calls.patch +jbd2-log_wait_for_space-2.6-rhel6.patch diff --git a/lustre/kernel_patches/series/3.0-sles11.series b/lustre/kernel_patches/series/3.0-sles11.series index 0b42b86..18549a9 100644 --- a/lustre/kernel_patches/series/3.0-sles11.series +++ b/lustre/kernel_patches/series/3.0-sles11.series @@ -3,3 +3,4 @@ quota-replace-dqptr-sem-sles11sp2.patch quota-avoid-dqget-calls-sles11sp2.patch blkdev_tunables-3.0-sles11.patch bh_lru_size_config.patch +jbd2-log_wait_for_space-2.6-rhel6.patch diff --git a/lustre/kernel_patches/series/3.0-sles11sp3.series b/lustre/kernel_patches/series/3.0-sles11sp3.series index 32e2d4a..c40c5f1 100644 --- a/lustre/kernel_patches/series/3.0-sles11sp3.series +++ b/lustre/kernel_patches/series/3.0-sles11sp3.series @@ -3,3 +3,4 @@ quota-replace-dqptr-sem-sles11sp2.patch quota-avoid-dqget-calls-sles11sp2.patch blkdev_tunables-3.0-sles11sp3.patch bh_lru_size_config.patch +jbd2-log_wait_for_space-2.6-rhel6.patch