LU-4090 kernel: a kernel patch for jbd2 hung

patch get from linux vanilla kernel commit
0ef54180e0187117062939202b96faf04c8673bc (v3.10-rc2)

jbd2:drop checkpoint mutex when waiting in __jbd2_log_wait_for_space()

While trying to debug an an issue under extreme I/O loading
on preempt-rt kernels, the following backtrace was observed
via SysRQ output:

rm              D ffff8802203afbc0  4600  4878   4748 0x00000000
ffff8802217bfb78 0000000000000082 ffff88021fc2bb80 ffff88021fc2bb80
ffff88021fc2bb80 ffff8802217bffd8 ffff8802217bffd8 ffff8802217bffd8
ffff88021f1d4c80 ffff88021fc2bb80 ffff8802217bfb88 ffff88022437b000
Call Trace:
[<ffffffff8172dc34>] schedule+0x24/0x70
[<ffffffff81225b5d>] jbd2_log_wait_commit+0xbd/0x140
[<ffffffff81060390>] ? __init_waitqueue_head+0x50/0x50
[<ffffffff81223635>] jbd2_log_do_checkpoint+0xf5/0x520
[<ffffffff81223b09>] __jbd2_log_wait_for_space+0xa9/0x1f0
[<ffffffff8121dc40>] start_this_handle.isra.10+0x2e0/0x530
[<ffffffff81060390>] ? __init_waitqueue_head+0x50/0x50
[<ffffffff8121e0a3>] jbd2__journal_start+0xc3/0x110
[<ffffffff811de7ce>] ? ext4_rmdir+0x6e/0x230
[<ffffffff8121e0fe>] jbd2_journal_start+0xe/0x10
[<ffffffff811f308b>] ext4_journal_start_sb+0x5b/0x160
[<ffffffff811de7ce>] ext4_rmdir+0x6e/0x230
[<ffffffff811435c5>] vfs_rmdir+0xd5/0x140
[<ffffffff8114370f>] do_rmdir+0xdf/0x120
[<ffffffff8105c6b4>] ? task_work_run+0x44/0x80
[<ffffffff81002889>] ? do_notify_resume+0x89/0x100
[<ffffffff817361ae>] ? int_signal+0x12/0x17
[<ffffffff81145d85>] sys_unlinkat+0x25/0x40
[<ffffffff81735f22>] system_call_fastpath+0x16/0x1b

What is interesting here, is that we call log_wait_commit, from
within wait_for_space, but we are still holding the checkpoint_mutex
as it surrounds mostly the whole of wait_for_space.  And then, as we
are waiting, journal_commit_transaction can run, and if the
JBD2_FLUSHED bit is set, then we will also try to take the same
checkpoint_mutex.

It seems that we need to drop the checkpoint_mutex while sitting in
jbd2_log_wait_commit, if we want to guarantee that progress can be
made by jbd2_journal_commit_transaction().  There does not seem to be
anything preempt-rt specific about this, other then perhaps increasing
the odds of it happening.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ifc1ebe6516ea349381331d22bd4f226255330d93
Reviewed-on: http://review.whamcloud.com/8060
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Li Xi <pkuelelixi@gmail.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>

diff --git a/lustre/kernel_patches/patches/jbd2-log_wait_for_space-2.6-rhel5.patch b/lustre/kernel_patches/patches/jbd2-log_wait_for_space-2.6-rhel5.patch
new file mode 100644 (file)
index 0000000..16db364
--- /dev/null
+++ b/lustre/kernel_patches/patches/jbd2-log_wait_for_space-2.6-rhel5.patch
@@ -0,0 +1,69 @@
+patch get from linux vanilla kernel commit
+0ef54180e0187117062939202b96faf04c8673bc (v3.10-rc2)
+
+jbd2: drop checkpoint mutex when waiting in __jbd2_log_wait_for_space()
+
+While trying to debug an an issue under extreme I/O loading
+on preempt-rt kernels, the following backtrace was observed
+via SysRQ output:
+
+rm              D ffff8802203afbc0  4600  4878   4748 0x00000000
+ffff8802217bfb78 0000000000000082 ffff88021fc2bb80 ffff88021fc2bb80
+ffff88021fc2bb80 ffff8802217bffd8 ffff8802217bffd8 ffff8802217bffd8
+ffff88021f1d4c80 ffff88021fc2bb80 ffff8802217bfb88 ffff88022437b000
+Call Trace:
+[<ffffffff8172dc34>] schedule+0x24/0x70
+[<ffffffff81225b5d>] jbd2_log_wait_commit+0xbd/0x140
+[<ffffffff81060390>] ? __init_waitqueue_head+0x50/0x50
+[<ffffffff81223635>] jbd2_log_do_checkpoint+0xf5/0x520
+[<ffffffff81223b09>] __jbd2_log_wait_for_space+0xa9/0x1f0
+[<ffffffff8121dc40>] start_this_handle.isra.10+0x2e0/0x530
+[<ffffffff81060390>] ? __init_waitqueue_head+0x50/0x50
+[<ffffffff8121e0a3>] jbd2__journal_start+0xc3/0x110
+[<ffffffff811de7ce>] ? ext4_rmdir+0x6e/0x230
+[<ffffffff8121e0fe>] jbd2_journal_start+0xe/0x10
+[<ffffffff811f308b>] ext4_journal_start_sb+0x5b/0x160
+[<ffffffff811de7ce>] ext4_rmdir+0x6e/0x230
+[<ffffffff811435c5>] vfs_rmdir+0xd5/0x140
+[<ffffffff8114370f>] do_rmdir+0xdf/0x120
+[<ffffffff8105c6b4>] ? task_work_run+0x44/0x80
+[<ffffffff81002889>] ? do_notify_resume+0x89/0x100
+[<ffffffff817361ae>] ? int_signal+0x12/0x17
+[<ffffffff81145d85>] sys_unlinkat+0x25/0x40
+[<ffffffff81735f22>] system_call_fastpath+0x16/0x1b
+
+What is interesting here, is that we call log_wait_commit, from
+within wait_for_space, but we are still holding the checkpoint_mutex
+as it surrounds mostly the whole of wait_for_space.  And then, as we
+are waiting, journal_commit_transaction can run, and if the JBD2_FLUSHED
+bit is set, then we will also try to take the same checkpoint_mutex.
+
+It seems that we need to drop the checkpoint_mutex while sitting in
+jbd2_log_wait_commit, if we want to guarantee that progress can be made
+by jbd2_journal_commit_transaction().  There does not seem to be
+anything preempt-rt specific about this, other then perhaps increasing
+the odds of it happening.
+
+Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>
+Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
+
+Index: linux-2.6.18-348.3.1.el5-b18/fs/jbd2/checkpoint.c
+===================================================================
+--- linux-2.6.18-348.3.1.el5-b18.orig/fs/jbd2/checkpoint.c
++++ linux-2.6.18-348.3.1.el5-b18/fs/jbd2/checkpoint.c
+@@ -155,7 +155,15 @@ void __jbd2_log_wait_for_space(journal_t
+                               /* We were able to recover space; yay! */
+                               ;
+                       } else if (tid) {
++                              /*
++                               * jbd2_journal_commit_transaction() may want
++                               * to take the checkpoint_mutex if JBD2_FLUSHED
++                               * is set.  So we need to temporarily drop it.
++                               */
++                              mutex_unlock(&journal->j_checkpoint_mutex);
+                               jbd2_log_wait_commit(journal, tid);
++                              spin_lock(&journal->j_state_lock);
++                              continue;
+                       } else {
+                               printk(KERN_ERR "%s: needed %d blocks and "
+                                      "only had %d space available\n",

diff --git a/lustre/kernel_patches/series/2.6-rhel5.series b/lustre/kernel_patches/series/2.6-rhel5.series
index 6e22e25..47014b2 100644 (file)
--- a/lustre/kernel_patches/series/2.6-rhel5.series
+++ b/lustre/kernel_patches/series/2.6-rhel5.series
@@ -25,3 +25,4 @@ mpt-fusion-max-sge.patch
 jbd2-jcberr-2.6-rhel5.patch
 jbd2-commit-timer-no-jiffies-rounding.diff
 make-bi_phys_segments-uint.patch
+jbd2-log_wait_for_space-2.6-rhel5.patch