1 Patch management scripts
2 Andrew Morton <akpm@digeo.com>
5 This is a description of a bunch of shell scripts which I use for
6 managing kernel patches. They are quite powerful. They can be used on
7 projects other than the linux kernel. They are easy to use, and fast.
9 You end up doing a ton of recompiling with these scripts, because
10 you're pushing and popping all the time. ccache takes away the pain of
11 all that. http://ccache.samba.org/ - be sure to put the cache
12 directory on the same fs as where you're working so that ccache can use
15 The key philosophical concept is that your primary output is patches.
16 Not ".c" files, not ".h" files. But patches. So patches are the
17 first-class object here.
22 You place all the scripts somewhere in your path, or in
23 /usr/lib/patch-scripts.
28 The patch scripts require three special directories called "pc",
31 If the environment variable PATCHSCRIPTS is set, it is taken to to be
32 the directory in which those three directories reside. Typically, it
33 would be a relative pathname. So
35 setenv PATCHSCRIPTS ./i-put-them-here
37 would tell the patch scripts to look in ./i-put-them-here/pc, etc.
39 If PATCHSCRIPTS is not set, and the directory ./patch-scripts is
40 present then the patch scripts will us ./patch-scripts/pc/,
41 ./patch-scripts/patches/ and ./patch-scripts/txt/.
43 Otherwise, the patch scripts use ./pc, ./patches and ./txt.
45 In this document, the symbol $P is used to describe the directory which
46 holds the pc/, patches/ and txt/ directories, as determined by the
49 It is expected that $P will always expand to a relative path.
54 All work occurs with a single directory tree. All commands are invoked
55 within the root of that tree. The scripts manage a "stack" of patches.
57 Each patch is a changeset against the base tree plus the preceding patches.
59 All patches are listed, in order, in the file ./series. You manage the
60 series file. Lines in the series file which start with `#' are ignored.
62 Any currently-applied patches are described in the file
63 ./applied-patches. The patch scripts manage this file.
65 Each patch affects a number of files in the tree. These files are
66 listed in a "patch control" file. These .pc files live in the
69 Patches are placed in the directory $P/patches/
71 Documentation for the patches is placed in $P/txt/
73 So for a particular patch "my-first-patch" the following will exist:
75 - An entry "my-first-patch.patch" in ./series
77 - An entry "my-first-patch" in ./applied-patches (if it's currently applied)
79 - A file $P/pc/my-first-patch.pc which contains the names of the
80 files which my-first-patch modifies, adds or removes
82 - A file $P/txt/my-first-patch.txt which contains the patch's
85 - A file $P/patches/my-first-patch.patch, which is the output of the
91 When a patch "my-patch" is applied with apatch, or with pushpatch
92 (which calls apatch), all the affected files (from $P/pc/my-patch.pc)
93 are copied to files with ~my-patch appended. So if $P/pc/my-patch.pc
99 then apatch will copy those files into kernel/sched.c~my-patch and
100 fs/inode.c~my-patch. It will then apply the patch to kernel/sched.c
103 When a diff is regenerated by refpatch (which calls mpatch), the diff
104 is made between kernel/sched.c and kernel/sched.c~my-patch. How do the
105 scripts know to use "~my-patch"? Because my-patch is the current
106 topmost patch. It's the last line in ./applied-patches.
108 In this way, the whole thing is stackable. If you have four patches
109 applied, say "patch-1", "patch-2", "patch-3" and "patch-4", and if
110 patch-2 and patch-4 both touch kernel/sched.c then you will have:
112 kernel/sched.c~patch-2 Original copy, before patch-2
113 kernel/sched.c~patch-4 Copy before patch-4. Contains changes
115 kernel/sched.c Current working copy. Contains changes
118 This means that your diff headers contain "~patch-name" in them, which
119 is convenient documentation.
126 Go into /usr/src/linux (or wherever)
130 Now let's generate a patch
132 fpatch my-patch kernel/sched.c
134 OK, we've copied kernel/sched.c to kernel/sched.c~my-patch. We've
135 appended "my-patch" to ./applied-patches and we've put "kernel/sched.c"
136 into the patch control file, pc/my-patch.pc.
138 Now edit kernel/sched.c a bit.
140 Now we're ready to document the patch
142 Now write txt/my-patch.txt
144 Now generate the patch
148 This will generate patches/my-patch.patch. Take a look.
154 applied-patches is now empty, and the patch is removed.
156 Now let's add a file to my-patch and then generate my-second-patch:
158 Add "my-patch.patch" to ./series (no blank lines in that file please)
162 OK, the patch is applied again. Let's add another file
164 fpatch kernel/printk.c
166 Note that here we gave fpatch a single argument. So rather than
167 opening a new patch, it adds kernel/printk.c to the existing topmost
168 patch. That's my-patch.
172 Refresh my-patch (you end up running refpatch a lot)
176 Now start a second patch:
178 fpatch my-second-patch kernel/sched.c
180 Now take a look at applied-patches. Also do an `ls kernel/sched*'.
182 Edit kernel/sched.c, to make some changes for my-second-patch
184 Generate my-second-patch:
188 Take a look in patches/my-second-patch.patch
190 Don't forget to add "my-second-patch.patch" to the series file.
192 And remove both patches:
198 That's pretty much it, really.
204 Generally, where any of these commands take a "patch-name", that can be
205 of the form txt/patch-name.txt, patch-name.pc, just patch-name or
206 whatever. The scripts will strip off a leading "txt/", "patches/" or
207 "pc/" and any trailing extension. This is so you can do
209 apatch patches/a<tab>
211 to conveniently use shell tabbing to select patch names.
219 apatch [-f] patch-name
221 This is the low-level function which adds patches. It does the
222 copying into ~-files and updates the applied-patches file. It
223 applies the actual patch.
225 apatch will do a patch --dry-run first and will refuse to apply the
226 patch if the dryrun fails.
228 So when you are getting rejects you do this:
230 pushpatch # This fails, due to rejects. Drat.
231 apatch -f patch-name # Force the patch
232 (or) pushpatch -f # Force the patch
234 OK, you've now applied patch-name, but you have rejects. Go fix
239 And you're ready to move on.
241 combine-series output-file
243 It incrementally combinediffs all the patches in series to make a
244 complete patch for the series. Requires combinediff frmo patchutils.
246 See http://cyberelk.net/tim/patchutils/ (Don't download the
247 "experimental" patchutils - it seems to only have half of the
248 commands in it. Go for "stable")
256 export the patches listed in ./series to a set of files which
257 are named in such a way that the sort order is the same as the
258 order of the series file.
260 Usage: export_patch directory [prefix]
264 Suppose ./series contains
272 export_patch ../mypatches fruit
274 The patches would be copied to
276 ../mypatches/p00001_fruit_mango.patch
277 ../mypatches/p00002_fruit_orange.patch
278 ../mypatches/p00003_fruit_banana.patch
279 ../mypatches/p00003_fruit_banana.patch
280 ../mypatches/p00003_fruit_banana.patch
282 Named in this way, someone may easily apply them:
284 cat mypatches/p*fruit* | patch -p1
286 If prefix is omitted, the patchnames will be transformed
287 such that "original.patch" becomes "pXXXXX_original.patch".
289 fpatch [patch-name] foo.c
291 If patch-name is given, fpatch will start a new patch which
292 modifies (or adds, or removes) the single file foo.c. It updates
293 ./applied-patches and creates pc/patch-name.pc. fpatch will copy
294 foo.c to foo.c~patch-name in preparation for edits of foo.c.
296 If patch-name is not given then fpatch will add foo.c to the
297 current topmost patch. It will add "foo.c" to $P/pc/$(toppatch).pc.
298 It will copy foo.c to foo.c~$(toppatch).
302 Imports a set of patch files, creating $P/pc, $P/txt, $P/patches and
303 ./series as necessary. It also creates $P/txt/*.txt by stripping
304 off the top of the patches (and removes any diffstat output it finds,
305 so that it can eat refpatch output and export_patch output.) The
306 imported patch names are appended to the series file.
308 In creating the $P/txt/*.txt files, mail headers are stripped with
309 formail, preserving the "From:" and "Subject:" lines. "DESC" and
310 "EDESC" markers are added if they are not already present, using the
311 "From:" and "Subject:" lines for the DESC portion, if they are present.
312 (See "patchdesc" command, below, for more on these markers.)
314 Also, it can rename the patch file as it is imported by stripping out
315 a pattern. This is useful if, as often is the case, you have patch
316 sets with filenames designed to help sort the patches into the correct
317 order, such as "p001_xxx_funky_stuff.patch" you can have it automatically
318 renamed to funky_stuff.patch on import, and let the series file manage
321 Import_patch will uncompress patches (*.Z, *.bz2, *.gz) as necessary.
325 import_patch [-p pattern] patchfile ...
329 % ls ../fruit/p*patch
330 ../fruit/p00001_northern_apple.patch
331 ../fruit/p00001_tropical_mango.patch
332 ../fruit/p00002_northern_pear.patch
333 ../fruit/p00002_tropical_orange.patch
334 ../fruit/p00003_tropical_banana.patch
335 % import_patch -p 'p[0-9]*_tropical_' ../fruit/p*tropical*
336 Recreated pc/mango.pc
337 Recreated pc/orange.pc
338 Recreated pc/banana.pc
339 % import_patch -p 'p[0-9]*_northern_' ../fruit/p*northern*
340 Recreated pc/apple.pc
343 Then you can "pushpatch; refpatch" 5 times.
347 List the names of ths files which are affected by the current
352 cat pc/$(toppatch).pc
356 "joins" the named patch to the current topmost patch.
358 Use this when you want to merge two patches into one. All the
359 files which `patchname' affects are added to pc/$(toppatch).pc (if
360 they are not already there) and patch `patchname' is applied. The
361 top patch remains unchanged. You'll need to run refpatch afterwards.
365 A low-level thing to generate patches
369 Some thing I use for importing a new kernel from kernel.org
373 Internal thing to convert patch -p0 form into patch -p1
377 Generates a single-line description of a patch.
379 The txt/my-patch.txt files have the following format:
383 some short description
391 patchdesc $(cat series)
393 to generate short-form summaries of the patch series.
401 Standalone tool to generate a .pc file from a patch.
403 Say someone sends you "his-patch.diff". What you do is:
405 cp ~/his-patch.diff patches/his-patch.patch
408 This generates $P/pc/his-patch.pc and you're all set. Add
409 "his-patch.patch" to ./series in the right place and start pushing.
417 Remove one or more patches from the current stack. This command
418 does *not* use the series file. It works purely against
424 Remove the topmost patch
427 poppatch some-patch-name[.patch]
428 Remove patches until "some-patch-name" is top patch
432 Shows status of patches
435 pstatus [patchfile ...]
437 One line per patch is output showing:
438 1: Patch number in the series file
439 2: Whether the patch is currently applied
441 4: Status of the patch (needs pcpatch, changelog, refpatch)
443 If no patchfiles are specified, $P/patches/*.patch
447 A patch set which contains separate patches to add a file
448 and modify that same file may give spurious "Needs refpatch"
449 status for the patch which adds the file or the topmost patch.
457 Run tkdiff against all the file affected
458 by $(toppatch). The diff is only for the changes made
459 by the top patch! ie: it's between "filename" and
460 "filename~toppatch-name".
464 Just run tkdiff against that file,
465 showing the changes which are due to toppatch.
469 Apply the next patch, from the series file.
471 This consults ./applied-patches to find out the top patch, then
472 consults ./series to find the next patch. And pushes it.
480 Apply the next ten patches
482 pushpatch some-patch-name
484 Keep pushing patches until "some-patch-name" is toppatch
488 Push the next patch, ignoring rejects.
492 regnerates the topmost patch. Reads all the affected files
493 from pc/$(toppatch).pc and diffs them against their tilde-files.
495 Also pastes into the patch your patch documentation and
496 generates a diffstat summary.
504 CVS rename for patches.
508 Bit of a hack. Is designed to generate a rolled-up diff of all
509 currently-applied patches. But it requires a ../linux-2.x.y tree to
510 diff against. Needs to be redone.
518 Some thing someone write to split patches up. I don't use it.
522 Assuming you keep pc/*, patches/* and txt/* under CVS revision
523 control, tag-series allows you to tag a patchset's individual
526 tag-series s2_5_44-mm3 pc/2.5.44-mm3-series
528 which will attach the cvs tag "s2_5_44-mm3" to every .pc, .patch
529 and .txt file which is mentioned in the series file
530 "pc/2.5.44-mm3-series".
532 It will also tag pc/2.5.44-mm3-series, which is a bit redundant
533 given that I use a different series file for each patchset release..
538 Print the name of the topmost patch. From ./applied-patches
540 touched-by-patch patch-filename
542 List the names of files which are affected by a diff.
546 Rasmus Andersen's script to convert a diff into minimum-context
547 form. This form has a better chance of applying if you're getting
548 nasty rejects. But patch can and will make mistakes when fed
555 I keep the kernel tree, the $P/pc/, $P/patches/ and $P/txt/ contents under
556 CVS control. This is important...
558 I have several "series" files. I keep these in $P/pc/foo-series and use
560 ln -s pc/foo-series series
562 when I'm working on foo.
564 If someone sends me a patch I'll do:
566 cp ~/whatever patches/his-patch.patch
570 If apatch fails then run `apatch -f his-patch' and fix the rejects.
574 to clean up any fuzz.
577 cvs add pc/his-patch.pc patches/his-patch.patch
578 cvs commit pc patches
580 Now edit ./series and place "his-patch.patch" in the appropriate place.
583 If you're working on a particular patch (say, "dud-patch") and you
584 balls something up, just run:
586 refpatch # Generate the crap patch
587 poppatch # Remove it all
588 rm patches/dud-patch.patch
589 cvs up patches/dud-patch.patch
594 Getting updates from Linus
595 ==========================
597 What I do is to grab the latest -bk diff from
598 http://www.kernel.org/pub/linux/kernel/people/dwmw2/bk-2.5/
601 gzip -d < cs<tab> > patches/linus.patch
603 apatch linus | grep diff
605 Now fix up all the files which got deleted,
606 because there's something wrong with bitkeeper diffs:
608 cvs up -ko <missing files from the above diff>
611 $EDITOR linus/linus.txt
613 Add the changeset number to txt/linus.txt
618 Now add "linus.patch" as the first entry in your ./series file and
619 start pushing your other patches on top of that.
624 Tons and tons. The scripts are fragile, the error handling is ungraceful and
625 if you do something silly you can end up in a pickle.
627 Generally the scripts are very careful to not wreck your files or your
628 patches. But they can get the ./applied-patches and ~-files into an
631 Usually you can sort it out by copying the ~-files back onto the originals
632 and removing the last line from ./applied-patches. Or do a "refpatch ;
633 poppatch ; rm patches/troublesome-patch.patch ; cvs up patches".
635 If it's really bad, just blow away the entire tree and do a new CVS checkout.
638 Working on non-kernel projects
639 ==============================
641 Well it's the same thing. Say you've downloaded a copy of util-linux
642 and you want to make a change:
645 tar xvfz ~/util-linux.tar.gz
648 fpatch my-patch sys-utils/rdev.c
649 fpatch sys-utils/ipcs.8
652 <ship patches/my-patch.patch>
654 How to balls things up
655 ======================
657 Well here's one way. Suppose you have 20 patches applied, and three of
658 them (say, "p1", "p6" and "p11") all modify "foo.c".
660 Now you go and change foo.c.
662 Well, to which patch does that change belong? You need to decide.
663 Let's say you decide "p6".
665 If you run `refpatch' when "p11" is toppatch then you lose. The diff
677 (See why ccache is looking good?)
684 poppatch p6 <hope like hell that the other patches remove cleanly>
688 Another good way of ballsing up is to cheat. Say "oh I just want to make
689 this one-line change". And "oh, and this one".
691 Now you're getting in a mess. It's much, much better to just use the system:
699 rm pc/junk.pc patches/junk.patch
701 Merging with -mm kernels
702 ========================
704 Haven't tried this, but it should work:
706 - Grab all the patches from broken-out/, place them in your $P/patches/
708 - Copy my series file into ./series (or $P/pc/akpm-series and symlink it)
712 And you're off and running. The nice thing about this is that you can
713 send me incremental diffs to diffs which I already have.
715 Or whatever. I'm fairly handy with diffs nowadays. Rejects are
716 expected. I just prefer to have "one concept per diff".