Patch management scripts Andrew Morton 18 October 2002 This is a description of a bunch of shell scripts which I use for managing kernel patches. They are quite powerful. They can be used on projects other than the linux kernel. They are easy to use, and fast. You end up doing a ton of recompiling with these scripts, because you're pushing and popping all the time. ccache takes away the pain of all that. http://ccache.samba.org/ - be sure to put the cache directory on the same fs as where you're working so that ccache can use hardlinks. The key philosophical concept is that your primary output is patches. Not ".c" files, not ".h" files. But patches. So patches are the first-class object here. Installation ============ You place all the scripts somewhere in your path, or in /usr/lib/patch-scripts. Terminology =========== The patch scripts require three special directories called "pc", "patches" and "txt". If the environment variable PATCHSCRIPTS is set, it is taken to to be the directory in which those three directories reside. Typically, it would be a relative pathname. So setenv PATCHSCRIPTS ./i-put-them-here would tell the patch scripts to look in ./i-put-them-here/pc, etc. If PATCHSCRIPTS is not set, and the directory ./patch-scripts is present then the patch scripts will us ./patch-scripts/pc/, ./patch-scripts/patches/ and ./patch-scripts/txt/. Otherwise, the patch scripts use ./pc, ./patches and ./txt. In this document, the symbol $P is used to describe the directory which holds the pc/, patches/ and txt/ directories, as determined by the above search. It is expected that $P will always expand to a relative path. Concepts ======== All work occurs with a single directory tree. All commands are invoked within the root of that tree. The scripts manage a "stack" of patches. Each patch is a changeset against the base tree plus the preceding patches. All patches are listed, in order, in the file ./series. You manage the series file. Lines in the series file which start with `#' are ignored. Any currently-applied patches are described in the file ./applied-patches. The patch scripts manage this file. Each patch affects a number of files in the tree. These files are listed in a "patch control" file. These .pc files live in the directory $P/pc/ Patches are placed in the directory $P/patches/ Documentation for the patches is placed in $P/txt/ So for a particular patch "my-first-patch" the following will exist: - An entry "my-first-patch.patch" in ./series - An entry "my-first-patch" in ./applied-patches (if it's currently applied) - A file $P/pc/my-first-patch.pc which contains the names of the files which my-first-patch modifies, adds or removes - A file $P/txt/my-first-patch.txt which contains the patch's changelog. - A file $P/patches/my-first-patch.patch, which is the output of the patch scripts. Operation ========= When a patch "my-patch" is applied with apatch, or with pushpatch (which calls apatch), all the affected files (from $P/pc/my-patch.pc) are copied to files with ~my-patch appended. So if $P/pc/my-patch.pc contained kernel/sched.c fs/inode.c then apatch will copy those files into kernel/sched.c~my-patch and fs/inode.c~my-patch. It will then apply the patch to kernel/sched.c and fs/inode.c When a diff is regenerated by refpatch (which calls mpatch), the diff is made between kernel/sched.c and kernel/sched.c~my-patch. How do the scripts know to use "~my-patch"? Because my-patch is the current topmost patch. It's the last line in ./applied-patches. In this way, the whole thing is stackable. If you have four patches applied, say "patch-1", "patch-2", "patch-3" and "patch-4", and if patch-2 and patch-4 both touch kernel/sched.c then you will have: kernel/sched.c~patch-2 Original copy, before patch-2 kernel/sched.c~patch-4 Copy before patch-4. Contains changes from patch-2 kernel/sched.c Current working copy. Contains changes from patch-4. This means that your diff headers contain "~patch-name" in them, which is convenient documentation. Walkthrough =========== Let's start. Go into /usr/src/linux (or wherever) mkdir pc patches txt Now let's generate a patch fpatch my-patch kernel/sched.c OK, we've copied kernel/sched.c to kernel/sched.c~my-patch. We've appended "my-patch" to ./applied-patches and we've put "kernel/sched.c" into the patch control file, pc/my-patch.pc. Now edit kernel/sched.c a bit. Now we're ready to document the patch Now write txt/my-patch.txt Now generate the patch refpatch This will generate patches/my-patch.patch. Take a look. Now remove the patch poppatch applied-patches is now empty, and the patch is removed. Now let's add a file to my-patch and then generate my-second-patch: Add "my-patch.patch" to ./series (no blank lines in that file please) pushpatch OK, the patch is applied again. Let's add another file fpatch kernel/printk.c Note that here we gave fpatch a single argument. So rather than opening a new patch, it adds kernel/printk.c to the existing topmost patch. That's my-patch. Edit kernel/printk.c Refresh my-patch (you end up running refpatch a lot) refpatch Now start a second patch: fpatch my-second-patch kernel/sched.c Now take a look at applied-patches. Also do an `ls kernel/sched*'. Edit kernel/sched.c, to make some changes for my-second-patch Generate my-second-patch: refpatch Take a look in patches/my-second-patch.patch Don't forget to add "my-second-patch.patch" to the series file. And remove both patches: poppatch poppatch That's pretty much it, really. Command reference ================= Generally, where any of these commands take a "patch-name", that can be of the form txt/patch-name.txt, patch-name.pc, just patch-name or whatever. The scripts will strip off a leading "txt/", "patches/" or "pc/" and any trailing extension. This is so you can do apatch patches/a to conveniently use shell tabbing to select patch names. added-by-patch Some internal thing. apatch [-f] patch-name This is the low-level function which adds patches. It does the copying into ~-files and updates the applied-patches file. It applies the actual patch. apatch will do a patch --dry-run first and will refuse to apply the patch if the dryrun fails. So when you are getting rejects you do this: pushpatch # This fails, due to rejects. Drat. apatch -f patch-name # Force the patch (or) pushpatch -f # Force the patch OK, you've now applied patch-name, but you have rejects. Go fix those up and do refpatch And you're ready to move on. combine-series output-file It incrementally combinediffs all the patches in series to make a complete patch for the series. Requires combinediff frmo patchutils. See http://cyberelk.net/tim/patchutils/ (Don't download the "experimental" patchutils - it seems to only have half of the commands in it. Go for "stable") cvs-take-patch I forget. export_patch export the patches listed in ./series to a set of files which are named in such a way that the sort order is the same as the order of the series file. Usage: export_patch directory [prefix] Example: Suppose ./series contains mango.patch orange.patch banana.patch apple.patch pear.patch export_patch ../mypatches fruit The patches would be copied to ../mypatches/p00001_fruit_mango.patch ../mypatches/p00002_fruit_orange.patch ../mypatches/p00003_fruit_banana.patch ../mypatches/p00003_fruit_banana.patch ../mypatches/p00003_fruit_banana.patch Named in this way, someone may easily apply them: cat mypatches/p*fruit* | patch -p1 If prefix is omitted, the patchnames will be transformed such that "original.patch" becomes "pXXXXX_original.patch". fpatch [patch-name] foo.c If patch-name is given, fpatch will start a new patch which modifies (or adds, or removes) the single file foo.c. It updates ./applied-patches and creates pc/patch-name.pc. fpatch will copy foo.c to foo.c~patch-name in preparation for edits of foo.c. If patch-name is not given then fpatch will add foo.c to the current topmost patch. It will add "foo.c" to $P/pc/$(toppatch).pc. It will copy foo.c to foo.c~$(toppatch). import_patch Imports a set of patch files, creating $P/pc, $P/txt, $P/patches and ./series as necessary. It also creates $P/txt/*.txt by stripping off the top of the patches (and removes any diffstat output it finds, so that it can eat refpatch output and export_patch output.) The imported patch names are appended to the series file. In creating the $P/txt/*.txt files, mail headers are stripped with formail, preserving the "From:" and "Subject:" lines. "DESC" and "EDESC" markers are added if they are not already present, using the "From:" and "Subject:" lines for the DESC portion, if they are present. (See "patchdesc" command, below, for more on these markers.) Also, it can rename the patch file as it is imported by stripping out a pattern. This is useful if, as often is the case, you have patch sets with filenames designed to help sort the patches into the correct order, such as "p001_xxx_funky_stuff.patch" you can have it automatically renamed to funky_stuff.patch on import, and let the series file manage the ordering. Import_patch will uncompress patches (*.Z, *.bz2, *.gz) as necessary. Usage: import_patch [-p pattern] patchfile ... Example: % ls ../fruit/p*patch ../fruit/p00001_northern_apple.patch ../fruit/p00001_tropical_mango.patch ../fruit/p00002_northern_pear.patch ../fruit/p00002_tropical_orange.patch ../fruit/p00003_tropical_banana.patch % import_patch -p 'p[0-9]*_tropical_' ../fruit/p*tropical* Recreated pc/mango.pc Recreated pc/orange.pc Recreated pc/banana.pc % import_patch -p 'p[0-9]*_northern_' ../fruit/p*northern* Recreated pc/apple.pc Recreated pc/pear.pc Then you can "pushpatch; refpatch" 5 times. inpatch List the names of ths files which are affected by the current topmost patch. This is basically cat pc/$(toppatch).pc join-patch patchname "joins" the named patch to the current topmost patch. Use this when you want to merge two patches into one. All the files which `patchname' affects are added to pc/$(toppatch).pc (if they are not already there) and patch `patchname' is applied. The top patch remains unchanged. You'll need to run refpatch afterwards. mpatch A low-level thing to generate patches new-kernel Some thing I use for importing a new kernel from kernel.org p0-2-p1 Internal thing to convert patch -p0 form into patch -p1 patchdesc Generates a single-line description of a patch. The txt/my-patch.txt files have the following format: DESC some short description EDESC The long description I use patchdesc $(cat series) to generate short-form summaries of the patch series. patchfns Internal utilities pcpatch Standalone tool to generate a .pc file from a patch. Say someone sends you "his-patch.diff". What you do is: cp ~/his-patch.diff patches/his-patch.patch pcpatch his-patch This generates $P/pc/his-patch.pc and you're all set. Add "his-patch.patch" to ./series in the right place and start pushing. p_diff I forget poppatch Remove one or more patches from the current stack. This command does *not* use the series file. It works purely against applied-patches. Usage: poppatch Remove the topmost patch poppatch 10 Remove ten patches poppatch some-patch-name[.patch] Remove patches until "some-patch-name" is top patch pstatus Shows status of patches Usage: pstatus [patchfile ...] One line per patch is output showing: 1: Patch number in the series file 2: Whether the patch is currently applied 3: Name of patch 4: Status of the patch (needs pcpatch, changelog, refpatch) If no patchfiles are specified, $P/patches/*.patch are assumed. Caveats: A patch set which contains separate patches to add a file and modify that same file may give spurious "Needs refpatch" status for the patch which adds the file or the topmost patch. ptkdiff Two modes: ptkdiff - Run tkdiff against all the file affected by $(toppatch). The diff is only for the changes made by the top patch! ie: it's between "filename" and "filename~toppatch-name". ptkdiff filename Just run tkdiff against that file, showing the changes which are due to toppatch. pushpatch [-f] Apply the next patch, from the series file. This consults ./applied-patches to find out the top patch, then consults ./series to find the next patch. And pushes it. pushpatch Apply the next patch pushpatch 10 Apply the next ten patches pushpatch some-patch-name Keep pushing patches until "some-patch-name" is toppatch pushpatch -f Push the next patch, ignoring rejects. refpatch regnerates the topmost patch. Reads all the affected files from pc/$(toppatch).pc and diffs them against their tilde-files. Also pastes into the patch your patch documentation and generates a diffstat summary. removed-by-patch Some thing. rename-patch CVS rename for patches. rolled-up-patch Bit of a hack. Is designed to generate a rolled-up diff of all currently-applied patches. But it requires a ../linux-2.x.y tree to diff against. Needs to be redone. rpatch Internal command split-patch Some thing someone write to split patches up. I don't use it. tag-series Assuming you keep pc/*, patches/* and txt/* under CVS revision control, tag-series allows you to tag a patchset's individual components. I use tag-series s2_5_44-mm3 pc/2.5.44-mm3-series which will attach the cvs tag "s2_5_44-mm3" to every .pc, .patch and .txt file which is mentioned in the series file "pc/2.5.44-mm3-series". It will also tag pc/2.5.44-mm3-series, which is a bit redundant given that I use a different series file for each patchset release.. toppatch Print the name of the topmost patch. From ./applied-patches touched-by-patch patch-filename List the names of files which are affected by a diff. unitdiff.py Rasmus Andersen's script to convert a diff into minimum-context form. This form has a better chance of applying if you're getting nasty rejects. But patch can and will make mistakes when fed small-context input. Work Practices ============== I keep the kernel tree, the $P/pc/, $P/patches/ and $P/txt/ contents under CVS control. This is important... I have several "series" files. I keep these in $P/pc/foo-series and use ln -s pc/foo-series series when I'm working on foo. If someone sends me a patch I'll do: cp ~/whatever patches/his-patch.patch pcpatch his-patch apatch his-patch If apatch fails then run `apatch -f his-patch' and fix the rejects. refpatch to clean up any fuzz. poppatch cvs add pc/his-patch.pc patches/his-patch.patch cvs commit pc patches Now edit ./series and place "his-patch.patch" in the appropriate place. If you're working on a particular patch (say, "dud-patch") and you balls something up, just run: refpatch # Generate the crap patch poppatch # Remove it all rm patches/dud-patch.patch cvs up patches/dud-patch.patch and all is well. Getting updates from Linus ========================== What I do is to grab the latest -bk diff from http://www.kernel.org/pub/linux/kernel/people/dwmw2/bk-2.5/ and do: gzip -d < cs > patches/linus.patch pcpatch linus apatch linus | grep diff Now fix up all the files which got deleted, because there's something wrong with bitkeeper diffs: cvs up -ko apatch linus $EDITOR linus/linus.txt Add the changeset number to txt/linus.txt refpatch poppatch Now add "linus.patch" as the first entry in your ./series file and start pushing your other patches on top of that. BUGS ==== Tons and tons. The scripts are fragile, the error handling is ungraceful and if you do something silly you can end up in a pickle. Generally the scripts are very careful to not wreck your files or your patches. But they can get the ./applied-patches and ~-files into an awkward state. Usually you can sort it out by copying the ~-files back onto the originals and removing the last line from ./applied-patches. Or do a "refpatch ; poppatch ; rm patches/troublesome-patch.patch ; cvs up patches". If it's really bad, just blow away the entire tree and do a new CVS checkout. Working on non-kernel projects ============================== Well it's the same thing. Say you've downloaded a copy of util-linux and you want to make a change: cd /usr/src tar xvfz ~/util-linux.tar.gz cd util-linux mkdir pc patches txt fpatch my-patch sys-utils/rdev.c fpatch sys-utils/ipcs.8 refpatch How to balls things up ====================== Well here's one way. Suppose you have 20 patches applied, and three of them (say, "p1", "p6" and "p11") all modify "foo.c". Now you go and change foo.c. Well, to which patch does that change belong? You need to decide. Let's say you decide "p6". If you run `refpatch' when "p11" is toppatch then you lose. The diff went into p11. What you can do is: 1: poppatch p6 refpatch pushpatch p11 (See why ccache is looking good?) or 2: poppatch p6 refpatch Another good way of ballsing up is to cheat. Say "oh I just want to make this one-line change". And "oh, and this one". Now you're getting in a mess. It's much, much better to just use the system: fpatch junk file1 fpatch file2 refpatch poppatch rm pc/junk.pc patches/junk.patch Merging with -mm kernels ======================== Haven't tried this, but it should work: - Grab all the patches from broken-out/, place them in your $P/patches/ - Copy my series file into ./series (or $P/pc/akpm-series and symlink it) - pushpatch 99 And you're off and running. The nice thing about this is that you can send me incremental diffs to diffs which I already have. Or whatever. I'm fairly handy with diffs nowadays. Rejects are expected. I just prefer to have "one concept per diff".