markusbe.com » background

How to read a patch or diff and understand its structure to apply it manually

Markus Bertheau — Sat, 05 Dec 2009 16:24:56 +0000

I wrote this article to show you how patch/diff files store difference information. After reading it you will be able to understand patch files when you read them, better understand certain patch error messages and apply changes manually.

Patch or diff files are just text files, so you can look at them with less or a text editor.

If you prefer to use a terminal, colordiff (in the colordiff package) provides syntax highlighting. If the patch is long, you may want to use less to look at it. You need the -R option for less, or else the colors won’t show. I also always use -S, which will enable horizontal scrolling in less instead of wrapping long lines. The whole command to view a patch with syntax highlighting through less in a terminal is:

markus@ubuntu:~$ cat file.patch | colordiff | less -RS

Patch formats

Each patch file comes in one of 3 formats, called normal, context and unified diff. You can identify them by looking at the contents of the patch file.

Normal diffs have a lot of lines that start with > or <:

...
51c51
< background-color: red;
---
> background-color: blue;
...

Skip to how to read a normal diff

Context diffs have a lot of lines with stars ***** in them and lines starting with !, + and -:

...
***************
*** 44,54 ****

  html, body { color: #242626; }
  html {
!   background-color: red;
    height: 100%;
    margin-bottom: 1px;
    overflow-y: scroll;
--- 44,54 ----

  html, body { color: #242626; }
  html {
!   background-color: blue;
    height: 100%;
    margin-bottom: 1px;
    overflow-y: scroll;
...

Skip to how to read a context diff

And finally unified diffs have a lot of lines with two @@s in them and lines starting with + and -:

...
@@ -44,11 +44,11 @@

 html, body { color: #242626; }
 html {
-  background-color: red;
+  background-color: blue;
   height: 100%;
   margin-bottom: 1px;
   overflow-y: scroll;
...

Skip to how to read a unified diff

How to read a normal diff

The patch file contains a section for each file that should be changed. That section starts with a line identifying the file. That line has the file name and path of first the old and then the new version in it:

diff violet-park-2.orig/style.css violet-park-2/style.css

Then follow sets of changes, called hunks. The first line of each hunk contains

the line or line range from,to in the file before the changes
a letter indicating a change (c), an addition (a) or a deletion (d)
the line or line range in the file after the changes

all without spaces inbetween.

A change looks like this:

73c73
<   red
---
>   blue

That means, change line 73, which contains red, to blue. The new line is line 73 in the file after all changes.

The new line number could also have been, say, 75. In that case all additions, deletions and changes before that line added up to two more lines in the file after all changes.

Instead of single line numbers, a line range can be specified, for example:

45,65c90,94
...

That means, change lines 45 to 65 in the original file to the following 5 lines, which will be lines 90 to 94 in the new file. The same applies for additions and deletions.

An addition looks like this:

78a80,81
> line1
> line2

That means, in the original file after line 78 add two lines. These will be lines 80 through 81 after all changes.

And finally, a deletion looks like this:

78d77
< line1

That means, delete line 78 in the original file. The line, that preceded line 78 in the original file will be line 77 in the patched file.

How to read a context diff

The patch file contains a section for each file that should be changed. That section starts with three lines that contain the file name and path of first the old and then the new version and some additional information:

diff -c violet-park-2.orig/archive.php violet-park-2/archive.php
*** violet-park-2.orig/archive.php      2009-10-10 07:37:43.000000000 +0200
--- violet-park-2/archive.php   2009-10-10 09:05:58.000000000 +0200

patch is pretty lax about the format of these lines, as long as it can find out the file names. In patches generated by version control systems like cvs, svn or git these three lines look a little different.

Then follow sets of changes, called hunks. Each hunk starts with a line with 15 asterisks. Then comes a line with the line range from,to and the lines from file before the changes. Lines that start with an ! are changed, lines that start with a - are deleted. Then comes the line range and the lines in the file after all changes. Lines that start with a ! are, again, changed, and lines that start with a + are added. Each line modified by the patch is surrounded with 3 lines of context before and after.

A change looks like this:

***************
*** 70,76 ****
  foo
  bar
  baz
! red
  more context
  and more
  still context
--- 70,76 ----
  foo
  bar
  baz
! blue
  more context
  and more
  still context

That means, change line 73 (= 70 + 3 lines of context) in the file before all changes, which contains red to blue. The changed line is also line 73 (= 70 + 3 lines of context) in the file after all changes.

Here’s an example for an addition:

***************
*** 75,80 ****
--- 77,84 ----
  foo
  bar
  baz
+ line1
+ line2
  more context
  and more
  and still context

That means, in the original file after line 78 (= 75 + 3 lines of context) add two lines. These will be lines 80 (= 77 + 3 lines of context) through 81 after all changes.

And that’s how a deletion looks like:

***************
*** 75,81 ****
  foo
  bar
  baz
- line1
  more context
  and more
  and still context
--- 75,80 ----

That means, delete line 78 (= 75 + 3 lines of context) in the original file. The unchanged context will be on lines 75 to 80 after all changes.

How to read a unified diff

diff -u violet-park-2.orig/archive.php violet-park-2/archive.php
--- violet-park-2.orig/archive.php      2009-10-10 07:37:43.000000000 +0200
+++ violet-park-2/archive.php   2009-10-10 09:05:58.000000000 +0200

Then follow sets of changes, called hunks. Each hunk starts with a line that contains, enclosed in @@, the line or line range from,no-of-lines in the file before (with a -) and after (with a +) the changes. After that come the lines from the file. Lines starting with a - are deleted, lines starting with a + are added. Each line modified by the patch is surrounded with 3 lines of context before and after.

An addition looks like this:

@@ -75,6 +77,8 @@
 foo
 bar
 baz
+line1
+line2
 more context
 and more
 and still context

That means, in the original file after line 78 (= 75 + 3 lines of context) add two lines. These will be lines 80 (= 77 + 3 lines of context) through 81 after all changes.

A deletion looks like this:

@@ -75,7 +75,6 @@
 foo
 bar
 baz
-line1
 more context
 and more
 and still context

That means, delete line 78 (= 75 + 3 lines of context) in the original file. The unchanged context will be on lines 75 to 80 after all changes.

Finally, a change looks like this:

@@ -70,7 +70,7 @@
 foo
 bar
 baz
-red
+blue
 more context
 and more
 still context

About running 32 bit programs on 64 bit Ubuntu and shared libraries

Markus Bertheau — Sun, 27 Sep 2009 19:27:37 +0000

Deutsch Русский

Quick answer

To run 32-bit programs on 64-bit installations of Ubuntu, install the package ia32-libs:

markus@ubuntu:~$ sudo apt-get install ia32-libs

Long story

I wrote this article to expand your knowledge and understanding of how Linux works. This knowledge should increase your problem solving skills and speed in the area of server administration and command line use. I show how I adapted an installation shell script of a commercial software package, Flash^® Media Server, which was written for RedHat Linux, to work on 64-bit Ubuntu Linux. For every symptom that occurs in the process, I explain the problem behind it and how to fix it. The end result is available in Installing Flash^® Media Server on Ubuntu Linux.

This article is the first part in a series on the subject. In it I write about running 32 bit programs on 64 bit operating systems and the concept of shared libraries.

The exact versions of the software used in the course are Ubuntu 8.04 LTS 64-Bit Server and Flash^® Media Server 3.5.2, if you want to follow along.

32 bits on 64 bits

When you run the original installer you’ll get an error message about the file fmsini not being found:

markus@ubuntu:~$ tar xfz FlashMediaServer3.5.tar.gz
markus@ubuntu:~$ sudo FMS_3_5_2_r654/installFMS
FMS_3_5_2_r654/installFMS: 172: FMS_3_5_2_r654/fmsini: not found

This error message is quite misleading. The file fmsini exists and it’s a 32 bit binary executable:

markus@ubuntu:~$ file FMS_3_5_2_r654/fmsini 
fmsini: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), for GNU/Linux
2.2.5, dynamically linked (uses shared libs), stripped

Why the error message “not found“?

The message does not refer to the file fmsini. Instead it refers to a helper program that’s needed to run the 32 bit dynamically linked executable fmsini.

Static and dynamic linkage

There are two types of binary executables: statically linked and dynamically linked ones. First about the statically linked ones: When a program wants to call a library function, it refers to it by name. When building the program from source, all library functions used in the program are copied from the library into the program. The program then contains its own code as well as the code of the library functions it uses. Then in the calling places the name is changed to the address of the corresponding function in the program. This process is called linking because it links together the name of a function with the function itself, its implementation. It’s called static, because the link cannot be changed after the program has been built.

Dynamically linked programs work differently: The program also refers to library functions by name. When building the program, two lists are assembled and stored together with the program: a list of which library functions are used in which places, and a list of the libraries that contain the functions used by the program. That’s all for building the program.

Later, at execution time, a special helper program, the so-called dynamic linker, looks in specific places in the file system for each library on the library list and loads it into memory. Now the dynamic linker knows at what memory addresses the library functions are available. It uses the first list to write the correct address in all places that call library functions. Then the dynamically linked program can be run.

Advantages and disadvantages

The biggest disadvantages of statically linked executables stem from the library sharing potential that is not realized. First, programs are bigger and require more space on the hard disk and in memory. That additional space often contains the same library functions over and over again, because they are used in a lot of programs. Second, security or bug fix updates of libraries require the program to be rebuilt from source and redistributed, which can be very costly.

Dynamically linked executables solve these problems: Shared libraries exist in one copy on the hard disk and in one copy in memory. The programs itself are smaller because they don’t contain library code. Libraries can be updated independently of the programs that use them. The programs themselves don’t have to be altered, just restarted.

They solve these problems at a price, though: More infrastructure is needed to support shared libraries: the libraries itself in the right version, and a more sophisticated program loader. In some scenarios this price is too high, and statically linked executables are used. They only need themselves to run and don’t have any dependencies or library compatibility problems.

The dynamic linker

The dynamic linker helper program has a manual page: ld.so(8). ld.so was the name of the dynamic linker in Linux before 1995/96. Now, for 32 bit, the dynamic linker is in the file /lib/ld-linux.so.2; for 64 bit it’s in /lib64/ld-linux-x86-64.so.2. The manual page still has the old name, though.

64 bit Ubuntu Linux has the ability built in to execute 32 bit statically linked programs. The dynamic linker for 32 bit programs though is not installed by default. It’s in the package libc6-i386. After installing that package the file /lib/ld-linux.so.2 exists, and the first step to executing 32 bit dynamically linked executables is done.

Shared libraries

After installing libc6-i386, you’ll get another error message when trying to execute fmsini:

markus@ubuntu:~$ FMS_3_5_2_r654/fmsini
FMS_3_5_2_r654/fmsini: error while loading shared libraries: libstdc++.so.6:
cannot open shared object file: No such file or directory

This is the dynamic linker, trying to load all the libraries in the list and not finding one of them.

Libraries used by dynamically linked executables are called shared libraries. When the dynamic linker finds that a certain library is already loaded, it can refer the new program to use that one instead of loading it again. That way, a library loaded one time is shared among several programs.

Shared libraries come in files that have a .so in their name. so stands for shared object. “Object” just means compiled, binary code.

ldd(1) shows what shared libraries a program needs and where the dynamic linker found these libraries, if it did:

markus@ubuntu:~$ ldd FMS_3_5_2_r654/fmsini
	linux-gate.so.1 =>  (0xf7fc8000)
	libpthread.so.0 => /lib32/libpthread.so.0 (0xf7fa0000)
	libdl.so.2 => /lib32/libdl.so.2 (0xf7f9c000)
	libstdc++.so.6 => not found
	libm.so.6 => /lib32/libm.so.6 (0xf7f75000)
	libgcc_s.so.1 => not found
	libc.so.6 => /lib32/libc.so.6 (0xf7e12000)
	/lib/ld-linux.so.2 (0xf7fc9000)

The dynamic linker looks for the libraries in a number of places – the manual page has all the details on that. The two missing libraries are in the packages lib32stdc++6 and lib32gcc1 respectively.

ldd, by the way, is just a shell script that calls the dynamic linker. Unless told otherwise, the dynamic linker loads all libraries and executes the program. ldd calls it with arguments that tell it to print information about the shared libraries needed by the program.

The majority of programs today are dynamically linked and use shared libraries. The Ubuntu installation comes with over 400 64 bit shared libraries. There’s a package ia32-libs in Ubuntu’s universe repository that contains the most commonly used shared libraries in 32 bit versions. Installing that package will allow many 32 bit programs to run on 64 bit Ubuntu. In particular, it allows the installer of Flash^® Media Server to run (but for other reasons it doesn’t function well yet). The Flash^® Media Server itself though needs libraries that aren’t in that package. How to solve that will be the topic of a later part in that series.

Feedback

Feedback is extremely important to me because it gives me clues about what I need to do differently and what to retain in order to reach my goal. My goal is to have this article be interesting and of use to you, as described in the first paragraph. Please don’t hesitate to leave a comment on which passages are written in a way that hinders comprehension, what should have been left out, or what is missing. I’d also like to hear from you if there’s something you liked especially about this article. Thank you.