The Index, Push, Fetch, Merge and Rebase

39.0 What this chapter gives you#

1. You will be able to name the three places a file can live in git, working directory, index and object database, and say which command moves content between which pair.
2. You will be able to open .git/index, read its header by hand, and explain why it is a full snapshot with stat data and not a list of differences.
3. You will be able to prove that git add writes a real object to disk immediately, before any commit exists, and find that object yourself.
4. You will be able to describe the three steps git commit performs, write a tree, make a commit, move the branch ref, and do all three by hand with plumbing commands.
5. You will be able to choose correctly between git restore, git reset in its three modes, git revert, git clean and git stash, and say for each one whether it can lose work.
6. You will be able to describe exactly what crosses the network on a push: capability advertisement, have and want negotiation, one packfile, pkt-line framing, and three sideband channels.
7. You will be able to explain the reader’s own failure, send-pack: unexpected disconnect while reading sideband packet, say which stage it happened at, and say why it tells you nothing about whether the write landed.
8. You will be able to state the rule for non-idempotent operations over unreliable transports, and apply it beyond git.
9. You will be able to explain merge base, three-way merge, fast-forward, conflict markers, and why a clean text merge can still produce broken code.
10. You will be able to explain why rebase changes a branch’s SHA even though the changes are identical, and decide when rebasing is safe.

39.1 The three states and three places#

PLAIN39.1.1 in simple words#

1. A file that git is tracking exists in three different places at once.
2. The first place is the working directory. That is the ordinary folder on your disk, holding the files you open and edit.
3. The second place is the index, also called the staging area. It holds the exact version of every file that your next commit will contain.
4. The third place is the object database, inside .git/objects. That is the permanent store of every version git has ever been told to keep.
5. A fourth thing sits alongside these three, called HEAD. It is a pointer saying which commit you currently have checked out.
6. The three copies of a file are often identical. When they are, git says the working tree is clean and prints nothing interesting.
7. When they differ, git tells you which pair differs. That is the whole of git status.
8. “Changes to be committed” means index differs from HEAD.
9. “Changes not staged for commit” means working directory differs from index.
10. “Untracked files” means the file is in the working directory and in neither of the other two places.

PLAIN39.1.2 a picture in your head#

1. Think of posting a parcel at a counter.
2. Your desk at home is the working directory. Things lie around, half sorted, in whatever state you left them.
3. The counter at the post office is the index. You put on the counter only the items you actually want to send.
4. You can put one item on the counter, go home, change your mind about a second item, and come back. The counter keeps what you already placed.
5. The sealed and stamped parcel is the commit. Once sealed, its contents are fixed forever.
6. The archive room behind the counter is the object database. Every parcel ever sealed is stored there.
7. The clerk’s note saying which parcel you are currently working from is HEAD.

Where this comparison breaks:

8. At a real post office the item leaves your desk when you put it on the counter. In git, staging a file copies it. Your working file stays put.
9. A real counter holds items. The git index holds a complete list of every tracked file, not only the changed ones.
10. A real parcel is opened by the receiver. A git commit is never opened and changed. It is replaced by a new one.

PLAIN39.1.3 a worked example#

1. Here is a real repository with one change of every kind in it, and its real git status output.

$ git status
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    new file:   TODO.md
    modified:   calc.py
    deleted:    notes.txt

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes)
    modified:   README.md

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    out.log

2. Map every line onto the three places.
3. new file: TODO.md. Not in HEAD. Present in the index. So index differs from HEAD, and the difference is an addition.
4. modified: calc.py. In HEAD with one content, in the index with another.
5. deleted: notes.txt. Present in HEAD, absent from the index.
6. modified: README.md under “not staged”. Index and HEAD agree. The working file differs from both.
7. out.log under “Untracked”. Not in HEAD, not in the index, only on disk.
8. The same state in short form, which is easier to read once you know it.

$ git status --short --branch
## main
 M README.md
A  TODO.md
M  calc.py
D  notes.txt
?? out.log

9. The two columns are index-versus-HEAD first, then working-versus-index.
10. So M means the index is clean and the working file changed. M means the index changed and the working file matches the index.

PLAIN39.1.4 what is really happening inside#

1. Here is the picture. The arrows are commands, and each one moves content in one direction between two of the three places.

   +------------------+   +------------------+   +----------------+
   | working directory|   | index / staging  |   | object database|
   | the files you    |   | .git/index       |   | .git/objects   |
   | actually edit    |   | one binary file  |   | blobs, trees,  |
   |                  |   |                  |   | commits        |
   +------------------+   +------------------+   +----------------+
            |                      |                      |
            |---- git add -------->|                      |
            |                      |---- git commit ----->|
            |                      |                      |
            |<--- git restore -----|                      |
            |     <file>           |                      |
            |                      |<-- git restore ------|
            |                      |    --staged <file>   |
            |                      |<-- git reset --mixed |
            |<--- git checkout <commit> -- <file> --------|
            |<--- git reset --hard <commit> --------------|
                                   ^
                                   |
                            HEAD -> refs/heads/main -> a commit

2. git add reads a working file, stores its content in the object database, and records the resulting name in the index.
3. That is worth reading twice. git add already writes to the object database. The index only records the name of what was written.
4. git commit reads the index, builds tree objects from it, builds one commit object, and moves the branch that HEAD points at.
5. git restore <file> copies from the index back over the working file.
6. git restore --staged <file> copies from HEAD back into the index.
7. git reset --hard <commit> moves HEAD and the branch, then overwrites both the index and the working directory from that commit.
8. Nothing in this picture ever edits a stored object. Objects are written once and never modified.

TECHNICAL39.1.5 the engineer’s version#

1. The three states are formally: modified, staged, committed. The three storage locations are the working tree, the index, and the object store.
2. The index is a single binary file at $GIT_DIR/index, by default .git/index. Its location can be overridden by GIT_INDEX_FILE.
3. git status is implemented as two diffs: diff-index --cached HEAD for the staged column and diff-files for the unstaged column, plus a directory walk for untracked paths.
4. The machine-readable form is git status --porcelain=v2, stable across versions and intended for scripts. Here is the same repository state.

$ git status --porcelain=v2 --branch
# branch.oid 66b84b4f79e5cce670a6a2c0f739d5433403423e
# branch.head main
1 .M N... 100644 100644 100644 1af06666... 1af06666... README.md
1 A. N... 000000 100644 100644 00000000... 1333ed77... TODO.md
1 M. N... 100644 100644 100644 4693ad3c... 6cddd28b... calc.py
1 D. N... 100644 000000 000000 fb188b9e... 00000000... notes.txt
? out.log

5. Columns after the XY code are: submodule state, mode in HEAD, mode in index, mode in worktree, object name in HEAD, object name in index, path. Hashes are abbreviated here to fit the page; the real output prints all 40 digits.
6. Note TODO.md has HEAD mode 000000 and HEAD hash all zeros. That is how git encodes “not present in HEAD”.

7. Commands that observe this: git status, git diff (worktree vs index), git diff --cached (index vs HEAD), git diff HEAD (worktree vs HEAD), git ls-files -s, git diff-index, git diff-files.

WORDS39.1.6 remember these#

1. Working directory — the files you edit — the checked-out worktree on the filesystem, outside .git.
2. Index — the counter where you put things to send — the binary staging file .git/index holding the next commit’s full snapshot.
3. Object database — the archive of everything kept — .git/objects, holding loose and packed blob, tree, commit and tag objects.
4. HEAD — the note saying where you are — a symbolic ref, usually pointing at a branch ref under refs/heads/.
5. Staged — placed on the counter — present in the index with content differing from HEAD.
6. Untracked — git has never been told about it — a path in the worktree with no index entry and no HEAD entry.

39.2 What the index actually is#

PLAIN39.2.1 in simple words#

1. The index is not an idea. It is a real file on your disk, and you can look at it.
2. It is at .git/index. It is binary, so opening it in a text editor shows rubbish, but its structure is documented and simple.
3. It holds one entry per tracked file. Not one entry per changed file. Every single tracked file, always.
4. Each entry stores the file’s path, the name of its stored content, its permission bits, and a copy of what the operating system said about the file the last time git looked.
5. That last part is the clever bit. Git remembers the file’s size, its last modified time, and its position on disk.
6. Next time you run git status, git asks the operating system for those same facts again. If they match, git assumes the file did not change and does not read it at all.
7. That is why git status is fast even in a project with tens of thousands of files. Most files are never opened.
8. The index stores whole snapshots, not differences. There is no “diff” anywhere in it.

PLAIN39.2.2 a picture in your head#

1. Think of a hotel cloakroom with a big register book.
2. Every guest’s coat has a ticket. The register lists, for each coat, the ticket number, the peg it hangs on, its colour, and its weight.
3. When you come back, the clerk does not take every coat down and inspect it.
4. The clerk glances at the register and at the pegs. If the weights and colours still match, nothing has changed and the clerk says so instantly.
5. Only if something looks different does the clerk take a coat down and check it properly.
6. The register is the index. The weights and colours are the stat data. The coats themselves are the stored objects.

Where this comparison breaks:

7. A clerk could be fooled by two coats of identical weight and colour. Git can be fooled too, and this is real: if a file changes within the same second and keeps the same size, git can miss it. Git has a “racy timestamp” rule to handle that case, described later in this section.
8. A cloakroom register lists only coats currently held. The git index lists every tracked file, even ones you have not touched in years.

PLAIN39.2.3 a worked example#

1. Here is a small repository with five tracked files. First its size, then its contents.

$ ls -l .git/index
-rw-r--r-- 1 root root 479 .git/index

$ git ls-files -s
100644 1af06666...9965 0    README.md
100644 1333ed77...ee69 0    TODO.md
100644 6cddd28b...7159 0    calc.py
100644 b04e7afa...7b19 0    docs/guide.md
100644 11b15b1a...a0d5 0    src/main.py

2. The columns are: file mode, object name of the content, stage number, path. Stage 0 means “not conflicted”. Hashes shortened here to fit the page.
3. Note docs/guide.md and src/main.py. The index has no directories. It has flat paths with slashes in them.
4. Now the stat cache. The --debug flag prints it.

$ git ls-files --debug | head -7
README.md
  ctime: 1786606646:156685960
  mtime: 1786606646:156685960
  dev: 65024    ino: 1049252
  uid: 0    gid: 0
  size: 13  flags: 0
TODO.md

5. mtime is when the content last changed. ctime is when the file’s metadata last changed. ino is the inode number, the file’s identity on the filesystem. size is 13 bytes.
6. Every one of those numbers came from a single stat call to the operating system, which is far cheaper than reading and hashing the file.
7. Here is the header of that binary file, byte by byte.

$ head -c 16 .git/index | od -A d -t x1z
0000000 44 49 52 43 00 00 00 02 00 00 00 05  >DIRC........<

8. 44 49 52 43 is the ASCII letters DIRC. Then 00 00 00 02 is version 2. Then 00 00 00 05 is five entries.

PLAIN39.2.4 what is really happening inside#

1. The file has three parts: a 12-byte header, then the entries, then optional extra sections, then a checksum at the end.
2. Each entry begins with 40 bytes of numbers copied from the operating system’s stat call: two timestamps with nanoseconds, device, inode, mode, user, group and size.
3. Then 20 bytes holding the content’s SHA-1 name. Then 2 bytes of flags, including the path length and the merge stage.
4. Then the path itself, followed by at least one zero byte, padded so the whole entry length is a multiple of eight.
5. The entries are sorted by path, as raw bytes. That sorting is what lets git find an entry by binary search instead of scanning.
6. Let us add it up for the five-file repository above. Fixed part is 62 bytes. README.md is 9 characters, so 62 plus 9 is 71, padded up to 72.
7. Doing that for all five paths gives 72, 72, 72, 80 and 80, which is 376.
8. Add the 12-byte header and the 20-byte trailing checksum and we have 408. The file is 479. The missing 71 bytes are one extension section.
9. That section is the cached tree, signature TREE, 63 bytes of data plus its own 8-byte header. It remembers tree objects already computed, so a commit does not have to rebuild trees for unchanged directories.
10. When git status runs, for each entry git calls lstat on the path. If size, mtime, ctime and inode all match, git marks the file unchanged without reading a single byte of it.

TECHNICAL39.2.5 the engineer’s version#

1. The format is specified in the git documentation page gitformat-index. The signature is the four bytes DIRC, standing for “directory cache”.
2. Supported versions are 2, 3 and 4. Version 3 adds a second flags word for skip-worktree and intent-to-add. Version 4 prefix-compresses path names.
3. All multi-byte numbers are big-endian (“network byte order”). The trailing checksum is a hash over everything before it.

4. Here is that exact five-entry index parsed by a short script, showing the real numbers.

total size: 479
signature: DIRC  version: 2  entries: 5
  README.md      mode=100644 size= 13 stage=0 bytes=72
  TODO.md        mode=100644 size=  5 stage=0 bytes=72
  calc.py        mode=100644 size= 65 stage=0 bytes=72
  docs/guide.md  mode=100644 size= 12 stage=0 bytes=80
  src/main.py    mode=100644 size= 15 stage=0 bytes=80
offset after entries: 388
extension: TREE size: 63

5. Only two file modes are legal for regular files, 100644 and 100755. 120000 is a symbolic link and 160000 is a gitlink (a submodule).
6. The stat cache is a cache, so it can be wrong. The known hazard is the “racy index” problem: a file written in the same second as the index itself may have a matching mtime yet different content. Git handles this by treating any entry whose mtime is not strictly older than the index’s own mtime as suspect, and re-reading it.
7. Real measurements, taken on a Linux sandbox with git 2.43.0, on a repository with 10,000 tracked files totalling 2,850,000 bytes:

8. The index for those 10,000 entries was 801,570 bytes, about 80 bytes per entry. The honest version: this repository is tiny, so the gap is only about four times. On a repository with large files the gap is far larger, because the cost of re-reading grows with total bytes while the stat calls do not.
9. Since git 2.36, released 18 April 2022, git ships a built-in filesystem monitor daemon, enabled with git config core.fsmonitor true. It replaces the per-file lstat scan with change notifications from the operating system, and matters most on macOS and Windows.
10. Sparse-checkout with a sparse index, extensions.sparseIndex, allows directory entries in the index with mode 040000, so that a huge repository does not need an index entry per file.

WORDS39.2.6 remember these#

1. Index entry — one line in the register — a 62-byte fixed record plus a padded path, holding stat data, OID, mode and flags.
2. Stat data — what the filesystem says about a file — st_mtime, st_ctime, st_dev, st_ino, st_size and mode from lstat(2).
3. Stat cache — remembering that so you need not re-read — the mechanism that makes git status skip unchanged files.
4. DIRC — the four letters at the start — the index file signature, short for “directory cache”, present since git’s first commit in April 2005.
5. Stage number — which side of a conflict — 0 for normal, 1 for merge base, 2 for ours, 3 for theirs.
6. Cached tree — the remembered directory hashes — the TREE index extension that lets git write-tree skip unchanged directories.

39.3 What git add really does#

PLAIN39.3.1 in simple words#

1. Most people believe git add marks a file for later. It does much more than that, and it does it at once.
2. When you run git add, git reads the file, compresses its content, works out its name from the content, and writes that content into .git/objects straight away.
3. Then it writes an entry in the index recording that name against the path.
4. So the content is already safely stored before you type git commit. The commit does not copy your files; the copy has already happened.
5. This is why staging a file and then editing it again gives you three different versions at once: the committed one, the staged one, and the one on disk.
6. It also means you can stage part of a file. Git can build a version that exists nowhere on disk, store it, and put that in the index.
7. That partial staging is done with git add -p, which walks you through each chunk of change and asks yes or no.

PLAIN39.3.2 a picture in your head#

1. Imagine a photocopier next to your desk that also files things.
2. When you say “add this page”, the machine does not put a sticky note on your page. It photocopies it immediately and files the copy in a cabinet.
3. It then writes in a ledger: “for the page called report.txt, use copy number eb2fc3ca”.
4. You can now scribble all over your original. The copy in the cabinet is untouched.
5. If you later say “add this page” again, a new copy is made and the ledger line is updated to point at the new copy. The old copy stays in the cabinet, unreferenced.

Where this comparison breaks:

6. A photocopier makes a new copy every time. Git checks first: if a copy with that exact content already exists, it stores nothing and reuses it.
7. The ledger in this picture holds one line per page you added. The real index holds one line per tracked file, added or not.

PLAIN39.3.3 a worked example#

1. Start a completely empty repository and prove that git add writes to .git/objects before any commit exists.

$ git init -q .
$ find .git/objects -type f
(nothing)

$ printf 'hello index\n' > greeting.txt
$ git hash-object greeting.txt
eb2fc3ca2f129a710df1a6c0fd5ebfd088a10bfd

$ git add greeting.txt

$ find .git/objects -type f
.git/objects/eb/2fc3ca2f129a710df1a6c0fd5ebfd088a10bfd

$ git log --oneline
fatal: your current branch 'main' does not have any commits yet

$ git ls-files -s
100644 eb2fc3ca2f...bfd 0   greeting.txt

2. Read that carefully. There is no commit. There is no branch tip. Yet the content is already a permanent object in the database.
3. git hash-object printed the name before the add, without writing anything. It is the same name, because the name is computed from the content.
4. Now the three-version demonstration. Take a file with two separate changes and stage only the first.

$ git add -p
@@ -1,6 +1,6 @@
 import os

-PORT = 8080
+PORT = 9090
 DEBUG = False

(1/2) Stage this hunk [y,n,q,a,d,j,J,g,/,e,?]? y
@@ -8,3 +8,4 @@ def start():
 def stop():
     print("stopping")
+    print("goodbye")
(2/2) Stage this hunk [y,n,q,a,d,K,g,/,e,?]? n

5. Now ask git for the name of server.py in all three places.

committed  : 6802ebd3762732a47322b13ed2d9510a9145a23f
index      : 345790221885e55f73d7e6c21d54fffc51998330
working dir: 154a7a3ee775a1746b842017a179c42cee649a30

6. Three different names, so three different contents. The middle one exists nowhere on your disk as a file. It is a real object in .git/objects.

PLAIN39.3.4 what is really happening inside#

1. git add path performs these steps, in order, for each path.
2. Call lstat on the path to get size, mode and timestamps.
3. Read the file’s bytes. Apply any content filters that are configured, for example line-ending conversion or a clean filter.
4. Build the object header: the word blob, a space, the byte count in decimal, and one zero byte. Put the content after it.
5. Compute SHA-1 over header plus content. That 40-character result is the object name.
6. Check whether an object with that name already exists. If it does, do nothing more; the content is already stored.
7. If it does not, compress the header-plus-content with zlib and write it to .git/objects/ab/cdef..., where ab is the first two characters.
8. Write or update the index entry: path, the new object name, the mode, and the freshly captured stat data.
9. Write the whole index back out, atomically, by writing .git/index.lock and renaming it over .git/index.
10. For git add -p, git first computes the diff, splits it into hunks, and for each accepted hunk applies it to a copy of the staged version. The result of applying the accepted subset is what gets hashed and stored.

TECHNICAL39.3.5 the engineer’s version#

1. The plumbing equivalents are git update-index --add --cacheinfo and git hash-object -w. The porcelain git add is a wrapper over both.
2. git hash-object without -w computes and prints the OID without writing. With -w it writes the loose object.
3. Loose objects are zlib-compressed with the default compression level set by core.compression, default -1 meaning zlib’s own default of 6.
4. Useful git add variants:

5. git add -N sets the intent-to-add flag, which requires index version 3. It makes an untracked file appear in git diff output without staging content.
6. In git add -p, the keys are: y stage, n skip, q quit, a stage this and all remaining in the file, d skip this and all remaining in the file, s split the hunk, e edit the hunk, j/k move without deciding.
7. git add --interactive was written by Junio C Hamano and merged on 18 December 2006; it shipped in git 1.5.0 on 14 February 2007.
8. Filters that change content on the way in are set by gitattributes: text and eol for line endings, and filter.<name>.clean for custom transforms such as Git LFS. The honest version: with a clean filter configured, the blob stored is not the bytes on your disk.
9. To see what would be staged without staging it, use git diff for unstaged changes and git add --dry-run -A.

WORDS39.3.6 remember these#

1. Blob — the stored content of one file — a git object of type blob, containing raw bytes with no filename.
2. Hunk — one chunk of change in a diff — a @@ section with its context lines, the unit git add -p operates on.
3. Loose object — one file per object — an object stored individually under .git/objects/xx/, as opposed to inside a packfile.
4. Intent to add — “this file is coming” — the index flag set by git add -N, registering a path with no content.
5. Clean filter — a transform applied on the way in — filter.<name>.clean from gitattributes, run before hashing.
6. Atomic index write — no half-written index — the lock-file-and-rename sequence git uses for .git/index.

39.4 What git commit really does#

PLAIN39.4.1 in simple words#

1. git commit does exactly three things, in this order.
2. First, it turns the index into tree objects. One tree for the top folder, and one for each subfolder, each listing names, modes and object names.
3. Second, it creates one commit object. That object holds the name of the top tree, the name of the commit you were on, who you are, when it is, and your message.
4. Third, it changes one small text file, the branch file, so it holds the new commit’s name.
5. That is all. No files are copied. The blobs were already written when you ran git add.
6. Because the branch file is the last thing to change, the commit is either fully there or not there at all.
7. If a step fails halfway, the objects written so far are simply unreferenced and will be cleaned up later. Nothing is corrupted.

PLAIN39.4.2 a picture in your head#

1. Think of publishing an edition of a newspaper.
2. All the articles are already typeset and stored, one file each. That happened when you staged them.
3. Making the edition means writing a contents page listing which article file goes on which page. That is the tree.
4. Then you write a cover sheet: edition number, date, editor’s name, a sentence about what is in it, and the number of the previous edition. That is the commit.
5. Then you update the single sign on the wall that says “current edition”. That is the branch file.

Where this comparison breaks:

6. A newspaper’s contents page is written fresh each time. Git reuses tree objects for any folder whose contents did not change at all.
7. A newspaper edition can be recalled and reprinted. A commit object can never be edited. git commit --amend makes a completely new one.

PLAIN39.4.3 a worked example#

1. Here is a commit done by hand, one plumbing command per step, so you can see the three stages separately.

$ git add greeting.txt

$ git write-tree
14a7508225642503557d7347c657e485225656a1

$ git commit-tree 14a75082... -p HEAD -m "second line"
d5d96f0f76b0571dd21802be5196808779e4b9ba

$ git update-ref refs/heads/main d5d96f0f...

$ git log --oneline
d5d96f0 second line
8c96e11 add greeting

2. Step one made a tree from the index. Step two made a commit pointing at that tree and at the old HEAD. Step three moved the branch.
3. git commit is those three commands with a message editor bolted on.
4. Now watch the three objects appear for a first commit in an empty repository. Before the commit there is one object, the blob.

before commit:
  eb2fc3ca...  (blob)

after commit:
  856b2713...  (tree)
  8c96e114...  (commit)
  eb2fc3ca...  (blob)

5. The commit object itself is plain text. Here it is.

$ git cat-file -p HEAD
tree 856b271361271d5c57e0ac557a10cc6961ea6806
author Reader <reader@example.com> 1772429400 +0530
committer Reader <reader@example.com> 1772429400 +0530

add greeting

6. And the branch file went from not existing at all to holding 40 characters plus a newline.

PLAIN39.4.4 what is really happening inside#

1. Step one, write-tree, walks the index in sorted path order.
2. Whenever the path prefix changes, it knows a directory ended, so it builds the tree object for that directory and adds it as an entry in its parent.
3. If the cached tree extension says a directory is unchanged, git skips all of that and reuses the remembered tree name.
4. Step two builds the commit’s bytes: the word tree and the tree name, then one parent line per parent, then author, then committer, then a blank line, then the message.
5. That text is hashed the same way a blob is, with a header saying commit and the byte count, and stored.
6. Step three writes the new commit’s name into .git/refs/heads/<branch> and appends a line to .git/logs/HEAD and .git/logs/refs/heads/<branch>. Those log files are the reflog.
7. If HEAD is detached, meaning it holds a commit name directly rather than a branch name, step three writes to .git/HEAD instead.
8. Why does the index exist at all? Because it lets you decide what goes in a commit separately from what is on your disk.
9. Without it, either every commit contains everything you have changed, or the tool must ask you a list of files every single time.
10. The honest counter-argument: other systems manage without it. Mercurial has no staging area by default and uses hg commit --interactive for the same job. Subversion and Perforce have change lists instead. Fossil and Darcs ask about patches at commit time. So the index is a design choice, not a necessity, and reasonable people call it an unnecessary extra concept.

TECHNICAL39.4.5 the engineer’s version#

1. The three plumbing commands are git write-tree, git commit-tree and git update-ref. Every porcelain commit path goes through equivalents.
2. git commit --amend does not modify the current commit. It builds a new commit with the same parents and moves the branch. The old commit remains reachable only through the reflog.
3. The commit object’s canonical byte layout, in order: tree <oid>, zero or more parent <oid>, author <name> <email> <unix-ts> <tz>, committer ..., optional headers such as gpgsig, a blank line, message.
4. Author and committer differ whenever a commit is replayed. Rebase, cherry pick and git am all keep the author and rewrite the committer.
5. Ref updates go through the ref transaction machinery. With the files backend this is a .lock file plus rename per ref. An alternative reftable backend was integrated in git 2.45, released April 2024, and is selected at creation time with git init --ref-format=reftable.
6. Commit-related knobs worth knowing:

7. Hooks that run: pre-commit, prepare-commit-msg, commit-msg, post-commit. A non-zero exit from the first three aborts the commit.
8. Historical note: the index was in git’s very first commit. On 7 April 2005 Linus Torvalds committed cache.h, containing #define CACHE_SIGNATURE 0x44495243 /* "DIRC" */, and the object store then lived in .dircache/objects. The tools were called update-cache, write-tree and commit-tree. The names changed; the design did not.

WORDS39.4.6 remember these#

1. Tree object — a folder listing — a git object mapping names to modes and object IDs, one per directory.
2. Commit object — the cover sheet — a git object naming one tree, zero or more parents, author, committer and message.
3. Ref — the sign on the wall — a file under .git/refs (or a line in packed-refs) holding an object name.
4. Amend — replace the top edition — create a new commit with the old commit’s parents and move the branch to it.
5. Committer versus author — who applied it versus who wrote it — two separate identity and timestamp fields in every commit.
6. Reflog — the local diary of ref movements — append-only files under .git/logs, local only, never pushed.

39.5 Undoing things, laid out clearly#

PLAIN39.5.1 in simple words#

1. Git has many undo commands because there are many different things to undo.
2. The trick is to ask: which of the three places do I want to change?
3. git restore <file> changes the working directory only. It copies the staged version over your file, throwing your edits away.
4. git restore --staged <file> changes the index only. It copies the HEAD version into the index, so the file is no longer staged.
5. git reset --soft <commit> moves HEAD and the branch only. Index and working directory are untouched.
6. git reset --mixed <commit>, which is the default, moves HEAD and the branch and resets the index. Working directory untouched.
7. git reset --hard <commit> moves all three. Your uncommitted work is gone.
8. git revert <commit> changes nothing that already exists. It makes a new commit that undoes an old one. This is the safe undo for shared history.
9. git clean deletes untracked files. Nothing about them was ever stored, so they cannot be recovered.
10. git stash puts your uncommitted changes aside as hidden commits and gives you a clean tree.

PLAIN39.5.2 a picture in your head#

1. Think of a kitchen with a chopping board, a tray, and a fridge.
2. The chopping board is the working directory. The tray is the index. The fridge is the object database.
3. git restore wipes the board and re-copies from the tray.
4. git restore --staged empties the tray and refills it from the fridge.
5. git reset --soft only moves the label saying which meal you are making. Board and tray keep everything.
6. git reset --mixed moves the label and clears the tray.
7. git reset --hard moves the label, clears the tray, and scrapes the board into the bin.
8. git revert does not touch anything. It cooks a new dish that cancels out an old one, and puts that in the fridge too.

Where this comparison breaks:

9. Scraping the board into a bin is final. git reset --hard losing a commit is not final, because the reflog remembers where the branch used to point for 90 days. Losing uncommitted edits, though, really is final.

PLAIN39.5.3 a worked example#

1. Three commits, c1, c2, c3, where the file goes A, then A B, then A B C. Now watch each reset mode from the same starting point.

BASELINE  (at c3)
  HEAD    = c3   index = "A B C"   workdir = "A B C"   status: clean

git reset --soft HEAD~1
  HEAD    = c2   index = "A B C"   workdir = "A B C"
  status: M  f.txt      (staged modification)

git reset --mixed HEAD~1
  HEAD    = c2   index = "A B"     workdir = "A B C"
  status:  M f.txt      (unstaged modification)

git reset --hard HEAD~1
  HEAD    = c2   index = "A B"     workdir = "A B"
  status: clean         (the "C" line is gone)

2. Read the three status lines. Soft leaves the change staged. Mixed leaves it unstaged. Hard destroys it.
3. Now the other four, in one real session.

$ git status --short
A  staged.txt
 M work.txt
?? build.tmp

$ git restore --staged staged.txt
 M work.txt
?? build.tmp
?? staged.txt

$ git restore work.txt
?? build.tmp
?? staged.txt

$ git clean -n
Would remove build.tmp
Would remove staged.txt

$ git clean -f
Removing build.tmp
Removing staged.txt

4. Note the -n dry run. Always run that first. git clean -f deletes files that git has never stored, so there is nothing to recover from.

PLAIN39.5.4 what is really happening inside#

1. Here is the table. Read the middle column as “which of the three places does this touch”.

2. Only two rows in that table can destroy something that git has never stored: git reset --hard and git clean. Everything else is recoverable.
3. git stash is not magic. It makes real commits. One commit records the index, another records the working tree, and a ref called refs/stash points at the second one.

$ git stash list
stash@{0}: On main: half finished idea

$ git log --oneline --graph refs/stash
*   69aa036 On main: half finished idea
|\
| * 77c7641 index on main: b068de2 three hours of important work
|/
* b068de2 three hours of important work

4. Now the recovery guide. Suppose you ran git reset --hard HEAD~1 and threw away a commit.

$ git log --oneline
f74729c base                       <- the good commit is gone

$ git reflog
f74729c HEAD@{0}: reset: moving to HEAD~1
b068de2 HEAD@{1}: commit: three hours of important work
f74729c HEAD@{2}: commit (initial): base

$ git reset --hard 'HEAD@{1}'

$ git log --oneline
b068de2 three hours of important work
f74729c base

5. The reflog is a local diary of every value HEAD and each branch has held. It is written on every commit, checkout, reset, merge, rebase and pull.
6. Because the commit object was never deleted, only unreferenced, pointing a ref at it again brings it straight back.

TECHNICAL39.5.5 the engineer’s version#

1. git reset has a fourth and fifth mode. --merge resets the index and updates worktree files that differ between HEAD and target while keeping unstaged changes. --keep is similar but aborts on conflict.
2. git reset <paths> never moves HEAD. Path form and commit form are different operations sharing a name, which is a known wart. git restore and git switch, introduced in git 2.23 on 16 August 2019, exist to split those roles. They were marked experimental at introduction and remain so in the documentation as of git 2.43.
3. Reflog retention defaults: gc.reflogExpire is 90 days for reachable entries, gc.reflogExpireUnreachable is 30 days. gc.pruneExpire is 2 weeks, so unreachable objects survive at least that long before deletion.
4. Recovery ladder, in the order to try:

5. ORIG_HEAD is written by merge, rebase, reset and pull before they move HEAD. git reset --hard ORIG_HEAD is the fastest undo for those four.
6. Since git 2.36 (April 2022), git branch --recurse-submodules and related plumbing exist, but stash and submodules still interact badly; git stash does not stash submodule changes by default.
7. git stash push --staged, added in git 2.35 (24 January 2022), stashes only what is in the index. git stash -u includes untracked files; git stash -a includes ignored files.
8. git stash was added to git on 30 June 2007 and shipped in git 1.5.3 on 2 September 2007.

WORDS39.5.6 remember these#

1. Soft reset — move the label only — move HEAD and branch, leave index and worktree.
2. Mixed reset — move the label and clear the tray — move HEAD and branch and reset the index; the default mode.
3. Hard reset — move everything and discard — overwrite index and worktree from the target commit.
4. Revert — undo by adding, not removing — create a new commit whose diff is the inverse of an existing commit.
5. ORIG_HEAD — where I was before that big operation — a ref written by merge, rebase, reset and pull.
6. Reflog — the local diary — per-ref append-only history of previous values, kept 90 days by default and never transferred over the network.

39.6 What actually crosses the wire on a push#

PLAIN39.6.1 in simple words#

1. When you push, git does not send files. It sends objects, and it sends only the ones the other side does not already have.
2. The first thing that happens is a conversation. Your git starts a program on the server called receive-pack.
3. The server speaks first. It lists every branch and tag it has, with the object name each one points at, and a list of features it supports.
4. Your git compares that list with what you are pushing. It works out which objects the server is missing.
5. Your git bundles those objects into one compressed file called a packfile, and sends it, together with a line saying “move branch main from this old name to this new name”.
6. The server reads the packfile, checks that nothing is missing or corrupt, then updates the branch.
7. Then the server sends a short report: whether the pack unpacked, and whether each branch update succeeded.
8. All of that is one connection and, for the object transfer, one round trip. Twenty commits cost the same number of round trips as one.

PLAIN39.6.2 a picture in your head#

1. Think of sending documents to a records office by courier.
2. You phone first. The clerk reads out an index of everything already filed, and says which formats they accept.
3. You compare against your own shelf and put only the missing documents into one box. You do not send twenty separate envelopes.
4. On the box you write one instruction slip: “file these, then change the pointer for folder main from document 1c8a3b1 to document 9978003”.
5. The courier carries the box. The clerk unpacks it, checks nothing is missing, files everything, updates the pointer, and phones you back.
6. That phone call back is the report. It is a separate thing from the box.

Where this comparison breaks:

7. A courier can tell you the box arrived. Over a network there is no such guarantee. If the return phone call is cut off, you have no idea whether the clerk filed the documents. That is the whole of section 39.7.

PLAIN39.6.3 a worked example#

1. This is a real push, traced with GIT_TRACE_PACKET=1. Lines are trimmed to fit the page; > means sent and < means received.

receive-pack> 9978003f... refs/heads/main\0report-status
              report-status-v2 delete-refs side-band-64k quiet
              atomic ofs-delta object-format=sha1 agent=git/2.43.0
receive-pack> 0000
push>         9978003f... e60afb23... refs/heads/main\0
              report-status-v2 side-band-64k quiet
              object-format=sha1 agent=git/2.43.0
push>         0000
   [ packfile bytes travel here ]
receive-pack> unpack ok
receive-pack> ok refs/heads/main
receive-pack> 0000
sideband<     \1000eunpack ok0017ok refs/heads/main0000

2. Line one is the advertisement. The server says what it has and what it can do. 0000 ends that list.
3. Line five is the command. Old name, new name, ref name, then the capabilities the client accepts.
4. Then the packfile is streamed.
5. Then the report: unpack ok and ok refs/heads/main.
6. Now look at the last line very carefully, because it shows the raw framing.
7. \1 is the sideband channel number: channel 1, meaning real data.
8. 000e is hexadecimal 14: this packet is 14 bytes long including its own 4-byte length. So 10 bytes of payload: unpack ok plus a newline.
9. 0017 is hexadecimal 23: 19 bytes of payload, ok refs/heads/main plus a newline.
10. 0000 is the flush packet, meaning “end of this stream”.
11. Now the round-trip claim, measured. Twenty commits, each changing one file.

$ git rev-list --count origin/main..main
20
$ git rev-list --objects origin/main..main | wc -l
60
$ git push --progress origin main
Enumerating objects: 62, done.
Counting objects: 100% (62/62), done.
Delta compression using up to 2 threads
Compressing objects: 100% (20/20), done.
Writing objects: 100% (60/60), 3.95 KiB | 674.00 KiB/s, done.
Total 60 (delta 0), reused 0 (delta 0), pack-reused 0
To file:///tmp/gw/srv.git
   1c8a3b1..9978003  main -> main

12. Twenty commits became sixty objects, twenty commits plus twenty trees plus twenty blobs, in one packfile of 3.95 KiB, in one exchange.

PLAIN39.6.4 what is really happening inside#

1. Everything on the wire is wrapped in a frame called a pkt-line.
2. A pkt-line is four hexadecimal characters giving the total length, then that many bytes minus four of payload.
3. So 0009hello is a complete packet: length 9, payload hello which is 5 bytes, plus the 4 length characters.
4. Three lengths are special. 0000 means flush, end of a section. 0001 is a delimiter used in the newer protocol. 0002 marks the end of a response.
5. On top of pkt-line sits the sideband. When both ends agree to it, the first byte of each packet’s payload is a channel number.
6. Channel 1 is real data, the packfile or the report. Channel 2 is progress text, which your git prints with the prefix remote:. Channel 3 is a fatal error, after which the stream stops.
7. This is why, on a real clone, you see lines beginning with remote:. Those came over channel 2.

$ git clone https://github.com/octocat/Hello-World.git
Cloning into 'Hello-World'...
remote: Enumerating objects: 13, done.
remote: Total 13 (delta 0), reused 0, pack-reused 13 (from 1)
Receiving objects: 100% (13/13), done.

8. remote: Enumerating objects is the server talking on channel 2. Receiving objects is your own git counting what it reads from channel 1.
9. For fetching, the two sides negotiate. The client says want <oid> for what it needs and have <oid> for tips it already holds. The server replies ACK for the ones it recognises.
10. That is how the server learns the common ancestor without either side sending the whole history.

TECHNICAL39.6.5 the engineer’s version#

1. Transports: SSH runs git-upload-pack or git-receive-pack as a remote command on port 22. Smart HTTP uses two requests on port 443. The git:// daemon protocol uses port 9418 and has no authentication or encryption.
2. Real smart-HTTP trace against github.com, taken with GIT_TRACE_CURL=1, trimmed to the git-relevant headers.

=> GET /octocat/Hello-World.git/info/refs?service=git-upload-pack
=> User-Agent: git/2.43.0
=> Git-Protocol: version=2
<= HTTP/1.1 200 OK
<= Content-Type: application/x-git-upload-pack-advertisement

=> POST /octocat/Hello-World.git/git-upload-pack
=> Content-Type: application/x-git-upload-pack-request
=> Git-Protocol: version=2
<= HTTP/1.1 200 OK
<= Content-Type: application/x-git-upload-pack-result
<= Transfer-Encoding: chunked

3. The GET fetches the advertisement, the POST carries the negotiation and receives the packfile. Push uses the same shape with git-receive-pack.
4. Real negotiation for a fetch, over protocol v2, trimmed:

fetch> command=fetch
fetch> thin-pack
fetch> no-progress
fetch> ofs-delta
fetch> want fec0e27c2650e36592a05049f34cae60fc2966c0
fetch> have 9f1a39e109aca3fc1cce3033a70a43553ebd2bb1
fetch> have f27fb125fe41194710c653bb5b43870a725be980
fetch> 0000
fetch< acknowledgments
fetch< ready
fetch< 0001
fetch< packfile

5. Exact framing limits, from the gitprotocol-common specification: maximum pkt-line data is 65,516 bytes; maximum total packet length is 65,520 bytes.
6. Sideband limits, from gitprotocol-capabilities: side-band allows up to 1,000 bytes per packet, 999 of payload plus one band byte. side-band-64k allows up to 65,520 bytes, 65,519 of payload plus one band byte.

7. Protocol v2 was introduced in git 2.18 on 21 June 2018, made default in git 2.26 on 22 March 2020, demoted again in 2.27 after bugs, and made default once more in git 2.29 on 19 October 2020.
8. Push still uses the v0-style advertisement in the trace above, taken with git 2.43.0. Since git 2.32 (6 June 2021), push.negotiate=true lets push use v2 negotiation to avoid sending objects the server already has.
9. Smart HTTP itself dates from git 1.6.6, released 23 December 2009, written largely by Shawn O. Pearce. Before that, HTTP fetch was a “dumb” walker that downloaded loose objects one at a time.
10. Observation tools: GIT_TRACE_PACKET=1, GIT_TRACE=1, GIT_TRACE_CURL=1 with GIT_TRACE_CURL_NO_DATA=1, GIT_SSH_COMMAND with ssh -v, and git ls-remote for a pure read of the server’s refs.
11. Be clear about what kind of rule each of these is. The pkt-line framing, the three sideband channels, the capability names and the smart-HTTP URL paths are a standard: they are written down in git’s protocol documentation, and other implementations follow them.
12. The remote: prefix on band 2 output is an implementation detail of the git client, defined by DISPLAY_PREFIX in sideband.c. Nothing on the wire contains those eight characters.
13. Naming your main remote origin and your default branch main is a convention. Git does not care; the tooling around it often does.

WORDS39.6.6 remember these#

1. pkt-line — a length-prefixed frame — 4 hex digits of total length followed by payload; 0000 is flush.
2. Sideband — three streams down one pipe — band 1 data, band 2 progress printed as remote:, band 3 fatal error.
3. Packfile — one compressed bundle of objects — the .pack format with deltas and an accompanying .idx.
4. receive-pack — the server side of push — the process that reads the pack and updates refs.
5. upload-pack — the server side of fetch — the process that negotiates and builds a pack for the client.
6. Capability advertisement — what each end can do — the space-separated list sent after a NUL byte on the first ref line.

39.7 The reader’s own failure, analysed properly#

PLAIN39.7.1 in simple words#

1. Twice in one session, the reader’s push stopped with this message.

send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

2. Here is what that sentence means, word by word.
3. send-pack is the program on your own machine that does the pushing. So the complaint is coming from your side, not the server’s.
4. reading sideband packet means your git had finished sending and had switched to listening for the answer.
5. unexpected disconnect means the connection went silent while it was listening. Not an error message. Silence.
6. So the stage is exactly this: the pack was already sent, and your git was waiting for the report that says whether it worked.
7. And now the crucial point. Your git never got the report. That is the only fact you have.
8. It does not tell you the push failed. It does not tell you the push succeeded. It tells you that the answer did not arrive.
9. The server may have taken the pack, checked it, moved the branch, and then died before it could say so.
10. Or the server may have died before doing any of that.
11. From your side those two look identical. The only way to find out is to ask the server again.

PLAIN39.7.2 a picture in your head#

1. You post a signed contract by registered mail and wait for the receipt.
2. The receipt never comes.
3. Did the office receive and file your contract, and then lose the receipt in the post? Or did the envelope never arrive at all?
4. You genuinely cannot tell from your end. Both stories produce exactly the same experience: you sent something and heard nothing.
5. The only sane move is to phone the office and ask what they have on file.
6. Note what you must not do. You must not assume it failed and post a second contract, because if the first one landed you now have two.
7. And you must not assume it succeeded and move on, because if it did not you have lost the work.

Where this comparison breaks:

8. With a contract, a duplicate is a real problem. With a git push, re-pushing the identical commits is harmless, because the ref update either has already happened or will happen. That is a lucky property of git, not a general one.

PLAIN39.7.3 a worked example#

1. This can be reproduced exactly. Here is a server that behaves normally, accepts the pack, updates the branch, and then drops the connection instead of delivering its report.

$ git ls-remote origin
(nothing: the server has no refs yet)

$ git push origin main
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
$ echo $?
128

2. That looks like a failure. Now ask the server.

$ git ls-remote origin
36e4614a1df6e4ff7839130240981cabb5b970ca    HEAD
36e4614a1df6e4ff7839130240981cabb5b970ca    refs/heads/main

$ git status -sb
## main...origin/main [gone]

3. The push worked. The branch is on the server. The error was about the reply, not about the write.
4. Also note origin/main [gone]. Your local copy of the server’s state was never updated, because the report never arrived. Your machine still thinks nothing is there.
5. Now the second server, which dies before applying anything. Same client, same commit, same command.

$ git push origin main
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly
$ echo $?
128

$ git ls-remote origin
(nothing)

6. Identical message. Identical exit code. Opposite outcome.
7. That is the whole lesson in one pair of transcripts. The error text carries no information about whether the write landed.
8. In the reader’s own session, both failures had NOT applied. That was discovered by running git ls-remote and looking, not by reasoning from the error message. If it had been deduced from the message, the conclusion would have been luck, not knowledge.

PLAIN39.7.4 what is really happening inside#

1. Sequence of events on a push, with the failure point marked.

1. client starts receive-pack on the server
2. server sends ref advertisement + capabilities  -> flush
3. client sends "old new refname"  -> flush
4. client streams the packfile
5. server reads pack, runs index-pack, verifies objects
6. server runs pre-receive hook
7. server updates refs/heads/main
8. server runs post-receive hook
9. server sends "unpack ok" and "ok refs/heads/main"
      <-- the connection dies anywhere from step 5 to step 9
10. client prints the report, updates origin/main

2. The client’s error is raised in step 9. Everything from step 5 to step 8 may or may not have completed.
3. Concretely: in git’s source, send-pack.c calls recv_sideband("send-pack", ...). Inside sideband.c, when the packet reader returns end-of-file, it produces exactly the message the reader saw.
4. The word send-pack in the message is simply the name the caller passed in. It is a label, not a diagnosis.
5. Why does the connection die? Any of: an idle timeout on a middlebox, a carrier-grade NAT table entry expiring, a proxy closing an idle connection, the server process being killed, a load balancer draining, or a genuinely dropped path like the one in the reader’s traceroute.
6. Note that step 4, streaming the pack, can take a long time with nothing coming back. That is exactly when idle-connection timers fire.
7. The general shape of the problem: a non-idempotent operation, meaning one where doing it twice is not the same as doing it once, sent over a transport that can fail after the request but before the reply.
8. Chapter 36 made the same point about HTTP APIs: POST is not idempotent, so a client that times out cannot safely retry, and that is precisely why idempotency keys exist. This is the same problem in a different suit.
9. Git is kinder than most, because pushing the same commits twice is harmless. But you still cannot know the state without asking.

TECHNICAL39.7.5 the engineer’s version#

1. The rule, stated properly: a failure while reading a response tells you nothing about whether the request was applied. Only the outcome of a fresh query does.
2. This is a two-generals situation. No number of extra acknowledgements makes the final one guaranteed to arrive.
3. Correct procedure after any interrupted push:

# what does the server actually hold right now?
git ls-remote origin refs/heads/<branch>

# what did I try to send?
git rev-parse HEAD

# refresh the remote-tracking refs, then read them
git fetch origin
git status -sb

4. Do not use git status alone. Before a fetch, origin/main is a cached value written by the last successful exchange, not a live fact.
5. Techniques that make this class of failure less likely or less painful:

6. git push --atomic uses the atomic capability so several ref updates land as one transaction. It does not solve the reporting problem; it only removes the partial-ref-update case.
7. --force-with-lease is the strongest safety net here, because it encodes an expectation about the server’s current value and fails if reality differs.
8. Generalisation beyond git, worth memorising:

9. The honest version: git push is described as “in practice safe to retry” because the second push either fast-forwards to the same value or reports “Everything up-to-date”. That is a property of the ref-update rule, not a guarantee written in a specification.
10. In the reader’s session, the diagnosis of the wider outage came from the same discipline: curl -v https://github.com printed Trying 20.207.73.82:443... and then timed out after 15 seconds with no reply of any kind, while the same request succeeded over mobile data. What was proven was “no response on this path”. What was not proven was who dropped it.

WORDS39.7.6 remember these#

1. Non-idempotent — doing it twice is not the same as once — an operation whose repetition changes the result.
2. Two generals problem — you can never be sure the last message arrived — the classic impossibility result for agreement over a lossy channel.
3. Re-query — go and look rather than guess — issue a fresh read to establish the true state after an ambiguous failure.
4. Stale remote-tracking ref — your cached idea of the server — origin/main as written by the last successful fetch or push.
5. Atomic push — all refs or none — the --atomic flag, backed by the server’s atomic capability.
6. Lease — an assertion about the server’s current value — the expected old OID carried by --force-with-lease.

39.8 Fetch versus pull#

PLAIN39.8.1 in simple words#

1. git fetch downloads. That is all it does.
2. It brings any objects you are missing into .git/objects, and it updates your record of where the server’s branches are.
3. It does not touch your files. It does not touch your branch. It does not touch your index. After a fetch, your working directory is byte for byte what it was before.
4. git pull is two commands in one: a fetch, then an integration step that combines the fetched work into your current branch.
5. That second step is where the surprises live, because it changes your files and can create commits or conflicts.
6. So git fetch is always safe to run. git pull is not always safe to run.
7. The usual advice is: fetch, look at what arrived, then decide how to integrate it.

PLAIN39.8.2 a picture in your head#

1. Think of a shared noticeboard in an office, and your own copy of it in a folder.
2. git fetch is walking over, photographing the board, and filing the photo. Your desk is untouched. You now know what the board says.
3. git pull is walking over, photographing the board, and then immediately rewriting your own working notes to match it.
4. If your notes and the board disagree, that second act is where the mess happens, right there on your desk, without warning.

Where this comparison breaks:

5. A photograph is passive. A fetched object is not: it is a real object in your database that other commands can now reach and use.
6. Copying the board means overwriting. Git never overwrites your commits; it merges or replays. But it does overwrite your working files.

PLAIN39.8.3 a worked example#

1. Here is a fetch, with the state of everything printed before and after.

before: HEAD=7f77069  origin/main=1303ba7
before: file f contains "A" and "from x1"

$ git fetch origin
From file:///tmp/gw/ff
   1303ba7..2c979a9  main       -> origin/main

after : HEAD=7f77069  origin/main=2c979a9
after : file f contains "A" and "from x1"

$ git status -sb
## main...origin/main [ahead 1, behind 1]

2. Only one thing changed: origin/main. HEAD did not move. The file did not change. And now git status can tell you the truth: one commit each way.
3. Now a pull in the same situation, on a modern git with nothing configured.

$ git pull
hint: You have divergent branches and need to specify how to
hint: reconcile them.
hint:
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
fatal: Need to specify how to reconcile divergent branches.

4. That refusal is a good thing, and it is recent. Older git versions silently merged, which is how so many repositories acquired hundreds of commits reading “Merge branch ‘main’ of …”.

PLAIN39.8.4 what is really happening inside#

1. git fetch origin does five things.
2. Connect and get the ref advertisement.
3. Work out which objects are missing, using want and have lines.
4. Receive one packfile and store it under .git/objects/pack/.
5. Update remote-tracking refs, that is refs/remotes/origin/*, to the server’s values.
6. Write FETCH_HEAD and append to the reflog. Nothing else moves.
7. git pull runs that, then runs either git merge FETCH_HEAD or git rebase FETCH_HEAD, depending on configuration.
8. The surprise has three separate causes.
9. First, the integration step is invisible in the command name. Nothing in the word “pull” says “and then merge”.
10. Second, which integration happens depends on configuration that may differ between your machine and a colleague’s.
11. Third, it acts on your working directory, so a bad outcome is immediately in your face, mid-task, possibly with conflicts.

TECHNICAL39.8.5 the engineer’s version#

1. git pull is documented as git fetch followed by either git merge or git rebase. The choice is made by pull.rebase, then branch.<name> .rebase, then the command-line flags.
2. Recommended settings, with what each buys you:

3. push.default = simple became the default in git 2.0, released 28 May 2014. push.autoSetupRemote arrived in git 2.37, released 27 June 2022.
4. git fetch --prune deletes remote-tracking refs whose branches no longer exist upstream. Without it, origin/old-feature lingers forever.
5. git fetch --all fetches from every remote. git remote update is the older spelling of the same thing.
6. FETCH_HEAD is a file listing everything the last fetch brought, with the ones marked “not-for-merge” excluded from the integration step.
7. The refspec controls what is fetched. The default for a clone is +refs/heads/*:refs/remotes/origin/*. The leading + means “allow non-fast-forward updates to my remote-tracking refs”, which is why a force-push upstream does not break your fetch.
8. Shallow and partial fetches change the object set: --depth=N truncates history, --filter=blob:none omits file contents until needed. Partial clone requires the server capability filter, which GitHub’s advertisement in the earlier trace does list.
9. The honest version: experts disagree about pull.rebase. Rebase-by-default gives a clean line but rewrites your local commits every pull, which is confusing if you have already shared the branch. Merge-by-default is truthful but noisy. pull.ff only sidesteps the argument by making you choose each time, and is the safest default to teach.

WORDS39.8.6 remember these#

1. Fetch — download and record — transfer missing objects and update refs/remotes/* only.
2. Pull — fetch plus integrate — fetch followed by merge or rebase into the current branch.
3. Remote-tracking ref — your note of where the server was — a ref under refs/remotes/<remote>/, updated only by fetch, push or remote commands.
4. Refspec — a mapping of their refs to mine — +<src>:<dst>, where + allows non-fast-forward updates.
5. FETCH_HEAD — what the last fetch brought — a file listing fetched tips, some marked not-for-merge.
6. Prune — forget branches that no longer exist — git fetch --prune or fetch.prune true.

39.9 Merge#

PLAIN39.9.1 in simple words#

1. Merging combines two lines of work into one.
2. Git starts by finding where the two lines split apart. That shared ancestor is called the merge base.
3. It then has three versions of every file: the base version, your version, and their version.
4. For each file it asks: did I change it, did they change it, or both?
5. If only one side changed a file, take that side. If neither changed it, keep the base. If both changed it, look closer.
6. If both changed different parts of the file, take both changes.
7. If both changed the same part, git stops and asks you. That is a conflict.
8. The result is stored as a new commit with two parents, one for each line that was joined.
9. There is one special case. If your branch has not moved at all since the split, there is nothing to combine. Git just moves your branch forward. That is called a fast-forward, and it creates no commit.

PLAIN39.9.2 a picture in your head#

1. Two people are given identical copies of a report and go away to edit it.
2. When they come back, an editor puts three copies on the desk: the original, copy A and copy B.
3. For each paragraph the editor compares all three. If only A touched it, take A’s. If only B touched it, take B’s.
4. If both touched the same paragraph in different ways, the editor cannot decide and leaves both versions marked up for a human.
5. If B never actually changed anything, the editor simply hands over A’s copy and calls it done. That is fast-forward.

Where this comparison breaks:

6. A human editor reads for meaning. Git compares lines of text and nothing else. Two changes that read fine together can still be wrong together, and git will never notice. Section 39.10 gives a real example.

PLAIN39.9.3 a worked example#

1. Fast-forward first. Branch feature is ahead; main has not moved.

$ git log --oneline --graph --all
* 80b9042 C: feature file
* 3fc5061 B: on main
* 0a83b6e A: base

$ git merge-base main feature
3fc5061b9e3a37928186aab85f0c7ed2fc6b73c4

$ git merge-base --is-ancestor main feature && echo "ancestor"
ancestor

$ git merge feature
Updating 3fc5061..80b9042
Fast-forward
 feature.txt | 1 +
 1 file changed, 1 insertion(+)

$ git rev-list --merges --count HEAD
0

2. Zero merge commits were created, because main was already an ancestor of feature. Moving the label forward is enough.
3. Now a real three-way merge. Both branches changed the same file in different places.

BEFORE
* 1f96d1d B: rename header       (main)
| * 8681015 C: add contact line  (feature)
|/
* 9db1830 A: base page

BASE (A)        OURS (main, B)   THEIRS (feature, C)
header          site header      header
body            body             body
footer          footer           footer
                                 contact

4. Line 1 changed on our side only. Line 4 was added on their side only. No overlap, so git can take both.

$ git merge --no-edit feature
Auto-merging page.txt
Merge made by the 'ort' strategy.
 page.txt | 1 +

AFTER
*   012c5e8 Merge branch 'feature'
|\
| * 8681015 C: add contact line
* | 1f96d1d B: rename header
|/
* 9db1830 A: base page

$ cat page.txt
site header
body
footer
contact

5. And the merge commit itself, which is the only commit in the repository with two parent lines.

$ git cat-file -p HEAD
tree ba2693f4d0086cd9a8eab7371433d6aa3acdd178
parent 1f96d1d9b440cb9f8b8d53ab39a8becb65a44eda
parent 86810156b0b336611ae572e45ad596dc623886a4
author Reader <reader@example.com> 1772772600 +0530
committer Reader <reader@example.com> 1772772600 +0530

Merge branch 'feature'

PLAIN39.9.4 what is really happening inside#

1. Finding the merge base is a graph problem, not a guess. Git walks backwards from both tips, marking which commits are reachable from which side.
2. The merge base is a commit reachable from both tips with no descendant that is also reachable from both. git merge-base A B prints it.
3. Sometimes there is more than one such commit. Git’s default strategy then merges the bases together first, recursively, and uses the result as the base. That is why the old strategy was called recursive.
4. Fast-forward is detected by a single question: is the current tip an ancestor of the target? git merge-base --is-ancestor A B answers it with an exit code.
5. If yes, there is nothing to combine, so git writes the target’s tree into the index and worktree and moves the branch. No new object is created.
6. --no-ff forces a merge commit even when a fast-forward is possible. The reason to use it is that the merge commit is the only record that these commits were a group.

with --no-ff:
*   a78fc04 Merge branch 'feature'
|\
| * 52ee392 C (feature)
| * 9aa4587 B (feature)
|/
* 7c451a3 A

$ git log --oneline --first-parent
a78fc04 Merge branch 'feature'
7c451a3 A

7. That last command is the payoff of --no-ff. --first-parent follows only the mainline, so a two-year history of merges reads as one line per feature.
8. The three-way merge itself is applied per file, comparing base, ours and theirs, then per region within the file.

TECHNICAL39.9.5 the engineer’s version#

1. Strategies: ort is the default, recursive is the old default, resolve handles a single base only, octopus merges more than two heads, ours discards the other side’s content entirely, subtree shifts paths.
2. ort, short for “ostensibly recursive’s twin”, became the default in git 2.34, released 15 November 2021. GitHub reported it outperforming recursive by roughly 500 times on rename-heavy merges and by more than 9,000 times on some rebases, largely because it does not use the index as its working data structure.
3. Do not confuse the ours strategy with the -X ours option. The strategy throws away the other branch’s changes. The option only decides conflicting hunks in your favour.

4. Rename detection is on by default, controlled by merge.renameLimit and diff.renames. It is why moving a file on one side and editing it on the other usually merges correctly.
5. git merge --abort restores the pre-merge state using MERGE_HEAD and ORIG_HEAD. It is safe while a merge is in progress.
6. During a conflicted merge, .git/MERGE_HEAD, .git/MERGE_MSG and .git/MERGE_MODE exist. Their presence is how git knows a merge is unfinished and why git status prints “You have unmerged paths”.
7. An octopus merge with more than two parents is legal and appears in the Linux kernel history. It refuses to run if any pair conflicts.

WORDS39.9.6 remember these#

1. Merge base — where the two lines split — the best common ancestor of two commits in the DAG.
2. Three-way merge — compare base, ours, theirs — the algorithm that decides per region which side’s change to keep.
3. Fast-forward — just move the label — an update where the old tip is an ancestor of the new one, so no merge commit is needed.
4. Merge commit — the join — a commit with two or more parents.
5. ort — the modern merge engine — the default merge strategy since git 2.34, November 2021.
6. First-parent history — the mainline only — git log --first-parent, which treats each merge as one entry.

39.10 Conflicts at the file level#

PLAIN39.10.1 in simple words#

1. A conflict is not git failing. It is git refusing to guess.
2. Git compares text line by line. It works on regions of lines, not on meaning.
3. If your change and their change are in different regions of the file, git takes both without asking.
4. If they overlap, git cannot know which one you want, so it writes both into the file surrounded by markers and stops.
5. The markers are ugly on purpose. They are meant to be impossible to miss and impossible to leave in by accident.
6. Your job is to edit the file until it is what you actually want, delete the markers, and tell git you are done with git add.
7. There is a second, nastier kind of conflict that git will never report. Two changes that merge perfectly as text, but break the program.
8. Git cannot see that, because git does not know what your code means.

PLAIN39.10.2 a picture in your head#

1. Two proofreaders mark up the same page.
2. One changes a word in paragraph two. The other adds a sentence to paragraph nine. The typesetter applies both without a thought.
3. Both change the same sentence in paragraph five, differently. The typesetter stops, prints both versions one above the other with a big line between them, and puts the page in the “needs a decision” tray.
4. Now the nasty case. One proofreader renames a character from Anna to Anya throughout chapter one. The other adds a new scene in chapter nine mentioning Anna.
5. No paragraph overlaps. The typesetter merges happily. The book is now inconsistent, and nothing in the process caught it.

Where this comparison breaks:

6. A typesetter might notice the name change by reading. Git never reads. It will merge a file into nonsense with complete confidence, every time.

PLAIN39.10.3 a worked example#

1. Both branches change the same line of a config file.

BASE            OURS (main)      THEIRS (feature)
PORT = 8080     PORT = 3000      PORT = 9090

$ git merge feature
Auto-merging config.py
CONFLICT (content): Merge conflict in config.py
Automatic merge failed; fix conflicts and then commit.

$ cat config.py
HOST = "localhost"
<<<<<<< HEAD
PORT = 3000
=======
PORT = 9090
>>>>>>> feature
TIMEOUT = 30

2. Line by line. <<<<<<< HEAD opens the conflicted region and names your side. Everything until ======= is your version.
3. ======= is the divider. Everything after it until >>>>>>> is their version. >>>>>>> feature closes the region and names their side.
4. The unconflicted lines, HOST and TIMEOUT, are outside the markers and have already been merged.
5. The default marker style hides the base. Turning on diff3 shows it, which is much more useful.

$ git checkout --conflict=diff3 config.py
$ cat config.py
HOST = "localhost"
<<<<<<< ours
PORT = 3000
||||||| base
PORT = 8080
=======
PORT = 9090
>>>>>>> theirs
TIMEOUT = 30

6. Now you can see that the original was 8080, so both sides changed it, and neither side is a superset of the other.
7. Meanwhile, the index holds all three versions at once.

$ git ls-files -u
100644 5c03df4f...9181 1    config.py
100644 bd085f95...ca9b 2    config.py
100644 ee2a92c1...4f85 3    config.py

8. Stage 1 is the base, stage 2 is ours, stage 3 is theirs. Resolving means replacing those three entries with one stage-0 entry, which is exactly what git add does.

PLAIN39.10.4 what is really happening inside#

1. Now the semantic conflict. This one is real, run end to end.
2. Start with two files. lib.py defines get_user. app.py calls it.
3. On a branch called refactor, rename the function to fetch_user and fix the one call site that exists.
4. On main, add a brand new file report.py that calls get_user.
5. Neither side touched a line the other touched. Watch git.

$ git merge refactor
Merge made by the 'ort' strategy.
 app.py | 4 ++--
 lib.py | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

$ python3 -c "import report; report.report()"
  File "/tmp/gw/sem/report.py", line 1, in <module>
    from lib import get_user
ImportError: cannot import name 'get_user' from 'lib'

6. Zero conflicts. Zero warnings. Broken program.
7. This is not a bug. Git did exactly what it promises: combine text changes that do not overlap. It never promised the result would compile.
8. The only defence against semantic conflicts is a test suite that runs on the merge result, not on either branch alone. That is precisely what continuous integration on the merged head is for.
9. Binary files are the other special case. Git cannot combine regions of a JPEG or a spreadsheet, so any change on both sides is a whole-file conflict.

$ git merge designer
warning: Cannot merge binary files: logo.png (HEAD vs. designer)
CONFLICT (content): Merge conflict in logo.png

$ git status --short
UU logo.png

10. There are no markers in the file. The working copy is left as one of the two versions. You choose a side with git checkout --ours or --theirs, then git add.

TECHNICAL39.10.5 the engineer’s version#

1. Conflict styles, set by merge.conflictStyle:

2. zdiff3 arrived in git 2.35, released 24 January 2022. It hoists lines common to both sides out of the conflict region, making the real disagreement smaller.
3. The resolution workflow, in full:

git status                  # which paths are unmerged
git diff                    # combined diff of conflicts only
git mergetool               # optional three-pane editor
<edit files, remove markers>
git add <path>              # collapses stages 1,2,3 into stage 0
git status                  # confirm nothing is unmerged
git commit                  # message is prefilled from MERGE_MSG

4. git mergetool drives an external tool named by merge.tool. It creates *.BASE, *.LOCAL and *.REMOTE temporary files from index stages 1, 2 and 3, and removes them afterwards unless mergetool.keepBackup is set.
5. git checkout --ours <path> and --theirs <path> pull stage 2 or stage 3 out of the index. During a rebase these are reversed relative to intuition, because the commit being replayed is “theirs”.
6. rerere stands for “reuse recorded resolution”. Enable it with git config rerere.enabled true. Git then records the conflict text and your resolution, and replays it automatically next time the same conflict appears.

$ git config rerere.enabled true
$ git merge feature
Recorded preimage for 'cfg.py'
<resolve by hand, git add, git commit>
Recorded resolution for 'cfg.py'.

$ git reset --hard HEAD~1
$ git merge feature
Resolved 'cfg.py' using previous resolution.

$ cat cfg.py
PORT = 3000

7. The recordings live in .git/rr-cache/<hash>/preimage and postimage. The hash is computed from the normalized conflict text, so an identical conflict anywhere gets the same resolution.
8. rerere was written by Junio C Hamano and merged on 6 February 2006, with the commit message “git-rerere: reuse recorded resolve”. It is most valuable during long rebases, where the same conflict recurs on every replayed commit.
9. For binary formats, .gitattributes with *.png binary or a custom merge= driver is the practical answer. Git LFS stores a pointer file, so conflicts appear on the pointer rather than the payload, which is smaller but no easier to merge.

WORDS39.10.6 remember these#

1. Conflict — git refusing to guess — overlapping changes recorded as index stages 1, 2 and 3 with markers in the worktree.
2. Conflict marker — the ugly fence — <<<<<<<, =======, >>>>>>>, plus ||||||| for the base in diff3 style.
3. Semantic conflict — merges fine, breaks anyway — a logically incompatible pair of changes that do not overlap textually.
4. rerere — remember how I fixed this — recorded conflict resolutions replayed automatically.
5. Unmerged path — a file still in dispute — a path with index entries at stages other than 0.
6. Merge driver — custom combining logic for a file type — a program named in gitattributes via merge=<name>.

39.11 Rebase#

PLAIN39.11.1 in simple words#

1. Rebase means: take my commits, and pretend I had started from somewhere else.
2. Git works out which commits are mine, sets them aside, moves my branch to the new starting point, and applies each of my changes again, one at a time.
3. Each time it applies one, it makes a new commit. Not the old one moved. A new object, with a new name.
4. It must be a new object, because a commit records the name of its parent, and the parent is now different. Change any byte and the name changes.
5. So after a rebase your work looks the same but every commit has a different identifier.
6. The result is a straight line instead of a fork, which many teams prefer to read.
7. Rebase can also edit history as it replays: reorder commits, reword messages, squash several into one, or drop one entirely. That is interactive rebase.

PLAIN39.11.2 a picture in your head#

1. You wrote three pages of notes on top of page 10 of a shared notebook.
2. Meanwhile the shared notebook grew to page 14.
3. Merging means gluing your three pages in as a side branch, with a note saying where they joined.
4. Rebasing means copying your three pages out by hand onto fresh sheets, placing them after page 14, and burning the originals.
5. The words on your new sheets are identical. The sheets are not the same sheets.

Where this comparison breaks:

6. Copying by hand can introduce mistakes. Rebase reapplies changes exactly, but it can hit conflicts, because the surrounding text is now different.
7. The originals are not really burned. They stay in the reflog for 90 days.

PLAIN39.11.3 a worked example#

1. A branch with two commits, and main has moved on.

BEFORE
* a91e93d M1: main moved on          (main)
| * 9ca2c92 F2: extend feat.txt      (feature)
| * 26e79b3 F1: add feat.txt
|/
* c32f284 A: base

$ git switch feature
$ git rebase main
Successfully rebased and updated refs/heads/feature.

AFTER
* 6a7f276 F2: extend feat.txt        (feature)
* 003972b F1: add feat.txt
* a91e93d M1: main moved on          (main)
* c32f284 A: base

2. The hashes side by side.

3. Two commits went in, two came out, and not one identifier survived.
4. Interactive rebase on a messy branch. This is the todo list git presents.

$ git rebase -i HEAD~4
pick 7644785 add parser
pick bd687cc wip
pick 894e838 fix typo in parser
pick 0ec5c9b add tests

5. Edit it to this, and save.

pick   7644785 add parser
fixup  bd687cc wip
fixup  894e838 fix typo in parser
reword 0ec5c9b add tests

6. The result: five commits become three, the file content is byte for byte identical, and the last message was rewritten.

$ git log --oneline
c868f3c add unit tests for the parser
4dcdb2e add parser
292e19e base

PLAIN39.11.4 what is really happening inside#

1. git rebase <upstream> runs these steps.
2. Work out the commit list: everything reachable from HEAD but not from upstream, oldest first. That is git rev-list upstream..HEAD --reverse.
3. Check out the upstream tip, with HEAD detached.
4. For each commit in the list, compute its change against its own parent and apply that change to the current state.
5. Create a new commit with the same message, the same author and author date, a new committer date, and the new parent.
6. When the list is finished, move the branch ref to the last new commit and re-attach HEAD to it.
7. Step 5 is why the hash changes: the parent line differs, the tree usually differs, and the committer timestamp differs. Chapter 38 showed that the hash is computed over exactly those bytes.
8. If applying a change fails, the rebase stops and asks you to fix it, then git rebase --continue. git rebase --abort restores everything.
9. Interactive rebase is the same loop, but the list is written to a todo file you may edit first.

10. git commit --fixup <sha> writes a message starting fixup!. Running git rebase -i --autosquash then reorders that commit next to its target and marks it fixup for you, with no editing by hand.

TECHNICAL39.11.5 the engineer’s version#

1. Two backends exist. The apply backend uses format-patch and git am. The merge backend replays with the merge machinery and is the default since git 2.26 (March 2020). --empty, --keep-empty and --reapply-cherry-picks control what happens to commits that become empty.
2. git rebase --onto <newbase> <upstream> <branch> moves only the commits that are on branch but not on upstream. It is the tool for detaching a topic built on another topic.

$ git rebase --onto main topic1 topic2

3. git rebase -i --autosquash was added by Nanako Shiraishi on 6 January 2010 and shipped in git 1.7.0 on 12 February 2010. Set rebase.autoSquash true to make it automatic.
4. git rebase --exec 'make test' runs a command after each replayed commit, so you can prove every commit in the branch builds, not just the tip.
5. rebase.autoStash true stashes uncommitted work before the rebase and restores it afterwards.
6. History: git-rebase-script was added by Junio C Hamano on 25 June 2005, with the message “rebase local commits to new upstream head”. Interactive mode was added by Johannes Schindelin on 24 June 2007 and shipped in git 1.5.3 on 2 September 2007.
7. git rebase sets ORIG_HEAD to the pre-rebase tip, and writes reflog entries labelled rebase (start), rebase (pick), rebase (fixup) and rebase (finish), so every intermediate state is recoverable.

WORDS39.11.6 remember these#

1. Rebase — replay my work on a new base — create new commits with the same changes and different parents.
2. Upstream — the thing I am rebasing onto — the commit that becomes the new parent of the first replayed commit.
3. Interactive rebase — edit the plan before replaying — git rebase -i with a todo list of verbs.
4. Autosquash — fold marked fixes automatically — reorder and mark fixup! and squash! commits during rebase -i.
5. Committer date — when this object was made — rewritten on every replay, unlike the author date.
6. Todo list — the rebase plan — the editable file of pick, squash, fixup and other verbs.

39.12 The reader’s rebase, explained#

PLAIN39.12.1 in simple words#

1. In the reader’s session, a local branch was rebased onto a main that had moved, and the branch’s SHA changed.
2. Nothing went wrong. That is the only possible outcome.
3. A commit’s name is computed from its exact bytes. Those bytes include the name of its parent.
4. Rebasing gives every replayed commit a different parent. Different parent means different bytes, which means a different name.
5. So the changes are the same and the commits are not the same objects.
6. The old commits still exist. They are simply no longer pointed at by the branch.

PLAIN39.12.2 a picture in your head#

1. Two identical recipes, each written on a card that also says which card comes before it in the box.
2. Move one recipe to a different position in the box and the “comes after” line must change.
3. The ingredients are identical. The card is not the same card, because the card includes that line.

Where this comparison breaks:

4. You could rub out the line on a real card. A git object’s name is derived from its bytes, so editing the bytes produces a different object rather than an edited one.

PLAIN39.12.3 a worked example#

1. The same commit, before and after rebase, printed in full.

OLD F1 (26e79b3)
tree   d18b4ea7a3728589bfef3b1836e869ac1db45faf
parent c32f2845c02885738408ebd404b13b9f2e3c7c4e
author    Reader <reader@example.com> 1772941200 +0530
committer Reader <reader@example.com> 1772941200 +0530

NEW F1 (003972b)
tree   ed1a2fa48bd51d2a79edc8c6e3628c892e5d4c9b
parent a91e93d9b813a53be8d8b7f51733815d0de72dea
author    Reader <reader@example.com> 1772941200 +0530
committer Reader <reader@example.com> 1772942400 +0530

2. Three of the four lines differ. The parent is a different commit. The tree is different because it now also contains main’s new version of core.txt. The committer timestamp is when the replay happened.
3. Only the author line survived, which is why git log still shows your original date.
4. Now recompute the new name by hand, exactly as Chapter 38 did.

$ git cat-file commit 003972b > c.raw
$ wc -c < c.raw
219
$ { printf 'commit %d\0' 219; cat c.raw; } | sha1sum
003972bffe4b385f957000613aa02ea8e5208924
$ git rev-parse 003972b
003972bffe4b385f957000613aa02ea8e5208924

5. The hash is not stored anywhere. It is arithmetic over those 219 bytes, and anyone can repeat it.

PLAIN39.12.4 what is really happening inside#

1. The change each commit introduces really is identical. Git can prove that with a patch identifier, a hash of the diff with line numbers and whitespace normalized away.

$ git show 26e79b3 | git patch-id --stable
3644a7626a72280f69af9830a10cf2f013748966 26e79b3d...
$ git show 003972b | git patch-id --stable
3644a7626a72280f69af9830a10cf2f013748966 003972bf...

2. Same patch identifier on the left. Different commit identifier on the right. That is the whole story in two lines.
3. The old commits are still in the object database, reachable through the reflog.

$ git reflog feature
6a7f276 feature@{0}: rebase (finish): onto a91e93d
9ca2c92 feature@{1}: commit: F2: extend feat.txt
26e79b3 feature@{2}: commit: F1: add feat.txt
c32f284 feature@{3}: branch: Created from HEAD

4. So the pre-rebase branch is one command away: git reset --hard feature@{1}.
5. Practical consequence for the reader: after that rebase, the branch on the server and the branch on the machine had no commits in common past the base. A plain push would be refused as non-fast-forward, and a forced push would be required. Section 39.13 covers when that is acceptable.

TECHNICAL39.12.5 the engineer’s version#

1. The invariant: SHA-1 over commit <len>\0 plus the commit body. Any change to tree, parent, author, committer or message yields a different object ID.
2. git patch-id --stable produces a diff-content identity that survives rebase, cherry-pick and reordering. git cherry and git rebase use it to skip changes already present upstream.
3. git range-diff <base>..<old> <base>..<new> compares two versions of the same series and is the correct tool for reviewing a rebased branch. It was added in git 2.19, September 2018.
4. What survives and what does not:

5. Signed commits are invalidated by rebase, because the signature covers the commit bytes. git rebase -S re-signs each replayed commit.
6. Because remote-tracking refs are updated only by network operations, a rebase makes git status report a large “diverged” count immediately. That is arithmetic on the graph, not a warning about anything being wrong.

WORDS39.12.6 remember these#

1. Patch identifier — the fingerprint of a change — a hash of the normalized diff, unchanged by rebase.
2. Rewritten history — the same work, new objects — a series replaced by a replayed series with different IDs.
3. Diverged — no longer a straight line — two refs whose merge base is behind both of them.
4. range-diff — compare two versions of a series — git range-diff, added in git 2.19, September 2018.
5. Re-signing — the signature does not follow — GPG signatures must be recreated after any rewrite, with git rebase -S.
6. Reflog recovery — the pre-rebase tip is still there — branch@{1} names the value the branch held before the last update.

39.13 When rebasing is safe and when it destroys work#

PLAIN39.13.1 in simple words#

1. There is one rule, and it is short. Do not rebase commits that other people already have.
2. Rebasing your own branch, before anyone else has fetched it, is safe and good practice. It makes your work easy to read and easy to review.
3. Once you have pushed a branch and a colleague has fetched it, their machine holds those exact commit objects.
4. If you rebase and force-push, the server now has different objects with the same branch name.
5. Your colleague’s git still has the originals, plus anything built on top of them. Their next pull tries to combine two versions of the same work.
6. The result is duplicated commits, avoidable conflicts, and confusion about which version is real.
7. Main, master, develop and release branches are shared by definition. Never rebase those.

PLAIN39.13.2 a picture in your head#

1. You hand out photocopies of a chapter to five reviewers.
2. Then you rewrite the chapter and renumber every paragraph.
3. Each reviewer’s comments now point at paragraph numbers that mean something different, or nothing at all.
4. Nobody’s work is destroyed, but reconciling it is now manual and tedious for five people instead of one.

Where this comparison breaks:

5. Reviewers can see that the chapter changed. Git cannot. It sees two unrelated sets of commits with the same branch name and tries to merge them, which is how duplicate commits appear in the log.

PLAIN39.13.3 a worked example#

1. Alice and Bob both have commit B. Bob builds C on top of it. Alice amends B, which is a one-commit rewrite, and force-pushes.

Alice: 2bd9139 B: with a better message   (rewritten)
       fac214d A

Bob:   4370404 C: bob's work on top of B
       426111e B                          (the original)
       fac214d A

2. Bob fetches, and his repository now contains both versions.

* 2bd9139 B: with a better message   (origin/main)
| * 4370404 C: bob's work on top of B  (main)
| * 426111e B
|/
* fac214d A

3. Bob pulls with the default merge behaviour, and gets a conflict in a file neither of them meant to fight over.

$ git pull
Auto-merging f
CONFLICT (content): Merge conflict in f

4. The fix Bob needs is git pull --rebase, which replays only his own C onto the new B.

$ git pull --rebase
Successfully rebased and updated refs/heads/main.
$ git log --oneline
0b6514b C: bob's work on top of B
2bd9139 B: with a better message
fac214d A

5. Note that Bob had to know what happened. Git gave him no message saying “the upstream branch was rewritten”.

PLAIN39.13.4 what is really happening inside#

1. A normal push is only allowed if the old value on the server is an ancestor of the new value. That is the fast-forward rule.

$ git push origin main
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs
hint: Updates were rejected because the remote contains work
hint: that you do not have locally.

2. --force switches that check off entirely. Whatever the server had is replaced, and any commit only reachable from the old value becomes garbage.
3. --force-with-lease replaces the check with a different one: the server’s current value must equal what your remote-tracking ref says it is.
4. So --force-with-lease says “overwrite, but only if nobody has pushed since I last looked”. Here is it refusing.

my origin/main says : 2bd9139
the server really has: 0b6514b

$ git push --force-with-lease origin main
 ! [rejected]        main -> main (stale info)
error: failed to push some refs

$ git push --force origin main
 + 0b6514b...a848939 main -> main (forced update)

5. The plain --force succeeded and Bob’s commit C is now unreachable on the server. Only Bob’s own clone still has it.
6. One honest warning: --force-with-lease compares against your remote-tracking ref, and a background git fetch updates that ref without you noticing. Then the lease is satisfied and the protection is gone. Use --force-with-lease=<ref>:<expected-oid> when it matters.

TECHNICAL39.13.5 the engineer’s version#

1. The decision table:

2. --force-if-includes, added in git 2.30 (December 2020), strengthens the lease by also requiring that your local history includes everything your remote-tracking ref has seen. Turn it on with push.useForceIfIncludes true.
3. Server-side defences: receive.denyNonFastForwards, and on hosted services, protected branches that reject force pushes outright.
4. Recovery when a force push destroys a colleague’s commit: the commit is still in that colleague’s clone, and on GitHub-style services it is often still reachable through the reflog on the server or through the events API.
5. git rebase on a branch that has an open pull request is common practice, and opinions differ. One camp rebases and force-pushes to keep the series clean; the other adds fixup commits during review and squashes at merge time so that review comments stay anchored. Both are defensible.
6. git range-diff makes the first camp’s approach reviewable, because it shows what changed between the old series and the new one.

WORDS39.13.6 remember these#

1. Published history — commits other people have — anything that has been pushed and fetched by someone else.
2. Fast-forward rule — you may only extend — the default push check that the old tip is an ancestor of the new one.
3. Force push — replace whatever is there — --force, which switches the ancestry check off.
4. Lease — only if it is still what I saw — --force-with-lease, which compares against your remote-tracking ref.
5. Protected branch — the server says no — a hosting-service rule rejecting force pushes and deletions.
6. Duplicate commits — the same work twice — the visible symptom of merging a rewritten branch with its original.

39.14 Merge versus rebase as a team policy#

PLAIN39.14.1 in simple words#

1. Both merge and rebase produce a repository containing the same final files. Neither is more correct than the other.
2. The argument is about what the history looks like afterwards, and how easy it is to work with.
3. Merging keeps a true record: this work happened on a side branch, and it was joined on this date.
4. Rebasing produces a straight line that reads like a story, but the story is tidied. It never happened quite that way.
5. Squash merging goes further: an entire branch becomes one commit on the mainline, and the individual steps are discarded.
6. Teams pick one, write it down, and enforce it with a setting on the hosting service. The worst outcome is a repository where everyone does something different.

PLAIN39.14.2 a picture in your head#

1. Three ways to write up a two-week experiment.
2. Merge: the full lab notebook, every dead end included, with dates.
3. Rebase: the same experiment rewritten as a clean sequence of steps that leads to the result.
4. Squash: a one-paragraph abstract.
5. All three are honest about the result. They differ in what you can go back and inspect a year later.

Where this comparison breaks:

6. A lab notebook cannot be rewritten without fraud. Git history is a tool for communication, and tidying it is normal and legitimate.

PLAIN39.14.3 a worked example#

1. The same feature, three policies, as git log --oneline --graph.

MERGE (--no-ff)
*   a78fc04 Merge branch 'feature'
|\
| * 52ee392 C (feature)
| * 9aa4587 B (feature)
|/
* 7c451a3 A

REBASE then fast-forward
* 6a7f276 C (feature)
* 003972b B (feature)
* 7c451a3 A

SQUASH
* 400e7f2 Add the whole feature (squashed)
* 040d29f A

2. One extra fact about squash merging that surprises people.

$ git merge-base --is-ancestor feature main && echo yes || echo no
no

3. After a squash merge, git does not know the branch was merged. Merging it again would try to replay B, C and D. Delete squashed branches immediately.

PLAIN39.14.4 what is really happening inside#

1. The arguments for merge.
2. It records what actually happened, including when parallel work existed.
3. It never rewrites objects, so nothing anybody has fetched ever becomes invalid.
4. git log --first-parent collapses each feature to one line, giving a clean mainline view on demand.
5. Reverting a whole feature is one command, git revert -m 1 <merge>.
6. The arguments for rebase.
7. A linear history is easier to read, and git log needs no graph rendering.
8. Every commit on the mainline was tested against the code it actually sits on top of, which makes bisect more reliable.
9. There are no merge commits whose diff is confusing to review.
10. The arguments for squash.
11. One commit per feature is the simplest possible mainline.
12. Reviewers’ intermediate “fix typo” commits never reach the main branch.
13. The cost: bisect lands on a large commit, and the intermediate steps are gone unless the branch is kept.
14. The honest summary: this is a readability and bisect-ability argument, not a correctness argument. Every option produces identical file contents. Experts disagree, and both large, successful projects and small teams run all three policies happily.

TECHNICAL39.14.5 the engineer’s version#

1. Comparison of the three:

2. The Linux kernel merges: Linus Torvalds pulls from maintainers, producing a deeply merged history, and explicitly asks maintainers not to rebase published trees. Many web and application teams squash-merge every pull request.
3. git config merge.ff false on a specific branch forces merge commits. git config pull.ff only prevents accidental merges on pull.
4. On hosting services the equivalent settings are the allowed merge methods for pull requests: merge commit, squash and merge, rebase and merge.
5. Bisect interacts with policy. With merge-based history, git bisect can land on a commit inside a feature branch that was never tested alone. git bisect --first-parent, added in git 2.29 (October 2020), restricts the search to the mainline and avoids that.
6. Commit hygiene matters more than the policy. A branch of ten commits each of which builds and passes tests is valuable under any of the three; a branch of ten commits named “wip” is not.

WORDS39.14.6 remember these#

1. Linear history — one line, no forks — a history where every commit has exactly one parent.
2. Squash merge — one commit per feature — combine a branch into a single commit with no merge link.
3. First-parent view — the mainline only — following only the first parent of each merge.
4. Bisect-ability — how precisely a search can land — a property of how large and how independently testable each commit is.
5. Merge method — the button the service offers — merge commit, squash and merge, or rebase and merge.
6. Commit hygiene — each commit is a whole, working step — the practice that makes any policy work.

39.15 Cherry-pick, revert, bisect and the rest#

PLAIN39.15.1 in simple words#

1. git cherry-pick copies one commit’s change onto your current branch. Use it to move a single fix without moving everything around it.
2. git revert creates a new commit that undoes an old one. Use it when the bad commit is already public.
3. git bisect finds which commit introduced a bug, by testing a shrinking range automatically.
4. git blame shows which commit last changed each line of a file.
5. git log -S"text" finds commits where a piece of text was added or removed.
6. git worktree gives you a second checked-out directory sharing one object database, so you can look at another branch without stashing.

PLAIN39.15.2 a picture in your head#

1. Bisect is the game of guessing a number between 1 and 1000 with yes-or-no questions.
2. Ask “is it above 500?” and half the possibilities vanish, whatever the answer.
3. Ten questions cover a thousand numbers; twenty cover a million.
4. Git plays the same game over commits. Each test halves the range.

Where this comparison breaks:

5. The number game assumes one answer. Bisect assumes the bug appeared once and stayed. If it comes and goes, bisect will mislead you. Mark such commits git bisect skip.

PLAIN39.15.3 a worked example#

1. Thirty-one commits. Somewhere in there, average([1,2,4]) started returning 2.0 instead of 2.333.

$ ./test.sh
AssertionError: average([1,2,4]) gave 2.0

2. Give git a known-bad commit and a known-good commit, and a script that exits 0 for good and non-zero for bad.

$ git bisect start HEAD 8c84483
$ git bisect run ./check.sh
Bisecting: 14 revisions left to test (roughly 4 steps)
[e07d528] c015: routine change
running './check.sh'
Bisecting: 7 revisions left to test (roughly 3 steps)
[56821a0] c022: routine change
running './check.sh'
Bisecting: 3 revisions left to test (roughly 2 steps)
[bb1e2b9] c018: routine change
running './check.sh'
Bisecting: 0 revisions left to test (roughly 1 step)
[a83bd2e] c017: routine change
running './check.sh'
bb1e2b9ca8de5e3324381ae15dec599b8c409a9f is the first bad commit

3. Four tests over thirty commits, because log base 2 of 30 is about 4.9.
4. And the guilty change, which was hidden among unrelated edits.

-    return sum(nums) / len(nums)
+    return sum(nums) // len(nums) + 0.0

5. Finish with git bisect reset, which puts you back where you started.

PLAIN39.15.4 what is really happening inside#

1. Cherry-pick computes the diff of one commit against its parent, applies it here, and makes a new commit. The change is the same; the object is new.

original S   = 9e25370ab0a266f7b25787b2a08e719f8dd9d964
copy on main = d673ae20eb28f1b43254cad402c16571ec99e9b3

patch-id of both = c92b260713d372c3c1c2babe11f8384d5319ef45

2. git cherry -v main hotfix uses that patch identifier to say which commits on hotfix are already on main. A leading - means already there, + means not yet.
3. Revert computes the inverse diff and commits it. Reverting a merge needs -m 1 to say which parent’s line of history to keep.

$ git revert HEAD
error: commit 88e4ee8... is a merge but no -m option was given.

$ git revert -m 1 --no-edit HEAD

4. Bisect works on the graph. git rev-list --bisect picks the commit that splits the remaining candidates most evenly, so each answer removes about half regardless of which answer it is.
5. git bisect run <cmd> automates it. Exit code 0 means good, 1 to 124 and 126 to 127 mean bad, and exit code 125 means “cannot test, skip this one”.

TECHNICAL39.15.5 the engineer’s version#

1. The wider toolbox:

2. History: git rev-list --bisect was added by Linus Torvalds on 17 June 2005, and git-bisect-script followed on 30 July 2005 with the commit message “Making it easier to find which change introduced a bug”.
3. git bisect run takes any command, so the test can be a unit test, a build, a grep on the output, or a shell one-liner. It is the highest-value five minutes in this chapter.
4. git bisect --first-parent (git 2.29, October 2020) confines the search to the mainline, which matters in merge-heavy repositories.
5. git worktree shares .git/objects and refs across checkouts, so a second worktree costs disk space for files only, not for history.
6. git log -S is a “pickaxe” search on the number of occurrences of a string; git log -G matches the diff text with a regular expression. They answer different questions, and -S is usually the one you want.

WORDS39.15.6 remember these#

1. Cherry-pick — copy one change here — apply one commit’s diff and create a new commit for it.
2. Revert — undo by adding — commit the inverse of an earlier commit.
3. Bisect — binary search over history — halve the candidate range with each test until one commit remains.
4. Pickaxe — find when text appeared — git log -S<string>.
5. Worktree — a second checkout of one repository — an extra working directory sharing the same object database.
6. Skip — this one cannot be judged — git bisect skip, or exit code 125 from a bisect run script.

39.98 Common wrong ideas#

1. Wrong: git add only marks a file for later. Right: it hashes the content and writes a real object into .git/objects immediately.
2. Wrong: the index stores differences. Right: it stores a complete snapshot, one entry per tracked file, with stat data and object IDs.
3. Wrong: git commit copies your files. Right: the blobs already exist; commit writes trees, one commit object, and moves a ref.
4. Wrong: a push failure means nothing was written. Right: a disconnect while reading the response says nothing about the write. Re-query with git ls-remote.
5. Wrong: git status shows the server’s current state. Right: it compares against a cached remote-tracking ref that only a fetch or push updates.
6. Wrong: git fetch can mess up my work. Right: fetch changes only refs/remotes/* and the object database. git pull is the one to be careful with.
7. Wrong: rebase moves my commits. Right: it creates new commit objects with new identifiers, and leaves the old ones in the reflog.
8. Wrong: a clean merge means the code works. Right: git merges text, not meaning. Two non-overlapping changes can break the program completely.
9. Wrong: git reset --hard loses commits forever. Right: commits survive in the reflog for 90 days. Uncommitted edits, however, are gone.
10. Wrong: --force-with-lease is always safe. Right: it compares against your remote-tracking ref, which a background fetch can silently refresh.

39.99 Chapter summary in 20 lines#

1. A tracked file exists in three places: working directory, index, object database. HEAD says which commit you are on.
2. git status is nothing more than two comparisons: index against HEAD, and working directory against index.
3. The index is a real binary file, .git/index, starting with the four bytes DIRC, present in git’s first commit on 7 April 2005.
4. It holds one entry per tracked file: path, object ID, mode, and cached stat data from the filesystem.
5. That stat cache is why git status is fast: most files are never opened, only asked about.
6. git add hashes the content and writes the object at once, then records its name in the index. git add -p can stage a version that exists nowhere.
7. git commit writes trees from the index, writes one commit object, and moves one ref. Three steps, in that order.
8. The index exists so you can choose what goes in a commit. Other systems manage without one; it is a design choice, not a necessity.
9. git restore, git reset --soft/--mixed/--hard, git revert, git clean and git stash differ only in which of the three places they touch.
10. Only git reset --hard and git clean can destroy work that git never stored. Everything else is recoverable through the reflog.
11. A push sends one packfile of only the missing objects, framed in pkt-lines, with three sideband channels: data, progress, fatal error.
12. Twenty commits is one round trip, not twenty. In the real measurement, sixty objects in 3.95 KiB.
13. send-pack: unexpected disconnect while reading sideband packet means the pack was sent and the reply never arrived.
14. That error is identical whether the server applied the write or not. Both cases were reproduced. Only git ls-remote can tell you which happened.
15. In the reader’s session both pushes had not applied, and that was established by re-querying, not by reading the error.
16. git fetch changes only remote-tracking refs and the object store. git pull adds an integration step that touches your files.
17. Merge finds the common ancestor and does a three-way merge; fast-forward needs no commit because the old tip is already an ancestor.
18. Conflicts are git refusing to guess about overlapping text. Semantic conflicts merge cleanly and still break the code.
19. Rebase replays each commit onto a new parent, so every commit gets a new identifier even though the patch identifier is unchanged.
20. Rebase your own unpushed work freely; never rewrite history other people already hold, and prefer --force-with-lease when you must force.