Software Heritage / SWHID

The only subject that must give one identity to trees arriving from git, Mercurial, Subversion and bare tarballs alike — and it does so by making git's blob and tree hashing the normalization target for all of them, including content that never passed through git.

Ecosystem	Software Heritage archive; swh.model; SWHID as an IANA-registered swh URI scheme
Level 1	Merkle DAG, extended above the tree: dir → rev → rel → snp
Level 2	whole blob — sha1_git, i.e. git's blob <len>\0<bytes> framing
Node model	five DentryPerms: content, executable content, symlink, directory, revision (submodule)
Entry ordering	byte-wise with / appended to directory names — git's rule, independently reimplemented
Digest	SHA-1, 40 lowercase hex digits, fixed by the identifier grammar itself
Specification	docs/persistent-identifiers.rst (SWHID v1.6, 2021-04-30); swh.model.git_objects for the manifests

Overview

What it solves

Citing a piece of source code by an identifier that is intrinsic — checkable without the archive that issued it — regardless of which version control system or distribution format the code was found in. The spec states the property that makes the identifier worth having (persistent-identifiers.rst):

An important property of any SWHID is that its core identifier is intrinsic: it can be computed from the object itself, without having to rely on any third party. […] when a software artifact is obtained from Software Heritage by resolving a SWHID, it is straightforward to verify that it is exactly the intended one: just compute the core identifier from the artefact itself, and check that it is the same as the core identifier part of the SHWID

That is the same verification contract git, OSTree and REAPI offer — but here it is the published, quotable form of the digest, printed in papers and registered as a URI scheme, rather than an implementation detail of a store.

Design philosophy

Deduplication across heterogeneous sources is the stated motivation, and the archive's own data model page names the sources it has to reconcile (Data model):

migration from one VCS to another—e.g., migrations from Subversion to Git, which are really popular these days—resulting in additional copies, but in a different distribution format, of the very same development histories.

[…] Realizing that principle, the Software Heritage archive is conceptually a single (big) Merkle Direct Acyclic Graph (DAG) […] In a Merkle structure each node is identified by an intrinsic identifier computed as a cryptographic hash of the node content.

The consequence is the axis this subject contributes: a Subversion checkout and a git clone of the same tree must land on the same dir node, so the serialization cannot be either system's native one. Software Heritage picked git's — and then had to extend it, because git's commit format cannot express everything Mercurial and Subversion carry (see Extending git's manifests).

Git compatibility is explicitly held to be accidental, not contractual (persistent-identifiers.rst):

Note that Git compatibility is incidental and is not guaranteed to be maintained in future versions of this scheme (or Git).

How it works

A SWHID is a mandatory core identifier plus optional qualifiers, generated by this grammar (persistent-identifiers.rst):

<identifier> ::= <identifier_core> [ <qualifiers> ] ;

<identifier_core> ::= "swh" ":" <scheme_version> ":" <object_type> ":" <object_id> ;
<scheme_version> ::= "1" ;
<object_type> ::=
    "snp"  (* snapshot *)
  | "rel"  (* release *)
  | "rev"  (* revision *)
  | "dir"  (* directory *)
  | "cnt"  (* content *)
  ;
<object_id> ::= 40 * <hex_digit> ;  (* intrinsic object id, as hex-encoded SHA1 *)

swh:1:cnt:94a9ed024d3859793618152ea559a168bbcbb5e2 is the content of a file holding the GPL3 text; swh:1:dir:d198bc9d7a6bcf6db04f476d29314f157507d505 a directory of Darktable sources. swhids.py implements exactly this as a regular expression, and adds two extended types that never appear in a published SWHID for source code — ori (origin) and emd (raw extrinsic metadata):

SWHID_TYPES = ["snp", "rel", "rev", "dir", "cnt"]
EXTENDED_SWHID_TYPES = SWHID_TYPES + ["ori", "emd"]
SWHID_SEP = ":"
SWHID_CTXT_SEP = ";"
SWHID_QUALIFIERS = {"origin", "anchor", "visit", "path", "lines"}

The five core types are hashed by five functions in git_objects.py; four of them reproduce git byte for byte, and the fifth has no git counterpart:

Type	Manifest	Git equivalent
cnt	blob <len>\0<bytes>, hashed with SHA-1 (sha1_git)	git blob id
dir	git tree body: <octal perms> <name>\0<20 raw bytes>, no separators	git tree id
rev	tree/parent/author/committer/<extra headers>, blank line, message	git commit id
rel	object/type/tag/tagger, blank line, message	git tag id
snp	SWH-specific, snapshot-typed manifest of branches	none

Qualifiers: identity separated from provenance

This is the part no other subject in the catalog has. The core identifier says what; the qualifiers say where it was seen, and they are explicitly not part of the hash (persistent-identifiers.rst):

swh:1:cnt:4d99d2d18326621ccdd70f5ea66c2e2ac236ad8b;
  origin=https://gitorious.org/ocamlp3l/ocamlp3l_cvs.git;
  visit=swh:1:snp:d7f1b9eb7ccb596c2622c4780febaa02549830f9;
  anchor=swh:1:rev:2db189928c94d62a3b4757b3eec68f0a4d4113f0;
  path=/Examples/SimpleFarm/simplefarm.ml;
  lines=9-15

Four context qualifiers — origin (the URI the object was observed at), visit (a snapshot of one crawl), anchor (a node the path is relative to), path (the absolute path from the anchor's root directory) — plus one fragment qualifier, lines. The spec's justification is a direct consequence of a Merkle DAG being a DAG:

This is particularly important, as the same object can be reached in the Merkle graph following different paths starting from different nodes (or anchors), and it may have been retrieved from different origins, that may evolve between different visits

A content-addressed blob has no path and no history; dedup is precisely the erasure of both. Qualifiers put them back outside the digest, which is why the spec's own recommendation is to prefer the most recomputable node and re-attach its context:

swh:1:dir: SWHIDs are the most robust SWHIDs, as they can be recomputed from the simplest objects (a directory structure on a filesystem), even when all metadata is lost […] you should add an anchor qualifier to swh:1:dir: SWHIDs whenever possible

Extending git's manifests to hold non-git history

sha1_git is only adequate for cnt and dir because bytes and trees are VCS-neutral. Revisions are not, and git_objects.py says so in its module docstring:

Most of the functions in this module take as argument an object from swh.model.model, and format it like a git object. […] but with extensions, as SWH's model is a superset of Git's:

• extensions of existing types (eg. revision/commit and release/tag dates can be expressed with precision up to milliseconds, to support formatting Mercurial objects)
• new types, for SWH's specific needs […]
• support for somewhat corrupted git objects that we need to reproduce

Two mechanisms carry the difference. Fractional timestamps: format_date emits an integer when there are no microseconds and a trailing-zero-stripped decimal otherwise, "to 'future-proof' our representation". extra_headers: an ordered list of key/value pairs appended after committer, with multi-line values escaped by indenting continuations one space — the same escaping git uses for GPG signatures, reused to smuggle Mercurial and Subversion metadata into a commit manifest.

So swh:1:rev:… for a Mercurial changeset is the SHA-1 of a git-shaped commit object that no git repository ever contained. It is a valid git commit by construction, and its identity is stable — but it is synthesized by the loader, not observed.

The snapshot manifest, and why it is length-prefixed

Snapshots are the one object type git has no analogue for, and their manifest does something the tree manifest does not (git_objects.py):

1. Branches are sorted using the name as key, in bytes order.
2. For each branch, the following bytes are output:

• the type of the branch target: content, directory, revision, release or snapshot […]; alias for branches referencing another branch; dangling for dangling branches
• an ascii space […] the branch name (as raw bytes) […] a null byte
• the length of the target identifier, as an ascii-encoded decimal number (20 for current intrinsic identifiers, 0 for dangling branches, the length of the target branch name for branch aliases)
• a colon (:) […] the identifier of the target object

Note that, akin to directory manifests, there is no separator between entries. Because of symbolic branches, identifiers are of arbitrary length but are length-encoded to avoid ambiguity.

Git trees get away with no length field because every target is exactly 20 bytes. The moment an entry's target became variable-length — a symbolic branch whose target is another branch's name — the format needed an explicit <decimal>: prefix. That is a canonical-form technique absent from every other subject here, and it generalizes: fixed-width targets are what let git, OSTree and Software Heritage's own dir manifests concatenate entries without delimiters at all.

Dimension 1 — node model

Exactly git's five modes, enumerated as DentryPerms in from_disk.py:

content = 0o100644
executable_content = 0o100755
symlink = 0o120000
directory = 0o040000
revision = 0o160000

mode_to_perms is the normalization that makes heterogeneous sources agree, and it is deliberately lossy: symlink and directory by S_ISLNK/S_ISDIR, then "file is executable in any way" — mode & 0o111 — collapsing to 0o100755, everything else to 0o100644. Group and other bits, uid, gid, xattrs and timestamps are gone before the tree is built. That is the same floor as NAR plus symlinks-as-modes, and far below OSTree's.

The richer node model lives above the tree instead: rev, rel and snp are first-class addressable nodes, where git has commits and tags but nothing naming a whole repository's ref set. A dir entry may target a rev (0o160000), so the level-1 tree and the history DAG are one graph.

Dimension 2 — canonical form and ordering

Git's implicit-slash rule, independently reimplemented (git_objects.py):

def directory_entry_sort_key(entry: model.DirectoryEntry):
    ...
    if type_ == "dir":
        return name + b"/"
    else:
        return name

This is the same function git's base_name_compare computes by substituting / for a directory name's terminating NUL, expressed the other way round — append the byte rather than substitute it. So Software Heritage is a second independent oracle for the survey's most consequential ordering finding, beside Nix's git.cc, and its docstring states the equivalence claim outright: "using the following algorithm, which is equivalent to the git algorithm for trees".

Snapshot manifests sort by plain byte order with no slash rule, because branch names are not paths. One system, two orderings, selected by object type.

Duplicate entries get a fifth answer to a question the catalog already has three of. A Directory cannot hold two entries with one name, but archived git repositories do. model.py's Directory.from_possibly_duplicated_entries resolves it by keeping the original bytes and repairing the model:

As it is impossible for a Directory instances to have more than one entry with a given names, this function computes a raw_manifest and renames one of the entries before constructing the Directory.

The renaming is not arbitrary — rev entries win over dir entries, which win over files, "because renaming them would likely break git submodules entirely" and because renaming a directory "affects the path of every file in the dir" — and the loser is renamed to <name>_<first 10 hex of its target>. The raw_manifest field then overrides hashing entirely: compute_hash returns sha1(raw_manifest) whenever one is present, and check() refuses a raw_manifest that was not needed. Identity stays byte-exact with the upstream object; the in-memory model is the thing that bends.

Its docstring is unambiguous that this is a containment mechanism, not a feature: "This should only be used as a last resort, and only set in the Git loader, for objects too corrupt to fit the data model."

Dimension 3 — level-1 composition

A Merkle DAG, and the largest one in the survey: not just directories into roots, but roots into revisions, revisions into releases, and releases and revisions into snapshots — plus ori and emd extended nodes for the crawling metadata. A directory subtree is shared not only across commits of one project but across projects, forges and VCSs, which is the entire storage argument for the archive.

The archive also runs the shared-subtree property in reverse: the ExtID node type records "a relationship between an original identifier of an artifact, in its native/upstream environment, and a core SWHID" (Data model), with an optional payload "useful for reconstructing the original tarball from the directory". That side table exists precisely because normalization is lossy — a Mercurial changeset id or a tarball's checksum is not recoverable from the dir/rev node it produced.

Dimension 4 — level-2 granularity

Whole blob, with git's blob <len>\0 framing, no chunk tree and no range addressing. sha1_git is however only one column of several: hashutil.py's MultiHash computes sha1, sha256, sha1_git and blake2s256 in one pass by default. Only sha1_git is the identity; the rest are lookup keys for matching artifacts found elsewhere (a distribution's published sha256, for instance). Distinguishing the identity digest from co-computed alternates is a pattern worth naming — REAPI's negotiated hash functions solve the same problem by agreement rather than by computing all of them.

Dimension 5 — digest

SHA-1, and the grammar freezes it: <object_id> ::= 40 * <hex_digit>. There is no algorithm tag, no multihash prefix, and no per-repository choice as in a SHA-256 git repository. The declared extension point is <scheme_version>, currently 1, with the promise that "future editions will use higher version numbers, possibly breaking backward compatibility, but without breaking the resolvability of SWHIDs that conform to previous versions". Agility is therefore a scheme-version migration, not a field — the strictest position in the survey, and the deliberate cost of an identifier meant to be printed in a journal article.

No length accompanies the digest, unlike REAPI's {hash, size_bytes} or Nix's NarSize. The one place Software Heritage does length-prefix is inside the snapshot manifest, and for disambiguation rather than for truncation detection.

Dimension 6 — partial verification

Per object, as for every Merkle level-1 scheme: a dir digest verifies that directory and transitively any subtree, without the siblings. Nothing finer — there is no intra-file tree, so a 2 GiB content is all or nothing.

What is unusual is that partial verification is exposed as an end-user contract rather than an internal store property: swh identify recomputes a core identifier locally, so a citation in a paper can be checked against a directory on disk with no network. The qualifiers carry no integrity at all — an origin= or path= can be wrong, or right at one visit and wrong at the next, and nothing in the identifier detects it. Identity is verifiable; provenance is asserted.

Strengths

• One identity across git, Mercurial, Subversion and tarballs — the axis nothing else in the survey has, and the only subject that treats cross-ecosystem agreement as the primary requirement.
• Intrinsic and publishable: a digest designed to be quoted, resolved by third parties (identifiers.org, N2T), and re-verified offline.
• Identity cleanly separated from provenance via qualifiers, instead of either smuggling context into the hash or losing it.
• An independent reimplementation of git's tree ordering, in a different language, by a different organization, with the equivalence claimed explicitly — a second oracle for the trickiest rule in the field.
• raw_manifest keeps identity byte-exact with upstream even for objects the data model cannot represent, instead of silently rehashing them.
• Reaches above the tree: snp names a whole repository's ref set, which git has no object for.

Weaknesses

• SHA-1, welded into the identifier grammar by 40 * <hex_digit>. A migration is a new scheme version, i.e. every published identifier changes form.
• Normalization is lossy and one-way — group/other permission bits, uid, gid, xattrs and any non-git VCS metadata outside extra_headers do not survive; ExtID is a side table, not a reversal.
• Synthesized revisions: a rev for a Mercurial or Subversion history is a git commit object that never existed, so its identity depends on the loader's conventions as much as on the upstream data.
• Qualifiers are unverifiable, and the spec's own advice ("equip identifiers with as many qualifiers as possible") maximizes the unverifiable surface.
• Git compatibility — the property most implementers actually rely on — is documented as incidental and not guaranteed.
• Inherits git's ordering surprise wholesale, including non-consecutive duplicates.

Key design decisions and trade-offs

Decision	Rationale	Trade-off
Reuse sha1_git as the normalization target for every source	Any git clone already has the right ids; the archive dedups against them for free	Bakes git's framing, five modes, ordering rule and SHA-1 into an identifier meant to outlive git
Extend the commit manifest (fractional dates, extra_headers) rather than define a new one	Mercurial and Subversion history becomes expressible without a second format	A rev id is loader-dependent and names an object no upstream repository holds
Qualifiers outside the hash	Identity stays intrinsic and dedup-friendly; context is still recorded	Provenance is asserted, never verified
Fix the digest width in the grammar	Identifiers are fixed-length, greppable, embeddable in URIs and papers	No algorithm agility short of a scheme-version break
raw_manifest escape hatch	A corrupt upstream object keeps its true hash instead of being silently rewritten	The model and the bytes can disagree; a field exists whose only job is to bypass the hash function
Length-prefix snapshot branch targets	Alias targets are branch names, so targets stopped being fixed-width	A second, different manifest grammar inside the same system
A new snp type with no git analogue	A repository's ref set is the thing a crawl actually observes	The one object type with no external oracle to check it against
ExtID as a side table	Native ids stay recoverable for re-export and for incremental crawling	Recoverability is a database property, not a property of the identifier

Sources

• docs/persistent-identifiers.rst — SWHID v1.6: the grammar, the object types, qualifiers, git compatibility, the "prefer swh:1:dir: plus an anchor" advice
• swh/model/git_objects.py — directory_entry_sort_key, directory_git_object, revision_git_object, snapshot_git_object, format_date, the module docstring on extending git's model
• swh/model/swhids.py — the SWHID regular expression, SWHID_QUALIFIERS, the ori/emd extended types
• swh/model/from_disk.py — DentryPerms and mode_to_perms, i.e. the lossy on-disk normalization
• swh/model/model.py — raw_manifest, Directory.from_possibly_duplicated_entries, ExtID, Origin
• swh/model/hashutil.py — git_object_header, MultiHash, the sha1_git algorithm name
• Data model — the Merkle DAG rationale, the VCS-migration duplication argument, ExtID and raw extrinsic metadata
• swh/core/nar.py — Software Heritage's own NAR implementation, used to match Nix-sourced artifacts