From a8ab9c3445eff718f8201ef2eefdaeb69a9fb867 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 09:10:21 -0500 Subject: [PATCH 01/16] document key files in a new Internals page this is still incomplete as it only describes key files, but doesn't clearly say how chunks are encrypted or decrypted. this address parts of #29 but eventually that document should also cover #27, #28 and maybe #45 --- docs/global.rst.inc | 4 +++ docs/internals.rst | 69 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+) create mode 100644 docs/internals.rst diff --git a/docs/global.rst.inc b/docs/global.rst.inc index 694f4d96..15f38ea0 100644 --- a/docs/global.rst.inc +++ b/docs/global.rst.inc @@ -12,6 +12,10 @@ .. _github: https://github.com/jborg/attic .. _OpenSSL: https://www.openssl.org/ .. _Python: http://www.python.org/ +.. _PBKDF2: https://en.wikipedia.org/wiki/PBKDF2 +.. _SHA256: https://en.wikipedia.org/wiki/SHA-256 +.. _HMAC: https://en.wikipedia.org/wiki/HMAC +.. _msgpack: http://msgpack.org/ .. _`msgpack-python`: https://pypi.python.org/pypi/msgpack-python/ .. _llfuse: https://pypi.python.org/pypi/llfuse/ .. _homebrew: http://mxcl.github.io/homebrew/ diff --git a/docs/internals.rst b/docs/internals.rst new file mode 100644 index 00000000..c2554872 --- /dev/null +++ b/docs/internals.rst @@ -0,0 +1,69 @@ +.. include:: global.rst.inc +.. _internals: + +Internals +========= + + +Key files +--------- + +When initialized with the ``init -e keyfile`` command, |project_name| +needs an associated file in ``$HOME/.attic/keys`` to read and write +the repository. As with most crypto code in |project_name|, the format +of those files is defined in `attic/key.py`_. The format is based on +msgpack_, base64 encoding and PBKDF2_ SHA256 encryption, which is +then encoded again in a msgpack_. + +The internal data structure is as follows: + +version + currently always an integer, 1 + +repository_id + the ``id`` field in the ``config`` ``INI`` file of the repository. + +enc_key + the AES encryption key + +enc_hmac_key + the HMAC key (32 bytes) + +id_key + another HMAC key? unclear. + +chunk_seed + unknown + +Those fields are encoded using msgpack_. The utf-8-encoded phassphrase +is encrypted with a PBKDF2_ and SHA256_ using 100000 iterations and a +random 32 bytes salt to give us a derived key. The derived key is 32 +bytes long. A HMAC_ SHA256_ checksum of the above fields is generated +with the derived key, then the derived key is also used to encrypt the +above pack of fields. Then the result is stored in a another msgpack_ +formatted as follows: + +version + currently always an integer, 1 + +salt + random 32 bytes salt used to encrypt the passphrase + +iterations + number of iterations used to encrypt the passphrase + +algorithm + the hashing algorithm used to encrypt the passphrase and do the HMAC + checksum + +hash + the HMAC checksum of the encrypted passphrase key + +data + the passphrase key, encrypted with AES over a PBKDF2_ SHA256 hash + described above + +The resulting msgpack_ is then encoded using base64 and written to the +key file, wrapped using the textwrap_ module with a header. The header +is a single line with the string ``ATTIC_KEY``, a space and a +hexadecimal representation of the repository id. From 9f0ed2a8c04c5dbed30b4d1ce1ef534ef4b0303b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:03:20 -0500 Subject: [PATCH 02/16] clarify some bits I missed --- docs/global.rst.inc | 1 + docs/index.rst | 1 + docs/internals.rst | 23 ++++++++++++----------- 3 files changed, 14 insertions(+), 11 deletions(-) diff --git a/docs/global.rst.inc b/docs/global.rst.inc index 15f38ea0..a6236f60 100644 --- a/docs/global.rst.inc +++ b/docs/global.rst.inc @@ -15,6 +15,7 @@ .. _PBKDF2: https://en.wikipedia.org/wiki/PBKDF2 .. _SHA256: https://en.wikipedia.org/wiki/SHA-256 .. _HMAC: https://en.wikipedia.org/wiki/HMAC +.. _AES: https://en.wikipedia.org/wiki/AES .. _msgpack: http://msgpack.org/ .. _`msgpack-python`: https://pypi.python.org/pypi/msgpack-python/ .. _llfuse: https://pypi.python.org/pypi/llfuse/ diff --git a/docs/index.rst b/docs/index.rst index 3d9f1198..711eaf15 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -50,6 +50,7 @@ User's Guide quickstart usage faq + internals Getting help ============ diff --git a/docs/internals.rst b/docs/internals.rst index c2554872..bdcf6aa0 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -24,21 +24,22 @@ repository_id the ``id`` field in the ``config`` ``INI`` file of the repository. enc_key - the AES encryption key + the key used to encrypt data with AES (256 bits) enc_hmac_key - the HMAC key (32 bytes) + the key used to HMAC the resulting AES-encrypted data (256 bits) id_key - another HMAC key? unclear. + the key used to HMAC the above chunks, the resulting hash is + stored out of band (256 bits) chunk_seed - unknown + the seed for the buzhash chunking table (signed 32 bit integer) Those fields are encoded using msgpack_. The utf-8-encoded phassphrase is encrypted with a PBKDF2_ and SHA256_ using 100000 iterations and a -random 32 bytes salt to give us a derived key. The derived key is 32 -bytes long. A HMAC_ SHA256_ checksum of the above fields is generated +random 256 bits salt to give us a derived key. The derived key is 256 +bits long. A HMAC_ SHA256_ checksum of the above fields is generated with the derived key, then the derived key is also used to encrypt the above pack of fields. Then the result is stored in a another msgpack_ formatted as follows: @@ -47,20 +48,20 @@ version currently always an integer, 1 salt - random 32 bytes salt used to encrypt the passphrase + random 256 bits salt used to encrypt the passphrase iterations - number of iterations used to encrypt the passphrase + number of iterations used to encrypt the passphrase (currently 100000) algorithm the hashing algorithm used to encrypt the passphrase and do the HMAC - checksum + checksum (currently the string ``sha256``) hash - the HMAC checksum of the encrypted passphrase key + the HMAC checksum of the encrypted derived key data - the passphrase key, encrypted with AES over a PBKDF2_ SHA256 hash + the derived key, encrypted with AES over a PBKDF2_ SHA256 hash described above The resulting msgpack_ is then encoded using base64 and written to the From 3f27c367fe644d2df8691e9cf532957ac6beedad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:04:35 -0500 Subject: [PATCH 03/16] document more internals, based on mailing list discussion this should address #27, #28 and #29 at least at a basic level it is mostly based on the mailing list discussion mentionned in #27, with some reformatting and merging of different posts. --- docs/global.rst.inc | 2 + docs/internals.rst | 105 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+) diff --git a/docs/global.rst.inc b/docs/global.rst.inc index a6236f60..72d15126 100644 --- a/docs/global.rst.inc +++ b/docs/global.rst.inc @@ -12,6 +12,7 @@ .. _github: https://github.com/jborg/attic .. _OpenSSL: https://www.openssl.org/ .. _Python: http://www.python.org/ +.. _Buzhash: https://en.wikipedia.org/wiki/Buzhash .. _PBKDF2: https://en.wikipedia.org/wiki/PBKDF2 .. _SHA256: https://en.wikipedia.org/wiki/SHA-256 .. _HMAC: https://en.wikipedia.org/wiki/HMAC @@ -28,3 +29,4 @@ .. _Arch Linux: https://aur.archlinux.org/packages/attic/ .. _Slackware: http://slackbuilds.org/result/?search=Attic .. _Cython: http://cython.org/ +.. _mailing list discussion about internals: http://librelist.com/browser/attic/2014/5/6/questions-and-suggestions-about-inner-working-of-attic> \ No newline at end of file diff --git a/docs/internals.rst b/docs/internals.rst index bdcf6aa0..94eef02f 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -4,6 +4,111 @@ Internals ========= +This page documents the internal data structures and storage +mechanisms of |project_name|. It is partly based on `mailing list +discussion about internals`_ and also on static code analysis. It may +not be exactly up to date with the current source code. + +Indexes and memory usage +------------------------ + +Repository index + 40 bytes x N ~ 200MB (If a remote repository is + used this will be allocated on the remote side) + +Chunk lookup index + 44 bytes x N ~ 220MB + +File chunk cache + probably 80-100 bytes x N ~ 400MB + +The chunk lookup index (chunk hash -> reference count, size, ciphered +size ; in file cache/chunk) and the repository index (chunk hash -> +segment, offset ; in file repo/index.%d) are stored in a sort of hash +table, directly mapped in memory from the file content, with only one +slot per bucket, but that spreads the collisions to the following +buckets. As a consequence the hash is just a start position for a linear +search, and if the element is not in the table the index is linearly +crossed until an empty bucket is found. When the table is full at 90% +its size is doubled, when it's empty at 25% its size is halfed. So +operations on it have a variable complexity between constant and linear +with low factor, and memory overhead varies between 10% and 300%. + +The file chunk cache (file path hash -> age, inode number, size, +mtime_ns, chunks hashes ; in file cache/files) is stored as a python +associative array storing python objects, which generate a lot of +overhead. This takes around 240 bytes per file without the chunk +list, to be compared to at most 64 bytes of real data (depending on data +alignment), and around 80 bytes per chunk hash (vs 32), with a minimum +of ~250 bytes even if only one chunck hash. The inode number is stored +to make sure we distinguish between different files, as a single path +may not be unique accross different archives in different setups. + +Repository structure +-------------------- + +|project_name| is a "filesystem based transactional key value store". + +Objects referenced by a key (256bits id/hash) are stored in line in +files (segments) of size approx 5MB in repo/data. They contain : +header size, crc, size, tag, key, data. Tag is either ``PUT``, +``DELETE``, or ``COMMIT``. Segments are built locally, and then +uploaded. + +A segment file is basically a transaction log where each repository +operation is appended to the file. So if an object is written to the +repository a ``PUT`` tag is written to the file followed by the object +id and data. And if an object is deleted a ``DELETE`` tag is appended +followed by the object id. A ``COMMIT`` tag is written when a +repository transaction is committed. When a repository is opened any +``PUT`` or ``DELETE`` operations not followed by a ``COMMIT`` tag are +discarded since they are part of a partial/uncommitted transaction. + +The manifest is an object with an id of only zeros (32 bytes), that +references all the archives. It contains : version, list of archives, +timestamp, config. Each archive contains: name, id, time. It is the last +object stored, in the last segment, and is replaced each time. + +The archive metadata does not contain the file items directly. Only +references to other objects that contain that data. An archive is an +object that contain metadata : version, name, items list, cmdline, +hostname, username, time. Each item represents a file or directory or +symlink is stored as a ``item`` dictionnary that contains: path, list +of chunks, user, group, uid, gid, mode (item type + permissions), +source (for links), rdev (for devices), mtime, xattrs, acl, +bsdfiles. ``ctime`` (change time) is not stored because there is no +API to set it and it is reset every time an inode's metadata is changed. + +All items are serialized using msgpack and the resulting byte stream +is fed into the same chunker used for regular file data and turned +into deduplicated chunks. The reference to these chunks is then added +to the archvive metadata. This allows the archive to store many files, +beyond the ``MAX_OBJECT_SIZE`` barrier of 20MB. + +A chunk is an object as well, of course, and its id is the hash of its +(unencrypted and uncompressed) content. + +Hints are stored in a file (repo/hints) and contain: version, list of +segments, compact. + +Chunks +------ + +|project_name| uses a rolling checksum with Buzhash_ algorithm, with +window size of 4095 bytes, with a minimum of 1024, and triggers when +the last 16 bits of the checksum are null, producing chunks of 64kB on +average. All these parameters are fixed. The buzhash table is altered +by XORing it with a seed randomly generated once for the archive, and +stored encrypted in the keyfile. + +Encryption +---------- + +AES_ is used with CTR mode of operation (so no need of padding). A 64 +bits initialization vector is used, a SHA256_ based HMAC_ is computed +on the encrypted chunk with a random 64 bits nonce and both are stored +in the chunk. The header of each chunk is actually : TYPE(1) + +HMAC(32) + NONCE(8). Encryption and HMAC use two different keys. Key files --------- From fd56bf0887d9097825f8a95961ac6d8dc325c023 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:20:23 -0500 Subject: [PATCH 04/16] document the repo config file and more storage properties again taken from the mailing list, mostly --- docs/internals.rst | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index 94eef02f..b4694034 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -24,7 +24,7 @@ File chunk cache The chunk lookup index (chunk hash -> reference count, size, ciphered size ; in file cache/chunk) and the repository index (chunk hash -> -segment, offset ; in file repo/index.%d) are stored in a sort of hash +segment, offset ; in file ``repo/index.%d``) are stored in a sort of hash table, directly mapped in memory from the file content, with only one slot per bucket, but that spreads the collisions to the following buckets. As a consequence the hash is just a start position for a linear @@ -44,16 +44,19 @@ of ~250 bytes even if only one chunck hash. The inode number is stored to make sure we distinguish between different files, as a single path may not be unique accross different archives in different setups. +The ``index.%d`` files are random access but those files can be +recreated if damaged or lost using "attic check --repair". + Repository structure -------------------- |project_name| is a "filesystem based transactional key value store". Objects referenced by a key (256bits id/hash) are stored in line in -files (segments) of size approx 5MB in repo/data. They contain : +files (segments) of size approx 5MB in ``repo/data``. They contain : header size, crc, size, tag, key, data. Tag is either ``PUT``, ``DELETE``, or ``COMMIT``. Segments are built locally, and then -uploaded. +uploaded. Those files are strictly append-only and modified only once. A segment file is basically a transaction log where each repository operation is appended to the file. So if an object is written to the @@ -101,6 +104,26 @@ average. All these parameters are fixed. The buzhash table is altered by XORing it with a seed randomly generated once for the archive, and stored encrypted in the keyfile. +Repository config file +---------------------- + +Each repository has a ``config`` file which which is a ``INI`` +formatted file which looks like this: + + [repository] + version = 1 + segments_per_dir = 10000 + max_segment_size = 5242880 + id = 57d6c1d52ce76a836b532b0e42e677dec6af9fca3673db511279358828a21ed6 + +This is where the ``repository.id`` is stored. It is a unique +identifier for repositories. It will not change if you move the +repository around so you can make a local transfer then decide to move +the repository in another (even remote) location at a later time. + +|project_name| will do a POSIX read lock on that file when operating +on the repository. + Encryption ---------- From 1fde2a97711b5894d59132495fe7e8ba095244e0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:20:52 -0500 Subject: [PATCH 05/16] add more details on how encryption works --- docs/internals.rst | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index b4694034..a31fbb1c 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -127,11 +127,28 @@ on the repository. Encryption ---------- -AES_ is used with CTR mode of operation (so no need of padding). A 64 +AES_ is used with CTR mode of operation (so no need for padding). A 64 bits initialization vector is used, a SHA256_ based HMAC_ is computed on the encrypted chunk with a random 64 bits nonce and both are stored -in the chunk. The header of each chunk is actually : TYPE(1) + -HMAC(32) + NONCE(8). Encryption and HMAC use two different keys. +in the chunk. The header of each chunk is : ``TYPE(1)` + +``HMAC(32)`` + ``NONCE(8)`` + ``CIPHERTEXT``. Encryption and HMAC use +two different keys. + +In AES CTR mode you can think of the IV as the start value for the +counter. The counter itself is incremented by one after each 16 byte +block. The IV/counter is not required to be random but it must NEVER be +reused. So to accomplish this Attic initializes the encryption counter +to be higher than any previously used counter value before encrypting +new data. + +To reduce payload size only 8 bytes of the 16 bytes nonce is saved in +the payload, the first 8 bytes are always zeros. This does not affect +security but limits the maximum repository capacity to only 295 +exabytes (2**64 * 16 bytes). + +Encryption keys are either a passphrase, passed through the +``ATTIC_PASSPHRASE`` environment or prompted on the commandline, or +stored in automatically generated key files. Key files --------- From ddca3b856bb41a33cc68a848f715282859b823a6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:30:57 -0500 Subject: [PATCH 06/16] add a more gentle introduction --- docs/internals.rst | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index a31fbb1c..0e01336b 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -9,6 +9,30 @@ mechanisms of |project_name|. It is partly based on `mailing list discussion about internals`_ and also on static code analysis. It may not be exactly up to date with the current source code. +|project_name| stores its data in a `Repository`. Each repository can +hold multiple `Archives`, which represent individual backups that +contain a full archive of the files specified when the backup was +performed. Deduplication is performed across multiple backups, both on +data and metadata, using `Segments` chunked with the Buzhash_ +algorithm. Each repository has the following file structure: + +README + simple text file describing the repository + +config + description of the repository, includes the unique identifier. also + acts as a lock file + +data/ + directory where the actual data (`segments`) is stored + +hints.%d + undocumented + +index.%d + cache of the file indexes. those files can be regenerated with + ``check --repair`` + Indexes and memory usage ------------------------ @@ -45,7 +69,7 @@ to make sure we distinguish between different files, as a single path may not be unique accross different archives in different setups. The ``index.%d`` files are random access but those files can be -recreated if damaged or lost using "attic check --repair". +recreated if damaged or lost using ``check --repair``. Repository structure -------------------- From 688ba109ef27a4e493dd8257e2eabb3a8c05ff58 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:35:48 -0500 Subject: [PATCH 07/16] reorder to be more logical and more gentle --- docs/internals.rst | 104 ++++++++++++++++++++++++--------------------- 1 file changed, 55 insertions(+), 49 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index 0e01336b..ede9fb0e 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -33,43 +33,25 @@ index.%d cache of the file indexes. those files can be regenerated with ``check --repair`` -Indexes and memory usage ------------------------- - -Repository index - 40 bytes x N ~ 200MB (If a remote repository is - used this will be allocated on the remote side) - -Chunk lookup index - 44 bytes x N ~ 220MB +Repository config file +---------------------- -File chunk cache - probably 80-100 bytes x N ~ 400MB +Each repository has a ``config`` file which which is a ``INI`` +formatted file which looks like this: -The chunk lookup index (chunk hash -> reference count, size, ciphered -size ; in file cache/chunk) and the repository index (chunk hash -> -segment, offset ; in file ``repo/index.%d``) are stored in a sort of hash -table, directly mapped in memory from the file content, with only one -slot per bucket, but that spreads the collisions to the following -buckets. As a consequence the hash is just a start position for a linear -search, and if the element is not in the table the index is linearly -crossed until an empty bucket is found. When the table is full at 90% -its size is doubled, when it's empty at 25% its size is halfed. So -operations on it have a variable complexity between constant and linear -with low factor, and memory overhead varies between 10% and 300%. + [repository] + version = 1 + segments_per_dir = 10000 + max_segment_size = 5242880 + id = 57d6c1d52ce76a836b532b0e42e677dec6af9fca3673db511279358828a21ed6 -The file chunk cache (file path hash -> age, inode number, size, -mtime_ns, chunks hashes ; in file cache/files) is stored as a python -associative array storing python objects, which generate a lot of -overhead. This takes around 240 bytes per file without the chunk -list, to be compared to at most 64 bytes of real data (depending on data -alignment), and around 80 bytes per chunk hash (vs 32), with a minimum -of ~250 bytes even if only one chunck hash. The inode number is stored -to make sure we distinguish between different files, as a single path -may not be unique accross different archives in different setups. +This is where the ``repository.id`` is stored. It is a unique +identifier for repositories. It will not change if you move the +repository around so you can make a local transfer then decide to move +the repository in another (even remote) location at a later time. -The ``index.%d`` files are random access but those files can be -recreated if damaged or lost using ``check --repair``. +|project_name| will do a POSIX read lock on that file when operating +on the repository. Repository structure -------------------- @@ -115,7 +97,7 @@ beyond the ``MAX_OBJECT_SIZE`` barrier of 20MB. A chunk is an object as well, of course, and its id is the hash of its (unencrypted and uncompressed) content. -Hints are stored in a file (repo/hints) and contain: version, list of +Hints are stored in a file (``repo/hints``) and contain: version, list of segments, compact. Chunks @@ -128,25 +110,49 @@ average. All these parameters are fixed. The buzhash table is altered by XORing it with a seed randomly generated once for the archive, and stored encrypted in the keyfile. -Repository config file ----------------------- +Indexes +------- -Each repository has a ``config`` file which which is a ``INI`` -formatted file which looks like this: +The chunk lookup index (chunk hash -> reference count, size, ciphered +size ; in file cache/chunk) and the repository index (chunk hash -> +segment, offset ; in file ``repo/index.%d``) are stored in a sort of hash +table, directly mapped in memory from the file content, with only one +slot per bucket, but that spreads the collisions to the following +buckets. As a consequence the hash is just a start position for a linear +search, and if the element is not in the table the index is linearly +crossed until an empty bucket is found. When the table is full at 90% +its size is doubled, when it's empty at 25% its size is halfed. So +operations on it have a variable complexity between constant and linear +with low factor, and memory overhead varies between 10% and 300%. - [repository] - version = 1 - segments_per_dir = 10000 - max_segment_size = 5242880 - id = 57d6c1d52ce76a836b532b0e42e677dec6af9fca3673db511279358828a21ed6 +The file chunk cache (file path hash -> age, inode number, size, +mtime_ns, chunks hashes ; in file cache/files) is stored as a python +associative array storing python objects, which generate a lot of +overhead. This takes around 240 bytes per file without the chunk +list, to be compared to at most 64 bytes of real data (depending on data +alignment), and around 80 bytes per chunk hash (vs 32), with a minimum +of ~250 bytes even if only one chunck hash. The inode number is stored +to make sure we distinguish between different files, as a single path +may not be unique accross different archives in different setups. -This is where the ``repository.id`` is stored. It is a unique -identifier for repositories. It will not change if you move the -repository around so you can make a local transfer then decide to move -the repository in another (even remote) location at a later time. +The ``index.%d`` files are random access but those files can be +recreated if damaged or lost using ``check --repair``. -|project_name| will do a POSIX read lock on that file when operating -on the repository. +Indexes memory usage +-------------------- + +Here is the estimated memory usage of |project_name| when using those +indexes: + +Repository index + 40 bytes x N ~ 200MB (If a remote repository is + used this will be allocated on the remote side) + +Chunk lookup index + 44 bytes x N ~ 220MB + +File chunk cache + probably 80-100 bytes x N ~ 400MB Encryption ---------- From d58b6ddf28ef07d87f77a10b472637ec7ea159fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:55:03 -0500 Subject: [PATCH 08/16] fix reference errors and remove reference to source code --- docs/internals.rst | 179 +++++++++++++++++++++++++++++++-------------- 1 file changed, 124 insertions(+), 55 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index ede9fb0e..585266a9 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -56,37 +56,81 @@ on the repository. Repository structure -------------------- -|project_name| is a "filesystem based transactional key value store". - -Objects referenced by a key (256bits id/hash) are stored in line in -files (segments) of size approx 5MB in ``repo/data``. They contain : -header size, crc, size, tag, key, data. Tag is either ``PUT``, -``DELETE``, or ``COMMIT``. Segments are built locally, and then -uploaded. Those files are strictly append-only and modified only once. - -A segment file is basically a transaction log where each repository -operation is appended to the file. So if an object is written to the -repository a ``PUT`` tag is written to the file followed by the object -id and data. And if an object is deleted a ``DELETE`` tag is appended +|project_name| is a "filesystem based transactional key value +store". It makes extensive use of msgpack_ to store data and, unless +otherwise noted, data is stored in msgpack_ encoded files. + +Objects referenced by a key (256bits id/hash) are stored inline in +files (`segments`) of size approx 5MB in ``repo/data``. They contain: + +* header size +* crc +* size +* tag +* key +* data + +Segments are built locally, and then uploaded. Those files are +strictly append-only and modified only once. + +Tag is either ``PUT``, ``DELETE``, or ``COMMIT``. A segment file is +basically a transaction log where each repository operation is +appended to the file. So if an object is written to the repository a +``PUT`` tag is written to the file followed by the object id and +data. And if an object is deleted a ``DELETE`` tag is appended followed by the object id. A ``COMMIT`` tag is written when a repository transaction is committed. When a repository is opened any ``PUT`` or ``DELETE`` operations not followed by a ``COMMIT`` tag are discarded since they are part of a partial/uncommitted transaction. The manifest is an object with an id of only zeros (32 bytes), that -references all the archives. It contains : version, list of archives, -timestamp, config. Each archive contains: name, id, time. It is the last -object stored, in the last segment, and is replaced each time. +references all the archives. It contains: + +* version +* list of archives +* timestamp +* config + +Each archive contains: + +* name +* id +* time + +It is the last object stored, in the last segment, and is replaced +each time. The archive metadata does not contain the file items directly. Only references to other objects that contain that data. An archive is an -object that contain metadata : version, name, items list, cmdline, -hostname, username, time. Each item represents a file or directory or -symlink is stored as a ``item`` dictionnary that contains: path, list -of chunks, user, group, uid, gid, mode (item type + permissions), -source (for links), rdev (for devices), mtime, xattrs, acl, -bsdfiles. ``ctime`` (change time) is not stored because there is no -API to set it and it is reset every time an inode's metadata is changed. +object that contain metadata: + +* version +* name +* items list +* cmdline +* hostname +* username +* time + +Each item represents a file or directory or +symlink is stored as a ``item`` dictionnary that contains: + +* path +* list of chunks +* user +* group +* uid +* gid +* mode (item type + permissions) +* source (for links) +* rdev (for devices) +* mtime +* xattrs +* acl +* bsdfiles + +``ctime`` (change time) is not stored because there is no API to set +it and it is reset every time an inode's metadata is changed. All items are serialized using msgpack and the resulting byte stream is fed into the same chunker used for regular file data and turned @@ -97,8 +141,11 @@ beyond the ``MAX_OBJECT_SIZE`` barrier of 20MB. A chunk is an object as well, of course, and its id is the hash of its (unencrypted and uncompressed) content. -Hints are stored in a file (``repo/hints``) and contain: version, list of -segments, compact. +Hints are stored in a file (``repo/hints``) and contain: + +* version +* list of segments +* compact Chunks ------ @@ -113,31 +160,55 @@ stored encrypted in the keyfile. Indexes ------- -The chunk lookup index (chunk hash -> reference count, size, ciphered -size ; in file cache/chunk) and the repository index (chunk hash -> -segment, offset ; in file ``repo/index.%d``) are stored in a sort of hash -table, directly mapped in memory from the file content, with only one -slot per bucket, but that spreads the collisions to the following -buckets. As a consequence the hash is just a start position for a linear -search, and if the element is not in the table the index is linearly -crossed until an empty bucket is found. When the table is full at 90% -its size is doubled, when it's empty at 25% its size is halfed. So -operations on it have a variable complexity between constant and linear -with low factor, and memory overhead varies between 10% and 300%. - -The file chunk cache (file path hash -> age, inode number, size, -mtime_ns, chunks hashes ; in file cache/files) is stored as a python -associative array storing python objects, which generate a lot of -overhead. This takes around 240 bytes per file without the chunk -list, to be compared to at most 64 bytes of real data (depending on data -alignment), and around 80 bytes per chunk hash (vs 32), with a minimum -of ~250 bytes even if only one chunck hash. The inode number is stored -to make sure we distinguish between different files, as a single path -may not be unique accross different archives in different setups. - -The ``index.%d`` files are random access but those files can be +There are two main indexes: the chunk lookup index and the repository +index. There is also the file chunk cache. + +The chunk lookup index is stored in ``cache/chunk`` and is indexed on +the ``chunk hash``. It contains: + +* reference count +* size +* ciphered size + +The repository index is stored in ``repo/index.%d`` and is also +indexed on ``chunk hash`` and contains: + +* segment +* offset + +The repository index files are random access but those files can be recreated if damaged or lost using ``check --repair``. +Both indexes are stored as hash tables, directly mapped in memory from +the file content, with only one slot per bucket, but that spreads the +collisions to the following buckets. As a consequence the hash is just +a start position for a linear search, and if the element is not in the +table the index is linearly crossed until an empty bucket is +found. When the table is full at 90% its size is doubled, when it's +empty at 25% its size is halfed. So operations on it have a variable +complexity between constant and linear with low factor, and memory +overhead varies between 10% and 300%. + +The file chunk cache is stored in ``cache/files`` and is indexed on +the ``file path hash`` and contains: + +* age +* inode number +* size +* mtime_ns +* chunks hashes + +The inode number is stored to make sure we distinguish between +different files, as a single path may not be unique accross different +archives in different setups. + +The file chunk cache is stored as a python associative array storing +python objects, which generate a lot of overhead. This takes around +240 bytes per file without the chunk list, to be compared to at most +64 bytes of real data (depending on data alignment), and around 80 +bytes per chunk hash (vs 32), with a minimum of ~250 bytes even if +only one chunck hash. + Indexes memory usage -------------------- @@ -158,9 +229,9 @@ Encryption ---------- AES_ is used with CTR mode of operation (so no need for padding). A 64 -bits initialization vector is used, a SHA256_ based HMAC_ is computed +bits initialization vector is used, a `HMAC-SHA256`_ is computed on the encrypted chunk with a random 64 bits nonce and both are stored -in the chunk. The header of each chunk is : ``TYPE(1)` + +in the chunk. The header of each chunk is : ``TYPE(1)`` + ``HMAC(32)`` + ``NONCE(8)`` + ``CIPHERTEXT``. Encryption and HMAC use two different keys. @@ -185,10 +256,8 @@ Key files When initialized with the ``init -e keyfile`` command, |project_name| needs an associated file in ``$HOME/.attic/keys`` to read and write -the repository. As with most crypto code in |project_name|, the format -of those files is defined in `attic/key.py`_. The format is based on -msgpack_, base64 encoding and PBKDF2_ SHA256 encryption, which is -then encoded again in a msgpack_. +the repository. The format is based on msgpack_, base64 encoding and +PBKDF2_ SHA256 encryption, which is then encoded again in a msgpack_. The internal data structure is as follows: @@ -212,9 +281,9 @@ chunk_seed the seed for the buzhash chunking table (signed 32 bit integer) Those fields are encoded using msgpack_. The utf-8-encoded phassphrase -is encrypted with a PBKDF2_ and SHA256_ using 100000 iterations and a +is encrypted with PBKDF2_ and SHA256_ using 100000 iterations and a random 256 bits salt to give us a derived key. The derived key is 256 -bits long. A HMAC_ SHA256_ checksum of the above fields is generated +bits long. A `HMAC-SHA256`_ checksum of the above fields is generated with the derived key, then the derived key is also used to encrypt the above pack of fields. Then the result is stored in a another msgpack_ formatted as follows: From b7c26735f77a228fd38ecb931ddb8b42b25178e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:59:02 -0500 Subject: [PATCH 09/16] fix formatting issues --- docs/global.rst.inc | 5 +---- docs/internals.rst | 18 +++++++++--------- 2 files changed, 10 insertions(+), 13 deletions(-) diff --git a/docs/global.rst.inc b/docs/global.rst.inc index 72d15126..08e54801 100644 --- a/docs/global.rst.inc +++ b/docs/global.rst.inc @@ -7,16 +7,13 @@ .. _deduplication: https://en.wikipedia.org/wiki/Data_deduplication .. _AES: https://en.wikipedia.org/wiki/Advanced_Encryption_Standard .. _HMAC-SHA256: http://en.wikipedia.org/wiki/HMAC +.. _SHA256: https://en.wikipedia.org/wiki/SHA-256 .. _PBKDF2: https://en.wikipedia.org/wiki/PBKDF2 .. _ACL: https://en.wikipedia.org/wiki/Access_control_list .. _github: https://github.com/jborg/attic .. _OpenSSL: https://www.openssl.org/ .. _Python: http://www.python.org/ .. _Buzhash: https://en.wikipedia.org/wiki/Buzhash -.. _PBKDF2: https://en.wikipedia.org/wiki/PBKDF2 -.. _SHA256: https://en.wikipedia.org/wiki/SHA-256 -.. _HMAC: https://en.wikipedia.org/wiki/HMAC -.. _AES: https://en.wikipedia.org/wiki/AES .. _msgpack: http://msgpack.org/ .. _`msgpack-python`: https://pypi.python.org/pypi/msgpack-python/ .. _llfuse: https://pypi.python.org/pypi/llfuse/ diff --git a/docs/internals.rst b/docs/internals.rst index 585266a9..9d6d8b7a 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -37,13 +37,13 @@ Repository config file ---------------------- Each repository has a ``config`` file which which is a ``INI`` -formatted file which looks like this: +formatted file which looks like this:: - [repository] - version = 1 - segments_per_dir = 10000 - max_segment_size = 5242880 - id = 57d6c1d52ce76a836b532b0e42e677dec6af9fca3673db511279358828a21ed6 + [repository] + version = 1 + segments_per_dir = 10000 + max_segment_size = 5242880 + id = 57d6c1d52ce76a836b532b0e42e677dec6af9fca3673db511279358828a21ed6 This is where the ``repository.id`` is stored. It is a unique identifier for repositories. It will not change if you move the @@ -309,6 +309,6 @@ data described above The resulting msgpack_ is then encoded using base64 and written to the -key file, wrapped using the textwrap_ module with a header. The header -is a single line with the string ``ATTIC_KEY``, a space and a -hexadecimal representation of the repository id. +key file, wrapped using the builtin ``textwrap`` module with a +header. The header is a single line with the string ``ATTIC_KEY``, a +space and a hexadecimal representation of the repository id. From e80e6c4dbb1cf2ca6630e377ab985dbd8fdcba87 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Antoine=20Beaupr=C3=A9?= Date: Tue, 16 Dec 2014 10:59:12 -0500 Subject: [PATCH 10/16] better titles --- docs/internals.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index 9d6d8b7a..cefa8968 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -33,8 +33,8 @@ index.%d cache of the file indexes. those files can be regenerated with ``check --repair`` -Repository config file ----------------------- +Config file +----------- Each repository has a ``config`` file which which is a ``INI`` formatted file which looks like this:: @@ -53,8 +53,8 @@ the repository in another (even remote) location at a later time. |project_name| will do a POSIX read lock on that file when operating on the repository. -Repository structure --------------------- +Segments and archives +--------------------- |project_name| is a "filesystem based transactional key value store". It makes extensive use of msgpack_ to store data and, unless From b7718f044ddf32b1ad1b3a7d1f0f786787161c4a Mon Sep 17 00:00:00 2001 From: anarcat Date: Wed, 17 Dec 2014 10:11:02 -0500 Subject: [PATCH 11/16] Update internals.rst --- docs/internals.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index cefa8968..ef43054f 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -309,6 +309,6 @@ data described above The resulting msgpack_ is then encoded using base64 and written to the -key file, wrapped using the builtin ``textwrap`` module with a +key file, wrapped using the standard ``textwrap`` module with a header. The header is a single line with the string ``ATTIC_KEY``, a space and a hexadecimal representation of the repository id. From 8f8a035e9322834bd1657b63b1124a172a3fcd36 Mon Sep 17 00:00:00 2001 From: anarcat Date: Thu, 5 Mar 2015 08:41:48 -0500 Subject: [PATCH 12/16] fix a bunch of typos this should fix the comments identified as `typo` and other small quirks found by @ThomasWaldmann. --- docs/internals.rst | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index ef43054f..598c26eb 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -113,7 +113,7 @@ object that contain metadata: * time Each item represents a file or directory or -symlink is stored as a ``item`` dictionnary that contains: +symlink is stored as an ``item`` dictionary that contains: * path * list of chunks @@ -135,7 +135,7 @@ it and it is reset every time an inode's metadata is changed. All items are serialized using msgpack and the resulting byte stream is fed into the same chunker used for regular file data and turned into deduplicated chunks. The reference to these chunks is then added -to the archvive metadata. This allows the archive to store many files, +to the archive metadata. This allows the archive to store many files, beyond the ``MAX_OBJECT_SIZE`` barrier of 20MB. A chunk is an object as well, of course, and its id is the hash of its @@ -199,7 +199,7 @@ the ``file path hash`` and contains: * chunks hashes The inode number is stored to make sure we distinguish between -different files, as a single path may not be unique accross different +different files, as a single path may not be unique across different archives in different setups. The file chunk cache is stored as a python associative array storing @@ -207,7 +207,7 @@ python objects, which generate a lot of overhead. This takes around 240 bytes per file without the chunk list, to be compared to at most 64 bytes of real data (depending on data alignment), and around 80 bytes per chunk hash (vs 32), with a minimum of ~250 bytes even if -only one chunck hash. +only one chunk hash. Indexes memory usage -------------------- @@ -238,12 +238,12 @@ two different keys. In AES CTR mode you can think of the IV as the start value for the counter. The counter itself is incremented by one after each 16 byte block. The IV/counter is not required to be random but it must NEVER be -reused. So to accomplish this Attic initializes the encryption counter +reused. So to accomplish this |project_name| initializes the encryption counter to be higher than any previously used counter value before encrypting new data. To reduce payload size only 8 bytes of the 16 bytes nonce is saved in -the payload, the first 8 bytes are always zeros. This does not affect +the payload, the first 8 bytes are always zeroes. This does not affect security but limits the maximum repository capacity to only 295 exabytes (2**64 * 16 bytes). @@ -280,7 +280,7 @@ id_key chunk_seed the seed for the buzhash chunking table (signed 32 bit integer) -Those fields are encoded using msgpack_. The utf-8-encoded phassphrase +Those fields are processed using msgpack_. The utf-8 encoded phassphrase is encrypted with PBKDF2_ and SHA256_ using 100000 iterations and a random 256 bits salt to give us a derived key. The derived key is 256 bits long. A `HMAC-SHA256`_ checksum of the above fields is generated @@ -292,20 +292,20 @@ version currently always an integer, 1 salt - random 256 bits salt used to encrypt the passphrase + random 256 bits salt used to process the passphrase iterations - number of iterations used to encrypt the passphrase (currently 100000) + number of iterations used to process the passphrase (currently 100000) algorithm - the hashing algorithm used to encrypt the passphrase and do the HMAC + the hashing algorithm used to process the passphrase and do the HMAC checksum (currently the string ``sha256``) hash - the HMAC checksum of the encrypted derived key + the HMAC of the encrypted derived key data - the derived key, encrypted with AES over a PBKDF2_ SHA256 hash + the derived key, encrypted with AES over a PBKDF2_ SHA256 key described above The resulting msgpack_ is then encoded using base64 and written to the From 87cb4a481335da75b00cfcacf78adff6a68ee496 Mon Sep 17 00:00:00 2001 From: anarcat Date: Thu, 5 Mar 2015 08:48:23 -0500 Subject: [PATCH 13/16] expand on the chunk id hash mechanism according to @ThomasWaldmann, the algorithm varies according to whether encryption is enabled. --- docs/internals.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/internals.rst b/docs/internals.rst index 598c26eb..45daa52b 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -138,8 +138,8 @@ into deduplicated chunks. The reference to these chunks is then added to the archive metadata. This allows the archive to store many files, beyond the ``MAX_OBJECT_SIZE`` barrier of 20MB. -A chunk is an object as well, of course, and its id is the hash of its -(unencrypted and uncompressed) content. +A chunk is an object as well, of course. The chunk id is either +HMAC-SHA256_, when encryption is used, or a SHA256_ hash otherwise. Hints are stored in a file (``repo/hints``) and contain: From 0ba86357d77e7b9e4c14017efff0d0ee613da54f Mon Sep 17 00:00:00 2001 From: anarcat Date: Thu, 5 Mar 2015 08:51:26 -0500 Subject: [PATCH 14/16] clarify that 4095 bytes is not a typo i am actually assuming this right now, i haven't double-checked --- docs/internals.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index 45daa52b..bd4022c1 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -151,7 +151,7 @@ Chunks ------ |project_name| uses a rolling checksum with Buzhash_ algorithm, with -window size of 4095 bytes, with a minimum of 1024, and triggers when +window size of 4095 bytes (`0xFFF`), with a minimum of 1024, and triggers when the last 16 bits of the checksum are null, producing chunks of 64kB on average. All these parameters are fixed. The buzhash table is altered by XORing it with a seed randomly generated once for the archive, and From 5f882e976d0d7746d8b9e54416175a4ef0bb4b50 Mon Sep 17 00:00:00 2001 From: anarcat Date: Thu, 5 Mar 2015 08:57:52 -0500 Subject: [PATCH 15/16] clarify the index memory usage analysis it seems I extracted that data from [this mailing list post][] which in turn takes it from [this github comment][]. [this mailing list post]: http://librelist.com/browser/attic/2014/5/6/questions-and-suggestions-about-inner-working-of-attic/ [this github comment]: https://github.com/jborg/attic/issues/26#issuecomment-35439254 --- docs/internals.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index bd4022c1..9172ec20 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -213,7 +213,7 @@ Indexes memory usage -------------------- Here is the estimated memory usage of |project_name| when using those -indexes: +indexes. Repository index 40 bytes x N ~ 200MB (If a remote repository is @@ -225,6 +225,9 @@ Chunk lookup index File chunk cache probably 80-100 bytes x N ~ 400MB +In the above we assume 350GB of data that we divide on an average 64KB +chunk size, so N is around 5.3 million. + Encryption ---------- From ecee5a0b514845e2957f0bb8055de89d2a25792c Mon Sep 17 00:00:00 2001 From: anarcat Date: Thu, 5 Mar 2015 09:00:06 -0500 Subject: [PATCH 16/16] PDKF is a key derivation function do not use the word "encryption", as it is actually closer to "hashing" anyways. --- docs/internals.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/internals.rst b/docs/internals.rst index 9172ec20..52e2938a 100644 --- a/docs/internals.rst +++ b/docs/internals.rst @@ -260,7 +260,7 @@ Key files When initialized with the ``init -e keyfile`` command, |project_name| needs an associated file in ``$HOME/.attic/keys`` to read and write the repository. The format is based on msgpack_, base64 encoding and -PBKDF2_ SHA256 encryption, which is then encoded again in a msgpack_. +PBKDF2_ SHA256 hashing, which is then encoded again in a msgpack_. The internal data structure is as follows: