Introduced xored-globals logic to fix fundamental problem with moves

This was a big roadblock for a while: with the new feature of inlined
files, the existing move logic was fundamentally flawed.

To pull off atomic moves between two different metadata-pairs, littlefs
uses a simple, if a bit clumsy trick.
1. Marks entry as "moving"
2. Copies entry to new metadata-pair
3. Deletes old entry

If power is lost before the move operation is completed, we will find the
"moving" tag. This means there may or may not be an incomplete move on
the filesystem. In this case, we simply search for the moved entry, if
we find it, we remove the old entry, otherwise we just remove the
"moving" tag.

This worked perfectly, until we introduced inlined files. See, unlike
the existing directory and ctz entries, inlined files have no guarantee
they are unique. There is nothing we can search for that will allow us
to find a moved file unless we assign entries globally-unique ids. (note
that moves are fundamentally rename operations, so searching for names
does not make sense).

---

Solving this problem required completely restructuring how littlefs
handled moves and pulled out a really old idea that had been left in the
cutting room floor back when littlefs was going through many
designs: xored-globals.

The problem xored-globals solves is the need to maintain some global state
via commits to these distributed, independent metadata-pairs. The idea
is that we can use some sort of symmetric operation, such as xor, to
introduces deltas of the global state that can be committed atomically
along with any other info to these metadata-pairs.

This means that to figure out our global state, we xor together the global
delta stored in every metadata-pair.

Which means any commit can update the global state atomically, opening
up a whole new set atomic possibilities.

There is a couple of downsides. These globals may end up with deltas on
every single metadata-pair, effectively duplicating the data for each
block. Additionally, these globals need to have multiple copies in RAM.
This means and globals need to be a bounded size and very small, since even
small globals will have a large footprint.

---

On top of xored-globals, it's trivial to fix our move logic. Here we've
added an indirect delete tag which allows us to atomically specify a
delete of any entry on the filesystem.

Our move operation is now:
1. Copy entry to new metadata-pair and atomically xor globals to
   indirectly delete our original entry.
2. Delete the original entry and xor globals to remove the indirect
   delete.

Extra exciting is that this now takes our relatively clumsy move
operation into a sexy guaranteed O(1) move operation with no searching
necessary (though we do need to xor globals during mount).

Also reintroduced entry struct, now with a specific purpose to describe
the metadata-pair + id combo needed by indirect deletes to locate an
entry.
This commit is contained in:
Christopher Haster
2018-05-29 12:35:23 -05:00
parent 116c1e76de
commit e39f7e99d1
2 changed files with 190 additions and 55 deletions

39
lfs.h
View File

@@ -102,9 +102,10 @@ enum lfs_type {
// internally used types
LFS_TYPE_NAME = 0x010,
LFS_TYPE_MOVE = 0x080,
LFS_TYPE_DELETE = 0x090,
LFS_TYPE_DELETE = 0x020,
LFS_TYPE_SUPERBLOCK = 0x0a0,
LFS_TYPE_SUPERBLOCK = 0x030,
LFS_TYPE_IDELETE = 0x0b0,
LFS_TYPE_SOFTTAIL = 0x0c0,
LFS_TYPE_HARDTAIL = 0x0d0,
LFS_TYPE_CRC = 0x0e0,
@@ -285,6 +286,25 @@ typedef struct lfs_mattrlist {
struct lfs_mattrlist *next;
} lfs_mattrlist_t;
typedef struct lfs_entry {
lfs_block_t pair[2];
uint16_t id;
} lfs_entry_t;
typedef struct lfs_mdir {
lfs_block_t pair[2];
lfs_block_t tail[2];
uint32_t rev;
lfs_off_t off;
uint32_t etag;
uint16_t count;
bool erased;
bool split;
lfs_entry_t idelete;
bool stop_at_commit; // TODO hmmm
uint16_t moveid; // TODO rm me
} lfs_mdir_t;
typedef struct lfs_cache {
lfs_block_t block;
lfs_off_t off;
@@ -307,19 +327,6 @@ typedef struct lfs_file {
lfs_mattrlist_t *attrs;
} lfs_file_t;
typedef struct lfs_mdir {
lfs_block_t pair[2];
lfs_block_t tail[2];
uint32_t rev;
lfs_off_t off;
uint32_t etag;
uint16_t count;
bool erased;
bool split;
bool stop_at_commit; // TODO hmmm
int16_t moveid;
} lfs_mdir_t;
typedef struct lfs_dir {
struct lfs_dir *next;
struct lfs_mdir m;
@@ -363,6 +370,8 @@ typedef struct lfs {
lfs_free_t free;
bool deorphaned;
lfs_entry_t idelete;
lfs_entry_t diff;
lfs_size_t inline_size;
lfs_size_t attrs_size;