e39f7e99d1
This was a big roadblock for a while: with the new feature of inlined files, the existing move logic was fundamentally flawed. To pull off atomic moves between two different metadata-pairs, littlefs uses a simple, if a bit clumsy trick. 1. Marks entry as "moving" 2. Copies entry to new metadata-pair 3. Deletes old entry If power is lost before the move operation is completed, we will find the "moving" tag. This means there may or may not be an incomplete move on the filesystem. In this case, we simply search for the moved entry, if we find it, we remove the old entry, otherwise we just remove the "moving" tag. This worked perfectly, until we introduced inlined files. See, unlike the existing directory and ctz entries, inlined files have no guarantee they are unique. There is nothing we can search for that will allow us to find a moved file unless we assign entries globally-unique ids. (note that moves are fundamentally rename operations, so searching for names does not make sense). --- Solving this problem required completely restructuring how littlefs handled moves and pulled out a really old idea that had been left in the cutting room floor back when littlefs was going through many designs: xored-globals. The problem xored-globals solves is the need to maintain some global state via commits to these distributed, independent metadata-pairs. The idea is that we can use some sort of symmetric operation, such as xor, to introduces deltas of the global state that can be committed atomically along with any other info to these metadata-pairs. This means that to figure out our global state, we xor together the global delta stored in every metadata-pair. Which means any commit can update the global state atomically, opening up a whole new set atomic possibilities. There is a couple of downsides. These globals may end up with deltas on every single metadata-pair, effectively duplicating the data for each block. Additionally, these globals need to have multiple copies in RAM. This means and globals need to be a bounded size and very small, since even small globals will have a large footprint. --- On top of xored-globals, it's trivial to fix our move logic. Here we've added an indirect delete tag which allows us to atomically specify a delete of any entry on the filesystem. Our move operation is now: 1. Copy entry to new metadata-pair and atomically xor globals to indirectly delete our original entry. 2. Delete the original entry and xor globals to remove the indirect delete. Extra exciting is that this now takes our relatively clumsy move operation into a sexy guaranteed O(1) move operation with no searching necessary (though we do need to xor globals during mount). Also reintroduced entry struct, now with a specific purpose to describe the metadata-pair + id combo needed by indirect deletes to locate an entry.
The little filesystem
A little fail-safe filesystem designed for embedded systems.
| | | .---._____
.-----. | |
--|o |---| littlefs |
--| |---| |
'-----' '----------'
| | |
Bounded RAM/ROM - The littlefs is designed to work with a limited amount of memory. Recursion is avoided and dynamic memory is limited to configurable buffers that can be provided statically.
Power-loss resilient - The littlefs is designed for systems that may have random power failures. The littlefs has strong copy-on-write guarantees and storage on disk is always kept in a valid state.
Wear leveling - Since the most common form of embedded storage is erodible flash memories, littlefs provides a form of dynamic wear leveling for systems that can not fit a full flash translation layer.
Example
Here's a simple example that updates a file named boot_count every time
main runs. The program can be interrupted at any time without losing track
of how many times it has been booted and without corrupting the filesystem:
#include "lfs.h"
// variables used by the filesystem
lfs_t lfs;
lfs_file_t file;
// configuration of the filesystem is provided by this struct
const struct lfs_config cfg = {
// block device operations
.read = user_provided_block_device_read,
.prog = user_provided_block_device_prog,
.erase = user_provided_block_device_erase,
.sync = user_provided_block_device_sync,
// block device configuration
.read_size = 16,
.prog_size = 16,
.block_size = 4096,
.block_count = 128,
.lookahead = 128,
};
// entry point
int main(void) {
// mount the filesystem
int err = lfs_mount(&lfs, &cfg);
// reformat if we can't mount the filesystem
// this should only happen on the first boot
if (err) {
lfs_format(&lfs, &cfg);
lfs_mount(&lfs, &cfg);
}
// read current count
uint32_t boot_count = 0;
lfs_file_open(&lfs, &file, "boot_count", LFS_O_RDWR | LFS_O_CREAT);
lfs_file_read(&lfs, &file, &boot_count, sizeof(boot_count));
// update boot count
boot_count += 1;
lfs_file_rewind(&lfs, &file);
lfs_file_write(&lfs, &file, &boot_count, sizeof(boot_count));
// remember the storage is not updated until the file is closed successfully
lfs_file_close(&lfs, &file);
// release any resources we were using
lfs_unmount(&lfs);
// print the boot count
printf("boot_count: %d\n", boot_count);
}
Usage
Detailed documentation (or at least as much detail as is currently available) can be found in the comments in lfs.h.
As you may have noticed, littlefs takes in a configuration structure that defines how the filesystem operates. The configuration struct provides the filesystem with the block device operations and dimensions, tweakable parameters that tradeoff memory usage for performance, and optional static buffers if the user wants to avoid dynamic memory.
The state of the littlefs is stored in the lfs_t type which is left up
to the user to allocate, allowing multiple filesystems to be in use
simultaneously. With the lfs_t and configuration struct, a user can
format a block device or mount the filesystem.
Once mounted, the littlefs provides a full set of POSIX-like file and directory functions, with the deviation that the allocation of filesystem structures must be provided by the user.
All POSIX operations, such as remove and rename, are atomic, even in event of power-loss. Additionally, no file updates are actually committed to the filesystem until sync or close is called on the file.
Other notes
All littlefs have the potential to return a negative error code. The errors
can be either one of those found in the enum lfs_error in lfs.h,
or an error returned by the user's block device operations.
In the configuration struct, the prog and erase function provided by the
user may return a LFS_ERR_CORRUPT error if the implementation already can
detect corrupt blocks. However, the wear leveling does not depend on the return
code of these functions, instead all data is read back and checked for
integrity.
If your storage caches writes, make sure that the provided sync function
flushes all the data to memory and ensures that the next read fetches the data
from memory, otherwise data integrity can not be guaranteed. If the write
function does not perform caching, and therefore each read or write call
hits the memory, the sync function can simply return 0.
Reference material
DESIGN.md - DESIGN.md contains a fully detailed dive into how littlefs actually works. I would encourage you to read it since the solutions and tradeoffs at work here are quite interesting.
SPEC.md - SPEC.md contains the on-disk specification of littlefs with all the nitty-gritty details. Can be useful for developing tooling.
Testing
The littlefs comes with a test suite designed to run on a PC using the emulated block device found in the emubd directory. The tests assume a Linux environment and can be started with make:
make test
Related projects
Mbed OS - The easiest way to get started with littlefs is to jump into Mbed, which already has block device drivers for most forms of embedded storage. The littlefs is available in Mbed OS as the LittleFileSystem class.
littlefs-fuse - A FUSE wrapper for littlefs. The project allows you to mount littlefs directly on a Linux machine. Can be useful for debugging littlefs if you have an SD card handy.
littlefs-js - A javascript wrapper for littlefs. I'm not sure why you would want this, but it is handy for demos. You can see it in action here.