Now, with the off, diff, and len parameters in each commit entry, we can build
up directory commits that resize entries. This adds complexity but opens
up the directory blocks to be much more flexible.
The main concern is that resizing entries can push around neighboring entries
in surprising ways, such as pushing them into new directory blocks when a
directory splits. This can break littlefs's internal logic in how it tracks
in-flight entries. The most problematic example being open files.
Fortunately, this is helped by a global linked-list of all files and
directories opened by the filesystem. As entries change size, the state
of open files/dirs may be updated as needed. Note this already needed to
exist for the ability to remove files/dirs, which has the same issue.
Expiremental implementation. This opens up the opportunity to use the same
commit description for both commits and appends, which effectively do the same
thing.
This should lead to better code reuse.
Really all this means is that the internal commit function was changed
from taking an array of "commit structures" to a linked-list of "commit
structures". The benefit of a linked-list is that layers of commit
functions can pull off some minor modifications to the description of
the commit. Most notably, commit functions can add additional entries
that will be atomically written out and CRCed along with the initial
commit.
Also a minor benefit, this is one less parameter when committing a
directory with zero entries.
Previously, commits could only come from memory in RAM. This meant any
entries had to be buffered in their entirety before they could be moved
to a different directory pair. By adding parameters for specifying
commits from existing entries stored on disk, we allow any sized entries
to be moved between directory pairs with a fixed RAM cost.
The separation of data-structure vs entry type has been implicit for a
while now, and even taken advantage of to simplify the traverse logic.
Explicitely separating the data-struct and entry types allows us to
introduce new data structures (inlined files).
The optional config structure options up the possibility of adding
file-level configuration in a backwards compatible manner.
Also adds possibility to open multiple files with LFS_NO_MALLOC
enabled thanks to dpgeorge
Also bumped minor version to v1.5
Before this, littlefs incorrectly assumed corrupt blocks were only the result
of our own modification. This would be fine for most cases of freshly
erased storage, but for storage with block-level ECC this wasn't always
true.
Fortunately, it's quite easy for littlefs to handle this case correctly,
as long as corrupt storage always reports that it is corrupt, which for
most forms of ECC is the case unless we perform a write on the storage.
found by rojer
When using "%d" or "%x" with uint32_t types, arm-none-eabi-gcc reports
warnings like below:
-- >8 -- >8 -- >8 -- >8 -- >8 -- >8 --
In file included from lfs.c:8:
lfs_util.h:45:12: warning: format '%d' expects argument of type 'int', but argument 4 has type 'lfs_block_t' {aka 'long unsigned int'} [-Wformat=]
printf("lfs debug:%d: " fmt "\n", __LINE__, __VA_ARGS__)
^~~~~~~~~~~~~~~~
lfs.c:2512:21: note: in expansion of macro 'LFS_DEBUG'
LFS_DEBUG("Found partial move %d %d",
^~~~~~~~~
lfs.c:2512:55: note: format string is defined here
LFS_DEBUG("Found partial move %d %d",
~^
%ld
-- >8 -- >8 -- >8 -- >8 -- >8 -- >8 --
Fix this by replacing "%d" and "%x" with `"%" PRIu32` and `"%" PRIx32`.
As a shortcut, littlefs never bother to zero any of the buffers is used.
It didn't need to because it would always write out the entirety of the
data it needed.
Unfortunately, this, combined with the extra padding used to align
buffers to the nearest prog size, would lead to uninitialized data
getting written out to disk.
This means unrelated file data could be written to different parts of
storage, or worse, information leaked from the malloc calls could be
written out to disk unnecessarily.
found by rojer
- Fixed shadowed variable warnings in lfs_dir_find.
- Fixed unused parameter warnings when LFS_NO_MALLOC is enabled.
- Added extra warning flags to CFLAGS.
- Updated tests so they don't shadow the "size" variable for -Wshadow
Opening multiple files simultaneously is not supported without dynamic memory,
but the previous behaviour would just let the files overwrite each other, which
could lead to bad errors down the line
found by husigeza
Paths such as the following were causing issues:
/tea/hottea/.
/tea/hottea/..
Unfortunately the existing structure for path lookup didn't make it very
easy to introduce proper handling in this case without duplicating the
entire skip logic for paths. So the lfs_dir_find function had to be
restructured a bit.
One odd side-effect of this is that now lfs_dir_find includes the
initial fetch operation. This kinda breaks the fetch -> op pattern of
the dir functions, but does come with a nice code size reduction.
As pointed out by davidefer, the lookahead pointer modular arithmetic
does not work around integer overflow when the pointer size is not a
multiple of the block count.
To avoid overflow problems, the easy solution is to stop trying to
work around integer overflows and keep the lookahead offset inside the
block device. To make this work, the ack was modified into a resetable
counter that is decremented every block allocation.
As a plus, quite a bit of the allocation logic ended up simplified.
One of the big simplifications in littlefs's implementation is the
complete lack of tracking free blocks, allowing operations to simply
drop blocks that are no longer in use.
However, this means the lookahead buffer can easily contain outdated
blocks that were previously deleted. This is usually fine, as littlefs
will rescan the storage if it can't find a free block in the lookahead
buffer, but after changes that caused littlefs to more conservatively
respect the alloc acks (e611cf5), any scanned blocks after an ack would
be incorrectly trusted.
The fix is to eagerly scan ahead in the lookahead when we allocate so
that alloc acks are better able to discredit old lookahead blocks. Since
usually alloc acks are tightly coupled to allocations of one or two blocks,
this allows littlefs to properly rescan every set of allocations.
This may still be a concern if there is a long series of worn out
blocks, but in the worst case littlefs will conservatively avoid using
blocks it's not sure about.
Found by davidefer
Like most of the lfs_dir_t functions, lfs_dir_append is responsible for
updating the lfs_dir_t struct if the underlying directory block is
moved. This property makes handling worn out blocks much easier by
removing the amount of state that needs to be considered during a
directory update.
However, extending the dir chain is a bit of a corner case. It's not
changing the old block, but callers of lfs_dir_append do assume the
"entry" will reside in "dir" after lfs_dir_append completes.
This issue only occurs when creating files, since mkdir does not use
the entry after lfs_dir_append. Unfortunately, the tests against
extending the directory chain were all made using mkdir.
Found by schouleu
Before this patch, when calling lfs_mkdir or lfs_file_open with root
as the target, littlefs wouldn't find the path properly and happily
run into undefined behaviour.
The fix is to populate a directory entry for root in the lfs_dir_find
function. As an added plus, this allowed several special cases around
root to be completely dropped.
Note: It's still expected to modify lfs_utils.h when porting littlefs
to a new target/system. There's just too much room for system-specific
improvements, such as taking advantage of CRC hardware.
Rather, encouraging modification of lfs_util.h and making it easy to
modify and debug should result in better integration with the consuming
systems.
This just adds a bunch of quality-of-life improvements that should help
development and integration in littlefs.
- Macros that require no side-effects are all-caps
- System includes are only brought in when needed
- Malloc/free wrappers
- LFS_NO_* checks for quickly disabling things at the command line
- At least a little-bit more docs
Required to support big-endian processors, with the most notable being
the PowerPC architecture.
On little-endian architectures, these conversions can be optimized out
and have no code impact.
Initial patch provided by gmouchard
Rather than tracking all in-flight blocks blocks during a lookahead,
littlefs uses an ack scheme to mark the first allocated block that
hasn't reached the disk yet. littlefs assumes all blocks since the
last ack are bad or in-flight, and uses this to know when it's out
of storage.
However, these unacked allocations were still being populated in the
lookahead buffer. If the whole block device fits in the lookahead
buffer, _and_ littlefs managed to scan around the whole storage while
an unacked block was still in-flight, it would assume the block was
free and misallocate it.
The fix is to only fill the lookahead buffer up to the last ack.
The internal free structure was restructured to simplify the runtime
calculation of lookahead size.
- Write on read-only file to return LFS_ERR_BADF
- Renaming directory onto file to return LFS_ERR_NOTEMPTY
- Changed LFS_ERR_INVAL in lfs_file_seek to assert
An annoying part of filesystems is that the software library can change
independently of the on-disk structures. For this reason versioning is
very important, and must be handled separately for the software and
on-disk parts.
In this patch, littlefs provides two version numbers at compile time,
with major and minor parts, in the form of 6 macros.
LFS_VERSION // Library version, uint32_t encoded
LFS_VERSION_MAJOR // Major - Backwards incompatible changes
LFS_VERSION_MINOR // Minor - Feature additions
LFS_DISK_VERSION // On-disk version, uint32_t encoded
LFS_DISK_VERSION_MAJOR // Major - Backwards incompatible changes
LFS_DISK_VERSION_MINOR // Minor - Feature additions
Note that littlefs will error if it finds a major version number that
is different, or a minor version number that has regressed.
As a copy-on-write filesystem, the truncate function is a very nice
function to have, as it can take advantage of reusing the data already
written out to disk.
Unfortunately for us, the ctz skip-list does not offer very much benefit
for full traversals. Since the information about which blocks are in
use are spread throughout the file, we can't use the fast-lanes
embedded in the skip-list without missing blocks.
However, it turns out we can at least use the 2nd level of the skip-list
without missing any blocks. From an asymptotic analysis, a constant speed
up isn't interesting, but from a pragmatic perspective, a 2x speedup is
not bad.
littlefs had an unwritten assumption that the block device's program
size would be a multiple of the read size, and the block size would
be a multiple of the program size. This has already caused confusion
for users. Added a note and assert to catch unexpected geometries
early.
Also found that the prog/erase functions indicated they must return
LFS_ERR_CORRUPT to catch bad blocks. This is no longer true as errors
are found by CRC.
In the open call, the LFS_O_TRUNC flag was correctly zeroing the file, but
it wasn't actually writing the change out to disk. This went unnoticed because
in the cases where the truncate was followed by a file write, the
updated contents would be written out correctly.
Marking the file as dirty if the file isn't already truncated fixes the
problem with the least impact. Also added better test cases around
truncating files.
This bug was a result of an annoying corner case around intermingling
signed and unsigned offsets. The boundary check that prevents seeking
a file to a position before the file was preventing valid seeks with
positive offsets.
This corner case is a bit more complicated than it looks because the
offset is signed, while the size of the file is unsigned. Simply
casting both to signed or unsigned offsets won't handle large files.
This was a small hole in the logic that handles initializing the
lookahead buffer. To imitate exhaustion (so the block allocator
will trigger a scan), the lookahead buffer is rewound a full
lookahead and set up to look like it is exhausted. However,
unlike normal allocation, this rewind was not kept aligned to
a multiple of the scan size, which is limited by both the
lookahead buffer and the total storage size.
This bug went unnoticed for so long because it only causes
problems when the block device is both:
1. Not aligned to the lookahead buffer (not a power of 2)
2. Smaller than the lookahead buffer
While this seems like a strange corner case for a block device,
this turned out to be very common for internal flash, especially
when a handleful of blocks are reserved for code.
As it was, if a user operated on a directory while at the same
time iterating over the directory, the directory objects could
fall out of sync. In the best case, files may be skipped while
removing everything in a file, in the worst case, a very poorly
timed directory relocate could be missed.
Simple fix is to add the same directory tracking that is currently
in use for files, at a small code+complexity cost.
Short story, files are no longer committed to directories during
file sync/close if the last write did not complete successfully.
This avoids a set of interesting user-experience issues related
to the end-of-life behaviour of the filesystem.
As a filesystem approaches end-of-life, the chances of running into
LFS_ERR_NOSPC grows rather quickly. Since this condition occurs after
at the end of a devices life, it's likely that operating in these
conditions hasn't been tested thoroughly.
In the specific case of file-writes, you can hit an LFS_ERR_NOSPC after
parts of the file have been written out. If the program simply continues
and closes the file, the file is written out half completed. Since
littlefs has a strong garuntee the prevents half-writes, it's unlikely
this state of the file would be expected.
To make things worse, since close is also responsible for memory
cleanup, it's actually _impossible_ to continue working as it was
without leaking memory.
By prevent the file commits, end-of-life behaviour should at least retain
a previous copy of the filesystem without any surprises.
Specifically around error handling. As is, incorrectly handled
errors could cause higher code to get uninitialized blocks,
potentially leading to writes to arbitray blocks on storage.
This is only an issue in the weird case that are worn down block is
left in the odd state of not being able to change the data that resides
on the block. That being said, this does pop up often when simulating
wear on block devices.
Currently, directory commits checked if the write succeeded by crcing the
block to avoid the additional RAM cost for another buffer. However,
before this commit, directory commits just checked if the block crc was
valid, rather than comparing to the expected crc. This would usually
work, unless the block was stuck in a state with valid crc.
The fix is to simply compare with the expected crc to find errors.
The previous math for determining if we scanned all of disk wasn't set
up correctly in the lfs_mount function. If lookahead == block_count
the lfs_alloc function would think we had already searched the entire
disk.
This is only an issue if we manage to exhaust a block on the first
pass after mount, since lfs_alloc_ack resets the lookahead region
into a valid state after a succesful block allocation.
The littlefs allows buffers to be passed statically in the case
that a system does not have a heap. Unfortunately, this means we
can't round up in the case of an unaligned lookahead buffer.
Double unfortunately, rounding down after clamping to the block device
size could result in a lookahead of zero for block devices < 32 blocks
large.
The assert in littlefs does catch this case, but rounding down prevents
support for < 32 block devices.
The solution is to simply require a 32-bit aligned buffer with an
assert. This avoids runtime problems while allowing a user to pass
in the correct buffer for < 32 block devices. Rounding up can be
handled at higher API levels.
Same runtime cost, however reduces the logic and avoids one of
the two big branches. See the DESIGN.md for more info.
Now uses these equations instead of the messy guess and correct method:
n = (N - w/8(popcount(N/(B-2w/8)) + 2)) / (B-2w/8)
off = N - (B-w2/8)n - w/8popcount(n)
This reduces the O(n^2logn) runtime to read a file to only O(nlog).
The extra O(n) did not touch the disk, so it isn't a problem until the
files become very large, but this solution comes with very little cost.
Long story short, you can find the block index + offset pair for a
CTZ linked-list with this series of formulas:
n' = floor(N / (B - 2w/8))
N' = (B - 2w/8)n' + (w/8)popcount(n')
off' = N - N'
n, off =
n'-1, off'+B if off' < 0
n', off'+(w/8)(ctz(n')+1) if off' >= 0
For the long story, you will need to see the updated DESIGN.md
Initially, I was concerned that the number of pointers in the ctz
linked-list could exceed the storage in a block. Long story short
this isn't really possible outside of extremely small block sizes.
Since clamping impacts the layout of files on disk, removing the
block size removed quite a bit of logic and corner cases. Replaced
with an assert on block size during initialization.
---
Long story long, the minimum block size needed to store all ctz
pointers in a filesystem can be found with this formula:
B = (w/8)*log2(2^w / (B - 2*(w/8)))
where:
B = block size in bytes
w = pointer width in bits
It's not a very pretty formula, but does give us some useful info
if we apply some math:
min block size:
32 bit ctz linked-list = 104 bytes
64 bit ctz linked-list = 448 bytes
For littlefs, 128 bytes is a perfectly reasonable minimum block size.
