Compare commits

...

36 Commits

Author SHA1 Message Date
Christopher Haster
97b5d04bf4 Switched to separate-tag encoding of forward-looking CRCs
Previously forward-looking CRCs was just two new CRC types, one for
commits with forward-looking CRCs, one without. These both contained the
CRC needed to complete the current commit (note that the commit CRC
must come last!).

         [--   32   --|--   32   --|--   32   --|--   32   --]
with:    [  crc3 tag  | nprog size |  nprog crc | commit crc ]
without: [  crc2 tag  | commit crc ]

This meant there had to be several checks for the two possible structure
sizes, messying up the implementation.

         [--   32   --|--   32   --|--   32   --|--   32   --|--   32   --]
with:    [nprogcrc tag| nprog size |  nprog crc | commit tag | commit crc ]
without: [ commit tag | commit crc ]

But we already have a mechanism for storing optional metadata! The
different metadata tags! So why not use a separate tage for the
forward-looking CRC, separate from the commit CRC?

I wasn't sure this would actually help that much, there are still
necessary conditions for wether or not a forward-looking CRC is there,
but in the end it simplified the code quite nicely, and resulted in a ~200 byte
code-cost saving.
2021-01-15 02:00:01 -06:00
Christopher Haster
7535795a44 Cleaned up a few additional commit corner cases
- General cleanup from integration, including cleaning up some older
  commit code
- Partial-prog tests do not make sense when prog_size == block_size
  (there can't be partial-progs!)
- Fixed signed-comparison issue in modified filebd
2020-12-07 01:10:28 -06:00
Christopher Haster
01a3b1f5f7 Initial implementation of forward-looking erase-state CRCs
This change is necessary to handle out-of-order writes found by pjsg's
fuzzing work.

The problem is that it is possible for (non-NOR) block devices to write
pages in any order, or to even write random data in the case of a
power-loss. This breaks littlefs's use of the first bit in a page to
indicate the erase-state.

pjsg notes this behavior is documented in the W25Q here:
https://community.cypress.com/docs/DOC-10507

---

The basic idea here is to CRC the next page, and use this "erase-state CRC" to
check if the next page is erased and ready to accept programs.

.------------------. \   commit
|     metadata     | |
|                  | +---.
|                  | |   |
|------------------| |   |
| erase-state CRC -----. |
|------------------| | | |
|   commit CRC    ---|-|-'
|------------------| / |
|     padding      |   | padding (doesn't need CRC)
|                  |   |
|------------------| \ | next prog
|     erased?      | +-'
|        |         | |
|        v         | /
|                  |
|                  |
'------------------'

This is made a bit annoying since littlefs doesn't actually store the
page (prog_size) in the superblock, since it doesn't need to know the
size for any other operation. We can work around this by storing both
the CRC and size of the next page when necessary.

Another interesting note is that we don't need to any bit tweaking
information, since we read the next page every time we would need to
know how to clobber the erase-state CRC. And since we only read
prog_size, this works really well with our caching, since the caches
must be a multiple of prog_size.

This also brings back the internal lfs_bd_crc function, in which we can
use some optimizations added to lfs_bd_cmp.

Needs some cleanup but the idea is passing most relevant tests.
2020-12-07 00:06:17 -06:00
Christopher Haster
288a5cbc8d Bumped minor version to v2.3 2020-12-04 01:31:27 -06:00
Christopher Haster
5783eea0de Merge pull request #490 from littlefs-project/fix-alloc-eviction
Fix allocation-eviction issue when erase state is multiple of block_cycles+1
2020-12-04 00:49:09 -06:00
Christopher Haster
2bb523421e Moved lfs_mlist_isopen checks into the API wrappers
This indirectly solves an issue with lfs_file_rawclose asserting
when lfs_file_opencfg errors since appending to the mlist occurs
after open. It also may speed up some of the internal operations such as
the lfs_file_write used to resolve unflushed data.

The idea behind adopting mlist over flags is that realistically it's
unlikely for the user to open a significant number of files (enough for
big O to kick in). That being said, moving the mlist asserts into the
API wrappers does protect some of the internal operations from scaling
based on the number of open files.
2020-12-04 00:42:32 -06:00
Noah Gorny
7388b2938a Deprecate LFS_F_OPENED and use lfs_mlist_isused instead
Instead of additional flag, we can just go through the mlist.
2020-12-04 00:26:19 -06:00
Christopher Haster
ce425a56c3 Merge pull request #470 from renesas/SWFLEX-1517-littlefs-thread-safe-option
Add thread safe wrappers
2020-12-03 23:47:32 -06:00
Christopher Haster
a99a93fb27 Added thread-safe build+size reporting to CI 2020-12-03 23:46:59 -06:00
Christopher Haster
45afded784 Moved LFS_TRACE calls to API wrapper functions
This removes quite a bit of extra code needed to entertwine the
LFS_TRACE calls into the original funcions.

Also changed temporary return type to match API declaration where
necessary.
2020-12-03 23:46:59 -06:00
Christopher Haster
00a9ba7826 Tweaked thread-safe implementation
- Stayed on non-system include for lfs_util.h for now
- Named internal functions "lfs_functionraw"
- Merged lfs_fs_traverseraw
- Added LFS_LOCK/UNLOCK macros
- Changed LFS_THREADSAFE from 1/0 to defined/undefined to
  match LFS_READONLY
2020-12-03 23:46:59 -06:00
Bill Gesner
fc6988c7c3 make raw functions static. formatting tweaks 2020-12-03 23:46:54 -06:00
Bill Gesner
d0f055d321 Squash of thread-safe PR cleanup
- expand functions
- add comment
- rename functions
- fix locking issue in format and mount
- use global include
- fix ac6 linker issue
- use the global config file
- address review comments
- minor cleanup
- minor cleanup
- review comments
2020-12-03 23:41:01 -06:00
Christopher Haster
b9fa33f9bc Merge pull request #480 from maximevince/master
Add LFS_READONLY define, to allow smaller builds providing read-only mode
2020-12-03 23:06:00 -06:00
Christopher Haster
2efebf8e9b Added read-only build+size reporting to CI 2020-12-03 23:04:48 -06:00
Maxime Vincent
754b4c3cda Squash of LFS_READONLY cleanup
- undef unavailable function declarations altogether
- even less code, assert on write attempts
- remove LFS_O_WRONLY and other flags when compiling with LFS_READONLY
- do not annotate #endif, as requested
- move ifdef before comments blocks, rework dangling opening bracket
- ifdef file flags that are not needed in read-only mode
- slight refactor
- ifdef LFS_F_ERRED out as well
2020-12-03 23:03:29 -06:00
Christopher Haster
584eb26efc Merge pull request #443 from NoahGorny/add-already-opened-assert
Assert that the file isnt open in lfs_file_opencfg
2020-12-03 22:43:10 -06:00
Noah Gorny
008ebc37df Add lfs_mlist_append/remove helper 2020-12-03 22:42:39 -06:00
Christopher Haster
66272067ab Merge pull request #395 from gmpy/improve-write-performance
lfs_bd_cmp() compares more bytes at one time
2020-12-03 22:34:47 -06:00
Christopher Haster
e273a82679 Merge pull request #487 from littlefs-project/fix-alloc-reset-modulus
Fix several wear-leveling issues found in lfs_alloc_reset
2020-12-03 22:33:47 -06:00
Christopher Haster
1dc6ae94b9 Merge pull request #486 from littlefs-project/fix-assert
Fix assert
2020-12-03 22:32:56 -06:00
Christopher Haster
817ef02d24 Merge pull request #412 from jrast/patch-3
Added littlefs-python to the related projects section
2020-12-03 22:32:04 -06:00
Christopher Haster
b8dcf10974 Changed lfs_dir_alloc to maximize block cycles for new metadata pairs
Previously we only bumped the revision count if an eviction would occur
immediately (and possibly corrupt littlefs). This works, but does risk
an unoptimal superblock size if an almost-exhausted superblock was
allocated during lfs_format.

As pointed out by tim-nordell-nimbelink, we can align the revision count
to maximize the number of block cycles without breaking the existing
requirements of increasing revision counts.

As an added benefit, littlefs's wear-leveling should behave more
consistently after this change.
2020-11-28 22:46:11 -06:00
Christopher Haster
0aba71d0d6 Fixed single unchecked bit during commit verification
This bug was exposed by the bad-block tests due to changes to block
allocation, but could have been hit before these changes.

In flash, when blocks fail, they don't fail in a predictable manner. To
account for this, the bad-block tests check a number of failure
behaviors. The interesting one here is "LFS_TESTBD_BADBLOCK_ERASENOOP",
in which bad blocks can not be erased or programmed, and are stuck with
the data written at the time the blocks go bad.

This is actually a pretty realistic failure behavior, since flash needs a
large voltage to force the electrons of the floating gates. Though
realistically, such a failure would like corrupt the data a bit, not leave the
underlying data perfectly intact.

LFS_TESTBD_BADBLOCK_ERASENOOP is rather interesting to test for because it
means bad blocks can end up with perfectly valid CRCs after a failed write,
confusing littlefs.

---

In this case, we had the perfect series of operations such that a test
was repeatedly writing the same sequence of metadata commits to the same
block, which eventually goes bad, leaving the block stuck with metadata
that occurs later in the sequence.

What this means is that after the first commit, the metadata block
contained both the first and second commits, even though the loop in the
test hadn't reached that point yet.

expected       actual
.----------.  .----------.
| commit 1 |  | commit 1 |
| crc 1    |  | crc 1    |
|          |  | commit 2 <-- (from previous iteration)
|          |  | crc 2    |
'----------'  '----------'

To protect against this, littlefs normally compares the written CRC
against the expected CRC, but because this was the exact same data that
it was going to write, this CRCs end up the same.

Ah! But doesn't littlefs also encode the state of the next page to keep
track of if the next page has been erased or not? Wouldn't that change
between iterations?

It does! In a single bit in the CRC-tag. But thanks to some incorrect
logic attempting to avoid an extra condition in the loop for writing out
padding commits, the CRC that littlefs checked against was the CRC
immediately before we include the "is-next-page-erased" bit.

Changing the verification check to use the same CRC as what is used to
verify commits on fetch solves this problem.
2020-11-22 15:07:16 -06:00
Christopher Haster
0ea2871e24 Fixed typo in scripts/readtree.py
Not sure how this went unnoticed, I guess this is the first bug that
needed in-depth inspection after the a last-minute argument cleanup
in the debug scripts.
2020-11-22 15:05:22 -06:00
Christopher Haster
d04c1392c0 Fixed allocation-eviction issue when erase state is multiple of block_cycles+1
This rather interesting corner-case arises in lfs_dir_alloc anytime the
uninitialized revision count happens to be a multiple of block_cycles+1.

For example, the source of the bug found by tim-nordell-nimbelink:

rev = 2742492087
block_cycles = 100

2742492087 % (100+1) = 0

The reason for this weird block_cycles+1 case is due to a fix for a
previous bug in fe957de. To avoid aliasing, which would cause metadata
pairs to wear unevenly, block_cycles incremented to the next odd number.

Normally, littlefs tweaks the revision count of blocks during
lfs_dir_alloc in order to make sure evictions can't happen on the first
compact. Otherwise, higher-level logic such as lfs_format would break.

However, this wasn't updated with the aliasing fix in fe957de, so
lfs_dir_alloc was only rounding the revision count to the nearest even
number.

The current fix is to change the logic in lfs_dir_alloc to explicitly
check for the eviction condition and increment if eviction would occur.

Found by tim-nordell-nimbelink
2020-11-22 00:40:58 -06:00
Christopher Haster
f215027fd4 Switched to CRC as seed collection function instead of xor
As noted by gtaska, we are sitting on a better hash-combining function
than xor: CRC. Previous issues with xor were solvable, but relying on
xor for this isn't really worth the risk when we already have a CRC
function readily available.

To quote a study found by gtaska:

https://michiel.buddingh.eu/distribution-of-hash-values

> CRC32 seems to score really well, but its graph is skewed by the results
> of Dataset 5 (binary numbers), which may or may not be too synthetic to
> be considered a fair benchmark. But even if you substract the results
> from that test, it does not fare significantly worse than other,
> cryptographic hash functions.
2020-11-20 00:38:41 -06:00
Christopher Haster
1ae4b36f2a Removed unnecessary randomization of offsets in lfs_alloc_reset
On first read, randomizing the allocators offset may seem appropriate
for lfs_alloc_reset. However, it ends up using the filesystem-fed
pseudorandom seed in situations it wasn't designed for.

As noted by gtaska, the combination of using xors for feeding the seed
and multiple traverses of the same CRCs can cause the seed to flip to
zeros with concerning frequency.

Removed the randomization from lfs_alloc_reset, leaving it in only
lfs_mount.

Found by gtaska
2020-11-20 00:18:13 -06:00
Christopher Haster
480cdd9f81 Fixed incorrect modulus in lfs_alloc_reset
Modulus of the offset by block_size was clearly a typo, and should be
block_count. Interesting to note that later moduluses during alloc
calculations prevents this from breaking anything, but as gtaska notes it
could skew the wear-leveling distribution.

Found by guiserle and gtaska
2020-11-20 00:02:19 -06:00
Noah Gorny
6303558aee Use LFS_O_RDWR instead of magic number in lfs_file_* asserts 2020-11-19 01:51:39 +02:00
Noah Gorny
4bd653dd00 Assert that file/dir struct is not reused in lfs_file_opencfg/lfs_dir_open 2020-11-19 01:51:39 +02:00
Maxime Vincent
8e6826c4e2 Add LFS_READYONLY define, to allow smaller builds providing read-only mode 2020-10-28 16:09:13 +01:00
Bill Gesner
10ac6b9cf0 add thread safe wrappers 2020-09-17 23:41:20 +00:00
Shiven Gupta
87a2cb0e41 Fix assert 2020-08-18 17:36:14 -04:00
Jürg Rast
6d0ec5e851 Added littlefs-python to the related projects section
As introduced in #297, I created a python wrapper for littlefs. The wrapper supports two API's: A C-like API which is the same as in C and a more pythonic API which is easier to use if you are more the python guy. The wrapper is built with littlefs 2.2.1 at the moment.
2020-04-13 21:33:30 +02:00
WeiXiong Liao
64f70f51b0 lfs_bd_cmp() compares more bytes at one time
It's very slowly to compare one byte at one time. Here are the
performance I get from 128M spinand with NFTL by sequential writing.

| file size | buffer size  | write speed  |
| 10 MB     | 0   B        | 3206.01 KB/s |
| 10 MB     | 1   B        | 2434.04 KB/s |
| 10 MB     | 2   B        | 2685.78 KB/s |
| 10 MB     | 4   B        | 2857.94 KB/s |
| 10 MB     | 8   B        | 3060.68 KB/s |
| 10 MB     | 16  B        | 3155.30 KB/s |
| 10 MB     | 64  B        | 3193.68 KB/s |
| 10 MB     | 128 B        | 3230.62 KB/s |
| 10 MB     | 256 B        | 3153.03 KB/s |

| 70 MB     | 0   B        | 2258.87 KB/s |
| 70 MB     | 1   B        | 1827.83 KB/s |
| 70 MB     | 2   B        | 1962.29 KB/s |
| 70 MB     | 4   B        | 2074.01 KB/s |
| 70 MB     | 8   B        | 2147.03 KB/s |
| 70 MB     | 64  B        | 2179.92 KB/s |
| 70 MB     | 256 B        | 2179.96 KB/s |

The 0 Byte size means no validation and the 1 Byte size is how
littlefs do before. Based on the above table and to save memory,
comparing 8 bytes at one time is more wonderful.

Signed-off-by: WeiXiong Liao <liaoweixiong@allwinnertech.com>
2020-03-13 15:23:20 +08:00
10 changed files with 1357 additions and 483 deletions

View File

@@ -208,6 +208,38 @@ jobs:
script:
- make test TFLAGS+="-k --valgrind"
# test compilation in read-only mode
- stage: test
env:
- NAME=littlefs-readonly
- CC="arm-linux-gnueabi-gcc --static -mthumb"
- CFLAGS="-Werror -DLFS_READONLY"
if: branch !~ -prefix$
install:
- *install-common
- sudo apt-get install
gcc-arm-linux-gnueabi
libc6-dev-armel-cross
- arm-linux-gnueabi-gcc --version
# report-size will compile littlefs and report the size
script: [*report-size]
# test compilation in thread-safe mode
- stage: test
env:
- NAME=littlefs-threadsafe
- CC="arm-linux-gnueabi-gcc --static -mthumb"
- CFLAGS="-Werror -DLFS_THREADSAFE"
if: branch !~ -prefix$
install:
- *install-common
- sudo apt-get install
gcc-arm-linux-gnueabi
libc6-dev-armel-cross
- arm-linux-gnueabi-gcc --version
# report-size will compile littlefs and report the size
script: [*report-size]
# self-host with littlefs-fuse for fuzz test
- stage: test
env:

View File

@@ -221,6 +221,11 @@ License Identifiers that are here available: http://spdx.org/licenses/
- [littlefs-js] - A javascript wrapper for littlefs. I'm not sure why you would
want this, but it is handy for demos. You can see it in action
[here][littlefs-js-demo].
- [littlefs-python] - A Python wrapper for littlefs. The project allows you
to create images of the filesystem on your PC. Check if littlefs will fit
your needs, create images for a later download to the target memory or
inspect the content of a binary image of the target memory.
- [mklfs] - A command line tool built by the [Lua RTOS] guys for making
littlefs images from a host PC. Supports Windows, Mac OS, and Linux.
@@ -250,3 +255,4 @@ License Identifiers that are here available: http://spdx.org/licenses/
[LittleFileSystem]: https://os.mbed.com/docs/mbed-os/v5.12/apis/littlefilesystem.html
[SPIFFS]: https://github.com/pellepl/spiffs
[Dhara]: https://github.com/dlbeer/dhara
[littlefs-python]: https://pypi.org/project/littlefs-python/

View File

@@ -80,11 +80,6 @@ int lfs_filebd_read(const struct lfs_config *cfg, lfs_block_t block,
LFS_ASSERT(size % cfg->read_size == 0);
LFS_ASSERT(block < cfg->block_count);
// zero for reproducability (in case file is truncated)
if (bd->cfg->erase_value != -1) {
memset(buffer, bd->cfg->erase_value, size);
}
// read
off_t res1 = lseek(bd->fd,
(off_t)block*cfg->block_size + (off_t)off, SEEK_SET);
@@ -101,6 +96,11 @@ int lfs_filebd_read(const struct lfs_config *cfg, lfs_block_t block,
return err;
}
// file truncated? zero for reproducability
if ((lfs_size_t)res2 < size) {
memset((uint8_t*)buffer + res2, 0, size-res2);
}
LFS_FILEBD_TRACE("lfs_filebd_read -> %d", 0);
return 0;
}

View File

@@ -32,11 +32,8 @@ int lfs_rambd_createcfg(const struct lfs_config *cfg,
}
}
// zero for reproducability?
if (bd->cfg->erase_value != -1) {
memset(bd->buffer, bd->cfg->erase_value,
cfg->block_size * cfg->block_count);
}
// zero for reproducability (this matches filebd)
memset(bd->buffer, 0, cfg->block_size * cfg->block_count);
LFS_RAMBD_TRACE("lfs_rambd_createcfg -> %d", 0);
return 0;

View File

@@ -207,7 +207,7 @@ int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block,
bd->power_cycles -= 1;
if (bd->power_cycles == 0) {
// sync to make sure we persist the last changes
assert(lfs_testbd_rawsync(cfg) == 0);
LFS_ASSERT(lfs_testbd_rawsync(cfg) == 0);
// simulate power loss
exit(33);
}
@@ -254,7 +254,7 @@ int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) {
bd->power_cycles -= 1;
if (bd->power_cycles == 0) {
// sync to make sure we persist the last changes
assert(lfs_testbd_rawsync(cfg) == 0);
LFS_ASSERT(lfs_testbd_rawsync(cfg) == 0);
// simulate power loss
exit(33);
}

1502
lfs.c

File diff suppressed because it is too large Load Diff

42
lfs.h
View File

@@ -9,6 +9,7 @@
#include <stdint.h>
#include <stdbool.h>
#include "lfs_util.h"
#ifdef __cplusplus
extern "C"
@@ -21,7 +22,7 @@ extern "C"
// Software library version
// Major (top-nibble), incremented on backwards incompatible changes
// Minor (bottom-nibble), incremented on feature additions
#define LFS_VERSION 0x00020002
#define LFS_VERSION 0x00020003
#define LFS_VERSION_MAJOR (0xffff & (LFS_VERSION >> 16))
#define LFS_VERSION_MINOR (0xffff & (LFS_VERSION >> 0))
@@ -112,6 +113,8 @@ enum lfs_type {
LFS_TYPE_SOFTTAIL = 0x600,
LFS_TYPE_HARDTAIL = 0x601,
LFS_TYPE_MOVESTATE = 0x7ff,
LFS_TYPE_COMMITCRC = 0x502,
LFS_TYPE_NPROGCRC = 0x5ff,
// internal chip sources
LFS_FROM_NOOP = 0x000,
@@ -123,20 +126,25 @@ enum lfs_type {
enum lfs_open_flags {
// open flags
LFS_O_RDONLY = 1, // Open a file as read only
#ifndef LFS_READONLY
LFS_O_WRONLY = 2, // Open a file as write only
LFS_O_RDWR = 3, // Open a file as read and write
LFS_O_CREAT = 0x0100, // Create a file if it does not exist
LFS_O_EXCL = 0x0200, // Fail if a file already exists
LFS_O_TRUNC = 0x0400, // Truncate the existing file to zero size
LFS_O_APPEND = 0x0800, // Move to end of file on every write
#endif
// internally used flags
#ifndef LFS_READONLY
LFS_F_DIRTY = 0x010000, // File does not match storage
LFS_F_WRITING = 0x020000, // File has been written since last flush
#endif
LFS_F_READING = 0x040000, // File has been read since last flush
LFS_F_ERRED = 0x080000, // An error occured during write
#ifndef LFS_READONLY
LFS_F_ERRED = 0x080000, // An error occurred during write
#endif
LFS_F_INLINE = 0x100000, // Currently inlined in directory entry
LFS_F_OPENED = 0x200000, // File has been opened
};
// File seek flags
@@ -174,6 +182,16 @@ struct lfs_config {
// are propogated to the user.
int (*sync)(const struct lfs_config *c);
#ifdef LFS_THREADSAFE
// Lock the underlying block device. Negative error codes
// are propogated to the user.
int (*lock)(const struct lfs_config *c);
// Unlock the underlying block device. Negative error codes
// are propogated to the user.
int (*unlock)(const struct lfs_config *c);
#endif
// Minimum size of a block read. All read operations will be a
// multiple of this value.
lfs_size_t read_size;
@@ -399,6 +417,7 @@ typedef struct lfs {
/// Filesystem functions ///
#ifndef LFS_READONLY
// Format a block device with the littlefs
//
// Requires a littlefs object and config struct. This clobbers the littlefs
@@ -407,6 +426,7 @@ typedef struct lfs {
//
// Returns a negative error code on failure.
int lfs_format(lfs_t *lfs, const struct lfs_config *config);
#endif
// Mounts a littlefs
//
@@ -426,12 +446,15 @@ int lfs_unmount(lfs_t *lfs);
/// General operations ///
#ifndef LFS_READONLY
// Removes a file or directory
//
// If removing a directory, the directory must be empty.
// Returns a negative error code on failure.
int lfs_remove(lfs_t *lfs, const char *path);
#endif
#ifndef LFS_READONLY
// Rename or move a file or directory
//
// If the destination exists, it must match the source in type.
@@ -439,6 +462,7 @@ int lfs_remove(lfs_t *lfs, const char *path);
//
// Returns a negative error code on failure.
int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath);
#endif
// Find info about a file or directory
//
@@ -461,6 +485,7 @@ int lfs_stat(lfs_t *lfs, const char *path, struct lfs_info *info);
lfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path,
uint8_t type, void *buffer, lfs_size_t size);
#ifndef LFS_READONLY
// Set custom attributes
//
// Custom attributes are uniquely identified by an 8-bit type and limited
@@ -470,13 +495,16 @@ lfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path,
// Returns a negative error code on failure.
int lfs_setattr(lfs_t *lfs, const char *path,
uint8_t type, const void *buffer, lfs_size_t size);
#endif
#ifndef LFS_READONLY
// Removes a custom attribute
//
// If an attribute is not found, nothing happens.
//
// Returns a negative error code on failure.
int lfs_removeattr(lfs_t *lfs, const char *path, uint8_t type);
#endif
/// File operations ///
@@ -525,6 +553,7 @@ int lfs_file_sync(lfs_t *lfs, lfs_file_t *file);
lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
void *buffer, lfs_size_t size);
#ifndef LFS_READONLY
// Write data to file
//
// Takes a buffer and size indicating the data to write. The file will not
@@ -533,6 +562,7 @@ lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
// Returns the number of bytes written, or a negative error code on failure.
lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
const void *buffer, lfs_size_t size);
#endif
// Change the position of the file
//
@@ -541,10 +571,12 @@ lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t off, int whence);
#ifndef LFS_READONLY
// Truncates the size of the file to the specified size
//
// Returns a negative error code on failure.
int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size);
#endif
// Return the position of the file
//
@@ -567,10 +599,12 @@ lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file);
/// Directory operations ///
#ifndef LFS_READONLY
// Create a directory
//
// Returns a negative error code on failure.
int lfs_mkdir(lfs_t *lfs, const char *path);
#endif
// Open a directory
//
@@ -632,6 +666,7 @@ lfs_ssize_t lfs_fs_size(lfs_t *lfs);
// Returns a negative error code on failure.
int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data);
#ifndef LFS_READONLY
#ifdef LFS_MIGRATE
// Attempts to migrate a previous version of littlefs
//
@@ -646,6 +681,7 @@ int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data);
// Returns a negative error code on failure.
int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg);
#endif
#endif
#ifdef __cplusplus

View File

@@ -24,6 +24,7 @@ TAG_TYPES = {
'gstate': (0x700, 0x700),
'movestate': (0x7ff, 0x7ff),
'crc': (0x700, 0x500),
'nprogcrc': (0x7ff, 0x5ff),
}
class Tag:
@@ -99,7 +100,16 @@ class Tag:
return struct.unpack('b', struct.pack('B', self.chunk))[0]
def is_(self, type):
return (self.type & TAG_TYPES[type][0]) == TAG_TYPES[type][1]
try:
if ' ' in type:
type1, type3 = type.split()
return (self.is_(type1) and
(self.type & ~TAG_TYPES[type1][0]) == int(type3, 0))
return self.type == int(type, 0)
except (ValueError, KeyError):
return (self.type & TAG_TYPES[type][0]) == TAG_TYPES[type][1]
def mkmask(self):
return Tag(
@@ -109,14 +119,20 @@ class Tag:
def chid(self, nid):
ntag = Tag(self.type, nid, self.size)
if hasattr(self, 'off'): ntag.off = self.off
if hasattr(self, 'data'): ntag.data = self.data
if hasattr(self, 'crc'): ntag.crc = self.crc
if hasattr(self, 'off'): ntag.off = self.off
if hasattr(self, 'data'): ntag.data = self.data
if hasattr(self, 'crc'): ntag.crc = self.crc
if hasattr(self, 'erased'): ntag.erased = self.erased
return ntag
def typerepr(self):
if self.is_('crc') and getattr(self, 'crc', 0xffffffff) != 0xffffffff:
return 'crc (bad)'
if (self.is_('crc') and not self.is_('nprogcrc') and
getattr(self, 'crc', 0xffffffff) != 0xffffffff):
crc_status = ' (bad)'
elif self.is_('nprogcrc') and getattr(self, 'erased', False):
crc_status = ' (era)'
else:
crc_status = ''
reverse_types = {v: k for k, v in TAG_TYPES.items()}
for prefix in range(12):
@@ -124,12 +140,12 @@ class Tag:
if (mask, self.type & mask) in reverse_types:
type = reverse_types[mask, self.type & mask]
if prefix > 0:
return '%s %#0*x' % (
type, prefix//4, self.type & ((1 << prefix)-1))
return '%s %#x%s' % (
type, self.type & ((1 << prefix)-1), crc_status)
else:
return type
return '%s%s' % (type, crc_status)
else:
return '%02x' % self.type
return '%02x%s' % (self.type, crc_status)
def idrepr(self):
return repr(self.id) if self.id != 0x3ff else '.'
@@ -172,6 +188,8 @@ class MetadataPair:
self.rev, = struct.unpack('<I', block[0:4])
crc = binascii.crc32(block[0:4])
etag = None
estate = None
# parse tags
corrupt = False
@@ -182,11 +200,11 @@ class MetadataPair:
while len(block) - off >= 4:
ntag, = struct.unpack('>I', block[off:off+4])
tag = Tag(int(tag) ^ ntag)
tag = Tag((int(tag) ^ ntag) & 0x7fffffff)
tag.off = off + 4
tag.data = block[off+4:off+tag.dsize]
if tag.is_('crc'):
crc = binascii.crc32(block[off:off+4+4], crc)
if tag.is_('crc') and not tag.is_('nprogcrc'):
crc = binascii.crc32(block[off:off+2*4], crc)
else:
crc = binascii.crc32(block[off:off+tag.dsize], crc)
tag.crc = crc
@@ -194,16 +212,29 @@ class MetadataPair:
self.all_.append(tag)
if tag.is_('crc'):
if tag.is_('nprogcrc') and len(tag.data) == 8:
etag = tag
estate = struct.unpack('<II', tag.data)
elif tag.is_('crc'):
# is valid commit?
if crc != 0xffffffff:
corrupt = True
if not corrupt:
self.log = self.all_.copy()
# end of commit?
if estate:
esize, ecrc = estate
dcrc = 0xffffffff ^ binascii.crc32(block[off:off+esize])
if ecrc == dcrc:
etag.erased = True
corrupt = True
elif not (tag.is_('crc 0x0') or tag.is_('crc 0x1')):
corrupt = True
# reset tag parsing
crc = 0
tag = Tag(int(tag) ^ ((tag.type & 1) << 31))
etag = None
estate = None
# find active ids
self.ids = list(it.takewhile(
@@ -280,7 +311,7 @@ class MetadataPair:
f.write('\n')
for tag in tags:
f.write("%08x: %08x %-13s %4s %4s" % (
f.write("%08x: %08x %-14s %3s %4s" % (
tag.off, tag,
tag.typerepr(), tag.idrepr(), tag.sizerepr()))
if truncate:

View File

@@ -106,7 +106,7 @@ def main(args):
struct.unpack('<HH', superblock[1].data[0:4].ljust(4, b'\xff'))))
print("%-47s%s" % ("littlefs v%s.%s" % version,
"data (truncated, if it fits)"
if not any([args.no_truncate, args.tags, args.log, args.all]) else ""))
if not any([args.no_truncate, args.log, args.all]) else ""))
# print gstate
print("gstate 0x%s" % ''.join('%02x' % c for c in gstate))

172
tests/test_powerloss.toml Normal file
View File

@@ -0,0 +1,172 @@
# There are already a number of tests that test general operations under
# power-loss (see the reentrant attribute). These tests are for explicitly
# testing specific corner cases.
[[case]] # only a revision count
code = '''
lfs_format(&lfs, &cfg) => 0;
lfs_mount(&lfs, &cfg) => 0;
lfs_mkdir(&lfs, "notebook") => 0;
lfs_file_open(&lfs, &file, "notebook/paper",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0;
strcpy((char*)buffer, "hello");
size = strlen("hello");
for (int i = 0; i < 5; i++) {
lfs_file_write(&lfs, &file, buffer, size) => size;
lfs_file_sync(&lfs, &file) => 0;
}
lfs_file_close(&lfs, &file) => 0;
char rbuffer[256];
lfs_file_open(&lfs, &file, "notebook/paper", LFS_O_RDONLY) => 0;
for (int i = 0; i < 5; i++) {
lfs_file_read(&lfs, &file, rbuffer, size) => size;
assert(memcmp(rbuffer, buffer, size) == 0);
}
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
// get pair/rev count
lfs_mount(&lfs, &cfg) => 0;
lfs_dir_open(&lfs, &dir, "notebook") => 0;
lfs_block_t pair[2] = {dir.m.pair[0], dir.m.pair[1]};
uint32_t rev = dir.m.rev;
lfs_dir_close(&lfs, &dir) => 0;
lfs_unmount(&lfs) => 0;
// write just the revision count
uint8_t bbuffer[LFS_BLOCK_SIZE];
cfg.read(&cfg, pair[1], 0, bbuffer, LFS_BLOCK_SIZE) => 0;
memcpy(bbuffer, &(uint32_t){lfs_tole32(rev+1)}, sizeof(uint32_t));
cfg.erase(&cfg, pair[1]) => 0;
cfg.prog(&cfg, pair[1], 0, bbuffer, LFS_BLOCK_SIZE) => 0;
lfs_mount(&lfs, &cfg) => 0;
// can read?
lfs_file_open(&lfs, &file, "notebook/paper", LFS_O_RDONLY) => 0;
for (int i = 0; i < 5; i++) {
lfs_file_read(&lfs, &file, rbuffer, size) => size;
assert(memcmp(rbuffer, buffer, size) == 0);
}
lfs_file_close(&lfs, &file) => 0;
// can write?
lfs_file_open(&lfs, &file, "notebook/paper",
LFS_O_WRONLY | LFS_O_APPEND) => 0;
strcpy((char*)buffer, "goodbye");
size = strlen("goodbye");
for (int i = 0; i < 5; i++) {
lfs_file_write(&lfs, &file, buffer, size) => size;
lfs_file_sync(&lfs, &file) => 0;
}
lfs_file_close(&lfs, &file) => 0;
lfs_file_open(&lfs, &file, "notebook/paper", LFS_O_RDONLY) => 0;
strcpy((char*)buffer, "hello");
size = strlen("hello");
for (int i = 0; i < 5; i++) {
lfs_file_read(&lfs, &file, rbuffer, size) => size;
assert(memcmp(rbuffer, buffer, size) == 0);
}
strcpy((char*)buffer, "goodbye");
size = strlen("goodbye");
for (int i = 0; i < 5; i++) {
lfs_file_read(&lfs, &file, rbuffer, size) => size;
assert(memcmp(rbuffer, buffer, size) == 0);
}
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
'''
[[case]] # partial prog, may not be byte in order!
if = "LFS_PROG_SIZE < LFS_BLOCK_SIZE"
define.BYTE_OFF = ["0", "LFS_PROG_SIZE-1", "LFS_PROG_SIZE/2"]
define.BYTE_VALUE = [0x33, 0xcc]
in = "lfs.c"
code = '''
lfs_format(&lfs, &cfg) => 0;
lfs_mount(&lfs, &cfg) => 0;
lfs_mkdir(&lfs, "notebook") => 0;
lfs_file_open(&lfs, &file, "notebook/paper",
LFS_O_WRONLY | LFS_O_CREAT | LFS_O_APPEND) => 0;
strcpy((char*)buffer, "hello");
size = strlen("hello");
for (int i = 0; i < 5; i++) {
lfs_file_write(&lfs, &file, buffer, size) => size;
lfs_file_sync(&lfs, &file) => 0;
}
lfs_file_close(&lfs, &file) => 0;
char rbuffer[256];
lfs_file_open(&lfs, &file, "notebook/paper", LFS_O_RDONLY) => 0;
for (int i = 0; i < 5; i++) {
lfs_file_read(&lfs, &file, rbuffer, size) => size;
assert(memcmp(rbuffer, buffer, size) == 0);
}
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
// imitate a partial prog, value should not matter, if littlefs
// doesn't notice the partial prog testbd will assert
// get offset to next prog
lfs_mount(&lfs, &cfg) => 0;
lfs_dir_open(&lfs, &dir, "notebook") => 0;
lfs_block_t block = dir.m.pair[0];
lfs_off_t off = dir.m.off;
lfs_dir_close(&lfs, &dir) => 0;
lfs_unmount(&lfs) => 0;
// tweak byte
uint8_t bbuffer[LFS_BLOCK_SIZE];
cfg.read(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0;
bbuffer[off + BYTE_OFF] = BYTE_VALUE;
cfg.erase(&cfg, block) => 0;
cfg.prog(&cfg, block, 0, bbuffer, LFS_BLOCK_SIZE) => 0;
lfs_mount(&lfs, &cfg) => 0;
// can read?
lfs_file_open(&lfs, &file, "notebook/paper", LFS_O_RDONLY) => 0;
for (int i = 0; i < 5; i++) {
lfs_file_read(&lfs, &file, rbuffer, size) => size;
assert(memcmp(rbuffer, buffer, size) == 0);
}
lfs_file_close(&lfs, &file) => 0;
// can write?
lfs_file_open(&lfs, &file, "notebook/paper",
LFS_O_WRONLY | LFS_O_APPEND) => 0;
strcpy((char*)buffer, "goodbye");
size = strlen("goodbye");
for (int i = 0; i < 5; i++) {
lfs_file_write(&lfs, &file, buffer, size) => size;
lfs_file_sync(&lfs, &file) => 0;
}
lfs_file_close(&lfs, &file) => 0;
lfs_file_open(&lfs, &file, "notebook/paper", LFS_O_RDONLY) => 0;
strcpy((char*)buffer, "hello");
size = strlen("hello");
for (int i = 0; i < 5; i++) {
lfs_file_read(&lfs, &file, rbuffer, size) => size;
assert(memcmp(rbuffer, buffer, size) == 0);
}
strcpy((char*)buffer, "goodbye");
size = strlen("goodbye");
for (int i = 0; i < 5; i++) {
lfs_file_read(&lfs, &file, rbuffer, size) => size;
assert(memcmp(rbuffer, buffer, size) == 0);
}
lfs_file_close(&lfs, &file) => 0;
lfs_unmount(&lfs) => 0;
'''