Compare commits

...

31 Commits

Author SHA1 Message Date
Christopher Haster
640f7efae7 Moved lfs_mdir_isopen behind LFS_NO_ASSERT
lfs_mdir_isopen goes unused if asserts are disabled, and this caused an
"unused function" warning on Clang (curiously not on GCC since the
function was static inline, commonly used for header-only functions).

Also removed "inline" from the lfs_mdir_* functions as these involve
linked-list operations and really shouldn't be inlined. And since they
are static, inlining should occur automatically if there is a benefit.

Found by dpgeorge
2020-12-08 20:24:58 -06:00
Christopher Haster
1a59954ec6 Merge pull request #495 from littlefs-project/devel
Minor release: v2.3
2020-12-07 20:50:31 -06:00
Christopher Haster
6a7012774d Renamed internal lfs_*raw -> lfs_raw* functions
- Prefixing with raw is slightly more readable, follows
  common-prefix rule
- Matches existing raw prefixes in testbd
2020-12-06 00:26:24 -06:00
Christopher Haster
288a5cbc8d Bumped minor version to v2.3 2020-12-04 01:31:27 -06:00
Christopher Haster
5783eea0de Merge pull request #490 from littlefs-project/fix-alloc-eviction
Fix allocation-eviction issue when erase state is multiple of block_cycles+1
2020-12-04 00:49:09 -06:00
Christopher Haster
2bb523421e Moved lfs_mlist_isopen checks into the API wrappers
This indirectly solves an issue with lfs_file_rawclose asserting
when lfs_file_opencfg errors since appending to the mlist occurs
after open. It also may speed up some of the internal operations such as
the lfs_file_write used to resolve unflushed data.

The idea behind adopting mlist over flags is that realistically it's
unlikely for the user to open a significant number of files (enough for
big O to kick in). That being said, moving the mlist asserts into the
API wrappers does protect some of the internal operations from scaling
based on the number of open files.
2020-12-04 00:42:32 -06:00
Noah Gorny
7388b2938a Deprecate LFS_F_OPENED and use lfs_mlist_isused instead
Instead of additional flag, we can just go through the mlist.
2020-12-04 00:26:19 -06:00
Christopher Haster
ce425a56c3 Merge pull request #470 from renesas/SWFLEX-1517-littlefs-thread-safe-option
Add thread safe wrappers
2020-12-03 23:47:32 -06:00
Christopher Haster
a99a93fb27 Added thread-safe build+size reporting to CI 2020-12-03 23:46:59 -06:00
Christopher Haster
45afded784 Moved LFS_TRACE calls to API wrapper functions
This removes quite a bit of extra code needed to entertwine the
LFS_TRACE calls into the original funcions.

Also changed temporary return type to match API declaration where
necessary.
2020-12-03 23:46:59 -06:00
Christopher Haster
00a9ba7826 Tweaked thread-safe implementation
- Stayed on non-system include for lfs_util.h for now
- Named internal functions "lfs_functionraw"
- Merged lfs_fs_traverseraw
- Added LFS_LOCK/UNLOCK macros
- Changed LFS_THREADSAFE from 1/0 to defined/undefined to
  match LFS_READONLY
2020-12-03 23:46:59 -06:00
Bill Gesner
fc6988c7c3 make raw functions static. formatting tweaks 2020-12-03 23:46:54 -06:00
Bill Gesner
d0f055d321 Squash of thread-safe PR cleanup
- expand functions
- add comment
- rename functions
- fix locking issue in format and mount
- use global include
- fix ac6 linker issue
- use the global config file
- address review comments
- minor cleanup
- minor cleanup
- review comments
2020-12-03 23:41:01 -06:00
Christopher Haster
b9fa33f9bc Merge pull request #480 from maximevince/master
Add LFS_READONLY define, to allow smaller builds providing read-only mode
2020-12-03 23:06:00 -06:00
Christopher Haster
2efebf8e9b Added read-only build+size reporting to CI 2020-12-03 23:04:48 -06:00
Maxime Vincent
754b4c3cda Squash of LFS_READONLY cleanup
- undef unavailable function declarations altogether
- even less code, assert on write attempts
- remove LFS_O_WRONLY and other flags when compiling with LFS_READONLY
- do not annotate #endif, as requested
- move ifdef before comments blocks, rework dangling opening bracket
- ifdef file flags that are not needed in read-only mode
- slight refactor
- ifdef LFS_F_ERRED out as well
2020-12-03 23:03:29 -06:00
Christopher Haster
584eb26efc Merge pull request #443 from NoahGorny/add-already-opened-assert
Assert that the file isnt open in lfs_file_opencfg
2020-12-03 22:43:10 -06:00
Noah Gorny
008ebc37df Add lfs_mlist_append/remove helper 2020-12-03 22:42:39 -06:00
Christopher Haster
66272067ab Merge pull request #395 from gmpy/improve-write-performance
lfs_bd_cmp() compares more bytes at one time
2020-12-03 22:34:47 -06:00
Christopher Haster
e273a82679 Merge pull request #487 from littlefs-project/fix-alloc-reset-modulus
Fix several wear-leveling issues found in lfs_alloc_reset
2020-12-03 22:33:47 -06:00
Christopher Haster
1dc6ae94b9 Merge pull request #486 from littlefs-project/fix-assert
Fix assert
2020-12-03 22:32:56 -06:00
Christopher Haster
817ef02d24 Merge pull request #412 from jrast/patch-3
Added littlefs-python to the related projects section
2020-12-03 22:32:04 -06:00
Christopher Haster
b8dcf10974 Changed lfs_dir_alloc to maximize block cycles for new metadata pairs
Previously we only bumped the revision count if an eviction would occur
immediately (and possibly corrupt littlefs). This works, but does risk
an unoptimal superblock size if an almost-exhausted superblock was
allocated during lfs_format.

As pointed out by tim-nordell-nimbelink, we can align the revision count
to maximize the number of block cycles without breaking the existing
requirements of increasing revision counts.

As an added benefit, littlefs's wear-leveling should behave more
consistently after this change.
2020-11-28 22:46:11 -06:00
Christopher Haster
d04c1392c0 Fixed allocation-eviction issue when erase state is multiple of block_cycles+1
This rather interesting corner-case arises in lfs_dir_alloc anytime the
uninitialized revision count happens to be a multiple of block_cycles+1.

For example, the source of the bug found by tim-nordell-nimbelink:

rev = 2742492087
block_cycles = 100

2742492087 % (100+1) = 0

The reason for this weird block_cycles+1 case is due to a fix for a
previous bug in fe957de. To avoid aliasing, which would cause metadata
pairs to wear unevenly, block_cycles incremented to the next odd number.

Normally, littlefs tweaks the revision count of blocks during
lfs_dir_alloc in order to make sure evictions can't happen on the first
compact. Otherwise, higher-level logic such as lfs_format would break.

However, this wasn't updated with the aliasing fix in fe957de, so
lfs_dir_alloc was only rounding the revision count to the nearest even
number.

The current fix is to change the logic in lfs_dir_alloc to explicitly
check for the eviction condition and increment if eviction would occur.

Found by tim-nordell-nimbelink
2020-11-22 00:40:58 -06:00
Noah Gorny
6303558aee Use LFS_O_RDWR instead of magic number in lfs_file_* asserts 2020-11-19 01:51:39 +02:00
Noah Gorny
4bd653dd00 Assert that file/dir struct is not reused in lfs_file_opencfg/lfs_dir_open 2020-11-19 01:51:39 +02:00
Maxime Vincent
8e6826c4e2 Add LFS_READYONLY define, to allow smaller builds providing read-only mode 2020-10-28 16:09:13 +01:00
Bill Gesner
10ac6b9cf0 add thread safe wrappers 2020-09-17 23:41:20 +00:00
Shiven Gupta
87a2cb0e41 Fix assert 2020-08-18 17:36:14 -04:00
Jürg Rast
6d0ec5e851 Added littlefs-python to the related projects section
As introduced in #297, I created a python wrapper for littlefs. The wrapper supports two API's: A C-like API which is the same as in C and a more pythonic API which is easier to use if you are more the python guy. The wrapper is built with littlefs 2.2.1 at the moment.
2020-04-13 21:33:30 +02:00
WeiXiong Liao
64f70f51b0 lfs_bd_cmp() compares more bytes at one time
It's very slowly to compare one byte at one time. Here are the
performance I get from 128M spinand with NFTL by sequential writing.

| file size | buffer size  | write speed  |
| 10 MB     | 0   B        | 3206.01 KB/s |
| 10 MB     | 1   B        | 2434.04 KB/s |
| 10 MB     | 2   B        | 2685.78 KB/s |
| 10 MB     | 4   B        | 2857.94 KB/s |
| 10 MB     | 8   B        | 3060.68 KB/s |
| 10 MB     | 16  B        | 3155.30 KB/s |
| 10 MB     | 64  B        | 3193.68 KB/s |
| 10 MB     | 128 B        | 3230.62 KB/s |
| 10 MB     | 256 B        | 3153.03 KB/s |

| 70 MB     | 0   B        | 2258.87 KB/s |
| 70 MB     | 1   B        | 1827.83 KB/s |
| 70 MB     | 2   B        | 1962.29 KB/s |
| 70 MB     | 4   B        | 2074.01 KB/s |
| 70 MB     | 8   B        | 2147.03 KB/s |
| 70 MB     | 64  B        | 2179.92 KB/s |
| 70 MB     | 256 B        | 2179.96 KB/s |

The 0 Byte size means no validation and the 1 Byte size is how
littlefs do before. Based on the above table and to save memory,
comparing 8 bytes at one time is more wonderful.

Signed-off-by: WeiXiong Liao <liaoweixiong@allwinnertech.com>
2020-03-13 15:23:20 +08:00
5 changed files with 866 additions and 328 deletions

View File

@@ -208,6 +208,38 @@ jobs:
script: script:
- make test TFLAGS+="-k --valgrind" - make test TFLAGS+="-k --valgrind"
# test compilation in read-only mode
- stage: test
env:
- NAME=littlefs-readonly
- CC="arm-linux-gnueabi-gcc --static -mthumb"
- CFLAGS="-Werror -DLFS_READONLY"
if: branch !~ -prefix$
install:
- *install-common
- sudo apt-get install
gcc-arm-linux-gnueabi
libc6-dev-armel-cross
- arm-linux-gnueabi-gcc --version
# report-size will compile littlefs and report the size
script: [*report-size]
# test compilation in thread-safe mode
- stage: test
env:
- NAME=littlefs-threadsafe
- CC="arm-linux-gnueabi-gcc --static -mthumb"
- CFLAGS="-Werror -DLFS_THREADSAFE"
if: branch !~ -prefix$
install:
- *install-common
- sudo apt-get install
gcc-arm-linux-gnueabi
libc6-dev-armel-cross
- arm-linux-gnueabi-gcc --version
# report-size will compile littlefs and report the size
script: [*report-size]
# self-host with littlefs-fuse for fuzz test # self-host with littlefs-fuse for fuzz test
- stage: test - stage: test
env: env:

View File

@@ -222,6 +222,11 @@ License Identifiers that are here available: http://spdx.org/licenses/
want this, but it is handy for demos. You can see it in action want this, but it is handy for demos. You can see it in action
[here][littlefs-js-demo]. [here][littlefs-js-demo].
- [littlefs-python] - A Python wrapper for littlefs. The project allows you
to create images of the filesystem on your PC. Check if littlefs will fit
your needs, create images for a later download to the target memory or
inspect the content of a binary image of the target memory.
- [mklfs] - A command line tool built by the [Lua RTOS] guys for making - [mklfs] - A command line tool built by the [Lua RTOS] guys for making
littlefs images from a host PC. Supports Windows, Mac OS, and Linux. littlefs images from a host PC. Supports Windows, Mac OS, and Linux.
@@ -250,3 +255,4 @@ License Identifiers that are here available: http://spdx.org/licenses/
[LittleFileSystem]: https://os.mbed.com/docs/mbed-os/v5.12/apis/littlefilesystem.html [LittleFileSystem]: https://os.mbed.com/docs/mbed-os/v5.12/apis/littlefilesystem.html
[SPIFFS]: https://github.com/pellepl/spiffs [SPIFFS]: https://github.com/pellepl/spiffs
[Dhara]: https://github.com/dlbeer/dhara [Dhara]: https://github.com/dlbeer/dhara
[littlefs-python]: https://pypi.org/project/littlefs-python/

View File

@@ -207,7 +207,7 @@ int lfs_testbd_prog(const struct lfs_config *cfg, lfs_block_t block,
bd->power_cycles -= 1; bd->power_cycles -= 1;
if (bd->power_cycles == 0) { if (bd->power_cycles == 0) {
// sync to make sure we persist the last changes // sync to make sure we persist the last changes
assert(lfs_testbd_rawsync(cfg) == 0); LFS_ASSERT(lfs_testbd_rawsync(cfg) == 0);
// simulate power loss // simulate power loss
exit(33); exit(33);
} }
@@ -254,7 +254,7 @@ int lfs_testbd_erase(const struct lfs_config *cfg, lfs_block_t block) {
bd->power_cycles -= 1; bd->power_cycles -= 1;
if (bd->power_cycles == 0) { if (bd->power_cycles == 0) {
// sync to make sure we persist the last changes // sync to make sure we persist the last changes
assert(lfs_testbd_rawsync(cfg) == 0); LFS_ASSERT(lfs_testbd_rawsync(cfg) == 0);
// simulate power loss // simulate power loss
exit(33); exit(33);
} }

1112
lfs.c

File diff suppressed because it is too large Load Diff

40
lfs.h
View File

@@ -9,6 +9,7 @@
#include <stdint.h> #include <stdint.h>
#include <stdbool.h> #include <stdbool.h>
#include "lfs_util.h"
#ifdef __cplusplus #ifdef __cplusplus
extern "C" extern "C"
@@ -21,7 +22,7 @@ extern "C"
// Software library version // Software library version
// Major (top-nibble), incremented on backwards incompatible changes // Major (top-nibble), incremented on backwards incompatible changes
// Minor (bottom-nibble), incremented on feature additions // Minor (bottom-nibble), incremented on feature additions
#define LFS_VERSION 0x00020002 #define LFS_VERSION 0x00020003
#define LFS_VERSION_MAJOR (0xffff & (LFS_VERSION >> 16)) #define LFS_VERSION_MAJOR (0xffff & (LFS_VERSION >> 16))
#define LFS_VERSION_MINOR (0xffff & (LFS_VERSION >> 0)) #define LFS_VERSION_MINOR (0xffff & (LFS_VERSION >> 0))
@@ -123,20 +124,25 @@ enum lfs_type {
enum lfs_open_flags { enum lfs_open_flags {
// open flags // open flags
LFS_O_RDONLY = 1, // Open a file as read only LFS_O_RDONLY = 1, // Open a file as read only
#ifndef LFS_READONLY
LFS_O_WRONLY = 2, // Open a file as write only LFS_O_WRONLY = 2, // Open a file as write only
LFS_O_RDWR = 3, // Open a file as read and write LFS_O_RDWR = 3, // Open a file as read and write
LFS_O_CREAT = 0x0100, // Create a file if it does not exist LFS_O_CREAT = 0x0100, // Create a file if it does not exist
LFS_O_EXCL = 0x0200, // Fail if a file already exists LFS_O_EXCL = 0x0200, // Fail if a file already exists
LFS_O_TRUNC = 0x0400, // Truncate the existing file to zero size LFS_O_TRUNC = 0x0400, // Truncate the existing file to zero size
LFS_O_APPEND = 0x0800, // Move to end of file on every write LFS_O_APPEND = 0x0800, // Move to end of file on every write
#endif
// internally used flags // internally used flags
#ifndef LFS_READONLY
LFS_F_DIRTY = 0x010000, // File does not match storage LFS_F_DIRTY = 0x010000, // File does not match storage
LFS_F_WRITING = 0x020000, // File has been written since last flush LFS_F_WRITING = 0x020000, // File has been written since last flush
#endif
LFS_F_READING = 0x040000, // File has been read since last flush LFS_F_READING = 0x040000, // File has been read since last flush
LFS_F_ERRED = 0x080000, // An error occured during write #ifndef LFS_READONLY
LFS_F_ERRED = 0x080000, // An error occurred during write
#endif
LFS_F_INLINE = 0x100000, // Currently inlined in directory entry LFS_F_INLINE = 0x100000, // Currently inlined in directory entry
LFS_F_OPENED = 0x200000, // File has been opened
}; };
// File seek flags // File seek flags
@@ -174,6 +180,16 @@ struct lfs_config {
// are propogated to the user. // are propogated to the user.
int (*sync)(const struct lfs_config *c); int (*sync)(const struct lfs_config *c);
#ifdef LFS_THREADSAFE
// Lock the underlying block device. Negative error codes
// are propogated to the user.
int (*lock)(const struct lfs_config *c);
// Unlock the underlying block device. Negative error codes
// are propogated to the user.
int (*unlock)(const struct lfs_config *c);
#endif
// Minimum size of a block read. All read operations will be a // Minimum size of a block read. All read operations will be a
// multiple of this value. // multiple of this value.
lfs_size_t read_size; lfs_size_t read_size;
@@ -399,6 +415,7 @@ typedef struct lfs {
/// Filesystem functions /// /// Filesystem functions ///
#ifndef LFS_READONLY
// Format a block device with the littlefs // Format a block device with the littlefs
// //
// Requires a littlefs object and config struct. This clobbers the littlefs // Requires a littlefs object and config struct. This clobbers the littlefs
@@ -407,6 +424,7 @@ typedef struct lfs {
// //
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_format(lfs_t *lfs, const struct lfs_config *config); int lfs_format(lfs_t *lfs, const struct lfs_config *config);
#endif
// Mounts a littlefs // Mounts a littlefs
// //
@@ -426,12 +444,15 @@ int lfs_unmount(lfs_t *lfs);
/// General operations /// /// General operations ///
#ifndef LFS_READONLY
// Removes a file or directory // Removes a file or directory
// //
// If removing a directory, the directory must be empty. // If removing a directory, the directory must be empty.
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_remove(lfs_t *lfs, const char *path); int lfs_remove(lfs_t *lfs, const char *path);
#endif
#ifndef LFS_READONLY
// Rename or move a file or directory // Rename or move a file or directory
// //
// If the destination exists, it must match the source in type. // If the destination exists, it must match the source in type.
@@ -439,6 +460,7 @@ int lfs_remove(lfs_t *lfs, const char *path);
// //
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath); int lfs_rename(lfs_t *lfs, const char *oldpath, const char *newpath);
#endif
// Find info about a file or directory // Find info about a file or directory
// //
@@ -461,6 +483,7 @@ int lfs_stat(lfs_t *lfs, const char *path, struct lfs_info *info);
lfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path, lfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path,
uint8_t type, void *buffer, lfs_size_t size); uint8_t type, void *buffer, lfs_size_t size);
#ifndef LFS_READONLY
// Set custom attributes // Set custom attributes
// //
// Custom attributes are uniquely identified by an 8-bit type and limited // Custom attributes are uniquely identified by an 8-bit type and limited
@@ -470,13 +493,16 @@ lfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path,
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_setattr(lfs_t *lfs, const char *path, int lfs_setattr(lfs_t *lfs, const char *path,
uint8_t type, const void *buffer, lfs_size_t size); uint8_t type, const void *buffer, lfs_size_t size);
#endif
#ifndef LFS_READONLY
// Removes a custom attribute // Removes a custom attribute
// //
// If an attribute is not found, nothing happens. // If an attribute is not found, nothing happens.
// //
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_removeattr(lfs_t *lfs, const char *path, uint8_t type); int lfs_removeattr(lfs_t *lfs, const char *path, uint8_t type);
#endif
/// File operations /// /// File operations ///
@@ -525,6 +551,7 @@ int lfs_file_sync(lfs_t *lfs, lfs_file_t *file);
lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file, lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
void *buffer, lfs_size_t size); void *buffer, lfs_size_t size);
#ifndef LFS_READONLY
// Write data to file // Write data to file
// //
// Takes a buffer and size indicating the data to write. The file will not // Takes a buffer and size indicating the data to write. The file will not
@@ -533,6 +560,7 @@ lfs_ssize_t lfs_file_read(lfs_t *lfs, lfs_file_t *file,
// Returns the number of bytes written, or a negative error code on failure. // Returns the number of bytes written, or a negative error code on failure.
lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file, lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
const void *buffer, lfs_size_t size); const void *buffer, lfs_size_t size);
#endif
// Change the position of the file // Change the position of the file
// //
@@ -541,10 +569,12 @@ lfs_ssize_t lfs_file_write(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file, lfs_soff_t lfs_file_seek(lfs_t *lfs, lfs_file_t *file,
lfs_soff_t off, int whence); lfs_soff_t off, int whence);
#ifndef LFS_READONLY
// Truncates the size of the file to the specified size // Truncates the size of the file to the specified size
// //
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size); int lfs_file_truncate(lfs_t *lfs, lfs_file_t *file, lfs_off_t size);
#endif
// Return the position of the file // Return the position of the file
// //
@@ -567,10 +597,12 @@ lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file);
/// Directory operations /// /// Directory operations ///
#ifndef LFS_READONLY
// Create a directory // Create a directory
// //
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_mkdir(lfs_t *lfs, const char *path); int lfs_mkdir(lfs_t *lfs, const char *path);
#endif
// Open a directory // Open a directory
// //
@@ -632,6 +664,7 @@ lfs_ssize_t lfs_fs_size(lfs_t *lfs);
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data); int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data);
#ifndef LFS_READONLY
#ifdef LFS_MIGRATE #ifdef LFS_MIGRATE
// Attempts to migrate a previous version of littlefs // Attempts to migrate a previous version of littlefs
// //
@@ -646,6 +679,7 @@ int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data);
// Returns a negative error code on failure. // Returns a negative error code on failure.
int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg); int lfs_migrate(lfs_t *lfs, const struct lfs_config *cfg);
#endif #endif
#endif
#ifdef __cplusplus #ifdef __cplusplus