diff --git a/docs/design-documents/features/filesystem/README.md b/docs/design-documents/features/filesystem/README.md deleted file mode 100644 index 98a692c..0000000 --- a/docs/design-documents/features/filesystem/README.md +++ /dev/null @@ -1 +0,0 @@ -# Design Documents - Filesystem diff --git a/docs/design-documents/features/security/README.md b/docs/design-documents/features/security/README.md deleted file mode 100644 index f79eb32..0000000 --- a/docs/design-documents/features/security/README.md +++ /dev/null @@ -1 +0,0 @@ -# Design Documents - Security diff --git a/docs/design-documents/features/storage/BlockDevices/get_type_method.md b/docs/design-documents/features/storage/BlockDevices/get_type_method.md deleted file mode 100644 index add8238..0000000 --- a/docs/design-documents/features/storage/BlockDevices/get_type_method.md +++ /dev/null @@ -1,78 +0,0 @@ -# Get type method addon to block devices class - -### Revision history -| Revision | Date | Authors | Mbed OS version | Comments | -|---------- |-----------------|-------------------------------------------------------------|------------------------|------------------| -| 1.0 | 04/12/2018 | Yossi Levy ([@yossi2le](https://github.com/yossi2le/)) | 5.11+ | Initial revision | - -## Introduction - -Most storage solutions use block devices, whether it is Filesystems, KVStore or any other solution -most of them will have some BlockDevice class underneath. However, sometimes a storage type or an application needs to know the physical -BlockDevice type in order to work smoothly or the BlockDevice type in order to decide what is -the best storage configuration to use. -To address this an add-on method of getting type is proposed for BlockDevice interface class. - -## The Motivation - -Below there is a list of some examples to explain the motivation and the need for the adding of get_type to BlockDevice interface. - -examples: -- TDBStore needs to know if there are flash characteristics for the block device and if there aren�t it should use - FlashSimBlockDevice to simulate a flash BlockDevice. -- When creating a file system you would prefer working with FAT on top of SD while LITTLEFS on top of any flash block device. - Those preference in favor of better performance. - -To summarize the above, it may be very useful when using block device to know the type of the instance and especially, but not only, -when using get_default_instace. Sometimes applications and tests would like to behave differently depending on the instance that has been created -or provided to them. - -In fact it might be worth to consider adding the get_type to any interface with get_default_instance at mbed-os. - -## Dive into details -we should add the following method to BlockDevice interface class. - -```virtual const char * get_type() const = 0;``` - -then every physical BlockDevice class which implements the interface should also implement this method and return a string -representing its type. Furthermore, a nonphysical BlockDevice like SlicingBlockDevice should return the underlying physical -BlockDevice type. - -### Physical BlockDevice: -``` -const char * HeapBlockDevice::get_type() const -{ - return "HEAP"; -} -``` - -### Logical BlockDevice: -``` -const char * SlicingBlockDevice::get_type() const -{ - if (_bd != NULL) { - return _bd->get_type(); - } - - return NULL; -} -``` - -### Open issue -The ChainingBlockDevice which chains different type of physical block devices into one block device is unable -to return the underneath physical as it contains two or more types. Therefore it will return CHAINING as its -identity and its left for the user to decide how the application will treat this information. - - -The below table describes physical BlockDevice and its tyep names - - -| BlockDevice class | String description | -|-----------------------------|--------------------| -| HeapBlockDevice | "HEAP" | -| SPIFBlockDevice | "SPIF" | -| QSPIFBlockDevice | "QSPIF" | -| SDBlockDevice | "SD" | -| FlashIAPBlockDevice | "FLASHIAP" | -| DataFlashBlockDevice | "DATAFLASH" | -| ChainingBlockDevice | "CHAINING" | diff --git a/docs/design-documents/features/storage/Configuration/CONFIGURATION.md b/docs/design-documents/features/storage/Configuration/CONFIGURATION.md deleted file mode 100644 index 8dcafa7..0000000 --- a/docs/design-documents/features/storage/Configuration/CONFIGURATION.md +++ /dev/null @@ -1,407 +0,0 @@ -# Storage configuration - -This document describes the configurations of the mbed OS storage based on kv_store interface. - -## New storage design - -![New storage design](./NewStorageDesign.jpg) - -This document describes the configuration for the KVStore part (left side of the diagram above) and its underlying components, such as file systems and block devices. - -The configuration concept is based on a number of predefined topologies (configurations) allowing customers to select one of them or to define a topology. - -The last section of this document explains how to override the configuration option in order to enable storage configuration that none of the configurations above support. - -To use the default configurations, set the `storage_type` parameter to one of the configurations options available. - -The implementation of the configuration is composed of a set of `.json` files and a set of functions instantiating and initializating the required components. - -## Configuration structure - -``` -kvstore -│ -└───conf - │ mbed_lib.json - │ - ├───tdb_external - │ mbed_lib.json - │ - ├───tdb_external_no_rbp - │ mbed_lib.json - │ - ├───filesystem - │ mbed_lib.json - │ - ├───filesystem_no_rbp - │ mbed_lib.json - │ - └───tdb_internal - mbed_lib.json -``` - -The KVStore configuration file structure includes six configuration files. The topmost configuration file is used to set up the full configuration of the storage by defining a single parameter (`storage_type`) to one of the predefined configurations. The configuration files in the subfolders are used to implement the above top level configurations. - -You can find the configuration files `conf/`: - -* `conf/tdb_internal` - storage type `TDB_INTERNAL` configuration is intended to be used when all data will be stored in internal memory only. No need for additional security features. A single TDBStore object will be allocated in internal flash. -* `conf/tdb_external` - storage type `TDB_EXTERNAL` configuration is providing full security and intended to be used when data is stored in external flash. It allocates: SecureStore, TDBStore in external flash and TDBStore in internal flash (for rollback protection - RBP). -* `conf/tdb_external_no_rbp` - storage type `TDB_EXTERNAL_NO_RBP` configuration allows security but without rollback protection. Similar to `tdb_external` but without the TDBStore in internal memory. -* `conf/filesystem` - This configuration will allocate: SecureStore, FileSystemStore, filesystem, TDBStore in internal memory and the required block devices. The allocated file system will be selected according to the COMPONENT set in `targets.json`, (FATFS for SD card and LittleFS for SPIF); however, you can set this differently by overriding the respective parameter. Use this configuration if you need the file system with a POSIX API in addition to the set/get API. -* `conf/filesystem_no_rbp` - storage type `FILESYSTEM_NO_RBP` configuration allows security like FILESYSTEM configuration but without rollback protection. - -A standalone block device is allocated for each component in internal and external memory and SD cards as required for the configurations. The full size of the memory allocated for each block device is used by the respective component. - -## Configuration parameters - -The following is a list of all storage parameters available and their description. - -* `storage_type` - Used to select one of the predefined configurations. - * `TDB_INTERNAL`. - * `TDB_EXTERNAL`. - * `TDB_EXTERNAL_NO_RBP`. - * `FILESYSTEM`. - * `FILESYSTEM_NO_RBP`. - * `default` - If the `default` configuration is set, the system will choose the type of storage TDB_INTERNAL. -* `default_kv` - This is a string representing the path for the default KVStore instantiation. Applications can pass an empty path (only the key name) or pass the generated name for this parameter (`MBED_CONF_STORAGE_DEFAULT_KV`) as the path to use this configuration. -* `internal_size` - The size in bytes for the internal FlashIAP block device. This, together with the `internal_base_address`, adjusts exactly the size and location where the block device resides on memory. If not defined, the block device will try to get the maximum size available. -* `internal_base_address` - The address where the internal FlashIAP blockDevice starts. This helps to prevent collisions with other needs, such as firmware updates. If not defined, the start address will be set to the first sector after the application code ends in `TDB_internal`. In any external configurations with rollback protection support, it will be set to end of flash - `rbp_internal_size`. -* `rbp_number_of_entries` - Sets the number of entries allowed for rollback protection. The default is set to 64. This parameter controls the maxmium number of different keys that can be created with rollback protection flag. -* `rbp_internal_size` - Sets the size for the rollback protection TDBStore in the internal memory. The base address is calculated as flash ends address - size. -* `filesystem` - Options are FAT, LITTLE or default. If not set or set to default, the file system type will be selected according to the storage component selected for the board in the `targets.json` file: FAT for "components": ["SD"] and Littlefs for "components": ["SPIF"]. -* `blockdevice` - Options are default, SPIF, DATAFLASH, QSPIF or SD. If file system is set to default, this parameter is ignored. -* `external_size` - The size of the external block device in bytes. If not set, the maximum available size will be used. -* `external_base_address` - The start address of the external block device. If not set, 0 address will be used. -* `mount_point` - Mount point for the file system. This parameter will be ignored if the file system is set to default. -* `folder_path` - Path for the working directory where the FileSystemStore stores the data. - -## Storage configuration - -Below is the main storage configuration `mbed_lib.json` file: - -``` -{ -"name": "storage", - "config": { - "storage_type": { - "help": "Options are TDB_INTERNAL, TDB_EXTERNAL, TDB_EXTERNAL_NO_RBP, FILESYSTEM, FILESYSTEM_NO_RBP or default. If default, the storage type will be chosen according to the component defined in targets.json", - "value": "NULL" - }, - "default_kv": { - "help": "A string name for the default kvstore configurtaion", - "value": "kv" - } - } -} -``` - -### `TDB_INTERNAL` - -The internal configuration should be used for targets willing to save all the data in internal flash. - -![TDB_Internal](./Internal.jpg) - -In this configuration, all KVStore C APIs are mapped to the TDBStore in the internal flash. To use this configuration, set the `storage_type` parameter in storage `mbed_lib.json` to `TDB_INTERNAL`. - -Below is the `TDB_INTERNAL` configuration `mbed_lib.json`: - -``` -{ - "name": "tdb_internal", - "config": { - "internal_size": { - "help": "Size of the FlashIAP block device", - "value": "NULL" - }, - "internal_base_address": { - "help": "If not defined the default is the first sector after the application code ends.", - "value": "NULL" - }, - "rbp_number_of_entries": { - "help": "If not defined default is 64", - "value": "64" - } - } -} -``` - -For this configuration, please define the section of the internal storage that will be used for data, by defining these parameters in your `app.config file`: `internal_base_address` and `internal_size`. If not defined, the storage will start in the first sector immediately after the end of the application. This can reduce the ability to update the application with a bigger one. - -### `TDB_External` - -![External](./TDB_External.jpg) - -`TDB_EXTERNAL` uses a TDBStore in the internal flash for security rollback protection and a TDBStore on the external flash for the data. In this configuration, all KVStore C API calls are mapped to work with the SecureStore class. This class handles the use of the two TDBStores. Tthe external TDBStore works on top of the default block device, and the internal TDBStore works with the FlashIAPBlockdevice. - -You can set the external TDBStore block device to any of the following block devices: SPIF, QSPIF, DATAFASH and SD. - -You can enable this configuration by setting `storage_type` in storage `mbed_lib.json` to `TDB_EXTERNAL`. - -Below is the `TDB_EXTERNAL` configuration `mbed_lib.json`: - -``` -{ - - "name": "tdb_external", - "config": { - "rbp_internal_size": { - "help": "If not defined default size is 4K*#enteries/32", - "value": "NULL" - }, - "rbp_number_of_entries": { - "help": "If not defined default is 64", - "value": "64" - }, - "internal_base_address": { - "help": "If not defined the default is the first sector after the application code ends.", - "value": "NULL" - }, - "blockdevice": { - "help": "Options are default, SPIF, DATAFASH, QSPIF or SD", - "value": "NULL" - }, - "external_size": { - "help": "Size of the external block device", - "value": "NULL" - }, - "external_base_address": { - "help": "If not defined the default is from address 0", - "value": "NULL" - } - } -} -``` - -### TDB_External_no_RBP - -![External](./TDB_External_no_rbp.jpg) - -`TDB_EXTERNAL_NO_RBF` configuration has no support for rollback protection and is therefore less secure. - -The `TDB_EXTERNAL_NO_RBP` uses only one TDBStore on the external flash for all data. In this configuration, all KVStore C API calls are mapped to work with the SecureStore class. The external TDBStore works on top of the default block device; however, you can set the external TDBStore block device to any of the following block devices: SPIF, QSPIF, DATAFASH and SD. - -You can enable this configuration by setting `storage_type` in storage `mbed_lib.json` to `TDB_EXTERNAL_NO_RBP`. - -Below is the `TDB_EXTERNAL_NO_RBP` configuration `mbed_lib.json`: - -``` -{ - "name": "tdb_external_no_rbp", - "config": { - "external_size": { - "help": "Size of the external block device", - "value": "NULL" - }, - "external_base_address": { - "help": "If not defined the default is from address 0", - "value": "NULL" - }, - "blockdevice": { - "help": "Options are default, SPIF, DATAFASH, QSPIF or FILESYSTEM", - "value": "NULL" - } - } -} -``` - -### FILESYSTEM - -![FILESYSTEM](./FILESYSTEM.jpg) - -The FILESYSTEM configuration resembles the EXTERNAL but uses FileSystemStore on the external flash. By default, FileSystemStore uses the default file system and the default block device. - -In this configuration, all KVStore C API paths are mapped to the SecureStore class. This class handles the use of the internal TDBStore or external FileSystemStore. - -You can enable this configuration by setting `storage_type` in storage `mbed_lib.json` to FILESYSTEM. - -Below is the FILESYSTEM configuration `mbed_lib.json`: - -``` -{ - "name": "filesystem_store", - "config": { - "rbp_internal_size": { - "help": "If not defined default size is 4K*#enteries/32", - "value": "NULL" - }, - "rbp_number_of_entries": { - "help": "If not defined default is 64", - "value": "64" - }, - "internal_base_address": { - "help": "If not defined the default is the first sector after the application code ends.", - "value": "NULL" - }, - "filesystem": { - "help": "Options are default, FAT or LITTLE. If not specified default filesystem will be used", - "value": "NULL" - }, - "blockdevice": { - "help": "Options are default, SPIF, DATAFASH, QSPIF or FILESYSTEM. If not set the default block device will be used", - "value": "NULL" - }, - "external_size": { - "help": "Size in bytes of the external block device, if not specified the maximum is the default.", - "value": "NULL" - }, - "external_base_address": { - "help": "If not defined the default is from address 0", - "value": "NULL" - }, - "mount_point": { - "help": "Where to mount the filesystem. Ignored if the default file system is applied.", - "value": "/sd" - }, - "folder_path": { - "help": "Path for the working directory where the FileSyetemStore stores the data", - "value": "/kvstore" - } - } -} -``` - -If file system is not set, the default file system and block device are applied and `blockdevice`, `external_size` and `external_base_address` are ignored. - -### FILESYSTEM_NO_RBP - -![FILESYSTEM](./FILESYSTEM_no_rbp.jpg) - -The `FILESYSTEM_NO_RBP` configuration resembles the `EXTERNAL_NO_RBP` but uses FileSystemStore on the external flash. By default, FileSystemStore uses the default file system and the default block device. This Configuration has no support for rollback protection and is therefore less secure. - -In this configuration, all KVStore C API calls are mapped to the SecureStore class. This class handles the use of the external FileSystemStore. - -You can enable this configuration by setting `storage_type` in `storage mbed_lib.json` to `FILESYSTEM_NO_RBF`. - -Below is the FILESYSTEM configuration `mbed_lib.json`: - -``` -{ - "name": "filesystem_store_no_rbp", - "config": { - "filesystem": { - "help": "Options are default, FAT or LITTLE. If not specified default filesystem will be used", - "value": "NULL" - }, - "blockdevice": { - "help": "Options are default, SPIF, DATAFASH, QSPIF or FILESYSTEM. If not set the default block device will be used", - "value": "NULL" - }, - "external_size": { - "help": "Size in bytes of the external block device, if not specified the maximum is the default.", - "value": "NULL" - }, - "external_base_address": { - "help": "If not defined the default is from address 0", - "value": "NULL" - }, - "mount_point": { - "help": "Where to mount the filesystem. Ignored if the default file system is applied.", - "value": "/sd" - }, - "folder_path": { - "help": "Path for the working directory where the FileSyetemStore stores the data", - "value": "/kvstore" - } - } -} -``` - -If file system is not set, the default file system and block device are applied and `blockdevice`, `external_size` and `external_base_address` are ignored. - -### Configuration functions API - -Applications must call the function **storage_configuration()** to instantiate the required configuration. This function is defined as weak to allow the replacement of this function with a completely different implementation of the instantiation of components. - -Below is a list of setup functions that `storage_configuration()` calls in each case, and their description: - -``` -#if MBED_CONF_STORAGE_STORAGE == NULL -define MBED_CONF_STORAGE_STORAGE USER_DEFINED -#endif - -#define _STORAGE_CONFIG_concat(dev) _storage_config_##dev() -#define _STORAGE_CONFIG(dev) _STORAGE_CONFIG_concat(dev) - -/** - * @brief This function initializes internal memory secure storage - * This includes a TDBStore instance with a FlashIAPBlockdevice - * as the supported storage. - * The following is a list of configuration parameter - * MBED_CONF_STORAGE_INTERNAL_SIZE - The size of the underlying FlashIAPBlockdevice - * MBED_CONF_STORAGE_INTERNAL_BASE_ADDRESS - The start address of the underlying FlashIAPBlockdevice - * MBED_CONF_STORAGE_INTERNAL_RBP_NUMBER_OF_ENTRIES - If not defined default is 64 - * @returns true on success or false on failure. - */ -bool _storage_config_TDB_INTERNAL(); - -/** - * @brief This function initialize external memory secure storage - * This includes a SecureStore class with TDBStore over FlashIAPBlockdevice - * and an external TDBStore over a default blockdevice unless configured differently. - * The following is a list of configuration parameter: - * MBED_CONF_STORAGE_TDB_EXTERNAL_RBP_INTERNAL_SIZE - Size of the internal FlashIAPBlockDevice and by default is set to 4K*#enteries/32. The start address will be set to end of flash - rbp_internal_size. - * MBED_CONF_STORAGE_TDB_EXTERNAL_RBP_NUMBER_OF_ENTRIES - If not defined default is 64 - * MBED_CONF_STORAGE_TDB_EXTERNAL_EXTERNAL_SIZE - Size of the external blockdevice in bytes or NULL for max possible size. - * MBED_CONF_STORAGE_TDB_EXTERNAL_EXTERNAL_BASE_ADDRESS - The block device start address. - * MBED_CONF_STORAGE_TDB_EXTERNAL_EXTERNAL_BLOCK_DEVICE - Alowed vlaues are: default, SPIF, DATAFASH, QSPIF or SD - * @returns true on success or false on failure. - */ -bool _storage_config_TDB_EXTERNAL(); - -/** - * @brief This function initialize a predefined external memory secure storage - * This includes a SecureStore class with external TDBStore over a blockdevice or, - * if no blockdevice was set the default blockdevice will be used. - * The following is a list of configuration parameter: - * MBED_CONF_STORAGE_TDB_EXTERNAL_NO_RBP_EXTERNAL_SIZE - Size of the external blockdevice in bytes or NULL for max possible size. - * MBED_CONF_STORAGE_TDB_EXTERNAL_NO_RBP_EXTERNAL_BASE_ADDRESS - The block device start address - * MBED_CONF_STORAGE_TDB_EXTERNAL_NO_RBP_EXTERNAL_BLOCK_DEVICE - Alowed vlaues are: default, SPIF, DATAFASH, QSPIF or SD - * @returns true on success or false on failure. - */ -bool _storage_config_TDB_EXTERNAL_NO_RBP(); - -/** - * @brief This function initialize a predefined FILESYSTEM memory secure storage - * This includes a SecureStore class with TDBStore over FlashIAPBlockdevice - * in the internal memory and an external FileSysteStore. If blockdevice and filesystem not set, - * the system will use the default block device and default filesystem - * The following is a list of configuration parameter: - * MBED_CONF_STORAGE_FILESYSTEM_RBP_INTERNAL_SIZE - Size of the internal FlashIAPBlockDevice and by default is set to 4K*#enteries/32. The start address will be set to end of flash - rbp_internal_size. - * MBED_CONF_STORAGE_FILESYSTEM_RBP_NUMBER_OF_ENTRIES - If not defined default is 64 - * MBED_CONF_STORAGE_FILESYSTEM_FILESYSTEM - Allowed values are: default, FAT or LITTLE - * MBED_CONF_STORAGE_FILESYSTEM_BLOCKDEVICE - Allowed values are: default, SPIF, DATAFASH, QSPIF or SD - * MBED_CONF_STORAGE_FILESYSTEM_EXTERNAL_SIZE - External Blockdevice size in bytes or NULL for max possible size. - * MBED_CONF_STORAGE_FILESYSTEM_EXTERNAL_BASE_ADDRESS - The block device start address. - * MBED_CONF_STORAGE_FILESYSTEM_MOUNT_POINT - Where to mount the filesystem - * MBED_CONF_STORAGE_FILESYSTEM_FOLDER_PATH - The working folder paths - * - * @returns true on success or false on failure. - */ -bool _storage_config_FILESYSTEM(); - -/** - * @brief This function initialize a predefined FILESYSTEM_NO_RBP memory secure storage with no - * rollback protection. This includes a SecureStore class an external FileSysteStore over a default - * filesystem with default blockdevice unless differently configured. - * The following is a list of configuration parameter: - * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_FILESYSTEM - Allowed values are: default, FAT or LITTLE - * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_BLOCKDEVICE - Allowed values are: default, SPIF, DATAFASH, QSPIF or SD - * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_EXTERNAL_SIZE - Blockdevice size in bytes. or NULL for max possible size. - * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_EXTERNAL_BASE_ADDRESS - The block device start address. - * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_MOUNT_POINT - Where to mount the filesystem - * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_FOLDER_PATH - The working folder paths - * - * @returns true on success or false on failure. - */ -bool _storage_config_FILESYSTEM_NO_RBP(); - -MBED_WEAK bool storage_configuration() -{ - return _STORAGE_CONFIG(MBED_CONF_STORAGE_STORAGE_TYPE); -} -``` - -### Override user-defined setup - -To create a more complex setup including using other block devices, such as MBRBlockDevice or SlicingBlockDevice, you need to override the `storage_configuration` function and generate the storage configuration you choose. diff --git a/docs/design-documents/features/storage/Configuration/FILESYSTEM.jpg b/docs/design-documents/features/storage/Configuration/FILESYSTEM.jpg deleted file mode 100644 index 52ea615..0000000 --- a/docs/design-documents/features/storage/Configuration/FILESYSTEM.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/Configuration/FILESYSTEM.xml b/docs/design-documents/features/storage/Configuration/FILESYSTEM.xml deleted file mode 100644 index c8434cb..0000000 --- a/docs/design-documents/features/storage/Configuration/FILESYSTEM.xml +++ /dev/null @@ -1 +0,0 @@ -7Vjfb9owEP5reNyU39DHUKBFYxITaFsf3eRIvJoYOQ4l++tnkwtJmrSqJho0NF6wP/vsu+++s5MM7Nvt4U6QXfyVh8AGlhEeBvZkYFmmY3nqTyN5gYwMpwAiQUOcVAEr+hsQNBDNaAhpY6LknEm6a4IBTxIIZAMjQvDn5rQNZ81ddySCFrAKCGujP2goY4zCNSr8HmgUlzubBo48kuApEjxLcL+BZW+Ov2J4S8q1cH4ak5A/1yB7OrBvBeeyaG0Pt8A0tyVthd3sldGT3wIS+S6DG/RD5mXsECoqsMuFjHnEE8KmFTo+xgd6BUP1YrllqmmqJhyo/Knhzy72HsqRRIq8NqS7D7jAL5AyRw2QTHIFVfsuON/hGu3YMNyUZyJA7y1UCxERnLJTYDqwmh0Scgd8C8obNUEAI5LumxogKKXoNK+iUzWQ0W520Zk9YRkuqkyMp30quQDV8pfzFv01Ol8NeA9CwuHNWHDULoWWlwWG/edK1uYIsbgmads4Q/hGK7p/W1ylkhrqsi+lrtKbmry+fD9uhP5ZBiM5iL4lNupTYsNrk9jZ5YSmS04TWWXNcptZs90X2Sg0hFYvEnJy4105ss3/OfqrHLlebzkqQ6qdJSsIMqHycbyqej1BLK/HE8S5tkvK6bikHOtSl5TTvqTWk3EfqjKtV6qnpqrTKXh2VbUf/WaMpPHcX44ZD54msKdB3ww4Zp8M2NdWV05HXQ0vVldOW2CUwSpPJWz7KC/HeUd5eR8lLrcV/QQ2JGPqHtSvWKvlfFa0vlXNib/2Zwt/da/aXOhZys5TJvb4UXW9SJ6Y6Y01Z9gna961leSwoyTdi5Xk8G1Rzvx1qbzFfL1eTPvWWtfx/2FaG7XImCcShMpyK2wVj2xqS0BKf5PH4wStmp1+xD16444H7kQhWjtpISNtQBiNEtVmsNFLaZJoQJiPsNS6Gqc7EtAkWh9F9sk5D8uW6zbPwY4vLF0kW+cg+aZF8vRwjSS7xovLxjA/imTVrT6/Fm9S1Tdue/oH \ No newline at end of file diff --git a/docs/design-documents/features/storage/Configuration/FILESYSTEM_no_rbp.jpg b/docs/design-documents/features/storage/Configuration/FILESYSTEM_no_rbp.jpg deleted file mode 100644 index 04659b7..0000000 --- a/docs/design-documents/features/storage/Configuration/FILESYSTEM_no_rbp.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/Configuration/FILESYSTEM_no_rbp.xml b/docs/design-documents/features/storage/Configuration/FILESYSTEM_no_rbp.xml deleted file mode 100644 index 6efe447..0000000 --- a/docs/design-documents/features/storage/Configuration/FILESYSTEM_no_rbp.xml +++ /dev/null @@ -1 +0,0 @@ -3ZjbcpswEIafhst2MALbufQxydSdSceeNrmUYQ1qZOQRwjF5+kqwnIKTSacpmcQ3Xv3alXZXn/DBIrP96VLSQ/RdBMAtxw5OFplbjjNwnaF+M0pWKGPbLYRQsgCdamHNHgFFG9WUBZC0HJUQXLFDW/RFHIOvWhqVUjy03XaCt3c90BA6wtqnvKv+YoGKsArPrvUrYGFU7jywcWZL/ftQijTG/SyH7PJXMb2n5Vron0Q0EA8NiSwsMpNCqMLan2bATW/LthVxy2dmq7wlxOpVAReYh8rK2iHQrcChkCoSoYgpX9TqNK8PzAq2HkVqz7U50CacmLo18lcPR3flTKxk1pgywztc4DcolSEDNFVCS/W+KyEOuEa3Niw3Ean0MXsHaaEyhOp0Cs0U1ojDhlyC2IPORjtI4FSxY5sBiiiFlV/dTm1gR893F5M5Up7iojrEvj8mSkjQ1uTmutP+RjufLfgIUsHpxVpwlpSgZeUFw/FDjfVgjFrUQJrYb1C+3anuY8NVktSii7wXXWU2Dby+/cw3wvwcm9MMZN+IjXtEjAw61X1wxM7j1GbOdf8RMQy9ESxW9Ul6w/ZJEu/JCRU5YNSTQ6rSeB27pMPuGvxU6jPKH429EusMeyTWJZ+MWNc9A+jovR6KZTYNsJaMwzpLFOz7gMsZP3OJGnBVwL05XF6n+jnsaMr1fTVfPdY318vC+lGb88lmslxN1lfaFtJ46bihDiHTrR4OQ1V1preuuaM+uzb8bFdydOZKeu92JUcvQ7mcbEryVtebzWrRN2uDPlm76DRjcVIg9Sl3ytb1qDZbEhL2SLe5g6HmYD6K82y8qeXNtWLYSQqMTADlLIy1zWFnljJNYvpn7gRlZbiaJgfqszjc5JB9cf/Tc7BcotHlc012/r7Jelj/Yi6+jNR/S5DFHw== \ No newline at end of file diff --git a/docs/design-documents/features/storage/Configuration/Internal.jpg b/docs/design-documents/features/storage/Configuration/Internal.jpg deleted file mode 100644 index 3aadaa1..0000000 --- a/docs/design-documents/features/storage/Configuration/Internal.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/Configuration/Internal.xml b/docs/design-documents/features/storage/Configuration/Internal.xml deleted file mode 100644 index d3e1441..0000000 --- a/docs/design-documents/features/storage/Configuration/Internal.xml +++ /dev/null @@ -1 +0,0 @@ -3VVNT8JAEP01vZptFwgeBfEjamKCUTmu7dBWth2yTKH117ulU9qmYjwoB7kw8+b78UodOU3ya6PW0QMGoB1PBLkjLx3PcwfeyH6VSFEhYzGogNDEASc1wDz+AAYFo1kcwKaTSIia4nUX9DFNwacOpozBXTdtibo7da1C6AFzX+k++hIHFPEVQ9HgNxCHUT3ZFRx5U/4qNJilPM/x5HL/qcKJqntx/iZSAe5akJw5cmoQqbKSfAq65Lamraq7OhI97G0gpR8VnPMeVNS3Q2CpYBcNRRhiqvSsQSf7+6DsIKwXUaKt6VoT8pheS/hsyN6ijqRkilaodBfc4B2ICtaAyggt1My9R1xzj2rTcr2jxzK0wcz4nOWxfJQJ4fBzHZi2CgZMwG5jcwxoRfG2216xlMJDXkOnNZjRr9nl2VulM25qS8RquyE0YK2Lx9se/S06uRoMQf79yf1buEAOWGhF/YCxv2tk7Y4Zi1qSluIXzhe96/6ZuNz6z6mtLnkiddXDW/K6e94P4nU8oVUB5tQSG59QYhXZHQ6eLidzMggnPtsb/d3Z1m1eCftY670rZ58= \ No newline at end of file diff --git a/docs/design-documents/features/storage/Configuration/NewStorageDesign.jpg b/docs/design-documents/features/storage/Configuration/NewStorageDesign.jpg deleted file mode 100644 index 601d575..0000000 --- a/docs/design-documents/features/storage/Configuration/NewStorageDesign.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/Configuration/TDB_External.jpg b/docs/design-documents/features/storage/Configuration/TDB_External.jpg deleted file mode 100644 index 10994b6..0000000 --- a/docs/design-documents/features/storage/Configuration/TDB_External.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/Configuration/TDB_External.xml b/docs/design-documents/features/storage/Configuration/TDB_External.xml deleted file mode 100644 index d2c83a2..0000000 --- a/docs/design-documents/features/storage/Configuration/TDB_External.xml +++ /dev/null @@ -1 +0,0 @@ -5VhRb5swEP41PHYCG9rmMYSkrdZJnVJt66MLDnh1cGRMmvTXz4YjQE2nTsqI1uUl9mdzvvv47nyJg2fr3ZUkm+yLSCh3kJvsHBw5CHk+OtdfBtnXyGWAayCVLIFNLbBkLxRAF9CSJbTobVRCcMU2fTAWeU5j1cOIlOK5v20leP/UDUmpBSxjwm30O0tU1kThtvg1ZWnWnOy5sPJI4qdUijKH8xyEV9WnXl6TxhbsLzKSiOcOhOcOnkkhVD1a72aUG24b2urnFm+sHvyWNFfvemACfqh9EztNNBUwFVJlIhU54fMWDav4qLHg6lmm1lwPPT2kO6Z+GPhTALOHZiVXct9ZMtMHMPCTKrUHDZBSCQ21594KsQEbdmwQbiFKGYP3CNRCZEoPb6fGTGCd54CQKyrWVHujN0jKiWLbvgYISCk97Gvp1ANgdJhdcGZLeAlG9SPu07ZQQlI9mt7dWPR36Hwz4C2Viu5+Gwus4kZo+ybBYP7cytq7BCzrSBq7RwjftaL7t8XVKKmnLnwqdTXedOT1+Vt1EPiHXE72VI4tscsxJXbx0SSGbYmhkxUw7P0P9J4sg7FvZfB8p6jUgVm862xUfTolLdgLeaw2GKI2guWq8iYInSDSCOEszTUQa0p0HcChyWqm25wpLKxZklTvjJNHysND8zITXMjq3KZ9GeIXv1UrDo0ZeNdrboZqiPvJnwSoV0aCevZuzsH2nSGhNXx20TN6FvQNiNWqMIXy1Ss7OPi+Ooytt7ikcSl1QlTX/KjVF52PWH0x+mDlAQ1c8E0Xd4L20b7g76NwDFV5r1SFA1tVyP9bd7rdNi84KbKb6V3IRfwU0S2Lx2bA90ZkAONx8uq4uTN0tfonyx27KI+UO/7FKXPH7igiuiIl15ec+e25NL84zSCa3k8Xt9PldT39qhcW9VBf/HpfNDJL4+ZXYLF0kx+z79K5VNRp5XXaME5XaqAJUybPwmJDYpan91XSnfnHYRkFr7qDycRi+XyAZPTnJOtp+7dV3UW1/w3i+S8= \ No newline at end of file diff --git a/docs/design-documents/features/storage/Configuration/TDB_External_no_rbp.jpg b/docs/design-documents/features/storage/Configuration/TDB_External_no_rbp.jpg deleted file mode 100644 index 06644f8..0000000 --- a/docs/design-documents/features/storage/Configuration/TDB_External_no_rbp.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/Configuration/TDB_External_no_rbp.xml b/docs/design-documents/features/storage/Configuration/TDB_External_no_rbp.xml deleted file mode 100644 index 414d36d..0000000 --- a/docs/design-documents/features/storage/Configuration/TDB_External_no_rbp.xml +++ /dev/null @@ -1 +0,0 @@ -1VfbbuIwEP2aPG4V7KSCRyDQrrYrdUW1u300yZB4a2JkHC79+nWScS4NVJVKU5UXPGfs8VyOx45Dp+vDjWKb5KeMQDjEjQ4ODRxCBh65Nn85ciyRoeuVQKx4hJNqYMGfAUEX0YxHsG1N1FIKzTdtMJRpCqFuYUwpuW9PW0nR3nXDYugAi5CJLvqHRzrBKHy3xm+Bx4ndeeCiZsnCp1jJLMX9HEJXxa9Ur5m1hfO3CYvkvgHRmUOnSkpdjtaHKYg8tzZt5br5GW3lt4JUv2nBCP3QRxs7RCYVKEqlExnLlIlZjU6K+CC34Bop0WthhgMzhAPXf3P4ykfp0WpSrY4NVS4+ooF/oPUROcAyLQ1U73sn5QZtdGPDcLcyUyF6T5AtTMVQVafE8sAa6zAhNyDXYLwxExQIpvmuzQGGVIqreXU6zQAzejq76MyOiQyNmiXu026rpQIzGt9/76S/kc6zAe9AaTi8GgtqqSXa0R4wlPc1rQdDxJIGpal7gfDdTnRfm1yWSS120c9il/WmQa8fv4uN0D/iCnYE1TfFhj1SjA460X1xitEuxcjFKYZL7yVPdV1Jct2uZFVZa6J0FFe9KFLlxtvq5nW4OztoUCZjnYIaHup2nRRs+TNbFhPyCmxyjwof/YnjBwZhgsepAUKTa3MC6CTnMzcX/BgVax5FBRkEW4KYVNf2VAqpin3txX2qcPTcKameJOhd61o/dXrcK2/kk1ba/VJ6ZzG/jfxTVq0BuVpt4b1VtGRtVHEBYabMSSsuuF77TsXeXvoO7afvXPZtdKq3eJ91fZEueR6CSR/MIaMXfc7zO8wh3kc9irqdL4AVy4Q5jPnrcJG/CfNBMH4Yz+/Gi9tS/GUU83JoGpSZF/ScJc/9uCwZsf7mKRtR/WFJZ/8B \ No newline at end of file diff --git a/docs/design-documents/features/storage/Configuration/UserDefined.jpg b/docs/design-documents/features/storage/Configuration/UserDefined.jpg deleted file mode 100644 index dbcfe8a..0000000 --- a/docs/design-documents/features/storage/Configuration/UserDefined.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/Configuration/UserDefined.xml b/docs/design-documents/features/storage/Configuration/UserDefined.xml deleted file mode 100644 index 08de035..0000000 --- a/docs/design-documents/features/storage/Configuration/UserDefined.xml +++ /dev/null @@ -1 +0,0 @@ -3VjLctowFP0aL9ux/CCwNAESprSTDjRtlop9Y6sRFiMLAvn6Srb8ikjbmYLdgQVIR1ePe3yOJWG51+v9Dceb5DOLgFqOHe0td2I5DvKcgfxRyKFAhrZXADEnkQ6qgSV5BQ3aGt2SCLJWoGCMCrJpgyFLUwhFC8Ocs5d22BOj7Vk3OAYDWIaYmuh3EolEZ+HbNX4LJE7KmZGtWx5x+Bxztk31fJbjPuWfonmNy7F0fJbgiL00IHdqudecMVGU1vtroIrbkrai3+yd1mrdHFLxNx2qdYhDmTtEkgpdZVwkLGYpptMaHef5gRrBlrVErKksIlmEPRE/FPzR17WHsiUV/NBoUtUHPcBPEOKgNYC3gkmonnfB2EaPYeam083Ylod69a5WC+Yx6KhhAam8Gt00HzfA1iAXIwM4UCzIri0BrJUUV3E1m7KgCT1Orl7LDtOtHvTTfT6PXp1jU3wAbjyBBqPv5rwDLmD/23x0q1s+Y+1FB/lF/aVWdmW8pKFq1/53CtDVhenLM/Xl9qUvz9CX7GI/7zLBOMhScDfvWFuofEN2oa3S2BejLd/UFupNXL4hrlmwstTznsnvxXy1Wkyr6pdvi8WZpeb7bal59pHXmHsuqSEjuxNIDTWE1tBWF1IbHpGa09s+eRZ27f+LXdQXu0PDyEsIt1wSWewShYNXk3Eb6MDSxsnkqsOTych8v1GcJfPgbkxZ+DyBHQk7ZWPY4/vNvbRrQGm3pgVHfTmwXExDa33Y7Y3AXK/LDfTS7gHlbtkUmN+bwJw/C2xGKGSHTMC6c+G9Pbp1qjxk3sK/Lu/ms4qAVmUSrILZIlje1s2TXk+5V0cuVGfbBcwL5TwVwKU9LGdA5UzjRy5LsaiSbRAhMxRtm3LIyCt+zAOUATeMpCJfnz+2/IlElA2zwpGqA6YkTmWZwpMaStFGQkwDDQtl0XG2wSFJ41Xu1w/eaXhHI7/FuzMYGbwPjtDunIJ286o13Ze0XxLJA/T2vGeK+0Qky2r9J2re1vin2p3+Ag== \ No newline at end of file diff --git a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_class_hierarchy.jpg b/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_class_hierarchy.jpg deleted file mode 100644 index ca9f62a..0000000 --- a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_class_hierarchy.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_class_hierarchy.xml b/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_class_hierarchy.xml deleted file mode 100644 index 173fdab..0000000 --- a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_class_hierarchy.xml +++ /dev/null @@ -1 +0,0 @@ -7VfbbtswDP0aP3bwJU7zmjRNCwwDBgS7PSo2bQuVLU9WEmdfP8qm41u8ZsDWbkDzEFuHFEUe8ZJY3l1aPiiWJx9kCMJy7bC0vLXluo7vz/BhkFON3C4IiBUPSakFtvwHEGgTuuchFD1FLaXQPO+DgcwyCHQPY0rJY18tkqJ/as5iGAHbgIkx+oWHOqnRhW+3+CPwOGlOdmyS7FjwFCu5z+g8y/Wi6lOLU9bYIv0iYaE8diDv3vLulJS6fkvLOxCG24a2et9mQnr2W0Gmr9pAfhyY2EPj8lzg3lVu3NMnomT+fW98WqVMxTyzvCVK7bzEbwSryAx+o2Vey2YdmYZS3zDBY9oXoHOgWpv4FtOzOnnXABsuYHsqNKRbLRU0cgxnN9yDWD7EEmVCaJKrOc2ZPjjkh4tBJ3TXxnnXBDZlAL2obDSo2zPnHkBpjlm2rLlYV2ytiJm1gMjskqgViSopIo6X5a0imWmqEcel9YalXJjqegRxAGPV+KlTYZTOZ3cTgXLD+ABlB6LEeACZglYnVCGp61FyUBG7M1of25JwGizplMOCMEZVGJ9Nt5mIL5SME4npjBLzUwEjRiELl6becSVzQBZXiHS5ClmRQEikXMMPhL3WMGanG/2cIlUgmOaHfve4FD6Z+yg5Hntm2rsdMD1ksJB7FQDt6pbzwJBvP2NIY5mCHhmqbuMc43UX5P4fneNXTWOn3trIX28jo5x80TYy+5ez9P3nt7n2+nNt8ZL56F/Ix+mZthMyeBpOMZRvKgLX9njedahChtTpq1F75zfLb8/RWA+IfvG8ykic6hq/OxJn84Eh70+NRFy2P9Rr9fbfkHf/Ew== \ No newline at end of file diff --git a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_design.md b/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_design.md deleted file mode 100644 index 521eaf2..0000000 --- a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_design.md +++ /dev/null @@ -1,410 +0,0 @@ -# FileSystemStore in Mbed OS - -- [FileSystemStore in Mbed OS](#filesystemstore-in-mbed-os) - + [Revision history](#revision-history) -- [Introduction](#introduction) - + [Overview and background](#overview-and-background) - + [Requirements and assumptions](#requirements-and-assumptions) -- [System architecture and high-level design](#system-architecture-and-high-level-design) - * [Design basics](#design-basics) -- [Detailed design](#detailed-design) - + [Class header](#class-header) - + [Important data structures](#important-data-structures) - + [Initialization and reset](#initialization-and-reset) - + [Core APIs](#core-apis) - + [Incremental set APIs](#incremental-set-apis) - + [Key iterator APIs](#key-iterator-apis) -- [Usage scenarios and examples](#usage-scenarios-and-examples) - + [Standard usage of the class](#standard-usage-of-the-class) -- [Other information](#other-information) - + [Open issues](#open-issues) - - -### Revision history - -| Revision | Date | Authors | Mbed OS version | Comments | -|---------- |---------------- |-------------------------------------------------------- |----------------- |------------------ | -| 1.0 | 20 September 2018 | David Saada ([@davidsaada](https://github.com/davidsaada/)) | 5.11+ | Initial revision | - -# Introduction - -### Overview and background - -FileSystemStore is a lightweight implementation of the [KVStore](../KVStore/KVStore_design.md) interface over file systems. - -### Requirements and assumptions - -FileSystemStore assumes the underlying file system qualities for resilience and file validation. This means that if the underlying file system has no protection against power failures, then neither would FileSystemStore have. - -When initializing this class, it is assumed that the underlying FileSystem is initialized and mounted. - -# System architecture and high-level design - -## Design basics - -FileSystemStore implements the get/set interface using files, where a single file represents each key. A key is represented by the file name, and its value is stored as file data. Therefore, FileSystemStore imitates the get/set actions using simple file operations. Set is achieved using open-write-close, get using open-read-close and so on. - -All files are concentrated under a single directory, whose name is hard coded. So actions such as "reset" are mapped to the deletion of all files under this directory, and iteration actions use file system APIs to traverse the directory. - -### Data layout - -When storing the data, it is stored with a preceding 16-byte metadata header. Metadata includes flags and other parameters for basic validity checks. - -![FileSystemStore Record](./FileSystemStore_record.jpg) - -Fields are: - -- Magic: A constant value, for quick validity checking. -- Metadata size: Size of metadata header. -- Revision: FileSystemStore revision (currently 1). -- User flags: Flags received from user. Currently only write once is dealt with (others are ignored). - -# Detailed design - -FileSystemStore fully implements the KVStore interface over a file system. As such, it uses the FileSystem class interface for file operations. - -![FileSystemStore Class Hierarchy](./FileSystemStore_class_hierarchy.jpg) - -Functionality, as defined by KVStore, includes the following: - -- Initialization and reset. -- Core actions: get, set and remove. -- Incremental set actions. -- Iterator actions. - -### Class header - -FileSystemStore has the following header: - -```C++ -class FileSystemStore : KVStore { - -public: - FileSystemStore(FileSystem *fs); - virtual ~FileSystemStore(); - - // Initialization and reset - virtual int init(); - virtual int deinit(); - virtual int reset(); - - // Core API - virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags); - virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0); - virtual int get_info(const char *key, info_t *info); - virtual int remove(const char *key); - - // Incremental set API - virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags); - virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size); - virtual int set_finalize(set_handle_t handle); - - // Key iterator - virtual int iterator_open(iterator_t *it, const char *prefix = NULL); - virtual int iterator_next(iterator_t it, char *key, size_t key_size); - virtual int iterator_close(iterator_t it); - -private: - Mutex _mutex; - FileSystem *_fs; - bool _is_initialized; -} -``` - -### Important data structures - -```C++ -// Key metadata -typedef struct { - uint32_t magic; - uint16_t metadata_size; - uint16_t revision; - uint32_t flags; -} key_metadata_t; - -// incremental set handle -typedef struct { - char *key; - uint32_t create_flags; -} inc_set_handle_t; - -// iterator handle -typedef struct { - void *dir_handle; - char *prefix; -} key_iterator_handle_t; -``` - -### Initialization and reset - -**init function** - -Header: - -`virtual int init();` - -Pseudo code: - -- If `_is_initialized`, return OK. -- Create and take `_mutex`. -- Create the FileSystemStore directory if it doesn't exist. -- Set `_is_initialized` to true. -- Release `_mutex`. - -**deinit function** - -Header: - -`virtual int deinit();` - -Pseudo code: - -- If not `_is_initialized`, return OK. -- Take `_mutex`. -- Set `_is_initialized` to false. -- Release `_mutex`. - -**reset function** - -Header: - -`virtual int reset();` - -Pseudo code: - -- Take `_mutex`. -- Delete all files under the FileSystemStore directory. -- Set `_num_keys` to 0. -- Release `_mutex`. - -### Core APIs - -**set function** - -Header: - -`virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags);` - -Pseudo code: - -- If not `_is_initialized`, return "not initialized" error. -- Call `set_start` with all fields and a local `set_handle_t` variable. -- Call `set_add_data` with `buffer` and `size`. -- Call `set_finalize`. -- Return OK. - -**get function** - -Header: - -`virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0);` - -Pseudo code: - -- If not `_is_initialized`, return "not initialized" error. -- Take `_mutex`. -- Using the `stat` API, extract file size. -- Open file `key` for reading to achieve a file handle. -- If failed, release `_mutex` and return "not found" error. -- Read from file into a `key_metadata_t` structure. -- Using `size` API, achieve file size. -- Seek to `offset` + metadata size. -- Set `actual_size` as the minimum of buffer size and remainder of data. -- Read data from file to `buffer`, size is `actual_size`. -- Close file. -- Release `_mutex`. -- Return OK. - -**get_info function** - -Header: - -`virtual int get_info(const char *key, info_t *info);` - -Pseudo code: - -- If not `_is_initialized`, return "not initialized" error. -- Find file `key` under the FileSystemStore directory. If not existing, return "not found" error. -- Take `_mutex`. -- Open file `key` for reading to achieve a file handle. -- If failed, release `_mutex`, and return "not found" error. -- Using `size` API, achieve file size. -- Read from file into a `key_metadata_t` structure. -- Fill `info` structure with all relevant fields. -- Close file. -- Return OK. - -**remove function** - -Header: - -`virtual int remove(const char *key);` - -Pseudo code: - -- If not `_is_initialized`, return "not initialized" error. -- Take `_mutex`. -- Open file `key` for reading, and read data into a `key_metadata_t` structure. -- If not existing, return "not found error". -- If flag "write once" is preset, return "write once" error. -- Delete file `key`. -- Release `_mutex`. -- Return OK. - -### Incremental set APIs - -**set_start function** - -Header: - -`virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags);` - -Pseudo code: - -- Find file `key` under the FileSystemStore directory. If not existing, increase `_num_keys` by 1. -- Take `_mutex`. -- Open file for reading, and read data into a `key_metadata_t` structure. -- If existing and flag "write once" is preset, return "write once" error. -- Close file. -- Allocate an `inc_set_handle_t` structure into `handle`. -- Duplicate `key` in `handle`. -- Update `create_flags` in `handle`. -- Fill `key_metadata_t` structure with all relevant values (`create_flags` from handle). -- Open file `key` for writing to achieve a file handle. -- Write metadata structure to the file. -- Close file. - -**set_add_data function** - -Header: - -`virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size);` - -Pseudo code: - -- Open file `key` for appending to achieve a file handle. -- Write `value_data` to the file. -- Close file. - -**set_finalize function** - -Header: - -`virtual int set_finalize(set_handle_t handle);` - -Pseudo code: - -- Free `key` in `handle` and then `handle`. -- Release `_mutex`. - -### Key iterator APIs - -**iterator_open function** - -Header: - -`virtual int iterator_open(iterator_t *it, const char *prefix = NULL);` - -Pseudo code: - -- Take `_mutex`. -- Allocate a `key_iterator_handle_t` structure into `it`. -- Duplicate `prefix` into same field in iterator. -- Using directory `open` API, open FileSystemStore directory, and store dir handle in the handle's `dir_handle` field. -- Release `_mutex`. - -**iterator_next function** - -Header: - -`virtual int iterator_next(iterator_t it, char *key, size_t key_size);` - -Pseudo code: - -- Take `_mutex`. -- Using direcory `read` API on handle's `dir_handle` field, read next file in directory. -- While not reached end of directory. - - If name matches prefix: - - Copy file name to `key`, and return OK. - - Using direcory `read` API on handle's `dir_handle` field, read next file in directory. -- Return "not found" error. -- Release `_mutex`. - -**iterator_close function** - -Header: - -`virtual int iterator_close(iterator_t it);` - -Pseudo code: - -- Using directory `close` API on `dir_handle` close handle. -- Release `prefix` field in iterator and structure allocated at `it`. - -# Usage scenarios and examples - -### Standard usage of the class - -The following example code shows standard use of the FileSystemStore class : - -**Standard usage example** - -```C++ - -// External file system of LittleFS type. Should be initialized. -extern LittleFileSystem fs; - -// Instantiate fsstore with our file system -FileSystemStore fsstore(&fs); - -int res; - -// Initialize fsstore -res = fsstore.init(); - -// Add "Key1" -const char *val1 = "Value of key 1"; -const char *val2 = "Updated value of key 1"; -res = fsstore.set("Key1", val1, sizeof(val1), 0); -// Update value of "Key1" -res = fsstore.set("Key1", val2, sizeof(val2), 0); - -uint_8 value[32]; -size_t actual_size; -// Get value of "Key1". Value should return the updated value. -res = fsstore.get("Key1", value, sizeof(value), &actual_size); - -// Remove "Key1" -res = fsstore.remove("Key1"); - -// Incremental write, if need to generate large data with a small buffer -const int data_size = 1024; -char buf[8]; - -KVSTore::set_handle_t handle; -res = fsstore.set_start(&handle, "Key2", data_size, 0); -for (int i = 0; i < data_size / sizeof(buf); i++) { - memset(buf, i, sizeof(buf)); - res = fsstore.set_add_data(handle, buf, sizeof(buf)); -} -res = fsstore.set_finalize(handle); - -// Iterate over all keys starting with "Key" -res = 0; -KVSTore::iterator_t it; -fsstore.iterator_open(&it, "Key*"); -char key[KVSTore::KV_MAX_KEY_LENGTH]; -while (!res) { - res = fsstore.iterator_next(&it, key, sizeof(key)); -} -res = fsstore.iterator_close(&it); - -// Deinitialize FileSystemStore -res = fsstore.deinit(); -``` - -# Other information - -### Open issues - -- Need to figure a way to prevent mutex abuse in incremental set APIs. diff --git a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_record.jpg b/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_record.jpg deleted file mode 100644 index 2d0d9d7..0000000 --- a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_record.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_record.xml b/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_record.xml deleted file mode 100644 index 6114ab0..0000000 --- a/docs/design-documents/features/storage/FileSystemStore/FileSystemStore_record.xml +++ /dev/null @@ -1 +0,0 @@ -3VdLk6IwEP41VO0eZgvCQz2Ozusyl7W29hxJhNQEwoQgur9+OxJeCo47pTXlejH5upM0X3/dBMtdJNtnibP4VRDKLWSTreU+WAg5vo/gTyO7CplMvQqIJCPGqQWW7A81oG3QghGa9xyVEFyxrA+GIk1pqHoYllKUfbe14P1TMxzRI2AZYn6M/mZExRU69e0Wf6EsiuuTHdtYVjh8i6QoUnOehdz1/leZE1zvZfzzGBNRdiD30XIXUghVjZLtgnLNbU1bte5pxNrELWmqzlkQTKoVG8wLWocccFg7J2wDw0gPa2hVA684YmGNwu6rAU87VzvDZ/Be6Aear0Wq7vJ9tu/BwbGzbWs82kIeIkMHATYQ6KnYv3nfT0Y+dGxvP9R7NrTPNtV82mAuY6boMsOhtpZQHYDFKuEwc5rV3fyYlG2oVHTbgUy+nqlIqJI7cDHWqZGOKS2nllLZCtUzUNzR6MRg2JRG1GzcygMGRiHDapnZn1MLVZhghW9UMPv2dFkJotuWYKO5ExoMrqVBZ1SDDZE/6ebHKYL/69yc0R/8a+XG+1R/+JVTedkCe+I4ym+03dz6+wm5XydAxx5XYMPkw+ir6KuZO6N0Hfdq1AUD1B0wAtfVTA/DQvLdXMJll6qPqWl5rGYKKyZSmN7N7As1vckhc94Rc2iAOGcWXIK56ajodEc5q9X4J1vN2E3q7B5UxTGicyBZ9TOWKyne6EJwIQFJRUp1yIzzAwhzFulEhpA66OHuXKeMwXfUvTEkjBB9zKBE+vVFmIRPuUoYuSh02q5xV0G1rLtFNaQN79+LCqbtp9ve1vk+dh//Ag== \ No newline at end of file diff --git a/docs/design-documents/features/storage/KVStore/KVStore_classes.jpg b/docs/design-documents/features/storage/KVStore/KVStore_classes.jpg deleted file mode 100644 index 72d89fa..0000000 --- a/docs/design-documents/features/storage/KVStore/KVStore_classes.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/KVStore/KVStore_classes.xml b/docs/design-documents/features/storage/KVStore/KVStore_classes.xml deleted file mode 100644 index 4e0a0ce..0000000 --- a/docs/design-documents/features/storage/KVStore/KVStore_classes.xml +++ /dev/null @@ -1 +0,0 @@ -7VjbbuIwEP2aPLZKnITSx1J6kVYrVWK1l0eTDIlVJ84aQ8N+/drxODdAi6ou7UN5gPh4PB6fOeOJ8MLbon6QtMq/ihS4R/y09sK5R0gQkYn+McjOIldBYIFMshSNOmDB/gCCPqIblsJ6YKiE4IpVQzARZQmJGmBUSvEyNFsJPty1ohnsAYuE8n30B0tVbtFp7Hf4I7AsdzsHPs4safKcSbEpcT+PhKvmY6cL6nyh/TqnqXjpQeGdF95KIZR9Kupb4IZbR5tdd39kto1bQqlOWoBxbCnfgAt5wvXaWWXCUzukZPJ7Y2KaFVRmrPTCGz3rV7X+1mBzMoNfKFHZuag3p6BWF5SzDNclOjiQnU/9lOFvs/PSAd/ms4USEtyEPsdybKyxaozl0sTuVOW2CY7vmLLtwdPmmGQTNTEnOuZAR9H4cCgZuCNbkIpped1YEuYNTTOkZM5hZVYJbbXijRpWTGcpnK1EqbA4AoLje1owbsrqEfgWjFcTpyq4MWr37isARWFigLoHoSIeQBSg5E6b4GwYoSqweokbv3S1EDgs79XBFDGK5Ze1rjsJ6gdU4RFFRh9ZkV++fwry/QU5Pace4wN6HLEJZXpjuo4eLblInk0C6TqHFCnQ8/cNgXPfjvoc9qjSDMndT2N2Gbvhr3/RqLTOQQ2LB9JBf9snts/cBFmSwKli22ELPEQdunsSTIfSJikKjtwazsVabGQCuKrfk0aO4quRo3DkyJ54z1GTyfaMpyX36iNfNlozsNitFRSfl86rLp29G+ZAAR29dIJ37YLTjyzMBSQbCZ+iPL8o4/gdRdk27jO0woN97CSm/kNrm5Dry/htmlsUDhPYhvj2zc1l5yxvLjVT9sUlinFoXlwu/EufOOAJJNPh6xvGejs1xTrghtx+yz5L2ok/Kjb/lUkPr0dJj6/fKOl62P1ZYM27f2TCu78= \ No newline at end of file diff --git a/docs/design-documents/features/storage/KVStore/KVStore_design.md b/docs/design-documents/features/storage/KVStore/KVStore_design.md deleted file mode 100644 index fc3fa08..0000000 --- a/docs/design-documents/features/storage/KVStore/KVStore_design.md +++ /dev/null @@ -1,488 +0,0 @@ -# KVStore in Mbed OS - -- [KVStore in Mbed OS](#kvstore-in-mbed-os) - + [Revision history](#revision-history) -- [Introduction](#introduction) - + [Overview and background](#overview-and-background) - + [Requirements and assumptions](#requirements-and-assumptions) -- [System architecture and high-level design](#system-architecture-and-high-level-design) - * [Design basics](#design-basics) - + [Derived implementations](#derived-implementations) - * [Global Key Value interface](#global-key-value-interface) -- [Detailed design](#detailed-design) - * [KVStore class design](#kvstore-class-design) - + [KVStore Class header](#kvstore-class-header) - * [Global Key Value interface design](#global-key-value-interface-design) - + [Global Key Value APIs](#global-key-value-apis) - * [Mapping APIs](#mapping-apis) - * [Implementation](#implementation) - + [Important data structures](#important-data-structures) - + [Global Key Value API implementation](#global-key-value-api-implementation) - + [Attachment API implementation](#attachment-api-implementation) -- [Usage scenarios and examples](#usage-scenarios-and-examples) - + [Standard usage of the KVStore class](#standard-usage-of-the-kvstore-class) - + [Standard usage of the Global Key Value interface](#standard-usage-of-the-global-key-value-interface) -- [Other information](#other-information) - + [Open issues](#open-issues) - - -### Revision history - -| Revision | Date | Authors | Mbed OS version | Comments | -|---------- |---------------- |-------------------------------------------------------- |----------------- |------------------ | -| 1.0 | 26 September 2018 | David Saada ([@davidsaada](https://github.com/davidsaada/)) | 5.11+ | Initial revision | - -# Introduction - -### Overview and background - -KVStore is an interface class whose purpose is to define APIs for a Key Value Store like storage over a block device. - -### Requirements and assumptions - -# System architecture and high-level design - -## Design basics - -KVStore defines a key value store like API set using this interface class. Classes implementing this interface store pairs of keys and values, where the keys are represented as character strings and the values are represented as binary blobs. Core APIs here are *get* and *set*, providing read and write access by key to the value in a single call. *remove* completes the set of core APIs. This simplifies the interface for the cases we need an actual key value store (like configurations). - -APIs also support an "incremental set" mode, allowing the implementing class to aggregate chunks of data for the set operation. This is for when the case the caller needs to generate large portions of data but doesn't wish to allocate large buffers for a single set operation. Note that *get* API doesn't have or require this functionality. Instead, it has an offset parameter (defaulting to 0) allowing the calling layer to extract portions of the data. - -Interface also includes iteration APIs, to let you iterate over all available keys, given a prefix. - -As some of the implementations use files as keys, key names must comply to file naming rules, meaning that characters like * , / etc. are not allowed in key names. - -### Derived implementations - -![KVStore Classes](./KVStore_classes.jpg) - -KVStore has a few derived implementations: - -- [TDBStore](../TDBStore/TDBStore_design.md) is the default solution because it gives the best performance, flash wear leveling and lowest overhead for a limited number of keys. -- [FileSystemStore](../FileSystemStore/FileSystemStore_design.md) is the preferred solution if you already have a file system and don't wish to have an additional one, or if specific POSIX features (such as file seeking) are required. It's also preferred if you don't have a limitation on the number of keys. -- [SecureStore](../SecureStore/SecureStore_design.md) adds security features such as encryption, rollback protection and authentication. It uses one of the other KVStore solutions as the underlying storage type. - -## Global Key Value interface - -A parallel key-value API is provided as global C-style functions (for all functions, except for the incremental set ones). This API performs a limited type of mapping of partition or mount point names present in the keys. For each of the APIs defined in KVStore, the global version extracts a partition prefix from the key name. The prefix must be in the form "/partition/key-name". Then a lookup is performed to map the partition name to a concrete KVStore instance, and the API call is routed to that instance. The routed key name has the partition prefix stripped, leaving only "key-name". - -In the case of iteration APIs, the prefix must include the partition (in the form of "/partition/prefix"). - -# Detailed design - -## KVStore class design - -As an interface class, KVStore has no implementation, just a class header. - -### KVStore class header - -```C++ -class KVStore { - - enum create_flags { - WRITE_ONCE_FLAG = (1 << 0), - REQUIRE_CONFIDENTIALITY_FLAG = (1 << 1), - REQUIRE_REPLAY_PROTECTION_FLAG = (1 << 3), - }; - - static const uint32_t MAX_KEY_LENGTH = 128; - - typedef struct _opaque_set_handle *set_handle_t; - - typedef struct _opaque_key_iterator *iterator_t; - - typedef struct info { - size_t size; - uint32_t flags; - } info_t; - - // Initialization and reset - virtual int init(); - virtual int deinit(); - virtual int reset(); - - // Core API - virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags); - virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0); - virtual int get_info(const char *key, info_t *info); - virtual int remove(const char *key); - - // Incremental set API - virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags); - virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size); - virtual int set_finalize(set_handle_t handle); - - // Key iterator - virtual int iterator_open(iterator_t *it, const char *prefix = NULL); - virtual int iterator_next(iterator_t it, char *key, size_t key_size); - virtual int iterator_close(iterator_t it); -} -``` - -## Global Key Value interface design - -As mentioned above, each KVStore API has a parallel C-style API, used globally with a partition name preceding the key name. - -### Global Key Value APIs - -```C -enum kv_create_flags { - KV_WRITE_ONCE_FLAG = (1 << 0), - KV_REQUIRE_CONFIDENTIALITY_FLAG = (1 << 1), - KV_REQUIRE_REPLAY_PROTECTION_FLAG = (1 << 3), -}; - -static const uint32_t KV_MAX_KEY_LENGTH = 128; -typedef struct _opaque_set_handle *kv_set_handle_t; -typedef struct _opaque_key_iterator *kv_key_iterator_handle_t; - -typedef struct info { - size_t size; - uint32_t flags; -} kv_info_t; - -// Core API -int kv_set(const char *full_name_key, const void *buffer, size_t size, uint32_t create_flags); -int kv_get(const char *full_name_key, void *buffer, size_t buffer_size, size_t *actual_size); -int kv_get_info(const char *full_name_key, kv_info_t *info); -int kv_remove(const char *full_name_key); - -// Key iterator -int kv_iterator_open(kv_key_iterator_handle_t *it, const char *full_prefix = nullptr); -int kv_iterator_next(kv_key_iterator_handle_t it, char *key, size_t key_size); -int kv_iterator_close(kv_key_iterator_handle_t it); -``` - -## Mapping APIs - -To use the global C style APIs, you need APIs to map the partition name to the instance of the implementing KVStore class, typically called once at initialization time. So, for example a `"/tdbstore/key1"` name means that you wish to access `"key1"` key in a TDBStore instance. This means that you need to attach `"tdbstore"` string to the TDBStore instance at initialization time. - -These APIs are part of a different header file ("kv_map.h") because they serve the integration code and not the KVStore user code: - -```C -// Attach and detach -int kv_init(); -int kv_attach(const char *partition_name, KVStore *kv_instance); -int kv_detach(const char *partition_name); - -// Full name lookup and then break it into KVStore instance and key -int kv_lookup(const char *full_name, KVStore& *kv_instance, char *key); -``` - -## Implementation - -Below is the implementation of the Global Key Value interface and of the attachment APIs. KVStore class has no implemetation because it's an interface class: - -### Important data structures - -```C -// incremental set handle -typedef struct { - KVStore *kvstore_intance; - KVStore::set_handle_t *set_handle; -} kv_inc_set_handle_t; - -// iterator handle -typedef struct { - KVStore *kvstore_intance; - KVStore::iterator_t *iterator_handle; -} kv_key_iterator_handle_t; - -const int MAX_ATTACHED_KVS 16 - -typedef struct { - char *partition_name; - KVStore *kvstore_instance; -} kv_map_entry_t; - -// Attachment table -kv_map_entry_t kv_map_table[MAX_ATTACHED_KVS]; -int kv_num_attached_kvs; -``` - -### Global Key Value API implementation - -**kv_set function** - -Header: - -`int kv_set(const char *full_name_key, const void *buffer, size_t size, uint32_t create_flags);` - -Pseudo code: - -- Using `kv_lookup`, break `full_name_key` into `key` and `kvs_instance`. -- Call `kvs_instance` `set` method with `key` and the rest of the arguments. - -**kv_get function** - -Header: - -`int kv_get(const char *full_name_key, void *buffer, size_t buffer_size, size_t *actual_size);` - -Pseudo code: - -- Using `kv_lookup`, break `full_name_key` into `key` and `kvs_instance`. -- Call `kvs_instance` `get` method with `key` and the rest of the arguments. - -**kv_get_info function** - -Header: - -`int kv_get_info(const char *full_name_key, kv_info_t *info);` - -Pseudo code: - -- Using `kv_lookup`, break `full_name_key` into `key` and `kvs_instance`. -- Call `kvs_instance` `get_info` method with `key` and the rest of the arguments. - -**kv_remove function** - -Header: - -`int kv_remove(const char *full_name_key);` - -Pseudo code: - -- Using `kv_lookup`, break `full_name_key` into `key` and `kvs_instance`. -- Call `kvs_instance` `remove` method with `key` and the rest of the arguments. - -**kv_set_start function** - -Header: - -`int kv_set_start(kv_set_handle_t *handle, const char *full_name_key, size_t final_data_size);` - -Pseudo code: - -- Allocate an `kv_inc_set_handle_t` structure into `handle`. -- Using `kv_lookup`, break `full_name_key` into allocated `key` and `kvs_instance` (in `handle`). -- Call `kvs_instance` `set_start` method with `key` and the rest of the arguments. - -**kv_set_add_data function** - -Header: - -`int kv_set_add_data(kv_set_handle_t handle, const void *value_data, size_t data_size);` - -Pseudo code: - -- Extract `kvs_instance` and `set_handle` from `handle`. -- Call `kvs_instance` `set_add_data` method with `set_handle` and the rest of the arguments. - -**kv_set_finalize function** - -Header: - -`int kv_set_finalize(kv_set_handle_t handle);` - -Pseudo code: - -- Extract `kvs_instance` and `set_handle` from `handle`. -- Call `kvs_instance` `set_finalize` method with `set_handle`. -- Free `key` and `handle`. - -**kv_iterator_open function** - -Header: - -`int kv_iterator_open(kv_key_iterator_handle_t *it, const char *full_prefix = nullptr);` - -Pseudo code: - -- Allocate a `kv_key_iterator_handle_t` structure into `it`. -- Using `kv_lookup`, break `full_name_key` into allocated `prefix` and `kvs_instance` (in `handle`). -- Call `kvs_instance` `iterator_open` method with `iterator_handle`, `prefix` and the rest of the arguments. - -**kv_iterator_next function** - -Header: - -`int kv_iterator_next(kv_key_iterator_handle_t it, char *key, size_t key_size);` - -Pseudo code: - -- Extract `kvs_instance` and `iterator_handle` from `handle`. -- Call `kvs_instance` `iterator_next` method with `iterator_handle` and the rest of the arguments. - -**kv_iterator_close function** - -Header: - -`int kv_iterator_close(kv_key_iterator_handle_t it);` - -Pseudo code: - -- Extract `kvs_instance` and `iterator_handle` from `handle`. -- Call `kvs_instance` `set_finalize` method with `iterator_handle`. -- Free `prefix` and `handle`. - -### Attachment API implementation - -**kv_init function** - -Header: - -`int kv_init();` - -Pseudo code: - -- Set `kv_num_attached_kvs` to 0. - -**kv_attach function** - -Header: - -`int kv_attach(const char *partition_name, KVStore *kv_instance);` - -Pseudo code: - -- Duplicate `partition_name` and `kv_instance` to last entry in `kv_map_table`. -- Increment `kv_num_attached_kvs`. - -**kv_detach function** - -Header: - -`int kv_detach(const char *partition_name);` - -Pseudo code: - -- Look for entry with `partition_name` in `kv_map_table`. -- Deallocate `partition_name` in this entry. -- Copy all preceding entries back one position. -- Decrement `kv_num_attached_kvs`. - -**kv_lookup function** - -Header: - -`int kv_lookup(const char *full_name, KVStore& *kv_instance, char *key);` - -Pseudo code: - -- Break `full_name` string to `partition_name` and `key`. -- Look for entry with `partition_name` in `kv_map_table`. -- Extract `kv_instance` from table entry. - -# Usage scenarios and examples - -### Standard use of the KVStore class - -The following example code shows standard use of the KVStore, using the TDBStore class: - -**Standard usage example - with class APIs** - -```C++ -// Underlying block device. Here, SPI Flash is fully used. -// One can use SlicingBlockDevice if we want a partition. -SPIFBlockDevice bd(PTE2, PTE4, PTE1, PTE5); - -// Instantiate tdbstore with our block device and a maximum of 64 keys -kvstore = new TDBStore(64, &bd); - -int res; - -// Initialize storage -res = kvstore->init(); - -const char *val1 = "Value of key 1"; -const char *val2 = "Updated value of key 1"; -// Add "Key1" -res = kvstore->set("Key1", val1, sizeof(val1), 0); -// Update value of "Key1" -res = kvstore->set("Key1", val2, sizeof(val2), 0); - -uint_8 value[32]; -size_t actual_size; -// Get value of "Key1". Value should return the updated value. -res = kvstore->get("Key1", value, sizeof(value), &actual_size); - -// Remove "Key1" -res = kvstore->remove("Key1"); - -// Incremental write, if need to generate large data with a small buffer -const int data_size = 1024; -char buf[8]; - -KVSTore::set_handle_t handle; -res = kvstore->set_start(&handle, "Key2", data_size, 0); -for (int i = 0; i < data_size / sizeof(buf); i++) { - memset(buf, i, sizeof(buf)); - res = kvstore->set_add_data(handle, buf, sizeof(buf)); -} -res = kvstore->set_finalize(handle); - -// Iterate over all keys starting with "Key" -res = 0; -KVSTore::iterator_t it; -kvstore->iterator_open(&it, "Key*"); -char key[KVSTore::KV_MAX_KEY_LENGTH]; -while (!res) { - res = kvstore->iterator_next(&it, key, sizeof(key)); -} - -// Deinitialize TDBStore -res = kvstore->deinit(); -``` - -### Standard usage of the Global Key Value interface - -The following example code shows how to use the previous example with the global key value interface. Here, `tdtbstore` is mapped to `"/tdbstore/"`. - -**Standard usage example - with global C-style APIs** - -This example assumes this code exists somewhere and is called during initialization: - -```C -// Assume TDBtore is already instantiated and initialized -extern TDBStore tdbstore; - -int res; - -// Attachment code. Should be called at initialization -res = kv_init(); -res = kv_attach("tdbstore", &tdbstore); - -``` - -This example shows how to access KVStore using C global APIs: - -```C -const char *val1 = "Value of key 1"; -const char *val2 = "Updated value of key 1"; - -// Add "Key1", now with full name, including "/tdbstore/" prefix. -res = kv_set("/tdbstore/Key1", val1, sizeof(val1), 0); -// Update value of "Key1" -res = kv_set("/tdbstore/Key1", val2, sizeof(val2), 0); - -uint_8 value[32]; -size_t actual_size; -// Get value of "Key1". Value should return the updated value. -res = kv_get("/tdbstore/Key1", value, sizeof(value), &actual_size); - -// Remove "Key1" -res = kv_remove("/tdbstore/Key1"); - -// Incremental write, if need to generate large data with a small buffer -const int data_size = 1024; -char buf[8]; - -kv_set_handle_t handle; -res = kv_set_start(&handle, "/tdbstore/Key2", data_size, 0); -for (int i = 0; i < data_size / sizeof(buf); i++) { - memset(buf, i, sizeof(buf)); - res = kv_set_add_data(handle, buf, sizeof(buf)); -} -res = kv_set_finalize(handle); - -// Iterate over all keys starting with "Key" in tdbstore -res = 0; -kv_key_iterator_handle_t it; -kv_iterator_open(&it, "/tdbstore/Key"); -char key[KV_MAX_KEY_LENGTH]; -while (!res) { - res = kv_iterator_next(&it, key, sizeof(key)); -} -res = kv_iterator_close(&it); -``` diff --git a/docs/design-documents/features/storage/README.md b/docs/design-documents/features/storage/README.md deleted file mode 100644 index 4cb03d7..0000000 --- a/docs/design-documents/features/storage/README.md +++ /dev/null @@ -1 +0,0 @@ -# Design Documents - Storage diff --git a/docs/design-documents/features/storage/SecureStore/SecureStore_class_hierarchy.jpg b/docs/design-documents/features/storage/SecureStore/SecureStore_class_hierarchy.jpg deleted file mode 100644 index 19047d5..0000000 --- a/docs/design-documents/features/storage/SecureStore/SecureStore_class_hierarchy.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/SecureStore/SecureStore_class_hierarchy.xml b/docs/design-documents/features/storage/SecureStore/SecureStore_class_hierarchy.xml deleted file mode 100644 index c19c4de..0000000 --- a/docs/design-documents/features/storage/SecureStore/SecureStore_class_hierarchy.xml +++ /dev/null @@ -1 +0,0 @@ -7VhZb+IwEP41eWwVHKD0kaOHVK1UCe31aJIhserEWWMo7K/fcTK5QVBp6bISPED8zXg8/jyHieNN4+2T5mn0RQUgHeYGW8ebOYz1BgOGPxbZ5cjdqJ8DoRYBKVXAXPwGAl1C1yKAVUPRKCWNSJugr5IEfNPAuNbqvam2VLK5aspD6ABzn8su+l0EJsrR0cCt8GcQYVSs3HNJsuD+W6jVOqH1HOYts08ujnlhi/RXEQ/Uew3yHhxvqpUy+VO8nYK03Ba05fMeD0hLvzUk5qQJ5MeGyzUULg8lzp2k1j2zI0qGv9bWp0nMdSgSxxuj1E23+I1gtjOL3xiV5rJ+TWZga264FCHN89E50JVNfArpN1t5UQBz8Nca5kZpKGS4lUVbH7G0jUXaul8EVrFS7/Cigdjs3XBE52wdZ3ZThwygF5mNAmUNc2wD2giMsHHOwyxjakKszCQs7SyFWkuZBcRS4EF5k6VKDOVHj9H4kcdC2sx6BrkBa9X6aWJplcq160FAcWF9gG0NoqB4AhWD0TtUISnzKDAogVmfxu9VOvQKLKqlwogwThkYlqarKMQHCsQDQdnrBOXXFXQYhSQY21zHkUoBWZwgUucq4KsIAiLlFH4gaJSFLjv13Q9ppxokN2LTrBz7tk/mXpXAZUumvbsW020GV2qtfaBZ9VRuGRq4RwwZTFEwHUPZaZR7PO2A2CVXjZdvRyvGQl9ryNlrSCcgP7WG9P/zEL0G5Nmb2ugz43GwJx4PN7SFVP5bu4Wh/DEjcOZ2m12NKmRI735YtdtBMfx5jMa8OzST55/0w0NV46P9sD9sGfLO1w/vL7nYzGAjfHiB3bUjXlhH9O4/sQIVfznPdauuqk695ri3fVYWoVfQAv3GqM21TitG95dzOS+L04cv56OWob92Ocdh9bogV6/eyXgPfwA= \ No newline at end of file diff --git a/docs/design-documents/features/storage/SecureStore/SecureStore_design.md b/docs/design-documents/features/storage/SecureStore/SecureStore_design.md deleted file mode 100644 index 451a112..0000000 --- a/docs/design-documents/features/storage/SecureStore/SecureStore_design.md +++ /dev/null @@ -1,472 +0,0 @@ -# SecureStore in Mbed OS - -- [SecureStore in Mbed OS](#securestore-in-mbed-os) - + [Revision history](#revision-history) -- [Introduction](#introduction) - + [Overview and background](#overview-and-background) - + [Requirements and assumptions](#requirements-and-assumptions) -- [System architecture and high-level design](#system-architecture-and-high-level-design) - * [Design basics](#design-basics) - + [Data layout](#data-layout) - + [Basic implementation concepts](#basic-implementation-concepts) -- [Detailed design](#detailed-design) - + [Class header](#class-header) - + [Important data structures](#important-data-structures) - + [Initialization and reset](#initialization-and-reset) - + [Core APIs](#core-apis) - + [Incremental set APIs](#incremental-set-apis) - + [Key iterator APIs](#key-iterator-apis) -- [Usage scenarios and examples](#usage-scenarios-and-examples) - + [Standard usage of the class](#standard-usage-of-the-class) -- [Other information](#other-information) - + [Open issues](#open-issues) - - -### Revision history - -| Revision | Date | Authors | Mbed OS version | Comments | -|---------- |---------------- |-------------------------------------------------------- |----------------- |------------------ | -| 1.0 | 02 October 2018 | David Saada ([@davidsaada](https://github.com/davidsaada/)) | 5.11+ | Initial revision | - -# Introduction - -### Overview and background - -SecureStore is a [KVStore](../KVStore/KVStore_design.md) based storage solution, providing security features on the stored data, such as encryption, authentication, rollback protection and write once, over an underlying KVStore class. It references an additional KVStore class for storing the rollback protection keys. - -### Requirements and assumptions - -SecureStore assumes that the underlying KVStore instances are instantiated and initialized. - -# System architecture and high-level design - -## Design basics - -SecureStore is a storage class, derived from KVStore. It adds security features to the underlying key value store. - -As such, it offers all KVStore APIs, with additional security options (which can be selected using the creation flags at set). These include: - -- Encryption: Data is encrypted using the AES-CTR encryption method, with a randomly generated 8-byte IV. Key is derived from [Device Key](../../../../../../mbed-os/drivers/device_key/README.md), using the NIST SP 800-108 KDF in counter mode spec, where salt is the key trimmed to 32 bytes, with "ENC" as prefix. Flag here is called "require confidentiality flag". -- Rollback protection: (Requires authentication) CMAC is stored in a designated rollback protected storage (also of KVStore type) and compared to when reading the data under the same KVStore key. A missing or different key in the rollback protected storage results in an error. The flag here is called "Require replay protection flag". -- Write once: Key can only be stored once and can't be removed. The flag here is called "Write once flag". - -SecureStore maintains data integrity using a record CMAC. This 16-byte CMAC is calculated on all stored data (including key & metadata) and stored at the end of the record. When reading the record, SecureStore compares the calculated CMAC with the stored one. In the case of encryption, CMAC is calculated on the encrypted data. The key used for generating the CMAC is derived from [Device Key](../../../../../../mbed-os/drivers/device_key/README.md), where salt is the key trimmed to 32 bytes, with "AUTH" as prefix. - -![SecureStore Layers](./SecureStore_layers.jpg) - -### Data layout - -When storing the data, it is stored with a preceding metadata header. Metadata includes flags and security related parameters, such as IV. The CMAC, calculated for authentication, is stored at the end of the data as it is calculated on the fly, so it can't be stored with the metadata. - -![SecureStore Record](./SecureStore_record.jpg) - -Fields are: - -- Metadata size: Size of metadata header. -- Revision: SecureStore revision (currently 1). -- Data size: Size of user data. -- Flags: User flags. -- IV: Random generated IV. -- Pad: Pad data to a multiple of 16 bytes (due to encryption). -- CMAC: CMAC calculated on key, metadata and data. - -### Basic implementation concepts - -Because the code can't construct a single buffer to store all data (including metadata and possibly encrypted data) in one shot, setting the data occurs in chunks, using the incremental set APIs. Get uses the offset argument to extract metadata, data and CMAC separately. - -Rollback protection (RBP) keys are stored in the designated rollback protection storage, which is also of KVStore type. RBP keys are the same as the SecureStore keys. -This RBP storage is also used for storing the CMAC in write once case, as otherwise an attacker can delete this key from the underlying storage and modify this flag. - -# Detailed design - -![SecureStore Class Hierarchy](./SecureStore_class_hierarchy.jpg) - -Functionality, as defined by KVStore, includes the following: - -- Initialization and reset. -- Core actions: get, set and remove. -- Incremental set actions. -- Iterator actions. - -### Class header - -SecureStore has the following header: - -```C++ -class SecureStore : KVStore { - -public: - SecureStore(KVStore *underlying_kv, KVStore *rbp_kv); - virtual ~SecureStore(); - - // Initialization and formatting - virtual int init(); - virtual int deinit(); - virtual int reset(); - - // Core API - virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags); - virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0); - virtual int get_info(const char *key, info_t *info); - virtual int remove(const char *key); - - // Incremental set API - virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags); - virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size); - virtual int set_finalize(set_handle_t handle); - - // Key iterator - virtual int iterator_open(iterator_t *it, const char *prefix = NULL); - virtual int iterator_next(iterator_t it, char *key, size_t key_size); - virtual int iterator_close(iterator_t it); - -private: - Mutex _mutex; - KVStore *_underlying_kv; - KVStore *_rbp_kv; - void *_entropy; - uint8_t *_scratch_buf; -} -``` - -### Important data structures - -```C++ -// Record header -typedef struct { - uint16_t metadata_size; - uint16_t revision; - uint32_t data_size; - uint32_t create_flags; - uint8_t iv[8]; -} record_metadata_t; - -// incremental set handle -typedef struct { - record_metadata_t metadata; - bd_size_t offset; - char *key; - void *encrypt_handle; - void *auth_handle; - KVStore::set_handle_t underlying_handle; -} inc_set_handle_t; - -// iterator handle -typedef struct { - KVStore::iterator_t underlying_it; -} key_iterator_handle_t; -``` - -### Initialization and reset - -**init function** - -Header: - -`virtual int init();` - -Pseudo code: - -- if `_is_initialized`, return OK. -- Take `_mutex`. -- Initialize `_entropy` with TLS entropy APIs. -- Using `DeviceKey` APIs, get the device key. -- Allocate `_scratch_buf` as a 32 byte array. -- Set `_is_initialized` to true. -- Release `_mutex`. - -**deinit function** - -Header: - -`virtual int deinit();` - -Pseudo code: - -- if not `_is_initialized`, return OK. -- Take `_mutex`. -- Deinitialize `_entropy`. -- Deallocate `_scratch_buf`. -- Release `_mutex`. - -**reset function** - -Header: - -`virtual int reset();` - -Pseudo code: - -- Take `_mutex`. -- Call `_underlying_kv` `reset` API. -- Call `_rbp_kv` `reset` API. -- Release `_mutex`. - -### Core APIs - -**set function** - -Header: - -`virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags);` - -Pseudo code: - -- Call `set_start` with all fields and a local `set_handle_t` variable. -- Call `set_add_data` with `buffer` and `size`. -- Call `set_finalize`. -- Return OK. - -**get function** - -Header: - -`virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0);` - -Pseudo code: - -- if not `_is_initialized` return error. -- Take `_mutex`. -- Call `_underlying_kv` `get` API with `metadata` size into a `metadata` local structure. -- If failure: - - If rollback protection or write once flag set: - - Call `_rbp_kv` `get` API on a local `rbp_cmac` variable, key is `key`, size 16. - - If no error, return "RBP authentication" error. - - Return "Key not found error". -- Derive a key from device key and `key`. -- Allocate and initialize `auth_handle` CMAC calculation local handle with derived key. -- Using `auth_handle` handle, calculate CMAC on `key` and `metadata`. -- If encrypt flag set: - - Derive a key from device key and `key`. - - Allocate and initialize a local `enc_handle` AES-CTR local handle with derived key and `iv` field. -- Set `data_size` local variable to data size in metadata. -- Set `actual_size` to the minimum of `buffer_size` and `data_size`. -- Set `current_offset` to 0. -- While `data_size` > 0: - - If `current_offset` between `offset` and `actual_size`. - - Set `dest_buf` to `buffer` and `chunk_size` to `actual_size`. - - Else: - - Set `dest_buf` to `_scratch_buf` and `chunk_size` to `actual_size`. - - Call `_underlying_kv` `get` API with `dest_buf` and `chunk_size`. - - Calculate CMAC on `dest_buf`, using `_auth_handle` handle. - - If encrypt flag set, decrypt `dest_buf` (in place) using `_enc_handle` handle. - - Decrement `data_size` by `chunk_size`. -- Call `_underlying_kv` `get` API with on a local `read_cmac` variable, size 16. -- Generate CMAC on local `cmac` variable . -- Using `mbedtls_ssl_safer_memcmp` function, compare `read_cmac` with `cmac`. Return "data corrupt error" if no match. -- If rollback protection or write once flags set: - - Call `_rbp_kv` `get` API on a local `rbp_cmac` variable, key is `key`, size 16. - - If `rbp_cmac` doesn't match `cmac`, clear `buffer` and return "RBP authentication" error. -- Deinitialize and free `auth_handle` and `enc_handle`. -- Release `_mutex`. -- Return OK. - -**get_info function** - -Header: - -`virtual int get_info(const char *key, info_t *info);` - -Pseudo code: - -- If not `_is_initialized`, return error. -- Call `get` API with `key` and 0 in `buffer_size` parameter. -- If failed, return error code. -- Call `_underlying_kv` `get` API with `metadata` size and `key`. -- Fill fields in `info` according to `metadata`. -- Return OK. - -**remove function** - -Header: - -`virtual int remove(const char *key);` - -Pseudo code: - -- If not `_is_initialized`, return error. -- Take `_mutex`. -- Call `_underlying_kv` `get` API with `metadata` size and `key`. -- If not found, return "Not found" error. -- If write once flag set, return "Already exists" error. -- Call `_underlying_kv` `remove` API with `key`. -- If rollback protect flag set, call `_rbp_kv` `remove` API with `key` as key. -- Return OK. - -### Incremental set APIs - -**set_start function** - -Header: - -`virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags);` - -Pseudo code: -- Take `_mutex`. -- Allocate an `inc_set_handle_t` and assign in handle. -- If flags include write once flag: - - Call `_underlying_kv` `get_info` API. - - If key exists, return "already exists" error. - - Call `_rbp_kv` `get` API with `key` as key. If key exists, return "already exists" error. -- If encrypt flag set: - - Derive a key from device key and `key` as salt (trimmed to 32 bytes with "ENC" as prefix). - - Using TLS entropy function on `_entropy` handle, randomly generate `iv` field. - - Allocate and initialize `enc_handle` AES-CTR handle field with derived key and `iv` field. -- Fill all available fields in `metadata`. -- Derive a key from device key and `key` as salt (trimmed to 32 bytes with "AUTH" as prefix). -- Allocate and initialize `auth_handle` CMAC calculation handle field with derived key. -- Using `auth_handle` handle, calculate CMAC on `key` and `metadata`. -- Call `_underlying_kv` `set_start` API. -- Call `_underlying_kv` `set_add_data` API with `metadata` field. -- Return OK. - -**set_add_data function** - -Header: - -`virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size);` - -Pseudo code: - -- If `offset` + `data_size` > data size in handle, return error. -- If flags include encryption: - - Iterate over `value_data` field in chunks of `_scratch_buf` size. - - Using `enc_handle` handle field, encrypt chunk into `_scratch_buf`. - - Using `auth_handle` handle field, update CMAC of `_scratch_buf`. - - Call `_underlying_kv` `set_add_data` API with `_scratch_buf`. -- Else: - - Using `auth_handle` handle field, update CMAC of `value_data`. - - Call `_underlying_kv` `set_add_data` API with `value_data`. -- Update `offset` field in handle. -- Return OK. - -**set_finalize function** - -Header: - -`virtual int set_finalize(set_handle_t handle);` - -Pseudo code: - -- Initialize a local `cmac` 16-byte array to 0. -- If authentication flag set, using `auth_handle` handle field, generate `cmac`. -- Call `_underlying_kv` `set_add_data` API with `cmac`. -- Call `_underlying_kv` `set_finalize`. -- If rollback protect or write once flags set, call `_rbp_kv` `set` API with `key` as key and `cmac` as data. -- Deinitialize and free `auth_handle` and `enc_handle`. -- Free `handle`. -- Release `_mutex`. -- Return OK. - -### Key iterator APIs - -**iterator_open function** - -Header: - -`virtual int iterator_open(iterator_t *it, const char *prefix = NULL);` - -Pseudo code: - -- Allocate a `key_iterator_handle_t` structure into `it`. -- Take `_mutex`. -- Call `_underlying_kv` `iterator_open` with `underlying_it` field. -- Release `_mutex`. -- Return OK. - -**iterator_next function** - -Header: - -`virtual int iterator_next(iterator_t it, char *key, size_t key_size);` - -Pseudo code: - -- Take `_mutex`. -- Call `_underlying_kv` `iterator_next` with `underlying_it` field. -- Release `_mutex`. -- Return OK. - -**iterator_close function** - -Header: - -`virtual int iterator_close(iterator_t it);` - -Pseudo code: - -- Take `_mutex`. -- Call `_underlying_kv` `iterator_close` with `underlying_it` field. -- Release `_mutex`. -- Deallocate `it`. -- Return OK. - -# Usage scenarios and examples - -### Standard use of the class - -The following example code shows standard use of the SecureStore class: - -**Standard usage example** - -```C++ -// Underlying key value store - here TDBStore (should be instantiated and initialized) -extern TDBStore tdbstore; - -// Rollback protect store - also of TDBStore type (should be instantiated and initialized) -extern TDBStore rbp_tdbstore; - -// Instantiate SecureStore with tdbstore as underlying key value store and rbp_tdbstore as RBP storage -SecureStore secure_store(&tdbstore, &rbp_tdbstore); - -int res; - -// Initialize secure_store -res = secure_store.init(); - -const char *val1 = "Value of key 1"; -const char *val2 = "Updated value of key 1"; -// Add "Key1" with encryption flag -res = secure_store.set("Key1", val1, sizeof(val1), KVSTore::REQUIRE_CONFIDENTIALITY_FLAG); -// Update value of "Key1" (flags must be the same per key) -res = secure_store.set("Key1", val2, sizeof(val2), KVSTore::REQUIRE_CONFIDENTIALITY_FLAG); - -uint_8 value[32]; -size_t actual_size; -// Get value of "Key1". Value should return the updated value. -res = secure_store.get("Key1", value, sizeof(value), &actual_size); - -// Remove "Key1" -res = secure_store.remove("Key1"); - -// Incremental write, if need to generate large data with a small buffer -const int data_size = 1024; -char buf[8]; - -KVSTore::set_handle_t handle; -res = secure_store.set_start(&handle, "Key2", data_size, 0); -for (int i = 0; i < data_size / sizeof(buf); i++) { - memset(buf, i, sizeof(buf)); - res = secure_store.set_add_data(handle, buf, sizeof(buf)); -} -res = secure_store.set_finalize(handle); - -// Iterate over all keys starting with "Key" -res = 0; -KVSTore::iterator_t it; -secure_store.iterator_open(&it, "Key*"); -char key[KVSTore::KV_MAX_KEY_LENGTH]; -while (!res) { - res = secure_store.iterator_next(&it, key, sizeof(key)e); -} -res = secure_store.iterator_close(&it); - -// Deinitialize SecureStore -res = secure_store.deinit(); -``` - -# Other information - -### Open issues - -- Need to figure a way to prevent mutex abuse in incremental set APIs. diff --git a/docs/design-documents/features/storage/SecureStore/SecureStore_layers.jpg b/docs/design-documents/features/storage/SecureStore/SecureStore_layers.jpg deleted file mode 100644 index 75b2c80..0000000 --- a/docs/design-documents/features/storage/SecureStore/SecureStore_layers.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/SecureStore/SecureStore_layers.xml b/docs/design-documents/features/storage/SecureStore/SecureStore_layers.xml deleted file mode 100644 index e4789c9..0000000 --- a/docs/design-documents/features/storage/SecureStore/SecureStore_layers.xml +++ /dev/null @@ -1 +0,0 @@ -7VlZc9owEP41zLQP6WD5iPMYkh4znc50Ss9HIS+2JsKiQlz99ZWwZLAtBxoMpUdeYq3O3e/bQ6jn301WrwWeZu94AqyH+smq59/3EPLCEKl/WrIuJNdxUAhSQRMzaCsY0h9ghH0jndMEZpWBknMm6bQqJDzPgciKDAvBl9VhY86qu05xCg3BkGDWlH6hicwKaRz2t/I3QNPM7uz1Tc8Ik4dU8Hlu9ushf7z5K7on2K5lxs8ynPDljsh/2fPvBOey+Jqs7oBp21qzFfNetfSW5xaQy0MmRNfFjAVmc7BHjpiaO0joQh9Qro1Rou9zfarBmOfyaraB7FYN8ILpatupvlLzn9nBx68ysoKhlSiNRq2jgMwFDCUXUHaJ+mDXAkpWnLch3tjCSlFFIbRBG7Q9+6p7mVEJwykmunepvEPJMjlhquVptSljd5xxsZmruYEIUfKZFPwBdnqSaBSFUbnfLqIG5AUICasdkUH4NfAJSLFWQ2xvbNi2LulatJdbbgeWwdkOry3fsXGntFx6Syn1YVjVQsl+3Eoxyw7r/xZ/71EqPJVNryiD4XomYVLhxkFM6BDzBEM8dmIekRhG424wD4Ia5tdNzL3gZJjf7MW8o4jw8X7wD6CJbg5A03egWSamo+C08eKCXPh34T2OCbgj9igOg7B/Gu8t64WzeK/ntcJ9iUXBgHHycA8LSi4+3ychxEngYk+MRn7UUb5vsOessd9DR7PH74Q9j65SkuHt51/PH26edKhj4wjHKf3sNl8/36PNwS5zMt84SyZt1MIu33DVwh1lUt/hHDUjQp7c6jusauU8B80pPMs2VvWqFlRmEOuv2tovQtv8ZvpabQVJ7e4rsUjBjkGB23w75gkfiRwCGJZ0Ud3AZTGzw3tON8S2kcuvouOjmtFnfC4ImFm799o9C5Uw24UKpRsLbQAs1T4Q0z/rDv0/Xe4prp3p0lVcd5IuUXtt/Z89fwB76gnFyZ6T/biCXKV6zYazDE/1J5kLth4ITB50tN9nzK3li5ZUcZ3nqnl109UlB4XVGH0TNiyHHIYr/fU4ywWtfnfeOy0Xx9xlla1lFbgq200FsesaRoQZTTWeRCEISj7QyFGC2a3pmNAk0ds4mVJ1zC7Y4NXYEDX9yBWE6xXC08gQPh6E3SH1PPz4wBnTzwo9rekJIu6FqPlehRj9lPPXKKSvlfqZqfOc+Ne4fFn72Grf4fLlk2D3Ph9dSAL4pKyqMjPNUzXkKay5YH40yOCgTCs//NrvWL6jtOqKH6q5fXUtboHbp23/5U8= \ No newline at end of file diff --git a/docs/design-documents/features/storage/SecureStore/SecureStore_record.jpg b/docs/design-documents/features/storage/SecureStore/SecureStore_record.jpg deleted file mode 100644 index f2de3c7..0000000 --- a/docs/design-documents/features/storage/SecureStore/SecureStore_record.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/SecureStore/SecureStore_record.xml b/docs/design-documents/features/storage/SecureStore/SecureStore_record.xml deleted file mode 100644 index df94e41..0000000 --- a/docs/design-documents/features/storage/SecureStore/SecureStore_record.xml +++ /dev/null @@ -1 +0,0 @@ -7VhNj5swEP01SNvDStgESI6bZHfbQ6SqkdqzAw5YazA1zld/fe1gAiQQshEoomouMW/8Mbx5nrExrFm0f+coCRfMx9SApr83rLkBIbBtKP8UcsgQdzzKgIATX3cqgCX5gzVoanRDfJxWOgrGqCBJFfRYHGNPVDDEOdtVu60Zra6aoABfAEsP0Uv0F/FFmKFj2yzwr5gEYb4yMLVlhbyPgLNNrNczoLU+/jJzhPK5dP80RD7blSDr1bBmnDGRtaL9DFPFbU5bNu6twXrym+NY3DJgov3YIrrBucsOlWOnPtnKZqCaObTKgQUWyEcC5Qa5wKqms5mKg6bU+b1R7zRds1g8p8eAv8gOwEz2hfFiCn6O1C0ksRpfr7l/FFynMz7BL1e5qHuRynywwhY8SgirIJnSvAuJwMsEecq6k1tOYqGIqHwCp9HloGsdbDEXeF+CtAjeMYuw4AfZRVvHWgd6v4Jcn7tC/a6GwpLwcwzp/RacJi40Jxtadg0SHN0lwb418kZRkA5U30+jYasROu1ytPuSo9MoxxORP/CWpITF10j+l7MFsB+YLty70sW8pVp1X0T+54f+8gNs19+oJ/0BE9wlwG8/hyqW8bDFYoF2sTi9iaX5cHNDano0c247c8Dqjbq6QnzGiLw+JarpbTg9TLm8fGHRTk3BY/YkkFDF3Jo/T8yOEhRwKsy57gVxsIY3OJ50QVxzhVQJ5aZMY1/NNDlwfg28OQVlfjTIXHIsqgFLBWcfeMYo4xKJWYyVy4TSMwhREqg4ejJyWOJTFTEir/Uv2hAR31fL1Cqkur26OCZZoKKCU00q759JnQ462T+Tu+rUd+QPOt3b5uPOpiAvNZ/kfLZ4mQ31dACcYR8P7BuKXEfHA/lYfNk72kqfT63Xvw== \ No newline at end of file diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_areas.jpg b/docs/design-documents/features/storage/TDBStore/TDBStore_areas.jpg deleted file mode 100644 index 6d30463..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_areas.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_areas.xml b/docs/design-documents/features/storage/TDBStore/TDBStore_areas.xml deleted file mode 100644 index 97ba384..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_areas.xml +++ /dev/null @@ -1 +0,0 @@ -7Zhdk5owFIZ/DZedgURQL127u53p7M160esIR8hsIDRExf76JhA+zY52yth1p94Y3pOQk+c9AaOD12n5LEievPAImIPcqHTwVwchz/eR+tLKqVbmi1ktxIJGplMnbOgvMKJr1D2NoBh0lJwzSfOhGPIsg1AONCIEPw677TgbzpqTGM6ETUjYufqDRjKp1YXvdvo3oHHSzOy5JrIl4Vss+D4z8zkI76pPHU5Jcy/Tv0hIxI89CT86eC04l3UrLdfANNsGWz3u6Z1om7eATF4zIPDrEQfC9tCkXCUmTw2MajmgB7gOfjgmVMImJ6GOHpX9SktkytSVp5rnCZicDiAklD3JJPQMPAUpTqpL2dKsh5jiQb5J8thZgRamT9KzYWn6EeN+3N66I6AaBoIdiD+7DOQCAlLkdU3uaKmxTcHEHzGZuWdIsGdB0op/xcRWJAFTUzzsuFpSH07wc8+bwJei2tgr1cHz87ILqlZsvqu7bBthI0kWbfW6VwJIE1b5bcdDlFbP3cgjjxRXObSlkIK/wZozLpSS8Qx0mpSxkUQYjTN1GSq3QOkP2iWqng0rE0hpFOlprFthuFmmcD4YOo8tu8GbW6xHUzg/v5Xzq1DSA/w3vvcY/KfGL941vnXkhRSaE3JfIeQispv2wd4lwZWvEjwFw+Vlht9BJ+vdBbvllez8Cdg1tX+ZXacLy6Pi49L0ZjcsxcC7Fie+D3jzW9Yi+uy1qDjdsBbxZZxPAvRvgRrCNQQ/zVsfuSNrLE/d9rQ59Vs/sB1+Rta8QgHioJY8sufOSt5ygLJWPPrzM6W67A7wVaz3Lwl+/A0= \ No newline at end of file diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_class_hierarchy.jpg b/docs/design-documents/features/storage/TDBStore/TDBStore_class_hierarchy.jpg deleted file mode 100644 index e865ee8..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_class_hierarchy.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_class_hierarchy.xml b/docs/design-documents/features/storage/TDBStore/TDBStore_class_hierarchy.xml deleted file mode 100644 index 0bcd9d2..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_class_hierarchy.xml +++ /dev/null @@ -1 +0,0 @@ -7VhLb9swDP41Prbwu9mxbtoVGAYMSPc6KhZtC5UtT1ZSZ79+ki35WbcNsGU5NIfE+kiRFEV+TGJ5N3n9kaMy+8wwUMu1cW15a8t1nSDw5YdCDi1ytdJAygnWSj2wIb9Bg7ZGdwRDNVIUjFFByjEYs6KAWIwwxDl7GqsljI69liiFGbCJEZ2j3wkWWYuuArvH74GkmfHs2FqyRfFjytmu0P4s10uaVyvOkbGl9asMYfY0gLxby7vhjIn2Ka9vgKrcmrS1++4WpF3cHArxpg06jj2iOzAhh1TujUoVnjjolIS/diqmKEc8JYXlXUupXdbyXYLNyRR+IVjZyvyBTEAtLhAlqd4Xy+CA9zblU6o/G89bAzyso41gHIxAnmM7VZZYOcUyrmI3VWXcOMseMdk/e9pMX7KK2lUnWjIgo2hsGNQdmXP3wAWR5XXdJmHdpCnSKVlTSNQuJrUS2lRDQuQteVHCCqGbw3H1+g7lhKq2uge6B2VVxSlyqpQ638oh1ItF4XSlJlsYWA6CH6SK3uB6uip097q+Xj/1veAYLBv0wUpjSLdf2pnuS1A+6CpcqEhnVpFfK5hlFAp8rRpdrlgJMouRRIa5wqjKAOukzPIDeEYAr2ZnePpQn5QDRYLsx7aeO74294UR6aXLtHc1yfQ0gxXb8Rj0rmEfTwwF9iuGhOxPEDNDzW10Z3zbBbnnTBkRZfHjGvYkfmeN07LGrARPyhr+ORflp2/vY+z/j7HVKesxeKYel0fYVrHWdGhJ+V2TwLU9H2+DVMl08MMPpXYZmOXPI6edLO5mQAy76TQTcIk1jp2Afjgx5P27CRieM9lEuyQBDvh9Ep4L8fgnnYRXs+J03vDtecQnI6apZOMIo40JylmBHzJVzq3I6PoG0KTVmJLEs9FuGRcZS1mB6G2PTu9wetFbJgTLlaWaiAHHydVP62X6s4+iv5ZmrNGP4hElhi9e+YV9GX5wvdG1O3+JIYNxNXXro38jOK8YWmDI3pBRZElSwZEsKpf93xqtev/fkXf7Bw== \ No newline at end of file diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_design.md b/docs/design-documents/features/storage/TDBStore/TDBStore_design.md deleted file mode 100644 index 2f53b3a..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_design.md +++ /dev/null @@ -1,528 +0,0 @@ -# TDBStore in Mbed OS - -- [TDBStore in Mbed OS](#tdbstore-in-mbed-os) - + [Revision history](#revision-history) -- [Introduction](#introduction) - + [Overview and background](#overview-and-background) - + [Requirements and assumptions](#requirements-and-assumptions) -- [System architecture and high-level design](#system-architecture-and-high-level-design) - * [Design basics](#design-basics) - + [Sequential writes](#sequential-writes) - + [Memory layout and areas](#memory-layout-and-areas) - + [Garbage collection](#garbage-collection) - + [RAM Table](#ram-table) -- [Detailed design](#detailed-design) - + [Class header](#class-header) - + [Important data structures](#important-data-structures) - + [Initialization and reset](#initialization-and-reset) - + [Core APIs](#core-apis) - + [Incremental set APIs](#incremental-set-apis) - + [Key iterator APIs](#key-iterator-apis) -- [Usage scenarios and examples](#usage-scenarios-and-examples) - + [Standard usage of the class](#standard-usage-of-the-class) -- [Other information](#other-information) - + [Open issues](#open-issues) - - -### Revision history - -| Revision | Date | Authors | Mbed OS version | Comments | -|---------- |---------------- |-------------------------------------------------------- |----------------- |------------------ | -| 1.0 | 16 September 2018 | David Saada ([@davidsaada](https://github.com/davidsaada/)) | 5.11+ | Initial revision | - -# Introduction - -### Overview and background - -Tiny Database Storage (TDBStore) is a lightweight module that stores data on flash storage. It is part of of the [KVStore](../KVStore/KVStore_design.md) class family, meaning it supports the get/set interface. It is designed to optimize performance (speed of access), reduce wearing of the flash and minimize storage overhead. It is also resilient to power failures. - -### Requirements and assumptions - -TDBStore assumes the underlying block device is fully dedicated to it (starting offset 0). If you want to dedicate only a part of the device to TDBStore, use a sliced block device, typically with `SlicingBlockDevice`. - -In addition, this feature requires a flash-based block device, such as `FlashIAPBlockDevice` or `SpifBlockDevice`. It can work on top of block devices that don't need erasing before writes, such as `HeapBlockDevice` or `SDBlockDevice`, but requires a flash simulator layer for this purpose, such as the one `FlashSimBlockDevice` offers. - -# System architecture and high-level design - -## Design basics - -TDBStore includes the following design basics: -- Sequential writes: TDBStore performs all writes sequentially on the physical storage as records, superseding the previous ones for the same key. -- Memory layout - areas: TDBStore divides the physical storage equally into two areas - active and standby. All writes are made to the end of the active area's free space. When the active area is exhausted, a garbage collection process is invoked, copying only the up to date values of all keys to the standby area, and turning it active. -- RAM table: Indexes all keys in RAM, thus allowing fast access to their records in the physical storage. - -### Sequential writes - -All writes occur sequentially on the physical storage as records, superseding the previous ones for the same key. Each data record is written right after the last written one. If a key is updated, a new record with this key is written, overriding the previous value of this key. If a key is deleted, a new record with a "deleted" flag is added. - -Writes expect the storage to be erased. However, TDBStore takes the "erase as you go" approach, meaning that when it crosses a sector boundary, it checks whether the next sector is erased. If not, it erases the next sector. This saves time during initialization and garbage collection. - -### Memory layout and areas - -![TDBStore Areas](./TDBStore_areas.jpg) - -Each key is stored in a separate record on the active area. The first record in the area is the master record. Its main purpose is to hold an area version, protecting you against a case in which there are two valid areas. (This can happen in the extreme cases of power failures.) - -![TDBStore Record](./TDBStore_record.jpg) - -A 24-byte header precedes a record key and data. Fields are: - -- Magic: a constant value, for quick validity checking. -- Header size: size of header. -- Revision: TDBStore revision (currently 1). -- User flags: Flags received from user. Currently only write once is dealt with (others are ignored). -- Internal flags: Internal TDBStore flags (currently only includes deleted flag). -- Key size: size of key. -- Data size: size of data. -- CRC: a 32-bit CRC, calculated on header (except CRC), key and data. -- Programming size pad: padding to the storage programming size. - -### Garbage collection - -Garbage collection (GC) is the process of compacting the records stored in the active area to the standby one, by copying only the most recent values of all the keys (without the ones marked as deleted). Then, the standby area becomes the active one and the previously active area is erased (not fully, only its first sector). - -GC is invoked in the following cases: - -- When the active area is exhausted. -- During initialization, when a corruption is found while scanning the active area. In this case, GC is performed up to the record preceding the corruption. - -### Reserved space - -The active area includes a fixed and small reserved space. This space is used for a quick storage and extraction of a write-once data (such as the device key). Its size is 32 bytes, aligned up to the underlying block device. Once it is written, nothing can modify it. It is also copied between the areas during garbage collection process. - -### RAM table - -All keys are indexed in memory using a RAM table. Key names are represented by a 32-bit hash. The table includes the hash (and sorted by it) and the offset to the key record in the block device. This allows both fast searching in the table and a low memory footprint. To keep code simple, the same CRC function used for recored validation is used for hash calculation (because TLS hash calculation is too heavy). - -![TDBStore RAM Table](./TDBStore_ram_table.jpg) - -Key names may produce duplicate hash values. This is OK because the hash is only used for fast access to the key, and the key needs to be verified when accessing the storage. If the key doesn't match, move to the next duplicate in the table. - -# Detailed design - -TDBStore fully implements the KVStore interface over a block device. Due to the fact it may write to the block device in program units that don't have to match the underlying device program units, it should use a `BufferedBlockDevice` for that purpose. - -![TDBStore Class Hierarchy](./TDBStore_class_hierarchy.jpg) - -Functionality, as defined by KVStore, includes the following: - -- Initialization and reset. -- Core actions: get, set and remove. -- Incremental set actions. -- Iterator actions. - -### Class header - -TDBStore has the following header: - -```C++ -class TDBStore : KVStore { - -public: - TDBSTore(BlockDevice *bd = 0); - virtual ~TDBSTore(); - - // Initialization and reset - virtual int init(); - virtual int deinit(); - virtual int reset(); - - // Core API - virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags); - virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0); - virtual int get_info(const char *key, info_t *info); - virtual int remove(const char *key); - - // Incremental set API - virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags); - virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size); - virtual int set_finalize(set_handle_t handle); - - // Key iterator - virtual int iterator_open(iterator_t *it, const char *prefix = NULL); - virtual int iterator_next(iterator_t it, char *key, size_t key_size); - virtual int iterator_close(iterator_t it); - - // Reserved space APIs - virtual int reserved_space_set(void *data); - virtual int reserved_space_get(void *data); - -private: - Mutex _mutex; - void *_ram_table; - size_t *_max_keys; - size_t *_num_keys; - BlockDevice *_bd; - bd_addr_t _free_space_offset; - BufferedBlockDevice *_buff_bd; - bool _is_initialized; - int _active_area; - - // Important internal functions - - // find record offset in flash - int find_record(const char *key, uint32_t *hash, bd_size_t *bd_offset, size_t ram_table_ind); - - // garbage collection - int garbage_collection(const char *key, const void *buffer, size_t size, uint32_t create_flags); -} -``` - -### Important data structures - -```C++ -// RAM table entry -typedef struct { - uint32_t hash; - bd_size_t bd_offset; -} ram_table_entry_t; - -// Record header -typedef struct { - uint32_t magic; - uint16_t header_size; - uint16_t revision; - uint32_t user_flags; - uint16_t int_flags; - uint16_t key_size; - uint32_t data_size; - uint32_t crc; -} record_header_t; - -// incremental set handle -typedef struct { - record_header_t header; - bd_size_t bd_base_offset; - bd_size_t bd_curr_offset; - uint32_t ram_table_ind; - uint32_t hash; - bool new_key; -} inc_set_handle_t; - -// iterator handle -typedef struct { - size_t ram_table_ind; - char *prefix; -} key_iterator_handle_t; -``` - -### Initialization and reset - -**init function** - -Header: - -`virtual int init();` - -Pseudo code: - -- If `_is_initialized` return OK. -- Take `_mutex`. -- Set `_max_keys` to an initial value of 32. -- Allocate `_ram_table` as an array of `_max_keys`. -- Allocate `_buff_bd` with `_bd` as the underlying block device, and initialize it. -- Check validity of master records on both areas. -- If one is valid, set its area as `_active_area`. -- If both are valid, set the one area whose master record has the higher version as `_active_area`. Erase first sector of the other one. -- If none are valid, set area 0 as `_active_area`, and write master record with version 0. -- Traverse active area until reaching an erased sector. - - Read current record and check its validity (calculte CRC). - - If not valid, perform garbage collection and exit loop. - - Advance `_free_space_offset`. - - Call `find_record` function to calculate hash and find key. - - If not found, add new RAM table entry with current hash. - - Update position of key in RAM table. -- Set `_is_initialized` to true. -- Release `_mutex`. - -**deinit function** - -Header: - -`virtual int deinit();` - -Pseudo code: - -- If not `_is_initialized`, return OK. -- Take `_mutex`. -- Deinitialize `_buff_bd`, and free it. -- Free `_ram_table`. -- Set `_is_initialized` to false. -- Release `_mutex`. - -**reset function** - -Header: - -`virtual int reset();` - -Pseudo code: - -- Take `_mutex`. -- Erase first sector in both areas. -- Set `_active_area` to 0. -- Write a master record with version 0. -- Set `_free_space_offset` to end of master record. -- Set `_num_keys` to 0. -- Release `_mutex`. - -### Core APIs - -**set function** - -Header: - -`virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags);` - -Pseudo code: - -- if not `_is_initialized`, return "not initialized" error. -- Call `set_start` with all fields and a local `set_handle_t` variable. -- Call `set_add_data` with `buffer` and `size`. -- Call `set_finalize`. -- Return OK. - -**get function** - -Header: - -`virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0);` - -Pseudo code: - -- if not `_is_initialized`, return "not initialized" error. -- Take `_mutex`. -- Call `find_record` to find record in storage. -- If not found, return "not found" error. -- Read header, and calculate CRC on it. -- Update CRC with key (if offset is 0). -- Read data into user buffer, starting offset. Actual size is minimum of buffer size and remainder of data. -- If offset is 0, - - Update CRC with buffer. - - Compare calculate CRC with header CRC. Return "data corrupt" error if different. -- Release `_mutex`. -- Return OK. - -**get_info function** - -Header: - -`virtual int get_info(const char *key, info_t *info);` - -Pseudo code: - -- if not `_is_initialized`, return "not initialized" error. -- Take `_mutex`. -- Call `find_record` to find record in storage. -- If not found, return "not found" error. -- Read header. -- Copy relevant fields from header into structure. -- Release `_mutex`. -- Return OK. - -**remove function** - -Header: - -`virtual int remove(const char *key);` - -Pseudo code: - -- Call `set` function with `key`, delete flag set in flags and empty data - -### Incremental set APIs - -**set_start function** - -Header: - -`virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags);` - -Pseudo code: - -- Take `_mutex`. -- Check if final size fits in free space; if not, call `garbage_collection`. -- Call `find_record` to find record in storage and achieve `ram_table_ind` and `hash`. -- If found and `flags` field in header includes write once flag, return "write once" error. -- Set `new_key` field in handle to true if not found and delete key not set. -- Allocate an `inc_set_handle_t` structure into `handle`. -- Calculate hash on `key` and update in `handle`. -- Update `bd_base_offset` in handle to `_free_space_offset`. -- Update a `record_header_t` structure with all relevant values. -- Update all header fields in `handle`. -- Calculate CRC on header. -- Update `ram_table_ind` and `hash` in `handle`. -- Program key in position after header. -- Advance `_free_space_offset` and update in `bd_curr_offset` field in handle. -- Set `_free_space_offset`, and update in `bd_curr_offset` field in handle. -- Call `find_record` to calculate hash, and find record in storage (with null key and current hash). - -**set_add_data function** - -Header: - -`virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size);` - -Pseudo code: - -- Calculate CRC on `value_data` and update in handle. -- Program `value_data` from `bd_curr_offset`. -- Advance `bd_curr_offset`. - -**set_finalize function** - -Header: - -`virtual int set_finalize(set_handle_t handle);` - -Pseudo code: - -- Advance `_free_space_offset` to padded offset. -- Update a `record_header_t` structure with all relevant values. -- Program header at `bd_base_offset` from handle with pads. -- Call `sync` on buffered block device. -- If delete flag set: - - Remove entry in index `ram_table_ind` from RAM table. -- Else if `new_key` field is true: - - If `_num_keys` = `_max_keys`: - - Increase _max_keys by 1. - - Duplicate ram table to with new `_max_keys` entries. - - Add entry `ram_table_ind`. -- Update `bd_offset` and `hash` in `ram_table_ind` position of RAM table. -- Free `handle`. -- Release `_mutex`. - -### Key iterator APIs - -**iterator_open function** - -Header: - -`virtual int iterator_open(iterator_t *it, const char *prefix = NULL);` - -Pseudo code: - -- Take `_mutex`. -- Allocate a `key_iterator_handle_t` structure into `it`. -- Set `ram_table_ind` field in iterator to 0. -- Duplicate `prefix` into same field in iterator. -- Release `_mutex`. - -**iterator_next function** - -Header: - -`virtual int iterator_next(iterator_t it, char *key, size_t key_size);` - -Pseudo code: - -- Take `_mutex`. -- While `ram_table_ind` field in iterator smaller than `_num_keys`: - - Read key RAM table points to in `ram_table_ind` into a local variable. - - If name matches prefix: - - Advance `ram_table_ind` field in iterator. - - Copy name to `key`, and return OK. - - Advance `ram_table_ind` field in iterator. -- Return "not found" error. -- Release `_mutex`. - -**iterator_close function** - -Header: - -`virtual int iterator_close(iterator_t it);` - -Pseudo code: - -- Release `prefix` field in iterator and structure allocated at `it`. - -### Reserved space - -**reserved_space_set function** - -Header: - -`virtual int reserved_space_set(void *data);` - -Pseudo code: - -- Check if reserved space is not empty; if it is, return a "reserved space programmed error". -- Copy `data` contents to reserved space location. - -**reserved_space_get function** - -Header: - -`virtual int reserved_space_get(void *data);` - -Pseudo code: - -- Copy contents from reserved space location `data`. - -# Usage scenarios and examples - -### Standard use of the class - -The following example code shows standard use of the TDBStore class: - -**Standard usage example** - -```C++ -// Underlying block device. Here, SPI Flash is fully used. -// One can use SlicingBlockDevice if we want a partition. -SPIFBlockDevice bd(PTE2, PTE4, PTE1, PTE5); - -// Instantiate tdbstore with our block device -TDBStore tdbstore(&bd); - -int res; - -// Initialize tdbstore -res = tdbstore.init(); - -// Add "Key1" -const char *val1 = "Value of key 1"; -const char *val2 = "Updated value of key 1"; -res = tdbstore.set("Key1", val1, sizeof(val1), 0); -// Update value of "Key1" -res = tdbstore.set("Key1", val2, sizeof(val2), 0); - -uint_8 value[32]; -size_t actual_size; -// Get value of "Key1". Value should return the updated value. -res = tdbstore.get("Key1", value, sizeof(value), &actual_size); - -// Remove "Key1" -res = tdbstore.remove("Key1"); - -// Incremental write, if need to generate large data with a small buffer -const int data_size = 1024; -char buf[8]; - -KVSTore::set_handle_t handle; -res = tdbstore.set_start(&handle, "Key2", data_size, 0); -for (int i = 0; i < data_size / sizeof(buf); i++) { - memset(buf, i, sizeof(buf)); - res = tdbstore.set_add_data(handle, buf, sizeof(buf)); -} -res = tdbstore.set_finalize(handle); - -// Iterate over all keys starting with "Key" -res = 0; -KVSTore::iterator_t it; -tdbstore.iterator_open(&it, "Key*"); -char key[KVSTore::KV_MAX_KEY_LENGTH]; -while (!res) { - res = tdbstore.iterator_next(&it, key, sizeof(key)); -} -res = tdbstore.iterator_close(&it); - -// Deinitialize TDBStore -res = tdbstore.deinit(); -``` - -# Other information - -### Open issues - -- Need to figure a way to prevent mutex abuse in incremental set APIs. diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_ram_table.jpg b/docs/design-documents/features/storage/TDBStore/TDBStore_ram_table.jpg deleted file mode 100644 index f370dc2..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_ram_table.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_ram_table.xml b/docs/design-documents/features/storage/TDBStore/TDBStore_ram_table.xml deleted file mode 100644 index 5638d79..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_ram_table.xml +++ /dev/null @@ -1 +0,0 @@ -3Zlbb5swGIZ/DZedABdCL5P0JE3VpmbSuksHHLDq4Mw4DdmvnwEbAjgN3SAtrVTFvDa2ed7PB4wB5uv0jsFN9EADRAzbDFIDXBu2bTnOpfjJlH2hTDwphAwHslAlLPAfJEVTqlscoKRWkFNKON7URZ/GMfJ5TYOM0V292IqSeqsbGKKWsPAhaas/ccCjQvUcs9LvEQ4j1bJlypwl9J9DRrexbM+wwSr/K7LXUNUlyycRDOjuQAI3BpgzSnmRWqdzRDK2Cltx3+2R3LLfDMW8yw2OW9zxAskWqS7nHeN7BWMXYY4WG+hn1zthuAFmEV8TcWWJJEw2hQUrnCJR7azdB9mtF8Q4Sg8k2ac7RNeIs70oInMdFQgyfhSuXWWG7UktOjCiFKEMgLCsuoIgEpLDESYTDROXiCZmKyoe6RCO+3tLVcZFksfxVBSwwCatMkUqlL95LUslPE4fROEfcClqU3msWVp0d6nRiq4ouWGZwMzrLiWc0Wc0p4QyocQ0RlmvMSENCRIcxuLSF+Yhoc8y07AYGVOZscZBkDUz04VFHvlZEFyb/QSCV48DSxMI1kQXCH3EgXc0DkpDHmCSYbLNR+RTFug9a5hTh3RidL3jYAJ9MLw6zfAryjprjYKd05Gd0wM71+zKzu46e3wwmuWyeY5QdK2uOME44IFzxqL96WPRO2csgtM4bxkSuwKzgNCF4KdZ9F2zsWBpIr2cOvpe9d3ju7/2imXewyQ6X8C3EGpAv0L11C6g3F31He6TyzfMHiNmqtkdDMfUecNyNmKmmml5OKZvGvvfVqsE8ZFQtbyTw3+oxW7S4T2qGv2jpqqZAAaj2vnNCoydqmYKGIqqp1upGjhQHEyzM8Zsw0RgkmC/TkA8Jds/SVr5xa/s4otT0kFB6/ixtV3ikIVI7eCv9LwOgDgaIEpjiECOX+pN6ijJFr5TnB9zSTuaMzJw6jUkdMt8JG+qULfqAbZ+ClL1FE/cqie3rHzobi7q1sZ3d7F4h/ooLrr/6KJlNaNhOBs7HE6ftDHF/OkgXZnYj8EFtvqUXDP9yFR3HtMbnl+aDa8+ouk9nL47nU7fF5yy7DuQbV6I/6mfcxYJhuBnOXv/r2XYUa8YyvQBD9/FZfXRq4ia6ssiuPkL \ No newline at end of file diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_record.jpg b/docs/design-documents/features/storage/TDBStore/TDBStore_record.jpg deleted file mode 100644 index acb4a9a..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_record.jpg +++ /dev/null Binary files differ diff --git a/docs/design-documents/features/storage/TDBStore/TDBStore_record.xml b/docs/design-documents/features/storage/TDBStore/TDBStore_record.xml deleted file mode 100644 index fd8b2b4..0000000 --- a/docs/design-documents/features/storage/TDBStore/TDBStore_record.xml +++ /dev/null @@ -1 +0,0 @@ -7VnbrqIwFP0ak5mHM4GWiz4qnksyOcmJZjLPFSo0VuqUepuvnyJFQAE9RIdhoi+WtXvZrr26uys96Cx3rxytgnfmYdoDmrfrwXEPAN00gfyKkX2C2H0jAXxOPNUpA6bkN1agptA18XBU6CgYo4KsiqDLwhC7ooAhztm22G3OaHHVFfLxGTB1ET1HfxJPBAnaN7UMf8PED9KVdU1ZZshd+JytQ7VeD8D54ZOYlyidS/WPAuSxbQ6Czz3ocMZE0lruHExjblPaknEvFdaj3xyH4poBlp2M2CC6xqnLFpVjRx7ZyKYfN1NolgLvyCduisrZZyU9tUjsFZ/Wr3X8g0ZzFoqn6BDtoeyga6tdZjybgp8iZQtJrMTROt+/GF9rPS9btjAfKPw2cIg2jvnUpHkbEIGnK+TG1q3cHRILxJLKJ/04Oh8fFbIN5gLvcpCK1ytmSyz4XnZR1r6SjtpaeiqlbSZUQ0FBTqO2wpDaGv5x4kwesqEUUq6WgdZILW8YeZh3VC6H5HRbAYJuC/CouBoFmvdSoF6pwCORE7z5Vkfwfx0bu8XYGJdj8yOqTwR1sXmhyI86mkW6fuiANve82ejUmWCptY3kqKOK6XgmAldkIvteirEaKeY73t+vdnjopFwn0Gixnm12+xkjgf5ukfkv56mOn2ywxfuUrlWXs3WcOxPnoZZW1GJcUQfdTy2gkVo+OPNrL0RN85X2MRx3O5pWm9GEl69LlRVJy8SZ2mXidHg35q64aFaf0S1TZ8FWqWt2l3qkkKpw2q2mkLJ7zgkhUYBWcdNdc7ofceQusLjMTEZj8iSQICyUj08D7TbEAQsUiIP6GW+ghDfY129BXHXhH5dJV9VPZm39lAIT7DLuyf7FP8Gvrq8SdyrELqkWxbhFgrMFdhhlXCIhC3HsOaH0BEKU+HE4XRlA6RUcxYEjLqJDZVgSz4uXKRVKcZPdQAy6bRfEcNwy+aQ4KJED+Pw2ko/Z+7WDLfcSEz7/AQ== \ No newline at end of file diff --git a/docs/design-documents/tools/README.md b/docs/design-documents/tools/README.md deleted file mode 100644 index 9eeb66d..0000000 --- a/docs/design-documents/tools/README.md +++ /dev/null @@ -1 +0,0 @@ -# Design Documents - Tools diff --git a/storage/docs/BlockDevices/get_type_method.md b/storage/docs/BlockDevices/get_type_method.md new file mode 100644 index 0000000..add8238 --- /dev/null +++ b/storage/docs/BlockDevices/get_type_method.md @@ -0,0 +1,78 @@ +# Get type method addon to block devices class + +### Revision history +| Revision | Date | Authors | Mbed OS version | Comments | +|---------- |-----------------|-------------------------------------------------------------|------------------------|------------------| +| 1.0 | 04/12/2018 | Yossi Levy ([@yossi2le](https://github.com/yossi2le/)) | 5.11+ | Initial revision | + +## Introduction + +Most storage solutions use block devices, whether it is Filesystems, KVStore or any other solution +most of them will have some BlockDevice class underneath. However, sometimes a storage type or an application needs to know the physical +BlockDevice type in order to work smoothly or the BlockDevice type in order to decide what is +the best storage configuration to use. +To address this an add-on method of getting type is proposed for BlockDevice interface class. + +## The Motivation + +Below there is a list of some examples to explain the motivation and the need for the adding of get_type to BlockDevice interface. + +examples: +- TDBStore needs to know if there are flash characteristics for the block device and if there aren�t it should use + FlashSimBlockDevice to simulate a flash BlockDevice. +- When creating a file system you would prefer working with FAT on top of SD while LITTLEFS on top of any flash block device. + Those preference in favor of better performance. + +To summarize the above, it may be very useful when using block device to know the type of the instance and especially, but not only, +when using get_default_instace. Sometimes applications and tests would like to behave differently depending on the instance that has been created +or provided to them. + +In fact it might be worth to consider adding the get_type to any interface with get_default_instance at mbed-os. + +## Dive into details +we should add the following method to BlockDevice interface class. + +```virtual const char * get_type() const = 0;``` + +then every physical BlockDevice class which implements the interface should also implement this method and return a string +representing its type. Furthermore, a nonphysical BlockDevice like SlicingBlockDevice should return the underlying physical +BlockDevice type. + +### Physical BlockDevice: +``` +const char * HeapBlockDevice::get_type() const +{ + return "HEAP"; +} +``` + +### Logical BlockDevice: +``` +const char * SlicingBlockDevice::get_type() const +{ + if (_bd != NULL) { + return _bd->get_type(); + } + + return NULL; +} +``` + +### Open issue +The ChainingBlockDevice which chains different type of physical block devices into one block device is unable +to return the underneath physical as it contains two or more types. Therefore it will return CHAINING as its +identity and its left for the user to decide how the application will treat this information. + + +The below table describes physical BlockDevice and its tyep names + + +| BlockDevice class | String description | +|-----------------------------|--------------------| +| HeapBlockDevice | "HEAP" | +| SPIFBlockDevice | "SPIF" | +| QSPIFBlockDevice | "QSPIF" | +| SDBlockDevice | "SD" | +| FlashIAPBlockDevice | "FLASHIAP" | +| DataFlashBlockDevice | "DATAFLASH" | +| ChainingBlockDevice | "CHAINING" | diff --git a/storage/docs/Configuration/CONFIGURATION.md b/storage/docs/Configuration/CONFIGURATION.md new file mode 100644 index 0000000..8dcafa7 --- /dev/null +++ b/storage/docs/Configuration/CONFIGURATION.md @@ -0,0 +1,407 @@ +# Storage configuration + +This document describes the configurations of the mbed OS storage based on kv_store interface. + +## New storage design + +![New storage design](./NewStorageDesign.jpg) + +This document describes the configuration for the KVStore part (left side of the diagram above) and its underlying components, such as file systems and block devices. + +The configuration concept is based on a number of predefined topologies (configurations) allowing customers to select one of them or to define a topology. + +The last section of this document explains how to override the configuration option in order to enable storage configuration that none of the configurations above support. + +To use the default configurations, set the `storage_type` parameter to one of the configurations options available. + +The implementation of the configuration is composed of a set of `.json` files and a set of functions instantiating and initializating the required components. + +## Configuration structure + +``` +kvstore +│ +└───conf + │ mbed_lib.json + │ + ├───tdb_external + │ mbed_lib.json + │ + ├───tdb_external_no_rbp + │ mbed_lib.json + │ + ├───filesystem + │ mbed_lib.json + │ + ├───filesystem_no_rbp + │ mbed_lib.json + │ + └───tdb_internal + mbed_lib.json +``` + +The KVStore configuration file structure includes six configuration files. The topmost configuration file is used to set up the full configuration of the storage by defining a single parameter (`storage_type`) to one of the predefined configurations. The configuration files in the subfolders are used to implement the above top level configurations. + +You can find the configuration files `conf/`: + +* `conf/tdb_internal` - storage type `TDB_INTERNAL` configuration is intended to be used when all data will be stored in internal memory only. No need for additional security features. A single TDBStore object will be allocated in internal flash. +* `conf/tdb_external` - storage type `TDB_EXTERNAL` configuration is providing full security and intended to be used when data is stored in external flash. It allocates: SecureStore, TDBStore in external flash and TDBStore in internal flash (for rollback protection - RBP). +* `conf/tdb_external_no_rbp` - storage type `TDB_EXTERNAL_NO_RBP` configuration allows security but without rollback protection. Similar to `tdb_external` but without the TDBStore in internal memory. +* `conf/filesystem` - This configuration will allocate: SecureStore, FileSystemStore, filesystem, TDBStore in internal memory and the required block devices. The allocated file system will be selected according to the COMPONENT set in `targets.json`, (FATFS for SD card and LittleFS for SPIF); however, you can set this differently by overriding the respective parameter. Use this configuration if you need the file system with a POSIX API in addition to the set/get API. +* `conf/filesystem_no_rbp` - storage type `FILESYSTEM_NO_RBP` configuration allows security like FILESYSTEM configuration but without rollback protection. + +A standalone block device is allocated for each component in internal and external memory and SD cards as required for the configurations. The full size of the memory allocated for each block device is used by the respective component. + +## Configuration parameters + +The following is a list of all storage parameters available and their description. + +* `storage_type` - Used to select one of the predefined configurations. + * `TDB_INTERNAL`. + * `TDB_EXTERNAL`. + * `TDB_EXTERNAL_NO_RBP`. + * `FILESYSTEM`. + * `FILESYSTEM_NO_RBP`. + * `default` + If the `default` configuration is set, the system will choose the type of storage TDB_INTERNAL. +* `default_kv` - This is a string representing the path for the default KVStore instantiation. Applications can pass an empty path (only the key name) or pass the generated name for this parameter (`MBED_CONF_STORAGE_DEFAULT_KV`) as the path to use this configuration. +* `internal_size` - The size in bytes for the internal FlashIAP block device. This, together with the `internal_base_address`, adjusts exactly the size and location where the block device resides on memory. If not defined, the block device will try to get the maximum size available. +* `internal_base_address` - The address where the internal FlashIAP blockDevice starts. This helps to prevent collisions with other needs, such as firmware updates. If not defined, the start address will be set to the first sector after the application code ends in `TDB_internal`. In any external configurations with rollback protection support, it will be set to end of flash - `rbp_internal_size`. +* `rbp_number_of_entries` - Sets the number of entries allowed for rollback protection. The default is set to 64. This parameter controls the maxmium number of different keys that can be created with rollback protection flag. +* `rbp_internal_size` - Sets the size for the rollback protection TDBStore in the internal memory. The base address is calculated as flash ends address - size. +* `filesystem` - Options are FAT, LITTLE or default. If not set or set to default, the file system type will be selected according to the storage component selected for the board in the `targets.json` file: FAT for "components": ["SD"] and Littlefs for "components": ["SPIF"]. +* `blockdevice` - Options are default, SPIF, DATAFLASH, QSPIF or SD. If file system is set to default, this parameter is ignored. +* `external_size` - The size of the external block device in bytes. If not set, the maximum available size will be used. +* `external_base_address` - The start address of the external block device. If not set, 0 address will be used. +* `mount_point` - Mount point for the file system. This parameter will be ignored if the file system is set to default. +* `folder_path` - Path for the working directory where the FileSystemStore stores the data. + +## Storage configuration + +Below is the main storage configuration `mbed_lib.json` file: + +``` +{ +"name": "storage", + "config": { + "storage_type": { + "help": "Options are TDB_INTERNAL, TDB_EXTERNAL, TDB_EXTERNAL_NO_RBP, FILESYSTEM, FILESYSTEM_NO_RBP or default. If default, the storage type will be chosen according to the component defined in targets.json", + "value": "NULL" + }, + "default_kv": { + "help": "A string name for the default kvstore configurtaion", + "value": "kv" + } + } +} +``` + +### `TDB_INTERNAL` + +The internal configuration should be used for targets willing to save all the data in internal flash. + +![TDB_Internal](./Internal.jpg) + +In this configuration, all KVStore C APIs are mapped to the TDBStore in the internal flash. To use this configuration, set the `storage_type` parameter in storage `mbed_lib.json` to `TDB_INTERNAL`. + +Below is the `TDB_INTERNAL` configuration `mbed_lib.json`: + +``` +{ + "name": "tdb_internal", + "config": { + "internal_size": { + "help": "Size of the FlashIAP block device", + "value": "NULL" + }, + "internal_base_address": { + "help": "If not defined the default is the first sector after the application code ends.", + "value": "NULL" + }, + "rbp_number_of_entries": { + "help": "If not defined default is 64", + "value": "64" + } + } +} +``` + +For this configuration, please define the section of the internal storage that will be used for data, by defining these parameters in your `app.config file`: `internal_base_address` and `internal_size`. If not defined, the storage will start in the first sector immediately after the end of the application. This can reduce the ability to update the application with a bigger one. + +### `TDB_External` + +![External](./TDB_External.jpg) + +`TDB_EXTERNAL` uses a TDBStore in the internal flash for security rollback protection and a TDBStore on the external flash for the data. In this configuration, all KVStore C API calls are mapped to work with the SecureStore class. This class handles the use of the two TDBStores. Tthe external TDBStore works on top of the default block device, and the internal TDBStore works with the FlashIAPBlockdevice. + +You can set the external TDBStore block device to any of the following block devices: SPIF, QSPIF, DATAFASH and SD. + +You can enable this configuration by setting `storage_type` in storage `mbed_lib.json` to `TDB_EXTERNAL`. + +Below is the `TDB_EXTERNAL` configuration `mbed_lib.json`: + +``` +{ + + "name": "tdb_external", + "config": { + "rbp_internal_size": { + "help": "If not defined default size is 4K*#enteries/32", + "value": "NULL" + }, + "rbp_number_of_entries": { + "help": "If not defined default is 64", + "value": "64" + }, + "internal_base_address": { + "help": "If not defined the default is the first sector after the application code ends.", + "value": "NULL" + }, + "blockdevice": { + "help": "Options are default, SPIF, DATAFASH, QSPIF or SD", + "value": "NULL" + }, + "external_size": { + "help": "Size of the external block device", + "value": "NULL" + }, + "external_base_address": { + "help": "If not defined the default is from address 0", + "value": "NULL" + } + } +} +``` + +### TDB_External_no_RBP + +![External](./TDB_External_no_rbp.jpg) + +`TDB_EXTERNAL_NO_RBF` configuration has no support for rollback protection and is therefore less secure. + +The `TDB_EXTERNAL_NO_RBP` uses only one TDBStore on the external flash for all data. In this configuration, all KVStore C API calls are mapped to work with the SecureStore class. The external TDBStore works on top of the default block device; however, you can set the external TDBStore block device to any of the following block devices: SPIF, QSPIF, DATAFASH and SD. + +You can enable this configuration by setting `storage_type` in storage `mbed_lib.json` to `TDB_EXTERNAL_NO_RBP`. + +Below is the `TDB_EXTERNAL_NO_RBP` configuration `mbed_lib.json`: + +``` +{ + "name": "tdb_external_no_rbp", + "config": { + "external_size": { + "help": "Size of the external block device", + "value": "NULL" + }, + "external_base_address": { + "help": "If not defined the default is from address 0", + "value": "NULL" + }, + "blockdevice": { + "help": "Options are default, SPIF, DATAFASH, QSPIF or FILESYSTEM", + "value": "NULL" + } + } +} +``` + +### FILESYSTEM + +![FILESYSTEM](./FILESYSTEM.jpg) + +The FILESYSTEM configuration resembles the EXTERNAL but uses FileSystemStore on the external flash. By default, FileSystemStore uses the default file system and the default block device. + +In this configuration, all KVStore C API paths are mapped to the SecureStore class. This class handles the use of the internal TDBStore or external FileSystemStore. + +You can enable this configuration by setting `storage_type` in storage `mbed_lib.json` to FILESYSTEM. + +Below is the FILESYSTEM configuration `mbed_lib.json`: + +``` +{ + "name": "filesystem_store", + "config": { + "rbp_internal_size": { + "help": "If not defined default size is 4K*#enteries/32", + "value": "NULL" + }, + "rbp_number_of_entries": { + "help": "If not defined default is 64", + "value": "64" + }, + "internal_base_address": { + "help": "If not defined the default is the first sector after the application code ends.", + "value": "NULL" + }, + "filesystem": { + "help": "Options are default, FAT or LITTLE. If not specified default filesystem will be used", + "value": "NULL" + }, + "blockdevice": { + "help": "Options are default, SPIF, DATAFASH, QSPIF or FILESYSTEM. If not set the default block device will be used", + "value": "NULL" + }, + "external_size": { + "help": "Size in bytes of the external block device, if not specified the maximum is the default.", + "value": "NULL" + }, + "external_base_address": { + "help": "If not defined the default is from address 0", + "value": "NULL" + }, + "mount_point": { + "help": "Where to mount the filesystem. Ignored if the default file system is applied.", + "value": "/sd" + }, + "folder_path": { + "help": "Path for the working directory where the FileSyetemStore stores the data", + "value": "/kvstore" + } + } +} +``` + +If file system is not set, the default file system and block device are applied and `blockdevice`, `external_size` and `external_base_address` are ignored. + +### FILESYSTEM_NO_RBP + +![FILESYSTEM](./FILESYSTEM_no_rbp.jpg) + +The `FILESYSTEM_NO_RBP` configuration resembles the `EXTERNAL_NO_RBP` but uses FileSystemStore on the external flash. By default, FileSystemStore uses the default file system and the default block device. This Configuration has no support for rollback protection and is therefore less secure. + +In this configuration, all KVStore C API calls are mapped to the SecureStore class. This class handles the use of the external FileSystemStore. + +You can enable this configuration by setting `storage_type` in `storage mbed_lib.json` to `FILESYSTEM_NO_RBF`. + +Below is the FILESYSTEM configuration `mbed_lib.json`: + +``` +{ + "name": "filesystem_store_no_rbp", + "config": { + "filesystem": { + "help": "Options are default, FAT or LITTLE. If not specified default filesystem will be used", + "value": "NULL" + }, + "blockdevice": { + "help": "Options are default, SPIF, DATAFASH, QSPIF or FILESYSTEM. If not set the default block device will be used", + "value": "NULL" + }, + "external_size": { + "help": "Size in bytes of the external block device, if not specified the maximum is the default.", + "value": "NULL" + }, + "external_base_address": { + "help": "If not defined the default is from address 0", + "value": "NULL" + }, + "mount_point": { + "help": "Where to mount the filesystem. Ignored if the default file system is applied.", + "value": "/sd" + }, + "folder_path": { + "help": "Path for the working directory where the FileSyetemStore stores the data", + "value": "/kvstore" + } + } +} +``` + +If file system is not set, the default file system and block device are applied and `blockdevice`, `external_size` and `external_base_address` are ignored. + +### Configuration functions API + +Applications must call the function **storage_configuration()** to instantiate the required configuration. This function is defined as weak to allow the replacement of this function with a completely different implementation of the instantiation of components. + +Below is a list of setup functions that `storage_configuration()` calls in each case, and their description: + +``` +#if MBED_CONF_STORAGE_STORAGE == NULL +define MBED_CONF_STORAGE_STORAGE USER_DEFINED +#endif + +#define _STORAGE_CONFIG_concat(dev) _storage_config_##dev() +#define _STORAGE_CONFIG(dev) _STORAGE_CONFIG_concat(dev) + +/** + * @brief This function initializes internal memory secure storage + * This includes a TDBStore instance with a FlashIAPBlockdevice + * as the supported storage. + * The following is a list of configuration parameter + * MBED_CONF_STORAGE_INTERNAL_SIZE - The size of the underlying FlashIAPBlockdevice + * MBED_CONF_STORAGE_INTERNAL_BASE_ADDRESS - The start address of the underlying FlashIAPBlockdevice + * MBED_CONF_STORAGE_INTERNAL_RBP_NUMBER_OF_ENTRIES - If not defined default is 64 + * @returns true on success or false on failure. + */ +bool _storage_config_TDB_INTERNAL(); + +/** + * @brief This function initialize external memory secure storage + * This includes a SecureStore class with TDBStore over FlashIAPBlockdevice + * and an external TDBStore over a default blockdevice unless configured differently. + * The following is a list of configuration parameter: + * MBED_CONF_STORAGE_TDB_EXTERNAL_RBP_INTERNAL_SIZE - Size of the internal FlashIAPBlockDevice and by default is set to 4K*#enteries/32. The start address will be set to end of flash - rbp_internal_size. + * MBED_CONF_STORAGE_TDB_EXTERNAL_RBP_NUMBER_OF_ENTRIES - If not defined default is 64 + * MBED_CONF_STORAGE_TDB_EXTERNAL_EXTERNAL_SIZE - Size of the external blockdevice in bytes or NULL for max possible size. + * MBED_CONF_STORAGE_TDB_EXTERNAL_EXTERNAL_BASE_ADDRESS - The block device start address. + * MBED_CONF_STORAGE_TDB_EXTERNAL_EXTERNAL_BLOCK_DEVICE - Alowed vlaues are: default, SPIF, DATAFASH, QSPIF or SD + * @returns true on success or false on failure. + */ +bool _storage_config_TDB_EXTERNAL(); + +/** + * @brief This function initialize a predefined external memory secure storage + * This includes a SecureStore class with external TDBStore over a blockdevice or, + * if no blockdevice was set the default blockdevice will be used. + * The following is a list of configuration parameter: + * MBED_CONF_STORAGE_TDB_EXTERNAL_NO_RBP_EXTERNAL_SIZE - Size of the external blockdevice in bytes or NULL for max possible size. + * MBED_CONF_STORAGE_TDB_EXTERNAL_NO_RBP_EXTERNAL_BASE_ADDRESS - The block device start address + * MBED_CONF_STORAGE_TDB_EXTERNAL_NO_RBP_EXTERNAL_BLOCK_DEVICE - Alowed vlaues are: default, SPIF, DATAFASH, QSPIF or SD + * @returns true on success or false on failure. + */ +bool _storage_config_TDB_EXTERNAL_NO_RBP(); + +/** + * @brief This function initialize a predefined FILESYSTEM memory secure storage + * This includes a SecureStore class with TDBStore over FlashIAPBlockdevice + * in the internal memory and an external FileSysteStore. If blockdevice and filesystem not set, + * the system will use the default block device and default filesystem + * The following is a list of configuration parameter: + * MBED_CONF_STORAGE_FILESYSTEM_RBP_INTERNAL_SIZE - Size of the internal FlashIAPBlockDevice and by default is set to 4K*#enteries/32. The start address will be set to end of flash - rbp_internal_size. + * MBED_CONF_STORAGE_FILESYSTEM_RBP_NUMBER_OF_ENTRIES - If not defined default is 64 + * MBED_CONF_STORAGE_FILESYSTEM_FILESYSTEM - Allowed values are: default, FAT or LITTLE + * MBED_CONF_STORAGE_FILESYSTEM_BLOCKDEVICE - Allowed values are: default, SPIF, DATAFASH, QSPIF or SD + * MBED_CONF_STORAGE_FILESYSTEM_EXTERNAL_SIZE - External Blockdevice size in bytes or NULL for max possible size. + * MBED_CONF_STORAGE_FILESYSTEM_EXTERNAL_BASE_ADDRESS - The block device start address. + * MBED_CONF_STORAGE_FILESYSTEM_MOUNT_POINT - Where to mount the filesystem + * MBED_CONF_STORAGE_FILESYSTEM_FOLDER_PATH - The working folder paths + * + * @returns true on success or false on failure. + */ +bool _storage_config_FILESYSTEM(); + +/** + * @brief This function initialize a predefined FILESYSTEM_NO_RBP memory secure storage with no + * rollback protection. This includes a SecureStore class an external FileSysteStore over a default + * filesystem with default blockdevice unless differently configured. + * The following is a list of configuration parameter: + * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_FILESYSTEM - Allowed values are: default, FAT or LITTLE + * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_BLOCKDEVICE - Allowed values are: default, SPIF, DATAFASH, QSPIF or SD + * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_EXTERNAL_SIZE - Blockdevice size in bytes. or NULL for max possible size. + * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_EXTERNAL_BASE_ADDRESS - The block device start address. + * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_MOUNT_POINT - Where to mount the filesystem + * MBED_CONF_STORAGE_FILESYSTEM_NO_RBP_FOLDER_PATH - The working folder paths + * + * @returns true on success or false on failure. + */ +bool _storage_config_FILESYSTEM_NO_RBP(); + +MBED_WEAK bool storage_configuration() +{ + return _STORAGE_CONFIG(MBED_CONF_STORAGE_STORAGE_TYPE); +} +``` + +### Override user-defined setup + +To create a more complex setup including using other block devices, such as MBRBlockDevice or SlicingBlockDevice, you need to override the `storage_configuration` function and generate the storage configuration you choose. diff --git a/storage/docs/Configuration/FILESYSTEM.jpg b/storage/docs/Configuration/FILESYSTEM.jpg new file mode 100644 index 0000000..52ea615 --- /dev/null +++ b/storage/docs/Configuration/FILESYSTEM.jpg Binary files differ diff --git a/storage/docs/Configuration/FILESYSTEM.xml b/storage/docs/Configuration/FILESYSTEM.xml new file mode 100644 index 0000000..c8434cb --- /dev/null +++ b/storage/docs/Configuration/FILESYSTEM.xml @@ -0,0 +1 @@ +7Vjfb9owEP5reNyU39DHUKBFYxITaFsf3eRIvJoYOQ4l++tnkwtJmrSqJho0NF6wP/vsu+++s5MM7Nvt4U6QXfyVh8AGlhEeBvZkYFmmY3nqTyN5gYwMpwAiQUOcVAEr+hsQNBDNaAhpY6LknEm6a4IBTxIIZAMjQvDn5rQNZ81ddySCFrAKCGujP2goY4zCNSr8HmgUlzubBo48kuApEjxLcL+BZW+Ov2J4S8q1cH4ak5A/1yB7OrBvBeeyaG0Pt8A0tyVthd3sldGT3wIS+S6DG/RD5mXsECoqsMuFjHnEE8KmFTo+xgd6BUP1YrllqmmqJhyo/Knhzy72HsqRRIq8NqS7D7jAL5AyRw2QTHIFVfsuON/hGu3YMNyUZyJA7y1UCxERnLJTYDqwmh0Scgd8C8obNUEAI5LumxogKKXoNK+iUzWQ0W520Zk9YRkuqkyMp30quQDV8pfzFv01Ol8NeA9CwuHNWHDULoWWlwWG/edK1uYIsbgmads4Q/hGK7p/W1ylkhrqsi+lrtKbmry+fD9uhP5ZBiM5iL4lNupTYsNrk9jZ5YSmS04TWWXNcptZs90X2Sg0hFYvEnJy4105ss3/OfqrHLlebzkqQ6qdJSsIMqHycbyqej1BLK/HE8S5tkvK6bikHOtSl5TTvqTWk3EfqjKtV6qnpqrTKXh2VbUf/WaMpPHcX44ZD54msKdB3ww4Zp8M2NdWV05HXQ0vVldOW2CUwSpPJWz7KC/HeUd5eR8lLrcV/QQ2JGPqHtSvWKvlfFa0vlXNib/2Zwt/da/aXOhZys5TJvb4UXW9SJ6Y6Y01Z9gna961leSwoyTdi5Xk8G1Rzvx1qbzFfL1eTPvWWtfx/2FaG7XImCcShMpyK2wVj2xqS0BKf5PH4wStmp1+xD16444H7kQhWjtpISNtQBiNEtVmsNFLaZJoQJiPsNS6Gqc7EtAkWh9F9sk5D8uW6zbPwY4vLF0kW+cg+aZF8vRwjSS7xovLxjA/imTVrT6/Fm9S1Tdue/oH \ No newline at end of file diff --git a/storage/docs/Configuration/FILESYSTEM_no_rbp.jpg b/storage/docs/Configuration/FILESYSTEM_no_rbp.jpg new file mode 100644 index 0000000..04659b7 --- /dev/null +++ b/storage/docs/Configuration/FILESYSTEM_no_rbp.jpg Binary files differ diff --git a/storage/docs/Configuration/FILESYSTEM_no_rbp.xml b/storage/docs/Configuration/FILESYSTEM_no_rbp.xml new file mode 100644 index 0000000..6efe447 --- /dev/null +++ b/storage/docs/Configuration/FILESYSTEM_no_rbp.xml @@ -0,0 +1 @@ +3ZjbcpswEIafhst2MALbufQxydSdSceeNrmUYQ1qZOQRwjF5+kqwnIKTSacpmcQ3Xv3alXZXn/DBIrP96VLSQ/RdBMAtxw5OFplbjjNwnaF+M0pWKGPbLYRQsgCdamHNHgFFG9WUBZC0HJUQXLFDW/RFHIOvWhqVUjy03XaCt3c90BA6wtqnvKv+YoGKsArPrvUrYGFU7jywcWZL/ftQijTG/SyH7PJXMb2n5Vron0Q0EA8NiSwsMpNCqMLan2bATW/LthVxy2dmq7wlxOpVAReYh8rK2iHQrcChkCoSoYgpX9TqNK8PzAq2HkVqz7U50CacmLo18lcPR3flTKxk1pgywztc4DcolSEDNFVCS/W+KyEOuEa3Niw3Ean0MXsHaaEyhOp0Cs0U1ojDhlyC2IPORjtI4FSxY5sBiiiFlV/dTm1gR893F5M5Up7iojrEvj8mSkjQ1uTmutP+RjufLfgIUsHpxVpwlpSgZeUFw/FDjfVgjFrUQJrYb1C+3anuY8NVktSii7wXXWU2Dby+/cw3wvwcm9MMZN+IjXtEjAw61X1wxM7j1GbOdf8RMQy9ESxW9Ul6w/ZJEu/JCRU5YNSTQ6rSeB27pMPuGvxU6jPKH429EusMeyTWJZ+MWNc9A+jovR6KZTYNsJaMwzpLFOz7gMsZP3OJGnBVwL05XF6n+jnsaMr1fTVfPdY318vC+lGb88lmslxN1lfaFtJ46bihDiHTrR4OQ1V1preuuaM+uzb8bFdydOZKeu92JUcvQ7mcbEryVtebzWrRN2uDPlm76DRjcVIg9Sl3ytb1qDZbEhL2SLe5g6HmYD6K82y8qeXNtWLYSQqMTADlLIy1zWFnljJNYvpn7gRlZbiaJgfqszjc5JB9cf/Tc7BcotHlc012/r7Jelj/Yi6+jNR/S5DFHw== \ No newline at end of file diff --git a/storage/docs/Configuration/Internal.jpg b/storage/docs/Configuration/Internal.jpg new file mode 100644 index 0000000..3aadaa1 --- /dev/null +++ b/storage/docs/Configuration/Internal.jpg Binary files differ diff --git a/storage/docs/Configuration/Internal.xml b/storage/docs/Configuration/Internal.xml new file mode 100644 index 0000000..d3e1441 --- /dev/null +++ b/storage/docs/Configuration/Internal.xml @@ -0,0 +1 @@ +3VVNT8JAEP01vZptFwgeBfEjamKCUTmu7dBWth2yTKH117ulU9qmYjwoB7kw8+b78UodOU3ya6PW0QMGoB1PBLkjLx3PcwfeyH6VSFEhYzGogNDEASc1wDz+AAYFo1kcwKaTSIia4nUX9DFNwacOpozBXTdtibo7da1C6AFzX+k++hIHFPEVQ9HgNxCHUT3ZFRx5U/4qNJilPM/x5HL/qcKJqntx/iZSAe5akJw5cmoQqbKSfAq65Lamraq7OhI97G0gpR8VnPMeVNS3Q2CpYBcNRRhiqvSsQSf7+6DsIKwXUaKt6VoT8pheS/hsyN6ijqRkilaodBfc4B2ICtaAyggt1My9R1xzj2rTcr2jxzK0wcz4nOWxfJQJ4fBzHZi2CgZMwG5jcwxoRfG2216xlMJDXkOnNZjRr9nl2VulM25qS8RquyE0YK2Lx9se/S06uRoMQf79yf1buEAOWGhF/YCxv2tk7Y4Zi1qSluIXzhe96/6ZuNz6z6mtLnkiddXDW/K6e94P4nU8oVUB5tQSG59QYhXZHQ6eLidzMggnPtsb/d3Z1m1eCftY670rZ58= \ No newline at end of file diff --git a/storage/docs/Configuration/NewStorageDesign.jpg b/storage/docs/Configuration/NewStorageDesign.jpg new file mode 100644 index 0000000..601d575 --- /dev/null +++ b/storage/docs/Configuration/NewStorageDesign.jpg Binary files differ diff --git a/storage/docs/Configuration/TDB_External.jpg b/storage/docs/Configuration/TDB_External.jpg new file mode 100644 index 0000000..10994b6 --- /dev/null +++ b/storage/docs/Configuration/TDB_External.jpg Binary files differ diff --git a/storage/docs/Configuration/TDB_External.xml b/storage/docs/Configuration/TDB_External.xml new file mode 100644 index 0000000..d2c83a2 --- /dev/null +++ b/storage/docs/Configuration/TDB_External.xml @@ -0,0 +1 @@ +5VhRb5swEP41PHYCG9rmMYSkrdZJnVJt66MLDnh1cGRMmvTXz4YjQE2nTsqI1uUl9mdzvvv47nyJg2fr3ZUkm+yLSCh3kJvsHBw5CHk+OtdfBtnXyGWAayCVLIFNLbBkLxRAF9CSJbTobVRCcMU2fTAWeU5j1cOIlOK5v20leP/UDUmpBSxjwm30O0tU1kThtvg1ZWnWnOy5sPJI4qdUijKH8xyEV9WnXl6TxhbsLzKSiOcOhOcOnkkhVD1a72aUG24b2urnFm+sHvyWNFfvemACfqh9EztNNBUwFVJlIhU54fMWDav4qLHg6lmm1lwPPT2kO6Z+GPhTALOHZiVXct9ZMtMHMPCTKrUHDZBSCQ21594KsQEbdmwQbiFKGYP3CNRCZEoPb6fGTGCd54CQKyrWVHujN0jKiWLbvgYISCk97Gvp1ANgdJhdcGZLeAlG9SPu07ZQQlI9mt7dWPR36Hwz4C2Viu5+Gwus4kZo+ybBYP7cytq7BCzrSBq7RwjftaL7t8XVKKmnLnwqdTXedOT1+Vt1EPiHXE72VI4tscsxJXbx0SSGbYmhkxUw7P0P9J4sg7FvZfB8p6jUgVm862xUfTolLdgLeaw2GKI2guWq8iYInSDSCOEszTUQa0p0HcChyWqm25wpLKxZklTvjJNHysND8zITXMjq3KZ9GeIXv1UrDo0ZeNdrboZqiPvJnwSoV0aCevZuzsH2nSGhNXx20TN6FvQNiNWqMIXy1Ss7OPi+Ooytt7ikcSl1QlTX/KjVF52PWH0x+mDlAQ1c8E0Xd4L20b7g76NwDFV5r1SFA1tVyP9bd7rdNi84KbKb6V3IRfwU0S2Lx2bA90ZkAONx8uq4uTN0tfonyx27KI+UO/7FKXPH7igiuiIl15ec+e25NL84zSCa3k8Xt9PldT39qhcW9VBf/HpfNDJL4+ZXYLF0kx+z79K5VNRp5XXaME5XaqAJUybPwmJDYpan91XSnfnHYRkFr7qDycRi+XyAZPTnJOtp+7dV3UW1/w3i+S8= \ No newline at end of file diff --git a/storage/docs/Configuration/TDB_External_no_rbp.jpg b/storage/docs/Configuration/TDB_External_no_rbp.jpg new file mode 100644 index 0000000..06644f8 --- /dev/null +++ b/storage/docs/Configuration/TDB_External_no_rbp.jpg Binary files differ diff --git a/storage/docs/Configuration/TDB_External_no_rbp.xml b/storage/docs/Configuration/TDB_External_no_rbp.xml new file mode 100644 index 0000000..414d36d --- /dev/null +++ b/storage/docs/Configuration/TDB_External_no_rbp.xml @@ -0,0 +1 @@ +1VfbbuIwEP2aPG4V7KSCRyDQrrYrdUW1u300yZB4a2JkHC79+nWScS4NVJVKU5UXPGfs8VyOx45Dp+vDjWKb5KeMQDjEjQ4ODRxCBh65Nn85ciyRoeuVQKx4hJNqYMGfAUEX0YxHsG1N1FIKzTdtMJRpCqFuYUwpuW9PW0nR3nXDYugAi5CJLvqHRzrBKHy3xm+Bx4ndeeCiZsnCp1jJLMX9HEJXxa9Ur5m1hfO3CYvkvgHRmUOnSkpdjtaHKYg8tzZt5br5GW3lt4JUv2nBCP3QRxs7RCYVKEqlExnLlIlZjU6K+CC34Bop0WthhgMzhAPXf3P4ykfp0WpSrY4NVS4+ooF/oPUROcAyLQ1U73sn5QZtdGPDcLcyUyF6T5AtTMVQVafE8sAa6zAhNyDXYLwxExQIpvmuzQGGVIqreXU6zQAzejq76MyOiQyNmiXu026rpQIzGt9/76S/kc6zAe9AaTi8GgtqqSXa0R4wlPc1rQdDxJIGpal7gfDdTnRfm1yWSS120c9il/WmQa8fv4uN0D/iCnYE1TfFhj1SjA460X1xitEuxcjFKYZL7yVPdV1Jct2uZFVZa6J0FFe9KFLlxtvq5nW4OztoUCZjnYIaHup2nRRs+TNbFhPyCmxyjwof/YnjBwZhgsepAUKTa3MC6CTnMzcX/BgVax5FBRkEW4KYVNf2VAqpin3txX2qcPTcKameJOhd61o/dXrcK2/kk1ba/VJ6ZzG/jfxTVq0BuVpt4b1VtGRtVHEBYabMSSsuuF77TsXeXvoO7afvXPZtdKq3eJ91fZEueR6CSR/MIaMXfc7zO8wh3kc9irqdL4AVy4Q5jPnrcJG/CfNBMH4Yz+/Gi9tS/GUU83JoGpSZF/ScJc/9uCwZsf7mKRtR/WFJZ/8B \ No newline at end of file diff --git a/storage/docs/Configuration/UserDefined.jpg b/storage/docs/Configuration/UserDefined.jpg new file mode 100644 index 0000000..dbcfe8a --- /dev/null +++ b/storage/docs/Configuration/UserDefined.jpg Binary files differ diff --git a/storage/docs/Configuration/UserDefined.xml b/storage/docs/Configuration/UserDefined.xml new file mode 100644 index 0000000..08de035 --- /dev/null +++ b/storage/docs/Configuration/UserDefined.xml @@ -0,0 +1 @@ +3VjLctowFP0aL9ux/CCwNAESprSTDjRtlop9Y6sRFiMLAvn6Srb8ikjbmYLdgQVIR1ePe3yOJWG51+v9Dceb5DOLgFqOHe0td2I5DvKcgfxRyKFAhrZXADEnkQ6qgSV5BQ3aGt2SCLJWoGCMCrJpgyFLUwhFC8Ocs5d22BOj7Vk3OAYDWIaYmuh3EolEZ+HbNX4LJE7KmZGtWx5x+Bxztk31fJbjPuWfonmNy7F0fJbgiL00IHdqudecMVGU1vtroIrbkrai3+yd1mrdHFLxNx2qdYhDmTtEkgpdZVwkLGYpptMaHef5gRrBlrVErKksIlmEPRE/FPzR17WHsiUV/NBoUtUHPcBPEOKgNYC3gkmonnfB2EaPYeam083Ylod69a5WC+Yx6KhhAam8Gt00HzfA1iAXIwM4UCzIri0BrJUUV3E1m7KgCT1Orl7LDtOtHvTTfT6PXp1jU3wAbjyBBqPv5rwDLmD/23x0q1s+Y+1FB/lF/aVWdmW8pKFq1/53CtDVhenLM/Xl9qUvz9CX7GI/7zLBOMhScDfvWFuofEN2oa3S2BejLd/UFupNXL4hrlmwstTznsnvxXy1Wkyr6pdvi8WZpeb7bal59pHXmHsuqSEjuxNIDTWE1tBWF1IbHpGa09s+eRZ27f+LXdQXu0PDyEsIt1wSWewShYNXk3Eb6MDSxsnkqsOTych8v1GcJfPgbkxZ+DyBHQk7ZWPY4/vNvbRrQGm3pgVHfTmwXExDa33Y7Y3AXK/LDfTS7gHlbtkUmN+bwJw/C2xGKGSHTMC6c+G9Pbp1qjxk3sK/Lu/ms4qAVmUSrILZIlje1s2TXk+5V0cuVGfbBcwL5TwVwKU9LGdA5UzjRy5LsaiSbRAhMxRtm3LIyCt+zAOUATeMpCJfnz+2/IlElA2zwpGqA6YkTmWZwpMaStFGQkwDDQtl0XG2wSFJ41Xu1w/eaXhHI7/FuzMYGbwPjtDunIJ286o13Ze0XxLJA/T2vGeK+0Qky2r9J2re1vin2p3+Ag== \ No newline at end of file diff --git a/storage/docs/FileSystemStore/FileSystemStore_class_hierarchy.jpg b/storage/docs/FileSystemStore/FileSystemStore_class_hierarchy.jpg new file mode 100644 index 0000000..ca9f62a --- /dev/null +++ b/storage/docs/FileSystemStore/FileSystemStore_class_hierarchy.jpg Binary files differ diff --git a/storage/docs/FileSystemStore/FileSystemStore_class_hierarchy.xml b/storage/docs/FileSystemStore/FileSystemStore_class_hierarchy.xml new file mode 100644 index 0000000..173fdab --- /dev/null +++ b/storage/docs/FileSystemStore/FileSystemStore_class_hierarchy.xml @@ -0,0 +1 @@ +7VfbbtswDP0aP3bwJU7zmjRNCwwDBgS7PSo2bQuVLU9WEmdfP8qm41u8ZsDWbkDzEFuHFEUe8ZJY3l1aPiiWJx9kCMJy7bC0vLXluo7vz/BhkFON3C4IiBUPSakFtvwHEGgTuuchFD1FLaXQPO+DgcwyCHQPY0rJY18tkqJ/as5iGAHbgIkx+oWHOqnRhW+3+CPwOGlOdmyS7FjwFCu5z+g8y/Wi6lOLU9bYIv0iYaE8diDv3vLulJS6fkvLOxCG24a2et9mQnr2W0Gmr9pAfhyY2EPj8lzg3lVu3NMnomT+fW98WqVMxTyzvCVK7bzEbwSryAx+o2Vey2YdmYZS3zDBY9oXoHOgWpv4FtOzOnnXABsuYHsqNKRbLRU0cgxnN9yDWD7EEmVCaJKrOc2ZPjjkh4tBJ3TXxnnXBDZlAL2obDSo2zPnHkBpjlm2rLlYV2ytiJm1gMjskqgViSopIo6X5a0imWmqEcel9YalXJjqegRxAGPV+KlTYZTOZ3cTgXLD+ABlB6LEeACZglYnVCGp61FyUBG7M1of25JwGizplMOCMEZVGJ9Nt5mIL5SME4npjBLzUwEjRiELl6becSVzQBZXiHS5ClmRQEikXMMPhL3WMGanG/2cIlUgmOaHfve4FD6Z+yg5Hntm2rsdMD1ksJB7FQDt6pbzwJBvP2NIY5mCHhmqbuMc43UX5P4fneNXTWOn3trIX28jo5x80TYy+5ez9P3nt7n2+nNt8ZL56F/Ix+mZthMyeBpOMZRvKgLX9njedahChtTpq1F75zfLb8/RWA+IfvG8ykic6hq/OxJn84Eh70+NRFy2P9Rr9fbfkHf/Ew== \ No newline at end of file diff --git a/storage/docs/FileSystemStore/FileSystemStore_design.md b/storage/docs/FileSystemStore/FileSystemStore_design.md new file mode 100644 index 0000000..521eaf2 --- /dev/null +++ b/storage/docs/FileSystemStore/FileSystemStore_design.md @@ -0,0 +1,410 @@ +# FileSystemStore in Mbed OS + +- [FileSystemStore in Mbed OS](#filesystemstore-in-mbed-os) + + [Revision history](#revision-history) +- [Introduction](#introduction) + + [Overview and background](#overview-and-background) + + [Requirements and assumptions](#requirements-and-assumptions) +- [System architecture and high-level design](#system-architecture-and-high-level-design) + * [Design basics](#design-basics) +- [Detailed design](#detailed-design) + + [Class header](#class-header) + + [Important data structures](#important-data-structures) + + [Initialization and reset](#initialization-and-reset) + + [Core APIs](#core-apis) + + [Incremental set APIs](#incremental-set-apis) + + [Key iterator APIs](#key-iterator-apis) +- [Usage scenarios and examples](#usage-scenarios-and-examples) + + [Standard usage of the class](#standard-usage-of-the-class) +- [Other information](#other-information) + + [Open issues](#open-issues) + + +### Revision history + +| Revision | Date | Authors | Mbed OS version | Comments | +|---------- |---------------- |-------------------------------------------------------- |----------------- |------------------ | +| 1.0 | 20 September 2018 | David Saada ([@davidsaada](https://github.com/davidsaada/)) | 5.11+ | Initial revision | + +# Introduction + +### Overview and background + +FileSystemStore is a lightweight implementation of the [KVStore](../KVStore/KVStore_design.md) interface over file systems. + +### Requirements and assumptions + +FileSystemStore assumes the underlying file system qualities for resilience and file validation. This means that if the underlying file system has no protection against power failures, then neither would FileSystemStore have. + +When initializing this class, it is assumed that the underlying FileSystem is initialized and mounted. + +# System architecture and high-level design + +## Design basics + +FileSystemStore implements the get/set interface using files, where a single file represents each key. A key is represented by the file name, and its value is stored as file data. Therefore, FileSystemStore imitates the get/set actions using simple file operations. Set is achieved using open-write-close, get using open-read-close and so on. + +All files are concentrated under a single directory, whose name is hard coded. So actions such as "reset" are mapped to the deletion of all files under this directory, and iteration actions use file system APIs to traverse the directory. + +### Data layout + +When storing the data, it is stored with a preceding 16-byte metadata header. Metadata includes flags and other parameters for basic validity checks. + +![FileSystemStore Record](./FileSystemStore_record.jpg) + +Fields are: + +- Magic: A constant value, for quick validity checking. +- Metadata size: Size of metadata header. +- Revision: FileSystemStore revision (currently 1). +- User flags: Flags received from user. Currently only write once is dealt with (others are ignored). + +# Detailed design + +FileSystemStore fully implements the KVStore interface over a file system. As such, it uses the FileSystem class interface for file operations. + +![FileSystemStore Class Hierarchy](./FileSystemStore_class_hierarchy.jpg) + +Functionality, as defined by KVStore, includes the following: + +- Initialization and reset. +- Core actions: get, set and remove. +- Incremental set actions. +- Iterator actions. + +### Class header + +FileSystemStore has the following header: + +```C++ +class FileSystemStore : KVStore { + +public: + FileSystemStore(FileSystem *fs); + virtual ~FileSystemStore(); + + // Initialization and reset + virtual int init(); + virtual int deinit(); + virtual int reset(); + + // Core API + virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags); + virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0); + virtual int get_info(const char *key, info_t *info); + virtual int remove(const char *key); + + // Incremental set API + virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags); + virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size); + virtual int set_finalize(set_handle_t handle); + + // Key iterator + virtual int iterator_open(iterator_t *it, const char *prefix = NULL); + virtual int iterator_next(iterator_t it, char *key, size_t key_size); + virtual int iterator_close(iterator_t it); + +private: + Mutex _mutex; + FileSystem *_fs; + bool _is_initialized; +} +``` + +### Important data structures + +```C++ +// Key metadata +typedef struct { + uint32_t magic; + uint16_t metadata_size; + uint16_t revision; + uint32_t flags; +} key_metadata_t; + +// incremental set handle +typedef struct { + char *key; + uint32_t create_flags; +} inc_set_handle_t; + +// iterator handle +typedef struct { + void *dir_handle; + char *prefix; +} key_iterator_handle_t; +``` + +### Initialization and reset + +**init function** + +Header: + +`virtual int init();` + +Pseudo code: + +- If `_is_initialized`, return OK. +- Create and take `_mutex`. +- Create the FileSystemStore directory if it doesn't exist. +- Set `_is_initialized` to true. +- Release `_mutex`. + +**deinit function** + +Header: + +`virtual int deinit();` + +Pseudo code: + +- If not `_is_initialized`, return OK. +- Take `_mutex`. +- Set `_is_initialized` to false. +- Release `_mutex`. + +**reset function** + +Header: + +`virtual int reset();` + +Pseudo code: + +- Take `_mutex`. +- Delete all files under the FileSystemStore directory. +- Set `_num_keys` to 0. +- Release `_mutex`. + +### Core APIs + +**set function** + +Header: + +`virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags);` + +Pseudo code: + +- If not `_is_initialized`, return "not initialized" error. +- Call `set_start` with all fields and a local `set_handle_t` variable. +- Call `set_add_data` with `buffer` and `size`. +- Call `set_finalize`. +- Return OK. + +**get function** + +Header: + +`virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0);` + +Pseudo code: + +- If not `_is_initialized`, return "not initialized" error. +- Take `_mutex`. +- Using the `stat` API, extract file size. +- Open file `key` for reading to achieve a file handle. +- If failed, release `_mutex` and return "not found" error. +- Read from file into a `key_metadata_t` structure. +- Using `size` API, achieve file size. +- Seek to `offset` + metadata size. +- Set `actual_size` as the minimum of buffer size and remainder of data. +- Read data from file to `buffer`, size is `actual_size`. +- Close file. +- Release `_mutex`. +- Return OK. + +**get_info function** + +Header: + +`virtual int get_info(const char *key, info_t *info);` + +Pseudo code: + +- If not `_is_initialized`, return "not initialized" error. +- Find file `key` under the FileSystemStore directory. If not existing, return "not found" error. +- Take `_mutex`. +- Open file `key` for reading to achieve a file handle. +- If failed, release `_mutex`, and return "not found" error. +- Using `size` API, achieve file size. +- Read from file into a `key_metadata_t` structure. +- Fill `info` structure with all relevant fields. +- Close file. +- Return OK. + +**remove function** + +Header: + +`virtual int remove(const char *key);` + +Pseudo code: + +- If not `_is_initialized`, return "not initialized" error. +- Take `_mutex`. +- Open file `key` for reading, and read data into a `key_metadata_t` structure. +- If not existing, return "not found error". +- If flag "write once" is preset, return "write once" error. +- Delete file `key`. +- Release `_mutex`. +- Return OK. + +### Incremental set APIs + +**set_start function** + +Header: + +`virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags);` + +Pseudo code: + +- Find file `key` under the FileSystemStore directory. If not existing, increase `_num_keys` by 1. +- Take `_mutex`. +- Open file for reading, and read data into a `key_metadata_t` structure. +- If existing and flag "write once" is preset, return "write once" error. +- Close file. +- Allocate an `inc_set_handle_t` structure into `handle`. +- Duplicate `key` in `handle`. +- Update `create_flags` in `handle`. +- Fill `key_metadata_t` structure with all relevant values (`create_flags` from handle). +- Open file `key` for writing to achieve a file handle. +- Write metadata structure to the file. +- Close file. + +**set_add_data function** + +Header: + +`virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size);` + +Pseudo code: + +- Open file `key` for appending to achieve a file handle. +- Write `value_data` to the file. +- Close file. + +**set_finalize function** + +Header: + +`virtual int set_finalize(set_handle_t handle);` + +Pseudo code: + +- Free `key` in `handle` and then `handle`. +- Release `_mutex`. + +### Key iterator APIs + +**iterator_open function** + +Header: + +`virtual int iterator_open(iterator_t *it, const char *prefix = NULL);` + +Pseudo code: + +- Take `_mutex`. +- Allocate a `key_iterator_handle_t` structure into `it`. +- Duplicate `prefix` into same field in iterator. +- Using directory `open` API, open FileSystemStore directory, and store dir handle in the handle's `dir_handle` field. +- Release `_mutex`. + +**iterator_next function** + +Header: + +`virtual int iterator_next(iterator_t it, char *key, size_t key_size);` + +Pseudo code: + +- Take `_mutex`. +- Using direcory `read` API on handle's `dir_handle` field, read next file in directory. +- While not reached end of directory. + - If name matches prefix: + - Copy file name to `key`, and return OK. + - Using direcory `read` API on handle's `dir_handle` field, read next file in directory. +- Return "not found" error. +- Release `_mutex`. + +**iterator_close function** + +Header: + +`virtual int iterator_close(iterator_t it);` + +Pseudo code: + +- Using directory `close` API on `dir_handle` close handle. +- Release `prefix` field in iterator and structure allocated at `it`. + +# Usage scenarios and examples + +### Standard usage of the class + +The following example code shows standard use of the FileSystemStore class : + +**Standard usage example** + +```C++ + +// External file system of LittleFS type. Should be initialized. +extern LittleFileSystem fs; + +// Instantiate fsstore with our file system +FileSystemStore fsstore(&fs); + +int res; + +// Initialize fsstore +res = fsstore.init(); + +// Add "Key1" +const char *val1 = "Value of key 1"; +const char *val2 = "Updated value of key 1"; +res = fsstore.set("Key1", val1, sizeof(val1), 0); +// Update value of "Key1" +res = fsstore.set("Key1", val2, sizeof(val2), 0); + +uint_8 value[32]; +size_t actual_size; +// Get value of "Key1". Value should return the updated value. +res = fsstore.get("Key1", value, sizeof(value), &actual_size); + +// Remove "Key1" +res = fsstore.remove("Key1"); + +// Incremental write, if need to generate large data with a small buffer +const int data_size = 1024; +char buf[8]; + +KVSTore::set_handle_t handle; +res = fsstore.set_start(&handle, "Key2", data_size, 0); +for (int i = 0; i < data_size / sizeof(buf); i++) { + memset(buf, i, sizeof(buf)); + res = fsstore.set_add_data(handle, buf, sizeof(buf)); +} +res = fsstore.set_finalize(handle); + +// Iterate over all keys starting with "Key" +res = 0; +KVSTore::iterator_t it; +fsstore.iterator_open(&it, "Key*"); +char key[KVSTore::KV_MAX_KEY_LENGTH]; +while (!res) { + res = fsstore.iterator_next(&it, key, sizeof(key)); +} +res = fsstore.iterator_close(&it); + +// Deinitialize FileSystemStore +res = fsstore.deinit(); +``` + +# Other information + +### Open issues + +- Need to figure a way to prevent mutex abuse in incremental set APIs. diff --git a/storage/docs/FileSystemStore/FileSystemStore_record.jpg b/storage/docs/FileSystemStore/FileSystemStore_record.jpg new file mode 100644 index 0000000..2d0d9d7 --- /dev/null +++ b/storage/docs/FileSystemStore/FileSystemStore_record.jpg Binary files differ diff --git a/storage/docs/FileSystemStore/FileSystemStore_record.xml b/storage/docs/FileSystemStore/FileSystemStore_record.xml new file mode 100644 index 0000000..6114ab0 --- /dev/null +++ b/storage/docs/FileSystemStore/FileSystemStore_record.xml @@ -0,0 +1 @@ +3VdLk6IwEP41VO0eZgvCQz2Ozusyl7W29hxJhNQEwoQgur9+OxJeCo47pTXlejH5upM0X3/dBMtdJNtnibP4VRDKLWSTreU+WAg5vo/gTyO7CplMvQqIJCPGqQWW7A81oG3QghGa9xyVEFyxrA+GIk1pqHoYllKUfbe14P1TMxzRI2AZYn6M/mZExRU69e0Wf6EsiuuTHdtYVjh8i6QoUnOehdz1/leZE1zvZfzzGBNRdiD30XIXUghVjZLtgnLNbU1bte5pxNrELWmqzlkQTKoVG8wLWocccFg7J2wDw0gPa2hVA684YmGNwu6rAU87VzvDZ/Be6Aear0Wq7vJ9tu/BwbGzbWs82kIeIkMHATYQ6KnYv3nfT0Y+dGxvP9R7NrTPNtV82mAuY6boMsOhtpZQHYDFKuEwc5rV3fyYlG2oVHTbgUy+nqlIqJI7cDHWqZGOKS2nllLZCtUzUNzR6MRg2JRG1GzcygMGRiHDapnZn1MLVZhghW9UMPv2dFkJotuWYKO5ExoMrqVBZ1SDDZE/6ebHKYL/69yc0R/8a+XG+1R/+JVTedkCe+I4ym+03dz6+wm5XydAxx5XYMPkw+ir6KuZO6N0Hfdq1AUD1B0wAtfVTA/DQvLdXMJll6qPqWl5rGYKKyZSmN7N7As1vckhc94Rc2iAOGcWXIK56ajodEc5q9X4J1vN2E3q7B5UxTGicyBZ9TOWKyne6EJwIQFJRUp1yIzzAwhzFulEhpA66OHuXKeMwXfUvTEkjBB9zKBE+vVFmIRPuUoYuSh02q5xV0G1rLtFNaQN79+LCqbtp9ve1vk+dh//Ag== \ No newline at end of file diff --git a/storage/docs/KVStore/KVStore_classes.jpg b/storage/docs/KVStore/KVStore_classes.jpg new file mode 100644 index 0000000..72d89fa --- /dev/null +++ b/storage/docs/KVStore/KVStore_classes.jpg Binary files differ diff --git a/storage/docs/KVStore/KVStore_classes.xml b/storage/docs/KVStore/KVStore_classes.xml new file mode 100644 index 0000000..4e0a0ce --- /dev/null +++ b/storage/docs/KVStore/KVStore_classes.xml @@ -0,0 +1 @@ +7VjbbuIwEP2aPLZKnITSx1J6kVYrVWK1l0eTDIlVJ84aQ8N+/drxODdAi6ou7UN5gPh4PB6fOeOJ8MLbon6QtMq/ihS4R/y09sK5R0gQkYn+McjOIldBYIFMshSNOmDB/gCCPqIblsJ6YKiE4IpVQzARZQmJGmBUSvEyNFsJPty1ohnsAYuE8n30B0tVbtFp7Hf4I7AsdzsHPs4safKcSbEpcT+PhKvmY6cL6nyh/TqnqXjpQeGdF95KIZR9Kupb4IZbR5tdd39kto1bQqlOWoBxbCnfgAt5wvXaWWXCUzukZPJ7Y2KaFVRmrPTCGz3rV7X+1mBzMoNfKFHZuag3p6BWF5SzDNclOjiQnU/9lOFvs/PSAd/ms4USEtyEPsdybKyxaozl0sTuVOW2CY7vmLLtwdPmmGQTNTEnOuZAR9H4cCgZuCNbkIpped1YEuYNTTOkZM5hZVYJbbXijRpWTGcpnK1EqbA4AoLje1owbsrqEfgWjFcTpyq4MWr37isARWFigLoHoSIeQBSg5E6b4GwYoSqweokbv3S1EDgs79XBFDGK5Ze1rjsJ6gdU4RFFRh9ZkV++fwry/QU5Pace4wN6HLEJZXpjuo4eLblInk0C6TqHFCnQ8/cNgXPfjvoc9qjSDMndT2N2Gbvhr3/RqLTOQQ2LB9JBf9snts/cBFmSwKli22ELPEQdunsSTIfSJikKjtwazsVabGQCuKrfk0aO4quRo3DkyJ54z1GTyfaMpyX36iNfNlozsNitFRSfl86rLp29G+ZAAR29dIJ37YLTjyzMBSQbCZ+iPL8o4/gdRdk27jO0woN97CSm/kNrm5Dry/htmlsUDhPYhvj2zc1l5yxvLjVT9sUlinFoXlwu/EufOOAJJNPh6xvGejs1xTrghtx+yz5L2ok/Kjb/lUkPr0dJj6/fKOl62P1ZYM27f2TCu78= \ No newline at end of file diff --git a/storage/docs/KVStore/KVStore_design.md b/storage/docs/KVStore/KVStore_design.md new file mode 100644 index 0000000..fc3fa08 --- /dev/null +++ b/storage/docs/KVStore/KVStore_design.md @@ -0,0 +1,488 @@ +# KVStore in Mbed OS + +- [KVStore in Mbed OS](#kvstore-in-mbed-os) + + [Revision history](#revision-history) +- [Introduction](#introduction) + + [Overview and background](#overview-and-background) + + [Requirements and assumptions](#requirements-and-assumptions) +- [System architecture and high-level design](#system-architecture-and-high-level-design) + * [Design basics](#design-basics) + + [Derived implementations](#derived-implementations) + * [Global Key Value interface](#global-key-value-interface) +- [Detailed design](#detailed-design) + * [KVStore class design](#kvstore-class-design) + + [KVStore Class header](#kvstore-class-header) + * [Global Key Value interface design](#global-key-value-interface-design) + + [Global Key Value APIs](#global-key-value-apis) + * [Mapping APIs](#mapping-apis) + * [Implementation](#implementation) + + [Important data structures](#important-data-structures) + + [Global Key Value API implementation](#global-key-value-api-implementation) + + [Attachment API implementation](#attachment-api-implementation) +- [Usage scenarios and examples](#usage-scenarios-and-examples) + + [Standard usage of the KVStore class](#standard-usage-of-the-kvstore-class) + + [Standard usage of the Global Key Value interface](#standard-usage-of-the-global-key-value-interface) +- [Other information](#other-information) + + [Open issues](#open-issues) + + +### Revision history + +| Revision | Date | Authors | Mbed OS version | Comments | +|---------- |---------------- |-------------------------------------------------------- |----------------- |------------------ | +| 1.0 | 26 September 2018 | David Saada ([@davidsaada](https://github.com/davidsaada/)) | 5.11+ | Initial revision | + +# Introduction + +### Overview and background + +KVStore is an interface class whose purpose is to define APIs for a Key Value Store like storage over a block device. + +### Requirements and assumptions + +# System architecture and high-level design + +## Design basics + +KVStore defines a key value store like API set using this interface class. Classes implementing this interface store pairs of keys and values, where the keys are represented as character strings and the values are represented as binary blobs. Core APIs here are *get* and *set*, providing read and write access by key to the value in a single call. *remove* completes the set of core APIs. This simplifies the interface for the cases we need an actual key value store (like configurations). + +APIs also support an "incremental set" mode, allowing the implementing class to aggregate chunks of data for the set operation. This is for when the case the caller needs to generate large portions of data but doesn't wish to allocate large buffers for a single set operation. Note that *get* API doesn't have or require this functionality. Instead, it has an offset parameter (defaulting to 0) allowing the calling layer to extract portions of the data. + +Interface also includes iteration APIs, to let you iterate over all available keys, given a prefix. + +As some of the implementations use files as keys, key names must comply to file naming rules, meaning that characters like * , / etc. are not allowed in key names. + +### Derived implementations + +![KVStore Classes](./KVStore_classes.jpg) + +KVStore has a few derived implementations: + +- [TDBStore](../TDBStore/TDBStore_design.md) is the default solution because it gives the best performance, flash wear leveling and lowest overhead for a limited number of keys. +- [FileSystemStore](../FileSystemStore/FileSystemStore_design.md) is the preferred solution if you already have a file system and don't wish to have an additional one, or if specific POSIX features (such as file seeking) are required. It's also preferred if you don't have a limitation on the number of keys. +- [SecureStore](../SecureStore/SecureStore_design.md) adds security features such as encryption, rollback protection and authentication. It uses one of the other KVStore solutions as the underlying storage type. + +## Global Key Value interface + +A parallel key-value API is provided as global C-style functions (for all functions, except for the incremental set ones). This API performs a limited type of mapping of partition or mount point names present in the keys. For each of the APIs defined in KVStore, the global version extracts a partition prefix from the key name. The prefix must be in the form "/partition/key-name". Then a lookup is performed to map the partition name to a concrete KVStore instance, and the API call is routed to that instance. The routed key name has the partition prefix stripped, leaving only "key-name". + +In the case of iteration APIs, the prefix must include the partition (in the form of "/partition/prefix"). + +# Detailed design + +## KVStore class design + +As an interface class, KVStore has no implementation, just a class header. + +### KVStore class header + +```C++ +class KVStore { + + enum create_flags { + WRITE_ONCE_FLAG = (1 << 0), + REQUIRE_CONFIDENTIALITY_FLAG = (1 << 1), + REQUIRE_REPLAY_PROTECTION_FLAG = (1 << 3), + }; + + static const uint32_t MAX_KEY_LENGTH = 128; + + typedef struct _opaque_set_handle *set_handle_t; + + typedef struct _opaque_key_iterator *iterator_t; + + typedef struct info { + size_t size; + uint32_t flags; + } info_t; + + // Initialization and reset + virtual int init(); + virtual int deinit(); + virtual int reset(); + + // Core API + virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags); + virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0); + virtual int get_info(const char *key, info_t *info); + virtual int remove(const char *key); + + // Incremental set API + virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags); + virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size); + virtual int set_finalize(set_handle_t handle); + + // Key iterator + virtual int iterator_open(iterator_t *it, const char *prefix = NULL); + virtual int iterator_next(iterator_t it, char *key, size_t key_size); + virtual int iterator_close(iterator_t it); +} +``` + +## Global Key Value interface design + +As mentioned above, each KVStore API has a parallel C-style API, used globally with a partition name preceding the key name. + +### Global Key Value APIs + +```C +enum kv_create_flags { + KV_WRITE_ONCE_FLAG = (1 << 0), + KV_REQUIRE_CONFIDENTIALITY_FLAG = (1 << 1), + KV_REQUIRE_REPLAY_PROTECTION_FLAG = (1 << 3), +}; + +static const uint32_t KV_MAX_KEY_LENGTH = 128; +typedef struct _opaque_set_handle *kv_set_handle_t; +typedef struct _opaque_key_iterator *kv_key_iterator_handle_t; + +typedef struct info { + size_t size; + uint32_t flags; +} kv_info_t; + +// Core API +int kv_set(const char *full_name_key, const void *buffer, size_t size, uint32_t create_flags); +int kv_get(const char *full_name_key, void *buffer, size_t buffer_size, size_t *actual_size); +int kv_get_info(const char *full_name_key, kv_info_t *info); +int kv_remove(const char *full_name_key); + +// Key iterator +int kv_iterator_open(kv_key_iterator_handle_t *it, const char *full_prefix = nullptr); +int kv_iterator_next(kv_key_iterator_handle_t it, char *key, size_t key_size); +int kv_iterator_close(kv_key_iterator_handle_t it); +``` + +## Mapping APIs + +To use the global C style APIs, you need APIs to map the partition name to the instance of the implementing KVStore class, typically called once at initialization time. So, for example a `"/tdbstore/key1"` name means that you wish to access `"key1"` key in a TDBStore instance. This means that you need to attach `"tdbstore"` string to the TDBStore instance at initialization time. + +These APIs are part of a different header file ("kv_map.h") because they serve the integration code and not the KVStore user code: + +```C +// Attach and detach +int kv_init(); +int kv_attach(const char *partition_name, KVStore *kv_instance); +int kv_detach(const char *partition_name); + +// Full name lookup and then break it into KVStore instance and key +int kv_lookup(const char *full_name, KVStore& *kv_instance, char *key); +``` + +## Implementation + +Below is the implementation of the Global Key Value interface and of the attachment APIs. KVStore class has no implemetation because it's an interface class: + +### Important data structures + +```C +// incremental set handle +typedef struct { + KVStore *kvstore_intance; + KVStore::set_handle_t *set_handle; +} kv_inc_set_handle_t; + +// iterator handle +typedef struct { + KVStore *kvstore_intance; + KVStore::iterator_t *iterator_handle; +} kv_key_iterator_handle_t; + +const int MAX_ATTACHED_KVS 16 + +typedef struct { + char *partition_name; + KVStore *kvstore_instance; +} kv_map_entry_t; + +// Attachment table +kv_map_entry_t kv_map_table[MAX_ATTACHED_KVS]; +int kv_num_attached_kvs; +``` + +### Global Key Value API implementation + +**kv_set function** + +Header: + +`int kv_set(const char *full_name_key, const void *buffer, size_t size, uint32_t create_flags);` + +Pseudo code: + +- Using `kv_lookup`, break `full_name_key` into `key` and `kvs_instance`. +- Call `kvs_instance` `set` method with `key` and the rest of the arguments. + +**kv_get function** + +Header: + +`int kv_get(const char *full_name_key, void *buffer, size_t buffer_size, size_t *actual_size);` + +Pseudo code: + +- Using `kv_lookup`, break `full_name_key` into `key` and `kvs_instance`. +- Call `kvs_instance` `get` method with `key` and the rest of the arguments. + +**kv_get_info function** + +Header: + +`int kv_get_info(const char *full_name_key, kv_info_t *info);` + +Pseudo code: + +- Using `kv_lookup`, break `full_name_key` into `key` and `kvs_instance`. +- Call `kvs_instance` `get_info` method with `key` and the rest of the arguments. + +**kv_remove function** + +Header: + +`int kv_remove(const char *full_name_key);` + +Pseudo code: + +- Using `kv_lookup`, break `full_name_key` into `key` and `kvs_instance`. +- Call `kvs_instance` `remove` method with `key` and the rest of the arguments. + +**kv_set_start function** + +Header: + +`int kv_set_start(kv_set_handle_t *handle, const char *full_name_key, size_t final_data_size);` + +Pseudo code: + +- Allocate an `kv_inc_set_handle_t` structure into `handle`. +- Using `kv_lookup`, break `full_name_key` into allocated `key` and `kvs_instance` (in `handle`). +- Call `kvs_instance` `set_start` method with `key` and the rest of the arguments. + +**kv_set_add_data function** + +Header: + +`int kv_set_add_data(kv_set_handle_t handle, const void *value_data, size_t data_size);` + +Pseudo code: + +- Extract `kvs_instance` and `set_handle` from `handle`. +- Call `kvs_instance` `set_add_data` method with `set_handle` and the rest of the arguments. + +**kv_set_finalize function** + +Header: + +`int kv_set_finalize(kv_set_handle_t handle);` + +Pseudo code: + +- Extract `kvs_instance` and `set_handle` from `handle`. +- Call `kvs_instance` `set_finalize` method with `set_handle`. +- Free `key` and `handle`. + +**kv_iterator_open function** + +Header: + +`int kv_iterator_open(kv_key_iterator_handle_t *it, const char *full_prefix = nullptr);` + +Pseudo code: + +- Allocate a `kv_key_iterator_handle_t` structure into `it`. +- Using `kv_lookup`, break `full_name_key` into allocated `prefix` and `kvs_instance` (in `handle`). +- Call `kvs_instance` `iterator_open` method with `iterator_handle`, `prefix` and the rest of the arguments. + +**kv_iterator_next function** + +Header: + +`int kv_iterator_next(kv_key_iterator_handle_t it, char *key, size_t key_size);` + +Pseudo code: + +- Extract `kvs_instance` and `iterator_handle` from `handle`. +- Call `kvs_instance` `iterator_next` method with `iterator_handle` and the rest of the arguments. + +**kv_iterator_close function** + +Header: + +`int kv_iterator_close(kv_key_iterator_handle_t it);` + +Pseudo code: + +- Extract `kvs_instance` and `iterator_handle` from `handle`. +- Call `kvs_instance` `set_finalize` method with `iterator_handle`. +- Free `prefix` and `handle`. + +### Attachment API implementation + +**kv_init function** + +Header: + +`int kv_init();` + +Pseudo code: + +- Set `kv_num_attached_kvs` to 0. + +**kv_attach function** + +Header: + +`int kv_attach(const char *partition_name, KVStore *kv_instance);` + +Pseudo code: + +- Duplicate `partition_name` and `kv_instance` to last entry in `kv_map_table`. +- Increment `kv_num_attached_kvs`. + +**kv_detach function** + +Header: + +`int kv_detach(const char *partition_name);` + +Pseudo code: + +- Look for entry with `partition_name` in `kv_map_table`. +- Deallocate `partition_name` in this entry. +- Copy all preceding entries back one position. +- Decrement `kv_num_attached_kvs`. + +**kv_lookup function** + +Header: + +`int kv_lookup(const char *full_name, KVStore& *kv_instance, char *key);` + +Pseudo code: + +- Break `full_name` string to `partition_name` and `key`. +- Look for entry with `partition_name` in `kv_map_table`. +- Extract `kv_instance` from table entry. + +# Usage scenarios and examples + +### Standard use of the KVStore class + +The following example code shows standard use of the KVStore, using the TDBStore class: + +**Standard usage example - with class APIs** + +```C++ +// Underlying block device. Here, SPI Flash is fully used. +// One can use SlicingBlockDevice if we want a partition. +SPIFBlockDevice bd(PTE2, PTE4, PTE1, PTE5); + +// Instantiate tdbstore with our block device and a maximum of 64 keys +kvstore = new TDBStore(64, &bd); + +int res; + +// Initialize storage +res = kvstore->init(); + +const char *val1 = "Value of key 1"; +const char *val2 = "Updated value of key 1"; +// Add "Key1" +res = kvstore->set("Key1", val1, sizeof(val1), 0); +// Update value of "Key1" +res = kvstore->set("Key1", val2, sizeof(val2), 0); + +uint_8 value[32]; +size_t actual_size; +// Get value of "Key1". Value should return the updated value. +res = kvstore->get("Key1", value, sizeof(value), &actual_size); + +// Remove "Key1" +res = kvstore->remove("Key1"); + +// Incremental write, if need to generate large data with a small buffer +const int data_size = 1024; +char buf[8]; + +KVSTore::set_handle_t handle; +res = kvstore->set_start(&handle, "Key2", data_size, 0); +for (int i = 0; i < data_size / sizeof(buf); i++) { + memset(buf, i, sizeof(buf)); + res = kvstore->set_add_data(handle, buf, sizeof(buf)); +} +res = kvstore->set_finalize(handle); + +// Iterate over all keys starting with "Key" +res = 0; +KVSTore::iterator_t it; +kvstore->iterator_open(&it, "Key*"); +char key[KVSTore::KV_MAX_KEY_LENGTH]; +while (!res) { + res = kvstore->iterator_next(&it, key, sizeof(key)); +} + +// Deinitialize TDBStore +res = kvstore->deinit(); +``` + +### Standard usage of the Global Key Value interface + +The following example code shows how to use the previous example with the global key value interface. Here, `tdtbstore` is mapped to `"/tdbstore/"`. + +**Standard usage example - with global C-style APIs** + +This example assumes this code exists somewhere and is called during initialization: + +```C +// Assume TDBtore is already instantiated and initialized +extern TDBStore tdbstore; + +int res; + +// Attachment code. Should be called at initialization +res = kv_init(); +res = kv_attach("tdbstore", &tdbstore); + +``` + +This example shows how to access KVStore using C global APIs: + +```C +const char *val1 = "Value of key 1"; +const char *val2 = "Updated value of key 1"; + +// Add "Key1", now with full name, including "/tdbstore/" prefix. +res = kv_set("/tdbstore/Key1", val1, sizeof(val1), 0); +// Update value of "Key1" +res = kv_set("/tdbstore/Key1", val2, sizeof(val2), 0); + +uint_8 value[32]; +size_t actual_size; +// Get value of "Key1". Value should return the updated value. +res = kv_get("/tdbstore/Key1", value, sizeof(value), &actual_size); + +// Remove "Key1" +res = kv_remove("/tdbstore/Key1"); + +// Incremental write, if need to generate large data with a small buffer +const int data_size = 1024; +char buf[8]; + +kv_set_handle_t handle; +res = kv_set_start(&handle, "/tdbstore/Key2", data_size, 0); +for (int i = 0; i < data_size / sizeof(buf); i++) { + memset(buf, i, sizeof(buf)); + res = kv_set_add_data(handle, buf, sizeof(buf)); +} +res = kv_set_finalize(handle); + +// Iterate over all keys starting with "Key" in tdbstore +res = 0; +kv_key_iterator_handle_t it; +kv_iterator_open(&it, "/tdbstore/Key"); +char key[KV_MAX_KEY_LENGTH]; +while (!res) { + res = kv_iterator_next(&it, key, sizeof(key)); +} +res = kv_iterator_close(&it); +``` diff --git a/storage/docs/SecureStore/SecureStore_class_hierarchy.jpg b/storage/docs/SecureStore/SecureStore_class_hierarchy.jpg new file mode 100644 index 0000000..19047d5 --- /dev/null +++ b/storage/docs/SecureStore/SecureStore_class_hierarchy.jpg Binary files differ diff --git a/storage/docs/SecureStore/SecureStore_class_hierarchy.xml b/storage/docs/SecureStore/SecureStore_class_hierarchy.xml new file mode 100644 index 0000000..c19c4de --- /dev/null +++ b/storage/docs/SecureStore/SecureStore_class_hierarchy.xml @@ -0,0 +1 @@ +7VhZb+IwEP41eWwVHKD0kaOHVK1UCe31aJIhserEWWMo7K/fcTK5QVBp6bISPED8zXg8/jyHieNN4+2T5mn0RQUgHeYGW8ebOYz1BgOGPxbZ5cjdqJ8DoRYBKVXAXPwGAl1C1yKAVUPRKCWNSJugr5IEfNPAuNbqvam2VLK5aspD6ABzn8su+l0EJsrR0cCt8GcQYVSs3HNJsuD+W6jVOqH1HOYts08ujnlhi/RXEQ/Uew3yHhxvqpUy+VO8nYK03Ba05fMeD0hLvzUk5qQJ5MeGyzUULg8lzp2k1j2zI0qGv9bWp0nMdSgSxxuj1E23+I1gtjOL3xiV5rJ+TWZga264FCHN89E50JVNfArpN1t5UQBz8Nca5kZpKGS4lUVbH7G0jUXaul8EVrFS7/Cigdjs3XBE52wdZ3ZThwygF5mNAmUNc2wD2giMsHHOwyxjakKszCQs7SyFWkuZBcRS4EF5k6VKDOVHj9H4kcdC2sx6BrkBa9X6aWJplcq160FAcWF9gG0NoqB4AhWD0TtUISnzKDAogVmfxu9VOvQKLKqlwogwThkYlqarKMQHCsQDQdnrBOXXFXQYhSQY21zHkUoBWZwgUucq4KsIAiLlFH4gaJSFLjv13Q9ppxokN2LTrBz7tk/mXpXAZUumvbsW020GV2qtfaBZ9VRuGRq4RwwZTFEwHUPZaZR7PO2A2CVXjZdvRyvGQl9ryNlrSCcgP7WG9P/zEL0G5Nmb2ugz43GwJx4PN7SFVP5bu4Wh/DEjcOZ2m12NKmRI735YtdtBMfx5jMa8OzST55/0w0NV46P9sD9sGfLO1w/vL7nYzGAjfHiB3bUjXlhH9O4/sQIVfznPdauuqk695ri3fVYWoVfQAv3GqM21TitG95dzOS+L04cv56OWob92Ocdh9bogV6/eyXgPfwA= \ No newline at end of file diff --git a/storage/docs/SecureStore/SecureStore_design.md b/storage/docs/SecureStore/SecureStore_design.md new file mode 100644 index 0000000..451a112 --- /dev/null +++ b/storage/docs/SecureStore/SecureStore_design.md @@ -0,0 +1,472 @@ +# SecureStore in Mbed OS + +- [SecureStore in Mbed OS](#securestore-in-mbed-os) + + [Revision history](#revision-history) +- [Introduction](#introduction) + + [Overview and background](#overview-and-background) + + [Requirements and assumptions](#requirements-and-assumptions) +- [System architecture and high-level design](#system-architecture-and-high-level-design) + * [Design basics](#design-basics) + + [Data layout](#data-layout) + + [Basic implementation concepts](#basic-implementation-concepts) +- [Detailed design](#detailed-design) + + [Class header](#class-header) + + [Important data structures](#important-data-structures) + + [Initialization and reset](#initialization-and-reset) + + [Core APIs](#core-apis) + + [Incremental set APIs](#incremental-set-apis) + + [Key iterator APIs](#key-iterator-apis) +- [Usage scenarios and examples](#usage-scenarios-and-examples) + + [Standard usage of the class](#standard-usage-of-the-class) +- [Other information](#other-information) + + [Open issues](#open-issues) + + +### Revision history + +| Revision | Date | Authors | Mbed OS version | Comments | +|---------- |---------------- |-------------------------------------------------------- |----------------- |------------------ | +| 1.0 | 02 October 2018 | David Saada ([@davidsaada](https://github.com/davidsaada/)) | 5.11+ | Initial revision | + +# Introduction + +### Overview and background + +SecureStore is a [KVStore](../KVStore/KVStore_design.md) based storage solution, providing security features on the stored data, such as encryption, authentication, rollback protection and write once, over an underlying KVStore class. It references an additional KVStore class for storing the rollback protection keys. + +### Requirements and assumptions + +SecureStore assumes that the underlying KVStore instances are instantiated and initialized. + +# System architecture and high-level design + +## Design basics + +SecureStore is a storage class, derived from KVStore. It adds security features to the underlying key value store. + +As such, it offers all KVStore APIs, with additional security options (which can be selected using the creation flags at set). These include: + +- Encryption: Data is encrypted using the AES-CTR encryption method, with a randomly generated 8-byte IV. Key is derived from [Device Key](../../../../../../mbed-os/drivers/device_key/README.md), using the NIST SP 800-108 KDF in counter mode spec, where salt is the key trimmed to 32 bytes, with "ENC" as prefix. Flag here is called "require confidentiality flag". +- Rollback protection: (Requires authentication) CMAC is stored in a designated rollback protected storage (also of KVStore type) and compared to when reading the data under the same KVStore key. A missing or different key in the rollback protected storage results in an error. The flag here is called "Require replay protection flag". +- Write once: Key can only be stored once and can't be removed. The flag here is called "Write once flag". + +SecureStore maintains data integrity using a record CMAC. This 16-byte CMAC is calculated on all stored data (including key & metadata) and stored at the end of the record. When reading the record, SecureStore compares the calculated CMAC with the stored one. In the case of encryption, CMAC is calculated on the encrypted data. The key used for generating the CMAC is derived from [Device Key](../../../../../../mbed-os/drivers/device_key/README.md), where salt is the key trimmed to 32 bytes, with "AUTH" as prefix. + +![SecureStore Layers](./SecureStore_layers.jpg) + +### Data layout + +When storing the data, it is stored with a preceding metadata header. Metadata includes flags and security related parameters, such as IV. The CMAC, calculated for authentication, is stored at the end of the data as it is calculated on the fly, so it can't be stored with the metadata. + +![SecureStore Record](./SecureStore_record.jpg) + +Fields are: + +- Metadata size: Size of metadata header. +- Revision: SecureStore revision (currently 1). +- Data size: Size of user data. +- Flags: User flags. +- IV: Random generated IV. +- Pad: Pad data to a multiple of 16 bytes (due to encryption). +- CMAC: CMAC calculated on key, metadata and data. + +### Basic implementation concepts + +Because the code can't construct a single buffer to store all data (including metadata and possibly encrypted data) in one shot, setting the data occurs in chunks, using the incremental set APIs. Get uses the offset argument to extract metadata, data and CMAC separately. + +Rollback protection (RBP) keys are stored in the designated rollback protection storage, which is also of KVStore type. RBP keys are the same as the SecureStore keys. +This RBP storage is also used for storing the CMAC in write once case, as otherwise an attacker can delete this key from the underlying storage and modify this flag. + +# Detailed design + +![SecureStore Class Hierarchy](./SecureStore_class_hierarchy.jpg) + +Functionality, as defined by KVStore, includes the following: + +- Initialization and reset. +- Core actions: get, set and remove. +- Incremental set actions. +- Iterator actions. + +### Class header + +SecureStore has the following header: + +```C++ +class SecureStore : KVStore { + +public: + SecureStore(KVStore *underlying_kv, KVStore *rbp_kv); + virtual ~SecureStore(); + + // Initialization and formatting + virtual int init(); + virtual int deinit(); + virtual int reset(); + + // Core API + virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags); + virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0); + virtual int get_info(const char *key, info_t *info); + virtual int remove(const char *key); + + // Incremental set API + virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags); + virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size); + virtual int set_finalize(set_handle_t handle); + + // Key iterator + virtual int iterator_open(iterator_t *it, const char *prefix = NULL); + virtual int iterator_next(iterator_t it, char *key, size_t key_size); + virtual int iterator_close(iterator_t it); + +private: + Mutex _mutex; + KVStore *_underlying_kv; + KVStore *_rbp_kv; + void *_entropy; + uint8_t *_scratch_buf; +} +``` + +### Important data structures + +```C++ +// Record header +typedef struct { + uint16_t metadata_size; + uint16_t revision; + uint32_t data_size; + uint32_t create_flags; + uint8_t iv[8]; +} record_metadata_t; + +// incremental set handle +typedef struct { + record_metadata_t metadata; + bd_size_t offset; + char *key; + void *encrypt_handle; + void *auth_handle; + KVStore::set_handle_t underlying_handle; +} inc_set_handle_t; + +// iterator handle +typedef struct { + KVStore::iterator_t underlying_it; +} key_iterator_handle_t; +``` + +### Initialization and reset + +**init function** + +Header: + +`virtual int init();` + +Pseudo code: + +- if `_is_initialized`, return OK. +- Take `_mutex`. +- Initialize `_entropy` with TLS entropy APIs. +- Using `DeviceKey` APIs, get the device key. +- Allocate `_scratch_buf` as a 32 byte array. +- Set `_is_initialized` to true. +- Release `_mutex`. + +**deinit function** + +Header: + +`virtual int deinit();` + +Pseudo code: + +- if not `_is_initialized`, return OK. +- Take `_mutex`. +- Deinitialize `_entropy`. +- Deallocate `_scratch_buf`. +- Release `_mutex`. + +**reset function** + +Header: + +`virtual int reset();` + +Pseudo code: + +- Take `_mutex`. +- Call `_underlying_kv` `reset` API. +- Call `_rbp_kv` `reset` API. +- Release `_mutex`. + +### Core APIs + +**set function** + +Header: + +`virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags);` + +Pseudo code: + +- Call `set_start` with all fields and a local `set_handle_t` variable. +- Call `set_add_data` with `buffer` and `size`. +- Call `set_finalize`. +- Return OK. + +**get function** + +Header: + +`virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0);` + +Pseudo code: + +- if not `_is_initialized` return error. +- Take `_mutex`. +- Call `_underlying_kv` `get` API with `metadata` size into a `metadata` local structure. +- If failure: + - If rollback protection or write once flag set: + - Call `_rbp_kv` `get` API on a local `rbp_cmac` variable, key is `key`, size 16. + - If no error, return "RBP authentication" error. + - Return "Key not found error". +- Derive a key from device key and `key`. +- Allocate and initialize `auth_handle` CMAC calculation local handle with derived key. +- Using `auth_handle` handle, calculate CMAC on `key` and `metadata`. +- If encrypt flag set: + - Derive a key from device key and `key`. + - Allocate and initialize a local `enc_handle` AES-CTR local handle with derived key and `iv` field. +- Set `data_size` local variable to data size in metadata. +- Set `actual_size` to the minimum of `buffer_size` and `data_size`. +- Set `current_offset` to 0. +- While `data_size` > 0: + - If `current_offset` between `offset` and `actual_size`. + - Set `dest_buf` to `buffer` and `chunk_size` to `actual_size`. + - Else: + - Set `dest_buf` to `_scratch_buf` and `chunk_size` to `actual_size`. + - Call `_underlying_kv` `get` API with `dest_buf` and `chunk_size`. + - Calculate CMAC on `dest_buf`, using `_auth_handle` handle. + - If encrypt flag set, decrypt `dest_buf` (in place) using `_enc_handle` handle. + - Decrement `data_size` by `chunk_size`. +- Call `_underlying_kv` `get` API with on a local `read_cmac` variable, size 16. +- Generate CMAC on local `cmac` variable . +- Using `mbedtls_ssl_safer_memcmp` function, compare `read_cmac` with `cmac`. Return "data corrupt error" if no match. +- If rollback protection or write once flags set: + - Call `_rbp_kv` `get` API on a local `rbp_cmac` variable, key is `key`, size 16. + - If `rbp_cmac` doesn't match `cmac`, clear `buffer` and return "RBP authentication" error. +- Deinitialize and free `auth_handle` and `enc_handle`. +- Release `_mutex`. +- Return OK. + +**get_info function** + +Header: + +`virtual int get_info(const char *key, info_t *info);` + +Pseudo code: + +- If not `_is_initialized`, return error. +- Call `get` API with `key` and 0 in `buffer_size` parameter. +- If failed, return error code. +- Call `_underlying_kv` `get` API with `metadata` size and `key`. +- Fill fields in `info` according to `metadata`. +- Return OK. + +**remove function** + +Header: + +`virtual int remove(const char *key);` + +Pseudo code: + +- If not `_is_initialized`, return error. +- Take `_mutex`. +- Call `_underlying_kv` `get` API with `metadata` size and `key`. +- If not found, return "Not found" error. +- If write once flag set, return "Already exists" error. +- Call `_underlying_kv` `remove` API with `key`. +- If rollback protect flag set, call `_rbp_kv` `remove` API with `key` as key. +- Return OK. + +### Incremental set APIs + +**set_start function** + +Header: + +`virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags);` + +Pseudo code: +- Take `_mutex`. +- Allocate an `inc_set_handle_t` and assign in handle. +- If flags include write once flag: + - Call `_underlying_kv` `get_info` API. + - If key exists, return "already exists" error. + - Call `_rbp_kv` `get` API with `key` as key. If key exists, return "already exists" error. +- If encrypt flag set: + - Derive a key from device key and `key` as salt (trimmed to 32 bytes with "ENC" as prefix). + - Using TLS entropy function on `_entropy` handle, randomly generate `iv` field. + - Allocate and initialize `enc_handle` AES-CTR handle field with derived key and `iv` field. +- Fill all available fields in `metadata`. +- Derive a key from device key and `key` as salt (trimmed to 32 bytes with "AUTH" as prefix). +- Allocate and initialize `auth_handle` CMAC calculation handle field with derived key. +- Using `auth_handle` handle, calculate CMAC on `key` and `metadata`. +- Call `_underlying_kv` `set_start` API. +- Call `_underlying_kv` `set_add_data` API with `metadata` field. +- Return OK. + +**set_add_data function** + +Header: + +`virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size);` + +Pseudo code: + +- If `offset` + `data_size` > data size in handle, return error. +- If flags include encryption: + - Iterate over `value_data` field in chunks of `_scratch_buf` size. + - Using `enc_handle` handle field, encrypt chunk into `_scratch_buf`. + - Using `auth_handle` handle field, update CMAC of `_scratch_buf`. + - Call `_underlying_kv` `set_add_data` API with `_scratch_buf`. +- Else: + - Using `auth_handle` handle field, update CMAC of `value_data`. + - Call `_underlying_kv` `set_add_data` API with `value_data`. +- Update `offset` field in handle. +- Return OK. + +**set_finalize function** + +Header: + +`virtual int set_finalize(set_handle_t handle);` + +Pseudo code: + +- Initialize a local `cmac` 16-byte array to 0. +- If authentication flag set, using `auth_handle` handle field, generate `cmac`. +- Call `_underlying_kv` `set_add_data` API with `cmac`. +- Call `_underlying_kv` `set_finalize`. +- If rollback protect or write once flags set, call `_rbp_kv` `set` API with `key` as key and `cmac` as data. +- Deinitialize and free `auth_handle` and `enc_handle`. +- Free `handle`. +- Release `_mutex`. +- Return OK. + +### Key iterator APIs + +**iterator_open function** + +Header: + +`virtual int iterator_open(iterator_t *it, const char *prefix = NULL);` + +Pseudo code: + +- Allocate a `key_iterator_handle_t` structure into `it`. +- Take `_mutex`. +- Call `_underlying_kv` `iterator_open` with `underlying_it` field. +- Release `_mutex`. +- Return OK. + +**iterator_next function** + +Header: + +`virtual int iterator_next(iterator_t it, char *key, size_t key_size);` + +Pseudo code: + +- Take `_mutex`. +- Call `_underlying_kv` `iterator_next` with `underlying_it` field. +- Release `_mutex`. +- Return OK. + +**iterator_close function** + +Header: + +`virtual int iterator_close(iterator_t it);` + +Pseudo code: + +- Take `_mutex`. +- Call `_underlying_kv` `iterator_close` with `underlying_it` field. +- Release `_mutex`. +- Deallocate `it`. +- Return OK. + +# Usage scenarios and examples + +### Standard use of the class + +The following example code shows standard use of the SecureStore class: + +**Standard usage example** + +```C++ +// Underlying key value store - here TDBStore (should be instantiated and initialized) +extern TDBStore tdbstore; + +// Rollback protect store - also of TDBStore type (should be instantiated and initialized) +extern TDBStore rbp_tdbstore; + +// Instantiate SecureStore with tdbstore as underlying key value store and rbp_tdbstore as RBP storage +SecureStore secure_store(&tdbstore, &rbp_tdbstore); + +int res; + +// Initialize secure_store +res = secure_store.init(); + +const char *val1 = "Value of key 1"; +const char *val2 = "Updated value of key 1"; +// Add "Key1" with encryption flag +res = secure_store.set("Key1", val1, sizeof(val1), KVSTore::REQUIRE_CONFIDENTIALITY_FLAG); +// Update value of "Key1" (flags must be the same per key) +res = secure_store.set("Key1", val2, sizeof(val2), KVSTore::REQUIRE_CONFIDENTIALITY_FLAG); + +uint_8 value[32]; +size_t actual_size; +// Get value of "Key1". Value should return the updated value. +res = secure_store.get("Key1", value, sizeof(value), &actual_size); + +// Remove "Key1" +res = secure_store.remove("Key1"); + +// Incremental write, if need to generate large data with a small buffer +const int data_size = 1024; +char buf[8]; + +KVSTore::set_handle_t handle; +res = secure_store.set_start(&handle, "Key2", data_size, 0); +for (int i = 0; i < data_size / sizeof(buf); i++) { + memset(buf, i, sizeof(buf)); + res = secure_store.set_add_data(handle, buf, sizeof(buf)); +} +res = secure_store.set_finalize(handle); + +// Iterate over all keys starting with "Key" +res = 0; +KVSTore::iterator_t it; +secure_store.iterator_open(&it, "Key*"); +char key[KVSTore::KV_MAX_KEY_LENGTH]; +while (!res) { + res = secure_store.iterator_next(&it, key, sizeof(key)e); +} +res = secure_store.iterator_close(&it); + +// Deinitialize SecureStore +res = secure_store.deinit(); +``` + +# Other information + +### Open issues + +- Need to figure a way to prevent mutex abuse in incremental set APIs. diff --git a/storage/docs/SecureStore/SecureStore_layers.jpg b/storage/docs/SecureStore/SecureStore_layers.jpg new file mode 100644 index 0000000..75b2c80 --- /dev/null +++ b/storage/docs/SecureStore/SecureStore_layers.jpg Binary files differ diff --git a/storage/docs/SecureStore/SecureStore_layers.xml b/storage/docs/SecureStore/SecureStore_layers.xml new file mode 100644 index 0000000..e4789c9 --- /dev/null +++ b/storage/docs/SecureStore/SecureStore_layers.xml @@ -0,0 +1 @@ +7VlZc9owEP41zLQP6WD5iPMYkh4znc50Ss9HIS+2JsKiQlz99ZWwZLAtBxoMpUdeYq3O3e/bQ6jn301WrwWeZu94AqyH+smq59/3EPLCEKl/WrIuJNdxUAhSQRMzaCsY0h9ghH0jndMEZpWBknMm6bQqJDzPgciKDAvBl9VhY86qu05xCg3BkGDWlH6hicwKaRz2t/I3QNPM7uz1Tc8Ik4dU8Hlu9ushf7z5K7on2K5lxs8ynPDljsh/2fPvBOey+Jqs7oBp21qzFfNetfSW5xaQy0MmRNfFjAVmc7BHjpiaO0joQh9Qro1Rou9zfarBmOfyaraB7FYN8ILpatupvlLzn9nBx68ysoKhlSiNRq2jgMwFDCUXUHaJ+mDXAkpWnLch3tjCSlFFIbRBG7Q9+6p7mVEJwykmunepvEPJMjlhquVptSljd5xxsZmruYEIUfKZFPwBdnqSaBSFUbnfLqIG5AUICasdkUH4NfAJSLFWQ2xvbNi2LulatJdbbgeWwdkOry3fsXGntFx6Syn1YVjVQsl+3Eoxyw7r/xZ/71EqPJVNryiD4XomYVLhxkFM6BDzBEM8dmIekRhG424wD4Ia5tdNzL3gZJjf7MW8o4jw8X7wD6CJbg5A03egWSamo+C08eKCXPh34T2OCbgj9igOg7B/Gu8t64WzeK/ntcJ9iUXBgHHycA8LSi4+3ychxEngYk+MRn7UUb5vsOessd9DR7PH74Q9j65SkuHt51/PH26edKhj4wjHKf3sNl8/36PNwS5zMt84SyZt1MIu33DVwh1lUt/hHDUjQp7c6jusauU8B80pPMs2VvWqFlRmEOuv2tovQtv8ZvpabQVJ7e4rsUjBjkGB23w75gkfiRwCGJZ0Ud3AZTGzw3tON8S2kcuvouOjmtFnfC4ImFm799o9C5Uw24UKpRsLbQAs1T4Q0z/rDv0/Xe4prp3p0lVcd5IuUXtt/Z89fwB76gnFyZ6T/biCXKV6zYazDE/1J5kLth4ITB50tN9nzK3li5ZUcZ3nqnl109UlB4XVGH0TNiyHHIYr/fU4ywWtfnfeOy0Xx9xlla1lFbgq200FsesaRoQZTTWeRCEISj7QyFGC2a3pmNAk0ds4mVJ1zC7Y4NXYEDX9yBWE6xXC08gQPh6E3SH1PPz4wBnTzwo9rekJIu6FqPlehRj9lPPXKKSvlfqZqfOc+Ne4fFn72Grf4fLlk2D3Ph9dSAL4pKyqMjPNUzXkKay5YH40yOCgTCs//NrvWL6jtOqKH6q5fXUtboHbp23/5U8= \ No newline at end of file diff --git a/storage/docs/SecureStore/SecureStore_record.jpg b/storage/docs/SecureStore/SecureStore_record.jpg new file mode 100644 index 0000000..f2de3c7 --- /dev/null +++ b/storage/docs/SecureStore/SecureStore_record.jpg Binary files differ diff --git a/storage/docs/SecureStore/SecureStore_record.xml b/storage/docs/SecureStore/SecureStore_record.xml new file mode 100644 index 0000000..df94e41 --- /dev/null +++ b/storage/docs/SecureStore/SecureStore_record.xml @@ -0,0 +1 @@ +7VhNj5swEP01SNvDStgESI6bZHfbQ6SqkdqzAw5YazA1zld/fe1gAiQQshEoomouMW/8Mbx5nrExrFm0f+coCRfMx9SApr83rLkBIbBtKP8UcsgQdzzKgIATX3cqgCX5gzVoanRDfJxWOgrGqCBJFfRYHGNPVDDEOdtVu60Zra6aoABfAEsP0Uv0F/FFmKFj2yzwr5gEYb4yMLVlhbyPgLNNrNczoLU+/jJzhPK5dP80RD7blSDr1bBmnDGRtaL9DFPFbU5bNu6twXrym+NY3DJgov3YIrrBucsOlWOnPtnKZqCaObTKgQUWyEcC5Qa5wKqms5mKg6bU+b1R7zRds1g8p8eAv8gOwEz2hfFiCn6O1C0ksRpfr7l/FFynMz7BL1e5qHuRynywwhY8SgirIJnSvAuJwMsEecq6k1tOYqGIqHwCp9HloGsdbDEXeF+CtAjeMYuw4AfZRVvHWgd6v4Jcn7tC/a6GwpLwcwzp/RacJi40Jxtadg0SHN0lwb418kZRkA5U30+jYasROu1ytPuSo9MoxxORP/CWpITF10j+l7MFsB+YLty70sW8pVp1X0T+54f+8gNs19+oJ/0BE9wlwG8/hyqW8bDFYoF2sTi9iaX5cHNDano0c247c8Dqjbq6QnzGiLw+JarpbTg9TLm8fGHRTk3BY/YkkFDF3Jo/T8yOEhRwKsy57gVxsIY3OJ50QVxzhVQJ5aZMY1/NNDlwfg28OQVlfjTIXHIsqgFLBWcfeMYo4xKJWYyVy4TSMwhREqg4ejJyWOJTFTEir/Uv2hAR31fL1Cqkur26OCZZoKKCU00q759JnQ462T+Tu+rUd+QPOt3b5uPOpiAvNZ/kfLZ4mQ31dACcYR8P7BuKXEfHA/lYfNk72kqfT63Xvw== \ No newline at end of file diff --git a/storage/docs/TDBStore/TDBStore_areas.jpg b/storage/docs/TDBStore/TDBStore_areas.jpg new file mode 100644 index 0000000..6d30463 --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_areas.jpg Binary files differ diff --git a/storage/docs/TDBStore/TDBStore_areas.xml b/storage/docs/TDBStore/TDBStore_areas.xml new file mode 100644 index 0000000..97ba384 --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_areas.xml @@ -0,0 +1 @@ +7Zhdk5owFIZ/DZedgURQL127u53p7M160esIR8hsIDRExf76JhA+zY52yth1p94Y3pOQk+c9AaOD12n5LEievPAImIPcqHTwVwchz/eR+tLKqVbmi1ktxIJGplMnbOgvMKJr1D2NoBh0lJwzSfOhGPIsg1AONCIEPw677TgbzpqTGM6ETUjYufqDRjKp1YXvdvo3oHHSzOy5JrIl4Vss+D4z8zkI76pPHU5Jcy/Tv0hIxI89CT86eC04l3UrLdfANNsGWz3u6Z1om7eATF4zIPDrEQfC9tCkXCUmTw2MajmgB7gOfjgmVMImJ6GOHpX9SktkytSVp5rnCZicDiAklD3JJPQMPAUpTqpL2dKsh5jiQb5J8thZgRamT9KzYWn6EeN+3N66I6AaBoIdiD+7DOQCAlLkdU3uaKmxTcHEHzGZuWdIsGdB0op/xcRWJAFTUzzsuFpSH07wc8+bwJei2tgr1cHz87ILqlZsvqu7bBthI0kWbfW6VwJIE1b5bcdDlFbP3cgjjxRXObSlkIK/wZozLpSS8Qx0mpSxkUQYjTN1GSq3QOkP2iWqng0rE0hpFOlprFthuFmmcD4YOo8tu8GbW6xHUzg/v5Xzq1DSA/w3vvcY/KfGL941vnXkhRSaE3JfIeQispv2wd4lwZWvEjwFw+Vlht9BJ+vdBbvllez8Cdg1tX+ZXacLy6Pi49L0ZjcsxcC7Fie+D3jzW9Yi+uy1qDjdsBbxZZxPAvRvgRrCNQQ/zVsfuSNrLE/d9rQ59Vs/sB1+Rta8QgHioJY8sufOSt5ygLJWPPrzM6W67A7wVaz3Lwl+/A0= \ No newline at end of file diff --git a/storage/docs/TDBStore/TDBStore_class_hierarchy.jpg b/storage/docs/TDBStore/TDBStore_class_hierarchy.jpg new file mode 100644 index 0000000..e865ee8 --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_class_hierarchy.jpg Binary files differ diff --git a/storage/docs/TDBStore/TDBStore_class_hierarchy.xml b/storage/docs/TDBStore/TDBStore_class_hierarchy.xml new file mode 100644 index 0000000..0bcd9d2 --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_class_hierarchy.xml @@ -0,0 +1 @@ +7VhLb9swDP41Prbwu9mxbtoVGAYMSPc6KhZtC5UtT1ZSZ79+ki35WbcNsGU5NIfE+kiRFEV+TGJ5N3n9kaMy+8wwUMu1cW15a8t1nSDw5YdCDi1ytdJAygnWSj2wIb9Bg7ZGdwRDNVIUjFFByjEYs6KAWIwwxDl7GqsljI69liiFGbCJEZ2j3wkWWYuuArvH74GkmfHs2FqyRfFjytmu0P4s10uaVyvOkbGl9asMYfY0gLxby7vhjIn2Ka9vgKrcmrS1++4WpF3cHArxpg06jj2iOzAhh1TujUoVnjjolIS/diqmKEc8JYXlXUupXdbyXYLNyRR+IVjZyvyBTEAtLhAlqd4Xy+CA9zblU6o/G89bAzyso41gHIxAnmM7VZZYOcUyrmI3VWXcOMseMdk/e9pMX7KK2lUnWjIgo2hsGNQdmXP3wAWR5XXdJmHdpCnSKVlTSNQuJrUS2lRDQuQteVHCCqGbw3H1+g7lhKq2uge6B2VVxSlyqpQ638oh1ItF4XSlJlsYWA6CH6SK3uB6uip097q+Xj/1veAYLBv0wUpjSLdf2pnuS1A+6CpcqEhnVpFfK5hlFAp8rRpdrlgJMouRRIa5wqjKAOukzPIDeEYAr2ZnePpQn5QDRYLsx7aeO74294UR6aXLtHc1yfQ0gxXb8Rj0rmEfTwwF9iuGhOxPEDNDzW10Z3zbBbnnTBkRZfHjGvYkfmeN07LGrARPyhr+ORflp2/vY+z/j7HVKesxeKYel0fYVrHWdGhJ+V2TwLU9H2+DVMl08MMPpXYZmOXPI6edLO5mQAy76TQTcIk1jp2Afjgx5P27CRieM9lEuyQBDvh9Ep4L8fgnnYRXs+J03vDtecQnI6apZOMIo40JylmBHzJVzq3I6PoG0KTVmJLEs9FuGRcZS1mB6G2PTu9wetFbJgTLlaWaiAHHydVP62X6s4+iv5ZmrNGP4hElhi9e+YV9GX5wvdG1O3+JIYNxNXXro38jOK8YWmDI3pBRZElSwZEsKpf93xqtev/fkXf7Bw== \ No newline at end of file diff --git a/storage/docs/TDBStore/TDBStore_design.md b/storage/docs/TDBStore/TDBStore_design.md new file mode 100644 index 0000000..2f53b3a --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_design.md @@ -0,0 +1,528 @@ +# TDBStore in Mbed OS + +- [TDBStore in Mbed OS](#tdbstore-in-mbed-os) + + [Revision history](#revision-history) +- [Introduction](#introduction) + + [Overview and background](#overview-and-background) + + [Requirements and assumptions](#requirements-and-assumptions) +- [System architecture and high-level design](#system-architecture-and-high-level-design) + * [Design basics](#design-basics) + + [Sequential writes](#sequential-writes) + + [Memory layout and areas](#memory-layout-and-areas) + + [Garbage collection](#garbage-collection) + + [RAM Table](#ram-table) +- [Detailed design](#detailed-design) + + [Class header](#class-header) + + [Important data structures](#important-data-structures) + + [Initialization and reset](#initialization-and-reset) + + [Core APIs](#core-apis) + + [Incremental set APIs](#incremental-set-apis) + + [Key iterator APIs](#key-iterator-apis) +- [Usage scenarios and examples](#usage-scenarios-and-examples) + + [Standard usage of the class](#standard-usage-of-the-class) +- [Other information](#other-information) + + [Open issues](#open-issues) + + +### Revision history + +| Revision | Date | Authors | Mbed OS version | Comments | +|---------- |---------------- |-------------------------------------------------------- |----------------- |------------------ | +| 1.0 | 16 September 2018 | David Saada ([@davidsaada](https://github.com/davidsaada/)) | 5.11+ | Initial revision | + +# Introduction + +### Overview and background + +Tiny Database Storage (TDBStore) is a lightweight module that stores data on flash storage. It is part of of the [KVStore](../KVStore/KVStore_design.md) class family, meaning it supports the get/set interface. It is designed to optimize performance (speed of access), reduce wearing of the flash and minimize storage overhead. It is also resilient to power failures. + +### Requirements and assumptions + +TDBStore assumes the underlying block device is fully dedicated to it (starting offset 0). If you want to dedicate only a part of the device to TDBStore, use a sliced block device, typically with `SlicingBlockDevice`. + +In addition, this feature requires a flash-based block device, such as `FlashIAPBlockDevice` or `SpifBlockDevice`. It can work on top of block devices that don't need erasing before writes, such as `HeapBlockDevice` or `SDBlockDevice`, but requires a flash simulator layer for this purpose, such as the one `FlashSimBlockDevice` offers. + +# System architecture and high-level design + +## Design basics + +TDBStore includes the following design basics: +- Sequential writes: TDBStore performs all writes sequentially on the physical storage as records, superseding the previous ones for the same key. +- Memory layout - areas: TDBStore divides the physical storage equally into two areas - active and standby. All writes are made to the end of the active area's free space. When the active area is exhausted, a garbage collection process is invoked, copying only the up to date values of all keys to the standby area, and turning it active. +- RAM table: Indexes all keys in RAM, thus allowing fast access to their records in the physical storage. + +### Sequential writes + +All writes occur sequentially on the physical storage as records, superseding the previous ones for the same key. Each data record is written right after the last written one. If a key is updated, a new record with this key is written, overriding the previous value of this key. If a key is deleted, a new record with a "deleted" flag is added. + +Writes expect the storage to be erased. However, TDBStore takes the "erase as you go" approach, meaning that when it crosses a sector boundary, it checks whether the next sector is erased. If not, it erases the next sector. This saves time during initialization and garbage collection. + +### Memory layout and areas + +![TDBStore Areas](./TDBStore_areas.jpg) + +Each key is stored in a separate record on the active area. The first record in the area is the master record. Its main purpose is to hold an area version, protecting you against a case in which there are two valid areas. (This can happen in the extreme cases of power failures.) + +![TDBStore Record](./TDBStore_record.jpg) + +A 24-byte header precedes a record key and data. Fields are: + +- Magic: a constant value, for quick validity checking. +- Header size: size of header. +- Revision: TDBStore revision (currently 1). +- User flags: Flags received from user. Currently only write once is dealt with (others are ignored). +- Internal flags: Internal TDBStore flags (currently only includes deleted flag). +- Key size: size of key. +- Data size: size of data. +- CRC: a 32-bit CRC, calculated on header (except CRC), key and data. +- Programming size pad: padding to the storage programming size. + +### Garbage collection + +Garbage collection (GC) is the process of compacting the records stored in the active area to the standby one, by copying only the most recent values of all the keys (without the ones marked as deleted). Then, the standby area becomes the active one and the previously active area is erased (not fully, only its first sector). + +GC is invoked in the following cases: + +- When the active area is exhausted. +- During initialization, when a corruption is found while scanning the active area. In this case, GC is performed up to the record preceding the corruption. + +### Reserved space + +The active area includes a fixed and small reserved space. This space is used for a quick storage and extraction of a write-once data (such as the device key). Its size is 32 bytes, aligned up to the underlying block device. Once it is written, nothing can modify it. It is also copied between the areas during garbage collection process. + +### RAM table + +All keys are indexed in memory using a RAM table. Key names are represented by a 32-bit hash. The table includes the hash (and sorted by it) and the offset to the key record in the block device. This allows both fast searching in the table and a low memory footprint. To keep code simple, the same CRC function used for recored validation is used for hash calculation (because TLS hash calculation is too heavy). + +![TDBStore RAM Table](./TDBStore_ram_table.jpg) + +Key names may produce duplicate hash values. This is OK because the hash is only used for fast access to the key, and the key needs to be verified when accessing the storage. If the key doesn't match, move to the next duplicate in the table. + +# Detailed design + +TDBStore fully implements the KVStore interface over a block device. Due to the fact it may write to the block device in program units that don't have to match the underlying device program units, it should use a `BufferedBlockDevice` for that purpose. + +![TDBStore Class Hierarchy](./TDBStore_class_hierarchy.jpg) + +Functionality, as defined by KVStore, includes the following: + +- Initialization and reset. +- Core actions: get, set and remove. +- Incremental set actions. +- Iterator actions. + +### Class header + +TDBStore has the following header: + +```C++ +class TDBStore : KVStore { + +public: + TDBSTore(BlockDevice *bd = 0); + virtual ~TDBSTore(); + + // Initialization and reset + virtual int init(); + virtual int deinit(); + virtual int reset(); + + // Core API + virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags); + virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0); + virtual int get_info(const char *key, info_t *info); + virtual int remove(const char *key); + + // Incremental set API + virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags); + virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size); + virtual int set_finalize(set_handle_t handle); + + // Key iterator + virtual int iterator_open(iterator_t *it, const char *prefix = NULL); + virtual int iterator_next(iterator_t it, char *key, size_t key_size); + virtual int iterator_close(iterator_t it); + + // Reserved space APIs + virtual int reserved_space_set(void *data); + virtual int reserved_space_get(void *data); + +private: + Mutex _mutex; + void *_ram_table; + size_t *_max_keys; + size_t *_num_keys; + BlockDevice *_bd; + bd_addr_t _free_space_offset; + BufferedBlockDevice *_buff_bd; + bool _is_initialized; + int _active_area; + + // Important internal functions + + // find record offset in flash + int find_record(const char *key, uint32_t *hash, bd_size_t *bd_offset, size_t ram_table_ind); + + // garbage collection + int garbage_collection(const char *key, const void *buffer, size_t size, uint32_t create_flags); +} +``` + +### Important data structures + +```C++ +// RAM table entry +typedef struct { + uint32_t hash; + bd_size_t bd_offset; +} ram_table_entry_t; + +// Record header +typedef struct { + uint32_t magic; + uint16_t header_size; + uint16_t revision; + uint32_t user_flags; + uint16_t int_flags; + uint16_t key_size; + uint32_t data_size; + uint32_t crc; +} record_header_t; + +// incremental set handle +typedef struct { + record_header_t header; + bd_size_t bd_base_offset; + bd_size_t bd_curr_offset; + uint32_t ram_table_ind; + uint32_t hash; + bool new_key; +} inc_set_handle_t; + +// iterator handle +typedef struct { + size_t ram_table_ind; + char *prefix; +} key_iterator_handle_t; +``` + +### Initialization and reset + +**init function** + +Header: + +`virtual int init();` + +Pseudo code: + +- If `_is_initialized` return OK. +- Take `_mutex`. +- Set `_max_keys` to an initial value of 32. +- Allocate `_ram_table` as an array of `_max_keys`. +- Allocate `_buff_bd` with `_bd` as the underlying block device, and initialize it. +- Check validity of master records on both areas. +- If one is valid, set its area as `_active_area`. +- If both are valid, set the one area whose master record has the higher version as `_active_area`. Erase first sector of the other one. +- If none are valid, set area 0 as `_active_area`, and write master record with version 0. +- Traverse active area until reaching an erased sector. + - Read current record and check its validity (calculte CRC). + - If not valid, perform garbage collection and exit loop. + - Advance `_free_space_offset`. + - Call `find_record` function to calculate hash and find key. + - If not found, add new RAM table entry with current hash. + - Update position of key in RAM table. +- Set `_is_initialized` to true. +- Release `_mutex`. + +**deinit function** + +Header: + +`virtual int deinit();` + +Pseudo code: + +- If not `_is_initialized`, return OK. +- Take `_mutex`. +- Deinitialize `_buff_bd`, and free it. +- Free `_ram_table`. +- Set `_is_initialized` to false. +- Release `_mutex`. + +**reset function** + +Header: + +`virtual int reset();` + +Pseudo code: + +- Take `_mutex`. +- Erase first sector in both areas. +- Set `_active_area` to 0. +- Write a master record with version 0. +- Set `_free_space_offset` to end of master record. +- Set `_num_keys` to 0. +- Release `_mutex`. + +### Core APIs + +**set function** + +Header: + +`virtual int set(const char *key, const void *buffer, size_t size, uint32_t create_flags);` + +Pseudo code: + +- if not `_is_initialized`, return "not initialized" error. +- Call `set_start` with all fields and a local `set_handle_t` variable. +- Call `set_add_data` with `buffer` and `size`. +- Call `set_finalize`. +- Return OK. + +**get function** + +Header: + +`virtual int get(const char *key, void *buffer, size_t buffer_size, size_t *actual_size = NULL, size_t offset = 0);` + +Pseudo code: + +- if not `_is_initialized`, return "not initialized" error. +- Take `_mutex`. +- Call `find_record` to find record in storage. +- If not found, return "not found" error. +- Read header, and calculate CRC on it. +- Update CRC with key (if offset is 0). +- Read data into user buffer, starting offset. Actual size is minimum of buffer size and remainder of data. +- If offset is 0, + - Update CRC with buffer. + - Compare calculate CRC with header CRC. Return "data corrupt" error if different. +- Release `_mutex`. +- Return OK. + +**get_info function** + +Header: + +`virtual int get_info(const char *key, info_t *info);` + +Pseudo code: + +- if not `_is_initialized`, return "not initialized" error. +- Take `_mutex`. +- Call `find_record` to find record in storage. +- If not found, return "not found" error. +- Read header. +- Copy relevant fields from header into structure. +- Release `_mutex`. +- Return OK. + +**remove function** + +Header: + +`virtual int remove(const char *key);` + +Pseudo code: + +- Call `set` function with `key`, delete flag set in flags and empty data + +### Incremental set APIs + +**set_start function** + +Header: + +`virtual int set_start(set_handle_t *handle, const char *key, size_t final_data_size, uint32_t create_flags);` + +Pseudo code: + +- Take `_mutex`. +- Check if final size fits in free space; if not, call `garbage_collection`. +- Call `find_record` to find record in storage and achieve `ram_table_ind` and `hash`. +- If found and `flags` field in header includes write once flag, return "write once" error. +- Set `new_key` field in handle to true if not found and delete key not set. +- Allocate an `inc_set_handle_t` structure into `handle`. +- Calculate hash on `key` and update in `handle`. +- Update `bd_base_offset` in handle to `_free_space_offset`. +- Update a `record_header_t` structure with all relevant values. +- Update all header fields in `handle`. +- Calculate CRC on header. +- Update `ram_table_ind` and `hash` in `handle`. +- Program key in position after header. +- Advance `_free_space_offset` and update in `bd_curr_offset` field in handle. +- Set `_free_space_offset`, and update in `bd_curr_offset` field in handle. +- Call `find_record` to calculate hash, and find record in storage (with null key and current hash). + +**set_add_data function** + +Header: + +`virtual int set_add_data(set_handle_t handle, const void *value_data, size_t data_size);` + +Pseudo code: + +- Calculate CRC on `value_data` and update in handle. +- Program `value_data` from `bd_curr_offset`. +- Advance `bd_curr_offset`. + +**set_finalize function** + +Header: + +`virtual int set_finalize(set_handle_t handle);` + +Pseudo code: + +- Advance `_free_space_offset` to padded offset. +- Update a `record_header_t` structure with all relevant values. +- Program header at `bd_base_offset` from handle with pads. +- Call `sync` on buffered block device. +- If delete flag set: + - Remove entry in index `ram_table_ind` from RAM table. +- Else if `new_key` field is true: + - If `_num_keys` = `_max_keys`: + - Increase _max_keys by 1. + - Duplicate ram table to with new `_max_keys` entries. + - Add entry `ram_table_ind`. +- Update `bd_offset` and `hash` in `ram_table_ind` position of RAM table. +- Free `handle`. +- Release `_mutex`. + +### Key iterator APIs + +**iterator_open function** + +Header: + +`virtual int iterator_open(iterator_t *it, const char *prefix = NULL);` + +Pseudo code: + +- Take `_mutex`. +- Allocate a `key_iterator_handle_t` structure into `it`. +- Set `ram_table_ind` field in iterator to 0. +- Duplicate `prefix` into same field in iterator. +- Release `_mutex`. + +**iterator_next function** + +Header: + +`virtual int iterator_next(iterator_t it, char *key, size_t key_size);` + +Pseudo code: + +- Take `_mutex`. +- While `ram_table_ind` field in iterator smaller than `_num_keys`: + - Read key RAM table points to in `ram_table_ind` into a local variable. + - If name matches prefix: + - Advance `ram_table_ind` field in iterator. + - Copy name to `key`, and return OK. + - Advance `ram_table_ind` field in iterator. +- Return "not found" error. +- Release `_mutex`. + +**iterator_close function** + +Header: + +`virtual int iterator_close(iterator_t it);` + +Pseudo code: + +- Release `prefix` field in iterator and structure allocated at `it`. + +### Reserved space + +**reserved_space_set function** + +Header: + +`virtual int reserved_space_set(void *data);` + +Pseudo code: + +- Check if reserved space is not empty; if it is, return a "reserved space programmed error". +- Copy `data` contents to reserved space location. + +**reserved_space_get function** + +Header: + +`virtual int reserved_space_get(void *data);` + +Pseudo code: + +- Copy contents from reserved space location `data`. + +# Usage scenarios and examples + +### Standard use of the class + +The following example code shows standard use of the TDBStore class: + +**Standard usage example** + +```C++ +// Underlying block device. Here, SPI Flash is fully used. +// One can use SlicingBlockDevice if we want a partition. +SPIFBlockDevice bd(PTE2, PTE4, PTE1, PTE5); + +// Instantiate tdbstore with our block device +TDBStore tdbstore(&bd); + +int res; + +// Initialize tdbstore +res = tdbstore.init(); + +// Add "Key1" +const char *val1 = "Value of key 1"; +const char *val2 = "Updated value of key 1"; +res = tdbstore.set("Key1", val1, sizeof(val1), 0); +// Update value of "Key1" +res = tdbstore.set("Key1", val2, sizeof(val2), 0); + +uint_8 value[32]; +size_t actual_size; +// Get value of "Key1". Value should return the updated value. +res = tdbstore.get("Key1", value, sizeof(value), &actual_size); + +// Remove "Key1" +res = tdbstore.remove("Key1"); + +// Incremental write, if need to generate large data with a small buffer +const int data_size = 1024; +char buf[8]; + +KVSTore::set_handle_t handle; +res = tdbstore.set_start(&handle, "Key2", data_size, 0); +for (int i = 0; i < data_size / sizeof(buf); i++) { + memset(buf, i, sizeof(buf)); + res = tdbstore.set_add_data(handle, buf, sizeof(buf)); +} +res = tdbstore.set_finalize(handle); + +// Iterate over all keys starting with "Key" +res = 0; +KVSTore::iterator_t it; +tdbstore.iterator_open(&it, "Key*"); +char key[KVSTore::KV_MAX_KEY_LENGTH]; +while (!res) { + res = tdbstore.iterator_next(&it, key, sizeof(key)); +} +res = tdbstore.iterator_close(&it); + +// Deinitialize TDBStore +res = tdbstore.deinit(); +``` + +# Other information + +### Open issues + +- Need to figure a way to prevent mutex abuse in incremental set APIs. diff --git a/storage/docs/TDBStore/TDBStore_ram_table.jpg b/storage/docs/TDBStore/TDBStore_ram_table.jpg new file mode 100644 index 0000000..f370dc2 --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_ram_table.jpg Binary files differ diff --git a/storage/docs/TDBStore/TDBStore_ram_table.xml b/storage/docs/TDBStore/TDBStore_ram_table.xml new file mode 100644 index 0000000..5638d79 --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_ram_table.xml @@ -0,0 +1 @@ +3Zlbb5swGIZ/DZedABdCL5P0JE3VpmbSuksHHLDq4Mw4DdmvnwEbAjgN3SAtrVTFvDa2ed7PB4wB5uv0jsFN9EADRAzbDFIDXBu2bTnOpfjJlH2hTDwphAwHslAlLPAfJEVTqlscoKRWkFNKON7URZ/GMfJ5TYOM0V292IqSeqsbGKKWsPAhaas/ccCjQvUcs9LvEQ4j1bJlypwl9J9DRrexbM+wwSr/K7LXUNUlyycRDOjuQAI3BpgzSnmRWqdzRDK2Cltx3+2R3LLfDMW8yw2OW9zxAskWqS7nHeN7BWMXYY4WG+hn1zthuAFmEV8TcWWJJEw2hQUrnCJR7azdB9mtF8Q4Sg8k2ac7RNeIs70oInMdFQgyfhSuXWWG7UktOjCiFKEMgLCsuoIgEpLDESYTDROXiCZmKyoe6RCO+3tLVcZFksfxVBSwwCatMkUqlL95LUslPE4fROEfcClqU3msWVp0d6nRiq4ouWGZwMzrLiWc0Wc0p4QyocQ0RlmvMSENCRIcxuLSF+Yhoc8y07AYGVOZscZBkDUz04VFHvlZEFyb/QSCV48DSxMI1kQXCH3EgXc0DkpDHmCSYbLNR+RTFug9a5hTh3RidL3jYAJ9MLw6zfAryjprjYKd05Gd0wM71+zKzu46e3wwmuWyeY5QdK2uOME44IFzxqL96WPRO2csgtM4bxkSuwKzgNCF4KdZ9F2zsWBpIr2cOvpe9d3ju7/2imXewyQ6X8C3EGpAv0L11C6g3F31He6TyzfMHiNmqtkdDMfUecNyNmKmmml5OKZvGvvfVqsE8ZFQtbyTw3+oxW7S4T2qGv2jpqqZAAaj2vnNCoydqmYKGIqqp1upGjhQHEyzM8Zsw0RgkmC/TkA8Jds/SVr5xa/s4otT0kFB6/ixtV3ikIVI7eCv9LwOgDgaIEpjiECOX+pN6ijJFr5TnB9zSTuaMzJw6jUkdMt8JG+qULfqAbZ+ClL1FE/cqie3rHzobi7q1sZ3d7F4h/ooLrr/6KJlNaNhOBs7HE6ftDHF/OkgXZnYj8EFtvqUXDP9yFR3HtMbnl+aDa8+ouk9nL47nU7fF5yy7DuQbV6I/6mfcxYJhuBnOXv/r2XYUa8YyvQBD9/FZfXRq4ia6ssiuPkL \ No newline at end of file diff --git a/storage/docs/TDBStore/TDBStore_record.jpg b/storage/docs/TDBStore/TDBStore_record.jpg new file mode 100644 index 0000000..acb4a9a --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_record.jpg Binary files differ diff --git a/storage/docs/TDBStore/TDBStore_record.xml b/storage/docs/TDBStore/TDBStore_record.xml new file mode 100644 index 0000000..fd8b2b4 --- /dev/null +++ b/storage/docs/TDBStore/TDBStore_record.xml @@ -0,0 +1 @@ +7VnbrqIwFP0ak5mHM4GWiz4qnksyOcmJZjLPFSo0VuqUepuvnyJFQAE9RIdhoi+WtXvZrr26uys96Cx3rxytgnfmYdoDmrfrwXEPAN00gfyKkX2C2H0jAXxOPNUpA6bkN1agptA18XBU6CgYo4KsiqDLwhC7ooAhztm22G3OaHHVFfLxGTB1ET1HfxJPBAnaN7UMf8PED9KVdU1ZZshd+JytQ7VeD8D54ZOYlyidS/WPAuSxbQ6Czz3ocMZE0lruHExjblPaknEvFdaj3xyH4poBlp2M2CC6xqnLFpVjRx7ZyKYfN1NolgLvyCduisrZZyU9tUjsFZ/Wr3X8g0ZzFoqn6BDtoeyga6tdZjybgp8iZQtJrMTROt+/GF9rPS9btjAfKPw2cIg2jvnUpHkbEIGnK+TG1q3cHRILxJLKJ/04Oh8fFbIN5gLvcpCK1ytmSyz4XnZR1r6SjtpaeiqlbSZUQ0FBTqO2wpDaGv5x4kwesqEUUq6WgdZILW8YeZh3VC6H5HRbAYJuC/CouBoFmvdSoF6pwCORE7z5Vkfwfx0bu8XYGJdj8yOqTwR1sXmhyI86mkW6fuiANve82ejUmWCptY3kqKOK6XgmAldkIvteirEaKeY73t+vdnjopFwn0Gixnm12+xkjgf5ukfkv56mOn2ywxfuUrlWXs3WcOxPnoZZW1GJcUQfdTy2gkVo+OPNrL0RN85X2MRx3O5pWm9GEl69LlRVJy8SZ2mXidHg35q64aFaf0S1TZ8FWqWt2l3qkkKpw2q2mkLJ7zgkhUYBWcdNdc7ofceQusLjMTEZj8iSQICyUj08D7TbEAQsUiIP6GW+ghDfY129BXHXhH5dJV9VPZm39lAIT7DLuyf7FP8Gvrq8SdyrELqkWxbhFgrMFdhhlXCIhC3HsOaH0BEKU+HE4XRlA6RUcxYEjLqJDZVgSz4uXKRVKcZPdQAy6bRfEcNwy+aQ4KJED+Pw2ko/Z+7WDLfcSEz7/AQ== \ No newline at end of file