Merge branch 'for-next/state'

state: backend_bucket_circular: promote old state debug message to info ...

A report about the changed on-storage state format might be interesting
to the user.

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>

state: backend_bucket_circular: fix old state reads ...

When the circular bucket meta magic was not found, it is assumed that
the whole written state is in the old on-storage format. In that case,
the size of the circular bucket meta must not be subtracted from the
read length, or the (complete_len > len) check in the raw backend's
verify function will fail:

    state: Error, invalid data_len 16 in header, have data of len 24

Signed-off-by: Philipp Zabel <p.zabel@pengutronix.de>
Signed-off-by: Lucas Stach <l.stach@pengutronix.de>

Merge branch 'for-next/misc'

state: backend_bucket_circular: drop unused free_pattern constant ...

Signed-off-by: Antony Pavlov <antonynpavlov@gmail.com>
Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: backend_bucket_circular: Do not leak memory ...

buf was just allocated, free it before returning an error.

Reported-by: Sam Ravnborg <sam@ravnborg.org>
Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: Remove -EUCLEAN check from userspace tool ...

The state code is used for the userspace tool aswell, kept in sync
manually. This patch only introduces a change for the userspace
tool, not for barebox.

In Linux userspace there is no direct possibility to check for -EUCLEAN.
To indirectly check for -EUCLEAN the state tool reads the number of
corrected bits before and after reading a block. Unfortunately it does
not take the number of acceptable bitflips into account, but instead
returns -EUCLEAN even when only a single bitflip occurred on a whole
page. To be correct the algorithm must be more complicated: We would
have to read the bitflip_threshold from sysfs. This value is per ECC
step (often 512 byte), not per page. We would have to read the page
in ECC step size chunks, testing for bitflips lower than the threshold
for each chunk. Even if we would do that, there's still another issue.
The eccstats ioctl delivers the eccstats for the whole device, so a
concurrent reader would falsify the result.

Let's decide that this is not worth the hassle and assume that no
device has enough uptime that a cleanup in barebox is not sufficient.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: backend bucket circular: Explain metadata ...

Explain why we have metadata and where it is used.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: backend_circular: Set minumum writesize to 8 ...

NOR flashes have a write size of 1. With this the metadata may
end up on non-4-byte-aligned offsets. Force the minimum writesize
to 8 so that the metadata is always at aligned offsets.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: backend: Add some documentation ...

Write some sentences to make the concepts clearer.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: Convert all bufs to void * ...

A void * is a much better type for a buffer than a u8 * as it
can be casted to any other type implicitly. Convert all buffers
used by the state framework to void *.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: replace len_hint logic ...

The len_hint mechanism is rather hard to understand as it's not clear
from where to where the hint is passed and also it's not clear what
happens if the hint is empty or wrong.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: drop lazy_init ...

lazy_init is an optimization that makes it possible to read only up to
the first valid bucket when starting. However, when restoring consistency,
immediately afterwards we have we have to initialize all buckets anyway,
so being lazy doesn't give us any gain. Remove it to simplify the code.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: backend_circular: Read whole PEB ...

When the circular backend searches for the last page written in the
eraseblock, it iterates backwards pagewise from the end of the block.
This is ok for NAND flash, but on NOR flash, which does not have pages,
the code ends up iterating bytewise backwards, calling into mtd each
time. This is very time consuming, so optimize this by reading the whole
eraseblock once and just iterate over the buffer in memory.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: backend circular: Fix wrong comment ...

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: bucket_circular: -EUCLEAN means data could be read ...

-EUCLEAN returned from state_mtd_peb_read() means that the
data shall still be used. This fixes initialization of buckets
which need cleanup

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: use packed attribute for on storage structs ...

These structs are used for on-storage data layouts. They should be not
affected by different integer precisions and alignment optimizations of
32bit or 64bit machines. Using the architecture independent integer data
types, like uint32_t, achieves the former, using the packed attribute
the later.

Signed-off-by: Stefan Lengfeld <s.lengfeld@phytec.de>
Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>

state: Refactor state framework ...

The state framework grew organically over the time. Unfortunately the
architecture and abstractions disappeared during this period.

This patch refactors the framework to recreate the abstractions. The
main focus was the backend with its storage. The main use-case was to
offer better NAND support with less erase cycles and interchangeable
data formats (dtb,raw).

The general architecture now has a backend which consists of a data
format and storage. The storage consists of multiple storage buckets
each holding exactly one copy of the state data. A data format describes
a data serialization for the state framework. This can be either dtb or
raw. A storage bucket is a storage location which is used to store any
data. There is a (new) circular type which writes changes behind the
last written data and therefore reduces the number of erases. The other
type is a direct bucket which writes directly to a storage offset for
all non-erase storage.

Furthermore this patch splits up all classes into different files in a
subdirectory.

This is currently all in one patch as I can't see a good way to split
the changes up without having a non-working state framework in between.

The following diagram shows the new architecture roughly:

           .----------.
           |  state   |
           '----------'
                 |
                 |
                 v
  .----------------------------.
  |       state_backend        |
  |----------------------------|
  | + state_load(*state);      |
  | + state_save(*state);      |
  | + state_backend_init(...); |
  |                            |
  |                            |
  '----------------------------'
    |            |                   The format describes
    |            |                   how the state data
    |            '------------->     is serialized
    |   .--------------------------------------------.
    |   |      state_backend_format <INTERFACE>      |
    |   |--------------------------------------------|
    |   | + verify(*format, magic, *buf, len);       |
    |   | + pack(*format, *state, **buf, len);       |
    |   | + unpack(*format, *state, *buf, len);      |
    |   | + get_packed_len(*format, *state);         |
    |   | + free(*format);                           |
    |   '--------------------------------------------'
    |              ^                      ^
    |              *                      *
    |              *                      *
    |   .--------------------. .--------------------.
    |   | backend_format_dtb | | backend_format_raw |
    |   '--------------------' '--------------------'
    |
    |
    |
    v
.----------------------------------------------------------.
|                  state_backend_storage                   |
|----------------------------------------------------------|
| + init(...);                                             |
| + free(*storage);                                        |
| + read(*storage, *format, magic, **buf, *len, len_hint); |
| + write(*storage, *buf, len);                            |
| + restore_consistency(*storage, *buf, len);              |
'----------------------------------------------------------'
                              |
     The backend storage is responsible to manage multiple
     data copies and distribute them onto several buckets.
     Read data is verified against the given format to
     ensure that the read data is correct.
                              |
                              |
                              |
                              |
                              |
                              v
        .------------------------------------------.
        | state_backend_storage_bucket <INTERFACE> |
        |------------------------------------------|
        | + init(*bucket);                         |
        | + write(*bucket, *buf, len);             |
        | + read(*bucket, **buf, len_hint);        |
        | + free(*bucket);                         |
        '------------------------------------------'
                      ^     ^      ^
                     *      *       *
                    *       *        *
 A storage bucket represents*exactly one data copy at one
 data location. A circular b*cket writes any new data to
 the end of the bucket (for *educed erases on NAND). A
 direct bucket directly writ*s at one location.
               *            *             *
              *             *              *
             *              *               *
 .-----------------------.  *  .-------------------------.
 | backend_bucket_direct |  *  | backend_bucket_circular |
 '-----------------------'  *  '-------------------------'
             ^              *               ^
             |              *               |
             |              *               |
             |              *               |
             |  .-----------------------.   |
             '--| backend_bucket_cached |---'
                '-----------------------'
             A backend_bucket_cached is a transparent
             bucket that directly uses another bucket
             as backend device and caches all accesses.

Signed-off-by: Markus Pargmann <mpa@pengutronix.de>
Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>