Examples testing script

The scripts in this folder are used for testing mbed-os official examples. It contains the following files:

examples.py - the main script that serves as the command-line interface.
examples.json - the default configuration file for the script, which contains the information to the examples. If required, you can pass a customized configuration file as an argument to the script.
examples_lib.py - the library file, which contains the main function of the testing scripts.

The scripts

examples.py provides command-line interface and subcommands that makes easier to test examples. Included subcommands are:

import - imports each of the repos and its dependencies (.lib files) associated with the specific examples name from the .json configuration file. If there is already a clone of the repo, it will first be removed to ensure a clean, up-to-date cloning.
clone - clones each of the repos associated with the specific examples name from the .json configuration file. If there is already a clone of the repo, it will first be removed to ensure a clean, up-to-date cloning.
deploy - if the example directory exists as provided by the .json configuration file, pulls in the examples dependencies by using mbed-cli deploy.
update - for each example repo identified in the config .json object, updates the version of example to that specified by the supplied GitHub tag. This function assumes that each example repo has already been cloned.
compile - compiles combinations of example programs, targets and compile chains.
export - exports and builds combinations of example programs, targets and IDEs.
list - displays examples in a configuration file in a table.
symlink - creates a symbolic link to a given mbed-os PATH.

For more detailed options, please use -h or --help.

The configuration file

Here is the section of default configuration file:

 {
  "examples": [
    {
      "name": "mbed-os-example-blinky",
      "github": "https://github.com/ARMmbed/mbed-os-example-blinky",
      "sub-repo-example": false,
      "subs": [],
      "features" : [],
      "targets" : [],
      "toolchains" : [],
      "exporters": [],
      "compile" : true,
      "export": true,
      "test" : true,
      "baud_rate": 9600,
      "compare_log": ["mbed-os-example-blinky/tests/blinky.log"],
      "auto-update" : true
    },
      
    ...
    
    {
      "name": "mbed-os-example-tls",
      "github": "https://github.com/ARMmbed/mbed-os-example-tls",
      "sub-repo-example": true,
      "subs": [
          "benchmark",
          "tls-client",
          "hashing",
          "authcrypt"
      ],
      "features" : [],
      "targets" : ["K66F", "NUCLEO_F429ZI"],
      "toolchains" : ["GCC_ARM", "ARM"],
      "exporters": [],
      "compile" : false,
      "export": false,
      "test" : false,
      "baud_rate": 9600,
      "compare_log": [
          "mbed-os-example-tls/tests/benchmark.log",
          "mbed-os-example-tls/tests/tls-client.log",
          "mbed-os-example-tls/tests/hashing.log",
          "mbed-os-example-tls/tests/authcrypt.log"
      ],
      "auto-update" : true
    }
  ]
}

Fields

name - name of the example. It should be the base name of the example repository address and will throw if it doesn't match.
github - example GitHub repository address.
sub-repo-example - specifies if the example repository has a subfolder for each example.
subs - array of subexample names.
features - the features that must be in the features array of a target in targets.json.
baud_rate - example default baud rate.
compare_log - array of log compared to command-line output during testing. If the example has many subexamples, the order of log should match the order of subexamples.
targets - list of mbed-os development boards that run this example.
targets - list of targeted development boards.
toolchains - toolchain to use for compiling.
exporters - allowed exporters.
compile - enables compiling.
export - enables exporting.
test - enables testing.

Values

[ ] means all possible alternatives.

Typical use

In the Mbed OS CI, we follow the below steps to compile and test Mbed OS examples.

Clone mbed-os repository to the current folder:

git clone https://github.com/ARMmbed/mbed-os.git

Clone the examples repo to the current folder. Users can pass an -e option to the script to filter out the rest of the examples, so the scripts only run on one particular example:
```
python mbed-os/tools/test/examples/examples.py clone
```
Create a symbolic link to mbed-os for every example. This step lets all the examples share a single mbed-os folder, rather than checking out the mbed-os folder many times. We highly recommend you pass an absolute path as the argument:
```
python mbed-os/tools/test/examples/examples.py symlink $PWD/mbed-os
```

Deploy other dependency libraries:

python mbed-os/tools/test/examples/examples.py deploy

Compile the test for the examples on a specific target:

python mbed-os/tools/test/examples/examples.py compile -m <target>

After the compile test finished, the scripts print the result table:

Passed example compilation:
+---------------------------------+--------+-----------+----------+--------------+
| EXAMPLE NAME                    | TARGET | TOOLCHAIN | TEST GEN | BUILD RESULT |
+---------------------------------+--------+-----------+----------+--------------+
| mbed-os-example-kvstore         |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-tls-socket      |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-blockdevice     |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-wifi            |  K64F  |  GCC_ARM  | TEST_OFF |    PASSED    |
| mbed-os-example-error-handling  |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-sd-driver       |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-crash-reporting |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-filesystem      |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-blinky          |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-cpu-stats       |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-sys-info        |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
| mbed-os-example-attestation     |  K64F  |  GCC_ARM  | TEST_ON  |    PASSED    |
+---------------------------------+--------+-----------+----------+--------------+
Number of failures = 0

After the compilation stage, a test_spec.json file is generated. Later, Greentea tests will consume this file. They test the compiled example on hardware platform.