misc_helpers: fix zero_normalmem() for BL2_AT_EL3 ...

The assertion in zero_normalmem() fails for BL2_AT_EL3.  This mode is
executed in EL3, so it should check sctlr_el3 instead of sctlr_el1.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

Introduce functions to disable the MMU in EL1 ...

The implementation is the same as those used to disable it in EL3.

Change-Id: Ibfe7e69034a691fbf57477c5a76a8cdca28f6b26
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>

Exit early if size zero for cache helpers ...

This patch enables cache helper functions `flush_dcache_range`,
`clean_dcache_range` and `invalidate_dcache_range` to exit early
if the size argument specified is zero

Change-Id: I0b63e8f4bd3d47ec08bf2a0b0b9a7ff8a269a9b0
Signed-off-by: Soby Mathew <soby.mathew@arm.com>

Use SPDX license identifiers ...

To make software license auditing simpler, use SPDX[0] license
identifiers instead of duplicating the license text in every file.

NOTE: Files that have been imported by FreeBSD have not been modified.

[0]: https://spdx.org/

Change-Id: I80a00e1f641b8cc075ca5a95b10607ed9ed8761a
Signed-off-by: dp-arm <dimitris.papastamos@arm.com>

Remove build option `ASM_ASSERTION` ...

The build option `ENABLE_ASSERTIONS` should be used instead. That way
both C and ASM assertions can be enabled or disabled together.

All occurrences of `ASM_ASSERTION` in common code and ARM platforms have
been replaced by `ENABLE_ASSERTIONS`.

ASM_ASSERTION has been removed from the user guide.

Change-Id: I51f1991f11b9b7ff83e787c9a3270c274748ec6f
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>

Replace ASM signed tests with unsigned ...

ge, lt, gt and le condition codes in assembly provide a signed test
whereas hs, lo, hi and ls provide the unsigned counterpart. Signed tests
should only be used when strictly necessary, as using them on logically
unsigned values can lead to inverting the test for high enough values.
All offsets, addresses and usually counters are actually unsigned
values, and should be tested as such.

Replace the occurrences of signed condition codes where it was
unnecessary by an unsigned test as the unsigned tests allow the full
range of unsigned values to be used without inverting the result with
some large operands.

Change-Id: I58b7e98d03e3a4476dfb45230311f296d224980a
Signed-off-by: Douglas Raillard <douglas.raillard@arm.com>

Introduce unified API to zero memory ...

Introduce zeromem_dczva function on AArch64 that can handle unaligned
addresses and make use of DC ZVA instruction to zero a whole block at a
time. This zeroing takes place directly in the cache to speed it up
without doing external memory access.

Remove the zeromem16 function on AArch64 and replace it with an alias to
zeromem. This zeromem16 function is now deprecated.

Remove the 16-bytes alignment constraint on __BSS_START__ in
firmware-design.md as it is now not mandatory anymore (it used to comply
with zeromem16 requirements).

Change the 16-bytes alignment constraints in SP min's linker script to a
8-bytes alignment constraint as the AArch32 zeromem implementation is now
more efficient on 8-bytes aligned addresses.

Introduce zero_normalmem and zeromem helpers in platform agnostic header
that are implemented this way:
* AArch32:
	* zero_normalmem: zero using usual data access
	* zeromem: alias for zero_normalmem
* AArch64:
	* zero_normalmem: zero normal memory  using DC ZVA instruction
	                  (needs MMU enabled)
	* zeromem: zero using usual data access

Usage guidelines: in most cases, zero_normalmem should be preferred.

There are 2 scenarios where zeromem (or memset) must be used instead:
* Code that must run with MMU disabled (which means all memory is
  considered device memory for data accesses).
* Code that fills device memory with null bytes.

Optionally, the following rule can be applied if performance is
important:
* Code zeroing small areas (few bytes) that are not secrets should use
  memset to take advantage of compiler optimizations.

  Note: Code zeroing security-related critical information should use
  zero_normalmem/zeromem instead of memset to avoid removal by
  compilers' optimizations in some cases or misbehaving versions of GCC.

Fixes ARM-software/tf-issues#408

Change-Id: Iafd9663fc1070413c3e1904e54091cf60effaa82
Signed-off-by: Douglas Raillard <douglas.raillard@arm.com>

Fix integer comparison in memcpy16 ...

Unsigned conditions should be used instead of signed ones when comparing
addresses or sizes in assembly.

Signed-off-by: Douglas Raillard <douglas.raillard@arm.com>
Change-Id: Id3bd9ccaf58c37037761af35ac600907c4bb0580

Refactor the xlat_tables library code ...

The AArch32 long descriptor format and the AArch64 descriptor format
correspond to each other which allows possible sharing of xlat_tables
library code between AArch64 and AArch32. This patch refactors the
xlat_tables library code to seperate the common functionality from
architecture specific code. Prior to this patch, all of the xlat_tables
library code were in `lib/aarch64/xlat_tables.c` file. The refactored code
is now in `lib/xlat_tables/` directory. The AArch64 specific programming
for xlat_tables is in `lib/xlat_tables/aarch64/xlat_tables.c` and the rest
of the code common to AArch64 and AArch32 is in
`lib/xlat_tables/xlat_tables_common.c`. Also the data types used in
xlat_tables library APIs are reworked to make it compatible between AArch64
and AArch32.

The `lib/aarch64/xlat_tables.c` file now includes the new xlat_tables
library files to retain compatibility for existing platform ports.
The macros related to xlat_tables library are also moved from
`include/lib/aarch64/arch.h` to the header `include/lib/xlat_tables.h`.

NOTE: THE `lib/aarch64/xlat_tables.c` FILE IS DEPRECATED AND PLATFORM PORTS
ARE EXPECTED TO INCLUDE THE NEW XLAT_TABLES LIBRARY FILES IN THEIR MAKEFILES.

Change-Id: I3d17217d24aaf3a05a4685d642a31d4d56255a0f

Remove xlat_helpers.c ...

lib/aarch64/xlat_helpers.c defines helper functions to build
translation descriptors, but no common code or upstream platform
port uses them. As the rest of the xlat_tables code evolves, there
may be conflicts with these helpers, therefore this code should be
removed.

Change-Id: I9f5be99720f929264818af33db8dada785368711

Merge pull request #542 from sandrine-bailleux-arm/km/pt-zero ...

Initialize all translation table entries

Initialize all translation table entries ...

The current translation table code maps in a series of regions, zeroing
the unmapped table entries before and in between the mapped regions. It
doesn't, however, zero the unmapped entries after the last mapped
region, leaving those entries at whatever value that memory has
initially.

This is bad because those values can look like valid translation table
entries, pointing to valid physical addresses. The CPU is allowed to do
speculative reads from any such addresses. If the addresses point to
device memory, the results can be unpredictable.

This patch zeroes the translation table entries following the last
mapped region, ensuring all table entries are either valid or zero
(invalid).

In addition, it limits the value of ADDR_SPACE_SIZE to those allowed by
the architecture and supported by the current code (see D4.2.5 in the
Architecture Reference Manual). This simplifies this patch a lot and
ensures existing code doesn't do unexpected things.

Change-Id: Ic28b6c3f89d73ef58fa80319a9466bb2c7131c21

Extend memory attributes to map non-cacheable memory ...

At the moment, the memory translation library allows to create memory
mappings of 2 types:

 - Device nGnRE memory (named MT_DEVICE in the library);

 - Normal, Inner Write-back non-transient, Outer Write-back
   non-transient memory (named MT_MEMORY in the library).

As a consequence, the library code treats the memory type field as a
boolean: everything that is not device memory is normal memory and
vice-versa.

In reality, the ARMv8 architecture allows up to 8 types of memory to
be used at a single time for a given exception level. This patch
reworks the memory attributes such that the memory type is now defined
as an integer ranging from 0 to 7 instead of a boolean. This makes it
possible to extend the list of memory types supported by the memory
translation library.

The priority system dictating memory attributes for overlapping
memory regions has been extended to cope with these changes but the
algorithm at its core has been preserved. When a memory region is
re-mapped with different memory attributes, the memory translation
library examines the former attributes and updates them only if
the new attributes create a more restrictive mapping. This behaviour
is unchanged, only the manipulation of the value has been modified
to cope with the new format.

This patch also introduces a new type of memory mapping in the memory
translation library: MT_NON_CACHEABLE, meaning Normal, Inner
Non-cacheable, Outer Non-cacheable memory. This can be useful to map
a non-cacheable memory region, such as a DMA buffer for example.

The rules around the Execute-Never (XN) bit in a translation table
for an MT_NON_CACHEABLE memory mapping have been aligned on the rules
used for MT_MEMORY mappings:
 - If the memory is read-only then it is also executable (XN = 0);
 - If the memory is read-write then it is not executable (XN = 1).

The shareability field for MT_NON_CACHEABLE mappings is always set as
'Outer-Shareable'. Note that this is not strictly needed since
shareability is only relevant if the memory is a Normal Cacheable
memory type, but this is to align with the existing device memory
mappings setup. All Device and Normal Non-cacheable memory regions
are always treated as Outer Shareable, regardless of the translation
table shareability attributes.

This patch also removes the 'ATTR_SO' and 'ATTR_SO_INDEX' #defines.
They were introduced to map memory as Device nGnRnE (formerly called
"Strongly-Ordered" memory in the ARMv7 architecture) but were not
used anywhere in the code base. Removing them avoids any confusion
about the memory types supported by the library.

Upstream platforms do not currently use the MT_NON_CACHEABLE memory
type.

NOTE: THIS CHANGE IS SOURCE COMPATIBLE BUT PLATFORMS THAT RELY ON THE
BINARY VALUES OF `mmap_attr_t` or the `attr` argument of
`mmap_add_region()` MAY BE BROKEN.

Change-Id: I717d6ed79b4c845a04e34132432f98b93d661d79

Use tf_printf() for debug logs from xlat_tables.c ...

The debug prints used to debug translation table setup in xlat_tables.c
used the `printf()` standard library function instead of the stack
optimized `tf_printf()` API. DEBUG_XLAT_TABLE option was used to enable
debug logs within xlat_tables.c and it configured a much larger stack
size for the platform in case it was enabled. This patch modifies these
debug prints within xlat_tables.c to use tf_printf() and modifies the format
specifiers to be compatible with tf_printf(). The debug prints are now enabled
if the VERBOSE prints are enabled in Trusted Firmware via LOG_LEVEL build
option.

The much larger stack size definition when DEBUG_XLAT_TABLE is defined
is no longer required and the platform ports are modified to remove this
stack size definition.

Change-Id: I2f7d77ea12a04b827fa15e2adc3125b1175e4c23

Remove direct usage of __attribute__((foo)) ...

Migrate all direct usage of __attribute__ to usage of their
corresponding macros from cdefs.h.
e.g.:
 - __attribute__((unused)) -> __unused

Signed-off-by: Soren Brinkmann <soren.brinkmann@xilinx.com>

Make generic code work in presence of system caches ...

On the ARMv8 architecture, cache maintenance operations by set/way on the last
level of integrated cache do not affect the system cache. This means that such a
flush or clean operation could result in the data being pushed out to the system
cache rather than main memory. Another CPU could access this data before it
enables its data cache or MMU. Such accesses could be serviced from the main
memory instead of the system cache. If the data in the sysem cache has not yet
been flushed or evicted to main memory then there could be a loss of
coherency. The only mechanism to guarantee that the main memory will be updated
is to use cache maintenance operations to the PoC by MVA(See section D3.4.11
(System level caches) of ARMv8-A Reference Manual (Issue A.g/ARM DDI0487A.G).

This patch removes the reliance of Trusted Firmware on the flush by set/way
operation to ensure visibility of data in the main memory. Cache maintenance
operations by MVA are now used instead. The following are the broad category of
changes:

1. The RW areas of BL2/BL31/BL32 are invalidated by MVA before the C runtime is
   initialised. This ensures that any stale cache lines at any level of cache
   are removed.

2. Updates to global data in runtime firmware (BL31) by the primary CPU are made
   visible to secondary CPUs using a cache clean operation by MVA.

3. Cache maintenance by set/way operations are only used prior to power down.

NOTE: NON-UPSTREAM TRUSTED FIRMWARE CODE SHOULD MAKE EQUIVALENT CHANGES IN
ORDER TO FUNCTION CORRECTLY ON PLATFORMS WITH SUPPORT FOR SYSTEM CACHES.

Fixes ARM-software/tf-issues#205

Change-Id: I64f1b398de0432813a0e0881d70f8337681f6e9a

Add support to indicate size and end of assembly functions ...

In order for the symbol table in the ELF file to contain the size of
functions written in assembly, it is necessary to report it to the
assembler using the .size directive.

To fulfil the above requirements, this patch introduces an 'endfunc'
macro which contains the .endfunc and .size directives. It also adds
a .func directive to the 'func' assembler macro.

The .func/.endfunc have been used so the assembler can fail if
endfunc is omitted.

Fixes ARM-Software/tf-issues#295

Change-Id: If8cb331b03d7f38fe7e3694d4de26f1075b278fc
Signed-off-by: Kévin Petit <kevin.petit@arm.com>

Add macro to calculate number of elements in an array ...

This patch defines the ARRAY_SIZE macro for calculating number of elements
in an array and uses it where appropriate.

Change-Id: I72746a9229f0b259323972b498b9a3999731bc9b

Add support for level specific cache maintenance operations ...

This patch adds level specific cache maintenance functions
to cache_helpers.S. The new functions 'dcsw_op_levelx',
where '1 <= x <= 3', allow to perform cache maintenance by
set/way for that particular level of cache.  With this patch,
functions to support cache maintenance upto level 3 have
been implemented since it is the highest cache level for
most ARM SoCs.

These functions are now utilized in CPU specific power down
sequences to implement them as mandated by processor specific
technical reference manual.

Change-Id: Icd90ce6b51cff5a12863bcda01b93601417fd45c

Introduce framework for CPU specific operations ...

This patch introduces a framework which will allow CPUs to perform
implementation defined actions after a CPU reset, during a CPU or cluster power
down, and when a crash occurs. CPU specific reset handlers have been implemented
in this patch. Other handlers will be implemented in subsequent patches.

Also moved cpu_helpers.S to the new directory lib/cpus/aarch64/.

Change-Id: I1ca1bade4d101d11a898fb30fea2669f9b37b956

Merge pull request #172 from soby-mathew/sm/asm_assert ...

Introduce asm assert and optimize crash reporting

Implement an assert() callable from assembly code ...

The patch implements a macro ASM_ASSERT() which can
be invoked from assembly code. When assertion happens,
file name and line number of the check is written
to the crash console.

Fixes ARM-software/tf-issues#95

Change-Id: I6f905a068e1c0fa4f746d723f18df60daaa00a86

Simplify management of SCTLR_EL3 and SCTLR_EL1 ...

This patch reworks the manner in which the M,A, C, SA, I, WXN & EE bits of
SCTLR_EL3 & SCTLR_EL1 are managed. The EE bit is cleared immediately after reset
in EL3. The I, A and SA bits are set next in EL3 and immediately upon entry in
S-EL1. These bits are no longer managed in the blX_arch_setup() functions. They
do not have to be saved and restored either. The M, WXN and optionally the C
bit are set in the enable_mmu_elX() function. This is done during both the warm
and cold boot paths.

Fixes ARM-software/tf-issues#226

Change-Id: Ie894d1a07b8697c116960d858cd138c50bc7a069

Make enablement of the MMU more flexible ...

This patch adds a 'flags' parameter to each exception level specific function
responsible for enabling the MMU. At present only a single flag which indicates
whether the data cache should also be enabled is implemented. Subsequent patches
will use this flag when enabling the MMU in the warm boot paths.

Change-Id: I0eafae1e678c9ecc604e680851093f1680e9cefa

Calculate TCR bits based on VA and PA ...

Currently the TCR bits are hardcoded in xlat_tables.c. In order to
map higher physical address into low virtual address, the TCR bits
need to be configured accordingly.

This patch is to save the max VA and PA and calculate the TCR.PS/IPS
and t0sz bits in init_xlat_tables function.

Change-Id: Ia7a58e5372b20200153057d457f4be5ddbb7dae4

Merge pull request #130 from athoelke/at/inline-asm-sysreg-v2 ...

Make system register functions inline assembly v2

Make system register functions inline assembly ...

Replace the current out-of-line assembler implementations of
the system register and system instruction operations with
inline assembler.

This enables better compiler optimisation and code generation
when accessing system registers.

Fixes ARM-software/tf-issues#91

Change-Id: I149af3a94e1e5e5140a3e44b9abfc37ba2324476

Enable mapping higher physical address ...

Current ATF uses a direct physical-to-virtual mapping, that is, a physical
address is mapped to the same address in the virtual space. For example,
physical address 0x8000_0000 is mapped to 0x8000_0000 virtual. This
approach works fine for FVP as all its physical addresses fall into 0 to
4GB range. But for other platform where all I/O addresses are 48-bit long,
If we follow the same direct mapping, we would need virtual address range
from 0 to 0x8fff_ffff_ffff, which is about 144TB. This requires a
significant amount of memory for MMU tables and it is not necessary to use
that much virtual space in ATF.

The patch is to enable mapping a physical address range to an arbitrary
virtual address range (instead of flat mapping)
Changed "base" to "base_va" and added "base_pa" in mmap_region_t and
modified functions such as mmap_add_region and init_xlation_table etc.
Fixes ARM-software/tf-issues#158

Add enable mmu platform porting interfaces ...

Previously, the enable_mmu_elX() functions were implicitly part of
the platform porting layer since they were included by generic
code. These functions have been placed behind 2 new platform
functions, bl31_plat_enable_mmu() and bl32_plat_enable_mmu().
These are weakly defined so that they can be optionally overridden
by platform ports.

Also, the enable_mmu_elX() functions have been moved to
lib/aarch64/xlat_tables.c for optional re-use by platform ports.
These functions are tightly coupled with the translation table
initialization code.

Fixes ARM-software/tf-issues#152

Change-Id: I0a2251ce76acfa3c27541f832a9efaa49135cc1c

Split platform.h into separate headers ...

Previously, platform.h contained many declarations and definitions
used for different purposes. This file has been split so that:

* Platform definitions used by common code that must be defined
  by the platform are now in platform_def.h. The exact include
  path is exported through $PLAT_INCLUDES in the platform makefile.

* Platform definitions specific to the FVP platform are now in
  /plat/fvp/fvp_def.h.

* Platform API declarations specific to the FVP platform are now
  in /plat/fvp/fvp_private.h.

* The remaining platform API declarations that must be ported by
  each platform are still in platform.h but this file has been
  moved to /include/plat/common since this can be shared by all
  platforms.

Change-Id: Ieb3bb22fbab3ee8027413c6b39a783534aee474a