Make generic code work in presence of system caches ...

On the ARMv8 architecture, cache maintenance operations by set/way on the last
level of integrated cache do not affect the system cache. This means that such a
flush or clean operation could result in the data being pushed out to the system
cache rather than main memory. Another CPU could access this data before it
enables its data cache or MMU. Such accesses could be serviced from the main
memory instead of the system cache. If the data in the sysem cache has not yet
been flushed or evicted to main memory then there could be a loss of
coherency. The only mechanism to guarantee that the main memory will be updated
is to use cache maintenance operations to the PoC by MVA(See section D3.4.11
(System level caches) of ARMv8-A Reference Manual (Issue A.g/ARM DDI0487A.G).

This patch removes the reliance of Trusted Firmware on the flush by set/way
operation to ensure visibility of data in the main memory. Cache maintenance
operations by MVA are now used instead. The following are the broad category of
changes:

1. The RW areas of BL2/BL31/BL32 are invalidated by MVA before the C runtime is
   initialised. This ensures that any stale cache lines at any level of cache
   are removed.

2. Updates to global data in runtime firmware (BL31) by the primary CPU are made
   visible to secondary CPUs using a cache clean operation by MVA.

3. Cache maintenance by set/way operations are only used prior to power down.

NOTE: NON-UPSTREAM TRUSTED FIRMWARE CODE SHOULD MAKE EQUIVALENT CHANGES IN
ORDER TO FUNCTION CORRECTLY ON PLATFORMS WITH SUPPORT FOR SYSTEM CACHES.

Fixes ARM-software/tf-issues#205

Change-Id: I64f1b398de0432813a0e0881d70f8337681f6e9a

Introduce PROGRAMMABLE_RESET_ADDRESS build option ...

This patch introduces a new platform build option, called
PROGRAMMABLE_RESET_ADDRESS, which tells whether the platform has
a programmable or fixed reset vector address.

If the reset vector address is fixed then the code relies on the
platform_get_entrypoint() mailbox mechanism to figure out where
it is supposed to jump. On the other hand, if it is programmable
then it is assumed that the platform code will program directly
the right address into the RVBAR register (instead of using the
mailbox redirection) so the mailbox is ignored in this case.

Change-Id: If59c3b11fb1f692976e1d8b96c7e2da0ebfba308

Rationalize reset handling code ...

The attempt to run the CPU reset code as soon as possible after reset
results in highly complex conditional code relating to the
RESET_TO_BL31 option.

This patch relaxes this requirement a little. In the BL1, BL3-1 and
PSCI entrypoints code, the sequence of operations is now as follows:
 1) Detect whether it is a cold or warm boot;
 2) For cold boot, detect whether it is the primary or a secondary
    CPU. This is needed to handle multiple CPUs entering cold reset
    simultaneously;
 3) Run the CPU init code.

This patch also abstracts the EL3 registers initialisation done by
the BL1, BL3-1 and PSCI entrypoints into common code.

This improves code re-use and consolidates the code flows for
different types of systems.

NOTE: THE FUNCTION plat_secondary_cold_boot() IS NOW EXPECTED TO
NEVER RETURN. THIS PATCH FORCES PLATFORM PORTS THAT RELIED ON THE
FORMER RETRY LOOP AT THE CALL SITE TO MODIFY THEIR IMPLEMENTATION.
OTHERWISE, SECONDARY CPUS WILL PANIC.

Change-Id: If5ecd74d75bee700b1bd718d23d7556b8f863546

Add support to indicate size and end of assembly functions ...

In order for the symbol table in the ELF file to contain the size of
functions written in assembly, it is necessary to report it to the
assembler using the .size directive.

To fulfil the above requirements, this patch introduces an 'endfunc'
macro which contains the .endfunc and .size directives. It also adds
a .func directive to the 'func' assembler macro.

The .func/.endfunc have been used so the assembler can fail if
endfunc is omitted.

Fixes ARM-Software/tf-issues#295

Change-Id: If8cb331b03d7f38fe7e3694d4de26f1075b278fc
Signed-off-by: Kévin Petit <kevin.petit@arm.com>

Initialise cpu ops after enabling data cache ...

The cpu-ops pointer was initialized before enabling the data cache in the cold
and warm boot paths. This required a DCIVAC cache maintenance operation to
invalidate any stale cache lines resident in other cpus.

This patch moves this initialization to the bl31_arch_setup() function
which is always called after the data cache and MMU has been enabled.

This change removes the need:
 1. for the DCIVAC cache maintenance operation.
 2. to initialise the CPU ops upon resumption from a PSCI CPU_SUSPEND
    call since memory contents are always preserved in this case.

Change-Id: Ibb2fa2f7460d1a1f1e721242025e382734c204c6

Call reset handlers upon BL3-1 entry. ...

This patch adds support to call the reset_handler() function in BL3-1 in the
cold and warm boot paths when another Boot ROM reset_handler() has already run.

This means the BL1 and BL3-1 versions of the CPU and platform specific reset
handlers may execute different code to each other. This enables a developer to
perform additional actions or undo actions already performed during the first
call of the reset handlers e.g. apply additional errata workarounds.

Typically, the reset handler will be first called from the BL1 Boot ROM. Any
additional functionality can be added to the reset handler when it is called
from BL3-1 resident in RW memory. The constant FIRST_RESET_HANDLER_CALL is used
to identify whether this is the first version of the reset handler code to be
executed or an overridden version of the code.

The Cortex-A57 errata workarounds are applied only if they have not already been
applied.

Fixes ARM-software/tf-issue#275

Change-Id: Id295f106e4fda23d6736debdade2ac7f2a9a9053

Remove coherent memory from the BL memory maps ...

This patch extends the build option `USE_COHERENT_MEMORY` to
conditionally remove coherent memory from the memory maps of
all boot loader stages. The patch also adds necessary
documentation for coherent memory removal in firmware-design,
porting and user guides.

Fixes ARM-Software/tf-issues#106

Change-Id: I260e8768c6a5c2efc402f5804a80657d8ce38773

Miscellaneous documentation fixes ...

This patch gathers miscellaneous minor fixes to the documentation, and comments
in the source code.

Change-Id: I631e3dda5abafa2d90f464edaee069a1e58b751b
Co-Authored-By: Soby Mathew <soby.mathew@arm.com>
Co-Authored-By: Dan Handley <dan.handley@arm.com>

Add CPU specific power management operations ...

This patch adds CPU core and cluster power down sequences to the CPU specific
operations framework introduced in a earlier patch. Cortex-A53, Cortex-A57 and
generic AEM sequences have been added. The latter is suitable for the
Foundation and Base AEM FVPs. A pointer to each CPU's operations structure is
saved in the per-cpu data so that it can be easily accessed during power down
seqeunces.

An optional platform API has been introduced to allow a platform to disable the
Accelerator Coherency Port (ACP) during a cluster power down sequence. The weak
definition of this function (plat_disable_acp()) does not take any action. It
should be overriden with a strong definition if the ACP is present on a
platform.

Change-Id: I8d09bd40d2f528a28d2d3f19b77101178778685d

Introduce framework for CPU specific operations ...

This patch introduces a framework which will allow CPUs to perform
implementation defined actions after a CPU reset, during a CPU or cluster power
down, and when a crash occurs. CPU specific reset handlers have been implemented
in this patch. Other handlers will be implemented in subsequent patches.

Also moved cpu_helpers.S to the new directory lib/cpus/aarch64/.

Change-Id: I1ca1bade4d101d11a898fb30fea2669f9b37b956

Unmask SError interrupt and clear SCR_EL3.EA bit ...

This patch disables routing of external aborts from lower exception levels to
EL3 and ensures that a SError interrupt generated as a result of execution in
EL3 is taken locally instead of a lower exception level.

The SError interrupt is enabled in the TSP code only when the operation has not
been directly initiated by the normal world. This is to prevent the possibility
of an asynchronous external abort which originated in normal world from being
taken when execution is in S-EL1.

Fixes ARM-software/tf-issues#153

Change-Id: I157b996c75996d12fd86d27e98bc73dd8bce6cd5

Call platform_is_primary_cpu() only from reset handler ...

The purpose of platform_is_primary_cpu() is to determine after reset
(BL1 or BL3-1 with reset handler) if the current CPU must follow the
cold boot path (primary CPU), or wait in a safe state (secondary CPU)
until the primary CPU has finished the system initialization.

This patch removes redundant calls to platform_is_primary_cpu() in
subsequent bootloader entrypoints since the reset handler already
guarantees that code is executed exclusively on the primary CPU.

Additionally, this patch removes the weak definition of
platform_is_primary_cpu(), so the implementation of this function
becomes mandatory. Removing the weak symbol avoids other
bootloaders accidentally picking up an invalid definition in case the
porting layer makes the real function available only to BL1.

The define PRIMARY_CPU is no longer mandatory in the platform porting
because platform_is_primary_cpu() hides the implementation details
(for instance, there may be platforms that report the primary CPU in
a system register). The primary CPU definition in FVP has been moved
to fvp_def.h.

The porting guide has been updated accordingly.

Fixes ARM-software/tf-issues#219

Change-Id: If675a1de8e8d25122b7fef147cb238d939f90b5e

Merge pull request #172 from soby-mathew/sm/asm_assert ...

Introduce asm assert and optimize crash reporting

Rework the crash reporting in BL3-1 to use less stack ...

This patch reworks the crash reporting mechanism to further
optimise the stack and code size. The reporting makes use
of assembly console functions to avoid calling C Runtime
to report the CPU state. The crash buffer requirement is
reduced to 64 bytes with this implementation. The crash
buffer is now part of per-cpu data which makes retrieving
the crash buffer trivial.

Also now panic() will use crash reporting if
invoked from BL3-1.

Fixes ARM-software/tf-issues#199

Change-Id: I79d27a4524583d723483165dc40801f45e627da5

Simplify management of SCTLR_EL3 and SCTLR_EL1 ...

This patch reworks the manner in which the M,A, C, SA, I, WXN & EE bits of
SCTLR_EL3 & SCTLR_EL1 are managed. The EE bit is cleared immediately after reset
in EL3. The I, A and SA bits are set next in EL3 and immediately upon entry in
S-EL1. These bits are no longer managed in the blX_arch_setup() functions. They
do not have to be saved and restored either. The M, WXN and optionally the C
bit are set in the enable_mmu_elX() function. This is done during both the warm
and cold boot paths.

Fixes ARM-software/tf-issues#226

Change-Id: Ie894d1a07b8697c116960d858cd138c50bc7a069

Remove coherent stack usage from the cold boot path ...

This patch reworks the cold boot path across the BL1, BL2, BL3-1 and BL3-2 boot
loader stages to not use stacks allocated in coherent memory for early platform
setup and enabling the MMU. Stacks allocated in normal memory are used instead.

Attributes for stack memory change from nGnRnE when the MMU is disabled to
Normal WBWA Inner-shareable when the MMU and data cache are enabled. It is
possible for the CPU to read stale stack memory after the MMU is enabled from
another CPUs cache. Hence, it is unsafe to turn on the MMU and data cache while
using normal stacks when multiple CPUs are a part of the same coherency
domain. It is safe to do so in the cold boot path as only the primary cpu
executes it. The secondary cpus are in a quiescent state.

This patch does not remove the allocation of coherent stack memory. That is done
in a subsequent patch.

Change-Id: I12c80b7c7ab23506d425c5b3a8a7de693498f830

Merge pull request #151 from vikramkanigiri/vk/t133-code-readability ...

Simplify entry point information generation code on FVP

Simplify entry point information generation code on FVP ...

This patch reworks FVP specific code responsible for determining
the entry point information for BL3-2 and BL3-3 stages when BL3-1
is configured as the reset handler.

Change-Id: Ia661ff0a6a44c7aabb0b6c1684b2e8d3642d11ec

Remove early_exceptions from BL3-1 ...

The crash reporting support and early initialisation of the
cpu_data allow the runtime_exception vectors to be used from
the start in BL3-1, removing the need for the additional
early_exception vectors and 2KB of code from BL3-1.

Change-Id: I5f8997dabbaafd8935a7455910b7db174a25d871

Per-cpu data cache restructuring ...

This patch prepares the per-cpu pointer cache for wider use by:
* renaming the structure to cpu_data and placing in new header
* providing accessors for this CPU, or other CPUs
* splitting the initialization of the TPIDR pointer from the
  initialization of the cpu_data content
* moving the crash stack initialization to a crash stack function
* setting the TPIDR pointer very early during boot

Change-Id: Icef9004ff88f8eb241d48c14be3158087d7e49a3

Introduce interrupt handling framework in BL3-1 ...

This patch adds a common handler for FIQ and IRQ exceptions in the
BL3-1 runtime exception vector table. This function determines the
interrupt type and calls its handler. A crash is reported if an
inconsistency in the interrupt management framework is detected. In
the event of a spurious interrupt, execution resumes from the
instruction where the interrupt was generated.

This patch also removes 'cm_macros.S' as its contents have been moved
to 'runtime_exceptions.S'

Change-Id: I3c85ecf8eaf43a3fac429b119ed0bd706d2e2093

Add support for BL3-1 as a reset vector ...

This change adds optional reset vector support to BL3-1
which means BL3-1 entry point can detect cold/warm boot,
initialise primary cpu, set up cci and mail box.

When using BL3-1 as a reset vector it is assumed that
the BL3-1 platform code can determine the location of
the BL3-2 images, or load them as there are no parameters
that can be passed to BL3-1 at reset.

It also fixes the incorrect initialisation of mailbox
registers on the FVP platform

This feature can be enabled by building the code with
make variable RESET_TO_BL31 set as 1

Fixes ARM-software/TF-issues#133
Fixes ARM-software/TF-issues#20

Change-Id: I4e23939b1c518614b899f549f1e8d412538ee570

Populate BL31 input parameters as per new spec ...

This patch is based on spec published at
https://github.com/ARM-software/tf-issues/issues/133

It rearranges the bl31_args struct into
bl31_params and bl31_plat_params which provide the
information needed for Trusted firmware and platform
specific data via x0 and x1

On the FVP platform BL3-1 params and BL3-1 plat params
and its constituents are stored at the start of TZDRAM.

The information about memory availability and size for
BL3-1, BL3-2 and BL3-3 is moved into platform specific data.

Change-Id: I8b32057a3d0dd3968ea26c2541a0714177820da9

Rework handover interface between BL stages ...

This patch reworks the handover interface from: BL1 to BL2 and
BL2 to BL3-1. It removes the raise_el(), change_el(), drop_el()
and run_image() functions as they catered for code paths that were
never exercised.
BL1 calls bl1_run_bl2() to jump into BL2 instead of doing the same
by calling run_image(). Similarly, BL2 issues the SMC to transfer
execution to BL3-1 through BL1 directly. Only x0 and x1 are used
to pass arguments to BL31. These arguments and parameters for
running BL3-1 are passed through a reference to a
'el_change_info_t' structure. They were being passed value in
general purpose registers earlier.

Change-Id: Id4fd019a19a9595de063766d4a66295a2c9307e1

Merge pull request #63 from soby-mathew/sm/save_callee_saved_registers_in_cpu_context-1 ...

Preserve x19-x29 across world switch for exception handling

Preserve x19-x29 across world switch for exception handling ...

Previously exception handlers in BL3-1, X19-X29 were not saved
and restored on every SMC/trap into EL3. Instead these registers
were 'saved as needed' as a side effect of the A64 ABI used by the C
compiler.

That approach failed when world switching but was not visible
with the TSP/TSPD code because the TSP is 64-bit, did not
clobber these registers when running and did not support pre-emption
by normal world interrupts. These scenarios showed
that the values in these registers can be passed through a world
switch, which broke the normal and trusted world assumptions
about these registers being preserved.

The Ideal solution saves and restores these registers when a
world switch occurs - but that type of implementation is more complex.
So this patch always saves and restores these registers on entry and
exit of EL3.

Fixes ARM-software/tf-issues#141

Change-Id: I9a727167bbc594454e81cf78a97ca899dfb11c27

Access system registers directly in assembler ...

Instead of using the system register helper functions to read
or write system registers, assembler coded functions should
use MRS/MSR instructions. This results in faster and more
compact code.

This change replaces all usage of the helper functions with
direct register accesses.

Change-Id: I791d5f11f257010bb3e6a72c6c5ab8779f1982b3

Correct usage of data and instruction barriers ...

The current code does not always use data and instruction
barriers as required by the architecture and frequently uses
barriers excessively due to their inclusion in all of the
write_*() helper functions.

Barriers should be used explicitly in assembler or C code
when modifying processor state that requires the barriers in
order to enable review of correctness of the code.

This patch removes the barriers from the helper functions and
introduces them as necessary elsewhere in the code.

PORTING NOTE: check any port of Trusted Firmware for use of
system register helper functions for reliance on the previous
barrier behaviour and add explicit barriers as necessary.

Fixes ARM-software/tf-issues#92

Change-Id: Ie63e187404ff10e0bdcb39292dd9066cb84c53bf

Reduce deep nesting of header files ...

Reduce the number of header files included from other header
files as much as possible without splitting the files. Use forward
declarations where possible. This allows removal of some unnecessary
"#ifndef __ASSEMBLY__" statements.

Also, review the .c and .S files for which header files really need
including and reorder the #include statements alphabetically.

Fixes ARM-software/tf-issues#31

Change-Id: Iec92fb976334c77453e010b60bcf56f3be72bd3e

Make use of user/system includes more consistent ...

Make codebase consistent in its use of #include "" syntax for
user includes and #include <> syntax for system includes.

Fixes ARM-software/tf-issues#65

Change-Id: If2f7c4885173b1fd05ac2cde5f1c8a07000c7a33