Make generic code work in presence of system caches ...

On the ARMv8 architecture, cache maintenance operations by set/way on the last
level of integrated cache do not affect the system cache. This means that such a
flush or clean operation could result in the data being pushed out to the system
cache rather than main memory. Another CPU could access this data before it
enables its data cache or MMU. Such accesses could be serviced from the main
memory instead of the system cache. If the data in the sysem cache has not yet
been flushed or evicted to main memory then there could be a loss of
coherency. The only mechanism to guarantee that the main memory will be updated
is to use cache maintenance operations to the PoC by MVA(See section D3.4.11
(System level caches) of ARMv8-A Reference Manual (Issue A.g/ARM DDI0487A.G).

This patch removes the reliance of Trusted Firmware on the flush by set/way
operation to ensure visibility of data in the main memory. Cache maintenance
operations by MVA are now used instead. The following are the broad category of
changes:

1. The RW areas of BL2/BL31/BL32 are invalidated by MVA before the C runtime is
   initialised. This ensures that any stale cache lines at any level of cache
   are removed.

2. Updates to global data in runtime firmware (BL31) by the primary CPU are made
   visible to secondary CPUs using a cache clean operation by MVA.

3. Cache maintenance by set/way operations are only used prior to power down.

NOTE: NON-UPSTREAM TRUSTED FIRMWARE CODE SHOULD MAKE EQUIVALENT CHANGES IN
ORDER TO FUNCTION CORRECTLY ON PLATFORMS WITH SUPPORT FOR SYSTEM CACHES.

Fixes ARM-software/tf-issues#205

Change-Id: I64f1b398de0432813a0e0881d70f8337681f6e9a

Re-design bakery lock memory allocation and algorithm ...

This patch unifies the bakery lock api's across coherent and normal
memory implementation of locks by using same data type `bakery_lock_t`
and similar arguments to functions.

A separate section `bakery_lock` has been created and used to allocate
memory for bakery locks using `DEFINE_BAKERY_LOCK`. When locks are
allocated in normal memory, each lock for a core has to spread
across multiple cache lines. By using the total size allocated in a
separate cache line for a single core at compile time, the memory for
other core locks is allocated at link time by multiplying the single
core locks size with (PLATFORM_CORE_COUNT - 1). The normal memory lock
algorithm now uses lock address instead of the `id` in the per_cpu_data.
For locks allocated in coherent memory, it moves locks from
tzfw_coherent_memory to bakery_lock section.

The bakery locks are allocated as part of bss or in coherent memory
depending on usage of coherent memory. Both these regions are
initialised to zero as part of run_time_init before locks are used.
Hence, bakery_lock_init() is made an empty function as the lock memory
is already initialised to zero.

The above design lead to the removal of psci bakery locks from
non_cpu_power_pd_node to psci_locks.

NOTE: THE BAKERY LOCK API WHEN USE_COHERENT_MEM IS NOT SET HAS CHANGED.
THIS IS A BREAKING CHANGE FOR ALL PLATFORM PORTS THAT ALLOCATE BAKERY
LOCKS IN NORMAL MEMORY.

Change-Id: Ic3751c0066b8032dcbf9d88f1d4dc73d15f61d8b

Add macros for retention control in Cortex-A53/A57 ...

This patch adds macros suitable for programming the Advanced
SIMD/Floating-point (only Cortex-A53), CPU and L2 dynamic
retention control policy in the CPUECTLR_EL1 and L2ECTLR
registers.

Signed-off-by: Varun Wadekar <vwadekar@nvidia.com>

cortex_a53: Add A53 errata #826319, #836870 ...

- Apply a53 errata #826319 to revision <= r0p2
- Apply a53 errata #836870 to revision <= r0p3
- Update docs/cpu-specific-build-macros.md for newly added errata build flags

Change-Id: I44918e36b47dca1fa29695b68700ff9bf888865e
Signed-off-by: Jimmy Huang <jimmy.huang@mediatek.com>

Add mmio utility functions ...

- Add mmio 16 bits read/write functions.
- Add clear/set/clear-and-set utility functions.

Change-Id: I00fdbdf24af537424f8666b1cadaa5f77a2a46ed
Signed-off-by: Jimmy Huang <jimmy.huang@mediatek.com>

Add "Project Denver" CPU support ...

Denver is NVIDIA's own custom-designed, 64-bit, dual-core CPU which is
fully ARMv8 architecture compatible.  Each of the two Denver cores
implements a 7-way superscalar microarchitecture (up to 7 concurrent
micro-ops can be executed per clock), and includes a 128KB 4-way L1
instruction cache, a 64KB 4-way L1 data cache, and a 2MB 16-way L2
cache, which services both cores.

Denver implements an innovative process called Dynamic Code Optimization,
which optimizes frequently used software routines at runtime into dense,
highly tuned microcode-equivalent routines. These are stored in a
dedicated, 128MB main-memory-based optimization cache. After being read
into the instruction cache, the optimized micro-ops are executed,
re-fetched and executed from the instruction cache as long as needed and
capacity allows.

Effectively, this reduces the need to re-optimize the software routines.
Instead of using hardware to extract the instruction-level parallelism
(ILP) inherent in the code, Denver extracts the ILP once via software
techniques, and then executes those routines repeatedly, thus amortizing
the cost of ILP extraction over the many execution instances.

Denver also features new low latency power-state transitions, in addition
to extensive power-gating and dynamic voltage and clock scaling based on
workloads.

Signed-off-by: Varun Wadekar <vwadekar@nvidia.com>

Add header guards to asm macro files ...

Some assembly files containing macros are included like header files
into other assembly files. This will cause assembler errors if they
are included multiple times.

Add header guards to assembly macro files to avoid assembler errors.

Change-Id: Ia632e767ed7df7bf507b294982b8d730a6f8fe69

Remove use of PLATFORM_CACHE_LINE_SIZE ...

The required platform constant PLATFORM_CACHE_LINE_SIZE is
unnecessary since CACHE_WRITEBACK_GRANULE effectively provides the
same information. CACHE_WRITEBACK_GRANULE is preferred since this
is an architecturally defined term and allows comparison with the
corresponding hardware register value.

Replace all usage of PLATFORM_CACHE_LINE_SIZE with
CACHE_WRITEBACK_GRANULE.

Also, add a runtime assert in BL1 to check that the provided
CACHE_WRITEBACK_GRANULE matches the value provided in CTR_EL0.

Change-Id: If87286be78068424217b9f3689be358356500dcd

Merge pull request #277 from soby-mathew/sm/coh_lock_opt ...

Optimize the bakery lock implementation

Merge pull request #280 from vwadekar/tlkd-fixed-v3 ...

TLK dispatcher

Translate secure/non-secure virtual addresses ...

This patch adds functionality to translate virtual addresses from
secure or non-secure worlds. This functionality helps Trusted Apps
to share virtual addresses directly and allows the NS world to
pass virtual addresses to TLK directly.

Change-Id: I77b0892963e0e839c448b5d0532920fb7e54dc8e
Signed-off-by: Varun Wadekar <vwadekar@nvidia.com>

Remove the `owner` field in bakery_lock_t data structure ...

This patch removes the `owner` field in bakery_lock_t structure which
is the data structure used in the bakery lock implementation that uses
coherent memory. The assertions to protect against recursive lock
acquisition were based on the 'owner' field. They are now done based
on the bakery lock ticket number. These assertions are also added
to the bakery lock implementation that uses normal memory as well.

Change-Id: If4850a00dffd3977e218c0f0a8d145808f36b470

Optimize the bakery lock structure for coherent memory ...

This patch optimizes the data structure used with the bakery lock
implementation for coherent memory to save memory and minimize memory
accesses. These optimizations were already part of the bakery lock
implementation for normal memory and this patch now implements
it for the coherent memory implementation as well. Also
included in the patch is a cleanup to use the do-while loop while
waiting for other contenders to finish choosing their tickets.

Change-Id: Iedb305473133dc8f12126726d8329b67888b70f1

Merge pull request #270 from vikramkanigiri/vk/a72_cpu_support ...

Add support for ARM Cortex-A72 processor

Add support for ARM Cortex-A72 processor ...

This patch adds support for ARM Cortex-A72 processor in the CPU
specific framework.

Change-Id: I5986855fc1b875aadf3eba8c36e989d8a05e5175

Use ARM CCI driver on FVP and Juno platforms ...

This patch updates the FVP and Juno platform ports to use the common
driver for ARM Cache Coherent Interconnects.

Change-Id: Ib142f456b9b673600592616a2ec99e9b230d6542

Call reset handlers upon BL3-1 entry. ...

This patch adds support to call the reset_handler() function in BL3-1 in the
cold and warm boot paths when another Boot ROM reset_handler() has already run.

This means the BL1 and BL3-1 versions of the CPU and platform specific reset
handlers may execute different code to each other. This enables a developer to
perform additional actions or undo actions already performed during the first
call of the reset handlers e.g. apply additional errata workarounds.

Typically, the reset handler will be first called from the BL1 Boot ROM. Any
additional functionality can be added to the reset handler when it is called
from BL3-1 resident in RW memory. The constant FIRST_RESET_HANDLER_CALL is used
to identify whether this is the first version of the reset handler code to be
executed or an overridden version of the code.

The Cortex-A57 errata workarounds are applied only if they have not already been
applied.

Fixes ARM-software/tf-issue#275

Change-Id: Id295f106e4fda23d6736debdade2ac7f2a9a9053

Return success if an interrupt is seen during PSCI CPU_SUSPEND ...

This patch adds support to return SUCCESS if a pending interrupt is
detected during a CPU_SUSPEND call to a power down state. The check
is performed as late as possible without losing the ability to return
to the caller. This reduces the overhead incurred by a CPU in
undergoing a complete power cycle when a wakeup interrupt is already
pending.

Fixes ARM-Software/tf-issues#102

Change-Id: I1aff04a74b704a2f529734428030d1d10750fd4b

Move bakery algorithm implementation out of coherent memory ...

This patch moves the bakery locks out of coherent memory to normal memory.
This implies that the lock information needs to be placed on a separate cache
line for each cpu. Hence the bakery_lock_info_t structure is allocated in the
per-cpu data so as to minimize memory wastage. A similar platform per-cpu
data is introduced for the platform locks.

As a result of the above changes, the bakery lock api is completely changed.
Earlier, a reference to the lock structure was passed to the lock implementation.
Now a unique-id (essentially an index into the per-cpu data array) and an offset
into the per-cpu data for bakery_info_t needs to be passed to the lock
implementation.

Change-Id: I1e76216277448713c6c98b4c2de4fb54198b39e0

Add macros for domain specific barriers. ...

This patch adds helper macros for barrier operations that specify
the type of barrier (dmb, dsb) and the shareability domain (system,
inner-shareable) it affects.

Change-Id: I4bf95103e79da212c4fbdbc13d91ad8ac385d9f5

Prevent optimisation of sysregs accessors calls ...

Calls to system register read accessors functions may be optimised
out by the compiler if called twice in a row for the same register.
This is because the compiler is not aware that the result from
the instruction may be modified by external agents. Therefore, if
nothing modifies the register between the 2 reads as far as the
compiler knows then it might consider that it is useless to read
it twice and emit only 1 call.

This behaviour is faulty for registers that may not have the same
value if read twice in succession. E.g.: counters, timer
control/countdown registers, GICv3 interrupt status registers and
so on.

The same problem happens for calls to system register write
accessors functions. The compiler might optimise out some calls
if it considers that it will produce the same result. Again, this
behaviour is faulty for cases where intermediate writes to these
registers make a difference in the system.

This patch fixes the problem by making these assembly register
accesses volatile.

Fixes ARM-software/tf-issues#273

Change-Id: I33903bc4cc4eea8a8d87bc2c757909fbb0138925

Fix the array size of mpidr_aff_map_nodes_t. ...

This patch fixes the array size of mpidr_aff_map_nodes_t which
was less by one element.

Fixes ARM-software/tf-issues#264

Change-Id: I48264f6f9e7046a3d0f4cbcd63b9ba49657e8818

Apply errata workarounds only when major/minor revisions match. ...

Prior to this patch, the errata workarounds were applied for any version
of the CPU in the release build and in the debug build an assert
failure resulted when the revision did not match. This patch applies
errata workarounds in the Cortex-A57 reset handler only if the 'variant'
and 'revision' fields read from the MIDR_EL1 match. In the debug build,
a warning message is printed for each errata workaround which is not
applied.

The patch modifies the register usage in 'reset_handler` so
as to adhere to ARM procedure calling standards.

Fixes ARM-software/tf-issues#242

Change-Id: I51b1f876474599db885afa03346e38a476f84c29

Merge pull request #206 from soby-mathew/sm/reset_cntvoff ...

Reset CNTVOFF_EL2 register before exit into EL1 on warm boot

Create BL stage specific translation tables ...

This patch uses the IMAGE_BL<x> constants to create translation tables specific
to a boot loader stage. This allows each stage to create mappings only for areas
in the memory map that it needs.

Fixes ARM-software/tf-issues#209

Change-Id: Ie4861407ddf9317f0fb890fc7575eaa88d0de51c

Initialize SCTLR_EL1 based on MODE_RW bit ...

Initializes SCTLR_EL1 based on MODE_RW bit in SPSR for the entry
point. The RES1 bits for SCTLR_EL1 differs for Aarch64 and Aarch32
mode.

Reset CNTVOFF_EL2 register before exit into EL1 on warm boot ...

This patch resets the value of CNTVOFF_EL2 before exit to EL1 on
warm boot. This needs to be done if only the Trusted Firmware exits
to EL1 instead of EL2, otherwise the hypervisor would be responsible
for this.

Fixes ARM-software/tf-issues#240

Change-Id: I79d54831356cf3215bcf1f251c373bd8f89db0e0

Juno: Implement initial platform port ...

This patch adds the initial port of the ARM Trusted Firmware on the Juno
development platform. This port does not support a BL3-2 image or any PSCI APIs
apart from PSCI_VERSION and PSCI_CPU_ON. It enables workarounds for selected
Cortex-A57 (#806969 & #813420) errata and implements the workaround for a Juno
platform errata (Defect id 831273).

Change-Id: Ib3d92df3af53820cfbb2977582ed0d7abf6ef893

Add support for selected Cortex-A57 errata workarounds ...

This patch adds workarounds for selected errata which affect the Cortex-A57 r0p0
part. Each workaround has a build time flag which should be used by the platform
port to enable or disable the corresponding workaround. The workarounds are
disabled by default. An assertion is raised if the platform enables a workaround
which does not match the CPU revision at runtime.

Change-Id: I9ae96b01c6ff733d04dc733bd4e67dbf77b29fb0

Add CPU specific crash reporting handlers ...

This patch adds handlers for dumping Cortex-A57 and Cortex-A53 specific register
state to the CPU specific operations framework. The contents of CPUECTLR_EL1 are
dumped currently.

Change-Id: I63d3dbfc4ac52fef5e25a8cf6b937c6f0975c8ab