Port of a Linux commit 4c458bb

  Both i.MX7D and i.MX8MQ have the same behaviour when it comes to
  clearing DIRECT_SPEED_CHANGE bit when no speed change occurs, so to
  handle variants correctly add a flag instead of checking the IP block
  variant.

  Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
  [lorenzo.pieralisi@arm.com: updated log]
  Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
  Reviewed-by: Lucas Stach <l.stach@pengutronix.de>
  Cc: Bjorn Helgaas <bhelgaas@google.com>
  Cc: Fabio Estevam <fabio.estevam@nxp.com>
  Cc: Chris Healy <cphealy@gmail.com>
  Cc: Lucas Stach <l.stach@pengutronix.de>
  Cc: Leonard Crestez <leonard.crestez@nxp.com>
  Cc: "A.s. Dong" <aisheng.dong@nxp.com>
  Cc: Richard Zhu <hongxing.zhu@nxp.com>

Signed-off-by: Andrey Smirnov <andrew.smirnov@gmail.com>
Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>