Don't disable clk to unrelated devices in the process. Signed-off-by: Lucas Stach <dev@lynxeye.de> Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>