The bus implementations currently call the drivers remove hook unconditionally, but this hook is seldomly populated. Only call it when it's actually populated. Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>