The bishop has not been charged, but has given interviews denying wrongdoing and saying he wanted to defend his name