The comedian and actor, who came under fire for past anti-gay tweets, said he refused to apologize and added he did not want to "be a distraction."