The Republican leader’s bid for speaker was stymied by a group of conservative hardliners as a historic battle for votes lasted hours.