A second appeal by a former priest against his dismissal from the priesthood over child sex abuse has been thrown out.