A former priest and school principal has had his appeal against his conviction for the sexual abuse rejected.