A man who raped a woman as she was drying her hands in a public toilet has lost an appeal against his conviction.