A man who “swindled” a vulnerable woman out of her life savings has had his three-year prison sentence upheld by the Court of Appeal.