A Nigerian man has lost his appeal over being refused permanent residence here after his EU citizen wife, whom he married here in 2005, returned to her native Poland in 2009.