A major search operation for a man who is thought to have fallen overboard from a ferry into the Irish Sea has been called off.