The Hong Kong Special Olympics team says the decision by the Minister for Health to request that they stay away is "disappointing and unfair".