The head of a private swimming club at the centre of a civil rights-style race row defended cancelling the memberships of dozens of minority children, saying it was for safety reasons.