After months of criticism, United Nations Secretary-General Kofi Annan decided that UN refugee chief Ruud Lubbers had to go because of the growing controversy over allegations that the former Dutch prime minister had sexually harassed female staffers.