A senior Fianna Fáil minister has said Ivor Callely should resign the party whip pending an official inquiry into his expenses - before he is forced out.