Almost 400,000 fewer people gave to charity in the last few months, including at Christmas, as recent scandals have hurt the sector.