More than a third of people say charity fundraisers on the street are "aggressive" and "annoying".