Stoke snatched a late win at Villa Park thanks to Victor Moses’ injury-time penalty.
Ron Vlaar was also sent off for bringing the forward down.
They slipped to second bottom thanks to Burnley’s surprise point at Chelsea but only remain in the Premier League drop zone on goal difference.
Sherwood has vowed to keep them up but revealed his disappointment:
“You look at the scoreboard and, though you set out wanting to win the game, you think ‘I’ll take that’. You come away with a point, something to build on, and then go away to Newcastle and go again and improve.
“We know we need to improve and we need to start improving very quickly.”
It was Villa’s sixth straight league defeat and they are winless in their last 11 games despite Scott Sinclair’s opener.
Mame Biram Diouf’s equaliser just before half-time allowed Stoke back into the the game and boss Mark Hughes believes the ex-Manchester United striker gives them an edge.
“He gives us that threat. His energy and his work rate is without exception,” he said.
“I made a little bit of a change late on, I thought he was creating a lot of chances for himself but not giving us that final product. So I moved Jon (Walters) out to the right hand side and Mame through the middle and that allowed him to threaten.
ASTON VILLA: Guzan, Hutton, Vlaar, Clark, Richardson (Cissokho 42), Gil (Weimann 58), Sanchez, Delph, Sinclair, Benteke, Agbonlahor.
STOKE: Begovic, Bardsley, Wollscheid, Wilson, Pieters (Dionatan 86), Ireland (Sidwell 86), Nzonzi, Whelan, Diouf, Walters, Moses.
Referee: Roger East (Wiltshire).