Juande Ramos admitted Tottenham's situation was "terrible" after Stoke's 2-1 win condemned his side to their sixth defeat in eight games.
Ramos has presided over Tottenham's worst ever start to a season and is feeling the pressure.
To make matters worse, Gareth Bale and Michael Dawson will almost certainly be suspended after picking up red cards.
Ramos said: "I am worried about the situation. At the moment it is terrible.
"Last season we won the Carling Cup and everyone was very happy.
"At this moment everyone is very sad and I am worried about it as I don't like this situation.
"We need to work very hard to change this situation and only hard work can change it.
"This is my job and I am working very hard each day to finish this situation.
"The responsibility is (with) all the people working in the club starting with me and finishing with the last person - we all have responsibility, when we win and when we lose."
Asked if he planned to meet Daniel Levy and whether he was confident he had the chairman's backing, he added: "I don't know - this is a question for the chairman.
"I speak to the chairman when he permits us to speak. He's a very busy person and I don't know when it will be possible to speak."
Bale conceded a penalty and was red-carded 17 minutes into the game at the Britannia Stadium.
Danny Higginbotham scored from the spot and though Darren Bent equalised, Rory Delap grabbed the winner before Michael Dawson was given a straight red card for a studs-up challenge.
Ramos added: "Sometimes football is difficult to explain. We had two penalties against us, two players sent off, some players injured.
"It was very difficult for us after the sending off in the first half but the attitude of the players was very good."
Spurs face Bolton, Arsenal, Liverpool and Manchester City as their next four league opponents but Ramos insisted he was not concerned at facing the big teams.
He added: "We played against Chelsea away and played okay and got a point, and I think playing against big teams, the team can concentrate well."
Stoke boss Tony Pulis called on the Tottenham board to stick with Ramos.
He said: "Let's see a bit of character by their board of directors and see them back him.
"We are all in the same boat and know the rules, and have to accept it. I just hope the chairman who got rid of a very popular manager before [Martin Jol] now backs him and gives him a chance to turn it around.
"They certainly have good enough players and a good depth of squad."
Pulis admitted that his players had disappointed him after going in front.
"We were like rabbits trapped in headlights for the rest of the first half," he added. "After they went down to 10 men we were garbage.
"We just told them to push up in the second half and not show them so much respect.
"The second half was more like us. We played with a bit of pace and could have scored more goals."