In an action packed game, the Blue Square South side stormed into a three-goal lead through Gary Monk’s own goal and strikes from Jamie Collins and Rocky Baptiste.
Swansea hit back through Guillem Bauza and Jason Scotland but it was Tom Jordan who sealed a fourth-round clash with Liverpool on 65 minutes.
Havant & Waterlooville boss Shaun Gale was pinching himself after setting up the Liverpool encounter.
Liverpool fan Gale hailed his non-league heroes: “We have turned the dream into reality. I wanted focus from my team and they showed me that tonight all over the pitch.
“It is a fantastic result. I am just so pleased for everyone involved in this football club.”
Gale felt Swansea had paid the price for underestimating a side ranked 83 places below them. He said: “Swansea played their strongest side but they underestimated us.
“We worked on our set pieces and they could not handle them tonight.
“We could have folded after they got their second goal but we dug deep.”
Swansea boss Roberto Martinez declined to speak to the media after the game.
Hereford set up a fourth round tie at home to Cardiff with a 1-0 victory over Tranmere at Edgar Street, with Simon Johnson scoring in the 72nd minute.