It was the controversial former Scotland supremo’s first visit to the stadium since he was sacked three years ago.
And yet again it proved an unhappy hunting ground for the man who is now at the Ulster helm.
Williams and Co arrived full of confidence in the wake of their shock away success against Munster. But they could not recover from two first-half tries by the hosts.
Edinburgh raced off to the ideal start, snatching the lead with less than a minute on the clock.
The entire visiting backline were offside as Phil Godman launched a probing cross-kick — and he followed up to calmly slot the penalty from almost 40 metres.
Even better was to come when Hugo Southwell completed a brilliant move with a try.
Shell-shocked Ulster broke their duck with a penalty by Ian Humphreys but Godman replied in identical fashion to preserve the Gunners’ lead.
Edinburgh’s enterprise was further rewarded midway through the half with a second touchdown.
A teasing chip from skipper Mike Blair sparked panic among the Ulster back ranks and debut flanker Roddy Grant underlined his talent by reaching the loose ball first.
They recycled quickly — enabling giant lock Jim Hamilton to flop over the line from a couple of paces out.
The Belfast brigade repaired some of the damage with two more penalties by Humphreys.
Gunners squandered a spate of chances after the restart and when Humphreys struck again the match was back in the melting pot.
But Godman eased the home jitters by finding the target at the other end.
A late Paddy Wallace penalty salvaged a bonus point for Ulster.
Edinburgh scorers: Tries: Southwell, J. Hamilton. Cons: Godman. Pens: Godman 3.
Ulster: Pens: Humphreys 4, Wallace.
EDINBURGH: Southwell, A. Turnbull, Cairns, Houston, Webster, Godman, M. Blair, Jacobsen, Ford, G. Cross, C. Hamilton, J. Hamilton, S. Cross, Grant, Hogg.
Replacements: De Luca for Cairns (61), D. Blair for Godman (68), Kelly for Ford (65), Kerr for G. Cross (56), Gissing for J. Hamilton (65), Newlands for Hogg (65).
ULSTER: Schifcofske, McCrea, Cave, Wallace, Trimble, Humphreys, Boss, Court, Best, Botha, O’Donoghue, McCullough, Ferris, Pollock, Diack.
Replacements: Cunningham for Cave (67), Steinmetz for Humphreys (69), Marshall for Boss (73), J. Fitzpatrick for Court (50), Brady for Best (68), Dawson for Pollock (62), Paulo for Diack (32).
Referee: Gwynn Morris (Swansea).