Kris Commons scored from the spot in the 18th minute after Gorgie midfielder Jamie Hamill was controversially penalised for handball before Jason Holt equalised in the 56th with a deflected drive from the edge of the box.
Anthony Stokes put Celtic back in front after 65 minutes with an unstoppable volley from 12 yards before substitute Pukki came on for Commons.
The transfer window signing from Schalke 04 made it 3-1 five minutes from time after great work from Stokes.
The win had lifted Celtic level on points with Inverness but Caley later reclaimed sole leadership when Billy McKay hit his sixth goal in as many games to give them a 2-1 victory over Kilmarnock at Rugby Park.
James Collins’ first strike for Hibernian just before the hour mark gave Pat Fenlon’s side their second away win in a row as they edged out St Johnstone 2-1 at McDiarmid Park. Aberdeen picked up their first league win since the opening day with a dominant 3-0 victory at Partick Thistle.
Dundee United moved into the top six of the Scottish Premiership yesterday with a clinical display of finishing as they beat Ross County 4-2 at Global Energy Stadium.