Menzi Masuku had given Phunya Sele Sele the lead just after the half time break, but with the picture looking bleak in the dying moments of the match on a wet night in Pietermaritzburg, Ugandan international Kateregga stole the show with two goals inside the final 10 minutes to earn a crucial three points.
The Team of Choice made a bright start to the game and had early chances as Mxolisi Kunene miscued a shot from the edge of the box before Mohau Mokate went well wide with a header following a cross from Thabiso Kutumela.
Kutumela was to have the ball in the net in the 14th minute but the goal was ruled out off-sides.
At the other end, Ndumiso Mabena nearly snatched the lead against the run of play for Celtic when he drove an angled shot into post.
There were also a couple of half chances for Kabelo Dlamini, but it was Maritzburg who were doing most of the attacking, as well as earning a series of corner kicks.
Celtic’s defence was however well organised and dealt well with a series of testing balls into the box from the home side for the remainder of the first half.
The Team of Choice were to suffer a setback just two minutes after the restart as Phunya Sele Sele broke on a counter attack through Mabena, who fed Dlamini who in turn set up Masuku for a simple finish.
United came back fighting for the equaliser, but fortune continued to elude them in front of goal as Kutumela had a shot saved by Patrick Tignyemb before his cross landed on the crossbar.
At the other end, Dlamini could have added a second goal, but fired the parting shot well wide.
Maritzburg meanwhile continued to try and get back into the match, but Yannick Zakri and Bandile Shandu both failed to capitalise in front of goal.
With the clock moving towards 80 minutes, time was running out. Enter Kateregga who struck with a sweetly-taken volley after Judas Moseamedi had combined with Shandu.
The winner was even more spectacular as Kateregga picked up possession all of 30-yards out before scorching a rocket of a shot into the top corner of the net, much to the delight of the full house who had turned out on Friday night.