The rebels were repelled after two hours of fierce fighting, a Malian army official said.
He said the insurgents had slipped past army checkpoints to enter the town. Gao residents had raised the alarm, saying rebels had entered their neighbourhood.
Gao was controlled by an Islamist group for several months before it was liberated in a French-led offensive in January.
The MUJWA Islamist group had attempted to impose an extreme form of sharia on the town.
Gao Mayor Sadou Diallo said the Islamist fighters had launched Saturday's attack inside the city's Quatrieme Quartier, or Fourth District, and retreated when they were engaged by Malian forces.
"There was heavy gunfire. The situation is under control now. The Islamists entered via Quatrieme Quartier, and the army went to meet them and was able to push them back," Mr Diallo told the Associated Press news agency.
"There is another group that entered via the river, but they too were pushed back. It's under control."
No death toll was immediately available after Saturday's firefight.
French intervention
Islamist rebels seized vast swathes of northern Mali a year ago after a military coup in the capital Bamako.
France intervened militarily in January amid fears that the militants were preparing to advance on Bamako. It currently has about 4,000 troops in Mali.
Mali's army and troops from several African countries, including 2,000 from Chad, have also been involved in the fighting.
Since the intervention began, major cities including Gao, Kidal and Timbuktu have been recaptured but fighting is still continuing in desert mountains.
France plans to withdraw its troops from Mali next month, with West African countries expected to take over in the run-up to elections due in July.
