Mr Jebali had been trying to form a new coalition in response to the political crisis sparked by the killing of opposition leader Chokri Belaid.
He had said he would quit if his Islamist Ennahda party did not back his plan for a cabinet of technocrats.
Mr Belaid's assassination on 6 February provoked mass protests and resignations from Tunisia's coalition government.'People disillusioned'
Tunisia in the Arab Spring
Dec 2010: Trader Mohammad Bouazizi sets himself on fire in Sidi Bouzid, sparking anti-government protests, and later dies
Jan 2011: Zine al-Abedine Ben Ali stands down as president after escalating protests and clashes
March 2011: New technocratic government announced by interim president
Oct 2011: Tunisians vote in first elections of Arab Spring
Dec 2011: Ex-dissident Moncef Marzouki chosen as president, with Ennahda's Hamadi Jebali as PM
May 2012: Salafi Islamists clash with police amid a row over alcohol sales
June 2012: Mr Ben Ali is sentenced to life in prison over the killing of protesters in the 2011 revolution. Saudi Arabia refuses extradition
Feb 2013: Opposition politician Chokri Belaid shot dead; PM announces plans for technocratic government, a move rejected by Ennahda.
"I vowed that if my initiative did not succeed, I would resign and I have done so," Mr Jebali told a news conference after meeting President Moncef Marzouki.
Describing his step as "a big disappointment", he said he was standing down to "fulfil a promise made to the people."
"Our people are disillusioned by the political class. We must restore confidence," he stressed.
And he added: "The failure of my initiative does not mean the failure of Tunisia or the failure of the revolution," in a reference to the popular unrest two years ago that ousted autocratic leader Zine al-Abidine Ben Ali.
Mr Jebali's resignation comes despite comments by Ennahda's leader Rached Ghannouchi on Monday that all parties involved in the coalition building talks had wanted the prime minister to remain in office.
The Ennahda has also indicated that it still hopes it can pull together a coalition to lead the country into early elections, the BBC's diplomatic correspondent Bridget Kendall reports.
Mr Ghannouchi is expected to meet President Moncef Marzouki on Wednesday to discuss the growing crisis, reports say.
But in the meantime, our correspondent says, the worry is that the continuing political stand-off and paralysed economy could lead to new mass protests and more violence.
On Tuesday, credit rating agency Standard and Poor's said it had downgraded Tunisia's rating, citing "a risk that the political situation could deteriorate further amid a worsening fiscal, external and economic outlook".
Supporters of the secular opposition have blamed Ennahda for Mr Belaid's assassination - an accusation the party denies.
It was the first political assassination in Tunisia since the popular unrest in 2011 forced the country's autocratic leader Zine al-Abidine Ben Ali from power and also inspired the so-called "Arab Spring" uprisings across the region.
