Mr Chavez's coffin set off in a procession to the Military Academy, where he will lie in state till Friday.
The government has announced seven days of mourning for the president, who died aged 58 after 14 years in the post.
Mr Chavez, a controversial figure and staunch critic of the US, was seriously ill with cancer for more than a year.
A self-proclaimed revolutionary, he inspired a left-wing revival across Latin America.
Dramatic images are being broadcast on Venezuelan TV stations, as the hearse of Hugo Chavez crosses the city surrounded by a sea of red flags. People are crying and sobbing, screaming the name of the late president, many wearing red T-shirts and carrying his images.
"We carry you in our heart," said one woman sobbing on live television. "Long live Hugo Chavez," she screamed.
Foreign presidents, such as Bolivia's Evo Morales, are taking part in the procession, together with Mr Chavez's family members, Vice President Nicolas Maduro and all top government officials.
This is just the beginning of public events to mourn Mr Chavez. The funeral will be held on Friday, and even greater crowds are expected.
Latin American leaders have begun arriving in Caracas to pay their respects - among them President Cristina Fernandez de Kirchner of Argentina, Jose Mujica of Uruguay and Evo Morales of Bolivia.
Ecuador, Argentina, Brazil, Bolivia, Uruguay, Chile, Cuba and the Caribbean island of Dominica have declared periods of official mourning.
Another Chavez ally, Iranian President Mahmoud Ahmadinejad, also announced a day of mourning, describing him as a "martyr".
Military units across the country have fired a 21-gun salute in his honour. They will fire another cannon shot each hour until he is buried, the armed forces said.
All schools and universities have been shut for the week.'To the pantheon'
On Wednesday morning, a priest prayed for eternal rest for Mr Chavez in a brief ceremony at the hospital where he died on Tuesday.
Officials then put the flag-draped coffin on top of a waiting hearse surrounded by crowds.
The procession began its slow journey through the streets of the city, led by officials including Vice-President Nicolas Maduro and accompanied by cheering red-clad supporters.
Some shouted "Chavez to the pantheon", referring to the mausoleum he built for revolutionary leader Simon Bolivar.Continue reading the main story
Previous
Life of people's hero and villainWatch
Chavez 'not dead to us'Watch
World reaction to deathWatch
Meet Chavez's chosen successorWatch
Hugo Chavez's death announcedWatch
Tears and fireworks on streetsWatch
'Eerie quiet' as Cuba mournsWatch
Chavez in his own wordsWatchNext"I'm here to say my final goodbye to my president. There will never be another Chavez. He is the greatest man that this fatherland gave us," said Jose Gregorio Conde, 34, an education worker, quoted by AFP news agency.
Mr Chavez's illness prevented him from taking the oath of office after he was re-elected for a fourth term in October.
Announcing the president's death on Tuesday, Vice-President Nicolas Maduro called on the nation to close ranks after its leader's demise.
"Let there be no weakness, no violence. Let there be no hate. In our hearts there should only be one sentiment: Love."
Police and troops would be deployed nationwide "to guarantee the peace", he added.
A statement from the military said it would remain loyal to the vice-president and to parliament, it added, urging people to remain calm.
Vice-President Maduro will assume the presidency until an election is called within 30 days.
Vice-President Nicolas Maduro was emotional as he announced Hugo Chavez's death
Foreign Minister Elias Jaua told state television that Mr Maduro would also be the candidate of the governing United Socialist Party (PSUV).
Opposition leader Henrique Capriles, whom Mr Chavez defeated in October's election, offered his condolences to Mr Chavez's family, saying "we were adversaries, but never enemies".
Mr Capriles is widely expected to be chosen to stand against the vice-president.
The BBC's Irene Caselli, in Caracas, says Mr Maduro will probably win, but the question remains whether he will be able to lead Venezuela following the loss of its charismatic president.Plot?
The exact nature of Mr Chavez's cancer was never officially disclosed, leading to continuing speculation about his health, and he had not been seen in public for several months.
Timeline: Hugo Chavez
1954: Born 28 July in Sabaneta, Barinas state, the son of schoolteachers
1975: Graduates from Venezuelan Academy of Military Sciences
1992: Leads doomed attempt to overthrow government of President Carlos Andres Perez, jailed for two years
1994: Relaunches his party as the Movement of the Fifth Republic
1999: Takes office after winning 1998 election
2002: Abortive coup. Returns to power after two days
2011: Reveals he is being treated for cancer
2012 (October): Re-elected for another six-year term
2012 (December): Has fourth cancer operation in Cuba
2013 (February): Returns to Venezuela to continue treatment
2013 (March): Death is announced by Venezuelan government
Last May, the former army paratrooper said he had recovered from an unspecified cancer, after undergoing surgery and chemotherapy in 2011 and a further operation in February 2012.
Despite this, he won another six-year presidential term in October 2012.
Mr Maduro has mentioned a plot against Venezuela, saying he had no doubt that Mr Chavez's cancer, first diagnosed in 2011, had been induced by foul play by Venezuela's enemies. The US promptly rejected the accusations as "absurd".
Two US diplomats had been expelled from the country for spying on Venezuela's military, Mr Maduro added.
Mr Chavez burst onto Venezuela's national stage in 1992 to lead a failed military coup.
After two years in prison, he returned to politics and was swept to power in a 1998 election.
A self-proclaimed socialist and revolutionary, he won enduring support among the poor and repeated election victories by using Venezuela's oil wealth to pursue socialist policies.
His government has implemented a number of "missions" or social programmes, including education and health services for all.
But his opponents accused him of mishandling the economy and taking the country towards dictatorship. Inequality has been reduced but growth overall has been lower than in some other Latin American economies.
Internationally, he was a staunch critic of US "imperialism" and accused Washington of backing a failed coup against him in 2002.
The US described the death as a "challenging time", reaffirming what it described as its support for the Venezuelan people and its interest in developing a constructive relationship with Caracas.
Analysts say Mr Chavez's death could alter the political balance in Latin America - dealing a blow to leftist states while favouring more centrist countries.
There could also be an economic impact given that Venezuela sells oil at below market prices to some neighbouring countries, especially in the Caribbean.
