The council unanimously voted for the probe, which will examine allegations of prison camps, slave labour and food deprivation in the country.
North Korea denounced the resolution as a political ploy.
It is highly unlikely the team will be granted access to North Korea, so they will have to rely on satellite imagery and accounts from defectors.
North Korea's human rights record will now be under intense scrutiny, and evidence gathered by the team could be used in future prosecutions for crimes against humanity.
UN special rapporteur Marzuki Darusman, who presented the initial report on North Korea and will be a member of the inquiry, said that a key focus should be the country's prison camps.
"The prison camps could qualify as crimes against humanity," he said. "These are camps which have the purpose of driving the people being detained there towards a slow death."
His report also described "widespread and systematic violations of human rights" including enforced disappearances and using food to control people.
UN High Commissioner for Human Rights Navi Pillay said the UN had evidence indicating that North Korea's political prisons held around 200,000 people, with many subjected to rape, torture and slave labour.
'Landmark step'
The resolution, which was presented by Japan and the European Union, was approved by all 47 members of the council.
North Korea's UN representative denounced the investigation as a "political ploy"
"For too long the population of the country has been subjected to widespread and systematic human rights violations and abuses," Ireland's Foreign Minister Eamon Gilmore said, speaking on behalf of the EU.
North Korea's ambassador to the UN, Pyong Se-so, called the resolution "a faked document full of political invective, with serious distortions."
He accused the council of seeking to "disgrace the image" of North Korea, adding that his country had "one of the best systems in the world for the protections of human rights".
The inquiry has been welcomed by activists. In a statement, Human Rights Watch described the move as a "landmark step", that would "help expose decades of abuse by the North Korean government".
The probe comes at a time of heightened tensions in the Korean peninsula, following North Korea's third nuclear test in February and the subsequent tightening of UN sanctions.
On Friday, the US voiced optimism that China would help enforce UN sanctions, a key part of which involves stopping money flow to Pyongyang's nuclear programme and illicit arms sales.
"We asked the Chinese for enhanced scrutiny of financial institutions in North Korea," said US Treasury Undersecretary David Cohen, who is visiting Beijing.
"It's no secret that there is a fair amount of financial relationship between China and North Korea and Chinese financial institutions in North Korea."
In recent days North Korean rhetoric against both the US and South Korea has escalated.
On Thursday, the North Korean army issued a statement threatening US military bases in Japan, in response to the US flying nuclear-capable B-52 bombers over South Korea as part of a joint military exercise.
