In his historic speech, Harper said the treatment of children at the schools, where they often suffered from physical and sexual abuse, was a sad chapter in the country's history.
"We now recognize that it was wrong to separate children from rich and vibrant cultures and traditions, and that it created a void in many lives and communities and we apologize," he said in an address to Parliament televised live across Canada.
From the 19th century until the 1970s, more than 150,000 aboriginal children were required to attend state-funded Christian schools as part of a program to strip them of their native culture and assimilate them into Canadian society.
"These institutions gave rise to abuse or neglect and were inadequately controlled and we apologize for failing to protect you," Harper said.
Hundreds of former students were invited to Ottawa to witness what native leaders say is a pivotal moment for Canada's more than 1 million aboriginals, who today remain the country's poorest and most disadvantaged group.
There are more than 80,000 surviving students of the schools.
Eleven aboriginal leaders watched the apology from the floor of the House of Commons and hundreds watched from the public gallery and from the front lawn of Parliament.
The apology came just months after Australian Prime Minister Kevin Rudd made a similar gesture to the so-called Stolen Generations - thousands of the continent's Aborigines who were forcibly taken from their families as children under assimilation policies that lasted from 1910 to 1970.