His death came on the day his eldest son, Andrew Cuomo, delivered inaugural addresses in Manhattan and Buffalo, New York, after being sworn in for his own second term as governor.
The governor's office said in a statement the elder Cuomo, who served as New York's 52nd chief executive from 1983 through 1994, had died of "natural causes due to heart failure this evening at home with his loving family at his side."
The former governor, long a celebrated orator who was a favorite of the Democratic Party's progressive contingent, was hospitalized on Nov. 30 for treatment of a heart condition.
His younger son, Chris Cuomo, of CNN's "New Day," informed the network shortly before 5 p.m. Thursday that his father had died, CNN reported. During that time Andrew Cuomo was speaking at the Erie and Buffalo County Historical Society.
In his inauguration address on Thursday, Andrew Cuomo said he had read his speech to his father the night before.
“He said it was good, especially for a second termer,” the younger Cuomo said. “He couldn’t be here physically today, my father. But my father is in this room. He is in the heart and mind of every person who is here."
President Barack Obama, in a statement issued from Hawaii, where he was vacationing, saluted the former governor as "an unflinching voice for tolerance, inclusiveness, fairness, dignity and opportunity."
White House spokesman Eric Schultz said Obama called Andrew Cuomo by telephone to extend condolences.
New York City Mayor Bill de Blasio, remembered Mario Cuomo as "a man who campaigned with poetry and governed with beautiful prose."
CHALLENGED 'SHINING CITY'
Mario Cuomo was first elected as governor in 1982 and came to national attention two years later when he gave an electrifying keynote address at the Democratic National Convention in San Francisco, criticizing the policies of then-President Ronald Reagan and challenged Reagan's metaphor likening America to a "shining city on a hill."
Cuomo countered by saying, "A shining city is perhaps all the president sees from the portico of the White House and the veranda of his ranch, where everyone seems to be doing well.
"But there's another city; there's another part to the shining city; the part where some people can't pay their mortgages, and most young people can't afford one; where students can't afford the education they need, and middle-class parents watch the dreams they hold for their children evaporate."
His speech defining Republicans as looking out only for the well-off and Democrats as champions of the middle class and the poor propelled Cuomo to the forefront of the party leadership.
After easily winning re-election to a second term as governor, Cuomo was the apparent front-runner for the 1988 Democratic presidential nomination.
But the filing deadline came and went, and Cuomo's reputation as a reticent, Hamlet-like figure began to grow.
A similar scenario came about in 1992 with Cuomo again the focus of Democratic presidential anticipation. But he said state budget problems needed his attention and declined to run again.
Mario Matthew Cuomo was born in the New York City borough of Queens on June 15, 1932, the son of parents who arrived from Salerno, Italy, during a wave of immigration in the 1920s.
His father dug ditches to earn money to buy a pushcart, then opened a small grocery store in a tough neighborhood.
Mario earned a name first as a baseball player, signing a minor league contract with the Pittsburgh Pirates after high school. An injury ended his career and he went back to college and law school at St. John's University.
His law practice brought him into politics in the mid-1960s when he helped residents fight a city plan to level small scrap-processing plants near the site of the 1964 World's Fair. Cuomo won the fight and saved hundreds of jobs.
For the next 10 years he repeatedly fought the power structure, winning concessions for residents in housing, education and welfare.
When a law school friend, Hugh Carey, was elected governor in 1974, Cuomo joined his administration as secretary of state. In 1977 he returned to New York City to run for mayor.
He finished a close second but lost the run-off to Ed Koch, who blistered Cuomo's opposition to the death penalty in what became a long-running Democratic rivalry. He then beat Koch in their run for the governorship in 1982 with heavy support from upstate New York.
As governor, Cuomo increased spending for education and social welfare, cut the state's highest income taxes and spoke out against the death penalty and for minority rights.
But budget problems and voter weariness after 12 years in power helped little-known George Pataki, a Republican, knock off Cuomo in 1994, and he later retired from politics.
In addition to his two sons, Cuomo is survived by his wife of 60 years, Matilda, and their three daughters, Maria, Margaret and Madeline, and 14 grandchildren.
The governor's office said funeral arrangements would be announced soon.