Gillmor was known for his quiet approach and spent nearly 20 years in Congress without drawing much attention to himself.
As a House member he was a solid Republican vote, a reliable conservative on social issues who was a strong proponent of the military and spoke out against abortion.
"He knew how to quietly get things done," said Mary Rose Oakar, a former Democratic U.S. representative from Cleveland. "I'm not saying he wasn't effective. He just was unassuming and quietly did his homework."
Aides found Gillmor's body at his town house in Arlington, Va., after he failed to show up at the Capitol for morning meetings, said John Lisle, a spokesman for Arlington County police.
Homicide investigators were called to the apartment, but foul play was not suspected, Lisle said. "It does not appear, at least preliminarily, to be anything suspicious," he said.
A cause of death had not been released.
"Congressman Gillmor's death comes as a great shock to us all," said a statement released by his office. "Representative Gillmor served the people of Ohio with every ounce of his soul and today he passed on doing the job he loved."
His seat will remain open until a special election is held, said John McClelland, a spokesman for the Ohio Republican Party. Gov. Ted Strickland will set the date.
Gillmor was first elected to Congress in 1988 to a seat he had eyed for years.
Although easily elected to each term, he did face significant primary fights during his career in the 5th District. He first won the GOP nomination by eking out a 27-vote victory over Robert Latta, who was trying to follow in the footsteps of his father, Republican Rep. Delbert Latta.
His long career crossed paths with many state leaders.
"Paul was one of the most conscientious public servants I've ever met," said GOP U.S. Sen. George Voinovich from Ohio. "Paul was a role model for all of us who serve in public office."
Strickland, a Democrat, was a member of the Ohio Congressional delegation for 12 years with Gillmor.
"Paul was a friend and a colleague," Strickland said. "Ohio has lost a truly decent and devoted public servant."
Gillmor led efforts to cleanup industrial contaminated sites and enact financial service reforms. He also was a strong advocate of a constitutional amendment to ban unfunded mandates on the states.
"He wasn't a headline grabber," said Tom Wiseman, a former mayor of Defiance, which is part of Gillmor's district. "He was a doer."
Wiseman said Gillmor also worked to support programs that help veterans. "That was one area that very few knew about," he said.
Gillmor served as a Vietnam War-era judge advocate in the Air Force before going into politics.
He had been in Ohio in recent weeks to attend a series of town meetings and tour areas of the state that were hit hard by flooding.
The son of a banker, he was born in Tiffin and grew up nearby in the small town of Old Fort.
"Born, raised and educated in our home state of Ohio, Paul never lost sight of the reason he came to Congress - to serve this great institution and his constituents with dedication and distinction," House Republican Leader John Boehner, also of Ohio, said in a statement.
A career politician, Gillmor became a state senator when he was 27, eventually rising to Senate president.
He spent 22 years in the Ohio senate and made an unsuccessful run for governor in 1986.
In recent years, Gillmor fought off accusations that he ignored his district and spent almost all of his time at his homes near Washington and Columbus.
"I was born and raised in the district," Gillmor said during a bitter 2002 primary campaign. "I consider it my home."
He is survived by his wife, Karen, two daughters and three sons.