Johnson City Mayor Phil Roe beat Davis by a 500-vote margin Thursday in the solidly Republican 1st District in the northeastern corner of the state.
Roe's victory came after a bruising campaign in which he accused Davis of selling out to "big oil."
Congressional incumbents from Tennessee are rarely voted out of office. Statewide, the last time an incumbent was defeated in a party primary was 1966 when Democrat Tom Murray lost to Ray Blanton in what was then the 7th District. Blanton won the general election then became governor in 1974.
"I will try to serve you with dignity and honesty, just like we ran this campaign," Roe said. "Ain't it fun to win one?"
With all precincts reporting, Roe had 25,916 votes, or 50 percent of the vote, to Davis' 25,416 votes, or 49 percent. He will face Democrat Rob Russell in November, although the district has voted strongly Republican in the past.
The race became increasingly acrimonious as the primary election neared. Roe ran a TV ad accusing Davis of selling out to "Big Oil" by accepting money from industry PACs and backing legislation supporting offshore drilling.
During the last month of the campaign, gas prices in the district hit a record high of $3.94, according to figures from AAA.
Davis countered with radio ads denying he "pocketed" oil money, accused Roe of deceptive campaign practices and said "the voters of East Tennessee deserve better."
Roe told reporters afterward that he was aware Davis hadn't acknowledged defeat, but said he considered his victory final.
"I knew the time was right, and I felt I was the right person to do it," Roe said of his challenge of Davis.
Davis, a health care business owner, was elected in 2006 after a crowded primary for a vacated seat. Roe came in fourth in that primary.
Norman Litchfield, 24, a medical student from Johnson City, said he voted for Roe because he thinks "change is good."
"I've been to D.C. several times with my sister for school. (Davis) kind of gave us a cold shoulder," Litchfield said. "I think your representative should take the time of day to come out and shake your hand. I like Roe's 'people not politics' platform."
Davis joins Rep. Chris Cannon, R-Utah; Rep. Wayne Gilchrest, R-Md.; and Rep. Albert Wynn, D-Md., who are also incumbents who lost their primary races this year.
Meanwhile, in the 9th District, in Memphis, a racially charged Democratic primary ended with an incumbent congressman trouncing the opponent who ran an ad linking him to the Ku Klux Klan.
Unofficial results showed Democrat Steve Cohen with 79 percent of the vote to 19 percent for Nikki Tinker, a black corporate lawyer who was his chief opponent in the district that covers Memphis, with all precincts reporting.
Cohen is the first white congressman from Memphis in more than three decades and one of only two white congressmen representing a majority black district.
"The results are pretty clear," Cohen told cheering supporters at a victory party. "I'm here to report that Tennessee and Tennessee (District) 9 voted firmly for the post-racial politics that has carried a new generation to power."
The campaign turned ugly in its final days, when Tinker ran a television ad juxtaposing photos of Cohen, who is Jewish, and a hooded Ku Klux Klan member. Tinker's supporters argued the district, which is 60 percent black and 35 percent white, should be represented by a black candidate. Cohen, a former state senator, has a long record as a civil rights supporter.
Tinker said her ad linking Cohen to the KKK for opposing a 2005 effort to remove a statue of Confederate Gen. Nathan Bedford Forrest from a downtown park "merely states the facts. I think the nation needs to know Steve Cohen's complete record."
But the ad drew condemnation Thursday from Democratic presidential candidate. It juxtaposed pictures of a statue of Forrest, an early leader of the KKK, and a hooded Klansman in front of a burning cross while asking, "Who is the real Steve Cohen?"