And that's pretty much the way it was for about 40 years - until a complaint from a disgruntled volunteer brought in the federal government, and investigators from the U.S. Department of Agriculture who insisted they have jurisdiction over the cats, CBS News investigative correspondent Sharyl Attkisson reports.
The complaint was about some of the cats, now numbering nearly four dozen, roaming off the property, which it turns out they were entitled to do since Key West has no leash law.
But once USDA got involved, it argued Hemingway cats were "exhibits" and needed the special protection of the same federal laws governing a zoo or circus.
Cara Higgins is an attorney for the Hemingway House and its feline population.
"They're not transferred, they're not moved, they're not disrupted, they're not eaten," Higgins said. "I can't imagine why the USDA, why the federal government, would have an interest in a handful of local cats."
First, USDA wanted the cats rounded up nightly and caged, according to the Hemingway House's Mike Morawski.
"Our vet who comes on the property weekly thought that was extremely traumatic for any of our cats, much less the cats that have lived on this property the last 10 to 15 years of their life," Morawski said.
Next, he says the USDA recommended an electric fence.
"We had one change after another, but the USDA demands kept coming," he said. "We had one cat that we did that experiment with and she ended up with two burn holes in her neck."
CBS News tried to scratch out exactly how many of your tax dollars are being spent on the whole cat fight, but USDA wouldn't provide the documentation or agree to an interview.
So we sifted through the litter and came up with this:
USDA agents even went so far as to go undercover.
"They, you know, pose as tourists and get pictures and surreptitiously tape the cats," Higgins said.
Has this all been a little silly on their part?
"It's been a lot silly," Morawski said.
Recently, the USDA hired its own cat expert. Even she made the Hemingway cats sound like some of the luckiest felines on the planet saying they're, "well cared for, healthy and content."
With the dispute entering its fifth year, the Key West sunset is unchanged and the cats oblivious.
But if they cared about the antics of humans, they might be amused by how much of your tax money is being spent trying to improve their charmed lives.