​Where have all the golfers gone?

Just a decade ago, it looked like the golf industry had hit a hole-in-one. Interest in the game was surging, and consumers snapped up equipment and booked tee times.

These days, though, the sport is caught in the rough.

Golf is losing more players than it's gaining, as 4.1 million people left the sport in 2013, outpacing the 3.7 million who picked it up last year. With an overall decline in the number of players, that's causing a ripple effect in the golfing world: Sales of clubs and other equipment are plunging, while some courses are pulling up their tees and calling it a day.

"The main issues that cause people to leave the game and not to try the game are the same. They boil down to time, cost and difficulty," said Steve Mona, the chief executive of the World Golf Foundation.

Golf reached what Mona calls its "high water mark" in 2005, when the sport had 30 million participants playing 550 million rounds of golf. Last year, that had shrunk to 25 million participants and 465 million rounds.

The recession had an impact, as well as tough winter and spring weather the past two years, Mona noted. But the industry is also aware that it needs to overcome some perception issues about the sport's expense, the length of a game and the time it takes to master golf, he added.

To tackle those perceptions, some courses are introducing variations on the game, such as "foot golf," which uses soccer balls instead of golf balls, and the industry is reaching out to under-represented groups, such as women, minorities and the Millennial generation, Mona added.

"The core group who loves golf will stick with it, but we're worried about those people who have not taken to it totally," said Trillium Sellers, a pro golfer who is a member of the PGA and is working on the golf association's task force for player development. "They are in that make-or-break state, where they might try it, but if they don't like it, they won't come back again."

With fewer people opting to play, that's hurting retailers and courses alike.

Take Dick's Sporting Goods (DKS), which on Tuesday said its golfing division had weighed down quarterly results, with same-store sales at its Golf Galaxy outposts slipping by 10.4 percent. Shares of Dick's Sporting Goods fell 13 percent the following day.

"We don't feel we found that bottom yet in the golf sales number," chief executive Ed Stack said on a conference call with analysts. "We now expect this trend to continue for the balance of the year."

Across the country, golf courses are calling it quits. More courses closed than opened in the U.S. last year: While 14 18-hole courses were christened in 2013, a whopping 157 closed. The majority of those closures were public courses.

What's causing golfers to put away their clubs?

The sport's high costs are one issue, given even a low-ball price tag pegs the cost at about $700 in course fees, apparel and equipment. At the same time, consumers are increasingly strapped for money, including those middle- and upper-middle-class consumers who are most likely to visit a golf course. Median per capita income for American households has barely budged since 1989 in inflation-adjusted dollars.

Women golfers are another problem spot for the sport because they're more likely to join but also more likely to quit, Sellers noted. Over the past five years, the number of women and junior golfers has declined 23 percent and 35 percent, respectively, according to a report she prepared for the Club Managers Association of America.

"The perception is that A, golf is an old white man's sport, and B, it's very hard to learn," she said. "Golf courses and golf clubs are realizing that it's not about him anymore. It's about the wife, because she's usually the one making the final decision about where the money is spent."

One benefit to the downturn? It's a buyer's market for equipment, Mona noted, adding, "You can find pretty good deals. If you shop smartly, you can consume golf pretty reasonably."