"One major negative that this market could endure is if we see oil heading towards $72 [a barrel]," said Peter Cardillo, chief market economist at Avalon Partners.
In recent action, crude oil was down 28 cents at $68.79. It earlier reached $69.13 amid concerns about production from Nigeria and refining capacity in the U.S.
The Dow Jones Industrial Average was down 14 points at 13,599, as 17 of its 30 components retreated, led by Hewlett-Packard Co. and and Procter & Gamble .
Among blue chips, home-improvement retailer Home Depot Inc. bucked the trend, rising 0.2%. after it agreed to sell its building supply unit for $10 billion.
Yet retailers came under pressure as Best Buy Co. Inc. fell 4.8%. The company posted an 18% decline in quarterly profit, as the popularity of lower-margin notebook computers and gaming hardware ate into profits, and pruned its forecast for the year.
The S&P 500 fell 3.4 points to 1,527, and the Nasdaq Composite lost 8.20 points to 2,618.
Among tech shares, Yahoo Inc. gained 0.6% after the Internet search giant and Google Inc. rival named co-founder Jerry Yang as chief executive, replacing Terry Semel.
Homes and rates
Starts of new U.S. homes fell by 2.1% to a seasonally adjusted annual pace of 1.47 million in May, as building permits for new construction rose 3% to 1.50 million on a jump in multifamily dwellings, the Commerce Department estimated Tuesday.
The figures were slightly stronger than economists expected.
"The fact that building permits are up is a sign that there might be some consolidation in housing," Cardillo said.. "But yesterday, the [National Association of Home Builders'] housing index was at its lowest in 16 years."
The bond market, where prices have come under pressure and yields have spiked over the past couple of weeks, failed to react much to the latest housing news. The benchmark 10-year Treasury bond was recently up 3/32 to 95 6/32, yielding 5.127%.
Higher bond yields had pressured the stock market over the past couple of weeks, as they provide a risk-free alternative to stocks, while also raising borrowing costs for consumers and businesses.
"Higher yields have now been discounted by the market," said Avalon's Cardillo. "Some of the economic data that we're continuing to see has come better than expected, and the market is now seeking a consolidation level."
In other markets, the dollar was lower against the euro and the yen.
Gold fell, losing $1.5 to $655.60 an ounce, tracking crude oil lower.
Bristol-Myers Squibb Co. gained 2.3% after the Food and Drug Administration granted priority review for its ixabepilone breast cancer drug.
By Nick Godt