Brian Rolston had tied the game earlier in the period on a power play and Johan Hedberg made 31 saves as the Devils posted their 12th win in 15 games (12-1-2) in keeping their improbable playoff push in gear.
Patrick Marleau scored for San Jose, which had a season-high five-game winning streak snapped and lost for the first time in regulation in 11 games (9-1-1).
Kovalchuk had given the Devils a 2-1 win over Toronto with 23.2 seconds left in overtime on Thursday night with a great rush up the ice.
This game winner was ordinary, except for the shot. Travis Zajac won a faceoff in the Sharks' end and Kovalchuk swooped in from the left side of the circle, skated to the right edge and snapped a shot that beat goaltender Antti Niemi.
Any chance the Sharks had of tying ended quickly when Torrey Mitchell picked up a four-minute high sticking penalty to put the Devils on a power play the rest of the game.
This is a game the Sharks seemed to have the upper hand for the first 47 minutes.
Marleau tapped in the rebound of a point shot by Niclas Wallin at 6:02 of the third period to give San Jose a 1-0 lead, and things looked bad for New Jersey when defenseman Henrik Tallinder picked up a hooking penalty a little more than a minute later.
Instead of capitalizing, the Sharks picked up their first two penalties as Joe Thornton was called for hooking and Logan Couture was whistled for a tripping penalty on an odd-man Devils' rush.
New Jersey cashed in quickly on the 4-on-3 advantage as Rolston took a pass from Kovalchuk and ripped a shot past Niemi, who was looking for his third shutout in four games.
Hedberg, starting his third straight game with Martin Brodeur sidelined with a sprained right knee, stopped Dany Heatley and Kyle Wellwood in close before Kovalchuk delivered again.
For much of the first two periods, the Devils showed the effects of playing their seventh game in 11 days. They looked a step slower than the Sharks and the puck stayed in their end much of the opening 40 minutes.
New Jersey's only action came in the opening two minutes of the second period, when Rolston had a shot between the circles and Tallinder had two shots from the right points. The shot total was one more than they had in the first period.
While San Jose had the territorial advantage and a 24-11 edge in shots, it didn't have a lot of great scoring chances. The best might have been Ryane Clowe's deflection of a Wellwood centering pass with 14:49 left in the second period.
Notes: Defenseman Jay Leach, who was acquired in a minor league trade between the Devils and Sharks on Wednesday, was called up from Albany for the game, but was a healthy scratch. ... Mitchell returned to the lineup after missing 10 games with a lower body injury. ... The Sharks beat New Jersey 5-2 in San Jose on Oct. 27. ... San Jose has not won in New Jersey since Nov. 14, 2000.