Olesz got the winner on a sharp-angle shot from the bottom of the right circle that slipped past Devils goaltender Johan Hedberg.
Chris Higgins, Dennis Wideman and Mike Santorelli also scored for the Panthers, 2-3-4 in their last nine games.
Florida cooled off the Devils, who are 8-2-1 in their last 11. Travis Zajac, Jason Arnott and Mark Fayne scored for New Jersey.
Higgins opened the scoring at 6:52, backhanding Steve Bernier's rebound past Hedberg.
The Devils scored the next two goals thanks to lucky bounces off the boards.
Zajac pulled New Jersey even with an unassisted tally at 8:52, capping a sequence that started with Vokoun sweeping the puck into the corner. Panthers defenseman Jason Garrison tried for a clear around the boards, only to have the puck take a weird bounce off the glass into the slot where Zajac tapped it home.
The Devils went up 2-1 as Arnott banked the rebound of Henrik Tallinder's point shot that deflected wide of the net. The puck again bounded New Jersey's way, hopping off the boards right to Arnott.
Wideman pulled the Panthers even at 12:35 with a power-play tally on a point shot with Michael Frolik providing a screen in front of Hedberg.
Fayne broke the deadlock at 6:20 of the second, chipping in a high shot from the blueline. Devils forward Nick Palmieri, positioned in the slot, ducked out of the way as the puck floated past Tomas Vokoun.
Later in the period, both goalies made big stops on short-handed breakaways to keep the score 3-2 in favor of the Devils.
Vokoun denied Patrik Elias' bid while Hedberg poke-checked the puck away from Marty Reasoner.
Hedberg was sharp early in the third, denying Rostislav Olesz when the Panthers charged the net. He later made a pair of bang-bang saves, stopping Wideman's point shot and Reasoner's rebound attempt.
Hedberg had no chance when Santorelli tied the game at 3 on a short-handed breakaway at 12:52.
Santorelli picked off a pass in the Panthers zone and sped toward Hedberg, who charged out to challenge the shooter. Santorelli faked him down and out before depositing the puck into the open net.
Zajac made a strong bid for the winner in regulation, hitting the post with 5 minutes left.
Notes: The Panthers played their third game in four nights coming out of the All-Star break. ... The Devils assigned D Alexander Urbom to Albany (AHL). He was recalled on Tuesday. ... The Panthers scratched RW Evgeny Dadonov (broken finger). ... The Devils' scratches were D Mark Fraser and C Adam Mair. ... Friday was the one-year anniversary of the trade that brought Ilya Kovalchuk to New Jersey from Atlanta. ... This concluded the Panthers' season-high six-game road trip (2-3-1).