With the Blue Jackets trailing 3-2 and 6:16 left, Umberger tied it with a nifty move and a snap shot. He then fed Huselius on a 2-on-1 break, with Huselius tucking a wrist shot inside the near post and past goalie Nikolai Khabibulin.
Magnus Paajarvi had a career-best two goals for the Oilers, who lost their fourth in a row - both goals coming on almost identical plays off assists from fellow rookie Linus Omark. Dustin Penner also scored.
Steve Mason was solid in goal for Columbus, allowing the Blue Jackets to come back from an early 2-0 deficit. He finished with 23 saves.
Huselius also assisted on one of Umberger's goals. Kris Russell also had two assists.
The win was critical for the Blue Jackets, who have little margin for error as they try to stay in the chase for a playoff spot in the Western Conference. They began the night six points out of the eighth and final qualifying spot.
On the tying goal, Umberger showed the puck to Omark and then pulled it back between the winger's skates before stepping into a shot from near the left dot.
Huselius had scored just one goal over the past 13 games and had been a healthy scratch in Friday night's 3-0 victory at Detroit.
Omark pivoted in the left corner near the goal line and whipped quick passes through the crease that Paajarvi converted for the Oilers' first and third goals. It was the first two-goal game of Paajarvi's NHL career.
The teams entered the third period tied at 2 after the Blue Jackets fought back to pull even on Huselius' first goal.
Umberger made a great effort play after Huselius was wide with a shot from the slot. Umberger collected the puck on the back boards, skated to the side of the net and sent a pass back to Huselius, who mishit the puck but it went in off Khabibulin's leg pad.
The Oilers came in last in the NHL with only 15 wins (in 51 games) and 38 points. But they needed just 7:44 to take the 2-0 lead.
After Omark's pass to Paajarvi for the first goal, Penner took a pass from Ales Hemsky and wound up on a hard one-timer from the high slot for his 17th goal while Shawn Horcoff supplied a screen of Mason.
Khabibulin, who had 30 saves, made several nice stops to preserve the lead. But the Blue Jackets finally broke through late in the first. Umberger carried the puck across the blue line along the right boards, then skated laterally through the high slot before unleashing a wrister that trickled in.
Notes: John Moore, a touted 20-year-old defenseman who was the Blue Jackets' first-round pick in the 2009 draft, made his NHL debut. ... Mason had stopped 34 shots on Friday night in Columbus' first-ever shutout win in Detroit.