The fire is burning on a 10-square-mile area and about half of the perimeter now has fire lines built around it.
National incident team spokesman Terry Krasko said Saturday that the fire isn't spreading and that crews are focusing on putting out fires burning inside the perimeter.
The fire has burned at least 169 homes within that perimeter.
Nearly 1,000 firefighters from 20 states have been working on the fire but Krasko says eight engine crews will start leaving Saturday.
Calmer winds on the fire lines and the all-clear for thousands of people to return to their homes had both firefighters and anxious residents feeling more optimistic.
"Really, the best news is that we're looking for this thing to wind down very quickly at this point," Terry Krasko, spokesman for the federal team managing the fire, said late Friday.
About 2,000 people in areas that didn't burn were cleared to return home Friday. Another 1,000 still waiting to go back live where the fire is still burning.
At least 169 homes have been destroyed since the fire erupted Monday.
"It's here! It's still here!" said Susan DiPrima as she entered her two-story log house with an Associated Press reporter.
Less than half a mile from her home below Sugarloaf Mountain, the ground was scorched and trees were blackened and ashy. DiPrima wandered outside and greeted returning neighbors as the wind began picking up and a puff of smoke floated over a ridge about two miles away. She acknowledged that she still didn't feel safe.
Sheila Gloe's house near Sugarloaf Mountain was smoky because she left the windows open before the fire started and she didn't get a chance to go back.
"We have been inconvenienced, but we are blessed," Gloe said. "Nothing but applause for the people who helped."
As happy as people were to find their homes standing, they were also dealing with the realities of living on the edge of an active wildfire. Many were without power and phones, although utility crews were starting to restore electricity.
"There's no power, no phone, no gas, no nothin'. Even staying up there is futile," Tom Bethke, a geologist, said at a YMCA fire shelter.
D. Greenwald was concerned about security. She said she saw only one patrol car and only one checkpoint when she returned to her house Friday.
"We want entrances to the mountain to be supervised," Greenwald said.
Boulder County sheriff's Cmdr. Rick Brough said patrols are being increased and blockades will be kept in place. He said authorities haven't been able to check all the homes in areas where the fire is still burning and don't know when the rest of the people will be able to return home.
But the calmer winds had one official speculating that the wildfire might be fully contained in three to five days. Jim Thomas, leader of the federal incident response team, said the low winds will make it easier for crews to attack the flames.
High winds quickly spread the flames and gusts of more than 40 mph led Boulder authorities Thursday to warn people in parts of the city to be ready to leave if the fire moved toward the city.
By Friday night, the winds had slowed to 5 to 10 mph, with gusts along the ridge tops of up to 20 mph.
The winds should remain light through Saturday, said David Barjenbruch, a meteorologist with the National Weather Service in Boulder. However, temperatures are expected to be warm and humidity low the next few days, which could be of concern to firefighters, he said.
Firefighters from 20 states were battling the blaze dubbed the Fourmile Canyon fire, which has cost $4 million to fight so far. Three firefighters suffered minor injuries such as a broken finger, but no one has been seriously hurt.
Authorities believe the blaze was human-caused. They are looking at whether a vehicle may have crashed into a propane tank.
The loss of homes surpassed that of the 2002 Hayman fire in southern Colorado, which destroyed 133 homes and 466 outbuildings over 138,000 acres.
Insurers had no immediate estimate on damages as rubble smoldered in mountain neighborhoods filled with a mix of million-dollar homes and more modest log homes and ranches. Carole Walker, executive director of the Rocky Mountain Insurance Information Association, said the blaze affected mostly primary residences, not vacation cabins, so the homes burned are more likely to be insured.
The wildfire has destroyed at least $76.9 million worth of property, according to a Boulder Daily Camera database of the buildings confirmed burned and their valuations listed in Boulder County property records.