Rescuers saw up to a dozen passengers climbing across the bottom of the 36-foot pontoon after winds gusting up to 50 mph flipped the boat over.
Petty Officer Edward Mendez of the Navy reserves said he watched violent winds toss the vessel "like a little toy boat getting blown out of control."
He and other reserve officers, stationed at a nearby center, saw the boat capsize about 1,000 yards from the shore, then rushed to the scene and pulled about a dozen people from the water, some of them unconscious.
Twenty-two people in all were removed from the water, including two women who died at nearby hospitals. The boat was carrying 25 people, including at least two crew members.
At 7 p.m., fire officials turned their mission from rescue to recovery, using helicopters to search for bodies of the missing.
"It's fortunate there was not a greater loss of life when you consider the force of the storm," said Baltimore Mayor Martin O'Malley.
The 44-degree water was choppy and winds up to 50 mph blew through the harbor when the boat capsized at around 4 p.m., said National Weather Service meteorologist Steve Rogowski.
"The wind just took it," said Master Chief Melvin Johnson, a commander with the reserves.
The water taxi was equipped with life preservers, but passengers are not required to wear them.
"No one on the craft had time to get their life preservers on,'' said Maj. Frederick Bealfeld of the Baltimore Police Department.
James Bond, president of the organization that operated the taxi, said the boat has a capacity of 25 passengers and two crew members.
"She was ready for an inspection on Monday and in shape the way she should be," said Bond, who heads the Living Classrooms Foundation.
The Inner Harbor is one of the nation's oldest seaports, and has experienced a renaissance in recent years. Millions of tourists visit the Inner Harbor each year, where they can walk along brick promenades on the shore and frequent the many shops, seafood restaurants, museums and other attractions in the harbor area.
Water taxis ferry thousands of visitors each year to the many points on the Inner Harbor, including Fort McHenry, Fells Point and the world-renowned National Aquarium.