He dismissed as "just baloney" and "a little disingenuous" claims by agency officials that they didn't know about the severity of the damage until the next day.
Testifying before a Senate committee, Brown said he agreed with members who characterized him as a scapegoat. "I feel somewhat abandoned," said Brown, who quit under fire as chief of the Federal Emergency Management Agency just days after the storm hit.
Brown suggested the administration's fixation with fighting terrorism may have been to blame, in part, for the slow government response.
Because of a focus on terrorism, natural disasters "had become the stepchild of the Department of Homeland Security," Brown said.
Had there been a report that "a terrorist had blown up the 17th Street Canal levee, then everybody would have jumped all over that," Brown said.
The administration's lackluster response to Katrina undermined Americans' confidence in President George W. Bush's leadership abilities and contributed to a decline in his opinion poll ratings.
The storm slammed into New Orleans and the Gulf coast on the morning of Monday, Aug. 29.
Brown said he spoke by phone to a top White House official, he said he believed it was Joe Hagin, "on at least two occasions on that day to inform him of what was going on."
Hagin was with the president, who was vacationing on his ranch in Crawford, Texas, at the time while Brown was in Baton Rouge, Louisiana.
"I think I told him that we were realizing our worst nightmare, that everything we had planned about, worried about, that FEMA, frankly, had worried about for 10 years was coming true," Brown said.
He said he made similar comments in an e-mail message to White House Chief of Staff Andrew Card.
In an appearance before the Senate Homeland Security and Government Affairs Committee that mixed cooperation and confrontation, Brown went far further than he had previously in blaming other elements of the Bush administration for the government's halting reaction to the massive storm. Brown had been widely expected to start naming names of "who knew what the day Katrina hit," CBS News correspondent Susan Roberts reports.
The Aug. 29 maelstrom killed more than 1,300 people, displaced hundreds of thousands of others, and caused tens of billions in damage, including widespread destruction in New Orleans and other Gulf Coast communities.
"There was a cultural clash that didn't recognize the absolute inherent science of preparing for a disaster," he testified. "Any time you break that cycle ... you're doomed to failure."
He added: "The policies and decisions implemented by the DHS put FEMA on a path to failure."
Earlier, the chairwoman of the panel, Sen. Susan Collins, said that FEMA missed early warning signs that emergency response teams were unprepared to handle a catastrophic disaster like Hurricane Katrina.
Brown, who is widely viewed as the public face of the government's missteps during and after the storm, staunchly defended his role and appeared eager to answer any questions, particularly those that shifted the blame elsewhere.
He insisted he provided information to White House and Homeland Security officials the day of the storm. Homeland Security Secretary Michael Chertoff has said he did not know the levees were breached until the following day.
Under pointed questioning by Sen. Joe Lieberman, Brown said several times he could not clearly recall what was said in some of those conversations.
"Do you specifically remember asking Hagin for the White House to take action?" said Lieberman, the Senate panel's top Democrat.