In a speech Monday to an American Restaurant Association conference, Ashcroft said people are being wrongly led to believe that libraries have been "surrounded by the FBI," with agents "dressed in raincoats, dark suits and sunglasses. They stop everyone and interrogate everyone like Joe Friday.
"Now, you may have thought with all this hysteria and hyperbole, something had to be wrong," Ashcroft said. "Do we at the Justice Department really care what you are reading? No."
A portion of the Patriot Act, passed shortly after the 2001 terror attacks, gives federal authorities access to library, bookstore and other business records as part of terrorism investigations. Some libraries have begun purging their records more frequently and posting signs warning that the records could be checked by the FBI.
The provision has drawn a legal challenge in a federal lawsuit filed in July by the American Civil Liberties Union and Islamic groups.
Emily Sheketoff, executive director of the American Library Association's Washington office, said library records should be treated differently from those of other businesses because of privacy rights and constitutional free speech protections.
The Justice Department should at least be required to publicly describe how often FBI agents subpoena library records, she said. That information is classified but is reported secretly to Congress twice a year.
"They are not taking this issue seriously, and the American people are upset about this," Sheketoff said. "At least they could give us some idea of the breadth of the problem."
Ashcroft said, however, that subpoenas of library records are closely scrutinized by federal judges, and the FBI, with 11,000 agents, could never begin to monitor the reading habits of millions of library patrons even if it wanted to.
The main reason an FBI agent would want library records is to track use of its publicly available computers, which terrorists have been known to use to communicate, Justice Department officials say. They say use of the power is extremely infrequent.
"The hysteria is ridiculous. Our job is not," Ashcroft said.
Ashcroft's speech followed a recent 16-city tour in which he repeatedly defended the Patriot Act as essential to the war on terror. President Bush asked last week for three new legal tools, including expansion of the "administrative subpoena" that effectively bypasses judicial or grand jury oversight.
By Curt Anderson