The suspect was taken into police custody after he was seen dropping the bag and fleeing on foot, Vienna police spokeswoman Michaela Raz said.
No one was injured, but police sealed off the neighborhood as a precaution and shut down or rerouted nearby bus and tramway lines. Officers patrolled the area with bomb-sniffing dogs.
The suspect was described only as a 42-year-old native of Bosnia-Herzegovina who now lives in the province of Lower Austria, which encircles most of the capital. Police said they made the arrest a short distance from the embassy building in a neighborhood where security is tight.
Raz said explosives experts were still examining the contents of the backpack.
"There were a lot of nails in that bag. Had it exploded, it would have had an enormous shrapnel effect," said Doris Edelbacher, of Austria's federal counterterrorism office.
The explosives resembled hand grenades, she told reporters, but that munitions experts were still working to determine whether the device had been properly rigged to explode.
The suspect spoke broken German and appeared to ramble during an initial interrogation, Edelbacher said.
It was unclear how far he made it into the embassy. Public broadcaster ORF, citing unidentified officials, reported that a metal detector sounded an alarm as the man was passing through security checks and that he fled immediately.
The nature of the Islamic material he allegedly was carrying also was unclear.
It was the second recent terrorism scare in Austria.
Last month, authorities arrested three people - all Austrian citizens of Arab origin in their 20s - in connection with a video posted online in March that had threatened Austria and Germany with attacks if they did not withdraw their military personnel from Afghanistan.
One of the suspects was released several days later for lack of evidence.
On Friday, the Interior Ministry said it found a list of politicians circulating online as potential targets for attacks. The individuals were notified, but authorities said there were no indications that any of them were ever in danger.
In January, the U.S. Embassy in Athens came under fire from a rocket that exploded inside the modern glass-front building but caused no injuries. That attack was blamed on Greek leftists.