The NYPD tells CBS New York he is 42 years old and has charges pending.
Police also say the incident was the second of that type that day in Manhattan's Upper East Side.
It wasn't clear whether the suspect would also be charged in connection with the second incident.
Area residents were repulsed by the vile acts.
"It's very scary," said Carole Kaplan. "I have to have my wits about me all the time."
Kaplan is avoiding using her cellphone as she walks after learning of the attack caught on video Monday evening.
Police say the 27-year-old woman was talking on her cell phone while walking down the street when the man came up from behind, grabbed her waist, shoved the bag in her pants and then grabbed her buttocks.
The man pulled off his rubber gloves and threw them into the street as he stomped away.
The victim sought help at a nearby parking garage. The attendant told the New York Daily News the woman was so shaken she couldn't even dial 911.
"I think it's the most traumatic thing, and I don't even know how anybody gets over something like that," Kaplan said. "It's just horrific."
"That's really gross," added Tara Pairan of the Upper East Side.
It's unknown if the suspect said anything during the incident.
Sources told CBS New York they believe the suspect targeted the woman at random.
Police said the same suspect is also believed to be involved in a second incident earlier in the day Monday.
In that one, a 33-year-old woman was walking along a Manhattan street when the suspect threw excrement at her.
The woman was hit in the face and torso, police said.
NYPD Chief of Detectives Robert Boyce said at a news conference that the suspect is believed to be homeless.
"Really disgusting attacks," he said. "We think it's the same male."
Lenny Burns, who is homeless, didn't mince words when talking about the suspect.
"He's a whack job!" Burns told CBS New York.