SAN FRANCISCO (CBS SF) – The nation's top housing official is catching some heat for comments he allegedly made about transgender people, while visiting San Francisco earlier this week.

According to The Washington Post, Secretary of Housing and Urban Development Ben Carson expressed concern about "big, hairy men" coming into women's homeless shelters. The Post reports he also said he's unhappy that society no longer seems to know the difference between men and women.

Carson visited HUD's office in San Francisco on Tuesday, the same day President Donald Trump was in the Bay Area for a fundraiser in Silicon Valley.

People who were there told the paper that many staffers were shocked.

Alphonso David, president of the Human Rights Campaign, condemned the HUD secretary's reported comments.

"From his comments on trans service members to his support for a proposal that would literally permit emergency shelters to turn away trans people who are homeless, hardly surprising that @SecretaryCarson would blatantly dehumanize trans people in his official capacity," David said in a tweet Thursday.

A HUD spokesperson responded to the Post, saying "The secretary does not use derogatory language to refer to transgendered individuals. Any reporting to the contrary is false."