Scalia's remarks came as he took questions from students during a lecture at the University of Connecticut's law school.
The case in question involved Cheney's request to keep private the details of closed-door White House strategy sessions that produced the administration's energy policy.
The administration fought a lawsuit brought by watchdog and environmental groups that contended that industry executives, including former Enron chairman Ken Lay, helped shape that policy. The Supreme Court upheld the administration position on a 7-2 vote.
Scalia refused to step aside from deliberations in the case, rejecting arguments by critics who said his impartiality was brought into question because of a hunting vacation that he took with Cheney while the court was considering the vice president's appeal.
"For Pete's sake, if you can't trust your Supreme Court justice more than that, get a life," he said Wednesday.
He told students he would have recused himself if the case had involved Cheney personally, but that he viewed the situation differently because the vice president was acting in his official capacity.
"I think the proudest thing I have done on the bench is not allow myself to be chased off that case," Scalia said.
The justice's appearance at the law school brought some protests.
Some gay rights activists set up a same-sex kissing booth outside the lecture hall. They said they believe some of Scalia's opinions amount to attacks on gays, women and other minorities.
"His visit opened a lot of conversation on this campus," said third-year law student Colby Smith, who was wearing an "I Kiss Boys" T-shirt. "We want to make sure people understand what the concerns are with him, and why his views are particularly offensive."
Inside the lecture hall, students whose names were drawn randomly from a lottery to attend said that having a chance to hear a sitting justice in person was a thrill. "Having a justice visit your campus is exciting no matter what you think about his views," said third-year student Hugo Tomasia of Norwalk, Conn.
Scalia, 70, was appointed in 1982 by President Ronald Reagan to the U.S. Court of Appeals for the District of Columbia Circuit. Reagan nominated him four years later to the U.S. Supreme Court, filling the opening that occurred when William H. Rehnquist was elevated to the position of chief justice.