Republican Gov. Linda Lingle's action came on the final day she had to either sign or veto the bill, which the Hawaii Legislature had approved in late April.
"There has not been a bill I have contemplated more or an issue I have thought more deeply about during my eight years as governor than House Bill 444 and the institution of marriage," Lingle said at a news conference. "I have been open and consistent in my opposition to same-sex marriage, and find that House Bill 444 is essentially same sex marriage by another name."
Had Lingle not vetoed it, the measure would have granted gay and lesbian couples the same rights and benefits that the state provides to married couples. It also would have made Hawaii one of six states that essentially grant the rights of marriage to same-sex couples without authorizing marriage itself. Five other states and the District of Columbia permit same-sex marriage.
Lingle's decision is expected to be the last say on the proposal this year because state House leaders have said they won't override any of Lingle's vetoes.
For weeks, Lingle has heard emotional views from both supporters and opponents of the bill and on Tuesday invited leaders from both sides to her standing-room only news conference.
Earlier, in the Capitol's ground-floor rotunda, dozens of supporters had gathered for a daylong vigil. Some waved flags and held signs along a busy street, to the honks of passing vehicles.
"I want to be able to get married," said Elizabeth Kline, a 22-year-old University of Hawaii student who quickly corrected herself to say she wants a civil union. "It's not marriage, but it's a step toward it."
Meanwhile, a group of about 20 civil unions opponents raised their hands, closed their eyes and said blessings in front of the office doors of key lawmakers. They wore white shirts in a show of unity and buttons declaring "iVote," a promise of consequences come November if civil unions become law.
"All we're doing is praying. We're not waving signs or playing music," like gay rights groups in the rotunda, said Dennis Arakaki, executive director for the Hawaii Family Forum.
About 60 percent of the more than 34,000 letters, telephone calls, e-mails and other communications from the public to the governor asked her to veto the measure, the governor's aides said late last week.
The Aloha State has been a battleground in the gay rights movement since the early 1990s.
A 1993 Hawaii Supreme Court ruling nearly made Hawaii the first state to legalize same-sex marriage before voters in the state overwhelmingly approved the nation's first "defense of marriage" constitutional amendment in 1998.
The measure gave the Legislature the power to reserve marriage to opposite-sex couples. Lawmakers responded by enacting a law banning gay marriage in Hawaii but left the door open for civil unions.
Last year, civil unions easily passed the House but stalled in the state Senate. When legislators reconvened in January, the Senate passed it but House leaders shelved it until the final day of the legislative session.