Unarmed security guards from the Hawaiian Kingdom Government group blocked all gates to the grounds of the palace, which is adjacent to the state Capitol. They did not enter the building itself.
After several hours, the protesters agreed to reopen the gates but said they would remain on the grounds until early evening and return Thursday. No arrests had been made as of mid-afternoon.
Laura Thielen, state land director who oversees the palace area, said some of the protesters could still be charged.
"This is public property and they can't block public access," she said.
Protest leaders had said they were prepared to be arrested and would go peacefully.
Mahealani Kahau, elected "head of state" of the group years ago, said the organization does not recognize Hawaii as a U.S. state but would keep the occupation peaceful.
"The Hawaiian Kingdom Government is here and it doesn't plan to leave. This is a continuity of the Hawaiian Kingdom of 1892 to today," Kahau said.
The group is one of several Hawaiian sovereignty organizations in the islands, which became the 50th U.S. state in 1959.
The ornate Iolani Palace is operated as a museum. Hawaiian King Kalakaua built it in 1882, and it also served as the residence for his sister and successor, Queen Liliuokalani, the islands' last ruling monarch.
It was neglected after the overthrow of the Hawaiian Kingdom in 1893 and restored in the 1970s as a National Historic Landmark. It includes a gift shop and is open for school groups and paid tours.
"The Hawaiian Kingdom Government is here and it doesn't plan to leave. This is a continuity of the Hawaiian Kingdom of 1892 to today," said Kahau, who was elected head of state of the group seven years ago.
The protesters are not damaging anything in the palace grounds, Kahau said. Workers inside the palace itself had locked the doors and were not letting them inside.
"We will not resist, we won't fight, we won't be aggressive. But we'll be back for sure," Kahau said.
No matter what happened Wednesday, the protesters planned to return to the palace Thursday, she said.
State Sen. Kalani English - a Native Hawaiian and a Democrat from East Maui-Lanai-Molokai - came over from the Capitol to speak with some of the protesters, and had his staff take them food.
"This is the manifestation of the frustration of the Hawaiian people for the loss of sovereignty and land," English said.
"It is symbolic. This made a statement. It got the word out about the plight of the Hawaiian people," he said.
Richard Kinney, who described himself as an independent Hawaiian nationalist, said he went to the Capitol to show his support. He carried an upside-down Hawaii state flag, signaling distress.
"The sovereignty of these islands is inherent to the Hawaiian people, and we've never relinquished that," he said.
"Occupying any land, including Iolani Palace, is the beginning," Kinney said.
Kippen de Alba Chu, executive director of Iolani Palace, issued a statement that said the protesters delivered a written message to palace officials claiming the grounds as the seat of their government.
"While we respect the freedom of Hawaiian groups to hold an opinion on the overthrow of the Hawaiian Kingdom, we believe that blocking public access to Iolani Palace is wrong and certainly detrimental to our mission to share the palace and its history with our residents, our keiki (children) and our visitors," Chu said.