Bayh told CBS News' Bob Schieffer on Washington Unplugged that he would not vote to proceed with the debate "if there are things in the bill I think are just beyond the pale."
"Some people argue that we should vote to go forward on a bill even if we don't like it," he said. "As we get further along in this, I view procedure and substance as being largely one and the same. I'd like to move forward, but some of that's going to depend on is it fiscally responsible."
Bayh said he didn't "think that's even worth starting a discussion on" provisions "that would explode the deficit... or would dramatically increase the premiums that ordinary families are paying."
When a bill is introduced on the Senate floor, any member may object to the "motion to proceed" with debate. In that case, it takes 60 votes of approval to proceed with the debate and consider amendments to the bill.
If there are provisions in the bill that are unacceptable, "I think your maximum leverage for getting them corrected is now," Bayh said. "It really is one of these things where I've just to look and see what's in there."
Given his current knowledge of the bill, Bayh said, "There are a couple elements in the bill that might actually increase premiums that typical families are paying… that's just not a direction in which we should go."
Bayh said he has asked the Congressional Budget Office for an analysis of those elements.
Yesterday, Sen. Joe Lieberman (I-Conn.), a member of the Democratic caucus, said he cannot support a bill that includes a government-run health insurance plan, or "public option." Bayh said today, however, that he is less focused on that issue.
Reid is going forward with a plan to include a public option from which states could opt out. Bayh said he would like to see more independence given to the states, but "I do think there's flexibility there now to allow states to consider their own specific circumstances."
In spite of the continuing controversy over the public option, Bayh said it was "highly likely legislation will be passed," and that if proponents of the public option are obstinate enough, they could pass it through the procedural loophole called "reconciliation."
"There are some pretty strong reasons not to go there... but if the people (who) want the public option in its fullest form are just adamant about that, they can always get that with 50 votes," he said.
Bayh told Schieffer that President Obama was wise to take a comprehensive approach to health care reform, rather than addressing issues on a piecemeal basis.
He added, "This is something we need to tackle, but we need to do it in a responsible way."
You can watch the interview above, along with a discussion with CSIS Senior Fellow Rick "Ozzie" Nelson about the deadly car bomb that struck a busy market in Peshawar, Pakistan, as well as coverage of some Capitol Hill hoops.
"Washington Unplugged" appears live on CBSNews.com each weekday at 12:30 p.m. ET. Click here to check out previous episodes.