Do airlines have too much power in booting passengers?

An Ivy League professor's mistaken removal from a flight is raising questions about passenger rights.

American Airlines questioned economist Guido Menzio after another passenger raised suspicions over his intense writing, which turned out to be complicated math. Menzio was allowed to fly, but the mix-up caused a two-hour delay.

Every time you buy a ticket, you give up some of your rights by agreeing to terms spelled out in the airline's "contract of carriage" -- a 40-plus-page document that covers all aspects of flying, including your behavior, reports CBS News correspondent Kris Van Cleave.

In November 2015, six people were booted from a Spirit Airlines flight after arguing with a flight attendant about double-booked seats.

Twenty-six-year-old college student Khairuldeen Makhzoomi was escorted from a Southwest Airlines flight in April. He said another passenger complained about a phone conversation he had with his uncle in Arabic.

"She turned around, actually, and she started staring at me and I knew that there's something wrong," Makhzoomi said.

Southwest said: "Our crew responded by following protocol... It was the content of the passenger's conversation, not the language used, that prompted the report leading to our investigation."

Airlines give themselves a lot of wiggle room in the ticket's fine print. Known as the "contract of carriage," Southwest can boot a passenger who is "disorderly, abusive, offensive, threatening, intimating [or] violent."

Most even reserve the right to remove a passenger for a "malodorous condition" -- meaning if you smell bad, you could be shown the door.

"They can kick you off for just about anything," said CBS News travel editor Peter Greenberg. "You can get kicked off the plane for how fat you are, if you stare, if someone overhears you in a language that they might not recognize -- the operative word right now here is fear."

Airline employees are given training on cultural sensitivity and how to respond to emotional passengers, but often, the deciders are the flight attendants and ultimately the captain.

Fight attendant Heather Poole said it all comes down to how passengers behave, and flight crews often err the side of caution.

"I've seen a lot of people who should have been thrown off, not thrown off," said Poole, who wrote "Cruising Attitude: Tales of Crash Pads, Crew Drama & Crazy Passengers" about her flight experience. "You have to stay calm, you have to be kind, you have to be nice. If you can't control yourself on the ground, I don't know what you're capable of doing in the air."

Passengers who are taken off a flight are due a refund. Truly unruly passengers can face fines or prosecution. Last year, 99 such cases were reported to the FAA; so far, there have been just nine this year.