1. Why do you believe what you believe? What would change your mind?
The longstanding precedent of millennia.
Societies instituting polygamy have always ended up with lots of men marrying lots of women, but few women marrying lots of men. This has applied even to supposedly-egalitarian societies like that of the ‘free love’ movement in the sixties, although of course it was worse in those societies where the one-sidedness was legally enforced (as in assorted ancient cultures, the Mormons, et cetera). And, where that imbalance occurs, problems have always followed.
First, women in polygamous societies have historically lost both romantic and personal autonomy. Polygamy, moreso than monogamy, has lent itself very well historically to families selling off their daughters to the well-paying, since the well-paying could afford so many more families’ daughters. But even in the context of ‘free love’, which escaped that particular problem, women who left their boyfriends were shunned and lost valuable social capital, to a far greater extent than happens in monogamous dating circles. The pattern is a robust one, and I see no reason to expect the polyamorists to do any better at avoiding it than the various past societies which have failed to do so did.
Second, even putting the aforementioned pattern aside—and it’s not a pattern to put aside lightly—where polygamy lives, jealousy, rivalry, and other forms of unpleasantness follow. People, in general, don’t want to settle for only a small sliver of a romantic partner’s attention, and if circumstances force them to do so anyway, they can get pretty vicious in their efforts to enlarge that sliver. This has, once again, been a consistent social pattern in polygamous societies throughout history, and I once again see no reason to think that the polyamorists’ attempt at it will fare any better than all those past attempts.
What could change my mind? Well, if the polyamorists, as a broad group, manage to last through the decades without losing most of their members to those failure modes and the consequent realization that monogamy is in fact preferable, that would be a decent reason to think I might be wrong. Short of that? I’m not sure there is anything. I don’t doubt that polygamy or its ideological relatives can create short-term feelings of satisfaction, when it’s new and exciting, before the downsides show themselves more clearly. I don’t doubt that a few psychologically-atypical people (free-love-movement remnants, some number of polyamorous rationalists, and presumably members of other communities I’m less aware of) will be able to make it work for a lifetime, even. My doubt is purely over whether polyamory will ever be able to work in the long run for the typical member of a polyamorous society, once one writes off the recent adopters and the psychological outliers as such.
2. A monogamous person is jealous of their partner (for example, because they’re afraid their partner has a romantic interest in someone else). In a healthy relationship, what would happen next?
In a healthy relationship? The monogamous person in question (let’s call her Jane) has a serious conversation about it with her partner, which she can do, because this is a healthy relationship where communication is an option and soap-opera-style dysfunction is correctly recognized as something to avoid. She lays out the reasons she’s jealous. Her partner listens and clarifies her own viewpoint on the jealousy-generating matters, and ideally, that’s the end of it.
If verbal reassurances aren’t enough, possibly they reorganize their lives in such a way as to ameliorate Jane’s anxieties. Involving Jane in more of her partner’s interactions with the person-who-Jane-is-worried-her-partner-is-into, or Jane and her partner organizing some sort of regular date night for themselves, or having Jane’s partner spend less time with the third party in question, or otherwise rearranging their lives to make it clear to Jane that her anxieties are unfounded.
If even that degree of reorganization isn’t enough, they possibly go see a therapist about it, although ‘partner’ is a vague term and I don’t know, offhand, to what extent marriage-counseling-like options are available to the unmarried.
If even therapy isn’t enough, they either find a further path to escalate their problem-solving, or they break up; because a relationship where one party is incurably jealous of the other is unlikely to stay healthy, even if it started out that way. Sometimes the healthiest response to a sufficiently-bad relationship problem really is to just let the relationship go and to try again with different people.
(If they have children, or other complicating factors, binding them together, then I suppose they stay together and do the best they can given the position they’re in. At that point they’re firmly beyond my area of expertise; my thoughts here are mostly just resignation at how hard relationship problems can sometimes get.)
3. What would happen if 90% of people in a society were polyamorous? (You may assume they all practice one style of polyamory, or different styles.)
Well, first off, you get the usual problem of women losing personal and romantic autonomy, and you get the usual problems with jealousy and rivalry, as discussed in my answer to 1. Gay people likely end up somewhat better off, since they get to avoid at least the gender-dynamics problem; but they remain susceptible to the problem of getting unsatisfactorily small slices of their partners’ attention, and the jealousy-and-rivalry problem which results from that. I have no idea where nonbinary people will fit into these dynamics, but I don’t see any avenues by which they’d come out well, either.
As a second-order effect of men dating/marrying more women apiece than women do men, romantic prospects become worse for low-status heterosexual men. Where, in our society, they can trust that there exist low-status heterosexual women who will remain viable prospects for them, in this hypothetical society, all the low-status heterosexual women will have already been swept up by mid-status men. This leaves a sizable contingent of the male population romantically worse-off than they would otherwise be, to their likely unhappiness. (Meanwhile various women get to date/marry higher-status men than they otherwise would, and various high-status men get to date/marry larger numbers of women than they otherwise would; this is technically to both groups’ benefit, but not nearly enough to offset the downsides for the women, so really it’s only the high-status men who particularly benefit here)
On the margins, you get some of the people who idealistic polyamorists tend to wield in counterargument against this picture of what a widely-polyamorous society would look like. The women with multiple boyfriends, the jealousy-immune psychological outliers who are perfectly happy to get only a sliver of their partners’ time and attention, et cetera. They exist; they presumably are romantically satisfied; they remain a small enough segment of the population that they don’t serve as a real counterbalance to all the problems elsewhere.
Eventually, as usual for polygamous societies, people realize that this is a bad idea and flock back to monogamy, maybe with legal backing or maybe just via large numbers of people individually choosing it.