Why shouldn’t you be forced to marry and have children against your will? Why shouldn’t you be forced to join and support a religious organization? Why shouldn’t you be forced into friendships to spend your leisure time with people you don’t like? Why shouldn’t you be forced to do unimportant and unproductive work for an abusive boss? Why shouldn’t you be forced to comply with the demands of politicians who steal your money and use it in unethical and counterproductive ways?
The answer is simple: nobody has a higher claim on your life than you do.
Argument against freedom of association constitutes a rejection of ethics. Politics is what you are left with after you reject ethics. It is the systematic violation of consent. It is an endless fight over oppressive control and stolen resources in which association must be either forced or prohibited.
When civilized people disagree, they don’t claim the moral high ground while violating consent to enforce their unprincipled demands on others. They respect the right of individuals to self-select into associations that seem most likely to result in their safety and happiness. If the internalized cost/benefit ratio is not satisfactory, they are free to disassociate.