Is it a red flag if a seller refuses to use a broker at all?

Not on its own. Plenty of legitimate sellers manage a sale directly — to save on commission, because they already have a specific buyer in mind, or simply because they prefer more direct control over the process. A seller's choice not to use a broker tells you about their preferences, not about the quality or honesty of the business itself.

What matters more as a buyer is whether the seller is still willing to support a properly run process even without a broker involved: providing financial statements and records for genuine due diligence, being straightforward about the business's history and any problems, and working through a normal purchase agreement with representations, warranties, and closing conditions rather than resisting standard protections because "there's no broker requiring it." A seller who refuses a broker and also resists ordinary due diligence and documentation is a very different situation than one who's simply handling the sale themselves competently. As a buyer, insist on the same level of diligence and legal documentation regardless of who's on the other side of the table, and have a Treadstone business lawyer involved either way.