Roughly 130 to 180 passengers is the practical sweet spot, and smaller isn't automatically better. Anything under about 200 passengers can run two landing sites a day with everyone getting ashore. Between 200 and 500 you'll still land, but usually at one site a day with groups rotating. Over 500 passengers and you don't land at all, you cruise past.
The sweet spot sits there because of everything around the shore time rather than the shore time itself. A ship in that range is steadier in a crossing than a very small one. It carries more guides, which means more Zodiac launch points and a wider spread of specialities aboard, and it can offer more activity variety to fill the three to four hour window ashore.
You'll notice numbers around 176 recurring across recent builds. That's not a coincidence, it's operators optimising around the guide to passenger ratio that the landing rules create.
The difference between a 130 passenger ship and a 199 passenger ship is much smaller than people assume. Maybe 5 to 15 minutes more in the mudroom. Both split their landings into groups, and modern ships are efficient at it.