They're different trips and the honest split is wildlife against landscape. South Georgia is the wildlife and history trip: king penguin colonies in the hundreds of thousands, beaches packed with seals, Shackleton's grave, and a green, mountainous, subantarctic feel. The Peninsula is the ice trip: glaciers, icebergs, the channels, and the continent itself. If you can only do one, decide which of those two you actually came for.
Most South Georgia itineraries include the Peninsula anyway, because the route runs that way, so in practice the question is usually whether to spend the extra week and money to add South Georgia rather than choosing one instead of the other.
Two practical constraints. South Georgia commits you to sailing, since flying rules it out almost entirely. And it's a longer, more expensive trip, typically by around a week.
If you want to set foot on the Antarctic continent specifically, check the itinerary carefully. Some South Georgia trips are heavy on the subantarctic and lighter on continental landings.