Avoid it wherever you can. Wind, snow, rain and sea spray all reach the sensor and the internal surfaces the moment the lens comes off, and doing it on a moving Zodiac adds a real chance of dropping something. You'll also miss the wildlife while you're fumbling.
Choose your lens before you leave the ship, based on what the expedition team has briefed for that landing. A broad-range zoom, two bodies, or using your phone for the wide shots all reduce how often the question comes up.
If a change is genuinely necessary, find shelter, turn your back to the wind and spray, point the camera opening downward, and work fast.