While the video transmitter (VTX) has a button to change channels (and "unlock" the VTX, per the manual), the easiest way to change video channels on your drone is via the On-Screen Display (OSD) if the drone is equipped with "Smart Audio". While it is a strange name for a feature that controls video channels, it is much easier than the old method of using the VTX button.
To enter the Betaflight OSD menu, the drone needs to be disarmed, and then move your left stick to center throttle and yaw left, while pitching forward on the right stick. Please see the video referenced below on how to navigate to the Smart Audio feature.
Also, note that Smart Audio changes the video channel through the OSD and flight controller. This is main control, while the VTX button is the secondary control. That is why your drone seems to forget what channel change you made with the VTX button after unplugging that flight's LiPo pack. It reverts back to whatever Smart Audio/OSD/flight controller had the video channel set to previously.
How to Use Smart Audio:
Additional Article on Smart Audio: