Hello and welcome to the forums:
I'm not a chemist, but I do compost a lot of horse manure
. I have a couple of suggestions.
First, composting can be anaerobic or aerobic (meaning whether it uses oxygen.) It's possible that having the balloon over the bottle caused some of your composting to go anaerobic. In any case, it's possible that your composting method removed more oxygen from the air (adding it to the nitrogen in the compost) than the composting produced CO2.
You might want to try composting with some air in the balloon to start with. This might keep the process from going anaerobic. If the process is actually removing more O2 than the CO2 produced, you can measure the "loss" in volume of the balloon as well as the gain with some of your other bottles. Be sure to have multiple trials and repeat your experiment.
A good idea would be to monitor the temperature of your compost, if possible. It's one of the best ways to know if your compost is going anaerobic. Also, if it's anaerobic, it will smell bad! You probably want all of your composting bottles to be either aerobic or anaerobic so that that's not another variable.