Ed Neu and Craig Bridge have given an excellent explanation of the chemistry for your project. Here is some additional information that should be helpful:
It sounds like you are doing this excellent project on the Science Buddies website:http://www.sciencebuddies.org/science-f ... p101.shtml
Proteins such as the casein found in milk are composed of amino acids that may have both positive and negative charges. The isoelectric point of a protein is the pH at which all of the charged groups have an overall net neutral charge. In general, proteins are more soluble at a pH above or below their isoelectic point and least soluble at the isoelectic point.
Craig had provided the key information for explaining your results. The isolectric point of casein is 4.6. The pH of milk is about 6.8. The pH of vinegar is about 2.4 to 3.4, depending on the concentration. So the casein is very soluble in milk because the pH is above the isoelectric point; the vinegar is used to lower the pH to the isoelectic point to precipitate it. If additional vinegar is added to lower the pH below the isoelectic point, then the casein starts to dissolve again.
If you repeated your experiment and measured the pH of the samples, the 15 ml sample would probably be closest to pH 4.6. You can find more information on this by doing internet searches for isoelectic point, casein, amino acids, and vinegar/acetic acid.
I’m sure the Science Fair judges will be impressed if you can explain what happened in your experiment on your project board, so do let us know if you have any more questions.