Amy - We need to know more about the 'unorthodox' results. If the student is getting consistent refraction measurements with each of the 10%, 20% and 30% solutions that is a good start. In a clear liquid containing having a sugar content, he/she should be able to compare the refraction measurement of the unknown and compare it to the test solutions. That will indicate whether the solution is between 10 to 30% sugar content. Perhaps the measurement is less than 10% or over 30%. Then it will be outside the range of the test solutions. In this case, interpolation will have to be used. In that we have 3 test points, each producing a different refraction of the laser beam, we can project from the data we have, what the concentration is in the unknown.
In the section of the description titled "Measuring the Index of Refraction of a Liquid", it shows the value of 'X' on the vertical scale as the refracted dot position. For each test measurement, if the prism is in the exact location relative to the place the refracted dot is measured, you should be able to plot 3 points on this 'X' axis representingr each of the 10%, 20% and 30% solutions. Now, plot where the unknown refracted dot is. If it's between the 3 dots, interpolate directly. If not, extrapolate the values you have based on the 3 dots and create your own refraction-to-concentration chart.
Until you can get to these observations, no amount of math is can be applied.