Need help on a small issue in my experiment

[mmpabc2000]

ok so im doing an experiment where im testing the speed of a VB.NET browser that i built against 5 C++ browsers to see which of these 6 browsers is the fastest at loading a website. Oh and im testing them on 5 categories of websites. Now im wondering should i test them each a couple of times and take the average? or is testing each once enough since i would have to keep re-emptying the cache? thanks in advanced!

[mpphlipot]

Legible handwriting

You really want to test them multiple times if it's at all possible. You'll see one of two results:

a) The multiple runs will have exactly the same time. This is great because it gives you confidence that your experiment is repeatable and that the data is reliable. It makes it more difficult for reviewers to question the validity of your results.

b) The multiple runs will have different times on most runs. This is also great because it points out some additional variables in your experiment that you need to be controlling or perhaps they simply aren’t under your control. Either way, it’s important to understand and be able to explain possible reasons for the variation.

Best of luck with your project.
Mike

[hhemken]

mmpabc200,

You definitely want to measure it several times. There will be issues such as caching of data by the browser, by the web server, and any proxy machines that may exist between the server and the browser. You may wish to test each page load at least 10 times and throw out the first two or three in each run so that you get stable results. Try googling these keywords to get a feel for what's going on:

Code: Select all

web page caching

You also should do a Student's t-Test on the results, testing each browser against each of the others. See:

http://udel.edu/~mcdonald/statttest.html
http://studentsttest.com/
http://www.usablestats.com/calcs/2samplet