Testing documentation
Load tests is the best way to really know about your web service facility and the real performance it has for servicing a lot of users. Load tests simulate an active population that will use the services and load the server with a fake load.
Tests are made in dedicated test instances of the service so no data generated by tests will polute your real production data.
While the tests are perfomed, the production system will be loaded and may not respond with full power, specially if the size of the test is high. System might not respond any more if a test is sized to find the "system breakage" limit. so it will be prudent to inform your normal users that the system will be tested.
How a test works
A test is an automaton that simulates a set of real users coming to use the service. Generally it seems difficult to modelize exactly the audience's behaviour. Scenarios will of course focus some type of visits, (generally purpose, or specific situation). Also real world load is the sum of users having very distinct needs, load tests using some definite typical scenario are useful to learn a lot about the hosting capacity.
Some aspects of the test need to be explicited. Usually, a test scenario is obtained capturing a real session with a capture tool (recorder). This will store all the queries and the site responses. The next step is parametrizing the query sequence to be repeatable for a lot of preloaded distinct users. Responses will be used to build some assertion tests that can determine if the response is what we expect for each query, or is a failure response generated by the load, or other error conditions.
Errors
Response errors during a load test can be of two types :
- Timeout errors : Usually this when the load starts influence the page response time so much that it passes over the acceptable timeout. Acceptable time is an important definition and it is important to set some values to them (also it generates errors in the results stats).
- Assertion errors : The response was given but with a content that is missing expected success conditions. Assertions errors usually denote a failure in the test design, or in the adequation of the test design to the target platform state.
Wait times
As a test is a sequence of preregistered queries, the simulation need to rate the query emission in a realistic throughput. We have analysed a big brunch of real platform logs to forge our default setup of the common tests we will provide here. Generaly we examined statistically user traces and could find appropriate settings in a
min time + rand() * offset time
modelisation.
Type of tests : Impulse and stale load
Impulse
Impulse test will generale big input load during a very short time, and let examine which time will be needed to resume a normal operation condition. Impulse tests are generally impacted by local "unit" perfomance of the local hosting system.
Stale load
the Stale load tests will generate a constant and durable load to test how much the system is stable during the load session and has no memory or resource leak or accumulation effect. A weak system may have performance degradation while there is no change in the input load.
Moodle tests
Our moodle tests are dedicated to moodle load testing. They are based on the standard tests provided by the Jmeter moodle test generator, but have been rebuilt to be more accurate in the simulation of real audiences.
Test types
Preheat
Preheat test is to be run at first into an empty testing site or the test site being used for the first time. It's purpose is to load all the persistant caches that would pump resources exceptionnaly and should not be part of the performance measure of an usual operatioon.