, (*1)
Provides a new type of test case base class which contains additional assertion
methods based on the statistical information gathered and analysed by Numerolog., (*2)
Numerolog uses a remote server (a free, public one is used by default) to store
values - this package integrates that with PHPUnit to create assertions which
for example cause a failure if a value increases; and learns over time when it
decreases so that the expected maximum value goes lower and lower., (*3)
Intended purposes
- Assertions based on statistical information rather than hardcoded values.
- An automatically "learning" way to measure performance of code during
iterative development where each assertion "raises the bar" for the next.
- Doing so in a distributed way that fits with CI platforms like Travis.
The assertions can be used for execution time tracking but should be used
very carefully for this: consistency is vital which means distributed tests
may not be a wise choice when asserting time-based statistics., (*4)
The assertions are naturally ideal for tracking values such as:, (*5)
- Memory usage of expensive functions.
- Call stack analysis; maximum call depth, number of functions called, etc.
- Cyclomatic complexity tracking; guarding against peak increases/decreases.
- Lines-of-code vs lines-of-comments ratio; guarding against decreases.
- File sizes.
- And much, much more.
A successful assertion performed on a system that has a Numerolog token will
result in a new value being recorded. In other words: authenticated users and
systems both run tests and record statistics., (*6)
Statistics generated by numerolog-phpunit can then be retrieved using standard
Numerolog commands and integrations (among other things for generating charts)., (*7)
Background knowledge
- Needless to say, but said nonetheless: this package communicates with a remote
host (that you can override). The data that is transmitted is only as anynomous
as you make it: a token is used which is semi-public information that you can
share on multiple hosts (but should be encrypting). And your package name will
be included, as will the name of the counter you are accessing. So, don't use
any secret information in counter names or package name and protect your tokens.
- Your project root folder must contain a
composer.json and this file must contain
a name entry, which must consist of a vendor/package-name format. The vendor
and package are used to identify your package against Numerolog.
- You'll need to know about Numerolog's tokens. Although public testing does not
require a token, only projects that have one will both test and update statistics.
Read about Numerlog tokens here.
- By default, your tests will be using the public Numerolog server, but you can
change this to one that you yourself set up. To learn more about that,
read about Numerolog's configuration.
- Numerolog may enforce rate-limiting based on your token when provided, and based on
your IP address when token is not provided. A very generous limit is given since
each request is extremely light and many requests are expected from each test run.
If your project generates more requests than are allowed, the only way to increase
or remove this limit is to use your own Numerlog endpoint.
If you start on the public server and later need to move, the author will happily
give you access to your data storage files for transfer - assuming they contain
significant amounts of data! If your storages are very light please just recreate
them on your own end point., (*8)
Usage
Require via composer using composer require namelesscoder/numerolog-phpunit, then
use \NamelessCoder\NumerologPhpunit\StatisticsTestCase as parent class for your
test cases (which otherwise follow all phpunit rules). Alternatively, you can use
the \NamelessCoder\NumerologPhpunit\StatisticsTestCaseTrait as trait in your class;
for those cases when that fits better with your unit tests' structure. Both methods
will provide the same functions for your test case., (*9)
There are five types of assertions to compare with various statistics:, (*10)
$this->assertLessThan******($counterName, $value, $count = 20);
$this->assertLessThanOrEqualTo******($counterName, $value, $count = 20);
$this->assertEqualTo******($counterName, $value, $count = 20);
$this->assertGreaterThan******($counterName, $value, $count = 20);
$this->assertGreaterThanOrEqualTo******($counterName, $value, $count = 20);
Where $counterName is a lowerCamelCase name of a single counter; where $value
is the new value you with to compare - and where $count is the number of values
to pull from history and use as data set in comparison., (*11)
And where the ****** can be one of the following four statistics parameters:, (*12)
AverageMinimumMaximumSum
Which means a total of 20 (5 x 4) simple statistical assertion methods., (*13)
Your test methods can also perform the following more advanced assertions:, (*14)
// Success only if $value has a standard deviation inside specified tolerance:
$this->assertWithinStandardDeviation($counterName, $value, $allowedStandardDeviation = 1, $count = 20);
// Success if $value is above current minimum and below current maximum:
$this->assertWithinSetRange($counterName, $value, $count = 20);
// Opposite of the above
$this->assertNotWithinSetRange($counterName, $value, $count = 20);
// Success only if $value exists as an exact match (also for floats!) in set:
$this->assertExactlyWithinSetRange($counterName, $value, $count = 20);
// Opposite of the above
$this->assertNotExactlyWithinSetRange($counterName, $value, $count = 20);
Example
When put together, a complete statistical unit test function can look like:, (*15)
public function testExpectedMemoryUsageOfMyFunctionOnMyClassIsSameOrLower() {
$subject = new MyClass('myconstructorvalue');
$monitor = new Monitor($subject);
$subject->doSomethingMemoryIntensive(1000);
$usage = $monitor->getMemoryUsage();
// method always uses memory; no usage or freed memory is an early failure:
$this->assertGreaterThan(0, $usage);
// assertion: no more than 2 standard deviations allowed. Include 40 values in set:
$this->assertWithinStandardDeviation('myFunctionMemoryUsage', $usage, 2, 40);
// assertion: value should be less than or equal to average recorded usage:
$this->assertLessThanOrEqualToAverage('myFunctionMemoryUsage', $usage, 40);
}
The Monitor class is not included and is hypothetical. Any measurement method
can be used. In this test, we have our $subject do something that's known to
cause a lot of memory usage - and then assert that, with the code base that we
currently are on, the usage does not deviate more than two standard deviations
from the recorded average. And we assert that the usage is either less than or
equal to the average recorded value., (*16)
Assuming the Numerolog token exists in the project doing the assertions, each
successful assertion adds to the statistical history. In this case, we are
continually testing that our memory usage does not increase; as well as
testing that it doesn't suddenly decrease drastically. Which means that as
you improve the code that gets tested, Numerolog ensures that your tests also
"learn" what to expect without you having to continually modify test cases to
change the expectations like you normally would with a unit test., (*17)
Pitfalls
If the code you are testing depends on a framework or has other dependencies, make
sure you sufficiently mock all of those dependencies in PHPUnit or the numbers may
be inadvertedly skewed by changes occurring in the framework or dependency. For
example, in the hypothetical case that you have a Symfony component as dependency
and that component suddenly decreases in performance (whatever the reason may be)
then your tests may fail if you did not sufficiently mock that dependency. Obviously
the PHPUnit code itself also counts unless you are careful, e.g. don't include
calls to assertion methods or mock generation in the code window you profile., (*18)
Essentially: use proper unit test design to avoid unexpected problems with
the variables you profile and track., (*19)
Wallogit.com
