Unit Testing

wiki

Unit testing is about testing your code during development, not in production. Typically you start by preparing the testing environment, writing some code that calls production code and check expected results with actual results.

Lua has several frameworks to do that:

[lunit]
- Fork with Lua 5.2 support: [lunitx]
[Lunity] (similar to lunit, but can only run a single file and doesn't distinguish between errors and test assertions)
[lunatest] (upwardly compatible from lunit, but no dependencies, and also supports randomized testing)
[LuaUnit] (supports Lua 5.1 and 5.2, no dependencies, see below)
[Shake] - source preprocessing using basic assert-like syntax (see below)
[build-test-deploy (BTD)] - Unit testing inspired by jUnit and originally based on LuaUnit
[luaspec][1] - Behavior Driven Development (BDD) test framework
[PenLight pl.test] - PenLightLibraries? has a very small unit testing library, mostly for internal testing. See the Penlight tests [here]
[telescope] - A highly customizable test library for Lua that allows for declarative tests with nested contexts. Uses BDD-style spec names.
[lua-TestMore] - port of the Perl5 module Test::More (Test Anything Protocol).
[busted] - Behavior Driven Development (BDD) unit testing library, with various outputs formats including TAP and focus on elegance.

They have mostly the same feature set, which is:

a special set of assert functions
facilities to group tests into test suites
facilities to detect where test fails

Commercial Lua testing tools are available from Software Verification[2].

Comparison: [3]

Tests of Lua itself (useful for testing LuaImplementations):

[Lua 5.2 test suite]
[Lua 5.1 test suite]
[Lua 5.0 test suite]
[lua-TestMore] contains Lua 5.1 and 5.2 test suites [announcement]

luaunit

I am going to provide an example for luaunit since I am more familiar with it. The example is pretty simple to understand.



-- Some super function to test

function my_super_function( arg1, arg2 ) return arg1 + arg2 end



-- Unit testing starts

local LuaUnit = require('luaunit')



TestMyStuff = {} --class

    function TestMyStuff:testWithNumbers()

        a = 1

        b = 2

        result = my_super_function( a, b )

        assertEquals( type(result), 'number' )

        assertEquals( result, 3 )

    end



    function TestMyStuff:testWithRealNumbers()

        a = 1.1

        b = 2.2

        result = my_super_function( a, b )

        assertEquals( type(result), 'number' )

        -- I would like the result to be always rounded to an integer

        -- but it won't work with my simple implementation

        -- thus, the test will fail

        assertEquals( result, 3 )

    end



-- class TestMyStuff



LuaUnit:run()

When you run it, you get:


shell $ lua use_luaunit1.lua

>>>>>> TestMyStuff

>>> TestMyStuff:testWithNumbers

Ok

>>> TestMyStuff:testWithRealNumbers

use_luaunit1.lua:25: expected: 3.3, actual: 3

Failed



Success : 50% - 1 / 2

The testing framework reports the file and line that caused the error, and some additional information.

When your program grows, you need more and more test cases. The next example is slightly more complicated.

-- Some super function to test

function my_super_function( arg1, arg2 ) return arg1 + arg2 end

function my_bad_function( arg1, arg2 ) return arg1 - arg2 end



-- Unit testing starts

require('luaunit')



-- now, we perform all the tests on int in one test class

-- and the tests on float in another one

-- when your test grows, you will have many test classes



TestWithInt = {} --class

    function TestWithInt:setUp() 

        -- this function is run before each test, so that multiple

        -- tests can share initialisations

        self.a = 1

        self.b = 2

    end



    function TestWithInt:tearDown() 

        -- this function is executed after each test

        -- here, we have nothing to do so we could have avoid

        -- declaring it

    end



    function TestWithInt:testSuperFunction()

        result = my_super_function( self.a, self.b )

        assertEquals( type(result), 'number' )

        assertEquals( result, 3 )

    end



    function TestWithInt:testBadFunction()

        result = my_bad_function( self.a, self.b )

        assertEquals( type(result), 'number' )

        assertEquals( result, -1 )

    end



    function TestWithInt:testThatFails()

        -- you can test anything with assertEquals

        assertEquals( self.a, 1 )

        assertEquals( type(self.a), 'number' )

        -- will fail

        assertEquals( 'hop', 'bof' )

    end

-- class TestWithInt



TestWithFloat = {} --class

    function TestWithFloat:setUp() 

        -- this function is run before each test, so that multiple

        -- tests can share initialisations

        self.a = 1.1

        self.b = 2.1

    end



    function TestWithFloat:tearDown() 

        -- this function is executed after each test

        -- here, we have nothing to do so we could have avoid

        -- declaring it

    end



    function TestWithFloat:testSuperFunction()

        result = my_super_function( self.a, self.b )

        assertEquals( type(result), 'number' )

        -- will fail

        assertEquals( result, 3 )

    end



    function TestWithFloat:testBadFunction()

        result = my_bad_function( self.a, self.b )

        assertEquals( type(result), 'number' )

        -- will work, but only by chance :-)

        assertEquals( result, -1 )

    end

-- class TestWithFloat



LuaUnit:run()

Run it:


shell $ lua use_luaunit2.lua

>>>>>> TestWithFloat

>>> TestWithFloat:testSuperFunction

use_luaunit2.lua:66: expected: 3.2, actual: 3

Failed

>>> TestWithFloat:testBadFunction

Ok



>>>>>> TestWithInt

>>> TestWithInt:testSuperFunction

Ok

>>> TestWithInt:testBadFunction

Ok

>>> TestWithInt:testThatFails

use_luaunit2.lua:44: expected: 'hop', actual: 'bof'

Failed



Success : 60% - 3 / 5

You can also run tests individually:


shell $ lua use_luaunit2.lua TestWithInt:testSuperFunction

>>>>>> TestWithInt

>>> TestWithInt:testSuperFunction

Ok



Success : 100% - 1 / 1





shell $ lua use_luaunit2.lua TestWithFloat

>>>>>> TestWithFloat

>>> TestWithFloat:testSuperFunction

use_luaunit2.lua:66: expected: 3.2, actual: 3

Failed

>>> TestWithFloat:testBadFunction

Ok



Success : 50% - 1 / 2

Shake

Shake is a simple and transparent test engine for Lua that assumes that tests only use standard assert and print calls. If you are looking for a xUnit style framework, check lunit and luaunit instead.

Many Lua modules and applications use a simple pattern for internal tests. They have a script (usually called test.lua) that exercises the module API using API calls and assert() calls to verify the results. It is also common practice in those modules to use print() to output information about the test progress and to use Lua comments to inform what is being tested to whoever reads the source. Although widespread this approach may be too crude for those wanting to test more than one module at a time or to have a more detailed view of the results.

Shake assumes that the tests have been implemented for the above scenario but offers a transparent test engine for those wanting the execution of the tests in batch mode or those wanting to get an overview of the various results. The Shake engine can inform not only the number of tests, failures and errors found in a group of test scripts, but it can infer information about the test context using the output and comments associated with the assert() calls.

The main characteristic of Shake is being transparent, which means that the module authors and test writers do not have to be aware that tests are going to be run with Shake. As long as the tests call assert() Shake can obtain quite a lot of information from the source and run time execution. This is done through the pre-processing of the tests source code using Leg (and therefore LPeg) to replace every call to assert() with one that can extract information about the expressions, values and operators involved in the assertion.

Assuming you have a module like LuaFileSystem installed and you go to its /tests directory and run Shake from there, the output would be:


>>>>~/workspace/luafilesystem/tests$ shake

->  test.lua OK!

_________________



Tests: 27

Failures: 0

Errors: 0

On the other hand, if you have a test script like the one below that includes two assertions that are supposed to fail (lines are numbered):

 1	items = 10

 2	-- checks the correct case

 3	assert (items == 10, "this should not fail")

 4

 5	items = 20

 6	-- checks an overflow case

 7	assert (items == 10, "wrong number of items")

 8

 9	print("Verifying the total")

10	items = 10

11	total = 30

12	assert (items == total, "wrong total")

Shake would register the failures but would run the whole test script, reporting at the end:


:~/workspace$ shake

----------------    test.lua failed!   ----------------



-- checks an overflow case

   #7 assert (items == 10, "wrong number of items")

   items -> 20



Verifying the total

   #12 assert (items == total, "wrong total")

   items -> 10

   total -> 30

_________________



Tests: 3

Failures: 2

Errors: 0

Note how much more informative this is when compared to the default output of running the test script with Lua:


:~/workspace$ lua5.1 test.lua

lua5.1: test.lua:7: wrong number of items

stack traceback:

        [C]: in function 'assert'

        test.lua:7: in main chunk

        [C]: ?

Unit Testing

luaunit

Shake

See Also