1. Introduction
PureUnit allows automated unit testing for PureBasic code with similar concepts to other xUnit frameworks like JUnit. The basic idea of unit testing is the creation of a set of tests that can run without user interaction so they can be repeated many times during the development process to ensure that the code still does what it is supposed to do.
The main reason for the creation of PureUnit was the testing of (user-)libraries for PureBasic, but it can just as well be used to test code written in PureBasic itself. Because the goal is testing without user interaction, PureUnit is not intended for testing entire application or a finished program, but rater for testing those parts of a program like individual functions, which can be executed separately from the rest of the program, in order to verify their correctness (for example an include file with commonly used functions, etc.).
PureUnit consists of a set of keywords (in the form of macros) to write the actual test code and a tool which will analyze the code to extract information, compile and run the test code and report its status back to the user. The tool comes with both a command-line interface, for use in makefiles or scrips, as well as with a graphical user interface, to allow testing directly from the PureBasic IDE, for example.
The Windows version comes as 2 separate executables for console/GUI use.
The reason for this is that a console application on Windows will always open a console window, even when started from File Explorer, which is annoying when using it as a GUI tool.
Therefore, these two files contain the exact same program: one being compiled in console mode, the other in GUI mode.
Similarly, the Mac OSX version comes as PureUnit.app (for GUI use) and as pureunit for console use.
|
2. Installation
-
The
PureUnit.resneeds to be copied theResidentsDirectory of your PureBasic installation. -
Optionally, the
PureUnitKeywords.txtfile can be selected in the preferences of the PureBasic IDE () as a keyword file to color all PureUnit specific keywords in a separate color. -
The PureUnit executables can be executed from anywhere.
| PureUnit requires a minimum compiler version of 4.10 due to its use of the compiler interface introduced in that version. |
3. Writing the Test Code
All test code must be written inside procedures. Code outside of the test procedures will not be executed. So just like in a DLL there should only be stuff like constant definitions, structures etc. on the main level. (Note though that PureUnit does not compile the code to a DLL for the tests; the code is compiled to an executable with extra embedded code to execute the tests.)
Test procedures are created just like normal procedures but with one of the below described keywords instead of Procedure.
Test procedures MUST have no arguments.
They should have descriptive names (representing what they actually test for), as the names are reported in case of a failure.
A test is considered to be passed if the procedure ends normally (or is left with ProcedureReturn).
A test is considered to be failed if the below described Assert() macro fails, the Fail() macro is called, or a runtime error (OnError library) happened.
Macros
PureUnit adds the following new keywords (as macros without arguments):
ProcedureUnit-
A procedure created with this keyword is considered to be a test procedure.
The test procedures in a given code will be executed in RANDOM order. The reason for this is to find problems from possible side-effects of the tests which would go unnoticed if the tests always get executed in the same order. (for example if some test always succeeds just because the test before it sets certain conditions that were not realized when the test code got written)
ProcedureUnitStartup-
All procedures created with this keyword will be executed at the startup of the compiled program (there can be more than one startup procedure). Use this to set up global conditions (variables etc.) for all tests.
ProcedureUnitShutdown-
All procedures with this keyword will be executed at the end of all tests. Note that they will also be executed when the program ends because of a failed test. Use this to do any cleanup after the tests.
ProcedureUnitBefore-
All procedures with this keyword will be executed before every single test. Use this to do preparations that should be done before every test separately.
ProcedureUnitAfter-
All procedures with this keyword will be executed after every single test. (also after a failed test.)
EndProcedureUnit-
This keyword has no special meaning to PureUnit. It is just defined as a macro for
EndProcedureto have a matching end keyword (ProcedureUnit/EndProcedureUnit). Using this orEndProceduremakes no difference whatsoever. It’s there just for aesthetic reasons.
Macros with Arguments
PureUnit adds the following new macros with arguments:
Assert( <expression> [, Message$] )-
This is the central macro of the testing concept. It is used to check all conditions that must be true for the test to succeed. If any of the conditions is found to be false, the entire test is considered to have failed.
<expression>can be any expression that evaluates to true (nonzero) or false (zero). Since the macro evaluates the expression with anIfstatement, the logical operatorsAnd,Or,XOrandNotcan be used.Message$is an optional argument that can contain a message to be shown (in the test protocol or in the console) if theAssertfails.ExampleAssert(MyVariable = 123 And aFunction() = 0) AssertString( String1$, String2$ [, Message$] )-
This is a special macro intended to test results from string functions inside (user-)libraries. For any other purpose,
Assert(String1$ = String2$)will work just fine.Detailed InformationIf a string function inside a library does not correctly allocate the internal string buffer (i.e. allocating more characters than are actually filled), the result will look just fine if seen as a single value. It will however become wrong if the string function result is added together with other strings (because there will be a NULL character inside the string then).
In order to test for this exact situation in a convenient way, this macro actually maps to:
Assert(Chr(1)+String1$+Chr(1) = Chr(1)+String2$+Chr(1))Since this situation cannot happen with PB code directly, there is no need to use this macro to test string equality in normal PB code.
Fail( [Message$] )-
If this macro is executed, the test will be considered as having failed, and the specified optional message will be reported. It is used to indicate failure if the test code reaches a position that it is not supposed to reach.
UnitDebug( String$ )-
Since the tests are executed without debugger (because the debugger is an interactive tool), there is no quick and dirty way like
Debug …to print information. This command will show the specified string in the GUI in a different color and also list this output in the HTML report.
|
This command should not be used for extensive output, like test results or similar — remember that the goal is unattended automated testing! This is rather meant as a quick way to provide some output when tracking a problem in the test code, etc. |
PureUnitOptions( Option1 [, Option2 [, …]])-
This macro controls the compilation of the source code for testing. It’s the only macro that should only appear once in each test code. If this macro is not present, or if no flags are specified, the code will be compiled/tested twice: once normally and once in Unicode mode. The following flags are possible:
NoUnicode→ no Unicode tests.
Thread→ test also in thread mode.
SubSystem[<name>]→ test also with the given subsystem.
The code will be compiled in all possible combinations of the given flags (on/off), so when using
PureUnitOptions(Thread), the code will be compiled once normally, once in Unicode mode, once in thread mode, and once in thread+Unicode mode. So every added option (except forNoUnicode) actually doubles the number of times the given test source is compiled/executed. This can be a speed concern when carrying out many tests, so only those options which are truly needed should be enabled here.
Handling of Included Files
PureUnit needs to parse the source code in order to know about the used test-procedures, etc.
The parser does not do a full analysis of the code, so there are some limits to what it can handle, especially when it comes to include files.
The parser does handle correctly IncludePath and (X)IncludeFile statements, but only as long as the paths are given as literal strings.
The parser does NOT resolve constants or macros and also it does not take care of compiler directives like CompilerIf.
So IncludePath and (X)IncludeFile can be used in test code (to include common startup code for example), even in nested files, but they may only contain literal strings.
Code Example
Note that everything in the example is optional, except that one ProcedureUnit must be present otherwise the file is skipped (which makes sense, since in that case there would be no tests).
; Set the options. This Code will be compiled/tested once normally and once with the OpenGl subsystem:
;
PureUnitOptions(NoUnicode, SubSystem[OpenGl])
; called once on program start
;
ProcedureUnitStartup startup()
UnitDebug("Startup...")
EndProcedureUnit
; called once on program end
;
ProcedureUnitShutdown shutdown()
UnitDebug("Shutdown...")
EndProcedureUnit
; Same goes for ProcedureUnitBefore / ProcedureUnitAfter
; Here comes our test procedure
;
ProcedureUnit mytest()
Assert(#True <> #False, "If this fails, the world has gone crazy :)")
For i = 1 To 10
Continue
Fail("This point should never be reaced")
Next i
; here our test will fail.
Assert(#True = #False, "This is bound to fail!")
EndProcedureUnit
Note that all PureUnit macros have a fallback to Messagerequester/Debugger display if the code is not executed from within PureUnit. This allows any PureUnit test code to be executed from the Editor or compiler without PureUnit for quick syntax checks or debugging while creating the test code.
4. Command-Line Arguments
PureUnit supports on all OSs the POSIX-style short and long versions of the command-line arguments described below (but not combining short versions: e.g. -x -y -z does not equal -xyz).
The Windows style command-line parameters (e.g. /HELP) are only supported by the Windows version though.
For portable testing, the POSIX-style arguments should be preferred.
On exit, PureUnit returns a standard exit code to indicate success (0) or failure (1) of the tests to allow running test from scripts or makefiles. When executed without any command-line options, PureUnit will enter GUI mode and display a window which allows the selections of files/folders for testing.
Usage
pureunit [Options] TestFiles
Wildcards are allowed to specify multiple TestFiles for testing easily.
Options
-c <compiler> --compiler <compiler> /COMPILER <compiler>
This option allows to specify the PureBasic compiler to use for compilation. If this option is not specified, PureUnit will try to locate a compiler to use in the following places in the given order:
-
The
PUREBASIC_HOMEenvironment variable. -
The
PATHenvironment variable. -
The Registry (Windows only).
-
The fixed path
/usr/share/purebasic(Linux and OSX only).
| The compiler must have a version of 4.10 or higher to work with PureUnit. |
-f <listfile> --files <listfile> /FILES <listfile>
Specify a file that contains a list of source files for testing. The file should contain one entry per line. Wildcards are allowed.
-g --gui /GUI
With this switch present, PureUnit will not output anything on the console, but rather display a window to show the progress and information messages. Without this switch, messages are printed on the console only. This switch is useful to configure PureUnit as a tool for the PureBasic IDE, for example.
-v --verbose /VERBOSE
Increases the verbosity of the output. (For example, prints the currently running test and similar information instead of just error messages as it is in normal mode.)
This affects the console mode only.
-q --quiet /QUIET
Hides all warning messages. (Warning messages appear when the parser skips a file and similar minor issues)
This affects both console and GUI mode.
-i --ignore /IGNORE
With this switch set, the testing does not stop with a failed test but continues until all tests have been executed.
The normal behavior is to quit testing at the first failed test, since a failed test is considered fatal. This switch, however, allows to change this in order to get an overview of how many tests actually do fail in a test code.
|
If a test fails, PureUnit will call the If this switch is set, PureUnit will restart the executable.
Call the |
-r <html file> --report <html file> /REPORT <html file>
This switch creates a test report in HTML format in the given file.
Note that in GUI mode, there is also an option to view/save this report after the tests have been done.
-v --version /VERSION
Displays version information and quits.
-h --help /HELP /?
Displays command-line arguments and quits.
5. Using the Graphic User Interface
Starting PureUnit without any command-line options will launch it in GUI mode.
The settings entered there will be saved in PureUnit.prefs in the standard location for PureBasic preference files.
The GUI provides the following options:
- Compiler Executable
-
Select the compiler to use. Must be of version 4.10 or above.
- Source Directory
-
Select the base directory in which to look for files to test.
- Source Pattern
-
Select the pattern to search for when processing a whole directory.
- Process the whole directory
-
With this switch set, all files in ‘Source Directory’ matching ‘Source Pattern’ will be tested. Without this switch, you will be prompted to select a file for testing when ‘Start’ is pressed.
- Include subdirectories
-
Recursively scans subdirectories for files to be tested.
- Do not abort on errors
-
Same as the
--ignorecommand-line switch. See there for more information. - Hide warnings
-
Hides warning messages. Same as the
--quietcommand-line switch.
After pressing ‘Start’ (and selecting a test file if needed), a progress window will be displayed showing log and debug messages while the tests are executed. After finishing or aborting the tests, a report in HTML format can be viewed or saved.
Starting PureUnit with options on the command-line (including the --gui switch) will cause PureUnit to directly show the progress window and start the testing with the provided command-line options.
Contributing and Support
Feel free to contribute to this document by submitting your own fixes and improvements via Git and creating a pull request on the PureBasic Open Source repository:
If you have any questions, remarks or suggestions, just write to: <support@purebasic.com>.