Continuing on the theme of my previous two posts, I’ve finally got around to getting some test results from Behat into Jenkins. The tests I’ll implement in this post are pretty simplistic acceptance tests against a pre-populated server, testing against a server populated by the Jenkins job will have to wait for another time.
The sources for the following can be found on my GitHub account.
In addition to the stack used in the previous posts, I’m adding Phing to run the build process in Jenkins, and using GitHub to host the test files. As before, the installation of Sahi and PhantomJS is not covered, but feel free to post a comment if you get stuck with it.
My version of Jenkins is 1.499, and I’m using these plugins:
- GitHub API
Running Sahi as a Service
To ease the pain of Sahi coming and going with server restarts, I recommend configuring Sahi as a service that runs automatically. How this is done can easily be found with Google.
Setting up the project
I’ve created a new project containing only the composer.phar and composer.json files needed to install Behat, Mink and the dependencies. Running composer as before creates the vendor directory and populates it.
The next thing to do is to create the behat.yml file. This is very similar to the one used in previous posts:
The difference is that now there are two profiles defined, default and jenkins. Behat profiles allow for different settings to be applied for different environments. The default profile is run by default (surprise), and subsequent profiles inherit from it. I’ve defined a jenkins profile, which adds an additional JUnit output format and writes files to the ./xml/ folder. I’ve left the default pretty format enabled as well, so we can see the tests running live in Jenkins.
Next I will create a features directory to contain my tests, and inside that I will create a bootstrap directory to hold my FeatureContext.php. Since my tests are very simple I don’t need to define any custom steps, but I will add the afterStep hook described in the previous post. This will make it possible to output screenshots when a scenario fails, although I’m not going to do that in this post.
Now to write some tests
Ok, so at this point Behat is configured to run headless though Sahi and PhantomJS, and the test suite is ready to actually test something. I’m going to do the most basic of acceptance tests to answer these questions:
- Is my site running?
- Can you see the page title?
- Can you get to the About Me page?
These questions are answered by the following two scenarios:
So now if I run
vendor/bin/behat I see these tests complete successfully. Even better, if I run
vendor/bin/behat --profile jenkins then I see the tests run, and afterwards I can look in the newly created xml directory to see the JUnit output. Cool!
So how do I put it into Jenkins?
In order for Jenkins to be able to install the composer dependencies and execute Behat all in one go, I need a build script. I’m using Phing for this, since I already have it installed on my Jenkins box and my laptop, but Ant or Maven would do the job just as well. My build script looks like this:
<?xml version="1.0" encoding="UTF-8" ?> <project name="Blog tester" default="runner" basedir="."> <!-- Run the individual steps required --> <target name="runner" description="Overall step executor" depends="prepare, test"> <echo msg="Running ..." /> </target> <!-- Prepare the environment with composer --> <target name="prepare" description="Use composer to install dependancies"> <echo msg="update composer" /> <exec command="php composer.phar self-update" passthru="true" /> <echo msg="install dependancies" /> <exec command="php composer.phar update" passthru="true" /> </target> <!-- Run the test suite --> <target name="test" description="Execute the behat tests"> <echo msg="Run behat tests" /> <exec command="vendor/bin/behat --profile jenkins" passthru="true" /> </target> </project>
I’m using passthru=”true” on all of my exec commands. This allows the output of the executables to be displayed, which means that I can see the messages from Composer and Behat mixed in with the Phing messages. This is really cool if you’re like me, and get a kick out of watching the Jenkins console during a build!
Jenkins needs somewhere to pull this test suite from, and I’m going to use GitHub. Since I don’t want to keep my vendor director or my test output under version control I’ll create a .gitignore file before I commit anything:
Now I can safely initialize my repo, commit everything that’s not ignored and push it all up to GitHub.
Creating the Job
The final thing to do is to create the Jenkins job to run the tests. I just created a new empty job and configured it like so:
- GitHub project URL: https://github.com/shanethehat/simple-behat-blog-test (allows direct linking to the sources on Github)
- Source Control Management: Select Git and enter the above GitHub URL as the repository location
- Build: Add a phing build step and point it at the runner target of the build script
- Post Build Actions: Add xUnit Test Result Report, add a JUnit report and provide the path to the generated JUnit XML file, relative to the project root.
If all is well, then the job will build and the tests will pass.
This is great in that I can now have Jenkins regularly check that my sites are appearing as expected. I can call my acceptance tests as a last step in a deployment process to ensure that everything appears just as it should in production.
To make this even more robust though, I could have Jenkins pushing my website files onto a staging server first, and use a separate Behat profile to run the same tests with my staging environment as the base URL.
I can also have Jenkins publish screenshots of the failures, and maybe even have them emailed directly to me in the event of a failure. But these are all experiments for another day.