Integrating Behat and Mink with JenkinsCI

Continuing on the theme of my previous two posts, I’ve finally got around to getting some test results from Behat into Jenkins. The tests I’ll implement in this post are pretty simplistic acceptance tests against a pre-populated server, testing against a server populated by the Jenkins job will have to wait for another time.

The sources for the following can be found on my GitHub account.

The Tools

In addition to the stack used in the previous posts, I’m adding Phing to run the build process in Jenkins, and using GitHub to host the test files. As before, the installation of Sahi and PhantomJS is not covered, but feel free to post a comment if you get stuck with it.

My version of Jenkins is 1.499, and I’m using these plugins:

  • GitHub API
  • Git
  • GitHub
  • Phing
  • xUnit

Running Sahi as a Service

To ease the pain of Sahi coming and going with server restarts, I recommend configuring Sahi as a service that runs automatically. How this is done can easily be found with Google.

Setting up the project

I’ve created a new project containing only the composer.phar and composer.json files needed to install Behat, Mink and the dependencies. Running composer as before creates the vendor directory and populates it.

The next thing to do is to create the behat.yml file. This is very similar to the one used in previous posts:

default:
  extensions:
    Behat\MinkExtension\Extension:
      base_url: 'http://shaneauckland.co.uk'
      javascript_session: sahi
      browser_name: phantomjs
      goutte: ~
      sahi: ~

jenkins:
    formatter:
        name: pretty,junit
        parameters:
            output_path: ,xml

The difference is that now there are two profiles defined, default and jenkins. Behat profiles allow for different settings to be applied for different environments. The default profile is run by default (surprise), and subsequent profiles inherit from it. I’ve defined a jenkins profile, which adds an additional JUnit output format and writes files to the ./xml/ folder. I’ve left the default pretty format enabled as well, so we can see the tests running live in Jenkins.

Next I will create a features directory to contain my tests, and inside that I will create a bootstrap directory to hold my FeatureContext.php. Since my tests are very simple I don’t need to define any custom steps, but I will add the afterStep hook described in the previous post. This will make it possible to output screenshots when a scenario fails, although I’m not going to do that in this post.

Now to write some tests

Ok, so at this point Behat is configured to run headless though Sahi and PhantomJS, and the test suite is ready to actually test something. I’m going to do the most basic of acceptance tests to answer these questions:

  1. Is my site running?
  2. Can you see the page title?
  3. Can you get to the About Me page?

These questions are answered by the following two scenarios:

@javascript
Feature: Test blog is up
  Test that my blog is up on the expected domain name

Scenario: Test site root is found

    Given I am on "/"
    Then I should see "shanethehat"
    And I should see "Where I write stuff about programming"

Scenario: Test about page exists

    Given I am on "/"
    And I follow "About Me"
    Then I should be on "/about-me/"
    And I should see "About Me"

So now if I run vendor/bin/behat I see these tests complete successfully. Even better, if I run vendor/bin/behat --profile jenkins then I see the tests run, and afterwards I can look in the newly created xml directory to see the JUnit output. Cool!

So how do I put it into Jenkins?

In order for Jenkins to be able to install the composer dependencies and execute Behat all in one go, I need a build script. I’m using Phing for this, since I already have it installed on my Jenkins box and my laptop, but Ant or Maven would do the job just as well. My build script looks like this:

<?xml version="1.0" encoding="UTF-8" ?>

<project name="Blog tester" default="runner" basedir=".">
    
    <!-- Run the individual steps required -->
    <target name="runner" description="Overall step executor" depends="prepare, test">
        <echo msg="Running ..." />
    </target>

    <!-- Prepare the environment with composer -->
    <target name="prepare" description="Use composer to install dependancies">
        <echo msg="update composer" />
        <exec command="php composer.phar self-update" passthru="true" />
        <echo msg="install dependancies" />
        <exec command="php composer.phar update" passthru="true" />
    </target>

    <!-- Run the test suite -->
    <target name="test" description="Execute the behat tests">
        <echo msg="Run behat tests" />
        <exec command="vendor/bin/behat --profile jenkins" passthru="true" />
    </target>

</project>

I’m using passthru=”true” on all of my exec commands. This allows the output of the executables to be displayed, which means that I can see the messages from Composer and Behat mixed in with the Phing messages. This is really cool if you’re like me, and get a kick out of watching the Jenkins console during a build!

Jenkins needs somewhere to pull this test suite from, and I’m going to use GitHub. Since I don’t want to keep my vendor director or my test output under version control I’ll create a .gitignore file before I commit anything:

vendor/
xml/

Now I can safely initialize my repo, commit everything that’s not ignored and push it all up to GitHub.

Creating the Job

The final thing to do is to create the Jenkins job to run the tests. I just created a new empty job and configured it like so:

  • GitHub project URL: https://github.com/shanethehat/simple-behat-blog-test (allows direct linking to the sources on Github)
  • Source Control Management: Select Git and enter the above GitHub URL as the repository location
  • Build: Add a phing build step and point it at the runner target of the build script
  • Post Build Actions: Add xUnit Test Result Report, add a JUnit report and provide the path to the generated JUnit XML file, relative to the project root.

If all is well, then the job will build and the tests will pass.

What next?

This is great in that I can now have Jenkins regularly check that my sites are appearing as expected. I can call my acceptance tests as a last step in a deployment process to ensure that everything appears just as it should in production.

To make this even more robust though, I could have Jenkins pushing my website files onto a staging server first, and use a separate Behat profile to run the same tests with my staging environment as the base URL.

I can also have Jenkins publish screenshots of the failures, and maybe even have them emailed directly to me in the event of a failure. But these are all experiments for another day.

2 Comments

  • 19th January, 2013 - 9:40 pm | Permalink

    Hi Shane, Thanks a lot for the great post. I was just wondering to install dependencies and update composer before every build trigger? It will be so time consuming. An idea about creating another profile is great but we still can confugure ANT file to generate report. I have one demo project using ANT, Please have a look and let me know if I am doing anything wrong?
    Here is a link https://github.com/Shashikant86/BehatDemo

    • 20th January, 2013 - 9:36 am | Permalink

      Hi Shashikant,

      The dependencies don’t actually install every time, the only delay on subsequent test passes is for Composer to verify that there are no updates to perform. Sometimes there will be, and that will slow things down a bit, but it’s a tradeoff against the portability of the tests.

      I looked briefly at your project, and I think that’s a perfectly valid way of doing things. I chose to keep my Behat configs in the Yaml file, but your approach works just as well. To be honest, I’m still a Behat novice, so I don’t know if either approach is better or if it’s just a case of personal preference.

  • Leave a Reply

    Your email address will not be published. Required fields are marked *

    You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>