silverstripe-behat-extension/README.md

# SilverStripe Integration for Behat

## Overview

[Behat](http://behat.org) is a testing framework for behaviour-driven development.
Because it primarily interacts with your website through a browser, 
you don't need any specific integration tools to get it going with
a basic SilverStripe website, simply follow the 
[standard Behat usage instructions](http://docs.behat.org/).

This extension comes in handy if you want to go beyond
interacting with an existing website and database,
for example make changes to your database content which
would need to be rolled back to a "clean slate" later on.

It provides the following helpers:

 * Provide access to SilverStripe classes in your Behat contexts
 * Set up a temporary database automatically
 * Reset the database content on every scenario
 * Prebuilt Contexts for SilverStripe's login forms and other common tasks
 * Creating of member fixtures with predefined permissions
 * YML fixture definitions inside your Behat scenarios
 * Waiting for jQuery Ajax responses (rather than fixed wait timers)
 * Captures JavaScript errors and logs them through Selenium
 * Saves screenshots to filesystem whenever an assertion error is detected

In order to achieve this, the extension makes one basic assumption:
Your Behat tests are run from the same application as the tested
SilverStripe codebase, on a locally hosted website from the same codebase.
This is important because we need access to the underlying SilverStripe
PHP classes. You can of course use a remote browser to do the actual testing.

Note: The extension has only been tested with the `selenium2` Mink driver.

## Installation

Simply [install SilverStripe through Composer](http://doc.silverstripe.org/framework/en/installation/composer).
Skip this step if adding the module to an existing project.

	composer create-project silverstripe/installer my-test-project 3.1.x-dev

Switch to the newly created webroot, and add the SilverStripe Behat extension.

	cd my-test-project
	composer require silverstripe/behat-extension:*

Now get the latest Selenium2 server (requires Java):

	wget http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar

We need to generate a token so the browser and commandline calls can interact with a "shared secret":

	php framework/cli-script.php dev/generatesecuretoken path=mysite/_config/behat.yml

Now install the SilverStripe project as usual by opening it in a browser and following the instructions.
Protip: You can skip this step by using `[SS_DATABASE_CHOOSE_NAME]` in a global 
[_ss_environment.php](http://doc.silverstripe.org/framework/en/topics/environment-management) 
file one level above the webroot.

Unless you have [$_FILE_TO_URL_MAPPING](http://doc.silverstripe.org/framework/en/topics/commandline#configuration)
set up, you also need to specify the URL for your webroot. Either add it to the existing `behat.yml` configuration file
in your project root, or set is as an environment variable in your terminal session: 

	export BEHAT_PARAMS="extensions[SilverStripe\BehatExtension\MinkExtension][base_url]=http://localhost/"

## Usage

### Starting the Selenium Server

You can run the server locally in a separate Terminal session:

    java -jar selenium-server-standalone-2.31.0.jar

### Running the Tests

Now you can run the tests (for example for the `framework` module):

	vendor/bin/behat @framework

In order to run specific tests only, use their feature file name:

	vendor/bin/behat @framework/login.feature

This will start a Firefox browser by default. Other browsers and profiles can be configured in `behat.yml`.

## Configuration

The SilverStripe installer already comes with a YML configuration
which is ready to run tests on a locally hosted Selenium server,
located in the project root as `behat.yml`.

You'll need to customize at least the `base_url` setting to match the URL where
the tested SilverStripe instance is hosted locally.  This 

Generic Mink configuration settings are placed in `SilverStripe\BehatExtension\MinkExtension`,
which is a subclass of `Behat\MinkExtension\Extension`.

Overview of settings (all in the `extensions.SilverStripe\BehatExtension\Extension` path):

 * `framework_path`: Path to the SilverStripe Framework folder. It supports both absolute and relative (to `behat.yml` file) paths.
 * `extensions.Behat\MinkExtension\Extension.base_url`: You will probably need to change the base URL that is used during the test process.
It is used every time you use relative URLs in your feature descriptions.
It will also be used by [file to URL mapping](http://doc.silverstripe.org/framework/en/topics/commandline#configuration) in `SilverStripeExtension`.
 * `extensions.Behat\MinkExtension\Extension.files_path`: Change to support file uploads in your tests.  Currently only absolute paths are supported.
 * `ajax_steps`: Because SilverStripe uses AJAX requests quite extensively, we had to invent a way
to deal with them more efficiently and less verbose than just
Optional `ajax_steps` is used to match steps defined there so they can be "caught" by
[special AJAX handlers](http://blog.scur.pl/2012/06/ajax-callback-support-behat-mink/) that tweak the delays. You can either use a pipe delimited string or a list of substrings that match step definition.
 * `ajax_timeout`: Milliseconds after which an Ajax request is regarded as timed out, 
 and the script continues with its assertions to avoid a deadlock (Default: 5000).
 * `screenshot_path`: Used to store screenshot of a last known state
of a failed step. It defaults to whatever is returned by PHP's `sys_get_temp_dir()`.
Screenshot names within that directory consist of feature file filename and line
number that failed.

Example: behat.yml 

	default:
	  context:
	    class: SilverStripe\MyModule\Test\Behaviour\FeatureContext
	  extensions:
	    SilverStripe\BehatExtension\Extension: ~
	    SilverStripe\BehatExtension\MinkExtension:
	      # Adjust this to your local environment
	      base_url:  http://localhost/
	      default_session: selenium2
	      javascript_session: selenium2
	      goutte: ~
	      selenium2:
	        browser: firefox

### Available Step Definitions

The extension comes with several `BehatContext` subclasses come with some extra step defintions.
Some of them are just helpful in general website testing, other's are specific to SilverStripe. 
To find out all available steps (and the files they are defined in), run the following:

	vendor/bin/behat @mymodule --definitions=i

Note: There are more specific step definitions in the SilverStripe `framework` module
for interacting with the CMS interfaces (see `framework/tests/behat/features/bootstrap`).

## Fixtures

Since each test run creates a new database, you can't rely on existing state unless
you explicitly define it. 

### Database Defaults

The easiest way to get default data is through `DataObject->requireDefaultRecords()`.
Many modules already have this method defined, e.g. the `blog` module automatically
creates a default `BlogHolder` entry in the page tree. Sometimes these defaults can
be counterproductive though, so you need to "opt-in" to them, via the `@database-defaults`
tag placed at the top of your feature definition. The defaults are reset after each
scenario automatically.

### Inline Definition

If you need more flexibility and transparency about which records are being created,
use the inline definition syntax. The following example shows some syntax variations:

	Feature: Do something with pages
		As an site owner
		I want to manage pages

		Background:
			# Creates a new page without data. Can be accessed later under this identifier
			Given a "page" "Page 1" 
			# Uses a custom RegistrationPage type
			And an "error page" "Register" 
			# Creates a page with inline properties 
			And a "page" "Page 2" with "URLSegment"="page-1" and "Content"="my page 1" 
			# Field names can be tabular, and based on DataObject::$field_labels 
			And the "page" "Page 3" has the following data
			 | Content | <blink> |
			 | My Property | foo |
			 | My Boolean | bar |
			# Pages are published by default, can be explicitly unpublished
			And the "page" "Page 1" is not published 
			# Create a hierarchy, and reference a record created earlier
			And the "page" "Page 1.1" is a child of a "page" "Page 1" 
			# Specific page type step 
			And a "page" "My Redirect" which redirects to a "page" "Page 1" 
			And a "member" "Website User" with "FavouritePage"="=>Page.Page 1"

		@javascript
		Scenario: View a page in the tree
			Given I am logged in with "ADMIN" permissions
			And I go to "/admin/pages"
			Then I should see "Page 1" in CMS Tree

 * Fixtures are created where you defined them. If you want the fixtures to be created
   before every scenario, define them in [Background](http://docs.behat.org/guides/1.gherkin.html#backgrounds). 
   If you want them to be created only when a particular scenario runs, define them there.
 * The basic syntax works for all `DataObject` subclasses, but some specific
   notations like "is not published" requires extensions like `Hierarchy` to be applied to the class
 * Record identifiers, property names and property values need to be quoted
 * Record types shouldn't be quoted, and can use more natural notation ("registration page" instead of "Registration Page") 
 * Fixtures are usually not cleared between scenarios. You can alter this behaviour
   by tagging the feature or scenario with `@database-defaults` tag.
 * Property values may also use a `=>` symbol to indicate relationships between records.
   The notation is `=><classname>.<identifier>`. For `has_many` or `many_many` relationships,
   multiple relationships can be separated by a comma.

The module runner empties the database before each scenario tagged with
`@database-defaults` and populates it with default records (usually a set of
default pages).

## Writing Behat Tests

### Directory Structure

As a convention, SilverStripe Behat tests live in a `tests/behat` subfolder
of your module. You can create it with the following command:

	mkdir -p mymodule/tests/behat/features/bootstrap/MyModule/Test/Behaviour

### FeatureContext

The generic [Behat usage instructions](http://docs.behat.org/) apply
here as well. The only major difference is the base class from which
to extend your own `FeatureContext`: It should be `SilverStripeContext`
rather than `BehatContext`.

Example: mymodule/tests/behat/features/bootstrap/MyModule/Test/Behaviour/FeatureContext.php

	<?php
	namespace MyModule\Test\Behaviour;

	use SilverStripe\BehatExtension\Context\SilverStripeContext,
	    SilverStripe\BehatExtension\Context\BasicContext,
	    SilverStripe\BehatExtension\Context\LoginContext;

	require_once 'PHPUnit/Autoload.php';
	require_once 'PHPUnit/Framework/Assert/Functions.php';

	class FeatureContext extends SilverStripeContext
	{
	    public function __construct(array $parameters)
	    {
	        $this->useContext('BasicContext', new BasicContext($parameters));
	        $this->useContext('LoginContext', new LoginContext($parameters));

	        parent::__construct($parameters);
	    }
	}

## FAQ

### FeatureContext not found

This is most likely a problem with Composer's autoloading generator.
Check that you have "SilverStripe" mentioned in the `vendor/composer/autoload_classmap.php` file,
and call `composer dump-autoload` if not.

### Why does the module need to know about the framework path on the filesystem?

Sometimes SilverStripe needs to know the URL of your site. When you're visiting
your site in a web browser this is easy to work out, but if you're executing
scripts on the command-line, it has no way of knowing.

To work this out, this module is using [file to URL mapping](http://doc.silverstripe.org/framework/en/topics/commandline#configuration).

### How does the module interact with the SS database?

The module creates temporary database on init and is switching to the alternative
database session before every scenario by using `/dev/tests/setdb` TestRunner
endpoint.

It also populates this temporary database with the default records if necessary.

It is possible to include your own fixtures, it is explained further.

### Why do tests pass in a fresh installation, but fail in my own project?

Because we're testing the interface directly, any changes to the
viewed elements have the potential to disrupt testing. 
By building a test database from scratch, we're trying to minimize this impact.
Some examples where things can go wrong nevertheless:

 * Thirdparty SilverStripe modules which install default data
 * Changes to the default interface language
 * Configurations which remove admin areas or specific fields

Currently there's no way to exclude offending modules from a test run.
You either have to adjust the tests to work around these changes,
or run tests on a "sandbox" projects without these modules.

### How do I debug when something goes wrong?

First, read the console output. Behat will tell you which steps have failed.

SilverStripe Behaviour Testing Framework also notifies you about some events.
It tries to catch some JavaScript errors and AJAX errors as well although it
is limited to errors that occur after the page is loaded.

Screenshot will be taken by the module every time the step is marked as failed.
Refer to configuration section above to know how to set up the screenshot path.

If you are unable to debug using the information collected with the above
methods, it is possible to delay the step execution by adding the following step:

    And I wait for "10000"

where `10000` is the number of millisecods you wish the session to wait.
It is very useful when you want to look at the error or developer console
inside the browser or if you want to interact with the session page manually.

### How do I use SauceLabs.com for remote Selenium2 testing?

Here's a sample profile for your `behat.yml`:

	# Saucelabs.com sample setup, use with "vendor/bin/behat --profile saucelabs"
	saucelabs:
	  extensions:
	    SilverStripe\BehatExtension\MinkExtension:
	      selenium2:
	        browser: firefox
	        # Add your own username and API token here
	        wd_host: <user>:<api-token>@ondemand.saucelabs.com/wd/hub
	        capabilities:
	          platform: "Windows 2008"
	          browser: "firefox"
	          version: "15"

## Useful resources

* [SilverStripe CMS architecture](http://doc.silverstripe.org/sapphire/en/trunk/reference/cms-architecture)
* [SilverStripe Framework Test Module](https://github.com/silverstripe-labs/silverstripe-frameworktest)
* [SilverStripe Unit and Integration Testing](http://doc.silverstripe.org/sapphire/en/trunk/topics/testing)