Docs for exec pipeline and manifests

- Added manifest logic docs
- More obvious pointers to flushing
- Removed Director page since it duplicated a lot of information already presenting in routing,
which in itself is already duplicated between routing.md and controllers.md.
- Merged Director execution rundown into main exec pipeline docs
- Moved YAML security note from main installation docs to "secure coding", making it sless obvious

docs/en/00_Getting_Started/01_Installation/index.md

@@ -1,45 +1,27 @@
 # Installation
 
 These instructions show you how to install SilverStripe on any web server. 
-
 The best way to install from the source code is to use [Composer](composer).
-
+Check out our operating system specific guides for [Linux](linux_unix),
-For additional information about installing SilverStripe on specific operation systems, refer to:
+[Windows Server](windows-pi) and [Mac OSX](mac-osx).
-
-*  [Installation on a Windows Server](windows-pi)
-*  [Installation on OSX](mac-osx)
 
 ## Installation Steps
 
 *  [Download](http://silverstripe.org/download) the installer package
-
+*  Make sure the webserver has MySQL and PHP support.  See [Server Requirements](server-requirements) for more information. 
-*  Make sure the webserver has MySQL and PHP support.  See [Server Requirements](server-requirements) for more
-information. 
-
 *  Unpack the installer somewhere into your web-root. Usually the www folder or similar. Most downloads from SilverStripe
 are compressed tarballs. To extract these files you can either do them natively (Unix) or with 7-Zip (Windows)
-
+*  Visit your sites domain or IP address in your web browser.
-*  Visit your sites Domain or IP Address in your web browser.
-
 *  You will be presented with a form where you enter your MySQL login details and are asked to give your site a 'project
 name' and the default login details. Follow the questions and select the *install* button at the bottom of the page.
-
 *  After a couple of minutes, your site will be set up. Visit your site and enjoy!
 
 ## Issues?
 
 If the above steps don't work for any reason have a read of the [Common Problems](common-problems) section.
 
-## Security notes
+<div class="notice" markdown="1">
-
+SilverStripe ships with default rewriting rules specific to your web server. Apart from
-### Yaml
+routing requests to the framework, they also prevent access to sensitive files in the webroot,
-
+for example YAML configuration files. Please refer to the [security](/topics/security) documentation for details.
-For the reasons explained in [security](/topics/security), Yaml files are blocked by default by the .htaccess file
+</div>
-provided by the SilverStripe installer module.
-
-To allow serving yaml files from a specific directory, add code like this to an .htaccess file in that directory
-
-	<Files *.yml>
-		Order allow,deny
-		Allow from all
-	</Files>

docs/en/00_Getting_Started/04_Directory_Structure.md

@@ -78,11 +78,13 @@ Example Forum Documentation:
 |`forum/docs/en/04_Some_Topic/00_Getting_Started.md`|Structure is created by use of numbered prefixes. This applies to nested folders and documentations pages, index.md should not have a prefix.|
 
 
-## PHP Include Paths
+## Autoloading
 
-Due to the way `[api:ManifestBuilder]` recursively detects php-files and includes them through PHP5's
+SilverStripe recursively detects classes in PHP files by building up a manifest used for autoloading,
-*__autoload()*-feature, you don't need to worry about include paths. Feel free to structure your php-code into
+as well as respecting Composer's built-in autoloading for libraries. This means
-subdirectories inside the *code*-directory.
+in most cases, you don't need to worry about include paths or `require()` calls
+in your own code - after adding a new class, simply regenerate the manifest
+by using a `flush=1` query parameter. See the ["Manifests" documentation](/developer_guides/execution_pipeline/manifests) for details.
 
 ## Best Practices
 

docs/en/02_Developer_Guides/02_Controllers/02_Routing.md

@@ -160,7 +160,9 @@ This is useful when you want to provide custom actions for the mapping of `teams
 
 The syntax for the `$url_handlers` array users the same pattern matches as the `YAML` configuration rules.
 
-## API Documentation
+## Links
 
-* [api:Controller]
+* [api:Controller] API documentation
-* [api:Director]
+* [api:Director] API documentation
+* [Example routes: framework](https://github.com/silverstripe/silverstripe-framework/blob/master/_config/routes.yml)
+* [Example routes: cms](https://github.com/silverstripe/silverstripe-cms/blob/master/_config/routes.yml)

docs/en/02_Developer_Guides/09_Security/04_Secure_Coding.md

@@ -406,17 +406,19 @@ file in the assets directory.  This requires PHP to be loaded as an Apache modul
 	php_flag engine off
 	Options -ExecCGI -Includes -Indexes 
 
-### Don't allow access to .yml files
+### Don't allow access to YAML files
 
-Yaml files are often used to store sensitive or semi-sensitive data for use by SilverStripe framework (for instance,
+YAML files are often used to store sensitive or semi-sensitive data for use by 
-configuration and test fixtures).
+SilverStripe, such as configuration files. We block access to any files
-
+with a `.yml` or `.yaml` extension through the default web server rewriting rules.
-You should therefore block access to all yaml files (extension .yml) by default, and white list only yaml files
+If you need users to access files with this extension,
-you need to serve directly.
+you can bypass the rules for a specific directory.
-
+Here's an example for a `.htaccess` file used by the Apache web server:
-See [Apache](/installation/webserver) and [Nginx](/installation/nginx) installation documentation for details 
-specific to your web server
 
+	<Files *.yml>
+		Order allow,deny
+		Allow from all
+	</Files>
 
 ## Passwords
 

docs/en/02_Developer_Guides/16_Execution_Pipeline/01_Director.md

@@ -1,87 +0,0 @@
-# Director
-
-## Introduction
-
-`[api:Director]` is the first step in the "execution pipeline". It parses the 
-URL, matching it to one of a number of patterns, and determines the controller, 
-action and any argument to be used. It then runs the controller, which will 
-finally run the viewer and/or perform processing steps.
-
-## Request processing
-
-The `[api:Director]` is the entry point in Silverstring Framework for processing 
-a request. You can read through the execution steps in `[api:Director]``::direct()`, 
-but in short
-
-* File uploads are first analysed to remove potentially harmful uploads (this 
-will likely change!)
-* The `[api:SS_HTTPRequest]` object is created
-* The session object is created
-* The `[api:Injector]` is first referenced, and asks the registered `[api:RequestProcessor]` 
-to pre-process the request object. This allows for analysis of the current 
-request, and allow filtering of parameters etc before any of the core of the 
-application executes.
-* The request is handled and response checked
-* The `[api:RequestProcessor]` is called to post-process the request to allow 
-further filtering before content is sent to the end user
-* The response is output
-
-The framework provides the ability to hook into the request both before and 
-after it is handled to allow developers to bind in their own custom pre- or 
-post- request logic; see the `[api:RequestFilter]` to see how this can be used 
-to authenticate the request before the request is handled. 
-
-## Routing
-
-You can influence the way URLs are resolved in the following ways
-
-1. Adding rules to `[api:Director]` in `<yourproject>/_config/routes.yml` 
-2. Adding rules to `[api:Director]` in `<yourproject>/_config.php (deprecated)
-3. Adding rules in your extended `[api:Controller]` class via the *$url_handlers* 
-static variable 
-
-See [controller](/topics/controller) for examples and explanations on how the 
-rules get processed for those methods.
-
-
-### Routing Rules
-
-SilverStripe comes with certain rules which map a URI to a `[api:Controller]`
-class (e.g. *dev/* -> DevelopmentAdmin). These routes are either stored in 
-a routes.yml configuration file located a `_config` directory or inside a 
-`_config.php` file (deprecated). 
-
-To add your own custom routes for your application create a routes.yml file 
-in `<yourproject>/_config/routes.yml` with the following format:
-
-	:::yaml
-	---
-	Name: customroutes
-	After: framework/routes#coreroutes
-	---
-	Director:
-  		rules:
-    		'subscriptions/$Action' : 'SubscriptionController'
-
-The [Controller](/topics/controller) documentation has a wide range of examples 
-and explanations on how the rules get processed for those methods.
-
-See:
-
-*  [framework/_config/routes.yml](https://github.com/silverstripe/silverstripe-framework/blob/master/_config/routes.yml)
-*  [cms/_config/routes.yml](https://github.com/silverstripe/silverstripe-cms/blob/master/_config/routes.yml)
-
-
-## Best Practices
-
-*  Checking for an Ajax-Request: Use Director::is_ajax() instead of checking 
-for $_REQUEST['ajax'].
-
-
-## Links
-
-*  See `[api:ModelAsController]` class for details on controller/model-coupling
-*  See [execution-pipeline](/reference/execution-pipeline) for custom routing
-
-## API Documentation
-`[api:Director]`

docs/en/02_Developer_Guides/16_Execution_Pipeline/01_Flushable.md

@@ -6,7 +6,6 @@ summary: Allows a class to define it's own flush functionality.
 ## Introduction
 
 Allows a class to define it's own flush functionality, which is triggered when `flush=1` is requested in the URL.
-
 `[api:FlushRequestFilter]` is run before a request is made, calling `flush()` statically on all
 implementors of `[api:Flushable]`.
 

docs/en/02_Developer_Guides/16_Execution_Pipeline/02_Manifests.md

@@ -0,0 +1,83 @@
+title: Manifests
+summary: Manage caches of file path maps and other expensive information
+
+# Manifests
+
+## Purpose
+
+Manifests help to cache information which is too expensive to generate on each request.
+Some manifests generate maps, e.g. class names to filesystem locations.
+Others store aggregate information like nested configuration graphs.
+
+## Storage
+
+By default, manifests are stored on the local filesystem through PHP's `serialize()` method. 
+Combined with PHP opcode caching this provides fast access.
+In order to share manifests between servers, or centralise cache management,
+other storage adapters are available. These can be configured by a `SS_MANIFESTCACHE` constant,
+placed in your `_ss_environment.php`.
+
+ * `ManifestCache_File`: The default adapter using PHP's `serialize()`
+ * `ManifestCache_File_PHP`: Using `var_export()`, which is faster when a PHP opcode cache is installed
+ * `ManifestCache_APC`: Use PHP's [APC object cache](http://php.net/manual/en/book.apc.php)
+
+You can write your own adapters by implementing the `ManifestCache` interface.
+
+## Traversing the Filesystem
+
+Since manifests usually extract their information from files in the webroot,
+they require a powerful traversal tool: `[api:SS_FileFinder]`.
+The class provides filtering abilities for files and folders, as well as
+callbacks for recursive traversal. Each manifest has its own implementation,
+for example `[api:ManifestFileFinder]`, adding more domain specific filtering
+like including themes, or excluding testss.
+
+## PHP Class Manifest
+
+The `[api:ClassManifest]` builds a manifest of all classes, interfaces and some
+additional items present in a directory, and caches it.
+
+It finds the following information:
+
+ * Class and interface names and paths
+ * All direct and indirect descendants of a class
+ * All implementors of an interface
+ * All module configuration files
+
+The gathered information can be accessed through `[api:SS_ClassLoader::instance()]`,
+as well as `[api:ClassInfo]`. Some useful commands of the `ClassInfo` API:
+
+ * `ClassInfo::subclassesFor($class)`: Returns a list of classes that inherit from the given class
+ * `ClassInfo::ancestry($class)`: Returns the passed class name along with all its parent class names
+ * `ClassInfo::implementorsOf($interfaceName)`: Returns all classes implementing the passed in interface
+
+In the absence of a generic module API, it is also the primary way to identify
+which modules are installed, through `[api:ClassManifest::getModules()]`.
+A module is defined as a toplevel folder in the webroot which contains
+either a `_config.php` file, or a `_config/` folder. Modules can be specifically
+excluded from manifests by creating a blank `_manifest_exclude` file in the module folder.
+
+By default, the finder implementation will exclude any classes stored in files within
+a `tests/` folder, unless tests are executed.
+
+## Template Manifest
+
+The `[api:SS_TemplateManifest]` class builds a manifest of all templates present in a directory,
+in both modules and themes. Templates in `tests/` folders are automatically excluded.
+The chapter on [template inheritance](../templates/template-inheritance) provides more details
+on its operation.
+
+## Config Manifest
+
+The `[api:SS_ConfigManifest]` loads builds a manifest of configuration items,
+for both PHP and YAML. It also takes care of ordering and merging configuration fragments.
+The chapter on [configuration](/topics/configuration) has more details.
+
+## Flushing
+
+If a `?flush=1` query parameter is added to a URL, a call to `flush()` will be triggered
+on any classes that implement the [Flushable](/reference/flushable) interface.
+This enables developers to clear [manifest caches](manifests),
+for example when adding new templates or PHP classes.
+Note that you need to be in [dev mode](/getting_started/environment_management)
+or logged-in as an administrator for flushing to take effect.

docs/en/02_Developer_Guides/16_Execution_Pipeline/index.md

@@ -4,103 +4,136 @@ summary: An overview of the steps involved in delivering a SilverStripe web page
 
 ## Introduction
 
-This page documents all the steps from an URL request to the delivered page.
+In order to transform a HTTP request or a commandline exeuction into a response,
+SilverStripe needs to boot its core and run through several stages of processing.
 
-## .htaccess and RewriteRule
+## Request Rewriting
 
-Silverstripe uses **[mod_rewrite](http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html)** to deal with page requests.
+The first step in most environments is a rewrite of a request path into parameters passed to a PHP script.
-So instead of having your normal everyday `index.php` file which tells all, you need to look elsewhere.
+This allows writing friendly URLs instead of linking directly to PHP files.
+The implementation depends on your web server, we'll show you the most common one here: 
+Apache with [mod_rewrite](http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html).
+Check our [installation guides](/getting_started/installation) on how other web servers like IIS or nginx handle rewriting.
 
-The basic .htaccess file after installing SilverStripe looks like this:
+The standard SilverStripe project ships with a `.htaccess` file in your webroot for this purpose.
+By default, requests will be passed through for files existing on the filesystem.
+Some access control is in place to deny access to potentially sensitive files in the webroot, such as YAML configuration files.
+If no file can be directly matched, control is handed off to `framework/main.php`.
 
-	<file>
 	### SILVERSTRIPE START ###
 
+	# Deny access to templates (but allow from localhost)
 	<Files *.ss>
 		Order deny,allow
 		Deny from all
 		Allow from 127.0.0.1
 	</Files>
 
+	# Deny access to IIS configuration
+	<Files web.config>
+		Order deny,allow
+		Deny from all
+	</Files>
+
+	# Deny access to YAML configuration files which might include sensitive information
+	<Files ~ "\.ya?ml$">
+		Order allow,deny
+		Deny from all
+	</Files>
+
+	# Route errors to static pages automatically generated by SilverStripe
+	ErrorDocument 404 /assets/error-404.html
+	ErrorDocument 500 /assets/error-500.html
+
 	<IfModule mod_rewrite.c>
+		SetEnv HTTP_MOD_REWRITE On
 		RewriteEngine On
 
+		# Deny access to potentially sensitive files and folders
 		RewriteRule ^vendor(/|$) - [F,L,NC]
 		RewriteRule silverstripe-cache(/|$) - [F,L,NC]
 		RewriteRule composer\.(json|lock) - [F,L,NC]
 
+		# Process through SilverStripe if no file with the requested name exists.
+		# Pass through the original path as a query parameter, and retain the existing parameters.
 		RewriteCond %{REQUEST_URI} ^(.*)$
 		RewriteCond %{REQUEST_FILENAME} !-f
 		RewriteRule .* framework/main.php?url=%1 [QSA]
+
+		# If requesting the main script directly, rewrite to the installer
+		RewriteCond %{REQUEST_URI} ^(.*)/framework/main.php$
+		RewriteCond %{REQUEST_FILENAME} !-f
+		RewriteRule . %1/install.php? [R,L]
+
 	</IfModule>
 	### SILVERSTRIPE END ###
-	</file>
 
-The `<Files>` section denies direct access to the template files from anywhere but the server itself.
+SilverStripe can also operate without this level of rewriting, in which case all dynamic requests go
+through an `index.php` script in the webroot.
 
-The next section enables the rewriting engine and rewrites requests to `framework/main.php` if they meet the following
+<div class="notice" markdown="1">
-criteria:
+Running SilverStripe without web server based rewriting is not recommended since it
+can leave sensitive files exposed to public access (the `RewriteRule` conditions from above don't apply).
+</div>
 
-*  URI doesn't end in .gif, .jpg, .png, .css, or .js
+## Boostrap
-*  The requested file doesn't exist on the filesystem `framework/main.php` is called with the REQUEST_FILENAME (%1) as the `url` parameter and also appends the original
-QUERY_STRING.
 
-See the [mod_rewrite documentation](http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html) for more information on how
+All requests go through `framework/main.php`, which sets up the execution environment:
-mod_rewrite works.
 
+ * Tries to locate an `_ss_environment.php` 
+   [configuration file](/getting_started/environment_management) in the webroot, 
+   or the two levels above it (to allow sharing configuration between multiple webroots).
+ * Sets constants based on the filesystem structure (e.g. `BASE_URL`, `BASE_PATH` and `TEMP_FOLDER`)
+ * Normalizes the `url` parameter in preparation for handing it off to `Director`
+ * Connects to a database, based on information stored in the global `$databaseConfig` variable.
+   The configuration is either defined in your `_config.php`, or through `_ss_environment.php`
+ * Sets up [error handlers](../debugging/error_handling)
+ * Optionally continues a [session](../cookies_and_sessions/sessions) if the request already contains a session identifier
+ * Loads manifests for PHP classes, templates, as well as any [YAML configuration](../configuration).
+ * Optionally regenerates these manifests (if a ["flush" query parameter](flushable) is set)
+ * Executes all procedural configuration defined through `_config.php` in all discovered modules
+ * Loads the Composer PHP class autoloader
+ * Hands control over to `[api:Director]`
 
-## main.php
+While you usually don't need to modify the bootstrap on this level, some deeper customizations like
+adding your own manifests or a performance-optimized routing might require it.
+An example of this can be found in the ["staticpublisher" module](https://github.com/silverstripe-labs/silverstripe-staticpublisher/blob/master/main.php).
+The modules instructs web servers to route through its own `main.php` to determine which requests can be cached
+before handing control off to SilverStripe's own `main.php`.
 
-All requests go through `main.`php, which sets up the environment and then hands control over to `Director`.
+## Routing and Request Handling
 
-## Director and URL patterns
+The `main.php` script relies on `[api:Director]` to work out which [controller](../controllers) should handle this request. It parses the URL, matching it to one of a number of patterns, 
+and determines the controller, action and any argument to be used ([Routing](../controllers/routing)).
 
-main.php relies on `[api:Director]` to work out which controller should handle this request.  `[api:Director]` will instantiate that
+ * Creates a `[api:SS_HTTPRequest]` object containing all request and environment information
-controller object and then call `[api:Controller::run()]`.
+ * The [session](../cookies_and_sessions/sessions) holds an abstraction of PHP session
+ * Instantiates a [controller](../Controllers) object
+ * The `[api:Injector]` is first referenced, and asks the registered 
+   [RequestFilter](../controller/request_filters)
+   to pre-process the request object (see below)
+ * The `Controller` executes the actual business logic and populates an `[api:SS_HTTPResponse]`
+ * The `Controller` can optionally hand off control to further nested controllers
+ * The `Controller` optionally renders a response body through `SSViewer` [templates](../templates)
+ * The `[api:RequestProcessor]` is called to post-process the request to allow 
+further filtering before content is sent to the end user
+ * The response is output to the client
 
-In general, the URL is build up as follows: `page/action/ID/otherID` - e.g. http://localhost/mypage/addToCart/12.
+## Request Preprocessing and Postprocessing
-This will add an object with ID 12 to the cart.
 
-When you create a function, you can access the ID like this:
+The framework provides the ability to hook into the request both before and 
+after it is handled to allow binding custom logic. This can be used
+to transform or filter request data, instanciate helpers, execute global logic,
+or even short-circuit execution (e.g. to enforce custom authentication schemes).
+The ["Request Filters" documentation](../controller/request_filters) shows you how.
 
-	:::php
+## Flushing Manifests
-	public function addToCart ($request) {
-		$param = $request->allParams();
-		echo "my ID = " . $param["ID"];
-		$obj = MyProduct::get()->byID($param["ID"]);
-		$obj->addNow();
-	}
-
-## Controllers and actions
-
-`[api:Controller]`s are the building blocks of your application.
-
-**See:** The API documentation for `[api:Controller]`
-
-You can access the following controller-method with /team/signup
-
-	:::php
-	class Team extends DataObject {}
-
-	class Team_Controller extends Controller {
-
-		private static $allowed_actions = array('signup');
-
-		public function signup($id, $otherId) {
-			return $this->renderWith('MyTemplate');
-		}
-
-	}
-
-## SSViewer template rendering
-
-See [templates](/reference/templates) for information on the SSViewer template system.
-
-## Flush requests
-
-If `?flush=1` is requested in the URL, e.g. http://mysite.com?flush=1, this will trigger a call to `flush()` on
-any classes that implement the `Flushable` interface.
-
-See [reference documentation on Flushable](/reference/flushable) for more details.
 
+If a `?flush=1` query parameter is added to a URL, a call to `flush()` will be triggered
+on any classes that implement the [Flushable](/reference/flushable) interface.
+This enables developers to clear [manifest caches](manifests),
+for example when adding new templates or PHP classes.
+Note that you need to be in [dev mode](/getting_started/environment_management)
+or logged-in as an administrator for flushing to take effect.
 
+[CHILDREN]