Merge pull request #9026 from open-sausages/pulls/4.0/file-migration-docs

Improve file migration information to documentation

docs/en/02_Developer_Guides/04_Configuration/00_Configuration.md

@@ -51,8 +51,6 @@ This can be done by calling the static method [Config::inst()](api:SilverStripe\
 
 
 ```php
-use SilverStripe\Core\Config\Config;
-
 $config = Config::inst()->get('MyClass', 'property');
 ```
 

docs/en/02_Developer_Guides/14_Files/04_File_Storage.md

@@ -136,7 +136,7 @@ assets/
             my-file.jpg
 ```
 
-## Versioned and archived files
+## Versioned and archived files {#archived}
 
 By default, when files are replaced or removed, their original file contents
 aren't retained in order to avoid bloat on the filesystem.

docs/en/02_Developer_Guides/14_Files/05_File_Migration.md

@@ -37,7 +37,26 @@ This task will perform a number of subtasks:
 One or more subtasks can be run individually through the `only` argument.
 Example: `only=move-files,move-thumbnails`
 
+The output is quite verbose by default. Look for `WARNING` and `ERROR` in the log files.
+When executing the task on CLI, you'll get colour coded error messages.
+
+## Background migration through the Queuedjobs module
+
 You can also run this task without CLI access through the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module.
+Open up `admin/queuedjobs`, then create a job of type `RunBuildTaskJob`.
+The only constructor parameter allowed is the full name of the task: `SilverStripe\Dev\Tasks\MigrateFileTask`.
+The task output will be progressively written to the job record, and can be inspected via the "Messages" tab within the job in the CMS.
+It attempts to continue running to "complete" status even if it encounters errors, so you'll need to review the logs
+to ensure if everything went smoothly. Note that it's currently not possible to run specific subtasks via a queuedjob.
+
+While you can run the job directly through the CMS, it'll usually be more constrained by PHP `max_execution_time` settings.
+Many platforms such as the New Zealand Government Common Web Platform or SilverStripe Platform
+are configured to run jobs automatically without time limits
+([1](https://docs.platform.silverstripe.com/development/platform-yml-file/#cron-tasks),
+[2](https://www.cwp.govt.nz/developer-docs/en/2/working_with_projects/infrastructural_considerations/)).
+It is not recommended to run
+[multiple processes](https://github.com/symbiote/silverstripe-queuedjobs/blob/master/docs/en/configuring-multi-process-execution.md)
+when executing the file migration job. 
 
 ## Migration of existing thumbnails
 
@@ -75,14 +94,14 @@ SilverStripe\Assets\Flysystem\FlysystemAssetStore:
   keep_archived_assets: true
 ```
 
-## Migrating substantial number of files {#performance}
+## Migrating substantial numbers of files {#performance}
 
 The time it takes to run the file migration will depend on the number of files and their size. The generation of thumbnails will depend on the number and dimension of your images.
 In general, the migration task can be restarted if it times out, and will continue where it left off. 
 
-If you are migrating a substantial number of files, you should run file migration task either as a queued job or on the command line. If the migration task fails or times out, you can start it again and it will pick up where it left off.
+If you are migrating a substantial number of files, you should run the file migration task either as a queued job or on the command line. If the migration task fails or times out, you can start it again and it will pick up where it left off.
 
-If your environement supports the _Imagick_ PHP library, you may want to use that library instead of _GD_. Imagick is considerably faster when resizing images. You can switch back to _GD_ after running the file migration task.
+If your environment supports the _Imagick_ PHP library, you may want to use that library instead of _GD_. Imagick is considerably faster when resizing images. You can switch back to _GD_ after running the file migration task.
 
 [Changing the image manipulation driver to Imagick](images#changing-the-manipulation-driver-to-imagick)
 
@@ -114,6 +133,121 @@ Use the following estimates to decide how you will run your file migration:
 
 Your exact experience will vary based on your host server, the size of your files and other conditions. If your site is hosted on a managed environement (e.g.: [Common Web Platform](https://www.cwp.govt.nz/service-desk) or [SilverStripe Platform](https://docs.platform.silverstripe.com/support/)), you may not have access to the command line to manually run the migration task. Contact your hosting provider's helpdesk if that's your case.
 
+
+## Natural path vs Hash Path
+
+Two types of paths are used by SilverStripe CMS when serving files:
+* Natural paths and
+* Hash paths.
+
+Natural paths will match the path displayed in SilverStripe's asset management section (e.g.: `assets/Folder/form.pdf`). Hash paths will include a partial hash as directory name (e.g.: `assets/Folder/282318025b/form.pdf`). The hash is generated by computing the sha1 hash of the provided file and keeping the first ten characters.
+
+The hash path has the advantage of being unique for every version of the file your users might upload. This allows multiple versions of the same file to co-exist, enabling features like [file archives](04_File_Storage#archived). This is why hash path is the default file format for the protected file store.
+
+The Natural path has the advantage of being easier to understand and permanent. When a newer version of a file is published to a natural path, it will be accessible at the same URL as the previous version. This is why natural path is the default file format for the public file store.
+
+[Learn more about SilverStripe File Storage](04_File_Storage)
+
+## Migration from SilverStripe 3 to Silverstripe 4.4 or later
+
+While the SilverStripe 3 file paths may look similar to the natural paths in SilverStripe 4.4, the database entries for each file have to be normalised to conform to the new SilverStripe 4 structure. File variants (e.g. image thumbnails) are stored differently in SilverStripe 4. They will automatically be moved to their new location.
+
+SilverStripe 3 did not support draft files which means that all migrated SilverStripe 3 files will automatically be published by the File Migration Task.
+
+File names can not contain "double underscores" in SilverStripe 4 (`__`) because those characters are used to indicate a "file variant". SilverStripe 3 files not conforming to this rule will automatically be renamed by the file migration task to remove extra underscores.
+
+### Will old URLs to SilverStripe 3 files still resolve?
+
+In most cases, the SilverStripe 3 file path will be identical to the public path of the file in SilverStripe 4. Links to SilverStripe 3 file variants (e.g. cropped images) will automatically be redirected to their new location.
+
+### What about file references in `HTMLText` database fields?
+
+The shortcode format in SilverStripe 4 has changed from SilverStripe 3. This means that when CMS editors try to edit existing content and alter image or file links, the reference to the file will not work.
+
+After running the file migration, you can run the short code migration task to update your existing HTML content in your database:
+
+```bash
+sake dev/tasks/TagsToShortcode
+``` 
+
+This will rewrite your existing shortcodes to the newer format SilverStripe 4 expects as well as convert `img` and `a` tags to use shortcodes..
+
+## Migrating from SilverStripe 4.3 or below to SilverStripe 4.4 or above
+
+Silverstripe 4.0 to 4.3 were using hash paths for the public file store as well as the protected file store. This meant that every time a new version of a file was published, it would be served under a different URL.
+
+SilverStripe 4.4 and later default to using natural paths for public store files. After upgrading to SilverStripe 4.4 or above, you can run the file migration task to normalise your existing files.
+
+This will take all existing files under a hash path and move them to their natural path.
+
+### Do I need to re-migrate my files?
+
+No, you do not need to run the file migration task when upgrading to SilverStripe 4.4 or later. 
+
+Your existing files will still resolve. Old hash paths will be redirected to the latest version of the file. As newer versions of existing files are published, they will be moved to their natural path.
+
+You can choose to run the migration task at any point.
+
+### What if I’m using `legacy_filenames`?
+
+SilverStripe 4.0 to 4.3 allowed you to enable a `legacy_filenames` mode. This would be equivalent to using natural paths for both the public store and the protected store.
+
+With the introduction of natural paths for the public store, the primary reason for `legacy_filenames` mode is no longer relevant. `legacy_filenames` mode is now considered deprecated, however leaving it enabled will not have any impact on your project – it will simply be ignored.
+
+After upgrading to SilverStripe 4.4 or later, you should run the file migration task to normalise your protected files. This will move your protected files to their equivalent hash path.
+
+###  What if I want to keep using hash paths for my public files?
+
+If you want to keep using hash paths for your public store, you can add the following entries to your YML configuration files.
+
+```yml
+SilverStripe\Core\Injector\Injector:
+  # Define public resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.public:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+    DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+    ResolutionFileIDHelpers:
+      - '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      VersionedStage: Live
+  # Define protected resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.protected:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+      DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      ResolutionFileIDHelpers:
+        - '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      VersionedStage: Stage
+```
+
+## Disabling old file resolution format
+
+When starting a brand new project, or after you’ve ran the file migration task, all your files will be in their default locations. This gives you the option to disable alternative resolution formats. This will slightly speed up file resolution.
+
+You can do this by adding this snippet to your YML configuration.
+
+```yml
+SilverStripe\Core\Injector\Injector:
+  # Define public resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.public:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+      ResolutionFileIDHelpers:
+      - '%$SilverStripe\Assets\FilenameParsing\NaturalFileIDHelper'
+      DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\NaturalFileIDHelper'
+      VersionedStage: Live
+    # Define protected resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.protected:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+      DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      ResolutionFileIDHelpers:
+       - '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      VersionedStage: Stage
+```
+
+If you're starting a brand new project using the `silverstripe/installer` 4.4 or above, this code snippet will already be part of your project.
+
 ## Customise the File Migration Task (Advanced)
 
 In some context, you may want to disable some other process when the file migration is running. For example, if you have a module that indexes files when they get modified, you'll probably want to wait until the file migration is done to reindex.

docs/en/04_Changelogs/4.4.0.md

@@ -102,7 +102,7 @@ introduced in this release, requests to these legacy thumbnails will automatical
 their new locations. Review the [Github issue](https://github.com/silverstripe/silverstripe-assets/issues/235)
 for more details. 
 
-### Optional migration tasks: Remove artefacts from "secureassets" module {#secureassets}
+### Optional migration tasks: Remove artifacts from "secureassets" module {#secureassets}
 
 The [silverstripe/secureassets](https://github.com/silverstripe/silverstripe-secureassets)
 module allowed opt-in file security on SilverStripe 3.x sites.
@@ -154,7 +154,7 @@ in your webserver (e.g. through `.htaccess`).
 The redirect code can be configured via `FlysystemAssetStore.permanent_redirect_response_code`.
 
 If you upgrade an older SilverStripe 4 project to SilverStripe 4.4 and choose not to run the file migration task, 
-your files will still be stored under an "hash" folder. Browsers who try to access this file via the "hashless" url will
+your files will still be stored under a "hash" folder. Browsers who try to access this file via the "hashless" url will
 be redirected to the "hash" URL with a `302 Temporary Redirect`.
 
 The temporary redirect code can be configured via `FlysystemAssetStore.redirect_response_code`.
@@ -171,12 +171,12 @@ We have implemented an automatic redirect for URLs pointing to asset
 locations prior to running the optional migration script,
 so there is no need to regenerate any content.
  
-#### Is there any data loss or data integrity issue?
+#### Are there any data loss or data integrity issues?
 
 There are no known issues around data integrity.
 All files that were available on a SS 3.x site are still available after upgrading to SS 4.x.
 
-#### Will this change affect my search engine ranking?
+#### Will this affect my search engine ranking?
 
 As long as your files are still linked on your website, 
 search engines will pick up the new links on any projects which have already been migrated.
@@ -194,7 +194,7 @@ the path of all existing assets.
 Technically yes, but you need to be aware of the tradeoffs. Once the regression has been fixed, we don’t see the need 
 for people to choose this option any more.
 
-SilverStripe 4.4, ignores the `legacy_filenames` in most situation. Enabling `legacy_filenames` on a fresh SilverStripe 
+SilverStripe 4.4, ignores `legacy_filenames` in most situations. Enabling `legacy_filenames` on a fresh SilverStripe 
 4.4 installation will have no affect.
 
 #### How do I test that the fix has worked on my site?
@@ -215,6 +215,77 @@ conditions.
 Yes, it will attempt to find the most recent public "hash-less" URL
 for this file and redirect to it.
 
+### Changing the file resolver
+
+The file resolution strategy describes how files are resolved. It relies on a list of file helpers and will iterate through them when looking up files.
+
+Out of the box, the public file resolver handles Natural File ID's, Hash File ID's and Legacy File ID's. But after you've ran the file migration task, all your files will be in the correct spot, so you may want to change your config to only resolve Natural File ID's:
+
+```yml
+  # Define public resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.public:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+      ResolutionFileIDHelpers:
+        - '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+        - '%$SilverStripe\Assets\FilenameParsing\NaturalFileIDHelper'
+        - '%$SilverStripe\Assets\FilenameParsing\LegacyFileIDHelper'
+      DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\NaturalFileIDHelper'
+      VersionedStage: Live
+  # Define protected resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.protected:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+      DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      ResolutionFileIDHelpers:
+        - '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+        - '%$SilverStripe\Assets\FilenameParsing\NaturalFileIDHelper'
+      VersionedStage: Stage
+```
+
+You can also configure it to only resolve Hash File ID's after the migration:
+
+```yml
+SilverStripe\Core\Injector\Injector:
+  # Define public resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.public:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+      ResolutionFileIDHelpers:
+        - '%$SilverStripe\Assets\FilenameParsing\NaturalFileIDHelper'
+      DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\NaturalFileIDHelper'
+      VersionedStage: Live
+  # Define protected resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.protected:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+      DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      ResolutionFileIDHelpers:
+        - '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      VersionedStage: Stage
+```
+
+If someone wanted to keep hash file id's for their public files, they could do something like this:
+
+```yml
+SilverStripe\Core\Injector\Injector:
+  # Define public resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.public:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+    DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+    ResolutionFileIDHelpers:
+      - '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      VersionedStage: Live
+  # Define protected resolution strategy
+  SilverStripe\Assets\FilenameParsing\FileResolutionStrategy.protected:
+    class: SilverStripe\Assets\FilenameParsing\FileIDHelperResolutionStrategy
+    properties:
+      DefaultFileIDHelper: '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      ResolutionFileIDHelpers:
+        - '%$SilverStripe\Assets\FilenameParsing\HashFileIDHelper'
+      VersionedStage: Stage
+```
 
 ## Changes to internal APIs