NEW Legacy thumbnail migration task (#8924)

* NEW Legacy thumbnail migration task

See https://github.com/silverstripe/silverstripe-assets/issues/235
Makes a start at https://github.com/silverstripe/silverstripe-assets/issues/219 as well

* API Removed migrate_legacy_file support

For the vast majority of sites, you really don't want to run your file migration as part of dev build.
The step is involved enough to warrant it's own task.
I don't think this is an API change, since the setting won't have affect
for anyone who has already enabled it - they would've already done the one-off migration.

See https://github.com/silverstripeltd/open-sourcerers/issues/91
and https://github.com/silverstripe/silverstripe-assets/issues/235

docs/en/02_Developer_Guides/14_Files/04_File_Storage.md

@@ -68,7 +68,7 @@ assets/
 The URL for this file will match the physical location on disk:
 `http://www.example.com/assets/my-public-folder/my-public-file.jpg`.
 
-## Variant file paths (e.g. resized images)
+## Variant file paths (e.g. resized images) {#variant-file-paths}
 
 Each file can have variants, most commonly resized versions of an image.
 These can be generated by resizing an image in the CMS rich text editor,

docs/en/02_Developer_Guides/14_Files/05_File_Migration.md

@@ -7,7 +7,7 @@ This section describes how to upgrade existing filesystems from earlier versions
 
 ## Running migration
 
-Since the structure of `File` dataobjects has changed between 3.x and 4.x, a new task `MigrateFileTask`
+Since the structure of `File` objects has changed between 3.x and 4.x, a new task `MigrateFileTask`
 has been added to assist in migration of legacy files.
 
 You can run this task on the command line:
@@ -16,35 +16,31 @@ You can run this task on the command line:
 $ ./vendor/bin/sake dev/tasks/MigrateFileTask
 ```
 
-This task will also support migration of existing File objects to file versioning. Any
+This task will perform a number of subtasks:
-pre-existing File objects will be automatically published to the live stage, to ensure
-that previously visible assets remain visible to the public site.
 
-If additional security or visibility rules should be applied to File dataobjects, then
+ - `move-files`: Migrates existing `File` objects by adding required metadata to the database (incl. versioning).
-make sure to correctly extend `canView` via extensions.
+   By default, it will not move files on the filesystem (starting with [4.4.0](/changelogs/4.4.0)).
+   Publishes to the live stage to ensure
+   that previously visible assets remain visible to the public site.
+   If additional security or visibility rules should be applied to `File`, then
+   make sure to correctly extend `canView` via extensions.
+ - `move-thumbnails`: Move existing thumbnails, rather than have them generated on the fly.
+   This task is optional, but helps to avoid growing your asset folder (no duplicate thumbnails)
+ - `generate-cms-thumbnails`: The new CMS UI needs different thumbnail sizes, which can be pregenerated.
+   This can be a CPU and memory intensive task for large asset stores.
+   See [Migrating substantial number of files](#performance)
 
-## Automatic migration
+One or more subtasks can be run individually through the `only` argument.
-
+Example: `only=move-files,move-thumbnails`
-Migration can be invoked by either this task, or can be configured to automatically run during dev build
-by setting the `File.migrate_legacy_file` config to true. However, it's recommended that this task is
-run manually during an explicit migration process, as this process could potentially consume large
-amounts of memory and run for an extended time.
-
-```yml
-SilverStripe\Assets\File:
-  migrate_legacy_file: true
-```
 
 You can also run this task without CLI access through the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module.
 
-## Migration of thumbnails
+## Migration of existing thumbnails
 
-If you have the [asset admin](https://github.com/silverstripe/silverstripe-asset-admin) module installed
+Thumbnails generated through SilverStripe's image manipulation layer can be created by authors
-this will also ensure that thumbnails for these images are generated when running 'MigrateFileTask'.
+resizing images in the rich text editor, through template or PHP code, or by SilverStripe's built-in CMS logic.
-Existing thumbnails will not be migrated however, and must be re-generated for use in the CMS.
+They are now called "variants", and are placed in a different folder structure. In order to avoid re-generating those thumbnails,
-
+and cluttering up your asset store with orphaned files, the task will move them to the new location by default.
-Note: Thumbnails can be regenerated on a one-by-one basis within the CMS by re-saving it
-within the file edit details form.
 
 ## Discarded files during migration
 
@@ -75,9 +71,10 @@ SilverStripe\Assets\Flysystem\FlysystemAssetStore:
   keep_archived_assets: true
 ```
 
-## Migrating substantial number of files
+## Migrating substantial number of files {#performance}
 
 The time it takes to run the file migration will depend on the number of files and their size. The generation of thumbnails will depend on the number and dimension of your images.
+In general, the migration task can be restarted if it times out, and will continue where it left off. 
 
 If you are migrating a substantial number of files, you should run file migration task either as a queued job or on the command line. If the migration task fails or times out, you can start it again and it will pick up where it left off.
 

docs/en/04_Changelogs/4.4.0.md

@@ -2,12 +2,16 @@
 
 ## Overview {#overview}
 
+ - [Optional migration to hash-less public asset URLs](#hash-less)
+ - [Optional migration of legacy thumbnail locations](#legacy-thumb)
  - [Correct PHP types are now returned from database queries](/developer_guides/model/sql_select#data-types)
  - [Server Requirements](/getting_started/server_requirements/#web-server-software-requirements) have been refined:
    MySQL 5.5 end of life reached in December 2018, thus SilverStripe 4.4 requires MySQL 5.6+.  
    SilverStripe 4.3 and prior still support MySQL 5.5 for their own lifetime.
  - The name of the directory where vendor module resources are exposed can now be configured by defining a `extra.resources-dir` key in your `composer.json` file. If the key is not set, it will automatically default to `resources`. New projects will be preset to `_resources`. This will avoid potential conflict with SiteTree URL Segments.
  - dev/build is now non-destructive for all Enums, not just ClassNames. This means your data won't be lost if you're switching between versions, but watch out for code that breaks when it sees an unrecognised value!
+ - Removed `File.migrate_legacy_file` config option. Migration tasks now need to run via `dev/tasks/`,
+   running them as part of `dev/build` is no longer supported
 
 ## Upgrading {#upgrading}
 
@@ -32,10 +36,25 @@ This will avoid potential conflict with SiteTree URL Segments.
 
 You may also need to update your server configuration if you have applied special conditions to the `resources` path.
 
-### Optional migration to hash-less public asset URLs {#hashless-urls}
+### Optional migration tasks: Overview
 
 SilverStripe 4.x introduced an [asset abstraction](https://docs.silverstripe.org/en/4/developer_guides/files/file_storage/)
 system which required a [file migration](https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/) task.
+
+We have ironed out some edge cases since then:
+
+ * [Hash-less public asset URLs](#hash-less)
+ * [Legacy thumbnail locations](#legacy-thumb)
+ * TODO Shortcode migration
+
+ 
+You can opt-in to performing these migration tasks on your already upgraded SilverStripe 4.x project.
+Projects which are upgraded from 3.x to 4.4 or newer will run these tasks by default.
+You can perform these tasks in the background without CLI access by installing and configuring
+the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module.
+
+### Optional migration tasks: Hash-less public asset URLs {#hash-less}
+
 It allowed files to be access protected, have a separate draft stage,
 and track changes to file metadata and their contents through the `Versioned` system.
 
@@ -58,10 +77,30 @@ In order to opt-in to moving these files, run the following command:
 vendor/bin/sake dev/tasks/MigrateFileTask
 ``` 
 
-Note that you can run this task without CLI access by installing and configuring
+Further information is provided in the [Hash-less Public Asset URLs FAQ](#hashless-faq) below,
-the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module.
+as well as the [Github issue](https://github.com/silverstripe/silverstripe-versioned/issues/177).
 
-Further information is provided in the [Hash-less Public Asset URLs FAQ](#hashless-faq) below.
+### Optional migration tasks: Legacy thumbnail locations {#legacy-thumb}
+
+Alongside the new [asset abstraction](https://docs.silverstripe.org/en/4/developer_guides/files/file_storage/),
+we've also changed where thumbnails are [stored](/developer_guides/files/file_storage#variant-file-paths).
+The file format is now generalised as a "variant", and no longer lives in a `_resampled` folder.
+Prior to this release, these thumbnails were left in place. For thumbnails generated
+through PHP or template code, these are generated on demand in their new location,
+causing duplication and unnecessary storage space.
+
+You can opt-in to moving these legacy thumbnails to their new locations
+with a subtask of `dev/tasks/MigrateFileTask`. If newer thumbnails have been generated,
+it'll keep those. New migrations from 3.x to 4.x will include this migration step by default.
+
+```
+vendor/bin/sake dev/tasks/MigrateFileTask only=move-thumbnails
+```
+
+Note that as part of the [hash-less public asset URLs](#hash-less)
+introduced in this release, requests to these legacy thumbnails will automatically redirect to
+their new locations. Review the [Github issue](https://github.com/silverstripe/silverstripe-assets/issues/235)
+for more details. 
 
 ### Hash-less Public Asset URLs FAQ {#hashless-faq}
 

src/Dev/Tasks/MigrateFileTask.php

@@ -6,11 +6,12 @@ use Monolog\Handler\StreamHandler;
 use Monolog\Logger;
 use Psr\Log\LoggerInterface;
 use SilverStripe\AssetAdmin\Helper\ImageThumbnailHelper;
+use SilverStripe\Assets\Dev\Tasks\LegacyThumbnailMigrationHelper;
+use SilverStripe\Assets\FileMigrationHelper;
+use SilverStripe\Assets\Storage\AssetStore;
 use SilverStripe\Control\Director;
 use SilverStripe\Core\Injector\Injector;
 use SilverStripe\Logging\PreformattedEchoHandler;
-use SilverStripe\ORM\DB;
-use SilverStripe\Assets\FileMigrationHelper;
 use SilverStripe\Dev\BuildTask;
 
 /**
@@ -20,38 +21,118 @@ class MigrateFileTask extends BuildTask
 {
     private static $segment = 'MigrateFileTask';
 
-    protected $title = 'Migrate File dataobjects from 3.x';
+    protected $title = 'Migrate File dataobjects from 3.x and successive iterations in 4.x';
 
-    protected $description =
+    protected $defaultSubtasks = [
-        'Imports all files referenced by File dataobjects into the new Asset Persistence Layer introduced in 4.0. ' .
+        'move-files',
-        'If the task fails or times out, run it again and it will start where it left off.';
+        'move-thumbnails',
+        'generate-cms-thumbnails'
+    ];
+
+    private static $dependencies = [
+        'logger' => '%$' . LoggerInterface::class,
+    ];
+
+    /** @var Logger */
+    private $logger;
 
     public function run($request)
     {
         $this->addLogHandlers();
 
-        if (!class_exists(FileMigrationHelper::class)) {
+        $args = $request->getVars();
-            DB::alteration_message("No file migration helper detected", "notice");
+        $this->validateArgs($args);
-            return;
+
+        $subtasks = !empty($args['only']) ? explode(',', $args['only']) : $this->defaultSubtasks;
+
+        if (in_array('move-files', $subtasks)) {
+            if (!class_exists(FileMigrationHelper::class)) {
+                $this->logger->error("No file migration helper detected");
+                return;
+            }
+
+            $this->logger->info('### Migrating filesystem and database records (move-files)');
+
+            $this->logger->info('If the task fails or times out, run it again and it will start where it left off.');
+
+            $migrated = FileMigrationHelper::singleton()->run();
+            if ($migrated) {
+                $this->logger->info("{$migrated} File DataObjects upgraded");
+            } else {
+                $this->logger->info("No File DataObjects need upgrading");
+            }
         }
 
-        DB::alteration_message(
+        if (in_array('move-thumbnails', $subtasks)) {
-            'If the task fails or times out, run it again and it will start where it left off.',
+            if (!class_exists(LegacyThumbnailMigrationHelper::class)) {
-            "info"
+                $this->logger->error("LegacyThumbnailMigrationHelper not found");
-        );
+                return;
+            }
 
-        $migrated = FileMigrationHelper::singleton()->run();
+            $this->logger->info('### Migrating existing thumbnails (move-thumbnails)');
-        if ($migrated) {
+
-            DB::alteration_message("{$migrated} File DataObjects upgraded", "changed");
+            $moved = LegacyThumbnailMigrationHelper::singleton()
-        } else {
+                ->setLogger($this->logger)
-            DB::alteration_message("No File DataObjects need upgrading", "notice");
+                ->run($this->getStore());
+
+            if ($moved) {
+                $this->logger->info(sprintf("%d thumbnails moved", count($moved)));
+            } else {
+                $this->logger->info("No thumbnails moved");
+            }
         }
 
-        if (!class_exists(ImageThumbnailHelper::class)) {
+        if (in_array('generate-cms-thumbnails', $subtasks)) {
-            DB::alteration_message("No image thumbnail helper detected", "notice");
+            if (!class_exists(ImageThumbnailHelper::class)) {
-            return;
+                $this->logger->error("ImageThumbnailHelper not found");
+                return;
+            }
+
+            $this->logger->info('### Generating new CMS UI thumbnails (generate-cms-thumbnails)');
+
+            ImageThumbnailHelper::singleton()->run();
+        }
+    }
+
+    public function getDescription()
+    {
+        return <<<TXT
+Imports all files referenced by File dataobjects into the new Asset Persistence Layer introduced in 4.0.
+Moves existing thumbnails, and generates new thumbnail sizes for the CMS UI.
+If the task fails or times out, run it again and it will start where it left off.
+See https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/.
+TXT;
+    }
+
+    /**
+     * @param LoggerInterface $logger
+     */
+    public function setLogger(LoggerInterface $logger)
+    {
+        $this->logger = $logger;
+
+        return $this;
+    }
+
+    /**
+     * @return AssetStore
+     */
+    protected function getStore()
+    {
+        return singleton(AssetStore::class);
+    }
+
+    /**
+     * @param array $args
+     * @throws \InvalidArgumentException
+     */
+    protected function validateArgs($args)
+    {
+        if (!empty($args['only'])) {
+            if (array_diff(explode(',', $args['only']), $this->defaultSubtasks)) {
+                throw new \InvalidArgumentException('Invalid subtasks detected: ' . $args['only']);
+            }
         }
-        ImageThumbnailHelper::singleton()->run();
     }
 
     /**