tonyair/silverstripe-framework
1
0
Fork 0
mirror of https://github.com/silverstripe/silverstripe-framework synced 2024-10-22 14:05:37 +02:00
Code Issues Projects Releases Wiki Activity

NEW Legacy thumbnail migration task (#8924)

Browse Source 
* NEW Legacy thumbnail migration task

See https://github.com/silverstripe/silverstripe-assets/issues/235
Makes a start at https://github.com/silverstripe/silverstripe-assets/issues/219 as well

* API Removed migrate_legacy_file support

For the vast majority of sites, you really don't want to run your file migration as part of dev build.
The step is involved enough to warrant it's own task.
I don't think this is an API change, since the setting won't have affect
for anyone who has already enabled it - they would've already done the one-off migration.

See https://github.com/silverstripeltd/open-sourcerers/issues/91
and https://github.com/silverstripe/silverstripe-assets/issues/235
This commit is contained in:
Ingo Schommer 2019-05-02 09:33:54 +12:00 committed by Maxime Rainville
parent 6b8f0eae30
commit 0696045e59
4 changed files with 169 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/en/02_Developer_Guides/14_Files/04_File_Storage.md
View File

@ -68,7 +68,7 @@ assets/
The URL for this file will match the physical location on disk:
`http://www.example.com/assets/my-public-folder/my-public-file.jpg`.
## Variant file paths (e.g. resized images)
## Variant file paths (e.g. resized images) {#variant-file-paths}
Each file can have variants, most commonly resized versions of an image.
These can be generated by resizing an image in the CMS rich text editor,

47
docs/en/02_Developer_Guides/14_Files/05_File_Migration.md
View File

@ -7,7 +7,7 @@ This section describes how to upgrade existing filesystems from earlier versions
## Running migration
Since the structure of `File` dataobjects has changed between 3.x and 4.x, a new task `MigrateFileTask`
Since the structure of `File` objects has changed between 3.x and 4.x, a new task `MigrateFileTask`
has been added to assist in migration of legacy files.
You can run this task on the command line:
@ -16,35 +16,31 @@ You can run this task on the command line:
$ ./vendor/bin/sake dev/tasks/MigrateFileTask
```
This task will also support migration of existing File objects to file versioning. Any
pre-existing File objects will be automatically published to the live stage, to ensure
that previously visible assets remain visible to the public site.
This task will perform a number of subtasks:
If additional security or visibility rules should be applied to File dataobjects, then
make sure to correctly extend `canView` via extensions.
- `move-files`: Migrates existing `File` objects by adding required metadata to the database (incl. versioning).
By default, it will not move files on the filesystem (starting with [4.4.0](/changelogs/4.4.0)).
Publishes to the live stage to ensure
that previously visible assets remain visible to the public site.
If additional security or visibility rules should be applied to `File`, then
make sure to correctly extend `canView` via extensions.
- `move-thumbnails`: Move existing thumbnails, rather than have them generated on the fly.
This task is optional, but helps to avoid growing your asset folder (no duplicate thumbnails)
- `generate-cms-thumbnails`: The new CMS UI needs different thumbnail sizes, which can be pregenerated.
This can be a CPU and memory intensive task for large asset stores.
See [Migrating substantial number of files](#performance)
## Automatic migration
Migration can be invoked by either this task, or can be configured to automatically run during dev build
by setting the `File.migrate_legacy_file` config to true. However, it's recommended that this task is
run manually during an explicit migration process, as this process could potentially consume large
amounts of memory and run for an extended time.
```yml
SilverStripe\Assets\File:
migrate_legacy_file: true
```
One or more subtasks can be run individually through the `only` argument.
Example: `only=move-files,move-thumbnails`
You can also run this task without CLI access through the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module.
## Migration of thumbnails
## Migration of existing thumbnails
If you have the [asset admin](https://github.com/silverstripe/silverstripe-asset-admin) module installed
this will also ensure that thumbnails for these images are generated when running 'MigrateFileTask'.
Existing thumbnails will not be migrated however, and must be re-generated for use in the CMS.
Note: Thumbnails can be regenerated on a one-by-one basis within the CMS by re-saving it
within the file edit details form.
Thumbnails generated through SilverStripe's image manipulation layer can be created by authors
resizing images in the rich text editor, through template or PHP code, or by SilverStripe's built-in CMS logic.
They are now called "variants", and are placed in a different folder structure. In order to avoid re-generating those thumbnails,
and cluttering up your asset store with orphaned files, the task will move them to the new location by default.
## Discarded files during migration
@ -75,9 +71,10 @@ SilverStripe\Assets\Flysystem\FlysystemAssetStore:
keep_archived_assets: true
```
## Migrating substantial number of files
## Migrating substantial number of files {#performance}
The time it takes to run the file migration will depend on the number of files and their size. The generation of thumbnails will depend on the number and dimension of your images.
In general, the migration task can be restarted if it times out, and will continue where it left off.
If you are migrating a substantial number of files, you should run file migration task either as a queued job or on the command line. If the migration task fails or times out, you can start it again and it will pick up where it left off.

47
docs/en/04_Changelogs/4.4.0.md
View File

@ -2,12 +2,16 @@
## Overview {#overview}
- [Optional migration to hash-less public asset URLs](#hash-less)
- [Optional migration of legacy thumbnail locations](#legacy-thumb)
- [Correct PHP types are now returned from database queries](/developer_guides/model/sql_select#data-types)
- [Server Requirements](/getting_started/server_requirements/#web-server-software-requirements) have been refined:
MySQL 5.5 end of life reached in December 2018, thus SilverStripe 4.4 requires MySQL 5.6+.
SilverStripe 4.3 and prior still support MySQL 5.5 for their own lifetime.
- The name of the directory where vendor module resources are exposed can now be configured by defining a `extra.resources-dir` key in your `composer.json` file. If the key is not set, it will automatically default to `resources`. New projects will be preset to `_resources`. This will avoid potential conflict with SiteTree URL Segments.
- dev/build is now non-destructive for all Enums, not just ClassNames. This means your data won't be lost if you're switching between versions, but watch out for code that breaks when it sees an unrecognised value!
- Removed `File.migrate_legacy_file` config option. Migration tasks now need to run via `dev/tasks/`,
running them as part of `dev/build` is no longer supported
## Upgrading {#upgrading}
@ -32,10 +36,25 @@ This will avoid potential conflict with SiteTree URL Segments.
You may also need to update your server configuration if you have applied special conditions to the `resources` path.
### Optional migration to hash-less public asset URLs {#hashless-urls}
### Optional migration tasks: Overview
SilverStripe 4.x introduced an [asset abstraction](https://docs.silverstripe.org/en/4/developer_guides/files/file_storage/)
system which required a [file migration](https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/) task.
We have ironed out some edge cases since then:
* [Hash-less public asset URLs](#hash-less)
* [Legacy thumbnail locations](#legacy-thumb)
* TODO Shortcode migration
You can opt-in to performing these migration tasks on your already upgraded SilverStripe 4.x project.
Projects which are upgraded from 3.x to 4.4 or newer will run these tasks by default.
You can perform these tasks in the background without CLI access by installing and configuring
the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module.
### Optional migration tasks: Hash-less public asset URLs {#hash-less}
It allowed files to be access protected, have a separate draft stage,
and track changes to file metadata and their contents through the `Versioned` system.
@ -58,10 +77,30 @@ In order to opt-in to moving these files, run the following command:
vendor/bin/sake dev/tasks/MigrateFileTask
```
Note that you can run this task without CLI access by installing and configuring
the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module.
Further information is provided in the [Hash-less Public Asset URLs FAQ](#hashless-faq) below,
as well as the [Github issue](https://github.com/silverstripe/silverstripe-versioned/issues/177).
Further information is provided in the [Hash-less Public Asset URLs FAQ](#hashless-faq) below.
### Optional migration tasks: Legacy thumbnail locations {#legacy-thumb}
Alongside the new [asset abstraction](https://docs.silverstripe.org/en/4/developer_guides/files/file_storage/),
we've also changed where thumbnails are [stored](/developer_guides/files/file_storage#variant-file-paths).
The file format is now generalised as a "variant", and no longer lives in a `_resampled` folder.
Prior to this release, these thumbnails were left in place. For thumbnails generated
through PHP or template code, these are generated on demand in their new location,
causing duplication and unnecessary storage space.
You can opt-in to moving these legacy thumbnails to their new locations
with a subtask of `dev/tasks/MigrateFileTask`. If newer thumbnails have been generated,
it'll keep those. New migrations from 3.x to 4.x will include this migration step by default.
```
vendor/bin/sake dev/tasks/MigrateFileTask only=move-thumbnails
```
Note that as part of the [hash-less public asset URLs](#hash-less)
introduced in this release, requests to these legacy thumbnails will automatically redirect to
their new locations. Review the [Github issue](https://github.com/silverstripe/silverstripe-assets/issues/235)
for more details.
### Hash-less Public Asset URLs FAQ {#hashless-faq}

125
src/Dev/Tasks/MigrateFileTask.php
View File

@ -6,11 +6,12 @@ use Monolog\Handler\StreamHandler;
use Monolog\Logger;
use Psr\Log\LoggerInterface;
use SilverStripe\AssetAdmin\Helper\ImageThumbnailHelper;
use SilverStripe\Assets\Dev\Tasks\LegacyThumbnailMigrationHelper;
use SilverStripe\Assets\FileMigrationHelper;
use SilverStripe\Assets\Storage\AssetStore;
use SilverStripe\Control\Director;
use SilverStripe\Core\Injector\Injector;
use SilverStripe\Logging\PreformattedEchoHandler;
use SilverStripe\ORM\DB;
use SilverStripe\Assets\FileMigrationHelper;
use SilverStripe\Dev\BuildTask;
/**
@ -20,38 +21,118 @@ class MigrateFileTask extends BuildTask
{
private static $segment = 'MigrateFileTask';
protected $title = 'Migrate File dataobjects from 3.x';
protected $title = 'Migrate File dataobjects from 3.x and successive iterations in 4.x';
protected $description =
'Imports all files referenced by File dataobjects into the new Asset Persistence Layer introduced in 4.0. ' .
'If the task fails or times out, run it again and it will start where it left off.';
protected $defaultSubtasks = [
'move-files',
'move-thumbnails',
'generate-cms-thumbnails'
];
private static $dependencies = [
'logger' => '%$' . LoggerInterface::class,
];
/** @var Logger */
private $logger;
public function run($request)
{
$this->addLogHandlers();
if (!class_exists(FileMigrationHelper::class)) {
DB::alteration_message("No file migration helper detected", "notice");
return;
$args = $request->getVars();
$this->validateArgs($args);
$subtasks = !empty($args['only']) ? explode(',', $args['only']) : $this->defaultSubtasks;
if (in_array('move-files', $subtasks)) {
if (!class_exists(FileMigrationHelper::class)) {
$this->logger->error("No file migration helper detected");
return;
}
$this->logger->info('### Migrating filesystem and database records (move-files)');
$this->logger->info('If the task fails or times out, run it again and it will start where it left off.');
$migrated = FileMigrationHelper::singleton()->run();
if ($migrated) {
$this->logger->info("{$migrated} File DataObjects upgraded");
} else {
$this->logger->info("No File DataObjects need upgrading");
}
}
DB::alteration_message(
'If the task fails or times out, run it again and it will start where it left off.',
"info"
);
if (in_array('move-thumbnails', $subtasks)) {
if (!class_exists(LegacyThumbnailMigrationHelper::class)) {
$this->logger->error("LegacyThumbnailMigrationHelper not found");
return;
}
$migrated = FileMigrationHelper::singleton()->run();
if ($migrated) {
DB::alteration_message("{$migrated} File DataObjects upgraded", "changed");
} else {
DB::alteration_message("No File DataObjects need upgrading", "notice");
$this->logger->info('### Migrating existing thumbnails (move-thumbnails)');
$moved = LegacyThumbnailMigrationHelper::singleton()
->setLogger($this->logger)
->run($this->getStore());
if ($moved) {
$this->logger->info(sprintf("%d thumbnails moved", count($moved)));
} else {
$this->logger->info("No thumbnails moved");
}
}
if (!class_exists(ImageThumbnailHelper::class)) {
DB::alteration_message("No image thumbnail helper detected", "notice");
return;
if (in_array('generate-cms-thumbnails', $subtasks)) {
if (!class_exists(ImageThumbnailHelper::class)) {
$this->logger->error("ImageThumbnailHelper not found");
return;
}
$this->logger->info('### Generating new CMS UI thumbnails (generate-cms-thumbnails)');
ImageThumbnailHelper::singleton()->run();
}
}
public function getDescription()
{
return <<<TXT
Imports all files referenced by File dataobjects into the new Asset Persistence Layer introduced in 4.0.
Moves existing thumbnails, and generates new thumbnail sizes for the CMS UI.
If the task fails or times out, run it again and it will start where it left off.
See https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/.
TXT;
}
/**
* @param LoggerInterface $logger
*/
public function setLogger(LoggerInterface $logger)
{
$this->logger = $logger;
return $this;
}
/**
* @return AssetStore
*/
protected function getStore()
{
return singleton(AssetStore::class);
}
/**
* @param array $args
* @throws \InvalidArgumentException
*/
protected function validateArgs($args)
{
if (!empty($args['only'])) {
if (array_diff(explode(',', $args['only']), $this->defaultSubtasks)) {
throw new \InvalidArgumentException('Invalid subtasks detected: ' . $args['only']);
}
}
ImageThumbnailHelper::singleton()->run();
}
/**