tonyair/silverstripe-framework
1
0
Fork 0
mirror of https://github.com/silverstripe/silverstripe-framework synced 2024-10-22 14:05:37 +02:00
Code Issues Projects Releases Wiki Activity

DOC Explain how to use the new file migation sub tasks in the 4.4.6/4.5.2 changelogs

Browse Source
This commit is contained in:
Maxime Rainville 2020-03-28 15:51:06 +13:00
parent a9598eec3f
commit 4b0345bee7
3 changed files with 945 additions and 18 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

61
docs/en/02_Developer_Guides/14_Files/05_File_Migration.md
View File

@ -18,28 +18,59 @@ $ ./vendor/bin/sake dev/tasks/MigrateFileTask
This task will perform a number of subtasks: This task will perform a number of subtasks:
- `move-files`: Migrates existing `File` objects by adding required metadata to the database (incl. versioning). - `move-files`: Migrates existing `File` objects by adding required metadata to the database (incl. versioning).
By default, it will not move files on the filesystem (starting with [4.4.0](/changelogs/4.4.0)). By default, it will not move files on the filesystem (starting with [4.4.0](/changelogs/4.4.0)).
Publishes to the live stage to ensure Publishes to the live stage to ensure
that previously visible assets remain visible to the public site. that previously visible assets remain visible to the public site.
If additional security or visibility rules should be applied to `File`, then If additional security or visibility rules should be applied to `File`, then
make sure to correctly extend `canView` via extensions. make sure to correctly extend `canView` via extensions.
- `move-thumbnails`: Move existing thumbnails, rather than have them generated on the fly. - `migrate-folders`: Migrates existing `Folder` objects by adding metadata to the
This task is optional, but helps to avoid growing your asset folder (no duplicate thumbnails) database (incl. versioning). This subtask does not perform any operations with the actual
- `generate-cms-thumbnails`: The new CMS UI needs different thumbnail sizes, which can be pregenerated. file system. It merely updates the database records.
This can be a CPU and memory intensive task for large asset stores. - `move-thumbnails`: Move existing thumbnails, rather than have them generated on the fly.
See [Migrating substantial number of files](#performance) This task is optional, but helps to avoid growing your asset folder (no duplicate thumbnails)
- `fix-secureassets`: Migrates files secured through the [silverstripe/secureassets](https://github.com/silverstripe/silverstripe-secureassets) module. - `generate-cms-thumbnails`: The new CMS UI needs different thumbnail sizes, which can be pregenerated.
Ensures that previous `.htaccess` folder protections don't interfere with 4.x-style asset protections. This can be a CPU and memory intensive task for large asset stores.
- `fix-folder-permissions`: Fixes folder permissions which might have been broken by See [Migrating substantial number of files](#performance)
previously using the [silverstripe/secureassets](https://github.com/silverstripe/silverstripe-secureassets) - `fix-secureassets`: Migrates files secured through the [silverstripe/secureassets](https://github.com/silverstripe/silverstripe-secureassets) module.
Ensures that previous `.htaccess` folder protections don't interfere with 4.x-style asset protections.
- `fix-folder-permissions`: Fixes folder permissions which might have been broken by
previously using the [silverstripe/secureassets](https://github.com/silverstripe/silverstripe-secureassets)
One or more subtasks can be run individually through the `only` argument. One or more subtasks can be run individually through the `only` argument.
Example: `only=move-files,move-thumbnails` Example: `only=move-files,move-thumbnails`
The `move-files` and `migrate-folders` subtasks are mandatory. Without running them,
your database will be left in an inconsistent state.
The output is quite verbose by default. Look for `WARNING` and `ERROR` in the log files. The output is quite verbose by default. Look for `WARNING` and `ERROR` in the log files.
When executing the task on CLI, you'll get colour coded error messages. When executing the task on CLI, you'll get colour coded error messages.
### Specialised opt-in file migration subtasks
Some subtasks are only necessary in specific situations. Those subtasks will not be run by default and must be explicitly referenced via the `only` parameter.
- `normalise-access`: This subtask identifies physical files that are
inadvertently exposed to the public and moves them to a protected location.
Until the release of Silverstripe CMS 4.3.5/4.4.4 in September 2019,
Silverstripe CMS stored protected files in publicly accessible locations
when the files were created from a "LIVE" versioned context. This commonly
occur when a user uploaded a file in a user form.
[CVE-2019-12245 patch](https://www.silverstripe.org/download/security-releases/CVE-2019-12245)
addressed this issue preventing the exposure of new files. However, this patch
did not retroactively protect files that had been exposed. You should run this
task at least once if your Silverstripe CMS project was created prior to
September 2019.
- `relocate-userform-uploads-2020-9280`: This subtask is specific to the
`silverstripe/userforms` module. Because of the
[CVE-2020-9280](https://www.silverstripe.org/download/security-releases/CVE-2020-9280)
vulnerability, if your project was migrated from Silverstripe CMS 3, the
migrated user forms might upload files in wrong folders. This subtask goes
through all files uploaded via user forms and moves them to their intended
locations.
Read the [Silverstripe CMS 4.4.6 change logs](/changelogs/4.4.6) to learn more
about the `normalise-access` and `relocate-userform-uploads-2020-9280` subtasks.
## Background migration through the Queuedjobs module ## Background migration through the Queuedjobs module
You can also run this task without CLI access through the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module. You can also run this task without CLI access through the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module.

448
docs/en/04_Changelogs/4.4.6.md Normal file
View File

@ -0,0 +1,448 @@
# 4.4.6
## Security patches
This release is a security hotfix. It only contains security fixes essential to
addressing the CVE-2020-9280 security issue and follow up work from the
CVE-2019-12245 patch which was released in September 2019.
* [Read the CVE-2020-9280 security disclosure](https://www.silverstripe.org/download/security-releases/CVE-2020-9280)
from April 2020
* [Read the CVE-2019-12245 security disclosure](https://www.silverstripe.org/download/security-releases/CVE-2019-12245)
from September 2019
**Simply applying the patch might NOT be enough.** Some files that were uploaded
or created prior to the application of the CVE-2020-9280/CVE-2019-12245 patches
may be left exposed to the public. You may need to run some additional file
migration tasks to protect those files.
This release is especially important for Silverstripe CMS projects using the
`silverstripe/userforms` module, but all project owners should consider
upgrading as soon as convenient.
If your project was migrated from Silverstripe CMS 3, you need to run the new
`migrate-folders` task.
## What does this release fix?
### CVE-2020-9280: Folders migrated from Silverstripe CMS 3 may be unsafe to upload to (CVSS 5.9)
Files uploaded via Forms to folders migrated from Silverstripe CMS 3.x may be
put to the default `/Uploads` folder instead. Uploads performed via the CMS UI
are not affected. This is a security issue because the default `/Uploads` folder
is publicly accessible by default, which means the uploaded files may be
accessed by unauthorised parties via HTTP by guessing the file name. This
affects installations which allowed upload folder protection via the optional
`silverstripe/secureassets` module under 3.x. This module is installed and
enabled by default on the Common Web Platform (CWP). The vulnerability only
affects files uploaded after an upgrade to 4.x.
[Read the CVE-2020-9280 security disclosure](https://www.silverstripe.org/download/security-releases/CVE-2020-9280)
### CVE-2019-12245 follow up work
In September 2019, Silverstripe CMS 4.3.6 and 4.4.4 were released to address an
issue that caused files to be stored in a publicly accessible location. Files
that were uploaded via the userforms module could have been affected. Files
uploaded via a custom form or programmatically created could also have been
affected.
While those older releases prevented the exposure of new files, they did not
retroactively protect files that had already been exposed prior to the patch.
These new releases provide a mechanism to protect files wrongly exposed to the
public.
* [Read the CVE-2019-12245 security disclosure](https://www.silverstripe.org/download/security-releases/CVE-2019-12245)
* [Read the Silverstripe CMS 4.3.6 change logs](4.3.6)
* [Read the Silverstripe CMS 4.4.4 change logs](4.4.4)
## What are the technical details of the vulnerabilities?
### CVE-2020-9280
The root cause of the problem is that the file migration task would not create
"Live" records for Folders migrated from Silverstripe CMS 3 prior to this patch.
This may lead to inconsistencies between the database state and what the actual
application gets from the ORM when performing its business logic.
This affects the `silverstripe/userforms` module and can cause user forms to
save uploaded files in the default "Uploads" folder. Custom forms built with
`Silverstripe\Forms` or `Silverstripe\Assets` components can also be
susceptible to this behaviour.
Considering user forms as an example. Let's say you have an EditableFileField,
that keeps an ID of a folder to upload to as an `has_one` relationship. When the
submission happens with a `Stage=Live` context, the ORM cannot fetch the
`Folder` instance, since it doesn't exist in the `Live` stage. When the upload
Folder is undefined, the default `Silverstripe\FileField` implementation falls
back to the default "Uploads" Folder, which is public by default.
### CVE-2019-12245
Prior the 4.3.6/4.4.4 release, the `File::canView()` method could
return an erroneous value for anonymous users when the file was being created or
uploaded from a "Live" stage. When creating a new File in a Live stage, this
could cause the file to be stored in a publicly accessible location when it was
meant to be protected.
This could occur when:
* a file was submitted in a user form upload field
* a file was submitted in a custom form
* a file was programmatically created by custom logic.
It would not occur when uploading a file via the asset administration area in
the CMS.
### How do I know if the files are actually exposed?
Even if your project creates files programmatically or receives them via a
custom form, your files might not have been exposed. Whether your files have
been exposed depends on your exact implementation.
There's two ways to verify if files are exposed:
* look directly at the file system
* try to access files in an incognito/private browser session.
If you have direct access to the file system or to a snapshot of your live data,
you can look directly at the files. To do that, access your `assets` directory
which will either be located in your project root or under the `public`
directory. The assets directory will contain an hidden `.protected` directory
which contains restricted and draft files. Other files which are not stored
under the `.protected` directory will be visible to anonymous users. If you can
see files that are meant to be confidential outside of the `.protected` folder,
then they are accessible by the general public.
Some files may be stored under a _hash path_. This means that the physical files
will be stored in a folder with a 10 character hexadecimal name. e.g.: If your
File is stored under `secret/folder/condfidential.pdf` in the CMS, the physical
file might be stored under a path similar to
`public/assets/secret/folder/c0f7cbd745/condfidential.pdf`.
[Learn what's the difference between _natural paths_ and _hash paths_](https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/#natural-path-vs-hash-path)
on the Silverstripe CMS documentation.
Alternatively, you can try accessing the file directly in a incognito/private
browser session. To do so, navigate to a restricted file in the asset
administration interface inside the CMS. On the right-hand panel,
right-click on the file preview and copy the link to the file. If you paste the
link into a private/incognito browser session and can access the file, then your
file can be viewed by the general public.
## Running the migration tasks
The 4.4.6/4.5.2 Silverstripe CMS releases add new file migration subtasks to
retroactively protect files that have been wrongly exposed. They are 3 new
subtasks.
* `migrate-folders` creates versioning metadata for Silverstripe CMS 3 Folder
records.
* `normalise-access` protect files that have been wrongly exposed.
* `relocate-userform-uploads-2020-9280` move files uploaded through the
userforms module to their intended folder.
### Do I need to run the migration tasks?
Based on the nature of your Silverstripe CMS project, you may need to run all of
the tasks, some of the them, or none of them.
You do not need to run any of the subtasks if the following three conditions are
all true:
* your project was not migrated from Silverstripe CMS 3
* your project does not include the userforms module, or if it does, no userform
has been configured to allow file uploads
* your project does not programmatically create files or allow users to upload
files via a custom form(s).
If your project was upgraded from a Silverstripe CMS 3 project, you should
minimally run the `migrate-folders` subtask.
If your project programmatically creates files or allows users to upload files
via a user form or a custom form, you should run the `normalise-access` task.
If your project allows users to upload files via a user form and was upgraded from
a Silverstripe CMS 3 project, you may need to run all 3 tasks.
If your project was migrated from Silverstripe CMS 3 and allows users to upload
files via custom forms, you should run the `migrate-folders` subtask. However,
you also will need to manually protect the affected files. The easiest way
could be to find all the folders affected through a SQL query before you run the
task. Then you may be able to find all the custom forms using those folder IDs
accordingly. Unfortunately, we cannot provide any automation for this scenario.
You should also consider making your `/Uploads` folder protected manually
(via CMS) as a best practice, since it is publicly accessible by default.
### What do the migration tasks do?
Take some time to read the following descriptions and understand how running
each subtask will affect your Silverstripe CMS project. `migrate-folders` and
`normalise-access` are relatively quick and have no side affects.
`relocate-userform-uploads-2020-9280` is the most complex of the three subtasks
and has the most potential side effects. You should take special care to
understand what it does before running it.
#### migrate-folders subtask
The `migrate-folders` subtask validates that every "Folder" record in the
`File` table has a matching entry in the `File_Live` table. This will prevent
future file uploads from being accidentally stored in the publicly accessible
"Uploads" folder.
It does not retroactively move files that have been stored in the wrong Folder
to their intended destination.
To find out what Folders on your Silverstripe CMS project will be affected by
running the `migrate-folders` subtask, you can run the following SQL query:
```sql
-- This query is targeted to MySQL. You may need to adapt it for other SQL databases.
SELECT
File.ID,
File.Name
FROM
File
LEFT JOIN
File_Live
ON File_Live.ID = File.ID
WHERE
File_Live.ID IS NULL
AND File.ClassName = 'SilverStripe\\Assets\\Folder'
```
Note that the file migration task will now run this step by default, so you
won't need to explicitly run the `migrate-folders` task for future
Silverstripe CMS 3 upgrades.
[Learn how to migrate files from Silverstripe CMS 3 to Silverstripe CMS 4](https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/#migration-from-silverstripe-3-to-silverstripe-4-4-or-later)
on the Silverstripe CMS documentation.
#### normalise-access subtask
The `normalise-access` subtask goes through each individual File record in a
Silverstripe CMS project and confirms that the `canView` result for anonymous
users matches the physical location of the file. If the physical file is
publicly visible while it's `canView` method returns `false` for anonymous
users, it will be moved to the protected location.
#### relocate-userform-uploads-2020-9280 task
Because of the `CVE-2020-9280` vulnerability, you may have Folders without a
matching entries in their "Live" table. User form file upload submissions
may end up being stored in an incorrect publicly-accessible location under the
following conditions:
* you have user forms that accept file uploads
* the file upload field is configured to save the file in a folder that was
created in Silverstripe CMS 3
* the targeted folder has not been manually saved since your Silverstripe CMS 4
upgrade.
If those 3 conditions are met, you should consider running the
`relocate-userform-uploads-2020-9280` subtask.
`relocate-userform-uploads-2020-9280` will go through each user form file upload
submission and verify that the uploaded file is stored in the folder specified
in the matching file upload user form field. If the file is stored in the wrong
folder, the task will move the file to the Folder specified in the user form
upload field configuration at the time the file was uploaded.
##### What if I have a custom upload form?
If you have a written a controller operating in the "Live" stage that creates
files or receives uploaded files, and that controller is configured to save
files in a folder created in Siverstripe CMS 3, your files may have been stored
in the wrong location. Unfortunately, we can not provide a migration task for
each custom scenario.
The `relocate-userform-uploads-2020-9280` only works because there's a record of
where the file was meant to be uploaded on the user form page.
If you have a small number of files, it might be practical to manually update
their location via the asset administration interface in the CMS. Otherwise, you
may have to write your own migration task to achieve the same purpose.
### How do I run the migration tasks?
`migrate-folders`, `normalise-access` and `relocate-userform-uploads-2020-9280`
have been written as subtasks of the regular file migration task. The file
migration task can be run in 3 different ways:
* in the browser under `dev/tasks`
* on the command line
* as a queued job.
You can run each subtask individually or you can run all three subtasks in one
go. The `only` parameter controls which subtasks will be run. You can get
multiple subtask to run by specifying them as a comma-separated list:
`only=migrate-folders,normalise-access,relocate-userform-uploads-2020-9280`. The subtasks
are executed in a pre-defined sequence, so the order they appeared in the
comma-separated list is irrelevant.
Which ever way you choose to run the sub tasks, you **MUST run `migrate-folders`
before `relocate-userform-uploads-2020-9280`**. The recommended order to run the subtasks
is:
1. `migrate-folders`
2. `normalise-access`
3. `relocate-userform-uploads-2020-9280`.
[Learn more about the File Migration Task on the Silverstripe CMS documentation](https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/)
#### Testing the file migration task first
If possible, you should consider running the task in a development or testing
environment first, either on a snapshot of your production environment or on a
subset of your live data. Make sure you have back ups configured on your
production environment and a recovery strategy in place in case things go badly.
#### Running the task in the browser
Running the file migration task in the browser should only be considered if you
have a small number of files. It is susceptible to timeout if the task runs for
too long. Depending on your application settings, you may not have the output
logs of the tasks run. Please consider the other options first.
To run the task in the browser, navigate to `/dev/tasks/MigrateFileTask` and
provide the `only` parameter as a GET parameter on your request. e.g.:
`https://example.com/dev/tasks/MigrateFileTask?only=migrate-folders`
If you're running this task on a production environment, you will be asked to
login using administrator credentials and you will be prompted to confirm you
want to execute the task.
#### Running the task on the command line
Navigate to your project root on the command line and use the sake utility to
execute the `MigrateFileTask` command.
```bash
./vendor/bin/sake dev/tasks/MigrateFileTask only=migrate-folders
```
[Learn how to execute Silverstripe CMS tasks on the command line](https://docs.silverstripe.org/en/4/developer_guides/cli/)
#### Running the task as a queued job
If you've installed the
[queued-jobs module](https://github.com/symbiote/silverstripe-queuedjobs) on
your Silverstripe CMS project, you can run the file migration task as a queued
job. There are two alternative ways to schedule a task to run as a queued job:
* by navigating to the task URL in your browser (e.g.:
`https://example.com/dev/tasks/queue/MigrateFileTask?only=migrate-folders`)
* via the command line (e.g.
`vendor/bin/sake dev/tasks/queue/MigrateFileTask only=migrate-folders`).
Once the task has been scheduled, you can monitor its progress in the queued job
administration area inside the CMS.
### How long should I expect the migration tasks to run for?
#### migrate-folders
The main factor in the execution time of the `migrate-folders` subtask is the
number of Folders to migrate.
This subtask does not perform any filesystem operations. It mostly performs read
and write SQL queries.
You can expect 20 to 30 Folders to be migrated per second. A project with
10,000 Folders to migrate should take less than 10 minutes.
#### normalise-access
The main factor in the execution time of the `normalise-access` subtask are:
* the number of files to protect
* the number of files in the same folder.
Protecting 1000 folders with 1 file each will be considerably faster than 1
folder with 1000 files. The total number of files in your project will also
impact the subtask runtime, but to a lesser extent.
This substask performs a lot of file system operations with some read SQL
queries. It does not write data to the database.
You can expect 10 to 20 files to be protected per seconds. Performance is
degraded if all those files are in a small number of folders.
* A project with 10,000 files to protected spread through a lot of folders
should take less than an hour.
* A project with 10,000 files to protected all in one folder can take several
hours.
#### relocate-userform-uploads-2020-9280
The main factor in the execution time of the `normalise-access` subtask are:
* the number of files to protect
* the number of files in the same folder.
Moving 1000 folders with 1 file each will be considerably faster than 1 folder
with 1000 files.
This substask performs a mix of file system operations and read/write SQL
queries.
The approximate speed of the task is:
- 500 files per minute with 1k files in the same folder (9 files per second)
- 60 files per minute with 10k files in the same folder (1 file per second).
## Edges cases and potential "gotchas"
### I don't have "/Uploads" folder in CMS
Either the default "Uploads" folder was changed via
[Silverstripe\Assets\Upload::uploads_folder](https://api.silverstripe.org/4/SilverStripe/Assets/Upload.html)
configuration parameter, or you don't have any files uploaded to it.
The issues covered by this patch may still affect your application.
### I have two "/Uploads" folders in CMS
If you see two /Uploads folders in CMS, most likely that means one of them was
migrated from Silverstripe CMS 3, whereas some application logic created the
second copy after the migration.
In that case your database would contain a record in the `File` table for each
of them, while `File_Live` table would only have a record about the other one.
After running the tasks provided in this patch, we suggest you manually move all
content from one Folder to the other via the CMS. Then remove the empty folder.
### What if I updated my files after upload?
Beware that draft files are present in 2 locations on the filesystem.
* The original live version of the file is still available.
* The draft version of the file should be present in a `.protected` folder that
is inaccessible for unauthenticated users.
This means that even if you moved a file but don't publish it, the live version
of the file will still be available.
To address this, "relocate-userform-uploads-2020-9280" task only works on the
very first version of a file if it's published. Subsequent versions of the file
are ignored by the task. For example, even if you didn't change the location of
a file, but updated the filename or its permissions, the script will ignore that
file.
If you have an unpublished draft, but the live version of the file is 1 (no
published changes after upload), the script will try to move the live version of
the file into its originally intended location, while preserving the current
draft version of the file as-is. This would increment numbers for both Live and
Draft versions of the file, while moving the live version to another location
(both on the file system and in the CMS).
### Files with draft access restriction
If you have a published file that is publicly available and you update the draft
version to restrict read access, the live file currently remains accessible
until you publish the draft version. However, the `canView` method on the live
file will immediately start returning `false` for anonymous users.
This will cause `normalise-access` to protect the live file. Beware that if
you have unpublished access restrictions on draft files, those will take effect
on the live file after running `normalise-access`.
Read the ["Files with access restriction in Draft" GitHub issue](https://github.com/silverstripe/silverstripe-assets/issues/385)
for more details.
<!--- Changes below this line will be automatically regenerated -->
<!--- Changes above this line will be automatically regenerated -->

448
docs/en/04_Changelogs/4.5.2.md Normal file
View File

@ -0,0 +1,448 @@
# 4.5.2
## Security patches
This release is a security hotfix. It only contains security fixes essential to
addressing the CVE-2020-9280 security issue and follow up work from the
CVE-2019-12245 patch which was released in September 2019.
* [Read the CVE-2020-9280 security disclosure](https://www.silverstripe.org/download/security-releases/CVE-2020-9280)
from April 2020
* [Read the CVE-2019-12245 security disclosure](https://www.silverstripe.org/download/security-releases/CVE-2019-12245)
from September 2019
**Simply applying the patch might NOT be enough.** Some files that were uploaded
or created prior to the application of the CVE-2020-9280/CVE-2019-12245 patches
may be left exposed to the public. You may need to run some additional file
migration tasks to protect those files.
This release is especially important for Silverstripe CMS projects using the
`silverstripe/userforms` module, but all project owners should consider
upgrading as soon as convenient.
If your project was migrated from Silverstripe CMS 3, you need to run the new
`migrate-folders` task.
## What does this release fix?
### CVE-2020-9280: Folders migrated from Silverstripe CMS 3 may be unsafe to upload to (CVSS 5.9)
Files uploaded via Forms to folders migrated from Silverstripe CMS 3.x may be
put to the default `/Uploads` folder instead. Uploads performed via the CMS UI
are not affected. This is a security issue because the default `/Uploads` folder
is publicly accessible by default, which means the uploaded files may be
accessed by unauthorised parties via HTTP by guessing the file name. This
affects installations which allowed upload folder protection via the optional
`silverstripe/secureassets` module under 3.x. This module is installed and
enabled by default on the Common Web Platform (CWP). The vulnerability only
affects files uploaded after an upgrade to 4.x.
[Read the CVE-2020-9280 security disclosure](https://www.silverstripe.org/download/security-releases/CVE-2020-9280)
### CVE-2019-12245 follow up work
In September 2019, Silverstripe CMS 4.3.6 and 4.4.4 were released to address an
issue that caused files to be stored in a publicly accessible location. Files
that were uploaded via the userforms module could have been affected. Files
uploaded via a custom form or programmatically created could also have been
affected.
While those older releases prevented the exposure of new files, they did not
retroactively protect files that had already been exposed prior to the patch.
These new releases provide a mechanism to protect files wrongly exposed to the
public.
* [Read the CVE-2019-12245 security disclosure](https://www.silverstripe.org/download/security-releases/CVE-2019-12245)
* [Read the Silverstripe CMS 4.3.6 change logs](4.3.6)
* [Read the Silverstripe CMS 4.4.4 change logs](4.4.4)
## What are the technical details of the vulnerabilities?
### CVE-2020-9280
The root cause of the problem is that the file migration task would not create
"Live" records for Folders migrated from Silverstripe CMS 3 prior to this patch.
This may lead to inconsistencies between the database state and what the actual
application gets from the ORM when performing its business logic.
This affects the `silverstripe/userforms` module and can cause user forms to
save uploaded files in the default "Uploads" folder. Custom forms built with
`Silverstripe\Forms` or `Silverstripe\Assets` components can also be
susceptible to this behaviour.
Considering user forms as an example. Let's say you have an EditableFileField,
that keeps an ID of a folder to upload to as an `has_one` relationship. When the
submission happens with a `Stage=Live` context, the ORM cannot fetch the
`Folder` instance, since it doesn't exist in the `Live` stage. When the upload
Folder is undefined, the default `Silverstripe\FileField` implementation falls
back to the default "Uploads" Folder, which is public by default.
### CVE-2019-12245
Prior the 4.3.6/4.4.4 release, the `File::canView()` method could
return an erroneous value for anonymous users when the file was being created or
uploaded from a "Live" stage. When creating a new File in a Live stage, this
could cause the file to be stored in a publicly accessible location when it was
meant to be protected.
This could occur when:
* a file was submitted in a user form upload field
* a file was submitted in a custom form
* a file was programmatically created by custom logic.
It would not occur when uploading a file via the asset administration area in
the CMS.
### How do I know if the files are actually exposed?
Even if your project creates files programmatically or receives them via a
custom form, your files might not have been exposed. Whether your files have
been exposed depends on your exact implementation.
There's two ways to verify if files are exposed:
* look directly at the file system
* try to access files in an incognito/private browser session.
If you have direct access to the file system or to a snapshot of your live data,
you can look directly at the files. To do that, access your `assets` directory
which will either be located in your project root or under the `public`
directory. The assets directory will contain an hidden `.protected` directory
which contains restricted and draft files. Other files which are not stored
under the `.protected` directory will be visible to anonymous users. If you can
see files that are meant to be confidential outside of the `.protected` folder,
then they are accessible by the general public.
Some files may be stored under a _hash path_. This means that the physical files
will be stored in a folder with a 10 character hexadecimal name. e.g.: If your
File is stored under `secret/folder/condfidential.pdf` in the CMS, the physical
file might be stored under a path similar to
`public/assets/secret/folder/c0f7cbd745/condfidential.pdf`.
[Learn what's the difference between _natural paths_ and _hash paths_](https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/#natural-path-vs-hash-path)
on the Silverstripe CMS documentation.
Alternatively, you can try accessing the file directly in a incognito/private
browser session. To do so, navigate to a restricted file in the asset
administration interface inside the CMS. On the right-hand panel,
right-click on the file preview and copy the link to the file. If you paste the
link into a private/incognito browser session and can access the file, then your
file can be viewed by the general public.
## Running the migration tasks
The 4.4.6/4.5.2 Silverstripe CMS releases add new file migration subtasks to
retroactively protect files that have been wrongly exposed. They are 3 new
subtasks.
* `migrate-folders` creates versioning metadata for Silverstripe CMS 3 Folder
records.
* `normalise-access` protect files that have been wrongly exposed.
* `relocate-userform-uploads-2020-9280` move files uploaded through the
userforms module to their intended folder.
### Do I need to run the migration tasks?
Based on the nature of your Silverstripe CMS project, you may need to run all of
the tasks, some of the them, or none of them.
You do not need to run any of the subtasks if the following three conditions are
all true:
* your project was not migrated from Silverstripe CMS 3
* your project does not include the userforms module, or if it does, no userform
has been configured to allow file uploads
* your project does not programmatically create files or allow users to upload
files via a custom form(s).
If your project was upgraded from a Silverstripe CMS 3 project, you should
minimally run the `migrate-folders` subtask.
If your project programmatically creates files or allows users to upload files
via a user form or a custom form, you should run the `normalise-access` task.
If your project allows users to upload files via a user form and was upgraded from
a Silverstripe CMS 3 project, you may need to run all 3 tasks.
If your project was migrated from Silverstripe CMS 3 and allows users to upload
files via custom forms, you should run the `migrate-folders` subtask. However,
you also will need to manually protect the affected files. The easiest way
could be to find all the folders affected through a SQL query before you run the
task. Then you may be able to find all the custom forms using those folder IDs
accordingly. Unfortunately, we cannot provide any automation for this scenario.
You should also consider making your `/Uploads` folder protected manually
(via CMS) as a best practice, since it is publicly accessible by default.
### What do the migration tasks do?
Take some time to read the following descriptions and understand how running
each subtask will affect your Silverstripe CMS project. `migrate-folders` and
`normalise-access` are relatively quick and have no side affects.
`relocate-userform-uploads-2020-9280` is the most complex of the three subtasks
and has the most potential side effects. You should take special care to
understand what it does before running it.
#### migrate-folders subtask
The `migrate-folders` subtask validates that every "Folder" record in the
`File` table has a matching entry in the `File_Live` table. This will prevent
future file uploads from being accidentally stored in the publicly accessible
"Uploads" folder.
It does not retroactively move files that have been stored in the wrong Folder
to their intended destination.
To find out what Folders on your Silverstripe CMS project will be affected by
running the `migrate-folders` subtask, you can run the following SQL query:
```sql
-- This query is targeted to MySQL. You may need to adapt it for other SQL databases.
SELECT
File.ID,
File.Name
FROM
File
LEFT JOIN
File_Live
ON File_Live.ID = File.ID
WHERE
File_Live.ID IS NULL
AND File.ClassName = 'SilverStripe\\Assets\\Folder'
```
Note that the file migration task will now run this step by default, so you
won't need to explicitly run the `migrate-folders` task for future
Silverstripe CMS 3 upgrades.
[Learn how to migrate files from Silverstripe CMS 3 to Silverstripe CMS 4](https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/#migration-from-silverstripe-3-to-silverstripe-4-4-or-later)
on the Silverstripe CMS documentation.
#### normalise-access subtask
The `normalise-access` subtask goes through each individual File record in a
Silverstripe CMS project and confirms that the `canView` result for anonymous
users matches the physical location of the file. If the physical file is
publicly visible while it's `canView` method returns `false` for anonymous
users, it will be moved to the protected location.
#### relocate-userform-uploads-2020-9280 task
Because of the `CVE-2020-9280` vulnerability, you may have Folders without a
matching entries in their "Live" table. User form file upload submissions
may end up being stored in an incorrect publicly-accessible location under the
following conditions:
* you have user forms that accept file uploads
* the file upload field is configured to save the file in a folder that was
created in Silverstripe CMS 3
* the targeted folder has not been manually saved since your Silverstripe CMS 4
upgrade.
If those 3 conditions are met, you should consider running the
`relocate-userform-uploads-2020-9280` subtask.
`relocate-userform-uploads-2020-9280` will go through each user form file upload
submission and verify that the uploaded file is stored in the folder specified
in the matching file upload user form field. If the file is stored in the wrong
folder, the task will move the file to the Folder specified in the user form
upload field configuration at the time the file was uploaded.
##### What if I have a custom upload form?
If you have a written a controller operating in the "Live" stage that creates
files or receives uploaded files, and that controller is configured to save
files in a folder created in Siverstripe CMS 3, your files may have been stored
in the wrong location. Unfortunately, we can not provide a migration task for
each custom scenario.
The `relocate-userform-uploads-2020-9280` only works because there's a record of
where the file was meant to be uploaded on the user form page.
If you have a small number of files, it might be practical to manually update
their location via the asset administration interface in the CMS. Otherwise, you
may have to write your own migration task to achieve the same purpose.
### How do I run the migration tasks?
`migrate-folders`, `normalise-access` and `relocate-userform-uploads-2020-9280`
have been written as subtasks of the regular file migration task. The file
migration task can be run in 3 different ways:
* in the browser under `dev/tasks`
* on the command line
* as a queued job.
You can run each subtask individually or you can run all three subtasks in one
go. The `only` parameter controls which subtasks will be run. You can get
multiple subtask to run by specifying them as a comma-separated list:
`only=migrate-folders,normalise-access,relocate-userform-uploads-2020-9280`. The subtasks
are executed in a pre-defined sequence, so the order they appeared in the
comma-separated list is irrelevant.
Which ever way you choose to run the sub tasks, you **MUST run `migrate-folders`
before `relocate-userform-uploads-2020-9280`**. The recommended order to run the subtasks
is:
1. `migrate-folders`
2. `normalise-access`
3. `relocate-userform-uploads-2020-9280`.
[Learn more about the File Migration Task on the Silverstripe CMS documentation](https://docs.silverstripe.org/en/4/developer_guides/files/file_migration/)
#### Testing the file migration task first
If possible, you should consider running the task in a development or testing
environment first, either on a snapshot of your production environment or on a
subset of your live data. Make sure you have back ups configured on your
production environment and a recovery strategy in place in case things go badly.
#### Running the task in the browser
Running the file migration task in the browser should only be considered if you
have a small number of files. It is susceptible to timeout if the task runs for
too long. Depending on your application settings, you may not have the output
logs of the tasks run. Please consider the other options first.
To run the task in the browser, navigate to `/dev/tasks/MigrateFileTask` and
provide the `only` parameter as a GET parameter on your request. e.g.:
`https://example.com/dev/tasks/MigrateFileTask?only=migrate-folders`
If you're running this task on a production environment, you will be asked to
login using administrator credentials and you will be prompted to confirm you
want to execute the task.
#### Running the task on the command line
Navigate to your project root on the command line and use the sake utility to
execute the `MigrateFileTask` command.
```bash
./vendor/bin/sake dev/tasks/MigrateFileTask only=migrate-folders
```
[Learn how to execute Silverstripe CMS tasks on the command line](https://docs.silverstripe.org/en/4/developer_guides/cli/)
#### Running the task as a queued job
If you've installed the
[queued-jobs module](https://github.com/symbiote/silverstripe-queuedjobs) on
your Silverstripe CMS project, you can run the file migration task as a queued
job. There are two alternative ways to schedule a task to run as a queued job:
* by navigating to the task URL in your browser (e.g.:
`https://example.com/dev/tasks/queue/MigrateFileTask?only=migrate-folders`)
* via the command line (e.g.
`vendor/bin/sake dev/tasks/queue/MigrateFileTask only=migrate-folders`).
Once the task has been scheduled, you can monitor its progress in the queued job
administration area inside the CMS.
### How long should I expect the migration tasks to run for?
#### migrate-folders
The main factor in the execution time of the `migrate-folders` subtask is the
number of Folders to migrate.
This subtask does not perform any filesystem operations. It mostly performs read
and write SQL queries.
You can expect 20 to 30 Folders to be migrated per second. A project with
10,000 Folders to migrate should take less than 10 minutes.
#### normalise-access
The main factor in the execution time of the `normalise-access` subtask are:
* the number of files to protect
* the number of files in the same folder.
Protecting 1000 folders with 1 file each will be considerably faster than 1
folder with 1000 files. The total number of files in your project will also
impact the subtask runtime, but to a lesser extent.
This substask performs a lot of file system operations with some read SQL
queries. It does not write data to the database.
You can expect 10 to 20 files to be protected per seconds. Performance is
degraded if all those files are in a small number of folders.
* A project with 10,000 files to protected spread through a lot of folders
should take less than an hour.
* A project with 10,000 files to protected all in one folder can take several
hours.
#### relocate-userform-uploads-2020-9280
The main factor in the execution time of the `normalise-access` subtask are:
* the number of files to protect
* the number of files in the same folder.
Moving 1000 folders with 1 file each will be considerably faster than 1 folder
with 1000 files.
This substask performs a mix of file system operations and read/write SQL
queries.
The approximate speed of the task is:
- 500 files per minute with 1k files in the same folder (9 files per second)
- 60 files per minute with 10k files in the same folder (1 file per second).
## Edges cases and potential "gotchas"
### I don't have "/Uploads" folder in CMS
Either the default "Uploads" folder was changed via
[Silverstripe\Assets\Upload::uploads_folder](https://api.silverstripe.org/4/SilverStripe/Assets/Upload.html)
configuration parameter, or you don't have any files uploaded to it.
The issues covered by this patch may still affect your application.
### I have two "/Uploads" folders in CMS
If you see two /Uploads folders in CMS, most likely that means one of them was
migrated from Silverstripe CMS 3, whereas some application logic created the
second copy after the migration.
In that case your database would contain a record in the `File` table for each
of them, while `File_Live` table would only have a record about the other one.
After running the tasks provided in this patch, we suggest you manually move all
content from one Folder to the other via the CMS. Then remove the empty folder.
### What if I updated my files after upload?
Beware that draft files are present in 2 locations on the filesystem.
* The original live version of the file is still available.
* The draft version of the file should be present in a `.protected` folder that
is inaccessible for unauthenticated users.
This means that even if you moved a file but don't publish it, the live version
of the file will still be available.
To address this, "relocate-userform-uploads-2020-9280" task only works on the
very first version of a file if it's published. Subsequent versions of the file
are ignored by the task. For example, even if you didn't change the location of
a file, but updated the filename or its permissions, the script will ignore that
file.
If you have an unpublished draft, but the live version of the file is 1 (no
published changes after upload), the script will try to move the live version of
the file into its originally intended location, while preserving the current
draft version of the file as-is. This would increment numbers for both Live and
Draft versions of the file, while moving the live version to another location
(both on the file system and in the CMS).
### Files with draft access restriction
If you have a published file that is publicly available and you update the draft
version to restrict read access, the live file currently remains accessible
until you publish the draft version. However, the `canView` method on the live
file will immediately start returning `false` for anonymous users.
This will cause `normalise-access` to protect the live file. Beware that if
you have unpublished access restrictions on draft files, those will take effect
on the live file after running `normalise-access`.
Read the ["Files with access restriction in Draft" GitHub issue](https://github.com/silverstripe/silverstripe-assets/issues/385)
for more details.
<!--- Changes below this line will be automatically regenerated -->
<!--- Changes above this line will be automatically regenerated -->