2011-02-07 07:48:44 +01:00
# Import CSV data
## Introduction
CSV import can be easily achieved through PHP's built-in `fgetcsv()` method,
but this method doesn't know anything about your datamodel. In SilverStripe,
this can be handled through the a specialized CSV importer class that can
2016-03-30 02:17:28 +02:00
be customised to fit your data.
2011-02-07 07:48:44 +01:00
## The CsvBulkLoader class
2017-07-03 03:22:12 +02:00
The [CsvBulkLoader ](api:SilverStripe\Dev\CsvBulkLoader ) class facilitate complex CSV-imports by defining column-mappings and custom converters.
2011-02-07 07:48:44 +01:00
It uses PHP's built-in `fgetcsv()` function to process CSV input, and accepts a file handle as an input.
Feature overview:
* Custom column mapping
* Auto-detection of CSV-header rows
* Duplicate detection based on custom criteria
* Automatic generation of relations based on one or more columns in the CSV-Data
* Definition of custom import methods (e.g. for date conversion or combining multiple columns)
* Optional deletion of existing records if they're not present in the CSV-file
* Results grouped by "imported", "updated" and "deleted"
## Usage
You can use the CsvBulkLoader without subclassing or other customizations, if the column names
in your CSV file match `$db` properties in your dataobject. E.g. a simple import for the
2017-07-03 03:22:12 +02:00
[Member ](api:SilverStripe\Security\Member ) class could have this data in a file:
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
FirstName,LastName,Email
Donald,Duck,donald@disney.com
Daisy,Duck,daisy@disney.com
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
The loader would be triggered through the `load()` method:
2017-08-03 02:51:32 +02:00
```php
2011-02-07 07:48:44 +01:00
$loader = new CsvBulkLoader('Member');
$result = $loader->load('< my-file-path > ');
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
2017-07-03 03:22:12 +02:00
By the way, you can import [Member ](api:SilverStripe\Security\Member ) and [Group ](api:SilverStripe\Security\Group ) data through `http://localhost/admin/security`
2011-02-07 07:48:44 +01:00
interface out of the box.
## Import through ModelAdmin
2017-07-03 03:22:12 +02:00
The simplest way to use [CsvBulkLoader ](api:SilverStripe\Dev\CsvBulkLoader ) is through a [ModelAdmin ](api:SilverStripe\Admin\ModelAdmin ) interface - you get an upload form out of the box.
2011-02-07 07:48:44 +01:00
2017-08-03 02:51:32 +02:00
```php
2017-08-03 05:35:09 +02:00
class PlayerAdmin extends ModelAdmin {
private static $managed_models = [
2011-02-07 07:48:44 +01:00
'Player'
2017-08-03 05:35:09 +02:00
];
private static $model_importers = [
2016-01-25 03:26:25 +01:00
'Player' => 'CsvBulkLoader',
2017-08-03 05:35:09 +02:00
];
2013-03-21 19:48:54 +01:00
private static $url_segment = 'players';
2011-02-07 07:48:44 +01:00
}
?>
2017-08-03 05:35:09 +02:00
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
The new admin interface will be available under `http://localhost/admin/players` , the import form is located
below the search form on the left.
## Import through a custom controller
2016-03-30 02:17:28 +02:00
You can have more customised logic and interface feedback through a custom controller.
2013-01-29 14:14:47 +01:00
Let's create a simple upload form (which is used for `MyDataObject` instances).
You'll need to add a route to your controller to make it accessible via URL
(see [director ](/reference/director )).
2011-02-07 07:48:44 +01:00
2017-08-03 02:51:32 +02:00
```php
2017-08-03 05:35:09 +02:00
class MyController extends Controller {
2017-08-03 02:51:32 +02:00
2017-08-03 05:35:09 +02:00
private static $allowed_actions = ['Form'];
2014-09-21 02:07:58 +02:00
2011-02-07 07:48:44 +01:00
protected $template = "BlankPage";
2014-09-21 02:07:58 +02:00
2012-01-30 23:13:42 +01:00
public function Link($action = null) {
2011-02-07 07:48:44 +01:00
return Controller::join_links('MyController', $action);
}
2014-09-21 02:07:58 +02:00
2012-01-30 23:13:42 +01:00
public function Form() {
2011-02-07 07:48:44 +01:00
$form = new Form(
$this,
'Form',
2011-10-28 03:37:27 +02:00
new FieldList(
2011-02-07 07:48:44 +01:00
new FileField('CsvFile', false)
),
2011-10-28 03:37:27 +02:00
new FieldList(
2011-02-07 07:48:44 +01:00
new FormAction('doUpload', 'Upload')
),
new RequiredFields()
);
return $form;
}
2014-09-21 02:07:58 +02:00
2012-01-30 23:13:42 +01:00
public function doUpload($data, $form) {
2011-02-07 07:48:44 +01:00
$loader = new CsvBulkLoader('MyDataObject');
$results = $loader->load($_FILES['CsvFile']['tmp_name']);
2017-08-03 05:35:09 +02:00
$messages = [];
2011-02-07 07:48:44 +01:00
if($results->CreatedCount()) $messages[] = sprintf('Imported %d items', $results->CreatedCount());
if($results->UpdatedCount()) $messages[] = sprintf('Updated %d items', $results->UpdatedCount());
if($results->DeletedCount()) $messages[] = sprintf('Deleted %d items', $results->DeletedCount());
if(!$messages) $messages[] = 'No changes';
$form->sessionMessage(implode(', ', $messages), 'good');
2014-09-21 02:07:58 +02:00
2011-02-07 07:48:44 +01:00
return $this->redirectBack();
}
}
2017-08-03 05:35:09 +02:00
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
2017-07-03 03:22:12 +02:00
Note: This interface is not secured, consider using [Permission::check() ](api:SilverStripe\Security\Permission::check( )) to limit the controller to users
2011-02-07 07:48:44 +01:00
with certain access rights.
## Column mapping and relation import
We're going to use our knowledge from the previous example to import a more sophisticated CSV file.
Sample CSV Content
2017-08-03 02:51:32 +02:00
```
2014-11-19 22:29:54 +01:00
"Number","Name","Birthday","Team"
2011-02-07 07:48:44 +01:00
11,"John Doe",1982-05-12,"FC Bayern"
12,"Jane Johnson", 1982-05-12,"FC Bayern"
13,"Jimmy Dole",,"Schalke 04"
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
Datamodel for Player
2017-08-03 02:51:32 +02:00
```php
2017-08-03 05:35:09 +02:00
class Player extends DataObject {
private static $db = [
2011-02-07 07:48:44 +01:00
'PlayerNumber' => 'Int',
2014-09-21 02:07:58 +02:00
'FirstName' => 'Text',
'LastName' => 'Text',
'Birthday' => 'Date',
2017-08-03 05:35:09 +02:00
];
private static $has_one = [
2011-02-07 07:48:44 +01:00
'Team' => 'FootballTeam'
2017-08-03 05:35:09 +02:00
];
2011-02-07 07:48:44 +01:00
}
?>
2017-08-03 05:35:09 +02:00
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
Datamodel for FootballTeam:
2017-08-03 02:51:32 +02:00
```php
2017-08-03 05:35:09 +02:00
class FootballTeam extends DataObject {
private static $db = [
2014-09-21 02:07:58 +02:00
'Title' => 'Text',
2017-08-03 05:35:09 +02:00
];
private static $has_many = [
2011-02-07 07:48:44 +01:00
'Players' => 'Player'
2017-08-03 05:35:09 +02:00
];
2011-02-07 07:48:44 +01:00
}
?>
2017-08-03 05:35:09 +02:00
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
Sample implementation of a custom loader. Assumes a CSV-file in a certain format (see below).
* Converts property names
* Splits a combined "Name" fields from the CSV-data into `FirstName` and `Lastname` by a custom importer method
* Avoids duplicate imports by a custom `$duplicateChecks` definition
2011-02-21 22:23:22 +01:00
* Creates `Team` relations automatically based on the `Gruppe` column in the CSV data
2017-08-03 02:51:32 +02:00
```php
2017-08-03 05:35:09 +02:00
class PlayerCsvBulkLoader extends CsvBulkLoader {
public $columnMap = [
2011-02-07 07:48:44 +01:00
'Number' => 'PlayerNumber',
'Name' => '->importFirstAndLastName',
2014-11-19 22:29:54 +01:00
'Birthday' => 'Birthday',
'Team' => 'Team.Title',
2017-08-03 05:35:09 +02:00
];
public $duplicateChecks = [
2014-11-19 22:29:54 +01:00
'Number' => 'PlayerNumber'
2017-08-03 05:35:09 +02:00
];
public $relationCallbacks = [
'Team.Title' => [
2011-02-07 07:48:44 +01:00
'relationname' => 'Team',
'callback' => 'getTeamByTitle'
2017-08-03 05:35:09 +02:00
]
];
2012-01-30 23:13:42 +01:00
public static function importFirstAndLastName(& $obj, $val, $record) {
2011-02-07 07:48:44 +01:00
$parts = explode(' ', $val);
if(count($parts) != 2) return false;
$obj->FirstName = $parts[0];
$obj->LastName = $parts[1];
}
2012-01-30 23:13:42 +01:00
public static function getTeamByTitle(& $obj, $val, $record) {
2012-06-23 00:32:43 +02:00
return FootballTeam::get()->filter('Title', $val)->First();
2011-02-07 07:48:44 +01:00
}
}
?>
2017-08-03 05:35:09 +02:00
2017-08-03 02:51:32 +02:00
```
2017-05-31 17:05:05 +02:00
Building off of the ModelAdmin example up top, use a custom loader instead of the default loader by adding it to `$model_importers` . In this example, `CsvBulkLoader` is replaced with `PlayerCsvBulkLoader` .
2017-08-03 02:51:32 +02:00
```php
2017-08-03 05:35:09 +02:00
class PlayerAdmin extends ModelAdmin {
private static $managed_models = [
2017-05-31 17:05:05 +02:00
'Player'
2017-08-03 05:35:09 +02:00
];
private static $model_importers = [
2017-05-31 17:05:05 +02:00
'Player' => 'PlayerCsvBulkLoader',
2017-08-03 05:35:09 +02:00
];
2017-05-31 17:05:05 +02:00
private static $url_segment = 'players';
}
?>
2017-08-03 05:35:09 +02:00
2017-08-03 02:51:32 +02:00
```
2011-02-07 07:48:44 +01:00
## Related
2017-07-03 03:22:12 +02:00
* [CsvParser ](api:SilverStripe\Dev\CsvParser )
* [ModelAdmin ](api:SilverStripe\Admin\ModelAdmin )