tonyair
/
silverstripe-fulltextsearch
mirror of https://github.com/silverstripe/silverstripe-fulltextsearch
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Merge pull request #215 from creative-commoners/pulls/new-docs 

[WIP] overhaul docs
This commit is contained in:
Daniel Hensby 2018-06-25 15:07:11 +01:00 committed by GitHub
parent 5fcaf87954 06c604c9f3
commit 65b780f629
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
13 changed files with 1156 additions and 1096 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@ -19,7 +19,9 @@ Adds support for fulltext search engines like Sphinx and Solr to SilverStripe CM
## Documentation
See docs/en/index.md
For pure Solr docs, check out [the Solr 4.10.4 guide](https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-4.10.pdf).
See [the docs](/docs/en/00_index.md) for configuration and setup, or for the quick version see [the quick start guide](/docs/en/01_getting_started.md#quick-start).
For details of updates, bugfixes, and features, please see the [changelog](CHANGELOG.md).
@ -48,8 +50,4 @@ maybe 'Content->Summary' to allow calling a specific method on the field object
- Allow user logic to cause triggering reindex of documents when field is user generated
* Add sphinx connector
* Add generic APIs for spell correction, file text extraction and snippet generation
* Better docs

3
_config/config.yml
View File

@ -4,6 +4,3 @@ Name: fulltextsearchconfig
SilverStripe\ORM\DataObject:
extensions:
- SilverStripe\FullTextSearch\Search\Extensions\SearchUpdater_ObjectHandler
SilverStripe\CMS\Controllers\ContentController:
extensions:
- SilverStripe\FullTextSearch\Solr\Control\ContentControllerExtension

87
bin/fts_quickstart Normal file
View File

@ -0,0 +1,87 @@
#!/usr/bin/env bash
echo "Installing Java SDK..."
# Find our package manager
if VERB="$( command -v apt-get )" 2> /dev/null; then
echo "Debian-based OS detected"
sudo apt-get install -y openjdk-8-jdk 2> /dev/null
elif VERB="$( command -v yum )" 2> /dev/null; then
echo "Modern Red Hat-based OS detected"
sudo yum install -y java-1.8.0-openjdk.x86_64 2> /dev/null
else
echo "No valid package manager detected; try one of apt-get, yum."
exit 1
fi
if [ ! -d "/opt/solr" ]; then
printf "Installing Solr 4"
# Acquire and unzip solr4
wget http://archive.apache.org/dist/lucene/solr/4.10.4/solr-4.10.4.tgz 2> /dev/null && printf "."
tar -xf solr-4.10.4.tgz 2> /dev/null && printf "."
rm solr-4.10.4.tgz 2> /dev/null && printf "."
# Set the defaults in /opt/solr
sudo mv solr-4.10.4 /opt/solr 2> /dev/null && printf "."
mv /opt/solr/example /opt/solr/core 2> /dev/null && echo "."
fi
if [ ! -f "/etc/init.d/solr" ]; then
echo "Installing Solr daemon..."
# Set up the daemon so that solr will run on startup
sudo cp vendor/silverstripe/fulltextsearch/docs/examples/daemon_script /etc/init.d/solr 2> /dev/null
sudo chmod +x /etc/init.d/solr 2> /dev/null
sudo chkconfig --add solr 2> /dev/null
fi
# Get solr running
sudo /etc/init.d/solr start 2> /dev/null
# Determine application dir
if [ -d app ]; then
APPDIR="app"
elif [ -d mysite ]; then
APPDIR="mysite"
else
echo "Can't detect application dir - skipping default index creating"
exit 1
fi
# Check to see if it has been configured in _config.php
grep -i "Solr::configure_server(" "$APPDIR/_config.php" 2> /dev/null
if [ "$?" != 0 ]; then
echo "Configuring Solr in _config.php..."
if [ ! -f "$APPDIR/_config.php" ]; then
echo "<?php" > "$APPDIR/_config.php"
echo "" >> "$APPDIR/_config.php"
fi
echo "" >> "$APPDIR/_config.php"
echo "# Enable Fulltextsearch" >> "$APPDIR/_config.php"
echo "\\SilverStripe\\FullTextSearch\\Solr\\Solr::configure_server([" >> "$APPDIR/_config.php"
echo " 'indexstore' => [" >> "$APPDIR/_config.php"
echo " 'mode' => 'file'," >> "$APPDIR/_config.php"
echo " 'path' => BASE_PATH . '/.solr'" >> "$APPDIR/_config.php"
echo " ]" >> "$APPDIR/_config.php"
echo "]);" >> "$APPDIR/_config.php"
fi
# Determine code dir
if [ -d "$APPDIR/src" ]; then
CODEDIR="$APPDIR/src"
elif [ -d "$APPDIR/code" ]; then
CODEDIR="$APPDIR/code"
else
echo "Can't detect code dir - skipping default index creating"
exit 1
fi
# Create a default index
if [ ! -f "$CODEDIR/FulltextSearch/DefaultIndex.php" ]; then
echo "Creating default index..."
mkdir -p "$CODEDIR/FulltextSearch"
cp vendor/silverstripe/fulltextsearch/docs/examples/default_index.php.example "$CODEDIR/FulltextSearch/DefaultIndex.php"
fi
vendor/bin/sake dev/tasks/Solr_Configure
vendor/bin/sake dev/tasks/Solr_Reindex
echo "Quickstart complete!"

34
docs/en/00_index.md Normal file
View File

@ -0,0 +1,34 @@
# Fulltext search documentation index
- Getting started
- [Module scope](01_getting_started.md#module-scope)
- [Quick start guide](01_getting_started.md#quick-start)
- Setup
- [Requirements](02_setup.md#requirements)
- [Installing Solr](02_setup.md#installing-solr)
- [Solr admin](02_setup.md#solr-admin)
- Configuration
- [Solr server parameters](03_configuration.md#solr-server-parameters)
- [Creating an index](03_configuration.md#creating-an-index)
- [Adding data to an index](03_configuration.md#adding-data-to-an-index)
- [Running the dev/tasks](03_configuration.md#dev-tasks)
- [File-based configuration](03_configuration.md#file-based-configuration)
- [Handling results](03_configuration.md#handling-results)
- Querying
- [Building a SearchQuery](04_querying.md#building-a-`searchquery`)
- [Searching value ranges](04_querying.md#searching-value-ranges)
- [Empty or existing values](04_querying.md#empty-or-existing-values)
- [Executing your query](04_querying.md#executing-your-query)
- Advanced configuration
- [Facets](05_advanced_configuration.md#facets)
- [Using multiple indexes](05_advanced_configuration.md#multiple-indexes)
- [Analyzers, tokens and token filters](05_advanced_configuration.md#analyzers,-tokenizers-and-token-filters)
- [Spellcheck](05_advanced_configuration.md#spell-check-("did-you-mean..."))
- [Highlighting](05_advanced_configuration.md#highlighting)
- [Boosting](05_advanced_configuration.md#boosting)
- [Indexing related objects](05_advanced_configuration.md#indexing-related-objects)
- [Subsites](05_advanced_configuration.md#subsites)
- [Custom field types](05_advanced_configuration.md#custom-field-types)
= [Text extraction](05_advanced_configuration.md#text-extraction)
- Troubleshooting
- [Gotchas](06_troubleshooting.md#common-gotchas)

66
docs/en/01_getting_started.md Normal file
View File

@ -0,0 +1,66 @@
# Getting started
## Module scope
### Introduction
This is a module aimed at adding support for standalone fulltext search engines to SilverStripe.
It contains several layers:
* A fulltext API, ignoring the actual provision of fulltext searching
* A connector API, providing common code to allow connecting a fulltext searching engine to the fulltext API
* Some connectors for common fulltext searching engines (currently only [Apache Solr](http://lucene.apache.org/solr/))
### Reasoning
There are several fulltext search engines that work in a similar manner. They build indexes of denormalized data that
are then searched through using some custom query syntax.
Traditionally, fulltext search connectors for SilverStripe have attempted to hide this design, instead presenting
fulltext searching as an extension of the object model. However, the disconnect between the fulltext search engine's
design and the object model meant that searching was inefficient. The abstraction would also often break and it was
hard to then figure out what was going on.
This module instead provides the ability to define those indexes and queries in PHP. The indexes are defined as a
mapping between the SilverStripe object model and the connector-specific fulltext engine index model. This module then
interrogates model metadata to build the specific index definition.
It also hooks into SilverStripe framework in order to update the indexes when the models change and connectors then
convert those index and query definitions into fulltext engine specific code.
The intent of this module is not to make changing fulltext search engines seamless. Where possible this module provides
common interfaces to fulltext engine functionality, abstracting out common behaviour. However, each connector also
offers its own extensions, and there is some behaviour (such as getting the fulltext search engines installed,
configured and running) that each connector deals with itself, in a way best suited to that search engine's design.
## Quick start
If you are running on a Linux-based system, you can get up and running quickly with the quickstart script, like so:
```bash
composer require silverstripe/fulltextsearch && vendor/bin/fts_quickstart
```
This will:
- Install the required Java SDK (using `apt-get` or `yum`)
- Install Solr 4
- Set up a daemon to run Solr on startup
- Start Solr
- Configure Solr in your `_config.php` (and create one if you don't have one)
- Create a DefaultIndex
- Run a [Solr Configure](03_configuration.md#solr-configure) and a [Solr Reindex](03_configuration.md#solr-reindex)
If you have the [CMS module](https://github.com/silverstripe/silverstripe-cms) installed, you will be able to simply add
`$SearchForm` to your template to add a Solr search form. Default configuration is added via the
[`ContentControllerExtension`](/src/Solr/Control/ContentControllerExtension.php) and alternative
[`SearchForm`](/src/Solr/Forms/SearchForm.php). With the
[Simple theme](https://github.com/silverstripe-themes/silverstripe-simple), this is in the
[`Header`](https://github.com/silverstripe-themes/silverstripe-simple/blob/master/templates/Includes/Header.ss#L10-L15)
by default.
Ensure that you _don't_ have `SilverStripe\ORM\Search\FulltextSearchable::enable()` set in `_config.php`, as the
`SearchForm` action provided by that class will conflict.
You can override the default template with a new one at `templates/Layout/Page_results_solr.ss`.

92
docs/en/02_setup.md Normal file
View File

@ -0,0 +1,92 @@
# Setup
The FulltextSearch module includes support for connecting to Solr.
It works with Solr in multi-core mode. It needs to be able to update Solr configuration files, and has modes for doing
so by direct file access (when Solr shares a server with SilverStripe) and by WebDAV (when it's on a different server).
See the helpful [Solr Tutorial](http://lucene.apache.org/solr/4_5_1/tutorial.html), for more on cores and querying.
## Requirements
Since Solr is Java based, it requires Java 1.5 or greater installed.
When you're installing it yourself, it also requires a servlet container such as Tomcat, Jetty, or Resin. For
development testing there is a standalone version that comes bundled with Jetty (see [Installing Solr](#installing-solr)
below).
See the official [Solr installation docs](http://wiki.apache.org/solr/SolrInstall) for more information.
Note that these requirements are for the Solr server environment, which doesn't have to be the same physical machine as
the SilverStripe webhost.
## Installing Solr
### Local installation
If you'll be running Solr on the same machine as your SilverStripe installation, and the
[quick start script](01_getting_started.md#quick-start) doesn't suit your needs, you can use the
[fulltextsearch-localsolr module](https://github.com/silverstripe-archive/silverstripe-fulltextsearch-localsolr). This
can also be useful as a development dependency. You can bring it in via composer (use `require-dev` if you plan to
install Solr remotely in Production):
```bash
composer require silverstripe/fulltextsearch-localsolr
```
Once installed, start the server via CLI:
```bash
cd fulltextsearch-localsolr/server
java -jar start.jar
```
Then configure the module to use `file` mode with the following configuration in your `app/_config.php`, making sure
that the `path` directory is writeable by the user that started the server (above):
```php
use SilverStripe\FullTextSearch\Solr\Solr;
Solr::configure_server([
'host' => 'localhost',
'indexstore' => [
'mode' => 'file',
'path' => BASE_PATH . '/.solr'
]
]);
```
### Remote installation
Alternatively, it can be beneficial to keep the Solr service contained on its own infrastructure, for performance and
security reasons. The [Common Web Platform (CWP)](www.cwp.govt.nz) uses Solr in this manner. To do so, you should
install the dependencies on the remote server, and then configure the module to use the `webdav` mode like so:
```php
use SilverStripe\FullTextSearch\Solr\Solr;
Solr::configure_server([
'host' => 'remotesolrserver.com', // IP address or hostname
'indexstore' => [
'mode' => 'webdav',
'path' => BASE_PATH . '/webdav',
]
]);
```
Check all the available [configuration options](03_configuration.md#solr-server-parameters) to fine-tune the module to
work with your desired setup.
This will mean that all configuration files, and the indexes themselves, are stored remotely.
## Solr admin
Solr provides an administration interface with a GUI to allow you to get at the finer details of your cores and
configuration. You can access it at example.com:<SOLR_PORT>/<SOLR_PATH>/#/ on a local installation
(usually example.com:8983/solr/#/).
There you can access logging, run raw queries against your stored indexes, and get some basic performance metrics.
Additionally, you can perform more drastic changes, such as dropping and reloading cores.
For a comprehensive look at the Solr admin interface, read the
[user guide for Solr 4.10](http://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-4.10.pdf#page=17)

264
docs/en/03_configuration.md Normal file
View File

@ -0,0 +1,264 @@
# Configuration
## Solr server parameters
Set these values inside your `app/_config.php` - the defaults are shown below:
```php
use SilverStripe\FullTextSearch\Solr\Solr;
Solr::configure_server([
'host' => 'localhost', // The host or IP address that Solr is listening on
'port' => '8983', // The port Solr is listening on
'path' => '/solr', // The suburl the Solr service is available on
'version' => '4', // Solr server version - currently only 3 and 4 supported
'service' => 'Solr4Service', // The class that provides actual communcation to the Solr server
'extraspath' => BASE_PATH .'/vendor/silverstripe/fulltextsearch/conf/solr/4/extras/', // Absolute path to the folder containing templates used for generating the schema and field definitions
'templates' => BASE_PATH . '/vendor/silverstripe/fulltextsearch/conf/solr/4/templates/', // Absolute path to the configuration default files, e.g. solrconfig.xml
'indexstore' => [
'mode' => NULL, // [REQUIRED] a classname which implements SolrConfigStore, or 'file' or 'webdav'
'path' => NULL, // [REQUIRED] The (locally accessible) path to write the index configurations to OR The suburl on the Solr host that is set up to accept index configurations via webdav (e.g. BASE_PATH . '/.solr')
'remotepath' => same as 'path' when using 'file' mode, // The path that the Solr server will read the index configurations from
'auth' => NULL, // Webdav only - A username:password pair string to use to auth against the webdav server (e.g. solr:solr)
'port' => '8983' // The port for WebDAV if different from the Solr port
]
]);
```
Note: We recommend to put the `indexstore['path']` directory outside of the webroot. If you place it inside of the webroot (as shown in the example), please ensure its contents are not accessible through the webserver.
This can be achieved by server configuration, or (in most configurations) also by marking the folder as hidden via a "dot" prefix.
### Disabling automatic configuration
If you have this module installed but do not have a Solr server running, you can disable the database manipulation
hooks that trigger automatic index updates:
```yaml
SilverStripe\FullTextSearch\Search\Updaters\SearchUpdater:
enabled: false
```
## Creating an index
An index can essentially be considered a database that contains all of your searchable content. By default, it will store everything in a field called `Content`, which is queried to find your search results. To create an index that you can query, you can define it like so:
```php
use Page;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addFulltextField('Title');
}
}
```
This will create a new `SolrIndex` called `MyIndex`, and it will store the `Title` field on all `Pages` for searching. To index more than one class,
you simply call `addClass()` multiple times. Fields that you add don't have to be present on all classes in the index, they will only apply to a class
if it is present.
```php
use Page;
use SilverStripe\Security\Member;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addClass(Member::class);
$this->addFulltextField('Content'); // only applies to Page class
$this->addFulltextField('FirstName'); // only applies to Member class
}
}
```
You can also skip listing all searchable fields, and have the index figure it out automatically via `addAllFulltextFields()`. This will add any database fields that are `instanceof DBString` to the index. Use this with caution, however, as you may inadvertently return sensitive information - it is often safer to declare your fields explicitly.
Once you've added this file, make sure you run a [Solr configure](#solr-configure) to set up your new index.
## Adding data to an index
Once you have [created your index](#creating-an-index), you can add data to it in a number of ways.
### Reindex the site
Running the [Solr reindex task](#solr-reindex) will crawl your site for classes that match those defined on your index, and add the defined fields to the index for searching. This is the most common method used to build the index the first time, or to perform a full rebuild of the index.
### Publish a page in the CMS
Every change, addition or removal of an indexed class instance triggers an index update through a "processor" object. The update is transparently handled through inspecting every executed database query and checking which database tables are involved in it.
A reindex event will trigger when you make a change in the CMS, via `SearchUpdater::handle_manipulation()`, or `ProxyDBExtension::updateProxy()`. This tracks changes to the database, so any alterations will trigger a reindex. In order to minimise delays to those users, the index update is deferred until after the actual request returns to the user, through PHP's `register_shutdown_function()` functionality.
### Manually
If the situation calls for it, you can add an object to the index directly:
```php
use Page;
$page = Page::create(['Content' => 'Help me. My house is on fire. This is less than optimal.']);
$page->write();
```
Depending on the size of the index and how much content needs to be processed, it could take a while for your search results to be updated, so your newly-updated page may not be available in your search results immediately. This approach is typically not recommended.
### Queued jobs
If the [Queued Jobs module](https://github.com/symbiote/silverstripe-queuedjobs/) is installed, updates are queued up instead of executed in the same request. Queued jobs are usually processed every minute. Large index updates will be batched into multiple queued jobs to ensure a job can run to completion within common constraints, such as memory and execution time limits. You can check the status of jobs in an administrative interface under `admin/queuedjobs/`.
### Excluding draft content
By default, the `SearchUpdater` class indexes all available "variant states", so in the case of the `Versioned` extension, both "draft" and "live".
For most cases, you'll want to exclude draft content from your search results.
You can either prevent the draft content from being indexed in the first place, by adding the following to your `SearchIndex::init()` method:
```php
use Page;
use SilverStripe\FullTextSearch\Search\Variants\SearchVariantVersioned;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
use SilverStripe\Versioned\Versioned;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addFulltextField('Title');
$this->excludeVariantState([SearchVariantVersioned::class => Versioned::DRAFT]);
}
}
```
Alternatively, you can index draft content, but simply exclude it from searches. This can be handy to preview search results on unpublished content, in case a CMS author is logged in. Before constructing your `SearchQuery`, conditionally switch to the "live" stage.
### Adding DataObjects
If you create a class that extends `DataObject` (and not `Page`) then it won't be automatically added to the search
index. You'll have to make some changes to add it in. The `DataObject` class will require the following minimum code
to render properly in the search results:
* `Link()` needs to return the URL to follow from the search results to actually view the object.
* `Name` (as a DB field) will be used as the result title.
* `Abstract` (as a DB field) will show under the search result title.
* `getShowInSearch()` is required to get the record to show in search, since all results are filtered by `ShowInSearch`.
So with that, you can add your class to your index:
```php
use My\Namespace\Model\SearchableDataObject;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
use Page;
class MySolrSearchIndex extends SolrIndex {
public function init()
{
$this->addClass(SearchableDataObject::class);
$this->addClass(Page::class);
$this->addAllFulltextFields();
}
}
```
Once you've created the above classes and run the [solr dev tasks](#solr-dev-tasks) to tell Solr about the new index
you've just created, this will add `SearchableDataObject` and the text fields it has to the index. Now when you search
on the site using `MySolrSearchIndex->search()`, the `SearchableDataObject` results will show alongside normal `Page`
results.
## Solr dev tasks
There are two dev/tasks that are central to the operation of the module - `Solr_Configure` and `Solr_Reindex`. You can access these through the web, or via CLI. Running via the web will return "quiet" output by default, but you can increase verbosity by adding `?verbose=1` to the `dev/tasks` URL; CLI will return verbose output by default.
It is often a good idea to run a configure, followed by a reindex, after a code change - for example, after a deployment.
### Solr configure
`dev/tasks/Solr_Configure`
This task will upload configuration to the Solr core, reloading it or creating it as necessary, and generate the schema. This should be run after every code change to your indexes, or after any configuration changes. This will convert the PHP-based abstraction layer into actual Solr XML. Assuming default configuration and the use of the `DefaultIndex`, it will:
- create the directory `BASE_PATH/.solr/DefaultIndex/` if it doesn't already exist
- copy configuration files from `vendor/silverstripe/fulltextsearch/conf/extras` to `BASE_PATH/.solr/DefaultIndex/conf/`
- generate a `schema.xml` in `BASE_PATH/.solr/DefaultIndex/conf/`
This task will overwrite these files every time it is run.
### Solr reindex
`dev/tasks/Solr_Reindex`
This task performs a reindex, which adds all the data specified in the index definition into the index store.
If you have the [Queued Jobs module](https://github.com/symbiote/silverstripe-queuedjobs/) installed, then this task will create multiple reindex jobs that are processed asynchronously; unless you are in `dev` mode, in which case the index will be processed immediately (see [processor.yml](/_config/processor.yml)). Otherwise, it will run in one process. Often, if you are running it via the web, the request will time out. Usually this means the actually process is still running in the background, but it can be alarming to the user, so bear that in mind.
Internally groups of records are grouped into sizes of 200. You can configure this group sizing by using the `Solr_Reindex.recordsPerRequest` config:
```yaml
SilverStripe\FullTextSearch\Solr\Tasks\Solr_Reindex:
recordsPerRequest: 150
```
The Solr indexes will be stored as binary files inside your SilverStripe project. You can also copy the `thirdparty/` Solr directory somewhere else, just set the `path` value in `mysite/_config.php` to point to the new location.
## File-based configuration
Many aspects of Solr are configured outside of the `schema.xml` file which SilverStripe generates based on the `SolrIndex` subclass that is defined. For example, stopwords are placed in their own `stopwords.txt` file, and advanced [spellchecking](05_advanced_configuration.md#spell-check-("did-you-mean...")) can be configured in `solrconfig.xml`.
By default, these files are copied from the `fulltextsearch/conf/extras/` directory over to the new index location. In order to use your own files, copy these files into a location of your choosing (for example `mysite/data/solr/`), and tell Solr to use this folder with the `extraspath` [configuration setting](#solr-server-parameters). Run a [`Solr_Configure](#solr-configure) to apply these changes.
You can also define these on an index-by-index basis by defining `SolrIndex->getExtrasPath()`.
## Handling results
In order to render search results, you need to return them from a controller. You can also drive this through a form response through standard SilverStripe forms. In this case we simply assume there's a GET parameter named `q` with a search term present.
```php
use SilverStripe\CMS\Controllers\ContentController;
use SilverStripe\Control\HTTPRequest;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use My\Namespace\Index\MyIndex;
class PageController extends ContentController
{
private static $allowed_actions = [
'search',
];
public function search(HTTPRequest $request)
{
$query = SearchQuery::create()->addSearchTerm($request->getVar('q'));
return $this->renderWith([
'SearchResult' => MyIndex::singleton()->search($query)
]);
}
}
```
In your template (e.g. `Page_results.ss`) you can access the results and loop through them. They're stored in the `$Matches` property of the search return object.
```silverstripe
<% if $SearchResult.Matches %>
<h2>Results for &quot;{$Query}&quot;</h2>
<p>Displaying Page $SearchResult.Matches.CurrentPage of $SearchResult.Matches.TotalPages</p>
<ol>
<% loop $SearchResult.Matches %>
<li>
<h3><a href="$Link">$Title</a></h3>
<p><% if $Abstract %>$Abstract.XML<% else %>$Content.ContextSummary<% end_if %></p>
</li>
<% end_loop %>
</ol>
<% else %>
<p>Sorry, your search query did not return any results.</p>
<% end_if %>
```
Please check the [pagination guide](https://docs.silverstripe.org/en/4/developer_guides/templates/how_tos/pagination/)
in the main SilverStripe documentation to learn how to paginate through search results.

130
docs/en/04_querying.md Normal file
View File

@ -0,0 +1,130 @@
# Querying
This is where the magic happens. You will construct the search terms and other parameters required to form a `SearchQuery` object, and pass that into a `SearchIndex` to get results.
## Building a `SearchQuery`
First, you'll need to construct a new `SearchQuery` object:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = SearchQuery::create();
```
You can then alter the `SearchQuery` with a number of methods:
### `addSearchTerm()`
The simplest - pass through a string to search your index for.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = SearchQuery::create()
->addSearchTerm('fire');
```
You can also limit this to specific fields by passing an array as the second argument, specified in the form of `{table}_{field}`:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use Page;
$query = SearchQuery::create()
->addSearchTerm('on fire', [Page::class . '_Title']);
```
### `addFuzzySearchTerm()`
Pass through a string to search your index for, with "fuzzier" matching - this means that a term like "fishing" would also likely find results containing "fish" or "fisher". Otherwise behaves the same as `addSearchTerm()`.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = SearchQuery::create()
->addFuzzySearchTerm('fire');
```
### `addClassFilter()`
Only query a specific class in the index, optionally including subclasses.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use My\Namespace\PageType\SpecialPage;
$query = SearchQuery::create()
->addClassFilter(SpecialPage::class, false); // only return results from SpecialPages, not subclasses
```
## Searching value ranges
Most values can be expressed as ranges, most commonly dates or numbers. To search for a range of values rather than an exact match,
use the `SearchQuery_Range` class. The range can include bounds on both sides, or stay open-ended by simply leaving the argument blank.
It takes arguments in the form of `SearchQuery_Range::create($start, $end))`:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery_Range;
use My\Namespace\Index\MyIndex;
use Page;
$query = SearchQuery::create()
->addSearchTerm('fire')
// Only include documents edited in 2011 or earlier
->addFilter(Page::class . '_LastEdited', SearchQuery_Range::create(null, '2011-12-31T23:59:59Z'));
$results = MyIndex::singleton()->search($query);
```
### How do I use date ranges where dates might not be defined?
The Solr index updater only includes dates with values, so the field might not exist in all your index entries. A simple bounded range query (`<field>:[* TO <date>]`) will fail in this case. In order to query the field, reverse the search conditions and exclude the ranges you don't want:
```php
// Wrong: Filter will ignore all empty field values
$query->addFilter('fieldname', SearchQuery_Range::create('*', 'somedate'));
// Right: Exclude the opposite range
$query->addExclude('fieldname', SearchQuery_Range::create('somedate', '*'));
```
Note: At the moment, the date format is specific to the search implementation.
## Empty or existing values
Since there's a type conversion between the SilverStripe database, object properties
and the search index persistence, it's often not clear which condition is searched for.
Should it equal an empty string, or only match if the field wasn't indexed at all?
The `SearchQuery` API has the concept of a "missing" and "present" field value for this:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use My\Namespace\Index\MyIndex;
use Page;
$query = SearchQuery::create()
->addSearchTerm('fire');
// Needs a value, although it can be false
->addFilter(Page::class . '_ShowInMenus', SearchQuery::$present);
$results = MyIndex::singleton()->search($query);
```
## Executing your query
Once you have your query constructed, you need to run it against your index.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use My\Namespace\Index\MyIndex;
$query = SearchQuery::create()->addSearchTerm('fire');
$results = MyIndex::singleton()->search($query);
```
The return value of a `search()` call is an object which contains a few properties:
* `Matches`: `ArrayList` of the current "page" of search results.
* `Suggestion`: (optional) Any suggested spelling corrections in the original query notation
* `SuggestionNice`: (optional) Any suggested spelling corrections for display (without query notation)
* `SuggestionQueryString` (optional) Link to repeat the search with suggested spelling corrections

465
docs/en/05_advanced_configuration.md Normal file
View File

@ -0,0 +1,465 @@
# Advanced configuration
## Facets
Inside the `SolrIndex->search()` function, the third-party library solr-php-client is used to send data to Solr and parse the response. Additional information can be pulled from this response and added to your results object for use in templates using the `updateSearchResults()` extension hook.
```php
use My\Namespace\Index\MyIndex;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$index = MyIndex::singleton();
$query = SearchQuery::create()
->addSearchTerm('My Term');
$params = [
'facet' => 'true',
'facet.field' => 'SiteTree_ClassName',
];
$results = $index->search($query, -1, -1, $params);
```
By adding facet fields into the query parameters, our response object from Solr now contains some additional information that we can add into the results sent to the page.
```php
namespace My\Namespace\Extension;
use SilverStripe\Core\Extension;
use SilverStripe\View\ArrayData;
use SilverStripe\ORM\ArrayList;
class FacetedResultsExtension extends Extension
{
/**
* Adds extra information from the solr-php-client repsonse
* into our search results.
* @param ArrayData $results The ArrayData that will be used to generate search
* results pages.
* @param stdClass $response The solr-php-client response object.
*/
public function updateSearchResults($results, $response)
{
if (!isset($response->facet_counts) || !isset($response->facet_counts->facet_fields)) {
return;
}
$facetCounts = ArrayList::create([]);
foreach($response->facet_counts->facet_fields as $name => $facets) {
$facetDetails = ArrayData::create([
'Name' => $name,
'Facets' => ArrayList::create([]),
]);
foreach($facets as $facetName => $facetCount) {
$facetDetails->Facets->push(ArrayData::create([
'Name' => $facetName,
'Count' => $facetCount,
]));
}
$facetCounts->push($facetDetails);
}
$results->setField('FacetCounts', $facetCounts);
}
}
```
And then apply the extension to your index via `yaml`:
```yaml
My\Namespace\Index\MyIndex:
extensions:
- My\Namespace\Extension\FacetedResultsExtension
```
We can now access the facet information inside our templates like so:
```silverstripe
<% if $Results.FacetCounts %>
<% loop $Results.FacetCounts.Facets %>
<% loop $Facets %>
<p>$Name: $Count</p>
<% end_loop %>
<% end_loop %>
<% end_if %>
```
## Multiple indexes
Multiple indexes can be created and searched independently, but if you wish to override an existing
index with another, you can use the `$hide_ancestor` config.
```php
use SilverStripe\Assets\File;
use My\Namespace\Index\MyIndex;
class MyReplacementIndex extends MyIndex
{
private static $hide_ancestor = MyIndex::class;
public function init()
{
parent::init();
$this->addClass(File::class);
$this->addFulltextField('Title');
}
}
```
You can also filter all indexes globally to a set of pre-defined classes if you wish to
prevent any unknown indexes from being automatically included.
```yaml
SilverStripe\FullTextSearch\Search\FullTextSearch:
indexes:
- MyReplacementIndex
- CoreSearchIndex
```
## Analyzers, Tokenizers and Token Filters
When a document is indexed, its individual fields are subject to the analyzing and tokenizing filters that can transform
and normalize the data in the fields. You can remove blank spaces, strip HTML, replace a particular character and much
more as described in the [Solr Wiki](http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters).
### Synonyms
To add synonym processing at query-time, you can add the `SynonymFilterFactory` as an `Analyzer`:
```php
use SilverStripe\FullTextSearch\Solr\SolrIndex;
use Page;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addField('Content');
$this->addAnalyzer('Content', 'filter', [
'class' => 'solr.SynonymFilterFactory',
'synonyms' => 'synonyms.txt',
'ignoreCase' => 'true',
'expand' => 'false'
]);
}
}
```
This generates the following XML schema definition:
```xml
<field name="Page_Content">
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="false"/>
</field>
```
In this case, you most likely also want to define your own synonyms list. You can define a mapping in one of two ways:
* A comma-separated list of words. If the token matches any of the words, then all the words in the list are
substituted, which will include the original token.
* Two comma-separated lists of words with the symbol "=>" between them. If the token matches any word on
the left, then the list on the right is substituted. The original token will not be included unless it is also in the
list on the right.
For example:
```text
couch,sofa,lounger
teh => the
small => teeny,tiny,weeny
```
Then you should update your [Solr configuration](03_configuration.md#solr-server-parameters) to include your synonyms
file via the `extraspath` parameter, for example:
```php
use SilverStripe\FullTextSearch\Solr\Solr;
Solr::configure_server([
'extraspath' => BASE_PATH . '/mysite/Solr/',
'indexstore' => [
'mode' => 'file',
'path' => BASE_PATH . '/.solr',
]
]);
```
Will include `/mysite/Solr/synonyms.txt` as your list after a [Solr configure](03_configuration.md#solr-configure)
## Spell check ("Did you mean...")
Solr has various spell checking strategies (see the ["SpellCheckComponent" docs](http://wiki.apache.org/solr/SpellCheckComponent)), all of which are configured through `solrconfig.xml`.
In the default config which is copied into your index, spell checking data is collected from all fulltext fields
(everything you added through `SolrIndex->addFulltextField()`). The values of these fields are collected in a special `_text` field.
```php
use My\Namespace\Index\MyIndex;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$index = MyIndex::singleton();
$query = SearchQuery::create()
->addSearchTerm('My Term');
$params = [
'spellcheck' => 'true',
'spellcheck.collate' => 'true',
];
$results = $index->search($query, -1, -1, $params);
$results->spellcheck;
```
The built-in `_text` data is better than nothing, but also has some problems: it's heavily processed, for example by
stemming filters which butcher words. So misspelling "Govnernance" will suggest "govern" rather than "Governance".
This can be fixed by aggregating spell checking data in a separate field.
```php
use SilverStripe\CMS\Model\SiteTree;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addCopyField(SiteTree::class . '_Title', 'spellcheckData');
$this->addCopyField(SomeModel::class . '_Title', 'spellcheckData');
$this->addCopyField(SiteTree::class . '_Content', 'spellcheckData');
$this->addCopyField(SomeModel::class . '_Content', 'spellcheckData');
}
public function getFieldDefinitions()
{
$xml = parent::getFieldDefinitions();
$xml .= "\n\n\t\t<!-- Additional custom fields for spell checking -->";
$xml .= "\n\t\t<field name='spellcheckData' type='textSpellHtml' indexed='true' stored='false' multiValued='true' />";
return $xml;
}
}
```
Now you need to tell Solr to use our new field for gathering spelling data. In order to customise the spell checking configuration,
create your own `solrconfig.xml` (see [File-based configuration](03_configuration.md#file-based-configuration)). In there, change the following directive:
```xml
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="field">spellcheckData</str>
</searchComponent>
```
Copy the new configuration via a the [`Solr_Configure` task](03_configuration.md#solr-configure), and reindex your data before using the spell checker.
## Highlighting
Solr can highlight the searched terms in context of the matched content, to help users determine the relevancy of results (e.g. in which part of a sentence the term is used). In order to use this feature, the full content of the field to be highlighted needs to be stored in the index,
by declaring it through `addStoredField()`:
```php
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addAllFulltextFields();
$this->addStoredField('Content');
}
}
```
To search with highlighting enabled, you need to pass in a custom query parameter.
There's a lot more parameters available for tweaking results detailed on the [Solr reference guide](https://archive.apache.org/dist/lucene/solr/ref-guide/apache-solr-ref-guide-4.10.pdf#page=270).
```php
use My\Namespace\Index\MyIndex;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$index = MyIndex::singleton();
$query = SearchQuery::create()
->addSearchTerm('My Term');
$params = [
'hl' => 'true',
];
$results = $index->search($query, -1, -1, $params);
```
Each result will automatically contain an `Excerpt` property which you can use in your own results template. The searched term is highlighted with an `<em>` tag by default.
> Note: It is recommended to strip out all HTML tags and convert entities on the indexed content,
to avoid matching HTML attributes, and cluttering highlighted content with unparsed HTML.
## Boosting/Weighting
Results aren't all created equal. Matches in some fields are more important than others; for example, a page `Title` might be considered more relevant to the user than terms in the `Content` field.
To account for this, a "weighting" (or "boosting") factor can be applied to each searched field. The default value is `1.0`, anything below that will decrease the relevance, anything above increases it. You can get more information on relevancy at the [Solr wiki](http://wiki.apache.org/solr/SolrRelevancyFAQ).
You can manage the boosting in two ways:
### Boosting on query
To adjust the relative values at the time of querying, pass them in as the third argument to your `addSearchTerm()` call:
```php
use My\Namespace\Index\MyIndex;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use Page;
$query = SearchQuery::create()
->addSearchTerm(
'fire',
null, // don't limit the classes to search
[
Page::class . '_Title' => 1.5,
Page::class . '_Content' => 1.0,
Page::class . '_SecretParagraph' => 0.1,
]
);
$results = MyIndex::singleton()->search($query);
```
This will ensure that `Title` is given higher priority for matches than `Content`, which is well above `SecretParagraph`.
### Boosting on index
Boost values for specific can also be specified directly on the `SolrIndex` class directly.
The following methods can be used to set one or more boosted fields:
* `addBoostedField()` - adds a field with a specific boosted value (defaults to 2)
* `setFieldBoosting()` - if a field has already been added to an index, the boosting
value can be customised, changed, or reset for a single field.
* `addFulltextField()` A boost can be set for a field using the `$extraOptions` parameter
with the key `boost` assigned to the desired value:
```php
use SilverStripe\CMS\Model\SiteTree;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class SolrSearchIndex extends SolrIndex
{
public function init()
{
$this->addClass(SiteTree::class);
// The following methods would all add the same boost of 1.5 to "Title"
$this->addBoostedField('Title', null, [], 1.5);
$this->addFulltextField('Title', null, [
'boost' => 1.5,
]);
$this->addFulltextField('Title');
$this->setFieldBoosting(SiteTree::class . '_Title', 1.5);
}
}
```
## Indexing related objects
To add a related object to your index.
## Subsites
When you are utilising the [subsites module](https://github.com/silverstripe/silverstripe-subsites) you
may want to add [boosting](#boosting/weighting) to results from the current subsite. To do so, you'll
need to use [eDisMax](https://lucene.apache.org/solr/guide/6_6/the-extended-dismax-query-parser.html)
and the supporting parameters `bq` and `bf`. You should add the following to your `SolrIndex`
extension:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use SilverStripe\Subsites\Model\Subsite;
public function search(SearchQuery $query, $offset = -1, $limit = -1, $params = [])) {
$params = array_merge($params, [
'defType' => 'edismax', // turn on eDisMax
'bq' => '_subsite:'.Subsite::currentSubsiteID(), // boost-query on current subsite ID
'bf' => '_subsite^2' // double the score of any document with that subsite ID
]);
return parent::search($query, $offset, $limit, $params);
}
```
## Custom field types
Solr supports custom field type definitions which are written to its XML schema. Many standard ones are already included
in the default schema. As the XML file is generated dynamically, we can add our own types by overloading the template
responsible for it: `types.ss`.
In the following example, we read our type definitions from a new file `mysite/solr/templates/types.ss` instead:
```php
use SilverStripe\Control\Director;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function getTypes()
{
return $this->renderWith(Director::baseFolder() . '/mysite/solr/templates/types.ss');
}
}
```
It's usually best to start with the existing definitions, and adjust from there. You can both add your own types and adjust the behaviour of existing definitions.
### Perform filtering on index
An example of something you can achieve with this is to move synonym filtering from performed on query, to being performed on index. To do this, you'd take
```xml
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
```
from inside the `<analyzer type="query">` block and move it to the `<analyzer type="index">` block. This can be advantageous as Solr does a better job of processing synonyms at index; however, it does mean that it requires a full Reindex to make a change, which - depending on the size of your site - could be overkill. See [this article](https://nolanlawson.com/2012/10/31/better-synonym-handling-in-solr/) for a good breakdown.
### Searching for words containing numbers
By default, the module is configured to split words containing numbers into multiple tokens. For example, the word "A1" would be interpreted as "A" "1", and since "a" is a common stopword, the term "A1" will be excluded from search.
To allow searches on words containing numeric tokens, you'll need to change the behaviour of the `WordDelimiterFilterFactory` with an overloaded template as described above. Each instance of `<filter class="solr.WordDelimiterFilterFactory">` needs to include the following attributes and values:
- add `splitOnNumerics="0"` on all `WordDelimiterFilterFactory` fields
- change `catenateNumbers="1"` to `catenateNumbers="0"` on all `WordDelimiterFilterFactory` fields
### Searching for macrons and other Unicode characters
The `ASCIIFoldingFilterFactory` filter converts alphabetic, numeric, and symbolic Unicode characters which are not in the Basic Latin Unicode block (the first 127 ASCII characters) to their ASCII equivalents, if one exists.
Find the fields in your overloaded `types.ss` that you want to enable this behaviour in, for example inside the `<fieldType name="htmltext">` block, add the following to both its index analyzer and query analyzer records.
```xml
<filter class="solr.ASCIIFoldingFilterFactory"/>
```
## Text extraction
Solr provides built-in text extraction capabilities for PDF and Office documents, and numerous other formats, through
the `ExtractingRequestHandler` API (see [the Solr wiki entry](http://wiki.apache.org/solr/ExtractingRequestHandler).
If you're using a default Solr installation, it's most likely already bundled and set up. But if you plan on running the
Solr server integrated into this module, you'll need to download the libraries and link them first. Run the following
commands from the webroot:
```
wget http://archive.apache.org/dist/lucene/solr/4.10.4/solr-4.10.4.tgz
tar -xvzf solr-4.10.4.tgz
mkdir .solr/PageSolrIndexboot/dist
mkdir .solr/PageSolrIndexboot/contrib
cp solr-4.10.4/dist/solr-cell-4.10.4.jar .solr/PageSolrIndexboot/dist/
cp -R solr-4.10.4/contrib/extraction .solr/PageSolrIndexboot/contrib/
rm -rf solr-4.10.4 solr-4.10.4.tgz
```
Create a custom `solrconfig.xml` (see [File-based configuration](03_configuration.md#file-based-configuration)).
Add the following XML configuration:
```xml
<lib dir="./contrib/extraction/lib/" />
<lib dir="./dist" />
```
Now run a [Solr configure](03_configuration.md#solr-configure). You can use Solr text extraction either directly through
the HTTP API, or through the [Text extraction module](https://github.com/silverstripe-labs/silverstripe-textextraction).

14
docs/en/06_troubleshooting.md Normal file
View File

@ -0,0 +1,14 @@
# Troubleshooting
## Common gotchas
* By default number-letter boundaries are treated as a word boundary. For example, `A1` is two words - `a` and `1` - when Solr parses the search term.
* Special characters and operators are not correctly escaped
* Multi-word synonym issues
* When Dolr indexes are reconfigured and reindexed, their content is trashed and rebuilt
### CWP-specific
* `solrconfig.xml` customisations fail silently
* Developers aren’t able to test raw queries or see output via the
[Solr admin interface](02_setup.md#solr-admin)

701
docs/en/Solr.md
View File

@ -1,701 +0,0 @@
# Solr connector for SilverStripe fulltextsearch module
## Introduction
The fulltextsearch module includes support for connecting to Solr.
It works with Solr in multi-core mode. It needs to be able to update Solr configuration files, and has modes for
doing this by direct file access (when Solr shares a server with SilverStripe) and by WebDAV (when it's on a different
server).
See the helpful [Solr Tutorial](http://lucene.apache.org/solr/4_5_1/tutorial.html), for more on cores
and querying.
## Requirements
Since Solr is Java based, it requires Java 1.5 or greater installed.
When you're installing it yourself, it also requires a servlet container such as Tomcat, Jetty, or Resin. For
development testing there is a standalone version that comes bundled with Jetty (see below).
See the official [Solr installation docs](http://wiki.apache.org/solr/SolrInstall) for more information.
Note that these requirements are for the Solr server environment, which doesn't have to be the same physical machine
as the SilverStripe webhost.
## Installation (Local)
### Get the Solr server
```
composer require silverstripe/fulltextsearch-localsolr
```
### Start the server (via CLI, in a separate terminal window or background process)
```
cd fulltextsearch-localsolr/server/
java -jar start.jar
```
### Configure the fulltextsearch Solr component to use the local server
Configure Solr in file mode. The 'path' directory has to be writeable
by the user the Solr search server is started with (see below).
```php
// File: mysite/_config.php:
use SilverStripe\FullTextSearch\Solr\Solr;
Solr::configure_server([
'host' => 'localhost',
'indexstore' => [
'mode' => 'file',
'path' => BASE_PATH . '/.solr'
]
]);
```
All possible parameters incl optional ones with example values:
```php
// File: mysite/_config.php:
use SilverStripe\FullTextSearch\Solr\Solr;
Solr::configure_server([
'host' => 'localhost', // default: localhost | The host or IP Solr is listening on
'port' => '8983', // default: 8983 | The port Solr is listening on
'path' => '/solr', // default: /solr | The suburl the solr service is available on
'version' => '4', // default: 4 | Solr server version - currently only 3 and 4 supported
'service' => 'Solr4Service', // default: depends on version, Solr3Service for 3, Solr4Service for 4 | the class that provides actual communcation to the Solr server
'extraspath' => BASE_PATH .'/fulltextsearch/conf/solr/4/extras/', // default: <basefolder>/fulltextsearch/conf/solr/{version}/extras/ | Absolute path to the folder containing templates which are used for generating the schema and field definitions.
'templates' => BASE_PATH . '/fulltextsearch/conf/solr/4/templates/', // default: <basefolder>/fulltextsearch/conf/solr/{version}/templates/ | Absolute path to the configuration default files, e.g. solrconfig.xml
'indexstore' => [
'mode' => 'file', // a classname which implements SolrConfigStore, or 'file' or 'webdav'
'path' => BASE_PATH . '/.solr', // The (locally accessible) path to write the index configurations to OR The suburl on the solr host that is set up to accept index configurations via webdav
'remotepath' => '/opt/solr/config', // default (file mode only): same as 'path' above | The path that the Solr server will read the index configurations from
'auth' => 'solr:solr', // default: none | Webdav only - A username:password pair string to use to auth against the webdav server
'port' => '80' // default: same as solr port | The port for WebDAV if different from the Solr port
]
]);
```
Note: We recommend to put the `indexstore.path` directory outside of the webroot.
If you place it inside of the webroot (as shown in the example),
please ensure its contents are not accessible through the webserver.
This can be achieved by server configuration, or (in most configurations)
also by marking the folder as hidden via a "dot" prefix.
## Configuration
### Create an index
```php
// File: mysite/code/MyIndex.php:
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addAllFulltextFields();
}
}
```
### Create the index schema
The PHP-based index definition is an abstraction layer for the actual Solr XML configuration.
In order to create or update it, you need to run the `Solr_Configure` task.
```
vendor/bin/sake dev/tasks/Solr_Configure
```
Based on the sample configuration above, this command will do the following:
- Create a `<BASE_PATH>/.solr/MyIndex` folder
- Copy configuration files from `vendor/silverstripe/fulltextsearch/conf/extras/` to `<BASE_PATH>/.solr/MyIndex/conf`
- Generate a `schema.xml`, and place it it in `<BASE_PATH>/.solr/MyIndex/conf`
If you call the task with an existing index folder,
it will overwrite all files from their default locations,
regenerate the `schema.xml`, and ask Solr to reload the configuration.
You can use the same command for updating an existing schema,
which will automatically apply without requiring a Solr server restart.
### Reindex
After configuring Solr, you have the option to add your existing
content to its indices. Run the following command:
```
vendor/bin/sake dev/tasks/Solr_Reindex
```
This will delete and rebuild all indices. Depending on your data,
this can take anywhere from minutes to hours.
Keep in mind that the normal mode of updating indices is
based on ORM manipulations of the underlying data.
For example, calling `$myPage->write()` will automatically
update the index entry for this record (and all its variants).
This task has the following options:
- `verbose`: Debug information
Internally, depending on what job processing backend you have configured (such as queuedjobs)
individual tasks for re-indexing groups of records may either be performed behind the scenes
as crontasks, or via separate processes initiated by the current request.
Internally groups of records are grouped into sizes of 200. You can configure this
group sizing by using the `Solr_Reindex.recordsPerRequest` config.
```yaml
SilverStripe\FullTextSearch\Solr\Tasks\Solr_Reindex:
recordsPerRequest: 150
```
Note: The Solr indexes will be stored as binary files inside your SilverStripe project.
You can also copy the `thirdparty/` solr directory somewhere else,
just set the `path` value in `mysite/_config.php` to point to the new location.
You can also run the reindex task through a web request.
By default, the web request won't receive any feedback while its running.
Depending on your PHP and web server configuration,
the web request itself might time out, but the reindex continues anyway.
This is possible because the actual index operations are run as separate
PHP sub-processes inside the main web request.
### File-based configuration (solrconfig.xml etc)
Many aspects of Solr are configured outside of the `schema.xml` file
which SilverStripe generates based on the index PHP file.
For example, stopwords are placed in their own `stopwords.txt` file,
and spell checks are configured in `solrconfig.xml`.
By default, these files are copied from the `fulltextsearch/conf/extras/`
directory over to the new index location. In order to use your own files,
copy these files into a location of your choosing (for example `mysite/data/solr/`),
and tell Solr to use this folder with the `extraspath` configuration setting.
```php
// mysite/_config.php
use SilverStripe\Control\Director;
use SilverStripe\FullTextSearch\Solr\Solr;
Solr::configure_server([
// ...
'extraspath' => Director::baseFolder() . '/mysite/data/solr/',
]);
```
Please run the `Solr_Configure` task for the changes to take effect.
Note: You can also define those on an index-by-index basis by
implementing `SolrIndex->getExtrasPath()`.
### Custom Types
Solr supports custom field type definitions which are written to its XML schema.
Many standard ones are already included in the default schema.
As the XML file is generated dynamically, we can add our own types
by overloading the template responsible for it: `types.ss`.
In the following example, we read out type definitions
from a new file `mysite/solr/templates/types.ss` instead:
```php
use SilverStripe\Control\Director;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function getTypes()
{
return $this->renderWith(Director::baseFolder() . '/mysite/solr/templates/types.ss');
}
}
```
#### Searching for words containing numbers
By default, the fulltextmodule is configured to split words containing numbers into multiple tokens. For example, the word "A1" would be interpreted as "A" "1"; since "a" is a common stopword, the term "A1" will be excluded from search.
To allow searches on words containing numeric tokens, you'll need to update your overloaded template to change the behaviour of the WordDelimiterFilterFactory. Each instance of `<filter class="solr.WordDelimiterFilterFactory">` needs to include the following attributes and values:
* add splitOnNumerics="0" on all WordDelimiterFilterFactory fields
* change catenateOnNumbers="1" on all WordDelimiterFilterFactory fields
Update your index to point to your overloaded template using the method described above.
#### Searching for macrons and other Unicode characters
The "ASCIIFoldingFilterFactory" filter converts alphabetic, numeric, and symbolic Unicode characters which are not in the Basic Latin Unicode block (the first 127 ASCII characters) to their ASCII equivalents, if one exists.
Find the fields in your overloaded `types.ss` that you want to enable this behaviour in. EG:
```xml
<fieldType name="htmltext" class="solr.TextField" ... >
```
Add the following to both its index analyzer and query analyzer records.
```xml
<filter class="solr.ASCIIFoldingFilterFactory"/>
```
Update your index to point to your overloaded template using the method described above.
### Spell Checking ("Did you mean...")
Solr has various spell checking strategies (see the ["SpellCheckComponent" docs](http://wiki.apache.org/solr/SpellCheckComponent)), all of which are configured through `solrconfig.xml`.
In the default config which is copied into your index,
spell checking data is collected from all fulltext fields
(everything you added through `SolrIndex->addFulltextField()`).
The values of these fields are collected in a special `_text` field.
```php
use SilverStripe\FullTextSearch\Search\Queries;
$index = new MyIndex();
$query = new SearchQuery();
$query->addSearchTerm('My Term');
$params = [
'spellcheck' => 'true',
'spellcheck.collate' => 'true',
];
$results = $index->search($query, -1, -1, $params);
$results->spellcheck;
```
The built-in `_text` data is better than nothing, but also has some problems:
Its heavily processed, for example by stemming filters which butcher words.
So misspelling "Govnernance" will suggest "govern" rather than "Governance".
This can be fixed by aggregating spell checking data in a separate
```php
use SilverStripe\CMS\Model\SiteTree;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
// ...
$this->addCopyField(SiteTree::class . '_Title', 'spellcheckData');
$this->addCopyField(SomeModel::class . '_Title', 'spellcheckData');
$this->addCopyField(SiteTree::class . '_Content', 'spellcheckData');
$this->addCopyField(SomeModel::class . '_Content', 'spellcheckData');
}
// ...
public function getFieldDefinitions()
{
$xml = parent::getFieldDefinitions();
$xml .= "\n\n\t\t<!-- Additional custom fields for spell checking -->";
$xml .= "\n\t\t<field name='spellcheckData' type='textSpellHtml' indexed='true' stored='false' multiValued='true' />";
return $xml;
}
}
```
Now you need to tell solr to use our new field for gathering spelling data.
In order to customize the spell checking configuration,
create your own `solrconfig.xml` (see "File-based configuration").
In there, change the following directive:
```xml
<!-- ... -->
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<!-- ... -->
<str name="field">spellcheckData</str>
</searchComponent>
```
Don't forget to copy the new configuration via a call to the `Solr_Configure`
task, and reindex your data before using the spell checker.
### Limiting search fields
Solr has a way of specifying which fields to search on. You specify these
fields as a parameter to `SearchQuery`.
In the following example, we're telling Solr to *only* search the
`Title` and `Content` fields. Note that the fields must be specified in
the search parameters as "composite fields", which means they should be
specified in the form of `{table}_{field}`.
These fields are defined in the schema.xml file that gets sent to Solr.
```php
use SilverStripe\CMS\Model\SiteTree;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addClassFilter(Page::class);
$query->addSearchTerm('someterms', [SiteTree::class . '_Title', SiteTree::class . '_Content']);
$result = singleton(SolrSearchIndex::class)->search($query, -1, -1);
// the request to Solr would be:
// q=(SiteTree_Title:Lorem+OR+SiteTree_Content:Lorem)
```
### Configuring boosts
There are several ways in which you can configure boosting on search fields or terms.
#### Boosting on search query
Solr has a way of specifying which fields should be boosted as a parameter to `SearchQuery`.
This means if you boost a certain field, search query matches on that field will be considered
higher relevance than other fields with matches, and therefore those results will be closer
to the top of the results.
In this example, we enter "Lorem" as the search term, and boost the `Content` field:
```php
use SilverStripe\CMS\Model\SiteTree;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addClassFilter(Page::class);
$query->addSearchTerm('Lorem', null, [SiteTree::class . '_Content' => 2]);
$result = singleton(SolrSearchIndex::class)->search($query, -1, -1);
// the request to Solr would be:
// q=SiteTree_Content:Lorem^2
```
More information on [relevancy on the Solr wiki](http://wiki.apache.org/solr/SolrRelevancyFAQ).
### Boosting on index fields
Boost values for specific can also be specified directly on the `SolrIndex` class directly.
The following methods can be used to set one or more boosted fields:
* `SolrIndex::addBoostedField` Adds a field with a specific boosted value (defaults to 2)
* `SolrIndex::setFieldBoosting` If a field has already been added to an index, the boosting
value can be customised, changed, or reset for a single field.
* `SolrIndex::addFulltextField` A boost can be set for a field using the `$extraOptions` parameter
with the key `boost` assigned to the desired value.
For example:
```php
use SilverStripe\CMS\Model\SiteTree;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class SolrSearchIndex extends SolrIndex
{
public function init()
{
$this->addClass(SiteTree::class);
$this->addAllFulltextFields();
$this->addFilterField('ShowInSearch');
$this->addBoostedField('Title', null, [], 1.5);
$this->setFieldBoosting(SiteTree::class . '_SearchBoost', 2);
}
}
```
### Custom Types
Solr supports custom field type definitions which are written to its XML schema.
Many standard ones are already included in the default schema.
As the XML file is generated dynamically, we can add our own types
by overloading the template responsible for it: `types.ss`.
In the following example, we read out type definitions
from a new file `mysite/solr/templates/types.ss` instead:
```php
use SilverStripe\Control\Director;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function getTemplatesPath()
{
return Director::baseFolder() . '/mysite/solr/templates/';
}
}
```
### Highlighting
Solr can highlight the searched terms in context of the matched content,
to help users determine the relevancy of results (e.g. in which part of a sentence
the term is used). In order to use this feature, the full content of the
field to be highlighted needs to be stored in the index,
by declaring it through `addStoredField()`.
```php
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addAllFulltextFields();
$this->addStoredField('Content');
}
}
```
To search with highlighting enabled, you need to pass in a custom query parameter.
There's a lot more parameters to tweak results on the [Solr Wiki](http://wiki.apache.org/solr/HighlightingParameters).
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$index = new MyIndex();
$query = new SearchQuery();
$query->addSearchTerm('My Term');
$results = $index->search($query, -1, -1, ['hl' => 'true']);
```
Each result will automatically contain an "Excerpt" property
which you can use in your own results template.
The searched term is highlighted with an `<em>` tag by default.
Note: It is recommended to strip out all HTML tags and convert entities on the indexed content,
to avoid matching HTML attributes, and cluttering highlighted content with unparsed HTML.
### Adding additional information into search results
Inside the SolrIndex::search() function, the third-party library solr-php-client
is used to send data to Solr and parse the response. Additional information can
be pulled from this response and added to your results object for use in templates
using the `updateSearchResults()` extension hook.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$index = new MyIndex();
$query = new SearchQuery();
$query->addSearchTerm('My Term');
$results = $index->search($query, -1, -1, [
'facet' => 'true',
'facet.field' => 'SiteTree_ClassName',
]);
```
By adding facet fields into the query parameters, our response object from Solr
now contains some additional information that we can add into the results sent
to the page.
```php
use SilverStripe\Core\Extension;
use SilverStripe\View\ArrayData;
use SilverStripe\ORM\ArrayList;
class MyResultsExtension extends Extension
{
/**
* Adds extra information from the solr-php-client repsonse
* into our search results.
* @param ArrayData $results The ArrayData that will be used to generate search
* results pages.
* @param stdClass $response The solr-php-client response object.
*/
public function updateSearchResults($results, $response)
{
if (!isset($response->facet_counts) || !isset($response->facet_counts->facet_fields)) {
return;
}
$facetCounts = ArrayList::create(array());
foreach($response->facet_counts->facet_fields as $name => $facets) {
$facetDetails = ArrayData::create([
'Name' => $name,
'Facets' => ArrayList::create([]),
]);
foreach($facets as $facetName => $facetCount) {
$facetDetails->Facets->push(ArrayData::create([
'Name' => $facetName,
'Count' => $facetCount,
]));
}
$facetCounts->push($facetDetails);
}
$results->setField('FacetCounts', $facetCounts);
}
}
```
We can now access the facet information inside our templates.
### Adding Analyzers, Tokenizers and Token Filters
When a document is indexed, its individual fields are subject to the analyzing and tokenizing filters that can transform and normalize the data in the fields. For example — removing blank spaces, removing html code, stemming, removing a particular character and replacing it with another
(see [Solr Wiki](http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters)).
Example: Replace synonyms on indexing (e.g. "i-pad" with "iPad")
```php
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addField('Content');
$this->addAnalyzer('Content', 'filter', ['class' => 'solr.SynonymFilterFactory']);
}
}
```
Generates the following XML schema definition:
```xml
<field name="Page_Content" ...>
<filter class="solr.SynonymFilterFactory" synonyms="syn.txt" ignoreCase="true" expand="false"/>
</field>
```
### Text Extraction
Solr provides built-in text extraction capabilities for PDF and Office documents,
and numerous other formats, through the `ExtractingRequestHandler` API
(see http://wiki.apache.org/solr/ExtractingRequestHandler).
If you're using a default Solr installation, it's most likely already
bundled and set up. But if you plan on running the Solr server integrated
into this module, you'll need to download the libraries and link the first.
```
wget http://archive.apache.org/dist/lucene/solr/3.1.0/apache-solr-3.1.0.tgz
mkdir tmp
tar -xvzf apache-solr-3.1.0.tgz
mkdir .solr/PageSolrIndexboot/dist
mkdir .solr/PageSolrIndexboot/contrib
cp apache-solr-3.1.0/dist/apache-solr-cell-3.1.0.jar .solr/PageSolrIndexboot/dist/
cp -R apache-solr-3.1.0/contrib/extraction .solr/PageSolrIndexboot/contrib/
rm -rf apache-solr-3.1.0 apache-solr-3.1.0.tgz
```
Create a custom `solrconfig.xml` (see "File-based configuration").
Add the following XML configuration.
```xml
<lib dir="./contrib/extraction/lib/" />
<lib dir="./dist" />
```
Now apply the configuration:
```
vendor/bin/sake dev/tasks/Solr_Configure
```
Now you can use Solr text extraction either directly through the HTTP API,
or indirectly through the ["textextraction" module](https://github.com/silverstripe-labs/silverstripe-textextraction).
## Adding DataObject classes to Solr search
If you create a class that extends `DataObject` (and not `Page`) then it won't be automatically added to the search
index. You'll have to make some changes to add it in.
So, let's take an example of `StaffMember`:
```php
use SilverStripe\Control\Controller;
use SilverStripe\ORM\DataObject;
class StaffMember extends DataObject
{
private static $db = [
'Name' => 'Varchar(255)',
'Abstract' => 'Text',
'PhoneNumber' => 'Varchar(50)',
];
public function Link($action = 'show')
{
return Controller::join_links('my-controller', $action, $this->ID);
}
public function getShowInSearch()
{
return 1;
}
}
```
This `DataObject` class has the minimum code necessary to allow it to be viewed in the site search.
`Link()` will return a URL for where a user goes to view the data in more detail in the search results.
`Name` will be used as the result title, and `Abstract` the summary of the staff member which will show under the
search result title.
`getShowInSearch` is required to get the record to show in search, since all results are filtered by `ShowInSearch`.
So with that, let's create a new class called `MySolrSearchIndex`:
```php
use StaffMember;
use SilverStripe\CMS\Model\SiteTree;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MySolrSearchIndex extends SolrIndex {
public function init()
{
$this->addClass(SiteTree::class);
$this->addClass(StaffMember::class);
$this->addAllFulltextFields();
$this->addFilterField('ShowInSearch');
}
}
```
This is a copy/paste of the existing configuration but with the addition of `StaffMember`.
Once you've created the above classes and run `flush=1`, access `dev/tasks/Solr_Configure` and `dev/tasks/Solr_Reindex`
to tell Solr about the new index you've just created. This will add `StaffMember` and the text fields it has to the
index. Now when you search on the site using `MySolrSearchIndex->search()`,
the `StaffMember` results will show alongside normal `Page` results.
## Debugging
### Using the web admin interface
You can visit `http://localhost:8983/solr`, which will show you a list
to the admin interfaces of all available indices.
There you can search the contents of the index via the native SOLR web interface.
It is possible to manually replicate the data automatically sent
to Solr when saving/publishing in SilverStripe,
which is useful when debugging front-end queries,
see `thirdparty/fulltextsearch/server/silverstripe-solr-test.xml`.
```
java -Durl=http://localhost:8983/solr/MyIndex/update/ -Dtype=text/xml -jar post.jar silverstripe-solr-test.xml
```
## FAQ
### How do I use date ranges where dates might not be defined?
The Solr index updater only includes dates with values,
so the field might not exist in all your index entries.
A simple bounded range query (`<field>:[* TO <date>]`) will fail in this case.
In order to query the field, reverse the search conditions and exclude the ranges you don't want:
```php
// Wrong: Filter will ignore all empty field values
$myQuery->addFilter('fieldname', new SearchQuery_Range('*', 'somedate'));
// Better: Exclude the opposite range
$myQuery->addExclude('fieldname', new SearchQuery_Range('somedate', '*'));
```

386
docs/en/index.md
View File

@ -1,386 +0,0 @@
## Introduction
This is a module aimed at adding support for standalone fulltext search engines to SilverStripe.
It contains several layers:
* A fulltext API, ignoring the actual provision of fulltext searching
* A connector API, providing common code to allow connecting a fulltext searching engine to the fulltext API, and
* Some connectors for common fulltext searching engines.
## Reasoning
There are several fulltext search engines that work in a similar manner. They build indexes of denormalized data that
is then searched through using some custom query syntax.
Traditionally, fulltext search connectors for SilverStripe have attempted to hide this design, instead presenting
fulltext searching as an extension of the object model. However the disconnect between the fulltext search engine's
design and the object model meant that searching was inefficient. The abstraction would also often break and it was
hard to then figure out what was going on.
This module instead provides the ability to define those indexes and queries in PHP. The indexes are defined as a mapping
between the SilverStripe object model and the connector-specific fulltext engine index model. This module then interrogates model metadata
to build the specific index definition.
It also hooks into SilverStripe framework in order to update the indexes when the models change and connectors then convert those index and query definitions
into fulltext engine specific code.
The intent of this module is not to make changing fulltext search engines seamless. Where possible this module provides
common interfaces to fulltext engine functionality, abstracting out common behaviour. However, each connector also
offers its own extensions, and there is some behaviour (such as getting the fulltext search engines installed, configured
and running) that each connector deals with itself, in a way best suited to that search engine's design.
## Disabling automatic configuration
If you have this module installed but do not have a Solr server running, you can disable the database manipulation
hooks that trigger automatic index updates:
```yaml
# File: mysite/_config/search.yml
---
Name: mysitesearch
---
SilverStripe\FullTextSearch\Search\Updaters\SearchUpdater:
enabled: false
```
## Basic usage
### Quick start
If you are running on a Linux-based system, you can get up and running quickly with the quickstart script, like so:
```bash
composer require silverstripe/fulltextsearch && vendor/bin/fulltextsearch_quickstart
```
This will:
- Install the required Java SDK (using `apt-get` or `yum`)
- Install Solr 4
- Set up a daemon to run Solr on startup
- Start Solr
- Configure Solr in your `_config.php` (and create one if you don't have one)
- Create a DefaultIndex
- Run a [Solr Configure](03_configuration.md#solr-configure) and a [Solr Reindex](03_configuration.md#solr-reindex)
If you have the [CMS module](https://github.com/silverstripe/silverstripe-cms) installed, you will be able to simply add `$SearchForm` to your template to add a Solr search form. Default configuration is added via the [`ContentControllerExtension`](/src/Solr/Control/ContentControllerExtension.php) and alternative [`SearchForm`](/src/Solr/Forms/SearchForm.php).
Ensure that you _don't_ have `SilverStripe\ORM\Search\FulltextSearchable::enable()` set in `_config.php`, as the `SearchForm` action provided by that class will conflict.
You can override the default template with a new one at `templates/Layout/Page_results_solr.ss`.
### "Slow" start
Otherwise, basic usage is a four step process:
1). Define an index in SilverStripe (Note: The specific connector index instance - that's what defines which engine gets used)
```php
// File: mysite/code/MyIndex.php:
use Page;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addFulltextField('Title');
$this->addFulltextField('Content');
}
}
```
You can also skip listing all searchable fields, and have the index
figure it out automatically via `addAllFulltextFields()`.
2). Add something to the index (Note: You can also just update an existing document in the CMS. but adding _existing_ objects to the index is connector specific)
```php
$page = Page::create(['Content' => 'Help me. My house is on fire. This is less than optimal.']);
$page->write();
```
Note: There's usually a connector-specific "reindex" task for this.
3). Build a query
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addSearchTerm('My house is on fire');
```
4). Apply that query to an index
```php
$results = singleton(MyIndex::class)->search($query);
```
Note that for most connectors, changes won't be searchable until _after_ the request that triggered the change.
The return value of a `search()` call is an object which contains a few properties:
* `Matches`: ArrayList of the current "page" of search results.
* `Suggestion`: (optional) Any suggested spelling corrections in the original query notation
* `SuggestionNice`: (optional) Any suggested spelling corrections for display (without query notation)
* `SuggestionQueryString` (optional) Link to repeat the search with suggested spelling corrections
## Controllers and Templates
In order to render search results, you need to return them from a controller.
You can also drive this through a form response through standard SilverStripe forms.
In this case we simply assume there's a GET parameter named `q` with a search term present.
```php
use SilverStripe\CMS\Controllers\ContentController;
use SilverStripe\Control\HTTPRequest;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
class PageController extends ContentController
{
private static $allowed_actions = [
'search',
];
public function search(HTTPRequest $request)
{
$query = new SearchQuery();
$query->addSearchTerm($request->getVar('q'));
return $this->renderWith([
'SearchResult' => singleton(MyIndex::class)->search($query)
]);
}
}
```
In your template (e.g. `Page_results.ss`) you can access the results and loop through them.
They're stored in the `$Matches` property of the search return object.
```ss
<% if $SearchResult.Matches %>
<h2>Results for &quot;{$Query}&quot;</h2>
<p>Displaying Page $SearchResult.Matches.CurrentPage of $SearchResult.Matches.TotalPages</p>
<ol>
<% loop $SearchResult.Matches %>
<li>
<h3><a href="$Link">$Title</a></h3>
<p><% if $Abstract %>$Abstract.XML<% else %>$Content.ContextSummary<% end_if %></p>
</li>
<% end_loop %>
</ol>
<% else %>
<p>Sorry, your search query did not return any results.</p>
<% end_if %>
```
Please check the [pagination guide](https://docs.silverstripe.org/en/4/developer_guides/templates/how_tos/pagination/)
in the main SilverStripe documentation to learn how to paginate through search results.
## Automatic Index Updates
Every change, addition or removal of an indexed class instance triggers an index update through a
"processor" object. The update is transparently handled through inspecting every executed database query
and checking which database tables are involved in it.
Index updates usually are executed in the same request which caused the index to become "dirty".
For example, a CMS author might have edited a page, or a user has left a new comment.
In order to minimise delays to those users, the index update is deferred until after
the actual request returns to the user, through PHP's `register_shutdown_function()` functionality.
If the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module is installed,
updates are queued up instead of executed in the same request. Queue jobs are usually processed every minute.
Large index updates will be batched into multiple queue jobs to ensure a job can run to completion within
common execution constraints (memory and time limits). You can check the status of jobs in
an administrative interface under `admin/queuedjobs/`.
## Manual Index Updates
Manual updates are connector specific, please check the connector docs for details.
## Searching Specific Fields
By default, the index searches through all indexed fields.
This can be limited by arguments to the `addSearchTerm()` call.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addSearchTerm('My house is on fire', [Page::class . '_Title']);
// No results, since we're searching in title rather than page content
$results = singleton(MyIndex::class)->search($query);
```
## Searching Value Ranges
Most values can be expressed as ranges, most commonly dates or numbers.
To search for a range of values rather than an exact match,
use the `SearchQuery_Range` class. The range can include bounds on both sides,
or stay open ended by simply leaving the argument blank.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery_Range;
$query = new SearchQuery();
$query->addSearchTerm('My house is on fire');
// Only include documents edited in 2011 or earlier
$query->addFilter(Page::class . '_LastEdited', new SearchQuery_Range(null, '2011-12-31T23:59:59Z'));
$results = singleton(MyIndex::class)->search($query);
```
Note: At the moment, the date format is specific to the search implementation.
## Searching Empty or Existing Values
Since there's a type conversion between the SilverStripe database, object properties
and the search index persistence, its often not clear which condition is searched for.
Should it equal an empty string, or only match if the field wasn't indexed at all?
The `SearchQuery` API has the concept of a "missing" and "present" field value for this:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addSearchTerm('My house is on fire');
// Needs a value, although it can be false
$query->addFilter(Page::class . '_ShowInMenus', SearchQuery::$present);
$results = singleton(MyIndex::class)->search($query);
```
## Indexing Multiple Classes
An index is a denormalized view of your data, so can hold data from more than one model.
As you can only search one index at a time, all searchable classes need to be included.
```php
// File: mysite/code/MyIndex.php
use SilverStripe\FullTextSearch\Solr\SolrIndex;
use SilverStripe\Security\Member;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addClass(Member::class);
$this->addFulltextField('Content'); // only applies to Page class
$this->addFulltextField('FirstName'); // only applies to Member class
}
}
```
## Using Multiple Indexes
Multiple indexes can be created and searched independently, but if you wish to override an existing
index with another, you can use the `$hide_ancestor` config.
```php
use SilverStripe\Assets\File;
class MyReplacementIndex extends MyIndex
{
private static $hide_ancestor = 'MyIndex';
public function init()
{
parent::init();
$this->addClass(File::class);
$this->addFulltextField('Title');
}
}
```
You can also filter all indexes globally to a set of pre-defined classes if you wish to
prevent any unknown indexes from being automatically included.
```yaml
SilverStripe\FullTextSearch\Search\FullTextSearch:
indexes:
- MyReplacementIndex
- CoreSearchIndex
```
## Indexing Relationships
TODO
## Weighting/Boosting Fields
Results aren't all created equal. Matches in some fields are more important
than others, for example terms in a page title rather than its content
might be considered more relevant to the user.
To account for this, a "weighting" (or "boosting") factor can be applied to each
searched field. The default is 1.0, anything below that will decrease the relevance,
anthing above increases it.
Example:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addSearchTerm(
'My house is on fire',
null,
[
Page::class . '_Title' => 1.5,
Page::class . '_Content' => 1.0,
]
);
$results = singleton(MyIndex::class)->search($query);
```
## Filtering
## Connectors
### Solr
See Solr.md
### Sphinx
Not written yet
## FAQ
### How do I exclude draft pages from the index?
By default, the `SearchUpdater` class indexes all available "variant states",
so in the case of the `Versioned` extension, both "draft" and "live".
For most cases, you'll want to exclude draft content from your search results.
You can either prevent the draft content from being indexed in the first place,
by adding the following to your `SearchIndex->init()` method:
```php
use SilverStripe\FullTextSearch\Search\Variants\SearchVariantVersioned;
$this->excludeVariantState([SearchVariantVersioned::class => 'Stage']);
```
Alternatively, you can index draft content, but simply exclude it from searches.
This can be handy to preview search results on unpublished content, in case a CMS author is logged in.
Before constructing your `SearchQuery`, conditionally switch to the "live" stage:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use SilverStripe\Security\Permission;
use SilverStripe\Versioned\Versioned;
if (!Permission::check('CMS_ACCESS_CMSMain')) {
Versioned::set_stage(Versioned::LIVE);
}
$query = new SearchQuery();
// ...
```
### How do I write nested/complex filters?
TODO

2
src/Solr/SolrIndex.php
View File

@ -137,7 +137,7 @@ abstract class SolrIndex extends SearchIndex
*
* @param string $field
* @param string $type
* @param Array $params Parameters for the analyzer, usually at least a "class"
* @param array $params parameters for the analyzer, usually at least a "class"
*/
public function addAnalyzer($field, $type, $params)
{