tonyair
/
silverstripe-fulltextsearch
mirror of https://github.com/silverstripe/silverstripe-fulltextsearch
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

prune module scope, add boosting docs

This commit is contained in:
Andrew Aitken-Fincham 2018-05-24 15:24:37 +01:00 committed by Daniel Hensby
parent 8b5a3dd2e7
commit 51656d94b9
No known key found for this signature in database
GPG Key ID: D8DEBC4C8E7BC8B9
9 changed files with 290 additions and 325 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/en/00_index.md
View File

@ -26,3 +26,4 @@
- [Adding new fields](04_advanced_configuration/47_adding_new_fields.md)
- Troubleshooting
- [Gotchas](05_troubleshooting/50_common_gotchas.md)
- [Test Anchor](05_troubleshooting.md#test-anchor)

329
docs/en/01_getting_started/10_module_scope.md
View File

@ -1,20 +1,20 @@
## Introduction
# Introduction
This is a module aimed at adding support for standalone fulltext search engines to SilverStripe.
It contains several layers:
* A fulltext API, ignoring the actual provision of fulltext searching
* A connector API, providing common code to allow connecting a fulltext searching engine to the fulltext API, and
* Some connectors for common fulltext searching engines.
* A connector API, providing common code to allow connecting a fulltext searching engine to the fulltext API
* Some connectors for common fulltext searching engines (currently only [Apache Solr](http://lucene.apache.org/solr/))
## Reasoning
There are several fulltext search engines that work in a similar manner. They build indexes of denormalized data that
is then searched through using some custom query syntax.
are then searched through using some custom query syntax.
Traditionally, fulltext search connectors for SilverStripe have attempted to hide this design, instead presenting
fulltext searching as an extension of the object model. However the disconnect between the fulltext search engine's
fulltext searching as an extension of the object model. However, the disconnect between the fulltext search engine's
design and the object model meant that searching was inefficient. The abstraction would also often break and it was
hard to then figure out what was going on.
@ -29,322 +29,3 @@ The intent of this module is not to make changing fulltext search engines seamle
common interfaces to fulltext engine functionality, abstracting out common behaviour. However, each connector also
offers its own extensions, and there is some behaviour (such as getting the fulltext search engines installed, configured
and running) that each connector deals with itself, in a way best suited to that search engine's design.
## Disabling automatic configuration
If you have this module installed but do not have a Solr server running, you can disable the database manipulation
hooks that trigger automatic index updates:
```yaml
# File: mysite/_config/search.yml
---
Name: mysitesearch
---
SilverStripe\FullTextSearch\Search\Updaters\SearchUpdater:
enabled: false
```
## Basic usage
### Quick start
If you are running on a Linux-based system, you can get up and running quickly with the quickstart script, like so:
```bash
composer require silverstripe/fulltextsearch && vendor/bin/fulltextsearch_quickstart
```
This will:
- Install the required Java SDK (using `apt-get` or `yum`)
- Install Solr 4
- Set up a daemon to run Solr on startup
- Start Solr
- Configure Solr in your `_config.php` (and create one if you don't have one)
- Create a DefaultIndex
- Run a [Solr Configure](03_configuration.md#solr-configure) and a [Solr Reindex](03_configuration.md#solr-reindex)
If you have the [CMS module](https://github.com/silverstripe/silverstripe-cms) installed, you will be able to simply add `$SearchForm` to your template to add a Solr search form. Default configuration is added via the [`ContentControllerExtension`](/src/Solr/Control/ContentControllerExtension.php) and alternative [`SearchForm`](/src/Solr/Forms/SearchForm.php).
Ensure that you _don't_ have `SilverStripe\ORM\Search\FulltextSearchable::enable()` set in `_config.php`, as the `SearchForm` action provided by that class will conflict.
You can override the default template with a new one at `templates/Layout/Page_results_solr.ss`.
### "Slow" start
Otherwise, basic usage is a four step process:
3). Build a query
4). Apply that query to an index
```php
$results = singleton(MyIndex::class)->search($query);
```
Note that for most connectors, changes won't be searchable until _after_ the request that triggered the change.
The return value of a `search()` call is an object which contains a few properties:
* `Matches`: ArrayList of the current "page" of search results.
* `Suggestion`: (optional) Any suggested spelling corrections in the original query notation
* `SuggestionNice`: (optional) Any suggested spelling corrections for display (without query notation)
* `SuggestionQueryString` (optional) Link to repeat the search with suggested spelling corrections
## Controllers and Templates
In order to render search results, you need to return them from a controller.
You can also drive this through a form response through standard SilverStripe forms.
In this case we simply assume there's a GET parameter named `q` with a search term present.
```php
use SilverStripe\CMS\Controllers\ContentController;
use SilverStripe\Control\HTTPRequest;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
class PageController extends ContentController
{
private static $allowed_actions = [
'search',
];
public function search(HTTPRequest $request)
{
$query = new SearchQuery();
$query->addSearchTerm($request->getVar('q'));
return $this->renderWith([
'SearchResult' => singleton(MyIndex::class)->search($query)
]);
}
}
```
In your template (e.g. `Page_results.ss`) you can access the results and loop through them.
They're stored in the `$Matches` property of the search return object.
```ss
<% if $SearchResult.Matches %>
<h2>Results for &quot;{$Query}&quot;</h2>
<p>Displaying Page $SearchResult.Matches.CurrentPage of $SearchResult.Matches.TotalPages</p>
<ol>
<% loop $SearchResult.Matches %>
<li>
<h3><a href="$Link">$Title</a></h3>
<p><% if $Abstract %>$Abstract.XML<% else %>$Content.ContextSummary<% end_if %></p>
</li>
<% end_loop %>
</ol>
<% else %>
<p>Sorry, your search query did not return any results.</p>
<% end_if %>
```
Please check the [pagination guide](https://docs.silverstripe.org/en/4/developer_guides/templates/how_tos/pagination/)
in the main SilverStripe documentation to learn how to paginate through search results.
## Automatic Index Updates
Every change, addition or removal of an indexed class instance triggers an index update through a
"processor" object. The update is transparently handled through inspecting every executed database query
and checking which database tables are involved in it.
Index updates usually are executed in the same request which caused the index to become "dirty".
For example, a CMS author might have edited a page, or a user has left a new comment.
In order to minimise delays to those users, the index update is deferred until after
the actual request returns to the user, through PHP's `register_shutdown_function()` functionality.
If the [queuedjobs](https://github.com/symbiote/silverstripe-queuedjobs) module is installed,
updates are queued up instead of executed in the same request. Queue jobs are usually processed every minute.
Large index updates will be batched into multiple queue jobs to ensure a job can run to completion within
common execution constraints (memory and time limits). You can check the status of jobs in
an administrative interface under `admin/queuedjobs/`.
## Manual Index Updates
Manual updates are connector specific, please check the connector docs for details.
## Searching Specific Fields
By default, the index searches through all indexed fields.
This can be limited by arguments to the `addSearchTerm()` call.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addSearchTerm('My house is on fire', [Page::class . '_Title']);
// No results, since we're searching in title rather than page content
$results = singleton(MyIndex::class)->search($query);
```
## Searching Value Ranges
Most values can be expressed as ranges, most commonly dates or numbers.
To search for a range of values rather than an exact match,
use the `SearchQuery_Range` class. The range can include bounds on both sides,
or stay open ended by simply leaving the argument blank.
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery_Range;
$query = new SearchQuery();
$query->addSearchTerm('My house is on fire');
// Only include documents edited in 2011 or earlier
$query->addFilter(Page::class . '_LastEdited', new SearchQuery_Range(null, '2011-12-31T23:59:59Z'));
$results = singleton(MyIndex::class)->search($query);
```
Note: At the moment, the date format is specific to the search implementation.
## Searching Empty or Existing Values
Since there's a type conversion between the SilverStripe database, object properties
and the search index persistence, its often not clear which condition is searched for.
Should it equal an empty string, or only match if the field wasn't indexed at all?
The `SearchQuery` API has the concept of a "missing" and "present" field value for this:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addSearchTerm('My house is on fire');
// Needs a value, although it can be false
$query->addFilter(Page::class . '_ShowInMenus', SearchQuery::$present);
$results = singleton(MyIndex::class)->search($query);
```
## Indexing Multiple Classes
An index is a denormalized view of your data, so can hold data from more than one model.
As you can only search one index at a time, all searchable classes need to be included.
```php
// File: mysite/code/MyIndex.php
use SilverStripe\FullTextSearch\Solr\SolrIndex;
use SilverStripe\Security\Member;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addClass(Member::class);
$this->addFulltextField('Content'); // only applies to Page class
$this->addFulltextField('FirstName'); // only applies to Member class
}
}
```
## Using Multiple Indexes
Multiple indexes can be created and searched independently, but if you wish to override an existing
index with another, you can use the `$hide_ancestor` config.
```php
use SilverStripe\Assets\File;
class MyReplacementIndex extends MyIndex
{
private static $hide_ancestor = 'MyIndex';
public function init()
{
parent::init();
$this->addClass(File::class);
$this->addFulltextField('Title');
}
}
```
You can also filter all indexes globally to a set of pre-defined classes if you wish to
prevent any unknown indexes from being automatically included.
```yaml
SilverStripe\FullTextSearch\Search\FullTextSearch:
indexes:
- MyReplacementIndex
- CoreSearchIndex
```
## Indexing Relationships
TODO
## Weighting/Boosting Fields
Results aren't all created equal. Matches in some fields are more important
than others, for example terms in a page title rather than its content
might be considered more relevant to the user.
To account for this, a "weighting" (or "boosting") factor can be applied to each
searched field. The default is 1.0, anything below that will decrease the relevance,
anthing above increases it.
Example:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
$query = new SearchQuery();
$query->addSearchTerm(
'My house is on fire',
null,
[
Page::class . '_Title' => 1.5,
Page::class . '_Content' => 1.0,
]
);
$results = singleton(MyIndex::class)->search($query);
```
## Filtering
## Connectors
### Solr
See Solr.md
### Sphinx
Not written yet
## FAQ
### How do I exclude draft pages from the index?
By default, the `SearchUpdater` class indexes all available "variant states",
so in the case of the `Versioned` extension, both "draft" and "live".
For most cases, you'll want to exclude draft content from your search results.
You can either prevent the draft content from being indexed in the first place,
by adding the following to your `SearchIndex->init()` method:
```php
use SilverStripe\FullTextSearch\Search\Variants\SearchVariantVersioned;
$this->excludeVariantState([SearchVariantVersioned::class => 'Stage']);
```
Alternatively, you can index draft content, but simply exclude it from searches.
This can be handy to preview search results on unpublished content, in case a CMS author is logged in.
Before constructing your `SearchQuery`, conditionally switch to the "live" stage:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use SilverStripe\Security\Permission;
use SilverStripe\Versioned\Versioned;
if (!Permission::check('CMS_ACCESS_CMSMain')) {
Versioned::set_stage(Versioned::LIVE);
}
$query = new SearchQuery();
// ...
```
### How do I write nested/complex filters?
TODO

14
docs/en/02_setup/22_installing_the_module.md
View File

@ -0,0 +1,14 @@
# Installing the module
## Disabling automatic configuration
If you have this module installed but do not have a Solr server running, you can disable the database manipulation
hooks that trigger automatic index updates:
```yaml
---
Name: mysitesearch
---
SilverStripe\FullTextSearch\Search\Updaters\SearchUpdater:
enabled: false
```

21
docs/en/03_configuration/30_creating_an_index.md
View File

@ -16,7 +16,26 @@ class MyIndex extends SolrIndex
}
```
This will create a new `SolrIndex` called `MyIndex`, and it will store the `Title` field on all `Pages` for searching.
This will create a new `SolrIndex` called `MyIndex`, and it will store the `Title` field on all `Pages` for searching. To index more than one class,
you simply call `addClass()` multiple times. Fields that you add don't have to be present on all classes in the index, they will only apply to a class
if it is present.
```php
use Page;
use SilverStripe\Security\Member;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addClass(Member::class);
$this->addFulltextField('Content'); // only applies to Page class
$this->addFulltextField('FirstName'); // only applies to Member class
}
}
```
You can also skip listing all searchable fields, and have the index figure it out automatically via `addAllFulltextFields()`. This will add any database fields that are `instanceof DBString` to the index. Use this with caution, however, as you may inadvertently return sensitive information - it is often safer to declare your fields explicitly.

38
docs/en/03_configuration/31_adding_data_to_an_index.md
View File

@ -28,3 +28,41 @@ Depending on the size of the index and how much content needs to be processed, i
## Queued jobs
If the [Queued Jobs module](https://github.com/symbiote/silverstripe-queuedjobs/) is installed, updates are queued up instead of executed in the same request. Queued jobs are usually processed every minute. Large index updates will be batched into multiple queued jobs to ensure a job can run to completion within common constraints, such as memory and execution time limits. You can check the status of jobs in an administrative interface under `admin/queuedjobs/`.
### Excluding draft content
By default, the `SearchUpdater` class indexes all available "variant states", so in the case of the `Versioned` extension, both "draft" and "live".
For most cases, you'll want to exclude draft content from your search results.
You can either prevent the draft content from being indexed in the first place, by adding the following to your `SearchIndex::init()` method:
```php
use Page;
use SilverStripe\FullTextSearch\Search\Variants\SearchVariantVersioned;
use SilverStripe\FullTextSearch\Solr\SolrIndex;
use SilverStripe\Versioned\Versioned;
class MyIndex extends SolrIndex
{
public function init()
{
$this->addClass(Page::class);
$this->addFulltextField('Title');
$this->excludeVariantState([SearchVariantVersioned::class => Versioned::DRAFT]);
}
}
```
Alternatively, you can index draft content, but simply exclude it from searches. This can be handy to preview search results on unpublished content, in case a CMS author is logged in. Before constructing your `SearchQuery`, conditionally switch to the "live" stage:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use SilverStripe\Security\Permission;
use SilverStripe\Versioned\Versioned;
if (!Permission::check('CMS_ACCESS_CMSMain')) {
Versioned::set_stage(Versioned::LIVE);
}
$query = SearchQuery::create();
// ...
```

40
docs/en/03_configuration/32_querying_the_index.md
View File

@ -58,6 +58,46 @@ $query = SearchQuery::create()
->addClassFilter(SpecialPage::class, false); // only return results from SpecialPages, not subclasses
```
### Searching value ranges
Most values can be expressed as ranges, most commonly dates or numbers. To search for a range of values rather than an exact match,
use the `SearchQuery_Range` class. The range can include bounds on both sides, or stay open-ended by simply leaving the argument blank.
It takes arguments in the form of `SearchQuery_Range::create($start, $end))`:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery_Range;
use My\Namespace\Index\MyIndex;
use Page;
$query = SearchQuery::create()
->addSearchTerm('fire')
// Only include documents edited in 2011 or earlier
->addFilter(Page::class . '_LastEdited', SearchQuery_Range::create(null, '2011-12-31T23:59:59Z'));
$results = singleton(MyIndex::class)->search($query);
```
Note: At the moment, the date format is specific to the search implementation.
### Searching for empty or existing values
Since there's a type conversion between the SilverStripe database, object properties
and the search index persistence, it's often not clear which condition is searched for.
Should it equal an empty string, or only match if the field wasn't indexed at all?
The `SearchQuery` API has the concept of a "missing" and "present" field value for this:
```php
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use My\Namespace\Index\MyIndex;
use Page;
$query = SearchQuery::create()
->addSearchTerm('fire');
// Needs a value, although it can be false
->addFilter(Page::class . '_ShowInMenus', SearchQuery::$present);
$results = singleton(MyIndex::class)->search($query);
```
## Querying an index
Once you have your query constructed, you need to run it against your index.

32
docs/en/04_advanced_configuration/41_multiple_indexes.md
View File

@ -0,0 +1,32 @@
# Using multiple indexes
Multiple indexes can be created and searched independently, but if you wish to override an existing
index with another, you can use the `$hide_ancestor` config.
```php
use SilverStripe\Assets\File;
use My\Namespace\Index\MyIndex;
class MyReplacementIndex extends MyIndex
{
private static $hide_ancestor = MyIndex::class;
public function init()
{
parent::init();
$this->addClass(File::class);
$this->addFulltextField('Title');
}
}
```
You can also filter all indexes globally to a set of pre-defined classes if you wish to
prevent any unknown indexes from being automatically included.
```yaml
SilverStripe\FullTextSearch\Search\FullTextSearch:
indexes:
- MyReplacementIndex
- CoreSearchIndex
```

28
docs/en/04_advanced_configuration/44_boosting.md
View File

@ -0,0 +1,28 @@
# Boosting/Weighting
Results aren't all created equal. Matches in some fields are more important
than others; for example, a page `Title` might be considered more relevant to the user than terms in the `Content` field.
To account for this, a "weighting" (or "boosting") factor can be applied to each searched field. The default value is `1.0`, anything below that will decrease the relevance, anything above increases it.
To adjust the relative values, pass them in as the third argument to your `addSearchTerm()` call:
```php
use My\Namespace\Index\MyIndex;
use SilverStripe\FullTextSearch\Search\Queries\SearchQuery;
use Page;
$query = SearchQuery::create()
->addSearchTerm(
'fire',
null, // don't limit the classes to search
[
Page::class . '_Title' => 1.5,
Page::class . '_Content' => 1.0,
Page::class . '_SecretParagraph' => 0.1,
]
);
$results = singleton(MyIndex::class)->search($query);
```
This will ensure that `Title` is given higher priority for matches than `Content`, which is well above `SecretParagraph`.

112
docs/en/05_troubleshooting.md Normal file
View File

@ -0,0 +1,112 @@
# Troubleshooting
## Common Gotchas
Oooh I gotcha
Here's some whitespace
## Test anchor
I'm on a boat