Commit Graph

76 Commits

Author SHA1 Message Date
Akihiko Odaki
d5cabfe5c6 Use CLD3 (#2949)
Compact Language Detector v3 (CLD3) is the successor of CLD2, which was
used in the previous implementation. CLD3 includes improvements since CLD2,
and supports newer compilers. On the other hand, it has additional
requirements and cld3-ruby, the FFI of CLD3 for Ruby, is still new and may
be still inmature.

Though CLD3 is named after CLD2, it is implemented with a neural network
model, different from the old implementation, which is based on a Naïve
Bayesian classifier.

CLD3 supports newer compilers, such as GCC 6. CLD2 is not compatible with
GCC 6 because it assigns negative values to varibales typed unsigned.
(see internal/cld_generated_cjk_uni_prop_80.cc) The support for GCC 6 and
newer compilers are essential today, when some server operating system
such as Ubuntu Server 16.10 has GCC 6 by default.

On the one hand, CLD3 requires C++11 support. Environments with old
compilers such as Ubuntu Server 14.04 needs to update the system or install
a newer compiler.

CLD3 needs protocol buffers as a new dependency. However,it is not
considered problematic because major server operating systems, CentOS and
Ubuntu Server provide them.

The FFI cld3-ruby was written by me (Akihiko Odaki) for use in Mastodon.
It is still new and may be inmature, but confirmed to pass existing tests.
2017-05-09 19:58:03 +02:00
abcang
08e94d1b19 Fixed bug that timeline can not be displayed by InvalidURIError (#2947) 2017-05-09 18:17:41 +02:00
Matt Jankowski
7a7d12d27f Delegate Account#user_locale method and allow nil (#2927) 2017-05-08 23:10:50 +02:00
masarakki
d08f1112d5 use Twitter::Extractor for creating links (#2502) 2017-05-05 19:48:22 +02:00
Matt Jankowski
8c5ad23b24 Language improvements, replace whatlanguage with CLD (#2753)
* add failing en specs

* add cld2 gem

* Replace WhatLanguage with CLD
2017-05-03 16:59:31 +02:00
Akihiko Odaki
a6788662b0 Revert "Use PostgreSQL inheritance for blocks and mutes (#2520)" (#2634)
This reverts commit 5135d609b7.
2017-04-30 08:49:24 -04:00
Akihiko Odaki
5135d609b7 Use PostgreSQL inheritance for blocks and mutes (#2520) 2017-04-30 00:27:31 +02:00
Yamagishi Kazutoshi
3ea5b948a4 Decodes URL containing IDN (#2436) 2017-04-25 15:03:51 +02:00
Eugen
338df98ddf Add more FeedManager#filter? tests (#2413) 2017-04-24 22:37:24 +02:00
Matt Jankowski
629d35e6f5 [WIP] Html lang on statuses (#2297)
* Add html lang attributes around statuses

* Remove urls from language detection
2017-04-22 04:26:25 +02:00
Matt Jankowski
2dda356e3f Clean up settings/preferences controller (#2237)
* Add missing fields group on preferences page

* Clean up settings/preferences controller

* Extract a UserSettingsDecorator
2017-04-21 03:26:52 +02:00
abcang
70891a99a9 Fix html escape characters in the URL (#2138)
* fix character escaping in URL

* add tests

* put a comma after the last item

* add HTML escape test
2017-04-19 14:52:18 +02:00
Eugen
96268e76e6 Fix wording on admin instances view, fix typos in language detector spec (#2106) 2017-04-18 23:39:51 +02:00
Eugen
6bd1019235 Add more tests for AtomSerializer (#2096) 2017-04-18 22:30:17 +02:00
Matt Jankowski
297c11dba2 Language detection refactor (#2099)
* Extract detect_language to separate class

* Use default locale, not just en

* Add spec to confirm that whatlanguage cant identify empty string

* Allow account locale to override default in language detector

* PostStatusService supplies an account to detect language
2017-04-18 22:20:12 +02:00
Eugen
f03fb6c0ae Add forgotten AtomSerializer spec (#2047) 2017-04-18 01:07:46 +02:00
Eugen
5dc73339ae Fix tests issue introduced in #1607 (#1639) 2017-04-13 02:17:40 +02:00
Matt Jankowski
aa90798386 Webfinger resource to extract username from resource string (#1607)
* Add WebfingerResource class to extract usernames

* Use WebfingerResource in xrd#webfinger
2017-04-12 18:22:38 +02:00
Eugen Rochko
3202bdd744 Fix #555 - Use a better URL parser 2017-02-22 19:35:11 +01:00
Eugen Rochko
0518492158 Stop trying to shoehorn all Salmon updates into the poor database-connected
StreamEntry model. Simply render Salmon slaps as they are needed
2017-02-12 01:19:14 +01:00
Eugen Rochko
920ba5fc4e Fix #61 - Add list of blocked users to the UI; clean up failed push notifications API
Try to fix Travis CI setup
2017-02-05 19:18:11 +01:00
Eugen Rochko
80cefd5b3c Fix #204, fix #515 - URL truncating is now a style so copypasting is not
affected, replaced onClick handler with onMouseUp/Down to detect text
selection not trigger onClick handler then
2017-01-24 17:05:44 +01:00
Eugen Rochko
76ec907993 Improved admin UI 2016-12-13 13:42:10 +01:00
Eugen Rochko
33f7e1cf99 Shorten rendered links (strip protocol and www, truncate to 30 chars), redirect
to sign in page after sign up instead of root path which redirects to /about
2016-10-13 16:51:34 +02:00
Eugen Rochko
be86d4e0a3 Preparing feeds for better filtering 2016-10-02 15:28:47 +02:00
Eugen Rochko
3cc47beb6e Refactored generation of unique tags, URIs and object URLs into own classes,
as well as formatting of content
2016-09-09 20:04:34 +02:00