Page MenuHomePhabricator
Create Task
Maniphest T154511

[Tracking] Research, test, and deploy new language analyzers
Open, LowPublic
Actions

Description

Two ways to start:

  • Languages that we really want to make big improvements on because we don't support them well (e.g. spaceless languages)
  • Test analysers that we know to be very mature (e.g. there's a Polish analyser that @dcausse knows about and likes)

Things to consider:

  • How much better the analyser is than what we've got
  • Maintainability of the code of the analyser
  • [add more!]

Languages/analyzers to consider (from T155549):

Previously a 2016/17 Q3 Goal.
Previously a 2016/17 Q4 Goal.
Currently a 2017/18 Q1 Goal.

Related Objects
Search...

StatusSubtypeAssignedTask
 InvalidNoneT174065 [FY 2017-18 Objective] Improve support for searching in multiple languages
 OpenNoneT154511 [Tracking] Research, test, and deploy new language analyzers
 ResolvedTJonesT154516 [Research spike, 4 hours] Research Polish language analysers
 ResolvedTJonesT154517 Test and analyse new Polish language analysers
 ResolvedEBernhardsonT158682 Deploy new Polish language analyser
ResolvedTJonesT155549 Create an ordered list of languages we want to find new analysers for
ResolvedTJonesT158202 [Research spike, 4 hours] Research Chinese language analyzers
 ResolvedTJonesT158203 Test and analyze new Chinese language analyzers
 ResolvedTJonesT163829 Enable BM25 for Chinese wikis
 ResolvedNoneT163832 Reindex Chinese wikis
 ResolvedTJonesT166722 Disable SmartCN for zh-hans
 ResolvedTJonesT160105 [Research spike, 4 hours] Research Ukrainian language analyzers
 ResolvedTJonesT160106 Test and analyze new Ukrainian language analyzers
 ResolvedEBernhardsonT162055 Deploy New Ukrainian Analyzer & Re-index Ukrainian Wikis
 ResolvedTJonesT162739 [Research spike, 4 hours] Research Hebrew language analyzers
 ResolvedTJonesT162741 Test and analyze new Hebrew language analyzers
 ResolvedGehelT167057 Deploy HebMorph Plugin to production
 ResolveddcausseT167058 Re-index Hebrew-language wikis
 ResolveddebtT71361 Search should normalize Niqqud diacritics in Hebrew characters
 ResolvedTJonesT166731 Test and analyze new Kuromoji Japanese language analyzer
 DeclinedTJonesT170423 Test and analyze Vietnamese language analyzer
 ResolvedTJonesT171516 Publish Language Analyzer Analysis Tools
 ResolvedTJonesT171652 Language Analysis Morphological Library Research Spike
 OpenNoneT178923 Review Japanese Morphological Libraries
 OpenNoneT178924 Review Vietnamese Morphological Libraries
 ResolvedTJonesT178925 Review Korean Morphological Libraries
 ResolvedTJonesT206874 Add Nori (Korean) configuration to AnalysisConfigBuilder
 ResolvedTJonesT216738 Reindex Korean-language wikis to enable Nori analyzer
 OpenNoneT219534 Test MLR models for zhwiki, jawiki and kowiki
 ResolvedTJonesT178926 Review Serbian Morphological Libraries
 ResolvedTJonesT183015 Create Serbian Elasticsearch Plugin/Analysis Chain Using Serbian Morphological Libraries
 ResolveddebtT189239 Deploy initial version of the extra-analysis plugin
 ResolveddebtT189265 Re-index Serbian Wikis
 ResolvedTJonesT196404 Re-Re-Index Serbian Wikis after refactored plugins are deployed
 ResolvedTJonesT192395 Create Croatian, Serbo-Croatian, and Bosnian Analysis Chains Using Serbian Morphological Libraries
 ResolvedTJonesT196658 Re-index Croatian, Serbo-Croatian, and Bosnian Wikis
 ResolvedTJonesT178928 Review Estonian Morphological Libraries
 ResolvedTJonesT178929 Review Slovak Morphological Libraries
 ResolvedTJonesT190815 Create Slovak Elasticsearch Plugin/Analysis Chain Using Slovak Stemming Algorithm
 ResolvedGehelT191543 Deploy updated search/extra plugin and search/extra-analysis-slovak plugin with Slovak Stemmer
 ResolvedTJonesT191544 Deploy the analysis config for the new Slovak stemmer
 ResolvedTJonesT191545 Re-index Slovak Wikis after analysis chain is deployed
 ResolvedTJonesT196780 Review Applying Indonesian Analysis Chain for Malay
 ResolvedTJonesT200204 Re-index Malay and Indonesian Wikis to use new unpacked analysis chain
 ResolvedTJonesT197240 Review Esperanto Morphological Libraries
 ResolvedTJonesT200099 Create Esperanto Elasticsearch Plugin Using Esperanto Morphological Libraries
 ResolvedTJonesT202173 Create Esperanto Analysis Chain using new Esperanto Plugin
 ResolvedTJonesT202662 Esperanto Stemmer Updates
 ResolvedTJonesT203005 Re-index Esperanto Wikis
 ResolvedTJonesT193734 Move Serbian language wikis from extra-analysis to extra-analysis-serbian plugin

Event Timeline

• Deskana created this task.Jan 3 2017, 7:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 3 2017, 7:43 PM
• Deskana renamed this task from [EPIC] Research, test, and deploy new language analysers to [Epic, Q3 Goal] Research, test, and deploy new language analysers.Jan 3 2017, 7:44 PM
• Deskana triaged this task as Medium priority.
• Deskana moved this task from needs triage to Current work on the Discovery-Search board.
• Deskana edited projects, added Discovery-Search (Current work); removed Discovery-Search.
• Deskana added a project: Epic.
• Deskana created subtask T154516: [Research spike, 4 hours] Research Polish language analysers.Jan 3 2017, 7:47 PM
TJones subscribed.Jan 11 2017, 6:04 PM
This comment was removed by TJones.
Matanya subscribed.Jan 11 2017, 6:25 PM
TJones updated the task description. (Show Details)Jan 11 2017, 6:40 PM
TJones added a subscriber: • Manybubbles.Jan 11 2017, 6:42 PM

HebMorph was recommended by @Matanya. It was investigated some time ago by Matanya and Nik (@Manybubbles). It's being actively developed and Matanya knows the developer.

• Deskana added a subtask: T155549: Create an ordered list of languages we want to find new analysers for .Jan 17 2017, 9:12 PM
TJones mentioned this in T155549: Create an ordered list of languages we want to find new analysers for .Jan 23 2017, 9:43 PM
TJones updated the task description. (Show Details)Jan 24 2017, 9:18 PM
Restricted Application added a subscriber: Base. · View Herald TranscriptJan 24 2017, 9:18 PM
TJones added a comment.EditedJan 26 2017, 5:29 PM

While researching analyzers, I came across others. I didn't really investigate most of them, so this list is just a starting point for anyone who wants to look more closely at any of these.

General
https://www.elastic.co/guide/en/elasticsearch/plugins/5.1/analysis.html (ES 5.1)
list of Elastic Analysis Plugins (internal and 3rd party)—Japanese, several for Chinese, Polish, Ukrainian, Hebrew, Russian, English, Vietnamese, & some technical ones.

Polish
See T154516.

Chinese
See T158202.

Ukrainian
See T160105.

Hebrew
See T162739.

Japanese
https://www.elastic.co/guide/en/elasticsearch/plugins/5.1/analysis-kuromoji.html (v5.1.2)
https://www.elastic.co/guide/en/elasticsearch/plugins/master/analysis-kuromoji.html (v6.0.0a)
test here (v?): http://www.atilika.org/

Vietnamese
https://github.com/duydo/elasticsearch-analysis-vietnamese (3 months)
linked by Elastic

Thai
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html
ICU Anlaysis plugin, "including better analysis of Asian languages"
Mentioned elsewhere that it covers Thai as well.

Phonetic analysis
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-phonetic.html (v5.1.2)
https://www.elastic.co/guide/en/elasticsearch/plugins/master/analysis-phonetic.html (v6.0.0.a)
“Soundex, Metaphone, and a variety of other algorithms”, presumably English

Misc
https://github.com/yakaz/elasticsearch-analysis-combo (2 years)
combines multiple language analyzers

TJones mentioned this in T154516: [Research spike, 4 hours] Research Polish language analysers.Jan 26 2017, 5:30 PM
• Deskana closed subtask T155549: Create an ordered list of languages we want to find new analysers for as Resolved.Jan 30 2017, 6:13 PM
• Deskana closed subtask T154516: [Research spike, 4 hours] Research Polish language analysers as Resolved.Jan 30 2017, 6:16 PM
TJones updated the task description. (Show Details)Feb 14 2017, 9:56 PM
TJones created subtask T158202: [Research spike, 4 hours] Research Chinese language analyzers.Feb 15 2017, 3:51 PM
TJones updated the task description. (Show Details)Feb 15 2017, 3:54 PM
TJones created subtask T160105: [Research spike, 4 hours] Research Ukrainian language analyzers.Mar 9 2017, 8:01 PM
TJones updated the task description. (Show Details)
• Deskana moved this task from Current work to This Quarter on the Discovery-Search board.Mar 16 2017, 10:18 PM
• Deskana edited projects, added Discovery-Search; removed Discovery-Search (Current work).
Liuxinyu970226 subscribed.Mar 25 2017, 4:53 AM
Shizhao added a project: Chinese-Sites.Apr 7 2017, 1:52 PM
mxn subscribed.Apr 11 2017, 7:15 AM
TJones created subtask T162739: [Research spike, 4 hours] Research Hebrew language analyzers.Apr 11 2017, 7:41 PM
TJones updated the task description. (Show Details)Apr 11 2017, 7:46 PM
TJones renamed this task from [Epic, Q3 Goal] Research, test, and deploy new language analysers to [Epic, Q3 Goal, Q4 Goal] Research, test, and deploy new language analysers.Apr 11 2017, 9:01 PM
TJones closed subtask T160105: [Research spike, 4 hours] Research Ukrainian language analyzers as Resolved.Apr 25 2017, 7:47 PM
TJones closed subtask T162739: [Research spike, 4 hours] Research Hebrew language analyzers as Resolved.
TJones closed subtask T158202: [Research spike, 4 hours] Research Chinese language analyzers as Resolved.
TJones created subtask T166731: Test and analyze new Kuromoji Japanese language analyzer.May 31 2017, 8:48 PM
TJones mentioned this in T154853: Consider asking communities which languages are analysed the poorest in search.Jun 21 2017, 9:34 PM
TJones updated the task description. (Show Details)Jun 23 2017, 6:41 PM
TJones renamed this task from [Epic, Q3 Goal, Q4 Goal] Research, test, and deploy new language analysers to [Epic, Q1 Goal] Research, test, and deploy new language analyzers.Jul 12 2017, 1:25 PM
TJones updated the task description. (Show Details)
TJones created subtask T170423: Test and analyze Vietnamese language analyzer.Jul 12 2017, 1:42 PM
Shizhao removed a project: Chinese-Sites.Jul 13 2017, 1:45 AM
debt closed subtask T166731: Test and analyze new Kuromoji Japanese language analyzer as Resolved.Jul 21 2017, 7:47 PM
TJones updated the task description. (Show Details)Jul 21 2017, 8:32 PM
TJones mentioned this in T171516: Publish Language Analyzer Analysis Tools.Jul 24 2017, 7:45 PM
TJones added a subtask: T171516: Publish Language Analyzer Analysis Tools.Jul 25 2017, 7:56 PM
TJones mentioned this in T171652: Language Analysis Morphological Library Research Spike.Jul 25 2017, 8:41 PM
TJones added a subtask: T171652: Language Analysis Morphological Library Research Spike.
debt closed subtask T171516: Publish Language Analyzer Analysis Tools as Resolved.Aug 4 2017, 8:21 PM
debt closed subtask T170423: Test and analyze Vietnamese language analyzer as Declined.Aug 6 2017, 2:07 PM
• EBjune added a parent task: T174065: [FY 2017-18 Objective] Improve support for searching in multiple languages.Aug 31 2017, 4:43 PM
Nemo_bis added a project: I18n.Sep 26 2017, 2:57 PM
• EBjune renamed this task from [Epic, Q1 Goal] Research, test, and deploy new language analyzers to [Epic] Research, test, and deploy new language analyzers.Oct 26 2017, 5:04 AM
TJones closed subtask T171652: Language Analysis Morphological Library Research Spike as Resolved.Feb 13 2018, 6:25 PM
Amire80 moved this task from Untriaged to Search on the I18n board.Mar 12 2018, 1:28 PM
debt closed subtask T193734: Move Serbian language wikis from extra-analysis to extra-analysis-serbian plugin as Resolved.Jun 14 2018, 8:11 PM
• Vvjjkkii reopened subtask T193734: Move Serbian language wikis from extra-analysis to extra-analysis-serbian plugin as Open.Jul 1 2018, 1:12 AM
CommunityTechBot closed subtask T193734: Move Serbian language wikis from extra-analysis to extra-analysis-serbian plugin as Resolved.Jul 2 2018, 2:09 PM
Smalyshev moved this task from This Quarter to [epic] on the Discovery-Search board.Jan 29 2019, 7:04 PM
Gehel renamed this task from [Epic] Research, test, and deploy new language analyzers to [Tracking] Research, test, and deploy new language analyzers.Sep 9 2020, 2:52 PM
Gehel lowered the priority of this task from Medium to Low.
MPhamWMF closed this task as Declined.Aug 1 2022, 9:22 PM
MPhamWMF subscribed.

Closing out low/est priority tasks over 6 months old with no activity within last 6 months in order to clean out the backlog of tickets we will not be addressing in the near term. Please feel free to reopen if you think a ticket is important, but bare in mind that given current priorities and resourcing, it is unlikely for the Search team to pick up these tasks for the indefinite future. We hope that the requested changes have either been addressed by or made irrelevant by work the team has done or is doing -- e.g. upgrading Elasticsearch to a newer version will solve various ES-related problems -- or will be subsumed by future work in a more generalized way.

RhinosF1 reopened this task as Open.Aug 2 2022, 1:33 PM
RhinosF1 removed a project: Discovery-Search.
RhinosF1 subscribed.

Re-opening tasks and removing from team workboard per IRC feedback given yesterday and discussion with MPham.

Gehel edited projects, added Discovery-Search; removed Discovery-ARCHIVED.Aug 11 2022, 8:29 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL