7.7.2.2. NormalizerNFKC100 — Groonga v10.1.1-31-g1e46ba6 documentation

# groonga - An open-source fulltext search engine and column store.

- 日本語

# 7.7.2.2. `NormalizerNFKC100`

## 7.7.2.2.1. Summary

New in version 8.0.2.

`NormalizerNFKC100` normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 10.0.

This normalizer can change behavior by specifying options.

## 7.7.2.2.2. Syntax

`NormalizerNFKC100` has optional parameter.

No options:

    NormalizerNFKC100

`NormalizerNFKC100` normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 10.0.

Specify option:

    NormalizerNFKC100("unify_kana", true)

    NormalizerNFKC100("unify_kana_case", true)

    NormalizerNFKC100("unify_kana_voiced_sound_mark", true)

    NormalizerNFKC100("unify_hyphen", true)

    NormalizerNFKC100("unify_prolonged_sound_mark", true)

    NormalizerNFKC100("unify_hyphen_and_prolonged_sound_mark", true)

    NormalizerNFKC100("unify_middle_dot", true)

    NormalizerNFKC100("unify_katakana_v_sounds", true)

    NormalizerNFKC100("unify_katakana_bu_sound", true)

    NormalizerNFKC100("unify_to_romaji", true)

New in version 8.0.3: unify_middle_dot is added.

unify_katakana_v_sounds is added.

unify_katakana_bu_sound is added.

New in version 8.0.9: unify_to_romaji is added.

Specify multiple options:

    NormalizerNFKC100("unify_to_romaji", true, "unify_kana_case", true, "unify_hyphen_and_prolonged_sound_mark", true)

`NormalizerNFKC100` also specify multiple options as above. You can also specify mingle multiple options except above example.

## 7.7.2.2.3. Usage

## 7.7.2.2.4. Simple usage

Here is an example of `NormalizerNFKC100`. `NormalizerNFKC100` normalizes text by Unicode NFKC (Normalization Form Compatibility Composition) for Unicode version 10.0.

Execution example:

    normalize NormalizerNFKC100 "©" WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840077.247264,
    #     0.0001680850982666016
    #   ],
    #   {
    #     "normalized": "©",
    #     "types": [
    #       "emoji"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_kana option.

This option enables that same pronounced characters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_kana", true)'   "あイｳｪおヽヾ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840210.809296,
    #     0.0002298355102539062
    #   ],
    #   {
    #     "normalized": "あいうぇおゝゞ",
    #     "types": [
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_kana_case option.

This option enables that large and small versions of same letters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_kana_case", true)'   "ぁあぃいぅうぇえぉおゃやゅゆょよゎわゕかゖけ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840389.78734,
    #     0.0001950263977050781
    #   ],
    #   {
    #     "normalized": "ああいいううええおおややゆゆよよわわかかけけ",
    #     "types": [
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Execution example:

    normalize   'NormalizerNFKC100("unify_kana_case", true)'   "ァアィイゥウェエォオャヤュユョヨヮワヵカヶケ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840469.179984,
    #     0.0002634525299072266
    #   ],
    #   {
    #     "normalized": "アアイイウウエエオオヤヤユユヨヨワワカカケケ",
    #     "types": [
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_kana_voiced_sound_mark option.

This option enables that letters with/without voiced sound mark and semi voiced sound mark in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_kana_voiced_sound_mark", true)'   "かがきぎくぐけげこごさざしじすずせぜそぞただちぢつづてでとどはばぱひびぴふぶぷへべぺほぼぽ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840544.908633,
    #     0.0002522468566894531
    #   ],
    #   {
    #     "normalized": "かかききくくけけここささししすすせせそそたたちちつつててととはははひひひふふふへへへほほほ",
    #     "types": [
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana",
    #       "hiragana"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Execution example:

    normalize   'NormalizerNFKC100("unify_kana_voiced_sound_mark", true)'   "カガキギクグケゲコゴサザシジスズセゼソゾタダチヂツヅテデトドハバパヒビピフブプヘベペホボポ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840704.477687,
    #     0.0002183914184570312
    #   ],
    #   {
    #     "normalized": "カカキキククケケココササシシススセセソソタタチチツツテテトトハハハヒヒヒフフフヘヘヘホホホ",
    #     "types": [
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_hyphen option. This option enables normalize hyphen to “-” (U+002D HYPHEN-MINUS) as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_hyphen", true)'   "-˗֊‐‑‒–⁃⁻₋−"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840778.422051,
    #     0.0001845359802246094
    #   ],
    #   {
    #     "normalized": "-----------",
    #     "types": [
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_prolonged_sound_mark option. This option enables normalize prolonged sound to “-” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK) as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_prolonged_sound_mark", true)'   "ー—―─━ｰ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840846.654316,
    #     0.0001988410949707031
    #   ],
    #   {
    #     "normalized": "ーーーーーー",
    #     "types": [
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_hyphen_and_prolonged_sound_mark option. This option enables normalize hyphen and prolonged sound to “-” (U+002D HYPHEN-MINUS) as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_hyphen_and_prolonged_sound_mark", true)'   "-˗֊‐‑‒–⁃⁻₋− ﹣－ ー—―─━ｰ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840930.462605,
    #     0.0001947879791259766
    #   ],
    #   {
    #     "normalized": "----------- -- ------",
    #     "types": [
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "others",
    #       "symbol",
    #       "symbol",
    #       "others",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_middle_dot option. This option enables normalize middle dot to “·” (U+00B7 MIDDLE DOT) as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_middle_dot", true)'   "·ᐧ•∙⋅⸱・･"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546840999.582769,
    #     0.0001835823059082031
    #   ],
    #   {
    #     "normalized": "········",
    #     "types": [
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol",
    #       "symbol"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_katakana_v_sounds option. This option enables normalize “ヴァヴィヴヴェヴォ” to “バビブベボ” as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_katakana_v_sounds", true)'   "ヴァヴィヴヴェヴォヴ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546841068.702912,
    #     0.0002088546752929688
    #   ],
    #   {
    #     "normalized": "バビブベボブ",
    #     "types": [
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_katakana_bu_sound option. This option enables normalize “ヴァヴィヴゥヴェヴォ” to “ブ” as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_katakana_bu_sound", true)'   "ヴァヴィヴヴェヴォヴ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546841138.543078,
    #     0.0001876354217529297
    #   ],
    #   {
    #     "normalized": "ブブブブブブ",
    #     "types": [
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana",
    #       "katakana"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

Here is an example of unify_to_romaji option. This option enables normalize hiragana and katakana to romaji as below.

Execution example:

    normalize   'NormalizerNFKC100("unify_to_romaji", true)'   "アァイィウゥエェオォ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546841200.15132,
    #     0.0001931190490722656
    #   ],
    #   {
    #     "normalized": "axaixiuxuexeoxo",
    #     "types": [
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

## 7.7.2.2.5. Advanced usage

You can output romaji of specific a part of speech with using to combine `TokenMecab` and `NormalizerNFKC100` as below.

First of all, you extract reading of a noun with excluding non-independent word and suffix of person name with `target_class` option and `include_reading` option.

Next, you normalize reading of the noun that extracted with `unify_to_romaji` option of `NormalizerNFKC100`.

Execution example:

    tokenize 'TokenMecab("target_class", "-名詞/非自立", "target_class", "-名詞/接尾/人名", "target_class", "名詞", "include_reading", true)' '彼の名前は山田さんのはずです。'
    # [
    #   [
    #     0,
    #     1546841272.495518,
    #     0.0003752708435058594
    #   ],
    #   [
    #     {
    #       "value": "彼",
    #       "position": 0,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "カレ"
    #       }
    #     },
    #     {
    #       "value": "名前",
    #       "position": 1,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ナマエ"
    #       }
    #     },
    #     {
    #       "value": "山田",
    #       "position": 2,
    #       "force_prefix": false,
    #       "force_prefix_search": false,
    #       "metadata": {
    #         "reading": "ヤマダ"
    #       }
    #     }
    #   ]
    # ]
    normalize   'NormalizerNFKC100("unify_to_romaji", true)'   "カレ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546841303.223331,
    #     0.000186920166015625
    #   ],
    #   {
    #     "normalized": "kare",
    #     "types": [
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]
    normalize   'NormalizerNFKC100("unify_to_romaji", true)'   "ナマエ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546841329.839442,
    #     0.0001835823059082031
    #   ],
    #   {
    #     "normalized": "namae",
    #     "types": [
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]
    normalize   'NormalizerNFKC100("unify_to_romaji", true)'   "ヤマダ"   WITH_TYPES
    # [
    #   [
    #     0,
    #     1546841358.479471,
    #     0.0001850128173828125
    #   ],
    #   {
    #     "normalized": "yamada",
    #     "types": [
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha",
    #       "alpha"
    #     ],
    #     "checks": [
    #     ]
    #   }
    # ]

## 7.7.2.2.6. Parameters

### 7.7.2.2.6.1. Optional parameter

There are optional parameters as below.

#### 7.7.2.2.6.1.1. `unify_kana`

This option enables that same pronounced characters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.

#### 7.7.2.2.6.1.2. `unify_kana_case`

This option enables that large and small versions of same letters in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.

#### 7.7.2.2.6.1.3. `unify_kana_voiced_sound_mark`

This option enables that letters with/without voiced sound mark and semi voiced sound mark in all of full-width Hiragana, full-width Katakana and half-width Katakana are regarded as the same character.

#### 7.7.2.2.6.1.4. `unify_hyphen`

This option enables normalize hyphen to “-” (U+002D HYPHEN-MINUS).

Hyphen of the target of normalizing is as below.

- “-” (U+002D HYPHEN-MINUS)

- “֊” (U+058A ARMENIAN HYPHEN)

- “˗” (U+02D7 MODIFIER LETTER MINUS SIGN)

- “‐” (U+2010 HYPHEN)

- “—” (U+2014 EM DASH)

- “⁃” (U+2043 HYPHEN BULLET)

- “⁻” (U+207B SUPERSCRIPT MINUS)

- “₋” (U+208B SUBSCRIPT MINUS)

- “−” (U+2212 MINUS SIGN)

#### 7.7.2.2.6.1.5. `unify_prolonged_sound_mark`

This option enables normalize prolonged sound to “-” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK).

Prolonged sound of the target of normalizing is as below.

- “—” (U+2014 EM DASH)

- “―” (U+2015 HORIZONTAL BAR)

- “─” (U+2500 BOX DRAWINGS LIGHT HORIZONTAL)

- “━” (U+2501 BOX DRAWINGS HEAVY HORIZONTAL)

- “ー” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK)

- “ｰ” (U+FF70 HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK)

#### 7.7.2.2.6.1.6. `unify_hyphen_and_prolonged_sound_mark`

This option enables normalize hyphen and prolonged sound to “-” (U+002D HYPHEN-MINUS).

Hyphen and prolonged sound of the target normalizing is below.

- “-” (U+002D HYPHEN-MINUS)

- “֊” (U+058A ARMENIAN HYPHEN)

- “˗” (U+02D7 MODIFIER LETTER MINUS SIGN)

- “‐” (U+2010 HYPHEN)

- “—” (U+2014 EM DASH)

- “⁃” (U+2043 HYPHEN BULLET)

- “⁻” (U+207B SUPERSCRIPT MINUS)

- “₋” (U+208B SUBSCRIPT MINUS)

- “−” (U+2212 MINUS SIGN)

- “—” (U+2014 EM DASH)

- “―” (U+2015 HORIZONTAL BAR)

- “─” (U+2500 BOX DRAWINGS LIGHT HORIZONTAL)

- “━” (U+2501 BOX DRAWINGS HEAVY HORIZONTAL)

- “ー” (U+30FC KATAKANA-HIRAGANA PROLONGED SOUND MARK)

- “ｰ” (U+FF70 HALFWIDTH KATAKANA-HIRAGANA PROLONGED SOUND MARK)

#### 7.7.2.2.6.1.7. `unify_middle_dot`

New in version 8.0.3.

This option enables normalize middle dot to “·” (U+00B7 MIDDLE DOT).

Middle dot of the target of normalizing is as below.

- “·” (U+00B7 MIDDLE DOT)

- “ᐧ” (U+1427 CANADIAN SYLLABICS FINAL MIDDLE DOT)

- “•” (U+2022 BULLET)

- “∙” (U+2219 BULLET OPERATOR)

- “⋅” (U+22C5 DOT OPERATOR)

- “⸱” (U+2E31 WORD SEPARATOR MIDDLE DOT)

- “・” (U+30FB KATAKANA MIDDLE DOT)

- “･” (U+FF65 HALFWIDTH KATAKANA MIDDLE DOT)

#### 7.7.2.2.6.1.8. `unify_katakana_v_sounds`

New in version 8.0.3.

This option enables normalize “ヴァヴィヴヴェヴォ” to “バビブベボ”.

#### 7.7.2.2.6.1.9. `unify_katakana_bu_sound`

New in version 8.0.3.

This option enables normalize “ヴァヴィヴゥヴェヴォ” to “ブ”.

#### 7.7.2.2.6.1.10. `unify_to_romaji`

New in version 8.0.9.

This option enables normalize hiragana and katakana to romaji.

## 7.7.2.2.7. See also

- normalize