FrequencyMan (Anki Plugin)

Overview

FrequencyMan allows you to sort your new cards by word frequency, familiarity, and other useful factors.

Features

• More than 50 default word frequency lists.
• Define multiple sorting targets for different decks or selection of cards.
• Customize the ranking factors for each target.
• It tracks word familiarity in a non-trivial way, allowing a more accurate sorting by i+1.
• Use multiple fields and languages (such as 'front' and 'back') to influence the ranking of a card.
• Multiple 'word frequency' lists can be used per language.

Installation

To download this add-on, please copy and paste the following code into Anki (Tools > Add-ons > Get Add-ons...): 909420026

Basic usage

1. Open the "FrequencyMan" menu option in the "Tools" menu of the main Anki window.
2. This will open FrequencyMan's main window where you can define your sorting targets.
3. Define the targets using a JSON array of objects. Each object represents a target to sort (a target can be a deck or a defined selection of cards).
4. Click the "Reorder Cards" button to apply the sorting.

Configuration examples

Example 1

Reorders a single deck. This will only match cards with note type Basic located in deck Spanish. It will also use the default ranking factors.

The content of the cards and all the ranking metrics will be analyzed per 'language'. The result of this will be combined to determine the final ranking of all new cards in the defined target.

[
    {
        "deck": "Spanish",
        "notes": [
            {
                "fields": {
                    "Front": "EN",
                    "Back": "ES"
                },
                "name": "Basic"
            }
        ]
    }
]

Example 2

Reorder the same deck twice, but the first target excludes the sorting of cards whose name matches "Speaking", while the second target only sorts those excluded cards.

The first target only modifies a single ranking factor, while the second target reduces the ranking factors used to only 2 factors.

Note: Both targets use the same 'main scope', which is the selection of cards used to create the data to calculate the ranking. This scope is reduced for each target by reorder_scope_query to limit which cards get repositioned.

[
    {
        "deck": "Spanish",
        "notes": [
            {
                "fields": {
                    "Meaning": "EN",
                    "Sentence": "ES"
                },
                "name": "Basic (customized note type)"
            }
        ],
        "reorder_scope_query": "-card:*Speaking*",
        "ranking_familiarity": 8
    },
    {
        "deck": "Spanish",
        "notes": [
            {
                "fields": {
                    "Meaning": "EN",
                    "Sentence": "ES"
                },
                "name": "Basic (customized note type)"
            }
        ],
        "reorder_scope_query": "card:*Speaking*",
        "ranking_factors": {
            "familiarity": 1,
            "word_frequency": 1
        }
    }
]

Example #3

Reorder only based on word frequency (using word frequency from both front and back):

[
    {
        "deck": "Spanish::Essential Spanish Vocabulary Top 5000",
        "notes": [
            {
                "name": "Basic-f4e28",
                "fields": {
                    "Front": "ES",
                    "Back": "EN"
                }
            }
        ],
        "ranking_factors": {
            "word_frequency": 1
        }
    }
]

Tokenizers

Custom tokenizers can be defined in user_files\tokenizers.

To use a custom tokenizer, or to see how one is defined, you can download here a working copy of Jieba (ZH), and here a version of Janome (JA).

If you download Janome (JA), you can place it in a directory like user_files\tokenizers\janome, which then should contain the file fm_init_janome.py and the subdirectory janome.

Automatic support

FrequencyMan will use tokenizers from other plugins, if there is no custom tokenizer for a given language:

• If ankimorphs-chinese-jieba is installed, Jieba can be used.
• If ankimorphs-japanese-mecab is installed, Mecab can be used.
• If AJT Japanese is installed, Mecab can be used.
• If Morphman is installed, Mecab and Jieba can be used (assuming those also work in Morphman itself).

Ranking factors

Default ranking factors

"ranking_factors" : {
    "word_frequency": 1.0,
    "internal_word_frequency": 0.0,
    "familiarity": 1.0,
    "familiarity_sweetspot": 0.5,
    "lexical_underexposure": 0.25,
    "ideal_focus_word_count": 4.0,
    "ideal_word_count": 1.0,
    "reinforce_learning_words": 1.5,
    "most_obscure_word": 0.5,
    "lowest_fr_least_familiar_word": 0.25,
    "lowest_word_frequency": 1.0,
    "lowest_internal_word_frequency": 0.0,
    "lowest_familiarity": 1.0,
    "new_words": 0.5,
    "no_new_words": 0.0,
    "ideal_new_word_count": 0.0,
    "proper_introduction": 0.1,
    "proper_introduction_dispersed": 0.0
}

Description

• word_frequency: Represents the word frequency of the words in the content, with a bias toward the lowest value. The word frequency values come from the provided word frequency lists.
• internal_word_frequency: Represents the word frequency of the words based on their occurrence within the notes themselves (the target's content), rather than external frequency lists. Like word_frequency, it has a bias toward the lowest value.
• familiarity: Represents how familiar you are with the words in the content. Like word_frequency, it has a bias toward the lowest value. How familiar you are with a word depends on how many times you have seen the word and in what context that specific word was present (the interval and ease of the card, the amount of words in the content etc.).
• familiarity_sweetspot: Promotes cards with words close to a specific 'sweetspot' of familiarity. This can be used to promote cards with words that have already been introduced to you by reviewed cards, but might benefit from 'reinforcement'. These can be recently introduced words, or words that are 'hidden' (non-prominent) in older cards. Use target setting familiarity_sweetspot_point to customize the sweetspot value.
• lexical_underexposure: Promotes cards with high-frequency words that you are not yet proportionally familiar with. Basically, lexical_underexposure = (word_frequency-word_familiarity).
• ideal_focus_word_count: Promotes cards with only a single 'focus word'. See also i+1: https://en.wikipedia.org/wiki/Input_hypothesis#Input_hypothesis. A focus word is a new word or a word you are not yet appropriately familiar with. Use target setting maturity_threshold to customize the maximum familiarity of the focus words.
• ideal_word_count: Represents how close the word count of the content is to the defined ideal range. By default this is 1 to 5, but you can customize it per target with:

  "ideal_word_count": [2, 8]
  

• reinforce_learning_words: Promotes cards with one or more 'learning' word (a reviewed, but not yet mature word), but only if there are no new words present.
• most_obscure_word: Represents the most obscure word. The non-obscurity of a word is defined by either word_frequency or word_familiarity (depending on which is higher, and thus less 'obscure').
• lowest_fr_least_familiar_word: Represents the lowest word frequency among the words with the lowest familiarity score.
• lowest_word_frequency: Represents the lowest word frequency found in the content of any targeted field. This is different from word_frequency, which reflect the average word frequency of all targeted fields.
• lowest_internal_word_frequency: Represents the lowest internal word frequency found in the content of any targeted field. This is different from internal_word_frequency, which reflects the average internal word frequency of all targeted fields.
• lowest_familiarity: Represents the lowest familiarity found in the content of any targeted field. This is different from familiarity, which reflect the average familiarity of all targeted fields.
• new_words: Promotes cards with one or more new words.
• no_new_words: Promotes cards with no new words. Put differently, it promotes cards who's words have all been seen before during review.
• ideal_new_word_count: Like ideal_focus_word_count, but promotes cards with only a single 'new word' (a word not found in any reviewed card).
• proper_introduction: Promotes cards that appear to be well suited to introduce a new word. Various factor are used, including the position of the new word and the word frequency + familiarity of the other words in the content. Cards without new words are not effected.
• proper_introduction_dispersed: Disperses cards using the proper_introduction factor. This is done per word selected by lowest_fr_least_familiar_word, thus dispersing cards with that same word selected. Cards with and without new words are effected.

Custom fields

The following fields will be automatically populated when you reorder your cards:

• fm_focus_words: A list of focus words for each field. (recommended!)
• fm_new_words: A list of new words (words not found in reviewed cards) for each field.
• fm_seen_words: A list of seen words (words found in reviewed cards) for each field.

Dynamic field names (the number at the end can be replaced with the index number of any field defined in the target):

• fm_main_focus_word_0: The focus word with the lowest familiarity for field 0.
• fm_main_focus_word_static_0: The focus word with the lowest familiarity for field 0. This field will not be updated once set.
• fm_lowest_fr_word_0: The word with the lowest word frequency for field 0.
• fm_lowest_internal_fr_word_0: The word with the lowest internal word frequency for field 0.
• fm_lowest_familiarity_word_0: The word with the lowest familiarity for field 0.
• fm_lowest_familiarity_word_static_0: The word with the lowest familiarity for field 0. This field will not be updated once set.

For debug purposes:

• fm_debug_info: Different metrics and data points for each field.
• fm_debug_ranking_info: The resulting score per ranking factor for the note.
• fm_debug_words_info The score's for each word for 'word frequency', 'lexical underexposure' and 'familiarity sweetspot'.

Display focus words on the back of your cards (html example)

{{#fm_main_focus_word_0}}
  <p style="color:darkred;">{{fm_main_focus_word_0}}</p>
{{/fm_main_focus_word_0}}

{{#fm_focus_words}}
  <p> <span style="opacity:0.65;">Focus:</span> {{fm_focus_words}} </p>
{{/fm_focus_words}}

Target settings

For each defined target, the following settings are available:

Notes:

• familiarity_sweetspot_point accepts a string starting with ~, such as "~0.5". This can be used to make it relative to the value of maturity_threshold. With the default settings, "~0.5" would result in a value of 0.14 (50% of 0.28). A string starting with ^ will make the number relative to the median word familiarity value.
• suspended_card_value and suspended_leech_card_value are used to devalue reviewed cards that are suspended when calculating 'word familiarity'. This is applied on top of the devaluing that happens if a card is due (devaluing due cards is done with both suspended and non-suspended cards).

Language data id

For each field a language_data_id must be defined. In most cases this should just be a two letter language code (ISO 639-1), such as EN or ES:

[
    {
        "deck": "Spanish::Essential Spanish Vocabulary Top 5000",
        "notes": [
            {
                "name": "Basic-f4e28",
                "fields": {
                    "Spanish": "ES",
                    "English": "EN"
                }
            }
        ]
    }
]

Alternatively, a language_data_id can also be an 'extended two letter language code':

[
    {
        "deck": "Medical",
        "notes": [
            {
                "name": "Basic-f4e28",
                "fields": {
                    "Front": "EN_MEDICAL",
                    "Back": "EN_MEDICAL"
                }
            },

        ]
    },
]

For every language data id defined, a directory should exist (although it could be empty). In the example above, \user_files\lang_data\en_medical should exist. If it does not exist, you will be prompted to automatically create one with a default word frequency list shipped with FrequencyMan.

Two different types of files can be placed in a language data id directory:

• word frequency lists: A text or csv file with words sorted to reflect the word frequency (in descending order). Only the position is used, not the (optional) word frequency value.
• ignore lists: A text file with words that will not be used to calculate the rankings. The file name should start with "ignore". FrequencyMan comes with a default ignore list (ignore_candidates.txt), which can be found in default_wf_lists/. This file contains words most found across different languages (words such as 'FBI', 'Steve', 'cool' etc.).

Language data folder

In the language data folder itself (\user_files\lang_data) the following type of files can be placed:

• names lists: A text file with names/words that will not be used to calculate the rankings. The file name should start with "names", such as names_to_ignore.txt.
• ignore lists: A text file with words that will not be used to calculate the rankings. The file name should start with "ignore".

Both 'name lists' and 'ignore lists' placed in the language data folder itself are used for all languages.

Reorder logging

Reorder logging is an optional feature that can be enabled by defining an id on a target. When enabled, information about the content of that target is logged each time the cards are reordered.

Display the amount of mature words

The information that is logged can be used to display the amount of 'mature' words a target has using the following plugin settings (Tools > Add-ons > (Select Frequencyman) > Config):

"show_info_deck_browser": [
    {
        "lang": "ES",
        "target": "*"
    },
    {
        "lang": "EN",
        "target": "*"
    },
    {
        "lang": "ES",
        "target": "id_of_target"
    },
    {
        "lang": "EN",
        "target": "id_of_target"
    }
],
"show_info_toolbar": [
    {
        "lang": "ES",
        "target": "*"
    }
]

Notes:

• * is used to show combined information about all logged targets.
• show_info_deck_browser wil create a table below the deck browser (below where you normally see "Studied N cards in N minutes today.").
• If there is no target with an id defined, nothing will be logged and thus no information will be shown.
• All logged information is stored in the file user_files\reorder_log.sqlite.

FrequencyMan plugin settings

Notes:

• To add or change any of the settings above, go to Tools > Add-ons > (Select Frequencyman) > Config.
• If reposition_shift_existing is set to True, the cards from the first reordered target will be positioned at the top of your collection.

Target Corpus data

A 'corpus data set' contains all the information related the the content of a note that is used to calculate the ranking of a card (such as the "familiarity" of a word).

Every target has one or more 'corpus data' sets, depending on how many fields are defined in the target and how the corpus_segmentation_strategy is set.

By default, corpus_segmentation_strategy is set to "by_lang_data_id", which means that a corpus data set will be created for every unique language_data_id:

{"Front": "EN", "Back": "EN"} // <- A single corpus data set
{"Front": "EN", "Back": "EN", "Extra": "ES"} // <- Two corpus data sets

To create separate corpus data sets for each field, you can set corpus_segmentation_strategy to "by_note_model_id_and_field_name". This will create a corpus data set for each field in the target:

{"Front": "EN", "Back": "EN"} // <- Two corpus data sets
{"Front": "EN", "Back": "EN", "Extra": "ES"} // <- Three corpus data sets

Notes:

• Using "by_note_model_id_and_field_name" also means that fields from different notes in the same target will not be 'joined' together.
• Using "by_note_model_id_and_field_name" can create multiple corpus data sets for the same language, which may not be desirable for language learning purposes.
• Using "by_lang_data_id" will join fields from all notes defined within a target if they have the same language_data_id.

Word frequency lists

FrequencyMan comes with 50+ default word frequency lists. These lists are generated using a combination of sources:

• Open Subtitles 2018: https://github.com/Rct567/top-open-subtitles-sentences/tree/frequencyman-edition
• AI generated children's stories: https://github.com/Rct567/wf_lists_lm_childrenstories
• Wortschatz 'News 2022', AnkiMorphs 'priority files': https://mortii.github.io/anki-morphs/user_guide/setup/prioritizing.html
• Google Books n-gram: https://github.com/orgtre/google-books-ngram-frequency

The default word frequency lists can be found in the \default_wf_lists. When prompted to create a new language data directory with a default word frequency list, the relevant file will be copied to the new language data directory, such as \user_files\lang_data\en.

The user_files directory

The user_files directory can be found inside Frequencyman's plugin directory, which can be accessed via: Tools > Add-ons > (Select Frequencyman) > View Files.

Any files placed in this folder will be preserved when the add-on is upgraded. All other files in the add-on folder are removed on upgrade.

Manual installation from GitHub

1. Go to the Anki plugin folder, such as C:\Users\%USERNAME%\AppData\Roaming\Anki2\addons21.
2. Create a new folder with the name FrequencyMan.
3. Make sure you are still in the directory addons21.
4. Run: git clone https://github.com/Rct567/FrequencyMan.git FrequencyMan
5. Start Anki.