Skip to content

By default use end-of-full-token character (</t>) instead of token boundaries (<w>, </w>) for all kinds of pre-processing for consistency  #3

@hlibbabii

Description

@hlibbabii

Currently:

>>> api.basic("getName")
['<w>', 'get', 'Name', '</w>']

To be done:

>>> api.basic("getName")
['get', 'Name', '</t>']

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions