Skip to content

Conversation

@JianlongCao
Copy link

1.4.0 chonkie changed the api and parameter which will cause the memory init fail by "SentenceChunker.init() got an unexpected keyword argument 'tokenizer_or_token_counter'"

Description

Summary:
Fix compatibility issue introduced by chonkie 1.4.0, where the API and parameter changes cause memory initialization to fail with the following error:
SentenceChunker.init() got an unexpected keyword argument 'tokenizer_or_token_counter'.

This PR updates the memory initialization module to align with the new chonkie API and corrects parameter handling during the chunking process.

Fix: #578

Docs Issue/PR: NA

Reviewer: TBA

Checklist:

  • [ x] I have performed a self-review of my own code | 我已自行检查了自己的代码
  • [ x] I have commented my code in hard-to-understand areas | 我已在难以理解的地方对代码进行了注释
  • [x ] I have added tests that prove my fix is effective or that my feature works | 我已添加测试以证明我的修复有效或功能正常
  • [x ] I have created related documentation issue/PR in MemOS-Docs (if applicable) | 我已在 MemOS-Docs 中创建了相关的文档 issue/PR(如果适用)
  • [x ] I have linked the issue to this PR (if applicable) | 我已将 issue 链接到此 PR(如果适用)
  • [x ] I have mentioned the person who will review this PR | 我已提及将审查此 PR 的人

1.4.0 chonkie changed the api and parameter which will cause the memory init fail by "SentenceChunker.__init__() got an unexpected keyword argument 'tokenizer_or_token_counter'"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: SentenceChunker.__init__() got an unexpected keyword argument 'tokenizer_or_token_counter'

1 participant