How to decide a grapheme break efficiently in mid-string?

[Old-Farmer]

Sorry if I ask naive question. I am quite new to unicode. As the warning part said: "utf8proc_grapheme_break_stateful must be called IN ORDER on ALL potential breaks in a string. However, it is safe to reset the state to zero after a grapheme break". So how to recognize a break in the middle of the str? Only start from the beginning of the str? I have to recognize a break before an offset.

After I read the uax29，I have found that there are only a few cases need the state. I think I can call stateless version and decide whether this is break based on codepoint category? Is it correct?

[stevengj]

Good question … if you scrutinize the algorithm described in the Unicode standard, it should be possible to start in the middle of the string and scan backwards for the last character from which we can determine the state … in most cases you shouldn't have to scan all the way to the beginning of the string.

The most rigorous version of this algorithm would be Unicode version-dependent, so ideally should be implemented within utf8proc itself.

However, a simplified version of the algorithm would be search backwards through the characters for a boundclass (pointed to by the utf8proc_get_property struct) that is equal to UTF8PROC_BOUNDCLASS_OTHER, which is the most common value for characters with no special break rules, and then start at that character with a reset state rather than at the beginning of the string.

[Old-Farmer]

Thanks for your reply. You mean a bound class other codepoint is safe to be passed as the codepoint1 arguement and with state 0?

[stevengj]

Yes, I think so.