sdgoij/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-05-10 02:54:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

unicode : fix? unicode_wstring_to_utf8

Browse Source
This commit is contained in:
Georgi Gerganov
2024-04-26 12:55:11 +03:00
parent 36d983262e
commit 06d3e693db
3 changed files with 19 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
unicode-data.cpp
View File

@@ -1666,4 +1666,4 @@ const std::map<std::string, std::wstring> unicode_regex_equivalent_wregex = {
const std::set<std::string> unicode_regex_with_custom_preprocessor = {
"'s|'t|'re|'ve|'m|'ll|'d| ?\\p{L}+| ?\\p{N}+| ?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)"
};
};