21 Commits

Author	SHA1	Message	Date
Georgi Gerganov	1545550ec2	unicode : normalize signatures	2024-04-28 21:40:36 +03:00
Georgi Gerganov	ee6d1b3fb4	unicode : simplify	2024-04-28 18:36:57 +03:00
Georgi Gerganov	e972e6cbf8	unicode : clean-up	2024-04-28 18:30:37 +03:00
Georgi Gerganov	b97add52a4	unicode : category support via std::regex	2024-04-28 15:15:57 +03:00
Georgi Gerganov	581c4a0239	unicode : try fix windows	2024-04-27 18:36:00 +03:00
Georgi Gerganov	91eaa414bf	unicode : support \p{N}, \p{L} and \p{P} natively	2024-04-27 17:48:38 +03:00
Georgi Gerganov	ce5485aee0	unicode : always use std::wregex	2024-04-27 17:11:34 +03:00
Georgi Gerganov	a22645c2a7	unicode : set bomb	2024-04-27 11:48:24 +03:00
Georgi Gerganov	c160818ec0	wip	2024-04-27 00:28:36 +03:00
Georgi Gerganov	e9891769ff	unicode : first try custom implementations	2024-04-26 15:09:07 +03:00
Georgi Gerganov	e8c206be61	unicode : shot in the dark to fix tests on Windows	2024-04-26 14:57:12 +03:00
Georgi Gerganov	06d3e693db	unicode : fix? unicode_wstring_to_utf8	2024-04-26 12:55:11 +03:00
Kazim Abrar Mahi	36d983262e	Fixed issue with gpt2 regex custom preprocessor	2024-04-26 11:43:29 +03:00
Kazim Abrar Mahi	feeaf4f39c	Added needed functionality, testing remains	2024-04-26 11:43:29 +03:00
Kazim Abrar Mahi	7e308ed212	Adding unicode regex function	2024-04-26 11:43:29 +03:00
Kazim Abrar Mahi	4056dc5b1e	added and refactored unicode_regex_split and related functions	2024-04-26 11:43:28 +03:00
Kazim Abrar Mahi	1c924e4b35	Resolved issues	2024-04-26 11:43:28 +03:00
Kazim Abrar Mahi	54f93eb50b	Moved header files	2024-04-26 11:43:28 +03:00
Kazim Abrar Mahi	d2cfc2225f	Moved regex patterns to unicode.cpp and updated unicode.h	2024-04-26 11:43:28 +03:00
Jared Van Bortel	32c8486e1f	wpm : portable unicode tolower (#6305) ... Also use C locale for ispunct/isspace, and split unicode-data.cpp from unicode.cpp.	2024-03-26 17:46:21 -04:00
Georgi Gerganov	83796e62bc	llama : refactor unicode stuff (#5992) ... * llama : refactor unicode stuff ggml-ci * unicode : names * make : fix c++ compiler * unicode : names * unicode : straighten tables * zig : fix build * unicode : put nfd normalization behind API ggml-ci * swift : fix build * unicode : add BOM * unicode : add <cstdint> ggml-ci * unicode : pass as cpts as const ref	2024-03-11 17:47:47 +02:00