sdgoij/llama.cpp
1
0
Fork 0
mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-05-13 12:34:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
a37696d4f1bdd66aa4bc9f8a1b86ccfff725c603
llama.cpp/examples/embedding/embedding.cpp
Jared Van Bortel 8803582721 llama : handle added special tokens like HF does 
Now the BERT tokenizer actually uses the SEP and CLS tokens from
SpecialVocab.
2024-04-04 16:01:26 -04:00

6.4 KiB
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink