senhtry
fd3abe849e
server: fixing naming conflict res_error in server-models.cpp ( #17679 )
2025-12-02 11:18:39 +01:00
Xuan-Son Nguyen
682e6658bb
server: explicitly set exec path when create new instance ( #17669 )
...
* Revert "rm unused fn"
This reverts commit f2dbe9c087
2025-12-02 10:25:11 +01:00
Adrien Gallouët
4574f2949e
ci : skip winget update when not in ggml-org ( #17465 )
...
Prevent forks from generating daily failure notifications.
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2025-12-02 10:15:01 +01:00
Adrien Gallouët
ab6726eeff
ggml : add fallback definition for HWCAP2_SVE2 ( #17683 )
...
This align with other HWCAP2 feature flags
See #17528
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2025-12-02 10:41:26 +02:00
Aleksander Grygier
cee92af553
Add context info to server error ( #17663 )
...
* fix: Add context info to server error
* chore: update webui build output
2025-12-02 09:20:57 +01:00
Aman Gupta
ed32089927
ggml-cuda: reorder only relevant nodes ( #17639 )
2025-12-02 12:36:31 +08:00
Aaron Teo
7b6d745364
release: fix duplicate libs, store symbolic links ( #17299 )
2025-12-02 11:52:05 +08:00
Neo Zhang Jianyu
98bd9ab1e4
enhance argsort for UT ( #17573 )
...
Co-authored-by: Neo Zhang <zhang.jianyu@outlook.com >
2025-12-02 08:56:46 +08:00
Piotr Wilkin (ilintar)
746f9ee889
Override SSM_A op for Qwen3 Next to reduce splits ( #17587 )
...
* Override SSM_A op for Qwen3 Next to reduce splits
* New tensor mapping SSM_A_NOSCAN for SSM_A used outside of OP_SSM_SCAN context.
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-02 00:43:13 +01:00
Jeff Bolz
9810cb8247
ops.md: update vulkan support ( #17661 )
2025-12-01 15:26:21 -06:00
Xuan-Son Nguyen
ecf74a8417
mtmd: add mtmd_context_params::warmup option ( #17652 )
...
* mtmd: add mtmd_context_params::warmup option
* reuse the common_params::warmup
2025-12-01 21:32:25 +01:00
Gilad S.
00c361fe53
fix: llama arch implementation ( #17665 )
2025-12-01 21:21:13 +01:00
Xuan-Son Nguyen
ec18edfcba
server: introduce API for serving / loading / unloading multiple models ( #17470 )
...
* server: add model management and proxy
* fix compile error
* does this fix windows?
* fix windows build
* use subprocess.h, better logging
* add test
* fix windows
* feat: Model/Router server architecture WIP
* more stable
* fix unsafe pointer
* also allow terminate loading model
* add is_active()
* refactor: Architecture improvements
* tmp apply upstream fix
* address most problems
* address thread safety issue
* address review comment
* add docs (first version)
* address review comment
* feat: Improved UX for model information, modality interactions etc
* chore: update webui build output
* refactor: Use only the message data `model` property for displaying model used info
* chore: update webui build output
* add --models-dir param
* feat: New Model Selection UX WIP
* chore: update webui build output
* feat: Add auto-mic setting
* feat: Attachments UX improvements
* implement LRU
* remove default model path
* better --models-dir
* add env for args
* address review comments
* fix compile
* refactor: Chat Form Submit component
* ad endpoint docs
* Merge remote-tracking branch 'webui/allozaur/server_model_management_v1_2' into xsn/server_model_maagement_v1_2
Co-authored-by: Aleksander <aleksander.grygier@gmail.com >
* feat: Add copy to clipboard to model name in model info dialog
* feat: Model unavailable UI state for model selector
* feat: Chat Form Actions UI logic improvements
* feat: Auto-select model from last assistant response
* chore: update webui build output
* expose args and exit_code in API
* add note
* support extra_args on loading model
* allow reusing args if auto_load
* typo docs
* oai-compat /models endpoint
* cleaner
* address review comments
* feat: Use `model` property for displaying the `repo/model-name` naming format
* refactor: Attachments data
* chore: update webui build output
* refactor: Enum imports
* feat: Improve Model Selector responsiveness
* chore: update webui build output
* refactor: Cleanup
* refactor: Cleanup
* refactor: Formatters
* chore: update webui build output
* refactor: Copy To Clipboard Icon component
* chore: update webui build output
* refactor: Cleanup
* chore: update webui build output
* refactor: UI badges
* chore: update webui build output
* refactor: Cleanup
* refactor: Cleanup
* chore: update webui build output
* add --models-allow-extra-args for security
* nits
* add stdin_file
* fix merge
* fix: Retrieve lost setting after resolving merge conflict
* refactor: DatabaseStore -> DatabaseService
* refactor: Database, Conversations & Chat services + stores architecture improvements (WIP)
* refactor: Remove redundant settings
* refactor: Multi-model business logic WIP
* chore: update webui build output
* feat: Switching models logic for ChatForm or when regenerating messges + modality detection logic
* chore: update webui build output
* fix: Add `untrack` inside chat processing info data logic to prevent infinite effect
* fix: Regenerate
* feat: Remove redundant settigns + rearrange
* fix: Audio attachments
* refactor: Icons
* chore: update webui build output
* feat: Model management and selection features WIP
* chore: update webui build output
* refactor: Improve server properties management
* refactor: Icons
* chore: update webui build output
* feat: Improve model loading/unloading status updates
* chore: update webui build output
* refactor: Improve API header management via utility functions
* remove support for extra args
* set hf_repo/docker_repo as model alias when posible
* refactor: Remove ConversationsService
* refactor: Chat requests abort handling
* refactor: Server store
* tmp webui build
* refactor: Model modality handling
* chore: update webui build output
* refactor: Processing state reactivity
* fix: UI
* refactor: Services/Stores syntax + logic improvements
Refactors components to access stores directly instead of using exported getter functions.
This change centralizes store access and logic, simplifying component code and improving maintainability by reducing the number of exported functions and promoting direct store interaction.
Removes exported getter functions from `chat.svelte.ts`, `conversations.svelte.ts`, `models.svelte.ts` and `settings.svelte.ts`.
* refactor: Architecture cleanup
* feat: Improve statistic badges
* feat: Condition available models based on modality + better model loading strategy & UX
* docs: Architecture documentation
* feat: Update logic for PDF as Image
* add TODO for http client
* refactor: Enhance model info and attachment handling
* chore: update webui build output
* refactor: Components naming
* chore: update webui build output
* refactor: Cleanup
* refactor: DRY `getAttachmentDisplayItems` function + fix UI
* chore: update webui build output
* fix: Modality detection improvement for text-based PDF attachments
* refactor: Cleanup
* docs: Add info comment
* refactor: Cleanup
* re
* refactor: Cleanup
* refactor: Cleanup
* feat: Attachment logic & UI improvements
* refactor: Constants
* feat: Improve UI sidebar background color
* chore: update webui build output
* refactor: Utils imports + move types to `app.d.ts`
* test: Fix Storybook mocks
* chore: update webui build output
* test: Update Chat Form UI tests
* refactor: Tooltip Provider from core layout
* refactor: Tests to separate location
* decouple server_models from server_routes
* test: Move demo test to tests/server
* refactor: Remove redundant method
* chore: update webui build output
* also route anthropic endpoints
* fix duplicated arg
* fix invalid ptr to shutdown_handler
* server : minor
* rm unused fn
* add ?autoload=true|false query param
* refactor: Remove redundant code
* docs: Update README documentations + architecture & data flow diagrams
* fix: Disable autoload on calling server props for the model
* chore: update webui build output
* fix ubuntu build
* fix: Model status reactivity
* fix: Modality detection for MODEL mode
* chore: update webui build output
---------
Co-authored-by: Aleksander Grygier <aleksander.grygier@gmail.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-12-01 19:41:04 +01:00
Xuan-Son Nguyen
7733409734
common: improve verbosity level definitions ( #17630 )
...
* common: improve verbosity level definitions
* string_format
* update autogen docs
2025-12-01 14:38:13 +01:00
Xuan-Son Nguyen
cd3c118908
model: support Ministral3 ( #17644 )
...
* conversion script
* support ministral 3
* maybe this is better?
* add TODO for rope_yarn_log_mul
* better ppl (tested on 14B-Instruct)
* Add Ministral3 support to Mistral format
* improve arch handling
* add sizes
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* nits
---------
Co-authored-by: Julien Denize <julien.denize@mistral.ai >
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
2025-12-01 12:26:52 +01:00
Georgi Gerganov
649495c9d9
metal : add FA head size 48 ( #17619 )
2025-12-01 12:49:53 +02:00
Georgi Gerganov
90c72a614a
ggml : extend the GGML_SCHED_NO_REALLOC debug logic of the scheduler ( #17617 )
2025-12-01 12:49:33 +02:00
Aman Gupta
6eea666912
llama-graph: avoid expand_forward for fusion ( #17633 )
2025-12-01 11:12:48 +02:00
Xuan-Son Nguyen
ff90508d68
contributing: update guidelines for AI-generated code ( #17625 )
...
* contributing: update guidelines for AI-generated code
* revise
2025-11-30 22:51:34 +01:00
Adrien Gallouët
0a4aeb927d
cmake : add option to build and link LibreSSL ( #17552 )
...
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2025-11-30 22:14:32 +01:00
Tarek Dakhran
2ba719519d
model: LFM2-VL fixes ( #17577 )
...
* Adjust to pytorch
* Add antialiasing upscale
* Increase number of patches to 1024
* Handle default marker insertion for LFM2
* Switch to flag
* Reformat
* Cuda implementation of antialias kernel
* Change placement in ops.cpp
* consistent float literals
* Pad only for LFM2
* Address PR feedback
* Rollback default marker placement changes
* Fallback to CPU implementation for antialias implementation of upscale
2025-11-30 21:57:31 +01:00
Xuan-Son Nguyen
7f8ef50cce
clip: fix nb calculation for qwen3-vl ( #17594 )
2025-11-30 15:33:55 +01:00
Xuan-Son Nguyen
3c136b21a3
cli: add migration warning ( #17620 )
2025-11-30 15:32:43 +01:00
Adrien Gallouët
beb1f0c503
common : throttle download progress output to reduce IO flush ( #17427 )
...
This change limits progress updates to approximately every 0.1% of the
file size to minimize stdio overhead.
Also fixes compiler warnings regarding __func__ in lambdas.
Signed-off-by: Adrien Gallouët <angt@huggingface.co >
2025-11-30 14:22:44 +02:00
Aaron Teo
def5404f26
common: add LLAMA_LOG_FILE env var ( #17609 )
...
Signed-off-by: Aaron Teo <aaron.teo1@ibm.com >
2025-11-30 12:12:32 +01:00
Gilad S.
fa0465954f
ggml: fix: macOS build with -DGGML_BACKEND_DL=ON ( #17581 )
2025-11-30 10:00:59 +08:00
ddh0
5a6241feb0
common: update env var name ( #17588 )
2025-11-30 09:59:25 +08:00
Aman Gupta
c7af376c29
CUDA: add stream-based concurrency ( #16991 )
...
* CUDA: add stream-based concurrency
* HIP: fix hipStreamWaitEvent define and nodiscard warnings
* ggml-cuda: fix fusion inside stream
* ggml-cuda: fix bug w.r.t first stream launch
* ggml-cuda: format
* ggml-cuda: improve assert message
* ggml-cuda: use lambda instead of duplicating code
* ggml-cuda: add some more comments
* ggml-cuda: add more detailed comments about concurrency
* ggml-cuda: rename + remove unused var
* ggml-cuda: fix condition for stream launch
* ggml-cuda: address review comments, add destructor
* common.cuh: add is_valid for concurrent events
* common.cuh: make comment better
* update comment
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
* update comment
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
* common.cuh: fix lower_bound condition + remove join_node data from write_ranges
* ggml-cuda: fix overlap condition + shadowing parameter
---------
Co-authored-by: Carl Philipp Klemm <carl@uvos.xyz >
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2025-11-30 08:17:55 +08:00
Mahekk Shaikh
00425e2ed1
cuda : add error checking for cudaMemcpyAsync in argsort ( #17599 )
...
* cuda : add error checking for cudaMemcpyAsync in argsort (#12836 )
* fix indentation
2025-11-30 08:16:28 +08:00
Acly
385c3da5e6
vulkan : fix FA mask load with bounds check (coopmat2) ( #17606 )
2025-11-30 01:03:21 +01:00
Xuan-Son Nguyen
ab49f094d2
server: move server-context to its own cpp|h ( #17595 )
...
* git mv
* add server-context.h
* add server-context.h
* clean up headers
* cont : cleanup
* also expose server_response_reader (to be used by CLI)
* fix windows build
* decouple server_routes and server_http
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-29 22:04:44 +01:00
Haiyue Wang
8c32d9d96d
server: explicitly set the function name in lambda ( #17538 )
...
As [1] explained, the real debug message will be like:
"res operator(): operator() : queue result stop"
Set the name explicitly, the message is easy for debugging:
"res operator(): recv : queue result stop"
The left "operator()" is generated by 'RES_DBG() ... __func__'
[1]: https://clang.llvm.org/extra/clang-tidy/checks/bugprone/lambda-function-name.html
Signed-off-by: Haiyue Wang <haiyuewa@163.com >
2025-11-29 18:43:29 +01:00
Igor Smirnov
0874693b44
common : fix json schema with '\' in literals ( #17307 )
...
* Fix json schema with '\' in literals
* Add "literal string with escapes" test
2025-11-29 17:06:32 +01:00
Neo Zhang
7d2add51d8
sycl : support to malloc memory on device more than 4GB, update the doc and script ( #17566 )
...
Co-authored-by: Neo Zhang Jianyu <jianyu.zhang@intel.com >
2025-11-29 14:59:44 +02:00
ixgbe
f698a79c63
ggml: replace hwcap with riscv_hwprobe for RVV detection ( #17567 )
...
Signed-off-by: Wang Yang <yangwang@iscas.ac.cn >
2025-11-29 14:56:31 +02:00
Ruben Ortlam
47a268ea50
Vulkan: MMVQ Integer Dot K-Quant and MUL_MAT_ID support ( #16900 )
...
* vulkan: split mul_mmq_funcs for mul_mat_vecq use
* add mxfp4 mmvq
* add q2_k mmvq
* add q3_k mmvq
* add q4_k and q5_k mmvq
* add q6_k mmvq
* handle 4x4 quants per mmvq thread
* enable MUL_MAT_ID mmvq support
* enable subgroup optimizations for mul_mat_vec_id shaders
* device tuning
* request prealloc_y sync after quantization
* fix indentation
* fix llvmpipe test failures
* fix mul_mat_id mmvq condition
* fix unused variable warning
2025-11-29 09:37:22 +01:00
Jeff Bolz
59d8d4e963
vulkan: improve topk perf for large k, fix overflow in unit tests ( #17582 )
2025-11-29 08:39:57 +01:00
Aleksei Nikiforov
d82b7a7c1d
gguf-py : fix passing non-native endian tensors (editor-gui and new-metadata) ( #17553 )
...
gguf_new_metadata.py reads data from reader.
Reader doesn't byteswap tensors to native endianness.
But writer does expect tensors in native endianness to convert them
into requested endianness.
There are two ways to fix this: update reader and do conversion to native endianness and back,
or skip converting endianness in writer in this particular USE-case.
gguf_editor_gui.py doesn't allow editing or viewing tensor data.
Let's go with skipping excessive byteswapping.
If eventually capability to view or edit tensor data is added,
tensor data should be instead byteswapped when reading it.
2025-11-28 20:53:01 +01:00
DAN™
03914c7ef8
common : move all common_chat_parse_* to chat-parser.cpp. ( #17481 )
2025-11-28 19:29:36 +01:00
o7si
3ce7a65c2f
server: fix: /metrics endpoint returning JSON-escaped Prometheus format ( #17386 )
...
* fix: /metrics endpoint returning JSON-escaped Prometheus format
* mod: remove string overload from ok() method
2025-11-28 19:14:00 +01:00
Diego Devesa
e072b2052e
ggml : add GGML_SCHED_NO_REALLOC option to disable reallocations in ggml_backend_sched ( #17276 )
...
* ggml : add GGML_SCHED_NO_REALLOC option to disable reallocations in ggml_backend_sched
Enabled in ggml-ci for testing.
* llama : update worst-case graph for unified cache
* ci : disable op offload in some tests
* fix spelling
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-28 17:33:23 +02:00
R0CKSTAR
c6f7a423c8
[MUSA] enable fp16/fast_fp16/bf16_mma on PH1 ( #17551 )
...
* [MUSA] enable fp16/fast_fp16/bf16_mma on PH1
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* Update ggml/src/ggml-cuda/fattn-vec.cuh
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
* Update ggml/src/ggml-cuda/fattn-vec.cuh
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
* Update ggml/src/ggml-cuda/fattn-tile.cuh
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
* Address review comments
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
---------
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2025-11-28 14:08:29 +01:00
Aman Gupta
2e7ef98f18
ggml-cuda: add stricter checking for fusion ( #17568 )
...
* ggml-cuda: make conditions for fusion more explicit
* ggml-cuda: remove size check as std::equal already does it
2025-11-28 20:34:51 +08:00
Fredrik Hultin
ddf9f94389
server : add Anthropic Messages API support ( #17570 )
...
* server : add Anthropic Messages API support
* remove -@pytest.mark.slow from tool calling/jinja tests
* server : remove unused code and slow/skip on test_anthropic_vision_base64_with_multimodal_model in test_anthropic_api.py
* server : removed redundant n field logic in anthropic_params_from_json
* server : use single error object instead of error_array in streaming response handler for /v1/chat/completions and use unordered_set instead of set in to_json_anthropic_stream()
* server : refactor Anthropic API to use OAI conversion
* make sure basic test always go first
* clean up
* clean up api key check, add test
---------
Co-authored-by: Xuan Son Nguyen <son@huggingface.co >
2025-11-28 12:57:04 +01:00
Piotr Wilkin (ilintar)
ff55414c42
model : Qwen3 Next ( #16095 )
...
* Qwen3 Next - cleaned up version
* Whitespaces and stuff
* Correct minor errors
* Update src/llama-model.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* Misc. fixes.
* Clean up code, add missing hybrid qualifier
* Did someone transpose the SOLVE_TRI result matrix? Perhaps...
* Whitespace
* Proper tensors for cb calls
* Use llama-graph.h vertical alignment
* BROKEN: chunking
* Set new tensors as inputs.
* Proper chunk logic
* It's the circle of life...
* More shenanigans for n_seq > 1
* Nail in the coffin?
* Fix Windows build
* Eh, one fails on Windows, the other fails on Mac... just use general capture.
* quant : cleanup
* model : cleanup
* qwen3 : cleanup
* cont : cleanup
* cont : cleanup
* ggml : revert change
* qwen3 : cleanup
* cont : cleanup
* Readd cmath
* qwen3 : fix typo
* Update convert_hf_to_gguf.py
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
* Usual suspects
* fix my bad suggestion
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2025-11-28 12:02:56 +01:00
Johannes Gäßler
73955f7d2a
CUDA: no FP16 arithmetic for vector FA kernel ( #17558 )
2025-11-28 10:29:09 +01:00
Jeff Bolz
35cf8887e1
vulkan: Implement GGML_OP_TRI ( #17503 )
...
* vulkan: Implement GGML_OP_TRI
* check types match
2025-11-28 10:07:29 +01:00
Radoslav Gerganov
15d2b46b4d
rpc : cache and reuse compute graphs ( #15405 )
...
Store the last computed graph and reuse it when possible.
Also do not return response from GRAPH_COMPUTE and assume it always
completes successfully. If this this is not the case, the server closes
the connection. This saves us a network round trip to the server.
2025-11-28 08:33:51 +00:00
yulo
6bca76ff5e
HIP: enable mul_mat_f for RDNA4 ( #17437 )
...
* enable mmf for rdna4
* move some mmvf to mmf
* revert lds128 for wmma loading
* Revert "revert lds128 for wmma loading"
This reverts commit db9ae8b6b4698c9f241899b92bd665you@example.com >
2025-11-28 08:24:30 +01:00
Piotr Wilkin (ilintar)
cd0e3a7a3b
SOLVE_TRI CUDA kernel for small matrices ( #17457 )
2025-11-28 12:15:32 +08:00
