7 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Radoslav Gerganov	db10f01310	rpc : track allocated buffers (#7411) ... * rpc : track allocated buffers ref: #7407 * rpc : pack rpc_tensor tightly	2024-05-20 16:36:55 +03:00
Radoslav Gerganov	f4bd8b3d26	rpc : set SO_REUSEADDR for the server socket (#7320) ... ref: #7293	2024-05-17 17:25:44 +03:00
Radoslav Gerganov	ee94172d33	server : add support for the RPC backend (#7305) ... ref: #7292	2024-05-17 10:00:17 +03:00
Radoslav Gerganov	9afdffe70e	rpc : get available mem for the CPU backend ... This can be overridden with the -m command line option ref: #7293	2024-05-16 12:04:08 +03:00
Radoslav Gerganov	3b3963c55c	rpc : add command line arg for specifying backend memory ... ref: #7293	2024-05-16 09:58:29 +03:00
Radoslav Gerganov	5e31828d3e	ggml : add RPC backend (#6829) ... * ggml : add RPC backend The RPC backend proxies all operations to a remote server which runs a regular backend (CPU, CUDA, Metal, etc). * set TCP_NODELAY * add CI workflows * Address review comments * fix warning * implement llama_max_devices() for RPC * Address review comments * Address review comments * wrap sockfd into a struct * implement get_alignment and get_max_size * add get_device_memory * fix warning * win32 support * add README * readme : trim trailing whitespace * Address review comments * win32 fix * Address review comments * fix compile warnings on macos	2024-05-14 14:27:19 +03:00
Radoslav Gerganov	60f819a2b1	Add section to README on how to run the project on Android (#130)	2023-03-14 15:30:08 +02:00