diff --git a/debian/README.Debian.md b/debian/README.Debian.md new file mode 100644 index 0000000000..317bdf5dc5 --- /dev/null +++ b/debian/README.Debian.md @@ -0,0 +1,107 @@ +# Debian package support + +This repository extends the upstream `llama-cpp-python` project with **Debian `.deb` packaging** for Debian 13 (Trixie). + +The packaging is designed around a modular backend architecture. The Python bindings and common runtime are separated from hardware-specific acceleration backends, allowing multiple backends to be installed independently and selected without rebuilding the Python package. + +Currently packaged backends include: + +- CPU / OpenBLAS, including multiple optimized CPU variants +- Vulkan +- NVIDIA CUDA 12.4 +- NVIDIA CUDA 12.9 +- NVIDIA CUDA 13.3 +- AMD ROCm Core SDK 10.1 / HIP +- Intel oneAPI SYCL + +Versioned CUDA and ROCm backends are designed to coexist where applicable, with `update-alternatives` used to select the active backend implementation. The packaging also includes transitional packages for compatibility with older package names. + +Although the packaging is developed and tested on **Debian 13 (Trixie)**, it can in practice be adapted and built on most Linux distributions based on Debian packages, provided that the required build dependencies and vendor toolkits/frameworks are available. Depending on the selected backends, this may include NVIDIA CUDA, AMD ROCm or Intel oneAPI/SYCL. + +### Vendor APT repositories + +The current Debian packaging builds all configured backends in one +source-package build. All build dependencies listed in `debian/control`, +including the CUDA, ROCm and SYCL toolchains, are required. + +At runtime, accelerator backend packages can be installed individually. + +#### NVIDIA CUDA + +The CUDA 12.9/13.x packages used by this packaging are obtained from NVIDIA's Debian repository. The repository currently used here is NVIDIA's Debian 12 x86_64 repository, which is also usable for these builds on Debian 13. + +Install the repository key: + +```bash +wget -qO- https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/3bf863cc.pub \ + | gpg --dearmor \ + | sudo tee /usr/share/keyrings/nvidia-cuda-debian12.gpg >/dev/null +``` + +Add the repository: + +```bash +echo "deb [signed-by=/usr/share/keyrings/nvidia-cuda-debian12.gpg] https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/ /" \ + | sudo tee /etc/apt/sources.list.d/nvidia-cuda.list +``` + +Then refresh APT metadata: + +```bash +sudo apt update +``` + +#### AMD ROCm + +AMD provides a Debian 13 multi-architecture ROCm repository. + +Install the repository key: + +```bash +wget -qO- https://repo.amd.com/rocm/packages-multi-arch/gpg/rocm.gpg \ + | sudo gpg --dearmor -o /etc/apt/keyrings/amdrocm.gpg +``` + +Add the repository: + +```bash +echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/amdrocm.gpg] https://repo.amd.com/rocm/packages-multi-arch/debian13 stable main" \ + | sudo tee /etc/apt/sources.list.d/rocm.list +``` + +Then refresh APT metadata: + +```bash +sudo apt update +``` + +#### Intel oneAPI / SYCL + +Intel provides the oneAPI APT repository containing the DPC++/C++ compiler and the required SYCL libraries. + +Install the repository key: + +```bash +wget -qO- https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB \ + | gpg --dearmor \ + | sudo tee /usr/share/keyrings/oneapi-archive-keyring.gpg >/dev/null +``` + +Add the repository: + +```bash +echo "deb [signed-by=/usr/share/keyrings/oneapi-archive-keyring.gpg] https://apt.repos.intel.com/oneapi all main" \ + | sudo tee /etc/apt/sources.list.d/oneapi.list +``` + +Then refresh APT metadata: + +```bash +sudo apt update +``` + +The Debian packaging files are maintained separately from the upstream Python package functionality and are intended to provide clean system integration, dependency handling and reproducible backend builds. + +**Debian packaging author / maintainer:** `ByCzech ` + +> The original `llama-cpp-python` project is developed and maintained by its upstream authors. The Debian packaging additions in this repository are maintained independently. diff --git a/debian/changelog b/debian/changelog new file mode 100644 index 0000000000..7e5f26ab72 --- /dev/null +++ b/debian/changelog @@ -0,0 +1,106 @@ +llama-cpp-python (0:0.3.36-7) stable; urgency=medium + + * Build against the matching upstream llama.cpp revision and remove + the obsolete ROCm 7.14 alternatives file. + + -- NemaChybu team APT archive Sun, 11 Oct 2026 10:15:34 +0200 + +llama-cpp-python (0:0.3.36-6) stable; urgency=medium + + * Apply the gfx1250 compatibility patch to vendored llama.cpp 4df29be too. + * Match the unique RDNA4 predicate without unrelated GCN/CDNA context, + preserving the GCN/CDNA predicates supplied by either upstream revision. + + -- NemaChybu team APT archive Sun, 11 Oct 2026 02:48:00 +0200 + +llama-cpp-python (0:0.3.36-5) stable; urgency=medium + + * Restore gfx1250 with a quilt patch for compatible HIP kernel dispatch. + * Keep native dot4 and GPU flash attention tile/vector kernels on gfx1250. + * Use FP32 arithmetic for its FP16 dot2 fallback; retain hipBLAS hardware + matrix support without enabling incompatible custom GGML WMMA kernels. + + -- NemaChybu team APT archive Sun, 11 Oct 2026 02:20:00 +0200 + +llama-cpp-python (0:0.3.36-4) stable; urgency=medium + + * Exclude gfx1250 from the ROCm 10.1 backend: the vendored GGML uses + unsupported dot2 and gfx12 WMMA instructions for this target. + * Retain all other 24 HIP targets, including gfx908 for MI100. + + -- NemaChybu team APT archive Sun, 11 Oct 2026 01:55:00 +0200 + +llama-cpp-python (0:0.3.36-3) stable; urgency=medium + + * Escape regex end anchors in the ROCm SDK and HIP version checks so + make passes the complete expressions to grep. + + -- NemaChybu team APT archive Sun, 11 Oct 2026 01:15:48 +0200 + +llama-cpp-python (0:0.3.36-2) stable; urgency=medium + + * Drop the unavailable ROCm Core SDK 7.14 build dependency and backend. + * Keep ROCm Core SDK 10.1 as the packaged HIP runtime. + + -- NemaChybu team APT archive Sat, 10 Oct 2026 22:55:00 +0200 + +llama-cpp-python (0:0.3.36-1) stable; urgency=medium + + * New upstream release 0.3.36; retain its pinned llama.cpp revision. + * Refresh the Debian dynamic backend installation patch. + * Add a co-installable AMD ROCm Core SDK 10.1 HIP backend. + * Verify the ROCm SDK and HIP component versions independently. + * Keep the existing CPU, Vulkan, CUDA, ROCm 7.14 and SYCL backends. + + -- NemaChybu team APT archive Sat, 10 Oct 2026 22:00:00 +0200 + +llama-cpp-python (0:0.3.35-3) stable; urgency=medium + + * Start new split-package layout. + * Add core package and dynamically dispatched CPU backend package. + * Add transitional packages for the old CPU package names. + * Add split Vulkan backend and transition from python3-llama-cpp-vulkan. + * Finalize phase 2 package layout and backend naming. + * Add Debian Trixie CUDA 12.4 dynamic backend and legacy CUDA transition. + * Add co-installable NVIDIA CUDA 12.9 and CUDA 13.3 dynamic backends. + * Add broad-target AMD ROCm 7.14 HIP dynamic backend. + + -- NemaChybu team APT archive Thu, 27 Aug 2026 01:00:00 +0200 + +python3-llama-cpp (0:0.3.35-1) stable; urgency=medium + + * New upstream release. + * Version 0.3.25 + + -- NemaChybu team APT archive Wed, 26 Aug 2026 17:49:47 +0200 + +python3-llama-cpp (0:0.3.23-1) stable; urgency=medium + + * Version 0.3.23 + + -- NemaChybu team APT archive Mon, 11 May 2026 23:39:39 +0200 + +python3-llama-cpp (0:0.3.21-1) stable; urgency=medium + + * Multipackage for all build + * New version of python3-llama-cpp + + -- NemaChybu team APT archive Tue, 28 Apr 2026 00:24:10 +0200 + +python3-llama-cpp (0:0.3.20-3~w2d1) stable; urgency=medium + + * Updated llama.cpp library to 0.0.8773 + + -- NemaChybu team APT archive Tue, 28 Apr 2026 00:23:54 +0200 + +python3-llama-cpp (0:0.3.20-2~w2d0) stable; urgency=medium + + * Updated dependencies + + -- NemaChybu team APT archive Thu, 16 Apr 2026 20:41:40 +0200 + +python3-llama-cpp (0:0.3.20-1~w2d0) stable; urgency=medium + + * Release 0:0.3.20-1~w2d0 + + -- NemaChybu team APT archive Tue, 07 May 2019 20:31:30 +0000 diff --git a/debian/control b/debian/control new file mode 100644 index 0000000000..9e8decb6b1 --- /dev/null +++ b/debian/control @@ -0,0 +1,259 @@ +Source: llama-cpp-python +Section: python +Priority: optional +Maintainer: NemaChybu team APT archive +Build-Depends: + debhelper-compat (= 13), + dh-python, + python3-all, + python3-all-dev, + python3-numpy, + python3-diskcache, + python3-jinja2, + python3-typing-extensions, + python3-setuptools, + python3-pip, + python3-wheel, + python3-scikit-build-core (>= 0.9.2), + cmake, + ninja-build, + pkg-config, + libopenblas-dev, + libvulkan-dev, + glslc, + spirv-headers, + nvidia-cuda-toolkit, + cuda-toolkit-12-9, + cuda-toolkit-13-3, + amdrocm-core-sdk10.1 (>= 10.1.0), + g++, + intel-oneapi-compiler-dpcpp-cpp, + intel-oneapi-mkl-sycl-devel, + intel-oneapi-dnnl-devel, + intel-oneapi-tbb-devel, + libze-dev, + patchelf +Build-Conflicts: + libamdhip64-dev +Standards-Version: 4.7.2 +Homepage: https://github.com/abetlen/llama-cpp-python +Rules-Requires-Root: no + +Package: python3-llama-cpp-python +Architecture: all +Depends: + python3-llama-cpp-python-core (= ${binary:Version}), + python3-llama-cpp-python-backend-cpu (= ${binary:Version}), + ${misc:Depends} +Recommends: python3-starlette-context (>= 0.5.1) +Description: Python bindings for llama.cpp (default runtime) + This package installs the llama-cpp-python runtime together with the + dynamically selected CPU backend. + . + Optional accelerator backends are provided by separate packages. + +Package: python3-llama-cpp-python-core +Architecture: any +Breaks: + python3-llama-cpp-cpu-generic (<< ${binary:Version}), + python3-llama-cpp-cpu-avx (<< ${binary:Version}), + python3-llama-cpp-cpu-avx2 (<< ${binary:Version}), + python3-llama-cpp-cpu-avx512 (<< ${binary:Version}), + python3-llama-cpp-vulkan (<< ${binary:Version}), + python3-llama-cpp-cuda (<< ${binary:Version}) +Replaces: + python3-llama-cpp-cpu-generic (<< ${binary:Version}), + python3-llama-cpp-cpu-avx (<< ${binary:Version}), + python3-llama-cpp-cpu-avx2 (<< ${binary:Version}), + python3-llama-cpp-cpu-avx512 (<< ${binary:Version}), + python3-llama-cpp-vulkan (<< ${binary:Version}), + python3-llama-cpp-cuda (<< ${binary:Version}) +Depends: + ${python3:Depends}, + ${shlibs:Depends}, + ${misc:Depends}, + python3-numpy, + python3-diskcache, + python3-jinja2, + python3-typing-extensions +Description: Python bindings for llama.cpp - core runtime + This package contains the Python modules and common llama.cpp/ggml shared + libraries. Hardware-specific backends are packaged separately. + +Package: python3-llama-cpp-python-backend-cpu +Architecture: any +Breaks: + python3-llama-cpp-vulkan (<< ${binary:Version}), + python3-llama-cpp-cuda (<< ${binary:Version}) +Replaces: + python3-llama-cpp-vulkan (<< ${binary:Version}), + python3-llama-cpp-cuda (<< ${binary:Version}) +Depends: + python3-llama-cpp-python-core (= ${binary:Version}), + ${shlibs:Depends}, + ${misc:Depends} +Description: llama-cpp-python dynamic CPU backend + This package contains dynamically selected CPU backends for llama-cpp-python. + On x86-64, llama.cpp selects the best supported implementation at runtime. + +Package: python3-llama-cpp-python-backend-vulkan +Architecture: any +Breaks: + python3-llama-cpp-vulkan (<< ${binary:Version}) +Replaces: + python3-llama-cpp-vulkan (<< ${binary:Version}) +Depends: + python3-llama-cpp-python (= ${binary:Version}), + ${shlibs:Depends}, + ${misc:Depends} +Description: llama-cpp-python dynamic Vulkan backend + This package contains the dynamically loadable Vulkan backend for + llama-cpp-python. The Vulkan loader runtime is pulled in through shared + library dependencies. + +Package: python3-llama-cpp-python-backend-cuda12.4 +Architecture: any +Provides: python3-llama-cpp-python-backend-cuda (= ${binary:Version}) +Depends: + python3-llama-cpp-python (= ${binary:Version}), + ${shlibs:Depends}, + ${misc:Depends} +Description: llama-cpp-python dynamic CUDA 12.4 backend + This package contains the dynamically loadable CUDA backend for + llama-cpp-python, built with the CUDA 12.4 toolkit from Debian Trixie. + Multiple versioned CUDA backend packages may be installed in parallel; + update-alternatives selects the default CUDA backend. + +Package: python3-llama-cpp-python-backend-cuda12.9 +Architecture: any +Provides: python3-llama-cpp-python-backend-cuda (= ${binary:Version}) +Depends: + python3-llama-cpp-python (= ${binary:Version}), + cuda-cudart-12-9, + libcublas-12-9, + ${shlibs:Depends}, + ${misc:Depends} +Description: llama-cpp-python dynamic CUDA 12.9 backend + This package contains the dynamically loadable CUDA backend for + llama-cpp-python, built with the NVIDIA CUDA 12.9 toolkit. + Multiple versioned CUDA backend packages may be installed in parallel; + update-alternatives selects the default CUDA backend. + +Package: python3-llama-cpp-python-backend-cuda13.3 +Architecture: any +Provides: python3-llama-cpp-python-backend-cuda (= ${binary:Version}) +Depends: + python3-llama-cpp-python (= ${binary:Version}), + cuda-cudart-13-3, + libcublas-13-3, + ${shlibs:Depends}, + ${misc:Depends} +Description: llama-cpp-python dynamic CUDA 13.3 backend + This package contains the dynamically loadable CUDA backend for + llama-cpp-python, built with the NVIDIA CUDA 13.3 toolkit. + Multiple versioned CUDA backend packages may be installed in parallel; + update-alternatives selects the default CUDA backend. + +Package: python3-llama-cpp-python-backend-rocm10.1 +Architecture: amd64 +Provides: python3-llama-cpp-python-backend-rocm (= ${binary:Version}) +Depends: + python3-llama-cpp-python (= ${binary:Version}), + amdrocm-runtime10.1, + amdrocm-blas10.1, + ${shlibs:Depends}, + ${misc:Depends} +Description: llama-cpp-python dynamic ROCm 10.1 HIP backend + This package contains the dynamically loadable HIP backend for + llama-cpp-python, built with AMD ROCm Core SDK 10.1. It contains code + objects for the RDNA and CDNA targets distributed by the SDK. Legacy + Vega gfx900/gfx906/gfx90c targets are not included in this backend. + Multiple versioned ROCm backend packages may be installed in parallel; + update-alternatives selects the default ROCm backend. + +Package: python3-llama-cpp-python-backend-sycl +Architecture: amd64 +Depends: + python3-llama-cpp-python (= ${binary:Version}), + intel-oneapi-compiler-dpcpp-cpp-runtime, + intel-oneapi-mkl-sycl-blas, + intel-oneapi-dnnl, + intel-oneapi-tbb, + ${shlibs:Depends}, + ${misc:Depends} +Description: llama-cpp-python dynamic Intel SYCL backend + This package contains the dynamically loadable SYCL backend for + llama-cpp-python, built with Intel oneAPI DPC++/C++. It targets Intel + GPUs through SYCL/Level Zero, uses FP16 kernels, oneMKL SYCL BLAS and + oneDNN, and keeps the backend separate from the common Python runtime. + +Package: python3-llama-cpp +Architecture: any +Section: oldlibs +Depends: + python3-llama-cpp-python (= ${binary:Version}), + ${misc:Depends} +Description: transitional package for python3-llama-cpp-python + This package is provided for a smooth upgrade to the new package layout. + It can be safely removed after the upgrade. + +Package: python3-llama-cpp-cpu-generic +Architecture: any +Section: oldlibs +Depends: + python3-llama-cpp-python (= ${binary:Version}), + ${misc:Depends} +Description: transitional package for python3-llama-cpp-python + This package is provided for a smooth upgrade from the old generic CPU build. + It can be safely removed after the upgrade. + +Package: python3-llama-cpp-cpu-avx +Architecture: any +Section: oldlibs +Depends: + python3-llama-cpp-python (= ${binary:Version}), + ${misc:Depends} +Description: transitional package for python3-llama-cpp-python + This package is provided for a smooth upgrade from the old AVX CPU build. + It can be safely removed after the upgrade. + +Package: python3-llama-cpp-cpu-avx2 +Architecture: any +Section: oldlibs +Depends: + python3-llama-cpp-python (= ${binary:Version}), + ${misc:Depends} +Description: transitional package for python3-llama-cpp-python + This package is provided for a smooth upgrade from the old AVX2 CPU build. + It can be safely removed after the upgrade. + +Package: python3-llama-cpp-cpu-avx512 +Architecture: any +Section: oldlibs +Depends: + python3-llama-cpp-python (= ${binary:Version}), + ${misc:Depends} +Description: transitional package for python3-llama-cpp-python + This package is provided for a smooth upgrade from the old AVX512 CPU build. + It can be safely removed after the upgrade. + +Package: python3-llama-cpp-vulkan +Architecture: any +Section: oldlibs +Depends: + python3-llama-cpp-python-backend-vulkan (= ${binary:Version}), + ${misc:Depends} +Description: transitional package for llama-cpp-python Vulkan backend + This package is provided for a smooth upgrade from the old Vulkan build. + It can be safely removed after the upgrade. + +Package: python3-llama-cpp-cuda +Architecture: any +Section: oldlibs +Depends: + python3-llama-cpp-python-backend-cuda12.4 (= ${binary:Version}), + ${misc:Depends} +Description: transitional package for llama-cpp-python CUDA backend + This package is provided for a smooth upgrade from the old CUDA build. + It installs the Debian Trixie CUDA 12.4 backend and can be safely removed + after the upgrade. diff --git a/debian/copyright b/debian/copyright new file mode 100644 index 0000000000..b425baf056 --- /dev/null +++ b/debian/copyright @@ -0,0 +1,16 @@ +Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ + +Files: * +Copyright: 2023 Andrei Betlen +License: MIT + +License: MIT + MIT License + + Copyright (c) 2023 Andrei Betlen + + Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: + + The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. \ No newline at end of file diff --git a/debian/cuda12.9-glibc2.41.py b/debian/cuda12.9-glibc2.41.py new file mode 100644 index 0000000000..2bab6ebdff --- /dev/null +++ b/debian/cuda12.9-glibc2.41.py @@ -0,0 +1,71 @@ +#!/usr/bin/python3 +import re +import sys +from pathlib import Path + + +def fail(message: str) -> None: + print(f"cuda12.9-glibc2.41: {message}", file=sys.stderr) + raise SystemExit(1) + + +if len(sys.argv) != 2: + fail("usage: cuda12.9-glibc2.41.py PATH/TO/math_functions.h") + +header = Path(sys.argv[1]) +try: + text = header.read_text() +except OSError as exc: + fail(f"cannot read {header}: {exc}") + +functions = ( + ("sinpi", "double"), + ("sinpif", "float"), + ("cospi", "double"), + ("cospif", "float"), +) + +for name, ctype in functions: + # CUDA 12.9 declares these device builtins without the exception + # specification used by glibc 2.41. Accept arbitrary whitespace so the + # workaround is independent of line-number and documentation changes in + # NVIDIA's header, but require the exact CUDA declaration shape. + declaration = ( + rf"(?m)^(?P\s*extern\s+__DEVICE_FUNCTIONS_DECL__\s+" + rf"__device_builtin__\s+{ctype}\s+{name}\s*\(\s*{ctype}\s+x\s*\))" + ) + fixed_re = re.compile(declaration + r"\s+noexcept\s*\(\s*true\s*\)\s*;\s*$") + old_re = re.compile(declaration + r"\s*;\s*$") + + fixed = list(fixed_re.finditer(text)) + if len(fixed) == 1: + print(f"cuda12.9-glibc2.41: {name} already has noexcept(true)") + continue + if len(fixed) > 1: + fail(f"found {len(fixed)} already-fixed declarations for {name}, expected one") + + old = list(old_re.finditer(text)) + if len(old) != 1: + fail(f"found {len(old)} unpatched declarations for {name}, expected one") + + text, count = old_re.subn(r"\g noexcept (true);", text, count=1) + if count != 1: + fail(f"failed to patch declaration for {name}") + print(f"cuda12.9-glibc2.41: patched {name}") + +# Final strict verification. This catches both a silently changed NVIDIA +# declaration and an incomplete transformation before nvcc is ever invoked. +for name, ctype in functions: + verify_re = re.compile( + rf"(?m)^\s*extern\s+__DEVICE_FUNCTIONS_DECL__\s+__device_builtin__\s+" + rf"{ctype}\s+{name}\s*\(\s*{ctype}\s+x\s*\)\s+" + rf"noexcept\s*\(\s*true\s*\)\s*;\s*$" + ) + matches = list(verify_re.finditer(text)) + if len(matches) != 1: + fail(f"verification failed for {name}: found {len(matches)} matching declarations") + +try: + header.write_text(text) +except OSError as exc: + fail(f"cannot write {header}: {exc}") diff --git a/debian/patches/0001-debian-dynamic-backend-path.patch b/debian/patches/0001-debian-dynamic-backend-path.patch new file mode 100644 index 0000000000..2d80c779bc --- /dev/null +++ b/debian/patches/0001-debian-dynamic-backend-path.patch @@ -0,0 +1,61 @@ +Description: Install and discover dynamic GGML backends in llama_cpp/lib + llama-cpp-python installs its shared libraries below the Python package, while + GGML's dynamic backend loader otherwise searches the executable directory and + current working directory. For Debian builds, use the package-local library + directory both as the CMake backend install directory and as the runtime search + path. Debian package builds also suppress the upstream source-tree install copy + so failed builds do not leave compiled libraries in llama_cpp/lib. +Author: NemaChybu team APT archive +Forwarded: not-needed + +--- a/CMakeLists.txt ++++ b/CMakeLists.txt +@@ -4,6 +4,7 @@ + + option(LLAMA_BUILD "Build llama.cpp shared library and install alongside python package" ON) + option(LLAVA_BUILD "Build llava shared library and install alongside python package" ON) ++option(LLAMA_CPP_PYTHON_DEBIAN_BUILD "Use Debian package installation layout" OFF) + + function(llama_cpp_python_install_target target) + if(NOT TARGET ${target}) +@@ -16,7 +17,7 @@ + ) + endif() + +- if(NOT EMSCRIPTEN) ++ if(NOT EMSCRIPTEN AND NOT LLAMA_CPP_PYTHON_DEBIAN_BUILD) + install( + TARGETS ${target} + LIBRARY DESTINATION ${CMAKE_CURRENT_SOURCE_DIR}/llama_cpp/lib +@@ -67,6 +68,16 @@ + set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE) + set(CMAKE_SKIP_RPATH FALSE) + ++ # GGML_BACKEND_DL installs generated backend modules through ++ # CMAKE_INSTALL_BINDIR. Point dynamic backends and the normal llama.cpp ++ # library install rules at the same package-local directory. Without the ++ # LIBDIR override, scikit-build also stages a duplicate copy in ++ # ${SKBUILD_PLATLIB_DIR}/lib. ++ if (GGML_BACKEND_DL) ++ set(CMAKE_INSTALL_BINDIR "${SKBUILD_PLATLIB_DIR}/llama_cpp/lib" CACHE PATH "Install dynamic GGML backends" FORCE) ++ set(CMAKE_INSTALL_LIBDIR "${SKBUILD_PLATLIB_DIR}/llama_cpp/lib" CACHE PATH "Install llama.cpp libraries" FORCE) ++ endif() ++ + # Enable building of the common library + set(LLAMA_BUILD_COMMON ON CACHE BOOL "Build llama.cpp common library" FORCE) + +@@ -132,6 +143,14 @@ + + + add_subdirectory(vendor/llama.cpp) ++ ++ # Do not set the GGML_BACKEND_DIR CMake cache variable here: upstream also ++ # uses it as an install destination. Define only the runtime loader macro, ++ # so installed libggml.so searches Debian's package-local backend directory. ++ if (GGML_BACKEND_DL AND TARGET ggml) ++ target_compile_definitions(ggml PRIVATE ++ GGML_BACKEND_DIR="/usr/lib/python3/dist-packages/llama_cpp/lib") ++ endif() + + if (WIN32) + if (TARGET llama) diff --git a/debian/patches/0002-hip-gfx1250-compatible-kernels.patch b/debian/patches/0002-hip-gfx1250-compatible-kernels.patch new file mode 100644 index 0000000000..aad0c0d4c4 --- /dev/null +++ b/debian/patches/0002-hip-gfx1250-compatible-kernels.patch @@ -0,0 +1,92 @@ +Description: Use compatible HIP kernels for gfx1250 + Separate gfx1250 from the RDNA4 instruction paths in both device macros and + host dispatch. Keep native integer dot4 and wave32 tile kernels, use FP32 + arithmetic instead of unsupported FP16 dot2 assembly, and let hipBLAS use + the hardware matrix units. Custom GGML gfx1250 WMMA kernels are not enabled. +Author: ByCzech +Origin: other, local Debian compatibility fix for vendored llama.cpp 4df29be and 0c1e570 +Forwarded: no +Last-Update: 2026-10-11 + +diff --git a/vendor/llama.cpp/ggml/src/ggml-cuda/common.cuh b/vendor/llama.cpp/ggml/src/ggml-cuda/common.cuh +index 50713c0..3220639 100644 +--- a/vendor/llama.cpp/ggml/src/ggml-cuda/common.cuh ++++ b/vendor/llama.cpp/ggml/src/ggml-cuda/common.cuh +@@ -82,6 +82,7 @@ + #define GGML_CUDA_CC_RDNA3 (GGML_CUDA_CC_OFFSET_AMD + 0x1100) // RX 7000, minimum for WMMA + #define GGML_CUDA_CC_RDNA3_5 (GGML_CUDA_CC_OFFSET_AMD + 0x1150) // AI 370, AI Max 395 laptops. + #define GGML_CUDA_CC_RDNA4 (GGML_CUDA_CC_OFFSET_AMD + 0x1200) // RX 9000 ++#define GGML_CUDA_CC_GFX1250 (GGML_CUDA_CC_OFFSET_AMD + 0x1250) + + #define GGML_CUDA_CC_IS_AMD(cc) (cc >= GGML_CUDA_CC_OFFSET_AMD) + #define GGML_CUDA_CC_IS_RDNA(cc) (cc >= GGML_CUDA_CC_RDNA1) +@@ -93 +94,2 @@ +-#define GGML_CUDA_CC_IS_RDNA4(cc) (cc >= GGML_CUDA_CC_RDNA4) ++#define GGML_CUDA_CC_IS_GFX1250(cc) ((cc) == GGML_CUDA_CC_GFX1250) ++#define GGML_CUDA_CC_IS_RDNA4(cc) (cc >= GGML_CUDA_CC_RDNA4 && !GGML_CUDA_CC_IS_GFX1250(cc)) +@@ -325,7 +327,7 @@ static bool fast_fp16_hardware_available(const int cc) { + // To be used for feature selection of external libraries, e.g. cuBLAS. + static bool fp16_mma_hardware_available(const int cc) { + return (GGML_CUDA_CC_IS_NVIDIA(cc) && cc >= GGML_CUDA_CC_VOLTA) || +- GGML_CUDA_CC_IS_CDNA(cc) || GGML_CUDA_CC_IS_RDNA3(cc) || GGML_CUDA_CC_IS_RDNA4(cc) || ++ GGML_CUDA_CC_IS_CDNA(cc) || GGML_CUDA_CC_IS_RDNA3(cc) || GGML_CUDA_CC_IS_RDNA4(cc) || GGML_CUDA_CC_IS_GFX1250(cc) || + (GGML_CUDA_CC_IS_MTHREADS(cc) && cc >= GGML_CUDA_CC_QY2); + } + +@@ -712,7 +714,7 @@ static __device__ __forceinline__ int ggml_cuda_dp4a(const int a, const int b, i + #if defined(GGML_USE_HIP) + #if defined(CDNA) || defined(RDNA2) || defined(__gfx906__) + c = __builtin_amdgcn_sdot4(a, b, c, false); +-#elif defined(RDNA3) || defined(RDNA4) ++#elif defined(RDNA3) || defined(RDNA4) || defined(GFX1250) + c = __builtin_amdgcn_sudot4( true, a, true, b, c, false); + #elif defined(RDNA1) || defined(__gfx900__) + int tmp1; +@@ -765,7 +767,7 @@ static __device__ __forceinline__ void ggml_cuda_mad(float & acc, const half2 v, + #ifdef V_DOT2_F32_F16_AVAILABLE + asm volatile("v_dot2_f32_f16 %0, %1, %2, %0" : "+v"(acc) : "v"(v), "v"(u)); + #else +-#ifdef FAST_FP16_AVAILABLE ++#if defined(FAST_FP16_AVAILABLE) && !defined(GFX1250) + const float2 tmp = __half22float2(v*u); + acc += tmp.x + tmp.y; + #else +@@ -773,7 +775,7 @@ static __device__ __forceinline__ void ggml_cuda_mad(float & acc, const half2 v, + const float2 tmpu = __half22float2(u); + acc += tmpv.x * tmpu.x; + acc += tmpv.y * tmpu.y; +-#endif // FAST_FP16_AVAILABLE ++#endif // defined(FAST_FP16_AVAILABLE) && !defined(GFX1250) + #endif // V_DOT2_F32_F16_AVAILABLE + } + +diff --git a/vendor/llama.cpp/ggml/src/ggml-cuda/vendors/hip.h b/vendor/llama.cpp/ggml/src/ggml-cuda/vendors/hip.h +index da7e3da..e1f26ee 100644 +--- a/vendor/llama.cpp/ggml/src/ggml-cuda/vendors/hip.h ++++ b/vendor/llama.cpp/ggml/src/ggml-cuda/vendors/hip.h +@@ -212,9 +212,11 @@ + #define CDNA // For the entire family + #endif // defined(CDNA4) || defined(CDNA3) || defined(CDNA2) || defined(CDNA1) + +-#if defined(__GFX12__) ++#if defined(__gfx1250__) ++#define GFX1250 ++#elif defined(__GFX12__) + #define RDNA4 +-#endif // defined(__GFX12__) ++#endif // defined(__gfx1250__) + + #if defined(__GFX11__) + #define RDNA3 +@@ -237,9 +239,9 @@ + #define RDNA1 + #endif // defined(__gfx1010__) || defined(__gfx1012__) + +-#if defined(RDNA4) || defined(RDNA3) || defined(RDNA2) || defined(RDNA1) ++#if defined(GFX1250) || defined(RDNA4) || defined(RDNA3) || defined(RDNA2) || defined(RDNA1) + #define RDNA // For the entire family +-#endif // defined(RDNA4) || defined(RDNA3) || defined(RDNA2) || defined(RDNA1) ++#endif // defined(GFX1250) || defined(RDNA4) || defined(RDNA3) || defined(RDNA2) || defined(RDNA1) + + #ifndef __has_builtin + #define __has_builtin(x) 0 diff --git a/debian/patches/series b/debian/patches/series new file mode 100644 index 0000000000..6f9918c012 --- /dev/null +++ b/debian/patches/series @@ -0,0 +1,2 @@ +0001-debian-dynamic-backend-path.patch +0002-hip-gfx1250-compatible-kernels.patch diff --git a/debian/python3-llama-cpp-python-backend-cuda12.4.alternatives b/debian/python3-llama-cpp-python-backend-cuda12.4.alternatives new file mode 100644 index 0000000000..f3a7f0d347 --- /dev/null +++ b/debian/python3-llama-cpp-python-backend-cuda12.4.alternatives @@ -0,0 +1,4 @@ +Name: llama-cpp-python-cuda +Link: /usr/lib/python3/dist-packages/llama_cpp/lib/libggml-cuda.so +Alternative: /usr/lib/python3/dist-packages/llama_cpp/lib/backends/cuda12.4/libggml-cuda.so +Priority: 124 diff --git a/debian/python3-llama-cpp-python-backend-cuda12.9.alternatives b/debian/python3-llama-cpp-python-backend-cuda12.9.alternatives new file mode 100644 index 0000000000..3fa44e5606 --- /dev/null +++ b/debian/python3-llama-cpp-python-backend-cuda12.9.alternatives @@ -0,0 +1,4 @@ +Name: llama-cpp-python-cuda +Link: /usr/lib/python3/dist-packages/llama_cpp/lib/libggml-cuda.so +Alternative: /usr/lib/python3/dist-packages/llama_cpp/lib/backends/cuda12.9/libggml-cuda.so +Priority: 129 diff --git a/debian/python3-llama-cpp-python-backend-cuda13.3.alternatives b/debian/python3-llama-cpp-python-backend-cuda13.3.alternatives new file mode 100644 index 0000000000..94db39d8b2 --- /dev/null +++ b/debian/python3-llama-cpp-python-backend-cuda13.3.alternatives @@ -0,0 +1,4 @@ +Name: llama-cpp-python-cuda +Link: /usr/lib/python3/dist-packages/llama_cpp/lib/libggml-cuda.so +Alternative: /usr/lib/python3/dist-packages/llama_cpp/lib/backends/cuda13.3/libggml-cuda.so +Priority: 133 diff --git a/debian/python3-llama-cpp-python-backend-rocm10.1.alternatives b/debian/python3-llama-cpp-python-backend-rocm10.1.alternatives new file mode 100644 index 0000000000..8b989274a9 --- /dev/null +++ b/debian/python3-llama-cpp-python-backend-rocm10.1.alternatives @@ -0,0 +1,4 @@ +Name: llama-cpp-python-rocm +Link: /usr/lib/python3/dist-packages/llama_cpp/lib/libggml-hip.so +Alternative: /usr/lib/python3/dist-packages/llama_cpp/lib/backends/rocm10.1/libggml-hip.so +Priority: 1010 diff --git a/debian/python3-llama-cpp-python-backend-sycl.alternatives b/debian/python3-llama-cpp-python-backend-sycl.alternatives new file mode 100644 index 0000000000..64be4fec74 --- /dev/null +++ b/debian/python3-llama-cpp-python-backend-sycl.alternatives @@ -0,0 +1,4 @@ +Name: llama-cpp-python-sycl +Link: /usr/lib/python3/dist-packages/llama_cpp/lib/libggml-sycl.so +Alternative: /usr/lib/python3/dist-packages/llama_cpp/lib/backends/sycl/libggml-sycl.so +Priority: 100 diff --git a/debian/rules b/debian/rules new file mode 100755 index 0000000000..214aff97a0 --- /dev/null +++ b/debian/rules @@ -0,0 +1,480 @@ +#!/usr/bin/make -f + +CORE_PACKAGE := python3-llama-cpp-python-core +CPU_PACKAGE := python3-llama-cpp-python-backend-cpu +VULKAN_PACKAGE := python3-llama-cpp-python-backend-vulkan +CUDA124_PACKAGE := python3-llama-cpp-python-backend-cuda12.4 +CUDA129_PACKAGE := python3-llama-cpp-python-backend-cuda12.9 +CUDA133_PACKAGE := python3-llama-cpp-python-backend-cuda13.3 +ROCM101_PACKAGE := python3-llama-cpp-python-backend-rocm10.1 +SYCL_PACKAGE := python3-llama-cpp-python-backend-sycl +PYLIB_DIR := usr/lib/python3/dist-packages/llama_cpp/lib +CUDA124_BUILD_DIR := build-cuda12.4 +CUDA129_BUILD_DIR := build-cuda12.9 +CUDA133_BUILD_DIR := build-cuda13.3 +ROCM101_BUILD_DIR := build-rocm10.1 +SYCL_BUILD_DIR := build-sycl +CUDA124_BACKEND_DIR := $(PYLIB_DIR)/backends/cuda12.4 +CUDA129_BACKEND_DIR := $(PYLIB_DIR)/backends/cuda12.9 +CUDA133_BACKEND_DIR := $(PYLIB_DIR)/backends/cuda13.3 +ROCM101_BACKEND_DIR := $(PYLIB_DIR)/backends/rocm10.1 +SYCL_BACKEND_DIR := $(PYLIB_DIR)/backends/sycl +CUDA129_ROOT := /usr/local/cuda-12.9 +CUDA129_SHADOW := $(CURDIR)/debian/.cuda-12.9-shadow +CUDA133_ROOT := /usr/local/cuda-13.3 +ROCM101_ROOT := /opt/rocm/core-10.1 +ROCM101_CMAKE_DIR := $(ROCM101_ROOT)/lib/cmake +ROCM101_INCLUDE_DIR := $(ROCM101_ROOT)/include +ROCM101_LLVM_DIR := $(ROCM101_ROOT)/lib/llvm +ROCM101_DEVICE_LIB_DIR := $(ROCM101_ROOT)/amdgcn/bitcode +ONEAPI_ROOT := /opt/intel/oneapi +ONEAPI_COMPILER_ROOT := $(ONEAPI_ROOT)/compiler/latest +ONEAPI_MKL_ROOT := $(ONEAPI_ROOT)/mkl/latest +ONEAPI_DNNL_ROOT := $(ONEAPI_ROOT)/dnnl/latest +ONEAPI_TBB_ROOT := $(ONEAPI_ROOT)/tbb/latest +GNU_GCC_INSTALL_DIR := $(shell dirname "$$(g++ -print-file-name=crtbeginS.o)") +ONEAPI_LD_LIBRARY_PATH := $(ONEAPI_COMPILER_ROOT)/lib:$(ONEAPI_MKL_ROOT)/lib:$(ONEAPI_DNNL_ROOT)/lib:$(ONEAPI_TBB_ROOT)/lib +ONEAPI_CMAKE_PREFIX_PATH := $(ONEAPI_COMPILER_ROOT):$(ONEAPI_MKL_ROOT):$(ONEAPI_DNNL_ROOT):$(ONEAPI_TBB_ROOT) +# ROCm 10.1 device/BLAS targets verified against AMD Debian 13 metadata. +# gfx900/gfx906/gfx90c are not distributed in this SDK. +# gfx1250 uses the compatible kernels provided by the Debian HIP patch. +ROCM101_TARGETS := gfx908;gfx90a;gfx942;gfx950;gfx1010;gfx1011;gfx1012;gfx1030;gfx1031;gfx1032;gfx1033;gfx1034;gfx1035;gfx1036;gfx1100;gfx1101;gfx1102;gfx1103;gfx1150;gfx1151;gfx1152;gfx1153;gfx1200;gfx1201;gfx1250 +DEB_VERSION := $(shell dpkg-parsechangelog -SVersion) +DEB_HOST_MULTIARCH := $(shell dpkg-architecture -qDEB_HOST_MULTIARCH) +DEB_SYSTEM_LIBDIR := /usr/lib/$(DEB_HOST_MULTIARCH) + +.PHONY: clean +clean: + # Never invoke dh_auto_clean/pybuild here. A failed pip/scikit-build build + # may leave partially configured CMake trees, and asking the build system + # to clean them can configure or build again. + dh_clean + rm -rf .pybuild build _skbuild dist wheelhouse $(CUDA124_BUILD_DIR) \ + $(CUDA129_BUILD_DIR) $(CUDA133_BUILD_DIR) $(ROCM101_BUILD_DIR) $(SYCL_BUILD_DIR) $(CUDA129_SHADOW) \ + .scikit-build .pytest_cache .mypy_cache .ruff_cache \ + llama_cpp_python.egg-info pip-wheel-metadata + find llama_cpp -type d -name '__pycache__' -prune -exec rm -rf {} + 2>/dev/null || true + find llama_cpp/lib -maxdepth 1 -type f \ + \( -name '*.so' -o -name '*.so.*' -o -name '*.dylib' \ + -o -name '*.dll' -o -name '*.a' -o -name '*.lib' \) \ + -delete 2>/dev/null || true + +%: + dh $@ --with python3 --buildsystem=pybuild + +# The actual CMake configure/build/install is driven by scikit-build-core below. +# Avoid creating a second, unused .pybuild CMake tree first. +override_dh_auto_configure: + : + +override_dh_auto_build: + : + +override_dh_auto_test: + : + +override_dh_auto_install: + # Upstream 0.3.36 bindings require this C API. Check the actual source + # headers too, so an outdated checkout or orig tarball fails immediately. + grep -q 'llama_adapter_lora_init_from_file_ptr' vendor/llama.cpp/include/llama.h || { \ + echo "llama-cpp-python 0.3.36 requires vendored llama.cpp 0c1e57098bba43ac29e6e3b677cdceebdd22334f; update the submodule and orig source archive" >&2; \ + exit 1; \ + } + + rm -rf debian/$(CORE_PACKAGE) debian/$(CPU_PACKAGE) debian/$(VULKAN_PACKAGE) \ + debian/$(CUDA124_PACKAGE) debian/$(CUDA129_PACKAGE) debian/$(CUDA133_PACKAGE) \ + debian/$(ROCM101_PACKAGE) debian/$(SYCL_PACKAGE) \ + $(CUDA124_BUILD_DIR) $(CUDA129_BUILD_DIR) $(CUDA133_BUILD_DIR) $(ROCM101_BUILD_DIR) $(SYCL_BUILD_DIR) + + CMAKE_ARGS="-DLLAMA_CPP_PYTHON_DEBIAN_BUILD=ON \ + -DGGML_NATIVE=OFF \ + -DGGML_BACKEND_DL=ON \ + -DGGML_CPU_ALL_VARIANTS=ON \ + -DGGML_VULKAN=ON \ + -DGGML_OPENMP=ON \ + -DGGML_BLAS=ON \ + -DGGML_BLAS_VENDOR=OpenBLAS" \ + DEB_PYTHON_INSTALL_LAYOUT=deb \ + python3 -m pip install . \ + --no-deps \ + --no-build-isolation \ + --prefix=/usr \ + --ignore-installed \ + --root=$(CURDIR)/debian/$(CORE_PACKAGE) + + # Validate the staged Python/C API before the expensive accelerator builds. + # Leave the source directory so Python cannot import its uninstalled package. + cd debian/$(CORE_PACKAGE) && \ + env -u LLAMA_CPP_LIB_PATH -u LD_LIBRARY_PATH \ + PYTHONPATH="$(CURDIR)/debian/$(CORE_PACKAGE)/usr/lib/python3/dist-packages" \ + python3 -c 'from llama_cpp import Llama; print("Staged llama_cpp import passed")' + + # Split dynamically loadable CPU/BLAS backends out of the common runtime. + mkdir -p debian/$(CPU_PACKAGE)/$(PYLIB_DIR) + find debian/$(CORE_PACKAGE)/$(PYLIB_DIR) -maxdepth 1 \ + \( -name 'libggml-cpu-*.so*' -o -name 'libggml-blas.so*' \) \ + -exec mv -t debian/$(CPU_PACKAGE)/$(PYLIB_DIR) {} + + + # Split the dynamically loadable Vulkan backend into its own package. + mkdir -p debian/$(VULKAN_PACKAGE)/$(PYLIB_DIR) + find debian/$(CORE_PACKAGE)/$(PYLIB_DIR) -maxdepth 1 \ + -name 'libggml-vulkan.so*' \ + -exec mv -t debian/$(VULKAN_PACKAGE)/$(PYLIB_DIR) {} + + + # Build the CUDA 12.4 backend separately from the common runtime. + # This keeps CUDA toolkit selection isolated so CUDA 12.9/13 backends can + # later be built from the same source tree and co-installed. + /usr/bin/nvcc --version | grep -Eq 'release 12\.4([, ])' + cmake -S vendor/llama.cpp -B $(CUDA124_BUILD_DIR) -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_CUDA_COMPILER=/usr/bin/nvcc \ + -DBUILD_SHARED_LIBS=ON \ + -DLLAMA_BUILD_COMMON=OFF \ + -DLLAMA_BUILD_TESTS=OFF \ + -DLLAMA_BUILD_TOOLS=OFF \ + -DLLAMA_BUILD_EXAMPLES=OFF \ + -DLLAMA_BUILD_SERVER=OFF \ + -DLLAMA_BUILD_APP=OFF \ + -DLLAMA_BUILD_UI=OFF \ + -DGGML_NATIVE=OFF \ + -DGGML_BACKEND_DL=ON \ + -DGGML_CPU=OFF \ + -DGGML_BLAS=OFF \ + -DGGML_CUDA=ON \ + -DGGML_CUDA_GRAPHS=ON \ + -DGGML_CUDA_NCCL=OFF \ + -DGGML_OPENMP=ON \ + -DGGML_BUILD_TESTS=OFF \ + -DGGML_BUILD_EXAMPLES=OFF + cmake --build $(CUDA124_BUILD_DIR) --target ggml-cuda + + # Keep the real backend in a versioned private directory. The public + # libggml-cuda.so path is managed by update-alternatives. + mkdir -p debian/$(CUDA124_PACKAGE)/$(CUDA124_BACKEND_DIR) + cp -a $(CUDA124_BUILD_DIR)/bin/libggml-cuda.so* \ + debian/$(CUDA124_PACKAGE)/$(CUDA124_BACKEND_DIR)/ + find debian/$(CUDA124_PACKAGE)/$(CUDA124_BACKEND_DIR) -maxdepth 1 \ + -type f -name 'libggml-cuda.so*' \ + -exec patchelf --set-rpath '$$ORIGIN/../..' {} + + + # Build CUDA 12.9 from a private shadow of NVIDIA's versioned toolkit. + # CUDA 12.9 headers conflict with glibc 2.41's noexcept declarations for + # sinpi/cospi. Never modify files owned by NVIDIA packages in /usr/local: + # create a symlink farm, materialize only nvcc and the affected header, and + # patch that private copy for this build. + $(CUDA129_ROOT)/bin/nvcc --version | grep -Eq 'release 12\.9([, ])' + rm -rf $(CUDA129_SHADOW) + cp -as $(CUDA129_ROOT)/. $(CUDA129_SHADOW)/ + rm -f $(CUDA129_SHADOW)/bin/nvcc + cp -a $(CUDA129_ROOT)/bin/nvcc $(CUDA129_SHADOW)/bin/nvcc + rm -f $(CUDA129_SHADOW)/targets/x86_64-linux/include/crt/math_functions.h + cp -a $(CUDA129_ROOT)/targets/x86_64-linux/include/crt/math_functions.h \ + $(CUDA129_SHADOW)/targets/x86_64-linux/include/crt/math_functions.h + python3 debian/cuda12.9-glibc2.41.py \ + $(CUDA129_SHADOW)/targets/x86_64-linux/include/crt/math_functions.h + $(CUDA129_SHADOW)/bin/nvcc --version | grep -Eq 'release 12\.9([, ])' + cmake -S vendor/llama.cpp -B $(CUDA129_BUILD_DIR) -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_CUDA_COMPILER=$(CUDA129_SHADOW)/bin/nvcc \ + -DCUDAToolkit_ROOT=$(CUDA129_SHADOW) \ + -DBUILD_SHARED_LIBS=ON \ + -DLLAMA_BUILD_COMMON=OFF \ + -DLLAMA_BUILD_TESTS=OFF \ + -DLLAMA_BUILD_TOOLS=OFF \ + -DLLAMA_BUILD_EXAMPLES=OFF \ + -DLLAMA_BUILD_SERVER=OFF \ + -DLLAMA_BUILD_APP=OFF \ + -DLLAMA_BUILD_UI=OFF \ + -DGGML_NATIVE=OFF \ + -DGGML_BACKEND_DL=ON \ + -DGGML_CPU=OFF \ + -DGGML_BLAS=OFF \ + -DGGML_CUDA=ON \ + -DGGML_CUDA_GRAPHS=ON \ + -DGGML_CUDA_NCCL=OFF \ + -DGGML_OPENMP=ON \ + -DGGML_BUILD_TESTS=OFF \ + -DGGML_BUILD_EXAMPLES=OFF + cmake --build $(CUDA129_BUILD_DIR) --target ggml-cuda + mkdir -p debian/$(CUDA129_PACKAGE)/$(CUDA129_BACKEND_DIR) + cp -a $(CUDA129_BUILD_DIR)/bin/libggml-cuda.so* \ + debian/$(CUDA129_PACKAGE)/$(CUDA129_BACKEND_DIR)/ + find debian/$(CUDA129_PACKAGE)/$(CUDA129_BACKEND_DIR) -maxdepth 1 \ + -type f -name 'libggml-cuda.so*' \ + -exec patchelf --set-rpath '$$ORIGIN/../..' {} + + + # Build CUDA 13.3 independently for Turing and newer NVIDIA GPUs. + $(CUDA133_ROOT)/bin/nvcc --version | grep -Eq 'release 13\.3([, ])' + cmake -S vendor/llama.cpp -B $(CUDA133_BUILD_DIR) -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_CUDA_COMPILER=$(CUDA133_ROOT)/bin/nvcc \ + -DCUDAToolkit_ROOT=$(CUDA133_ROOT) \ + -DBUILD_SHARED_LIBS=ON \ + -DLLAMA_BUILD_COMMON=OFF \ + -DLLAMA_BUILD_TESTS=OFF \ + -DLLAMA_BUILD_TOOLS=OFF \ + -DLLAMA_BUILD_EXAMPLES=OFF \ + -DLLAMA_BUILD_SERVER=OFF \ + -DLLAMA_BUILD_APP=OFF \ + -DLLAMA_BUILD_UI=OFF \ + -DGGML_NATIVE=OFF \ + -DGGML_BACKEND_DL=ON \ + -DGGML_CPU=OFF \ + -DGGML_BLAS=OFF \ + -DGGML_CUDA=ON \ + -DGGML_CUDA_GRAPHS=ON \ + -DGGML_CUDA_NCCL=OFF \ + -DGGML_OPENMP=ON \ + -DGGML_BUILD_TESTS=OFF \ + -DGGML_BUILD_EXAMPLES=OFF + cmake --build $(CUDA133_BUILD_DIR) --target ggml-cuda + mkdir -p debian/$(CUDA133_PACKAGE)/$(CUDA133_BACKEND_DIR) + cp -a $(CUDA133_BUILD_DIR)/bin/libggml-cuda.so* \ + debian/$(CUDA133_PACKAGE)/$(CUDA133_BACKEND_DIR)/ + find debian/$(CUDA133_PACKAGE)/$(CUDA133_BACKEND_DIR) -maxdepth 1 \ + -type f -name 'libggml-cuda.so*' \ + -exec patchelf --set-rpath '$$ORIGIN/../..' {} + + + # Build the ROCm 10.1 HIP backend independently. The new TheRock SDK lives + # under a versioned /opt/rocm/core-X.Y prefix. Debian's legacy ROCm 5.7 + # development headers in /usr/include/hip take precedence over TheRock's + # compiler-provided -idirafter include path, so they must not be installed + # while this source package is built (also declared via Build-Conflicts). + test -x $(ROCM101_ROOT)/bin/hipcc + test -x $(ROCM101_LLVM_DIR)/bin/clang++ + test -f $(ROCM101_INCLUDE_DIR)/hip/hip_fp16.h + test ! -e /usr/include/hip/hip_fp16.h || { \ + echo "legacy Debian HIP headers found in /usr/include/hip; remove libamdhip64-dev (or run apt-get build-dep . to satisfy Build-Conflicts)" >&2; \ + exit 1; \ + } + test -d $(ROCM101_DEVICE_LIB_DIR) + test -f $(ROCM101_CMAKE_DIR)/hip/hip-config.cmake + test -f $(ROCM101_CMAKE_DIR)/hip-lang/hip-lang-config.cmake + test -f $(ROCM101_CMAKE_DIR)/hipblas/hipblas-config.cmake + test -f $(ROCM101_CMAKE_DIR)/rocblas/rocblas-config.cmake + # SDK version and HIP component version have different numbering. + grep -Eq '^10\.1([.]|$$)' $(ROCM101_ROOT)/.info/version + $(ROCM101_ROOT)/bin/hipconfig --version | grep -Eq '(^|[^0-9])7\.16([.]|[^0-9]|$$)' + + # Cheap compiler smoke test before the expensive multi-architecture build. + mkdir -p $(ROCM101_BUILD_DIR) + printf '%s\n' '#include ' '__global__ void rocm_path_probe() {}' \ + > $(ROCM101_BUILD_DIR)/hip-path-probe.hip + HIP_PLATFORM=amd HIP_PATH=$(ROCM101_ROOT) ROCM_PATH=$(ROCM101_ROOT) \ + HIP_CLANG_PATH=$(ROCM101_LLVM_DIR)/bin \ + HIP_INCLUDE_PATH=$(ROCM101_INCLUDE_DIR) \ + HIP_DEVICE_LIB_PATH=$(ROCM101_DEVICE_LIB_DIR) DEVICE_LIB_PATH=$(ROCM101_DEVICE_LIB_DIR) \ + $(ROCM101_LLVM_DIR)/bin/clang++ \ + --offload-arch=gfx908 \ + -x hip -c $(ROCM101_BUILD_DIR)/hip-path-probe.hip \ + -o $(ROCM101_BUILD_DIR)/hip-path-probe.o + rm -f $(ROCM101_BUILD_DIR)/hip-path-probe.hip $(ROCM101_BUILD_DIR)/hip-path-probe.o + + HIP_PLATFORM=amd HIP_PATH=$(ROCM101_ROOT) ROCM_PATH=$(ROCM101_ROOT) \ + HIP_CLANG_PATH=$(ROCM101_LLVM_DIR)/bin \ + HIP_INCLUDE_PATH=$(ROCM101_INCLUDE_DIR) \ + HIP_DEVICE_LIB_PATH=$(ROCM101_DEVICE_LIB_DIR) DEVICE_LIB_PATH=$(ROCM101_DEVICE_LIB_DIR) \ + PATH=$(ROCM101_ROOT)/bin:$(ROCM101_LLVM_DIR)/bin:$$PATH \ + cmake -S vendor/llama.cpp -B $(ROCM101_BUILD_DIR) -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_HIP_COMPILER=$(ROCM101_LLVM_DIR)/bin/clang++ \ + -DCMAKE_HIP_COMPILER_ROCM_ROOT=$(ROCM101_ROOT) \ + -DCMAKE_PREFIX_PATH=$(ROCM101_CMAKE_DIR) \ + -Dhip_DIR=$(ROCM101_CMAKE_DIR)/hip \ + -Dhipblas_DIR=$(ROCM101_CMAKE_DIR)/hipblas \ + -Drocblas_DIR=$(ROCM101_CMAKE_DIR)/rocblas \ + -DHIP_PLATFORM=amd \ + -DCMAKE_HIP_ARCHITECTURES="$(ROCM101_TARGETS)" \ + -DBUILD_SHARED_LIBS=ON \ + -DLLAMA_BUILD_COMMON=OFF \ + -DLLAMA_BUILD_TESTS=OFF \ + -DLLAMA_BUILD_TOOLS=OFF \ + -DLLAMA_BUILD_EXAMPLES=OFF \ + -DLLAMA_BUILD_SERVER=OFF \ + -DLLAMA_BUILD_APP=OFF \ + -DLLAMA_BUILD_UI=OFF \ + -DGGML_NATIVE=OFF \ + -DGGML_BACKEND_DL=ON \ + -DGGML_CPU=OFF \ + -DGGML_BLAS=OFF \ + -DGGML_HIP=ON \ + -DGGML_HIP_GRAPHS=ON \ + -DGGML_HIP_RCCL=OFF \ + -DGGML_HIP_NO_VMM=ON \ + -DGGML_OPENMP=ON \ + -DGGML_BUILD_TESTS=OFF \ + -DGGML_BUILD_EXAMPLES=OFF + # Fail before the expensive compile if CMake resolved any HIP package from + # Debian's legacy ROCm in /usr instead of the pinned ROCm 10.1 SDK. + grep -Fxq 'hip_DIR:UNINITIALIZED=$(ROCM101_CMAKE_DIR)/hip' $(ROCM101_BUILD_DIR)/CMakeCache.txt || \ + grep -Fxq 'hip_DIR:PATH=$(ROCM101_CMAKE_DIR)/hip' $(ROCM101_BUILD_DIR)/CMakeCache.txt + grep -Fxq 'hipblas_DIR:UNINITIALIZED=$(ROCM101_CMAKE_DIR)/hipblas' $(ROCM101_BUILD_DIR)/CMakeCache.txt || \ + grep -Fxq 'hipblas_DIR:PATH=$(ROCM101_CMAKE_DIR)/hipblas' $(ROCM101_BUILD_DIR)/CMakeCache.txt + grep -Fxq 'rocblas_DIR:UNINITIALIZED=$(ROCM101_CMAKE_DIR)/rocblas' $(ROCM101_BUILD_DIR)/CMakeCache.txt || \ + grep -Fxq 'rocblas_DIR:PATH=$(ROCM101_CMAKE_DIR)/rocblas' $(ROCM101_BUILD_DIR)/CMakeCache.txt + cmake --build $(ROCM101_BUILD_DIR) --target ggml-hip + mkdir -p debian/$(ROCM101_PACKAGE)/$(ROCM101_BACKEND_DIR) + cp -a $(ROCM101_BUILD_DIR)/bin/libggml-hip.so* \ + debian/$(ROCM101_PACKAGE)/$(ROCM101_BACKEND_DIR)/ + find debian/$(ROCM101_PACKAGE)/$(ROCM101_BACKEND_DIR) -maxdepth 1 \ + -type f -name 'libggml-hip.so*' \ + -exec patchelf --set-rpath '$$ORIGIN/../..:$(ROCM101_ROOT)/lib' {} + + + # Build the Intel SYCL backend independently with the oneAPI DPC++ compiler. + # Keep compiler and library roots explicit instead of relying on a sourced + # interactive setvars.sh environment. + test -x $(ONEAPI_COMPILER_ROOT)/bin/icx + test -x $(ONEAPI_COMPILER_ROOT)/bin/icpx + test -x /usr/bin/g++ + test -f $(GNU_GCC_INSTALL_DIR)/crtbeginS.o + test -f $(GNU_GCC_INSTALL_DIR)/libstdc++.so + test -d $(ONEAPI_MKL_ROOT) + test -d $(ONEAPI_DNNL_ROOT) + test -d $(ONEAPI_TBB_ROOT) + test -f $(ONEAPI_TBB_ROOT)/lib/cmake/tbb/TBBConfig.cmake + $(ONEAPI_COMPILER_ROOT)/bin/icpx --version + rm -rf $(SYCL_BUILD_DIR) + mkdir -p $(SYCL_BUILD_DIR) + # IntelLLVM can otherwise select an unrelated gcc-cross installation when + # multiple Debian cross toolchains are installed. Resolve the native GCC + # installation through the default g++ driver and pin it exactly. + printf '%s\n' '#include ' 'int main() { std::cout << "SYCL toolchain OK\\n"; }' \ + > $(SYCL_BUILD_DIR)/icpx-sycl-probe.cpp + LD_LIBRARY_PATH=$(ONEAPI_LD_LIBRARY_PATH):$${LD_LIBRARY_PATH:-} \ + $(ONEAPI_COMPILER_ROOT)/bin/icpx -fsycl \ + --gcc-install-dir=$(GNU_GCC_INSTALL_DIR) \ + $(SYCL_BUILD_DIR)/icpx-sycl-probe.cpp \ + -o $(SYCL_BUILD_DIR)/icpx-sycl-probe + LD_LIBRARY_PATH=$(ONEAPI_LD_LIBRARY_PATH):$${LD_LIBRARY_PATH:-} \ + $(SYCL_BUILD_DIR)/icpx-sycl-probe + rm -f $(SYCL_BUILD_DIR)/icpx-sycl-probe.cpp $(SYCL_BUILD_DIR)/icpx-sycl-probe + ONEAPI_ROOT=$(ONEAPI_ROOT) \ + MKLROOT=$(ONEAPI_MKL_ROOT) \ + DNNLROOT=$(ONEAPI_DNNL_ROOT) \ + TBBROOT=$(ONEAPI_TBB_ROOT) \ + CFLAGS=--gcc-install-dir=$(GNU_GCC_INSTALL_DIR) \ + CXXFLAGS=--gcc-install-dir=$(GNU_GCC_INSTALL_DIR) \ + CMAKE_PREFIX_PATH=$(ONEAPI_CMAKE_PREFIX_PATH) \ + PATH=$(ONEAPI_COMPILER_ROOT)/bin:$$PATH \ + LD_LIBRARY_PATH=$(ONEAPI_LD_LIBRARY_PATH):$${LD_LIBRARY_PATH:-} \ + cmake -S vendor/llama.cpp -B $(SYCL_BUILD_DIR) -G Ninja \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_C_COMPILER=$(ONEAPI_COMPILER_ROOT)/bin/icx \ + -DCMAKE_CXX_COMPILER=$(ONEAPI_COMPILER_ROOT)/bin/icpx \ + -DCMAKE_C_FLAGS=--gcc-install-dir=$(GNU_GCC_INSTALL_DIR) \ + -DCMAKE_CXX_FLAGS=--gcc-install-dir=$(GNU_GCC_INSTALL_DIR) \ + "-DCMAKE_PREFIX_PATH=$(ONEAPI_COMPILER_ROOT);$(ONEAPI_MKL_ROOT);$(ONEAPI_DNNL_ROOT);$(ONEAPI_TBB_ROOT)" \ + -DTBB_DIR=$(ONEAPI_TBB_ROOT)/lib/cmake/tbb \ + -DBUILD_SHARED_LIBS=ON \ + -DLLAMA_BUILD_COMMON=OFF \ + -DLLAMA_BUILD_TESTS=OFF \ + -DLLAMA_BUILD_TOOLS=OFF \ + -DLLAMA_BUILD_EXAMPLES=OFF \ + -DLLAMA_BUILD_SERVER=OFF \ + -DLLAMA_BUILD_APP=OFF \ + -DLLAMA_BUILD_UI=OFF \ + -DGGML_NATIVE=OFF \ + -DGGML_BACKEND_DL=ON \ + -DGGML_CPU=OFF \ + -DGGML_BLAS=OFF \ + -DGGML_SYCL=ON \ + -DGGML_SYCL_TARGET=INTEL \ + -DGGML_SYCL_F16=ON \ + -DGGML_SYCL_GRAPH=ON \ + -DGGML_SYCL_HOST_MEM_FALLBACK=ON \ + -DGGML_SYCL_SUPPORT_LEVEL_ZERO_API=ON \ + -DGGML_SYCL_DNN=ON \ + -DGGML_OPENMP=ON \ + -DGGML_BUILD_TESTS=OFF \ + -DGGML_BUILD_EXAMPLES=OFF + grep -Fxq 'GGML_SYCL:BOOL=ON' $(SYCL_BUILD_DIR)/CMakeCache.txt + grep -Fxq 'GGML_SYCL_TARGET:STRING=INTEL' $(SYCL_BUILD_DIR)/CMakeCache.txt + ONEAPI_ROOT=$(ONEAPI_ROOT) \ + MKLROOT=$(ONEAPI_MKL_ROOT) \ + DNNLROOT=$(ONEAPI_DNNL_ROOT) \ + TBBROOT=$(ONEAPI_TBB_ROOT) \ + PATH=$(ONEAPI_COMPILER_ROOT)/bin:$$PATH \ + LD_LIBRARY_PATH=$(ONEAPI_LD_LIBRARY_PATH):$${LD_LIBRARY_PATH:-} \ + cmake --build $(SYCL_BUILD_DIR) --target ggml-sycl + mkdir -p debian/$(SYCL_PACKAGE)/$(SYCL_BACKEND_DIR) + cp -a $(SYCL_BUILD_DIR)/bin/libggml-sycl.so* \ + debian/$(SYCL_PACKAGE)/$(SYCL_BACKEND_DIR)/ + find debian/$(SYCL_PACKAGE)/$(SYCL_BACKEND_DIR) -maxdepth 1 \ + -type f -name 'libggml-sycl.so*' \ + -exec patchelf --set-rpath '$$ORIGIN/../..:$(ONEAPI_COMPILER_ROOT)/lib:$(ONEAPI_MKL_ROOT)/lib:$(ONEAPI_DNNL_ROOT)/lib:$(ONEAPI_TBB_ROOT)/lib' {} + + +override_dh_python3: + dh_python3 -p$(CORE_PACKAGE) --no-ext-rename + +# libggml-base has a SONAME and is shared across our split backend packages, +# but it lives in a private Python package directory that dh_makeshlibs does +# not automatically describe. Provide exact in-tree shlibs metadata for it. +override_dh_makeshlibs: + dh_makeshlibs -a + install -d debian/$(CORE_PACKAGE)/DEBIAN + { \ + if [ -f debian/$(CORE_PACKAGE)/DEBIAN/shlibs ]; then \ + grep -v '^libggml-base[[:space:]]' debian/$(CORE_PACKAGE)/DEBIAN/shlibs || true; \ + fi; \ + printf '%s\n' 'libggml-base 0 $(CORE_PACKAGE) (= $(DEB_VERSION))'; \ + } > debian/$(CORE_PACKAGE)/DEBIAN/shlibs.tmp + mv debian/$(CORE_PACKAGE)/DEBIAN/shlibs.tmp \ + debian/$(CORE_PACKAGE)/DEBIAN/shlibs + +override_dh_shlibdeps: + # Common libraries first. + dh_shlibdeps -p$(CORE_PACKAGE) \ + -l/$(PYLIB_DIR) + + # Dynamic CPU backends link against libggml-base from the core package. + dh_shlibdeps -p$(CPU_PACKAGE) \ + -L$(CORE_PACKAGE) \ + -l/$(PYLIB_DIR) + + # Vulkan backend links against the common ggml libraries from core and + # against the system Vulkan loader. + dh_shlibdeps -p$(VULKAN_PACKAGE) \ + -L$(CORE_PACKAGE) \ + -l/$(PYLIB_DIR) + + # CUDA backend links against libggml-base from core and the CUDA 12.4 + # runtime libraries from Debian. + dh_shlibdeps -p$(CUDA124_PACKAGE) \ + -L$(CORE_PACKAGE) \ + -l/$(PYLIB_DIR):$(DEB_SYSTEM_LIBDIR) \ + -- -Ldebian/shlibs.cuda12 + + # NVIDIA CUDA toolkits are installed side-by-side under /usr/local. + # dh_shlibdeps accepts one -l option containing a colon-separated path list. + # Keep the private llama_cpp library directory first so libggml-base is + # resolved from the core package, then add the matching NVIDIA toolkit. + dh_shlibdeps -p$(CUDA129_PACKAGE) \ + -L$(CORE_PACKAGE) \ + -l/$(PYLIB_DIR):$(CUDA129_ROOT)/targets/x86_64-linux/lib \ + -- -Ldebian/shlibs.cuda12 --ignore-missing-info + + dh_shlibdeps -p$(CUDA133_PACKAGE) \ + -L$(CORE_PACKAGE) \ + -l/$(PYLIB_DIR):$(CUDA133_ROOT)/targets/x86_64-linux/lib \ + -- -Ldebian/shlibs.cuda13 --ignore-missing-info + + # ROCm Core SDK 10.1 is an external versioned runtime under /opt. Keep + # explicit runtime/BLAS Depends in debian/control and still let shlibdeps + # collect normal Debian dependencies. AMD's external packages do not + # consistently provide Debian shlibs/symbols metadata for every ROCm DSO. + dh_shlibdeps -p$(ROCM101_PACKAGE) \ + -L$(CORE_PACKAGE) \ + -l/$(PYLIB_DIR):$(ROCM101_ROOT)/lib \ + -- --ignore-missing-info + + # Intel oneAPI libraries live under /opt and external oneAPI packages do not + # consistently provide Debian shlibs/symbols metadata. Runtime packages are + # explicit in debian/control; still collect ordinary Debian dependencies + # (including Level Zero) here. + dh_shlibdeps -p$(SYCL_PACKAGE) \ + -L$(CORE_PACKAGE) \ + -l/$(PYLIB_DIR):$(ONEAPI_COMPILER_ROOT)/lib:$(ONEAPI_MKL_ROOT)/lib:$(ONEAPI_DNNL_ROOT)/lib:$(ONEAPI_TBB_ROOT)/lib \ + -- --ignore-missing-info + +override_dh_missing: + dh_missing --fail-missing diff --git a/debian/shlibs.cuda12 b/debian/shlibs.cuda12 new file mode 100644 index 0000000000..8471765f65 --- /dev/null +++ b/debian/shlibs.cuda12 @@ -0,0 +1 @@ +libcuda 1 libcuda1 (>= 525.60.13) diff --git a/debian/shlibs.cuda13 b/debian/shlibs.cuda13 new file mode 100644 index 0000000000..1ffce591d5 --- /dev/null +++ b/debian/shlibs.cuda13 @@ -0,0 +1 @@ +libcuda 1 libcuda1 (>= 580) diff --git a/debian/source/format b/debian/source/format new file mode 100644 index 0000000000..163aaf8d82 --- /dev/null +++ b/debian/source/format @@ -0,0 +1 @@ +3.0 (quilt)