A critical caching vulnerability has been identified in llama-server, leading to the reuse of incorrect Key-Value (KV) data when working with LoRA adapters. The core issue is that the RAM caching mechanism saves computation states without recording the specific LoRA adapter scale settings used during the initial processing. Consequently, when a request returns to a slot, the server may apply stale data computed for a different scale, which corrupts the final generation output.
Technically, the bug arises because the LoRA verification mechanism compares the new request against the slot's previous state rather than the cached entry itself. If a slot is vacated by another request and then reoccupied, the server mistakenly assumes the cached data is valid. This leads to a scenario where the model generates text based on mixed configurations: the context is processed under one adapter scale, while the continuation uses another, resulting in unpredictable and incorrect outputs.
This vulnerability is particularly insidious because the server does not report any errors, and the generated tokens appear plausible. On smaller models like stories15M, the discrepancies are obvious, but on large language models using complex stylistic adapters, the error might manifest as a subtle degradation in quality or a shift in tone. Users may remain unaware that they are receiving incorrect data, as the system reports successful cache hits throughout the process.
Research indicates that the problem manifests under a specific sequence: a request with an adapter, its eviction from the cache by another request, and a subsequent request with a different adapter scale. If the server has enough slots and requests do not evict each other, the error might not occur, complicating diagnostics. This is why standard tests, which execute requests sequentially without intermediate operations, often fail to detect this critical flaw in the caching logic.
As a temporary workaround, developers recommend starting the server with the --cache-ram 0 parameter. This disables RAM caching, ensuring the correctness of computations for every request, albeit at the cost of some performance. Alternatively, users can set cache_prompt: false for each individual request. These measures prevent the use of contaminated data until the developers update the caching logic to include adapter parameters in the cache keys, ensuring full state integrity.
This case serves as a vital lesson for developers of caching systems: a cache key must include all variables that affect the computation result. For LLMs, this includes not only input tokens but also adapter configurations. The llama-server experience demonstrates that verifying state based solely on tokens is insufficient when the architecture supports dynamic parameter tuning. Such vulnerabilities highlight the necessity of rigorous testing for caching mechanisms in scenarios involving frequent context switching.
🔗 Original Source: Read Original Article