Amazon AWS News, Entwicklungen, Updates, HowTos - Seite 59 von 720

Today, Amazon SageMaker HyperPod announces the general availability of new APIs that enable programmatic rebooting and replacement of SageMaker HyperPod cluster nodes. SageMaker HyperPod helps you provision resilient clusters for running machine learning (ML) workloads and developing state-of-the-art models such as large language models (LLMs), diffusion models, and foundation models (FMs). The new BatchRebootClusterNodes and BatchReplaceClusterNodes APIs enable customers to programmatically reboot or replace unresponsive or degraded cluster nodes, providing a consistent, orchestrator agnostic approach to node recovery operations. The new APIs enhance node management capabilities for both Slurm and EKS orchestrated clusters complementing existing node reboot and replacement workflows. Existing orchestrator-specific methods, such as Kubernetes labels for EKS clusters and Slurm commands for Slurm clusters, remain available alongside the newly introduced programmatic capabilities for reboot and replace operations through these purpose-built APIs. When cluster nodes become unresponsive due to issues such as memory overruns or hardware degradation, recovery operations such as node reboots and replacements maybe be necessary and can be initiated through these new APIs. These capabilities are particularly valuable when running time-sensitive workloads. For instance, when a Slurm controller, login or compute node becomes unresponsive, administrators can trigger a reboot operation using the API and monitor its progress to get nodes back to operational status. Similarly, EKS cluster administrators can replace degraded worker nodes programmatically. Each API supports batch operations of up to 25 instances, enabling efficient management of large-scale recovery scenarios. The reboot and replace APIs are currently supported in three AWS regions where SageMaker HyperPod is available: US East (Ohio), Asia Pacific (Mumbai), and Asia Pacific (Tokyo).The APIs can be accessed through the AWS CLI, SDK, or API calls. For more information, see the Amazon SageMaker HyperPod documentation for BatchRebootClusterNodes and BatchReplaceClusterNodes.
Quelle: aws.amazon.com

27. November 2025

da Agency

SageMaker HyperPod now supports Managed tiered KV cache and intelligent routing

Amazon SageMaker HyperPod now supports Managed Tiered KV Cache and Intelligent Routing for large language model (LLM) inference, enabling customers to optimize inference performance for long-context prompts and multi-turn conversations. Customers deploying production LLM applications need fast response times while processing lengthy documents or maintaining conversation context, but traditional inference approaches require recalculating attention mechanisms for all previous tokens with each new token generation, creating computational overhead and escalating costs. Managed Tiered KV Cache addresses this challenge by intelligently caching and reusing computed values, while Intelligent Routing directs requests to optimal instances. These capabilities deliver up to 40% latency reduction, 25% throughput improvement, and 25% cost savings compared to baseline configurations. The Managed Tiered KV Cache feature uses a two-tier architecture combining local CPU memory (L1) with disaggregated cluster-wide storage (L2). AWS-native disaggregated tiered storage is the recommended backend, providing scalable terabyte-scale capacity and automatic tiering from CPU memory to local SSD for optimal memory and storage utilization. We also offer Redis as an alternative L2 cache option. The architecture enables efficient reuse of previously computed key-value pairs across requests. The newly introduced Intelligent Routing maximizes cache utilization through three configurable strategies: prefix-aware routing for common prompt patterns, KV-aware routing for maximum cache efficiency with real-time cache tracking, and round-robin for stateless workloads. These features work seamlessly together. Intelligent routing directs requests to instances with relevant cached data, reducing time to first token in document analysis and maintaining natural conversation flow in multi-turn dialogues. Built-in observability integration with Amazon Managed Grafana provides metrics for monitoring performance. You can enable these features through InferenceEndpointConfig or SageMaker JumpStart when deploying models via the HyperPod Inference Operator on EKS-orchestrated clusters. These features are available in all regions where SageMaker HyperPod is available. To learn more, see the user guide.
Quelle: aws.amazon.com