Monat: November 2025

Amazon SageMaker HyperPod now supports custom Kubernetes labels and taints

Amazon SageMaker HyperPod now supports custom Kubernetes labels and taints, enabling customers to control pod scheduling and integrate seamlessly with existing Kubernetes infrastructure. Customers deploying AI workloads on HyperPod clusters orcehstrated with EKS need precise control over workload placement to prevent expensive GPU resources from being consumed by system pods and non-AI workloads, while ensuring compatibility with custom device plugins such as EFA and NVIDIA GPU operators. Previously, customers had to manually apply labels and taints using kubectl and reapply them after every node replacement, scaling, or patching operation, creating significant operational overhead. This capability allows you to configure labels and taints at the instance group level through the CreateCluster and UpdateCluster APIs, providing a managed approach to defining and maintaining scheduling policies across the entire node lifecycle. Using the new KubernetesConfig parameter, you can specify up to 50 labels and 50 taints per instance group. Labels enable resource organization and pod targeting through node selectors, while taints repel pods without matching tolerations to protect specialized nodes. For example, you can apply NoSchedule taints to GPU instance groups to ensure only AI training jobs with explicit tolerations consume high-cost compute resources, or add custom labels that enable device plugin pods to schedule correctly. HyperPod automatically applies these configurations during node creation and maintains them across replacement, scaling, and patching operations, eliminating manual intervention and reducing operational overhead. This feature is available in all AWS Regions where Amazon SageMaker HyperPod is available. To learn more about custom labels and taints, see the user guide.
Quelle: aws.amazon.com

SageMaker HyperPod now supports Managed tiered KV cache and intelligent routing

Amazon SageMaker HyperPod now supports Managed Tiered KV Cache and Intelligent Routing for large language model (LLM) inference, enabling customers to optimize inference performance for long-context prompts and multi-turn conversations. Customers deploying production LLM applications need fast response times while processing lengthy documents or maintaining conversation context, but traditional inference approaches require recalculating attention mechanisms for all previous tokens with each new token generation, creating computational overhead and escalating costs. Managed Tiered KV Cache addresses this challenge by intelligently caching and reusing computed values, while Intelligent Routing directs requests to optimal instances. These capabilities deliver up to 40% latency reduction, 25% throughput improvement, and 25% cost savings compared to baseline configurations. The Managed Tiered KV Cache feature uses a two-tier architecture combining local CPU memory (L1) with disaggregated cluster-wide storage (L2). AWS-native disaggregated tiered storage is the recommended backend, providing scalable terabyte-scale capacity and automatic tiering from CPU memory to local SSD for optimal memory and storage utilization. We also offer Redis as an alternative L2 cache option. The architecture enables efficient reuse of previously computed key-value pairs across requests. The newly introduced Intelligent Routing maximizes cache utilization through three configurable strategies: prefix-aware routing for common prompt patterns, KV-aware routing for maximum cache efficiency with real-time cache tracking, and round-robin for stateless workloads. These features work seamlessly together. Intelligent routing directs requests to instances with relevant cached data, reducing time to first token in document analysis and maintaining natural conversation flow in multi-turn dialogues. Built-in observability integration with Amazon Managed Grafana provides metrics for monitoring performance. You can enable these features through InferenceEndpointConfig or SageMaker JumpStart when deploying models via the HyperPod Inference Operator on EKS-orchestrated clusters. These features are available in all regions where SageMaker HyperPod is available. To learn more, see the user guide.
Quelle: aws.amazon.com

Amazon SageMaker AI now supports EAGLE speculative decoding

Amazon SageMaker AI now supports EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) speculative decoding to improve large language model inference throughput by up to 2.5x. This capability enables models to predict and validate multiple tokens simultaneously rather than one at a time, improving response times for AI applications. As customers deploy AI applications to production, they need capabilities to serve models with low latency and high throughput to deliver responsive user experiences. Data scientists and ML engineers lack efficient methods to accelerate token generation without sacrificing output quality or requiring complex model re-architecture, making it hard to meet performance expectations under real-world traffic. Teams spend significant time optimizing infrastructure rather than improving their AI applications. With EAGLE speculative decoding, SageMaker AI enables customers to accelerate inference throughput by allowing models to generate and verify multiple tokens in parallel rather than one at a time, maintaining the same output quality while dramatically increasing throughput. SageMaker AI automatically selects between EAGLE 2 and EAGLE 3 based on your model architecture, and provides built-in optimization jobs that use either curated datasets or your own application data to train specialized prediction heads. You can then deploy optimized models through your existing SageMaker AI inference workflow without infrastructure changes, enabling you to deliver faster AI applications with predictable performance. You can use EAGLE speculative decoding in the following AWS Regions: US East (N. Virginia), US West (Oregon), US East (Ohio), Asia Pacific (Tokyo), Europe (Ireland), Asia Pacific (Singapore), and Europe (Frankfurt) To learn more about EAGLE speculative decoding, visit AWS News Blog here, and SageMaker AI documentation here.
Quelle: aws.amazon.com

AWS Lambda adds support for Node.js 24

AWS Lambda now supports creating serverless applications using Node.js 24. Developers can use Node.js 24 as both a managed runtime and a container base image, and AWS will automatically apply updates to the managed runtime and base image as they become available. Node.js 24 is the latest long-term support release of Node.js and is expected to be supported for security and bug fixes until April 2028. With this release, Lambda has simplified the developer experience, focusing on the modern async/await programming pattern and no longer supports callback-based function handlers. You can use Node.js 24 with Lambda@Edge (in supported Regions), allowing you to customize low-latency content delivered through Amazon CloudFront. Powertools for AWS Lambda (TypeScript), a developer toolkit to implement serverless best practices and increase developer velocity, also supports Node.js 24. You can use the full range of AWS deployment tools, including the Lambda console, AWS CLI, AWS Serverless Application Model (AWS SAM), AWS CDK, and AWS CloudFormation to deploy and manage serverless applications written in Node.js 24. The Node.js 24 runtime is available in all Regions, including the AWS GovCloud (US) Regions and China Regions. For more information, including guidance on upgrading existing Lambda functions, see our blog post. For more information about AWS Lambda, visit our product page. 
Quelle: aws.amazon.com

Manage Amazon SageMaker HyperPod clusters with the new Amazon SageMaker AI MCP Server

The Amazon SageMaker AI MCP Server now supports tools that help you setup and manage HyperPod clusters. Amazon SageMaker HyperPod removes the undifferentiated heavy lifting involved in building generative AI models by quickly scaling model development tasks such as training, fine-tuning, or deployment across a cluster of AI accelerators. The SageMaker AI MCP Server now empowers AI coding assistants to provision and operate AI/ML clusters for model training and deployment. MCP servers in AWS provide a standard interface to enhance AI-assisted application development by equipping AI code assistants with real-time, contextual understanding of various AWS services. The SageMaker AI MCP server comes with tools that streamline end-to-end AI/ML cluster operations using the AI assistant of your choice—from initial setup through ongoing management. It enables AI agents to reliably setup HyperPod clusters orchestrated by Amazon EKS or Slurm complete with pre-requisites, powered by CloudFormation templates that optimize networking, storage, and compute resources. Clusters created via this MCP server are fully optimized for high-performance distributed training and inference workloads, leveraging best practice architectures to maximize throughput and minimize latency at scale. Additionally, it provides comprehensive tools for cluster and node management—including scaling operations, applying software patches, and performing various maintenance tasks. When used in conjunction with AWS API MCP Server, AWS Knowledge MCP Server, and Amazon EKS MCP Server you gain complete coverage for all SageMaker HyperPod APIs and you can effectively troubleshoot common issues, such as diagnosing why a cluster node became inaccessible. For cluster administrators, these tools streamline daily operations. For data scientists, they enable you to set up AI/ML clusters at scale without requiring infrastructure expertise, allowing you to focus on what matters most—training and deploying models. You can manage your AI/ML clusters through the SageMaker AI MCP server in all regions where SageMaker HyperPod is available. To get started, visit the AWS MCP Servers documentation.
Quelle: aws.amazon.com