da Agency

What I learned about Kubernetes and Knative Serverless

If you happened to miss this year’s Kubernetes Summer Camp, there’s some good news! The sessions were recorded and are available for on-demand viewing. Along with those, you’ll also get access to a variety of downloadable content, including a free O’Reilly e-book. Here’s some of what you’ll learn.
Quelle: CloudForms

da Agency

Spark on Google Cloud: Serverless Spark jobs made seamless for all data users

Apache Spark has become a popular platform as it can serve all of data engineering, data exploration, and machine learning use cases. However, Spark still requires the on-premises way of managing clusters and tuning infrastructure for each job. Also, end to end use cases require Spark to be used along with technologies like TensorFlow, and programming languages like SQL and Python. Today, these operate in silos, with Spark on unstructured data lakes, SQL on data warehouses, and TensorFlow in completely separate machine learning platforms. This increases costs, reduces agility, and makes governance extremely hard; prohibiting enterprises from making insights available to the right users at the right time.Announcing Spark on Google Cloud, now serverless and integratedWe are excited to announce Spark on Google Cloud, bringing industry’s first autoscaling serverless Spark, seamlessly integrated with the best of Google Cloud and open source tools, so you can effortlessly power ETL, data science, and data analytics use cases at scale. Google Cloud has been running large scale business critical Spark workloads for enterprise customers for 6+ years, using open source Spark in Dataproc. Today, we are furthering our commitment by enabling customers to:Eliminate time spent managing Spark clusters: With serverless Spark, users submit their Spark jobs, and let them do auto-provision, and autoscale to finish.Enable data users of all levels: Connect, analyze, and execute Spark jobs from the interface of users’ choice including BigQuery, Vertex AI or Dataplex, in 2 clicks, without any custom integrations.Retain flexibility of consumption: No one size fits all. Use Spark as serverless, deploy on Google Kubernetes Engine (GKE), or on compute clusters based on the requirements.With Spark on Google Cloud, we are providing a way for customers to use Spark in a cloud native manner (serverless), and seamlessly with tools used by data engineers, data analysts, and data scientists for their use cases. These tools will help customers on their way to realize the data platform redesign they have embarked on.”Deutsche Bank is using Spark for a variety of different use cases. Migrating to GCP and adopting Serverless Spark for Dataproc allows us to optimize our resource utilization and reduce manual effort so our engineering teams can focus on delivering data products for our business instead of managing infrastructure. At the same time we can retain the existing code base and knowhow of our engineers, thus boosting adoption and making the migration a seamless experience.”—Balaji Maragalla, Director Big Data Platform, Deutsche Bank“We see serverless Spark playing a central role in our data strategy. Serverless Spark will provide an efficient, seamless solution for teams that aren’t familiar with big data technology or don’t need to bother with idiosyncrasies of Spark to solve their own processing needs. We’re excited about the serverless aspect of the offering, as well as the seamless integration with BigQuery, Vertex AI, Dataplex and other data services.” —Saral Jain, Director of Engineering, Infrastructure and Data, Snap Inc.Dataproc Serverless for SparkPer IDC, developers spend 40% time writing code, and 60% of the time tuning infrastructure and managing clusters. Furthermore, not all Spark developers are infrastructure experts, resulting in higher costs and productivity impact. With serverless Spark, developers can spend all their time on the code and logic. They do not need to manage clusters or tune infrastructure. They submit Spark jobs from their interface of choice, and processing is auto-scaled to match the needs of the job. Furthermore, while Spark users today pay for the time the infrastructure is running, with serverless Spark they only pay for the job duration.Spark through BigQueryBigQuery, the leading data warehouse, now provides a unified interface for data analysts to write SQL or PySpark. The code is executed using serverless Spark seamlessly, without the need for infrastructure provisioning. BigQuery has been the pioneer for serverless data warehousing, and now supports serverless Spark for Spark-based analytics.Spark through Vertex AIData scientists no longer need to go through custom integrations to use Spark with their notebooks. Through Vertex AI Workbench, they can connect to Spark with a single click, and do interactive development. With Vertex AI, Spark can easily be used together with other ML frameworks like TensorFlow, Pytorch, Sci-kit learn, and BigQuery ML. All the Google Cloud security, compliance, and IAM are automatically applied across Vertex AI and Spark. Once you are ready to deploy the ML models, the notebook can be executed as a Spark job in Dataproc, and scheduled as part of Vertex AI Pipelines.Spark through DataplexDataplex is an intelligent data fabric that enables organizations to centrally manage, monitor, and govern their data across data lakes, data warehouses, and data marts with consistent controls, providing access to trusted data and powering analytics at scale. Now, you can use Spark on distributed data natively through Dataplex. Dataplex provides a collaborative analytics interface, with 1-click access to SparkSQL, Notebooks, or PySpark, and the ability to save, share, search notebooks and scripts alongside data.Flexibility of consumptionWe understand one size does not fit all. Spark is available for consumption in 3 different ways based on your specific needs. For customers standardizing on Kubernetes for infrastructure management, run Spark on Google Kubernetes Engine (GKE) to improve resource utilization and simplify infrastructure management. For customers looking for Hadoop style infrastructure management, run Spark on Google Compute Engine (GCE). For customers, who’re looking for no-ops Spark deployment, use serverless Spark! ESG Senior Analyst Mike Leone commented, “Google Cloud is making Spark easier to use and more accessible to a wide range of users through a single, integrated platform. The ability to run Spark in a serverless manner, and through BigQuery and Vertex AI will create significant productivity improvement for customers. Further, Google’s focus on security and governance makes this Spark portfolio useful to all enterprises as they continue migrating to the Cloud.”Getting startedDataproc Serverless for Spark will be Generally Available within a few weeks. BigQuery and Dataplex integration is in Private Preview. Vertex AI workbench is available in Public Preview, you can get started here. For all capabilities, you can request for Preview access through this form.You can work with Google Cloud partners to get started as well.“We are excited to partner with Google Cloud as we look to provide our joint customers with the latest innovations on Spark. We see Spark being used for a variety of analytics and ML use cases. Google is taking Spark a step further by making it serverless, and available through BigQuery, Vertex AI and Dataplex for a wide spectrum of users.” —Sharad Kumar, Cloud First data and AI Lead at AccentureFor more information, visit our website or the watch announcement video and our conversation with Snap at Next 2021.
Quelle: Google Cloud Platform

da Agency

Best practices for securing your applications and APIs using Apigee

Enterprises across the globe are seeing surging demand for digital experiences from their customers, employees, and partners. For many of these enterprises, hundreds of business applications are hosted in private or public clouds that interact with their users (customers, partners, and employees) spread across geographies, channels (web, mobile, APIs, VPNs, and cloud services), and time zones. As a consequence of this surge in demand, enterprises are also experiencing increased pressure to fortify their technical infrastructure against cyber attacks. The number of reported cyber attacks on U.S. companies rose 69% in 2020 from the previous year, according to the Federal Bureau of Investigation. Web and API attacks cannot be prevented but can be mitigated—a recent study showed that 55% of organizations experience a DDoS attack at least every month. While many enterprises are accelerating digital transformation to build omnichannel experiences, they need to keep security and privacy top of mind across all of these channels. This goal can only be supported by implementing a robust security architecture and organizational policy enforcement model that enables enterprises to prevent, detect, and react to newer threats, in near-real time. While it is easy to say, implementation of such a system can be extremely challenging. Best practices for securing your applications and APIsTo help organizations navigate these challenges, we recently published, “Best practices for securing your applications and APIs using Apigee,” which describes the best practices and approaches that can help companies secure their applications and APIs using Apigee API management, Google Cloud Armor,reCAPTCHA Enterprise, andCloud CDNThese best practices include using Apigee as a proxy layer to protect backend APIs, Google Cloud Armor as a Web Application Firewall (WAF), Cloud CDN for caching, and comprehensive web app and API protection with the Google Cloud solution. Use Apigee as a proxy layerIn this pattern, Apigee is a facade layer that can secure and protect your backend APIs with its out-of -box capabilities.Apigee offers a wide range of security features that can be applied consistently across all your APIs. It can be used to route the requests to different backends, which helps with your migration effort too. Use Google Cloud Armor as a WAF layer along with ApigeeTo increase your security footprint, you could easily enable Google Cloud Armor along with Apigee. Google Cloud Armor provides web application firewall (WAF) capabilities and helps to prevent distributed denial of service (DDoS) attacks. It can also help you to mitigate the threat to applications from the risks listed in the OWASP Top 10. For more information on how to configure rules in Google Cloud Armor, see the Google Cloud Armor How-to guidesor check out this blog post about Apigee and Google Cloud Armor.Use Cloud CDN for cachingBy using Cloud CDN: Content Delivery Network, you can use the Google global network to serve content closer to users, which accelerates response times for your websites and applications. Cloud CDN also offers caching capabilities to provide responses much faster. It helps you to secure the backend by returning the response from its cache and handling traffic spikes. It can also help to minimize web server load, compute, and network usage. To implement this architecture, you must enable Cloud CDN on the load balancer that’s serving the Apigee traffic. To learn more, check out this blog post.Implement comprehensive Web App and API Protection (WAAP)To further enhance your security profile, you can also use WAAP, which brings together Google Cloud Armor, reCAPTCHA Enterprise, and Apigee to help protect your system against DDoS attacks and bots. It also provides web application firewall (WAF) and API protection. We recommended WAAP for enterprise use cases where the API calls are made from a website or mobile applications. You can set applications to load the reCAPTCHA libraries to generate a reCAPTCHA token and send it along when they make a request. For more information on WAAP, check out this blog post or read this whitepaper.Next stepsAs more and more organizations get into and accelerate their digital transformation journey, systems and business channels will rely more on digital interactions, and the need for tightened levels of security and protection will continue to rise significantly. Building an architecture that can help your organizations deliver fast and efficiently with improved threat protection and visibility is of the utmost importance.  Get started with the “Best practices for securing your applications and APIs using Apigee” Cloud Architecture Pattern Read about OWASP Top 10 mitigation options on Google Cloud from our Cloud Architecture Center and find out how Apigee and other GCP products can help mitigate OWASP Top 10 attacks.View the Enhance API security with Apigee and Cloud Armor videoWatch this video to learn How to protect your APIs against these 6 security threatsRead and ask questions in the Apigee community.Explore the Apigee repository on GitHub.Related ArticleBetter protect your web apps and APIs against threats and fraud with Google CloudHow Google Cloud’s Web App and API Protection (WAAP) solution protects enterprises from rising security & fraud threatsRead Article
Quelle: Google Cloud Platform

da Agency

Cloud Data Loss Prevention is now automatic!

Data is one of your most valuable assets; understanding and using data effectively powers your business. However, it can also be a source of privacy, security, and compliance risk.  The data discovery and classification capabilities of Cloud Data Loss Prevention (DLP) have helped many Google Cloud customers identify where sensitive data resides. However, data growth is outpacing the ability to manually inspect data, and data sprawl means sensitive data is increasingly appearing in places it’s not expected. You need a solution for managing data risk that can automatically scale with the rapid growth of your data without additional management overhead.To help, we are happy to announce that we’re making Cloud DLP automatic: automatic discovery, automatic inspection, automatic classification, automatic data profiling. Now available in preview for BigQuery, you can enable Cloud DLP across your entire organization to gain visibility into your data risk.  With rich insights for each table and column, you can focus on the outcome, manage data risk, and ultimately help to safely accelerate your business. Automatic DLP is an example of Google Cloud’s Invisible Security vision, where the capability to understand and protect your data is engineered into the platform. Here are the benefits of Automatic DLP:Continuous monitoring: Cloud DLP automatically profiles new tables as you create them across your org. Also, it periodically reprofiles tables that you modify.Low overhead: No jobs to manage.  Enable it directly in the Cloud Console for an entire organization or select folders or projects.Data residency: Cloud DLP will inspect your data and generate data profiles in the same geographic region that your data lives in (as configured in BigQuery).Google-driven: Powered by industry leading Cloud DLP, we figure out how to inspect and profile your tables and columns.  You can focus on the outcomes.Rich insights: Table and column profiles give you details about the data risk and sensitivity of your data including Cloud DLP’s predicted infoType.Data profilesA data profile is a set of metrics and insights that Cloud DLP gathers from scanning your data. Among these metrics are the predicted infoTypes found in BigQuery tables, “free text” score, uniqueness score, and the data risk level.  Use these insights to make informed decisions about how you protect, share, and use your data.  You can get results directly in the Cloud Console or export profile details to BigQuery for custom analysis and reporting:Managing your data riskHere are few example scenarios of how DLP profiles can help you understand and manage data risk:Scenario 1: Table found with credit card numbers and a high uniqueness scoreLet’s say that a column in a table with 10M rows was classified with a predicted infoType of “CREDIT_CARD_NUMBER” and a high uniqueness score. This indicates that you likely have 10M unique credit card numbers in this table. A lower uniqueness score might indicate that you have fewer numbers repeated in the table. Potential Action to Take: If this type of data is acceptable for you to store and process, you can lower this data risk by applying a BigQuery Policy Tag which would restrict access to this column to only those with specific permission.  Alternatively, if you do not want to store this raw information, consider tokenizing the data using Cloud DLP’s de-identification methods or solutions for PCI Tokenization. Scenario 2: Table found with several infoTypes and a high free text score. Let’s say that a column in a table does not have a strong predicted infoType but has hints of PHONE_NUMBER, US_SOCIAL_SECURITY_NUMBER, and DATE_OF_BIRTH along with a high “free text” score. This indicates that you may have a column of unstructured data in your table that has occasional instances of PII.  This could, for example, be a note field or comment field where someone types in PII such as “customer was born on 1/1/1985” and is an indication of potential risk.Potential Action to Take:  Consider running a deep scan of this column using Cloud DLP’s on-demand inspection for BigQuery so that you can understand where instances of PII may exist in specific rows or cells.  Or consider using Cloud DLP’s masking capability to replace this table with a de-identified version.Scenario 3: Table found with sensitive data and shared publiclyLet’s say that a table contains customer EMAIL_ADDRESS and PHONE_NUMBER and it was shared with a marketing partner. However, instead of being shared directly, this table was made public. This greatly increases the risks of exposure of this sensitive information. Potential Action to Take:  Adjust permissions to this table to remove public access groups like AllUsers or AllAuthenticatedUsers.  Instead, add the specific users or groups that should have access to the data. Get started with Automatic DLPAutomatic DLP Profiling is available now in preview for BigQuery.    To get started, open the Cloud DLP page in the Cloud Console and check out our documentation.Related ArticleTake charge of your data: using Cloud DLP to de-identify and obfuscate sensitive informationIn our previous “Taking charge of your data” post, we talked about how to gain visibility into your data using the Cloud Data Loss Preven…Read Article
Quelle: Google Cloud Platform