da Agency

Top 5 use cases for Google Cloud Spot VMs explained + best practices

Cloud was built on the premise of flexible infrastructure that grows and shrinks with your application demands. Applications that can take advantage of this elastic infrastructure and scale horizontally with the demands of your application offer significant advantages over competitors by allowing infrastructure costs to scale up and down along with the demand. Google Cloud’s Spot VMs enable our customers to make the most of our idle capacity where and when it is available. Spot VMs are offered at a significant discount from list price to drive maximum savings provided customers have flexible, stateless workloads that can handle preemption. Spot VMs can be reclaimed by Google (with a 30 second notice). When you deploy the right workloads on Spot VMs, you are able to maintain elasticity while also taking advantage of the best discounts Google has to offer.This blog discusses a few common use cases and design patterns we have seen customers utilize Spot VMs for and discusses the best practices for these use cases. While this is not an exhaustive list, this blog serves as a template to help customers make the most of the Spot VM savings while still reaching their application and workload objectives. Media renderingRendering workloads (such as rendering 2D or 3D elements) can be both compute and time intensive, requiring skilled IT resources to manage render farms. Job management becomes even more difficult when the render farm is at 100% utilization. Spot VMs are ideal resources for fault-tolerant rendering workloads; when combined with a queuing system customers can integrate the preemption notice to track preempted jobs. This allows you to build a render farm which benefits from reduced TCO. If your renderer supports taking snapshots of in-progress renders at specified intervals, writing these snapshots to a persistent data store (Cloud Storage) will limit any loss in work in the event the Spot VM is preempted. As subsequent Spot VMs are created, they can pick up where the old ones left off by using the snapshots on Cloud Storage. You can also leverage the new “suspend and resume a VM” feature which allows you to keep the VM instances during the preemption event but not incur any charges for it while the VM is not in use.Additionally, we have helped customers combine local render farms in their existing datacenters with cloud-based render farms, allowing a hybrid approach for large or numerous render workloads without increasing their investment in their physical datacenters. Not only does this reduce their capital expenses, but it adds flexible scalability to the existing farm and provides a better experience for their business partners. Financial modelingCapital market firms have significant investments in their infrastructure to create state-of-the-art, world-class compute grids. Since compute grids began, in-house researchers leverage these large grids in physical datacenters to test their trading hypotheses and perform backtesting. But as the business grows, what happens when all the researchers each have a brilliant idea and want to test that out at the same time? Researchers then have to compete with one another for the same limited resources, which leads to queueing their jobs and increased lead times for testing their ideas. And in financial markets, time is always scarce. Enter cloud computing and Spot VMs. Capital market firms can use Google Cloud as an extension of their on-premises grid by spinning up temporary compute resources. Or they can go all in on cloud and build their grid in Google Cloud entirely. In either scenario, Spot VMs are ideal candidates for bursting research workloads given the transient nature of the workload and heavily discounted prices of VMs. This enables researchers to test more hypotheses at a lower cost per test, in turn producing better models for firms. Google Cloud Spot VM discounts not only apply to the VMs themselves, but also to any GPU accelerator attached to them, providing even more processing power to a firm looking to process larger more complex models. Once these jobs have completed, Spot VMs can be quickly spun down, maintaining strict control on costs. CI/CD pipelinesContinuous integration (CI) and Continuous delivery (CD) tools are very common for the modern application developer. These tools allow developers to create a testing pipeline that enables developers and quality engineers to ensure the newly created code works with their environment and that the deployment process does not break anything during deployment. CI/CD tools and test environments are great workloads to run on Spot VMs since CI/CD pipelines are not mission-critical for most companies — a delay in deployment or testing by 15 minutes, or even a few hours, is not material to their business. This means that companies can lower the cost of operating their CI/CD pipeline significantly through the use of Spot VMs. A simple example of this would be to install the Jenkins Master Server in a Managed Instance Group (MIG) with the VM type set to Spot. If the VM gets preempted, the CI/CD pipelines will stall until the MIG can find resources again to spin up a new VM. The first reaction may be concern that Jenkins persists data locally, which is problematic for Spot VMs. However, customers can move the Jenkins directory (/var/lib/Jenkins) to Google Cloud Filestore and preserve this data. Then when the new Spot VM spins up, it will reconnect to the directory. In the case of a large-scale Jenkins deployment, build VMs can utilize Spot VMs as part of a MIG to scale as necessary while ensuring that the builds can be maintained with on-demand VMs. This blended approach removes any risk to the builds, while still allowing customers to save up to 91% in costs of the additional VMs versus traditional on-demand VMs.Web services and appsLarge online retailers have found ways to drive massive increases in order volume. Typically companies like this target a specific time each month, such as the last day of the month, through a unique promotion process. This means that they are in many cases creating a Black Friday/Cyber Monday-style event, each and every month! In order to support this, companies traditionally used a “Build it like a stadium for Super Bowl Sunday” model. The issue with that, and a reason most professional sports teams have practice facilities, is that it’s very expensive to keep all the lights, climate control, and ancillary equipment running for the sole purpose of practice. 29-30 days of a month most infrastructure sits idle, wasting HVAC, electricity, etc. However, using the elasticity of cloud, we could manage this capacity and turn it up only when necessary. But to drive even more optimization and savings, we turn to Spot VMs. Spot VMs really shine during these kinds of scale-out events. Imagine the above scenario: what if behind a load balancer we could have:One MIG to help scale the web frontends. This MIG will be sized with on-demand VMs to handle day-to-day traffic.A second MIG for Spot VMs that scales up starting at 11:45pm the night prior to the end of month. The first and second MIG can now handle ~80-90% of the workload. A third MIG of on-demand VMs that spins up as a workload bursts to handle any remaining traffic, should the Spot MIG not be able to find enough capacity, thus ensuring we’re meeting our SLAs as well as keeping costs as tight as possible. KubernetesNow you may say “Well that’s all well and good, but we’re a fully modernized container shop, using Google Kubernetes Engine (GKE).” You are in luck — Spot VMs are integrated with GKE, enabling you to quickly and easily save on your GKE workloads by using Spot VMs with standard GKE clusters or Spot Pods with your Autopilotclusters. GKE supports gracefully shutting downSpot VMs, notifying your workloads that they will be shut down and giving them time to cleanly exit. GKE then automatically reschedules your deployments. With Spot Pods, you can use Kubernetes nodeSelectors and/or Node affinity to control the placement of spot workloads, striking the right balance between cost and availability across spot and on-demand compute.General best practicesTo take advantage of Spot VMs, your use case doesn’t have to be an exact match to any of those described above. If the workload is stateless, scalable, can be stopped and checkpointed in less than 30 seconds, or is location- and hardware-flexible, then they may be a good fit for Spot VMs.There are many several actions you can take to help ensure your Spot workloads run as smoothly as possible. Below we outline a few best practices you should consider:1. Deploy Spot behindRegional Managed Instance Groups (RMIGs):RMIGs are a great fit for Spot workloads given the RMIG’s ability to recreate instances which are preempted.Using your workload’s profile, determine the RMIG’s target distribution shape. For example, with a batch research workload, you might select an ANY target distribution shape. This will allow for Spot instances to be distributed in any manner across the various zones, thereby taking advantage of any underutilized resources. You can use a mix of on-demand RMIGs and Spot RMIGs to maintain stateful applications while increasing availability in a cost effective manner.2. Ensure you have a shutdown script:In the event of Spot VM preemptions, use a shutdown script to enable checkpointing to Cloud Storage for your workloads as well as perform any graceful shutdown processes.When drafting your shutdown script, test it out on an instance by either manually stopping or deleting the instance with the shutdown script attached and validate the intended behavior.3. Write check-point files to Cloud Storage.4. Consider using multiple MIGs behind your load balancer.Whether your workload is graphics rendering, financial modeling, scaled-out ecommerce, or any other stateless use case, Spot VMs are the best and easiest way to reduce your cost of operating it by more than 60%. By following the examples and best practices above, you can ensure that Spot VMs will create the right outcome. Get started today with a free trial of Google Cloud. AcknowledgementSpecial thanks to Dan Sheppard, Product Manager for Cloud Compute, for contributing to this post.
Quelle: Google Cloud Platform

da Agency

Building a Mobility Dashboard with Cloud Run and Firestore

Visualization is the key to understanding massive amounts of data. Today we have BigQuery and Looker to analyze petabytes scale data and to extract insights in a sophisticated way. But how about monitoring data that actively changes every second? In this post, we will walk through how to build a real-time dashboard with Cloud Run and Firestore.Mobility DashboardThere are many business use cases that require real-time updates. For example, inventory monitoring in retail stores, security cameras, and MaaS (Mobility as a Service) applications such as share ride. In the MaaS business area, locations of vehicles are very useful in making business decisions. In this post, we are going to build a mobility dashboard, monitoring vehicles on a map in real-time.The ArchitectureThe dashboard should be accessible from the web browser without any setups on the client side. Cloud Run is a good fit because it can generate URLs, and of course, scalable that can handle millions of users. Now we need to implement an app that can plot geospatial data, and a database that can broadcast its update. Here are my choices and architecture.Cloud Run — Hosting a web app (dashboard)(streamlit — a library to visualize data and to make web app)(pydeck — a library to plot geospatial data)Firestore — a full managed database that keeps your data in syncThe diagram below illustrates a brief architecture of the system. In the production environment, you may also need to implement a data ingestion and transform pipeline.Before going to the final form, let’s take some steps to understand each component.Step 1: Build a data visualization web app with Cloud Run + streamlitstreamlit is an OSS web app framework that can create beautiful data visualization apps without knowledge of the front-end (e.g. HTML, JS). If you are familiar with pandas DataFrame for your data analytics, it won’t take time to implement. For example, you can easily visualize your DataFrame in a few lines of code.code_block[StructValue([(u’code’, u”import streamlit as strnchart_data = pd.DataFrame(rn np.random.randn(20, 3),rn columns=[‘a’, ‘b’, ‘c’])rnst.line_chart(chart_data)”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e54a26e3d90>)])]The chart on the webapp (Source)Making this app runnable on Cloud Run is easy. Just add streamlit in requirements.txt, and make Dockerfile from a typical python webapp image. If you are not familiar with Docker, buildpacks can do the job. Instead of making Dockerfile, make Procfile with just 1 line as below.code_block[StructValue([(u’code’, u’web: streamlit run app.py –server.port $PORT –server.enableCORS=false’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e54a3261350>)])]To summarize, the minimum required files are only as below.code_block[StructValue([(u’code’, u’.rn|– app.pyrn|– Procfilern|– requirements.txt’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e54b4265b10>)])]Deployment is also easy. You can deploy this app to Cloud Run with just a command.code_block[StructValue([(u’code’, u’$ gcloud run deploy mydashboard –source .’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e54b4265310>)])]This command will build and make your image with buildpacks and Cloud Build, thus you don’t need to set up a build environment in your local system. Once deployment is completed, you can access your web app with the generated URL like https://xxx-[…].run.app. Copy and paste the URL into your web browser, and you will see your first dashboard webapp.Step 2: Add a callback function that receive changes in Firestore databaseIn the STEP 1, you can visualize your data with fixed conditions or interactively with UI functions on streamlit. Now we want it to update by itself.Firestore is a scalable NoSQL database, and it keeps your data in sync across client apps through real-time listeners. Firestore is available on Android and iOS, and also provides SDKs in major programming languages. Since we use streamlit in Python, let us use a Python client.In this post we don’t cover detailed usage of Firestore though, it is easy to implement a callback function that is called when a specific “Collection” has been changed. [reference]code_block[StructValue([(u’code’, u”from google.cloud import firestore_v1rnrndb = firestore_v1.Client()rncollection_ref = db.collection(u’users’)rnrndef on_snapshot(collection_snapshot, changes, read_time):rn for doc in collection_snapshot.documents:rn print(u'{} => {}’.format(doc.id, doc.to_dict()))rnrn# Watch this collectionrncollection_watch = collection_ref.on_snapshot(on_snapshot)”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e54b42658d0>)])]In this code, on_snapshot callback function is called when users Collection has been changed. You can also watch changes of Document.Since Firestore is a fully managed database, you would not need to provision the service ahead. You only need to choose “mode” and location. To use real-time sync functionality, select “Native mode”. Also select nearest or desired location.Using Firestore with streamlitNow let’s implement Firestore with streamlit. We add on_snapshot callback and update a chart with the latest data sent from Firestore. Here is one quick note when you use the callback function with streamlit. on_snapshot function is executed in a sub thread, instead UI manipulation in streamlit must be executed in a main thread. Therefore, we use Queue to sync the data between threads. The code will be something like below.code_block[StructValue([(u’code’, u’from queue import Queuernrnq = Queue()rndef on_snapshot(collection_snapshot, changes, read_time):rn for doc in collection_snapshot.documents:rn q.put(doc.to_dict()) # Put data into the Queuernrn# below will run in main threadrnsnap = st.empty() # placeholderrnrnwhile True:rn # q.get() is a blocking function. thus recommend to add timeoutrn doc = q.get() # Read from the Queuern snap.write(doc) # Change the UI’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e54a36850d0>)])]Deploy this app and write something in the collection you refer to. You will see the updated data on your webapp.Step 3: Plot a geospatial data with streamlitWe learned how to host web apps on Cloud Run, then how to update data with Firestore. Now we want to know how to plot geospatial data with streamlit. streamlit has multiple ways to plot geospatial data which includes latitude and longitude, we here used pydeck_plot(). This function is a wrapper of deck.gl, a geospatial visualization library.For example, provide data in latitude and longitude as to plot, add layers to visualize them.code_block[StructValue([(u’code’, u’import streamlit as strnimport pydeck as pdkrnimport pandas as pdrnimport numpy as nprnrndf = pd.DataFrame(rn np.random.randn(1000, 2) / [50, 50] + [37.76, -122.4],rn columns=[‘lat’, ‘lon’])rnst.pydeck_chart(pdk.Deck(rn map_provider=”carto”,rn map_style=’road’,rn initial_view_state=pdk.ViewState(rn latitude=37.76,rn longitude=-122.4,rn zoom=11,rn pitch=50,rn ),rn layers=[rn pdk.Layer(rn ‘HexagonLayer’,rn data=df,rn get_position='[lon, lat]’,rn radius=200,rn elevation_scale=4,rn elevation_range=[0, 1000],rn pickable=True,rn extruded=True,rn ),rn pdk.Layer(rn ‘ScatterplotLayer’,rn data=df,rn get_position='[lon, lat]’,rn get_color='[200, 30, 0, 160]’,rn get_radius=200,rn ),rn ],rn ))’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e54a3685750>)])]Plotting with pydeck_plot (Source)pydeck supports multiple map platforms. We here chose CARTO. If you would like to know more about great examples using CARTO and deck.gl, please refer to this blog.Step 4: Plot mobility dataWe are very close to the goal. Now we want to plot locations of vehicles. pydeck supports some ways to plot data, and TripsLayer would be a good fit to plot mobility data.Demo using Google Maps JavaScript API (Source)TripsLayer can visualize location data in time sequential. That means, when selecting a specific timestamp, it plots lines from location data in the time including last n periods. It also draws like an animation when you change the time in sequential order.In the final form, we also add IconLayer to identify the latest location. This layer is also useful when you want to plot a static location, and it just works like a “pin” on Google Maps.Now we need to think about how to use this plot with Firestore. Let’s make Document per vehicle, and only save the latest latitude, longitude, and timestamp of every vehicle. Why not save the history of locations? In that case, we should rather use BigQuery. We just want to see the latest locations that update in realtime.Firestore is useful and scalable, yet NoSQL. Note that there are some good fits and bad fits in NoSQL.Location data in Firestore ConsoleStep 5: RunFinally, we are here. Now let’s ride in a car and record data… if possible.For demo purposes, now we ingest dummy data into Firestore. It is easy to write data by using a client library.code_block[StructValue([(u’code’, u”db = firestore.Client()rncol_ref = db.collection(‘connected’)rncol_ref.document(str(vehicle_ind)).set({rn ‘lonlat': [-74, 40.72],rn ‘timestamp': 0rn})”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e54a350c950>)])]With writing dummy data, open the web page hosted on Cloud Run. you will see the map is updated upon new data coming.Firestore syncs data on streamlitNote that we used dummy data and manipulated the timestamps. Consequently, the location data updates much faster than actual time. This can be fixed once you use proper data and update cycle.Try it with your dataIn this post, we learned how to build a dashboard updated in real-time with Cloud Run and Firestore. Let us know when you find other use-cases with those nice Google Cloud products.Find out more automotive solutions here.Haven’t used Google Cloud yet? Try it from here.Check out the source code on GitHub.Related ArticleDiscover our new edge concepts at Hannover Messe that bring smart factories to lifeIntel and Google Cloud demonstrate edge-to-cloud technology at Hannover Messe.Read Article
Quelle: Google Cloud Platform

da Agency

Announcing new BigQuery capabilities to help secure sensitive data

In order to better serve their customers and users, digital applications and platforms continue to store and use sensitive data such as Personally Identifiable Information (PII), genetic and biometric information, and credit card information. Many organizations that provide data for analytics use cases face evolving regulatory and privacy mandates, ongoing risks from data breaches and data leakage, and a growing need to control data access. Data access control and masking of sensitive information is even more complex for large enterprises that are building massive data ecosystems. Copies of datasets often are created to manage access to different groups. Sometimes, copies of data are obfuscated while other copies aren’t. This creates an inconsistent approach to protecting data, which can be expensive to manage. To fully address these concerns, sensitive data needs to be protected with the right defense mechanism at the base table itself so that data can be kept secure throughout its entire lifecycle. Today, we’re excited to introduce two new capabilities in BigQuery that add a second layer of defense on top of access controls to help secure and manage sensitive data. 1. General availability of BigQuery column-level encryption functionsBigQuery column-level encryption SQL functions enable you to encrypt and decrypt data at the column level in BigQuery. These functions unlock use cases where data is natively encrypted in BigQuery and must be decrypted when accessed. It also supports use cases where data is externally encrypted, stored in BigQuery, and must then be decrypted when accessed. SQL functions support industry standard encryption algorithms AES-GCM (non-deterministic) and AES-SIV (deterministic).  Functions supporting AES-SIV allow for grouping, aggregation, and joins on encrypted data. In addition to these SQL functions, we also integrated BigQuery with Cloud Key Management Service (Cloud KMS). This gives you additional control, and allows you to manage your encryption keys in KMS and enables on-access secure key retrieval as well as detailed logging. An additional layer of envelope encryption enables generations of wrapped key sets to decrypt data. Only users with permission to access the Cloud KMS key and the wrapped keyset can unwrap the keyset and decrypt the ciphertext. “Enabling dynamic field level encryption is paramount for our data fabric platform to manage highly secure, regulated assets with rigorous security policies complying with several regulations including FedRAMP, PCI, GDPR, CCPA and more. BigQuery column-level encryption capability provides us with a secure path for decrypting externally encrypted data in BigQuery unblocking analytical use cases across more than 800+ analysts,” said Kumar Menon, CTO of Equifax.Users can also leverage available SQL functions to support both non-deterministic encryption and deterministic encryption to enable joins and grouping of encrypted data columns.The following query sample uses non-deterministic SQL functions to decrypt ciphertext.code_block[StructValue([(u’code’, u’SELECTrn AEAD.DECRYPT_STRING(KEYS.KEYSET_CHAIN(rn @kms_resource_name,rn @wrapped_keyset),rn ciphertext,rn additional_data)rnFROMrn ciphertext_tablernWHERErn …’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3edc9b976150>)])]The following query sample uses deterministic SQL functions to decrypt ciphertext.code_block[StructValue([(u’code’, u’SELECTrn DETERMINISTIC_DECRYPT_STRING(KEYS.KEYSET_CHAIN(rn @kms_resource_name,rn @wrapped_keyset),rn ciphertext,rn additional_data)rn FROMrn ciphertext_tablernWHERErn …’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3edc9b9764d0>)])]2. Preview of dynamic data masking in BigQueryExtending BigQuery’s column-level security, dynamic data masking allows you to obfuscate sensitive data and control user access while mitigating the risk of data leakage. This capability selectively masks column level data at query time based on the defined masking rules, user roles and privileges. Masking eliminates the need to duplicate data and allows you to define different masking rules on a single copy of data to desensitize data, simplify user access to sensitive data, and protect against compliance, privacy regulations, or confidentiality issues. Dynamic data masking allows for different transformations of underlying sensitive data to obfuscate data at query time. Masking rules can be defined on the policy tag in the taxonomy to grant varying levels of access based on the role and function of the user and the type of sensitive data. Masking adds to the existing access controls to allow customers a wide gamut of options around controlling access. An administrator can grant a user full access, no access or partial access with a particular masked value based on data sharing use case.For the preview of data masking, three different masking policies are being supported: ALWAYS_NULL. Nullifies the content regardless of column data types.SHA256. Applies SHA256 to STRING or BYTES data types. Note that the same restrictions apply to the SHA256 function.Default_VALUE. Returns the default value based on the data type.A user must first have all of the permissions necessary to run a query job against a BigQuery table to query it. In addition, for users to view the masked data of a column tagged with a policy tag they need to have a MaskedReader role.When to use dynamic data masking vs encryption functions?Common scenarios for using data masking or column level encryption are: protect against unauthorized data leakage access control management compliance against data privacy laws for PII, PHI, PCI datacreate safe test datasetsSpecifically, masking can be used for real-time transactions whereas encryption provides additional security for data at rest or in motion where real-time usability is not required.  Any masking policies or encryption applied on the base tables are carried over to authorized views and materialized views, and masking or encryption is compatible with other security features such as row-level security. These newly added BQ security features along with automatic DLP can help to scan your data across your entire organization, give you visibility into where sensitive data is stored, and enable you to manage access and usability of data for different use cases across your user base. We’re always working to enhance BigQuery’s (and Google Cloud’s) data governance capabilities, to enable end to end management of your sensitive data. With the new releases, we are adding deeper protections for your data in BigQuery. Related ArticleBuild a secure data warehouse with the new security blueprintIntroducing our new security blueprint that helps enterprises build a secure data warehouse.Read Article
Quelle: Google Cloud Platform

da Agency

Introducing Firehose: An open source tool from Gojek for seamless data ingestion to BigQuery and Cloud Storage

Indonesia’s largest hyperlocal company, Gojek has evolved from a motorcycle ride-hailing service into an on-demand mobile platform, providing a range of services that include transportation, logistics, food delivery, and payments. A total of 2 million driver-partners collectively cover an average distance of 16.5 million kilometers each day, making Gojek Indonesia’s de-facto transportation partner.To continue supporting this growth, Gojek runs hundreds of microservices that communicate across multiple data centers. Applications are based on an event-driven architecture and produce billions of events every day. To empower data-driven decision-making, Gojek uses these events across products and services for analytics, machine learning, and more.Data warehouse ingestion challenges To make sense of large amounts of data — and to better understand customers for the purpose of app development, customer support, growth, and marketing purposes — data must first be ingested into a data warehouse. Gojek uses BigQuery as its primary data warehouse. But ingesting events at Gojek’s scale, with rapid changes, poses the following challenges:With multiple products and microservices offered, Gojek releases new Kafka topics almost every day and they need to be ingested for analytical purposes. This can quickly result in significant operational overhead for the data engineering team that is deploying new jobs to load data into BigQuery and Cloud Storage. Frequent schema changes in Kafka topics require consumers of those topics to load the new schema to avoid data loss and capture more recent changes. Data volumes can vary and grow exponentially as people start building new products and logging new activities on top of a new topic. Each topic can also have a different load during peak business hours. Customers need to handle the rising volume of data to quickly scale per their business needs.Firehose and Google Cloud to the rescue To solve these challenges, Gojek uses Firehose, a cloud-native service to deliver real-time streaming data to destinations like service endpoints, managed databases, data lakes, and data warehouses like Cloud Storage and BigQuery. Firehose is part of the Open Data Ops Foundation (ODPF), and is fully open source. Gojek is one of the major contributors to ODPF.Here are Firehose’s key features:Sinks – Firehose supports sinking stream data to the log console, HTTP, GRPC, PostgresDB (JDBC), InfluxDB, Elastic Search, Redis, Prometheus, MongoDB, GCS, and BigQuery.Extensibility – Firehose allows users to add a custom sink with a clearly defined interface, or choose from existing sinks.Scale – Firehose scales in an instant, both vertically and horizontally, for a high-performance streaming sink with zero data drops.Runtime – Firehose can run inside containers or VMs in a fully-managed runtime environment like Kubernetes.Metrics – Firehose always lets you know what’s going on with your deployment, with built-in monitoring of throughput, response times, errors, and more.Key advantagesUsing Firehose for ingesting data in BigQuery and Cloud Storage has multiple advantages. Reliability Firehose is battle-tested for large-scale data ingestion. At Gojek, Firehose streams 600 Kafka topics in BigQuery and 700 Kafka topics in Cloud Storage. On average, 6 billion events are ingested daily in BigQuery, resulting in more than 10 terabytes of daily data ingestion.  Streaming ingestionA single Kafka topic can produce up to billions of records in a day. Depending on the nature of the business, scalability and data freshness are key to ensuring the usability of that data, regardless of the load. Firehose uses BigQuery streaming ingestion to load data in near-real-time. This allows analysts to query data within five minutes of it being produced.Schema evolutionWith multiple products and microservices offered, new Kafka topics are released almost every day, and the schema of Kafka topics constantly evolves as new data is produced. A common challenge is ensuring that as these topics evolve, their schema changes are adjusted in BigQuery tables and Cloud Storage. Firehose tracks schema changes by integrating with Stencil, a cloud-native schema registry, and automatically updates the schema of BigQuery tables without human intervention. This reduces data errors and saves developers hundreds of hours. Elastic infrastructureFirehose can be deployed on Kubernetes and runs as a stateless service. This allows Firehose to scale horizontally as data volumes vary.Organizing data in cloud storage Firehose GCS Sink provides capabilities to store data based on specific timestamp information, allowing users to customize how their data is partitioned in Cloud Storage.Supporting a wide range of open source softwareBuilt for flexibility and reliability, Google Cloud products like BigQuery and Cloud Storage are made to support a multi-cloud architecture. Open source software like Firehose is just one of many examples that can help developers and engineers optimize productivity. Taken together, these tools can deliver a seamless data ingestion process, with less maintenance and better automation.How you can contributeDevelopment of Firehose happens in the open on GitHub, and we are grateful to the community for contributing bug fixes and improvements. We would love to hear your feedback via GitHub discussions or Slack.Related ArticleTransform satellite imagery from Earth Engine into tabular data in BigQueryWith Geobeam on Dataflow, you can transform Geospatial data from raster format in Earth Engine to vector format in BigQuery.Read Article
Quelle: Google Cloud Platform

da Agency

Pride Month: Q&A with bunny.money founders about saving for good

June is Pride Month—a time for us to come together to bring visibility and belonging, and celebrate the diverse set of experiences, perspectives, and identities of the LGBTQ+ community. This month, Lindsey Scrase, Managing Director, Global SMB and Startups at Google Cloud, is showcasing conversations with startups led by LGBTQ+ founders and how they use Google Cloud to grow their businesses. This feature highlights bunny.money and its founders, Fabien Lamaison, CEO, Thomas Ramé, Technology Lead, and Cyril Goust, Engineering Lead. Lindsey: Thanks Fabien, Thomas, and Cyril. It’s great to connect with you and talk about bunny.money. I love how you’re bringing a creative twist to fintech and giving back to communities. What inspired you to found the company?Fabien: One of my favorite childhood toys was an old-fashioned piggy bank. I remember staring at it and trying to figure out how much of my allowance should be saved, spent, or given to charity. As you can imagine, there were lots of ideas racing through my mind but saving and giving back were always important to me. Years later, I realized I could combine my passions for banking, technology, and helping others by creating a fintech service that makes it easy for people to save while donating to their favorite causes.Fabien Lamaison, CEO of bunny.moneyLindsey: My brothers and I did something similar where we allocated a portion of any money we made as kids to giving. And I too had a piggy bank – a beautiful one that could only be opened by breaking it. Needless to say it was a good saving mechanism! It’s inspiring to see you carrying your personal value forward into bunny.money to help others do the same. Tell us more about bunny.money?Fabien: bunny.money plays with the concept of reimagining saving—and offers a way to positively disrupt conventional banking. For us bunnybankers, financial and social responsibility go hand in hand. We empower people to build more sustainable, inclusive financial futures. Looking ahead, we not only want to help people set up recurring schedules for saving and donating, but also offer more options for socially responsible investing and help companies better match employee donations to charitable causes and build out retirement plans.Lindsey: It sounds like you’re not only disrupting traditional banking services but also how people manage their finances. How does bunny.money serve its customers?Fabien: bunny.money is a fintech company founded on the principles of providing easy, free, and ethical banking services. Our comprehensive banking platform enables customers to quickly open savings wallet and schedule recurring deposits.Thomas: bunny.money is also a fintech bridge that connects people and businesses to the communities and causes they care about. With bunny.money, customers can make one-time or recurring donations to the nonprofits of their choice. bunny.money doesn’t charge recipients fees to process donations. We give customers the option of offering us a tip, but it’s not required.Lindsey: So with bunny.money, what are some of the nonprofits people can donate to?Fabien: Over 30 organizations have already joined bunny.money’s nonprofit marketplace, includingStartOut,TurnOut,Trans Lifeline, and Techqueria. Some are seeing donations increase by up to 20 percent as they leverage bunny.money to gamify fundraising, promote social sharing, and encourage micro-donations from their members and supporters.Cyril: bunny.money also helps people discover local causes and nonprofits such as food banks requesting volunteers, parks that need to be cleaned, and mentoring opportunities. I’m particularly excited to see bunny.money help people build a fairer, greener society by donating to environmental nonprofits, including, Carbon Lighthouse,Sustainable Conservation, Public Land Water Association, back2earth andFARMS. We also decided to “lead by the example” and pledge to give 1% of our revenues to 1% for the Planet.Lindsey: Given your business and the services you offer, I imagine you’ve encountered immense complexity along the way. What were some of the biggest challenges that you had to overcome?Fabien: One of our biggest challenges was helping people understand saving for good, and purpose-led banking, which is a relatively new idea in fintech. Although there are plenty of mobile banking apps, most don’t offer an easy way for people to improve their personal finances and donate to their favorite causes in one convenient place.Cyril: On the technical side, we needed to comply with strict industry regulations, including all applicable requirements under the Bank Secrecy Act and the USA PATRIOT Act. These regulations protect sensitive financial data and help fight against fraudulent activities such as money laundering.Lindsey: Can you talk about how Google Cloud is helping you address these challenges?  Thomas: Protecting client data is a top priority for us, so we built bunny.money on thehighly secure-by-design infrastructure of Google Cloud. Google Cloud automatically encrypts data in transit and at rest, and the solutions comply with all major international security standards and regulations right out of the box. Although we serve customers in the U.S. today, Google Cloud distributed data centers will allow us to meet regional security requirements and eventually reach customers worldwide with quality financial services.Thomas Ramé, Technology Lead at bunny.moneyFabien: We wanted to build a reliable, feature-rich fintech platform and design a responsive mobile app with an intuitive user interface (UI). We knew from experience that Google Cloud is easy to use and offers integrated tools, APIs, and solutions. We also wanted to tap into the deep technical knowledge of theGoogle for Startups team to help us scale bunny.money and affordably trial different solutions with Google for Startups Cloud Program credits.Cyril: As aCertified Benefit Corporation™ (B Corp™), it is also important for us to work with companies that align with the values we champion such as diversity and environmental sustainability. Google Cloud iscarbon neutral and enables us to accuratelymeasure, report, and reduce our cloud carbon emissions. Lindsey: This is exactly how we strive to support startups at all stages – with the right technology, offerings, and support to help you scale quickly and securely, all while being the cleanest cloud in the industry. Can you go into more detail about the Google Cloud solutions you use—and how they all come together to support your business and customers? Fabien: Our save for good® mobile app enables customers to securely create accounts, verify identities, and connect to external banks in just under four minutes. Thomas: With Google Cloud, bunny.money consistently delivers a reliable, secure, and seamless banking experience. Since recently launching our fintech app, we’ve already seen an incredible amount of interest in our services that enable people to grow financially while contributing to causes they are passionate about. Right now, we’re seeing customers typically allocate about 10 percent of each deposit to their favorite charities.Cyril: The extensive Google Cloud technology stack helps us make it happen. We can useBigQuery to unlock data insights,Cloud SQL to seamlessly manage relational database services, andGoogle Kubernetes Engine (GKE) to automatically deploy and scale Kubernetes. These solutions enable us to cost-effectively scale bunny.money and build out a profitable fintech platform.Cyril Goust, Engineering Lead at bunny.moneyThomas: In addition to the solutions Cyril mentioned, we useCloud Scheduler to manage cron job services,Dataflow to unify stream and batch data processing, andContainer Registry to securely store Docker container images. We’re always innovating, and Google Cloud helps our small team accelerate the development and deployment of new services.Lindsey: It’s exciting to hear your story and the many different ways that Google Cloud technology has been able to support you along the way. You’re creating something that affects change on many levels—from how people save and give to how businesses and nonprofits can engage.Since it is also Pride month, I want to change focus for a minute and talk about how being part of the LGBTQ+ community impacted your approach to starting bunny.money?Fabien: I believe we all belong to several communities (family, friends “tribes,” sports, group of interests) that are different layers of our own identity and way of life. I’m part of the LGBTQ+ community, and I’m also an immigrant for example. I’m now a French-American, as is my husband, and we live in San Francisco. But even as a couple, we still had to live apart for several years—he in Paris and I in San Francisco—as we worked through issues with his U.S. work visa (same sex weddings were not possible at that time at the federal level, we couldn’t be under the same visa application).Fortunately, the LGBTQ+ community can be like an extended family, both professionally and personally. Personally, I’ve had the support of friends as my husband and I dealt with immigration and work challenges. And professionally, I’ve experienced incredible support in the startup world with nonprofits such asStartOut, which provides key resources to help LGBTQ+ entrepreneurs grow their businesses.Lindsey: I can only imagine the emotional toll that being apart created for you and your husband and I’m so glad that it eventually worked out. My wife is Austrian and while we are fortunate to be here together, this intersectionality has created an additional layer of complexity for us over the years as we have started a family. Do you have any advice for others in the LGBTQ+ community looking to start and grow their own companies? You mentioned StartOut, and I know there are additional organizations LGBTQ+ entrepreneurs can turn to for help, includingLesbians who Tech,Out in Tech,High Tech Gays (HTG) – Queer Silicon Valley, andQueerTech NYC (Meetup).Fabien: I would suggest really exploring what you’re passionate about. I’ve enjoyed focusing on saving and finances since I was young and have always been passionate about giving back. Being part of the LGBTQ+ community—or really any community that’s viewed as an “outsider”—gives you the opportunity to think differently. When you bring your passion and life experiences together, you can start to imagine new ways of doing things. By engaging in your communities, it can be easier to find others who share your experiences, interests, and even values. You bring the best from each world.Since LGBTQ+ founders and entrepreneurs might belong to several groups, it’s good to explore all available avenues and resources, including the organizations you mentioned earlier. We can always learn and accomplish more when we work together. I’ve experienced that both in the LGTBQ+, immigrant and Fintech communities.Lindsey: The importance of community underlies so many aspects of your identity as a founder, as someone who has moved to the US from France, and as a member of the LGBTQ+ community. I’m so glad that you’ve sought out – and received – support along the way. I agree it’s so important  for others to seek out this community and support.  And to close, would you be able to share any next steps for bunny.money?Fabien: We’re looking forward to helping customers build more sustainable and inclusive financial futures on our platform. We’ll continue contributing to positive change in the world by rolling out new AI-powered services to enable ethical investing and personalized giving and impact programs. As we build this first banking app for personal and workplace giving, our goal is to benefit all communities by bridging the gap between businesses and people—which is why we’re excited to continue working with partners like Google for Startups andGV (GV offers us valuable mentor sessions during our accelerator program at StartOut).If you want to learn more about how Google Cloud can help your startup, visit our pagehere to get more information about our program, and sign up for our communications to get a look at our community activities, digital events, special offers, and more.Related ArticlePride Month: Q&A with Beepboop founders about more creative, effective approaches to learning a new languageRead how Beepboop democratizes language instruction by helping students learn to speak Spanish and English in dynamic, fun environments l…Read Article
Quelle: Google Cloud Platform

da Agency

Commerzbank has Reimagined the Customer Experience with Google Contact Center AI

Digital channels and on-demand banking have led customers to expect instant and helpful access to managing their finances, with minimal friction. Google Cloud built Contact Center AI (CCAI) and DialogFlow CX to help banks and other enterprises deliver these services, replacing phone trees or sometimes confusing digital menus with intelligent chatbots that let customers interact conversationally, just as they would with human agents. Leaders at Germany-based Commerzbank, which operates in over 50 countries, saw potential for these technologies to enhance customer experiences, providing more curated and helpful interactions that would build trust in and satisfaction with their brand. Commerzbank’s implementation speaks to how conversational artificial intelligence (AI) services can help businesses better serve customers, and in this article, we’ll explore their story and what their example means for your business. Commerzbank: Disrupting Customer Interactions with Google’s Contact Center AI and Dialog Flow CXTokyo, 7:00 AM. Vanessa is on a business trip in Japan, closing a new deal for her company, one of Commerzbank´s more than 30,000 corporate customers throughout Germany. She has been preparing for weeks, and is going through her points a final time in a downtown coffee shop. Glancing at her watch, she realizes she must leave immediately to get to the meeting.Intending to pay, she realizes the chip in her credit card is not functioning. Due to the time difference with Germany, Vanessa is now concerned she will not be able to contact someone from customer support. She opens the Commerzbank mobile app and contacts the customer center through chat. The access point she needs is available, but how can it help her most efficiently? Building excellent conversational experiencesCustomers like Vanessa need an answer right away. With that in mind, Commerzbank aims to provide customers with integrated support via the use of chatbots in the quest to deliver efficiency, high quality, and information consistency. This goal is where the Google Cloud virtual agent platform Dialogflow CX comes into play, providing us with an enormous number of features to build conversation dialogue through accurate intent recognition, a robust visual flow creator, and automated testing—all while significantly improving our time to market. In just nine weeks, the Commerzbank team set-up an agile proof-of-value project by developing a chatbot solution designed to deliver a reliable conversation experience. Commerz Direktservices Chatbot Agent is now able to identify the touchpoint the customer is using (App or Web) and detect more than 100 suitable FAQs and answer them properly. The Chatbot Agent also identifies leads and sales prospects, enabling it to provide support on open questions in relation to products and services, thus performing a graceful handover to the human agent with the enrichment of value parameters. Commerz Direktservices has also broadened the ability  of the Chatbot to handle different customer types (keyword-based vs. context-based customers) by constructing an intelligent dialog architecture that lets the Chatbot Agent flow elegantly through prompts and intent questioning.Commerzbank has integrated Google Dialogflow CX with Genesys Platform, helping to make use of the full capabilities of the existing contact center infrastructure and more efficiently orchestrate the incoming interactions. A very versatile architecture bridges the potential of Google Cloud with a variety of on-premise applications and components, while also providing system resiliency and supporting data security compliance. The support of the entire Google team has been invaluable to accelerate the bank’s journey to the cloud. Commerzbank is seeing a number of benefits as it expands its AI platform, including:Enhanced ability to deliver innovationImproved operational efficienciesBetter customer experience through reduced wait times and self-serve capabilities, leading to reduced churnGreater productivity for CommerzBank employees who are able to support customer queries with enriched Google CCAI data The creation of an integrated cross-channel strategyGoing beyond support into an active conversational experienceNow, Commerzbank wants to move beyond great customer support to continue to increase the value-add to the customer. Customers like Vanessa are looking for their bank to go the extra mile by optimizing their finances,  providing personalized financial products and solutions, and  offering more control over their investment portfolio, among other needs. With this in mind, Commerzbank aims to continue moving away from a scenario where chatbots are only passive entities waiting to be triggered, into a new and more innovative one whereby they become an active key enabler of enhanced customer interactions across the customer value chain. Commerzbank is already mapping active dialog paths to: Make tailored product suggestions to prospects, giving them the possibility to acquire a product that suits their particular needsIdentify customer requirements for financing or investment, inviting them to get advice and benefit from the existing opportunitiesGenerate prospects based on the business potential, thus providing the human agents with a framework to prioritize their interactions Commerzbank leaders anticipate the impact of this solution will be significant. It will let the company fulfill the first advisory touchpoint for financial needs and perform a fast conversation hand-over to specialists as soon as the customer requires it. As a result, leaders expect to exponentially increase conversion rates via more fruitful customer journeys.Helping Vanessa with a delightful customer experienceGoing back to Vanessa’s example: how can Commerzbank help Vanessa efficiently? When she contacts support through chat, the chatbot welcomes her and offers help with any question she may have. Vanessa explains the situation and the digital agent explains that delivering a replacement card would take many days, and that the most practical solution would be to activate a virtual debit card, e. g., with Google Pay on her phone. Vanessa gladly accepts this solution, prompting the Chatbot to deliver a short explanation on how to carry out the process, as well as two additional links: one for downloading the Google Pay App from the Google Play Store and another for digital self-service in the Commerzbank App, which she can intuitively use to synchronize the Commerzbank App and Google Pay. After just 5 minutes, Vanessa is able to pay comfortably using her phone and get to her meeting in time. This engagement is how Commerzbank wants to deliver digital customer experiences that fascinate their customers, allowing their customers to perform their daily banking activities faster, better, and easier. To learn more about how Google Cloud AI solutions can help your company, visit the product page or check out this report that explores the total economic impact of Google Cloud CCAI.Related ArticleHSBC deploys Dialogflow, easing call burden on policy expertsHSBC uses AI and machine learning to reduce the time employees spend on manually intensive queries and improve the consistency of policy …Read Article
Quelle: Google Cloud Platform

da Agency

Introducing new commitments on the processing of service data for our cloud customers

At Google, we engage regularly with customers, regulators, policymakers, and other stakeholders to provide transparency into our operations, policies, and practices and to further strengthen our commitment to privacy compliance. One such engagement is our ongoing work with the Dutch government regarding its Data Protection Impact Assessment (DPIA) of Google Workspace and Workspace for Education.As a result of that engagement, today Google is announcing our intention to offer new  contractual privacy commitments for service data1 that align with the commitments we offer for customer data.2 Once those new commitments become generally available, we will process service data as a processor under customers’ instructions, with the exception of limited processing3 that we will continue to undertake as a controller. We will provide further details as we implement these updates – planned for Google Workspace, Google Workspace for Education and Google Cloud4 services – beginning in 2023 and in successive phases through 2024.In parallel, Google is working to develop a version of Chrome OS (including Chrome browser running on managed Chrome OS devices) for which Google will offer similar processor commitments. In line with our goal of giving customers greater transparency and control over their data, we’re aiming to provide this updated version of Chrome OS, once it’s complete, to our enterprise and education customers around the world. We recognise that privacy compliance plays a crucial role in earning and maintaining your trust, and we will continue to work diligently to help make compliance easier for your business as you use our cloud services. To learn more about our approach to privacy compliance, please visit our Privacy Resource Center.1. Service Data is defined in the Google Cloud Privacy Notice as the personal information Google collects or generates during the provision and administration of the Cloud Services, excluding any Customer Data and Partner Data.2. Customer Data means data submitted, stored, sent or received via the services by customer or end users, as further described in the applicable data processing terms.3. For example, billing and account management, capacity planning and forecast modeling, detecting, preventing and responding to security risks and technical issues.4. Formerly known as Google Cloud Platform.Related ArticleAn update on Google Cloud’s commitments to E.U. businesses in light of the new E.U.-U.S. data transfer frameworkGoogle Cloud welcomes the new data transfer framework deal agreed by the E.U./U.S. and explains how we support customers to further prote…Read Article
Quelle: Google Cloud Platform

da Agency

Google Workspace, GKE help startup CAST AI grow faster and optimize cloud costs

In many ways, serial entrepreneur Gil Laurent and his technology startups have grown alongside Google Workspace and Google Cloud. When he was CEO and co-founder of Ukraine-based Viewdle — a machine learning and computer vision startup that was acquired by Google in 2012 — the organization relied on Google Workspace for many of its collaboration needs, trading the complexity of email attachments and file versions for the cloud-synced availability of documents in Google Drive. A similar story played out a few years later when he co-founded Zenedge — a cybersecurity company focused on the edge of the network — which was acquired by Oracle in 2018. Zenedge still used a handful of other services to round out meetings and collaboration, but Google Workspace was the foundation. In 2019, when co-founding his latest venture — cloud cost management startup CAST AI — Laurent saw that he didn’t have to pay for additional services, as Google Workspace’s product suite included everything needed to connect his teams and workstreams. From onboarding new employees and getting them connected to their corporate email, to real-time collaboration and video conferencing, Google Workspace offered everything. “As a young startup, there was only one place to start—Google Workspace,”  recalled Laurent, who now serves as the company’s chief product officer. “We did not even consider anything else.”Google Workspace is only one part of CAST AI’s Google product adoption, however. “Our whole business runs on GKE on Google Cloud,” Laurent said. The company was up and running on GKE (Google Kubernetes Engine) almost immediately after rolling out Google Workspace, and Laurent recalls a smooth transition. “It was very natural for everyone.” CAST AI is an end-to-end Kubernetes Automation and Management platform that helps businesses optimize their cloud costs by 63% on average. With an approach built on container orchestration, a product like GKE was necessary to efficiently run the company’s workloads and services.Laurent explained that at Zenedge, the company struggled to understand how to control its cloud costs as it experienced growth: “We started out spending thousands per month with 10 engineers, which seemed right. But three years later, after continued growth, we were spending millions. We didn’t understand why. The bill could be 100 pages long.” When founding CAST AI, Laurent addressed this frustration head on, using containers to ensure their customers’ cloud resources weren’t going unused at such high rates. “Containers can be moved around, so you can optimize deployment to make them busy most of the time while eliminating waste,” Laurent said. “We knew we had to include automation. You can tell someone that they’re using 1,000 VMs and that 50 could be used better or more efficiently if moved to a different instance type — but in DevOps, who does this? The opportunities for optimization change daily and people are afraid of breaking things. We knew we had to find a way to offer not just observability but automated management.”Choosing GKE was “easy because Google invented Kubernetes, and GKE is the state of the art, with its implementation of the full Kubernetes API, autoscaling, multi-cluster support, and other features that set the trend.” Laurent added that the company also took advantage of the Google for Startups Cloud Program to scale up its business by tapping into extended benefits like tailored mentorship and coverage for their Google Cloud usage for two years. Many startups adopt Google Workspace to connect and engage in real-time with their teams, but quickly learn that leveraging other Google offerings — such as cloud solutions and the Google for Startups Cloud Program — can be very helpful to further their startup’s growth. For CAST AI, the combination of GKE on Google Cloud and Google Workspace proved especially valuable because the company was founded in late 2019, just months before the global pandemic began. The CAST AI team needed sophisticated cloud services to build their product, in addition to collaboration and productivity tools that could accommodate remote workers in different countries. “The idea that you can work in any place at any time without tradeoffs, whether you’re in Madrid or Miami — that helps a lot,” Laurent said. “Without GKE and Google Workspace, I am not sure we could have achieved all that we have so far.”To learn more about how Google Workspace and Google Cloud help startups like CAST AIaccelerate their journey — from connecting and collaborating to building and innovating — visit our startups solutions pages for Google Workspace and Google Cloud.Related ArticleWhy managed container services help startups and tech companies build smarterWhy managed container services such as GKE are crucial for startups and tech companies.Read Article
Quelle: Google Cloud Platform

da Agency

Mercari leverages Google's vector search technology to create a new marketplace

Mercari is one of the most successful marketplace services in recent years, with 5.3 million active users in the US and 20 million active users in Japan. In Oct 2021, the company launched a new service Mercari Shops in Japan that allows small business owners and individuals to open their e-commerce portal in 3 minutes. At the core of the new service, Mercari introduced Google’s vector search technology to realize the crucial part: creating a new marketplace for small shops using “similarity”.Mercari has 5.3M active users in the USThe Challenge: collection of shops doesn’t make a marketplaceAt the time of the launch, Mercari Shops was just a collection of small e-commerce sites where shoppers could only see the items sold by each shop one by one. For the shoppers, it was a somewhat painful experience to go back to the top page and choose a shop each time. This loses the most important value of the service; an enjoyable shopping experience for the shoppers.The challenge of Mercari Shops: shoppers were only able to browse the items from the selected shopShoppers would love something like “a real marketplace on smartphones” where they can easily browse hundreds of items from a wide variety of shops with a single finger gesture. But how do you manage the relationships across all the items to realize the experience? You would need to carefully define millions of item categories and SKUs shared across the thousands of sellers, and keep maintaining it all by manual operation of support staff. It also requires the sellers to search and choose the exact category for each item to sell. This is the way traditional marketplace services are built, involving much operational cost, and also losing another key value of Mercari Shops that anyone can build an e-commerce site within 3 minutes.How about using a recommendation system? The popular recommendation algorithm such as collaborative filtering usually requires large purchase or click histories to recommend other items, and doesn’t work well for recommending new items or long-tail items that don’t have any relationship with existing items. Also, collaborative filtering only memorizes the relationships between the items, such as “many customers purchase/view these other items also”. Meaning, it doesn’t actually make any recommendations with insights by looking at the item descriptions, names, images or many other side features.So Mercari decided to introduce a new way: using “similarity” to create a marketplace. A new marketplace created by similarityWhat does it mean by similarity? For example, you can define a vector (a list of numbers) with three elements (0.1, 0.02, 0.03) to represent an item that has 10% affinity to the concept of “fresh”, 2% to “vegetable”, and 30% to “tomato”. This vector represents the meaning or semantics of “a fresh tomato” as an item. If you search near vectors around it, those items would also have similar meaning or semantics – you will find other fresh tomatoes (note: this is a simplified explanation of the concept and the actual vectors have much complex vector space). Vector search finds items with similar meaningThis similarity between vectors exemplifies the marketplace in Mericari Shops that allows the shopper to browse all the similar items collected on a page. You don’t need to define and update item categories and SKUs manually to connect between the millions of items from thousands of sellers. Instead, machine learning (ML) algorithms extract the vectors from each item automatically, every time a seller adds a new item or updates an item. This is exactly the same way Google uses for finding relevant contents on Search, YouTube, Play and other services; called Vector Search.Enabled by the technology, now the shoppers of Mercari Shops can easily browse relevant items sold by different shops on the same page.The marketplace created with the similarity: shoppers can easily browse the relevant itemsVector search made easy with Matching EngineLet’s take a look at how Mercari built the marketplace using the vector search technology. With analytics results and experiments, they found that the item description written by the sellers represents the value of each item well, compared to other features such as the item images. So they decided to use item description texts to extract the feature vector of each item. Thus, the marketplace of Mercari Shops is organized by “how items are similar to each other in the text description”.Extracting feature vectors from the item description textsFor extracting the text feature vector, they used a word2vec model combined with TF-IDF. Mercari also tried other models such as BERT, but they decided to use word2vec as it’s simple and lightweighted, suitable for production use with less GPU cost for prediction.There was another challenge. Building a production vector search infrastructure is not an easy task. In the past, Mercari built their own vector search from scratch for an image search service. It took for them to assign a dedicated DevOps engineer, let them build Kubernetes servers, design and maintain the service. Also, they had to build and operate a data pipeline for continuous index update. To keep the search results fresh, you need to update the vector search index every hour with newly added items using the data pipeline. This pipeline had some incidents in the past and consumed DevOps engineers’ resources. Considering these factors, it was almost impossible for Mercari Shops to add a new vector search under a limited resource. Instead of building it from scratch, they introduced Vertex AI Matching Engine. It’s a fully managed service that shares the same vector search backend with the major Google services such as Google Search, YouTube and Play. So there is no need to implement the infrastructure from scratch, maintain it, and design and run the index update pipeline by yourself. Yet, you can quickly take advantage of the responsiveness, accuracy, scalability and availability of Google’s latest vector search technology.The feature extraction pipelineMercari Shops’ search service has two components: 1) feature extraction pipeline and 2) vector search service. Let’s see how each component works.The feature extraction pipelineThe feature extraction pipeline is defined with Vertex AI Pipelines, and is invoked by Cloud Scheduler and Cloud Functions periodically to initiate the following process:Get item data: The pipeline makes a query BigQuery to fetch the updated item dataExtract feature vector: The pipeline runs predictions on the data with the word2vec model to extract feature vectorsUpdate index: The pipeline calls Matching Engine APIs for adding the feature vectors to the vector index. The vectors are also saved to Cloud BigtableThe following is the actual definition of the feature extraction pipeline on Vertex AI Pipelines:The feature extraction pipeline definition on Vertex AI PipelinesVector search serviceThe second component is the vector search service that works in the following manner:The vector search serviceClient makes a query: a client makes a query to the Cloud Run frontend specifying an item idGet the feature vector: get a feature vector of the item from BigtableFind similar items: using Matching Engine API, find similar items with the feature vectorReturns the similar items: returns item ids of the similar itemsBy introducing Matching Engine, Mercari Shops was able to build the production vector search service within a couple of months. As of one month after launching the service, they haven’t seen any incidents. From development to production, only a single ML engineer (the author) implements and operates the whole service.Looking aheadWith the successful introduction, Mercari Shops is now working on adding more functionalities and extending the service to future shop projects. For example, Matching Engine has a filter vector match function that applies simple filters to the search results.  With this function, they may only show “on sale” items, or exclude items from specific shops. Also, Matching Engine will support a streaming index update soon that would allow the users to find items as soon as they are added by the sellers. Vertex AI Feature Store looks attractive too as a replacement for the Cloud Bigtable as the repository of feature vectors with its additional functionality including feature monitoring for better observability on the service quality. With those Google Cloud technologies and products, Mercari can turn their new ideas into reality with less time and resources, adding significant value to their business. 
Quelle: Google Cloud Platform

da Agency

Announcing general availability of Confidential GKE Nodes

Today, we’re excited to announce the general availability of Confidential GKE Nodes. Many organizations have made Google Kubernetes Engine (GKE) the foundation of their modern application architectures. While the benefits of containers and Kubernetes can outweigh that of traditional architectures, moving to and running those apps in the cloud often entails careful planning to minimize risk and potential data exposure. To help increase security of your GKE clusters, Confidential GKE Nodes can be used.Part of the growing Confidential Computing product portfolio, Confidential GKE Nodes leverage hardware to make sure your data is encrypted in memory. The GKE workloads you run today can run confidentially without any code changes on your end. Bringing confidential computing to your container workloadsWith Confidential GKE Nodes, you can achieve encryption in-use for data processed inside your GKE cluster, without significant performance degradation. Confidential GKE Nodes are built on the same technology foundation as Confidential VM and utilize AMD Secure Encrypted Virtualization (SEV). This feature allows you to keep data encrypted in memory with node-specific, dedicated keys that are generated and managed by the processor. The keys are generated in hardware during node creation and reside solely within the processor, making them unavailable to Google or other nodes running on the host. Confidential GKE Nodes also leverage Shielded GKE nodes to offer additional protection against rootkit and bootkits, helping to ensure the integrity of the operating system you run on your Confidential GKE Nodes.Mixed node pools and stateful workloads Two new features have been added for the general availability release of Confidential GKE Nodes: mixed node pool support and PersistentVolumes. Mixing confidential node pools with non-confidential node poolsConfidential GKE Nodes can be enabled as a cluster-level security setting or a node pool-level security setting. When enabled at the cluster level, Confidential GKE Nodes enforce the use of Confidential VMs on all worker nodes. Worker nodes in a cluster can only use confidential nodes, and confidential computing can not be disabled on individual node pools. All worker nodes, including the workloads running on them, are encrypted in-use. When enabled at the node level, Confidential GKE Nodes enforce the use of Confidential VMs on specific node pools, so only worker nodes in specified node pools are running confidentially. This new capability can allow a single GKE cluster to run both confidential and non-confidential workloads. Creating regular node pools and confidential node pools in a single cluster can help minimize cluster management. To learn more, see our guide to enabling Confidential GKE Nodes on node pools.Supporting PersistentVolumes for stateful container workloadsConfidential GKE Nodes are great for protecting data in stateless and stateful workloads. Confidential GKE Nodes recently added support for PersistentVolume resources. In GKE, a PersistentVolume is a cluster resource that Pods can use for durable storage and is typically backed by a persistent disk. The pairing of PersistentVolumes with Confidential GKE Nodes is ideal for containerized applications that require block storage.PricingThere is no additional cost to deploy Confidential GKE Nodes, other than the cost of Compute Engine Confidential VM.Get started with this game-changing technologyCreating a GKE cluster that uses Confidential GKE Nodes on all nodes is easy. Simply go to the Cloud Console, click Kubernetes Engine and then click Clusters. Select “Create” and then “Configure” on GKE Standard. Under Cluster, there is a security section where you click the checkbox that says “Enable Confidential GKE Nodes.”GKE clusters can be enabled to run as Confidential under the Security Setting for Kubernetes Engine.Confidential computing transforms the way organizations process data in the cloud while preserving confidentiality and privacy. To learn more, read about our Confidential VMs and get started using your own confidential GKE Nodes today.Related ArticleA deeper dive into Confidential GKE Nodes—now available in previewConfidential GKE Nodes, now in preview, encrypt the memory of your nodes and the workloads that run on top of them.Read Article
Quelle: Google Cloud Platform