da Agency

2021 resolutions: Kick off the new year with free Google Cloud training

Tackle your New Year’s resolutions with our new skills challenge, which will provide you with no cost training to build cloud knowledge and an opportunity to earn Google Cloud skill badges to showcase your cloud competencies. There are 4 initial tracks in the skills challenge: Getting Started, Data Analytics, Hybrid and Multi-cloud, Machine Learning (ML) and Artificial Intelligence (AI). To begin, sign up for the skills challenge you’re interested in most to receive 30 days free access to Google Cloud labs. Each track will give you a chance to earn different skill badges such as the Foundational Infrastructure skill badge or Foundational Data, ML, and AI skill badge, which you can share with your network. To earn a skill badge, complete a series of hands-on labs on Google Cloud labs to learn new cloud skills and take a final assessment challenge lab to test your skills. Read on to find out which track in the skills challenge is best for you. Getting Started trackNew to Google Cloud? Select the Getting Started track and use your 30 days access to Google Cloud labs to demonstrate your core infrastructure skills. You’ll learn how to write cloud shell commands, deploy your first virtual machine, and run applications on Kubernetes. It’s a great place to start for cloud engineers, cloud architects, IT practitioners, or anyone with some cloud computing foundational knowledge.Data Analytics track This track is for data analysts ready to expand their skills into AI and machine learning. You will have a chance to demonstrate your understanding of BigQuery. You’ll learn how to do everything from writing and troubleshooting SQL queries and using Apps Script, to building classification and forecasting models.Hybrid and Multi-cloud trackThis track is for hybrid and multi-cloud architects ready to showcase their skills in managing containers with Google Kubernetes Engine and Anthos. You will also test your skills in security when deploying and managing production environments with Google Kubernetes Engine.ML and AI track This track is for data scientists and machine learning engineers ready to prove their skills with Google Cloud tools like BigQuery, Cloud Speech API, AI Platform, and Cloud Vision API.Registerfor our January 22 webinar for an introduction to Google Cloud, including a walk-through of the Google Cloud Console and a tour of the labs included in the Getting Started track of the skills challenge. Ready to jump into the skills challenge? Sign uphere.Related ArticlePrepare for Google Cloud certification with one free month of new Professional Certificates on CourseraTrain for Google Cloud certifications with one free month of Professional Certificates on CourseraRead Article
Quelle: Google Cloud Platform

da Agency

Using machine learning to improve road maintenance

There’s a new way to look out for potholes in the road and it doesn’t involve better eyeglasses or dispatching costly repair crews. Bus-mounted cameras and machine learning can do it for you, as the City of Memphis discovered.    Staying on top of deteriorating roads when you can’t add more personnel is a never ending cycle of patching holes as increasing traffic only worsens the problem.     Google Cloud Partner, SpringML worked with the City of Memphis to tackle this problem, assisting in repairing 63,000 potholes in one year, a massive improvement in pothole detection over previous manual efforts.  Advances in analytics and machine learning are making it possible for authorities to not only fix roads faster but actually prevent damage from occurring in the first place.Memphis Area Transit Authority busUsing machine learning for road maintenance The City of Memphis struggled with a problem many cities have to face: the continuous degradation of paved roads and the formation, through usage and weather, of potholes. These gaps in the road not only frustrate drivers, they slow down traffic, delaying commutes and mass transit, and they lead to greater wear and tear on vehicles. They’re just no good. Potholes are inevitable, so the challenge for Memphis, and other cities, becomes how to keep up, putting repair resources where they can be most helpful. With limited hardware and staff, they can’t tackle every report from citizens. And those public reports don’t always present a full picture of the problem either.Enter SpringML, who partners with public sector customers to solve problems with technology in creative ways. As the SpringML team joined with Memphis to figure this out, they first looked at what sorts of data they could get access to. And voila: bus cameras!“Look for data you already have that can fuel your decision making, before you go out and try to acquire new data sets.” Eric Clark, AI Practice, SpringMLThe city buses in Memphis all have front-mounted cameras, gathering data the entire time that the bus is running, mostly for traffic purposes. Every bus in the city was watching the roads every day! Immediately the team had a treasure trove of data: every road covered by the mass transit system has daily recordings being captured. The bus routes are well defined and each bus has GPS to help correlate the footage with precise locations. The team set to work.At the end of the day they retrieved videos from each bus and uploaded them to on-prem storage —a fairly manual process.Downloading the video data manually from drives that were on the busesBus system IT rack, tracking location and camera data as it travels its routeThen a script checked for new files in the video directory nightly, and uploaded the new videos to Google Cloud Storage, to begin processing.From there the Google Cloud Video Intelligence API could start to work, running its detection model on the new videos to look for possible pothole images. To make the initial pothole detection AI model the SpringML team took existing images and manually picked out potholes. They also used data from higher quality cameras to improve detection and accuracy of the model, and continued to feed new data from the bus routes to improve the model over time. Results from the Video Intelligence model inference were sent to BigQuery, where the images, annotations, file metadata, location and scoring were kept and easily sorted or queried.Some of the data from the BigQuery model, as it outputs pothole location and severityApplication used by public works employees to evaluate potholesThe custom web-app presented possible potholes to public works employees, who could help correct the model when it made mistakes (frequently caused by stains, shadows or animals), or confirm a pothole and then trigger the next automated flow. Once a pothole is detected and confirmed, the team needs a work ticket to track the actual repair. So the web-app submits information about confirmed potholes to the city’s 311 information system, which can then generate a ticket, which will dispatch a work crew and repair vehicle to actually repair the road.The full process of pothole data collection and detectionA smooth road aheadAs well as detecting and fixing potholes faster, this effort has paved the way for future projects that can improve public infrastructure, as more of the data gets gathered and applied to decision making. Want to learn more? Read this Video Intelligence API quickstart to try it out. Listen to the interview with SpringML’s Eric Clark on the GCP Podcast, and check out more machine learning tools in our AI Platform.Related ArticleAnnouncing updates to AutoML Vision Edge, AutoML Video, and Video Intelligence APIWe’re introducing enhancements to our AI vision and video intelligence portfolio to help even more customers take advantage of machine le…Read Article
Quelle: Google Cloud Platform

da Agency

Implementing leader election on Google Cloud Storage

Leader election is a commonly applied pattern for implementing distributed systems. For example, replicated relational databases such as MySQL, or distributed key-value stores such as Apache Zookeeper, choose a leader (sometimes referred to as master) among the replicas. All write operations go through the leader, so only a single node is writing to the system at any time. This is done to ensure no writes are lost and the database is not corrupted.It can be challenging to choose a leader among the nodes of a distributed system due to the nature of networked systems and time synchronization. In this article, we’ll discuss why you need leader election (or more generally, “distributed locks”), explain why they are difficult to implement, and provide an example implementation that uses a strongly consistent storage system, in this case Google Cloud Storage.Why do we need distributed locks?Imagine a multithreaded program where each thread is interacting with a shared variable or data structure. To prevent data loss or corrupting the data structure, multiple threads should block and wait on each other while modifying the state. We ensure this with mutexes in a single-process application. Distributed locks are no different in this regard than mutexes in single-process systems.A distributed system working on shared data still needs a locking mechanism to safely take turns while modifying shared data. However, we no longer have the notion of mutexes while working in a distributed environment. This is where distributed locks and leader elections come into the picture.Use cases for leader electionTypically leader election is used to ensure exclusive access by a single node to shared data, or to ensure a single node coordinates the work in a system.For replicated database systems such as MySQL, Apache Zookeeper, or Cassandra, we need to make sure only one “leader” exists at any given time. All writes go through this leader to ensure writes happen in one place. Meanwhile, the reads can be served from the follower nodes.Here’s another example. You have three nodes for an application that consumes messages from a message queue; however, only one of these nodes is to process messages at any time. By choosing a leader, you can appoint a node to fulfill that responsibility. If the leader becomes unavailable, other nodes can take over and continue the work. In this case, a leader election is needed to coordinate the work.Many distributed systems take advantage of leader election or distributed lock patterns. However, choosing a leader is a nontrivial problem.Why is distributed locking difficult?Distributed systems are like threads of a single-process program, except they are on different machines and they talk to each other over the network (which can be unreliable). As a result, they cannot rely on mutexes or similar locking mechanisms that use atomic CPU instructions and shared memory to implement the lock.The distributed locking problem requires the participants to agree on who is holding the lock. We also expect a leader to be elected while some nodes in the system are unavailable. This may sound simple, but implementing such a system correctly can be quite difficult, in part due to the many edge cases. This is where distributed consensus algorithms come into the picture.To implement distributed locking, you need a strongly consistent system to decide which node holds the lock. Because this must be an atomic operation, it requires consensus protocols such as Paxos, Raft, or the two-phase commit protocol. However, implementing these algorithms correctly is quite difficult, as the implementations must be extensively tested and formally proved. Furthermore, the theoretical properties of these algorithms often fail to withstand real-world conditions, which has led to more advanced research on the topic.At Google, we achieve distributed locking using a service called Chubby. Across our stack, Chubby helps many teams at Google make use of distributed consensus without having to worry about implementing a locking service from scratch (and doing so correctly).Cheating a bit: Leveraging other storage primitivesInstead of implementing your own consensus protocol, you can easily take advantage of a strongly consistent storage system that provides the same guarantees through a single key or record. By delegating the responsibility for atomicity to an external storage system, we no longer need the participating nodes to form a quorum and vote on a new leader.For example, a distributed database record (or file) can be used to name the current leader, and when the leader has renewed its leadership lock. If there’s no leader in the record, or the leader has not renewed its lock, other nodes can run for election by attempting to write their name to the record. First one to come will win, because this record or file allows atomic writes.Such atomic writes on files or database records are typically implemented using optimistic concurrency control, which lets you atomically update the record by providing its version number (if the record has changed since then, the write will be rejected). Similarly, the writes become immediately available to any readers. Using these two primitives (atomic updates and consistent reads), we can implement a leader election on top of any storage system.In fact, many Google Cloud storage products, such as Cloud Storage and Cloud Spanner, can be used to implement such a distributed lock. Similarly, open source storage systems like Zookeeper (Paxos), etcd (Raft), Consul (Raft), or even properly configured RDBMS systems like MySQL or PostgreSQL can provide the needed primitives.Example: Leader election with Cloud StorageWe can implement leader election using a single object (file) on Cloud Storage that contains the leader data, and require each node to read that file, or run for election based on the file. In this setup, the leader must renew its leadership by updating this file with its heartbeat.My colleague Seth Vargo published such a leader election implementation – written in Go and using Cloud Storage – as a package within the HashiCorp Vault project. (Vault also has a leader election on top of other storage backends).To implement leader election among distributed nodes of our application in Go, we can write a program that makes use of this package in just 50 lines of code:This example program creates a lock using a file in Cloud Storage, and continually runs for election.In this example, the Lock() call blocks until the calling program becomes a leader (or the context is cancelled). This call may block indefinitely since there might be another leader in the system.If a process is elected as the leader, the library periodically sends heartbeats keeping the lock active. The leader then must finish work and give up the lock by calling the Unlock() method. If the leader loses the leadership, the doneCh channel will receive a message and the process can tell that it has lost the lock, as there might be a new leader.Fortunately for us, the library we’re using implements a heartbeat mechanism to ensure the elected leader remains available and active. If the elected leader fails abruptly without giving up the lock, after the TTL (time-to-live) on the lock expires, the remaining nodes then select a new leader, ensuring the overall system’s availability.Fortunately, this library implements the mentioned details around sending so-called periodic heartbeats, or how frequently the followers should check if the leader has died and if they should run for election. Similarly, the library employs various optimizations via storing the leadership data in object metadata instead of object contents, which is costlier to read frequently.If you need to ensure coordination between your nodes, using leader election in your distributed systems can help you safely achieve there’s at most one node that has this responsibility. Using Cloud Storage or other strongly consistent systems, you can implement your own leader election. However, make sure you are aware of all the corner cases before implementing a new such library.Further reading:Implementing leader election using Kubernetes APILeader election in distributed systems – AWS Builders LibraryLeader election –  Azure Design Patterns LibraryThanks to Seth Vargo for reading drafts of this article. You can follow me on Twitter.
Quelle: Google Cloud Platform