da Agency

Using machine learning to improve road maintenance

There’s a new way to look out for potholes in the road and it doesn’t involve better eyeglasses or dispatching costly repair crews. Bus-mounted cameras and machine learning can do it for you, as the City of Memphis discovered.    Staying on top of deteriorating roads when you can’t add more personnel is a never ending cycle of patching holes as increasing traffic only worsens the problem.     Google Cloud Partner, SpringML worked with the City of Memphis to tackle this problem, assisting in repairing 63,000 potholes in one year, a massive improvement in pothole detection over previous manual efforts.  Advances in analytics and machine learning are making it possible for authorities to not only fix roads faster but actually prevent damage from occurring in the first place.Memphis Area Transit Authority busUsing machine learning for road maintenance The City of Memphis struggled with a problem many cities have to face: the continuous degradation of paved roads and the formation, through usage and weather, of potholes. These gaps in the road not only frustrate drivers, they slow down traffic, delaying commutes and mass transit, and they lead to greater wear and tear on vehicles. They’re just no good. Potholes are inevitable, so the challenge for Memphis, and other cities, becomes how to keep up, putting repair resources where they can be most helpful. With limited hardware and staff, they can’t tackle every report from citizens. And those public reports don’t always present a full picture of the problem either.Enter SpringML, who partners with public sector customers to solve problems with technology in creative ways. As the SpringML team joined with Memphis to figure this out, they first looked at what sorts of data they could get access to. And voila: bus cameras!“Look for data you already have that can fuel your decision making, before you go out and try to acquire new data sets.” Eric Clark, AI Practice, SpringMLThe city buses in Memphis all have front-mounted cameras, gathering data the entire time that the bus is running, mostly for traffic purposes. Every bus in the city was watching the roads every day! Immediately the team had a treasure trove of data: every road covered by the mass transit system has daily recordings being captured. The bus routes are well defined and each bus has GPS to help correlate the footage with precise locations. The team set to work.At the end of the day they retrieved videos from each bus and uploaded them to on-prem storage —a fairly manual process.Downloading the video data manually from drives that were on the busesBus system IT rack, tracking location and camera data as it travels its routeThen a script checked for new files in the video directory nightly, and uploaded the new videos to Google Cloud Storage, to begin processing.From there the Google Cloud Video Intelligence API could start to work, running its detection model on the new videos to look for possible pothole images. To make the initial pothole detection AI model the SpringML team took existing images and manually picked out potholes. They also used data from higher quality cameras to improve detection and accuracy of the model, and continued to feed new data from the bus routes to improve the model over time. Results from the Video Intelligence model inference were sent to BigQuery, where the images, annotations, file metadata, location and scoring were kept and easily sorted or queried.Some of the data from the BigQuery model, as it outputs pothole location and severityApplication used by public works employees to evaluate potholesThe custom web-app presented possible potholes to public works employees, who could help correct the model when it made mistakes (frequently caused by stains, shadows or animals), or confirm a pothole and then trigger the next automated flow. Once a pothole is detected and confirmed, the team needs a work ticket to track the actual repair. So the web-app submits information about confirmed potholes to the city’s 311 information system, which can then generate a ticket, which will dispatch a work crew and repair vehicle to actually repair the road.The full process of pothole data collection and detectionA smooth road aheadAs well as detecting and fixing potholes faster, this effort has paved the way for future projects that can improve public infrastructure, as more of the data gets gathered and applied to decision making. Want to learn more? Read this Video Intelligence API quickstart to try it out. Listen to the interview with SpringML’s Eric Clark on the GCP Podcast, and check out more machine learning tools in our AI Platform.Related ArticleAnnouncing updates to AutoML Vision Edge, AutoML Video, and Video Intelligence APIWe’re introducing enhancements to our AI vision and video intelligence portfolio to help even more customers take advantage of machine le…Read Article
Quelle: Google Cloud Platform

da Agency

Implementing leader election on Google Cloud Storage

Leader election is a commonly applied pattern for implementing distributed systems. For example, replicated relational databases such as MySQL, or distributed key-value stores such as Apache Zookeeper, choose a leader (sometimes referred to as master) among the replicas. All write operations go through the leader, so only a single node is writing to the system at any time. This is done to ensure no writes are lost and the database is not corrupted.It can be challenging to choose a leader among the nodes of a distributed system due to the nature of networked systems and time synchronization. In this article, we’ll discuss why you need leader election (or more generally, “distributed locks”), explain why they are difficult to implement, and provide an example implementation that uses a strongly consistent storage system, in this case Google Cloud Storage.Why do we need distributed locks?Imagine a multithreaded program where each thread is interacting with a shared variable or data structure. To prevent data loss or corrupting the data structure, multiple threads should block and wait on each other while modifying the state. We ensure this with mutexes in a single-process application. Distributed locks are no different in this regard than mutexes in single-process systems.A distributed system working on shared data still needs a locking mechanism to safely take turns while modifying shared data. However, we no longer have the notion of mutexes while working in a distributed environment. This is where distributed locks and leader elections come into the picture.Use cases for leader electionTypically leader election is used to ensure exclusive access by a single node to shared data, or to ensure a single node coordinates the work in a system.For replicated database systems such as MySQL, Apache Zookeeper, or Cassandra, we need to make sure only one “leader” exists at any given time. All writes go through this leader to ensure writes happen in one place. Meanwhile, the reads can be served from the follower nodes.Here’s another example. You have three nodes for an application that consumes messages from a message queue; however, only one of these nodes is to process messages at any time. By choosing a leader, you can appoint a node to fulfill that responsibility. If the leader becomes unavailable, other nodes can take over and continue the work. In this case, a leader election is needed to coordinate the work.Many distributed systems take advantage of leader election or distributed lock patterns. However, choosing a leader is a nontrivial problem.Why is distributed locking difficult?Distributed systems are like threads of a single-process program, except they are on different machines and they talk to each other over the network (which can be unreliable). As a result, they cannot rely on mutexes or similar locking mechanisms that use atomic CPU instructions and shared memory to implement the lock.The distributed locking problem requires the participants to agree on who is holding the lock. We also expect a leader to be elected while some nodes in the system are unavailable. This may sound simple, but implementing such a system correctly can be quite difficult, in part due to the many edge cases. This is where distributed consensus algorithms come into the picture.To implement distributed locking, you need a strongly consistent system to decide which node holds the lock. Because this must be an atomic operation, it requires consensus protocols such as Paxos, Raft, or the two-phase commit protocol. However, implementing these algorithms correctly is quite difficult, as the implementations must be extensively tested and formally proved. Furthermore, the theoretical properties of these algorithms often fail to withstand real-world conditions, which has led to more advanced research on the topic.At Google, we achieve distributed locking using a service called Chubby. Across our stack, Chubby helps many teams at Google make use of distributed consensus without having to worry about implementing a locking service from scratch (and doing so correctly).Cheating a bit: Leveraging other storage primitivesInstead of implementing your own consensus protocol, you can easily take advantage of a strongly consistent storage system that provides the same guarantees through a single key or record. By delegating the responsibility for atomicity to an external storage system, we no longer need the participating nodes to form a quorum and vote on a new leader.For example, a distributed database record (or file) can be used to name the current leader, and when the leader has renewed its leadership lock. If there’s no leader in the record, or the leader has not renewed its lock, other nodes can run for election by attempting to write their name to the record. First one to come will win, because this record or file allows atomic writes.Such atomic writes on files or database records are typically implemented using optimistic concurrency control, which lets you atomically update the record by providing its version number (if the record has changed since then, the write will be rejected). Similarly, the writes become immediately available to any readers. Using these two primitives (atomic updates and consistent reads), we can implement a leader election on top of any storage system.In fact, many Google Cloud storage products, such as Cloud Storage and Cloud Spanner, can be used to implement such a distributed lock. Similarly, open source storage systems like Zookeeper (Paxos), etcd (Raft), Consul (Raft), or even properly configured RDBMS systems like MySQL or PostgreSQL can provide the needed primitives.Example: Leader election with Cloud StorageWe can implement leader election using a single object (file) on Cloud Storage that contains the leader data, and require each node to read that file, or run for election based on the file. In this setup, the leader must renew its leadership by updating this file with its heartbeat.My colleague Seth Vargo published such a leader election implementation – written in Go and using Cloud Storage – as a package within the HashiCorp Vault project. (Vault also has a leader election on top of other storage backends).To implement leader election among distributed nodes of our application in Go, we can write a program that makes use of this package in just 50 lines of code:This example program creates a lock using a file in Cloud Storage, and continually runs for election.In this example, the Lock() call blocks until the calling program becomes a leader (or the context is cancelled). This call may block indefinitely since there might be another leader in the system.If a process is elected as the leader, the library periodically sends heartbeats keeping the lock active. The leader then must finish work and give up the lock by calling the Unlock() method. If the leader loses the leadership, the doneCh channel will receive a message and the process can tell that it has lost the lock, as there might be a new leader.Fortunately for us, the library we’re using implements a heartbeat mechanism to ensure the elected leader remains available and active. If the elected leader fails abruptly without giving up the lock, after the TTL (time-to-live) on the lock expires, the remaining nodes then select a new leader, ensuring the overall system’s availability.Fortunately, this library implements the mentioned details around sending so-called periodic heartbeats, or how frequently the followers should check if the leader has died and if they should run for election. Similarly, the library employs various optimizations via storing the leadership data in object metadata instead of object contents, which is costlier to read frequently.If you need to ensure coordination between your nodes, using leader election in your distributed systems can help you safely achieve there’s at most one node that has this responsibility. Using Cloud Storage or other strongly consistent systems, you can implement your own leader election. However, make sure you are aware of all the corner cases before implementing a new such library.Further reading:Implementing leader election using Kubernetes APILeader election in distributed systems – AWS Builders LibraryLeader election –  Azure Design Patterns LibraryThanks to Seth Vargo for reading drafts of this article. You can follow me on Twitter.
Quelle: Google Cloud Platform

da Agency

Top Developer Trends for 2021

The year 2020 will go down in the history books for so many reasons. For Docker, despite the challenges of our November 2019 restructuring, we were fortunate to see 70% growth in activity from our 11.3 million monthly active users sharing 7.9 million apps pulled 13.6 billion times per month. Thank you, Docker team, community, customers, and partners!

But with 2020 behind us it’s natural to ask, “What’s next?” Here in the second week of January, we couldn’t be more excited about 2021. Why? Because the step-function shift from offline to online of every dimension of human activity brought about by the global pandemic is accelerating opportunities and challenges for development teams. What are the key trends relevant to development teams in 2021? Here are our top picks:

The New Normal: Open, Distributed Collaboration

While already a familiar teamwork model for many open source projects and Internet companies, the global pandemic seemingly overnight drove all software development teams to adopt new ways of working together. In fact, our 2020 survey of thousands of Docker developers about their ways of working found that 51% prefer to work mostly remote and only sometimes in an office if/when given a choice. 

While not without its challenges, we are seeing success in teams that not only embrace fully-distributed collaboration – they use it to do what they couldn’t do before. This includes the opportunity to find the absolute best talent, staying closer to customers in their own timezones, following-the-sun collaborations, and more. Remote development work is here to stay in 2021. 

Critical to remote teamwork efficiency is the ability of team members to share and keep consistent their development environments and application stacks. To achieve this we’re seeing developers use docker-compose.yml locally in Docker Desktop to define and pull their team’s shared images of their app stacks from Docker Hub. Centralized team sharing and visibility of image versions, vulnerabilities, test results, and more reduce “works on my machine” ambiguities to speed up delivery.

More Microservices, Less Complexity

Microservices-based applications proved their worth many times over in 2020. Specifically, they compress “idea-to-production” timeframes and enable rapid, dynamic reactions to changes in demand – both incredibly valuable in 2020. And in our developer community, 65% say their organizations are already leveraging microservices.

But one friction to adoption is the complexity introduced by microservices architectures and the underlying infrastructure. To address this, we see many development teams reaching for Docker Compose as a means for defining multi-service apps capable of running on any infrastructure. Specifically, as a result of open sourcing the Compose spec almost a year ago, Compose-defined apps can now be built locally in Docker Desktop and then deployed, unchanged, to run on Kubernetes, AWS ECS, Microsoft Azure ACI, and Swarm. This allows development teams to focus their energies on building their apps vs. wrestling with the complexities of app dependencies on the underlying infrastructure.

Accelerating Democratization of AI/ML

The exponentially-changing daily landscape of the global pandemic puts a premium on data velocity and the ability to rapidly extract insights to drive decisions. Automating via AI/ML the “collect > analyze > decide” cycle is no longer a “nice-to-have” for organizations – it’s becoming a “must-have” tool in every development team’s toolkit.

Docker simplifies the adoption of AI/ML tools for development teams and data scientists, helping them build shareable, reproducible environments. GPU support in both Docker Desktop and our Docker – AWS ECS integration ensures consistency between local and remote deployments. Furthermore, to jumpstart any AI/ML project TensorFlow, the open source ML platform, is freely available as a Docker Official Image, and the Pytorch and Juypter community images are some of the most popular on Docker Hub.

While 35% of survey respondents said their organizations are currently using machine learning, a number of developers listed ML as the technology they most want their organization to adopt. Thus, in 2021 we expect to see increased adoption of AI/ML tools by development teams for use in their applications.

—

The above are the key major trends triggered by 2020 that the Docker team anticipates accelerating in 2021. As always, our developer community plays a big role in determining where and how Docker can further help their teams build, share, and run modern apps, and thus we encourage you to contribute to our product roadmap early and often. Finally, to learn more about how Docker is helping development teams take advantage of these trends, be sure to attend DockerCon Live 2021.  

There’s a big year ahead of us – let’s have a great one!
The post Top Developer Trends for 2021 appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/