Cloud Computing Köln

Introducing Unattended Project Recommender: discover, reclaim, or deprecate abandoned projects under your organization

In fast-moving organizations, it’s not uncommon for cloud resources, including entire projects, to occasionally be forgotten about. Not only such unattended resources can be difficult to identify, but they also tend to create a lot of headaches for product teams down the road, including unnecessary waste and security risks. To help you prune your idle cloud resources, we’re excited to introduce Unattended Project Recommender. It’s a new feature of Active Assist that provides you with a one-stop shop for discovering, reclaiming, and shutting down unattended projects. With actionable and automatic recommendations, you no longer have to worry about wasting money or mitigating security risks presented by your idle resources. Unattended Project Recommender uses machine learning to identify, with a high degree of confidence, projects that are likely abandoned based on API and networking activity, billing, usage of cloud services, and other signals. This feature is available via the Recommender API today, making it easy for you to integrate with your company’s existing workflow management and communication tools, or export results to a BigQuery table for custom analysis.Thousands of projects can be unattended in large organizations, presenting major security risksYour cloud projects can go abandoned or unattended for a number of reasons — ranging from a test environment that’s no longer needed, to project cancellation, to project owner switching jobs, and more. Not only can such projects contribute to your cloud bill (waste) but they may contain security issues such as open firewalls or privileged service account keys that attackers can exploit to get a hold of your cloud resources for cryptocurrency mining or, worse, compromise your company’s sensitive data. These security risks tend to grow over time because the latest best practices and patches are usually not applied to unattended projects. We experience this issue here at Google, too… In fact, it has been on Google’s internal security team’s radar for some time now, so we joined forces and looked into this problem together, starting with our very own “google.com”organization cloud projects. We quickly found some projects that were unattended, but remediating this issue was easier said than done due to challenges in several areas:Detection: With lots of signals available to you via sources like Cloud Monitoring, what are the right ones you should look at (e.g. API, networking, user activity)? How can you tell the difference between an unattended project and a project that has a low level of activity by design (e.g. a “shell” project that holds an auth token)?Remediation: Once you have identified a project that seems abandoned, how do you go about ensuring that it’s indeed an unattended project? How do you reduce the risk of deleting something that might be essential to a production workload, causing irreversible data loss? How do you solve this at the scale of your entire organization, beyond a one-time cleanup? Over the course of 2021 we built and tested a Google-internal prototype first, cleaning up many of our internal unattended projects, and then worked with a number of Google Cloud customers to build and tune this feature based on real-life data (thank you to all of our early adopters for working with us and your generous feedback that helped us shape this feature!) It was not uncommon for us to come across organizations with thousands of unattended projects, and we’re very excited to bring Unattended Project Recommender to all customers, in public preview.Discovering and acting on unattended project recommendationsUnattended Project Recommender analyzes usage activity across all projects under your organization, including the following data:API activity (e.g. service accounts with authentication activity, API calls consumed)Networking activity (ingress and egress)Billing activity (e.g. services with billable usage)User activity (e.g. active project owners)Cloud services usage (e.g. number of active VMs, BigQuery jobs, storage requests)Based on these signals, it can generate recommendations to clean up projects that have low usage activity (where “low usage” is defined using a machine learning model that ranks projects in your organization by level of usage), or recommendations to reclaim projects that have high usage activity but no active project owners. Here’s what an example post-processed summary list of recommendations can look like for the “foobar” organization that has 3 projects:In addition to the recommendations, you can also examine the underlying project activity insights that the recommendations are based upon. The insights provide additional information that can be useful for integration with your organization’s existing workflows and automation (e.g. send an auto-generated email or chat message to project owners based on the list provided by the owners field). Here’s an example insight payload:GCP projects are used in many different ways and for many different purposes. In case you get a recommendation to delete a project that’s being used in a way that’s out of the scope for this feature, you can dismiss the recommendation and it will stop showing up for the given project. Restoring deleted projectsWhen you choose to shut down a project using the projects.delete() method, it gets marked for deletion. After a project is marked for deletion, it becomes unusable, all resources within that project are shut down, and a 30-day wait period for the project and all of its data to get fully deleted begins.In case a useful project is accidentally shut down, you have the option to restore the project within that 30-day wait period. Since restoring allows you to recover most but not necessarily all of your project data and resources, we recommend carefully examining the utilization insights associated with a project and considering any additional utilization signals that may not be captured by the Unattended Project Recommender before taking the cleanup action.Early customer success storiesA number of enterprise customers are already using Unattended Project Recommender to keep their organizations clean of unattended projects and resources.Decathlon, a French sporting goods retailer, is excited for the insight Unattended Project Recommender will bring to their environment, and are already deploying it as a part of their latest cloud security initiatives.”After a thorough test of this feature and the validation of our CISO, we ended up deleting our first 775 projects, and no one complained! A great help to improve our security. The next step for us will be to operationalize it at scale, and implement a company wide policy for unattended resource management.” —Adeline Villette, Cloud Security OfficerFor Veolia, one of the world’s largest water, waste and energy management companies, not only does this feature reduce security risks and waste, but also helps drive cultural shift and alignment with its ecological transformation strategy.“This feature allows us to reduce our costs and security debt on assets that are no longer in use, and is also fully in line with Veolia’s philosophy of limiting its carbon footprint. After having tested Unattended Project Recommender on more than 3,000 projects throughout our organization, we are looking to bring it as proactive alerts to our project owners at scale.”—Thomas Meriadec, Product ManagerBox, a secure cloud content management provider, views it as a foundation for building a repeatable process to remediate unused resources.“Unattended Project Recommender is a great fit for us. It gives us a unified view of project usage across our entire organization and enables us to address security risks of legacy projects in a systematic and organized manner, ensuring an even safer environment.” —Matt Bowes, Staff Security EngineerGetting started with the Unattended Project RecommenderTo help you get started, we’ve prepared a Cloud Shell tutorial (source code) that you can use to find unattended project recommendations within your own Projects/Folders/Organization. Click this button to clone the tutorial from GitHub and run in your Cloud Shell environment:As you can see, listing recommendations for your projects only takes a few clicks with the tutorial (special thanks to Lanre Ogunmola, Security & Compliance Specialist, for making this look so easy)! For additional detail on using the gcloud CLI or API to discover unattended project recommendations, please refer to the documentation page.You can also automatically export all recommendations from your Organization to BigQuery and then investigate the recommendations with DataStudio or Looker, or use Connected Sheets that let you use Google Workspace Sheets to interact with the data stored in BigQuery without having to write SQL queries.As with any other Recommender, you can choose to opt out of data processing at any time by disabling the appropriate data groups in the Transparency & control tab under Privacy & Security settings.We hope that you can leverage Unattended Project Recommender to improve your cloud security posture and reduce cost, and can’t wait to hear your feedback and thoughts about this feature! Please feel free to reach us at active-assist-feedback@google.com and we also invite you to sign up for our Active Assist Trusted Tester Group if you would like to get early access to the newest features as they are developed.Related ArticleIntroducing Active Assist: Reduce complexity, maximize your cloud ROIIntroducing Active Assist, a family of tools to help you easily optimize your Google Cloud environment.Read Article
Quelle: Google Cloud Platform

Try a tutorial in the Google Cloud Console

When it comes to learning how to implement some technology, we all have our own version of what I call the “tab game”—that is, your setup for all the tabs and windows you need open at once. You may have several monitors so you can see documentation, your IDE, and terminal windows at the same time. You may have several guides and references open at once in one window to get all the information you need.Personally, I like to work just from my laptop because I like to move around and work from various comfy spots. I think my tab game would probably enrage most devs because it involves a lot of swiping back and forth between windows *and* toggling tabs. It’s not pretty. That is, it wasn’t pretty until I discovered tutorials in the Google Cloud Console!Jen really didn’t know about tutorials in the Google Cloud Console?Yes, I honestly didn’t know about them! I’m sharing about it because if I can work for Google and not know, then I can’t be the only one, and it would be a shame to miss out on this because it’s a brilliant idea. Also I wrote some pretty sweet tutorials for the console, but I swear that the main reason I’m telling you is because it’s a cool thing!There are several reasons that these tutorials are great:You can view the instructions and the console at the same time. No more playing the tab game!The tutorials include links and highlights, making it easy to find the screens and buttons you’re looking forYou can run code from Cloud Shell, so you don’t need a separate window for an IDEYou can use the demo data provided to try things out, or you can apply the steps to your existing projects using data that suits your app’s needsFirestore tutorialsI’m developing a series of tutorials in the Google Cloud Console designed to take you through everything you need to know about Firestore–from manually adding data in the Google Cloud Console to triggering Cloud Functions to make changes for you. Below are links and summaries for the currently available tutorials. Check back regularly to find the latest additions as they’re released!Add Data to FirestoreEnable Firestore on a projectLearn about the Firestore data modelAdd a collection of documentsAdd fields to a documentDelete documents and collectionsUpdating Data in Firestore using Node.js or using PythonAdd a collection of documentsExplore available data typesReplace the data of documentReplace fields in a documentHandle special cases: incrementing, timestamps, and arraysReading Data from Firestore using Node.js or using PythonAdd a collection of documentsExplore available data typesRead a collectionRead a single documentOrder documentsQuery documentsTransactions in Firestore using Node.jsAdd a collection of documentsUpdate data without a transaction to observe issueComplete a transactionComplete a batched writeBatched Writes in Firestore using Node.js or using PythonUse Cloud Shell and Cloud Shell Editor to write a Node.js or Python appComplete a batched writeFirestore triggers for Cloud FunctionsInitialize Cloud Functions using the Firebase CLIWrite a Cloud Function triggered by a new document write to FirestoreOffline Data in FirestoreAdd data to Firestore in the Cloud console Firestore dashboardCreate a web app that uses Firestore using the Firebase SDKDeploy Firestore security rules that enable access to the required dataEnable data persistence in the web appObserve app behavior with and without network connectionChime inIs there a particular action or concept in Firestore that you’d like to see a tutorial for? Is there another Google Cloud product that you want to learn more about? Tweet @ThatJenPerson and you may just see your suggestion come to life in the Google Cloud Console!Related ArticleBuilding scalable web applications with Firestore — new reference guideGoogle’s Firestore is a scalable, serverless document database that lets you build web or mobile apps. This new guide compares Firestore …Read Article
Quelle: Google Cloud Platform

MySQL major version upgrade using Database Migration Service

As hopefully you all know by now, migrating your SQL database got easier this past November when Google’s Database Migration Service went GA for MySQL. If you didn’t know, migrating your SQL database to Cloud SQL is now easier! Something you might have noticed, or I’ll happily point out if you haven’t noticed, is that Database Migration Service lets you pull from a number of different sources, including on premises, AWS, and even Cloud SQL for MySQL.Side plug – we just announced DMS supports Cloud SQL for PostgreSQL as a source, so you can use DMS for your PostgreSQL major version upgrades too. But THIS post is about MySQL…Because something ELSE I wanted to call out… notice this, it appears on the “Create a destination” page of the DMS process:So if you’re following along here, I’m telling you that you can ALSO use DMS to do major version upgrades on your MySQL database. We’ll all still wait with baited breath for in-place upgrades of course, but while we’re waiting this is a great option.There’s a catch, because of course there is. The key here is that because we’re using a migration tool to upgrade, there’s two dimensions of complications to deal with. We have the migration piece and the upgrade piece. DMS helps manage a good amount of this, but there’s still some things you need to consider. In this post, I’m going to pull together all the pieces you need to think about and link to everything you need to do a major version upgrade for MySQL using Database Migration Service.Why are we here?Just to cover why you might want to upgrade MySQL in the first place. Between performance upgrades and feature updates, there’s plenty of reasons to do it. If you need convincing, here’s a nice list of the enhancements in MySQL 8.0. There’s also the elephant in the room: officially 5.6 was deprecated this past February. You might have been on 5.6 for the last 8 years, saw it’s officially end-of-lifed this year and are panicking a little bit. The good news here is that Cloud SQL will support 5.6 for a while longer, but that doesn’t mean now isn’t a good time to upgrade.Version compatibilityThe first thing to look at is what versions you’re upgrading between. So 5.6 to 5.7 or 5.7 to 8.0. 5.6 to 8.0 is right out. MySQL has significant changes between major versions that are likely to break compatibility, so you need to triple check your database for some of these incompatible changes.For example, between 5.6 and 5.7 you need to keep an eye out for any system or status variables in the INFORMATION_SCHEMA tables in your 5.6 database. Those were all replaced by the Performance Schema tables in 5.7.6. There are also a lot of little things–like if you have a column with the YEAR(2) data type, you’ll need to update all those values to a 4-digit YEAR column before you’ll be able to use those columns again. If you’re going 5.6 to 5.7, you can go over the full list of changes here.Between 5.7 and 8.0 of course there are yet more changes to watch out for. Big ones for me are that default flags were changed quite a bit between the two. While a lot of them might not break things with a segmentation fault, they could lead to some unintended behavior in your application. Also, you might want to take a peek at your AUTO_INCREMENT columns. It’s been deprecated for FLOAT and DOUBLE types. For a full list of what changed between these two versions, check out here.Connecting the dotsIf you’re planning on doing the major version upgrade from one Cloud SQL instance to another, you can skip this section because you’ve already done what I’m going to talk about. If, however, you’re changing from an on premises database, or in some small edge cases, a GCE (virtual machine) instance to Cloud SQL, there’s a few extra things to watch out for.#1 thing is latency. If you’ve gotten comfortable having a near-instantaneous response from your database because it lived right next to your application, be prepared for a bit of a wakeup call. Unless your application also lives in the Cloud, there’s now some extra time that will pass between your application and your database. This might be totally fine! Or, it could introduce some really difficult-to-debug re-entrancy bugs into your application. A colleague of mine wrote a great post that covers all the things you need to watch out and potentially change for when moving your database to Cloud SQL from another source here.#2 is how do I even now get to my database?! Network connectivity can be really difficult. Firewalls, routing and DNS can all throw a wrench in the works. You had everything all nicely connected before you did this, and now all of a sudden your database lives (potentially) in an entirely different part of the internet with different requirements to connect to it. This particular rabbit hole I’ve gone very deep on already. I wrote an in-depth blog post on connecting source to destination instances with Database Migration Service here. If you run into problems with connectivity, that post should cover most situations.More migrating!DMS makes the moving your data piece of the process easy, but since we rely on the database engine’s native migration mechanisms, we’re also subject to its limitations. This means there’s some steps to complete once you’re done migrating your database regardless of where it came from. For example, you know, your IP address will change. Because we’re using a migration method to upgrade our version, we’re instantiating a new instance. Once we get in-place upgrades this is a non-issue, but in the meantime we have to remember to update any applications, load-balancers, etc that are pointing at the old IP address.Another thing is, don’t forget, your users are stored in system tables that aren’t migrated either. Those will need to be brought over once the DMS portion is complete. There’s a few more bits and bobs, and there’s a complete post here that lists the things you’ll need to finish off before your new instance is ready for use.What’s next?Now that you’ve got your newly upgraded instance all set, it’s time to get it ready for production. There are a few things that you’ll likely want that can’t be set up until the Cloud SQL instance is fully online.By default, for example, SSL-only connections are off by default and we know many organizations enforce SSL-only connections. This will need to be enabled on the server and certificates will need to be set up. This can be done after the data is migrated but before the DMS target instance is promoted.Disaster recovery is another must-have for production instances. It’s also a requirement for our SLA which guarantees uptime. Absolutely need to set this up. If you edit your instance it’ll be on the main Overview page:Setting to “Multiple zones” enables this feature.Also falling into this category are read replicas. Optimizing your data infrastructure to support your application is key and this is an integral part of that. If you have a read-heavy application, you’ll want to set up your read replicas at this stage of the game as well.Setting up high availability and the read replicas can only be done after you’ve promoted the instance at the end of the DMS process.One last bit before I leave you. You’ve done all this work to upgrade your instance to the next version. You’ve set up all the extra pieces you need. You’ve connected your application and it can totally talk to your database. Now is NOT the time to hit the push to prod button. This may absolutely come across as obvious to many of you reading this. But I’m going to say it anyway. TEST! Test all the things. Test all the edge cases you can think of. Test adding even more latency into your system arbitrarily to be sure that you can handle random internet lag spikes between application and database. The high availability feature once it’s on allows you to test automatic failover to simulate your primary instance going down and the fallback instance taking over. So test. Test all these scenarios to ensure that you have a smooth launch of your new major version SQL instance.Hopefully this all helped to guide you to successfully using Database Migration Service to upgrade your MySQL instance to a new version! If you have questions or comments, please feel free to reach out to me on Twitter, my DMs are open! Thanks for reading.Related ArticleClosing the gap: Migration completeness when using Database Migration ServiceLearn what is and isn’t included when migrating a MySQL database to Cloud SQL using Database Migration Service (DMS).Read Article
Quelle: Google Cloud Platform