da Agency

APIs 101: Everything you need to know about API design

Application programming interfaces, or APIs, are how software talks to other software. They abstract the complexity of underlying systems so the systems can connect in novel ways even if they were never intended to interoperate. Consequently, APIs are key ingredients in both most modern digital experiences and the execution of many of today’s most exciting business opportunities. How much value an API provides, however, involves not only the functionality and data to which the API provides access, but also how the API is designed. Many APIs are designed for integration—that is, as one-off projects that connect systems but do not anticipate future use of the API. The most valuable APIs tend to be designed to make developers’ jobs easier, which typically means designing them with the expectation they’ll be used by other developers in the future. This is designing for consumption. The distinction can significantly impact a business’s efficiency and ability to innovate. APIs designed for consumption make valuable functionality and data reusable, letting developers modularly mix and match different APIs to create new digital experiences or enable new strategies. For goals such as streamlining partner engagement or facilitating participation in digital ecosystems, being able to leverage APIs in this way is crucial. APIs designed for integration, in contrast, may serve the needs of an immediate project but do not help developers do much with the APIs going forward. These APIs may not be designed in ways that future developers will easily understand and may not behave in ways future developers expect. This can lead to the creation of new APIs, adding work and delay that could have been avoided if the older APIs had been designed with broader vision in the first place. How can API designers ensure they’re building APIs that will maximize value and developer productivity? We’ve explored this topic many times in the Google Cloud Blog, and in this post, we offer a collection of some of our most useful API design tips and best practices.Different approaches: REST, RPC, and GraphQLWhat an API even entails can involve different approaches to system interaction and different design norms, so overviewing various API design models is a great starting point. Check out API Design: Understanding gRPC, OpenAPI and REST and When to Use Them, Rest vs. RPC: What Problems are You Trying to Solve with Your APIs?, GraphQL: Building a Consistent Approach for the API Consumer, and Why Your Web APIs Should Be Entity-Oriented to dive in. Holistic API design overviewsFor deep overviews of many API design topics, our ebook API Web Design: The Missing Link: Best Practices for Crafting. Interfaces that Developers Love provides a strong foundation, as does the Google Cloud API Design Guide, which has been used inside Google since 2014 and is the guide Google follows when designing Cloud APIs and other Google APIs. We also encourage you to read API Design Best Practices & Common Pitfalls, which summarizes a wide-ranging Q&A on API design. Specific Challenges and Best Practices Whereas the above articles address numerous broad topics, we’ve also explored many of the more granular and specific API design challenges that can impact an API’s long-term value. For example, APIs are in many ways about defining relationships. In a retailer’s API, the information model might address relationships among entities such as customers, orders, catalog items, carts, and so on. Similarly, a banking API expresses which customer an account belongs to or which account each credit or debit applies to. The most common way that API developers express relationships is to expose database keys, or proxies for them, in the fields of the entities they expose. However, at least for web APIs, that approach has several disadvantages over the alternative: the web link. To learn why, check out API Design: Why You Should use Links, Not Keys, to Represent Relationships in APIs. Similarly, when creating API URLs, it can be confusing — but impactful — to know when to use URLs built around names that are easy for humans to read versus when to use URLs that rely on static numeric identifiers. A bank account, for example, may be difficult to reliably reference if a numeric identifier is not used. Details about the account’s owner are all subject to change (e.g., name, address, marital status), or subject to ambiguity (date and place of birth), or both. Even if we have a reliable identifier for the owner, ownership of the account can change. A static numeric identifier is the most reliable choice—to dig deeper, read API Design: Choosing Between Names and Identifiers in URLs and, for a specific discussion of API design tradeoffs between human readability and system stability, don’t miss The False Dichotomy of URL Design in Web APIs.APIs designed for consumption are essentially software products for developers, which means they may be iterated and improved, just as any software product is. Handling versioning of new APIs can involve some nuances, however, so be sure to peruse API Design: Which Version of Versioning is Right for You? and Common Misconceptions about API Versioning. Finally, APIs can impact how single-page applications are indexed by search engines—so if this sounds pertinent to your business’s needs, our two-part series on API design and SEO might be right up your alley. Doing more with APIsYou’re now well on your way to creating more powerful, user-friendly, and versatile APIs—and because the point of these consumption-focused APIs is partly to facilitate reuse, you’re also on your way to enabling myriad new ways to build richer applications more quickly. To learn even more about how APIs are driving business results, be sure to check out our round-up of API-related topics from Google Cloud Next ‘20: OnAir. We can’t wait to see what you build.Related ArticleNo-code momentum: Accelerating app development and automationLearn why no-code development is gaining momentum and empowering citizen developers to create powerful automations and line-of-business a…Read Article
Quelle: Google Cloud Platform

da Agency

Meeting the need for speed with a cloud data warehouse

In our work with Google Cloud customers, we hear great stories of growth, change, and cloud success. We worked closely with Trendyol Group, a fast-growing e-commerce company based in Turkey, on their data warehouse migration project. Trendyol employs about 2,000 people, and the company’s e-commerce site gets about 50 billion page views per year, 5 billion visits per year, and 50 million monthly unique users. For this digital-native company, data has been at the heart of their business. Trendyol Group was facing unprecedented growth, and the Trendyol data warehouse (DWH) team had been challenged with the performance and scalability of their existing Vertica data warehouse, especially during the holiday shopping season and other busy retail seasons. Performance issues had become critical over the past 18 months, and had business impact. The DWH team realized that not being able to process data and deliver internal reports and dashboards on time was causing lost revenue and inaccurate supplier management. For example, the business couldn’t react quickly when suppliers made bad decisions or sold a product that didn’t actually have inventory.  The capacity limitations of the on-premises data warehouse forced the IT team to constantly tune performance and plan and scale capacity instead of focusing on business insights. Trendyol’s reporting team serves more than 600 users with roughly 2,000 workbooks and 7,000 views in Tableau. Prior to migration, Trendyol stored over 30 TB of data in their Vertica environment. In addition, there were over 300 slowly changing dimensions (SCDs) in the ETL pipelines, requiring the team to update 10% of the data every day, which led to an 11 TB truncate-insert during the ELT process. The size of the data was weighing down the business. Business users couldn’t meet SLAs for their Monday morning financial reports required by the executives. To meet those busy periods, their IT team had to spend time tuning workloads by killing long-running queries to ensure timely completion of the reports. For example, business users couldn’t run queries that spanned a three-year period for aggregations due to capacity issues; they could only do a one-year time frame. By the time that business users accessed the report, the data was already stale. Then, when Thursdays came and users weren’t running as many queries, the DWH team found themselves with excess capacity. With the impact of COVID-19, Trendyol needed to be able to react quickly and cut off non-compliant products or suppliers to be able to meet the sudden increase in demand. The DWH team knew they needed to auto-scale the workloads in a cost-effective way. They extended their Vertica environment for one more year while they started evaluating cloud data warehousing alternatives. Cloud data warehouse decision criteria The Trendyol team decided to look into a number of vendors, including Snowflake and Google Cloud. Their decision criteria for a cloud data warehouse included:Instant scalability. Given the variability in their analytical workload, this was a critical need so they could have capacity on demand to run the Monday morning reports. Reduced operational costs. Since the retail business is seasonal, Trendyol needed to keep their costs low, in line with demand. Uptime SLAs. Their analytical platform needed to be highly available to meet business needs, especially in these critical times. BigQuery now offers a 99.99% SLA.Backup and recovery. This is important so the team can look back in time, in case there are errors in processing. Security. This is a key requirement for them, since they need to restrict access to personally identifiable information (PII) and sensitive data depending on roles. Ease of use. It was very important that business users could transition to the new cloud data warehouse platform without a learning curve and could be productive immediately.Evaluating cloud data warehousesBigQuery’s comprehensive documentation and simple management interface let the Trendyol team set up BigQuery and fine-tune queries for their proof-of-concept trial. Other data warehouse vendors’ trials required a consultant to optimize and tune the environment. They were able to move the data into BigQuery by themselves and it just worked. They also used BigQuery features like time travel, which met backup and recovery requirements out of the box, and integrated Cloud Identity and Access Management (Cloud IAM) roles that met security requirements easily.The most important feature in BigQuery for Trendyol was the separation of storage and compute, so that they would not have to pay for compute when not in use. Furthermore, it was easy to scale their workload up and down without the need for startup or shutdown time, which other tools required. The DWH team ran through a comprehensive evaluation of alternative data warehouse tools with a variety of benchmarks to represent their main workloads, including ELT, end-to-end BI, integration with BI, and a number of different OLAP queries. BigQuery was the preferred option for price and performance for each of the workloads. Here are three example queries—OLAP-type queries with joins involving a billion rows (Regex, 20+ analytical functions). 1. Ad hoc queries representing power users: Join four tables; high cartesian joins: 632m, 162m, 13m, 23k Regex functions, 2x dist. count2. ELT publish layer with analytical functions: Join five tables; rows: 800m, 13m, 11m, 10m, 360k 20+ analytical functions, first/last_value group by3. Example of publish layer: Join 13 tables (including subqueries), rows: 274m, 262m, 135m, 13m,10x group byTesting results from TrendyolConcurrent queries: BigQuery was the most cost-effective and faster compared to the alternative solution. BigQuery allowed testing increased slots and sharing resources seamlessly across reservations, which wasn’t possible in the alternative solution.DML statements performance: Similar across platforms, including CTAS/updates/inserts, but BigQuery was the most cost-effective. End-to-end runtime: With BI run, BigQuery was faster.Ingestion times: BigQuery was an order of magnitude faster. Data ingestion benchmark: 492 Parquet files with size of 63GB (400 million rows, 50 columns, snappy compressed)ELT’s SCD phase: With one of the largest dimensions, creates more than 2.1 million updates and approximately 1 million inserts.Overall, BigQuery provided the best price for performance, and its predictable flat-rate pricing was key for the decision. In the past, the DWH team had purchased capacity ahead of time, and often thought that it would end up being utilized, but didn’t—creating significant cost and unpredictability. The team would now be able to predict how much capacity they would use at the end of each month. And, the ability to scale up and down in minute intervals with Flex Slots was not available from any other vendor.  Migrating to BigQueryThe Trendyol DWH team separated the migration into three main categories:BI/Tableau migration was done in two weeks. The team changed 50 source tables accessed by 2,000 worksheets and 7,000 weave reports. Since Tableau has a native connection to BigQuery, it was easy to migrate. They used the same table and column names in BigQuery, matching the ones Tableau reports are using, and it just worked. They also avoided using custom SQL in Tableau, eliminating the need to rewrite most of the reports. The team found BigQuery’s ANSI SQL-compliant dialect to be compatible with most of their requirements. Additionally they had some custom SQL with a good amount of regular expressions, which were easily addressed by writing around 10 UDFs. ETL: More than 1,500 ETL jobs are scattered across three tools (Attunity, custom Python scripts, and Kafka Connect). The team has been doing ETL on-prem, and now, in the second phase of the migration, they’ve started migrating ETL to BigQuery.Data: There was 22 TB to start in Vertica that the team moved to BigQuery. They used Attunity for SQL Server and Kafka Connect for cloud-based sources. In addition, custom Python code integrated natively with the BigQuery JDBC driver. Within three months, the team ingested 600 TB into BigQuery, an order of magnitude larger than they had expected.  Currently, the Trendyol team stores 650 TB of data and 300+ SCDs in BigQuery and processes 11 TB of data daily. They mix and match flat-rate, slot reservations, and Flex Slots to get the best pricing at any given time. For example, they can now handle fluctuations in demand by purchasing Flex Slots on an on-demand basis. The data team can now concentrate on creating value rather than spending time on operationalizing the data warehouse. The relationship between the IT and business teams has been transformed, too. There are now plenty of compliments on speed and scalability. The data team can now produce reports in an hour on Monday morning, meeting their SLAs comfortably. The ODS pipeline previously took two to three hours, depending on the day. Trendyol’s BigQuery migration has helped restore trust between the IT and business teams, enable data-driven decision making, save on costs, and meet customer needs quickly. Learn more about Trendyol and about BigQuery’s data warehouse migration program.
Quelle: Google Cloud Platform

da Agency

Docker and Snyk Extend Partnership to Docker Official and Certified Images

Today we are pleased to announce that Docker and Snyk have extended our existing partnership to bring vulnerability scanning to Docker Official and certified images. As the exclusive scanning partner for these two image categories, Snyk will work with Docker to provide developers with insights into our most popular images. It builds on our previous announcement earlier this year where Snyk scanning was integrated into the Docker Desktop and Docker Hub. This means that developers can now incorporate vulnerability assessment along each step of the container development and deployment process.

Docker Official images represent approximately 25% of all of the pull activity on Docker Hub. Docker Official images are used extensively by millions of developers and developer world wide teams to build and run tens of millions of containerized applications. By integrating vulnerability scanning from Snyk users are now able to get more visibility into the images and have a higher level of confidence that their applications are secure and ready for production.

Docker Official images that have been scanned by Snyk will be available early next year.

You can read more about it from Snyk here and you can catch Docker CEO Scott Johnson and Snyk CEO Peter McKay discuss the partnership during the Snykcon user conference keynote Thursday morning October 22 at 8:30 AM Pacific. You can register for Snykcon at http://bit.ly/SnykConDocker

Additional Resources

Get started with scanning in the desktop nowhttps://www.docker.com/get-started

Learn more about scanning in Docker Hubhttps://goto.docker.com/on-demand-adding-container-security.htmlLearn more about scanning in Docker Desktop https://goto.docker.com/on-demand-find-fix-container-image-vulnerabilities.html
The post Docker and Snyk Extend Partnership to Docker Official and Certified Images appeared first on Docker Blog.
Quelle: https://blog.docker.com/feed/

da Agency

Graceful shutdowns on Cloud Run: Deep dive

Cloud Run now sends a SIGTERM signal to your container instance before the container instance terminates, due to an event like scale down or deleted revision.  By handling this signal, you can now gracefully terminate your applications and do some cleanup tasks –as opposed to an abrupt shutdown of the container.In this blog, we will explore some use cases for this and how you can try it out.Graceful shutdownsWhen a container instance is shut down on Cloud Run, a SIGTERM signal will be sent to the container and your application will have 10 seconds to exit. If the container does not exit by then, a SIGKILL signal (which you cannot capture) will be sent to abruptly close your application. If you choose not to write a signal handler for SIGTERM, your process is terminated instantly.Using this termination signal, you can perform various “graceful shutdown” tasks in your application code:Flush monitoring data: If you use Cloud Trace or upload metrics from your application, you can develop a signal handler and call the function that flushes out the trace spans collected before your container quits and loses these in-memory trace spans that are not uploaded.Log termination of your container: By logging the termination event of the container, you can refer to your application logs to see when a specific container instance has started and exited, and get full visibility into the lifecycle of individual container instances.Close file descriptors or database connections: Some abruptly quit connections can confuse the connected servers and cause them to keep connections open for a long time than gracefully disconnecting.The graceful termination signal is primarily sent to your application when it’s scaling down container instances that are not getting traffic. Therefore, you don’t need to handle draining in-flight requests in your signal handler. However, you might sometimes receive this signal before your container will be shut down due to underlying infrastructure reasons and your container might still have in-flight connections. The graceful termination is therefore not always guaranteed.Trapping signals, the right wayMost programming languages provide libraries to trap termination signals like SIGTERM and run routines before your program terminates.If your application does not receive the termination signal on Cloud Run, the most prominent reason for this might be because your application is not running as the init process (PID 1) and its parent process is not forwarding the signal appropriately.The leading reason why this happens is that the ENTRYPOINT statement in your container image’s Dockerfile is not set directly to your application process. For example, the Dockerfile statement:ENTRYPOINT node server.jsinternally is translated to:ENTRYPOINT [“/bin/sh”, “-c”, “node server.js”]when your Dockerfile is executed to build a container image.Most notably, the GNU /bin/sh and other shells like bash do not forward signals to child processes by default. Therefore, you should write your entrypoint statements in the vector form, like the following, to prevent your app to be executed as the sub-process of a shell:ENTRYPOINT [“python”, “server.js”]Similarly, if you use an entrypoint script to kick off background processes in your containers, consider using a proper init process that can forward signals to child processes, such as tini, dumb-init or supervisord. (I have compared init process alternatives for the multi-process container use case here.)See graceful shutdowns in actionTo try trapping SIGTERM signals in Cloud Run, let’s a small Node.js server app to try this out. For this purpose, let’s take the Node.js sample application for Cloud Run. You can download it from this repository on GitHub.Add this snippet of code to index.js:process.on(‘SIGTERM’, function () {    console.log(‘helloworld: received SIGTERM, exiting gracefully’);    process.exit(0); });After completing this step, you can now build and push this container image, and deploy it to Cloud Run. As part of your new deployment a container instance is spun up to handle the request.After some time passes, your container will scale to zero since it is not getting any requests. (If you want to trigger a scale-to-zero event, you can also edit your Cloud Run application’s settings such as CPU or memory on the Google Cloud Console. This will deploy a new revision, and the old revision will be turned off.)The scale-to-zero will trigger a SIGTERM signal to be sent to your container before it is shut down, and you can see the graceful shutdown routine executed in the Logs tab:As you can see, Cloud Run does not require additional settings to enable graceful shutdowns. It’s turned on by default for all Cloud Run services. ConclusionIf you have cleanup tasks or have monitoring data to push out before your serverless container instances on Cloud Run shut down, give termination signals a try. Check out our documentation for more information about this feature.
Quelle: Google Cloud Platform

da Agency

Unifiedpost and Google collaborate on Document AI to automate procurement data capture

Belgian fintech company, Unifiedpost Group has deployedProcurement DocAI to process nearly 350 million invoices and other procure-to-pay docs per year, in 15 countries across Europe. Procurement DocAI, which was announced at Google Cloud Next OnAir, automates the capture of invoices, receipts and other procurement data at scale. It takes unstructured documents across a variety of formats and turns them into cleanly structured data, increasing operational efficiency and improving document processing accuracy.    “At Unifiedpost, we believe that administrative and financial processes should be simple and smart. Google Cloud’s Procurement DocAI solution helps us achieve that goal by providing the best in class document automation processing with high accuracy and global, multi-language support.” – Hans Leybaert, CEO, Unifiedpost GroupLarge enterprise procurement and distribution networks as well as small and medium enterprises (SMEs) generate millions of invoices, receipts, and other related documents a year. These processes generate significant overhead for every procured item. With competitive pressures increasing, businesses are finding new ways to automate one of their highest volume business processes — the procurement cycle.A key part of our strategy at Google Cloud is the creation of industry-specific solutions that address vertical needs. Cross-industry solutions like Document AI are built to plug into your existing workflows and deliver business results from AI without having to hire an army of AI experts, or manage cloud infrastructure to get there.Procurement DocAI is one of our newest solutions, and it’s deployed by many customers and partners including Unifiedpost. The company currently serves 400k SMEs and 250+ corporations; their offering includes: documents (e.g., invoice PDF to XML conversion, e-invoicing), identity (e.g., KYC), and payments (e.g., IBAN accounts, online collection services, PSD2 payment services). With a large, rapidly growing, and heavily multi-language footprint, Unifiedpost sought the best industry solutions to meet their automation challenges and connect their customers such as Billtobox.com or JeFacture.com. Google Cloud’s Procurement DocAI delivered two key benefits to address Unifiedpost’s business needs:Lower TCO of procure-to-pay processing. Unifiedpost will be able to lower their TCO of procure-to-pay processing costs by up to 60% with Procurement DocAI. This solution provided them a cost effective approach for data extraction for invoices, receipts, and other valuable documents in the procurement cycle of procure to invoice, and invoice to pay.Improve procurement document processing accuracy. Procurement DocAI also helped boost data accuracy by 250% for Unifiedpost’s document extraction through specialized DocAI parsers with advanced OCR, computer vision, and Natural Language Processing. Additionally, Unifiedpost needed rapid multi-language expansion to enable regionalization support across Europe, particularly for French and Dutch to begin with, which Google delivered in less than a month.The collaboration between Google Cloud and Unifiedpost is just one of the latest examples of how we’re providing AI-powered functional solutions to solve business problems by leveraging our Deployed AI approach.Let’s connectFor more customer stories from EMEA, check out Google Cloud Next OnAir EMEA, which has tailored content to support the needs of our customers in Europe, Middle East, and Africa, kicking off from 29 September – 27 October.
Quelle: Google Cloud Platform