Eine Universität verlangt Videoüberwachung und Spyware bei Onlineprüfungen. Die GFF hält dies für grundrechtswidrig und klagt. (Gesellschaft für Freiheitsrechte, Datenschutz)
Quelle: Golem
The WordPress.com team is always working on new design ideas to bring your website to life. Below you’ll find the four newest themes that we’ve added to our library, with beautiful options for food-based businesses, podcasts, and bloggers.
To install any of the below themes, click the the name of the theme you like, which brings you right to the installation page. Then simply click the “Activate this design” button. You can also click “Open live demo,” which brings up a clickable, scrollable version of the theme for you to preview.
Premium themes are free to use for any user on a Premium plan or above, or can be purchased individually by those with free sites or Personal plans.
You can explore all of our themes by navigating to the “Themes” page, which is found under “Appearance” in the left-side menu of your WordPress.com dashboard. Or, just click here:
All WordPress.com Themes
Masu
Masu is a blogging-focused theme inspired by the traditional square wooden box used to measure rice in Japan. It has a warm tone and, naturally, features a square grid alignment.
Click here to view a demo of this theme.
Varese (Premium Theme)
Made for restaurants, bakeries, and other small businesses, Varese is a versatile and modern theme. It comes with a number of patterns, allowing you quickly design and build your pages and launch your website. With the support of OpenTable Block, get bookings and reservations to your business.
Click here to view a demo of this theme.
Spearhead Blocks
Spearhead Blocks is a podcasting starter theme. A modernized version of the original Spearhead, this theme allows you to easily customize and scale your podcast website as needed.
Click here to view a demo of this theme.
Munchies (Premium Theme)
Designed with portable business in mind, Munchies comes with patterns for food menus, a schedule for upcoming events, and more. Use this theme to share your story with your customers and to make it easier for them to connect with you.
Click here to view a demo of this theme.
Stay tuned for more updates about new themes, patterns, blocks, and other exciting product updates! And be sure to click below to take a look at the entire showcase of themes we offer:
All WordPress.com Themes
Quelle: RedHat Stack
Frequently, your application may need to count the number of matches for a given query. For example, if you’re developing a social application, you may need to count the number of friends an individual has. With today’s Preview launch of count(), you can easily and cost-efficiently perform a count() directly in Firestore. It’s accessible via all our server, client SDKs, Google Cloud and Firebase console.Use CasesSupport for count() now enables you to utilize Firestore for building applications which need aggregation support eg. a dashboard displaying the number of active players in a gaming app or counting the number of friends on a social media app etc., without reading all the docs. Count() can be leveraged to simply count the number of items in a collection or to count the results based on certain conditions in any application.For example, Let’s say you have a collection called “employees”. To get a count() of total number of employees, you run the below query:code_block[StructValue([(u’code’, u”async function getNumEmployees(db: Firestore): Promise<number> {rn const collection_ = collection(db, ‘employees’);rn const snapshot = await getCountFromServer(collection_);rn return snapshot.data().count;rn}”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e778e148790>)])]Now, let’s say you want to get a total count of “developers” among your employees. To get a count of total developers among your employees,you run the below query:code_block[StructValue([(u’code’, u”async function getNumDevelopers(db: Firestore): Promise<number> {rn const collection_ = collection(db, ‘employees’);rn const query_ = query(collection_, where(‘role’, ‘==’, ‘dev’));rn const snapshot = await getCountFromServer(query_);rn return snapshot.data().count;rn}”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e774ba88fd0>)])]How does COUNT work?Firestore computes count() based on index entry matches. Before we discuss the methodology used by the query planner, it’s important to familiarize yourself with two new terms:Index entries scanned: Number of index entries scanned to execute a query. Index entries matched: Number of index entries that match the query. When a query with a count() function is executed, we scan relevant index entries, and the index entries that match the query are then counted on the server. Since the matching documents never have to be retrieved, the performance of the query will primarily depend on the size of the number of index entries scanned. Pricing for COUNT()Firestore’s count() isn’t just easy to use, it is also cost-efficient! You’ll be charged for the number of index entries matched during the computation. Index entry matches will map to the existing Doc Reads SKU, where up to 1000 index entry matches will be equal to 1 doc read, 1,001 to 2,000 index entry matches will be equal to 2 document reads, etc.For example, a query utilizing a count aggregation function resulting in 1500 index entry matches, will be charged 2 document reads.Count () is also available for free-tier users. Since it is charged using document reads, it adheres to the existing free-tier quotas. COUNT() behavior For web and mobile users, during the Preview, count() will be an online only experience. You can use the count() function on any query when you are online. Support for real time listeners, and offline access to the count() function is not available at the moment. The count() function can be used with all our existing query patterns, including transactions. Next Steps Please refer to the documentation for more information.With the ease and cost-efficiency of this feature, we hope you’ll agree this is the one that really counts. Get counting now.
Quelle: Google Cloud Platform
To all Apache Beam and Dataflow users:If you’ve experimented with Beam, prototyped a pipeline, or verified assumptions about a dataset, you might have used Beam Notebooks or other interactive alternatives such as Google Colab or Jupyter Notebooks.You might also have noticed a gap between running a small prototype pipeline in a notebook and a production pipeline on Dataflow: What if you want to interactively process and inspect aggregations of bigger production datasets from within the notebook, but at scale? You cannot rely on the single machine that’s running your notebook to execute the pipeline because it simply lacks the capacity to do so.Allow me to introduce Interactive FlinkRunner on notebook-managed clusters. It lets you execute pipelines at scale and inspect results interactively with FlinkRunner on notebook-managed clusters. Under the hood, it uses Dataproc with its Flink and Docker components to provision long-lasting clusters.This post will introduce you to Interactive FlinkRunner using three examples:A starter word count example with a small notebook-managed cluster.An example using a much bigger cluster to process tens of millions of flight records to see how many flights are delayed for each airline.An example reusing the bigger cluster to run ML inference against 50,000 images with a pre-trained model – all from within a notebook.If you want to control the cost of these examples, you are free to use pipeline options to reduce the size of the data and the cluster. The starter example costs ~$1/hr and the other two cost ~$20/hr (estimated from Dataproc pricing and VM instance pricing). The actual cost may vary. Optionally, you can reduce the cost by configuring source data and pipeline options. PrerequisitesPrerequisitesOnce you have Beam Notebooks instantiated, create an empty notebook (ipynb) file and open it with a notebook kernel selected. Always use a notebook/IPython kernel with the newer Beam version to take advantage of bug fixes, optimizations and new features. For Dataflow-hosted Beam Notebooks, use notebook kernels with Beam versions >= 2.40.0.To get started, you have to check whether your project has the necessary services activated and permissions granted. You can find relevant information about the current user by executing the following in the notebook.code_block[StructValue([(u’code’, u’# Describe the user currently authenticated.rn!gcloud iam service-accounts describe $(gcloud config get-value account)rnrn# List the IAM roles granted to the user. If it’s already a Project Editor,rn# it should have all required IAM permissions. Otherwise, look for a projectrn# admin for missing grants if you encounter any permission issues in the examples.rn!gcloud projects get-iam-policy $(gcloud config get-value project) \rn –flatten=”bindings[].members” \rn –format=’table(bindings.role)’ \rn –filter=”bindings.members:$(gcloud config get-value account)”‘), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d3e6dd0>)])]Interactive Flink on notebook-managed clusters uses Dataproc under the hood.code_block[StructValue([(u’code’, u’!gcloud services enable dataproc.googleapis.com’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d3e6b10>)])]A starter example – Word CountYou’ve probably already seen the word count example multiple times. You know how to process and inspect the counted words with an InteractiveRunner or a DirectRunner on a single machine.And you are able to run the pipeline on Dataflow as a one-shot job from within the exact same notebook without copying/pasting, moving across workspaces, or setting up the Cloud SDK.To run it interactively with Flink on a notebook-managed cluster, you only need to change the runner and optionally modify some pipeline options.The notebook-managed Flink cluster is configurable through pipeline options. You need these imports for this and the other examples.code_block[StructValue([(u’code’, u’from apache_beam.options.pipeline_options import FlinkRunnerOptionsrnfrom apache_beam.options.pipeline_options import GoogleCloudOptionsrnfrom apache_beam.options.pipeline_options import PipelineOptionsrnfrom apache_beam.options.pipeline_options import PortableOptionsrnfrom apache_beam.options.pipeline_options import SetupOptionsrnfrom apache_beam.options.pipeline_options import WorkerOptionsrnfrom apache_beam.runners.interactive.interactive_runner import InteractiveRunnerrnfrom apache_beam.runners.portability.flink_runner import FlinkRunner’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d3e6590>)])]You can then set up the configurations for development and execution.code_block[StructValue([(u’code’, u”import loggingrnlogging.getLogger().setLevel(logging.ERROR)rnrnimport google.authrnproject = google.auth.default()[1]rnrn# IMPORTANT! Adjust the following to choose a Cloud Storage location.rn# Used to cache source recordings and computed PCollections.rnib.options.cache_root = ‘gs://YOUR-BUCKET/’rnrn# Define an InteractiveRunner that uses the FlinkRunner under the hood.rninteractive_flink_runner = InteractiveRunner(underlying_runner=FlinkRunner())rnrn# Set up the Apache Beam pipeline options.rnoptions = PipelineOptions()rnoptions.view_as(GoogleCloudOptions).project = project”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d21a9d0>)])]Above are the minimum configurations needed; you’ll further customize them in later examples.You can find the source code of the word count example here. Modify it with the interactive_flink_runner to build the pipeline in the notebook. The example uses gs://apache-beam-samples/shakespeare/kinglear.txt as the input file.Inspecting the PCollection counts would implicitly start a Flink cluster, execute the pipeline, and render the result in the notebook.Example 2 – Find out how many flights are delayedThis example reads more than 17 million records from a public BigQuery dataset, bigquery-samples.airline_ontime_data.flights, and counts how many flights have been delayed since 2010 for all the airlines.On a normal InteractiveRunner running directly on a single notebook instance, it could take more than an hour for reading and processing due to the number of records (though the size of data is relatively small, ~ 1GB), and the pipeline can OOM or run out of disk space when the data is even bigger. With interactive Flink on notebook-managed clusters, you work with a higher capacity and performance (~ 4 mins for the example) while still being able to construct the pipeline step by step and inspect the results one by one within a notebook.You need to have BigQuery service activated.code_block[StructValue([(u’code’, u’!gcloud services enable bigquery.googleapis.com’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c7b9a44d0>)])]Configure a much bigger cluster with the options below. You may add a “LIMIT 1000” or similar constraints in the BigQuery read query to limit the records read. Based on the size of data read from BigQuery, you may reduce the values of the options.code_block[StructValue([(u’code’, u”# Use cloudpickle to alleviate the burden of staging things in the main module.rnoptions.view_as(SetupOptions).pickle_library = ‘cloudpickle’rn# As a rule of thumb, the Flink cluster has about vCPU * #TMs = 8 * 40 = 320 slots.rnoptions.view_as(WorkerOptions).machine_type = ‘n1-highmem-8’rnoptions.view_as(WorkerOptions).num_workers = 40″), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c7b9a4b10>)])]Whenever you inspect the result of a PCollection through ib.show() or ib.collect() in a notebook, Beam implicitly runs a fragment of the pipeline to compute the data. You can adjust the parallelism of the execution interactively.code_block[StructValue([(u’code’, u’# The parallelism is applied to each step, so if your pipeline has 10 steps, yourn# end up having 150 * 10 tasks scheduled that can theoretically be executed in parallel byrn# the 320 (upper bound) slots/workers/threads.rnoptions.view_as(FlinkRunnerOptions).parallelism = 150′), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c7bf0fa50>)])]With the above configurations, when you inspect data in the notebook, you are instructing Beam to implicitly start or reuse a Flink cluster on Google Cloud (Dataproc under the hood) with 40 VMs and run pipelines with parallelism set to 150.code_block[StructValue([(u’code’, u’options.view_as(GoogleCloudOptions).temp_location = ib.options.cache_rootrnbq_p = beam.Pipeline(runner=interactive_flink_runner, options=options)rnrndelays_by_airline = (rn bq_prn | ‘Read Dataset from BQ’ >> beam.io.ReadFromBigQuery(rn project=project, use_standard_sql=True,rn # Read 17,692,149 records, ~1GB worth of datarn query=(‘SELECT airline, arrival_delay ‘rn ‘FROM `bigquery-samples.airline_ontime_data.flights` ‘rn ‘WHERE date >= “2010-01-01″‘))rn | ‘Rebalance Data to TM Slots’ >> beam.Reshuffle(num_buckets=1000)rn | ‘Extract Delay Info’ >> beam.Map(rn lambda e: (e[‘airline’], e[‘arrival_delay’] > 0))rn | ‘Filter Delayed’ >> beam.Filter(lambda e: e[1])rn | ‘Count Delayed Flights Per Airline’ >> beam.combiners.Count.PerKey())’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c7bf0fe10>)])]You can include visualize_data=True when inspecting data through ib.show(). Binning the visualized data by their count, you can see that WN airline has the most delayed flights recorded in the dataset.Example 3 – Run ML inference at scale interactivelyThe RunInference example classifies 50,000 image files (~280GB) from within the notebook.The workload normally takes half a day for a single notebook instance or worker. With interactive Flink on notebook-managed clusters, it shows the result in ~1 minute. Looking at the Flink job dashboard, the actual inference only took a dozen seconds. The rest of the running time is overhead from staging the job, scheduling the tasks, writing the aggregated result to ib.options.cache_root, transferring the result back to the notebook, and rendering it in the browser.SetupFor the RunInference example, you need to build a container image. You can find more information about building a container image from a notebook in this guide.The extra Python dependencies needed for this example are:code_block[StructValue([(u’code’, u’%pip install torchrn%pip install torchvisionrn%pip install pillowrn%pip install transformers’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d3e9fd0>)])]The example uses the validation image set from ImageNet and the PyTorch pre-trained ImageNetV2 model. You can download similar dependencies or use your own image dataset and model. Make sure you copy the pre-trained model to the container and use its file path in the Beam pipeline. You can find many image datasets from places such as ImageNet or COCO (Common Objects in Context) and pre-trained models such as MobileNetV2 in the ImageNet Models package.Configure the pipeline options to use the custom container you build.code_block[StructValue([(u’code’, u”options.view_as(PortableOptions).environment_config = f’gcr.io/{project}/flink'”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d3e9ad0>)])]Build the pipelineTo run inference with a Beam pipeline, you need the following imports:code_block[StructValue([(u’code’, u’import iornfrom typing import Iterablernfrom typing import Optionalrnfrom typing import Tuplernrnimport torchrnfrom PIL import Imagernfrom torchvision import modelsrnfrom torchvision import transformsrnfrom torchvision.models.mobilenetv2 import MobileNetV2rnrnimport apache_beam as beamrnfrom apache_beam.io.filesystems import FileSystemsrnfrom apache_beam.ml.inference.base import KeyedModelHandlerrnfrom apache_beam.ml.inference.base import PredictionResultrnfrom apache_beam.ml.inference.base import RunInferencernfrom apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerTensor’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d3e9f50>)])]Then you can define processing logic for each step of the pipeline. You can use a mixture of DoFns and normal functions that yield or return and later incorporate them into the pipeline with different transforms.code_block[StructValue([(u’code’, u”def filter_empty_text(text: str) -> Iterable[str]:rn if len(text.strip()) > 0:rn yield textrnrndef preprocess_image(data: Image.Image) -> torch.Tensor:rn image_size = (224, 224)rn # Pre-trained PyTorch models expect input images normalized with thern # below values (see: https://pytorch.org/vision/stable/models.html)rn normalize = transforms.Normalize(rn mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])rn transform = transforms.Compose([rn transforms.Resize(image_size),rn transforms.ToTensor(),rn normalize,rn ])rn return transform(data)rnrndef read_image(image_file_name: str) -> Tuple[str, torch.Tensor]:rn with FileSystems().open(image_file_name, ‘r’) as file:rn data = Image.open(io.BytesIO(file.read())).convert(‘RGB’)rn return image_file_name, preprocess_image(data)rnrnclass PostProcessor(beam.DoFn):rn def process(self, element: Tuple[str, PredictionResult]) -> Iterable[str]:rn filename, prediction_result = elementrn prediction = torch.argmax(prediction_result.inference, dim=0)rn yield str(prediction.item())”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d3e9d10>)])]Now define a few variables.code_block[StructValue([(u’code’, u”# Replace this with a file containing paths to your image files.rnimage_file_names = ‘gs://runinference/it_mobilenetv2_imagenet_validation_inputs.txt’rnmodel_state_dict_path = ‘/tmp/mobilenet_v2.pt’rnmodel_class = MobileNetV2rnmodel_params = {‘num_classes': 1000}rnrn# In this example we pass keyed inputs to the RunInference transform.rn# Therefore, we use KeyedModelHandler wrapper over PytorchModelHandler.rnmodel_handler = KeyedModelHandler(rn PytorchModelHandlerTensor(rn state_dict_path=model_state_dict_path,rn model_class=model_class,rn model_params=model_params))”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8d6a26d0>)])]And build the pipeline with the above building blocks.code_block[StructValue([(u’code’, u”pipeline = beam.Pipeline(interactive_flink_runner, options=options)rnrncounts = (rn pipelinern | ‘Read Image File Names’ >> beam.io.ReadFromText(rn image_file_names)rn | ‘Filter Empty File Names’ >> beam.ParDo(filter_empty_text)rn | ‘Shuffle Files to Read’ >> beam.Reshuffle(num_buckets=900)rn | ‘Read Image Data’ >> beam.Map(read_image)rn | ‘PyTorch Run Inference’ >> RunInference(model_handler)rn | ‘Process Output’ >> beam.ParDo(PostProcessor())rn | ‘Count Per Classification’ >> beam.combiners.Count.PerElement())rnrn# Further increase the parallelism from the starter example.rnoptions.view_as(FlinkRunnerOptions).parallelism = 300″), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8c893610>)])]The pipeline reads a text file with 50,000 image file names in it. The Reshuffle is necessary to rebalance the image file names to all the workers before reading the image files. Without it, all 50,000 files will be read from a single task/thread/worker no matter how high the parallelism is.Once read, each image will be classified into 1 of 1000 classes (e.g., a cat, a dog, a flower). The final aggregation counts how many images there are for each class.In notebooks, Beam tries to cache the computed data of each PCollection that is assigned to a variable defined in the main module or watched by ib.watch({‘pcoll_name’: pcoll}). Here, to speed everything up, you only assign the final aggregation to a PCollection variable named counts as it’s the only data worth inspection.To inspect the data, you can use either ib.show or ib.collect. If it’s the first time you inspect the data, a Flink cluster is implicitly started. For later inspections, computed PCollections do not incur executions. For inspections of data by newly appended transforms, the same cluster will be reused (unless instructed otherwise).You can also inspect the cluster by running ib.clusters.describe(pipeline).And you can follow the link in the output to the Flink dashboard where you can review finished jobs or future running jobs.As you can see, the process took 1m45s to run inference for 50,000 images (~280GB).You can further enrich the data if you know the mappings between classifications and their human-readable labels.code_block[StructValue([(u’code’, u”idx_to_label = pipeline | ‘A sample class idx to label’ >> beam.Create(list({rn ‘242’: ‘boxer’,rn ‘243’: ‘bull mastiff’,rn ‘244’: ‘Tibetan mastiff’,rn ‘245’: ‘French bulldog’,rn ‘246’: ‘Great Dane’,rn ‘247’: ‘Saint Bernard, St Bernard’,rn ‘248’: ‘Eskimo dog, husky’,rn ‘249’: ‘malamute, malemute, Alaskan malamute’,rn ‘250’: ‘Siberian husky’,rn ‘251’: ‘dalmatian, coach dog, carriage dog’,rn ‘252’: ‘affenpinscher, monkey pinscher, monkey dog’,rn ‘253’: ‘basenji’,rn ‘254’: ‘pug, pug-dog’,rn}.items()))rnrndef cross_join(idx_count, idx_labels):rn idx, count = idx_countrn if idx in idx_labels:rn return {‘class': idx, ‘label': idx_labels[idx], ‘count': count}rnrnlabel_counts = (rn countsrn | ‘Enrich with human-readable labels’ >> beam.Map(rn cross_join, idx_labels=beam.pvalue.AsDict(idx_to_label))rn | ‘Keep only enriched data’ >> beam.Filter(lambda x: x is not None))”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c7b45d4d0>)])]When inspecting the label_counts, the computed counts will be reused for the newly added transforms. After an aggregation, the output data size can be tiny compared with the input data. High parallelism does not help with processing small data and could introduce unnecessary overhead. You can interactively tune down the parallelism to inspect the result of processing only a handful of elements with the newly added transform.Clean UpExecute the code below to clean up clusters created by the notebook and avoid unintended charges.code_block[StructValue([(u’code’, u’ib.clusters.cleanup(force=True)’), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8c893110>)])]Optionally, you can go to the Dataproc UI to manually manage your clusters.Open Source SupportApache Beam is open source software. The interactive features work with all IPython kernel-backed notebook runtimes. This also means the interactive FlinkRunner feature can be adapted to your own notebook and cluster setups.For example, you can use Google Colab (a free alternative to Dataflow-hosted Beam Notebooks) connected with a local runtime (kernel) on your own workstation and then interactively submit jobs to a Flink cluster that you host and manage.Set up Google Colab with local runtimeSet up a Flink cluster locallyTo use your own Flink cluster, simply specify the necessary options:code_block[StructValue([(u’code’, u”flink_options = options.view_as(FlinkRunnerOptions)rnflink_options.flink_master = ‘localhost:8081′ # Or any resolvable URL of your clusterrnflink_options.flink_version = ‘1.12’ # Or the version of Flink you use”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8c893390>)])]If you use Beam built from source code (a dev version), you can configure a compatible container image.code_block[StructValue([(u’code’, u”# Or any custom container you build to run the Python code you define.rnoptions.view_as(PortableOptions).environment_config = ‘apache/beam_python3.8_sdk:2.41.0′”), (u’language’, u”), (u’caption’, <wagtail.wagtailcore.rich_text.RichText object at 0x3e7c8c893a10>)])]Now you can run Beam pipelines interactively at scale on your own setup.CompatibilitiesInteractive Flink features are not patched back to older versions of (Interactive) Beam. Here is a compatibility table.Beam VersionsDataflow-hosted Beam NotebooksOther notebook and cluster setups<2.40.0Not supportedNot supported>=2.40.0,<2.43.0SupportedParallelism fixed to 1>=2.43.0SupportedSupportedThere is also a cluster manager UI widget in the JupyterLab extension apache-beam-jupyterlab-sidepanel. Dataflow-hosted Beam Notebooks have it pre-installed. If you use your own JupyterLab setup, you can install it from either NPM or source code. It’s not supported in other notebook runtime environments such as Colab or classic Jupyter Notebooks.Next StepsGo to the Vertex AI workbench and get started using Dataflow-hosted Beam Notebooks! You can create, share, and collaborate on your notebooks with ease. And you have the flexibility to control who can access your notebook and what resources to use any time you want to make a change.For the interactive Flink feature, check the public documentation for tips, caveats and FAQs when you run into issues.Your feedback, suggestions, and open source contributions are welcomed.
Quelle: Google Cloud Platform
We understand that for startups in the build phase, the highest priority task is to continuously ship features based on your users’ needs. There are three main focus areas when building applications: Development: When it comes to development, focus on tasks that make your app unique by offloading backend setup and processing to someone else. For example, instead of setting up your own API servers and managing backend services, Firebase offers a managed experience. Hosting: Once you’ve built your app, the next step is to host it. Containers have become the de facto way of packaging applications today. You can easily run your containers in managed environments, such as Google Kubernetes Engine or Cloud Run. Improvements: A one-time deployment is not enough. Growth is about taking in feedback from the market and improving our applications based on the same. We recommend incorporating CI/CD and automating improvements in your software delivery pipelines.In this blog post, you can learn more about the tools that help you with the above three focus areas.Develop apps faster by shifting focus to business logic with FirebaseIn a traditional app architecture, you would need to set up and manage an API server to direct requests to your backend. With Firebase, you can easily add features in your mobile or web app with a few lines of code, without worrying about the infrastructure. The products on Firebase help you Build, Release & Monitor, and Engage. Doing so will allow your teams to:Add features like authentication and databases with only a few lines of codeUnderstand your users and apps better using Google Analytics for Firebase, Crashlytics, and Performance MonitoringSend messages to engage your users with Firebase Cloud Messaging and In-App MessagingWith simple-to-use cross-platform SDKs, Firebase can help you develop applications quicker and reduce your time to market, improve app quality in less time with less effort, and optimize your app experience. Find out how you can put together these building blocks in our video on Working with Firebase.Host apps easily with managed container platforms on Google CloudFor startups who are looking to utilize resources better, containerization becomes the next step. With our investment in Google Kubernetes Engine (GKE) and Cloud Run, Google Cloud gives you the freedom to build with containers on a tech stack based on open source tools like Kubernetes, Knative and Istio. This means no vendor lock-in for you.Google Kubernetes EngineWe understand that our customers are looking for autonomous and extensible platforms that are expertly run. GKE gives you a managed environment to run applications, simplified consoles to create/update your clusters with a single click, and lets you deploy applications with minimal operational overhead.Google manages your control plane, and 4-way autoscaling gives you the option to fine tune to get the most optimized utilization for the resources used.These best practices are applied by default with the second mode of operation for GKE – Autopilot. It dynamically adjusts compute resources so you don’t have to worry about unused capacity and you pay only for the pods you use , billed per second for vCPU, memory and disk resource requests. This means that you can reduce operational costs, while still optimizing for production and higher workload availability.Head to Compute with Google Kubernetes Engine to quickly get started with GKE.Cloud RunCloud Run lets you run containers in a fully managed serverless environment and gives you the ability to scale down to ‘zero’ when there are no requests coming in. It is a great fit for stateless applications like web frontends, REST APIs, lightweight data transformation jobs, etc. There are 3 steps to any Cloud Run deployment –Create a build using your source code. Submit the build to store it in a container registry.Deploy the application using a simple command. This process is very similar to the usual steps followed for deployments on other platforms but what makes Cloud Run special is that all of this can be achieved in one single command – `gcloud run deploy –source . `Watch this in action in the video to Get started on Cloud RunImprove and iterate more often with CI/CD solutionsSoftware systems are living things and need to adapt to reflect your changing priorities. Continuous integration/Continuous deployments (CI/CD) as the term suggests, means that you are adding code updates and deploying them continuously. Our developer’s time should be spent writing code, so CI/CD steps should be triggered and run in the background when code is pushed. Let’s look at the components of a CI/CD pipeline and how Google Cloud tools support them – Cloud Code integrates with your IDE and lets you easily write, run and debug your applications.Cloud Build lets you run your build steps to package and deploy your applications on any platform on Google Cloud. You can set up triggers to start builds automatically. Artifact Registry is where we store the intermediate artifacts created during a build. Container images stored here can be used to create newer deployments to other platforms as well.Cloud Deploy automates the delivery of your updated application to target environments specified by you. Both Cloud Run and GKE come integrated with Cloud Operations Suite so you can monitor your application for any errors or performance issues. We know that you want to deliver bug-free features to your customers. So when you are shipping code, consider how a CI/CD pipeline can help you catch performance issues early and improve developer workflows. To set up your CI/CD pipeline on Google Cloud, refer to CI/CD on Google Cloud.Stay in touch for moreThe Google Cloud Technical Guides for Startups series has many more detailed videos and resources to support you on all steps of your growth journey. Check out our full playlist on the Google Cloud Tech channel and handbooks and sample architectures on our website. Don’t forget to subscribe to stay up to date. If you’re ready to get started with Google Cloud, apply now for the Google for Startups Cloud Program.See you in the cloud.Related ArticleBootstrap your startup with the Google Cloud Technical Guides for Startups : A Look into the Start SeriesAnnouncing the summary of the first phase of the Google Cloud Technical Guides for Startups, a video series for technical enablement aime…Read Article
Quelle: Google Cloud Platform
Flexera’sState of the Cloud Report 2022 pointed out that significant cloud spending is wasted, a major issue that is getting more critical as cloud costs continue to rise. In the current macroeconomic conditions, companies focus on identifying ways to reduce spending. To effectively do that, we need to understand the pricing model. We can then work towards the challenges of cost monitoring, optimization, and forecasting. One area that often gets overlooked in budgeting is observability—logging, monitoring, tracing. This can represent a significant cost, especially if it’s not optimized. Let’s explore how to understand and optimize our most voluminous data source—logs—within Google Cloud.Cloud Logging is a fully managed real-time log solution that allows you to ingest, route, store, search and analyze your logs to easily troubleshoot incidents using your log data. It can collect data from on-prem, Google Cloud and other clouds with open source agents that support over 150+ services. Unlike traditional licensing models or self hosted logging solutions, Cloud Logging pricing model is simple and based on actual usage. Let’s explore the various components of Cloud Logging and address a few commonly asked questions about pricing. Cloud Logging – Components & PurposeTo understand pricing better and be able to predict future costs, we need to understand the high-level components of Cloud Logging and where billing occurs in our system. There are three important components within Cloud Logging: Cloud Logging API, Cloud Logging Router (Log Router) and log buckets (Log Storage).The below table outlines the high-level components, purpose and pricing information for Cloud Logging. As indicated above, today billing in Cloud Logging occurs only for a log that is routed and ingested into a log bucket. “Ingestion” in Cloud Logging is the process of saving log data into a log bucket, not simply processing it in the Log Router. There are three options for log buckets – RequiredDefault User-defined or Custom. Only Default and User-defined buckets are billed.Today, ourlogging pricing is based on the volume of logs ingested in a chargeable log bucket—default or user-defined. All charges in Cloud Logging occur at the log bucket and all log types incur the same cost. Logs dropped using sink filters or exclusion filters are not charged by Cloud Logging, even if these logs are routed to a destination outside of Cloud Logging. Now, we’ll address frequently asked questions about the Cloud Logging pricing model. What Cloud Logging charges will I see on my bill?There are two types of charges your logs can potentially incur: An ingestion charge of 0.50 cents/GB which includes default storage of 30 days. Note that the first 50 GB in a project fall under the free tier quota. You get charged based on the volume of logs ingested into the Default and User-defined log buckets.Logs stored beyond 30 days will incur a retention charge of $0.01/GiB/month for non-required buckets. Note that this pricing is not currently enforced. We will begin charging in early 2023.For the latest pricing, check here.How can I reduce my bill?Because Cloud Logging pricing is based on actual usage, you can reduce your pricing by adjusting the ingestion volume or retention period.Reduce the volume of logs ingested per log bucket by identifying and keeping (ingesting) only valuable log data for analysis. If you do not need to keep data beyond the included 30 days, reduce the retention period. Because the first 30 days of retention are included with ingestion, reducing retention to less than 30 days will have no impact on your bill.Does Cloud Logging charge based on the number of queries, searches either from Cloud Logging UI or Client SDK/APIs?No, Cloud Logging does not charge for the number of queries, searches, logs read from disks during queries, or varied log types. There is a quota limit for querying logs, though, so for integrations with SIEMs or other logging tools, it’s a best practice to set up a log sink via Pub/Sub to push the logs to the downstream system.Can I incur multiple ingestion charges?It is possible to be charged for ingesting the same log entry into Cloud Logging log buckets multiple times. For example, if your sinks route a log entry to two log buckets, you will pay ingestion costs at two buckets. You may choose to do this to have independent retention of logs or to keep copies of logs in multiple regions for compliance reasons. Are there different costs for hot and cold storage?No, there are no differences between hot and cold storage. The beauty of Cloud Logging is that all logs are accessible throughout their lifespan. Cloud Logging is designed to scale easily and efficiently, which makes logs accessible for troubleshooting, investigating and compliance reasons whether they are seconds or years old. How much does it cost to route logs to other destinations?Today, Cloud Logging does not charge for centrally collecting and routing logs to other destinations like Cloud Storage, BigQuery, Pub/Sub. Usage rates for the destination services, Cloud Storage, BigQuery and Pub/Sub apply.Do Logs have a generation fee?For network telemetry logs such as VPC Logs, Firewall rules logs and Cloud NAT logs, you might incur an additional network generation charge if logs are not stored in Cloud Logging. If you store your logs in Cloud Logging, networking logs generation charges are waived, and only Cloud Logging charges apply. How do I understand my ingestion volume in Cloud Billing?To determine the cost per Project:Go to Cloud Console -> Billing -> Select the Billing Account -> Reports (left pane) On the right side, under filters -> Services -> select “Cloud Logging”Now, Let’s drill down to learn about the cost incurred by each log bucket. Select the Project on the top bar. On the Left pane, go to Logging -> Logs Storage. Now you should be able to see the log volume per bucket.Putting it all togetherNow that we understand pricing for Cloud Logging, we can optimize our usage. Here are four best practices:Recommendation #1: Use a log router to centralize your collection; get a 360 view of your log world and then use an exclusion filter to reduce noisy logs and send only valuable logs to the log bucket. Logs dropped using sink filters or exclusion filters are not charged by Cloud Logging, even if these logs are routed to a destination outside of Cloud Logging. Recommendation #2: Admin activity audit logs are captured by default for all GCP services for no additional cost. Leverage the audit logs from Required Bucket by identifying use-cases for your organizations and configure log-based-alerts on them. Recommendation #3: Logs can be stored cost effectively for up to 10 years and easily accessed via Cloud Logging. Cloud Logging will begin charging customers for long term log retention starting Jan 2023. Between now and Jan 2023, determine the required lifespan of a log and set the appropriate retention period for each log bucket.Recommendation #4: If you are a new customer, estimate your bills. This is a great way to compare costs with your current Cloud Logging solution. If you are an existing customer, create a budgetand set up alerts on your Cloud Logging bills. In addition to analyzing log volumes by buckets, customers may want to analyze the sources, projects, etc. Metrics explorer in Cloud Monitoring can also be used to identify costs. We will discuss this in the next blog. For more information, join us in ourdiscussion forum. As always, we welcome your feedback. Interested in using Cloud Logging to save costs in your organization, contact us here. We are hosting a webinar to talk about how you can leverage Log Analytics, powered by BigQuery in Cloud Logging for no additional cost. Register here.
Quelle: Google Cloud Platform
Editor’s note: KP Philpot is the Environmental Health & Safety Manager at Alphabet’s data center campus in Douglas County, Georgia. It’s a long way from both a childhood in Chicago’s South Side, and standing in football stadiums with thousands of fans, but one thing has always held true for him: The importance of personal and team performance. How did you come to Google?At surface level, it was pretty direct. I was working as a site safety engineer for a contractor that was building a Google data center, and I was offered a job at Google. On a deeper level, it was a long and unexpected journey. I grew up in inner city Chicago, and we didn’t hear a lot about data center technicians and environmental engineering. We had blue collar jobs you stuck to, or you played sports. I played football and basketball, and was recruited by colleges for both. I set three NCAA records playing linebacker at Eastern Michigan University, and then I was with the Detroit Lions and played some Arena Football. A few years after that someone I played with in college brought me into the construction industry, and that’s what I did at three other companies before arriving at Google. How different is Google from other places?One thing that’s a breath of fresh air is that when you come to Google, it’s okay to not have all the answers. I think you work more freely and more confidently when there’s no expectation to know everything from day one. If someone you ask doesn’t know the answer, they’re interested in finding it out. There’s a healthy curiosity that you don’t find in most places. One other difference is that Google tends to be team oriented. That part comes naturally to me, even if it is tech. I’ve played on teams since I was a kid, and both my parents were athletes. On a team, everyone has a part to play. You have different people, with different skill sets, but everyone belongs. Their contributions are different, but the goal is the same.What is a typical day like?Many people see data centers as rooms full of servers and switches, but I assure you no two days are alike. There are many things to think about in terms of safety, since a data center has a lot of moving parts, especially when working with electricity, we have rigorous protocols to ensure safety for everyone on the site. We also take our environmental impact seriously. A big part of our environmental work is the innovative cooling system we have here in Douglas County — we recycle local sewer water that would otherwise be put in the Chattahoochee River. As for leftover water that does not evaporate, we treat it before returning it to the river. More than that, though, it’s the diversity of people you find in a data center. There may be construction people, who tend to have a lot of hands-on experience and are task focused; there are engineers and managers, who are more focused on how to optimize a process;and of course, there are Googlers. We all become interesting to each other. I get to coordinate and work alongside all of them, which I enjoy a lot.So is team building part of the job?Teamwork is the lens through which I see the world. I was raised by very principled people, who taught me how much your individual actions impact everyone. A family is a team as well. My grandfather would point at his first name, and say, “That’s my name,” then point at my last name, and say, “that’s our name. Every time you walk out the door, that’s who you are.” When I work, I see the world the same way, the need to be a principled person who’s part of a larger team, and constantly working to build respect and trust. Being in the NFL was more expected than being at Google, but these things don’t change.Related ArticleSales specialist, mentor, and woman in Web3: Anella Bokhari is building community and helping others tell their story along the waySales Specialist, Mentor, and Woman in Web3: Anella Bokhari Wears Many Hats But Has the Same “Why” – Helping Others Find & Tell Their Sto…Read Article
Quelle: Google Cloud Platform
There is a critical need for increased visibility and control over the operational state of complex networks running sophisticated workloads. Multi-cloud and hybrid network environments power new demands of remote work, 5G/Edge connectivity, microservices based workloads, and increased cloud adoption. The advent of the cloud has added agility, cost benefits, and brought along the need for management of the infrastructure. Management and monitoring of the network underlying these complex applications plays a key role in ensuring end-user satisfaction.
Azure Network Watcher provides an entire suite of tools to visualize, monitor, diagnose, and troubleshoot network issues across Azure and Hybrid cloud environments. Network Watcher enables customers to detect anomalies across Azure and hybrid networks with comprehensive wide coverage, through a guided and intuitive drilled-down experience. Network Watcher helps customers monitor, manage, and understand their own networks for performance, connectivity, security, and compliance issues and furthermore, empowers customers to troubleshoot efficiently with actionable insights and proactive alerting, thus effectively reducing the mean time to resolve network issues.
The following new feature enhancements across Network Watcher suite aim to provide timely and complete visibility and actionable insights to customers of their hybrid networks in a manner that is easily accessible, readily usable, and reliable.
Visualize resource and network health with Topology
Topology enables users to quickly acquire system context, comprehend state, and troubleshoot issues efficiently by visualizing the resources in a network. It offers a visually connected experience for monitoring and managing inventory.
This new topology experience in Azure, which replaces the Network Watcher topology, will enable customers to create a consistent and dynamic topology across multiple subscriptions, regions, and resource groups (RGs)—comprising of numerous resources.
Allowing deep dives into the customer’s environment, Topology lets users drill down from regions, VNETs to subnets, and resource view diagram of resources supported in Azure.
Stitching the end-to-end monitoring and diagnostics story for all Network Monitoring needs, topology offers the capability to run Next Hop directly from a VM selected in the topology.
Significant features available with this preview:
Multi-region and multi-subscription–dynamic drill-down visualization.
Health status of resources using resource health (RHC) status.
Diagnostics tool Next Hop integration.
Resource view diagram for all supported resources.
Monitor connectivity using Azure Monitor Agent with Connection Monitor
Integration of Azure Monitor Agent’s support consolidates multi-monitoring agents into a single connectivity monitoring agent in Azure Network Watcher’s Connection Monitor.
Connection Monitor, a multi-agent solution, monitors connectivity at regular intervals across Azure and Hybrid endpoints and provides aggregated data for packet loss, latency, and status codes over TCP, ICMP, and HTTP(s) pings.
Connection Monitor helps you troubleshoot network issues with faster alerts for lack of connectivity or reachability to the endpoints. The unified topology rendered provides a complete end-to-end visualization of the network path from source to destination, with actionable insights.
This agent integration enhancement addresses connectivity monitoring logs and metrics data collection needs across Azure and ARC-enabled on-premises machines, thus eliminating the overhead of management and enablement of multiple monitoring agents. Additionally, Azure Monitor Agent provides enhanced security and performance capabilities, effective cost savings, and ease of troubleshooting with simpler management of data collection. With this support, dependency on the soon-to-be-retired Log Analytics agent is eliminated, while increasing the coverage for on-premises machines with support for ARC-enabled endpoints.
Significant features available with preview:
Connectivity monitoring support for ARC-enabled on-premises endpoints.
Simpler management of monitoring extension.
One agent for monitoring Azure and non-Azure endpoints.
Enhanced security through Managed Identity and Azure Active Directory (Azure AD) tokens.
Learn More
Please navigate to the Network Insights portal to try out Azure Topology.
Refer here to learn more about Network Insights.
Please navigate to the Connection Monitor portal to try out Connection Monitor with Azure Monitor Agent.
Refer here to learn more about Connection Monitor.
Quelle: Azure
This blog has been co-authored by Anupam Vij, Principal PDM Manager.
Distributed denial of service (DDoS) attacks are some of the largest availability and security concerns facing customers that are moving their applications to the cloud. While cyber-attacks are on the rise, they typically make the news only when a large organization has fallen victim to an attack. However, contrary to what many may think, small and medium businesses (SMBs) are just as enticing to cybercriminals.1 While large organizations have the resources needed to protect themselves, small businesses often lack the budget and qualified staff to defend against DDoS attacks.
At Microsoft, we continuously enhance our product offerings to meet the needs of all organizations, including helping SMBs on their digital transformation journey by ensuring that they are protected against the latest DDoS attack vectors. As we shared at Microsoft Ignite, Azure DDoS IP Protection SKU, a new SKU of Azure DDoS Protection built for SMBs, is now available in preview.
Cost-effective, enterprise-grade DDoS protection for small businesses
DDoS IP Protection is designed to meet the needs of SMBs, providing enterprise-grade DDoS protection at an affordable price point. It offers the same essential capabilities as Azure DDoS Network Protection (previously known as Azure DDoS Protection Standard) to protect your resources and applications against evolving DDoS attacks, including L3/L4 automatic attack detection and mitigation, metrics and alerts, mitigation flow logs, mitigation policies tuned to customer applications, and tight integration with Azure Firewall Manager, Microsoft Sentinel, and Microsoft Defender for Cloud.
With the DDoS IP Protection SKU, customers now have the flexibility to enable DDoS protection on individual public IP addresses. SMB customers who have a few public IP addresses to protect will benefit from this cost-effective DDoS protection option.
Key features of Azure DDoS IP Protection
Massive mitigation capacity and scale: Defend your workloads against the largest and most sophisticated attacks with cloud-scale DDoS protection backed by Azure’s global network.
Adaptive tuning: Protect your apps and resources while minimizing false negatives with adaptive tuning tuned to the scale and actual traffic patterns of your application.
Attack analytics, metrics, and logging: Monitor DDoS attacks near real-time and respond quickly to attacks with visibility into the attack lifecycle, vectors, and mitigation.
Integration with Azure Firewall Manager: Centrally manage your DDoS protection across your environment alongside other network security services.
Integration with Microsoft Sentinel and Microsoft Defender for Cloud: Strengthen your security posture with rich attack analytics and telemetry integrated with Microsoft Sentinel and security alerts and recommendations provided by Microsoft Defender for Cloud.
Choosing the right DDoS protection SKU for your needs
Azure DDoS protection now offers two SKUs:
DDoS IP Protection is recommended for SMB customers with a few public IP resources who need a comprehensive DDoS protection solution that is fully managed, and easy to deploy and monitor.
DDoS Network Protection (previously known as Azure DDoS Protection Standard) is recommended for larger enterprises and organizations looking to protect their entire deployment that spans multiple virtual networks and includes many public IP addresses. It also offers value-added features like cost protection, DDoS Rapid Response, and discounts on Azure Web Application Firewall.
Let’s see a detailed comparison of these two SKUs:
Azure DDoS IP Protection pricing
With DDoS IP Protection SKU, you only pay for the public IP resources protected. The monthly cost is fixed for each public IP resource protected with no additional variable costs. Prices may vary by region. Billing for IP Protection will be effective starting on February 1, 2023. For more details on pricing, visit the Azure DDoS Protection pricing page.
Get Started
DDoS IP Protection is currently available in preview in select regions and can only be enabled on Public IP Standard SKU. DDoS IP Protection is currently only available in the Azure Preview Portal and will be made available on the Azure Portal soon.
For more information on DDoS IP Protection, see the following:
Azure DDoS IP Protection documentation.
DDoS Protection pricing page.
1Diving back into SMB breaches, Data Breach Investigation Report, 2021, Verizon.com.
Quelle: Azure
You may have seen some hype around WebAssembly, or Wasm, as it’s often called. It’s a relatively new technology that allows you to compile application code written in languages like Rust, C, C++, Javascript, and Golang to byte code, then run it inside a sandboxed environment.
So why all the hype? Well, those sandboxed environments can run in a large variety of locations — including your web browser using a Javascript virtual machine. Not only does this mean the sandbox benefits from billions of dollars of investment in security, speed, and cross compatibility, it also means you can run existing code in your browser with some minor changes. And before you ask, yes, it can run Doom.
But running Doom in the browser is just one use case. Companies like WasmEdge are using Tensorflow to push the boundaries of what can be run with Wasm. Fermyon is building tools for Wasm to be used in microservices, while Vercel, Fastly, Shopify, and Cloudflare use Wasm for running code at the edge. Figma is using Wasm to provide higher performance in the browser for their application, and their new parent company Adobe is bringing their desktop applications to the Web using Wasm.
If all those examples don’t excite you about what’s possible with Wasm, I’m not sure what will!
How do Docker and Wasm fit together?
So what is Docker doing with Wasm? We see Wasm and containers as complementary technologies. The problem you’re solving will make one or the other more applicable, but they’re compatible, and should work well together in your cloud native application.
It really comes down to the use case. For example, Wasm’s quick startup time is great for short lived operations, and its isolation is a good match when you need strict security guarantees enforceable at the code level. But as of now, it doesn’t have multithreading or garbage collection capabilities, so any use case with those requirements isn’t a good fit. It also requires that you rebuild your software from source to work.
Join Docker at Cloud Native Wasm Day
We’ll be at the Cloud Native Wasm Day NA in Detroit on October 24, as a Diamond sponsor, to talk about how we’re providing developers the tooling they need using development experiences they already know and love.
Justin Cormack, our CTO, will be presenting during the keynote. In his presentation, he’ll talk about how the container, Docker, and cloud native communities are embracing Wasm — and give some insights as to where we can go from here. If you’ve never seen him speak, I highly recommend it!
Michael Yuan (WasmEdge) and I will also be giving a talk to show how WASI and container workloads work together in Docker Desktop. You’ll find out when to use Wasm, the current tooling options for Wasm, and how to use Docker and Wasm together. We’ll even share download links to the Docker + Wasm preview so you can give it a try yourself!
If you’re attending the Cloud Native Wasm Day don’t miss our keynote and talk!
Quelle: https://blog.docker.com/feed/
