Dataflow

Deploy and manage machine learning (ML) pipelines with ease. Use ML models to do local and remote inference with batch and streaming pipelines. Use data processing tools to prepare your data for model training and to process the results of the models.

Data processing system optimized for performance and cost of your GPU usage. Support for a wide range of NVIDIA GPUs.

Dynamically adjusts the compute capacity allocated to each worker based on utilization. Vertical autoscaling works hand in hand with horizontal autoscaling to seamlessly scale workers to best fit the needs of the pipeline.

Horizontal autoscaling lets the Dataflow service automatically choose the appropriate number of worker instances required to run your job. The Dataflow service may also dynamically reallocate more workers or fewer workers during runtime to account for the characteristics of your job.

Right fitting creates stage-specific pools of resources that are optimized for each stage to reduce resource wastage.

A suite of features including 1) SLO-based data pipeline management, 2) Job visualization capabilities that provide users a visual way to inspect their job graph and identify bottlenecks, 3) Automatic recommendations to identify and tune performance and availability problems. 

Streaming Engine separates compute from state storage and moves parts of pipeline execution out of the worker VMs and into the Dataflow service back end, significantly improving autoscaling and data latency.

Service-based Dataflow Shuffle moves the shuffle operation, used for grouping and joining data, out of the worker VMs and into the Dataflow service back end for batch pipelines. Batch pipelines scale seamlessly, without any tuning required, into hundreds of terabytes.

Dataflow SQL lets you use your SQL skills to develop streaming Dataflow pipelines right from the BigQuery web UI. You can join streaming data from Pub/Sub with files in Cloud Storage or tables in BigQuery, write results into BigQuery, and build real-time dashboards using Google Sheets or other BI tools.

Dataflow FlexRS reduces batch processing costs by using advanced scheduling techniques, the Dataflow Shuffle service, and a combination of preemptible virtual machine (VM) instances and regular VMs. 

Dataflow templates allow you to easily share your pipelines with team members and across your organization or take advantage of many Google-provided templates to implement simple but useful data processing tasks. This includes Change Data Capture templates for streaming analytics use cases. With Flex Templates, you can create a template out of any Dataflow pipeline.

Iteratively build pipelines from the ground up with Vertex AI Notebooks and deploy with the Dataflow runner. Author Apache Beam pipelines step by step by inspecting pipeline graphs in a read-eval-print-loop (REPL) workflow. Available through Google’s Vertex AI, Notebooks allows you to write pipelines in an intuitive environment with the latest data science and machine learning frameworks.

Synchronize or replicate data reliably and with minimal latency across heterogeneous data sources to power streaming analytics. Extensible Dataflow templates integrate with Datastream to replicate data from Cloud Storage into BigQuery, PostgreSQL, or Spanner. Apache Beam’s Debezium connector gives an open source option to ingest data changes from MySQL, PostgreSQL, SQL Server, and Db2.

Dataflow inline monitoring lets you directly access job metrics to help with troubleshooting batch and streaming pipelines. You can access monitoring charts at both the step and worker level visibility and set alerts for conditions such as stale data and high system latency.

You can create a batch or streaming pipeline that is protected with a customer-managed encryption key (CMEK) or access CMEK-protected data in sources and sinks.

Dataflow’s integration with VPC Service Controls provides additional security for your data processing environment by improving your ability to mitigate the risk of data exfiltration.

Turning off public IPs allows you to better secure your data processing infrastructure. By not using public IP addresses for your Dataflow workers, you also lower the number of public IP addresses you consume against your Google Cloud project quota.