The concept of incremental processing can have a major impact on the design of data analytics pipelines. Processing large amounts of data in increments introduces resource efficiencies, faster processing time and inherently reduced processing cost. However, not all analytical functions have incremental properties. A common example is the count distinct

There’s a lot going on at Google. Our customers solve challenging problems using Google Cloud’s services and solutions. What has been obvious to me is that the challenges that our customers solve are not always unique. There are common repeatable patterns that can be applied across different customers. For that reason I’ve decided to blog about some of the common data & analytics patterns and solutions that we solve on behalf of our customers. If there are challenges that you like to solve I would love to hear from you. Find me on twitter: pdeyhim@

Parviz Deyhim

Data lover and cloud architect. ex-aws, ex-databricks, and now a Googler

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store