The holy grail of data platforms and why I rejoined Databricks I have recently left Google to rejoin Databricks. I first joined Databricks in 2014 when the company was about 20 people in Berkeley and left a little more than a year later. I landed at Google and had an…

How to Simplify data lake architectures Data lakes surrounded by specialized data marts, connected by complex workflows supported by specialized and cross-functional teams of engineers are challenging for several reasons discussed in the previous post. Ideally, a single system could provide the flexibility of a data lake for data producers while in parallel serving data consumers…

The data ecosystem has come a long way since the days of using databases and expensive traditional data warehouses to make sense of large amounts of data. Organizations commonly store and analyze data using open-source tools hosted on-prem or in the cloud on platforms capable of processing more data than…

In my previous blog, I demonstrated how to leverage BigQuery’s AEAD encryption functions to achieve data deletion, also referred to as crypto-deletion. However, I limited this demonstration to data in BigQuery. But data rarely exists only in one system. What if we have to delete data from all of the…

The concept of incremental processing can have a major impact on the design of data analytics pipelines. Processing large amounts of data in increments introduces resource efficiencies, faster processing time and inherently reduced processing cost. However, not all analytical functions have incremental properties. A common example is the count distinct…

There are a number of features that set BigQuery apart from other data warehouses — large-scale streaming ingest, automatic data archival without performance penalties, and integrated machine learning functions are just a few. …

There’s a lot going on at Google. Our customers solve challenging problems using Google Cloud’s services and solutions. What has been obvious to me is that the challenges that our customers solve are not always unique. There are common repeatable patterns that can be applied across different customers. For that reason I’ve decided to blog about some of the common data & analytics patterns and solutions that we solve on behalf of our customers. If there are challenges that you like to solve I would love to hear from you. Find me on twitter: pdeyhim@

Not long ago, I worked on a project where the client was looking to reduce their AWS cost by optimizing their AWS cloud environment. After further investigation, I concluded that applying AutoScaling to their AWS infrastructure would have substantially reduce customer’s monthly AWS cost. And the reason for that was…

After spending the past 6 years helping clients build cloud data processing platforms, I’ve decided to share my thoughts and vision. I’m by no means a thought leader in this space. Far from it actually. But I’ve worked with interesting clients and have been invovled in challenging projects where what I’ve learned and experienced could provide different point of view for others who are walking down the same path.