Build Private Pool CloudBuild for Accessing A Resource in Different Project GCP

I will share something new in my experience as a data engineer. So, let’s start.

I have a tableau service using a private network inside the organization. Someday, my stakeholder need the data from the tableau dashboard and need to send it daily to our channel communication. Of course, what thought first was using python for the win! Using tableau API + python is enough for me to extract that data. But here comes the problem. First I am the type of person who does not really understand the network haha fortunately, my lead gives me clear direction and a helping hand for that. At that time I was using cloudbuild to execute our task.

That is why I got the problem, the network between cloudbuild and private network. It does not easily understand each other. We need something so they can connect. Especially if you have those services in a different project. So here comes the private pool https://cloud.google.com/build/docs/private-pools/private-pools-overview. What I remember is that this feature is quite new at that time, so there are not many references to read.

Thanks to Johanes Glenn story here https://medium.com/enlear-academy/cloudbuild-private-pool-for-reaching-out-private-resources-outside-gcp-2d0391828f1f I can understand the flow from this private pool. Also some talks with him about the network, I just knew about the transitive connection in GCP haha Before that, I just connect that cloudbuild worker pool and private network tableau with VPC peering turns out it did not work.

So here is what I have done to run this task

• I created a worker pool address and peer it with tableau network
• Then I created cloudbuild worker pool and set the network config as before
• Those 2 steps happened in the same project with tableau itself. Do not forget to set the firewall configuration if any.
• Lastly, I call the worker pool from my cloudbuild file (in a different project from tableau project). Here is the example from google cloud documentation

steps:
- name: 'bash'
  args: ['echo', 'I am running in a private pool!']
options:
  pool:
    name: 'projects/PRIVATEPOOL_PROJECT_ID/locations/REGION/workerPools/PRIVATEPOOL_ID'

So that is my story, this is quite fun to know that we can build a connection in that way and especially those who really love to use cloudbuild haha Do you have other stories about this one? let’s discuss and share it with me!