I'm trying to reproduce the Cloud SQL Proxy in Cloud Shell that both you and the OP have mentioned, but I've been unable to reproduce (and thus file a bug). I'm sure I'm missing a step. If you're willing, I would love if you could drop me a quick note (sethvargo at google dot com) with the steps you're taking, commands you're running, and output you're seeing. This will help me escalate it to the right team.
I'm also going to share your feedback about Stackdriver with our product team as well. Thank you for that!
Just to clarify with some more detail (for the HN record), the issue with Cloud SQL is that while you can whitelist an individual internal IP, you can't whitelist an internal network (like 10/8). This means you need a sidecar service of some sort to add the whitelist rules when a new instance/pod comes up that should be able to access the DB.
In AWS for a simple app I'd just whitelist 10/8 and have the DB open to all internal instances (with TLS for added security as required).
I'll follow up by email too, happy to provide detailed feedback on this sort of thing.
Thanks for taking the time to engage with the community on this stuff!
We have similar experience with StackDriver Logging. It's felt unfinished since day one. Two problems come to mind:
(1) Some time ago I needed to get a month of logs (filtered by an expression) in order to compare the logged data against a database. The Console does not have any such option. You can set up "exporting", but that only starts replication from the moment you set it up; no historical data. You can use the APIs or "gcloud beta logging read", but the performance is terrible. I got maybe 200-500KB/sec. After keeping the command running continuously for 3 days (!), I still only had 20% of the log data.
These days, we have exporting set up to pipe all the logs into BigQuery and GCS, just to get around the crappiness. That gives some query capabilities, as well as the ability to quickly grab the original data for processing with other tools.
(2) We're on GKE, and none of the container's labels end up in log entries. Surely this is pretty key stuff. There seems to be no way to customize the payloads that GKE's Fluentd sends to StackDriver.
(3) There's no way to tail logs. The gcloud command doesn't have it, and the Console doesn't either. It feels like such a missed opportunity.
I think Google would be better off ditching StackDriver Logging entirely and instead let the user set up rules to send logs to different services that actually work. The default could be BigQuery. I don't understand the point of having a separate, proprietary query language for logs when you have SQL.
I've looked briefly at the other StackDriver stuff (most of which is still on stackdriver.com, always causing a confusing redirect and often re-authentication), and it's been similarly underwhelming. We use Prometheus with Grafana, and StackDriver seems pointless in comparison.
the biggest problem I had with cloud sql proxy or cloud sql in generell is that postgresql is limited to 100 connections.
since I'm a small customer hosting many low traffic projects I want to share my instance as much as possible, however running pgbouncer for a lot of ups is painful since cloudsql does not allow to query the auth table (what's used by pgbouncer if you don't add all database username/password on your own).
I think a lot of problems would be solved if there would be a way to setup cloud sql proxy that comes with transaction pooling, especially when using k8s and having a pooling namespace (multiple load balancers for HA) where all other apps connect to.
also cloud sql and ipv6 is not funny. (i.e. a lot of things in gcp is ipv4 only which is sad)
I'm also going to share your feedback about Stackdriver with our product team as well. Thank you for that!