Is Netflix still using their Titus container platform? It’s mentioned at the start of the post but the rest of the post seems to be able Kubernetes. Also the Titus repos on GitHub were archived a year ago.
Interesting to learn about last gasp packets, seems like a useful mechanism for any system.
I generally find that observability into kubernetes is not that great especially for analytical queries. Answering the question of “where where my pods running for a replicaset at time X” is surprisingly not easy to answer with just events. Also, events are designed to have a short ttl so you need to extract them out of etcd and into some sort of long term storage and indexed in a way which you can run useful queries. At larger scales kube-state-metrics will blow up about any prometheus server outside of a full blown billion metric stream scale mimir deployment.
Sloop is pretty good for helping you understand events over time. It has an embedded database so you need to be careful with resources (apparently it's possible to split separately if required). Filtering out extraneous events can help minimise resource utilisation - it's amazing how many events a real-world cluster can generate running things like Kyverno, Flux etc. https://github.com/salesforce/sloop
Curious why this isn’t published to Pod events, seems like that would be the more natural place. They mentioned not wanting to patch Status but that too seems non-standard?
For the problem of where to run this if this was upstreamed, it’s an interesting problem - seems like you could make the case for the kubelet node agent baking this behavior in and gossiping to neighbors when the node terminates.
Probably be way less work and more reliable to emit node event when it comes back and scan for that. Seems like a case of google cargo culting. Remember kids, Google runs real machines and you probably dont
If you mean traffic from outside of the cluster, there is no reason for their cluster nodes to have public IP addresses, so they wouldn't even be routable from outside the VPC. I would expect this controller to run on the control plane and probably not be Internet-accessible by any means, but even if it were, I'm sure the security group only allows UDP port 6666 from inside the cluster, presumably based on source security group, not IP, so even if you could somehow spoof a private IP address and get it into the VPC, it would still not be accepted.
If you mean malicious traffic coming from their own pods, they could simply add Kubernetes network policies not allowing UDP port 6666 egress from any pod since the traffic they're actually hoping for would always come from a kernel, not a pod, and thus egress from a different interface.
So now you're left with the remaining attack vectors being spoofed IPs, or potentially even the correct IP but a spoofed packet, coming directly from EC2 instances in the same VPC, either by launching separate instances or compromising clusters nodes at the OS level. An insider with that level of access is always a threat pretty much no matter what you do because they could just delete your nodes through the EC2 service. You mitigate that with thorough audit logging, no shared or long-lived credentials, deep vetting of a very small number of personnel you give that level of access to, and the threat of near-certain jail time if someone does it on purpose anyway.
> Luckily the netconsole-setup command makes the setup pretty easy. All the configuration options can be set dynamically as well, so that when the endpoint changes one can point to the new IP.
I'm not sure I understand the purpose of using netconsole in virtual machines. The serial console is easier to use, has fewer moving parts, and doesn't rely on a functional network stack.
Shouldn't ec2 kernel panic be handled on the ec2-level instead? It's not like they're running hardware nodes, after all. And surely kernel panics on k8 nodes shouldn't really be a thing? Are they loading custom kernel modules at random times in there?
EC2 it appears does not do this (capture a kernel panic reason and put it somewhere that an orchestration system can find it later).
Kernel panics are a thing. Whether a kernel is hosting k8s processes seems not terribly relevant to the probability it will panic? k8s seems neither likely to induce panics more often, nor to reduce their probability, imho.
This is probably a case of "at scale, everything happens", including kernel panics in VMs that most people assume never panic.
Kernel panics are more common in Kubernetes because the container memory limitations mean the kernel is more often trying to reclaim memory, or failing to allocate, and that exercises little-tested error paths that are full of bugs.
https://github.com/Netflix/titus