I would say James Mickens sums things up nicely in "The Night Watch[0]." For example, you mention debugging with logs and metrics -- this snippet came to mind:

     “Yeah, that sounds bad. Have you checked the log files for
     errors?” I said, “Indeed, I would do that if I hadn’t broken every
     component that a logging system needs to log data. I have a
     network file system, and I have broken the network, and I have
     broken the file system, and my machines crash when I make
     eye contact with them. I HAVE NO TOOLS BECAUSE I’VE
     DESTROYED MY TOOLS WITH MY TOOLS. My only logging
     option is to hire monks to transcribe the subjective experience
     of watching my machines die as I weep tears of blood.”

[0]: https://www.usenix.org/system/files/1311_05-08_mickens.pdf

https://mickens.seas.harvard.edu/wisdom-james-mickens

I also have gotten a lot slack for it. But I see it as a way to convey that new feeling of wonder and awe a child has upon entering a new world. Challenge the assumptions, make up some nice words, pretend very hard that your academic title is an actual thing that gives you some blessed superpowers (as is the way that people often treat it!). I love it. It's truth wrapped in humor to soften the blow to the ones that think themselves overly important. The "shocked ones" are the ones that I try to avoid in life anyway, nice discussions begin at the edge of your comfort zone.

This might be my all-time favorite quote that I never to get use in relevant situations because nobody is around who would get the reference. I think of it almost every time I hear the word "tool"

I know I can; I don't even need any of my fingers.

Such luxury! I just spent a couple weeks getting FreeBSD booting in the Firecracker VM and most of my debugging was performed by inserting hlt instructions into the FreeBSD kernel and looking at whether virtual CPU halted or hit a triple fault.

JFYI:

https://tinyapps.org/blog/200702250700_why_in_my_day.html

Ah, I see the gentleman works directly in binary.

Only comparable experience I can think of is "breaking" the terminal via bash_profile and then not being able to fix bash_profile via the terminal. Or locking yourself out of a web server via security configuration update.

That: uh-oh! moment.

We loaded the connection in varying patterns for about an hour. ZERO CRC errors. Scratching our heads, we decided to punt. Then suddenly the CRC error count started climbing on my scope. I shout into the speakerphone, excitedly, what did you all just do? They’re like “nothing but unplug the scope from the patch panel”. “Please plug it back in”. CRC error count stopped.

Diagnosis: faulty patch panel.

Kernel code is amazing, especially the parts written by the early members like DaveC, MarkZ and others.

For me the biggest part was working with a group of extremely smart people who were very nearly the best programmers in the world.

I really miss that outside of Microsoft. I would imagine that you can get the same experience if you worked with the Linux kernel dev team or some of the other few places in the world like FAMGA where you can work.

My suggestion is to go for it. After leaving MS I found peace by working with the open source community and open source software.

Hope that helps, happy to discuss further.

> 2. Debugging exclusively via metrics and logs, since I can't just attach a debugger to a running server.

You often can't do this in kernel/OS development work either. Serial printf logging is often required. It can be a real "my tools to debug my tools are broken" slog.

> 3. Designing systems in general. Some people love the challenge of distributed transactions, eventual consistency and all that jazz, but it just rubs my brain the wrong way. I'm not interested at all in that problem space [1].

I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there.

Tools like windbg probably work great until you are debugging the code that runs when windbg tries to take over. Or you’re debugging something sensitive to interrupts and it’s literally impossible to keep up with a tool like windbg. Or you a debugging something that overwrites windbg in memory because you have a corrupt pointer. Or you triple-fault the machine and it reboots with so much prejudice that windbg doesn’t have a chance. Or you’re poking at hardware that the debugger can’t usefully interact with. Etc.

Basically, in the kernel, a lot of the things that a working kernel does to hold your hand aren’t available. So there’s a lot of time spent just thinking or adding logging statements or otherwise using low tech tools. (Or using high tech instrumentation or sanitizers!)

Kernel development is great :)

These days I mostly use gdb (which is incredibly buggy for such things) attached to QEMU’s gdbserver. It works except when I doesn’t.

https://docs.microsoft.com/en-us/windows-hardware/drivers/de...

https://designintools.intel.com/Silicon_View_Technology_Clos...

Most of the time you can do kernel dev with a serial port and printf though.

>I'm not sure I understand this issue. I mean, kernels have subsystems for doing stuff; you might need to design one someday? But it won't be dull-as-dishwater web technology stacks, it'll be you writing data structures directly in C/C++ or Rust if MS goes there.

Ah, I meant designing distributed systems. Like, figuring out the difference services, storage requirements, databases, then planning out the infrastructure. It's really not my cup of tea. I do enjoy designing systems/subsystems/components as long as they're within a single host :)

  Serial printf logging is often required

1. There will be a lot of infrastructure complexity in the kernel, just prepare yourself for that. Even worse bugs! You'll be fixing a lot of bugs, or looking at a lot of bugs, and most of these bugs are from other teams who are interacting with your component! Just order a copy of Windows Internals and get yourself familiar with how thing work.

2. Old ass engineering systems. Just as the interviewers said you will spend a lot of your time waiting for Windows builds.

Good stuff:

1. Work life balance is amazing actually. Most of the time there's very little pressure for you to get work done, as long as you're doing something no one really bothers you.

2. Since you said you hate designing systems, good news, everything has been designed for you! Your, job will mostly be implementing new features for a component.

3. Windbg is actually great and I will die on this hill. You might have to print some debug logs but you won't be looking at metrics because there's whole teams dedicated to doing that stuff. There's also tools which quickly spin up VMs for you to do some live kernel debugging.

Meh stuff:

1. The pay, could be better. That might change soon though.

  >  Windbg is actually great and I will die on this hill.

Came here to say this. (Actually kd, but lots of overlap.)

I think you are coming at this from the wrong perspective. Rather than thinking about how to avoid work you DON'T like, think about what you DO like and then decide if the new job would offer more or less of that.

Personally, I've found that every five years I end up sick of working on the same kind of problem and I have to go work on something completely different. Maybe that's where you're at.

Is it difficult to transition? Yes, in that if you spend 10 years learning SystemA, then it will take some time with SystemB to build up to the _same level of expertise_ that you enjoyed on SystemA.

The neat thing about learning one OS or system _deeply_ is that the deep knowledge gifts you with frameworks for learning the next-system.

I wrote software on various *nix'es during my university time, then worked on OS/2 (yes, "OS-who?"), Windows, and now I'm at Google, (my work involves 80% Linux 20% Win). It took me some time to re-build my skills on each environment/platform, but it's been a great ride and a lot of fun.

(P.S. earhart@ pointed me to this thread, and says hi to all y'all: Evan, JonO, JVert, LandyW? Dan?)

Source: 33 years experience hopping all over the spectrum

Ultimately, it's all just code and problem solving. And you can usually find a way to leverage your expertise in one domain into a different domain.

Most importantly, MAKE SURE YOU KNOW WHAT THE JOB IS! Microsoft people don't try to be dishonest, but there can be misunderstandings between you and your future coworkers about your role, and if you take a job that turns out to be different from what you expected, you will be unhappy.

If you haven't already, you should ask the hiring manager more about what the team does. Try to get enough specifics that you might not know everything the manager refers to, but can easily Google what you don't know: "For example, in Windows 10 version 2004, we shipped the API and implementation for the Windows hypervisor feature that lets third-party VM host software like VirtualBox force their VM guests' virtualized RAM to be paged into the host machine's physical RAM all the time." (Not an actual feature, at least as far as I know.) At this level of detail, you'll be able to judge whether the work is really what you think it is.

Talk to your other interviewers to learn more about the work and the team, if they gave you their contact info or otherwise seemed inclined to hear from you. 3 out of 4 of them are likely going to be your peers, and the 4th is either the hiring manager, another mid-level to senior leader, or a team architect - all will be at least close enough to your team that they won't give you vague generalities.

Getting the chance to work with people like that was one of the highlights of my Microsoft career. Some of them are famous or semi-famous outside Microsoft, like David Cutler (mentioned repeatedly in this comment thread), while others are not known outside MSFT at all but arguably should be, while others are respected among a small geeky community (I'm thinking here of 2 Linux kernel subsystem maintainers who joined MSFT after making their names in Linux, and continue Linux work today). If making those connections is something you'd want to do as well, I'd definitely see that as a big plus of an MSFT job.

> 2. Debugging exclusively via metrics and logs, since I can't just attach a debugger to a running server.

I find debugging Windows issues fun, but it might not be for you. On rare occasions, you might be lucky even to have telemetry and logs, for customer issues that can't be reliably reproduced. "We noticed that 30% of our module's crashes in the last Windows Insider Preview Dev build had this new stack, so we used Watson Portal to request more crash dumps from devices that ran into that particular crash, and a week later, we've accumulated this set of dump cabs...."

On the other hand, if you get an automated email saying "test case XYZ broke due to a crash in your module", you'll probably get a live kernel debugger remote - an email link or copy-paste command to open a kernel debugger on the dead VM, preserved for your debugging. But of course, bugs aren't necessarily caught that early, and even if they are, finding things via kernel debugging is a needle-in-a-haystack problem because you're debugging the entire computer, not just one process.

As an aside, if you are a third-party Windows application developer, I highly recommend you register your apps with the Windows Desktop Application Program: https://docs.microsoft.com/windows/win32/appxpkg/windows-des... - this will give you access to the same app reliability telemetry, including user-mode crash dumps (.DMP / .CAB files), that Microsoft has. This is only the latest of several iterations of this data access over the years, but it and its predecessors are still badly underused in my non-Microsoft experience.

The "repros often but doesn't reliably produce a useful dump" is particularly frustrating. Like, you're seeing all those crashes, and every one of them is likely to be some poor user who is at best annoyed, and at worst just lost some data. And you have no clue as to what the bug is or how to fix it to help them.

That name is misleading, but only partly so: shipping the Windows desktop and Windows Server products are a major part of that division's mission, but so is building Microsoft's internal-use Linux distribution CBL-Mariner, all of Microsoft's embedded and Internet of Things software products (Azure ThreadX RTOS, Windows IoT, etc.), and various Microsoft-internal software and hardware products.

It's very possible that the team you'd be joining would be working on an Azure product or Windows kernel-mode code for an Azure product, which means all your 5 issues could be a concern, especially:

> 4. The insane amount of work required to stand up even the smallest microservice: infrastructure provisioning, certificates, security reviews, GDPR compliance, etc.

> 5. Anything I build will end up paging some poor soul at 3am some day when something is down or under heavy traffic.

(Point 5 is even worse in Azure because you will get paged yourself if you're on-call. You can't just assume that the operations or site reliability engineering teams will take care of problems without pulling in the original engineers, especially when the product is new and buggy :)

The concern for me would be the fact that windows is no longer a fully offline experience running on people's desks. A large part of the kernel team is working on features that only benefit Azure and you may get some of the same services exposure there as you claim to be so burnt out on.

Asking to discuss the role more with the team to get a better sense of what they're doing, how they're doing it, and who their customers are seems like the path forward IMO.

And the IIS team, and the networking team, and…

In general, go for it. The level of understanding you obtain by working on the kernel about how things -really- work will make you a better engineer even if it turns out not to be for you.

Things I would caution you about based on the downsides of your current job:

1. Kernels make k8s look simple. Not just simple, childs-play.

2. Debugging experience in kernel-land can vary widely depending on what layer of the stack you are working on and that nature of the bugs. Highly concurrent pieces of any kernel are a nightmare to debug because such bugs are normally race conditions and timings are incredibly sensitive when you get this close to the metal.

3. Whilst you won't have to worry about this initially as you will probably need a few years experience at this level before you design new kernel subsystems I definitely wouldn't consider architecture at this level less complicated than distributed systems. This is because computers these days -are- distributed system. NUMA essentially means you have all the same problems. You do have much more convenient tools for solving them though (at the cost of performance) like HW coherency, etc.

4. Ok yeah, you shouldn't need to worry about this one. You won't be standing up new build systems or anything.

5. Well.. this is the rough part of kernel bugs. When you fk something up you potentially fk over everyone, usually in a very subtle, hard to diagnose and even harder to workaround way.

So yeah, go do it but don't do it because you think you will be getting away from those things because you aren't really. Do it because you think it will be enriching/fun/whatever you want more of in your life.

1. Yeah, I have no doubt about that. My peeve with k8s is not its inherent complexity, but just how much worse it made our day-to-day work. When we ran things on managed services, everything was a lot more understandable and straightforward. k8s is piles and piles of hard-to-discover yamls and I (and other folks in the team) feel that it made our infra a lot harder to understand and change.

2. This was my most unclear point. What turns me off about my current experience debugging distributed systems is having to trudge through millions of log messages coming in constantly. I just can't say I like the ELK stack, despite its popularity.

3. Fair point, though I'm aware of the distributed nature of things even within a single host. I do expect it to be different though. Will I still have to account for "the response was lost, you don't know what happened" situations? My understanding is that being close to the metal I can at least trust that the wires are all still there and working, and if they aren't, well, I can say the machine (or something within it) is broken and needs to be replaced. Do you have to do things like sagas or deal with eventual consistency at the kernel level? Hardware guarantees seem a lot stronger than what you get in networked systems.

4. Yay!

5. The difference here is that even a distributed system that's built correctly will still page someone one day. There'll be more traffic than it was built for, or some network dependency will be down (like, not the things you accounted for, but something fundamental like a DNS server or whatever), etc. My expectation (and past experience) is that in code that's intended to run on a single host, I'll either get it right and it'll just work as intended, or there'll be bugs that I have to fix, and once I do things will just work. What I find extremely frustrating about services is that it seems that no matter how much effort you put into quality, there will still be times when it just goes down and people have to put out a fire.

Definitely going for it after all the responses here!

The other great part is the sheer amount of engineering that will go into every feature you will own. I say “engineering” because it’ll be different from any services work you have done.

Tools that are at your disposal as part of Windows Core team is top notch.

Good luck! You will not regret this.

I enjoyed my time in the Windows team the most.

If you can read and enjoy these books, consider it:

https://docs.microsoft.com/en-us/sysinternals/resources/wind...

The ugh!: You will be a tiny cog in a big machine, spending very little time writing code, mostly banging your head over someone else's bugs. And M$FT still has a certain stink to it these days and that may cling to you.

The windows org has a reputation of being dysfunctional although I never personally observed this, and personally think even if it were true it would be in the higher level parts of the stack.

Kernel dev was my original dream but I fell in love with Excel during my internship. I definitely think you should go for it.

It leads to code being Complicated, and a lot of Kobayashi Maru situations where there is no good solution to an engineering problem, just a bunch of bad compromises. Depending on your area, you might run into this all the time, or you might not.

You said you were looking for things that you might not like about kernel development, so here’s a few:

- Distributed systems. Everything is distributed these days with NUMA, cache coherency, PCI transactions, etc. You’ll have to know how to use the right kind of atomic memory ordering, debug lockless algorithms, and know how the OS scheduler works at a deep level. If this doesn’t excite you, the job might not be a good fit.

- Working with physical hardware. Although in most cases you can get away with a VM, you will almost certainly have to use an actual device for some of your work. The hardware you use might not be 100% functional or even correct- I’ve had to debug kernel issues that ultimately were hardware errata. At Apple it’s easy to get in contact with the silicon design teams, but I’m not sure about Microsoft and Intel/Qualcomm. Working with physical hardware also limits your ability to work remotely easily. Lugging around a bunch of laptops, phones, and tablets in a carry-on suitcase is no fun, especially when you have to take them all out at TSA checkpoints.

- Lack of user-visible impact. Kernel dev is vitally important but it doesn’t get much visibility, except when something goes wrong. New kernel features hardly make the headlines. It can be a little annoying to see your colleagues in app/web dev get recognized for the features they worked on at WWDC or Microsoft Build, while you’re toiling away working on kernel features that very few people care about.

- Debugging. I actually think debugging kernels is not too difficult if you have the right set of tools. At Apple we use lldb and hardware debuggers (JTAG/SWD), and have the ability to take a core dump of the kernel after a panic to analyze later. But since kernel dev is at the core of the operating system, you’ll have to learn to debug other parts of the stack too. For example, you might make a kernel change that breaks the file explorer but only when you visit a specific directory. So you’ll need to know how to debug both the user space process to know what’s going wrong, and the kernel to know why that is happening.

Although there are some negative aspects of the job, there are many more positives. I really like my job and can see myself staying at Apple for many years. I’m constantly learning and working with some of the smartest people I have ever met.

Give the Microsoft job a shot; you can always leave if it’s not a good fit but you will have left a better engineer than when you started.

I'm familiar with the things you mentioned, except PCI transactions. In this context, do you also have to deal with distributed transactions, lost responses, eventual consistency, and the like? Do you have to account for "response lost, don't know if it went through" situations?

Definitely going for it after all the responses here :)

In my experience the kernel devs are the best of the best developers. But know that working on the windows kernel will be complicated and require a lot of domain knowledge. Expect to learn about it for _years_ to come.

> 1. Infrastructure complexity - Sure, but there would be a LOT of complexity in kernel dev

> 2. Debugging - :) MS kernel is ancient and must be full of cruft. Systems dev is notorious for all sorts of weird timing, coordination bugs.

> 3. Designing systems - You'd have transactions, concurrency, race condition kind of problems which imo tickle the same part of the brain as distributed txns, eventual consistency etc

That said, if you like it, you like it. Can't know without trying. If I were you, and if I had the opportunity - I'd go for a newer variant of this which could look like Apple's M1 team, Tesla's systems teams etc. simply to have a fresh slate to build on.

You might even be able to return to your old company if you hate it. Tell them directly "hey I need to give this a shot but I'd love to check-in in a year".

Curious how you get an offer such as this? I've thought about changing job roles, but I really suck at leet coding so I've never really bothered. I figured that as a 40 year old male, no one would hire me for a role unless I was already experienced. Is that not the case?

Have you considered applying for entry level jobs of a different domain? As long as you are ok with a pay cut, there is not such thing as over-qualifying.

MIT 6.172 Performance Engineering of Software Systems, Fall 2018 https://www.youtube.com/playlist?list=PLUl4u3cNGP63VIBQVWguX...

I feel what is underpinning this thread is a job dissatisfaction issues from the ratio between "using other people's software" vs. "writing software for others". The former is inevitably frustrating as I'd become a consumer of software, rather than a creator. Kernel programming, and low-level system engineering in general is more aligned with the latter where you have more creative freedom and less constraints from other people's abstractions. It's like driving an automatic vs. manual car. The latter is just more fun, despite its inefficiency.

Enjoy the courses!

The main thing I'd consider in this is the work/life balance. Will it allow the same stellar level of that? People vary in their priorities. For some it's interesting/fun problems to solve, for some it's impact, but for me it's work/life balance, presuming of course that the work side isn't a hateful stressfest. No amount of remote work or flex schedule will make up for absolutely hellish work environment, and I had something like that once. (I ended up taking a 30% pay cut that I could just barely afford and got the heck out)

So I'm sorry that I can't speak to the kernel work itself, but instead the framework for making the decision. If the work/life balance will remain constant then you're only risking the possibility that the work won't be any better, and hopefully no worse. If the work/life balance for the new job is also uncertain then (for me) that would be a bigger risk consideration.

Finally, you should consider the worst case scenario: The work is worse, the work/life balance is also worse. How easily can you shift to something better? If you have the chops to work on the Windows kernel then I'd wager that if you hit the worst case scenario you could get out of it without too much trouble, but you're the only one who knows whether or not that's true.

Best of luck to you, sláinte, and may the wind be always at your back.

Microsoft's work/life balance is good in my experience; managers respect your time, product deadlines rarely or never cause long-hour rushes, and if your role lacks an on-call rotation then you won't be woken up in the middle of the night. But other companies may be far worse, and Microsoft is big enough that one person's experience never truly generalizes.

For time off in particular, as of my last experience 2 years ago, MSFT's US policy on paid time off was stingy by American tech company standards:

* There is no scheduled period of the year where everyone gets paid time off. (LinkedIn has 2 such weeks a year but it manages its HR independently of the rest of Microsoft.)

* You accumulate 3 weeks of vacation PTO per year of work for your first 6 years at the company, then 4 weeks per year for years 7-12, then 5 weeks per year for years 13+.

* Unused vacation days expire at the end of the calendar year following the calendar year in which you accumulated them.

* When you leave the company, you get a "vacation cash-out:" you are paid a lump sum equal to all your unused, unexpired vacation days.

* In addition to vacation, each calendar year you get 2 weeks of paid sick leave and 2 "floating holidays" to use like vacation days.

For comparison, at Google you accumulate 4 weeks of vacation per year starting just in year 2, while at Netflix and many smaller companies you get unlimited vacation at the cost of no cash-out.

In practice, there is more leeway at Microsoft on PTO than this policy allows. Even when I had an otherwise poor relationship with my manager, they still looked the other way when I went on Christmas/New Year's week vacation without filing an absence request or using up my vacation days. And I have heard of divisions where after a particularly stressful and overtime-heavy year, the division Vice President told everyone to take such an off-the-books holiday vacation.

Congrats on your kernel development offer from Microsoft! I imagine these to be very scarce these days, considering that today's Microsoft seems to be more into services and less about low-level software development.

If you like to chat more about this, just drop me a line. Always interested to hear from low-level devs and aspiring ones. My website/e-mail is in my HN profile.

[1]: https://news.ycombinator.com/item?id=30892773

It’s very readable, if a bit idiosyncratic. I would work on it again, given the right opportunity.

if (offered_nt_kernel_job) accept_right_away()

You can always quit if it doesn't turn out well.

You'll work with people great at their job whom you can learn a lot from.

I also loved having my own window-side private office. I've since heard that Microsoft was trying to adopt open offices, but if it's still there, awesome. Remote work is okay too of course.

https://www.amazon.com/Old-New-Thing-Development-Throughout/...

Also read - Showstopper

I know, it gets a lot of trashing all around, the ads, the ui, the whatever - I like the UI and I really like what it offers me for a technical person: ETW traces, Performance monitor, Event fowarding, PowerShell, Windows Terminal, GPO, Windows Defender with advanced features, Resource Monitor, ACLs (Well, auditing file access could have been better :) ), wf.msc, RSAT, HyperV and whatever features under the hood that enables Process Monitor!, ..., ... ,...

I’m not sure this is really a difference between kernel and backend service development. You can run the server on a dev machine, attach a debugger, and reproduce the bug. If your problem is you can’t reproduce the bug, you may encounter the same issue when working on the kernel. If someone reports a kernel bug, you may not be able to attach a debugger to their computer, so you will need to reproduce it.

- a lot of bikeshedding

- a mailing list driven workflow

- poor continuous integration

Raymond Chen is a favourite of mine:

https://devblogs.microsoft.com/oldnewthing/

You'll run into a lot of the same issues on the Windows kernel that you're getting irritated with in service development land. A lot of infrastructure is already out there, but figuring out how to use it effectively ends up being almost as much work as just rewriting it from scratch. Things are documented but poorly.

For Microsoft specifically you'll often get stuck because of problems with some other team's code, and then find yourself embroiled in multi-week battles with multiple engineers, managers, project managers, and product managers all at each other's throats.

All that said, the Windows kernel itself is a work of art IMO, and kernel development is a lot of fun. I think that if you want to make the big bucks but still work with c/c++, it's getting to the point where your choices are very limited: Microsoft, Google, Facebook, Apple, or Tableau.

Microsoft was terrible to work for, but they aren't as bad as a bad small company. At least there's HR to report gross violations to, they have a TON of great perks, the money is really good, they have good work-life balance, and it's not too hard to change teams (but if you find yourself wanting to jump to a different team, do it well before performance reviews come along, since your manager will definitely give you a bad review and hamstring your ability to move around within the company.. so you'll end up having to change companies. Not the worst thing if you do so, I guess)

EDIT: I just reread your list of complaints.. timing and consistency in the kernel are WAY more complicated than your average web services. There's also a ludicrous amount of red tape around getting anything done. I'm starting to think that you should look at Microsoft as a stepping stone in your career rather than its final destination.. it's still a good place to work for overall compared to random small companies, it'll give a boost to your resume, but it sounds like what you actually want to work on is game emulators or raspi home automation gizmos in your free time.

This is documented in Zachary's Showstoppers book:

https://www.flyingpigbooks.com/book/9780759285781

It appears that performance and architecture improvements are very hard to get accepted (and the kernel suffers for it):

https://blog.zorinaq.com/i-contribute-to-the-windows-kernel-...

It is a great piece of engineering, but it is likely for a developer that is more comfortable making small changes within the hierarchy than for any revolutionary ideas (unfortunately).

If Cutler hadn't found his way to Microsoft, then they probably would have ended up on a BSD kernel, as Apple did. As it was, Microsoft sold their Xenix business around the time of Cutler's arrival.

Judging by the presenters at CPPCon, a significant fraction of highly-skilled and highly-paid C++ devs are employed in the High Frequency Trading (HFT) industry.

They meet OP's criteria in many ways as well, as HFT trading tends to be "very close to the metal", optimised to death, and often involves solving deep technical problems. It's diametrically opposed to modern web development practices.

I’m pretty sure that you could donate a big chunk of the million+ ill-gotten dollars to charity and make a lot of good things happen

I’m not telling you or anyone else in these companies to suck it up, but there’s a point where employees and teams in these situations need to step back and realise what they’re dealing with, and adjust their mindset accordingly.

Personally, I like identifying and unfucking bottlenecks. Sometimes explaining why something is so fucked to the people being fucked can go a long way.

I don’t like it either, but I can’t really do much about it. If you realise you’re in a difficult organisation, you can at least start to take action, or take a different tact (or leave). Being annoyed about the situation is probably just going to make you more and more unhappy because the system won’t (or can’t) change in your desired timeframe.

What's funny is that the guy agreed with everything I thought, he saw the problems that were hamstringing me (and everyone else), he had complete authority to fix the problems.. and so far as I can tell, he wasn't having any luck with it

The best fix seems to be to tell people to quit being jerks to each other and focus on making good products and making customers happy, but good luck getting that to happen.

One of the main criteria for performance reviews was suddenly something along the lines of "Teamwork" that captured what a non-toxic, non-Ballmerized good person you were. Of course, all that happened is that the best backstabbers and the weaseliest scumbags manipulated their way into high "Teamwork" scores, and anyone who sat down and focused on productive work/helping others got screwed on that score just like everything else.

The pity is often how the most qualified talent for resolving poor behaviour is so often subordinated into slopping out the pigsty. There needs to be a foundational next step.

Edit: since Fairchild we've been scared of merely assembling raw talent under a thin layer of management capabilities and from this is developed the pseudo matrix management system that e.g. Microsoft operates.

There’s going to be SOME of that stuff any time you have more than one person in the same building, but a decade under the iron fist of Ballmer has permanently ruined Microsoft as an organization.

It’s a fractal shape too. You see the same sort of intrapersonal weaponry at the division level, the org level, the group level, and even within teams

Maybe it even continues to a Herman’s Head-style inner conflict for good company men

Presumably Tesla and Uber must have some “native” code being written somewhere in-house too, and someone mentioned the HFT industry

There are a lot of decent jobs out there that will keep you comfortable, too, even if you can’t bring in 300k+.

Microsoft is great. It's an amazing place to make impacts in many different areas of technology. Work teams used to be more siloed, but that's changed a lot in the last 6-7 years. We regularly work across teams at GitHub and Microsoft to get things done with great cooperation.

As for Teams vs. Slack vs. Discord, it's a personal taste thing. I'd rather have Teams over Slack + Zoom 1000x.

Take the Windows kernel job! Worst case, it's not a fit and you move on, like any other job. Best case, you stick around for a long time and have a great career working on tech that 1B+ people use.

When Robert Downey Jr. was trying to talk Gwyneth Paltrow into joining "Iron Man", he said to her something like, "do you want to work on art house films the rest of your life or do you want to be in something that people actually see?"

Windows is massive... it's the largest code base that I'm aware of. It has a ton of process and procedures and test cycles because... it's Windows. Of course you can't just freelance there. But you can "be in something that people actually see", and most SV startups will never even get 1,000 users, much less 1,000,000,000+, and more if you count the users on Azure services using Windows indirectly.

Sounds like a bastardization of Jobs on convicing John Sculley manufactured by a PR person:

Steve Jobs and John Sculley, then PepsiCo president, were sitting on a balcony overlooking New York’s Central Park. Jobs turned to Sculley and said, “Do you want to sell sugar water for the rest of your life or come with me and change the world?”

https://www.forbes.com/sites/carminegallo/2016/11/12/how-ste...

That's like asking someone who cooks at a fine-dining restaurant to work at McDonalds, "do you want to make pretentious food for 40 cashed up foodies and critics a day, or do you want to make food for hundreds of regular people a day?"

People pretending that Facebook, Google, Microsoft, et al are noble endevours are deranged. It's like rooting for Walmart to win the World Series of Retail.

A pet peeve of mine is pizza joints who serve a different pie every time, depending on who's in the kitchen. If I have a great pizza, I expect to get it again, but rarely do. Consistency is hard.

The culture in games is very healthy from my experience and one of the best compensation in the games industry.

Microsoft is incredibly siloed. Each team is essentially their own mini company and they're mainly guided by large top line metrics but there's no top down overall vision on what something should look like. It's like those party games where everyone has to draw a portion of a drawing. It comes out looking like a disaster even if every individual portion is good. This also means that product management doesn't work with any particular team either. You essentially get some random big metric and are told "make this metric better" without any context. Everyone is duplicating work and there's nowhere to learn from

Microsoft Teams is a disaster in so many ways. In the most obvious way, it's very very slow and a pain to use. This subtly hinders teamwork because no one wants to use teams. In other ways, there's no global search so finding stuff in other orgs is impossible. The teams "channels" are essentially shitty forums that are unintuitive to use. You'll never get a channel about hobbies and stuff and even if you did it's hard to find and they're usually dead. Everyone uses private group chats including each team but these have zero discoverability. What this means is you'll never have a golang channel or something where people share and chat about stuff. Most people have a facebook group where they chat about stuff (wild).

Every team has their own onboarding down to what hardware you should get. In theory this allows for some flexibility but what this actually means is that no one has any idea how you should be onboarded and you essentially are sent to flounder until you pick stuff up.

Everyone is doing everything. Every person is their own product manager, scrum master, manager, and also programmer. There's so much duplicated process overhead it's wild.

They have not handled remote well. They insisted on trying to send me a desktop computer. I asked for a laptop and they couldn't give it to me and they instead sent me a used intern laptop. They gave my sibling a used surface tablet. This is a 2 trillion dollar company and they're unwilling to shell out 2k for a basic workstation computer with 6-8 cores and 32gb of ram. Not a huge ask. Also some stuff is only accessible through a direct hardline in the office. Whether you want to use a desktop or not is irrelevant. It's mainly how cheap they are when it comes to hardware.

EVERYTHING has to be Microsoft software for the most part. If you think nih syndrome is bad at your company, imagine you're at a company where they've been making mostly mediocre versions of other software for the past 30 years. Yeah. I'm not a huge splunk fan but trust me when I say the azure equivalent is much shittier.

The pay isn't top notch. In fact it's pretty bottom barrel for a big company and if you can pass the microsoft interviews you can pass somewhere else. Their interview process is also a nightmare. I went through 4 different recruiters and it took me 2 months between passing to get an offer letter. The whole thing was insane. They initially offered me such a paltry amount it made me laugh.

Everyone is a lifer because anyone else has left. Imagine talking to your boss about docker or talking to him about IntelliJ and he's never heard of it because he's been at Microsoft for 20 years.

There's a lot of weird "not racism" but might as well be where certain ethnic groups have taken over certain orgs and speak in their primarily mother tongue despite being in the US in a US based company. It makes teammwork really hard.

You need a separate laptop to log into any production resource. Production resource is a loose term because that also includes int environments and anything on azure. I have 2 laptops and a desktop that I'm forced to remote into it's insane and so slow.

There is no one to talk to about anything. You literally cannot find what anyone is working on or what's happening. Discoverability at this company is literally 0

HR processes are like actually completely broken down to me not receiving healthcare for 3 weeks after I joined forcing me to pay some very big expenses out of pocket that would have been 0 after the fact. Getting reimbursed doesn't work because after the fact those expenses only applied to my deductible which wouldn't have been the case if I paid initially with my healthcare.

Most things are a huge heaping pile of legacy crap that absolutely cannot be changed for backwards compatibility. Imagine working on a c++ code base with no local environment, no unit tests, and the only way it can be changed is to make a change and upload it to a build server (1hr + build times) and then deploy it. Yeah. There's no room to improve things because it's all so delicate.

There's simply so much. Thankfully I'm leaving but it's been ass

I think what you should do is, if possible, arrange for a short call with some of the team mates that you are going to work with. Just a casual one about how the kernel team day to day work looks like there. If not with the team, then maybe with your manager.

Unless someone from that team or some ex-kernel team member replies here, all the rest of these posts are just speculative and not worth basing your decision on.

I don't hate the competent developers at Microsoft (of course they exist, there have been source code leaks) - I just very strongly dislike the machine that is Microsoft. I don't think the machine has reformed.