I once had an issue where it appeared my servers were leaking less than a byte of memory per request (WTF? less than a byte?).
It took months to find the bug. Turns out it started leaking once the filesystem quota was reached, the log tried to roll and failed. All logging messages were just building up in a queue. It was a definite face-palm moment once found, but yeah, took months to get there.
I had a similar issue w/ a high-res timer that was causing spurious socket timeouts. Only found because...logs were saying operations were taking +200 years to complete. That was about 6 weeks of time during development.
It took months to find the bug. Turns out it started leaking once the filesystem quota was reached, the log tried to roll and failed. All logging messages were just building up in a queue. It was a definite face-palm moment once found, but yeah, took months to get there.
I had a similar issue w/ a high-res timer that was causing spurious socket timeouts. Only found because...logs were saying operations were taking +200 years to complete. That was about 6 weeks of time during development.