Yes, if your adversary has physical access to your home, your computer, or other methods of installing backdoor software on it then the question of password security is rendered moot.
You can't have a secret and type it on a compromised computer too.
My point was to put the question in its appropriate context: it really depends on your threat model. And if that model includes those whom you'd prefer not acquire information having ready access to your house, then no, it's not safe.
Similarly: if that's not a problem for you, it's a perfectly reasonable practice.
That said: I'd probably try to find a slightly more obscure and/or secure location than in plain sight.
Your threat model matters. It includes possible attackers, their modes of access, likeliness of access, the assets you're trying to protect, and how they might be used in ways damaging to you. Any significant discussion or assessment of security should be framed in this context, and it's very much generalizable beyond online, electronic, or data systems.
I completely agree it depends upon your threat model. But I find the term 'threat model' isn't that useful when a simpler answer is possible. The term is great for leading you to ask more questions.
One huge advantage of the paper system is people have thousands of years of collective experience dealing with the security of paper documents. For example, the 4th Amendment to the US Constitution reads "The right of the people to be secure in their persons, houses, papers, and effects, ... shall not be violated...".
The question "is it safe on my desk next to my computer?" illustrates how users tend to discount their own experience and common sense once computer security gets involved. This is certainly reasonable from the users' perspective. We've all had absurdly counterintuitive experiences with computers, heard astonishing stories about hackers, and gotten plenty of nonsensical advice from the 'experts'.
It's domain-specific language. It is a model. Of your threat profile. Of risks, exposures, etc. Understand the concept, it's useful.
One huge advantage of the paper system ... Paper has many advantages. I own a great deal of paper. I love paper. It's tremendously stable.
It's also hard to search, expensive to duplicate, and carries a risk of single-copy loss. Even misplacing (without destroying) a document can be a crisis.
Those are all parts of the paper threat model.
Your mention of the fourth amendment brings up s great many other issues, and I won't discuss them, but generally pointing in the direction of:
Are electronic records "papers", and in what contexts and locations are they treated as such.
Do protections against unreasonable search protect against warrented searches? Or warrantless searches?
As for practical experience: I've had some in the areas of which I write here.
You can't have a secret and type it on a compromised computer too.