Elegance doesn't matter; simplicity does in the face of changing requirements. Or rather, elegance falls out of this simplicity. It's an emergent property, rather than an explicit aim. Thus, my favorite designs have traditionally resembled a microkernel in approach, where the kernel of the app exists only to facilitate disparate services to find and use one another. Interfaces are kept to an absolute minimum, and concerns like UIs or persistence are pushed to the edge of the system, where they should be. This design facilitates division of labor well; more novice developers can make a mess within their subsystem, yet have the damage contained. I usually bolt a pub/sub mechanism on relatively quickly, as this requirement is almost a given with a UI.

Dynamic loading is indeed the ultimate modularity. Combined with this microkernel approach, extensions are just as powerful as built-in services. The only problem is you enter a new hell: managing dependencies/versions of the core vs the extensions. If you can control the release of the extensions as well as the core, you're set.