I don't mind merging new modules at the moment, so would be happy if you try it out and contribute! But it's also possible to use your module without merging in the upstream HPI package [0]

In the long run, I'm not sure how sustainable it is, since there are many different data sources and ways of representing them. This is also particularly difficult when you don't use the service, e.g. as a maintaner I wouldn't have any means of testing Traktv.

So far I've kept all HPI modules close, because a monolith is easier for prototyping and refactoring. But my fear is the fate of oh-my-zsh, or spacemacs, which are overwhelmed by the pull requests.

Ideally I think it should be a simple core, only containing the common utility functions, extraction helpers, error handling, caching, logging, that sort of thing; and make the rest third-party.

It's possible to achieve this in Python, thanks to the namespace packages. The only problem I see is managing these small individual packages, and declaring dependencies between them. This is possible with PIP and setuptools, but there is certain overhead involved, I feel like this step is ought to be simpler, especially to people who don't want to fully dive into Python.

[0] https://github.com/karlicoss/HPI/blob/master/doc/DEVELOPMENT...

Have you looked at Pluggy much?

It's the Python plugin system that was spun out of pytest. I'm really impressed by it - it's a very clean design, and integrates great with Python packaging.

I'm using it extensively for Datasette, which means myself or others can add new features to the core software without needing to ask permission and in a way which supports trying out crackpot new ideas without sullying the design of the core software.

I've been thinking about usi it for my Dogsheep personal analytics suite too, which is currently split up into a bunch of separate tools in separate repos (since the only unifying interface is that they all spit out SQLite databases).

Thanks, looks great, I'll check this out! So far I haven't even done proper research of potential solutions, because I feel like it would be overengineering at this stage, I'm still figuring out the core.

When I decided to add plugins to Datasette I asked around and pluggy was pretty much the universal recommendation - with hindsight it's worked out really well so I'm happy to pass on the recommendation.

> since there are many different data sources and ways of representing them

I've been working on something quite similar for a while (in Rust).

I decided to normalize data to schema.org types where available and store JSON-LD documents in a key/value store. While the schema.org types are far from perfect, it at least morphs data into a standardized format.