GNU file doesn't just check magic numbers against libmagic, it also defines a scripting language and a series of tests/printers written in it.
That's what allows it to do complex things, eg identify all the flavors of ELF objects even though they share a magic number or determine if something is JSON or CSV without one.
The point is that, fundamentally, the concept of a file type is undecidable or not well defined.
Think about it: A JSON file can also be considered a text file. It could also be some higher level type of file, depending on whether it conforms to some application-specific JSON schema. Thus the kind of file it is has more to do with what you want to do with it; it's not some intrinsic property of the file.
Ok? I was replying to a comment asking about how such systems can work. GNU file is example of a program that makes a best effort to classify file type in a useful way.
This is the authors website. Apparently yeah its not part of GNU utils, I had no idea, I knew it came with most Linux systems so I looked for the Debian package and found the site linked above.
That's what allows it to do complex things, eg identify all the flavors of ELF objects even though they share a magic number or determine if something is JSON or CSV without one.