I had a lot of interesting chats with people about South last week at EuroDjangoCon, and several eyebrows have been raised at me both parsing models and then storing their definitions as dicts.
While this is a design that is partially born of backwards compatibility, it was mostly a conscious design choice I had to make, and so I'd like to now describe why I made it.
Firstly, for those who aren't aware, South's magic autodetection and auto-writing-migration code is based off of a file called 'modelsparser.py', which does what it says on the tin; it opens models.py, parses it, and extracts definitions directly.
Now, my first response to this is also 'ew', but there's a reason for it: custom fields. If it wasn't for those, introspection of the existing fields would work just fine, given a long list of special cases (I in fact have a file called modelsinspector.py sitting around that does just that - if you're interested, it's in the 'noparsing' branch on BitBucket. It doesn't do much, but it proves the concept).
However, with custom fields in the game, you have two options: hope they follow Django conventions and try to introspect common options, or put the onus on custom field authors to add a hook that makes it spit out a definition. While I would obviously choose the second one (possibly added to some of the first), it simply doesn't provide the drop-in compatability I want South to provide.
However, as Alex Gaynor and several others have surmised, I could just pickle everything and stick that at the bottom. No introspection needed, no weird namespace errors because the models dictionary is simply a collection of eval-able strings.
However, I don't like pickle (the Python kind; the delicious food kind, on the other hand, is more than welcome in my household). The security issues don't matter here - after all, we're passing raw strings to eval instead - but it has two major features I don't like. Firstly, it's not at all human-readable - and while I don't imagine people editing their migrations by hand, I like to be able to see what is going on, and I like the fact that South still doesn't depend on you using startmigration - a design choice I'm very proud of.
Secondly, I've been burned by pickles in the past. If you remove a module that a pickled thing uses, pickle will crash and burn, and if you've not done them right, you won't know quite where the error happened. (For the purposes of argument, of course, I'm assuming I'd do it right, so this isn't really a point. Still...)
There's yet another option (see, my job isn't easy) - custom-pickling, which Alex suggested last Wednesday; it consists of roughly freezing the __dict__ of the field. This has some advantages - it's more human-readable - but you get the errors similar to my current approach (where datetime.datetime wasn't unfreezing if people didn't have datetime imported).
My main gripe, with both this and pickle, is that the internals of fields aren't guaranteed to be consistent between releases of Django, just the external API, and so South uses a method that lets it use the external API - in my mind, this is less hassle, since that's the thing that's really not going to change very much (and even if it does, you can easily see what South is trying to call, and run that in a shell yourself to see where the problem lies).
So, that's why models are frozen as a massive lump of dictionaries and triples. As with many things, it probably wouldn't be necessary if models had been done slightly differently (so there was a method that always spat out a nice, portable way to reincarnate a field), but with no reason for that previously, I would have been thoroughly impressed by the core developers' foresight if it existed.
As for parsing versus introspection... I'm still divided. Parsing always has, and still feels, wrong, but introspection might just be less reliable. If you're interested, go peek at the branch I mentioned above, and see what you make of it.
And now, I will stop blathering and return to my small hole at the base of the mighty Django Mountain, where I am attempting to fit square pegs into round holes. Good day.