In C#, it is really easy to convert between class and JSON object, which brings tremendous convenience for Web development. Recently, I have to use a Python framework Django, and found that there’s no such feature as C#, so I implemented a shabby replica for that by myself.

What you will learn

In this article, you will get both a serializer and a de-serializer in Python to convert JSON string or dict object into or from a Python object. 😋

Prerequisites

To start with, you must keep one concept in mind,

"Python object is merely a dictionary!"

Not that absolute, but generally speaking, yes.

We do not begin with scratch, and Python already make json package part of its standard. So we just use it for fundamental parsing.

So… Let’s start serialization! 😆

1. JSON Encoder & Decoder

1.1 JSON Encoder

For basic conversion between JSON string and Python dict, we can simply use json module to do the stuffs. However, if you have datetime field in your Python object, you have to manually convert it to string, as JSON doesn’t have corresponding datetime value.

When you use json.dump, you need a custom encoder to handle datetime field.

1
2
3
4
5
6
7
class AdvancedEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, datetime.datetime):
            return obj.strftime("%Y-%m-%d %H:%M:%S")
        elif isinstance(obj, datetime.date):
            return obj.strftime("%Y-%m-%d")
        return obj.__dict__

Notice that, datetime is derived from date, so you should place instance check of datetime ahead of date. And the format is custom, just make sure they corresponds each other in Encoder and Decoder. 🫡

If you are time zone aware, for example in Django project with USE_TZ true, you may need to add an extra conversion. Just add a astimezone method before strftime for datetime.datetime class.

1
obj.astimezone(datetime.timezone(datetime.timedelta(hours=8))).strftime("%Y-%m-%d %H:%M:%S")

Emm… datetime.date doesn’t seem to have such a method. Perhaps because it does not support timezone.

1.2 JSON Decoder

Then, correspondingly, when you use json.loads, you need a custom decoder. However, what is different is that we need to override its object_hook to tell it to try convert str into datetime object. (It seems, in previous version, it is a method, instead of a member.)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
def object_hook(obj):
    for key, value in obj.items():
        if isinstance(value, str):
            if re.match(r'^\d{4}-\d{2}-\d{2} \d\d:\d\d:\d\d$', value) is not None:
                try:
                    obj[key] = datetime.datetime.strptime(value, '%Y-%m-%d %H:%M:%S')
                except ValueError:
                    pass
            if re.match(r'^\d{4}-\d{2}-\d{2}$', value) is not None:
                try:
                    _date = datetime.datetime.strptime(value, '%Y-%m-%d')
                    obj[key] = datetime.date(_date.year, _date.month, _date.day)
                except ValueError:
                    pass
    return obj

After you spot a str member, you should first check the format of it. Because strptime(value, '%Y-%m-%d %H:%M:%S') will also treat 2023-06-20 as a valid one and return 2023-06-20 00:00:00, which will leave no chance for datetime.date.

And if you find out that the str should be a datetime.date, you still have to parse it with datetime.datetime, because the other doesn’t support this method. Then, you need to convert it do datetime.date manually.

2. Serialization

Then, with JSONEncoder, you can simply serialize a object into JSON string.

1
2
3
4
5
def serialize(obj) -> str:
    try:
        return json.dumps(obj, cls=AdvancedEncoder)
    except Exception:
        raise JsonSerializeException("Failed to serialize", obj)

Serialization related exceptions will be talked about at the end, since they are not the main topic.

If you don’t want to serialize a object into raw string, you can also serialize it into a dict. Here I use deserialize that will be elaborated later.

1
2
def serialize_as_dict(obj):
    return deserialize(serialize(obj))

3. Deserialization

3.1 Deserialization

Compared to serialization, deserialization got one more problem - how to deserialize a JSON object into a object with desired class type? And this is the key point of this article.

Then, the deserialization can be implemented as such. If cls is assigned, it will try to convert JSON string or object into the given class, and raise exception if type mismatch. Otherwise, it will simply return a dict object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
def deserialize(obj, cls=None):
    if isinstance(obj, dict):
        dict_obj = obj
    else:
        try:
            dict_obj = json.loads(obj, cls=AdvancedDecoder)
        except JSONDecodeError as e:
            raise JsonDeserializeException(f"Failed to deserialize!\n\t{e}", obj)

    if cls is None:
        return dict_obj

    try:
        _check_type(dict_obj, cls)
        obj = _construct_cls(dict_obj, cls)
    except AttributeError as e:
        raise JsonDeserializeException(f"Type mismatch, should be {cls.__name__}\n\t{e}", obj)

    return obj

There are two functions that play an important role here, _check_type and _construct_cls. They make sure the JSON object strictly match the given class, and try to build such a class from the JSON object. The implementation of them may be a little hard to understand, though. 😣

3.2 Check Type

Well, although we can just construct class directly, and raise exception if anything wrong happens, we check the type first to give more detailed information if type mismatch occurs. The principle of this process is to convert target class into dict first, and then compare all fields recursively. You just need to pay more attention to dict and list.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
def _check_type(src, cls):
    __check_type(src, serialize_as_dict(cls()))
    
def __check_type(src, model):
    t = type(model)
    if issubclass(t, dict):   # can be simpler as t == dict
        __check_type_dict(src, model)
    elif issubclass(t, list): # can be simpler as t == list
        __check_type_list(src, model)
    else:
        # So ugly
        if not isinstance(src, t) and not isinstance(model, datetime.date):
            hint = f"Value '{src}' type mismatch:\n\t"
            hint += f"Expected: {type(model)}\n\t"
            hint += f"     Got: {type(src)}"
            raise AttributeError(hint)

And here is how exactly dict and list are handled here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
def __check_type_dict(src, model):
    if not isinstance(src, dict):
        hint = f"Type mismatch, '{src}' should be 'dict'"
        raise AttributeError(hint)

    if not src.keys() == model.keys():
        hint = "Attribute set does not match:\n\t"
        hint += f"Expected: {model.keys()}\n\t"
        hint += f"     Got: {src.keys()}"
        raise AttributeError(hint)

    for key in model.keys():
        __check_type(src[key], model[key])


def __check_type_list(src, model):
    if not isinstance(src, list):
        hint = f"Type mismatch, '{src}' should be 'list'"
        raise AttributeError(hint)
    try:
        m = model[0]
    except IndexError:
        raise AttributeError(f"Missing default value for '{src}'")
    for v in src:
        __check_type(v, m)

Now, you can check if a dict is exactly a desired class or not.

3.3 Construct Class

After you check the type, you can go build a class from it. This is a little tricky, and was buggy. I refined it many times, and I guess it now should work in most cases. At least, no bug encountered ever after.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
def _construct_cls(src, cls):
    try:
        return __construct_cls(src, cls, cls())
    except Exception as e:
        raise AttributeError(f"Unexpected error: '{e}'")

def __construct_cls(src, cls, model):
    if issubclass(cls, list):
        if not isinstance(src, list):
            return None
        obj = []
        try:
            _type = type(model[0])
        except IndexError:
            raise AttributeError(f"Missing default value for '{cls}'")
        for v in src:
            obj.append(__construct_cls(v, _type, model[0]))
    elif isinstance(src, cls):
        obj = src
    else:
        if not isinstance(src, dict):
            return None
        obj = cls()
        for (k, v) in model.__dict__.items():
            obj.__dict__[k] = __construct_cls(src.get(k), type(v), v)

    return obj

So… again, Python object is just a dict with some extra fields. Emm… Hope it could make it easier for you to understand the code. 🥺

4. Exception Declaration

At last, I present to you the definition of custom exceptions that I used, to fit the last piece.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class JsonException(Exception):
    def __init__(self, msg):
        self.msg = msg

    def __str__(self):
        return f"JSON Error: {self.msg}"


class JsonSerializeException(JsonException):
    def __init__(self, msg, obj):
        super().__init__(msg)
        self.obj = obj

    def __str__(self):
        _str = super().__str__()
        _str += f"\n\tOn object: {'None' if self.obj is None else self.obj.__dict__}"
        return _str
    
    
class JsonDeserializeException(JsonException):
    def __init__(self, msg, obj):
        super().__init__(msg)
        self.obj = obj

    def __str__(self):
        _str = super().__str__()
        _str += f"\n\tOn string: {'None' if self.obj is None else self.obj}"
        return _str

So, this is it. And… I guess that Python is not that diabolical. It can be convenient sometimes. Only, some times. 😶‍🌫️