Encoders

Encoders are basically converters between datastructures and json. Its main use is in response were different return values are automatically parsed into json objects.

For understanding what they are doing, it is helpful to go through the manual usage

Manual usage

Encoders provide two public methods:

• json_encode
• apply_structure

json_encode

json_encode is basically an enhanced json.dumps. It provides via the encoder converters which parse additional types to json. By default only a simplification is done, therefore the json string is deserialized again so a simple datastructure is returned.

from lilya.encoders import json_encode

json_string = json_encode({"hello": "world"}, post_transform_fn=None)
# or
json_string = json_encode({"hello": "world"}, post_transform_fn=lambda x: x)

apply_structure

apply_structure is basically the inverse of json_encode. It assumes it is known from which structure a value was serlialized. If one encoder matches via is_type_structure (for Encoder by default isclass(structure) and issubclass(structure, self.__type__)) the encoder is used. First is checked via is_type if the value is already converted and if yes simply returned. Otherwise the value is molded via the structure in an instance and returned.

from typing import Any

from dataclasses import dataclass

from lilya.encoders import apply_structure, json_encode

@dataclass
class Foo:
    a: int
    b: int

simplified = json_encode(Foo(a=3, b=5))
# dict {"a": 3, "b": 5}
foo = apply_structure(Foo, simplified)
# now a Foo object again
assert foo == Foo(a=3, b=5)

Default Encoders

In order to understand how to serialize a specific object into json, Lilya has some default encoders that evaluates when tries to guess the response type.

• DataclassEncoder - Serialises dataclass objects.
• NamedTupleEncoder - Serialises NamedTuple objects.
• ModelDumpEncoder - Serialises objects by calling its model_dump method. This allows serializing pydantic objects out of the box.
• EnumEncoder - Serialises Enum objects.
• PurePathEncoder - Serializes PurePath objects.
• DateEncoder - Serializes date and datetime objects.
• StructureEncoder - Serializes more complex data types which implement Iterable.

What if a brand new encoder is needed and it is not natively supported by Lilya? Well, building a custom encoder is extremly easy and possible.

All default encoders are also implementing the molding protocol.

Build a custom encoder

As mentioned before, Lilya has default encoders that are used to transform a response into a json serializable response.

To build a custom encoder you must implement the EncoderProtocol or MoldingProtocol. You can use the Encoder helper class from Lilya for that and provide the serialize() and/or encode function where it applies the serialisation process of the encoder type. If the encoder should also be able to deserialize a value in an provided object, you need additionally the method: encode and maybe the method is_type_structure.

Then you must register the encoder for Lilya to use it.

When defining an encoder the def is_type(self, value: Any) -> bool: or (Encoder helper class only)__type__ must be declared or overridden.

When in an Encoder subclass the __type__ is properly declared, the default is_type and is_type_structure (MoldingProtocol) will evaluate the object against the type and return True or False. __type__ can be a single type or a tuple of types.

This is used internally to understand the type of encoder that will be applied to a given object.

Warning

If you are not able to provide the __type__ for any reason and you just want to override the default evaluation process, simple use EncoderProtocol, override the is_type() and apply your custom logic there.

E.g.: In Python 3.8, for a Pydantic BaseModel if passed in the __type__, it will throw an error due to Pydantic internals, so to workaround this issue, you can simply override the is_type() and apply the logic that validates the type of the object and returns a boolean.

Example

Create and register an encoder that handles msgspec.Struct types.

from typing import Any

import msgspec
from msgspec import Struct

from lilya.encoders import Encoder, register_encoder


class MsgSpecEncoder(Encoder):
    __type__ = Struct

    def serialize(self, obj: Any) -> Any:
        """
        When a `msgspec.Struct` is serialised,
        it will call this function.
        """
        return msgspec.json.decode(msgspec.json.encode(obj))

    def encode(
        self,
        structure: Any,
        obj: Any,
    ) -> Any:
        return msgspec.json.decode(obj, type=structure)


# A normal way
register_encoder(MsgSpecEncoder())

# As alternative
register_encoder(MsgSpecEncoder)

Simple right? Because now the MsgSpecEncoder is registered, you can simply do this in your handlers and return directly the msgspec.Struct object type.

from msgspec import Struct

from lilya.routing import Path


class User(Struct):
    name: str
    email: str


def msgspec_struct():
    return User(name="lilya", url="example@lilya.dev")

Design specific custom encoders

Lilya being 100% pure python and not tight to any particular validation library allows you to design custom encoders that are later used by Lilya responses.

Ok, this sounds a bit confusing right? I bet it does so let us go slowly.

Imagine you want to use a particular validation library such as Pydantic, msgspec or even attrs or something else at your choice.

You want to make sure that if you return a pydantic model or a msgspec Struct or even a define attr class.

Let us see how it would look like for all of them.

For pydantic

Nothing required anymore. Works out of the box thanks to the ModelDumpEncoder. But we can do an instance check instead:

from __future__ import annotations

from typing import Any

from pydantic import BaseModel

from lilya.encoders import Encoder, register_encoder


class PydanticEncoder(Encoder):
    __type__ = BaseModel
    # optional a name can be provided, so same named encoders are removed
    name = "ModelDumpEncoder"

    # is_type and is_type_structure are provided by Encoder.
    # checked is the type provided by __type__.

    def serialize(self, obj: BaseModel) -> dict[str, Any]:
        return obj.model_dump()

    def encode(self, structure: type[BaseModel], value: Any) -> Any:
        return structure(**value)


# A normal way
register_encoder(PydanticEncoder())

# As alternative
register_encoder(PydanticEncoder)

For msgspec Struct

from typing import Any

import msgspec
from msgspec import Struct

from lilya.encoders import Encoder, register_encoder


class MsgSpecEncoder(Encoder):
    __type__ = Struct

    def serialize(self, obj: Any) -> Any:
        """
        When a `msgspec.Struct` is serialised,
        it will call this function.
        """
        return msgspec.json.decode(msgspec.json.encode(obj))

    def encode(
        self,
        structure: Any,
        obj: Any,
    ) -> Any:
        return msgspec.json.decode(obj, type=structure)


# A normal way
register_encoder(MsgSpecEncoder())

# As alternative
register_encoder(MsgSpecEncoder)

For attrs

from typing import Any

from attrs import asdict, has

from lilya.encoders import Encoder, register_encoder


class AttrsEncoder(Encoder):

    def is_type(self, value: Any) -> bool:
        """
        You can use this function instead of declaring
        the `__type__`.
        """
        return has(value)

    def serialize(self, obj: Any) -> Any:
        return asdict(obj)

    def encode(self, structure: type[Any], obj: Any) -> Any:
        if isinstance(obj, dict):
            return structure(**obj)
        return structure(*obj)



# A normal way
register_encoder(AttrsEncoder())

# As alternative
register_encoder(AttrsEncoder)

Easy and powerful, right? Yes.

Do you understand what does this mean? Means you can design any encoder at your choice using also any library of your choice as well.

The flexibility of Lilya allows you to be free and for Lilya not to be tight to any particular library.

Tip

You can replace other Encoders by providing a name attribute. By default all encoders use their class-name as name.

Custom encoders and responses

After the custom encoders in the examples are created, this allows to do something like this directly.

from attrs import define
from msgspec import Struct
from pydantic import BaseModel

from lilya.apps import Lilya
from lilya.routing import Path


class User(BaseModel):
    name: str
    age: int


class Item(Struct):
    name: str
    age: int


@define
class AttrItem:
    name: str
    age: int


def pydantic_response():
    return User(name="lilya", age=24)


def pydantic_response_list():
    return [User(name="lilya", age=24)]


def msgspec_struct():
    return Item(name="lilya", age=24)


def msgspec_struct_list():
    return [Item(name="lilya", age=24)]


def attrs_response():
    return AttrItem(name="lilya", age=24)


def attrs_response_list():
    return [AttrItem(name="lilya", age=24)]


app = Lilya(
    routes=[
        Path("/pydantic", pydantic_response),
        Path("/pydantic-list", pydantic_response_list),
        Path("/msgspec", msgspec_struct),
        Path("/msgspec-list", pydantic_response_list),
        Path("/attrs", attrs_response),
        Path("/attrs-list", attrs_response_list),
    ]
)

Custom encoders and the make_response

Well, here its where the make_response helps you. The make_response will generate a JSONResponse by default and when you return a custom encoder type, there are some limitations to it.

For example, what if you want to return with a different status_code? Or even attach a task to it?

The custom encoder does not handle that for you but the make_response does!

Let us see how it would look like now using the make_response.

from attrs import define
from msgspec import Struct
from pydantic import BaseModel

from lilya import status
from lilya.apps import Lilya
from lilya.responses import make_response
from lilya.routing import Path


class User(BaseModel):
    name: str
    age: int


class Item(Struct):
    name: str
    age: int


@define
class AttrItem:
    name: str
    age: int


def pydantic_response():
    data = User(name="lilya", age=24)
    return make_response(
        data,
        status_code=status.HTTP_200_OK,
    )


def pydantic_response_list():
    data = [User(name="lilya", age=24)]
    return make_response(
        data,
        status_code=status.HTTP_201_CREATED,
        background=...,
        headers=...,
    )


def msgspec_struct():
    return make_response(Item(name="lilya", age=24))


def msgspec_struct_list():
    return make_response(
        [Item(name="lilya", age=24)],
        status_code=...,
    )


def attrs_response():
    return make_response(
        AttrItem(name="lilya", age=24),
        status_code=...,
    )


def attrs_response_list():
    return make_response(
        [AttrItem(name="lilya", age=24)],
        status_code=...,
    )


app = Lilya(
    routes=[
        Path("/pydantic", pydantic_response),
        Path("/pydantic-list", pydantic_response_list),
        Path("/msgspec", msgspec_struct),
        Path("/msgspec-list", pydantic_response_list),
        Path("/attrs", attrs_response),
        Path("/attrs-list", attrs_response_list),
    ]
)

Use with custom json encoder

Sometimes you might want a more performant json parser that is not the default python json built-in, for example, the orjson json serializer/deserializer.

This is no problem for Lilya:

from lilya.encoders import json_encode
import orjson

# orjson serializes to bytes, so apply str
json_string = json_encode({"hello": "world"}, json_encode_fn=orjson.dumps, post_transform_fn=str)
# or for simplifying
json_simplified = json_encode({"hello": "world"}, json_encode_fn=orjson.dumps, post_transform_fn=orjson.loads)

Or the make_response example with orjson

from attrs import define
from msgspec import Struct
from pydantic import BaseModel

import orjson
from lilya import status
from lilya.apps import Lilya
from lilya.responses import make_response
from lilya.routing import Path


class User(BaseModel):
    name: str
    age: int


class Item(Struct):
    name: str
    age: int


@define
class AttrItem:
    name: str
    age: int


def pydantic_response():
    data = User(name="lilya", age=24)
    return make_response(
        data,
        status_code=status.HTTP_200_OK,
        json_encode_extra_kwargs={
            "json_encode_fn": orjson.dumps, "post_transform_fn": orjson.loads
        }
    )


def pydantic_response_list():
    data = [User(name="lilya", age=24)]
    return make_response(
        data,
        status_code=status.HTTP_201_CREATED,
        background=...,
        headers=...,
        json_encode_extra_kwargs={
            "json_encode_fn": orjson.dumps, "post_transform_fn": orjson.loads
        }
    )


def msgspec_struct():
    return make_response(Item(name="lilya", age=24))


def msgspec_struct_list():
    return make_response(
        [Item(name="lilya", age=24)],
        status_code=...,
        json_encode_extra_kwargs={
            "json_encode_fn": orjson.dumps, "post_transform_fn": orjson.loads
        }
    )


def attrs_response():
    return make_response(
        AttrItem(name="lilya", age=24),
        status_code=...,
        json_encode_extra_kwargs={
            "json_encode_fn": orjson.dumps, "post_transform_fn": orjson.loads
        }
    )


def attrs_response_list():
    return make_response(
        [AttrItem(name="lilya", age=24)],
        status_code=...,
        json_encode_extra_kwargs={
            "json_encode_fn": orjson.dumps, "post_transform_fn": orjson.loads
        }
    )


app = Lilya(
    routes=[
        Path("/pydantic", pydantic_response),
        Path("/pydantic-list", pydantic_response_list),
        Path("/msgspec", msgspec_struct),
        Path("/msgspec-list", pydantic_response_list),
        Path("/attrs", attrs_response),
        Path("/attrs-list", attrs_response_list),
    ]
)