Protocols in Python: Why You Need Them

Python 3.8, released in October 2019, introduced a multitude of enhancements that excited developers around the globe. Among the standout features are assignment expressions and positional-only arguments. However, one of the most powerful yet underappreciated additions is Python protocols, also known as static duck typing. But what exactly are Python protocols, and how can they enhance your coding practices?

In order to give you a good overview of where protocols fit in, and why they are useful, I’ll first discuss the following subjects:

• Dynamic versus Static Typing
• Type Hints
• ABCs
• And finally: Protocols

Dynamic versus Static Typing and Static Type Checker

Python is a dynamically typed language. What does that mean?

Firstly: type declarations are not required. I can define the following function without ever specifying what types I expect the arguments to be, nor do I have to name a return type:

def my_function(a, b, c):
    return a + b - c  

COPY

Python

Copy

Secondly: types are handled – and checked – at runtime. I can run my_function with either integers, floats or a mix of both as input. The return type depends on the input:

result = my_function(5, 3, 2)
# type(result) -> int

result = my_function(5.1, 3, 2)
# type(result) -> float

COPY

Python

Copy

Comparing this to C, which is a statically typed language, we see we have to provide type declarations:

int my_function(int a, int b, int c) { return a + b - c; }

COPY

C

Copy

Providing any other type would be illegal. The following would not compile:

int result = my_function(5.1, 3, 2); 

COPY

C

Copy

That is a benefit of statically typed language: types are checked at compile time, so you can not run into any issues with types at runtime. In Python you may encounter issues at runtime that you would never have with a statically typed language. On the other hand, dynamically typed languages are more flexible when it comes to types that are accepted. And they do not require type declarations, which is great for lazy programmers.

Duck Typing

Dynamic typing is also called duck typing, because

If it walks like a duck and it quacks like a duck, then it must be a duck.

Or in other words: if any object has the required functionality, then we should accept it as an argument.

For example, suppose that we have a class named Duck, and this class can walk and quack:

class Duck:
    def walk(self):
        ...

    def quack(self):
        ...   

COPY

Python

Copy

Then we can make an instance of this class and make it walk and quack:

duck = Duck()
duck.walk()
duck.quack() 

Python

Now if we have a class Donkey that can walk, but it can not quack:

class Donkey:
    def walk(self):
        ...

Python

Then if we try to make an instance of the donkey walk and quack:

duck = Donkey()
duck.walk()
duck.quack() 

COPY

Python

Copy

We will get an >> AttributeError: ‘Donkey’ object has no attribute ‘quack’. Note that we only get this at runtime!

However, we can replace duck with any other class that can walk and can quack. For example:

class ImpostorDuck:
    def walk(self):
        ...

    def quack(self):
        not_quite_quacking()

duck = ImpostorDuck()
duck.walk()
duck.quack() 

COPY

Python

Copy

So, wrapping up: Python is a dynamically typed language, which is great because it gives lots of flexibility and type declarations are not required. But no type checking happens except at runtime, which can lead to unexpected issues. Which leads us to…

Type Hints

Type Hints, or optional static typing, were introduced in Python 3.5 to overcome this downside. It lets you optionally specify types of arguments and return values, which can then be checked by a static type checker such as mypy.

The typing module plays a crucial role in defining type hints and protocols, allowing developers to specify the types of variables, function arguments, and return values.

For example, suppose we have a type Duck that can swim and eat bread: class Duck: def eat_bread(self): …

def swim(self):
    ... 

Python

We can then define a function feed_bread that makes a Duck eat bread. We can specify the type of the argument to be of type Duck def feed_the_duck(duck: Duck): duck.eat_bread()

duck = Duck() feed_the_duck(duck) COPY

Python

Copy

Now trying to feed bread to a Monkey, for example, will not work: class Monkey: def eat_bananas(self): …

def climb_tree(self):
    ...

monkey = Monkey() feed_the_duck(monkey) Python

At runtime, this will give you: >> AttributeError: ‘Monkey’ object has no attribute ‘eat_bread’.

But mypy can spot issues like these before you run your code. In this case, it will tell you: error: Argument 1 to "feed_the_duck" has incompatible type "Monkey"; expected "Duck" These type hints can make your life as developer much easier, but they aren’t perfect. For example, if we want to make feed_bread more generic such that it can also accept other types of animal, we need to explicitly list all accepted types: from typing import Union

class Pig: def eat_bread(self): pass

def feed_bread(animal: Union[Duck, Pig]): animal.eat_bread() Python

And another downside: if you use code as above that is provided by an external package that is not under your control (let’s suppose it is called animals), you can not use it for your own types. For example, my baby son Mees’ favourite activities include eating bread and drinking milk: animals feed_bread

class Mees: def eat_bread(self): pass

def drink_milk(self):
    pass 

mees Mees() feed_breadmees COPY

Python

Copy

At runtime the above code will work perfectly fine, but mypy will complain:

error: Argument 1 to “feed_bread” has incompatible type “Mees”; expected “Union[Duck, Pig]” If we do not have control over the animals package, there is nothing be can do about this – except tell mypy to ignore the offending line.

So, wrapping up: type hints are great because they give you the option to have static type checking. Still there are no obligations to add type declarations, but if you do, you get some of the benefits of a statically typed language. But the inability to adapt type hints of imported code gives conflicts between the dynamic typing nature of Python and the static type hints. Static analysis tools like mypy can help by verifying type correctness and detecting issues before runtime.

ABCs and Abstract Methods

Abstract Base Classes take away some of the pain of the conflict described above. As the name says, they are base classes-classes that you are supposed to inherit from- but they can not be instantiated. They are used to define the interface of what the subclasses of the ABC should look like.

For example (and forgive me for assuming that all animals can walk): class Animal(metaclass=ABCMeta): @abstractmethod def walk(self): pass # Needs implementation by subclass COPY

Python

Copy

Instantiating this class is impossible: my_animal = Animal() will yield >> TypeError: Can’t instantiate abstract class Animal with abstract methods walk.

However, if we define a subclass, we can instantiate it: class Duck(Animal): def walk(self): ..

duck = Duck() https://essentials.xebia.com/assertions/ isinstance(duck, Animal) # <-- True COPY

Python

Copy

For a more practical example, one may create an ABC called EatsBread that defines that its subclasses can indeed eat bread (or, in other words, they must have a method with the signature eat_bread(self)): from abc import ABCMeta, abstractmethod

class EatsBread(metaclass=ABCMeta): @abstractmethod def eat_bread(self): pass

class Duck(EatsBread): def eat_bread(self): ..

class Pig(EatsBread): def eat_bread(self): ...

def feed_bread(animal: EatsBread): animal.eat_bread() Now if I were to use this implementation of feed_bread in my code of Mees – I can make Mees a subclass of EatsBread and all will be fine: from animals import EatsBread, feed_bread

class Mees(EatsBread): def eat_bread(self): ...

def drink_milk(self):
    ...
feed_bread(Mees()) # <-- OK at runtime and for mypy Python

Although this is much better – this still is not perfect. Often base classes are not easily exposed, meaning I have to have ugly imports to get what I need: from animals import feed_bread from animals.base.eats import EatsBread Python

In addition you have to either inherit from the base class (or explicitly register your class as a subclass, e.g. EatsBread.register(Mees)) for this to work – which is not as nice as the implicit behaviour of duck typing.

And still there can be situations which would not quite work. Suppose we use two external packages:

From package animals: class Animal(metaclass=ABCMeta): @abstractmethod def walk(self): pass

class Dog(Animal): def walk(self): …

def walk_animal(animal: Animal): animal.walk() Python

And from package llamas: class Llama: def walk(self): … COPY

Python

Copy

Now if you combine those in your code: from animals import walk_animal from llamas import Llama

llama = Llama() walk_animal(llama) # <– Not OK for mypy COPY

Python

Copy

This last line will work fine at runtime – but as Llama does not inherit from Animal, mypy will complain.

One can solve this by making Llama a virtual subclass of Animal: Animal.register(Llama)

llama = Llama() walk_animal(llama) # <– OK COPY

Python

Copy

But I would say that this is far from pretty.

Wrapping up once more: ABCs provide structure to your types, which is great. This means that type hints do not need updates for new subclasses. But we may still have issues combining classes from multiple packages. And all this typing -be it inheriting or as virtual subclasses- need to happen explicitly, which conflicts with the dynamic and implicit nature of dynamic typing. ABCs also impose limitations on multiple inheritance, which can be restrictive in some scenarios.

Using multiple protocols, a class can explicitly inherit from multiple protocols and resolve methods using normal MRO. This ensures that the type checker verifies all subtyping is correct, providing a more flexible solution compared to ABCs.

Python Protocols and Protocol Methods

And this is where protocols come in. A protocol is a special case of ABC which works implicitly: from typing import Protocol

class EatsBread(Protocol): eat_bread(self): pass

def feed_bread(animal: EatsBread): animal.eat_bread()

class Duck: def eat_breadself …

feed_breadDuck # <– OK In the above code, Duck is implicitly considered to be a subtype of EatsBread. There is no need to explicitly inherit from the protocol in the class definition. Any class that implements all attributes and methods defined in the protocol (with matching signatures) is seen as a subtype of that protocol.

So if we were to use the feed_bread function from package animals: from animals import feed_bread

class Mees: def eat_bread(self): …

def drink_milk(self):
    ...

feed_bread(Mees()) # <– OK Python

Here Mees is also implicitly a subtype of EatsBread. Again there is no need to explicitly specify that: as long as the signatures match it just works! This is why protocols are also called static duck typing. Additionally, class objects can be used for introspection and type hints to determine compatibility with protocol members.

Note that all this only works while type checking: not at runtime! If you do want this to work at runtime, you can use the runtime_checkable-decorator

The protocol class body is where all methods defined in a protocol are considered as protocol members, including normal and decorated methods. This is also where the bodies of protocol methods are type-checked.

Wrapping up one last time: protocols are awesome, because:

• There is no need to explicitly inherit from a protocol or register your class as a virtual subclass.
• There are no more difficulties combining packages: it works as long as the signatures match.
• We now have the best of both worlds: static type checking of dynamic types.

The protocol class implementation defines rules and guidelines for implementing protocol methods within a class, including the use of variable annotations.

Python Protocol class variables are defined within the class body of a protocol, and there is a distinction between protocol class variables and protocol instance variables.

Protocol classes behave similarly to regular classes in the context of static type checking and structural subtyping.

Python Protocol instance variables are defined and used within the protocol class body, with specific annotations and rules.

Python Protocol methods include static methods, class methods, and properties as allowed protocol members.

FAQs

What is the main advantage of using Python protocols?
Protocols provide a flexible way to perform static type checking without requiring explicit inheritance or registration, offering the benefits of both static and dynamic typing.

How do protocols differ from Abstract Base Classes (ABCs)?
Unlike ABCs, protocols do not require explicit inheritance. Any class that implements the required methods and attributes can be considered a subtype of the protocol.

Can Python protocols be used at runtime?
By default, protocols are only checked at compile-time. However, you can use the runtime_checkable decorator to make protocols enforceable at runtime.

What is static duck typing?
Static duck typing, enabled by protocols, allows Python to perform type checking based on the structure and methods of an object rather than its explicit class inheritance.

How do protocols improve code flexibility?
Protocols allow different classes to be compatible with functions or methods as long as they implement the required methods, promoting code reuse and flexibility.

Do protocols impact the performance of Python code?
Protocols primarily affect type checking during development and do not significantly impact the runtime performance of Python code.