3r33939. 3r3-31.
Python: metaprogramming in production. Part One
3r3903.  3r33939.
Many believe that metaprogramming in Python unnecessarily complicates the code, but if you use it correctly, you can quickly and elegantly implement complex design patterns. In addition, well-known Python frameworks such as Django, DRF, and SQLAlchemy use metaclasses to provide easy extensibility and simple code reuse. 3r3908. 3r3903.  3r33939.
In this article I’ll tell you why you shouldn’t be afraid to use metaprogramming in your projects and show you what tasks it is best for. You can learn more about metaprogramming capabilities on course Advanced Python . 3r3908. type , which has a rather interesting call signature (it will be discussed a little later). The same effect can be achieved if the display attribute __class__ at any object. 3r3908. 3r3903.  3r33939.

So, to create a function is a built-in class function . Let's see what we can do with it. To do this, take the workpiece from the built-in module 3r3357. types :

3r3903.  3r33939. 3r33880. 3r33867. from types import FunctionType
FunctionType
3r366. 3r33939. help (FunctionType)
3r33939. class function (object)
| function (code, globals[, name[, argdefs[, closure]]])
| 3r33939. | Create a function object from a code. 3r33939. | The optional name string overrides the name of the code object. 3r33939. | The optional argdefs tuple specifies the default argument values. 3r33939. | The bindings for free variables. 3r3886. 3r3903.  3r33939.

As we can see, any function in Python is an instance of the class described above. Let's now try to create a new function without resorting to its declaration through def . To do this, we need to learn how to create code objects using the built-in function of the function 3r3383. compile :

3r3903.  3r33939. 3r33880. 3r33867. # create a code object that prints the string "Hello, world!" 3r33939. code = compile ('print ("Hello, world!")', ' ', 'eval')
code
3r3394. at 0xdeadbeef, file " ", line 1>
# create a function by passing a code object to the constructor,
# global variables and function name 3r3393919. func = FunctionType (code, globals (), 'greetings')
func
at 0xcafefeed>
func .__ name__
'greetings'
func ()
Hello, world! 3r301901. 3r3886. 3r3903.  3r33939.

Excellent! With the help of meta-tools, we learned to create functions on the fly, but in practice this knowledge is rarely used. Now let's take a look at how class objects and instance objects of these classes are created:

3r3903.  3r33939. 3r33880. 3r33867. class User: pass
user = User ()
type (user)
3r3178. 3r33939. type (User)
3r3122. 3r301901. 3r3886. 3r3903.  3r33939.

It is quite obvious that the class is User used to create an instance of r3r3900. user It is much more interesting to look at the class type which is used to create the class itself User . This is where we turn to the second option for calling the built-in function 3r-300. type which is also a metaclass for any class in Python. A metaclass is, by definition, a class whose instance is another class. Metaclasses allow us to customize the class creation process and partially manage the class instance creation process. 3r3908. 3r3903.  3r33939.

According to the documentation, the second version of the signature type (name, bases, attrs) - returns a new data type or, in simple terms, a new class, with the attribute 3r-3900. name will become an attribute 3r300. __name__ for the returned class, bases - A list of parent classes will be available as 3r30000. __bases__ Well, attrs - a dict-like object containing all the attributes and methods of the class will go to 3r-3900. __dict . The principle of operation of the function can be described as a simple pseudocode in Python:

3r3903.  3r33939. 3r33880. 3r33867. type (name, bases, attrs)
~
class name (bases):
attrs 3r3886. 3r3903.  3r33939.

Let's see how you can, using only the 3r300 call. type , construct a brand new class:

3r3903.  3r33939. 3r33880. 3r33867. User = type ('User', (), {})
User
3r3178. 3r301901. 3r3886. 3r3903.  3r33939.

As you can see, we do not want to use the keyword class to create a new class, function type copes without it, now let's consider an example more difficult:

3r3903.  3r33939. 3r33880. 3r33867. class User:
def __init __ (self, name):
self.name = name
3r33939. class SuperUser (User):
"" "" Encapsulate domain logic to work with super users ". group_name = 'admin'
3r33939. @property
def login (self):
return f '{self.group_name} /{self.name}'. lower ()
3r33939. # Now let's create an analogue of the class SuperUser "dynamically"
CustomSuperUser = type (
# The name of the class
'SuperUser',
# The list of classes from which the new class is inherited
(Attributes, methods of the new class in the form of a dictionary
{
' __doc__ ':' Encapsulate domain users to super users, '
' group_name ':' admin ',
' login ': property (lambda self: f' {self.group_name} /{self.name} '. lower ()),
}
)
3r33939. assert SuperUser .__ doc__ == CustomSuperUser .__ doc__
assert SuperUser ('Vladimir'). login == CustomSuperUser ('Vladimir'). login 3r3886. 3r3903.  3r33939.

As you can see from the examples above, the description of classes and functions using keywords 3-333900. class and 3r300. def - this is just a syntactic sugar and any types of objects can be created with ordinary calls of built-in functions. And now, finally, let's talk about how you can use dynamic class creation in real projects. 3r3908. 3r3903.  3r33939. 3r33232. Dynamic creation of forms and validators 3r3903.  

Sometimes we need to validate information from the user or from other external sources according to a previously known data scheme. For example, we want to change the user login form from the admin panel — remove and add fields, change their validation strategy, etc. 3r3908. 3r3903.  3r33939.

To illustrate, let's try to dynamically create Django -form, the disclosure of which is stored in the following scheme json Format: 3r3908. 3r3903.  3r33939. 3r33880. 3r33900. {
"fist_name": {"type": "str", "max_length": 25},
"last_name": {"type": "str", "max_length": 30},
"age": {"type": "int", "min_value": 1? "max_value": 99}
} 3r3886. 3r3903.  3r33939.

Now, on the basis of the description above, we will create a set of fields and a new form using the function already known to us, 3r-300. type :

3r3903.  3r33939. 3r33880. 3r33867. import json
from django import forms
3r33939. fields_type_map = {
'Str': forms.CharField,
'int': forms.IntegerField,
}
3r33939. # form_description - our json with format description 3r3393919. deserialized_form_description: dict = json.loads (form_description)
form_attrs = {}
3r33939. # select the class of the object of the field in the form, depending on its type
for field_name, field_description in deserialized_form_description.items ():
field_class = fields_type_map[field_description.pop('type')]3r33939. form_attrs[field_name]= field_class (** field_description)
3r33939. user_form_class = type ('DynamicForm', (forms.Form,), form_attrs)
3r33939. form = user_form_class ({'age': 101})
form
3r33939. form.is_valid ()
False
form.errors
{'fist_name':['This field is required.'], 3r33939. 'last_name':['This field is required.'], 3r33939. 'age':['Ensure this value is less than or equal to 99.']} 3r3886. 3r3903.  3r33939.

Super! Now you can transfer the created form to a template and render it to the user. The same approach can be used with other frameworks for validation and data representation (3r3-33299. DRF Serializers , Marshmallow And others). 3r3908. 3r3903.  3r33939. 3r3306. Configurable through the creation of a new class metaclass 3r3903.  3r33939.

Above, we looked at the already “ready” metaclass 3-333900. type , but more often in the code you will create your own metaclasses and use them to configure the creation of new classes and their instances. In general, the “blank” metaclass looks like this:

3r3903.  3r33939. 3r33880. 3r33867. class MetaClass (type):
"" "3r3-3919. Description of accepted parameters: 3r3393919. 3r-3919. Mcs is a metaclass object, for example 3r3-3322.
Name is a string, the name of the class for which 3r31919 is used. This metaclass, for example" User "
parents, for example (SomeMixin, AbstractUser) 3-333919. attrs - dict-like object, stores the values ​​of attributes and methods of the class 3r-3919. cls - the created class, for example <__main__.User> 3r-3919. extra_kwargs - additional keyword arguments passed to the signature of the class
and kwargs are the arguments passed to the constructor of the
class when created but 3r31939. "" "
def __new __ (mcs, name, bases, attrs, ** extra_kwargs):
return super () .__ new __ (mcs, name, bases, attrs)
3r33939. def __init __ (cls, name, bases, attrs, ** extra_kwargs):
super () .__ init __ (cls)
3r33939. @classmethod
def __prepare __ (mcs, cls, bases, ** extra_kwargs):
return super () .__ prepare __ (mcs, cls, bases, ** kwargs)
3r33939. def __call __ (cls, * args, ** kwargs):
return super () .__ call __ (* args, ** kwargs) 3r3886. 3r3903.  3r33939.

To use this metaclass Class configuration. User , the following syntax is used:

3r3903.  3r33939. 3r33880. 3r33867. class User (metaclass = MetaClass):
3r33939. def __new __ (cls, name):
return super () .__ new __ (cls)
3r33939. def __init __ (self, name):
self.name = name 3r3886. 3r3903.  3r33939.

The most interesting is the order in which the Python interpreter invokes the metamethods of the metaclass at the time of creating the class itself:

3r3903.  3r33939.

1.  3r33939. 3r33846. The interpreter identifies and finds parent classes for the current class (if any). 3r33857.  3r33939. 3r33846. The interpreter defines the metaclass (

MetaClass In our case). 3r33857.  3r33939. 3r33846. The method r3r3900 is called. MetaClass .__ prepare__ - it must return a dict-like object in which the attributes and methods of the class will be written. After that, the object will be passed to method MetaClass .__ new__ through argument 3r300. attrs . We will talk about the practical use of this method a little later in the examples. 3r33857.  3r33939. 3r33846. The interpreter reads the body of the class User and generates parameters to pass them in metaclass MetaClass . 3r33857.  3r33939. 3r33846. The method r3r3900 is called. MetaClass .__ new__ - method-constructor, returns the created class object. C arguments 3r300. name , 3r300. bases and 3r300. attrs we already met when we passed them to the function 3r300. type , and about the parameter 3-333900. ** extra_kwargs we'll talk a little later. If the type of the argument is 3r30000. attrs was modified using 3r30000. __prepare__ then it needs to be converted to 3r30000. dict before passing to the 3r300 method call. super () 3r301901. . 3r33857.  3r33939. 3r33846. The method r3r3900 is called. MetaClass .__ init__ - initialization method, with which you can add additional attributes and methods to the class object. In practice, it is used in cases when metaclasses are inherited from other metaclasses, otherwise everything that can be done in 3r-300. __init__ It is better to do in 3r300. __new__3r3901. . For example, parameter 3r300. __slots__ can be set only 3r33430. in method 3r300. __new__3r3901. by writing it to the object 3r300. attrs . 3r33857.  3r33939. 3r33846. At this step, the class is considered to be created. 3r33857.  3r33939. 3r33490. 3r3903.  3r33939.

Now we create an instance of our class User and look at the call chain:

3r3903.  3r33939. 3r33880. 3r33867. user = User (name = 'Alyosha') 3r3886. 3r3903.  3r33939.  3r33939. 3r33846. At the time of the call 3r30000. User () the interpreter calls the method. MetaClass .__ call __ (name = 'Alyosha') to which the class object and the passed arguments are passed. 3r33857.  3r33939. 3r33846. 3r33900. MetaClass .__ call__ calls User .__ new __ (name = 'Alyosha') - the constructor method that creates and returns an instance of the class 3r30000. User 3r33857.  3r33939. 3r33846. Next MetaClass .__ call__ calls User .__ init __ (name = 'Alyosha') - initialization method that adds new attributes to the created instance. 3r33857.  3r33939. 3r33846. 3r33900. MetaClass .__ call__ returns the created and initialized instance of the class 3r30000. User . 3r33857.  3r33939. 3r33846. At this point, an instance of the class is considered to be created. 3r33857.  3r33939. 3r33490. 3r3903.  3r33939.

This description, of course, does not cover all the nuances of the use of metaclasses, but it is enough to start applying metaprogramming to implement some architectural patterns. Forward to the examples! 3r3908. 3r3903.  3r33939.

Abstract classes

3r3903.  3r33939.

And the very first example can be found in the standard library: 3r3502. ABCMeta - the metaclass allows you to declare any of our classes to be abstract and to force all of its heirs to implement predefined methods, properties and attributes, look at this:

3r3903.  3r33939. 3r33880. 3r33867. from abc import ABCMeta, abstractmethod
3r33939. class BasePlugin (metaclass = ABCMeta):
"" "
The attribute of the supported_formats class and the run method must be implemented in
In the heirs of this class, 3r3r1919." ""
@property
@abstractmethod
def supported_formats (self) -> list:
pass
3r33939. @abstractmethod
def run (self, input_dаta: dict):
pass 3r3886. 3r3903.  3r33939.

If all abstract methods and attributes are not implemented in the heir, then when we try to create an instance of the heir class, we will get 3r3009. TypeError :

3r3903.  3r33939. 3r33880. 3r33867. class VideoPlugin (BasePlugin):
3r33939. def run (self):
print ( 'Processing video ')
3r33939. plugin = VideoPlugin ()
# TypeError: Can't instantiate abstract class VideoPlugin
# With abstract methods supported_formats 3r3886. 3r3903.  3r33939.

Using abstract classes helps to fix the interface of the base class immediately and to avoid errors in future inheritance, for example, typos in the name of the overridden method. 3r3908. 3r3903.  3r33939. 3r33550. plug-in system with automatic registration 3r3903.  3r33939.

Quite often, metaprogramming is used to implement various design patterns. Almost any known framework uses metaclasses to create 3r33555. registry -objects. Such objects store references to other objects and allow them to be quickly received anywhere in the program. Consider a simple example of auto-registration of plug-ins for playing media files of various formats. 3r3908. 3r3903.  3r33939.

Implementation of the metaclass:

3r3903.  3r33939. 3r33880. 3r33867. class RegistryMeta (ABCMeta):
"" "3r33939. The metaclass that creates the registry from the classes of heirs. 3r-33919. The registry stores links like" file format "->" plugin class "3r3393919." "" 3r3393919. _registry_formats = {}
3r33939. def __new __ (mcs, name, bases, attrs):
cls: 'BasePlugin' = super () .__ new __ (mcs, name, bases, attrs)
3r33939. # do not handle abstract classes (BasePlugin)
if inspect.isabstract (cls):
return cls
3r33939. for media_format in cls.supported_formats:
if media_format in mcs._registry_formats:
raise ValueError (f'Format {media_format} is already registered ')
3r33939. # save the link to the plugin in the registry 3r3393919. mcs._registry_formats[media_format]= cls
3r33939. return cls
3r33939. @classmethod
def get_plugin (mcs, media_format: str):
try:
return mcs._registry_formats[media_format]3r33939. except KeyError:
raise RuntimeError (f'Plugin is not defined for {media_format} ')
3r33939. @classmethod
def show_registry (mcs):
from pprint import pprint
pprint (mcs._registry_formats) 3r3886. 3r3903.  3r33939.

And here is the plugins themselves, implementation BasePlugin take the previous example:

3r3903.  3r33939. 3r33880. 3r33867. class BasePlugin (metaclass = RegistryMeta):
3r33939. 3r33939. class VideoPlugin (BasePlugin):
supported_formats =['mpg', 'mov']3r33939. def run (self):
3r33939. class AudioPlugin (BasePlugin):
supported_formats =['mp3', 'flac']3r33939. def run (self):
3r3886. 3r3903.  3r33939.

After the interpreter executes this code, 4 formats and 2 plug-ins that can handle these formats will be registered in our registry: 3r3908. 3r3903.  3r33939. 3r33880. 3r33867. RegistryMeta.show_registry ()
{'flac': , 3r33939. 'mov': , 3r33939. 'mp3': , 3r33939. 'mpg': }
plugin_class = RegistryMeta.get_plugin ('mov')
plugin_class
3r33939. plugin_class (). run ()
Processing video 3r3393901. 3r3886. 3r3903.  3r33939.

Here it is worth noting another interesting nuance of working with metaclasses, thanks to the unobvious method resolution order, we can call the method. show_registry not only in class 3r300. RegistyMeta , but also in any other class the metaclass of which it is:

3r3903.  3r33939. 3r33880. 3r33867. AudioPlugin.get_plugin ('avi')
# RuntimeError: Plugin is not found for avi 3r3886. 3r3903.  3r33939. 3r3663. Using attribute names as metadata 3r3903.  3r33939.

Using metaclasses, you can use class attribute names as metadata for other objects. Can not understand anything? But I’m sure you’ve already seen this approach many times, such as the declarative declaration of model fields in Django:

3r3903.  3r33939. 3r33880. 3r33867. class Book (models.Model):
title = models.Charfield (max_length = 250) 3r3886. 3r3903.  3r33939.

In the example above, r3r3900. title Is the name of the Python identifier, it is also used for the name of the column in table 3r-3900. book , although we did not explicitly indicate this anywhere. Yes, such “magic” can be implemented using metaprogramming. Let us, for example, implement an application error transfer system on the front-end, so that each message has readable code that can be used to translate a message into another language. So, we have a message object that can be converted to 3r30000. json :

3r3903.  3r33939. 3r33880. 3r33867. class Message:
def __init __ (self, text, code = None):
self.text = text
self.code = code
3r33939. def to_json (self):
return json.dumps ({'text': self.text, 'code': self.code}) 3r3886. 3r3903.  3r33939.

All our error messages will be stored in a separate “namespace”:

3r3903.  3r33939. 3r33880. 3r33867. class Messages:
not_found = Message ('Resource not found')
bad_request = Message ('Request body is invalid')
3r33939. 3r33939. Messages.not_found.to_json ()
{"text": "Resource not found", "code": null} 3r3886. 3r3903.  3r33939.

Now we want code became not 3r3009. null , and 3r300. not_found , for this we write the following metaclass:

3r3903.  3r33939. 3r33880. 3r33867. class MetaMessage (type):
3r33939. def __new __ (mcs, name, bases, attrs):
for attr, value in attrs.items ():
# pass through all the attributes described in the class with the type Message
# and replace the code field with the attribute name 3r3393919. # (If the code is not set in advance)
if isinstance (value, Message) and value.code is None:
value.code = attr
3r33939. return super () .__ new __ (mcs, name, bases, attrs)
3r33939. class Messages (metaclass = MetaMessage):
3r33939. 3r3886. 3r3903.  3r33939.

Let's see how our messages look now:

3r3903.  3r33939. 3r33880. 3r33867. Messages.not_found.to_json ()
{"text": "Resource not found", "code": "not_found"}
Messages.bad_request.to_json ()
{"text": "Request body is invalid", "code": "bad_request"} 3r3886. 3r3903.  3r33939.

What you need! Now you know what to do so that by the format of the data you can easily find the code that processes them. 3r3908. 3r3903.  3r33939. 3r33762. Caching metadata about the class and its heirs 3r3903.  3r33939.

Another frequent case is caching of any static data at the stage of class creation, in order not to waste time on their calculation while the application is running. In addition, some data can be updated when creating new instances of classes, for example, the count of the number of objects created. 3r3908. 3r3903.  3r33939.

How can this be used? Suppose you are developing a framework for building reports and tables, and you have such an object:

3r3903.  3r33939. 3r33880. 3r33867. class Row (metaclass = MetaRow):
name: str
age: int 3r3393919. 3r33939. def __init __ (self, ** kwargs):
self.counter = None
for attr, value in kwargs.items ():
setattr (self, attr, value)
3r33939. def __str __ (self):
out =[self.counter]3r33939. 3r33939. # The __header__ attribute will be dynamically added in the metaclass
for name in self .__ header__[1:]:
out.append (getattr (self, name, 'N ​​/A'))
3r33939. return '| '.join (map (str, out)) 3r3886. 3r3903.  3r33939.

We want to save and increase the counter when creating a new series, and also want to generate the header of the resulting table in advance. Metaclass to the rescue! 3r3908. 3r3903.  3r33939. 3r33880. 3r33867. class MetaRow (type):
# global counter of all created3r3-3919 rows. row_count = 0
3r33939. def __new __ (mcs, name, bases, attrs):
cls = super () .__ new __ (mcs, name, bases, attrs)
3r33939. # Cache a list of all the fields in the row alphabetically sorted
cls .__ header__ =['№']+ sorted (attrs['__annotations__']keys ())
return cls
3r33939. def __call __ (cls, * args, ** kwargs):
# creating a new series takes place here
row: 'Row' = super () .__ call __ (* args, ** kwargs) 3r3393919. # increment the global counter
cls.row_count + = 1
3r33939. # set the current row number to
row.counter = cls.row_count
return row 3r3886. 3r3903.  3r33939.

Here you need to clarify 2 things:

3r3903.  3r33939.  3r33939. 3r33846. Have class 3r300. Row no class attributes named r3r3900. name and 3r300. age - this is 3r33838. type annotations , so they are not in the keys of the dictionary attrs , and to get the list of fields, we use the attribute of the class 3r3009. __annotations__ . 3r33857.  3r33939. 3r33846. Operation 3r300. cls.row_count + = 1 should have misled you: how is that? After all, cls This is class 3r300. Row it does not have the 3r300 attribute. row_count . That's right, but as I explained above - if the created class does not have an attribute or method that they are trying to call, then the interpreter goes further along the chain of base classes - if they don’t exist, the metaclass is searched. In such cases, in order not to confuse anyone, it is better to use another record: MetaRow.row_count + = 1 . 3r33857.  3r33939. 3r33859. 3r3903.  3r33939.

See how elegantly you can now display the entire table:

3r3903.  3r33939. 3r33880. 3r33867. rows =[
Row(name='Valentin', age=25),
Row(name='Sergey', age=33),
Row(name='Gosha'),
]3r33939. 3r33939. print ('|' .join (Row .__ header__))
for row in rows:
print (row) 3r3886. 3r3903.  3r33939. 3r33880. 3r33900. No. | age | name
1 | 25 | Valentin
2 | 33 | Sergey 3r3393919. 3 | N /A | Gosha 3r3886. 3r3903.  3r33939.

By the way, the display and work with the table can be encapsulated in a separate class Sheet . 3r3908. 3r3903.  3r33939. 3r33895. To be continued
3r3903.  3r33939.

In the next part of this article, I will explain how to use metaclasses to debug the code of your application, how to parameterize the creation of a metaclass, and show the main examples of using the 3-333900 method. __prepare__ . Stay tuned! 3r3908. 3r3903.  3r33939.
In more detail about metaclasses and descriptors in Python, I will talk in the framework of the intensive Advanced Python . 3r3908. 3r33915. 3r33939. 3r33939. 3r33912. ! function (e) {function t (t, n) {if (! (n in e)) {for (var r, a = e.document, i = a.scripts, o = i.length; o-- ;) if (-1! == i[o].src.indexOf (t)) {r = i[o]; break} if (! r) {r = a.createElement ("script"), r.type = "text /jаvascript", r.async =! ? r.defer =! ? r.src = t, r.charset = "UTF-8"; var d = function () {var e = a.getElementsByTagName ("script")[0]; e.parentNode.insertBefore (r, e)}; "[object Opera]" == e.opera? a.addEventListener? a.addEventListener ("DOMContentLoaded", d,! 1): e.attachEvent ("onload", d ): d ()}}} t ("//mediator.mail.ru/script/2820404/"""_mediator") () (); 3r33939. 3r33939. 3r33915. 3r33939. 3r33939. 3r33939. 3r33939.