12. Decorator and Descriptors

12.1. Decorators

Decorator is a function that creates a wrapper around another function. This wrapper adds some additional functionality to existing code. In this tutorial, various types of decorators are discussed.

12.1.1. Function inside the function and Decorator

Following is the example of function inside the function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# funcEx.py

def addOne(myFunc):
    def addOneInside(x):
        print("adding One")
        return myFunc(x) + 1
    return addOneInside

def subThree(x):
    return x - 3

result = addOne(subThree)

print(subThree(5))
print(result(5))

# outputs
# 2
# adding One
# 3

Above code works as follows,

  • Function ‘subThree’ is defined at lines 9-10, which subtract the given number with 3.
  • Function ‘addOne’ (Line 3) has one argument i.e. myFunc, which indicates that ‘addOne’ takes the function as input. Since, subThree function has only one input argument, therefore one argument is set in the function ‘addOneInside’ (Line 4); which is used in return statement (Line 6). Also, “adding One” is printed before returning the value (Line 5).
  • In line 12, return value of addOne (i.e. function ‘addOneInside’) is stored in ‘result’. Hence, ‘result’ is a function which takes one input.
  • Lastly, values are printed at line 13 and 14. Note that “adding One” is printed by the result(5) and value is incremented by one i.e. 2 to 3.

Another nice way of writing above code is shown below. Here (*args and **kwargs) are used, which takes all the arguments and keyword arguments of the function.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
# funcEx.py

def addOne(myFunc):
    def addOneInside(*args, **kwargs):
        print("adding One")
        return myFunc(*args, **kwargs) + 1
    return addOneInside

def subThree(x):
    return x - 3

result = addOne(subThree)

print(subThree(5))
print(result(5))

# outputs
# 2
# adding One
# 3

Now, in the below code, the return value of addOne is stored in the ‘subThree’ function itself (Line 12),

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# funcEx.py

def addOne(myFunc):
    def addOneInside(*args, **kwargs):
        print("adding One")
        return myFunc(*args, **kwargs) + 1
    return addOneInside

def subThree(x):
    return x - 3

subThree = addOne(subThree)

print(subThree(5))
# outputs
# adding One
# 3

Lastly, in Python, the line 12 in above code, can be replaced by using decorator, as shown in Line 9 of below code,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# funcEx.py

def addOne(myFunc):
    def addOneInside(*args, **kwargs):
        print("adding One")
        return myFunc(*args, **kwargs) + 1
    return addOneInside

@addOne
def subThree(x):
    return x - 3

print(subThree(5))
# outputs
# adding One
# 3

In this section, we saw the basics of the decorator, which we will be used in this tutorial.

12.1.2. Decorator without arguments

In following code, Decorator takes a function as the input and print the name of the function and return the function.

# debugEx.py

def printName(func):
    # func is the function to be wrapped
    def pn(*args, **kwargs):
        print(func.__name__)
        return func(*args, **kwargs)
    return pn

Next, put the printName function as decorator in the mathEx.py file as below,

# mathEx.py

from debugEx import printName

@printName
def add2Num(x, y):
    # add two numbers
    # print("add")
    return(x+y)

print(add2Num(2, 4))
help(add2Num)

Finally, execute the code and the name of each function will be printed before calculation as shown below,

$ python mathEx.py
add
6

Help on function pn in module debugEx:

pn(*args, **kwargs)
    # func is the function to be wrapped

Important

Decorator brings all the debugging code at one places. Now we can add more debugging features to ‘debugEx.py’ file and all the changes will be applied immediately to all the functions.

Warning

Decorators remove the help features of the function along with name etc. Therefore, we need to fix it using functools as shown next.

Rewrite the decorator using wraps function in functools as below,

# debugEx.py

from functools import wraps

def printName(func):
    # func is the function to be wrapped

    # wrap is used to exchange metadata between functions
    @wraps(func)
    def pn(*args, **kwargs):
        print(func.__name__)
        return func(*args, **kwargs)
    return pn

If we execute the mathEx.py again, it will show the help features again.

Note

@wraps exchanges the metadata between the functions as shown in above example.

12.1.3. Decorators with arguments

Suppose, we want to pass some argument to the decorator as shown below,

# mathEx.py

from debugEx import printName

@printName('**')
def add2Num(x, y):
    '''add two numbers'''
    return(x+y)

print(add2Num(2, 4))
# help(add2Num)

Note

To pass the argument to the decorator, all we need to write a outer function which takes the input arguments and then write the normal decorator inside that function as shown below,

# debugEx.py

from functools import wraps

def printName(prefix=""):
    def addPrefix(func):
        msg = prefix + func.__name__
        # func is the function to be wrapped

        # wrap is used to exchange metadata between functions
        @wraps(func)
        def pn(*args, **kwargs):
            print(msg)
            return func(*args, **kwargs)
        return pn
    return addPrefix

Now, run the above code,

$ python mathEx.py
**add2Num
6

Error

  • But, above code will generate error if we do not pass the argument to the decorator as shown below,
# mathEx.py

from debugEx import printName

@printName
def add2Num(x, y):
    '''add two numbers'''
    return(x+y)

print(add2Num(2, 4))
# help(add2Num)

Following error will be generate after running the code,

$ python mathEx.py
Traceback (most recent call last):
  File "mathEx.py", line 10, in <module>
    print(add2Num(2, 4))
TypeError: addPrefix() takes 1 positional argument but 2 were given
  • One solution is to write the two different codes e.g. ‘printName’ and ‘printNameArg’; then use these decorators as required. But this will make code repetitive as shown below,
# debugEx.py

from functools import wraps

def printName(func):
    # func is the function to be wrapped

    # wrap is used to exchange metadata between functions
    @wraps(func)
    def pn(*args, **kwargs):
        print(func.__name__)
        return func(*args, **kwargs)
    return pn


def printNameArg(prefix=""):
    def printName(func):
        # func is the function to be wrapped

        # wrap is used to exchange metadata between functions
        @wraps(func)
        def pn(*args, **kwargs):
            print(prefix + func.__name__)
            return func(*args, **kwargs)
        return pn
    return printName

Now, modify the math.py as below,

# mathEx.py

from debugEx import printName, printNameArg

@printNameArg('**')
def add2Num(x, y):
    '''add two numbers'''
    return(x+y)

@printName
def diff2Num(x, y):
    '''subtract two integers only'''
    return(x-y)

print(add2Num(2, 4))
print(diff2Num(2, 4))
# help(add2Num)

Next execute the code,

$ python mathEx.py
**add2Num
6
diff2Num
-2

12.1.4. DRY decorator with arguments

In previous code, we repeated the same code two time for creating the decorator with and without arguments. But, there is a better way to combine both the functionality in one decorator using partial function as shown below,

# debugEx.py

from functools import wraps, partial

def printName(func=None, *, prefix=""):
    if func is None:
        return partial(printName, prefix=prefix)
    # wrap is used to exchange metadata between functions
    @wraps(func)
    def pn(*args, **kwargs):
        print(prefix + func.__name__)
        return func(*args, **kwargs)
    return pn

Now, modify the mathEx.py i.e. remove printNameArg decorator from the code, as below,

# mathEx.py

from debugEx import printName

@printName(prefix='**')
def add2Num(x, y):
    '''add two numbers'''
    return(x+y)

@printName
def diff2Num(x, y):
    '''subtract two integers only'''
    return(x-y)

print(add2Num(2, 4))
print(diff2Num(2, 4))
# help(add2Num)

Next, run the code and it will display following results,

$ python mathEx.py
**add2Num
6
diff2Num
-2

Partial function is required because, when we pass argument to the decorator i.e. @printName(prifix=’**’), then decorator will not find any function argument at first place, hence return func(*arg, **kwargs) will generate error as there is no ‘func’.

To solve this problem, partial is used which returns the an new function, with modified parameters i.e. newFunc(func = printName, prefix = prefix).

12.1.5. Decorators inside the class

In previous sections, decorators are defined as functions. In this section, decorators will be defined as class methods.

class method and instance method decorator

In the following code, two types of decorators are defined inside the class i.e. using class method and instance method,

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# clsDecorator.py

from datetime import datetime

class DateDecorator(object):
    # instance method decorator
    def instantMethodDecorator(self, func):
        def printDate(*args, **kwargs):
            print("Instance method decorator at time : \n", datetime.today())
            return func(*args, **kwargs)
        return printDate

    # class method decorator
    @classmethod
    def classMethodDecorator(self, func):
        def printDate(*args, **kwargs):
            print("Class method decorator at time : \n", datetime.today())
            return func(*args, **kwargs)
        return printDate


# decorator: instance method
a = DateDecorator()
@a.instantMethodDecorator
def add(a, b):
    return a+b

# decorator: class method
@DateDecorator.classMethodDecorator
def sub(a, b):
    return a-b

sum = add(2, 3)
# Instance method decorator at time :
#  2017-02-04 13:31:27.742283

diff = sub(2, 3)
# Class method decorator at time :
#  2017-02-04 13:31:27.742435

Note that, we need to instantiate the instance mehtod decorator before using it as shown at line 23; whereas class decorator can be used as ClassName.DecoratorName.

12.1.6. Conclusion

In this section, we saw the relation between ‘function inside the function’ and decorator. Then, decorator with and without arguments are discussed. Lastly, class decorators are shown with examples.

12.2. Descriptors

Descriptor gives us the fine control over the attribute access. It allows us to write reusable code that can be shared between the classes as shown in this tutorial.

12.2.1. Problems with @property

Before beginning the descriptors, let’s look at the @property decorator and it’s usage along with the problem. Then we will see the descriptors in next section.

# square.py

class Square(object):
    def __init__(self, side):
        self.side = side

    def aget(self):
        return self.side ** 2

    def aset(self, value):
        print("can not set area")

    def adel(self):
        print("Can not delete area")

    area = property(aget, aset, adel, doc="Area of sqare")


s = Square(10)
print(s.area)  # 100

s.area = 10  # can not set area

del s.area  # can not delete area

Note that in above code, no bracket is used for calculating the area i.e. ‘s.area’ is used, instead of s.area(); because area is defined as property, not as a method.

Above code can be rewritten using @property decorator as below,

# square.py

class Square(object):
    """ A square using property with decorator"""
    def __init__(self, side):
        self.side = side

    @property
    def area(self):
        """Calculate the area of the square"""
        return self.side ** 2

    # name for setter and deleter (i.e. @area) must be same
    # as the method for which @property is used i.e. area here
    @area.setter
    def area(self, value):
        """ Do not allow set area directly"""
        print("Can not set area")

    @area.deleter
    def area(self):
        """Do not allow deleting"""
        print("Can not delete area")


s = Square(10)
print(s.area)  # 100

s.area = 10  # can not set area

del s.area  # can not delete area

Note that, @area.setter and @area.deleter are optional here. We can stop writing the code after defining the @property. In that, case setter and deleter option will generate standard exception i.e. attribute error here. If we want to perform some operations, then setter and deleter options are required.

Further, code is repetitive because @property, setter and deleter are the part of same method here i.e. area.

We can merge all three methods inside one method as shown next. But before looking at that example, let’s understand two python features first i.e. **kwargs and locals().

**kwargs

**kwargs converts the keyword arguments into dictionary as shown below,

def func(**kwargs):
    print(kwargs)

func(a=1, b=2)  # {'a': 1, 'b': 2}

locals()

Locals return the dictionary of local variables, as shown below,

def mathEx(a, b):
    add = a + b
    diff = a - b
    print(locals())

mathEx(3, 2)  # {'a': 3, 'add': 5, 'b': 2, 'diff': 1}

Now, we can implement all the get, set and del method inside one method as shown below,

# square.py

def nested_property(func):
    """ Nest getter, setter and deleter"""
    names = func()

    names['doc'] = func.__doc__
    return property(**names)


class Square(object):
    """ A square using property with decorator"""
    def __init__(self, side):
        self.side = side

    @nested_property
    def area():
        """Calculate the area of the square"""

        def fget(self):
            """ Calculate area """
            return self.side ** 2

        def fset(self, value):
            """ Do not allow set area directly"""
            print("Can not set area")

        def fdel(self):
            """Do not allow deleting"""
            print("Can not delete area")

        return locals()

s = Square(10)
print(s.area)  # 100

s.area = 10  # can not set area

del s.area  # can not delete area

Note

@propery is good for performing certain operations before get, set and delete operation. But, we need to implement it for all the functions separately and code becomes repetitive for larger number of methods. In such cases, descriptors can be useful.

12.2.2. Data Descriptors

# data_descriptor.py

class DataDescriptor(object):
    """ descriptor example """
    def __init__(self):
        self.value = 0

    def __get__(self, instance, cls):
        print("data descriptor __get__")
        return self.value

    def __set__(self, instance, value):
        print("data descriptor __set__")
        try:
            self.value = value.upper()
        except AttributeError:
            self.value = value

    def __delete__(self, instance):
        print("Can not delete")

class A(object):
    attr =  DataDescriptor()



d = DataDescriptor()
print(d.value)  # 0

a = A()
print(a.attr)
# data descriptor __get__
# 0

# a.attr is equivalent to below code
print(type(a).__dict__['attr'].__get__(a, type(a)))
# data descriptor __get__
# 0

# set will upper case the string
a.attr = 2  # 2
# lazy loading: above o/p will not display if
# below line is uncommented
a.attr = 'tiger' # TIGER
print(a.__dict__) # {}

# Following are the outputs of above three commands
# data descriptor __set__
# data descriptor __set__
# {}
# data descriptor __get__
# data descriptor __get__
# TIGER

Note

  • Note that object ‘d’ does not print the ‘data descriptor __get__’ but object of other class i.e. A prints the message. In the other words, descriptor can not use there methods by its’ own. Other’s class-attributes can use descriptor’s methods as shown in above example.

Also, see the outputs of last three commands. We will notice that,

Note

  • The set values are not store in the instance dictionary i.e. print(a.__dict__) results in empty dictionary.
  • Further, a.attr = 2 and a.attr ‘tiger’ performs the set operation immediately (see the __set__ message at outputs), but __get__ operations are performed at the end of the code, i.e. first print(a.__dict__) outputs are shown and then get operations is performed.
  • Lastly, set operation stores only last executed value, i.e. only TIGER is printed at the end, but not 2.

12.2.3. non-data descriptor

non-data descriptor stores the assigned values in the dictionary as shown below,

# non_data_descriptor.py

class NonDataDescriptor(object):
    """ descriptor example """
    def __init__(self):
        self.value = 0

    def __get__(self, instance, cls):
        print("non-data descriptor __get__")
        return self.value + 10

class A(object):
    attr =  NonDataDescriptor()

a = A()
print(a.attr)
# non-data descriptor __get__
# 10

a.attr = 3
a.attr = 3
print(a.__dict__) # {'attr': 4}

Important

  • In Non-data descriptor, the assigned values are stored in instance dictionary (and only last assigned value is stored in dictionary); whereas data descriptor assigned values are stored in descriptor dictionary because the set method of descriptor is invoked.

12.2.4. __getattribute__ breaks the descriptor usage

In below code, __getattribute__ method is overridden in class Overriden. Then, instance of class Overriden is created and finally the descriptor is called at Line 18. In this case, __getattribute__ method is invoked first and does not give access to descriptor.

# non_data_descriptor.py

class NonDataDescriptor(object):
    """ descriptor example """
    def __init__(self):
        self.value = 0

    def __get__(self, instance, cls):
        print("non-data descriptor __get__")
        return self.value + 10

class Overriden(object):
    attr =  NonDataDescriptor()
    def __getattribute__(self, name):
        print("Sorry, No way to reach to descriptor!")

o = Overriden()
o.attr  # Sorry, No way to reach to descriptor!

Note

Descriptors can not be invoked if __getattribute__ method is used in the class as shown in above example. We need to find some other ways in such cases.

12.2.5. Use more than one instance for testing

Following is the good example, which shows that test must be performed on more than one object of a classes. As following code, will work fine for one object, but error can be caught with two or more objects only.

# Examples.py

class DescriptorClassStorage(object):
    """ descriptor example """
    def __init__(self, default = None):
        self.value = default

    def __get__(self, instance, cls):
        return self.value
    def __set__(self, instance, value):
        self.value = value

class StoreClass(object):
    attr =  DescriptorClassStorage(10)


store1 =  StoreClass()
store2 =  StoreClass()

print(store1.attr, store2.attr)  # 10, 10

store1.attr = 30

print(store1.attr, store2.attr) # 30, 30

In above code, only store1.attr is set to 30, but value of store2.attr is also changes. This is happening because, in data-descriptors values are stored in descriptors only (not in instance dictionary as mentioned in previous section).

12.2.6. Examples

12.2.6.1. Write a descriptor which allows only positive values

Following is the code to test the positive number using descriptor,

# positiveValueDescriptor.py

class PositiveValueOnly(object):
    """ Allows only positive values """

    def __init__(self):
        self.value = 0

    def __get__(self, instance, cls):
        return self.value

    def __set__(self, instance, value):
        if value < 0 :
            raise ValueError ('Only positive values can be used')
        else:
            self.value = value

class Number(object):
    """ sample class that uses PositiveValueOnly descriptor"""

    value = PositiveValueOnly()


test1 = Number()
print(test1.value)  #  0

test1.value = 10
print(test1.value)  #  0

test1.value = -1
# [...]
# ValueError: Only positive values can be used

12.2.6.2. Passing arguments to decorator

IN previous codes, no arguments were passed in the decorator. In this example, taxrate is passed in the decorator and total price is calculated based on tax rate.

# taxrate.py

class Total(object):
    """ Calculate total values """

    def __init__(self, taxrate = 1.20):
        self.rate = taxrate

    def __get__(self, instance, cls):
        # net_price * rate
        return instance.net * self.rate

    # override __set__, so that there will be no way to set the value
    def __set__(self, instance, value):
        raise NoImplementationError("Can not change value")

class PriceNZ(object):
    total = Total(1.5)

    def __init__(self, net, comment=""):
        self.net = net
        self.comment = comment

class PriceAustralia(object):
    total = Total(1.3)

    def __init__(self, net):
        self.net = net


priceNZ = PriceNZ(100, "NZD")
print(priceNZ.total)  # 150.0

priceAustralia = PriceAustralia(100)
print(priceAustralia.total)  # 130.0

Note

In above example, look for the PriceNZ class, where init function takes two arguments and one of which is used by descriptor using ‘instance.net’ command. Further, init function in class Total need one argument i.e. taxrate, which is passed by individual class which creating the object of the descriptor.

12.2.7. Conclusion

In this section, we discussed data-descriptors and non-data-descriptors. Also, we saw the way values are stored in these two types of descriptors. Further, we saw that __getattribute__ method breaks the descriptor calls.