The other day, while reviewing the Assignment Section exercises for a training course I was about to deliver, wanting to type dir() into a Python REPL, my fingers went for () instead. At first I didn’t even notice it but then, something in the back of my head called for attention: I had indeed typed del() which, to my surprise, Python happily executed, producing no exceptions.

Since I knew that del is a Python statement and a reserved keyword, there was no way I was calling a del function that I could have accidentally created. This left me wondering, hmmm… What’s going on here?!

I this article I’ll describe the steps I went through in the process of grasping why del() is valid Python and exactly what it does, sharing a few techniques that can prove to be useful to others in understanding funny looking Python code tidbits, like this one felt to me.

Let the games begin: del() works

It all started with a freshly launched CPython 3.7 REPL, where I entered:

Observations:

  • No exception was raised.
  • Whatever del() did, no value was returned.

Conclusion:

  • Whatever Python did, it did something I wasn’t grasping.

Since del is a reserved keyword and a statement, I immediately went for the following hypothesis “The Python tokenizer and parser are interpreting del() as del () — note the space between del and (). That seemed reasonable but, given that () is an empty tuple literal and that del deletes references to objects (like variables, attributes, and more), the logic behind deleting an object literal still made no sense to me.

Deleting a literal makes no sense

From the hypothesis that del() was being parsed as del (), I then confirmed my second expectation: you really can’t delete a literal.

>>> del {}
  File "<stdin>", line 1
SyntaxError: can't delete literal
>>> del 42
  File "<stdin>", line 1
SyntaxError: can't delete literal
>>>

Interestingly, these two forms work:

Knowing how Python assignment works and what the del statement does, I was positive that deleting literals should fail, like it did for the dict and int literals. It still provocatively seemed to work with tuple and list literals… What was going on here?

Digging Deeper

It was time to bring in a powerful tool. I decided to disassemble the mysterious statement and see what kind of byte-code was being generated which, apparently, was being happily executed by the Python VM. To do that, I created the function containing a single del () statement, and then used the dis module, from the Standard Library:

>>> import dis
>>>
>>> def mystery():
...     del ()
... 
>>> dis.dis(mystery)
  2           0 LOAD_CONST               0 (None)
              2 RETURN_VALUE
>>> 

Interesting! The del statement in the mystery function seemed to have been completely eliminated… Was I getting it right? I went further with two other tests. I first created a do_nothing function, disassembling it…

>>> def do_nothing():
...     pass
... 
>>> 
>>> dis.dis(do_nothing)
  2           0 LOAD_CONST               0 (None)
              2 RETURN_VALUE
>>> 

…and then the triple_mystery function, calling del () three times in a row and, again, disassembled it:

>>> def triple_mystery():
...     del ()
...     del ()
...     del ()
... 
>>>
>>> dis.dis(triple_mystery)
  4           0 LOAD_CONST               0 (None)
              2 RETURN_VALUE
>>> 

Hopefully, even someone who is not familiar with Python byte-code can verify that both the mystery and triple_mystery functions compile down to the same exact byte-code, which does precisely nothing, much like the do_nothing function I created for comparison.

I confess that these results, while consistent with the initial observations — nothing is returned and not exception is raised — puzzled me even more, leading me to direct my quest somewhere else…

Has the compiler gone crazy?

When in doubt, the answer to such a question is most probably “no, the compiler has not gone crazy, there’s something you’re misunderstanding”. Especially when dealing with widely-used, time-tested compilers like the one in CPython.

To figure out why the del () statement was generating no byte-code at all, I went for a divide and conquer diagnostic approach. Here’s what we know, at this point:

  1. The input code is del ().
  2. It is fed to the compiler.
  3. The output byte-code is empty.

What’s going on in-between, inside the compiler itself? Well, between the source Python code and the executable Python byte-code, the Python compiler generates an intermediate representation called an Abstract Syntax Tree — knowing that, in simple terms, we can break Step 2., above, into:

  1. The input code is del ().
  2. It is fed to the compiler:
    1. The input code is parsed into an Abstract Syntax Tree.
    2. The Abstract Syntax Tree is used to generate byte-code.
  3. The output byte-code is empty.

Fortunately, the Python Standard Library includes the ast module which allows us to easily create an Abstract Syntax Tree from a given piece of valid Python code. That’s precisely where I went next:

>>> import ast
>>> node = ast.parse('del ()')

The node object is an ast.Module object which includes a body attribute…

>>> node.body
[<_ast.Delete object at 0x10c1b1b00>]

…which is a list of nodes containing, in our case, unsurprisingly, an instance of a ast.Delete object. The ast.Delete object, in turn, has a targets attribute which is a list of the targets for the delete statement:

>>> delete = node.body[0]
>>> delete.targets
[<_ast.Tuple object at 0x10c1bc8d0>]

From the above, it seems clear that the parsed del () statement has been understood to mean: a) delete a single target, b) the target is a tuple. Let’s check that tuple’s AST representation by taking a look at the elts attribute in the ast.Tuple object:

>>> delete_target = delete.targets[0]
>>> delete_target.elts
[]

It comes out as an empty list, pretty much inline with what we could expect to represent the elements in our source empty tuple.

Compacting the exploratory code a bit, we could write:

>>> ast.parse('del ()').body[0].targets
[<_ast.Tuple object at 0x10c424c88>]
>>> ast.parse('del ()').body[0].targets[0].elts
[]

Preliminary Conclusion

The above observations clearly confirm that parsing del () results in an AST representing the deletion of a single target, which is a tuple with no elements. Thus, something is going on, in the later stages of the compiler, that is “clever enough” to somehow discard such statement, given that no byte-code is produced from it… The question is: should it? Does it make any sense? It probably does, let’s see.

Stepping back for a while, let’s see what kind of AST’s we get from more common delete statements:

>>> ast.parse('del n').body[0].targets
[<_ast.Name object at 0x10c424278>]
>>> ast.parse('del n, m').body[0].targets
[<_ast.Name object at 0x10c424c88>, <_ast.Name object at 0x10c424860>]

The targets are ast.Name objects, which makes sense, even if we don’t dig any deeper. What about deleting targets in a tuple?

>>> ast.parse('del (n,)').body[0].targets
[<_ast.Tuple object at 0x10b9b5908>]
>>> ast.parse('del (n,)').body[0].targets[0].elts
[<_ast.Name object at 0x10ba50cf8>]
>>> ast.parse('del (n, m)').body[0].targets
[<_ast.Tuple object at 0x10b9b58d0>]
>>> ast.parse('del (n, m)').body[0].targets[0].elts
[<_ast.Name object at 0x10ba50cf8>, <_ast.Name object at 0x10ba50d68>]

Interesting, the target is a single ast.Tuple, containing ast.Name objects as elements. Do these forms even work?

>>> n = 42
>>> del (n,)
>>> n
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'n' is not defined
>>>
>>> n, m = 42, 24
>>> del (n, m)
>>> 'n' in dir(), 'm' in dir()
(False, False)

Yes, they certainly do.

What about something somewhat crazy like…

>>> n, m, o, p = range(4)
>>> del [n, m,], (o,), p
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'ast']

…or…

>>> n, m, o, p, q, r = range(6)
>>> del (n, [m, o, (p, (q, (r,)))])
>>> dir()
['__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'ast']

It all works. As I progressed through these attempts, I developed an intuition like I believe some of the readers will. This all felt familiar, especially in the context of Python assignment, around which the del statement is an important tool.

That’s when I decided to do something I probably never did before: I went reading the del statement documentation, where the first line reads del_stmt ::= "del" target_list, and target_list links here, under the Assignment Statements section of the documentation.

Conclusion

The del statement grammar rule clearly states that the del keyword is to be followed by a target list which, in turn, can be as simple as a single name, or as complex as an arbitrarily nested structure of tuples, lists, and more, with lots of possibilites in-between.

Such target lists are used in many other places, such as on the left side of an assignment statement, in for statements, in function arguments, and eventually more.

The specification for target lists includes empty tuples and lists as valid targets. That’s why del () and its funny cousin del [] work the way they do: both include valid target lists with no targets to delete.

Wrap-up

In retrospect, the reason why del() works, and what it does, is pretty obvious and natural to me. Some people, faced with the same code, might have understood it immediately, while others might have taken a completely different path in understanding what could be going on.

I don’t believe in a right vs. wrong way of understanding things and, more importantly, of investigating unexpected behaviours. I decided to write these words sharing the process I took and how useful the dis and ast modules were for me. Maybe these tools will be useful to someone else having the need to understand some funny behaving code they don’t immediately understand.

Lastly, something I usually say a lot during my training sessions: “The Python documentation is very good. When in doubt, refer to it!”

Now, why didn’t I go there straight away? 😊

Post Scriptum

Interestingly, the del () statement does not seem to run on CPython 2.7, CPython 3.5, PyPy 6.0.0 2.7.13, nor PyPy 6.0.0 3.5.3, resulting in SyntaxError: can't delete () all around. I did not dig further, but after my findings, and even though del () is obviously a no-op, I kind of like its consistency.



Source link

LEAVE A REPLY

Please enter your comment!
Please enter your name here