How Does Python Work?

Hello again, everyone! Today I’ll explain how Python works, what an interpreter is, how a compiler works, and what bytecode is. Then I’ll talk about the virtual machine (PVM) and Python’s performance, as well as alternative implementations of the interpreter.

Now that you have installed Python, let’s move on to the theoretical and practical side, starting with what an interpreter actually is.

The Interpreter

An interpreter is a program that executes other programs. When you write a program in Python, the interpreter reads your program and executes the instructions it contains. In effect, the interpreter is a layer of software logic between your program code and your computer’s hardware.

Depending on the version of Python you use, the interpreter itself may be implemented as a program written in C, as a collection of Java classes, or in some other form, but more on that later.

Running a Script in the Console

Let’s start the interpreter in the console:

# python 

It is now waiting for commands. Enter the following instruction:

print 'hello world!'

Hooray, our first program! :D

Running a Script from a File

Create a file called “test.py” with the following contents:

# вывести "hello world"
print "hello world"

# вывести 2 в 10 степени
print 2 ** 10

and execute this file:

# python /path/to/test.py

You’ll see the result in the console. Let’s move on!

Dynamic Compilation and Bytecode

After you launch a script, Python first compiles its source code into bytecode for the virtual machine. Compilation is simply a translation step, and bytecode is a low-level, platform-independent representation of a program’s source code. Python translates each instruction in the script’s source code into groups of bytecode instructions to improve the program’s execution speed, since bytecode runs much faster. After compilation into bytecode, a file with the ".pyc" extension is created alongside the script’s source code.

The next time you run your program, the interpreter skips the compilation step and passes the compiled file with the “.pyc” extension on for execution. However, if you have changed your program’s source code, the bytecode compilation step takes place again, since Python automatically monitors the source file’s modification date.

If Python is unable to write the bytecode file, for example because it lacks permission to write to disk, the program will not be affected. The bytecode will simply be built in memory and removed from there when the program terminates.

The Python Virtual Machine (PVM)

Once compilation is complete, the bytecode is passed to a mechanism called the virtual machine, which executes the bytecode instructions. The virtual machine is a runtime engine. It is always present as part of the Python system and is the final component of the system called the “Python Interpreter.”

To reinforce what we’ve covered, let’s clarify things once more: compilation into bytecode happens automatically, and the PVM is simply part of the Python system you installed along with the interpreter and compiler. Everything happens transparently to the programmer, and you do not need to perform these operations manually.

Performance

Programmers with experience in languages such as C and C++ may notice some differences in Python’s execution model. The first is the absence of a build step or a call to the “make” utility: Python programs can be run as soon as their source code has been written. The second difference is that bytecode is not binary machine code (such as instructions for an Intel microprocessor); it is an internal representation of a Python program.

For these reasons, Python programs cannot run as fast as C/C++ programs. A virtual system steps through the instructions rather than a microprocessor, and executing bytecode requires additional interpretation, whose instructions take more time than the microprocessor’s machine instructions.

On the other hand, unlike traditional interpreters, such as the one in PHP, there is an additional compilation step here: the interpreter does not need to analyze the program’s source code every time.

As a result, in terms of performance, Python sits between traditional compiled and traditional interpreted programming languages.

Alternative Python Implementations

What I said above about the compiler and virtual machine applies to the standard Python implementation, known as CPython (the implementation in ANSI C). However, there are also alternative implementations, such as Jython and IronPython, which I’ll discuss now.

CPython

This is the standard and original implementation of Python, named this way because it is written in ANSI C. It is the implementation we installed when we chose the ActivePython package or installed it from the FreeBSD ports collection. Since it is the reference implementation, it generally runs faster, is more stable, and works better than the alternative implementations.

Jython

Originally named JPython, its main goal is tight integration with the Java programming language. The Jython implementation consists of Java classes that compile Python program code into Java bytecode and then pass the resulting bytecode to the Java Virtual Machine (JVM).

Jython’s goal is to allow Python programs to control Java applications, just as CPython can control components written in C/C++. This implementation offers seamless integration with Java. Since Python program code is translated into Java bytecode, it behaves exactly like a real Java program at runtime. Jython programs can act as applets and servlets, create graphical interfaces using Java mechanisms, and so on. What’s more, Jython supports importing and using Java classes in Python program code.

Nevertheless, because the Jython implementation runs more slowly and is less stable than CPython, it is primarily of interest to Java developers who need a scripting language as an interface to Java code.

IronPython

This implementation is intended to integrate Python programs with applications built to run in the Microsoft .NET Framework environment on Windows, as well as in Mono, its open-source equivalent for Linux. The .NET platform and the C# runtime environment are designed to enable interaction between software objects, regardless of the programming language used, in the spirit of Microsoft’s earlier COM model.

IronPython allows Python programs to serve as both client and server components accessible from other .NET programming languages. Since development is being carried out by Microsoft, one might expect significant performance optimizations from IronPython, among other things.

Tools for Optimizing Execution Speed

There are other implementations, including the Psyco dynamic compiler and the Shedskin C++ translator, that attempt to optimize the basic execution model.

The Psyco Dynamic Compiler

The Psyco system is a component that extends the bytecode execution model, allowing programs to run faster. Psyco is an extension to the PVM that gathers and uses type information to translate parts of a program’s bytecode into actual binary machine code, which runs much faster. This translation requires no changes to the source code or additional compilation during development.

While a program is running, Psyco gathers information about object types. This information is then used to generate highly efficient machine code optimized for objects of those types. The generated machine code then replaces the corresponding sections of bytecode, thereby increasing execution speed.

Ideally, some sections of program code running under Psyco can execute as fast as compiled C code.

Psyco provides speed increases ranging from 2 to 100 times, but typically around 4 times, when used with an unmodified Python interpreter. Psyco’s only downside is that it can currently generate machine code only for the Intel x86 architecture.

Psyco is not included in the standard distribution; it must be downloaded and installed separately. There is also the PyPy project, which is an attempt to rewrite the PVM to optimize code as Psyco does. The PyPy project intends to largely absorb the Psyco project.

The Shedskin C++ Translator

Shedskin is a system that converts Python source code into C++ source code, which can then be compiled into machine code. In addition, the system implements a platform-independent approach to executing Python program code.

Frozen Binaries

Sometimes you need to turn your Python programs into standalone executables. This is mainly useful for packaging and distributing programs.

Frozen binaries combine a program’s bytecode, the PVM, and the support files the program needs into a single package file. The result is a single executable file, such as a file with the “.exe” extension for Windows.

At present, there are three main tools for creating “frozen binaries”:

• py2exe — it can create standalone Windows programs that use the Tkinter, PMW, wxPython, and PyGTK libraries to build graphical interfaces, programs that use the PyGame game development tools, win32com client programs, and many others;
• PyInstaller — similar to py2exe, but it also works on Linux and UNIX and can produce self-installing executables;
• freeze — the original version.

You need to download these tools separately from Python; they are distributed free of charge.

Frozen binaries are fairly large because they contain the PVM, but by modern standards they still cannot be called unusually large. Since the Python interpreter is embedded directly in frozen binaries, installing it is not a prerequisite for running the programs on the recipient’s end.

Summary

That’s all for today. In the next article, I’ll talk about Python’s standard data types, and in subsequent articles we’ll look at each type individually, as well as the functions and operators used to work with them.