Skip to content
CodeMDD.io | Coding Tutorials

Effortlessly Mastering subprocess Popen

[

The subprocess Module: Wrapping Programs With Python

by Ian Currie

If you’ve ever wanted to simplify your command-line scripting or use Python alongside command-line applications—or any applications for that matter—then the Python subprocess module can help. From running shell commands and command-line applications to launching GUI applications, the Python subprocess module can help.

By the end of this tutorial, you’ll be able to:

  • Understand how the Python subprocess module interacts with the operating system
  • Issue shell commands like ls or dir
  • Feed input into a process and use its output
  • Handle errors when using subprocess
  • Understand the use cases for subprocess by considering practical examples

In this tutorial, you’ll get a high-level mental model for understanding processes, subprocesses, and Python before getting stuck into the subprocess module and experimenting with an example. After that, you’ll start exploring the shell and learn how you can leverage Python’s subprocess with Windows and UNIX-based shells and systems. Specifically, you’ll cover communication with processes, pipes, and error handling.

Note: subprocess isn’t a GUI automation module or a way to achieve concurrency. For GUI automation, you might want to look at PyAutoGUI. For concurrency, take a look at this tutorial’s section on modules related to subprocess.

Once you have the basics down, you’ll be exploring some practical ideas for how to leverage the subprocess module in your projects. You’ll learn about creating a new project and changing extended attributes. Additionally, this tutorial will introduce you to other Python modules associated with subprocess, and walk you through using the Popen class to connect two processes together with pipes and interact dynamically with a process.

Processes and Subprocesses

Processes and the Operating System

Before diving into Python’s subprocess module, it’s important to understand what processes and subprocesses are, and how they relate to the operating system.

A process refers to a running program on your computer. Each process has a unique ID, known as a process ID, which is assigned by the operating system. You can think of a process as an instance of an application running on your computer. When you open a text editor or a web browser, each of these instances is a different process.

The operating system is responsible for managing processes. It oversees the creation, execution, and termination of processes. It also schedules the execution of multiple processes to ensure optimal use of system resources.

Process Lifetime

The lifetime of a process can be divided into different stages:

  1. Creation: A process is created when you start an application or run a command. The operating system creates a new process to execute the application or command.
  2. Execution: Once a process is created, it starts executing the instructions provided by the application or command.
  3. Termination: A process may terminate when it has finished executing its instructions or when it is terminated forcefully by the operating system.

Active Processes on Your System

To get a list of active processes on your system using Python, you can utilize the subprocess module along with the appropriate command for your operating system.

Here’s an example that uses the ps command on UNIX-based systems (like macOS and Linux) to list the active processes:

import subprocess


output = subprocess.run(['ps', 'aux'], capture_output=True, text=True)
print(output.stdout)

This code uses the run() function from the subprocess module to execute the ps command (which lists the active processes) with the aux option. The capture_output parameter is set to True to capture the command’s output, and the text parameter is set to True to indicate that the captured output should be returned as a string.

On Windows, the tasklist command can be used instead:

import subprocess


output = subprocess.run(['tasklist'], capture_output=True, text=True)
print(output.stdout)

This code uses the tasklist command to list the active processes on Windows. The capture_output and text parameters have the same purpose as in the previous example.

By running these code snippets, you’ll see the output containing information about all the currently active processes on your system.

Overview of the Python subprocess Module

The subprocess module in Python provides a way to create new processes, interact with them, and handle their input and output streams. It allows you to execute shell commands, run command-line applications, and even launch GUI applications.

To use the subprocess module, you’ll need to import it at the beginning of your script:

import subprocess

Once you have imported the module, you can start exploring its various functions and classes.

Basic Usage of the Python subprocess Module

The subprocess module provides several functions and classes for executing and managing processes. In this section, you’ll learn about some basic usage of the module.

The Timer Example

To get started with the subprocess module, let’s create a simple example that demonstrates the run() function. This function allows you to run a command and capture its output.

Suppose you want to measure the time it takes to execute a command. One way to do this is by creating a timer script that uses the run() function to execute the command and captures the elapsed time.

Here’s an example that shows how to use the run() function to measure the time it takes to execute the ls command (which lists the contents of a directory):

import subprocess
import time


start_time = time.time()
output = subprocess.run(['ls', '-l'], capture_output=True, text=True)
end_time = time.time()


elapsed_time = end_time - start_time
print(output.stdout)
print(f"Elapsed Time: {elapsed_time} seconds")

In this code, the run() function is used to execute the ls command with the -l option. The capture_output and text parameters are set to True to capture the output of the command and return it as a string.

The start_time variable is assigned the current time using the time.time() function before the run() function is called. Similarly, the end_time variable is assigned the current time after the run() function completes.

The elapsed time is calculated by subtracting the start_time from the end_time. Finally, the output of the ls command and the elapsed time are printed.

When you run this code, you’ll see the output of the ls command and the elapsed time it took to execute the command.

The Use of subprocess to Run Any App

The subprocess module is not limited to executing shell commands like ls. It can also run any program or application installed on your system, including Python scripts, compiled executables, and even GUI applications.

Here’s an example that demonstrates how to use the subprocess module to execute a Python script:

import subprocess


output = subprocess.run(['python', 'script.py'], capture_output=True, text=True)
print(output.stdout)

In this example, the run() function is used to execute the script.py Python script. The python command is used to run the script. The capture_output and text parameters are set to True to capture the output of the script and return it as a string.

Similarly, you can use the subprocess module to run other programs or applications by replacing 'python' with the desired program or application name and providing any additional command-line arguments as needed.

The CompletedProcess Object

The run() function returns a CompletedProcess object that represents the completed process. This object contains useful information about the executed command, such as the return code, the captured output, and more.

You can access this information by assigning the result of the run() function to a variable and accessing the attributes of the CompletedProcess object.

Here’s an example that demonstrates how to access the return code and the captured output of the run() function:

import subprocess


result = subprocess.run(['ls', '-l'], capture_output=True, text=True)


print(f"Return Code: {result.returncode}")
print(f"Captured Output:\n{result.stdout}")

In this code, the returncode attribute of the result object is accessed to get the return code of the ls command. The stdout attribute is accessed to get the captured output of the command.

By running this code, you’ll see the return code and the captured output of the ls command printed.

subprocess Exceptions

When working with the subprocess module, it’s important to handle potential exceptions that may occur during the execution of processes. The module provides several exception classes to handle specific types of errors.

CalledProcessError for Non-Zero Exit Code

One common exception that you may encounter is the CalledProcessError. This exception is raised when a process exits with a non-zero exit code, indicating an error.

To handle this exception, you can use a try…except block. Here’s an example that demonstrates how to handle the CalledProcessError:

import subprocess


try:
    subprocess.run(['ls', 'nonexistent_dir'], check=True)
except subprocess.CalledProcessError as error:
    print(f"Command Failed with Exit Code {error.returncode}")
    print(f"Error Output:\n{error.output}")

In this code, the run() function is used to execute the ls command with the nonexistent_dir argument. Since the directory doesn’t exist, the command will fail, and a CalledProcessError exception will be raised.

The check parameter is set to True to raise an exception if the command fails. If the check parameter is set to False (the default value), the exception won’t be raised, and you’ll have to manually check the return code to determine if the command succeeded or failed.

Inside the except block, the attributes of the CalledProcessError object are accessed to get the return code and the error output.

By running this code, you’ll see the error message and the error output printed.

TimeoutExpired for Processes That Take Too Long

Another useful exception class provided by the subprocess module is TimeoutExpired. This exception is raised when a process takes too long to complete.

You can specify a timeout value using the timeout parameter of the run() function. If the process doesn’t complete within the specified timeout, a TimeoutExpired exception will be raised.

Here’s an example that demonstrates how to use the timeout parameter and handle the TimeoutExpired exception:

import subprocess


try:
    subprocess.run(['sleep', '10'], timeout=5)
except subprocess.TimeoutExpired as error:
    print(f"Command Timed Out After {error.timeout} Seconds")

In this code, the sleep command is used to pause the execution for 10 seconds. However, the timeout parameter is set to 5 seconds. Therefore, the process will exceed the specified timeout, and a TimeoutExpired exception will be raised.

Inside the except block, the timeout attribute of the TimeoutExpired object is accessed to get the timeout value specified in the run() function.

By running this code, you’ll see the timeout message printed.

FileNotFoundError for Programs That Don’t Exist

Sometimes, you may encounter a situation where you’re trying to execute a program or application that doesn’t exist on your system. In such cases, the subprocess module raises a FileNotFoundError exception.

To handle this exception, you can use a try…except block. Here’s an example that demonstrates how to handle the FileNotFoundError:

import subprocess


try:
    subprocess.run(['nonexistent_program'], check=True)
except FileNotFoundError:
    print("Program Not Found")

In this code, the run() function is used to execute a nonexistent_program. Since the program doesn’t exist, a FileNotFoundError exception will be raised.

Inside the except block, a simple error message is printed.

By running this code, you’ll see the “Program Not Found” message printed.

An Example of Exception Handling

Now that you’re familiar with handling exceptions in the subprocess module, let’s put it all together with a more comprehensive example:

import subprocess


try:
    # Execute the command
    result = subprocess.run(['ls', 'nonexistent_dir'], capture_output=True, text=True, check=True)


    # Print the captured output
    print(result.stdout)
except subprocess.CalledProcessError as error:
    # Print the error message and output
    print(f"Command Failed with Exit Code {error.returncode}")
    print(f"Error Output:\n{error.output}")
except subprocess.TimeoutExpired as error:
    # Print the timeout message
    print(f"Command Timed Out After {error.timeout} Seconds")
except FileNotFoundError:
    # Print the program not found message
    print("Program Not Found")

In this code, the run() function is used to execute the ls command with the nonexistent_dir argument. The capture_output, text, and check parameters are set to capture the output, return it as a string, and raise an exception if the command fails.

The code includes except blocks to handle the CalledProcessError, TimeoutExpired, and FileNotFoundError exceptions.

By running this code, you can see the appropriate error message and output based on the specific exception that is raised.

Introduction to the Shell and Text-Based Programs With subprocess

In addition to executing shell commands, the subprocess module can also interact with text-based programs and scripts. This allows you to leverage the power and flexibility of the shell and use Python to automate tasks, process data, and more.

Use Cases for the Shell and subprocess

There are many use cases where using the shell alongside Python can be beneficial:

  • Automation: You can automate repetitive tasks by scripting command-line operations. For example, you can write a Python script that runs a series of commands to update files, process data, or perform system administration tasks.
  • Integration: You can integrate third-party command-line tools or libraries into your Python applications. This allows you to leverage the functionality of these tools without having to reinvent the wheel.
  • Data Processing: You can use text-based programs or scripts to process and manipulate data. Python can be used to automate the execution of these programs and handle their input and output.

To interact with the shell and text-based programs, the subprocess module provides functions and classes that allow you to execute commands, handle input and output, and even automate user interactions.

Basic Usage of subprocess With UNIX-Based Shells

When working with UNIX-based shells (like the Bash shell on macOS and Linux), you can execute shell commands directly using the run() function.

Here’s an example that demonstrates how to use the run() function with a UNIX-based shell:

import subprocess


output = subprocess.run(['echo', 'Hello, World!'], capture_output=True, text=True)
print(output.stdout)

In this code, the echo command is used to print the string "Hello, World!". The run() function is called with the command as a list and the capture_output and text parameters set to capture the output as a string.

By running this code, you’ll see the output of the echo command printed.

Basic Usage of subprocess With Windows Shells

When working with Windows shells (like Command Prompt or PowerShell), you can use the run() function with a slight modification. Instead of passing the command as a list, you need to pass it as a single string.

Here’s an example that demonstrates how to use the run() function with a Windows shell:

import subprocess


output = subprocess.run('echo Hello, World!', capture_output=True, text=True, shell=True)
print(output.stdout)

In this code, the echo command is used to print the string "Hello, World!". The command is passed as a single string to the run() function. The capture_output and text parameters are set to capture the output as a string, and the shell parameter is set to True to use the Windows shell to execute the command.

By running this code, you’ll see the output of the echo command printed.

A Security Warning

When using the run() function with a shell, be cautious when passing user-supplied input as part of the command. Never use string concatenation or interpolation to build the command string, as it can potentially lead to security vulnerabilities.

For example, consider the following code:

import subprocess


user_input = input("Enter a filename: ")


# DON'T DO THIS!
output = subprocess.run(['cat', user_input], capture_output=True, text=True)
print(output.stdout)

In this code, the user is prompted to enter a filename. The supplied filename is then passed directly to the cat command using string interpolation. This code is dangerous because it allows the user to execute arbitrary commands by manipulating the input.

To prevent this, you should sanitize and validate any user input that is used in commands. One way to do this is by using the shlex.quote() function to properly quote any user input, ensuring it is treated as a single argument. Here’s an example that demonstrates how to sanitize user input:

import shlex
import subprocess


user_input = input("Enter a filename: ")


# Sanitize and validate the user input
quoted_input = shlex.quote(user_input)


# Execute the command
output = subprocess.run(['cat', quoted_input], capture_output=True, text=True)
print(output.stdout)

In this code, the shlex.quote() function is used to properly quote the user input. This ensures that the user input is treated as a single argument and prevents any potential command injection attacks.

By using proper input sanitation and validation techniques, you can reduce the risk of security vulnerabilities when interacting with shells and text-based programs.

Communication With Processes

The subprocess module provides several ways to interact with processes, including reading and writing to their standard I/O streams. This allows you to communicate with processes and automate tasks that require input and receive output from external programs.

The Standard I/O Streams

When a program or application is executed, it has three standard I/O streams associated with it:

  • Standard Input (stdin): This stream accepts input to the program. By default, it is connected to the keyboard, allowing the user to provide input interactively. However, it can also be redirected to read input from a file or a pipe.
  • Standard Output (stdout): This stream represents the normal output of the program. By default, it is connected to the console, allowing the program to print output that the user can see. However, it can also be redirected to write output to a file or a pipe.
  • Standard Error (stderr): This stream is used to print error messages and diagnostic output. By default, it is also connected to the console, separate from the normal output. However, like stdout, it can be redirected to write output to a file or a pipe.

The subprocess module provides ways to access and communicate with these standard I/O streams of a process, allowing you to automate tasks and handle input and output.

The Magic Number Generator Example

To demonstrate communication with processes, let’s create a simple example that generates a magic number. The magic number is a random integer between 1 and 100, and the program waits for the user to guess the number. It provides feedback based on whether the guess is too high, too low, or correct.

Here’s an example that uses the subprocess module to communicate with the magic number generator:

import random
import subprocess


# Generate a random magic number
magic_number = random.randint(1, 100)


# Create the process to generate the magic number
process = subprocess.Popen(['python', 'magic_number_generator.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)


# Communicate with the process
while True:
    # Get the user's guess
    guess = input("Enter your guess (1-100): ")


    # Send the guess to the process
    process.stdin.write(f"{guess}\n")
    process.stdin.flush()


    # Get the response from the process
    response = process.stdout.readline().strip()


    # Print the response
    print(response)


    # Check if the guess is correct
    if response == "You guessed it!":
        break


# Close the process
process.stdin.close()
process.stdout.close()
process.wait()

In this code, the magic_number_generator.py script generates the magic number. The subprocess.Popen() function is used to create the process, specifying the command to run as a list.

Inside the while loop, the user is prompted to enter a guess. The guess is sent to the process using process.stdin.write() and process.stdin.flush(). The response from the process is read using process.stdout.readline(), stripped of leading and trailing whitespace using strip(), and printed.

The loop continues until the guess is correct, as indicated by the response.

Finally, the process is closed by closing the stdin and stdout pipes, and waiting for the process to finish using process.wait().

When you run this code, you’ll be prompted to enter guesses until you guess the correct number, and you’ll see feedback from the process indicating whether your guess is too high, too low, or correct.

The Decoding of Standard Streams

When interacting with processes in Python, it’s important to properly handle the encoding and decoding of the standard I/O streams. By default, these streams use the system default encoding, which may not be suitable for all situations.

To specify a different encoding for the standard I/O streams, you can use the encoding parameter of the subprocess.Popen() function. This parameter specifies the text encoding to use, such as 'utf-8' or 'latin1'. If you’re unsure which encoding to use, you can use 'utf-8', which is commonly used and supports a wide range of characters.

Here’s an example that demonstrates how to specify the encoding for the standard I/O streams:

import subprocess


# Create the process with the specified encoding
process = subprocess.Popen(['python', 'script.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True, encoding='utf-8')


# ...

In this code, the encoding parameter is set to 'utf-8' to use the UTF-8 encoding for the standard I/O streams of the process.

By specifying the correct encoding, you can ensure that the input and output of the process are properly encoded and decoded, preventing any potential encoding errors.

Reaction Game Example

To further illustrate communication with processes, let’s create another example: a reaction game. In this game, the program waits for a ready signal and measures the time it takes for the user to press a key. The program then provides feedback on the reaction time.

Here’s an example that demonstrates how to use the subprocess module to communicate with the reaction game:

import subprocess
import time


# Create the process to run the reaction game
process = subprocess.Popen(['python', 'reaction_game.py'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)


# Wait for the ready signal from the process
ready_signal = process.stdout.readline().strip()
print(ready_signal)


# Prompt the user to press a key
input("Press Enter to start...")


# Get the start time
start_time = time.time()


# Send the start signal to the process
process.stdin.write("start\n")
process.stdin.flush()


# Get the result from the process
result = process.stdout.readline().strip()


# Get the end time
end_time = time.time()


# Calculate the reaction time
reaction_time = end_time - start_time


# Print the result and reaction time
print(result)
print(f"Reaction Time: {reaction_time:.2f} seconds")


# Close the process
process.stdin.close()
process.stdout.close()
process.wait()

In this code, the reaction_game.py script implements the reaction game. The subprocess.Popen() function is used to create the process, specifying the command to run as a list.

The program waits for the ready signal from the process using process.stdout.readline(), strips leading and trailing whitespace using strip(), and prints the signal.

The user is prompted to press the Enter key to start the game. The start time is recorded using time.time().

The start signal ("start\n") is sent to the process using process.stdin.write() and process.stdin.flush().

The result of the game is read from the process using process.stdout.readline(), stripped of leading and trailing whitespace, and printed.

The end time is recorded using time.time(), and the reaction time is calculated by subtracting the start time from the end time.

Finally, the process is closed by closing the stdin and stdout pipes, and waiting for the process to finish.

When you run this code, you’ll see the ready signal, prompted to press Enter, and provided with feedback on your reaction time.

Pipes and the Shell

The subprocess module provides ways to create and manage pipes, allowing you to connect processes together and transfer data between them.

Introduction to Pipes

A pipe is a communication channel between two processes. It allows the output of one process to be connected to the input of another process. Data written to the output of the first process is received as input by the second process.

Pipes are commonly used to connect processes together to form pipelines. A pipeline is a series of connected processes, where the output of one process is piped directly into the input of the next process.

By using pipes, you can combine the functionality of multiple processes and perform complex tasks by chaining them together sequentially, each process consuming the output of the previous process and producing output for the next process.

The Pipes of subprocess

The subprocess module provides several functions and classes to create, manage, and connect processes with pipes.

To create a pipe, you can use the subprocess.PIPE constant as the value for the stdin, stdout, or stderr parameter of the subprocess.Popen() or subprocess.run() functions. This designates that the respective standard I/O stream should be connected to a pipe.

Here’s an example that demonstrates how to create a pipe and connect processes together:

import subprocess


# Create the first process with a pipe for stdout
process1 = subprocess.Popen(['echo', 'Hello, World!'], stdout=subprocess.PIPE, text=True)


# Create the second process with a pipe for stdin
process2 = subprocess.Popen(['grep', 'World'], stdin=process1.stdout, stdout=subprocess.PIPE, text=True)


# Close the first process's stdout pipe, as it won't be used anymore
process1.stdout.close()


# Wait for the second process to finish and get the output
output = process2.communicate()[0]


# Print the output
print(output)

In this code, two processes are created: process1 and process2.

process1 is created using subprocess.Popen() with the stdout=subprocess.PIPE parameter, which creates a pipe for the standard output of the process. This allows the output of process1 to be received as input by process2.

process2 is created using subprocess.Popen() with the stdin=process1.stdout parameter. This specifies that the standard input of process2 should be connected to the output of process1. This effectively directs the output of process1 to the input of process2.

The stdout pipe of process1 is closed using the stdout.close() method, as it won’t be used anymore.

The communicate() method of process2 is called to wait for process2 to finish and get its output. The [0] index is used to access the output, as communicate() returns a tuple containing the output and the error output.

Finally, the output is printed.

When you run this code, you’ll see the output of process2, which contains the line "Hello, World!" as it matches the pattern "World".

Pipe Simulation With run()

The subprocess module also provides the subprocess.run() function, which is a simplified interface for executing commands and capturing their output. This function can also be used to simulate pipes by connecting processes together.

Here’s an example that demonstrates how to use subprocess.run() to simulate pipes:

import subprocess


# Simulate a pipe using the `echo` and `grep` commands
output = subprocess.run(['echo', 'Hello, World!'], capture_output=True, text=True).stdout
output = subprocess.run(['grep', 'World'], input=output, capture_output=True, text=True).stdout


# Print the output
print(output)

In this code, the echo command is executed using subprocess.run() with the capture_output=True and text=True parameters. This captures the output of the echo command and returns it as a string.

The captured output is then passed as the input to the grep command using the input parameter of subprocess.run(). This effectively simulates a pipe, directing the output of the echo command to the input of the grep command.

The output of the grep command is captured and printed.

When you run this code, you’ll see the output of the grep command, which contains the line "Hello, World!" as it matches the pattern "World".

Practical Ideas

In addition to the basic usage of the subprocess module, there are many practical ideas for leveraging it in your projects. Here are a couple of examples:

Creating a New Project: An Example

When starting a new project, you often need to set up a directory structure and create initial files. Instead of manually creating each file and directory, you can automate this process using the subprocess module.

Here’s an example that demonstrates how to create a new project using the subprocess module:

import os
import subprocess


# Prompt the user for the project name
project_name = input("Enter the project name: ")


# Create the project directory
os.mkdir(project_name)


# Change to the project directory
os.chdir(project_name)


# Create the source directory
os.mkdir('src')


# Create the main Python file
with open('src/main.py', 'w') as file:
    file.write('# This is the main file of the project\n')


# Create the README file
with open('README.md', 'w') as file:
    file.write(f'# {project_name}\n\nA new project created with Python\n')


# Open the project directory in the default file manager
subprocess.run(['open', '.'])

In this code, the user is prompted to enter a project name using input(). The project directory is created using os.mkdir(), and the process’s working directory is changed using os.chdir().

The src directory is created inside the project directory using os.mkdir().

The main Python file is created using open() in write mode, and the README.md file is created in a similar way. The contents of the files are written using the file object’s write() method, which creates the initial content of the files.

Finally, the project directory is opened in the default file manager using subprocess.run() and the open command.

When you run this code, you’ll see a new project directory created with the specified name. Inside the directory, you’ll find a src directory containing a main.py file, and a README.md file with the project name as the heading.

Changing Extended Attributes

Extended attributes are additional metadata associated with files and directories on certain filesystems. They provide a way to attach additional information to files and directories beyond the standard attributes like permissions and timestamps.

The subprocess module can be used to interact with the xattr command-line tool on macOS and Linux systems, allowing you to manage extended attributes programmatically.

Here’s an example that demonstrates how to change extended attributes using the subprocess module:

import subprocess


# Set an extended attribute on a file
subprocess.run(['xattr', '-w', 'com.example.metadata', 'value', 'file.txt'])


# Get the value of an extended attribute
output = subprocess.run(['xattr', '-p', 'com.example.metadata', 'file.txt'], capture_output=True, text=True).stdout


# Print the output
print(output)

In this code, the xattr command is used to set an extended attribute on a file using subprocess.run() with the appropriate parameters.

The value of the extended attribute is then retrieved using subprocess.run() with the -p option.

The output is captured and printed.

To run this code, you need to have the xattr command-line tool installed on your system. You can install it using the appropriate package manager for your system (brew for macOS, apt for Ubuntu, etc.).

When you run this code, you’ll see the output of the xattr command, which is the value of the extended attribute associated with the file.

Python Modules Associated With subprocess

The subprocess module is a powerful tool for executing and interacting with external processes. However, it’s not the only Python module available for these tasks. There are several other modules you can explore for more specific use cases or expanded functionality.

Here are a few notable Python modules associated with subprocess:

  • shlex: The shlex module provides a way to parse shell-style strings into tokens, making it easier to work with shell commands and scripts.
  • argparse: The argparse module allows you to define and parse command-line arguments and options, making it easier to create command-line interfaces for your Python scripts.
  • pty: The pty module provides a way to run a command in a pseudo-terminal, which can be useful when interacting with programs that expect to be run in a terminal environment.

By exploring these modules and the subprocess module, you can expand your capabilities when working with external processes and creating command-line interfaces in Python.

The Popen Class

The Popen class is another important component of the subprocess module. It provides more control and flexibility when working with processes, allowing you to perform advanced process management and handle complex scenarios.

Using Popen()

To create a new process using the Popen class, you need to instantiate the class and pass the command to run as a list to the constructor. You can also specify additional parameters to customize the behavior of the process.

Here’s an example that demonstrates how to use the Popen class to create a new process:

import subprocess


# Create the process
process = subprocess.Popen(['echo', 'Hello, World!'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)


# Wait for the process to finish and get the output
output, error = process.communicate()


# Print the output
print(output)

In this code, the Popen class is instantiated with ['echo', 'Hello, World!'] as the command to run. The stdout=subprocess.PIPE parameter specifies that the standard output of the process should be captured and returned as a string. Similarly, the stderr=subprocess.PIPE parameter specifies that the standard error of the process should also be captured.

The communicate() method of the process object is called to wait for the process to finish and get its output. The output and error output are returned as tuples, with the output assigned to output and the error output assigned to error. In this example, since the command echo doesn’t produce any error output, error will be an empty string.

Finally, the output is printed.

When you run this code, you’ll see the output of the echo command printed.

Connecting Two Processes Together With Pipes

One powerful feature of the Popen class is the ability to connect two processes together with pipes. This allows you to build complex pipelines and pass data between processes.

Here’s an example that demonstrates how to connect two processes together with pipes using the Popen class:

import subprocess


# Create the first process with a pipe for stdout
process1 = subprocess.Popen(['echo', 'Hello, World!'], stdout=subprocess.PIPE, text=True)


# Create the second process with a pipe for stdin
process2 = subprocess.Popen(['grep', 'World'], stdin=process1.stdout, stdout=subprocess.PIPE, text=True)


# Close the first process's stdout pipe, as it won't be used anymore
process1.stdout.close()


# Wait for the second process to finish and get the output
output, _ = process2.communicate()


# Print the output
print(output)

In this code, two Popen instances are created: process1 and process2.

process1 is created with ['echo', 'Hello, World!'] as the command to run and stdout=subprocess.PIPE to create a pipe for the standard output of the process.

process2 is created with ['grep', 'World'] as the command to run and stdin=process1.stdout to connect the standard input of process2 to the output of process1’s pipe.

The stdout pipe of process1 is closed using the stdout.close() method, as it won’t be used anymore.

The communicate() method of process2 is called to wait for process2 to finish and get its output. The error output is discarded using the _ variable, as the echo command doesn’t produce any error output.

Finally, the output is printed.

When you run this code, you’ll see the output of process2, which contains the line "Hello, World!" as it matches the pattern "World".

Interacting Dynamically With a Process

The Popen class also allows you to interact with a process dynamically, sending input and receiving output as needed. This can be useful for long-running processes or interactive programs.

Here’s an example that demonstrates how to interact dynamically with a process using the Popen class:

import subprocess


# Create the process
process = subprocess.Popen(['python', '-i'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)


# Send input to the process
process.stdin.write("print('Hello, World!')\n")
process.stdin.flush()


# Get the output from the process
output = process.stdout.readline().strip()


# Print the output
print(output)


# Close the process's stdin pipe
process.stdin.close()


# Wait for the process to finish
process.wait()

In this code, the Popen class is instantiated with ['python', '-i'] as the command to run. The -i option starts the Python interpreter in interactive mode, allowing multiple commands to be executed.

After creating the process, input is sent to the process using process.stdin.write() and process.stdin.flush(). These methods write the input to the process’s stdin pipe and ensure the data is flushed and sent immediately.

The output of the process is read using process.stdout.readline(), which reads a line of output until a newline character is encountered. The leading and trailing whitespace is stripped using strip().

Finally, the output is printed. The stdin pipe is closed using process.stdin.close(), and the process is waited for using process.wait().

When you run this code, you’ll see the output of the process, which contains the line "Hello, World!" as it was executed in the interactive mode.

Conclusion

The subprocess module in Python provides a powerful way to interact with external processes, allowing you to execute shell commands, run command-line applications, and even launch GUI applications. With the knowledge and techniques covered in this tutorial, you can now leverage the subprocess module to simplify your command-line scripting, automate tasks, and handle input and output for external programs.

By understanding the basics of processes and subprocesses, exploring the Python subprocess module, and examining practical examples, you can enhance your Python skills and become more proficient in working with external processes in your projects.