Profile Python processes across multiple cores

Q&A
,
1

I am running many different processes with Python’s multiprocessing module. What would be the best way to profile each process so that I could determine the limiting process for my Python program? I would like to see the memory usage and time taken on each core for each process. What would be the easiest method to display this information?

2

One approach to display this information is by using external libraries such as psutil for resource monitoring and time' or timeit’ for time measurements. psutil is a cross-platform library that can be used to monitor memory and CPU usage for each process.

  1. install them using pip:

pip install psutil

  1. Here is a minimum code for Resource Monitoring:
import psutil
import multiprocessing
import time

def worker_function():
    process = psutil.Process()
    while True:
        print(f"Memory Usage: {process.memory_info().rss / 1024 / 1024:.2f} MB")
        print(f"CPU Usage: {psutil.cpu_percent()}%")
        time.sleep(1)

if __name__ == "__main__":
    num_processes = multiprocessing.cpu_count()
    processes = []
    for _ in range(num_processes):
        process = multiprocessing.Process(target=worker_function)
        process.start()
        processes.append(process)

    for process in processes:
        process.join()
  1. Measure Time Taken:
    You can use the built-in time module to measure the execution time of each process. Create a timestamp at the beginning and end of the process to calculate the elapsed time.

import time

def worker_function():
    start_time = time.time()
    # Your process code here
    end_time = time.time()
    elapsed_time = end_time - start_time
    print(f"Process time: {elapsed_time:.2f} seconds")
  1. Aggregate Data:
    Since you are running multiple processes, you might want to aggregate the collected data to determine which process is the limiting factor.
3

There has been a lot of development in this Area especially as the C language has been integrated into python.

That been said you will choose the application that meets your needs. Go over the writeups