multi-threaded & multi-process in Python

Use Python multiprocessing (multiprocessing) enables multiple CPU can work at the same time, greatly enhance the utilization efficiency.

IO-intensive and CPU intensive

CPU-intensive: linear program execution, CPU intensive, always close to 100% (eg: replacing a large number of regular text matching)
IO-intensive: a lot of time spent waiting for program I/O operation, CPU always idle, at about 10% (such as network request)

multi-threaded and multi-process

Python multi-threaded and other programming languages are very different from the multi-threaded, Python multi-threaded even in the multi-core CPU host can only use a CPU, not the true sense of the concurrent implementation.

For IO-intensive programs, the use of multi-threaded technology can wait at intervals of CPU blocking the implementation of other operations, you can improve CPU efficiency.

For CPU-intensive programs, the original CPU is a busy state, forced the use of multi-threaded technology, will not increase the efficiency of CPU use, but increase the cost of thread switching, may lead to multi-threaded implementation efficiency is not as single-threaded Implementation efficiency. In the multi-core CPU, a host can use python multi-process (multiprocessing) makes the program using multiple CPU, you can achieve the program in the real sense of the parallel (the number of processes does not exceed the CPU core).

Example

Multi-threaded

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import Queue

import threading

queueLock = threading.Lock()

workQueue = Queue.Queue()

thread_num = 11

threads = []

class MyThread (threading.Thread):

    def __init__(self, Queue,id):

        threading.Thread.__init__(self)

        self.q = Queue

        self.id = id

    def run(self):

        while not workQueue.empty():

            get_info(self.q.get(),self.id)

def get_info(data,threading_id):

    for i in range(1000000):

        a=str(threading)+data

def main():

    for i in range(1000):

        workQueue.put(str(i))

    for i in range(thread_num):

        thread = MyThread(workQueue, i)

        thread.start()

        threads.append(thread)

    for t in threads:

        t.join()

if __name__ == '__main__':

    main()

Run the Python program to monitor CPU usage, Only one CPU is in use

Multi-process

#!/usr/bin/env python

# -*- coding: utf-8 -*-

import Queue

from multiprocessing import Process

processors = []

processor_num = 11

workQueue = Queue.Queue()

def single_processor(q,id):

    while not workQueue.empty():

            get_info(q.get(),id)

def get_info(data,processor_id):

    for i in range(1000000):

        a=str(processor_id)+data

def main():

    for i in range(1000):

        workQueue.put(str(i))

    for i in range(processor_num):

        p = Process(target=single_processor, args=(workQueue,i))

        p.start()

        processors.append(p)

    for i in processors:

         i.join()

if __name__ == '__main__':

    main()

Run the Python program, monitor the use of the CPU can be seen that 11 CPU are in 100% of the use of state