ThreadPool + 딕셔너리 컴프리헨션

참조

https://docs.python.org/ko/3/library/concurrent.futures.html?highlight=threadpool#concurrent.futures.ThreadPoolExecutor 

concurrent.futures — 병렬 작업 실행하기 — Python 3.10.5 문서

---

일단 Pool 개념은 https://wonhyeok1994.tistory.com/163?category=1079515 

POOL + Cursor

위의  DB connection Pool 이랑 같은 개념

과부하를 막기위해 미리 특정 수만큼 생성해 두고 사용하는 방식

프로세스 중 병렬 작업처리가 많아지면 스레드 개수가 증가되고 그에따른 스레드생성과 스케줄링으로 인해 CPU가 바빠져 메모리 사용량이 늘어난다. 따라서 시스템성능이 저하되고, 갑작스러운 병렬작업의 폭증에 따른 스레드 폭증을 막으려고 스레드 풀 Thread Pool을 사용한다.

 

스레드 풀은 작업처리에 사용되는 스레드를 제한된 개수만큼 정해놓고 작업큐 (Queue)에 들어오는 작업들을 하나씩 스레드가 맡아 처리한다. 그렇게 하면 작업처리 요청이 폭증되어도 스레드의 전체개수가 늘어나지 않으므로(제한해서 하나씩 처리하기 때문)  시스템 성능이 급격히 저하되지 않음

 

예제_1

먼저 알 부분 :  딕셔너리 컴프리 헨션 사용했음 // { func_1(x) : x for x in X_list }

future_to_url = {executor.submit(load_url, url, 60): url for url in URLS} : 뜻 <- URLS을 하나씩 url에 넣고 넣는대로 executor.submit(load_url, url, 60) 함수를 실행하고 리턴값을 Key에 url값을 Value에 넣어서 dictionary 만듬

 

키워드 :

concurrent.futures.ThreadPoolExecutor(max_workers=5).submit(load_url, url, 60)  :  <- load_url(target)함수의 인자에 (url,60)을 넣어서 쓰레드풀Queue에 put 해줌 + load_url(target)에 대한 Future object를 리턴해줍니다.

Future obj : <class 'concurrent.futures._base.Future'>

concurrent.futures.as_completed( future객체 )  : <- future객체가 완료(ThreadPool에서)되는 대로 future객체인스턴스의 이터레이터(Future obj)를 반환함

 

import concurrent.futures
import urllib.request

URLS = ['http://www.daum.net/',
        'http://www.naver.com/',
        'http://www.naver.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

# Retrieve a single page and report the URL and contents
def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

# We can use a with statement to ensure threads are cleaned up promptly
with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    # Start the load operations and mark each future with its URL
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    print('future_to_url type : ',type(future_to_url))
    print('future_to_url : ',future_to_url)
    print()
    for future in concurrent.futures.as_completed(future_to_url):
        print('future : ',future)
        url = future_to_url[future]
        print('url : ',url)
        try:
            data = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r page is %d bytes' % (url, len(data)))