今天写爬虫的时候看bs4的官方文档,说使用lxml作为解析器比使用内置解析器速度快很多,于是写了个函数测试了一下,但不知道为什么,测试跑出来的结果刚好相反,暂时未找到原因,先做下记录后续继续研究。
测试代码如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
| from bs4 import BeautifulSoup import time from wswp.WebScraping import download def time_test(): FIELDS = ('area', 'population', 'iso', 'country', 'capital', 'continent', 'tld', 'currency_code', 'currency_name', 'phone', 'postal_code_format', 'postal_code_regex', 'languages', 'neighbours') NUM_ITERATIONS = 1000 # number of times to test each scraper html = download('http://example.webscraping.com/view/Aland-Islands-2') start = time.time() for name in ['html.parser', 'lxml']: for i in range(NUM_ITERATIONS): result = {} soup = BeautifulSoup(html, name) for field in FIELDS: result[field]=soup.find('tr', id='places_%s__row' % field).find('td', class_='w2p_fw').text assert(result['area'] == '1,580 square kilometres') end = time.time() print('%s the time is: %.2f seconds' % (name, (end - start))) time_test()
|