When users run Apify.launchPuppetter() on Docker image without Chromium, they see:

Error: Failed to launch chrome! spawn /usr/src/app/node_modules/puppeteer/.local-chromium/linux-706915/chrome-linux/chrome ENOENT

We should show some better error telling them how to fix it.

Affected file: grab/document.py

>>> import libgenapi
... /usr/local/lib/python3.9/site-packages/grab/document.py:35: DeprecationWarning: defusedxml.lxml is no longer supported and will be removed in a future release.
  import defusedxml.lxml

The defusedxml.lxml subpackage will be removed in a future release, so be

Hello,

Bit a silly comment and possibly it has been given before (though I could not find it in the issues list).
But the Conditions of Use from IMDb explicitly state that:

Robots and Screen Scraping: You may not use data mining, robots, screen scraping, or similar data gathering and extraction tools on this site, except with our express written consent as noted below.

Hence, u

It took me hours to figure this out so I want to help anyone else having trouble getting this running on Heroku.

Kimurai uses lsof, so an Aptfile with the single line lsof needs to be included in the root folder along with the heroku buildpack. Can you add this to the docs? Thanks!

Might be good to add this to the documentation :

• user-agent strings provided is not "random" because user-agents.json.gz contains browser fingerprints not user-agents strings so it will give you a representation of what is most used at the time period depending of the version of the lib.
• the db is updated in full not incremental, so old UA are ventilated

For example :

Top user agen

I have to admit I haven't spent any time troubleshooting, but it does look like this doesn't function as is anymore.

wayback-machine-scraper -f 20080623 -t 20080623 news.ycombinator.com
2019-03-21 11:50:11 [scrapy.utils.log] INFO: Scrapy 1.6.0 started (bot: scrapybot)
2019-03-21 11:50:11 [scrapy.utils.log] INFO: Versions: lxml 4.3.2.0, libxml2 2.9.5, cssselect 1.0.3, parsel 1.5.1, w3li

URL: https://www.il-fa.com/
Documents URL: https://www.il-fa.com/public-access/board-documents/
Spider Name: il_finance_authority
Agency Name: Illinois Finance Authority

See the contribution guide for information on how to get started