Natural support for multiprocessing in crawlers

[Pijukatel]

Currently the design is based around async code and it works well for many cases. There are also cases where multiprocessing would be beneficial. This is just a discussion to group some of the related points together. It is not an issue.

Here are some existing issues and some ad hoc points related to it:

• Autoscaling based on multiple cpu utilization for single process crawlers? #1119)
• Add get_run_state method to BasicCrawler #925
• Request handler should be able to timeout blocking sync code in user defined handler #908
• scaling http-based crawlers: When running the crawlers it can't use all the resources available. The crawler performance saturates at some point and giving it more resources does not improve the performance further. On Apify platform it is currently for example 8 GB ~ 2 cores, beyond that there is no improvement in speed.
• cpu heavy processing in request handlers