Stop and resume the spider where it stopped

[shellking4]

Can we stop and resume the spider at a given point ? How is it done ?

[Ziinc]

You can call Crawly.Engine.stop_spider(MySpider) from anywhere in your project code.

[Ziinc]

please use discussions for such question

[mvkvc]

Would it be possible if writing to CSV to continue later with the same file and only add new items that are not already there?

[oltarasenko]

Hey. Crawly does not support it right now. We may build it as a feature in the future. However, this approach might have some limitations (I am thinking of the cases when the website is being updated in between crawls) compared to daily crawls.

[mvkvc]

Thank you! Good to understand.