In general, the scraping cycle for spiders follows this basic pattern:

- Initial Requests are generated along with the function that should be called on those Requests (callback function)
- The callback function parses responses and returns item objects, Request objects, or an iterable of either aforementioned objects
- In the callback functions, page contents are parsed to generate data
- Items returned by the functions are added to a database or written to a file

Scrapy Spider Scraping Cycle

Spiders are custom classes that determine how your web crawler will scrape information. They are accessed with scrapy.spiders.Spider

University of Michigan - Ann Arbor

Scrapy is a Python framework that is used to create web crawlers. These web crawlers can extract structured data directly from web sites or by using APIs. 

Scrapy

Scrapy Documentation

To install Scrapy using pip, type the following into your terminal:
```bash
$ pip install scrapy
```

Installing Scrapy

https://docs.scrapy.org/en/latest/intro/tutorial.html

Learn Before

Related

Learn After