watzon
/
arachnid
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases 3 Wiki Activity

Updated readme

This commit is contained in:
Chris Watson 2019-06-26 20:23:18 -07:00
parent 7fa5205096
commit 32bf4d89b8
No known key found for this signature in database
GPG Key ID: 37DAEF5F446370A4
1 changed files with 6 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
README.md
View File

@ -8,9 +8,9 @@ Arachnid is a fast and powerful web scraping framework for Crystal. It provides
- [Usage](#Usage) - [Usage](#Usage)
- [Configuration](#Configuration) - [Configuration](#Configuration)
- [Crawling](#Crawling) - [Crawling](#Crawling)
- [Arachnid#start_at(url, **options, &block : Agent ->)](#Arachnidstartaturl-options-block--Agent) - [`Arachnid#start_at(url, **options, &block : Agent ->)`](#Arachnidstartaturl-options-block--Agent)
- [Arachnid#site(url, **options, &block : Agent ->)](#Arachnidsiteurl-options-block--Agent) - [`Arachnid#site(url, **options, &block : Agent ->)`](#Arachnidsiteurl-options-block--Agent)
- [Arachnid#host(name, **options, &block : Agent ->)](#Arachnidhostname-options-block--Agent) - [`Arachnid#host(name, **options, &block : Agent ->)`](#Arachnidhostname-options-block--Agent)
- [Crawling Rules](#Crawling-Rules) - [Crawling Rules](#Crawling-Rules)
- [Events](#Events) - [Events](#Events)
- [`every_url(&block : URI ->)`](#everyurlblock--URI) - [`every_url(&block : URI ->)`](#everyurlblock--URI)
@ -167,15 +167,15 @@ There are also a few class properties on `Arachnid` itself which are used as the
Arachnid provides 3 interfaces to use for crawling: Arachnid provides 3 interfaces to use for crawling:
#### Arachnid#start_at(url, **options, &block : Agent ->) #### `Arachnid#start_at(url, **options, &block : Agent ->)`
`start_at` is what you want to use if you're going to be doing a full crawl of multiple sites. It doesn't filter any urls by default and will scan every link it encounters. `start_at` is what you want to use if you're going to be doing a full crawl of multiple sites. It doesn't filter any urls by default and will scan every link it encounters.
#### Arachnid#site(url, **options, &block : Agent ->) #### `Arachnid#site(url, **options, &block : Agent ->)`
`site` constrains the crawl to a specific site. "site" in this case is defined as all paths within a domain and it's subdomains. `site` constrains the crawl to a specific site. "site" in this case is defined as all paths within a domain and it's subdomains.
#### Arachnid#host(name, **options, &block : Agent ->) #### `Arachnid#host(name, **options, &block : Agent ->)`
`host` is similar to site, but stays within the domain, not crawling subdomains. `host` is similar to site, but stays within the domain, not crawling subdomains.