2019-06-30 23:53:02 +00:00
|
|
|
crystal_doc_search_index_callback({"repository_name":"github.com/watzon/arachnid","body":"# Arachnid\n\nArachnid is a fast and powerful web scraping framework for Crystal. It provides an easy to use DSL for scraping webpages and processing all of the things you might come across.\n\n- [Arachnid](#Arachnid)\n - [Installation](#Installation)\n - [Examples](#Examples)\n - [Usage](#Usage)\n - [Configuration](#Configuration)\n - [Crawling](#Crawling)\n - [`Arachnid#start_at(url, **options, &block : Agent ->)`](#Arachnidstartaturl-options-block--Agent)\n - [`Arachnid#site(url, **options, &block : Agent ->)`](#Arachnidsiteurl-options-block--Agent)\n - [`Arachnid#host(name, **options, &block : Agent ->)`](#Arachnidhostname-options-block--Agent)\n - [Crawling Rules](#Crawling-Rules)\n - [Events](#Events)\n - [`every_url(&block : URI ->)`](#everyurlblock--URI)\n - [`every_failed_url(&block : URI ->)`](#everyfailedurlblock--URI)\n - [`every_url_like(pattern, &block : URI ->)`](#everyurllikepattern-block--URI)\n - [`urls_like(pattern, &block : URI ->)`](#urlslikepattern-block--URI)\n - [`all_headers(&block : HTTP::Headers)`](#allheadersblock--HTTPHeaders)\n - [`every_resource(&block : Resource ->)`](#everyresourceblock--Resource)\n - [`every_ok_page(&block : Resource ->)`](#everyokpageblock--Resource)\n - [`every_redirect_page(&block : Resource ->)`](#everyredirectpageblock--Resource)\n - [`every_timedout_page(&block : Resource ->)`](#everytimedoutpageblock--Resource)\n - [`every_bad_request_page(&block : Resource ->)`](#everybadrequestpageblock--Resource)\n - [`def every_unauthorized_page(&block : Resource ->)`](#def-everyunauthorizedpageblock--Resource)\n - [`every_forbidden_page(&block : Resource ->)`](#everyforbiddenpageblock--Resource)\n - [`every_missing_page(&block : Resource ->)`](#everymissingpageblock--Resource)\n - [`every_internal_server_error_page(&block : Resource ->)`](#everyinternalservererrorpageblock--Resource)\n - [`every_txt_page(&block : Resource ->)`](#everytxtpageblock--Resource)\n - [`every_html_page(&block : Resource ->)`](#everyhtmlpageblock--Resource)\n - [`every_xml_page(&block : Resource ->)`](#everyxmlpageblock--Resource)\n - [`every_xsl_page(&block : Resource ->)`](#everyxslpageblock--Resource)\n - [`every_doc(&block : Document::HTML | XML::Node ->)`](#everydocblock--DocumentHTML--XMLNode)\n - [`every_html_doc(&block : Document::HTML | XML::Node ->)`](#everyhtmldocblock--DocumentHTML--XMLNode)\n - [`every_xml_doc(&block : XML::Node ->)`](#everyxmldocblock--XMLNode)\n - [`every_xsl_doc(&block : XML::Node ->)`](#everyxsldocblock--XMLNode)\n - [`every_rss_doc(&block : XML::Node ->)`](#everyrssdocblock--XMLNode)\n - [`every_atom_doc(&block : XML::Node ->)`](#everyatomdocblock--XMLNode)\n - [`every_javascript(&block : Resource ->)`](#everyjavascriptblock--Resource)\n - [`every_css(&block : Resource ->)`](#everycssblock--Resource)\n - [`every_rss(&block : Resource ->)`](#everyrssblock--Resource)\n - [`every_atom(&block : Resource ->)`](#everyatomblock--Resource)\n - [`every_ms_word(&block : Resource ->)`](#everymswordblock--Resource)\n - [`every_pdf(&block : Resource ->)`](#everypdfblock--Resource)\n - [`every_zip(&block : Resource ->)`](#everyzipblock--Resource)\n - [`every_image(&block : Resource ->)`](#everyimageblock--Resource)\n - [`every_content_type(content_type : String | Regex, &block : Resource ->)`](#everycontenttypecontenttype--String--Regex-block--Resource)\n - [`every_link(&block : URI, URI ->)`](#everylinkblock--URI-URI)\n - [Content Types](#Content-Types)\n - [Parsing HTML](#Parsing-HTML)\n - [Contributing](#Contributing)\n - [Contributors](#Contributors)\n\n## Installation\n\n1. Add the dependency to your `shard.yml`:\n\n ```yaml\n dependencies:\n arachnid:\n github: watzon/arachnid\n version: ~> 0.1.0\n ```\n\n2. Run `shards install`\n\n## Examples\n\nArachnid pro
|
