yii2-crawler

Advanced web crawler module, _(*1)

Features

Multiple connections on single object
Define and use multiple proxys
Define and use multiple identities (usarname/password)

Install

Install using composer:, _(*2)

The preferred way to install this extension is through composer., _(*3)

Either run, _(*4)

php composer.phar require --prefer-dist macklus/yii2-geoselect "*"

or add, _(*5)

"macklus/yii2-geoselect": "*"

to the require section of your composer.json file., _(*6)

Usage

Create new object:, _(*7)

<?php

namespace app\controllers;

use macklus\Crawler\Crawler;

class TestController extends Controller
{
    public function actionCraw()
    {
        $crawler = new Crawler();
    }
}

UserAgent

$crawler->setUA($browser)
/*
 * Browser could be chrome, firefox or explorer
 */

Identities

$crawler->setUser($name, $user, $password)
$crawler->setUsers(['user1' => ['username' => 'jhon', 'password' => 'doe']])

Proxys

$crawler->setProxy($name, $string)
$crawler->setProxys([['proxy1' => 'proxystring', 'proxy2' => 'proxy2string]])
/*
 * string should contain all proxy info, like http://username:password@proxy.thing.com:8080/
 */

01/04 2016

dev-master

9999999-dev

Advanced web crawler

Sources Download

GPL-3.0+

The Requires

by macklus

extension yii2 crawler goutte proxy

yii2-extension yii2-crawler