hedii/php-crawler

A crawler application written with php and Laravel that finds email addresses on the internets.

php-crawler

This repository has not been updated since a long time and I am not using it anymore. Please feel free to fork it and do whatever you want with it., _(*1)

, _(*2)

A crawler application with a php backend using Laravel, and a js frontend using vuejs, that finds email addresses on the internets., _(*3)

Given an entry point url, the crawler will search for emails in all the urls available from this entry point domain name. The emails are downloadable as a text file., _(*4)

Multiple users can start searching for emails without being able see the other users' searches (searches are related to a single user)., _(*5)

Screenshots

, _(*6)

, _(*7)

, _(*8)

, _(*9)

Server requirements

• PHP >= 7.2.0
• OpenSSL PHP Extension
• PDO PHP Extension
• Mbstring PHP Extension
• Tokenizer PHP Extension
• XML PHP Extension

Installation

• Create a mysql database (default name: crawler)
• Install the project with composer:

composer create-project hedii/php-crawler crawler
cd crawler

• Open the .env file, check the database credentials, and modify it if needed:

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=crawler
DB_USERNAME=root
DB_PASSWORD=your_password_here

• In the .env file, set the application url:

APP_URL=http://crawler.localhost

• Build the crawler application

php artisan crawler:build

• Point your web server's document / web root to be the public directory: /some/path/crawler/public. The index.php in this directory serves as the front controller for all HTTP requests entering your application. See Laravel documentation. I highly recommend using Laravel Valet if you are using a Mac. Otherwise, check Laravel Homestead.
• Done

Usage

• Navigate to your php-crawler website
• Register a new account
• Create a new search
• Create more searches
• Download the found emails

Testing

composer test

Contributing

All contributions are welcome :), _(*10)

Please write some tests if you are adding or modifying features., _(*11)

License

php-crawler is open-sourced software licensed under the MIT license., _(*12)