2017 © Pedro Peláez
 

library php-csv-db-import

Handling of large bulk data into database tables.

image

keboola/php-csv-db-import

Handling of large bulk data into database tables.

  • Thursday, May 3, 2018
  • by Keboola
  • Repository
  • 5 Watchers
  • 3 Stars
  • 4,088 Installations
  • PHP
  • 1 Dependents
  • 0 Suggesters
  • 1 Forks
  • 2 Open issues
  • 53 Versions
  • 6 % Grown

The README.md

Keboola Database table importer

Build Status, (*1)

Handling of large bulk data into database tables., (*2)

Supported engines:

  • AWS Redshift, Snowflake
    • Load data from CSV stored in AWS S3
    • Load data from another Redshift table in same database

Features

  • Full load - destination table is truncated before load
  • Incremental load - data are merged
  • Primary key dedup for all engines
  • Convert empty values to NULL (using convertEmptyValuesToNull option)

Development

Preparation

  • Create AWS S3 bucket and IAM user using aws-services.json cloudformation template.
  • Create Redshift cluster
  • Download Snowflake driver (.deb) and place it into root of repository (./snowflake-odbc.deb)
  • Create .env file. Use output of aws-services cloudfront stack to fill the variables and your Redshift credentials.
REDSHIFT_HOST=
REDSHIFT_PORT=5439
REDSHIFT_USER=
REDSHIFT_DATABASE=
REDSHIFT_PASSWORD=

SNOWFLAKE_HOST=
SNOWFLAKE_PORT=
SNOWFLAKE_USER=
SNOWFLAKE_PASSWORD=
SNOWFLAKE_DATABASE=
SNOWFLAKE_WAREHOUSE=

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_S3_BUCKET=
AWS_REGION=

Upload test fixtures to S3:, (*3)

docker-compose run php php ./tests/loadS3.php

Redshift settings

User and database are required for tests. You can create them:, (*4)

CREATE USER keboola_db_import PASSWORD 'YOUR_PASSWORD';
CREATE DATABASE keboola_db_import;
GRANT ALL ON DATABASE keboola_db_import TO keboola_db_import;

Snowflake settings

Role, user, database and warehouse are required for tests. You can create them:, (*5)

CREATE ROLE "KEBOOLA_DB_IMPORT";
CREATE DATABASE "KEBOOLA_DB_IMPORT";
GRANT ALL PRIVILEGES ON DATABASE "KEBOOLA_DB_IMPORT" TO ROLE "KEBOOLA_DB_IMPORT";

CREATE WAREHOUSE "KEBOOLA_DB_IMPORT" WITH WAREHOUSE_SIZE = 'XSMALL' WAREHOUSE_TYPE = 'STANDARD' AUTO_SUSPEND = 3600 AUTO_RESUME = TRUE;
GRANT USAGE ON WAREHOUSE "KEBOOLA_DB_IMPORT" TO ROLE "KEBOOLA_DB_IMPORT" WITH GRANT OPTION;

CREATE USER "KEBOOLA_DB_IMPORT"
PASSWORD = "YOUR_PASSWORD"
DEFAULT_ROLE = "KEBOOLA_DB_IMPORT";

GRANT ROLE "KEBOOLA_DB_IMPORT" TO USER "KEBOOLA_DB_IMPORT";

Tests Execution

Run tests with following command., (*6)

docker-compose run --rm tests

Redshift and S3 credentials have to be provided., (*7)

The Versions

03/05 2018

dev-master

9999999-dev

Handling of large bulk data into database tables.

  Sources   Download

MIT

The Requires

 

The Development Requires

by Martin Halamicek

03/05 2018

4.1.0

4.1.0.0

Handling of large bulk data into database tables.

  Sources   Download

MIT

The Requires

 

The Development Requires

02/05 2018

dev-martin-snowflake-perf-debug

dev-martin-snowflake-perf-debug

Handling of large bulk data into database tables.

  Sources   Download

MIT

The Requires

 

The Development Requires

27/04 2018

dev-martin-snowflake-slices-limit

dev-martin-snowflake-slices-limit

Handling of large bulk data into database tables.

  Sources   Download

MIT

The Requires

 

The Development Requires

27/04 2018

4.0.0

4.0.0.0

Handling of large bulk data into database tables.

  Sources   Download

MIT

The Requires

 

The Development Requires

27/04 2018

dev-martin-php-require

dev-martin-php-require

Handling of large bulk data into database tables.

  Sources   Download

MIT

The Requires

 

The Development Requires

27/04 2018

dev-martin-php-7

dev-martin-php-7

Handling of large bulk data into database tables.

  Sources   Download

MIT

The Requires

 

The Development Requires

23/04 2018

dev-martin-cleanup

dev-martin-cleanup

  Sources   Download

The Requires

 

The Development Requires

20/11 2017

3.0.1

3.0.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

12/07 2017

3.0.0

3.0.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

10/07 2017

dev-tests-utf

dev-tests-utf

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

03/05 2017

2.4.4

2.4.4.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

28/04 2017

2.4.3

2.4.3.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

24/04 2017

2.4.1

2.4.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

24/04 2017

2.4.2

2.4.2.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

19/04 2017

2.4.0

2.4.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

28/11 2016

2.3.1

2.3.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

15/11 2016

2.3.0

2.3.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

23/09 2016

dev-snflkBindingQuotes

dev-snflkBindingQuotes

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

16/09 2016

2.2.1

2.2.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

02/09 2016

2.2.0

2.2.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

16/08 2016

2.1.0

2.1.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

15/08 2016

2.0.0

2.0.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

02/08 2016

dev-snowflakeBindingTest

dev-snowflakeBindingTest

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

22/07 2016

1.5.1

1.5.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

20/07 2016

1.5.0

1.5.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

23/06 2016

1.4.1

1.4.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

13/06 2016

1.4.0

1.4.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

03/06 2016

1.3.11

1.3.11.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

03/06 2016

1.3.10

1.3.10.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

02/06 2016

1.3.9

1.3.9.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

02/06 2016

1.3.8

1.3.8.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

31/05 2016

1.3.7

1.3.7.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

31/05 2016

1.3.6

1.3.6.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

30/05 2016

1.3.5

1.3.5.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

24/05 2016

1.3.4

1.3.4.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

18/05 2016

1.3.3

1.3.3.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

12/05 2016

1.3.2

1.3.2.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

11/05 2016

1.3.1

1.3.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

10/05 2016

1.3.0

1.3.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

09/05 2016

1.1.1

1.1.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

09/05 2016

1.2.0

1.2.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

28/04 2016

1.1.0

1.1.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

26/01 2016

1.0.9

1.0.9.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

18/01 2016

1.0.8

1.0.8.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

13/01 2016

1.0.7

1.0.7.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

11/01 2016

1.0.6

1.0.6.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

11/01 2016

1.0.5

1.0.5.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

11/01 2016

1.0.4

1.0.4.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

11/01 2016

1.0.3

1.0.3.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

08/01 2016

1.0.2

1.0.2.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

08/01 2016

1.0.1

1.0.1.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek

08/01 2016

1.0.0

1.0.0.0

  Sources   Download

The Requires

 

The Development Requires

by Martin Halamicek