Skip to content

Allow the user to change the numbers of rows to analyze for data structure and types #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
runarbu wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Allow the user to change the numbers of rows to analyze for data structure and types #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ Use the `--help` option to get a full, up-to-date look at what the options are,
* `--id`: ID to use for resuming stream; if an input file is provided, the filename is used by default.
* `--restart`: Restart any resuming as well as remove existing data and tables. **WARNING: DELETES DATA** Use this is you don't have a unique key defined or if your model has changed.
* `--batch-size`: Numbers of rows to import at once. Default is 1000. Use lower or higher numbers depending the database and how it is configured.
* `--guess-limit`: Numbers of rows to analyze for data structure and types. Default is 1000. Use a higher number than the default if your data have many inconsistencies.
* `--type`: Force type of parsing. This is determined from filename and defaults to `csv`. Valid values are `csv`, `tsv`, `json`, `ndjson`, `html`, or `custom`.
* `--csv-headers`: Use the keyword, false, if there are no headers. Or use a comma separated list of headers. Defaults to reading headers from file. Forces type to CSV.
* `--csv-delimiter`: CSV delimiter character. Defaults to `,`. If you need to use a tab character, use `--csv-delimiter=$'\t'`. Forces type to CSV.
Expand Down
2 changes: 2 additions & 0 deletions bin/tables
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ command.option("-p, --data [file]", "Path to data file for resuming streams. De
command.option("-z, --id [name]", "ID to use for resuming stream; if an input file is provided, " +
ls + "this is used. For CSV only.");
command.option("-b, --batch-size [num]", "Numbers of rows to import at once. Default is 1000");
command.option("-g, --guess-limit [num]", "Numbers of rows to analyze for data structure and types. Default is 1000");
command.option("-r, --restart", "WARNING Restarts any resuming, deletes existing data, and existing tables in DB.");
command.option("-s, --silent", "Suppress output besides errors.");
command.option("-u, --debug", "Output stack trace if available.");
Expand Down Expand Up @@ -82,6 +83,7 @@ var options = {
dateFormat: command.dateFormat ? command.dateFormat : undefined,
datetimeFormat: command.datetimeFormat ? command.datetimeFormat : undefined,
batchSize: command.batchSize ? command.batchSize : undefined,
guessLimit: command.guessLimit ? command.guessLimit : undefined,
outputConfig: !!command.outputConfig,
restart: !!command.restart,
output: !command.silent
Expand Down
6 changes: 3 additions & 3 deletions lib/tables.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ function Tables(options) {
}
},
batchSize: 1000,
guessLimit: 1000,
dataPath: this.defaultDataPath(),
dateFormat: "MM/DD/YYYY",
datetimeFormat: "MM/DD/YYYY HH:mm:ss a",
Expand Down Expand Up @@ -335,7 +336,6 @@ Tables.prototype.insertBatch = function(data, streamProcess, parseProcess, bytes
// Guess model
Tables.prototype.guessModel = function() {
var thisTables = this;
var guessLimit = 1000;
var guessCount = 0;
var columns = {};
var bar = this.bar();
Expand All @@ -346,7 +346,7 @@ Tables.prototype.guessModel = function() {

// Title
this.output.heading("Model guessing");
this.output.item("No models options provided, guessing data structure and types from first " + guessLimit + " rows");
this.output.item("No models options provided, guessing data structure and types from first " + this.options.guessLimit + " rows");

// Start stream
if (!this.options.stdin) {
Expand All @@ -373,7 +373,7 @@ Tables.prototype.guessModel = function() {
guess.pipe(this.inputParser())
.on("data", function(data) {
// Add to column list
if (guessCount < guessLimit) {
if (guessCount < thisTables.options.guessLimit) {
_.each(data, function(d, di) {
columns[di] = columns[di] || [];
columns[di].push(d);
Expand Down