Conversation
This data all fits into memory in one run and takes less than a minute to load into snowflake so it will completely reset nightly. Overview of (max) granularity of each table - One time: -- Databases, Tables - Daily: -- DatabaseStorageUsageHistory, StorageUsage - Hourly: -- LoadHistory, LoginHistory - Minute: -- QueryHistory
|
@brln-looker - ready for you to review. I didn't end up making any comments on this PR, hopefully it's all reasonably self explanatory |
| } | ||
| ]; | ||
| } | ||
| } |
| } | ||
| ]; | ||
| } | ||
| } |
| this.ERROR_COUNT = Math.random() < 0.1 ? Math.round(Math.random() * 10) : 0; | ||
| } | ||
|
|
||
| static oddsNew() { |
There was a problem hiding this comment.
personally I would define this as a const at the top of the module, const ODDS_NEW 0.2 but I don't really have any good reason to do it one way or the other.
There was a problem hiding this comment.
Early on I thought I'd have a lot more "dice rolling" to create an object than I ended up with. So I made this DataRow.rollDice() / SubClass.oddsNew() construct that ended up not being used very much (and actually DataRow.oddsNew() should be static). What I'd like is class level properties but doesn't seem like those exist in JS?
I could set const ODDS_NEW = 0.2 at the module level and then reference it in the static oddsNew() method in each class to help with the "magic number" issue, is that what you're referring to? or would you do the check differently?
| } | ||
| ]; | ||
| } | ||
| } |
data_rows/QueryHistory.js
Outdated
| constructor(databaseName, date) { | ||
| super(); | ||
| this.QUERY_ID = uuid4(); | ||
| this.setQueryText(); |
There was a problem hiding this comment.
in DatabaseStorageUsageHistory you use the _convention to annotate private methods. that's not super standard for node, but I like it.
There was a problem hiding this comment.
oops, sorry for lack of consistency. I'll switch everything applicable to "private"
data_rows/QueryHistory.js
Outdated
| import uuid4 from "uuid/v4"; | ||
|
|
||
| import DataRow from "./DataRow"; | ||
| import { LoginHistory } from "./LoginHistory"; |
There was a problem hiding this comment.
if I'm only exporting one thing from a module, I like to use export default, then you don't have to put braces around it like this.
There was a problem hiding this comment.
when I switched to using the "package" (./index.js) I got errors and thought I had to remove the default. But that's not the case, not sure what I was doing wrong but I'll go back to defaults
| [this.WAREHOUSE_NAME, this.WAREHOUSE_SIZE] = helpers.randomFromArray([ | ||
| ["BIG_WH", "X-Large"], | ||
| ["SMALL_WH", "Small"] | ||
| ]); |
There was a problem hiding this comment.
are you referring to my super original names/sizes or the "de-structuring" feature of JS (python's unpacking, php's list())? I'm happy to see JS has it too :-)
| } | ||
| ]; | ||
| } | ||
| } |
| } | ||
| ]; | ||
| } | ||
| } |
| if (this.WAREHOUSE_NAME == "SMALL_WH") { | ||
| this.CREDITS_USED *= 0.3; | ||
| } | ||
| this.CREDITS_USED.toFixed(2); |
There was a problem hiding this comment.
heads up that .toFixed outputs a string, but the column type here is marked as a float. it probably gets cast somewhere in the data upload, but just making sure you're aware.
There was a problem hiding this comment.
I didn't know that. yeah, it just "sort of works" and makes the data be 2 points of precision instead of a butt load when it finally gets into the db. I think native type gets kind of lost in writing the csv file and the Types stuff we have is just for creating the table columns in snowflake
There was a problem hiding this comment.
a/ there's a bug I fixed here (needed to assign the result of toFixed, it doesn't mutate in place)
b/ now I see the string problem:
> const n1 = 0.3252.toFixed(2);
undefined
> const n2 = 0.3652.toFixed(2);
undefined
> n1 + n2
'0.330.37'
oops...
main.js
Outdated
| } | ||
| } | ||
|
|
||
| backfill().catch(e => console.log(e)); |
|
lgtm. i should have marked this as a "review" but started doing individual comments and didn't want to switch halfway. oh well next time. |
|
I added a few feedback commits. I also still have a couple tweaks to make after showing the data to Jeffty but I'll put those up tomorrow. |
and fix toFixed bug
Also add weight to different queries
|
@brln-looker - data tweak fixes are up, I think this is ready to go |
2 participants
This data all fits into memory in one run and takes less than a minute
to load into snowflake so it will completely reset nightly.
Overview of (max) granularity of each table
-- Databases, Tables
-- DatabaseStorageUsageHistory, StorageUsage
-- LoadHistory, LoginHistory
-- QueryHistory