Move parquet cascading schemes to subprojects#1514
Conversation
| @@ -394,17 +395,25 @@ lazy val scaldingParquet = module("parquet").settings( | |||
| exclude("com.twitter.elephantbird", "elephant-bird-pig") | |||
| exclude("com.twitter.elephantbird", "elephant-bird-core"), | |||
| "org.apache.thrift" % "libthrift" % "0.7.0", | |||
There was a problem hiding this comment.
can we move this raw version string with the rest at the top of this file?
There was a problem hiding this comment.
Updated. The global thriftVersion is already present and set to 0.5.0, which I think should be okay.
|
👍 |
|
This seems useful for the cascading 3 upgrade, and makes sense there. Do we want it in the 0.16 release cycle ? Or keep it closer to the big painful cascading one? @rubanm would this be hard to take as an update at twitter? (just as an exemplar of issues elsewhere possibly). |
|
With the aliases and deps in place it should be handled mostly by ivy resolves I think? Can we have the new types/classes be package private to scalding? They seem like an internal implementation detail as far as scalding is concerned, would be nice to not have them in our ABI if we could? thoughts @johnynek ? |
|
Re cascading3, I could cherry pick the sbt update commit (moves from Build.scala to build.sbt) to cascading3 branch, and apply this on top of it. If we don't merge this to develop, it will be one more thing to look at in the diff in the eventual cascading3 -> develop merge. Yes given the pre-existing sub-projects will still pull in classes in the corresponding new *-cascading ones, the upgrade should be fine I think. I'm in favor of merging to develop (and cherry picking to cascading3 branch) if this seems reasonable as a general thing to do. |
|
Some of Schemes are consumed at Twitter in multiformat sources, for example. The rest could be made package private I think. |
|
@rubanm we will need help from you and @isnotinvain to make sure we keep twitter up to date and not let any forks happen! So, if you think you can get the merge in at Twitter, I'm fine with a merge for 0.16. |
|
Yep, good way to put it. Merge when green from me if the hassle level is ok On Friday, February 12, 2016, P. Oscar Boykin notifications@github.com
Sent from Gmail Mobile |
|
Sorry, closed by mistake. |
|
should we merge this now? I kind of lost track of it. |
|
|
4 participants
This moves all the parquet schemes to separate sub-projects than the parquet sources. This is mainly for easier upgrade to cascading3 and future versions perhaps. (As things stand now, we need to upgrade scalding-core in our cascading3 branch, before scalding-parquet for example.)