Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SBT Support #21

Merged
merged 41 commits into from
Feb 16, 2017
Merged
Changes from 2 commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
8d142ec
clean up and formatting
karthikvadla Feb 8, 2017
b79ccc9
formatting testcases
karthikvadla Feb 8, 2017
cb4656e
upgraded to scala 2.11.8
karthikvadla Feb 9, 2017
651ae54
Merge branch 'master' into tensorflow
karthikvadla Feb 9, 2017
b57acb9
merge
karthikvadla Feb 9, 2017
6919a17
formatting merge
karthikvadla Feb 9, 2017
9591da4
update readme
karthikvadla Feb 9, 2017
7c6a0db
Merge branch 'master' of github.com:karthikvadla16/spark-tensorflow-c…
karthikvadla Feb 9, 2017
c30085b
Clean up .gitignore file, remove tf folder inside test folder
karthikvadla Feb 10, 2017
4128deb
Merge remote-tracking branch 'origin/master' into tensorflow
karthikvadla Feb 10, 2017
09508d3
Merge remote-tracking branch 'origin/tensorflow' into newtens
karthikvadla Feb 10, 2017
3a3fbef
Rename spark-tf-core to core, and update all references
karthikvadla Feb 10, 2017
9dcb951
Remove core module, add License file and make pom changes
karthikvadla Feb 10, 2017
37b31c2
Renaming namespace, update all files with new namespace
karthikvadla Feb 10, 2017
71b427f
Fix custom schema, correct pom
karthikvadla Feb 10, 2017
b3691b3
update readme
karthikvadla Feb 10, 2017
ebbd0d9
Merge branch 'master' into tensorflow
karthikvadla Feb 10, 2017
6b5ffda
update readme
karthikvadla Feb 10, 2017
bab9e3a
merge from master
karthikvadla Feb 14, 2017
1822369
Merge branch 'tensorflow' of github.com:karthikvadla16/spark-tensorfl…
karthikvadla Feb 14, 2017
f6fee5f
add sbt build files
karthikvadla Feb 14, 2017
9bf2e70
Add conversion from mvn to sbt (#15)
karthikvadla Feb 15, 2017
ee16f57
Add classifier to bring in correct shaded jar and class
joyeshmishra Feb 15, 2017
efaf1ed
Add classifier to bring in correct shaded jar and class (#16)
joyeshmishra Feb 15, 2017
c2b23f1
Merge branch 'sbt' of github.com:tapanalyticstoolkit/spark-tensorflow…
joyeshmishra Feb 15, 2017
2cc06c3
Add travis.yml file
joyeshmishra Feb 15, 2017
e2f5add
Refactor travis file
joyeshmishra Feb 15, 2017
5645965
Refactor travis file
joyeshmishra Feb 15, 2017
d7da13c
Update README.md
joyeshmishra Feb 15, 2017
5a9f563
Add Travis support to sbt branch (#17)
joyeshmishra Feb 15, 2017
b349f6a
Cleanup
joyeshmishra Feb 15, 2017
4b3c8c4
Merge branch 'sbt' of github.com:joyeshmishra/spark-tensorflow-connec…
joyeshmishra Feb 15, 2017
c479efd
merge
joyeshmishra Feb 15, 2017
b8503a1
Remove central1 dependency in sbt and sudo requirement from travis.ym…
joyeshmishra Feb 15, 2017
b369d84
Merge branch 'sbt' of github.com:tapanalyticstoolkit/spark-tensorflow…
joyeshmishra Feb 15, 2017
562beab
SBT working, Cleaned up (#19)
karthikvadla Feb 15, 2017
f710331
Merge branch 'sbt' of github.com:tapanalyticstoolkit/spark-tensorflow…
joyeshmishra Feb 15, 2017
2df8580
use filterNot
joyeshmishra Feb 15, 2017
ac9afcc
Refactor to use filterNot (#20)
joyeshmishra Feb 15, 2017
9fb9fff
Merge branch 'sbt' of github.com:tapanalyticstoolkit/spark-tensorflow…
joyeshmishra Feb 16, 2017
6171354
Add sbt-spark-package plugin support
joyeshmishra Feb 16, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ class TensorflowSuite extends TestingSparkSessionWordSpec with Matchers with Bef
val testRows: Array[Row] = Array(
new GenericRow(Array[Any](11, 1, 23L, 10.0F, 14.0, List(1.0, 2.0), "r1")),
new GenericRow(Array[Any](21, 2, 24L, 12.0F, 15.0, List(2.0, 2.0), "r2")))

val schema = StructType(List(
StructField("id", IntegerType),
StructField("IntegerTypelabel", IntegerType),
Expand All @@ -63,9 +64,7 @@ class TensorflowSuite extends TestingSparkSessionWordSpec with Matchers with Bef
StructField("DoubleTypelabel", DoubleType),
StructField("vectorlabel", ArrayType(DoubleType, true)),
StructField("name", StringType)))

val rdd = sparkSession.sparkContext.parallelize(testRows)


val df: DataFrame = sparkSession.createDataFrame(rdd, schema)
df.write.format("tensorflow").save(path)

Expand Down Expand Up @@ -148,7 +147,7 @@ class TensorflowSuite extends TestingSparkSessionWordSpec with Matchers with Bef

//Here Vector with null's are not supported
val expectedRow = new GenericRow(Array[Any](1, 23L, 10.0F, 14.0, List(1.0, 2.0), "r1"))

val schema = StructType(List(
StructField("IntegerTypelabel", IntegerType),
StructField("LongTypelabel", LongType),
Expand Down