Skip to content

Commit

Permalink
[AIRFLOW-378] Add string casting to params of spark-sql operator
Browse files Browse the repository at this point in the history
For parameters num_executors and executor_cores
add casts to strings to prevent issues when these
parameters are passed as integers (as comments specify).
Also fix minor typo that breaks the use of num-executors param.

Closes apache#1694 from
danielvdende/spark_sql_operator_bugfixes
  • Loading branch information
danielvdende authored and bolkedebruin committed Oct 5, 2016
1 parent 2aaa629 commit 573fb99
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions airflow/contrib/hooks/spark_sql_hook.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,13 +90,13 @@ def _prepare_command(self, cmd):
for conf_el in self._conf.split(","):
connection_cmd += ["--conf", conf_el]
if self._executor_cores:
connection_cmd += ["--executor-cores", self._executor_cores]
connection_cmd += ["--executor-cores", str(self._executor_cores)]
if self._executor_memory:
connection_cmd += ["--executor-memory", self._executor_memory]
if self._keytab:
connection_cmd += ["--keytab", self._keytab]
if self._num_executors:
connection_cmd += ["--num_executors", self._num_executors]
connection_cmd += ["--num-executors", str(self._num_executors)]
if self._sql:
if self._sql.endswith('.sql'):
connection_cmd += ["-f", self._sql]
Expand Down

0 comments on commit 573fb99

Please sign in to comment.