root@f60b2ab2d796:/mnt# t2t-decoder --data_dir=t2t_data --problem=translate_envi_iwslt32k --model=transformer --decode_hparams="beam_size=4,alpha=0.6" --decode_from_file=tst2013.en --decode_to_file=system.output --hparams_set=transformer_base --checkpoint_path t2t_avg/model.ckpt-50000 WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons If you depend on functionality not listed there, please file an issue. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/trainer_lib.py:278: __init__ (from tensorflow.contrib.learn.python.learn.estimators.run_config) is deprecated and will be removed in a future version. Instructions for updating: When switching to tf.estimator.Estimator, use tf.estimator.RunConfig instead. INFO:tensorflow:Configuring DataParallelism to replicate the model. INFO:tensorflow:schedule=continuous_train_and_eval INFO:tensorflow:worker_gpu=1 INFO:tensorflow:sync=False WARNING:tensorflow:Schedule=continuous_train_and_eval. Assuming that training is running on a single machine. INFO:tensorflow:datashard_devices: ['gpu:0'] INFO:tensorflow:caching_devices: None INFO:tensorflow:ps_devices: ['gpu:0'] WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpRMR8O1 INFO:tensorflow:Using config: {'_save_checkpoints_secs': None, '_num_ps_replicas': 0, '_keep_checkpoint_max': 20, '_task_type': None, '_train_distribute': None, '_is_chief': True, '_cluster_spec': , '_tf_config': gpu_options { per_process_gpu_memory_fraction: 1.0 } , '_protocol': None, '_save_checkpoints_steps': 1000, '_keep_checkpoint_every_n_hours': 10000, '_session_config': gpu_options { per_process_gpu_memory_fraction: 0.95 } allow_soft_placement: true graph_options { optimizer_options { global_jit_level: OFF } } , '_model_dir': '/tmp/tmpRMR8O1', 'use_tpu': False, '_tf_random_seed': None, '_master': '', '_device_fn': None, '_num_worker_replicas': 0, '_task_id': 0, '_log_step_count_steps': 100, '_evaluation_master': '', '_eval_distribute': None, 'data_parallelism': , '_environment': 'local', '_save_summary_steps': 100, 't2t_device_info': {'num_async_replicas': 1}} WARNING:tensorflow:Estimator's model_fn () includes params argument, but params are not passed to Estimator. INFO:tensorflow:decode_hp.batch_size not specified; default=32 INFO:tensorflow:Performing decoding from file (tst2013.en). INFO:tensorflow:Getting sorted inputs WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. INFO:tensorflow: batch 40 INFO:tensorflow:Decoding batch 0 WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/decoding.py:567: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, use tf.py_function, which takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means `tf.py_function`s can use accelerators such as GPUs as well as being differentiable using a gradient tape. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/decoding.py:900: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:Input graph does not use tf.data.Dataset or contain a QueueRunner. That means predict yields forever. This is probably a mistake. INFO:tensorflow:Calling model_fn. INFO:tensorflow:Setting T2TModel mode to 'infer' INFO:tensorflow:Setting hparams.layer_prepostprocess_dropout to 0.0 INFO:tensorflow:Setting hparams.symbol_dropout to 0.0 INFO:tensorflow:Setting hparams.label_smoothing to 0.0 INFO:tensorflow:Setting hparams.attention_dropout to 0.0 INFO:tensorflow:Setting hparams.dropout to 0.0 INFO:tensorflow:Setting hparams.relu_dropout to 0.0 INFO:tensorflow:Beam Decoding with beam size 4 WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/function.py:1007: calling create_op (from tensorflow.python.framework.ops) with compute_shapes is deprecated and will be removed in a future version. Instructions for updating: Shapes are always computed; don't use the compute_shapes as it has no effect. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/modalities.py:114: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.cast instead. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/models/transformer.py:93: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version. Instructions for updating: Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`. WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensor2tensor/layers/common_layers.py:2935: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version. Instructions for updating: Use keras.layers.dense instead. INFO:tensorflow:Done calling model_fn. INFO:tensorflow:Graph was finalized. 2019-03-02 09:46:46.611689: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2019-03-02 09:46:46.617695: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2992965000 Hz 2019-03-02 09:46:46.619629: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x9c8b100 executing computations on platform Host. Devices: 2019-03-02 09:46:46.619653: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): , WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. INFO:tensorflow:Restoring parameters from t2t_avg/model.ckpt-50000 2019-03-02 09:46:46.997473: W tensorflow/core/framework/op_kernel.cc:1401] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key transformer/symbol_modality_20428_512/shared/weights_0 not found in checkpoint Traceback (most recent call last): File "/usr/local/bin/t2t-decoder", line 17, in tf.app.run() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "/usr/local/bin/t2t-decoder", line 12, in main t2t_decoder.main(argv) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_decoder.py", line 192, in main decode(estimator, hp, decode_hp) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_decoder.py", line 92, in decode checkpoint_path=FLAGS.checkpoint_path) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/decoding.py", line 435, in decode_from_file for elapsed_time, result in timer(result_iter): File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/decoding.py", line 429, in timer item = next(gen) File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 627, in predict hooks=all_hooks) as mon_sess: File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 934, in __init__ stop_grace_period_secs=stop_grace_period_secs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 648, in __init__ self._sess = _RecoverableSession(self._coordinated_creator) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1122, in __init__ _WrappedSession.__init__(self, self._create_session()) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1127, in _create_session return self._sess_creator.create_session() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 805, in create_session self.tf_sess = self._session_creator.create_session() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 571, in create_session init_fn=self._scaffold.init_fn) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 281, in prepare_session config=config) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 195, in _restore_checkpoint saver.restore(sess, checkpoint_filename_with_path) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1292, in restore err, "a Variable name or other graph key that is missing") tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: Key transformer/symbol_modality_20428_512/shared/weights_0 not found in checkpoint [[node save/RestoreV2_1 (defined at /usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py:627) ]] Caused by op u'save/RestoreV2_1', defined at: File "/usr/local/bin/t2t-decoder", line 17, in tf.app.run() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 125, in run _sys.exit(main(argv)) File "/usr/local/bin/t2t-decoder", line 12, in main t2t_decoder.main(argv) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_decoder.py", line 192, in main decode(estimator, hp, decode_hp) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/bin/t2t_decoder.py", line 92, in decode checkpoint_path=FLAGS.checkpoint_path) File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/decoding.py", line 435, in decode_from_file for elapsed_time, result in timer(result_iter): File "/usr/local/lib/python2.7/dist-packages/tensor2tensor/utils/decoding.py", line 429, in timer item = next(gen) File "/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 627, in predict hooks=all_hooks) as mon_sess: File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 934, in __init__ stop_grace_period_secs=stop_grace_period_secs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 648, in __init__ self._sess = _RecoverableSession(self._coordinated_creator) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1122, in __init__ _WrappedSession.__init__(self, self._create_session()) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 1127, in _create_session return self._sess_creator.create_session() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 805, in create_session self.tf_sess = self._session_creator.create_session() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 562, in create_session self._scaffold.finalize() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/monitored_session.py", line 217, in finalize self._saver = training_saver._get_saver_or_default() # pylint: disable=protected-access File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 604, in _get_saver_or_default saver = Saver(sharded=True, allow_empty=True) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 832, in __init__ self.build() File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 844, in build self._build(self._filename, build_save=True, build_restore=True) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 881, in _build build_save=build_save, build_restore=build_restore) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 507, in _build_internal restore_sequentially, reshape) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 385, in _AddShardedRestoreOps name="restore_shard")) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 332, in _AddRestoreOps restore_sequentially) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 580, in bulk_restore return io_ops.restore_v2(filename_tensor, names, slices, dtypes) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_io_ops.py", line 1572, in restore_v2 name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ self._traceback = tf_stack.extract_stack() NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: Key transformer/symbol_modality_20428_512/shared/weights_0 not found in checkpoint [[node save/RestoreV2_1 (defined at /usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/estimator.py:627) ]] root@f60b2ab2d796:/mnt#