Skip to content

memory errors with SpectralCF #21

New issue
New issue
Open
Open
memory errors with SpectralCF#21
@duncanmcelfresh

Description

@duncanmcelfresh
duncanmcelfresh
opened on Jul 29, 2022

Below are the the traces. These occurred on a GCP n1-highmem-2 node with 1x tesla T4. Saw this when running experiments with the following datasets:

  • AmazonMoviesTV
  • Dating
  • GoogleLocalReviews
  • Gowalla
  • YahooMusic
SpectralCF_RecommenderWrapper: Instantiating model...
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_time = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 170, in fit
    self.data_generator = Data(self.URM_train, batch_size=self.batch_size)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 34, in __init__
    self.R = np.zeros((self.n_users, self.n_items), dtype=np.float32)
MemoryError

and the following memory error on

  • BookCrossing
  • Epinions
  • NetflixPrize
  • Recipes
SpectralCF_RecommenderWrapper: Instantiating model...
SpectralCF: Computing adjacient_matrix...
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_time = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 181, in fit
    self.model.compute_eigenvalues()
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF.py", line 25, in compute_eigenvalues
    self.A = self._adjacient_matrix(self_connection=True)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF.py", line 118, in _adjacient_matrix
    A = np.zeros([self.n_users+self.n_items, self.n_users+self.n_items], dtype=np.float32)
MemoryError

and the following on:

  • CiaoDVD
  • LastFM
  • YahooMovies
SpectralCF_RecommenderWrapper: Training SpectralCF...
SpectralCF_RecommenderWrapper: Saving model in file './result_experiments/__Temp
_SpectralCF_RecommenderWrapper_1185/_best_model'
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_tim
e = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 203, in fit
    self._update_best_model()
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 227, in _update_best_model
    self.save_model(self.temp_file_folder, file_name="_best_model")
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 287, in save_model
    saver.save(self.sess, folder_path + file_name + "_session")
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1196, in save
    meta_graph_filename, strip_default_attrs=strip_default_attrs)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1234, in export_meta_graph
    graph_def=ops.get_default_graph().as_graph_def(add_shapes=True),
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3150, in as_graph_def
    result, _ = self._as_graph_def(from_version, add_shapes)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3108, in _as_graph_def
    c_api.TF_GraphToGraphDef(self._c_graph, buf)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot serialize protocol buffer of type tensorflow.GraphDef as the serialized size (5223729696bytes) would be larger than the limit (2147483647 bytes)

and this one on MarketBiasAmazon

SpectralCF: Initialization complete in 1.49 hour
SpectralCF: Building Tensorflow graph...
/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/Spec
tralCF_our_interface/SpectralCF.py:74: ComplexWarning: Casting complex values to
 real discards the imaginary part
  A_hat = A_hat.astype(np.float32)
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_tim
e = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning
/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 187, in fit
    self.model.build_graph()
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF.py", line 80, in build_graph
    embeddings = tf.matmul(A_hat, embeddings)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 2399, in matmul
    a = ops.convert_to_tensor(a, name="a")
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1039, in convert_to_tensor
    return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1097, in convert_to_tensor_v2
    as_ref=False)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1175, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 304, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 245, in constant
    allow_broadcast=True)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl
    allow_broadcast=allow_broadcast))
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 537, in make_tensor_proto
    "Cannot create a tensor proto whose content is larger than 2GB.")
ValueError: Cannot create a tensor proto whose content is larger than 2GB.

and the following on MovieTweetings and Movielens10M

SpectralCF: Computing laplacian_matrix...
SpectralCF: Computing eigenvalues...
Traceback (most recent call last):
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 402, in _objective_function
    result_dict, result_string, recommender_instance, train_time, evaluation_time = self._evaluate_on_validation(current_fit_parameters_dict)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/RandomSearch.py", line 50, in _evaluate_on_validation
    current_fit_parameters
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 293, in _evaluate_on_validation
    recommender_instance, train_time = self._fit_model(current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/ParameterTuning/SearchAbstractClass.py", line 283, in _fit_model
    **current_fit_parameters)
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF_RecommenderWrapper.py", line 181, in fit
    self.model.compute_eigenvalues()
  File "/home/shared/reczilla/RecSys2019_DeepLearning_Evaluation/Conferences/RecSys/SpectralCF_our_interface/SpectralCF.py", line 34, in compute_eigenvalues
    self.lamda, self.U = np.linalg.eig(self.L)
  File "/home/shared/miniconda3/envs/reczilla/lib/python3.6/site-packages/numpy/linalg/linalg.py", line 1311, in eig
    w, vt = _umath_linalg.eig(a, signature=signature, extobj=extobj)
MemoryError

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions