SkeletonRNN

SkeletonRNN is a variable length custom Recurrent Deep Neural Network to estimate miss or wrongly detected 3D skeleton joints.
Explore the docs »

Report Bug · Request Feature

Table of Contents

1. About The Project
   • Citation
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
   • Dataset
   • Training
   • Testing
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About The Project

The extraction of 2D or 3D skeletons from images is a well-known procedure to perform data anonymization. The skeleton estimation has been shown to be sometimes disordered or not available and can provide wrong human identification. Indeed, many points are likely to miss causing issues in related tasks. We present here the Deep Learning architecture proposed in "Continuous Person Identification and Tracking in Healthcare by Integrating Accelerometer Data and Deep Learning Filled 3D Skeletons" to estimate missing 3D skeleton joints. In SkeletonRNN, the Long Short-Term Memory (LSTM) cells are modified such that the RNN hidden states are N x h matrices where N is the number of joints per skeleton and h is the hidden state size. In order to decode such hidden states into the prediction of the 3D skeleton joints for the next instance, a convolutinal decoder is introduced. To overcome the issue of missing points in the input data, starting from the second RNN iteration, we introduce an input refinement module.

Citation

Our paper is available in IEEE Sensors or

  @ARTICLE{9813452,
  author={Bastico, Matteo and Belmonte-Hernández, Alberto and García, Federico Álvarez},
  journal={IEEE Sensors Journal}, 
  title={Continuous Person Identification and Tracking in Healthcare by Integrating Accelerometer Data and Deep Learning Filled 3D Skeletons}, 
  year={2022},
  volume={22},
  number={15},
  pages={15402-15409},
  doi={10.1109/JSEN.2022.3186499}}

(back to top)

Built With

Our released implementation is tested on:

• Ubuntu 20.04 / macOS 11.5
• Python 3.9.7
• Pytorch 1.10 / torchvison 0.11.1
• NVIDIA CUDA 11.3
• TensorboardX 2.5
• Scikit-learn 1.0.1
• SciPy 1.7.2
• tqdm 4.64
• 1x NVIDIA GeForce GTX 1080 Ti

(back to top)

Getting Started

Prerequisites

• Create and lunch conda environment

  conda create -n SkeletonRNN python=3.9
  conda activate SkeletonRNN

Installation

• Clone project

  git clone https://github.com/matteo-bastico/SkeletonRNN.git
  cd SkeletonRNN

• Install dependencies

  pip install -r requirements.txt

  Note: for Pytorch CUDA installation follow https://pytorch.org/get-started/locally/. Example with CUDA 11.3:

  conda install pytorch==1.10 torchvision=0.11 torchaudio cudatoolkit=11.3 -c pytorch

(back to top)

Usage

Dataset

The train set contains 1035 sequences of complete skeletons which are augmented during the training to simulate loss of joints. The test set contains 259 sequences of skeletons with missing points and the ground-truth is provided. Both dataset are in the Data folder.

Data are stored in .npy files. Each of them contains a list of skeleton sequences saved as Numpy array with shape (L, N, D) where L is the sequence length, N is the number of skeleton points (18 for Intel RealSense) and D is the dimensionality (3). In the testing dataset missing points are represented with [-1, -1, -1]. Training skeletons are all complete, i.e. without missing points. The original dataset also for Person Identification and Tracking (PIT) can be downloaded from https://drive.upm.es/s/3zgeHKhlbWYcow1.

Training

To replicate best results of the paper

  python train.py -d "Data/train/examples.npy" -e 1000 -i 18 3 -hs 1024 -lr 1e-6 1e-5 -tb 8 -eb 8 -s 100

Parameters:

• -d : Training data path
• -e : Number of training epochs
• -i : Input format (N:Number of joints, D:Dimensions)
• -hs : LSTM hidden layer size. Options: 256, 512, 1024
• -lr : One-Cycle policy learning rates (max_lr, min_lr)
• -tb : Test batch size
• -eb : Evaluation batch size
• -s : Every how many epochs save checkpoint of the model

The results of the training (args, tensorboard summary and checkpoints) are saved in the runs folder. Note: The GPU is automatically detected for training.

Testing

  python test.py -d "Data/test/examples.npy" -l "Data/test/labels.npy" -i 18 3 -hs 1024 -tb 8 -chk path_to_last_chk

Parameters:

• -d : Testing data path
• -l : Ground-truth data path
• -i : Input format (N:Number of joints, D:Dimensions)
• -hs : LSTM hidden layer size. Options: 256, 512, 1024
• -tb : Test batch size
• -chk : Path to checkpoint to test

Note: The GPU is automatically detected for testing.

(back to top)

Roadmap

• CUDA distributed implementation
• Skeletons graphical visualization

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/my_feature)
3. Commit your Changes (git commit -m 'Add my_feature')
4. Push to the Branch (git push origin feature/my_feature)
5. Open a Pull Request

(back to top)

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

Matteo Bastico - @matteobastico - matteo.bastico@gmail.com

Project Link: https://github.com/matteo-bastico/SkeletonRNN

(back to top)

Acknowledgments

This work was supported by the H2020 European Project: Procare4Life https://procare4life.eu/ web Grant no. 875221. The authors are with the Escuela Técnica Superior de Ingenieros de Telecomunicación, Universidad Politécnica de Madrid, 28040 Madrid, Spain (e-mail: mab@gatv.ssr.upm.es, abh@gatv.ssr.upm.es, fag@gatv.ssr.upm.es).

(back to top)