Skip to content

Last changes to repository #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
stonescenter wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Last changes to repository #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7676df4
Moved config files
stonescenter Sep 16, 2020
644473b
Last implementations for gaussian model
stonescenter Oct 4, 2020
ccc9b46
Purge some unuseful notebooks
stonescenter Oct 4, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 62 additions & 34 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,45 +38,60 @@ Considering the mentioned regions and the symmetry of the detector, each dataset
<img width="600" height="450" src="./imgs/polar_last.png"></img>
</p>

A short datasets are in `dataset` directory.
Short datasets are in `dataset` directory. We split a region in three parts (this is too big to be here).

# Running
## Training
There are some predefined config files to train diferents models (MLP, CNN, LSTM, CNN-parallel and others). If you need to change the parameters then change the config_*.json file. We used internal barrel as dataset, this dataset is previously transformed and linked in json file:
```sh
$ python main_train.py --config config_lstm_parallel_internal.json
$ python main_train.py --config configs/config_lstm_parallel_internal.json
```
There are other configurations for example a CNN model:

```sh
$ python main_train.py --config config_cnn_parallel_internal.json
$ python main_train.py --config configs/config_cnn_parallel_internal.json
```

If you want to see the training process when ajust any parameters of .json file. Run the notebook:
If you want to use a gaussian model

```sh
$ main_train.ipynb
$ python main_train.py --config configs/config_lstm_gaussian.json
```
For many trainings and testings with scripts. You can run it, with the default configuration:

If you want to see the training process when ajust any parameters of .json file. Run the notebook:

```sh
$ ./run_trains.sh
$ ./run_tests.show
$ main_train.ipynb
```

## Inference

You can inference data test:
You can inference the previous compiled model with the same config file:
```sh
$ python main_inference.py --config config_lstm_parallel_internal.json
$ python main_inference.py --config configs/config_lstm_parallel_internal.json --load True
```
This will produce a `results/encrypt_name/results-test.txt` file.
This will produce an report at `results/encrypt_name/results-test.txt` file. With the number of reconstructed tracks and quantity of hits correctly predicted by layer.

If you want to inference with a short dataset:
```sh
$ python main_inference.py --config configs/config_lstm_parallel_internal.json --load True --samples 500
```
or
```sh
$ python main_inference.py --config configs/config_lstm_gaussian.json --load True
```
## Auxiliary Scripts

You can automatizate trainings and testings with scripts. there are some default configurations:

```sh
$ ./run_trains.sh
$ ./run_tests.sh
```

# Performance
## Accuracy of Algorithm

We are using regressions metrics for accuracy of models. We show 2 groups of metrics.

- The principal metrics is a scoring. Scoring counts how many correct hits were found per layer and comparates with original truth hits. Finally we count the quantity of tracks reconstructed.
Expand All @@ -89,18 +104,24 @@ Output test file:
```
[Output] Results
---Parameters---
Model Name : lstm
Dataset : phi025-025_eta025-025_train1_lasthit_20200219.csv
Tracks : 528
Model saved : /compiled/model-lstm-DCtuvkiXn32hugVsTaokcp-coord-xyz-normalise-true-epochs-21-batch-6.h5
Test date : 10/06/2020 12:09:34
Coordenates : xyz
Model Name : gaussian-lstm
Dataset : eta_n0.5-0.5_phi_n0.5-0.5_20200518171238_tracks.csv
Tracks : 3000
Model saved : compiled/model-gaussian-lstm-MEnGHa5DCmJGE9C9BpAg4i-coord-cylin-normalise-true-epochs-30-batch-30.h5
Test date : 02/10/2020 20:23:55
Coordenates : cylin
Coordenate 3D : True
Model Scaled : True
Model Optimizer : adam
Prediction Opt : nearest
Total correct hits per layer [256. 251. 213. 194. 157. 126.] of 528 tracks tolerance=0.0:
Total porcentage correct hits : ['48.48%', '47.54%', '40.34%', '36.74%', '29.73%', '23.86%']
Reconstructed tracks: 74 of 528 tracks
Prediction Opt : gaussian
Distance metric : ['polar', 'euclidean', 'mahalanobis', 'cosine']
Remove hit : False
Correct hits per layer (polar) ['191.0(6.37%)', '135.0(4.5%)', '182.0(6.07%)', '167.0(5.57%)', '121.0(4.03%)', '127.0(4.23%)'] of 3000 tracks:
Correct hits per layer (euclidean) ['61.0(2.03%)', '54.0(1.8%)', '64.0(2.13%)', '62.0(2.07%)', '49.0(1.63%)', '36.0(1.2%)'] of 3000 tracks:
Correct hits per layer (mahalanobis) ['191.0(6.37%)', '193.0(6.43%)', '178.0(5.93%)', '193.0(6.43%)', '165.0(5.5%)', '133.0(4.43%)'] of 3000 tracks:
Correct hits per layer (cosine) ['2323.0(77.43%)', '1980.0(66.0%)', '1792.0(59.73%)', '1702.0(56.73%)', '1487.0(49.57%)', '1351.0(45.03%)'] of 3000 tracks:
Reconstructed tracks: 887 of 3000 tracks (29.566666666666666)


```
Above output shows scoring per layer for example 48% with 256 hits were matched at the first layer, results are 74 tracks reconstructed of 528 tracks(it is a short dataset just). We also write other info like what kind of coordinate, if we use the nearest optimization, epochs, batchs, optimazer used, model name etc.
Expand Down Expand Up @@ -138,23 +159,30 @@ layer 7
The last output shows one geral metric for all hits and four (R^2, MSE, RMSE, MAE) metrics per layer.


## Vizualization
If you want to see the results with plots, go to the plot_prediction.ipynb file at `notebooks` directory.
## Plots
If you want to see the results with plots, go to the plot_prediction.ipynb file at `notebooks` directory. This example plots cartesian coordinates.

This plot is 10 tracks reconstructed.
<p align="center">
<img width="650" height="500" src="./imgs/internal_reconstruction_cartesian_zy.png"></img>
</p>
<table><tr>
<td align="center"> <img src="./imgs/internal_reconstruction_cartesian_zy.png" alt="Drawing" style="width: 250px;"/> </td>
</tr></table>

The next plot shows all hits.
<p align="center">
<img width="650" height="500" src="./imgs/internal_reconstruction_3d.png"></img>
</p>
<table><tr>
<td align="center"> <img src="./imgs/internal_reconstruction_3d.png" alt="Drawing" style="width: 250px;"/> </td>
</tr></table>

The next plot is the prediction of all hits.
<p align="center">
<img width="650" height="500" src="./imgs/all_tracks_pred.png"></img>
</p>
### Plot Distributions
There is other notebook for plot the gaussian distributions plot_distributions.ipynb

<table><tr>
<td align="center"> <img src="./imgs/plot_polar.png" alt="Drawing" style="width: 250px;"/> </td>
</tr></table>

The pull of distributions for $etha$ and $phi$.

<table><tr>
<td> <img src="./imgs/plot_eta.png" alt="Drawing" style="width: 250px;"/> </td>
<td> <img src="./imgs/plot_phi.png" alt="Drawing" style="width: 250px;"/> </td>
</tr></table>


10 changes: 5 additions & 5 deletions config_cnn.json → configs/config_cnn.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"log_dir": "logs"
},
"data": {
"filename": "/data/track-ml/eramia/dataset/eta-05_05.csv",
"filename": "dataset/eta_n0.5-0.5_phi_n0.5-0.5_short.csv",
"train_split": 0.80,
"normalise": true,
"cylindrical": false,
Expand All @@ -21,12 +21,12 @@
"use_gpu": true,
"show_metrics": false,
"shuffle": true,
"earlystopping": true
"earlystopping": true
},
"testing": {
"type_optimization" : "normal",
"type_optimization" : "nearest",
"normalise": false,
"tolerance": 0.02
"tolerance": 0.01
},
"model": {
"name": "cnn",
Expand All @@ -53,7 +53,7 @@
},
{
"type": "dense",
"neurons": 18,
"neurons": 12,
"activation": "relu"
},
{
Expand Down
15 changes: 9 additions & 6 deletions config_cnn_parallel.json → configs/config_cnn_parallel.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,19 @@
"log_dir": "logs"
},
"data": {
"filename": "./dataset/phi025-025_eta025-025_train1_lasthit_20200219.csv",
"filename": "dataset/eta_n0.5-0.5_phi_n0.5-0.5_short.csv",
"train_split": 0.80,
"normalise": true,
"cylindrical": false,
"type_norm": "maxmin",
"cylindrical": true,
"points_3d": true,
"num_hits": 10,
"features": 3
},
"training": {
"epochs": 15,
"epochs": 18,
"batch_size": 6,
"validation": 0.2,
"validation": 0.3,
"save_model": true,
"load_model": false,
"use_gpu": true,
Expand All @@ -25,8 +27,9 @@
},
"testing": {
"type_optimization" : "nearest",
"metric": "cosine",
"normalise": false,
"tolerance": 0.02
"tolerance": 0.01
},
"model": {
"name": "cnn-parallel",
Expand All @@ -53,7 +56,7 @@
},
{
"type": "dense",
"neurons": 64,
"neurons": 32,
"activation": "relu"
},
{
Expand Down
0 config_cnn_parallel_internal.json → configs/config_cnn_parallel_internal.json
View file
Open in desktop
File renamed without changes.
0 config_default.json → configs/config_default.json
View file
Open in desktop
File renamed without changes.
8 changes: 4 additions & 4 deletions config_lstm.json → configs/config_lstm.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,17 @@
"log_dir": "logs"
},
"data": {
"filename": "/data/track-ml/eramia/dataset/phi025-025_eta025-025_train1_lasthit_20200219.csv",
"filename": "dataset/eta_n0.5-0.5_phi_n0.5-0.5_short.csv",
"train_split": 0.80,
"normalise": true,
"cylindrical": false,
"num_hits": 10,
"features": 3
},
"training": {
"epochs": 21,
"epochs": 20,
"batch_size": 6,
"validation": 0.2,
"validation": 0.33,
"save_model": true,
"load_model": false,
"use_gpu": true,
Expand Down Expand Up @@ -46,7 +46,7 @@
},
{
"type": "dropout",
"rate": 0.25
"rate": 0.2
},
{
"type": "lstm",
Expand Down
34 changes: 18 additions & 16 deletions config_lstm_gaussian.json → configs/config_lstm_gaussian.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,19 @@
"log_dir": "logs"
},
"data": {
"filename": "./dataset/eta_n0.5-0.5_phi_n0.5-0.5_internal_short1.csv",
"filename": "/data/track-ml/bracis/datasets/eta_n0.5-0.5_phi_n0.5-0.5/eta_n0.5-0.5_phi_n0.5-0.5_20200518171238_tracks.csv",
"train_split": 0.80,
"normalise": true,
"type_norm": "zscore",
"type_norm": "maxmin",
"cylindrical": true,
"points_3d": false,
"points_3d": true,
"num_hits": 10,
"features": 2
"features": 3
},
"training": {
"epochs": 30,
"batch_size": 18,
"validation": 0.3,
"batch_size": 30,
"validation": 0.35,
"save_model": true,
"load_model": false,
"use_gpu": true,
Expand All @@ -26,27 +26,29 @@
"earlystopping": true
},
"testing": {
"type_optimization" : "nearest",
"metric": "cosine",
"type_optimization" : "gaussian",
"metric": ["polar", "euclidean", "mahalanobis", "cosine"],
"normalise": false,
"tolerance": 0.02
},
"tolerance": 0.02,
"remove_hit": false,
"show_metrics": false
},
"model": {
"name": "gaussian-lstm",
"overwrite": true,
"isparallel": false,
"isparallel": false,
"loss": "mse",
"metrics": ["acc"],
"optimizer": "adam",
"learningrate": 0.01,
"learningrate": 0.0005,
"layers": [
{
"type": "input",
"neurons": 500,
"input_features": 2,
"neurons": 700,
"input_features": 3,
"input_timesteps": 4,
"activation": "tanh"
}
"activation": "tanh"
}
]
}
}
29 changes: 19 additions & 10 deletions config_lstm_parallel.json → configs/config_lstm_parallel.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,27 @@
"log_dir": "logs"
},
"data": {
"filename": "./dataset/phi025-025_eta025-025_train1_lasthit_20200219.csv",
"filename": "./dataset/eta_2.0-3.0_phi_n0.5-0.5_external_short.csv",
"train_split": 0.80,
"normalise": true,
"cylindrical": false,
"num_hits": 10,
"features": 3

},
"training": {
"epochs": 18,
"batch_size": 24,
"epochs": 10,
"batch_size": 30,
"validation": 0.2,
"save_model": true,
"load_model": false,
"use_gpu": true,
"show_metrics": false,
"shuffle": true,
"shuffle": false,
"earlystopping": true
},
"testing": {
"type_optimization" : "nearest",
"metric": "euclidean",
"normalise": false,
"tolerance": 0.01
},
Expand All @@ -39,21 +39,30 @@
"layers": [
{
"type": "lstm",
"neurons": 500,
"neurons": 800,
"input_timesteps": 4,
"input_features": 1,
"input_features": 1,
"return_seq": false,
"stateful": false
},
{
"type": "dropout",
"rate": 0.25
"rate": 0.5
},
{
"type": "dense",
"neurons": 64,
"neurons": 100,
"activation": "relu"
},
},
{
"type": "dropout",
"rate": 0.2
},
{
"type": "dense",
"neurons": 18,
"activation": "relu"
},
{
"type": "dense",
"neurons": 3,
Expand Down
0 config_lstm_parallel_internal.json → configs/config_lstm_parallel_internal.json
View file
Open in desktop
File renamed without changes.
Loading