Leaky ReLUs by echoline · Pull Request #105 · libfann/fann

echoline · 2018-06-29T21:43:10Z

leaky rectifying units are a new type of activation function

broken-pen · 2018-07-03T21:17:18Z

This implements a subset of the changes introduced in #80. It has the advantage that it looks ready to be merged to me, whereas #80 still requires some changes and the author has been silent for a while. This PR has the (small) disadvantage that it only includes the Leaky ReLU, not the regular one.

Should we merge this in favor of #80 or should we wait for action on #80 instead?

mrrstrat · 2020-02-03T14:25:23Z

So what is going on with ReLU? This seems to be a major feature FANN is lacking. I see Merge is blocked - does anyone know why the pending merge is still pending?
What does regression testing show the new ReLU does compared to other transfer functions?

bukka · 2021-01-23T18:08:17Z

CMakeLists.txt


 configure_file (cmake/config.h.in ${CMAKE_CURRENT_BINARY_DIR}/src/include/config.h)
-include_directories (${CMAKE_CURRENT_BINARY_DIR}/src/include/)
+include_directories (${CMAKE_CURRENT_BINARY_DIR}/src/include/ ${CMAKE_CURRENT_BINARY_DIR}/src/include/optimized/opencl/)


OpenCL is not used so this shouldn't be here.

bukka · 2021-01-23T18:09:14Z

examples/mkfile

@@ -0,0 +1,11 @@
+</$objtype/mkfile


What's the reason of adding this file?

Portability to Plan 9 I guess

bukka · 2021-01-23T18:09:36Z

src/CMakeLists.txt

+FIND_PACKAGE(OpenCL)
+IF(OPENCL_FOUND)
+  SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -I${OpenCL_INCLUDE_DIR}")
+  SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -I${OpenCL_INCLUDE_DIR}")
+ENDIF(OPENCL_FOUND)


Can this be deleted as it's not used anywhere?

bukka · 2021-01-23T18:09:46Z

src/CMakeLists.txt

+#	optimized/opencl/fann.c
+#	optimized/opencl/fann_cl.cpp
+#	optimized/opencl/fann_cl_kernel.c
+#	optimized/opencl/fann_cl_train.c
+#	optimized/opencl/fann_cl_ann.c
+#	optimized/opencl/fann_cl_run.c	


please delete it.

bukka · 2021-01-23T18:09:55Z

src/CMakeLists.txt

+
 ADD_SUBDIRECTORY( include )

+#INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CURRENT_SOURCE_DIR}/include/optimized/opencl)


please delete it

bukka · 2021-01-23T18:10:15Z

src/fann.c

+#ifdef PLAN9
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/time.h>
+#include <time.h>
+#include <math.h>
+#else


what's the purpose of this?

Portability to Plan 9 I guess

bukka · 2021-01-23T18:11:00Z

src/fann.c

 				}

-				for(; i != num_connections; i += 4)
+				#pragma omp parallel for reduction(+:neuron_sum)


omp shouldn't be added as part of this PR as it's not used in fann.c

bukka · 2021-01-23T18:11:39Z

src/include/fann_data_cpp.h

+	LINEAR_PIECE_LEAKY,
+	LINEAR_PIECE_RECT


indent slightly wrong

bukka · 2021-01-23T18:12:03Z

src/mkfile

@@ -0,0 +1,31 @@
+</$objtype/mkfile


again not sure what's the reason for this to be added...?

bukka · 2021-01-23T18:15:00Z

@echoline I have added few comments as I saw some recent commits. Is this PR still active? If so it would be also good to add a test for this.

echoline · 2021-01-23T23:40:33Z

I have continued making unrelated changes since the initial pull request. I am attaching a patch for leaky and normal ReLUs

…

On Sat, Jan 23, 2021 at 10:15 AM Jakub Zelenka ***@***.***> wrote: @echoline <https://github.com/echoline> I have added few comments as I saw some recent commits. Is this PR still active? If so it would be also good to add a test for this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#105 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADCAWP73MKG2Q7WURJ7RFDS3MG3DANCNFSM4FHWU4BA> .

echoline · 2021-01-23T23:58:55Z

@bukka hmmm github seems to have filtered out my email attachment...

relus.patch.txt
.

…on another system

mrrstrat · 2023-02-05T23:50:21Z

Hey gang - long time (since early 2000s) FANN user and developer. I have been watching the ReLU effort for a while now - is there a batter Github thread for asking some questions on the ReLU changes?

I can help with regression testing on my DL380 SAN cluster. I have a 20 core server I run things like this on.

@echoline

Original patch by @echoline (libfann#105 (comment)) plus a test case.

bukka · 2026-01-27T09:32:08Z

I'm going to close this - leaky relu has been added in different PR and the rest is not in the state that it could be merged. Feel free to create a new PR if you would like some parts of this get integrated.

Leaky ReLUs

ca8938b

eli and others added 5 commits July 7, 2018 08:58

Merge https://github.com/libfann/fann

eb7915e

Merge https://github.com/libfann/fann

204006f

Rectifying Linear Units

1fb4697

openmp - opencl

7c70a7e

Merge branch 'master' of https://github.com/echoline/fann

6a7c361

glenda added 2 commits January 15, 2021 19:28

Plan 9 port

3c7dbba

examples mkfile

8cec194

bukka requested changes Jan 23, 2021

View reviewed changes

Eli added 6 commits June 10, 2022 12:28

start on opencl. intel hardware sucks for this so committing to work …

a30ca99

…on another system

blah

9e87141

Merge branch 'master' of github.com:echoline/fann

a0c90c4

;)

00bd0dc

opengl compute shader for run

b918024

train shader (?)

5d09aa0

Eli added 9 commits February 5, 2023 18:23

gl train (leaky relus only)

6c2215a

PLAN9 ifndefs, fann_from_gpu for weights

0bdc64e

PLAN9 ifndef

0294836

PLAN9 ifndef

6c7ed9e

PLAN9 ifndef

d707c8d

PLAN9 ifndef

54eb717

PLAN9 ifdef

0a3a84e

PLAN9 ifdef

7162449

fixes

48ee736

Eli and others added 7 commits February 10, 2023 12:21

glMapBufferRange(... GL_MAP_PERSISTENT_BIT)

94c860f

glMemoryBarrier

012c672

NVIDIA RTX

058bfc8

gpu test

1c41188

is this a problem with GL work group sizes?

95d000d

blah

411c9cd

xor test

9483fba

DrDub added a commit to DrDub/fann that referenced this pull request Dec 6, 2023

Leaky ReLUs with test case

44bfcee

Original patch by @echoline (libfann#105 (comment)) plus a test case.

DrDub mentioned this pull request Dec 6, 2023

Leaky ReLUs with test case #147

Merged

bukka closed this Jan 27, 2026


		ADD_SUBDIRECTORY( include )

		#INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}/include ${CMAKE_CURRENT_SOURCE_DIR}/include/optimized/opencl)

Conversation

echoline commented Jun 29, 2018

Uh oh!

broken-pen commented Jul 3, 2018

Uh oh!

mrrstrat commented Feb 3, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bukka commented Jan 23, 2021

Uh oh!

echoline commented Jan 23, 2021 via email

Uh oh!

echoline commented Jan 23, 2021

Uh oh!

mrrstrat commented Feb 5, 2023

Uh oh!

bukka commented Jan 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants