-
Notifications
You must be signed in to change notification settings - Fork 2
Expand file tree
/
Copy pathguidelines.html
More file actions
executable file
·129 lines (99 loc) · 223 KB
/
guidelines.html
File metadata and controls
executable file
·129 lines (99 loc) · 223 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
---
layout: guidelines
title: "ESIP Software Guidelines Draft"
---
<html><head><meta content="text/html; charset=UTF-8" http-equiv="content-type"><style type="text/css">ul.lst-kix_w9dmjfpkrgmp-3{list-style-type:none}ul.lst-kix_w9dmjfpkrgmp-4{list-style-type:none}ul.lst-kix_w9dmjfpkrgmp-5{list-style-type:none}ul.lst-kix_w9dmjfpkrgmp-6{list-style-type:none}ul.lst-kix_w9dmjfpkrgmp-0{list-style-type:none}ul.lst-kix_w9dmjfpkrgmp-1{list-style-type:none}ul.lst-kix_w9dmjfpkrgmp-2{list-style-type:none}ol.lst-kix_shfd0z5borlh-1.start{counter-reset:lst-ctn-kix_shfd0z5borlh-1 0}ol.lst-kix_bz8hwtl93rmi-7.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-7 0}.lst-kix_xcfwdxntv4m-3>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-3}ol.lst-kix_4cboiy4h3etg-8.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-8 0}ol.lst-kix_kz9r144lwdvm-5.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-5 0}ol.lst-kix_bz8hwtl93rmi-2.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-2 0}.lst-kix_bz8hwtl93rmi-4>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-4}ol.lst-kix_2qefniq611a3-0.start{counter-reset:lst-ctn-kix_2qefniq611a3-0 0}ul.lst-kix_w9dmjfpkrgmp-7{list-style-type:none}ul.lst-kix_w9dmjfpkrgmp-8{list-style-type:none}ol.lst-kix_shfd0z5borlh-6.start{counter-reset:lst-ctn-kix_shfd0z5borlh-6 0}ol.lst-kix_2qefniq611a3-5.start{counter-reset:lst-ctn-kix_2qefniq611a3-5 0}.lst-kix_4cboiy4h3etg-5>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-5}.lst-kix_2qefniq611a3-3>li{counter-increment:lst-ctn-kix_2qefniq611a3-3}.lst-kix_kt2trrwrjj4x-7>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-7}ol.lst-kix_4cboiy4h3etg-3.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-3 0}.lst-kix_kt2trrwrjj4x-0>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-0}.lst-kix_kpzm26rv33it-8>li{counter-increment:lst-ctn-kix_kpzm26rv33it-8}ol.lst-kix_pe1x2tciq74l-4.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-4 0}ol.lst-kix_kz9r144lwdvm-0.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-0 0}ol.lst-kix_shfd0z5borlh-8{list-style-type:none}ol.lst-kix_shfd0z5borlh-6{list-style-type:none}ol.lst-kix_shfd0z5borlh-7{list-style-type:none}ul.lst-kix_j4i5j7saczsg-3{list-style-type:none}ol.lst-kix_shfd0z5borlh-4{list-style-type:none}ul.lst-kix_j4i5j7saczsg-4{list-style-type:none}ol.lst-kix_shfd0z5borlh-5{list-style-type:none}ul.lst-kix_j4i5j7saczsg-1{list-style-type:none}ol.lst-kix_shfd0z5borlh-2{list-style-type:none}ul.lst-kix_j4i5j7saczsg-2{list-style-type:none}ol.lst-kix_shfd0z5borlh-3{list-style-type:none}ol.lst-kix_shfd0z5borlh-0{list-style-type:none}ul.lst-kix_j4i5j7saczsg-0{list-style-type:none}ol.lst-kix_shfd0z5borlh-1{list-style-type:none}ul.lst-kix_ce68zz3v1vbp-0{list-style-type:none}ul.lst-kix_ce68zz3v1vbp-2{list-style-type:none}ul.lst-kix_ce68zz3v1vbp-1{list-style-type:none}.lst-kix_kt2trrwrjj4x-4>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-4}ul.lst-kix_ce68zz3v1vbp-4{list-style-type:none}ul.lst-kix_ce68zz3v1vbp-3{list-style-type:none}ul.lst-kix_ce68zz3v1vbp-6{list-style-type:none}.lst-kix_kt2trrwrjj4x-1>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-1,lower-latin) ". "}ul.lst-kix_ce68zz3v1vbp-5{list-style-type:none}.lst-kix_4cboiy4h3etg-1>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-1,lower-latin) ". "}.lst-kix_kt2trrwrjj4x-7>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-7,lower-latin) ". "}.lst-kix_pe1x2tciq74l-2>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-2}.lst-kix_kpzm26rv33it-5>li{counter-increment:lst-ctn-kix_kpzm26rv33it-5}.lst-kix_shfd0z5borlh-7>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-7,lower-latin) ". "}.lst-kix_w9dmjfpkrgmp-0>li:before{content:"\0025cf "}ol.lst-kix_xcfwdxntv4m-3.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-3 0}.lst-kix_pe1x2tciq74l-3>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-3}ol.lst-kix_kpzm26rv33it-6.start{counter-reset:lst-ctn-kix_kpzm26rv33it-6 0}.lst-kix_bz8hwtl93rmi-7>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-7}.lst-kix_4cboiy4h3etg-7>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-7,lower-latin) ". "}.lst-kix_4cboiy4h3etg-5>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-5,lower-roman) ". "}ol.lst-kix_shfd0z5borlh-8.start{counter-reset:lst-ctn-kix_shfd0z5borlh-8 0}.lst-kix_kz9r144lwdvm-2>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-2}ul.lst-kix_ce68zz3v1vbp-8{list-style-type:none}ul.lst-kix_ce68zz3v1vbp-7{list-style-type:none}.lst-kix_4cboiy4h3etg-2>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-2}ol.lst-kix_kt2trrwrjj4x-2.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-2 0}.lst-kix_4cboiy4h3etg-3>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-3,decimal) ". "}.lst-kix_kz9r144lwdvm-7>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-7,lower-latin) ". "}.lst-kix_kz9r144lwdvm-5>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-5,lower-roman) ". "}.lst-kix_shfd0z5borlh-5>li{counter-increment:lst-ctn-kix_shfd0z5borlh-5}ol.lst-kix_xcfwdxntv4m-5.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-5 0}ol.lst-kix_kt2trrwrjj4x-0.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-0 0}.lst-kix_kz9r144lwdvm-1>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-1,lower-latin) ". "}.lst-kix_kz9r144lwdvm-3>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-3,decimal) ". "}ol.lst-kix_pe1x2tciq74l-1.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-1 0}ol.lst-kix_2qefniq611a3-2.start{counter-reset:lst-ctn-kix_2qefniq611a3-2 0}ol.lst-kix_bz8hwtl93rmi-4.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-4 0}ol.lst-kix_pe1x2tciq74l-2{list-style-type:none}ol.lst-kix_pe1x2tciq74l-1{list-style-type:none}.lst-kix_2qefniq611a3-4>li{counter-increment:lst-ctn-kix_2qefniq611a3-4}.lst-kix_kpzm26rv33it-4>li{counter-increment:lst-ctn-kix_kpzm26rv33it-4}ol.lst-kix_pe1x2tciq74l-4{list-style-type:none}ol.lst-kix_bz8hwtl93rmi-0{list-style-type:none}ol.lst-kix_pe1x2tciq74l-3{list-style-type:none}ol.lst-kix_bz8hwtl93rmi-1{list-style-type:none}ol.lst-kix_pe1x2tciq74l-0{list-style-type:none}ol.lst-kix_bz8hwtl93rmi-6{list-style-type:none}ol.lst-kix_bz8hwtl93rmi-7{list-style-type:none}ol.lst-kix_bz8hwtl93rmi-8{list-style-type:none}ol.lst-kix_bz8hwtl93rmi-2{list-style-type:none}ol.lst-kix_4cboiy4h3etg-6.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-6 0}ol.lst-kix_bz8hwtl93rmi-3{list-style-type:none}.lst-kix_shfd0z5borlh-5>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-5,lower-roman) ". "}ol.lst-kix_bz8hwtl93rmi-4{list-style-type:none}ol.lst-kix_kpzm26rv33it-4.start{counter-reset:lst-ctn-kix_kpzm26rv33it-4 0}ol.lst-kix_bz8hwtl93rmi-5{list-style-type:none}.lst-kix_kz9r144lwdvm-3>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-3}.lst-kix_kt2trrwrjj4x-5>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-5,lower-roman) ". "}.lst-kix_shfd0z5borlh-3>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-3,decimal) ". "}.lst-kix_bz8hwtl93rmi-0>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-0}.lst-kix_kt2trrwrjj4x-3>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-3,decimal) ". "}.lst-kix_shfd0z5borlh-1>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-1,lower-latin) ". "}.lst-kix_xpy554lpybad-4>li:before{content:"\0025cb "}.lst-kix_xpy554lpybad-8>li:before{content:"\0025a0 "}.lst-kix_xpy554lpybad-5>li:before{content:"\0025a0 "}.lst-kix_xpy554lpybad-0>li:before{content:"\0025cf "}.lst-kix_shfd0z5borlh-8>li{counter-increment:lst-ctn-kix_shfd0z5borlh-8}ol.lst-kix_4cboiy4h3etg-5.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-5 0}ol.lst-kix_pe1x2tciq74l-6{list-style-type:none}ol.lst-kix_pe1x2tciq74l-5{list-style-type:none}.lst-kix_xpy554lpybad-1>li:before{content:"\0025cb "}ol.lst-kix_pe1x2tciq74l-8{list-style-type:none}ol.lst-kix_pe1x2tciq74l-7{list-style-type:none}.lst-kix_2qefniq611a3-1>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-1,lower-latin) ". "}.lst-kix_bz8hwtl93rmi-7>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-7,lower-latin) ". "}.lst-kix_j4i5j7saczsg-7>li:before{content:"\0025cb "}.lst-kix_2qefniq611a3-2>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-2,lower-roman) ". "}.lst-kix_bz8hwtl93rmi-6>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-6,decimal) ". "}ol.lst-kix_kpzm26rv33it-1{list-style-type:none}ol.lst-kix_kpzm26rv33it-0{list-style-type:none}ol.lst-kix_kpzm26rv33it-3{list-style-type:none}ol.lst-kix_kpzm26rv33it-2{list-style-type:none}.lst-kix_2qefniq611a3-5>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-5,lower-roman) ". "}ol.lst-kix_kpzm26rv33it-5{list-style-type:none}.lst-kix_j4i5j7saczsg-3>li:before{content:"\0025cf "}ol.lst-kix_kpzm26rv33it-4{list-style-type:none}.lst-kix_xcfwdxntv4m-3>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-3,decimal) ". "}ol.lst-kix_kpzm26rv33it-7{list-style-type:none}ol.lst-kix_kpzm26rv33it-6{list-style-type:none}.lst-kix_ukky5gftyjz0-1>li:before{content:"\0025cb "}.lst-kix_xcfwdxntv4m-2>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-2,lower-roman) ". "}.lst-kix_j4i5j7saczsg-2>li:before{content:"\0025a0 "}ol.lst-kix_kpzm26rv33it-8{list-style-type:none}.lst-kix_ukky5gftyjz0-0>li:before{content:"\0025cf "}.lst-kix_xcfwdxntv4m-7>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-7,lower-latin) ". "}.lst-kix_pe1x2tciq74l-7>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-7}ol.lst-kix_bz8hwtl93rmi-5.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-5 0}.lst-kix_2qefniq611a3-6>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-6,decimal) ". "}ol.lst-kix_xcfwdxntv4m-2.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-2 0}.lst-kix_xcfwdxntv4m-6>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-6,decimal) ". "}ol.lst-kix_kpzm26rv33it-2.start{counter-reset:lst-ctn-kix_kpzm26rv33it-2 0}.lst-kix_j4i5j7saczsg-6>li:before{content:"\0025cf "}ol.lst-kix_pe1x2tciq74l-2.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-2 0}.lst-kix_shfd0z5borlh-6>li{counter-increment:lst-ctn-kix_shfd0z5borlh-6}.lst-kix_kt2trrwrjj4x-1>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-1}.lst-kix_pe1x2tciq74l-5>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-5}.lst-kix_xcfwdxntv4m-6>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-6}ol.lst-kix_kpzm26rv33it-1.start{counter-reset:lst-ctn-kix_kpzm26rv33it-1 0}.lst-kix_kt2trrwrjj4x-6>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-6}.lst-kix_2qefniq611a3-2>li{counter-increment:lst-ctn-kix_2qefniq611a3-2}.lst-kix_ukky5gftyjz0-4>li:before{content:"\0025cb "}ol.lst-kix_2qefniq611a3-2{list-style-type:none}.lst-kix_jf0gg8kp4t2b-4>li:before{content:"\0025cb "}ol.lst-kix_2qefniq611a3-1{list-style-type:none}.lst-kix_pe1x2tciq74l-6>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-6,decimal) ". "}ol.lst-kix_xcfwdxntv4m-1.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-1 0}ol.lst-kix_2qefniq611a3-4{list-style-type:none}ol.lst-kix_2qefniq611a3-3{list-style-type:none}.lst-kix_ukky5gftyjz0-5>li:before{content:"\0025a0 "}.lst-kix_ce68zz3v1vbp-8>li:before{content:"\0025a0 "}.lst-kix_w9dmjfpkrgmp-5>li:before{content:"\0025a0 "}.lst-kix_shfd0z5borlh-1>li{counter-increment:lst-ctn-kix_shfd0z5borlh-1}.lst-kix_pe1x2tciq74l-3>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-3,decimal) ". "}.lst-kix_pe1x2tciq74l-7>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-7,lower-latin) ". "}ol.lst-kix_2qefniq611a3-0{list-style-type:none}.lst-kix_jf0gg8kp4t2b-3>li:before{content:"\0025cf "}.lst-kix_jf0gg8kp4t2b-7>li:before{content:"\0025cb "}.lst-kix_w9dmjfpkrgmp-4>li:before{content:"\0025cb "}.lst-kix_w9dmjfpkrgmp-8>li:before{content:"\0025a0 "}.lst-kix_jf0gg8kp4t2b-0>li:before{content:"\0025cf "}.lst-kix_jf0gg8kp4t2b-8>li:before{content:"\0025a0 "}.lst-kix_ce68zz3v1vbp-4>li:before{content:"\0025cb "}.lst-kix_w9dmjfpkrgmp-1>li:before{content:"\0025cb "}.lst-kix_ukky5gftyjz0-8>li:before{content:"\0025a0 "}.lst-kix_ce68zz3v1vbp-3>li:before{content:"\0025cf "}.lst-kix_pe1x2tciq74l-2>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-2,lower-roman) ". "}ol.lst-kix_2qefniq611a3-6{list-style-type:none}ol.lst-kix_2qefniq611a3-5{list-style-type:none}ol.lst-kix_2qefniq611a3-8{list-style-type:none}ol.lst-kix_2qefniq611a3-7{list-style-type:none}.lst-kix_ce68zz3v1vbp-7>li:before{content:"\0025cb "}ol.lst-kix_kpzm26rv33it-0.start{counter-reset:lst-ctn-kix_kpzm26rv33it-0 0}ul.lst-kix_j4i5j7saczsg-7{list-style-type:none}.lst-kix_rxqk3bi4h806-6>li:before{content:"\0025cf "}ul.lst-kix_j4i5j7saczsg-8{list-style-type:none}ul.lst-kix_j4i5j7saczsg-5{list-style-type:none}ul.lst-kix_j4i5j7saczsg-6{list-style-type:none}ol.lst-kix_xcfwdxntv4m-0.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-0 0}.lst-kix_2qefniq611a3-0>li{counter-increment:lst-ctn-kix_2qefniq611a3-0}.lst-kix_m4hdmy5b3bhc-7>li:before{content:"- "}.lst-kix_kt2trrwrjj4x-6>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-6,decimal) ". "}.lst-kix_m4hdmy5b3bhc-3>li:before{content:"- "}.lst-kix_ce68zz3v1vbp-0>li:before{content:"\0025cf "}ol.lst-kix_pe1x2tciq74l-6.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-6 0}.lst-kix_kt2trrwrjj4x-3>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-3}ol.lst-kix_4cboiy4h3etg-1.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-1 0}.lst-kix_kpzm26rv33it-0>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-0,decimal) ". "}.lst-kix_hputbrx9prs6-1>li:before{content:"\0025cb "}.lst-kix_rxqk3bi4h806-2>li:before{content:"\0025a0 "}.lst-kix_4cboiy4h3etg-2>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-2,lower-roman) ". "}.lst-kix_4cboiy4h3etg-6>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-6,decimal) ". "}.lst-kix_kpzm26rv33it-6>li{counter-increment:lst-ctn-kix_kpzm26rv33it-6}.lst-kix_shfd0z5borlh-3>li{counter-increment:lst-ctn-kix_shfd0z5borlh-3}.lst-kix_xcfwdxntv4m-8>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-8}ol.lst-kix_bz8hwtl93rmi-8.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-8 0}.lst-kix_kz9r144lwdvm-8>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-8,lower-roman) ". "}ol.lst-kix_4cboiy4h3etg-0.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-0 0}ol.lst-kix_pe1x2tciq74l-8.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-8 0}.lst-kix_kz9r144lwdvm-4>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-4,lower-latin) ". "}.lst-kix_4cboiy4h3etg-0>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-0}.lst-kix_kz9r144lwdvm-0>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-0,decimal) ". "}.lst-kix_4ons6eimy48-3>li:before{content:"\0025cf "}.lst-kix_4ons6eimy48-7>li:before{content:"\0025cb "}.lst-kix_bz8hwtl93rmi-3>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-3,decimal) ". "}.lst-kix_hputbrx9prs6-5>li:before{content:"\0025a0 "}.lst-kix_kpzm26rv33it-4>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-4,lower-latin) ". "}.lst-kix_2qefniq611a3-7>li{counter-increment:lst-ctn-kix_2qefniq611a3-7}.lst-kix_kpzm26rv33it-1>li{counter-increment:lst-ctn-kix_kpzm26rv33it-1}.lst-kix_pe1x2tciq74l-0>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-0}.lst-kix_shfd0z5borlh-6>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-6,decimal) ". "}.lst-kix_kpzm26rv33it-8>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-8,lower-roman) ". "}.lst-kix_xcfwdxntv4m-1>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-1}.lst-kix_kz9r144lwdvm-6>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-6}ol.lst-kix_pe1x2tciq74l-7.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-7 0}.lst-kix_kt2trrwrjj4x-2>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-2,lower-roman) ". "}.lst-kix_kt2trrwrjj4x-8>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-8}.lst-kix_shfd0z5borlh-2>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-2,lower-roman) ". "}ul.lst-kix_rxqk3bi4h806-0{list-style-type:none}ol.lst-kix_4cboiy4h3etg-2.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-2 0}.lst-kix_kz9r144lwdvm-8>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-8}ul.lst-kix_rxqk3bi4h806-8{list-style-type:none}ul.lst-kix_rxqk3bi4h806-7{list-style-type:none}.lst-kix_bz8hwtl93rmi-2>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-2}ul.lst-kix_rxqk3bi4h806-6{list-style-type:none}ul.lst-kix_rxqk3bi4h806-5{list-style-type:none}ul.lst-kix_rxqk3bi4h806-4{list-style-type:none}.lst-kix_2qefniq611a3-5>li{counter-increment:lst-ctn-kix_2qefniq611a3-5}ul.lst-kix_rxqk3bi4h806-3{list-style-type:none}ul.lst-kix_rxqk3bi4h806-2{list-style-type:none}ul.lst-kix_rxqk3bi4h806-1{list-style-type:none}ol.lst-kix_shfd0z5borlh-7.start{counter-reset:lst-ctn-kix_shfd0z5borlh-7 0}.lst-kix_4cboiy4h3etg-7>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-7}ul.lst-kix_4ons6eimy48-5{list-style-type:none}ul.lst-kix_4ons6eimy48-6{list-style-type:none}ul.lst-kix_4ons6eimy48-3{list-style-type:none}ul.lst-kix_4ons6eimy48-4{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-6.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-6 0}.lst-kix_xcfwdxntv4m-5>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-5}ul.lst-kix_4ons6eimy48-1{list-style-type:none}ul.lst-kix_4ons6eimy48-2{list-style-type:none}ul.lst-kix_4ons6eimy48-0{list-style-type:none}ul.lst-kix_4ons6eimy48-7{list-style-type:none}ul.lst-kix_4ons6eimy48-8{list-style-type:none}ol.lst-kix_2qefniq611a3-6.start{counter-reset:lst-ctn-kix_2qefniq611a3-6 0}ol.lst-kix_pe1x2tciq74l-5.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-5 0}ul.lst-kix_hputbrx9prs6-8{list-style-type:none}ul.lst-kix_hputbrx9prs6-7{list-style-type:none}ul.lst-kix_hputbrx9prs6-6{list-style-type:none}ul.lst-kix_hputbrx9prs6-5{list-style-type:none}ul.lst-kix_hputbrx9prs6-4{list-style-type:none}.lst-kix_shfd0z5borlh-0>li{counter-increment:lst-ctn-kix_shfd0z5borlh-0}ol.lst-kix_bz8hwtl93rmi-1.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-1 0}ol.lst-kix_kt2trrwrjj4x-1{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-2{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-3{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-4{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-0{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-5{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-6{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-7{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-8{list-style-type:none}.lst-kix_kz9r144lwdvm-1>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-1}.lst-kix_shfd0z5borlh-7>li{counter-increment:lst-ctn-kix_shfd0z5borlh-7}ol.lst-kix_shfd0z5borlh-0.start{counter-reset:lst-ctn-kix_shfd0z5borlh-0 0}ol.lst-kix_bz8hwtl93rmi-6.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-6 0}.lst-kix_pe1x2tciq74l-8>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-8}ol.lst-kix_kz9r144lwdvm-6.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-6 0}ol.lst-kix_kz9r144lwdvm-4{list-style-type:none}.lst-kix_rxqk3bi4h806-7>li:before{content:"\0025cb "}ol.lst-kix_kz9r144lwdvm-5{list-style-type:none}ol.lst-kix_bz8hwtl93rmi-3.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-3 0}ol.lst-kix_kz9r144lwdvm-6{list-style-type:none}ol.lst-kix_kz9r144lwdvm-7{list-style-type:none}ol.lst-kix_kz9r144lwdvm-0{list-style-type:none}.lst-kix_m4hdmy5b3bhc-8>li:before{content:"- "}.lst-kix_rxqk3bi4h806-5>li:before{content:"\0025a0 "}ol.lst-kix_kz9r144lwdvm-1{list-style-type:none}ol.lst-kix_kz9r144lwdvm-2{list-style-type:none}ol.lst-kix_pe1x2tciq74l-3.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-3 0}ol.lst-kix_kz9r144lwdvm-3{list-style-type:none}.lst-kix_m4hdmy5b3bhc-6>li:before{content:"- "}ol.lst-kix_kz9r144lwdvm-8{list-style-type:none}.lst-kix_4cboiy4h3etg-3>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-3}.lst-kix_m4hdmy5b3bhc-2>li:before{content:"- "}.lst-kix_m4hdmy5b3bhc-0>li:before{content:"- "}.lst-kix_m4hdmy5b3bhc-4>li:before{content:"- "}ol.lst-kix_kpzm26rv33it-3.start{counter-reset:lst-ctn-kix_kpzm26rv33it-3 0}.lst-kix_ce68zz3v1vbp-1>li:before{content:"\0025cb "}.lst-kix_kpzm26rv33it-3>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-3,decimal) ". "}.lst-kix_kz9r144lwdvm-4>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-4}.lst-kix_kpzm26rv33it-1>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-1,lower-latin) ". "}.lst-kix_rxqk3bi4h806-1>li:before{content:"\0025cb "}.lst-kix_hputbrx9prs6-0>li:before{content:"\0025cf "}.lst-kix_pe1x2tciq74l-4>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-4}.lst-kix_kpzm26rv33it-3>li{counter-increment:lst-ctn-kix_kpzm26rv33it-3}.lst-kix_rxqk3bi4h806-3>li:before{content:"\0025cf "}ol.lst-kix_4cboiy4h3etg-4.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-4 0}ul.lst-kix_hputbrx9prs6-3{list-style-type:none}ul.lst-kix_hputbrx9prs6-2{list-style-type:none}ul.lst-kix_hputbrx9prs6-1{list-style-type:none}ol.lst-kix_4cboiy4h3etg-7.start{counter-reset:lst-ctn-kix_4cboiy4h3etg-7 0}ul.lst-kix_hputbrx9prs6-0{list-style-type:none}ol.lst-kix_kpzm26rv33it-5.start{counter-reset:lst-ctn-kix_kpzm26rv33it-5 0}.lst-kix_hputbrx9prs6-8>li:before{content:"\0025a0 "}.lst-kix_bz8hwtl93rmi-6>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-6}.lst-kix_4ons6eimy48-8>li:before{content:"\0025a0 "}.lst-kix_hputbrx9prs6-2>li:before{content:"\0025a0 "}.lst-kix_4ons6eimy48-6>li:before{content:"\0025cf "}.lst-kix_hputbrx9prs6-4>li:before{content:"\0025cb "}.lst-kix_bz8hwtl93rmi-4>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-4,lower-latin) ". "}.lst-kix_4ons6eimy48-4>li:before{content:"\0025cb "}.lst-kix_hputbrx9prs6-6>li:before{content:"\0025cf "}.lst-kix_kpzm26rv33it-5>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-5,lower-roman) ". "}ol.lst-kix_kt2trrwrjj4x-1.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-1 0}.lst-kix_kpzm26rv33it-7>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-7,lower-latin) ". "}.lst-kix_4ons6eimy48-2>li:before{content:"\0025a0 "}.lst-kix_bz8hwtl93rmi-2>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-2,lower-roman) ". "}.lst-kix_4ons6eimy48-0>li:before{content:"\0025cf "}ol.lst-kix_pe1x2tciq74l-0.start{counter-reset:lst-ctn-kix_pe1x2tciq74l-0 0}ol.lst-kix_xcfwdxntv4m-4.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-4 0}.lst-kix_xcfwdxntv4m-4>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-4}.lst-kix_bz8hwtl93rmi-0>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-0,decimal) ". "}ol.lst-kix_2qefniq611a3-1.start{counter-reset:lst-ctn-kix_2qefniq611a3-1 0}.lst-kix_kt2trrwrjj4x-5>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-5}.lst-kix_xpy554lpybad-6>li:before{content:"\0025cf "}ol.lst-kix_2qefniq611a3-3.start{counter-reset:lst-ctn-kix_2qefniq611a3-3 0}.lst-kix_xpy554lpybad-3>li:before{content:"\0025cf "}.lst-kix_xpy554lpybad-7>li:before{content:"\0025cb "}.lst-kix_4cboiy4h3etg-8>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-8}.lst-kix_2qefniq611a3-6>li{counter-increment:lst-ctn-kix_2qefniq611a3-6}ol.lst-kix_shfd0z5borlh-4.start{counter-reset:lst-ctn-kix_shfd0z5borlh-4 0}.lst-kix_j4i5j7saczsg-0>li:before{content:"\0025cf "}.lst-kix_xpy554lpybad-2>li:before{content:"\0025a0 "}ol.lst-kix_kz9r144lwdvm-2.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-2 0}.lst-kix_xcfwdxntv4m-0>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-0,decimal) ". "}.lst-kix_j4i5j7saczsg-8>li:before{content:"\0025a0 "}.lst-kix_xcfwdxntv4m-1>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-1,lower-latin) ". "}.lst-kix_xcfwdxntv4m-2>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-2}.lst-kix_2qefniq611a3-0>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-0,decimal) ". "}.lst-kix_bz8hwtl93rmi-8>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-8,lower-roman) ". "}.lst-kix_ukky5gftyjz0-3>li:before{content:"\0025cf "}.lst-kix_kz9r144lwdvm-7>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-7}.lst-kix_ukky5gftyjz0-2>li:before{content:"\0025a0 "}.lst-kix_xcfwdxntv4m-4>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-4,lower-latin) ". "}.lst-kix_2qefniq611a3-4>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-4,lower-latin) ". "}.lst-kix_2qefniq611a3-3>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-3,decimal) ". "}.lst-kix_j4i5j7saczsg-1>li:before{content:"\0025cb "}.lst-kix_bz8hwtl93rmi-1>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-1}.lst-kix_xcfwdxntv4m-8>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-8,lower-roman) ". "}ul.lst-kix_xpy554lpybad-8{list-style-type:none}.lst-kix_2qefniq611a3-8>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-8,lower-roman) ". "}ul.lst-kix_xpy554lpybad-6{list-style-type:none}ul.lst-kix_xpy554lpybad-7{list-style-type:none}ol.lst-kix_kz9r144lwdvm-8.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-8 0}ul.lst-kix_xpy554lpybad-0{list-style-type:none}ul.lst-kix_xpy554lpybad-1{list-style-type:none}.lst-kix_j4i5j7saczsg-4>li:before{content:"\0025cb "}.lst-kix_xcfwdxntv4m-5>li:before{content:"" counter(lst-ctn-kix_xcfwdxntv4m-5,lower-roman) ". "}.lst-kix_kpzm26rv33it-0>li{counter-increment:lst-ctn-kix_kpzm26rv33it-0}.lst-kix_2qefniq611a3-7>li:before{content:"" counter(lst-ctn-kix_2qefniq611a3-7,lower-latin) ". "}ul.lst-kix_xpy554lpybad-4{list-style-type:none}ul.lst-kix_xpy554lpybad-5{list-style-type:none}.lst-kix_j4i5j7saczsg-5>li:before{content:"\0025a0 "}ul.lst-kix_xpy554lpybad-2{list-style-type:none}ul.lst-kix_xpy554lpybad-3{list-style-type:none}.lst-kix_kz9r144lwdvm-5>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-5}.lst-kix_2qefniq611a3-8>li{counter-increment:lst-ctn-kix_2qefniq611a3-8}ul.lst-kix_jf0gg8kp4t2b-8{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-3.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-3 0}.lst-kix_kpzm26rv33it-2>li{counter-increment:lst-ctn-kix_kpzm26rv33it-2}ol.lst-kix_kz9r144lwdvm-7.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-7 0}ol.lst-kix_xcfwdxntv4m-6.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-6 0}.lst-kix_xcfwdxntv4m-0>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-0}.lst-kix_bz8hwtl93rmi-5>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-5}.lst-kix_4cboiy4h3etg-1>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-1}.lst-kix_pe1x2tciq74l-5>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-5,lower-roman) ". "}.lst-kix_4cboiy4h3etg-4>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-4}.lst-kix_w9dmjfpkrgmp-7>li:before{content:"\0025cb "}ol.lst-kix_2qefniq611a3-4.start{counter-reset:lst-ctn-kix_2qefniq611a3-4 0}.lst-kix_pe1x2tciq74l-4>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-4,lower-latin) ". "}.lst-kix_pe1x2tciq74l-8>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-8,lower-roman) ". "}.lst-kix_ukky5gftyjz0-6>li:before{content:"\0025cf "}.lst-kix_ce68zz3v1vbp-5>li:before{content:"\0025a0 "}.lst-kix_shfd0z5borlh-4>li{counter-increment:lst-ctn-kix_shfd0z5borlh-4}ul.lst-kix_jf0gg8kp4t2b-0{list-style-type:none}ul.lst-kix_jf0gg8kp4t2b-1{list-style-type:none}ul.lst-kix_jf0gg8kp4t2b-2{list-style-type:none}ol.lst-kix_kt2trrwrjj4x-4.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-4 0}.lst-kix_ukky5gftyjz0-7>li:before{content:"\0025cb "}ul.lst-kix_jf0gg8kp4t2b-3{list-style-type:none}.lst-kix_w9dmjfpkrgmp-3>li:before{content:"\0025cf "}.lst-kix_jf0gg8kp4t2b-2>li:before{content:"\0025a0 "}ul.lst-kix_jf0gg8kp4t2b-4{list-style-type:none}ul.lst-kix_jf0gg8kp4t2b-5{list-style-type:none}.lst-kix_jf0gg8kp4t2b-1>li:before{content:"\0025cb "}ul.lst-kix_jf0gg8kp4t2b-6{list-style-type:none}ul.lst-kix_jf0gg8kp4t2b-7{list-style-type:none}.lst-kix_w9dmjfpkrgmp-2>li:before{content:"\0025a0 "}ol.lst-kix_bz8hwtl93rmi-0.start{counter-reset:lst-ctn-kix_bz8hwtl93rmi-0 0}.lst-kix_pe1x2tciq74l-0>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-0,decimal) ". "}ol.lst-kix_kpzm26rv33it-7.start{counter-reset:lst-ctn-kix_kpzm26rv33it-7 0}.lst-kix_pe1x2tciq74l-1>li:before{content:"" counter(lst-ctn-kix_pe1x2tciq74l-1,lower-latin) ". "}.lst-kix_ce68zz3v1vbp-6>li:before{content:"\0025cf "}.lst-kix_jf0gg8kp4t2b-6>li:before{content:"\0025cf "}ol.lst-kix_xcfwdxntv4m-7.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-7 0}.lst-kix_jf0gg8kp4t2b-5>li:before{content:"\0025a0 "}.lst-kix_w9dmjfpkrgmp-6>li:before{content:"\0025cf "}.lst-kix_kt2trrwrjj4x-0>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-0,decimal) ". "}.lst-kix_rxqk3bi4h806-8>li:before{content:"\0025a0 "}ol.lst-kix_kt2trrwrjj4x-5.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-5 0}ol.lst-kix_4cboiy4h3etg-5{list-style-type:none}.lst-kix_4cboiy4h3etg-0>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-0,decimal) ". "}ol.lst-kix_4cboiy4h3etg-4{list-style-type:none}ol.lst-kix_4cboiy4h3etg-3{list-style-type:none}.lst-kix_m4hdmy5b3bhc-5>li:before{content:"- "}ol.lst-kix_4cboiy4h3etg-2{list-style-type:none}ol.lst-kix_4cboiy4h3etg-8{list-style-type:none}ol.lst-kix_4cboiy4h3etg-7{list-style-type:none}ol.lst-kix_4cboiy4h3etg-6{list-style-type:none}.lst-kix_shfd0z5borlh-2>li{counter-increment:lst-ctn-kix_shfd0z5borlh-2}.lst-kix_kt2trrwrjj4x-8>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-8,lower-roman) ". "}.lst-kix_m4hdmy5b3bhc-1>li:before{content:"- "}.lst-kix_ce68zz3v1vbp-2>li:before{content:"\0025a0 "}ol.lst-kix_kt2trrwrjj4x-8.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-8 0}.lst-kix_shfd0z5borlh-8>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-8,lower-roman) ". "}.lst-kix_bz8hwtl93rmi-8>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-8}ol.lst-kix_2qefniq611a3-7.start{counter-reset:lst-ctn-kix_2qefniq611a3-7 0}.lst-kix_kpzm26rv33it-2>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-2,lower-roman) ". "}ol.lst-kix_xcfwdxntv4m-8.start{counter-reset:lst-ctn-kix_xcfwdxntv4m-8 0}.lst-kix_4cboiy4h3etg-8>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-8,lower-roman) ". "}ol.lst-kix_kpzm26rv33it-8.start{counter-reset:lst-ctn-kix_kpzm26rv33it-8 0}ol.lst-kix_shfd0z5borlh-5.start{counter-reset:lst-ctn-kix_shfd0z5borlh-5 0}.lst-kix_pe1x2tciq74l-1>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-1}.lst-kix_rxqk3bi4h806-0>li:before{content:"\0025cf "}.lst-kix_4cboiy4h3etg-4>li:before{content:"" counter(lst-ctn-kix_4cboiy4h3etg-4,lower-latin) ". "}ol.lst-kix_kz9r144lwdvm-1.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-1 0}.lst-kix_rxqk3bi4h806-4>li:before{content:"\0025cb "}ul.lst-kix_m4hdmy5b3bhc-4{list-style-type:none}ul.lst-kix_m4hdmy5b3bhc-3{list-style-type:none}ul.lst-kix_ukky5gftyjz0-1{list-style-type:none}ul.lst-kix_m4hdmy5b3bhc-6{list-style-type:none}ul.lst-kix_ukky5gftyjz0-0{list-style-type:none}ul.lst-kix_m4hdmy5b3bhc-5{list-style-type:none}ul.lst-kix_ukky5gftyjz0-3{list-style-type:none}ul.lst-kix_m4hdmy5b3bhc-8{list-style-type:none}ul.lst-kix_ukky5gftyjz0-2{list-style-type:none}ul.lst-kix_m4hdmy5b3bhc-7{list-style-type:none}ul.lst-kix_ukky5gftyjz0-5{list-style-type:none}ul.lst-kix_ukky5gftyjz0-4{list-style-type:none}.lst-kix_kz9r144lwdvm-6>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-6,decimal) ". "}ul.lst-kix_m4hdmy5b3bhc-0{list-style-type:none}.lst-kix_kz9r144lwdvm-0>li{counter-increment:lst-ctn-kix_kz9r144lwdvm-0}ol.lst-kix_kz9r144lwdvm-4.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-4 0}ul.lst-kix_m4hdmy5b3bhc-2{list-style-type:none}ul.lst-kix_m4hdmy5b3bhc-1{list-style-type:none}.lst-kix_xcfwdxntv4m-7>li{counter-increment:lst-ctn-kix_xcfwdxntv4m-7}.lst-kix_kz9r144lwdvm-2>li:before{content:"" counter(lst-ctn-kix_kz9r144lwdvm-2,lower-roman) ". "}.lst-kix_hputbrx9prs6-3>li:before{content:"\0025cf "}ol.lst-kix_xcfwdxntv4m-1{list-style-type:none}ol.lst-kix_xcfwdxntv4m-0{list-style-type:none}ol.lst-kix_xcfwdxntv4m-3{list-style-type:none}ul.lst-kix_ukky5gftyjz0-7{list-style-type:none}ol.lst-kix_xcfwdxntv4m-2{list-style-type:none}ul.lst-kix_ukky5gftyjz0-6{list-style-type:none}ol.lst-kix_xcfwdxntv4m-5{list-style-type:none}ol.lst-kix_xcfwdxntv4m-4{list-style-type:none}.lst-kix_hputbrx9prs6-7>li:before{content:"\0025cb "}ul.lst-kix_ukky5gftyjz0-8{list-style-type:none}ol.lst-kix_xcfwdxntv4m-7{list-style-type:none}ol.lst-kix_shfd0z5borlh-2.start{counter-reset:lst-ctn-kix_shfd0z5borlh-2 0}.lst-kix_bz8hwtl93rmi-5>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-5,lower-roman) ". "}ol.lst-kix_xcfwdxntv4m-6{list-style-type:none}.lst-kix_4ons6eimy48-5>li:before{content:"\0025a0 "}ol.lst-kix_xcfwdxntv4m-8{list-style-type:none}.lst-kix_pe1x2tciq74l-6>li{counter-increment:lst-ctn-kix_pe1x2tciq74l-6}.lst-kix_kpzm26rv33it-7>li{counter-increment:lst-ctn-kix_kpzm26rv33it-7}.lst-kix_2qefniq611a3-1>li{counter-increment:lst-ctn-kix_2qefniq611a3-1}.lst-kix_kpzm26rv33it-6>li:before{content:"" counter(lst-ctn-kix_kpzm26rv33it-6,decimal) ". "}.lst-kix_bz8hwtl93rmi-1>li:before{content:"" counter(lst-ctn-kix_bz8hwtl93rmi-1,lower-latin) ". "}.lst-kix_4ons6eimy48-1>li:before{content:"\0025cb "}.lst-kix_shfd0z5borlh-4>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-4,lower-latin) ". "}.lst-kix_4cboiy4h3etg-6>li{counter-increment:lst-ctn-kix_4cboiy4h3etg-6}ol.lst-kix_shfd0z5borlh-3.start{counter-reset:lst-ctn-kix_shfd0z5borlh-3 0}.lst-kix_kt2trrwrjj4x-2>li{counter-increment:lst-ctn-kix_kt2trrwrjj4x-2}ol.lst-kix_4cboiy4h3etg-1{list-style-type:none}ol.lst-kix_4cboiy4h3etg-0{list-style-type:none}ol.lst-kix_2qefniq611a3-8.start{counter-reset:lst-ctn-kix_2qefniq611a3-8 0}.lst-kix_kt2trrwrjj4x-4>li:before{content:"" counter(lst-ctn-kix_kt2trrwrjj4x-4,lower-latin) ". "}ol.lst-kix_kz9r144lwdvm-3.start{counter-reset:lst-ctn-kix_kz9r144lwdvm-3 0}.lst-kix_shfd0z5borlh-0>li:before{content:"" counter(lst-ctn-kix_shfd0z5borlh-0,decimal) ". "}.lst-kix_bz8hwtl93rmi-3>li{counter-increment:lst-ctn-kix_bz8hwtl93rmi-3}ol.lst-kix_kt2trrwrjj4x-7.start{counter-reset:lst-ctn-kix_kt2trrwrjj4x-7 0}ol{margin:0;padding:0}table td,table th{padding:0}.c20{border-right-style:solid;padding:5pt 5pt 5pt 5pt;border-bottom-color:#000000;border-top-width:1pt;border-right-width:1pt;border-left-color:#000000;vertical-align:top;border-right-color:#000000;border-left-width:1pt;border-top-style:solid;border-left-style:solid;border-bottom-width:1pt;width:87.8pt;border-top-color:#000000;border-bottom-style:solid}.c33{border-right-style:solid;padding:5pt 5pt 5pt 5pt;border-bottom-color:#000000;border-top-width:1pt;border-right-width:1pt;border-left-color:#000000;vertical-align:top;border-right-color:#000000;border-left-width:1pt;border-top-style:solid;border-left-style:solid;border-bottom-width:1pt;width:156pt;border-top-color:#000000;border-bottom-style:solid}.c24{border-right-style:solid;padding:5pt 5pt 5pt 5pt;border-bottom-color:#000000;border-top-width:1pt;border-right-width:1pt;border-left-color:#000000;vertical-align:top;border-right-color:#000000;border-left-width:1pt;border-top-style:solid;border-left-style:solid;border-bottom-width:1pt;width:224.2pt;border-top-color:#000000;border-bottom-style:solid}.c16{border-right-style:solid;padding:5pt 5pt 5pt 5pt;border-bottom-color:#000000;border-top-width:1pt;border-right-width:1pt;border-left-color:#000000;vertical-align:top;border-right-color:#000000;border-left-width:1pt;border-top-style:solid;border-left-style:solid;border-bottom-width:1pt;width:93.6pt;border-top-color:#000000;border-bottom-style:solid}.c6{border-right-style:solid;padding:8pt 8pt 8pt 8pt;border-bottom-color:#555555;border-top-width:1pt;border-right-width:1pt;border-left-color:#555555;vertical-align:middle;border-right-color:#555555;border-left-width:1pt;border-top-style:solid;border-left-style:solid;border-bottom-width:1pt;width:108pt;border-top-color:#555555;border-bottom-style:solid}.c17{border-right-style:solid;padding:5pt 5pt 5pt 5pt;border-bottom-color:#000000;border-top-width:1pt;border-right-width:1pt;border-left-color:#000000;vertical-align:top;border-right-color:#000000;border-left-width:1pt;border-top-style:solid;border-left-style:solid;border-bottom-width:1pt;width:234pt;border-top-color:#000000;border-bottom-style:solid}.c5{border-right-style:solid;padding:8pt 8pt 8pt 8pt;border-bottom-color:#555555;border-top-width:1pt;border-right-width:1pt;border-left-color:#555555;vertical-align:middle;border-right-color:#555555;border-left-width:1pt;border-top-style:solid;border-left-style:solid;border-bottom-width:1pt;width:179.2pt;border-top-color:#555555;border-bottom-style:solid}.c30{border-right-style:solid;padding:8pt 8pt 8pt 8pt;border-bottom-color:#555555;border-top-width:1pt;border-right-width:1pt;border-left-color:#555555;vertical-align:middle;border-right-color:#555555;border-left-width:1pt;border-top-style:solid;border-left-style:solid;border-bottom-width:1pt;width:177pt;border-top-color:#555555;border-bottom-style:solid}.c13{background-color:#fdfdfd;color:#111111;font-weight:400;text-decoration:none;vertical-align:baseline;font-size:10pt;font-family:"Arial";font-style:normal}.c0{color:#000000;font-weight:400;text-decoration:none;vertical-align:baseline;font-size:10pt;font-family:"Arial";font-style:normal}.c7{color:#000000;font-weight:700;text-decoration:none;vertical-align:baseline;font-size:11pt;font-family:"Arial";font-style:normal}.c10{color:#000000;font-weight:400;text-decoration:none;vertical-align:baseline;font-size:11pt;font-family:"Arial";font-style:normal}.c32{color:#000000;text-decoration:none;vertical-align:baseline;font-size:10pt;font-family:"Arial";font-style:normal}.c38{border-spacing:0;border-collapse:collapse;margin-right:auto}.c19{padding-top:0pt;padding-bottom:0pt;line-height:1.15;text-align:left}.c12{padding-top:0pt;padding-bottom:0pt;line-height:1.0;text-align:left}.c31{padding-top:0pt;padding-bottom:0pt;line-height:1.15;text-align:center}.c34{background-color:#ffffff;max-width:468pt;padding:72pt 72pt 72pt 72pt}.c36{background-color:#ffffff;font-size:11.5pt}.c1{color:#1155cc;text-decoration:underline}.c2{orphans:2;widows:2}.c3{margin-left:36pt;padding-left:0pt}.c14{padding:0;margin:0}.c39{background-color:#fdfdfd;color:#111111}.c4{color:inherit;text-decoration:inherit}.c37{font-size:10pt}.c28{height:21pt}.c18{margin-left:72pt}.c8{page-break-after:avoid}.c41{text-align:right}.c29{padding-left:0pt}.c21{color:#cc0000}.c22{font-weight:700}.c35{height:28pt}.c26{color:#ff0000}.c11{height:0pt}.c15{margin-left:36pt}.c25{margin-left:54pt}.c9{height:11pt}.c27{padding-bottom:10pt}.c23{margin-left:18pt}.c40{height:40pt}.title{padding-top:0pt;color:#000000;font-size:26pt;padding-bottom:3pt;font-family:"Arial";line-height:1.15;page-break-after:avoid;orphans:2;widows:2;text-align:left}.subtitle{padding-top:0pt;color:#666666;font-size:15pt;padding-bottom:16pt;font-family:"Arial";line-height:1.15;page-break-after:avoid;orphans:2;widows:2;text-align:left}li{color:#000000;font-size:11pt;font-family:"Arial"}p{margin:0;color:#000000;font-size:11pt;font-family:"Arial"}h1{padding-top:20pt;color:#000000;font-size:20pt;padding-bottom:6pt;font-family:"Arial";line-height:1.15;page-break-after:avoid;orphans:2;widows:2;text-align:left}h2{padding-top:18pt;color:#000000;font-size:16pt;padding-bottom:6pt;font-family:"Arial";line-height:1.15;page-break-after:avoid;orphans:2;widows:2;text-align:left}h3{padding-top:16pt;color:#434343;font-size:14pt;padding-bottom:4pt;font-family:"Arial";line-height:1.15;page-break-after:avoid;orphans:2;widows:2;text-align:left}h4{padding-top:14pt;color:#666666;font-size:12pt;padding-bottom:4pt;font-family:"Arial";line-height:1.15;page-break-after:avoid;orphans:2;widows:2;text-align:left}h5{padding-top:12pt;color:#666666;font-size:11pt;padding-bottom:4pt;font-family:"Arial";line-height:1.15;page-break-after:avoid;orphans:2;widows:2;text-align:left}h6{padding-top:12pt;color:#666666;font-size:11pt;padding-bottom:4pt;font-family:"Arial";line-height:1.15;page-break-after:avoid;font-style:italic;orphans:2;widows:2;text-align:left}</style></head>
<body><p class="c2 c8 c9 title" id="h.7nzd33k2plcx"><span></span></p><p class="c2 c8 title" id="h.6rttn4epaywi"><span>ESIP Software Guidelines</span></p><p class="c2"><span>Editor: Soren Scott</span></p><p class="c2 c9"><span></span></p><p class="c2 c9"><span></span></p><p class="c2"><span>IN DRAFT Oct 2016</span>
<p class="c2 c9"><span></span></p>
<p>Please refer to the <a href="{{ site.baseurl }}/index.html">contribution guidelines</a> if you have comments or suggestions.</p>
<p class="c2 c9"><span></span></p>
<p>Materials referenced in this document can be found in <a href="{{ site.baseurl }}/resources.html">References and Bibliography</a></p>
<p class="c2 c9"><span></span></p>
<p>Licensed under: <a rel="license" href="http://creativecommons.org/licenses/by/3.0/us/">CC-BY 3.0 U.S.</a></p>
<p class="c2 c9"><span></span></p>
<hr />
<p class="c2 c9"><span></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.vljbal76mq9u">Motivation</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.kzncxgycnjwc">Background</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.t69p8mskjiy9">Developing Software Guidelines: A Community-driven Process</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.ktnrb9wswuob">What Is Research Code?</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.6bi4jecr96n">Community and Stakeholders</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.rtkadvvyvaul">Research Software Projects</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.djec5033ho1u">Scenarios</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.x1l28fc1ejv">Development Process and Project Maturity</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.girxi3be7fh7">Systems</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.3qhixgp3adkd">Goals of this Document</a></span></p><p class="c2 c23"><span class="c1"><a class="c4" href="#h.jto1fyx4wiqs">Guidelines</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.anjaj5gfn1k">Principles</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.jc42b2hcl6wm">Sustainable Code</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.w7q3addxbtzy">Clean, standardized code</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.pt7zrmqte23n">The codebase is well-structured and consistently structured.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.tekrxe5wegtn">Code follows the language’s recommended style guide or the research group’s style guide.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.1zkmnhg3e5oe">Conformance to the style is verified using a linter.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.xacmpjv4c3t">Linting is included during automated testing.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.n0j2lapycvbr">Errors and exception handling follows recommends practices for the language used.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.g8dldwawc4gi">Versioned code</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.2tdya1yrzmc6">Source code is maintained in a version control system (VCS).</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.6c0j1lpk4dir">Source code uses a versioning scheme.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.huu50eug1a5f">Source code uses tagged releases.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.tlmgsr2d8yip">Redistributable</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.dxcrxyr9q443">Source code includes build scripts to automate builds.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.v88x0wfkpugl">Scripts or configurations are provided for creating binary installers.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.hi3h4uzdhzde">For desktop GUIs, provide an installer.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.hlj4s6z7m6fe">Tested</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.r7veqzthne4v">Source code includes unit and integration tests.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.qfvkud70t782">Tests limit dependence on external services where possible.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.vekc78g7nqxx">Testing is automated through a continuous integration system.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.66jeksgqtmig">For web applications, include automated GUI testing.</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.k8wt6ghlvp2k">Interoperable</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.jdkbivybsbw3">For data services</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.jwbj4ujf85sn">Applications, desktop or web-based, supports community data standards for inputs and outputs.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.9c8tsx28txfx">For Semantic Services</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.j3o37luj1lmk">Usable</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.jf59v21hi69u">Software has a clear, understandable interface.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.6foli2yv92b3">For software with a visual interface, such as a desktop GUI, plugin/extension or web/mobile application, the interface provides a clean and clear layout.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.tvmzy7swlrrs">For web applications or any software accepting user inputs, Unicode is well-supported throughout the system.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.qxz6nnr14i6e">For web applications, the application implements responsive design, i.e. support for multiple screen sizes.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.ga5mu41sst4w">For web applications, the application adheres to progressive enhancement practices.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.42fc5mhfrgk0">For web applications, the interface follows established guidelines for web accessibility.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.fdfrt2xf1lu2">For web applications, support current browser versions.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.vft1b61bp3vj">For a web service, the API is RESTful or follows a known community standard or specification.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.1gxhk9ppul8o">For collaboration tools, ensure that the interface design does not encourage sensitive information leaks.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.7mb3s9l49d19">Software is performant and stable.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.nwh0jy5c06kn">Documentation describes the current development status.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.mpxwzayw9bzb">For web applications or web services, the documentation or project website includes service level qualities.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.1z1daqwsh16q">For web applications and web services, systems are monitored.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.l9dc6388w79f">For high performance computing, GPU or other cluster-based software, the project provides basic benchmarking statistics.</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.278bwz69j1fc">Documented</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.qlvssx94m25r">Source code is documented.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.muhqkex415r9">Source code comments use a standard comment style, related to the selected style guide or language or related to a document generation tool.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.ycrdao5dntt5">API documentation is automatically generated.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.4nph8cwunjf5">The documentation describes how to automatically generate the documentation.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.tq0ajo5926em">API Documentation is versioned.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.gjmwihq5i3uk">The codebase includes documentation to support adoption and reuse.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.eyumqunuhqwt">The source code repository is documented using plain text file formats.</a></span></p>
<p class="c2 c18"><span class="c1"><a class="c4" href="#h.jo1cauupk7qk">The build procedure or installation process is described.</a></span></p>
<p class="c2 c18"><span class="c1"><a class="c4" href="#h.nkh4hefff5qr">The documentation describes how to run simple examples.</a></span></p>
<p class="c2 c18"><span class="c1"><a class="c4" href="#h.4pnm2fq7ygzv">The documentation describes how to execute the test suite.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.6qp4s0y7ha51">The versioning scheme is described.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.csa26k8fwk2r">Release notes or changelogs are provided for each release.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.90q5gnajhkbj">Software is documented.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.5wf9rcv7ts53">Any Documentation</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.m7qbd04yq214">User documentation</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.6kfwca5ceoo6">Code or software requirements are documented.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.gqpohaprkcbq">Project is documented.</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.kevi2ogx4kct">Secure</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.c0ntdoo6m887">Source code, including automation configurations and scripts, have been sanitized for public release.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.bwddr3yneyvr">For web applications and services, software follows industry recommendations for secure practices.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.2qasdya4g2dq">For any code or software system, follow the recommended security practices for each system component.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.v4eyawiuk7gf">Keep systems and dependencies up-to-date.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.wcbbon9uy2j6">Collaborative platforms include tests for permissions and integrations.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.ltbzsn7tud66">Containers follow recommended practices for security.</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.j3cd3qemat79">Sharable</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.hhtvml7dyjhk">Source code is licensed.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.561dz5rrclj0">Source code includes configurations for automated systems.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.oofsvittp0hd">Project name is discoverable.</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.1ogf2zcgktnr">Governed</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.qderu9pnkzxc">Contribution policies are provided.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.j2u8c0tf34q2">For any project, describe the workflow used by the project for contributions to the source code (or other versioned items).</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.ns2tcmg58ars">For any project, describe commit message, pull request, and issue preferences.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.p0kh0cpf78eq">For any project, define the code review process.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.ffvx3h4l2hb7">For open source projects, provide a code of conduct or state clear expectations of behavior and communication.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.oy3ktoh4p9gb">For open source projects, provide guidelines about what contributions will or will not be accepted.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.b4mxlh97r1ja">Development activities are transparent.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.foxopgq1esv4">Development activities are managed through an issue tracker or similar software.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.l1vbrrmurts7">Project provides support mechanism(s).</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.tux2f6u3qd4a">Project provides a development roadmap</a></span></p><p class="c2 c15"><span class="c1"><a class="c4" href="#h.nag33fat6sc8">Code as Research Products</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.tk3e9fopeigk">Publication and Citation</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.llz3mjenwmpr">Software, as binaries and source code, are published to a sustainable third-party repository.</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.42ni1tsctkct">Documentation includes citation details.</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.pzfz7hkzxqus">Preservation/Archiving</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.rdevd43xfrh4">Credit</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.26zgiufdtz5">Provenance</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.zgp24m9oygk7">Reproducibility/Replicability</a></span></p><p class="c2 c23"><span class="c1"><a class="c4" href="#h.g3e5ybizjqjh">Progression, Sustainability and Reusability/Adoption</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.b2e39m7jtud5">Progression</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.ekrcac4t6s00">Sustainability</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.ytbqx2fpn94q">Sustainability based on potential</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.7c0vnqsr803n">Sustainability of broadly applicable cyberinfrastructure</a></span></p><p class="c2 c18"><span class="c1"><a class="c4" href="#h.yd76uhfdnc1j">Sustainability of research development</a></span></p><p class="c2 c25"><span class="c1"><a class="c4" href="#h.hftwhr372i88">Adoption and Reuse</a></span></p><p class="c2 c23"><span class="c1"><a class="c4" href="#h.gpan0aidj5ex">Using this guidance for assessment</a></span></p><p class="c2 c23"><span class="c1"><a class="c4" href="#h.aep6uf41nqbc">Conclusions</a></span></p><p class="c2 c23"><span class="c1"><a class="c4" href="#h.qyi0cpttkurs">Contributors</a></span></p><p class="c2 c23"><span class="c1"><a class="c4" href="#h.iy5k73rbewd1">Acknowledgements</a></span></p><p class="c2 c9"><span></span></p>
<p class="c2"><span class="c36">Code, software, and other such products are often developed in the course of publicly-funded research. “Code-wise,” there is nothing special about these research products in themselves; however, unlike commercially-derived code and software, research products bear the additional burdens of reproducibility, publication, and preservation, because of the nature of the funding that produced them.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The Federation for Earth Science Information Partners (ESIP) recognizes the role code plays in contemporary research practices as well as the complex issues community members face when approaching technology-dependent research. With the cross-disciplinary nature of the ESIP membership, we feel well positioned to develop guidance and recommended practices for the Earth observation and geosciences communities in a way that recognizes the domain expertise within these communities and that supports the communities’ desire for improved practices around research code and software. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Current discussions around code and software practices in the broader research community often do not include representations from either the Earth observation or geosciences communities. And while we agree that code is code in any domain, public or private, we believe that it is important that these discussions include a broad range of researchers, including faculty, research staff, developers, graduate students and postdoctoral fellows. Such inclusion is necessary to ensure that the practices evolving from these discussions have the broadest possible usage and application. Our goal is to develop meaningful and practical assessment procedures that address the concerns of all researchers, and to develop training and support networks for early career and future researchers.</span></p><h2 class="c2 c8" id="h.vljbal76mq9u"><span>Motivation</span></h2><p class="c2"><span>Our motivation for developing these guidelines is twofold. First, we recognize, both within our research communities and within funding agencies, that code is vital to many of our research activities. It is, therefore, crucial to support researchers, at all levels, in developing high quality code to meet the needs of these activities, and to continue to do so as capabilities and requirements grow in complexity and scale. We also have the additional burden to understand and support our research community practices. For researchers and research groups, integrating these guidelines can save time, improve trust in the code product and its outputs, and reduce the amount of effort needed to meet community expectations for publication and reproducibility, among other benefits.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Second</span><span>, with the increased focus on return on investment for grant-funded cyberinfrastructure and, perhaps more crucially, a new focus by funding agencies on “time to science” (where some cyberinfrastructure, or software product, reduces the time or effort needed to address the primary research goal), we see renewed interest in evaluating the adoption and impact of cyberinfrastructure solutions. We stress the need to develop evaluation processes in ways that reflect current code outputs and that encourage adoption of development and management practices to support improved research outcomes and impact.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>This document will provide a broad overview of current development practices and provide guidelines for improving those practices. We will then discuss the relationship of these guidelines to the larger research development ecosystem as well as their limitations. We hope this discussion leads to equitable evaluation practices, and, more critically, to new approaches to supporting high quality, sustainable research development.</span></p>
<h2 class="c2 c8" id="h.kzncxgycnjwc"><span>Background</span></h2><p class="c2 c9"><span></span></p><p class="c2"><span>This work uses </span><span>the</span><span> criteria for research code developed by the </span><span class="c1"><a class="c4" href="http://software.ac.uk/sites/default/files/SSI-SoftwareEvaluationCriteria.pdf">Software Sustainability Institute</a></span><span> (SSI, Jackson et al. 2011) and extends their use through a larger evaluation framework developed by ESIP in collaboration with NASA AIST to verify </span><span class="c1"><a class="c4" href="https://www.nasa.gov/directorates/heo/scan/engineering/technology/txt_accordion1.html">Technology Readiness Levels</a></span><span> (TRL, NASA 2012). The evaluation framework was used in a </span><span class="c1"><a class="c4" href="http://testbed.esipfed.org/sites/default/files/2015_AIST_Evaluations_Overview.pdf">pilot effort</a></span><span> (Burgess 2016, Graybeal 2016) over the course of autumn 2015. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>This effort results from both community discussions around these criteria and from ESIP’s larger interest in technology evaluation as a mentoring activity rather than as a verification activity. The guidelines outlined in this document will serve as the foundation for future assessment and training implementations.</span></p><h2 class="c2 c8" id="h.t69p8mskjiy9"><span>Developing Software Guidelines: A Community-driven Process</span></h2><p class="c2"><span>We began developing these guidelines by adapting the SSI’s criteria so that they would better meet the diverse needs of the interdisciplinary ESIP community. During this process, we made a conscious effort to solicit input from active developers. Our aim was to have roughly even representation of developers on one hand and researchers, managers and/or principal investigators on the other. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Formal revision of the initial draft was undertaken through three different activities. The first involved a remote sprint where ESIP members volunteered to review different sections of the original criteria based on their domain expertise and interest. The remote sprint ended with a half day workshop during the ESIP Summer Meeting, July 2016. The second activity involved community discussions with members of the Boulder Earth and Space Science Informatics Group (BESSIG). The last activity was the Joint EarthCube and ESIP Workshop on Software Assessment (Boulder, CO; June 2016). Two achievements that resulted from this workshop were, first, that the existing guidelines were revised and new guidelines for interoperability were established, and second, that EarthCube and ESIP's roles were established for addressing the larger EO and geoscience needs for improving both research code and software practices.</span></p><h2 class="c2 c8" id="h.ktnrb9wswuob"><span>What Is Research Code?</span></h2><p class="c2"><span>For our purposes, research code and research software are any code outputs developed through grant-funded activities. This understanding is broader and more inclusive than that found in more traditional “science code” definitions, and it is a necessary expansion for supporting the current needs of our research communities. Research code and research software are critical to many contemporary research activities (Hettrick et al. 2014). </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Here, we also make a distinction between research code and research software. Research code can be scripts, data processing or analysis workflows, or other codes developed to generate data, perform analyses, or perform other project-specific tasks leading to </span><span>artifacts</span><span> published through journal articles or otherwise. This kind of code is usually not intended for community adoption through open sourcing or public release, nor is it included as an explicit deliverable on a grant. The potential for long term support for these codes is low, but these codes are nevertheless important to the community for provenance and reproducibility, replicability and publication.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For its part, research software is any code product, whether as scripts or as some packaged object, that is intended to be released for community adoption and use. Intention of release is not a complete description of what we mean by research software as we also include intentional design or systems understanding for a defined purpose. This software is most likely an explicit deliverable for a grant. These outputs are valuable for provenance and other community practices, and have the additional potential of longer term support through both ongoing maintenance by the research group or other contributors and through continued funding. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The distinctions here are related to the purpose of the release and the prospect for ongoing maintenance. In practice, these factors usually do not affect the guidance provided, and for most of this text, the terms research code and research software will be simplified to “codes”. However, there are certain circumstances in which the differences matter, and these situations will be noted.</span></p><h2 class="c2 c8" id="h.6bi4jecr96n"><span>Community and Stakeholders</span></h2><p class="c2"><span>Our focus on assessment for verification and guided educational activities encompasses a wide range of stakeholders. Table 1 outlines the main stakeholder communities and related use cases.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Table 1. Stakeholder use cases. Note that different stakeholders may have overlap in desired outcomes. </span></p><a id="t.dcb2d0b75dcabd32bed21a1ff284ced36b434167"></a><a id="t.0"></a><table class="c38"><tbody><tr class="c11"><td class="c6" colspan="1" rowspan="1"><p class="c2 c31"><span class="c39 c37 c22">Stakeholder</span></p></td><td class="c5" colspan="1" rowspan="1"><p class="c31 c2"><span class="c37 c22 c39">Use Case</span></p></td><td class="c30" colspan="1" rowspan="1"><p class="c31 c2"><span class="c39 c37 c22">Desired Outcome</span></p></td></tr><tr class="c11"><td class="c6" colspan="1" rowspan="1"><p class="c19 c2"><span class="c13">Funder</span></p></td><td class="c5" colspan="1" rowspan="1"><p class="c19"><span class="c13">As a funding agency, we're interested in evaluating the software projects we fund.</span></p></td><td class="c30" colspan="1" rowspan="1"><p class="c19"><span class="c13">A functional evaluation system based on accepted metrics.</span></p></td></tr><tr class="c11"><td class="c6" colspan="1" rowspan="1"><p class="c19"><span class="c13">Project Manager, Principal Investigator (manager in practice)</span></p></td><td class="c5" colspan="1" rowspan="1"><p class="c19"><span class="c13">As a manager, I’m interested in using the rubric/progression as a learning tool to help improve the development practices in my research group.</span></p></td><td class="c30" colspan="1" rowspan="1"><p class="c19"><span class="c13">A checklist or other informal assessment to help the research group meet funder's expectations and to determine the next steps for training or related activities in the research group.</span></p></td></tr><tr class="c11"><td class="c6" colspan="1" rowspan="1"><p class="c19"><span class="c13">Principal Investigator</span></p></td><td class="c5" colspan="1" rowspan="1"><p class="c19"><span class="c13">As a PI, I would like a tool to assess our progress and to ensure we’re meeting our funder’s expectations for a software project based on the readiness level stated in the original proposal and as defined by the funder.</span></p></td><td class="c30" colspan="1" rowspan="1"><p class="c19"><span class="c13">A checklist or other informal assessment to help the research group meet funder's expectations, and to determine the next steps for training or related activities in the research group. This informal assessment would also provide aid for formal reviews. </span></p></td></tr><tr class="c11"><td class="c6" colspan="1" rowspan="1"><p class="c19"><span class="c13">Science Software Developer, Researcher who codes</span></p></td><td class="c5" colspan="1" rowspan="1"><p class="c19"><span class="c13">As a science software developer, I’m interested in using the recommended practices to improve my own workflow and skillsets.</span></p></td><td class="c30" colspan="1" rowspan="1"><p class="c19"><span class="c13">A checklist or mentoring activity to help guide me towards training options to meet my research and skillset goals.</span></p></td></tr><tr class="c11"><td class="c6" colspan="1" rowspan="1"><p class="c19"><span class="c13">Developer</span></p></td><td class="c5" colspan="1" rowspan="1"><p class="c19"><span class="c13">As a developer, I would like community-supported guidelines to support requests to change our current dev team practices.</span></p></td><td class="c30" colspan="1" rowspan="1"><p class="c19"><span class="c13">A checklist or informal assessment to encourage my manager or PI to allow the development team to adopt appropriate practices.</span></p></td></tr><tr class="c11"><td class="c6" colspan="1" rowspan="1"><p class="c19"><span class="c13">Grad Student, Post-Doc, Researcher interested in continuing code education</span></p></td><td class="c5" colspan="1" rowspan="1"><p class="c19"><span class="c13">I’ve taken the introductory courses and want to continue to improve my skills but don’t know what steps to take next, and I’d like guidance based on my skillset.</span></p></td><td class="c30" colspan="1" rowspan="1"><p class="c19"><span class="c13">A checklist or mentoring activity to help guide me towards training options to meet my research and skillset goals.</span></p></td></tr><tr class="c11"><td class="c6" colspan="1" rowspan="1"><p class="c19"><span class="c13">Research Community</span></p></td><td class="c5" colspan="1" rowspan="1"><p class="c19"><span class="c13">We want to provide educational materials or other support for community members to meet their goals regarding research software implementation and career growth.</span></p></td><td class="c30" colspan="1" rowspan="1"><p class="c19"><span class="c13">A set of guidelines for technology assessment, and the framework for using those guidelines as educational tools.</span></p></td></tr></tbody></table><p class="c2 c9"><span></span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We must understand that research software projects are undertaken by a variety of research groups that have varying degrees of resources available. These groups range from well-supported institutions with dedicated developer teams to individual PIs with only part-time developer support, or with small teams of graduate students with varying levels of experience in software development. With that understanding in mind, we have structured this guidance to demonstrate worthwhile and concrete actions that can be taken to improve development practices within a given research group’s current resource limitations. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Over time, some individuals will take on different stakeholder roles at different stages of their academic careers, and their educational goals will take on a different slant. They may find, for instance, that the skills that were adequate to manage code at the graduate student or post-doc level are no longer adequate when they become principal investigators of more complex technical projects. Likewise, in transitioning to a program officer role at a funding agency, one may find that one’s technical experience is not adequate for developing functional and realistic metrics for project success. While these issues are not the explicit focus of this document, we hope that the discussions here bring some awareness to these different needs.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Use cases are more fully explained in the </span><span class="c1"><a class="c4" href="#h.djec5033ho1u">Scenarios</a></span><span> section below to provide more concrete examples and to address the assessment implications for maturity and for education in real world situations.</span></p>
<h2 class="c2 c8" id="h.rtkadvvyvaul"><span>Research Software Projects</span></h2><h4 class="c2 c8" id="h.djec5033ho1u"><span>Scenarios</span></h4><p class="c2"><span>In this section, we will describe a number of common scenarios where research produces some code or software, and where the expectations for assessment, development process and project maturity, researcher education, and code sustainability will play out differently. </span></p><p class="c2 c9"><span></span></p>
<p class="c2"><span class="c22">Scenario 1.</span><span> An individual developing data processing scripts. </span></p><p class="c2 c15"><span>I am a grad student or post-doc without any computer science or software development training and I need to develop some data processing scripts for my analysis. I know these scripts will be made public so I’d like try to follow good code practices but don’t know where to start.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Assessment: The developed scripts are secondary research products where the desired products are the data and analyses. The code is unlikely to undergo formal review but is expected to meet basic code standards and publication requirements.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Maturity: Few expectations for high levels of process maturity although this is dependent on the nature of the scripting. However, since it has data production or analysis as its primary goal, correctness, and demonstrations of that correctness, are expected. </span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Education: Good code practices for clean code and tested code. Accepted publication practices for the community. </span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Sustainability: No ongoing support from the grad student, post-doc or research group is expected if the code is published to an external archive. </span></p><p class="c2 c9"><span></span></p>
<p class="c2"><span class="c22">Scenario 2.</span><span> A new </span><span>system</span><span> based on a single grant.</span></p><p class="c2 c15"><span>Our research group is starting a new project to develop an analysis platform as a web application. It will use a well-known JavaScript framework and will be developed as open source. We plan on developing our community to support active contributions and to support the application’s adoption in areas where the analysis is useful. </span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Assessment: Verification that the new system meets the research goals and meets expectations to support reuse and adoption. </span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Maturity: Processes must be mature enough to allow external developers to set up a development environment and contribute features or patches. The group considers it beta software so stability is not expected.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Education: Guidance on developing and supporting community contributions. Identification of next steps for an operational system.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Sustainability: Demonstrations of community activity and successful engagement to show adoption potential. Could consider outreach for early adopters in a specific research community. </span></p><p class="c2 c9"><span></span></p>
<p class="c2"><span class="c22">Scenario 3.</span><span> An existing system with a single grant to add new features.</span></p><p class="c2 c15"><span>Our research group has an existing project, a geospatial processing library. Our grant includes development time for several new features that we’ll release as a new major version.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Assessment: An initial assessment of the existing system is necessary to establish a baseline maturity level before evaluating the current grant work. Assess the new features against this baseline with the expectation that the implementation meets the baseline maturity and raises concerns if lower. Include evaluation of the new features against the stated research goals. </span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Maturity: As noted, the maturity of the new features are expected to be at or above the level of the existing system. Process improvements may also occur during the grant cycle.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Education: Identification of next steps for an operational system.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Sustainability: We could assume this is managed as an open source project where improvements are made with a research goal in mind. Steps towards gaining contributions of funding or effort from other sources have been undertaken.</span></p><p class="c2 c9"><span></span></p>
<p class="c2"><span class="c22">Scenario 4.</span><span> An existing system with multiple grants, one of which adds a new standalone component.</span></p><p class="c2 c15"><span>We have an existing platform for data access that is currently receiving support from multiple grants. One of the grants is funding an extension to the platform that will be developed as a standalone application (middleware). Our platform is based on an open source project developed by a different group not supported by this funding (we are reusing the OS system as a base for our own efforts).</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Assessment: Evaluate the standalone component for completeness and functionality at the maturity levels laid out in the grant. The integration of the standalone component with the existing system may also be evaluated.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Maturity: Started at known levels at the start of the grant with no expectations of improvement. </span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Education: Project manager would like information on the assessment criteria based on the stated project goals. </span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Sustainability: Project has reached a stable state, continues to look for adoption at other institutions and funding opportunities.</span></p><p class="c2 c9"><span></span></p>
<p class="c2"><span class="c22">Scenario 5. </span><span>An existing system with multiple grants, one of which supports a new data product.</span></p><p class="c2 c15"><span>We currently maintain a data portal with legacy code. In our new grant, we are collaborating with other groups, but we are not responsible for the main development tasks. Part of our requirements are to add new data products in support of the larger collaboration. We need to write processing code for those products; the code won’t be integrated into our legacy platform and won’t be needed long-term.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Assessment: Evaluation of the larger collaborative project where the successful generation of the data product is relevant to demonstrate interoperability/integration, but the code itself is not evaluated. </span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Maturity: Has reached some stable maturity levels, but the state of the legacy platform is not relevant to the assessment.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Education: Research group would like guidance on automation practices to better support data pipelines in the future.</span></p><p class="c2 c15 c9"><span></span></p><p class="c2 c15"><span>Sustainability: None. Code will be published as an archive for provenance.</span></p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span class="c22">Scenario 6. </span><span>A single grant with multiple collaborators providing outputs/functionality through their independent systems.</span></p>
<p class="c2 c15"><span>We are demonstrating a new analysis platform that relies on distributed data sources. One collaboration group is developing the new analysis platform, one is implementing new features in an existing platform and the third developing a data pipeline process. The success of the project depends on the integration of the three systems and the groups are not contributing to one codebase.
</span></p>
<p class="c2 c15 c9"><span></span></p>
<p class="c2 c15"><span>Assessment: Assessment at the project level, focusing on the successful integration of the three activities. If the analysis platform is intended for potential reuse, additional assessment may be considered for that component only.
</span></p>
<p class="c2 c15 c9"><span></span></p>
<p class="c2 c15"><span>Maturity: The new development, seen in the analysis platform, is expected to meet maturity expectations for a prototype web application while the two data providing systems should work within the maturity level of the existing systems.</span></p>
<p class="c2 c15 c9"><span></span></p>
<p class="c2 c15"><span>Education: The collaboration groups providing data may want information on the interoperable service implementations. The project collaborators are interested in understanding the assessment process.</span></p>
<p class="c2 c15 c9"><span></span></p>
<p class="c2 c15"><span>Sustainability: Sustainability is likely relevant only to the analysis platform as the component intended for reuse/adoption, certainly from a funder’s perspective. The two data systems ideally would continue to support the new functionality, as early adopters of the system or to support other downstream activities.</span></p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span class="c22">Scenario 7. </span><span>An existing system with a grant specifically to provide maintenance.
</span></p>
<p class="c2 c15"><span>We are the maintainers of a successful visualization tool that has reached a point in its lifecycle where major refactoring is required to continue to support future activities. The grant doesn’t include any new functionality, but may include upgrades to the language version, bug fixes and refactoring to manage technical debt.
</span></p>
<p class="c2 c15 c9"><span></span></p>
<p class="c2 c15"><span>Assessment: Assessment requirements unlikely unless revisions are extensive.</span></p>
<p class="c2 c15 c9"><span></span></p>
<p class="c2 c15"><span>Maturity: Expected to remain at the maturity level at the start of the maintenance effort or improve based on the stated goals.</span></p>
<p class="c2 c15 c9"><span></span></p>
<p class="c2 c15"><span>Education: No explicit education requirements. </span></p>
<p class="c2 c15 c9"><span></span></p>
<p class="c2 c15"><span>Sustainability: Project is expected to continue community engagement and support activities (assuming major upgrade) and continue pursuing additional funding, whether through additional maintenance grants or for extending the functionality.
</span></p>
<p class="c2 c9"><span></span></p><p class="c2"><span>Hopefully these scenarios</span><span>, while clearly not a comprehensive list of all possibilities,</span><span> provide some insight into how different project types and phases might be assessed. </span></p><h4 class="c2 c8" id="h.x1l28fc1ejv"><span>Development Process and Project </span><span>Maturity</span></h4><p class="c2"><span>Development process maturity and project maturity, identified either through longevity or by some other defined criteria, are not linear, nor do they necessarily coincide. A project can start with a high level of development process maturity in an initial proof of concept phase. For its part, project maturity, often presented as governance patterns similar to traditional open source projects, can begin the same way. Improvements in both can occur within the lifespan of a single grant, or they can happen through a series of grants. Here we can see the potential mismatch between the way that funding agencies often assess and categorize a project and the way that software develops in reality.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Additionally, a mismatch between assessment and maturity can occur when, for example, a codebase does not meet the desired level of code maturity or project maturity, but is nevertheless successfully serving a communal need. Maturity markers are indicators of project health, and we strongly encourage their use, but not to the detriment of the larger goals of providing a product that meets a research community’s needs.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Within the diversity of research software projects, </span><span>two overarching categories</span><span> can be distinguished that bear on the assessment of maturity: first, code/software intentionally developed for reusability and adoption, and second, code/software developed for highly specific project needs. Neither category can neglect the recent requirements for openness, notably the U.S. Office of Science and Technology Office’s policies on Open Science (Holdren, 2013) and the increasing number of academic journals requiring data and code publication; however, the methods used for achieving those requirements, and expectations around development process or project maturity, are different in each category. For projects developing code or software not intended for reuse and adoption, the focus for assessment is on development process maturity, documentation and preservation or publication. (We should note that “not intended” reflects stated project milestones, sustainability and community engagement activities as per the funded proposal.) For projects developing code or software that is intended for adoption, the focus for assessment is on project maturity related to governance and documentation as well as development process maturity.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We are focused here on code/software developed by a research group. For the purposes of assessment, we exclude from discussion of maturity any software products reused by a research group,while noting the value in assessing project maturity. More specifically, we support, and encourage, a research group’s decision to reuse a current, outcome-appropriate technology over reimplementing a similar technology. We do not, however, consider the reused technology to be an assessment target itself. A simple example is developing modules for an existing web publication framework instead of developing an entirely new framework.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>On a related issue, in cases where a project employs an existing technology to support the publication or release of a research product that is not itself code or software, the pre-existing technology again would not be considered during the assessment of the research software or code. A very broad example of this would be developing datasets and providing them publicly through ESRI’s ArcOnline platform. In this case, a project assessment would consider the datasets, but not ArcOnline. (Guidance for quality and maturity for other research product types, such as datasets or ontologies, is beyond the scope of this document.)</span></p><p class="c2 c9"><span></span></p><p class="c2"><span class="c1"><a class="c4" href="https://www.usability.gov/what-and-why/web-analytics.html">Analytics</a></span><span> and </span><span class="c1"><a class="c4" href="http://altmetrics.org/manifesto/">altmetrics</a></span><span> are included here to highlight two aspects of analytics for assessment. First, projects at different stages of their lifecycles are characterized by different outcomes, and those outcomes may not occur within a specific funding period. Second, our guidance focuses on those activities that a research group can usefully act upon to meet its own maturity goals. The success of these activities may, at least in part, be demonstrated through analytics and altmetrics. Understanding how those metrics relate to funding cycle timelines is necessary before including them in an assessment process focused on activities within a cycle. We suggest caution, then, in interpreting what is being measured, and in being aware of when in the cycle it is being measured. </span></p><h4 class="c2 c8" id="h.girxi3be7fh7"><span>Systems</span></h4><p class="c2"><span>As exemplified by the scenarios, when we talk about maturity and/or assessment, we are often discussing existing systems maintained by a research group or a larger collaboration. From the funder’s perspective, a formal evaluation of existing products is not cost effective; however, there are instances when assessment requires an understanding of the existing system’s baseline maturities. We discuss this requirement further in </span><span class="c1"><a class="c4" href="#h.b2e39m7jtud5">Progression</a></span><span class="c22">,</span><span> where the context and research goals of the grant play a role in determining the assessment need. Regardless of assessment requirements, understanding the current state of a project is necessary if a research group wants to evaluate whether process changes are leading to improvements or to regressions. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We are not starting with a tabula rasa for research cyberinfrastructure in general or when beginning new grant-funded activities. Nor do we often start cyberinfrastructure projects without taking advantage of existing platforms, frameworks or libraries. There is a need to assess the selection of these dependencies as they relate to the overall project goals, even if we are not evaluating the maturity of the dependencies themselves.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>One final note regarding these systems: technology changes rapidly. Our cyberinfrastructure and research development challenges often require complex solutions, and guiding system architectures is beyond the scope of this document. There is often no single solution for many of our research activities. It is the responsibility of the researcher or research group to investigate current technologies that may address the specific project needs and select those technologies accordingly. Researchers must consider the project requirements, previous efforts, and the pros and cons of available platforms and solutions. </span></p><h2 class="c2 c8" id="h.3qhixgp3adkd"><span>Goals of this Document</span></h2><ol class="c14 lst-kix_shfd0z5borlh-0 start" start="1"><li class="c2 c3"><span>Provide resources and guidance to different kinds of research groups and other stakeholders to support research code/software development within our research communities.</span></li><li class="c2 c3"><span>Provide a progression model and related discussion around sustainability, adoption and reuse, and assessment that reflects the variation in research development products. </span></li></ol><h1 class="c2 c8" id="h.jto1fyx4wiqs"><span>Guidelines</span></h1><p class="c2"><span>Research code, in itself, is not special as code; however, as a product of research funding, it comes with the burdens of reproducibility, publication and preservation.</span><span> The guidelines presented here are intended to help researchers solve problems they encounter while addressing these requirements. We will structure our discussion around the reasons for the requirements, and wherever possible, we will refer to current industry and/or research community practices and recommendations. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Some aspects of development process or project maturity may not relate to every type of software or code activity. Table 2 describes the software types we considered when developing these guidelines. Where necessary, additional guidance is provided for a specific type if the general guidance neglects some aspect important for that software type. </span></p><p class="c2 c9"><span></span></p>
<p class="c2"><span>Table 2. Code and Software Categories.</span></p><a id="t.f8e98be5ca560e426b4fec126df0d72d056fe59a"></a><a id="t.1"></a><table class="c38"><tbody><tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c32 c22">Code/Software Type</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c22 c32">Description</span></p></td></tr><tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Processing scripts</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Code developed for one-off processing or other project-specific activities, not generally intended for adoption and reuse.</span></p></td></tr><tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c37">Notebooks</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Hybrid documents containing code, documentation, and analyses.</span></p></td></tr><tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Module/library/command line interface (CLI)</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Packaged or buildable codebase, usually serving a specific purpose, often intended for adoption and reuse.</span></p></td></tr><tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Plugin/extension</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Packaged or installable software provided as a component of another piece of software.</span></p></td></tr><tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Desktop Application (GUI)</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Installable software developed for use on a desktop with a visual interface.</span></p></td></tr><tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Web Application</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Software deployed to a web server with functionality provided through a visual interface.</span></p></td></tr><tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Web Service</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Software deployed to a web server, providing data access or other functionality through a Web API.</span></p></td></tr>
<tr class="c11"><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Platforms/Systems Software*</span></p></td><td class="c17" colspan="1" rowspan="1"><p class="c12"><span class="c0">Operating systems, utilities, servers, database engines or other systems developed to serve or support application software.</span></p></td></tr>
</tbody></table>
<p class="c2 c9"><span></span></p>
<p>* Programming languages, such as <span class="c1"><a class="c4" href="http://julialang.org/">Julia</a></span> or <span class="c1"><a class="c4" href="https://www.r-project.org/">R</a></span>, are outside the scope of this document; however, the guidelines provided here support development activities within the specific language ecosystems. </p>
<p class="c2 c9"><span></span></p><p class="c2"><span>The guidance provided below strives to be language- and platform-agnostic. Wherever possible, the examples provided are not an endorsement of a language or product and are intended solely as examples of the larger concept. When choices must be made by a research group on how to implement a concept, we suggest that in almost all cases there are many acceptable paths. Choose one, support it and move on to the project implementation. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Code notebooks, supported in platforms like <a class="c4" href="http://jupyter.org/">Jupyter</a>, <a class="c4" href="http://rmarkdown.rstudio.com/r_notebooks.html">R Notebook</a>, <a class="c4" href="http://zeppelin.apache.org/">Apache Zeppelin</a>, are hybrid products containing code, documentation and analysis outputs. They can serve as processing scripts and workflows for generating data and analyses or as documentation and tutorials for a code or software project. When approaching the guidelines below, consider the different elements within a notebook, as code or as documentation, when considering the guidelines.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Finally, while we strongly support open science and the principles of accessibility, reproducibility and preservation of research software and code, we acknowledge that there are resource limitations, differing levels of experience and differing needs, both within and among communities. Choice of language, choice of platform, choice of tools—these are driven by the specific research and community needs rather than any dogmatic adherence to a particular philosophy. </span></p>
<h2 class="c2 c8" id="h.anjaj5gfn1k"><span>Principles</span></h2><p class="c2"><span>The Guidelines section is organized into eight principles to define the characteristics we support for research code and software activities, regardless of the specific products.</span></p><p class="c2 c9"><span></span></p>
<p class="c2 c15"><span class="c22">Sustainable Code:</span> The code itself is well-structured and readable and is developed to support understandability and maintenance over time.</p>
<p class="c2 c15"><span class="c22">Interoperable:</span> The code or software uses, generates or serves data using recognized community standards or formal standards.</p>
<p class="c2 c15"><span class="c22">Usable:</span> The software is developed using recommended practices for the given software type to create functional, performant and stable interfaces in order to promote adoption and improve research practices.</p>
<p class="c2 c15"><span class="c22">Documented:</span> The project includes documentation to support understanding at different levels throughout (code, reuse and adoption, project description).</p>
<p class="c2 c15"><span class="c22">Secure:</span> The code/software is developed using recommended practices for securing deployments, systems and user interactions.</p>
<p class="c2 c15"><span class="c22">Sharable:</span> Project incorporates practices to ensure the code/software products are open to reuse and adoption for future research activities as well as for preservation.</p>
<p class="c2 c15"><span class="c22">Governed:</span> The project uses established practices to support community engagement, growth and transparency to encourage adoption and reuse, especially for those projects released as open source.</p>
<p class="c2 c15"><span class="c22">Code as Research Products:</span> The project supports community practices to encourage and support the practice of research, including citation, preservation, provenance and reproducibility/replicability.</p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span>From these principles and our understanding of the range of research product development activities, we can provide a core model to reflect the minimum set of criteria for any research development product to meet:
</span></p><ol class="c14 lst-kix_xcfwdxntv4m-0 start" start="1"><li class="c2 c3"><span>Sustainable Code</span></li></ol><ol class="c14 lst-kix_xcfwdxntv4m-1 start" start="1"><li class="c2 c18 c29"><span>Clean, standardized code</span></li><li class="c2 c18 c29"><span>Versioned code</span></li><li class="c2 c18 c29"><span>Tested* (strongly recommended)</span></li></ol><ol class="c14 lst-kix_xcfwdxntv4m-0" start="2"><li class="c2 c3"><span>Documented</span></li></ol><ol class="c14 lst-kix_xcfwdxntv4m-1 start" start="1"><li class="c2 c18 c29"><span>Source code is documented.</span></li><li class="c2 c18 c29"><span>The codebase includes documentation to support adoption and reuse.</span></li></ol><ol class="c14 lst-kix_xcfwdxntv4m-0" start="3"><li class="c2 c3"><span>Sharable</span></li></ol><ol class="c14 lst-kix_xcfwdxntv4m-1 start" start="1"><li class="c2 c18 c29"><span>Code is licensed.</span></li></ol><ol class="c14 lst-kix_xcfwdxntv4m-0" start="4"><li class="c2 c3"><span>Code as Research Products</span></li></ol><ol class="c14 lst-kix_xcfwdxntv4m-1 start" start="1"><li class="c2 c18 c29"><span>Publication and Citation</span></li></ol><p class="c2 c9"><span></span></p><p class="c2"><span>These cover the basic requirements to support open science activities. For graduate students and those new to development projects, we encourage you to start with these core guidelines.</span></p>
<h2 class="c2 c8" id="h.jc42b2hcl6wm"><span>Sustainable Code</span></h2><p class="c2"><span>Sustainable code deals with the characteristics that make source code maintainable </span><span>as</span><span> time </span><span>passes </span><span>and contributors change.</span></p><h3 class="c2 c8" id="h.w7q3addxbtzy"><span>Clean, standardized code</span></h3><h4 class="c2 c8" id="h.pt7zrmqte23n"><span>The codebase is well-structured and consistently structured. </span></h4><p class="c2"><span>The text of the code is formatted in a clean and consistent manner throughout the codebase. Function (method) names and parameter (variable) names follow a standard structure such as CamelCase. </span></p><h4 class="c2 c8" id="h.tekrxe5wegtn"><span>Code follows the language’s recommended style guide or the research group’s style guide.</span></h4><p class="c2"><span>A code style guide describes the formatting conventions preferred within a language community or within a research group. Other organizations, such as Google or Mozilla, also publish style guides for different languages. The key is in selecting a style guide and consistently applying it. </span></p><h4 class="c2 c8" id="h.1zkmnhg3e5oe"><span>Conformance to the style is verified using a linter.</span></h4><p class="c2"><span>A linter is a small utility used to check for style discrepancies and other potential issues in source code. These can be integrated into a preferred development environment, such as the vi or emacs text editors or into a GUI-based IDE like Eclipse. Using a linter catches syntax or formatting errors quickly and conveniently.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>These are language dependent, for example, Python’s </span><span class="c1"><a class="c4" href="https://www.pylint.org/">Pylint</a></span><span>, Javascript’s </span><span class="c1"><a class="c4" href="https://github.com/douglascrockford/JSLint">JSLint</a></span><span> or Oracle’s </span><span class="c1"><a class="c4" href="http://www.oracle.com/technetwork/java/codeconvtoc-136057.html">Java guide</a></span><span>.</span></p><h4 class="c2 c8" id="h.xacmpjv4c3t"><span>Linting is included during automated testing.</span></h4><p class="c2"><span>Once an automated testing system is in use for a project, ensure that linting is one of the tests included. Often, this is included in continuous integration processes. See </span><span class="c1"><a class="c4" href="#h.vekc78g7nqxx">Tested</a></span><span> below. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Regardless of whether a codebase is open sourced, i.e. accepting outside contributions, having a code style guide for your research group or for a project is encouraged. Emphasizing clean code practices aids in maintenance during active development or during long-term support. Code is the interface for developers so maintaining clean code over the lifespan of a project and as contributors change is encouraged.</span></p><h4 class="c2 c8" id="h.n0j2lapycvbr"><span>Errors and exception handling follows recommends practices for the language used.</span></h4><p class="c2"><span>Use language and module-defined exceptions in most cases. Catch specific errors and handle those appropriately for the code/software. For web applications and services, do not publicly expose full stack traces in any operational environment.</span></p><h3 class="c2 c8" id="h.g8dldwawc4gi"><span>Versioned code</span></h3><p class="c2"><span>Software (or code) versioning is important for open sourced code and for research activities related to citation, reproducibility (Stodden and Miguez 2014) and provenance.</span></p><h4 class="c2 c8" id="h.2tdya1yrzmc6"><span>Source code is maintained in a version control system (VCS).</span></h4><p class="c2"><span>Any code should be versioned in a version control system of your choice, i.e. </span><span class="c1"><a class="c4" href="https://git-scm.com/">git</a></span><span>, Apache </span><span class="c1"><a class="c4" href="https://subversion.apache.org/">Subversion</a></span><span>, or </span><span class="c1"><a class="c4" href="https://www.mercurial-scm.org/">Mercurial</a></span><span>. It is, for individual researchers or local teams, not critical which system you choose so long as it’s being used. For code or software intended to be publicly released and released for adoption, it is important to note that, at the time of this document, git-based systems are widely available and widely used. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For code notebooks, include the exported code (for easier comparison of modifications) and the exported HTML (for visual comparisons of generated outputs). The generation of these files can be automated depending on notebook platform.</span></p><p class="c2 c9"><span></span></p><p class="c2 c9"><span></span></p><p class="c2"><span>These repositories do not need to be public to start a project or during active development. VCS’s can be run locally, i.e. installing git on your laptop, on a private system for local team development, using a third party VCS system such as BitBucket or GitHub that offers private repositories or through larger institutional systems. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>When considering a VCS, consider the funder’s delivery and assessment requirements. Code repositories can be migrated from one VCS to another, they can be forked or cloned and, in the process, the repository can become decoupled from issue tracking systems or other related structure that are part of an assessment. See </span><span class="c1"><a class="c4" href="#h.b4mxlh97r1ja">Governed</a></span><span> and </span><span class="c1"><a class="c4" href="#h.rdevd43xfrh4">Code as Research Products</a></span><span> for information on related guidelines. We note this issue, and it is unresolved for distributed VCS. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Note that versioning, through remote VCS systems, is not a replacement for backups. </span></p><h4 class="c2 c8" id="h.6c0j1lpk4dir"><span>Source code uses a versioning scheme.</span></h4><p class="c2"><span>For code that is intended for adoption and ongoing maintenance, implement a versioning scheme. A versioning scheme, such as </span><span class="c1"><a class="c4" href="http://semver.org/">Semantic Versioning</a></span><span> or the </span><span class="c1"><a class="c4" href="https://www.gnu.org/prep/standards/html_node/Releases.html%23index-version-numbers_002c-for-releases">GNU preference</a></span><span>, here refers to the release version used in packaging and generally offers major and minor numbers, for example, “version 1.2” has a major version number of 1 and a minor number 2. This process gives adopters, using packaging tools or other automation tools for managing dependencies, a way to more effectively manage those dependencies and a way to manage those dependencies within their development cycles. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>See </span><span class="c1"><a class="c4" href="#h.278bwz69j1fc">Documented</a></span><span> for related activities.</span></p><h4 class="c2 c8" id="h.huu50eug1a5f"><span>Source code uses tagged releases.</span></h4><p class="c2"><span>Depending on the version control system in use, use tagged releases. A release is basically a specific commit in the code repository that meets some milestone conditions (defined by the project) and is stable (or not yet stable but released for beta testing). This provides adopters both an indication of when to upgrade to a stable version and, when combined with release notes (see </span><span class="c1"><a class="c4" href="#h.278bwz69j1fc">Documented</a></span><span>), some understanding of the effort needed to incorporate that new version into their own codebase or workflows.</span></p><h3 class="c2 c8" id="h.tlmgsr2d8yip"><span>Redistributable</span></h3><p class="c2"><span>The method for delivering the software depends on the project requirements. For software projects implementing desktop GUIs, for example, pre-built installers may be made available. For other projects, the preferred method may be providing scripts for automated build tools and installation instructions.</span></p><h4 class="c2 c8" id="h.dxcrxyr9q443"><span>Source code includes build scripts to automate builds.</span></h4><p class="c2"><span>Indicate what system is used to perform the build and provide configuration files or related information. For some languages, builds are described as package installs.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For dependencies that aren’t installed as part of the build, provide information about those requirements and procedures. Otherwise, use the dependency management of your selected build tool to ensure any required dependencies are installed during the build process.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>If your software has optional dependencies, indicate that and provide installation information. Note what features or capabilities are supported by each optional dependency. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>In some languages, packaging tools can provide binaries that serve effectively as installers. Consider providing this option if it meets project goals and community practices for that language or your target audience.</span></p><h4 class="c2 c8" id="h.v88x0wfkpugl"><span>Scripts or configurations are provided for creating binary installers.</span></h4><p class="c2"><span>For software delivered as an installer, provide the language-specific configurations and tool information to generate a new installer. (This assumes the code is released as open source.)</span></p><h4 class="c2 c8" id="h.hi3h4uzdhzde"><span>For desktop GUIs, provide an installer.</span></h4><p class="c2"><span>Provide an installer, tied to a versioned release, for any desktop GUI. Indicate which operating systems are supported and provide an installer for each.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Whenever possible, include an uninstaller or provide documentation to allow someone to perform a clean uninstall of the software.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Note that systems can be delivered as runnable containers,</span><span> Unix/Linux distributions or other options. These prepackaged options (not to be confused with code packages) should follow recommended security and creation guidelines as provided by the platform used. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>If the product is a code notebook, redistribution is often publication to a hosting platform like Jupyter’s <a href="https://nbviewer.jupyter.org/">NBViewer</a> or another public platform that can render the notebooks like GitHub.</span></p><h3 class="c2 c8" id="h.hlj4s6z7m6fe"><span>Tested</span></h3><p class="c2"><span>Testing provides numerous benefits throughout the code or software lifecycle like basic quality control, verification of builds or installation, and effective and efficient means of assessing contributions to the code base from internal team members or external contributors. In a well-designed test, you are setting the expectation of success and writing your code to meet that expectation so it is documenting intent and providing a means of verifying that intent as the code changes.</span></p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span>We promote code/software testing regardless of method. The style of testing during the active development cycle, whether test first (often using Test Driven Development (Beck 2003; Ambler 2013) or Behavior Driven Development) or test last (where testing is an integral part of the development process without adhering to the TDD methodology), is not as critical as encouraging and maintaining a strong testing culture in the research group. Integrate testing early in development (where possible), select a testing workflow that fits the group’s development workflow (and be open to adjusting the testing workflow to support continued, good quality testing rather than strict adherence to any particular methodology) and focus on developing high quality tests—these efforts provide direct benefit to a research group in their development efforts (as executable documentation, early bug detection, workflow improvements with automation and promoting overall trust in the product) and longer term benefits for the community as a whole (promoting trustworthy products, as code/software or the data/analyses generated from it, and supporting reproducibility, replicability, and preservation goals).</span></p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span>Tests, like documentation, are only usable if kept up to date. A reasonable criteria for adoption is that a code base does not include failing tests and certainly not failing tests that have been in the code for a long period of time. Code coverage can promote unsustainable practices; we recommend, if nothing else, testing key aspects and integrations. This includes testing for success states as well as exceptions or exceptional conditions.</span></p>
<h4 class="c2 c8" id="h.r7veqzthne4v"><span>Source code includes unit and integration tests.</span></h4>
<p class="c2"><span>Unit and integration tests (and, further, functional tests) help ensure that the code is meeting expectations at different levels of abstraction in the system: unit tests at a narrow component level and integration tests to confirm the behavior of some small set of components. Testing frameworks (or harnesses) are available for most programming languages, for example <a class="c4" href="https://github.com/substack/tape">Tape</a> (Javascript), <a class="c4" href="https://github.com/MOxUnit/MOxUnit">MOxUnit</a> (Matlab and GNU Octave), or <a class="c4" href="https://cran.r-project.org/web/packages/RUnit/index.html">RUnit</a> (R) as well as those provided in the core packages of a language (<a class="c4" href="https://docs.python.org/3.5/library/unittest.html">unittest</a> (Python) or <a class="c4" href="https://golang.org/pkg/testing/">testing</a> (Go)).</span></p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span>If the project provides specifications, ensure that tests cover the functional requirements within that document. Otherwise, consider ensuring coverage for those areas most likely to affect the user, your research or science goal.</span></p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span>For code notebooks, testing is often considered at the notebook level rather than the code block level. Some platforms may have third party modules to provide support for unit testing within the code blocks, but this is platform and language dependent. If the task benefits from testing at the code block level, take advantage of these modules; if not, consider the notebook the unit and the successful execution of all code blocks in order as the test.</span></p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span>For code and software projects requiring a higher degree of correctness, include assertions to help verify the code and diagnose bugs (Regehr 2014; Kudrjavets et al. 2006). Assertions can tell us if assumptions we’ve made about the code at some point in the execution are correct as we execute the code and without becoming disconnected from the code, as might happen with providing those assumptions in a comment block or specification. Assertions don’t replace testing as they each serve different and valuable functions in the codebase.</span></p>
<h4 class="c2 c8" id="h.qfvkud70t782"><span>Tests limit dependence on external services where possible.</span></h4><p class="c2"><span>Where possible, limit the use of external resources as inputs for the tests. Use <span class="c1"><a class="c4" href="http://martinfowler.com/articles/mocksArentStubs.html">mocking</a></span> (Fowler 2007) or include <span class="c1"><a class="c4" href="https://testing.googleblog.com/2012/10/hermetic-servers.html">test infrastructure</a></span> to build database dependencies (Narla and Salas 2012) or similar tasks. Tests should be as realistic as possible with regard to the system and with an eye towards efficient testing. Related, mocking allows tests to be developed for conditions that may be difficult or impossible to repeat from an external system.</span></p><h4 class="c2 c8" id="h.vekc78g7nqxx"><span>Testing is automated through a continuous integration system.</span></h4><p class="c2"><span class="c1"><a class="c4" href="http://martinfowler.com/articles/continuousIntegration.html">Continuous integration</a></span><span> is a development process that allows developers to commit small, discrete changes (bug patches or a new feature), into a shared codebase in a way that doesn’t “break the build” for other members of the team. </span><span>Continuous integration</span><span> is a way to enforce code quality, (code isn’t merged into the main branch if tests fail), to avoid large merge issues (conflicts in the code submitted by two developers),</span><span> to</span><span> catch integration bugs early in the development process</span><span>,</span><span> and </span><span>to </span><span>provide a process for acceptance testing in reproducible environments. Common tools include </span><span class="c1"><a class="c4" href="https://jenkins.io/index.html">Jenkins</a></span><span>, </span><span class="c1"><a class="c4" href="https://circleci.com/">Circle CI</a></span><span> and </span><span class="c1"><a class="c4" href="https://travis-ci.org/">Travis CI</a></span><span> which can be integrated into a GitHub workflow or, depending on the tool, run in a locally-hosted environment</span><span>.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Code notebooks can also be tested in continuous integration systems. This automation is encouraged for notebooks considered research products or documentation.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Note that continuous integration relies on several of the concepts outlined in this document before it can be implemented</span><span>, such as </span><span>version control and versioning, a test suite, and automated builds</span><span>. Nor is it limited to simple unit testing.</span><span> </span></p><h4 class="c2 c8" id="h.66jeksgqtmig"><span>For web applications, include automated GUI testing.</span></h4><p class="c2"><span>Automated GUI testing extends code tests to the interface, giving developers a means of ensuring that the user interface remains functional. This includes repeatable web form interactions, consistent interactions across browsers, and visual tests to ensure web pages render consistently and as expected. Tools such as </span><span class="c1"><a class="c4" href="http://www.seleniumhq.org/">Selenium</a></span><span> or <span class="c1"><a class="c4" href="https://watir.com/">Watir</a></span><span> provide options for automated testing against web interfaces and across browsers. </span></p><h2 class="c2 c8" id="h.k8wt6ghlvp2k"><span>Interoperable</span></h2><p class="c2"><span>These guidelines are intended for any code or software that provide</span><span>s</span><span> web services or data for consumption by other platforms; however, we are mostly focusing on the web application/web service projects.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Interoperable services fall under the same guidelines for performance and stability. See </span><span class="c1"><a class="c4" href="#h.7mb3s9l49d19">Usable</a></span><span> for information.</span></p><h3 class="c2 c8" id="h.jdkbivybsbw3"><span>For data services</span></h3><p class="c2"><span>Use community standards or conventions for data and metadata formats as well as the web services delivering those data. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Ensure that the data is usable by validating the data format during testing, for example, ensuring that an ISO 19115 file is valid XML.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Ensure that the web service is valid. Often this means validating a capabilities document against the schema provided by the standards body or through a web validator (</span><span class="c1"><a class="c4" href="http://testbed.echo.nasa.gov/cwic-smart/validations">CWICSmart</a></span><span> for OpenSearch, the </span><span class="c1"><a class="c4" href="http://cite.opengeospatial.org/teamengine/">OGC Web Validator</a></span><span>). At a lower level, ensure that the web service returns valid HTTP status codes and uses them for their intended purposes. Verify that content headers, especially those related to language and character encodings, are present and correct. Include service validation as part of the test suite for any generated service.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Ensure that the web service is complete with regard to the goals and use cases of the project. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Indicate, in the web API documentation for a service, that the service has been validated and the validation method, and that it is valid.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We note that often the development of systems supporting interoperability include external frameworks or platforms that are integrated into the project system. Given that these guidelines are intended only for code and software developed by the project team, some of the above may not be feasible. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>See </span><span class="c1"><a class="c4" href="#h.278bwz69j1fc">Documented</a></span><span> for API documentation discussions.</span></p><h3 class="c2 c8" id="h.jwbj4ujf85sn"><span>Applications, desktop or web-based, supports community data standards for inputs and outputs.</span></h3><p class="c2"><span>Whenever possible, use open standards for data inputs and outputs. </span></p><h3 class="c2 c8" id="h.9c8tsx28txfx"><span>For Semantic Services</span></h3><p class="c2"><span>Data dictionaries for each appropriate data output is provided in the documentation.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>If implementing a standard that supports it, provide codelists or controlled vocabularies.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Provide ontology or vocabulary services. Every collection and concept/term published online should have a persistent, dereferencable URI and a definition.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>If providing RDF or JSON-LD formats (or other Linked Open Data formats), provide the ontology. Follow recommended practices for LOD services such as including a license, ensuring forward and back-linked URIs are available, providing provenance metadata and providing performant services (Zaveri et al. 2015).</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Publish the ontology in a stable, persistent community repository.</span></p><h2 class="c2 c8" id="h.j3o37luj1lmk"><span>Usable</span></h2><p class="c2"><span>Usable software meets the needs of the communities it has been developed for</span><span>; i</span><span>deally</span><span>,</span><span> meeting those needs </span><span>also </span><span>matches the goals of the project. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>See </span><span class="c1"><a class="c4" href="#h.jc42b2hcl6wm">Sustainable Code</a></span><span> for guidelines regarding the usability of code.</span></p><h3 class="c2 c8" id="h.jf59v21hi69u"><span>Software has a clear, understandable interface.</span></h3><h4 class="c2 c8" id="h.6foli2yv92b3"><span>For software with a visual interface, such as a desktop GUI, plugin/extension or web/mobile application, the interface provides a clean and clear layout.</span></h4><p class="c2"><span>For desktop GUIs, this often means compliance with recommended design patterns for an operating system (</span><span class="c1"><a class="c4" href="https://developer.apple.com/library/mac/documentation/UserExperience/Conceptual/OSXHIGuidelines/">Apple OS X Human Interface Guidelines</a></span><span>) or for the interface toolkit of a programming language (Tkinter for Python or Swing for Java). </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For plugins or extensions, the interface components, such as forms and menus, conform to the design of the parent GUI.</span><span> If the parent platform provides guidelines for extensions or plugins, the extension follows those guidelines (</span><span class="c1"><a class="c4" href="http://docs.qgis.org/testing/en/docs/pyqgis_developer_cookbook/plugins.html">QGIS: Developing Python Plugins</a></span><span>).</span><span> </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For web applications, the interface design depends on the goals of the project. Web interface design is not as constrained as desktop GUIs, nor do we want to impose visual design preferences here. However, for research groups with limited resources, consider using an existing framework, whether an extensible content delivery framework (ie, CMS or portal) or a more generic front-end framework (</span><span class="c1"><a class="c4" href="http://getbootstrap.com/">Bootstrap</a></span><span> or </span><span class="c1"><a class="c4" href="http://foundation.zurb.com/">Foundation</a></span><span>).</span><span> </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>When assessing web applications, visual design is important only as it helps or hinders the functionality of the site. Unless a stated research goal involves developing a new interaction method, use common interface designs.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Provide clear labels and easily identifiable means of accessing help. Apply styles consistently across the application. Provide a style guide for visual styles, for example the </span><span class="c1"><a class="c4" href="https://standards.usa.gov/">USDS 18F Web Design Standards</a></span><span> (advanced option and more for those projects developed for adoption). </span></p><h4 class="c2 c8" id="h.tvmzy7swlrrs"><span>For web applications or any software accepting user inputs, Unicode is well-supported throughout the system.</span></h4><p class="c2"><span>We include this as an example of a what may appear to be a fairly specific detail but one that can, in web applications and services, indicate issues in the development process and, more importantly, in the understanding of the research group to understand and meet the needs of their intended audience. It is, in effect, an implementation canary for potential adopters and reusers.</span></p><h4 class="c2 c8" id="h.qxz6nnr14i6e"><span>For web applications, the application implements responsive design, i.e. support for multiple screen sizes.</span></h4><p class="c2"><span class="c1"><a class="c4" href="http://alistapart.com/article/responsive-web-design">Responsive design</a></span><span> adjusts an application’s layout based on the screen size (mobile, tablet, desktop) without requiring support for multiple application versions.</span></p><h4 class="c2 c8" id="h.ga5mu41sst4w"><span>For web applications, the application adheres to progressive enhancement practices. </span></h4><p class="c2"><span class="c1"><a class="c4" href="https://www.w3.org/wiki/Graceful_degradation_versus_progressive_enhancement">Progressive enhancement</a></span><span> adapts the supported features of the application based on the detected device. In practice, this means adjusting functionality based on slow connections, prevalence of feature phones (instead of smartphones) in the community being addressed, or other concerns related to the execution of Javascript code.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>This depends on the stated goals of the project and requires an understanding of the community needs </span><span>before</span><span> implementation. </span></p><h4 class="c2 c8" id="h.42fc5mhfrgk0"><span>For web applications, the interface follows established guidelines for web accessibility.</span></h4><p class="c2"><span>Web accessibility guidelines provide information to help develop web sites and applications that are open and inclusive for people with a range of abilities. Visit the W3C’s </span><span class="c1"><a class="c4" href="https://www.w3.org/standards/webdesign/accessibility%23wai">Web Accessibility Initiative</a></span><span> (</span><span class="c1"><a class="c4" href="https://www.w3.org/standards/techs/wcag%23w3c_all">WCAG</a></span><span>, </span><span class="c1"><a class="c4" href="https://www.w3.org/WAI/intro/aria.php">WAI-ARIA</a></span><span>) and the U.S. GSA’s </span><span class="c1"><a class="c4" href="https://section508.gov/content/learn/best-practice-library">Section 508</a></span><span> guidelines.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>These guidelines discuss structural concerns, for example semantic and well-structured HTML to support screen readers, and navigation or interaction requirements, such as keyboard bindings. We consider semantic and well-structured HTML in a similar context to that described in the </span><span class="c1"><a class="c4" href="#h.jc42b2hcl6wm">Sustainable Code</a></span><span> section, namely that it is beneficial to meet guidelines related to clean, well-structured code and is not considered an onerous or additional burden.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For web applications supporting dynamic mapping functions, try to use mapping libraries that include support for accessibility whenever possible. We note that support for colorblind users, represented only in data color schemes only, does not meet the minimum expectations for supporting web accessibility. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Project websites should also follow </span><span>web accessibility</span><span> guidelines. This may not be in the research group’s control, given institutional requirements or existing cyberinfrastructure; however, we encourage research groups to consider web accessibility support as an important adoption criterion should the choice of framework be available.</span></p><h4 class="c2 c8" id="h.fdfrt2xf1lu2"><span>For web applications, support current browser versions.</span></h4><p class="c2"><span>Front-end frameworks and libraries have greatly improved cross-browser support, but it’s still important to ensure that the web application functions as intended across any and all browsers supported by the project. Any modern Javascript framework, whether the more generic <a class="c4" href="https://jquery.com/">JQuery</a> or interface frameworks like <a class="c4" href="https://facebook.github.io/react/">React</a>, <a class="c4" href="http://backbonejs.org/">Backbone</a>, or <a class="c4" href="http://emberjs.com/">Ember</a>, will provide cross-browser support.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Sometimes, project goals and community needs may require supporting older browser versions. In this case, state the supported versions and take steps to meet those requirements </span><span>safely</span><span> (noting vulnerabilities in older browser versions, etc). Progressive enhancement principles can be considered to provide equivalent functionality for users limited to older browser versions and users using current versions.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The opposite may also hold true, where a research goal is to develop a proof of concept or prototype application using cutting edge functionality that may only be supported in certain browsers. In this case, progressive enhancement is not necessary but the project documentation and web application should indicate that limited support clearly.</span></p><h4 class="c2 c8" id="h.vft1b61bp3vj"><span>For a web service, the API is </span><span>RESTful</span><span> </span><span>or follows a known community standard or specification.</span></h4><p class="c2"><span>API design is a usability concern. As with other areas, an API should be clear and legible, with meaningful names and </span><span>using the </span><span class="c1"><a class="c4" href="https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm">RESTful design pattern</a></span><span> </span><span>(Fielding 2000) or </span><span>HTTP API pattern</span><span>. See </span><span class="c1"><a class="c4" href="#h.k8wt6ghlvp2k">Interoperable</a></span><span> for guidance on implementing community standards, understanding that those standards may not be RESTful or conform to the conceptual models found in Web API documentation platforms. For the purposes of assessment, correct implementation of the community standard outweighs any RESTful recommendations.</span></p><h4 class="c2 c8" id="h.1gxhk9ppul8o"><span>For collaboration tools, ensure that the </span><span>interface design does not encourage sensitive information leaks</span><span>.</span></h4><p class="c2"><span>Carefully consider interface design choices related to an individual’s permissions and access selections. These characteristics, related to layout, label text and other details, are included in general web interface guidelines but they are important to highlight here due to privacy concerns. Permissions and access options should not result in unexpected behavior from the viewpoint of the individual using the platform. </span></p><h3 class="c2 c8" id="h.7mb3s9l49d19"><span>Software is performant and stable.</span></h3><h4 class="c2 c8" id="h.nwh0jy5c06kn"><span>Documentation describes the current development status.</span></h4><p class="c2"><span>For open or public codebases, this gives potential adopters an understanding of where the project is in its lifecycle. This can be as simple as adding a note stating that the source code is under very active development (high rate of commits, known breaking bugs remain, or rough interface), the source code is not being maintained or is in a reasonably stable in-between state.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For code or software that will not be maintained or has reached end-of-life, update the README and other documentation to indicate this status. This can be text or with a badge (see @potch </span><span class="c1"><a class="c4" href="http://unmaintained.tech/">No Maintenance Intended</a></span><span>).</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Please refer to </span><span class="c1"><a class="c4" href="#h.1ogf2zcgktnr">Governed</a></span><span> for managing contributions.</span></p><h4 class="c2 c8" id="h.mpxwzayw9bzb"><span>For web applications or web services, the documentation or project website includes service level qualities.</span></h4><p class="c2"><span>Simply put, provide potential adopters of the public-facing system information about the reliability and performance of that system. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For web services, include limitations for requests (rate limiting) and other policies related to automated access. This gives adopters guidelines for developing clients safely, i.e. they understand the system limitations and can work within them, and gives adopters of the system some indication of system limitations when taking the </span><span>deployment requirements</span><span> into consideration. It also gives you, as the maintainer of a system, a </span><span>clear process</span><span> for dealing with violations of these limits and an understanding of the system performance for planning future work. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For web applications and services, use an appropriate method for managing human and automated access. Consider the service level qualities and project goals with the understanding that potential adopters may be approaching your services with very different, but valid, use cases. This is an inherent feature of publishing to the open web! Use robots.txt, rate limiting via server configuration, access token policies or other connection configuration options.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For web applications and services, provide information regarding uptime, methods for receiving outage notifications, expected response times for responses to support requests.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>These service level qualities are dependent on </span><span>effective monitoring</span><span>. Providing service level quality information is not, in itself, enough to build and maintain trust in a system. See </span><span class="c1"><a class="c4" href="#h.1z1daqwsh16q">Monitoring</a></span><span> below. </span></p><h4 class="c2 c8" id="h.1z1daqwsh16q"><span>For web applications and web services, systems are monitored.</span></h4><p class="c2"><span>Monitor your systems for security and infrastructure (anomalous activity), for service level quality metrics and performance (uptime, service response times), for sustainability metrics (growth, retention, service usage). Note that these are different concepts and use different platforms, for example, </span><span class="c1"><a class="c4" href="https://www.nagios.org/">Nagios</a></span><span> for infrastructure, the </span><span class="c1"><a class="c4" href="https://www.elastic.co/products">Elastic Stack</a></span><span> (ElasticSearch, Logstash and Kibana) for log analytics, and </span><span class="c1"><a class="c4" href="https://www.google.com/analytics/%23?modal_active%3Dnone">Google Analytics</a></span><span> for application events. You may also be able to take advantage of solutions provided by your cloud resource provider. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The level of monitoring depends on where in the project lifecycle you are and what expectations you have for adoption and use. For production (beta) or operational systems, monitoring is not considered “nice to have” as access issues, performance issues or other issues affecting the use of the system has a negative impact on the trust of the system—instability or </span><span>persistent performance problems</span><span> (Brutlag, 2009) drives people and potential adopters away.</span></p><h4 class="c2 c8" id="h.l9dc6388w79f"><span>For high performance computing, GPU or other cluster-based software, the project provides basic benchmarking </span><span>statistics.</span></h4><p class="c2"><span>For algorithm development or similar code, provide benchmarks with comparisons to similar algorithms. Describe the benchmarking process. We note that this activity can be the topic of a research paper in certain domains. References to the paper and related research products can be provided instead of a detailed write-up in the documentation. Cite it appropriately (see </span><span class="c1"><a class="c4" href="#h.tk3e9fopeigk">Publication and Citation</a></span><span>).</span></p><h2 class="c2 c8" id="h.278bwz69j1fc"><span>Documented</span></h2><p class="c2"><span>Documentation is important across project areas and audiences. Research code and software has a higher documentation burden. This is generally seen in two areas: the temporary and variable nature of the workforce and the expectations of reproducibility and contributions to the larger research community. Both can be </span><span>mitigated</span><span> by improved source code and project documentation where an effective knowledge base protects the project against unexpected churn and aids in onboarding new participants. The ideal of self-documenting code must be considered against the realities of the research development workforce and conditions.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>When in doubt, a good rule of thumb is to document things that can’t be found in any dependency or framework documentation and that might appear counter-intuitive to another developer, even if it’s you in six months. </span></p></p><h3 class="c2 c8" id="h.qlvssx94m25r"><span>Source code is documented.</span></h3><p class="c2"><span>Comment blocks are included in the source code. Minimum documentation describes methods and the input and output parameters. In addition, we recommend code comments to describe sections that may not be clear, for example, bug patches that appear counter-intuitive or business decisions that shouldn’t be modified without clear external reasons. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For code notebooks as a research product, include text blocks to describe the workflow and any outputs. </span></p><h4 class="c2 c8" id="h.muhqkex415r9"><span>Source code comments use a standard comment style, related to the selected style guide or language or related to a document generation tool. </span></h4><p class="c2"><span>As with the code itself, the comment style should follow the chosen style guide or language conventions. </span></p><h4 class="c2 c8" id="h.ycrdao5dntt5"><span>API documentation is automatically generated.</span></h4><p class="c2"><span>In this case, the API can refer to the methods and properties of the codebase or a web API. Document generators, such as </span><span class="c1"><a class="c4" href="http://www.sphinx-doc.org/en/stable/index.html">Sphinx</a></span><span> (Python) or </span><span class="c1"><a class="c4" href="https://github.com/weavejester/codox">Codox</a></span><span> (Clojure), expect explicitly formatted comment blocks within the code or as standalone files. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For web APIs, especially RESTful APIs, use a documentation specification geared to that such as </span><span class="c1"><a class="c4" href="http://swagger.io/">Swagger</a></span><span> (or </span><span class="c1"><a class="c4" href="https://openapis.org/">Open API Initiative</a></span><span>), </span><span class="c1"><a class="c4" href="http://raml.org/">RAML</a></span><span> or </span><span class="c1"><a class="c4" href="https://apiblueprint.org/">API Blueprint</a></span><span>. In some cases, document generation tools may support the API specification so check that documentation for possible integrations. Keep in mind that public web API documentation does not replace source code documentation as one serves adoption for client development and reuse and the other supports further contributions. </span></p><h4 class="c2 c8" id="h.4nph8cwunjf5"><span>The documentation describes how to automatically generate the documentation.</span></h4><p class="c2"><span>When using a documentation generator, describe the tools and process for generating the documents. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For traditionally open sourced projects, this description is related to other </span><span class="c1"><a class="c4" href="#h.qderu9pnkzxc">contribution guidelines</a></span><span> and offers contribution options beyond code.</span></p><h4 class="c2 c8" id="h.tq0ajo5926em"><span>API Documentation is versioned.</span></h4><p class="c2"><span>Like the code, the generated API documentation is versioned and maintained in a VCS. API documentation can be hosted under the project website or an external documentation host such as </span><span class="c1"><a class="c4" href="https://readthedocs.org/">Read the Docs</a></span><span>. If possible, link the documentation version with a code release (for reproducibility, preservation and community support). </span></p><h3 class="c2 c8" id="h.gjmwihq5i3uk"><span>The codebase includes documentation to support adoption and reuse.</span></h3><p class="c2"><span>This documentation is related to those information items that describe how to deploy the code or software. Reuse and adoption here refers to the developer community but it can also benefit preservation needs. </span></p><h4 class="c2 c8" id="h.eyumqunuhqwt"><span>The source code repository is documented using plain text file formats.</span></h4><p class="c2"><span>Current practice is to provide certain documentation through </span><span class="c1"><a class="c4" href="https://daringfireball.net/projects/markdown/">Markdown</a></span><span> or </span><span class="c1"><a class="c4" href="http://docutils.sourceforge.net/rst.html">reStructuredText</a></span><span>. Expected files include a project-level README, LICENSE, and INSTALL and may include a code of conduct, CONTRIBUTING, AUTHOR or other files common to the GNU recommended file structure. </span></p><h4 class="c2 c8" id="h.jo1cauupk7qk"><span>The build procedure or installation process is described.</span></h4><p class="c2"><span>Describe the build procedure if building from the source code. Describe the installation process if providing binaries. Be sure to include any dependency requirements, including version information. Also include system requirements, including version information.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Often, build/installation details are provided in the source code’s README or INSTALL files. </span></p>
<h4 class="c2 c8" id="h.nkh4hefff5qr"><span>The documentation describes how to run simple examples.</span></h4>
<p class="c2"><span>Select a representative task and provide configuration and parameters or input data to execute the required code. Include a brief description of the task. Provide the expected outcome or where to locate any generated outputs as appropriate. This documentations can also be provided as code notebooks.</span></p>
<p class="c2 c9"><span></span></p>
<p class="c2"><span>See </span><span class="c1"><a class="c4" href="#h.5wf9rcv7ts53">Software is documented > Any Documentation</a></span> for information regarding similar documentation with quick start guides.</span></p>
<h4 class="c2 c8" id="h.4pnm2fq7ygzv"><span>The documentation describes how to execute the test suite.</span></h4><p class="c2"><span>The documentation describes the tool used for testing and how to execute tests. Often this is included as a post-install or post-build step to demonstrate that that process was successful. For automated deployments where the configurations are provided, safely, with the code base, indicate that tests are automatically run.</span></p><h4 class="c2 c8" id="h.6qp4s0y7ha51"><span>The versioning scheme is described.</span></h4><p class="c2"><span>For source code in a VCS and with a defined versioning scheme, provide the versioning scheme used in the documentation. This is most likely included in the CONTRIBUTING file.</span></p><h4 class="c2 c8" id="h.csa26k8fwk2r"><span>Release notes or changelogs are provided for each release.</span></h4><p class="c2"><span>Clear and complete release notes provide adopters some understanding of how a major or minor release may affect their own systems and workflows. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For preservation and archiving concerns, provide these items in the code repository or other documentation. Consider a future scenario in which a potential issue with a generated dataset comes up and, while the code is accessible (locatable and viewable), it may not be possible to run. Release notes in these situations provide avenues for understanding when and where the issue may have occurred.</span></p><h3 class="c2 c8" id="h.90q5gnajhkbj"><span>Software is documented.</span></h3><p class="c2"><span>Here we are discussing documentation guidelines for higher level guides for user or developer documentation, such as how-to guides, quick start guides or tutorials.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Certain code or software types, like APIs and libraries/modules, lend themselves to integration into code notebooks to provide interactive examples and tutorials. Notebooks can be a valuable publication option for some of the guidelines below.</span></p><h4 class="c2 c8" id="h.5wf9rcv7ts53"><span>Any Documentation</span></h4><p class="c2"><span>Any restrictions or constraints for a particular method, request, or process is included in that item’s documentation. This includes file size limits or rate limiting for web APIs.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Quick start guides are provided to provide any user an introduction to the code or software. The selected example should reflect a normal interaction, provide a description of common possible errors and how to respond to each, and a successful outcome. If warranted, provide a simple dataset for this demonstration.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Developer Documentation</span></p><p class="c2"><span>Note that developer documentation can simply be API documentation for modules/libraries, command line interfaces, or web APIs.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For command line tools and web APIs, all input and output parameters for each command or request are defined, including default values. Every command or request is likewise described.</span></p><h4 class="c2 c8" id="h.m7qbd04yq214"><span>User </span><span>documentation </span></h4><p class="c2"><span>This documentation is aimed at users of research software, whether a desktop GUI, plugin/extension, or web application. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The software documentation is complete. It describes every possible action with annotated screenshots of interfaces and step-by-step processes. This includes possible errors and recovery options. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The images and method or parameter names are up-to-date, i.e. they match what the user sees in the software. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Provide this documentation in a manner that is searchable through a web search engine (preferred) for both desktop GUIs and web applications. For desktop GUIs, provide documentation in a manner supported by the application or the operating system. </span></p><h3 class="c2 c8" id="h.6kfwca5ceoo6"><span>Code or software requirements are documented.</span></h3><p class="c2"><span>Formal specifications are not always required but, for those projects that have a clear need for formal correctness (algorithm implementations, data processing, etc), provide the specifications used for implementation. If not provided as a published document, ensure that the expectations are described in the appropriate tests (see </span><span class="c1"><a class="c4" href="#h.hlj4s6z7m6fe">Tested</a></span><span> for related). </span></p><h3 class="c2 c8" id="h.gqpohaprkcbq"><span>Project is documented.</span></h3><p class="c2"><span>This is the highest level documentation and often the first information made public. It can, in some ways, be considered pre-registration. This documentation is maintained on a project website separate from the code.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>This website should include the grant award numbers and agencies, initial project milestones, a description of the project and the expected software or code deliverables and a listing of project team members.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>As the project continues, include reports, articles, conference presentations or other materials on the site. If possible, include testimonials from satisfied users.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Include links to the project source code or software installers and documentation.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Provide copyright information, commonly in the footer of the project site. Consider licensing the project web site content through </span><span class="c1"><a class="c4" href="https://creativecommons.org/">Creative Commons</a></span><span> or under the license of the department or organization.</span><span> </span></p><h2 class="c2 c8" id="h.kevi2ogx4kct"><span>Secure</span></h2><p class="c2"><span>Any code or software</span><span> should be developed following the current recommended practices for the systems being used. Security must be considered across functional areas, from system administration and backend development to frontend development and user access. Consider using tools such as </span><span class="c1"><a class="c4" href="https://www.synopsys.com/software-integrity/products/static-code-analysis.html">Coverity</a></span><span>, </span><span class="c1"><a class="c4" href="https://www.sourceclear.com/">SourceClear</a></span><span> or </span><span class="c1"><a class="c4" href="https://hackage.haskell.org/package/QuickCheck">QuickCheck</a></span><span> variants.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The principles outlined below are not inclusive and are meant to highlight certain areas of concern and point to resources to provide more detailed information. Security through obscurity is not a viable method. </span></p><h3 class="c2 c8" id="h.c0ntdoo6m887"><span>Source code, including automation configurations and scripts, have been sanitized for public release.</span></h3><p class="c2"><span>Whether you release source code at the end of the funded project for preservation or use a version control platform during development, it is important to ensure that sensitive information, such as access credentials for cloud services or database logins, are not ever publicly available. This starts before the first commit to a repository — the information remains in the history without additional effort. Take steps to ensure that you are not leaking credentials through your version control histories, through backups, through any automation configuration, or any other potential avenue. </span></p><h3 class="c2 c8" id="h.bwddr3yneyvr"><span>For web applications and services, software follows industry recommendations for secure practices.</span></h3><p class="c2"><span>It is beyond the scope of this document to cover all aspects of web application or web service security. However, we do strongly recommend following those put forth by the </span><span class="c1"><a class="c4" href="http://trustedci.org/">Center for Trustworthy Scientific Cyberinfrastructure</a></span><span> and </span><span class="c1"><a class="c4" href="https://www.owasp.org/index.php/Main_Page">The Open Web Application Security Project</a></span><span> (OWASP) as well as taking advantage of the </span><span class="c1"><a class="c4" href="https://github.com/google/vsaq">Vendor Security Assessment Questionnaire</a></span><span> (Google VSAQ). </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Some considerations:</span></p><ul class="c14 lst-kix_hputbrx9prs6-0 start"><li class="c2 c3"><span>Enable HTTPS on every web application or service (note, Google uses HTTPS for search results ranking (Google Security Blog 2015) and, as such, is recommended beyond security requirements).</span></li><li class="c2 c3"><span>Sanitize web form inputs to prevent cross-site scripting (XSS) or SQL injection attacks.</span></li><li class="c2 c3"><span>Protect against Cross-site Request Forgery (CSRF) attacks.</span></li><li class="c2 c3"><span>Sanitize responses related to errors and exceptions, i.e. do not return stack traces or similar from any public-facing platform.</span></li></ul><p class="c2 c9"><span></span></p><p class="c2"><span>For HTTPS</span><span>, check with your home institution or department for certificate providers in use (</span><span class="c1"><a class="c4" href="https://www.incommon.org/certificates/">InCommon</a></span><span> is a common university option). Small research groups may also want to consider an open certificate authority such as </span><span class="c1"><a class="c4" href="https://letsencrypt.org/">Let’s Encrypt</a></span><span>. </span></p><h3 class="c2 c8" id="h.2qasdya4g2dq"><span>For any code or software system, follow the recommended security practices for each system component.</span></h3><p class="c2"><span>For any system or framework or similar dependency used, follow the recommended security practices for that system for deployment and for ongoing management. Consider databases, web frameworks, authentication systems (such as </span><span class="c1"><a class="c4" href="https://developers.google.com/identity/protocols/OAuth2">OAuth2</a></span><span>), web servers, cloud infrastructure access. </span></p><h3 class="c2 c8" id="h.v4eyawiuk7gf"><span>Keep systems and dependencies up-to-date.</span></h3><p class="c2"><span>Security vulnerabilities can occur in any part of the implementation stack. For software or code currently being maintained under a grant, take note of vulnerability notifications and update dependencies and code accordingly. Indicate to adopters when a version contains a security patch. Indicate in the release notes which dependencies require updating—do not force a potentially breaking update without some warning.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>See </span><span class="c1"><a class="c4" href="#h.1ogf2zcgktnr">Governed</a></span><span> for information regarding sunsetting web applications or services. If resources are not available to upgrade the application to manage vulnerabilities in the dependencies, including the programming language version, operating system or web server version, we suggest archiving the system.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>If the project is releasing runnable containers</span><span>, we encourage you to provide upgrade paths for the container or the application it runs. If nothing else, refrain from presenting a runnable container as one-time task and provide estimates for system maintenance for potential adopters, particularly if presenting the product as a solution for those without dedicated technical resources.</span></p><h3 class="c2 c8" id="h.wcbbon9uy2j6"><span>Collaborative platfo</span><span>rm</span><span>s include tests for permissions and integrations.</span></h3><p class="c2"><span>Include extensive tests for permissions conflicts in the test suite, particularly those that might make public content the person or group marked as private. Consider the interactions of all the content privacy settings. Consider carefully any new setting that might be added to the system, as </span><span>potential conflicts and errors grow accordingly</span><span>.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Indicate in the documentation, at a very high level, that this kind of testing is performed. This is not dissimilar to a service level quality in that the statement alone is without merit. It must be backed up with valid tests of the system.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Include additional testing for integrations, or the use of third party APIs such as Google Docs or GitHub. Only ask for permissions from the third party platforms that are required for use in the collaboration platform. Follow any recommended practices for securely connecting to those services.</span></p><h3 class="c2 c8" id="h.ltbzsn7tud66"><span>Containers follow recommended</span><span> practices for security.</span></h3><p class="c2"><span>Container systems, such as </span><span class="c1"><a class="c4" href="http://www.docker.com/">Docker</a></span><span> (Docker 2015) or the </span><span class="c1"><a class="c4" href="https://linuxcontainers.org/">Linux Container</a></span><span> project, are not secure by default. For any public-facing system or any system deployed to a cloud provider, follow the recommended security practices for that container platform. </span><span>This applies to containers deployed and maintained by the project team and those built as a project deliverable or with the intent to share (“Project in a Box” situations). </span></p><p class="c2"><span><br>See </span><span class="c1"><a class="c4" href="#h.561dz5rrclj0">Sharable</a></span><span> for additional guidelines on automation. </span></p><h2 class="c2 c8" id="h.j3cd3qemat79"><span>Sharable</span></h2><p class="c2"><span>The guidelines promote practices for adoption and reusability of the code or software’s codebase whether it was born-open or publicly released at a later time. These are intended for code released for publication and reproducibility as well as code released as an active open source project.</span></p><h3 class="c2 c8" id="h.hhtvml7dyjhk"><span>Source code is licensed.</span></h3><p class="c2"><span>Any code or software product, whether released for publication/preservation only or for ongoing development activities, should include a license. ESIP cannot provide recommendations for which license to apply but we encourage you to refer to your institution’s intellectual property policies with consideration for your funding agency’s policies or any policies put forth in the solicitation of the award. Unlicensed code is</span><span> unusable code</span><span>—you haven’t provided a potential adopter any information regarding permissions or limitations on its use so the assumption is that an adopter is not permitted to use or modify that source code. Unusable code is unsustainable code in the context of a funder’s return on investment.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>License selection is affected by the intended use of the software and the licenses of the dependencies you use. For this reason, we encourage you to consider your license options early in the development process and to take that choice into consideration throughout implementation.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For projects allowing outside contributions, you may also want to consider </span><span class="c1"><a class="c4" href="https://www.clahub.com/">Contributor License Agreements</a></span><span>. A CLA states that the contributor agrees to contribute and that the contributor grants rights to the project so that the project can use the contribution in distributions and that the contributor can’t revoke that right.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For a description of open source licenses, please visit the </span><span class="c1"><a class="c4" href="https://opensource.org/licenses">Open Source Initiative</a></span><span> (OSI) or </span><span class="c1"><a class="c4" href="http://choosealicense.com/">Choose a License</a></span><span>. We strongly encourage the adoption of an OSI-approved license whenever possible.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For source code published for preservation, at a minimum, the selected license should allow someone to rerun your code for the purposes of reproducibility or replication. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>One final note on software licenses in academic workplaces—intellectual property restrictions are related to your job status (faculty, student, staff). Again, ESIP cannot provide legal advice; however, we encourage research groups to familiarize themselves with these issues and agree to internal procedures for handling intellectual property issues (licensing or copyright). For a related discussion about research data, please refer to the </span><span class="c1"><a class="c4" href="https://rd-alliance.org/sites/default/files/attachment/Legal%2520Interoperability%2520Principles%2520and%2520Implementation%2520Guidelines_Final.pdf">RDA-CODATA Legal Interoperability for Research Data</a></span><span> report</span><span> (Uhlir and Clement, eds, 2016).</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>See </span><span class="c1"><a class="c4" href="#h.278bwz69j1fc">Documented</a></span><span> for related.</span></p><h3 class="c2 c8" id="h.561dz5rrclj0"><span>Source code includes configurations for automated systems.</span></h3><p class="c2"><span>Automation can include test tools, build tools or packagers, provisioning and/or orchestration tools (</span><span class="c1"><a class="c4" href="https://www.chef.io/">Chef</a></span><span>, </span><span class="c1"><a class="c4" href="https://www.vagrantup.com/">Vagrant</a></span><span>, </span><span class="c1"><a class="c4" href="https://puppet.com/">Puppet</a></span><span>), or continuous integration systems. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For tests, this includes providing verified inputs and outputs as necessary. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Follow the tool’s security practices for preventing credential leaks or other vulnerability concerns.</span></p><h3 class="c2 c8" id="h.oofsvittp0hd"><span>Project name is discoverable.</span></h3><p class="c2"><span>When discussing project names, ensure that the name or acronym is unique within a domain, at a minimum, and that it is searchable as is or with some additional context. Common words or highly similar acronyms/names can make a project difficult or impossible to find through a search engine. </span></p><h2 class="c2 c8" id="h.1ogf2zcgktnr"><span>Governed</span></h2><p class="c2"><span>Governance guidelines refer to the processes around active development and maintenance as well as communications related to those.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Project governance guidelines can be applied without requiring a project to be traditionally open sourced. Indeed, many of the guidelines are simply explicit statements of process. We also note that it is not uncommon to consider a project “internal open source,” i.e. it’s managed in a way similar to a public OS project without being public. As with </span><span class="c1"><a class="c4" href="#h.278bwz69j1fc">Documented</a></span><span>, these guidelines are also meant to mitigate some of the issues related to the nature of our staff resources. These, then, can be applied to internal teams, larger collaborative groups and to traditionally open sourced projects. </span></p><h3 class="c2 c8" id="h.qderu9pnkzxc"><span>Contribution policies are provided.</span></h3><p class="c2"><span>This </span><span class="c1"><a class="c4" href="https://github.com/nayafia/contributing-template/blob/master/CONTRIBUTING-template.md">example template</a></span><span> for contribution policies covers many of the recommendations and provides examples of existing documents.</span></p><h4 class="c2 c8" id="h.j2u8c0tf34q2"><span>For any project, describe the workflow used by the project for contributions to the source code (or other versioned items).</span></h4><p class="c2"><span>This assumes that the source code is maintained in a VCS. Choose a workflow for contributions to the code repository. For git-based systems, this most likely involves a branch-based workflow such as </span><span class="c1"><a class="c4" href="https://guides.github.com/introduction/flow/">GitHub Flow</a></span><span>. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For local teams (or even single contributors), select a workflow that is workable for all of the team. For public processes, select a commonly used workflow for your VCS of choice and your review process.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Part of this workflow definition should include expectations about testing and documentation related to the contribution. </span></p><h4 class="c2 c8" id="h.ns2tcmg58ars"><span>For any project, describe commit message, pull request, and issue preferences.</span></h4><p class="c2"><span>If your workflow or automation processes support it (or for legibility), define commit message conventions and branch naming conventions. Set expectations for issue content. Define what is expected for bug reports or new features. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>See </span><span class="c1"><a class="c4" href="#h.278bwz69j1fc">Documented</a></span><span> for related.</span></p><h4 class="c2 c8" id="h.p0kh0cpf78eq"><span>For any project, define the code review process.</span></h4><p class="c2"><span>This describes the process for accepting a contribution into the repository. It applies to any development team and sets the roles and responsibilities of the reviewers and an expected timeframe for response.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Note that this response time is not dissimilar to that described for service level qualities as outstanding pull requests can be a warning flag for potential contributors and for sustainability concerns.</span></p><h4 class="c2 c8" id="h.ffvx3h4l2hb7"><span>For open source projects, provide a code of conduct or state clear expectations of behavior and communication.</span></h4><p class="c2"><span>This is a sign of openness to potential contributors. </span><span>A common structure is the </span><span class="c1"><a class="c4" href="http://contributor-covenant.org/">Contributor Covenant</a></span><span>. Consider the statements made in any code of conduct you develop or reuse and ensure that you agree with and will enforce the statements in the document. </span></p><h4 class="c2 c8" id="h.oy3ktoh4p9gb"><span>For open source projects, provide guidelines about what contributions will or will not be accepted.</span></h4><p class="c2"><span>This includes contributions not related to source code, contributions that don’t fit within stated milestones or near-term feature implementation goals, or contributions that don’t comply with other stated expectations. A common example of what won’t be accepted are contributions based solely on style guide modifications (tabs changed to spaces, for example). </span></p><h3 class="c2 c8" id="h.b4mxlh97r1ja"><span>Development activities are transparent.</span></h3><p class="c2"><span>The section involves those activities related to active development and the communication processes in places for development and support.</span></p><h4 class="c2 c8" id="h.foxopgq1esv4"><span>Development activities are managed through an issue tracker or similar software.</span></h4><p class="c2"><span>For internal teams, this can be the tracker provided through an external or self-hosted VCS (GitHub, </span><span class="c1"><a class="c4" href="https://bitbucket.org/">BitBucket</a></span><span>, </span><span class="c1"><a class="c4" href="https://about.gitlab.com/">GitLab</a></span><span>), through a project management tool (</span><span class="c1"><a class="c4" href="https://www.pivotaltracker.com/">Pivotal Tracker</a></span><span>) or a TODO manager (</span><span class="c1"><a class="c4" href="https://trello.com/">Trello</a></span><span>).</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For traditionally open sourced projects, the issue tracker provided through the external VCS is preferred, although any public tracker is acceptable. This allows adopters to submit bug reports or external contributors to submit pull requests for patches and new features. </span></p><h4 class="c2 c8" id="h.l1vbrrmurts7"><span>Project provides support mechanism(s).</span></h4><p class="c2"><span>For project support beyond a specific code base (a project can support multiple codebases), provide a dedicated support email or contact form. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For security concerns, a support contact option is recommended over the public issue tracker. This provides a way for someone to describe a potential vulnerability privately, giving the project team time to respond. Include this contact information and preference regarding vulnerability concerns in your CONTRIBUTING documentation or other appropriate documentation location. We strongly encourage this for collaborative platforms.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Other communication methods include Twitter, listservs or mailing lists and IRC. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Any public-facing support mechanism should be actively used by project team members. This does not mean enforced chatter; it means that support requests are addressed in a timely manner. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>In the event that a web application or service reaches the end of its lifespan, project members use these support mechanisms and other public-facing avenues (the website and/or code repository README, for example) to notify the community of adopters of service sunsetting and expectations for the cessation of services. Include information about what will happen with crowdsourced data, account information, site access and other relevant details based on the nature of the project.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Related: </span><span class="c1"><a class="c4" href="#h.mpxwzayw9bzb">service level qualities</a></span><span>.</span></p><h4 class="c2 c8" id="h.tux2f6u3qd4a"><span>Project provides a development roadmap</span></h4><p class="c2"><span>A roadmap document describes the philosophical underpinnings of the project and its future technical path. This gives potential contributors and other adopters insight into how well the project may fit their needs longer term and the types of features or enhancements that are likely to be accepted. This is higher level than the contribution guidelines.</span></p><h2 class="c2 c8" id="h.nag33fat6sc8"><span>Code as Research Products</span></h2><p class="c2"><span>These guidelines discuss activities important to the research community for preservation, reproducibility, provenance and credit. When discussing accessibility in this section, we are referring to the ability of an individual to locate a project and, more importantly, its software or code products. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The Geoscience Paper of the Future (Gil et. al 2016) outlines recommended practices and the relationships of research code and software within the larger scholarly ecosystem. Academic journals may also provide guidance on their expectations for research code and software related to accepted papers. Support journals that are signatories of or have implemented the Transparency and Openness Promotion Guidelines (Alter et al. 2016).</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The ongoing shift towards considering code and software as research products in their own right is a rapidly changing environment. The guidance below starts with currently actionable steps, defaulting to capturing the information in the code repository at least.</span></p><h3 class="c2 c8" id="h.tk3e9fopeigk"><span>Publication and Citation</span></h3><h4 class="c2 c8" id="h.llz3mjenwmpr"><span>Software, as binaries and source code, are published to a sustainable third-party repository.</span></h4><p class="c2 c9"><span></span></p><p class="c2"><span>This criteria can be met for projects hosting source code in an external VCS platform during development and simply leaving those publicly accessible at the end of the project. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Some research communities have developed domain-specific software and code registries for publication. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Other options include depositing software/code and related documentation with your institutional repository. </span></p><h4 class="c2 c8" id="h.42ni1tsctkct"><span>Documentation includes citation details.</span></h4><p class="c2"><span>As noted earlier regarding other metrics for assessing the sustainability (or viability) of a research software package includes impact related to citations. Provide a suggested citation in the README of the source code and the project website.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>If publishing a software paper to a dedicated research software journal such as </span><span class="c1"><a class="c4" href="http://www.journals.elsevier.com/softwarex/">SoftwareX</a></span><span> , </span><span class="c1"><a class="c4" href="http://openresearchsoftware.metajnl.com/">Journal of Open Research Software</a></span>, or <span class="c1"><a class="c4" href="http://joss.theoj.org/">The Journal of Open Source Software</a></span><span>, provide the citation in the README and on the project website. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We encourage the adoption</span><span> of the </span><span>Software Citation Principles</span><span> (Smith et al. 2016) outlined by the Force11 Software Citation Working Group, noting that currently we do not have a normalized citation structure for code. In some areas, community practices exist; for those areas without, the F11 document provides references for currently used options. Providing citation details is only one part of fully realizing the value of software citations. Citing others’ code and software you use for your research activities, for datasets or analyses, is part of being a good research citizen.</span></p><h3 class="c2 c8" id="h.pzfz7hkzxqus"><span>Preservation/Archiving</span></h3><p class="c2"><span>As noted in </span><span class="c1"><a class="c4" href="#h.tk3e9fopeigk">Publication and Citation</a></span><span>, depositing a stable or final version in an external repository is the minimum option. Meeting most, if not all, of the documentation guidelines and ensuring that the documentation is deposited with the source code provides a more useful archive. If using certain version control systems, such as GitHub, export and archive all of the metadata related to the repository (wikis, issues, etc). </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Containerization is being explored as another archive option (</span><span class="c1"><a class="c4" href="https://daspos.crc.nd.edu/">DASPOS</a></span><span>). Here, the software is provided in a runnable container, preserving the system and dependencies. In some cases, this kind of preservation is not possible due to license constraints related to redistribution (for proprietary software, for example).</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Realistically, black box archiving or a container providing only installed or built software is not as future-proof as providing both the source code and documentation archive and a containerized version. Runnable archives address reproducibility and replicability concerns; they do not address other aspects of code that are of value. Code, maintained in its basic text format, is valuable for education and for other research activities as data. </span></p><h3 class="c2 c8" id="h.rdevd43xfrh4"><span>Credit</span></h3><p class="c2"><span>Authorship and credit for software is overlooked in traditional scholarly spaces. Citation is one method for addressing that. Projects such as </span><span class="c1"><a class="c4" href="http://depsy.org/">depsy</a></span><span>, CASRAI's <a href="http://casrai.org/credit ">CRediT</a>, <a href="http://casrai.org/credit">OntoSoft</a> or Dagstuhl's <a href="https://dagstuhleas.github.io/SoftwareCreditRoles/doc/index-en.html">Software Credit Ontology</a> offer new opportunities for understanding contributions and transitive credit. Often these options are limited to certain programming languages, VCS platforms or package indexes and may require implementation through the software publication platforms. To ensure that the authorship information is available regardless of these current limitations, include an AUTHOR file in the code repository. Indicate major contributions, important contributions to code and other components. </span></p><h3 class="c2 c8" id="h.26zgiufdtz5"><span>Provenance</span></h3><p class="c2"><span>Several of the guidelines outlined throughout this section support provenance efforts (consider publication and versioning, for example). Rather than reiterate those details, we discuss two higher level guidelines here.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For code/software creating or serving data, integrate provenance trace generation into the process. This might mean developing the product in an existing workflow management system (</span><span class="c1"><a class="c4" href="https://www.vistrails.org/index.php/Main_Page">VisTrails</a></span><span>, </span><span class="c1"><a class="c4" href="http://www.taverna.org.uk/">Taverna</a></span><span> or </span><span class="c1"><a class="c4" href="https://kepler-project.org/">Kepler</a></span><span>), provenance capture integrated into code notebooks as workflows (Ma et al. 2017, ECO-OP 2015), or integrating a provenance capture package into the codebase (</span><span class="c1"><a class="c4" href="https://github.com/End-to-end-provenance/RDataTracker">RDataTracker</a></span><span> or </span><span class="c1"><a class="c4" href="https://github.com/recipy/recipy">recipy</a></span><span>). Different domains and research activities may implement traces at different abstraction levels. Whenever possible, we encourage generating original traces with a high level of detail to better support integration across activities as provenance activities mature.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Once provenance traces are implemented within a system, publish those traces through a system supported by the research group, in a domain-specific provenance store, or in a public provenance store (</span><span class="c1"><a class="c4" href="https://provenance.ecs.soton.ac.uk/store/">ProvStore</a></span><span>). Traces, regardless of publication method, should be provided using the </span><span class="c1"><a class="c4" href="https://www.w3.org/TR/prov-overview/">W3C PROV</a></span><span> specification with an ontology reflecting the abstraction model used for generation.</span></p><h3 class="c2 c8" id="h.zgp24m9oygk7"><span>Reproducibility/Replicability</span></h3><p class="c2"><span>Rather than providing any additional guidelines specific to reproducibility and replicability, we simply note that following guidelines for clean code, automation, documentation and publication will provide a solid working baseline for meeting the requirements for a specific publication or community practices.</span></p><h1 class="c2 c8" id="h.g3e5ybizjqjh"><span>Progression, Sustainability and Reusability/Adoption</span></h1><p class="c2"><span>In this section, we will discuss how we can </span><span>best </span><span>place the concepts outlined in the guidelines </span><span>into</span><span> three larger areas of concern</span><span> for research code and software, namely </span><span>progression, sustainability, and reusability. We will mostly be discussing these areas in terms of publicly released research software as </span><span>such products are</span><span> often the main area of concern</span><span>,</span><span> but </span><span>this discussion should not be taken as</span><span> discount</span><span>ing</span><span> the importance of research code.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Before we discuss each topic in detail, we will pause to reflect on some of the gaps in our collective knowledge of the larger geoscience and research development communities. We do not have an understanding of the </span><span>full </span><span>breadth of research code/software activities</span><span>,</span><span> nor of how those activities have been sustained over time</span><span>,</span><span> </span><span>but we do </span><span>have </span><span>a wealth of </span><span>anecdotal evidence based on our own project activities, our interactions with collaborators in other research groups, and our participation at conferences and meetings. From these local viewpoints, we </span><span>know that we </span><span>can’t form a realistic picture of the overall landscape and, without that, it </span><span>will be </span><span>difficult to provide meaningful measures for sustainability and reuse. We risk artificially limiting the discussion to a subset of situations</span><span>,</span><span> or providing recommendations </span><span>based on survivorship bias and </span><span>false equivalencies</span><span>. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>As we have approached this process of developing guidelines, we were struck by the ways in which the limited representations of code and software in research prevented many from seeing how their activities are situated in this ecosystem. </span><span>A</span><span>gain, it is worthwhile to consider, even with largely imperfect knowledge, how our approaches to sustainability, progression and reusability can promote equitable assessment processes and actively support those </span><span>researchers who are </span><span>developing and managing code.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>With that in mind, we offer a number of outstanding questions regarding our research code/software activities.</span></p><p class="c2 c9"><span></span></p><ul class="c14 lst-kix_j4i5j7saczsg-0 start"><li class="c2 c3"><span>How often are projects developing systems from scratch?</span></li></ul><ul class="c14 lst-kix_j4i5j7saczsg-1 start"><li class="c2 c18 c29"><span>How do we define “systems from scratch”?</span></li><li class="c2 c18 c29"><span>Are there material differences in how we assess and adopt systems?</span></li><li class="c2 c18 c29"><span>How do we support incremental improvements in cyberinfrastructure to sustain research activities as well as supporting innovation?</span></li></ul><ul class="c14 lst-kix_j4i5j7saczsg-0"><li class="c2 c3"><span>What are the distributions for the types of research code/software projects currently active?</span></li><li class="c2 c3"><span>How can we evaluate adoption and reuse for…</span></li></ul><ul class="c14 lst-kix_j4i5j7saczsg-1 start"><li class="c2 c18 c29"><span>Library modules/packages?</span></li><li class="c2 c18 c29"><span>Standalone but integratable platforms/frameworks?</span></li></ul><ul class="c14 lst-kix_j4i5j7saczsg-0"><li class="c2 c3"><span>How do we identify and support projects that…</span></li></ul><ul class="c14 lst-kix_j4i5j7saczsg-1 start"><li class="c2 c18 c29"><span>have successfully supported a research activity within their community?</span></li><li class="c2 c18 c29"><span>have transitioned into a successful general product to support research activities across different domains and support activities in private sector efforts?</span></li></ul><ul class="c14 lst-kix_j4i5j7saczsg-0"><li class="c2 c3"><span>Do our funding structures support alternative means of sustaining valuable but niche research software?</span></li></ul><ul class="c14 lst-kix_j4i5j7saczsg-1 start"><li class="c2 c18 c29"><span>Can we apply specific crowdfunding support to other</span><span> projects in our budgets</span><span>, as one time payments or subscription fees?</span></li><li class="c2 c18 c29"><span>Can our organizations accept funds from crowdfunding services, as one time payments or subscription fees?</span></li></ul><p class="c2 c9"><span></span></p><p class="c2"><span>I</span><span>t is clear that there is no one-size-fits-all solution to any sustainability or assessment question given the variations in projects and in research goals. Our discussion here is a renewed call to explore viable and equitable solutions for both concerns.</span></p><h3 class="c2 c8" id="h.b2e39m7jtud5"><span>Progression</span></h3><p class="c2"><span>As described earlier in this document, the initial evaluation efforts relied on NASA’s Technology Readiness Levels to describe progression and maturity. In reframing the effort to focus more on education, we determined that the </span><span>TRL structure did not lend itself well to this approach</span><span>. Born out of a particular management style, the TRL’s rigidity and linearity are out of sync with common development management practices today, and </span><span>with </span><span>practices modified for groups with limited resources. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We therefore suggest a structure that is different from the TRL approach, one that will better address each of the following three main concerns:</span></p><ol class="c14 lst-kix_kpzm26rv33it-0 start" start="1"><li class="c2 c3"><span>The need to align guidelines and development activities within the scope of a grant, that partially or fully funds the activities;</span></li><li class="c2 c3"><span>The need to align long-term projects with continued grant support;</span></li><li class="c2 c3"><span>The need to apply context-driven models of maturity appropriate to different project stages.</span></li></ol><p class="c2 c9"><span></span></p><p class="c2"><span>The first concern acknowledges that many of the concepts outlined in the guidelines </span><span>will be implemented </span><span>during the active development cycle of a grant</span><span>,</span><span> and that </span><span>these guidelines </span><span>may be applied unevenly. For example, a codebase may be configured for continuous integration early on in the development process but never provide high-level documentation at any point during the cycle. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The second </span><span>concern acknowledges </span><span>the complex</span><span>ities of</span><span> project lifecycles, </span><span>and that</span><span> </span><span>different stages in that lifecycle </span><span>may map to </span><span>different </span><span>solicitation requirements (or expectations)</span><span>.</span><span> </span><span>T</span><span>ransitions </span><span>between different </span><span>phase</span><span>s</span><span> </span><span>may also require different expectations.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The third concern acknowledges the importance of </span><span>context-driven </span><span>assessments of </span><span>maturity levels</span><span>, which has been </span><span>noted previously in this document in relation to different kinds of code/software project types. Here</span><span>,</span><span> we are highlighting the need to address maturity expectations at different project phases and with respect to the different products from research code and software. We will be describing these phases in terms of four characteristics: intent, utility, correctness and criticality.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Table 3. Definition of progression characteristics.</span></p><a id="t.465a739bb47f39854dac07fc7c99b5fe5f8cbb68"></a><a id="t.2"></a><table class="c38"><tbody><tr class="c11"><td class="c20" colspan="1" rowspan="1"><p class="c12"><span class="c7">Characteristic</span></p></td><td class="c24" colspan="1" rowspan="1"><p class="c12"><span class="c7">Description</span></p></td><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c7">Values</span></p></td></tr><tr class="c40"><td class="c20" colspan="1" rowspan="2"><p class="c12"><span>Intent</span></p></td><td class="c24" colspan="1" rowspan="2"><p class="c19 c2"><span>D</span><span>escribes the nature of the code or software as internal</span><span>ly</span><span> or externally focused. Internally</span><span> </span><span>focused </span><span>describes</span><span> </span><span>much that we have called</span><span class="c10"> research code, but is not limited to that definition.</span></p></td><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">Internal: product is very specific to the project, heavily integrated with a legacy system or otherwise not intended to be released as reusable code but can be published.</span></p></td></tr><tr class="c28"><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">External: product is a stated deliverable or is intended for adoption/reuse outside of the research group.</span></p></td></tr><tr class="c28"><td class="c20" colspan="1" rowspan="2"><p class="c12"><span class="c10">Utility</span></p></td><td class="c24" colspan="1" rowspan="2"><p class="c19 c2"><span>D</span><span>escribes the function of the code or software in relation to the project’s outputs. </span><span>Is the code </span><span>itself </span><span>a stated deliverable (primary) or does it generate a deliverable (secondary)?</span></p></td><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">Primary: product is one of the project’s stated deliverables or goals. </span></p></td></tr><tr class="c28"><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">Secondary: product is used to generate or support an output of the project but is not, itself, a stated deliverable or goal.</span></p></td></tr><tr class="c28"><td class="c20" colspan="1" rowspan="3"><p class="c12"><span class="c10">Correctness</span></p></td><td class="c24" colspan="1" rowspan="3"><p class="c19 c2"><span>D</span><span>escribes the importance of the correctness of the code’s output</span><span>,</span><span> particularly as it relates to other project outputs.</span><span> </span><span>Correctness</span><span> is not necessarily an indication of bug-free code, more that the code </span><span>meets the </span><span>provided </span><span>specifications.</span></p></td><td class="c33" colspan="1" rowspan="1"><p class="c12"><span>Low</span><span class="c10">: limited need for formal specifications.This does not preclude the inclusion of testing.</span></p></td></tr><tr class="c28"><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">Medium: formal specifications and testing, while not required for research goals, are useful for infusion or community goals.</span></p></td></tr><tr class="c28"><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">High: research goal requires demonstration of correct implementation through formal specifications and testing.</span></p></td></tr><tr class="c35"><td class="c20" colspan="1" rowspan="3"><p class="c12"><span class="c10">Criticality</span></p></td><td class="c24" colspan="1" rowspan="3"><p class="c19 c2"><span>D</span><span>escribes the level of operational support, in part related to project goals and type. </span><span>Criticality pertains to </span><span class="c10">expectations for support as well as actual support, depending on the project.</span></p></td><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">Low: limited or no need for ongoing support or maintenance of the code product or availability (if hosted web application or service).</span></p></td></tr><tr class="c35"><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">Medium: product may receive support at varying intervals but little ongoing support.</span></p></td></tr><tr class="c35"><td class="c33" colspan="1" rowspan="1"><p class="c12"><span class="c10">High: product is expected to be available for internal or external users according to the service level qualities or release cycles, with continued maintenance and support. (This is the equivalent of “operational” in many respects.)</span></p></td></tr></tbody></table><p class="c2 c9"><span></span></p><p class="c2"><span>We encounter difficulties in </span><span>defining progressions </span><span>when</span><span> there is a tension between more traditional understandings of science code and </span><span>the code that</span><span> is more likely to come out of research grants in the broader ecosystem. Realistically, production code is anything that supports a public outcome</span><span>,</span><span> whether that </span><span>outcome </span><span>is a generated data product, a tool or framework to be adopted</span><span>,</span><span> or a web application. </span><span>The</span><span> distinction between internal </span><span>and</span><span> external support, categorized </span><span>above </span><span>as “Intent,” matters to the development of training and assessment tools </span><span>more than it does </span><span>to any functional differences with industry versus research software. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Table 4 outlines a four phase progression model </span><span>for</span><span> </span><span>capturing</span><span> common patterns in research code and software using our four characteristics. </span><span>The model</span><span> is not meant to be comprehensive and, indeed, we can </span><span>imagine</span><span> </span><span>other </span><span>situations where the assigned values may be quite different. </span><span>Instead, the table </span><span>is meant to provide common ground for building relationships between process and project maturity as indicated by the guidelines</span><span>,</span><span> with the kinds of activities pursued as research development</span><span>,</span><span> and as understood from grant solicitations.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Table 4. Context-driven progression.</span></p><a id="t.ec6e4b259ff37f24a0d97a55fc7a9f67e17e870a"></a><a id="t.3"></a><table class="c38"><tbody><tr class="c11"><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c7">Phase</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c7">Intent</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c7">Utility</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c7">Correctness</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c22">Criticality</span></p></td></tr><tr class="c11"><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Proof of Concept</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Internal/External</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Primary</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Low</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Low</span></p></td></tr><tr class="c11"><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Prototype</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">External</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Primary</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">> Medium</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">> Medium</span></p></td></tr><tr class="c28"><td class="c16" colspan="1" rowspan="2"><p class="c12"><span class="c10">Research Production</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Internal</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Secondary</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">High</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span>Low</span><span class="c10">*</span></p></td></tr><tr class="c28"><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Internal </span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Primary</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">High</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">High</span></p></td></tr><tr class="c11"><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Production/Operations</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">External</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">Primary</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">> High</span></p></td><td class="c16" colspan="1" rowspan="1"><p class="c12"><span class="c10">> High</span></p></td></tr></tbody></table><p class="c2"><span>*The generated output may be critical but the code, and the need to continue to support that code, may not be.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Proofs of concept may be released publicly but with limited expectations of community support. One can consider the public activity to be demonstrations or collaborator integration. As such, active operational support is not expected.</span><span> </span><span>Correctness here may be limited to core functionality. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Prototypes have higher expectations than proofs of concept in terms of support (criticality) and correctness</span><span>,</span><span> as these projects are likely to have higher expectations for adoption. </span><span>Prototypes</span><span> are not, however, considered operational products.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Research Production encompasses those projects or codes that are internally focused, i.e. not intended for adoption or reuse on their own, but </span><span>that </span><span>have generated primary research products. Criticality here depends on whether the need </span><span>(and support) </span><span>for those products is ongoing. Correctness of the code is important </span><span>for</span><span> </span><span>generating</span><span> high quality primary products, whether </span><span>those are </span><span>data or analyses. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Production or operational systems are public-facing products with priority placed on both criticality and correctness. These systems have been adopted as reusable components, standalone software or web applications within one or more communities. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>This progression structure more explicitly captures the realities of research development activities. </span><span>Using this structure, we can consider innovative solutions with the understanding that</span><span> additional resources will be required to operationalize those </span><span>early-stage products </span><span>showing promise. We can also more usefully consider </span><span>a </span><span>long-running software and </span><span>its</span><span> need for dedicated, ongoing maintenance to ensure continued successful research outcomes </span><span>for</span><span> those </span><span>who</span><span> rely on </span><span>it</span><span>. </span><span>And we</span><span> can consider the </span><span>utility of other </span><span>codes </span><span>that </span><span>do not become part of any dedicated cyberinfrastructure</span><span>,</span><span> but that </span><span>instead </span><span>produce data and analyses. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>When we describe the </span><span>different progression </span><span>phases as context-driven, we are creating a framework that gives us a flexible and functional method of applying the guidelines in practice, for both training and assessment activities. For training and education, we can now design practical guides for meeting community expectations for divergent goals (with the understanding that code is code, regardless). And for assessment, we can set expectations for our research products early in the funding cycle, with a common understanding to negotiate that space according to the requirements of the solicitation and the relationship of the new research products to existing systems. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>If nothing else, removing languag</span><span>e specific to development process or project maturity from the </span><span>progression </span><span>phases leaves us free to adapt code practices to meet new technical requirements and new methodologies while providing funders or other researchers a means of </span><span>assessing improvements across</span><span> research communities and research development activities. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Development process maturity and project maturity is integrated in this progression model based on the specific assessment requirements of an implementation. That being said, the characteristics do assume certain levels of maturity for both aspects where we expect production systems with high criticality and correctness values to reflect higher maturity levels. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>While the structure of the guidelines implies certain paths to more mature practices within the sections, we have not developed an explicit model here. We will discuss the application of these guidelines and the progression model further in </span><span class="c1"><a class="c4" href="#h.gpan0aidj5ex">Using this guidance for assessment</a></span><span>. </span></p><h3 class="c2 c8" id="h.ekrcac4t6s00"><span>Sustainability</span></h3><p class="c2"><span>The medium- to long-term sustainability of research software projects is often tied to a traditional open source model</span><span>, a</span><span>nd the guidance provided here draws heavily on those kinds of governance and contribution models. That being said, research software and code </span><span>have</span><span> additional expectations as part of the larger scholarly and research community. Traditional open source might be a big hammer for research projects that may not actually </span><span>use</span><span> nails. It is </span><span>of course </span><span>appropriate and necessary to understand the context in which open source activities </span><span>and research </span><span>are </span><span>each </span><span>undertaken</span><span>,</span><span> and adapt these processes and potential funding avenues to reflect different needs. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>To support the different expectations and development activities, we propose a three-tiered approach:</span></p><ul class="c14 lst-kix_jf0gg8kp4t2b-0 start"><li class="c2 c3"><span class="c22">Public Source:</span><span> meets the publication and provenance requirements. Code </span><span>artifacts</span><span> are public with no maintenance expectations.</span></li><li class="c2 c3"><span class="c22">Research Open Source:</span><span> similar to internal open source in that open source practices are used but with no expectations of outside contributions due to community structure,research goals, or research group governance. Critical to support community adoption of the product.</span></li><li class="c2 c3"><span class="c22">Open Source:</span><span> traditional open source with expectations of external contributors.</span></li></ul><p class="c2 c9"><span></span></p><p class="c2"><span>Public Source does not exclude R</span><span>esearch Open Source or Open Source</span><span> and it is possible to meet the Public Source requirements through either Research Open Source or Open Source governance. The key difference is the publication requirement.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Within these three categories, we immediately note the difference between Public Source and the two Open Source categories. Sustainability for Public Source products is most often the responsibility of some external platform, like a domain-specific software repository, a hosted VCS or an institutional archive. The sustainability of the projects developing such products is certainly of concern, but largely out of scope of this discussion.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We can, based on our stakeholder expectations, speak to sustainability for Research Open Source and Open Source activities. We can suggest activities to support the adoption and growth of a piece of research software, as would be required for its longer term viability, </span><span>but there are no guarantees of success. This is as true of the larger </span><span>O</span><span>pen </span><span>S</span><span>ource community (Eghbal 2016) as it is for research software. Realistically, most projects are unlikely to draw from a large pool of potential contributors, </span><span>which </span><span>limit</span><span>s</span><span> their ability to diversify the contributor base beyond the original research group. This </span><span>limitation </span><span>does not mean that the research goals </span><span>will</span><span> not </span><span>be </span><span>met</span><span>,</span><span> nor that the project is not valuable to the audience it is aimed at; it simply means that we cannot expect high levels of external contributions to sustain future development efforts. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We are faced again with a situation that does not provide a single solution. </span><span>Furthermore</span><span>, without a comprehensive understanding of the research development landscape, it is irresponsible to suggest that there is one. But we can consider sustainability in three broad areas, with an eye towards a healthy research development culture and broader impacts.</span></p><h4 class="c2 c8" id="h.ytbqx2fpn94q"><span>Sustainability based on potential</span></h4><p class="c2"><span>For projects in early phases, e.g. proofs of concept or prototypes, sustainability may be more accurately called opportunity, i.e. whether the project has met its research goals and whether it is being managed in such a way to indicate future success. As we described in </span><span class="c1"><a class="c4" href="#h.b2e39m7jtud5">Progression</a></span><span>, one of the goals of these projects is to demonstrate the viability of an implementation. Resources might not be dedicated to supporting activities related to adoption; or, in other cases, adoption is a goal but whether any resources can be used to support adoption will depend on timing. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Sustainability here can mean either additional funding to operationalize the software, or to provide time to allow for community adoption and reuse. </span></p><h4 class="c2 c8" id="h.7c0vnqsr803n"><span>Sustainability of broadly applicable cyberinfrastructure</span></h4><p class="c2"><span>It is telling that, in conversations about research code or software, we speak of certain open source projects as models for adoption. The kinds of projects we talk about, however, are often not good comparisons. </span><span>We point to data formats</span><span>, database systems, visualization libraries or analysis packages, these more general products, as exemplars for any research code/software open source project. These are products that can be adopted outside the research community.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Discussions around sustainability and sustainability plans are also where open source expectations around community management and engagement play a larger role. Outreach for a project requires resources and effort. The guidelines describe the technologies and platform needs to support community activities; the mere existence of those technologies won’t grow a viable community. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Alternative funding models in the open source community include crowdfunding (one time payments or subscription), foundation funding, or corporate patronage (Eghbal 2016). Of course, pursuing funding outside of federal research grants is also resource-intensive. </span></p><h4 class="c2 c8" id="h.yd76uhfdnc1j"><span>Sustainability of research development</span></h4><p class="c2"><span>What about development that cannot be broadly adopted across research communities or beyond? Here, we are talking about t</span><span>hose projects that </span><span>develop domain-specific products such as models, those that develop and maintain citizen science applications for ongoing data collection, or those that provide infrastructure for data or analysis.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>In some cases, a high priority goal is to have the product integrated into an agency’s operational systems. </span><span>Reaching this goal however </span><span>does not guarantee </span><span>that there will be </span><span>ongoing maintenance support.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>The alternative funding models listed for open source projects may be applicable here as well. Anecdotal stories from the open source community, particularly </span><span>stories involving </span><span>patronage (</span><span>monies from a </span><span>foundation, </span><span>whether </span><span>corporate or other</span><span>wise</span><span>), describe limitations on that funding</span><span>,</span><span> suggest</span><span>ing that</span><span> these models may not be effective long-term solutions (Eghbal 2016). If the patronage model revolves around funding specific features </span><span>that are </span><span>required by the patron, one must consider the larger research goals of the project</span><span>,</span><span> both in terms of the available resources for the maintenance of those features</span><span>,</span><span> and </span><span>in terms of </span><span>the philosophical underpinnings of the project. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>It is, then, a question for our research communities as to how we develop sustainable funding models for projects that cannot make the transition from research product to a more widely applicable product, projects that are nonetheless successful in supporting valuable science and research activities.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Finally, concepts recommended for sustainability related to financial support are also valuable within a research group. That is, for groups that rely heavily on potentially transient contributors (like graduate students and post-docs) for development activities, internal project governance and solid documentation can provide some stability for projects and improve the transitions to new research staff. </span></p><h3 class="c2 c8" id="h.hftwhr372i88"><span>Adoption</span><span> and</span><span> Reuse</span></h3><p class="c2"><span>This discussion centers on research software as that set of products aimed for use beyond the original research group. This may be considered a technology infusion problem; here we will consider two aspects that fall under that larger concept: adoption (by consumers) and reuse (by developers). For funders, the distinction is not important; for researchers and research groups, the distinction can help clarify what monitoring and other metrics they may need to collect based on the project’s goals for infusion. We also consider the timing of the infusion signals in the metrics, both in term of the software phase and of the grant cycle. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We don’t have research to describe growth rates or patterns in adoption of open source </span><span>research software as applications or as reusable modules</span><span>. This obviously makes it difficult to provide timeframes for assessing these metrics for a project. In some cases, it may be possible to evaluate some infusion metrics by project grant’s end, for instance, if the release occurred well within the grant. For projects that release software products late in the funding cycle, there is simply no time to grow. This is as true for analytics as it is for altmetrics. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>To provide some flexibility in assessing infusion that is aware of these timing concerns, we offer three general phases (durations are estimated and refer to the time since public release):</span></p><ol class="c14 lst-kix_pe1x2tciq74l-0 start" start="1"><li class="c2 c3"><span>Early phase (<6 months): analytics provides little information, perhaps capturing early adopters or known project collaborators.</span></li><li class="c2 c3"><span>Middle phase (6 months < 18 months): analytics show active growth (adoption and/or reuse) and some sustained use. Potential altmetrics activity.</span></li><li class="c2 c3"><span>Advanced phase (> 18 months): analytics show continued growth, if limited, but sustained active user base (adoption and/or reuse). Altmetrics activity expected.</span></li></ol><p class="c2 c9"><span></span></p><p class="c2"><span>We do not provide any expectations of scale in these phases as these must be considered in terms of the research goals and the communities being served. For example, the adoption of a hydrology model will display different characteristics than would a citizen science data collection web application. These considerations become one more aspect of of our larger context-driven approach, tied to progression, to product type, to research goals, that need to be addressed to support equitable and realistic assessment efforts when addressing “time to science” and ROI from the funder’s perspective. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>Returning to research code specifically, we want to consider adoption and reuse not necessarily of the code as the primary concern but of the dataset generated or analysis performed using that code. Here again, the research code is a secondary output but one that is still very key to the research goals. The code’s impact is tied to the impact of the dataset and it’s this situation where the value of strong cultural practices around software and data citation as well as use and quality analyses using provenance traces (Car 2016) comes to the forefront. How we value those measures for code as secondary outputs is an open question.</span></p><h1 class="c2 c8" id="h.gpan0aidj5ex"><span>Using this guidance for assessment</span></h1><p class="c2"><span>We have touched upon assessment throughout this document as we describe some of the ways that development intersects with our research environments and practices. We have touched on the various stakeholder expectations, development progression and maturity, sustainability and infusion, and the guidelines themselves. These provide a foundation for implementing an assessment instrument when taking into account the different stakeholder needs and different community needs, although providing an explicit model to address each is outside the scope of this document.</span></p><p class="c2 c9"><span></span></p>
<p class="c2"><span>We err here in favor of externally verifiable indicators for any assessment task, whether that assessment is done in-house or through a more formal process. In some cases, these indicators will support other sustainability and adoption goals. For most assessment, such as those done by a funding agency, sustainability and adoption/reuse are often the only metrics of interest for evaluating a piece of research software.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>There are also recommended practices, invisible to this kind of external review, for improving code quality. Code reviews, pair programming, Agile or Lean management practices—these activities can be difficult to discern from the project </span><span>artifacts</span><span> that we consider in the guidance above. While there are practices described there that are commonly used in Agile management, for example, we are not advocating any one management style. We are, rather, noting that there are processes involved in the development of these products that are difficult or impossible to assess at a project’s end or at all. </span></p><p class="c2 c9"><span></span></p><p class="c2"><span>For those wishing to use these guidelines for assessing research products for adoption or reuse, we n</span><span>ote that there is no one measure for fitness of use provided by these guidelines. Each researcher or research group must weigh their own project’s goals and resources (every dependency is a maintenance cost) when evaluating these products for reuse. We do not recommend the use of these guidelines alone when evaluating projects for adoption as development process maturity is not the most relevant measure of whether a research product is a worthwhile tool to add to your research workflow. Each project, and project phase, requires its own analysis of potential reuse/adoption, an analysis that can be aided by the discussion here and by metrics such as those found in the Reuse Readiness Levels (RRLs) put forth by NASA’s Earth Science Data Systems’ Software Reuse Working Group (2010).</span></p>
<p class="c2 c9"><span></span></p><p class="c2"><span>
Returning to our scenarios, we would like to address concerns around just what is likely to be assessed. We are not speaking for any funders in this; however, we all recognize that it is simply not feasible to review every code/software product and that no one assessment effort will address the different stakeholder needs and expectations. From the ESIP community, we place efforts to improve the technical capabilities of our researchers and research groups as key to approaching the underlying concerns found in the most frequently discussed needs for assessment: “rigor” in code-dependent research activities (from the research community) and return on investment or “time to science” (from the funding agencies).
</span></p>
<h1 class="c2 c8" id="h.aep6uf41nqbc"><span>Conclusions</span></h1><p class="c2"><span>These, or any code/software, guidelines alone reflect only one aspect of a research project involving code or software development. We never want to lose sight of the research goals in favor of a purely technical answer. That is, from the viewpoint of a developer, the fastest way to reduce “time to science” and one that disregards the necessities of our research communities. Our research activities rely heavily on code and, while code is code and these guidelines reflect that understanding, we do have additional expectations of this code because it is written for research activities. We expect rigor, inspection, trust. To promote that, we need to support our communities and provide the education and training to produce, manage and understand the code/software throughout the research lifecycle.</span></p><p class="c2 c9"><span></span></p><p class="c2"><span>We will end with one final note. This document is as much about creating a shared understanding around what it takes to produce research code/software. We are not advocating that every researcher must also be a trained software engineer. It is more important that our researchers have the technical literacy and knowledge to understand how code supports their research, the limitations of code and the ways in which it can fail. Those that continue to code and continue to learn have our support. But those that don’t continue on that path may still have active roles in shaping the technologies that we create. That may include product owners (for those working in an Agile system), project managers, domain researchers, principal investigators, or federal program officers. When we consider the ways in which producing code and software has grown more complex, in terms of technologies and in the number of skillsets required, we cannot ignore the very real need to better support interdisciplinary teams and more collaborative approaches to research development. We start by building these shared understandings of our research development activities so that we can better support our research communities.</span></p><h1 class="c2 c8" id="h.qyi0cpttkurs"><span>Contributors</span></h1><p class="c2"><span class="c22">ESIP</span></p><p class="c2"><span>Valerie Toner (NOAA)</span></p><p class="c2"><span>James Gallagher (OPeNDAP)</span></p><p class="c2"><span>Robert Downs (CIESIN)</span></p><p class="c2"><span>Anne Wilson (LASP)<br>Kim Kokkonen (LASP)</span></p><p class="c2"><span>Joshua Elliott (LASP)</span></p><p class="c2"><span>Chris Pankratz (LASP)<br>Ruth Duerr (Ronin Institute)</span></p><p class="c2"><span>Ryan Bowe (Geospatial Metadata)</span></p><p class="c2"><span>Chris Kotfila (Kitware)</span></p><p class="c2"><span>Leslie Hsu (USGS)</span></p><p class="c2"><span>Sam Silva (MIT)</span></p><p class="c2"><span>Bon Simons (NOAA)</span></p><p class="c2"><span>Ben Barton (University of Alaska Fairbanks)</span></p><p class="c2"><span>Greg Janée (UC Santa Barbara)</span></p><p class="c2 c9"><span></span></p><p class="c2"><span class="c22">EarthCube/ESIP Software Assessment Workshop, June 1-3, Boulder, CO</span></p><p class="c2"><span>Anne Wilson (LASP)</span></p><p class="c2"><span>Emily Law (JPL NASA)</span></p><p class="c2"><span>Ken Keiser (UAL-H)</span></p><p class="c2"><span>Aleksandar Jelenak (The HDF Group)</span></p><p class="c2"><span>Nathan Hook (NCAR-CISL)</span></p><p class="c2"><span>Ward Fisher (UCAR/Unidata)</span></p><p class="c2"><span>Bob Arko (UNOLS/R2R)</span></p><p class="c2"><span>Adam Shephard (WHOI)</span></p><p class="c2"><span>Matt Tricomi (Xentity)</span></p><p class="c2"><span>Lindsay Powers (The HDF Group)</span></p><p class="c2"><span>Arika Virapongse (University of Florida)</span></p><p class="c2 c9"><span></span></p><p class="c2"><span class="c22">Community</span></p><p class="c2"><span>Neil P. Chue Hong, Software Sustainability Institute, University of Edinburgh</span></p><p class="c2 c9"><span></span></p>
<h1 class="c2 c8" id="h.iy5k73rbewd1"><span>Acknowledgements</span></h1><p class="c2"><span>Special thanks to Jennifer Ast for editorial support and Anne Wilson for her patience while rubberducking. </span></p><p class="c2 c9 c27"><span class="c37"></span></p><div><p class="c2 c9 c41"><span></span></p></div>
</body>
</html>