VPTS calculated relative to ground height

[adokter]

Hi @peterdesmet, I'm looking into implementing profiles calculated relative to ground level instead of sea level. When grouping radar pixels into profile height bins, the height of a radar pixel will be calculated relative to the ground using a digital elevation map, instead of relative to sea level. In regions with topography, height relative to sea level will no longer be well-defined.

I'm wondering how to fit this into the VPTS CSV format, I see a few options:

1. relax the requirement that height has to be relative to sea level, and store heights relative to ground level in the same column; however in that case it becomes unclear from the VPTS CSV itself how height is defined, therefore not ideal.
2. add an additional (optional) height_agl column for this specific use case, not including a height column. Current specification has height as a required column, so then the requirement should change to one of height or height_agl.
3. add an additional (optional) height_agl column for this specific use case, but still calculate height as the average height above sea level for the pixels in each height_agl bin, such that it is always has a value as currently required. But then height will no longer be regularly spaced, breaking things in e.g. bioRad.
4. add an additional categorical height_reference column, that specifies the reference point for the height definition. Could be one of "ground", "sea", "antenna". If height_reference column is absent, it defaults to sea_level.

Any thoughts or preferences?

[peterdesmet]

Can you clarify in the third option if height would no longer be evenly spaced within a radar (40, 240, 480, 500, …) or when comparing radars (because they start at a different height)?

[adokter]

within a radar - the spatial sampling for different height layers isn't always the same, and therefore you get irregularly spaced layers (relative to sea level) like your example (40, 240, 480, 500, …). Also it wouldn't be very accurate. Say the terrain around a radar is a linear slope from 0-1000m, then the 0-100 m bin relative to ground level would correspond on average to a 500m bin relative to sea level, but with a large variance around that mean from 0-1000.

What about the fourth option I just added to #49 (comment)?

[adokter]

@peterdesmet any objections to me implementing option 4) above in bioRad and vol2bird? Would you want me to always add an additional height_reference column in new releases, or only when a profile is not referenced to sea level (the current default). I think adding it by default is cleaner, but let me know if that could break anything on your end. Could you add the new (optional) field to the documentation?

[jshamoun]

Not sure if this message reaches everyone. I think including it by default is much cleaner and will reduce risk of misinterpretation.

[peterdesmet]

@adokter the height_reference column is probably the most backwards compatible solution. Some questions and comments:

1. How do you plan to combine data with a different value for height_reference? E.g. can you convert data with height_reference: antenna to height_reference: sea or is that not planned/supported?

2. Are we fine with the 3 values? Can you give me the definitions?

   • sea: height above mean sea level (?)
   • ground
   • antenna: is that the same as radar_height?

3. Where in the order of fields do we want to add this column? I suggest directly after height

radar,datetime,height,u,v,w,ff,dd,sd_vvp,gap,eta,dens,dbz,dbz_all,n,n_dbz,n_all,n_dbz_all,rcs,sd_vvp_threshold,vcp,radar_latitude,radar_longitude,radar_height,radar_wavelength,source_file

4. As this is a change to the standard, we should create a new release. I suggest a minor release: VPTS CSV 1.1

5. Software like bioRad will have to be able to read VPTS 1.0 and VPTS 1.1 data. I'd also suggest to always upgrade to VPTS 1.1, which means adding the height_reference column and setting it to sea for VPTS 1.0 data.

Ping @CeciliaNilsson709 @YohanSassi

[bart1]

I'm try to completely comprehend the change here I might misunderstand things.

Would height reference antenna not be very similar to sea except for the addition of the antenna height above sea level? I'm not sure how the calculations would differ, is is it that the antenna version does not account for the curvature of the earth? If there essentially the same would it not be better to directly reflect this in the height column?

For height reference ground the elevation model used could have a strong influence also if you want to translate the measurements back to actual heights above sea level. It would be good to retain this somehow, I guess the vp attributes could be one way but then we loose the information in the vpts case.

Would it be better to call it terrain instead of ground? Somehow terrain makes me think more of the whole elevation map in the domain of a radar.

[CeciliaNilsson709]

Just so I understand correctly, this new height value would then be new altitude bins that are relative to the ground at each point, and therefore not "even" across the radar volume? That sounds both cool and like a big change!

I think there might be some risk for confusion with older data (since the height_reference column would not be included there). And maybe make it more difficult to compare different data, because there wouldn't be a way to recalculate it back to sea level once you have a vertical profile right?

But still sounds like an interesting variable to have!

Maybe worth discussing in an upcoming meeting?

[adokter]

@CeciliaNilsson709 that's right, the height of each pixel is defined relative to the local terrain. It's mostly relevant for habitat association / take-off studies, shortly after take-off it often makes more sense to define the profile relative to the ground.

@CeciliaNilsson709 for older data without the height_reference column we can assume it's relative to sea level, as we introduce the new options only now. I can adjust bioRad such that it will recognize both and add the height_reference column for older data that is missing that column

@jshamoun agree on including it always going forward

@bart1 antenna and sea level are closely related, but the difference shows up especially when generating profiles for multiple radars that have different antenna heights. In the "sea" case you make sure the height bins match exactly, in the "antenna" case the height bins no longer align, but there are sometimes advantages in terms of making sure the number of samples is similar for different heigh bins. It kind of depends on what you want to do, which is why I thought including both is worth it.

@bart1 My reasoning so far has been to leave it to the user which digital elevation map to use, it's really a decision by the analyst how you do this. I have added function add_param() in adokter/bioRad#748, that effectively maps data from a map as a new param to each sweep, so it becomes part of the pvol. So the source of the digital elevation map isn't tracked.

[adokter]

@peterdesmet responses to #49 (comment)

1. not straightforward to combine vpts data with different references. Option "ground" cannot be converted into one of the others, potentially you could do some interpolation to align "antenna" and "sea" profiles. My vote would be to not allow combinations for now.

2. definitions:

   • sea: height above mean sea level in m
   • ground: height above the ground height of the local terrain in m
   • antenna: height above the antenna height of the radar in m, as defined in column radar_height.

yes the heigh reference for antenna is identical to the value given in radar_height. Let me know if you prefer a different name than "antenna", like "radar" or "radar_height"

3. after height works for me

4. sounds good

5. agreed

[adokter]

@fyiguo could you please review this conversation as well

[fyiguo]

@bart1 antenna and sea level are closely related, but the difference shows up especially when generating profiles for multiple radars that have different antenna heights. In the "sea" case you make sure the height bins match exactly, in the "antenna" case the height bins no longer align, but there are sometimes advantages in terms of making sure the number of samples is similar for different heigh bins. It kind of depends on what you want to do, which is why I thought including both is worth it.

I'm still a bit confused about the reasoning behind keeping both "sea" and "antenna" profiles. If height bins are defined relative to antenna (say at 100m intervals), they can still be aligned right? It's just that the same height bin might refer to different altitude above sea level depending on the antenna height, kind of similar to when comparing AGL profiles for multiple radars.

Also, looks like all antenna towers in the US are less than 35m in height, smaller than the typical height bin (100m) used for generating vertical profiles. So it probably wound't matter that much when rounded?

Finally, it also appears that the radar_height column in the old vertical profiles refers to the height of radar only, not the antenna height (antenna_height = radar_height + tower_height). See the NEXRAD metadata for KOKX as an example (elevation in ft)

[adokter]

I'm still a bit confused about the reasoning behind keeping both "sea" and "antenna" profiles.

Let's assume radar A has an antenna level at 210 m asl and radar B at 190 asl, and say we define altitude bins of 100 m width. height encodes the bottom of the height bin. And let's assume the antenna profile is such that we have about 10 data points per meter of altitude. In that scenario you get different sample sizes for the height bins, like in this table indicating the number of data points for each layer.

height	sea A	antenna A	sea B	antenna B
0	0	1000	0	1000
100	0	1000	100	1000
200	900	1000	1000	1000
300	1000	1000	1000	1000

So even though you can convert an "antenna" profile into a "sea" profile by adding the antenna height to the height column, the profiles will still be different, because the altitude bin specification is different.

An advantage of the "sea" profile is that for all radars, the bins will be aligned and cover the same altitudes relative to sea level, but the downside is that the sample size for the lowest bin especially will vary, depending on the antenna height and where it falls within the height bin. It depends what is preferred, therefore I think adding both as an option is worth it.

it also appears that the radar_height column in the old vertical profiles refers to the height of radar only

It shouldn't - but unfortunately the rsl library we use to read nexrad data uses an external file with hard-coded antenna heights and locations, see https://github.com/adokter/vol2birdR/blob/main/inst/librsl/wsr88d_locations.dat, instead of reading that information from the file metadata. I'm not sure why that choice was made, potentially older data don't have the location metadata. @fyiguo could you start an issue here to document mismatches for other stations.

[CeciliaNilsson709]

Thanks for explaining Adriaan!

Still thinking about how to implement this without causing too much confusion. For data going forward the hight_reference column will make it clear what kind of data it is, but we do now have large archives of VPTS data that are missing this column, so I wonder if people using that data would then be confused about which kind it is. Even if biorad can handle it we would still need to clearly indicate it for the "old" data, maybe this is something @peterdesmet has thoughts about when he is back from his trip.

I also assume that this will be an additional option while processing right, and won't change the default? For the altitude work we are doing right now we are quite keen on keeping it over sea level, and of course the more interoperable data that is available the better 😄

[bart1]

I do understand the need idea for "ground", it would be nice to have some recording of the elevation model used in the data besides what the user documents itself.

For sea vs antenna would it not be more general to be able to provide a minheight where the vp calculations start, currently you specify a number of height bins and their size (n_layer & h_layer). But the sequence always starts from 0, having a minheight would provide some extra flexibility to for example start at ground height (avoiding variation in tower height) or make sure bins are still aligned between radars (e.g. use 50 meter bins and round the antenna height down to the nearest 50 meters).

[fyiguo]

Agree that the differences largely stem from rounding and depend on the height bin size. In the example @adokter provided, the range correction based on these four profiles will have different assumptions:

sea A: no birds below 200m asl
antenna A: no birds below 210m asl
sea B: no birds below 100m asl
antenna B: no birds below 190m asl

So even though the actual elevation differences between radar A (210m) and B (190m) is only 20m, a "sea" profile using a height bin of 100m will exaggerate such differences, whereas the "antenna" profile will not have such problem.

Let's assume radar A has an antenna level at 210 m asl and radar B at 190 asl, and say we define altitude bins of 100 m width. height encodes the bottom of the height bin. And let's assume the antenna profile is such that we have about 10 data points per meter of altitude. In that scenario you get different sample sizes for the height bins, like in this table indicating the number of data points for each layer.

height sea A antenna A sea B antenna B
0 0 1000 0 1000
100 0 1000 100 1000
200 900 1000 1000 1000
300 1000 1000 1000 1000

radar_height mismatch issue reported in: adokter/vol2birdR#144