`overlapped-lists` feature causes parser errors to `xsi:nil` fields after array fields

[weiznich]

We have a rather complex XML parser written with quick-xml that started failing some tests while trying to update from quick-xml 0.39.0 to 0.39.1. The old version did successful parse the document, the new version returned an error.

I minimized both the data and the code down to the following example:

pub mod bml {
    use super::*;

    #[derive(Debug, Clone, serde::Deserialize)]
    pub struct BoreholeType {
        #[serde(rename = "boreholeSegment")]
        pub borehole_segment: Vec<bml::BoreholeSegmentPropertyType>,
        #[serde(rename = "drillingProcess")]
        pub drilling_process: Option<Vec<DrillingProcess>>,
    }

    #[derive(Debug, Clone, serde::Deserialize)]
    pub struct BoreholeSegmentType {}

    #[derive(Debug, Clone, serde::Deserialize)]
    pub struct DrillingProcess {
        #[serde(rename = "DrillingProcess")]
        pub drilling_process: bml::DrillingProcessType,
    }

    #[derive(Debug, Clone, serde::Deserialize)]
    pub struct DrillingProcessType {
        #[serde(rename = "to")]
        pub to: gml::LengthType,
    }

    #[derive(Debug, Clone, serde::Deserialize)]
    pub struct BoreholeSegmentPropertyType {
        #[serde(rename = "BoreholeSegment")]
        pub borehole_segment: bml::BoreholeSegmentType,
    }
}

pub mod gml {

    #[derive(Debug, Clone, serde::Deserialize)]
    pub struct LengthType {
        #[serde(rename = "$value")]
        pub value: Option<f64>,
    }
}

fn main() {
    let input = r#"
<?xml version='1.0' encoding='UTF-8'?>
    <bml:Borehole xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.opengis.net/gml/3.2 http://schemas.opengis.net/gml/3.2.1/deprecatedTypes.xsd http://www.infogeo.de/boreholeml/3.0 https://www.infogeo.de/boreholeml/3.0/BoreholeML.xsd" xmlns:gml="http://www.opengis.net/gml/3.2" xmlns:bml="http://www.infogeo.de/boreholeml/3.0">
      <bml:boreholeSegment>
        <bml:BoreholeSegment>
          <bml:from>0.0</bml:from>
        </bml:BoreholeSegment>
      </bml:boreholeSegment>
      <bml:drillingProcess xsi:nil="true"/>
    </bml:Borehole>
"#;
    let res = quick_xml::de::from_str::<bml::BoreholeType>(&input);
    assert!(res.is_ok(), "Error: {}", res.unwrap_err());
}

With that I run git bisect against all revisions between the 0.39.0 and the 0.39.1 tag. This reported 210d3e1 as the first failing commit.

For me this looks like something around the parsing of xsi:nil="true" must have changed there as the parser now tries to parse inner fields of a null value.

[Mingun]

That change seems to not be the root cause (it just exposed another problem), the self_closed test also failed before 210d3e1

/// Regression test for https://github.com/tafia/quick-xml/issues/953.
mod issue953 {
    use super::*;

    #[derive(Debug, Deserialize)]
    struct BoreholeType {
        #[serde(rename = "boreholeSegment")]
        borehole_segment: Vec<()>,

        #[serde(rename = "drillingProcess")]
        drilling_process: Option<DrillingProcess>,
    }

    #[derive(Debug, Deserialize)]
    struct DrillingProcess {
        #[serde(rename = "DrillingProcess")]
        drilling_process: (),
    }

    // success before 210d3e1d460a4c63e13d455b0b15623b77fa669c
    // failed since 210d3e1d460a4c63e13d455b0b15623b77fa669c
    #[test]
    fn open_close() {
        let input = r#"
        <Borehole xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
          <boreholeSegment> </boreholeSegment>
          <drillingProcess xsi:nil="true"/>
        </Borehole>
        "#;
        let _: BoreholeType = from_str(&input).unwrap();
    }

    // failed always
    #[test]
    fn self_closed() {
        let input = r#"
        <Borehole xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
          <boreholeSegment/>
          <drillingProcess xsi:nil="true"/>
        </Borehole>
        "#;
        let _: BoreholeType = from_str(&input).unwrap();
    }
}

Changing Vec<()> to () fixes the problem, so this is somehow related to process arrays. Tests failed only when overlapped-lists feature is enabled. When the feature is not enabled, both tests are always succeeded.

[Mingun]

I found the root case. The error was introduced when switch to namespace=aware reader in deserializer in b8ce862 (part of #850, included in 0.37.3).

When overlapped-lists feature is active, we buffer events until we get a closing tag. When close tag is encountered, we pop all namespace bindings that was introduced by it's opening tag. And when we rewind buffered events, that namespace bindings unavailible, so xsi:nil does not threated in special manner, introduced in #850.

Before #598 we did not clear bindings at </boreholeSegment> close tag that may be introduced in <boreholeSegment> open tag, that is why open_close test passed:

Before #598

Event	Level	Description
<Borehole xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">	1	xsi binding added at level 1
<boreholeSegment>	2	Nothing added, but nesting level increased
</boreholeSegment>	2	Nothing removed, but also nesting level does not decreased. xsi binding still available
<drillingProcess xsi:nil="true"/>	2	Nothing added, nesting level not changed, event buffered. xsi binding still available
</Borehole>	1	Nesting level decreased and nothing removed, because we pop bindings from <boreholeSegment> tag. Event buffered. xsi binding still available
<drillingProcess xsi:nil="true"/>	1	Event rewinded, xsi binging still available
</Borehole>	0	Event rewinded, nesting level decreased and xsi binding removed, because we pop bindings from <Borehole> tag
EOF	0	Parsing done

After #598

Event	Level	Description
<Borehole xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">	1	xsi binding added at level 1
<boreholeSegment>	2	Nothing added, but nesting level increased
</boreholeSegment>	1	Nothing removed, nesting level decreased. xsi binding still available
<drillingProcess xsi:nil="true"/>	1	Nothing added, nesting level not changed, event buffered. xsi binding still available
</Borehole>	0	Nesting level decreased and xsi binding removed, because we pop bindings from <Borehole> tag
<drillingProcess xsi:nil="true"/>	0	Event rewinded, xsi binging not available
</Borehole>	0	Event rewinded, nesting level note changed because of saturating_sub and nothing removed, because there are no bindings
EOF	0	Parsing done