duplicate statements across named graphs

[VladimirAlexiev]

(cc @nikolatulechki)
Split from this issue, which shows duplicate statements in Nordic44:

• duplicate type statements across named graphs Sveino/Inst4CIM-KG#164

This query shows that there are a lot more duplicates in Nordic44+Telemark120:

PREFIX cim: <https://cim.ucaiug.io/ns#>
select ?p ?o ?files (count(*) as ?count) {
  {select ?s ?p ?o (count(*) as ?rows) {
    graph ?g1 {?s ?p ?o}
    graph ?g2 {?s ?p ?o}
    filter(str(?g1)<str(?g2))
  } group by ?s ?p ?o}
  values (?rows ?files) {(1 2) (3 3) (4 6) (5 10) (6 15) (7 21)}
} group by ?p ?o ?files order by ?p ?o

The values list are the "triangle numbers". Eg if a triple is duplicated in 3 files (graphs) g1,g2,g3, the query will return it 3 times for these combinations of graphs: g1,g2; g1,g3; g2,g3.

p	o	files	count
rdf:type	http://www.opengis.net/ont/geosparql#Feature	2	1
rdf:type	cim:ACLineSegment	3	35
rdf:type	cim:ActivePowerLimit	2	5
rdf:type	cim:ActivePowerLimit	3	1
rdf:type	cim:BaseVoltage	2	2
rdf:type	cim:BaseVoltage	3	1
rdf:type	cim:Bay	2	1
rdf:type	cim:Breaker	2	42
rdf:type	cim:BusbarSection	3	7
rdf:type	cim:BusbarSection	2	8
rdf:type	cim:ConformLoad	2	50
rdf:type	cim:ConnectivityNode	2	183
rdf:type	cim:ConnectivityNode	15	1
rdf:type	cim:ControlArea	2	4
rdf:type	cim:CurrentLimit	2	532
rdf:type	cim:Disconnector	2	41
rdf:type	cim:EquivalentInjection	3	3
rdf:type	cim:Fuse	2	29
rdf:type	cim:GeneratingUnit	3	80
rdf:type	cim:GeneratingUnit	2	1
rdf:type	cim:GeographicalRegion	2	1
rdf:type	cim:LinearShuntCompensator	2	9
rdf:type	cim:LoadBreakSwitch	2	2
rdf:type	cim:NameType	2	1
rdf:type	cim:NameTypeAuthority	2	1
rdf:type	cim:NonConformLoad	2	13
rdf:type	cim:OperationalLimitSet	2	1
rdf:type	cim:PetersenCoil	3	2
rdf:type	cim:PowerTransformer	3	4
rdf:type	cim:PowerTransformerEnd	2	8
rdf:type	cim:RatioTapChanger	2	16
rdf:type	cim:RegulatingControl	2	20
rdf:type	cim:SubGeographicalRegion	2	10
rdf:type	cim:Substation	3	1
rdf:type	cim:Substation	2	1
rdf:type	cim:SynchronousMachine	2	80
rdf:type	cim:SynchronousMachine	3	1
rdf:type	cim:TapChangerControl	2	13
rdf:type	cim:Terminal	3	569
rdf:type	cim:Terminal	2	118
rdf:type	cim:VoltageLevel	3	1
rdf:type	cim:VoltageLevel	2	1
rdf:type	cim:VoltageLimit	2	184
rdf:type	https://cim4.eu/ns/nc#CircuitShare	2	2
rdf:type	https://cim4.eu/ns/nc#InServiceAction	2	1
rdf:type	https://cim4.eu/ns/nc#LoadAction	2	1
rdf:type	https://cim4.eu/ns/nc#OrdinaryContingency	2	1
rdf:type	https://cim4.eu/ns/nc#OrdinaryPowerTransferCorridor	2	1
rdf:type	https://cim4.eu/ns/nc#RemedialActionScheme	2	3
rdf:type	https://cim4.eu/ns/nc#RotatingMachineAction	2	2
rdf:type	https://cim4.eu/ns/nc#SchedulingArea	2	2
rdf:type	https://cim4.eu/ns/nc#StageTrigger	2	3
rdf:type	https://cim4.eu/ns/nc#StaticPropertyRange	2	3
rdf:type	https://cim4.eu/ns/nc#TopologyAction	2	4
cim:ActivePowerLimit.normalValue	1600	2	1
cim:BaseVoltage.nominalVoltage	300	2	1
cim:BaseVoltage.nominalVoltage	420	2	1
cim:Bay.VoltageLevel	urn:uuid:681a21b6-5a55-11eb-a658-74e5f963e191	2	1
cim:ConnectivityNode.ConnectivityNodeContainer	urn:uuid:1f4fabd4-8c57-c245-b107-4e64c9007ae3	2	1
cim:EquivalentInjection.r	3.16799999E-1	2	1
cim:EquivalentInjection.r	4.3445E-1	2	1
cim:EquivalentInjection.x	7.4644242E-2	2	1
cim:EquivalentInjection.x	4.3445E0	2	1
cim:IdentifiedObject.description	Base Voltage 132 kV	2	1
cim:IdentifiedObject.description	Nedenes 400V Low Voltage Cable 1 Bay	2	1
cim:IdentifiedObject.description	Nordic	2	1
cim:IdentifiedObject.description	PTC 420RINGHALS-HALDEN LimitSet for direction Halden to Ringhals	2	1
cim:IdentifiedObject.description	Secondary Substation LV	2	1
cim:IdentifiedObject.description	Secondary Substation Nedenes	2	1
cim:IdentifiedObject.mRID	1f19556f-4604-8545-b930-0d96b163c57a	2	1
cim:IdentifiedObject.mRID	1f4fabd4-8c57-c245-b107-4e64c9007ae3	3	1
cim:IdentifiedObject.mRID	2dd90159-bdfb-11e5-94fa-c8f73332c8f4	2	1
cim:IdentifiedObject.mRID	2dd90169-bdfb-11e5-94fa-c8f73332c8f4	2	1
cim:IdentifiedObject.mRID	2dd90174-bdfb-11e5-94fa-c8f73332c8f4	3	1
cim:IdentifiedObject.mRID	2dd9048c-bdfb-11e5-94fa-c8f73332c8f4	2	1
cim:IdentifiedObject.mRID	33666962-c2f9-4f6b-af5a-2ef1982ac282	2	1
cim:IdentifiedObject.mRID	3d51330c-7528-4efb-8edc-e61dcc145ac4	2	1
cim:IdentifiedObject.mRID	62585c75-b020-614c-89a1-fec7c118c3fa	2	1
cim:IdentifiedObject.mRID	681a2179-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a21b6-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a26cf-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a26db-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a26f3-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a2a13-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a2a7f-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a2fcb-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a2fce-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a2fd8-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a2fdd-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a2fe0-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a2fea-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a3087-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a308a-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a3094-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a309b-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a309e-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	681a30a8-5a55-11eb-a658-74e5f963e191	2	1
cim:IdentifiedObject.mRID	92d33b99-7034-11eb-a65a-74e5f963e191	2	1
cim:IdentifiedObject.mRID	92d33bcc-7034-11eb-a65a-74e5f963e191	2	1
cim:IdentifiedObject.mRID	92d33be4-7034-11eb-a65a-74e5f963e191	2	1
cim:IdentifiedObject.mRID	ab5aff23-cc0c-40a9-ab1b-7c113822358e	2	1
cim:IdentifiedObject.mRID	b1b4714b-a160-7742-84d5-d6486d776c94	2	1
cim:IdentifiedObject.mRID	c70c1ea0-c58b-624b-ab6a-0072ad2eea25	2	1
cim:IdentifiedObject.mRID	c9d5445c-9ad6-3849-9e02-08762805826b	2	1
cim:IdentifiedObject.mRID	cb837454-5c66-d341-be63-d0c044e5fd3c	2	1
cim:IdentifiedObject.mRID	cf484bc9-2ef9-4116-a5a9-9a5db3e09df9	2	1
cim:IdentifiedObject.mRID	d1afcb93-f7d9-4e93-8c91-bbeb35f2a322	2	1
cim:IdentifiedObject.mRID	ed8865cf-8b36-42ce-acdc-0627b5f55802	2	1
cim:IdentifiedObject.mRID	f17695af-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f17695c3-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f17695cd-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f1769609-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f1769619-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f176965f-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f1769670-9aeb-11e5-91da-b8763fd99c5f	3	1
cim:IdentifiedObject.mRID	f1769699-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f17696a3-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f17696b3-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f17696cf-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:IdentifiedObject.mRID	f6ab9ccd-10fa-4531-8427-763e11163b28	2	1
cim:IdentifiedObject.mRID	f802a83f-b9fb-48fb-9dde-0f5132e5f886	2	1
cim:IdentifiedObject.name	AC-132kV	2	1
cim:IdentifiedObject.name	APL STABILITY	2	1
cim:IdentifiedObject.name	ARENDAL	3	1
cim:IdentifiedObject.name	ARENDAL 420	3	1
cim:IdentifiedObject.name	FI1 SGR	2	1
cim:IdentifiedObject.name	LS PTC 420RING-HALDEN F HA	2	1
cim:IdentifiedObject.name	NEDENES	2	1
cim:IdentifiedObject.name	NEDENES 04LC1	2	1
cim:IdentifiedObject.name	NEDENES 04	2	1
cim:IdentifiedObject.name	NO	2	1
cim:IdentifiedObject.name	NO1 SGR	2	1
cim:IdentifiedObject.name	NO2 SGR	2	1
cim:IdentifiedObject.name	NO3 SGR	2	1
cim:IdentifiedObject.name	NO4 SGR	2	1
cim:IdentifiedObject.name	NO5 SGR	2	1
cim:IdentifiedObject.name	SE1 SGR	2	1
cim:IdentifiedObject.name	SE2 SGR	2	1
cim:IdentifiedObject.name	SE3 SGR	2	1
cim:IdentifiedObject.name	SE4 SGR	2	1
cim:OperationalLimit.OperationalLimitSet	urn:uuid:f802a83f-b9fb-48fb-9dde-0f5132e5f886	2	1
cim:OperationalLimit.OperationalLimitType	urn:uuid:df0cea8a-8e5a-4254-a040-410fa4797154	2	1
cim:OperationalLimitSet.Terminal	urn:uuid:2dd9039f-bdfb-11e5-94fa-c8f73332c8f4	2	1
cim:PowerTransformer.isPartOfGeneratorUnit	false	2	4
cim:SubGeographicalRegion.Region	urn:uuid:2dd9048c-bdfb-11e5-94fa-c8f73332c8f4	2	10
cim:Substation.Region	urn:uuid:f176965f-9aeb-11e5-91da-b8763fd99c5f	3	1
cim:Substation.Region	urn:uuid:f176965f-9aeb-11e5-91da-b8763fd99c5f	2	1
cim:SynchronousMachine.r	4.3445E-1	2	1
cim:TransformerEnd.grounded	false	2	8
cim:VoltageLevel.BaseVoltage	urn:uuid:2dd90159-bdfb-11e5-94fa-c8f73332c8f4	3	1
cim:VoltageLevel.BaseVoltage	urn:uuid:9598e4a0-67e5-4ad7-879c-c85a1f63159c	2	1
cim:VoltageLevel.Substation	urn:uuid:681a2179-5a55-11eb-a658-74e5f963e191	2	1
cim:VoltageLevel.Substation	urn:uuid:f1769670-9aeb-11e5-91da-b8763fd99c5f	3	1

• 1 ConnectivityNode is in 15 files (!!)
• 1 Substation (ARENDAL ) is in 3 files, and 1 (NEDENES) is in 2 files
• similar for voltage levels

For example the 1f4fabd4-8c57-c245-b107-4e64c9007ae3 — "ARENDAL 420" VoltageLevel:

File	Type
Telemark-120-HV1-MV1_BM.xml	Boundary (BM)
Telemark-120-LV1_EQ.xml	Equipment (EQ)
Telemark-120-MV1_EQ.xml	Equipment (EQ) - referenced

This causes problems when loading all these files in PowSyBl:

• Unexpected CgmesContainer for containerId on Telemark120 data powsybl/powsybl-core#3682

An LLM diagnosed it as follows:
Root cause: The Telemark model is structured as separate grid segments (HV1, MV1, LV1) with Boundary files (BM) defining the connection points. These boundary files re-define VoltageLevels that also exist in the Equipment files, causing pypowsybl to fail with "duplicate object" or "unexpected container" errors when loading all files together.

[VladimirAlexiev]

I did some more analysis of the duplication.
Count of dup triples, discounting rdf:type (which are duplicated by the standard, see Sveino/Inst4CIM-KG#164)

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX cim: <https://cim.ucaiug.io/ns#>
select ?g1 ?g2 ?g1name ?g2name (count(*) as ?triples) {
    graph ?g1 {?s ?p ?o}
    graph ?g2 {?s ?p ?o}
    filter(?p != rdf:type)
    filter(str(?g1)<str(?g2))
    ?g1 dct:title ?g1name.
    ?g2 dct:title ?g2name.
} group by ?g1 ?g1name ?g2 ?g2name

All duplicates involve:

• BM: no wonder, because it's a Boundary Model, so it repeats the boundary resources
• RD: reference data. Hopefully that has only BaseVoltages and PowSyBl hasn't complained about BaseVoltages so far
• "DIGIN10-30-MV1_SC" and "DIGIN10-30-MV1_EQ":

PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX cim: <https://cim.ucaiug.io/ns#>
select ?p (count(*) as ?c) {
    bind("DIGIN10-30-MV1_SC" as ?g1name)
    ?g1 dct:title ?g1name.
    ?g2 dct:title ?g2name.
    graph ?g1 {?s ?p ?o}
    graph ?g2 {?s ?p ?o}
    filter(str(?g1)<str(?g2))
} group by ?p order by ?p

This involves 4 float props (r,x), 2 booleans, MRID, and rd:type: