Updating `seg[@subtype="cr"]` in PN

[samosafuz]

I demo'd the program for @jcowey this morning (he was very impressed), and we identified two features that will improve its functionality. I'll present them in separate issue tickets.

The first involves cases where the BP file has seg[@subtype="cr"] but the corresponding PN file does not (or does not have the same information).

Where PN has no entry for C.R. (or an incomplete entry for C.R.) and the user indicates that BP is correct for the line number corresponding to C.R., we need to do two things:

1. Update the corresponding PN entry with the new data (which the program is already designed to facilitate)
2. Create a new XML file for each review detailed in seg[@subtype="cr"], unless such a review already appears in PN. This would be a new feature: yes, it is the case that reviews get their own individual XML files in PN Biblio.

So, for example, in 2010-0043, one finds a number of reviews, which are tokenized by -:

Regarding 1., the PN version of this file (20984) does not have seg[@subtype="cr"]. The program as currently designed will remedy that.

But 2. is trickier, since we need to do a few things:

• determine whether the change in BP indicates the publication of new review(s) (i.e., whether the better information in BP is cosmetic or substantive)
• check whether each review already exists in PN, as its own XML file
• where a review doesn't already exist, to create a new XML file for it

Only the user can determine whether BP's seg[@subtype="cr"] indicates the publication of a new review: if it does, that will trigger a new workflow.

Checking for an existing review should be straightforward: reviews are flagged in XML by relatedItem[@type="reviews"], where the item reviewed is identified via bibl/ptr/@target (see snippet for 97327, below, which points to 20984):

<?xml version="1.0" encoding="UTF-8"?>
<bibl xmlns="http://www.tei-c.org/ns/1.0" xml:id="b97327" type="review">
  <author>
      <forename>Peter</forename>
      <surname>Van Minnen</surname>
   </author>
   <date>2013</date>
  <biblScope type="pp" from="323" to="326">323-326</biblScope>
  <biblScope type="no">Nos 171-172</biblScope>
  <relatedItem type="appearsIn">
      <bibl>
         <ptr target="https://papyri.info/biblio/110"/>
         <!--ignore - start, i.e. SoSOL users may not edit this-->
         <!--ignore - stop-->
      </bibl>
  </relatedItem>
  <biblScope type="issue">50</biblScope>
  <relatedItem type="reviews" n="1">
      <bibl>
         <ptr target="https://papyri.info/biblio/20984"/>
         <!--ignore - start, i.e. SoSOL users may not edit this-->
         <!--ignore - stop-->
      </bibl>
  </relatedItem>
  <idno type="pi">97327</idno>
  <seg type="original" subtype="cr" resp="#BP">Peter van Minnen, BASP 50 (2013) pp. 323-326.</seg>
</bibl>

Simply looking for a review that points to the item in question will get us pretty far: a quick check reveals that both the Straus (82888) and van Minnen (97327) reviews already exist in PN (but that there are three others in the BP fiche that do not).

• If the answer is 'there are no matches/reviews of this work in PN Biblio', then we will need to create the file(s).
• If the answer is 'there is one or more matches', then we will need to determine whether one is correct, or, where none is correct, to create the file.

Where one is correct, the user can identify it and the program can move on (since nothing needs to change). The easiest way to identify the correct file is probably via the page range, which will take the format of \d+-\d+ before the token-delimiting . - : the user could confirm that a review file pointing at the desired target whose page range matches that of the BP fiche is correct (or maybe the computer could simply move on if the page range is a match?). And, again, if there is no match, then we'd have to create the file.

For things that need to be created/added, I would certainly like a log file generated, in which the PN Biblio target for the review is indicated first, followed by the contents of seg[@subtype="cr"] (tokenized by -). Each token would appear on its own line, and we would only need the reviews that do not yet appear in PN Biblio as their own files. So, for 20984, the log file would contain the following output

20984: Thomas Schmidt, MusHelv 68 (2011) pp. 232-233.
20984: Lajos Berkes, Gnomon 85 (2013) pp. 464-466.
20984: Rudolf Stefec, Gymnasium 119 (2012) pp. 302-304.

Via XSLT, it will be possible to parse the log, and to populate the new XML with pretty much everything we need (as per the **** in the following template). But if you want to to try doing that using non-XSLT tools, I can provide a list of journal abbreviations used by BP and the corresponding PN Biblio idno that is stated in relatedItem[@type="appearsIn"].

<?xml version="1.0" encoding="UTF-8"?>
<bibl xmlns="http://www.tei-c.org/ns/1.0" xml:id="b****" type="review">
  <author>
      <forename>****</forename>
      <surname>****</surname>
   </author>
   <date>****</date>
  <biblScope type="pp" from="****" to="****">****</biblScope>
  <relatedItem type="appearsIn">
      <bibl>
         <ptr target="https://papyri.info/biblio/****"/>
         <!--ignore - start, i.e. SoSOL users may not edit this-->
         <!--ignore - stop-->
      </bibl>
  </relatedItem>
  <biblScope type="issue">****</biblScope>
  <relatedItem type="reviews" n="1">
      <bibl>
         <ptr target="https://papyri.info/biblio/****"/>
         <!--ignore - start, i.e. SoSOL users may not edit this-->
         <!--ignore - stop-->
      </bibl>
  </relatedItem>
  <idno type="pi">****</idno>
  <seg type="original" subtype="cr" resp="#BP">****</seg>
</bibl>

[halosm1th]

I think for this issue we'll need to talk about it on Wednesday so I can get a better sense of what you're looking for in particular and how to integrate it into the program as it exists now, because this is a pretty substantive addition and modification to key parts of the program.

It is certainly doable, I just basically need to better understand what you're looking for and figure out how to integrate it.

For example, one of the questions I have right off the bat is: does this need to happen when user selects the the CR line in BP is correct? Or can I handle it later?

Then to make sure I'm understanding the request correctly:

1. "determine whether the change in BP indicates the publication of new review" - I understand what you've written to indicate that this is done by the user. Or basically that "Only the user can determine whether BP's seg[@subtype="cr"] indicates the publication of a new review: if it does, that will trigger a new workflow." means that when the program is processing that update to the record, that is when we should run the code for the new features.
2. When selected, load the PN file again, and this time search it for the relatedItems field, then into the biblio node, and then extract a list of all the ptr nodes.
3. For those nodes, return to the earlier list we already have of all the xml entrise from the PN folder, and find those where the PN_Number matches the last numbers of the url in the ptr.
4. Then, take the list of entries from BP's cr, split it based on ' - ' (personal note: need the spaces around the - otherwise it will split page ranges), store in CR_Tokens
5. Loop through list of Cr_Tokens. Each token should have its page range extracted using "\d+-\d+". Compare these tokens to page range values of all the ptr_nodes. If there are no matches, add to new_XML list. If there is one or more matches, store them in a match_list. (match list should contain: CR_TOken matched again, match value (the extracted page range) and then a list of all the files which matched it)
6. Once all the CR_Tokens have been parsed, there should be two lsits: new_XML, and match_list. Now, loop through the match list, and ask the user which match is the correct one for the relevant CR. If only one was correct, note it but don't ask for any user comment. If the user says none are correct, then move the entry to the new_XML list.
7. Then, loop through the list of new_XML entries. when processing the entry, add the entry and its ptrs to the log file in the format described above.

This leaves me with one major question: what is the review xml supposed to look like? Am I creating an essentially blank xml document with the basic framework there, and a single seg with the info from CR in it?

For now I'm just starting to plot out what these changes will look like, and am trying to get a better sense of what I'm implementing. If I've missed a step or misunderstood, please let me know. Otherwise Wednesday will probably me confirming and refining the above outline of how to implement the changes.

[samosafuz]

To your basic question, it is certainly possible to have a separate C.R. program, which could be run independently of the Data Compiler you're working on. But since the Data Compiler already has the user flag when BP's C.R. field is better, it seemed appropriate to build it in.

As for your other questions:

1. Yes, the user currently indicates whether BP, PN, Neither, or Both/Shared is correct for a given line in the UI table. When BP is selected for the C.R. seg, that is what will trigger the workflow for this new feature. The UI should immediately ask the reviewer: Are there any new reviews in BP that we need to add to PN? If the answer is yes, we trigger the workflow. If the answer is no, we simply update the PN biblio as we'd do for other segs where BP is correct.
2. When triggered, the program begins by looking through PN biblio for a match on the value of idno[@type="pi"] at the end of ptr/@target. NB: be sure to look only on the ptr children of relatedItem[@type="reviews"] (and not to relatedItem[@type="mentions"]). We want to know how many matches for that idno there are.
3. See above: the PN file whose idno we're looking for is open when the workflow is triggered; you could grab it either from the filename (*.xml) or via idno[@type="pi"].
4. Yes, this seems right. We'll want to know how many items are in the BP C.R. seg.
5. Yes, this seems right. But if the number of matches corresponds to the number of C.R. tokens AND the page ranges correspond, we don't need to do anything further. But if there are no matches or if the page numbers don't line up, we're going to have to do something with the outlying/orphaned C.R. seg. (I'm a bit fuzzy here about your last sentence, since there should only be one match.)
6. Let's chat Wednesday about what the easiest way to achieve 6 and 7 will be. Essentially, the idea is to identify any new reviews that are recorded in seg[@subtype="cr"] but aren't yet in PN. If there's a way for the Data Compiler program to create those new files and populate the various XML elements with the correct data by parsing what's in BP, that's great: if not, we can simply log the information and I'll write some XSLT to do that work. The log as illustrated in the sample above can be the default option.

[halosm1th]

On the Working-On-CR branch, I believe I've added this feature.

When the user selects a CR element and has marked BP as being correct, the pair of BP and PN entries are added to a special list.

Then that list is returned, and later processed. While processed, the BP's CR entries are extracted and stored in a list. Then the PN file has it's xml called up, the PTR elements are found, and then the PN number from those are extracted. Then the big list of previously loaded PN files is searched for that PN. Then a list of the PN reviews is developed (this is a feature which I have not fully tested)

Once the two lists are assembled, the year and page ranges are extracted from the CR entries, and there is a check if any of the review entries we previously collected matched (I have not yet confirmed this works). For any CR entries which don't match, we then move onto checking the whole PN list previously collected, first filtering for all entires in the CR's publication year, and then checking if any of those entries share it's page range. Any entries which don't match are collected in a list and returned.

Those returned entries are then processed for their various component parts. So that the following XML is generated for the example entry in the original request (note, my PN data is now technically out of date so the number generated for this is likely incorrect):

<?xml version="1.0" encoding="UTF-8"?>
<bibl xmlns="http://www.tei-c.org/ns/1.0" xml:id="b9999" type="review">
  <author>
      <forename>Thomas</forename>
      <surname>Schmidt</surname>
   </author>
   <date>2011</date>
  <biblScope type="pp" from="232" to="233">232-233</biblScope>
  <relatedItem type="appearsIn">
      <bibl>
         <ptr target="https://papyri.info/biblio/322"/>
         <!--ignore - start, i.e. SoSOL users may not edit this-->
         <!--ignore - stop-->
      </bibl>
  </relatedItem>
  <biblScope type="issue"> 68 </biblScope>
  <relatedItem type="reviews" n="1">
      <bibl>
         <ptr target="https://papyri.info/biblio/20984"/>
         <!--ignore - start, i.e. SoSOL users may not edit this-->
         <!--ignore - stop-->
      </bibl>
  </relatedItem>
  <idno type="pi">9999</idno>
  <seg type="original" subtype="cr" resp="#BP">Thomas Schmidt, MusHelv 68 (2011) pp. 232-233.</seg>
</bibl>

[samosafuz]

For numbering the new entries, NB: currently the highest number in Biblio is 97355.

[jcowey]

Please be careful here: the new highest number in Biblio is 97357.

Perhaps best for now, @samosafuz and @halosm1th, that you should start numbering your new entries from, let us say, 97500

That way we can continue to add hand entered bibliographic entries from 97358 without potential risk of overwriting.

[halosm1th]

I've updated the code so that the numbering should be properly done now. Using the data from @samosafuz idp.data version, it put the next entry as being 97356.

[jcowey]

Can you, @halosm1th, please take back that commit just now. I had written above in #7 (comment) that 97356 should not be used as the starting point.

I had already made commits up unitl 97372 in the meantime.

THat is why I had asked, please, to make these commits starting from 97500. Flagging you as well again, @samosafuz

[jcowey]

OK, I panicked. They have not been committed to master yet. Master repo is now sitting at 97371

https://github.com/papyri/idp.data/blob/master/Biblio/98/97371.xml

I still think it would be better to have a designated area for these super helpful commits, so that we do our best to avoid possible overwrites.

Sorry for the panic, @halosm1th . Apologies again.

[halosm1th]

Apologizes for any issues arising from my code.

To make sure the ID numbers being generated here don't over write existing values I can:

1. arbitrarily set the start number to 97500
2. Instead of starting at the last found ID in the xml data +1, I can add +501 or something to it so that it should would be notably out out line.

I just need to know which to go with.

[samosafuz]

Anything the program produces will sit in the user's local directory; it won't write directly to GitHub. It'll be on the user (i.e., me) to pull from papyri/idp.data:master before doing anything.

I think easiest way forward (for you, @halosm1th), will be to enumerate based on the highest number in the user's local idp.data/Biblio collection, +1. I'll make sure to pull from papyri/idp.data:master beforehand, and since I'll be initiating the pull request, I can always modify on my end as needed. I'd rather avoid arbitrarily skipping numbers, if possible.

[jcowey]

Super. I will leave it all in safe hands. That is fine. I certainly understand the desire to avoid "holes" in the numbering.

[samosafuz]

I ran the Compiler on the year 2010 today (some 1295 entries), and it worked splendidly throughout, until the time came for the CR subroutine to do its job, at which point it crashed:

We've discussed that it is desirable to overwrite the local file immediately (i.e., when the f command is entered for a file), when BP is selected as the correct version for one or more given seg elements. It would also be good to write any new XML files to the NewXMLEntries folder, and to log any of the changes needed on the BP's end, before moving on to the CR subroutine.

I'm not sure why the CR subroutine crashed in this case: Graeco-Latina Brunensia appears in the list of journals that is being searched (1410). Debugging is necessary, but so too do I think that the program needs to log and then move on to the next file when this error is thrown. The message 'the reviews for {filename}.xml may need to be created manually' is probably sufficient to let me know what was skipped.