Link to source data in results table

[ghing]

Race table should display links to source data either on state site or github, so that journalists can verify the data.

This is somewhat difficult because there isn't currently a consumable map of results to source URLs.

The dashboard provides some information about source URLs but it's not comprehensive. I've found it often lacks:

• URLs when there are consistent machine-guessable URLs for data (these are usually generated by code in the datasource).
• URLs when there are a slew of per-county results files with totally inconsistent URLs (these are usual defined in url_paths.csv).

The best source of this data is the datasource for each state, but we don't yet aggregate our output this data in a way that could be used by the results front end.

The path forward for this is to either

• Decide to use the dashboard and decide to only link to portal URLs or backfill the source URLs to be more comprehensive.
• Bake out source metadata from datasource.

In either case, this is a non-trivial amount of work.

[zstumgoren]

yeah, this is a problem due to inconsistency in how we've handled data sources. The data admin (specifically the direct_links field) emerged as our preferred solution for the ETL process, and could be a partial (and easy) source of raw data links. At the datasource layer, we could output the mappings dict.

However, this still doesn't address a third case -- where the data came from a non-web source, such as a database dump emailed or mailed to us.

I do think it's important we provide some way of linking our results back to original source file(s), but I don't see us being able to do a one-to-one linkage across the board.

I think we should discuss a strategy for this offline since it could be quite involved, and then document our proposed strategy here or in a new ticket.

[zstumgoren]

The more I think about this, the more it seems that Datasource could be the driving force behind this feature. It already exposes the mapping of standardized filenames to source URLs, whether dynamically generated or fetched from the metadata API.

The main problem with this approach involves pre-processed data such as database dumps (Washington) and data extracted manually from PDFs ( Mississippi). In such cases, the Datasource obviously points at the preprocessed version of the data living on Github. So we'd need some mapping between preprocessed files and source data file(s).

@dwillis @ghing You all have done the preprocessing work so far. Any thoughts on what, if any, changes we could make to the preprocessing workflow to more easily expose the relationship between preprocessed files and the source data?

[ghing]

@zstumgoren, I agree that the Datasource is where this information comes from.

It actually handles the data required by processing pretty well for cases like MS or IA where the data files are published, but require processing. A while back, I updated the datasource API so it returns records for both the unprocessed data from the official website and the processed versions from our GitHub repos. My motivation was to put knowledge of source data in a consistent place and to be able to use the fetch task to download the unprocessed data for processing. However, it serves us well here too. As long as the datasources follow the conventions, we should be able to use them to get URLs for source data.

The only case we don't have covered is things like pre-2007 Washington or Colorado (I think?) where we got results over email or DVD rather than from the web. I don't know the right way to handle this, but the Datasource still seems like the integration point since we have to create records for the preprocessed results files.

In many cases I, and I believe @dwillis has been saving the unprocessed data to the repos, so they often do have URLs. We just need to define a convention for representing this in the datasource.

[dwillis]

Yeah, I've saved unprocessed data for WY, but for PA I had to split a very large file into segments. I think we can come up with the convention based on the url_paths.csv files in state repos.

[ghing]

I opened up openelections/openelections-core#177 for the convention discussion.