Implement build framework for repository

[jmshapir]

@rdahis my goal in this issue is to get us all comfortable working in the same environment, as a prelude to bringing the protests project into github.

I will get you access to the underlying data. The code is written to assume that this is accessible via a symbolic link in the root directory called /raw. The raw data will not be versioned.

Steps:

• Review carefully the rules laid out in the Wiki, especially those pertaining to storage policy.
• Implement SCons as a build framework for this repository

[jmshapir]

@rdahis just checking to make sure everything is looking ok here.

[rdahis]

I'm on it, although it's taking me more time than I expected to get it right. Hoping to finish soon.

[jmshapir]

Ok thanks @rdahis, just let me know if questions come up.

[rdahis]

Should I commit the basic Scons structure into the Github folder?

Plus: I recommend all to install Scons, and the best online tutorial I found is http://scons.org/doc/2.1.0/HTML/scons-user/index.html

[jmshapir]

@rdahis I envision that you will follow the workflow outlined in the Wiki, i.e. branch the repo, implement your changes, and then make a pull request.

[rdahis]

At this point we have a (very simple) working SCons structure in the scons_implementation branch.

The SConstruct and source/SConscript files create an environment with each user's paths, run the make.py files and compiles Report.tex with pdftex.

I'm still working on (1) having a more robust Builder object to run python scripts and (2) compiling pdfs with latexmk.

If this is enough for now, please close the issue and merge branches.

Ps: Other useful resources are http://www.wiwi.uni-bonn.de/gaudecker/teaching/prog_econ/2010/07_automating_project_workflows_with_scons.pdf

[jmshapir]

@rdahis I haven't reviewed the branch in detail but I think we can simplify it a bit. I envision that we will replace each make.py with a (very simple) SConscript file. (We can still call it "make"). Then we can have a master SConstruct file that lives in the root and builds the whole thing. Let me know if that makes sense and if I'm using SCons terminology correctly.

[rdahis]

@jmshapir Yes, the terminology is correct. And that's indeed what we have now. The SConstruct file lives in the root, and the SConscript file lower in the hierarchy lives inside /source.

The SConscript file doesn't replace the make.py files yet but I believe it is just a matter of copy-pasting the code into the new structure.

A question I have is that I haven't understood well yet how the storage policy and symbolic /raw link will relate to the build structure itself. But maybe it will become clear when we bring in the data.

[jmshapir]

Thanks @rdahis. We shouldn't need most of what make.py is doing--we just want the SConscript to execute the Stata script or compile the TEX file in the respective source directory.

In terms of the storage policy, what I'd ideally like to see is that every user has the raw data locally in a place that is referenced by symbolic link. Then the remaining issue is just deciding how to coordinate centralized storage so that we all have the same thing. But for now you can just extract the raw data from the GoogleDrive link and work from there so you can run the code.

Let me know if that all makes sense.

[rdahis]

@jmshapir I have included my local path to the Ethnicity project in the SCons environment variables. So then when calling /raw/some/file I believe it will work. When we have more people editing the scripts, it should be a matter of including everyone's paths to /Ethnicity Project so we could all just call /raw. Symbolic links don't seem necessary inside this environment (they make more sense in a command line script).

However I haven't tested it since I would have to understand better and change the make.py files. Should I go for testing anyway, maybe only inside my branch?

[jmshapir]

Thanks @rdahis. A few thoughts:

• Definitely feel free to test it. That's the point of having a test repo and working in branches!
• Do we need to keep make.py at all? Or can we just replace it with an SConscript file?
• The reason I like symbolic links is that with those the user can make a purely local modification--there's no need to version a growing list of environment variables as more users touch the code. If we use a symbolic link, then everyone can set that up on her own machine and treat /raw as a relative path even if it is not.

I'm happy to discuss any of this further if you think I'm missing something.

[rdahis]

Ok great! Let me work a little more on it and I will get back to you.

We don't need to keep the make.py, so I will replace them with SConscript files. And you're right about the symbolic link vs. environment variables. I will implement that too.