Complicated rules to build protobuf files in directory hierarchies.

[coryan]

This is mostly to document the rationale for the complicated rules that compile protobuf files.

Some background:

protoc uses -I options for two things. One is to find other proto files in imports. A second is to find the canonical name of your proto file. So if you your directory has a file named a/b/c/d.proto which imports b/c/e.proto then you need to add -Ia option that points to the a

Also, if you try to compile with:

protoc ../../a/b/c/d.proto

the protoc compiler will require you to have the ../.. path in your -I options. Because JayBeams often compiles in build directories that becomes a must.

Now, because some of the googleapis protos import other protos without the googleapis prefix, we need to add the -I../../googleapis option to the build. And then, if you say something like:

protoc -I../../a ../../a/b/c/d.proto

The protoc compiler then generates b/c/d.pb.h and b/c/d.pb.cc. Notice how the initial path is stripped. That confuses the Makefile rules, because when we are searching for the generated a/b/c/d.pb.cc we do not expect to find it under b/c/d.pb.cc. We could change the rules to generate protobuf files to say:

protoc -I../../a  --cpp_out=a ../../a/b/c/d.proto

But the etcd protos do not follow the same convention, their imports include the etcd prefix! Fine, we can avoid some of this by playing with the CXXFLAGS. One could write the rules to compile a proto file as follows:

protoc -I../.. -I../../a  --cpp_out=. ../../a/b/c/d.proto

That means the protobuf canonical name becomes a/b/c/d.proto, the file is generated with the path we want, and the imports are found using the second -I flag. Then we can add -I$(builddir)/a to the search path in the C++ compiler so the generate headers are found too.

That breaks because protoc uses the canonical path to generate some internal symbols. When compiling a/b/c/e.proto which imports b/c/d.proto it assumes that the generated files for d.proto have a symbol where b_c_d is part of the name. While if we generate the file with the full path the symbol has a_b_c_d in its name.

All of this means that the proto compiler has extremely strong opinions on the location of its files. All the files must have the same "canonical name" (the path stripped of the prefix in the -I option used to find it).

Fortunately one cal use a vpath with a pattern in GNU Make to solve this problem. For example, one can say:

vpath %.proto $(srcdir) $(srcdir)/googleapis $(srcdir)/gogoproto

Which instructs GNU Make to search for proto files in those directories, even if the target names are shorter. So for example we could have a/b/c/d.proto in the $(srcdir) directory, and then make b/c/d.pb.cc a dependency of one of our targets, and finally include $(srcdir)/a in the vpath for proto files.

GNU Make would find the dependency and apply the

%.pb.cc %.pb.h: %.proto

rule, calling the proto compiler with -I$(srcdir)/a in its search path.

Alternative

An alternative would be to have links in the top level source directory that point to the right place, in our example:

ln -s a/b b

or in practice:

ln -s googleapis/google google
ln -s gogoproto/gogoproto gogoproto # Ooops, need to rename module