Protocol Buffers: Introduction and Installation

Overview

What is protocol buffer? What is protocol buffer for? This post intends to give an introduction about protocol buffer and its installation on Mac.

Protocol Buffer Introduction

We can find the guide/overview about protocol buffer at Google Protocol Buffer, some key concepts about protocol buffer:

  1. The stable version is proto2 and it officially only supports Java/C++/Python
  2. What are protocol buffers?Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data, think XML, but smaller, faster, and simpler. We define how we want our data to be structured once, then we can use special generated source code to easily write and read our structured data to and from a variety of data streams and using a variety of languages. We can even update our data structure without breaking deployed programs that are compiled against the “old” format
  3. required, optional, repeated these three keywords specify the quantification of the fields
  4. Protocol buffers have many advantages over XML for serializing structured data. Protocol buffers:are simpler
    are 3 to 10 times smaller
    are 20 to 100 times faster
    are less ambiguous
    generate data access classes that are easier to use programmatically
  5. However, protocol buffers are not always a better solution than XML, for instance, protocol buffers would not be a good way to model a text-based document with markup (e.g. HTML), since we cannot easily interleave structure with text. In addition, XML is human-readable and human-editable; protocol buffers, at least in their native format, are not. XML is also to some extent self-describing. A protocol buffer is only meaningful if we have the message definition (the .proto file).
  6. proto2 only does serializing/deserialization, we have to come up with our own network layer/RPC framework to achieve RPC
  7. proto3 and gRPC are quite new and in alpha stage, when these two are stable, protocol buffer will be more enriched and more like thrift which supports both serialization and rpc framework.
  8. backward compatibility: We can add new fields to our message formats without breaking backwards-compatibility; old binaries simply ignore the new field when parsing.

Protocol Buffer Installation on Mac

To install Protocol Buffer in Unix like Systems, here I use Mac for example, go to git hub google protocol buffer for more detailed instructions and here is my procedures:

  1. The first step is to go to the source code root folder and execute `./autogen.sh`, since the default Mac system does not have Unix tools available, the complicated solution (I will give a much easier alternative soon after) is given by the README.md:We will first need to install Xcode from the Mac AppStore and then run the following command from a terminal: $ sudo xcode-select –install
To install Unix tools, we can install “port” following the instructions at [https://www.macports.org][1] . This will reside in /opt/local/bin/port for most Mac installations.

    $ sudo /opt/local/bin/port install autoconf automake libtool 

In fact, my installation experience is that, if we directly autogen.sh I only encountered 2 problems, these 2 problems are due to the lack of the Unix tool autoconf and libtool, the first error message I got is:  
 
+ sed -i -e 's/RuntimeLibrary="5"/RuntimeLibrary="3"/g;
s/RuntimeLibrary="4"/RuntimeLibrary="2"/g;' gtest/msvc/gtest-md.vcproj gtest/msvc/gtest.vcproj gtest/msvc/gtest_main-md.vcproj gtest/msvc/gtest_main.vcproj gtest/msvc/gtest_prod_test-md.vcproj gtest/msvc/gtest_prod_test.vcproj gtest/msvc/gtest_unittest-md.vcproj gtest/msvc/gtest_unittest.vcproj
+ autoreconf -f -i -Wall,no-obsolete
./autogen.sh: line 38: autoreconf: command not found

The solution for this problem is just to install automake:  brew install automake which should also install autoconf and allow rvm to finish installing.

The second problem I got is this

./autogen.sh
+ sed -i -e 's/RuntimeLibrary="5"/RuntimeLibrary="3"/g;
s/RuntimeLibrary="4"/RuntimeLibrary="2"/g;' gtest/msvc/gtest-md.vcproj gtest/msvc/gtest.vcproj gtest/msvc/gtest_main-md.vcproj gtest/msvc/gtest_main.vcproj gtest/msvc/gtest_prod_test-md.vcproj gtest/msvc/gtest_prod_test.vcproj gtest/msvc/gtest_unittest-md.vcproj gtest/msvc/gtest_unittest.vcproj
+ autoreconf -f -i -Wall,no-obsolete
Can't exec "glibtoolize": No such file or directory at /usr/local/Cellar/autoconf/2.69/share/autoconf/Autom4te/FileUtils.pm line 345, <GEN7> line 6.
autoreconf: failed to run glibtoolize: No such file or directory
autoreconf: glibtoolize is needed because this package uses Libtool

The solution for this problem is just to install libtool: brew install libtool 

To summarize, run brew install automake && brew install libtool && ./autogen.sh, instead of directly run `./autogen.sh`

After step 1 is successful, we shall see the output ended wit this line “+ exit 0″ and the exit code zero indicates no error occurred. Then use sudo as follows so we can write/install files in our system folder

$ sudo bash

Next run the following command line to configure and specify the installation folder to be /usr. This is because by default, the package will be installed to /usr/local. But on many platforms, /usr/local/lib is not part of LD_LIBRARY_PATH (In Linux, the environment variable LD_LIBRARY_PATH is a colon-separated set of directories where libraries should be searched for first, before the standard set of directories, see this link Shared Libraries for more details).

$ ./configure --prefix=/usr

Next run the following to check whether everything is OK before make install

$ make && make check

if there is no problem, every test passes, you should be able to see the final lines of the output as (Warnning is fine, just ignore them):

/Library/Developer/CommandLineTools/usr/bin/make check-TESTS
PASS: protobuf-test
PASS: protobuf-lazy-descriptor-test
PASS: protobuf-lite-test
PASS: google/protobuf/compiler/zip_output_unittest.sh
PASS: google/protobuf/io/gzip_stream_unittest.sh
============================================================================
Testsuite summary for Protocol Buffers 2.6.1
============================================================================
# TOTAL: 5

# PASS: 5

# SKIP: 0

# XFAIL: 0

# FAIL: 0

# XPASS: 0

# ERROR: 0

===========================================================================

Next run the following to install

$ make install

If it is successful, then run the following

$ which protoc 

you should be able to see the output as “/usr/bin/protoc”

Now you can copy the sample from the guides of protocol buffer into say test.proto, and run protoc to generate source code. For example, the following will generate the .h and .cc files in your current folder, you should be able to see the generated files and that indicates you installed protocol buffer successfully.

$ protoc --cpp_out=./addressbook.proto

I would like to explain a bit more about other concepts mentioned in the official website:

  1. Compiling dependent packages: use  pkg-config to pass various flags to our compiler and linker so we can compile a package that uses Protocol Buffers. The primary use of pkg-config is to provide the necessary details for compiling and linking a program to a library. This metadata is stored in pkg-config files.  See pkg-config for mote details about pkg-config
  2. Java and Python InstallationThe Java and Python** runtime libraries** for Protocol Buffers are located in the java and python directories. See the README file in each directory for more information on how to compile and install them. Note that both of them require us to first install the Protocol Buffer compiler (protoc), which is part of the C++ packageoThe so called run time library here is the low-level routines for handling like memory management, exceptions and so on. See this for more explanation: run time library

TODO

There are some advanced topics I would like to research more on and write later posts about them:

  1. protocol buffer encoding: https://developers.google.com/protocol-buffers/docs/encoding
  2. gRPC, proto3: http://www.grpc.io/docs/ and  https://developers.google.com/protocol-buffers/docs/proto3
  3. C++/Java Samples https://developers.google.com/protocol-buffers/docs/cpptutorial and https://developers.google.com/protocol-buffers/docs/javatutorial
  4. Compare with Thrift, integrate into Thrift ? Is this possible?

Summary

This post gave an introduction about protocol buffer about what is protocol buffer? What is protocol buffer for?  and its installation on Mac.

Written on October 14, 2015