Did you know ... Search Documentation:
protobufs.pl -- Google's Protocol Buffers
PublicShow source

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data -- think XML, but smaller, faster, and simpler. You define how you want your data to be structured once. This takes the form of a template that describes the data structure. You use this template to encode and decode your data structure into wire-streams that may be sent-to or read-from your peers. The underlying wire stream is platform independent, lossless, and may be used to interwork with a variety of languages and systems regardless of word size or endianness. Techniques exist to safely extend your data structure without breaking deployed programs that are compiled against the "old" format.

The idea behind Google's Protocol Buffers is that you define your structured messages using a domain-specific language and tool set. In SWI-Prolog, you define your message template as a list of predefined Prolog terms that correspond to production rules in the Definite Clause Grammar (DCG) that realizes the interpreter. Each production rule has an equivalent rule in the protobuf grammar. The process is not unlike specifiying the format of a regular expression. To encode a template to a wire-stream, you pass a grounded template, X, and variable, Y, to protobuf_message/2. To decode a wire-stream, Y, you pass an ungrounded template, X, along with a grounded wire-stream, Y, to protobuf_message/2. The interpreter will unify the unbound variables in the template with values decoded from the wire-stream.

For an overview and tutorial with examples, see protobufs_overview.md. Examples of usage may also be found by inspecting test_protobufs.pl and the demo directory.

- : Jeffrey Rosenwald (JeffRose@acm.org)
- : Peter Ludemann (peter.ludemann@gmail.org)
See also
- https://developers.google.com/protocol-buffers
- https://developers.google.com/protocol-buffers/docs/encoding
- : SWI-Prolog
Source protobuf_message(?Template, ?Wire_stream) is semidet
Source protobuf_message(?Template, ?Wire_stream, ?Rest) is nondet
Marshals and unmarshals byte streams encoded using Google's Protobuf grammars. protobuf_message/2 provides a bi-directional parser that marshals a Prolog structure to Wire_stream, according to rules specified by Template. It can also unmarshal Wire_stream into a Prolog structure according to the same grammar. protobuf_message/3 provides a difference list version.
Template- is a protobuf grammar specification. On decode, unbound variables in the Template are unified with their respective values in the Wire_stream. On encode, Template must be ground.
Wire_stream- is a code list that was generated by a protobuf encoder using an equivalent template.
- The protobuf specification states that the wire-stream can have the fields in any order and that unknown fields are to be ignored. This implementation assumes that the fields are in the exact order of the definition and match exactly.
- "Packed" repeated fields are not supported.
Source protobuf_segment_message(+Segments:list, -WireStream:list(int)) is det
protobuf_segment_message(-Segments:list, +WireStream:list(int)) is det
Low level marshalling and unmarshalling of byte streams. The processing is independent of the .proto description, similar to the processing done by protoc --decode_raw.

For unmarshalling, a simple heuristic is used on length-delimited segments: first interpret it as a message; if that fails, try to interpret as a UTF8 string; otherwise, leave it as a "blob" (if the heuristic was wrong, you can convert to a string or a blob by using protobuf_segment_convert/2). 32-bit and 64-bit numbers are left as codes because they could be either integers or floating point (use int32_codes/2, float32_codes/2, int64_codes/2, float64_codes/2 as appropriate); variable-length numbers ("varint" in the Protocol Buffers encoding documentation), might require "zigzag" conversion, integer_zigzag/2.

For marshalling, use the predicates int32_codes/2, float32_codes/2, int64_codes/2, float64_codes/2, integer_zigzag/2 to put integer and floating point values into the appropriate form.

Segments- a list containing terms of the following form (Tag is the field number; Codes is a list of integers):
  • varint(Tag,Varint) - Varint may need integer_zigzag/2
  • fixed64(Tag,Codes) - Codes is of length 8, (un)marshalled by int64_codes/2 or float64_codes/2
  • start_group(Tag)
  • end_group(Tag)
  • fixed32(Tag,Codes) - Codes is of length 4, (un)marshalled by int32_codes/2 or float32_codes/2
  • message(Tag,Segments)
  • string(Tag,String) - String is a SWI-Prolog string
  • length_delimited(Tag,Codes)

Of these, start_group and end_group are deprecated in the protobuf documentation and shouldn't appear in modern code, having been replaced by nested message types.

For deciding how to interpret a length-delimited item (when Segments is a variable), an attempt is made to parse the item as a meeesage; if that succeeds, it is put into a message/2 term, if it fails and it's of the formof a UTF8 string, it is put into a string/2 term (as a string, not as codes), otherwise into a length_delimited/2 term.

The interpretation of length-delimited items can sometimes guess wrong; the interpretation can be undone by using protobuf_segment_convert/2 to convert the incorrect segment to a string or a list of codes.

WireStream- a code list that was generated by a protobuf endoder.
See also
- https://developers.google.com/protocol-buffers/docs/encoding
- This predicate is preliminary and may change as additional functionality is added.
To be done
- Expansion of this code to allow generalized handling of wire streams with fields in arbitrary order. (See bugs for protobuf_message/2).
Source protobuf_segment_convert(+Form1, -Form2) is multi
A convenience predicate for dealing with the situation where protobuf_segment_message/2 interprets a segment of the wire stream as a form that you don't want (e.g., as a message but it should have been a UTF8 string).

For example:

?- protobuf_segment_convert(
?- protobuf_segment_convert(
?- protobuf_segment_convert(
       string(10, "inputType"),
- This predicate is preliminary and may change as additional functionality is added.
To be done
- Make this bidirectional, so it fits better with the style of protobuf_meessage/2.
Form1- message(Tag,Pieces) or string(Tag,String).
Form2- string(Tag,String) or length_delimited(Tag,Codes).

Undocumented predicates

The following predicates are exported, but not or incorrectly documented.

Source integer_zigzag(Arg1, Arg2)
Source protobuf_message(Arg1, Arg2, Arg3)
Source float64_codes(Arg1, Arg2)
Source int64_codes(Arg1, Arg2)
Source float32_codes(Arg1, Arg2)