Did you know ... Search Documentation:
protobufs.pl -- Google's Protocol Buffers ("protobufs")
PublicShow source

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data -- think XML, but smaller, faster, and simpler. You define how you want your data to be structured once. This takes the form of a template that describes the data structure. You use this template to encode and decode your data structure into wire-streams that may be sent-to or read-from your peers. The underlying wire stream is platform independent, lossless, and may be used to interwork with a variety of languages and systems regardless of word size or endianness. Techniques exist to safely extend your data structure without breaking deployed programs that are compiled against the "old" format.

The idea behind Google's Protocol Buffers is that you define your structured messages using a domain-specific language and tool set. Further documentation on this is at https://developers.google.com/protocol-buffers.

There are two ways you can use protobufs in Prolog:

The protobuf_parse_from_codes/3 and protobuf_serialize_to_codes/3 interface translates between a "wire stream" and a Prolog term. This interface takes advantage of SWI-Prolog's dict. There is a protoc plugin (protoc-gen-swipl) that generates a Prolog file of meta-information that captures the .proto file's definition in the protobufs module:

  • proto_meta_normalize(Unnormalized, Normalized)
  • proto_meta_package(Package, FileName, Options)
  • proto_meta_message_type( Fqn, Package, Name)
  • proto_meta_field_name( Fqn, FieldNumber, FieldName, FqnName)
  • proto_meta_field_json_name( FqnName, JsonName)
  • proto_meta_field_label( FqnName, LabelRepeatOptional) % 'LABEL_OPTIONAL', 'LABEL_REQUIRED', 'LABEL_REPEATED'
  • proto_meta_field_type( FqnName, Type) % 'TYPE_INT32', 'TYPE_MESSAGE', etc
  • proto_meta_field_type_name( FqnName, TypeName)
  • proto_meta_field_default_value(FqnName, DefaultValue)
  • proto_meta_field_option_packed(FqnName)
  • proto_meta_enum_type( FqnName, Fqn, Name)
  • proto_meta_enum_value( FqnName, Name, Number)

The protobuf_message/2 interface allows you to define your message template as a list of predefined Prolog terms that correspond to production rules in the Definite Clause Grammar (DCG) that realizes the interpreter. Each production rule has an equivalent rule in the protobuf grammar. The process is not unlike specifiying the format of a regular expression. To encode a template to a wire-stream, you pass a grounded template, X, and variable, Y, to protobuf_message/2. To decode a wire-stream, Y, you pass an ungrounded template, X, along with a grounded wire-stream, Y, to protobuf_message/2. The interpreter will unify the unbound variables in the template with values decoded from the wire-stream.

For an overview and tutorial with examples, see library(protobufs): Google's Protocol Buffers Examples of usage may also be found by inspecting test_protobufs.pl and the demo directory, or by looking at the "addressbook" example that is typically installed at /usr/lib/swi-prolog/doc/packages/examples/protobufs/interop/addressbook.pl

author
- Jeffrey Rosenwald (JeffRose@acm.org)
- Peter Ludemann (peter.ludemann@gmail.org)
See also
- https://developers.google.com/protocol-buffers
- https://developers.google.com/protocol-buffers/docs/encoding
Compatibility
- SWI-Prolog
Source protobuf_parse_from_codes(+WireCodes:list(int), +MessageType:atom, -Term) is semidet
Process bytes (list of int) that is the serialized form of a message (designated by MessageType), creating a Prolog term.

Fails if the message can't be parsed or if the appropriate meta-data from protoc hasn't been loaded.

Arguments:
WireCodes- Wire format of the message from e.g., read_stream_to_codes/2. (The stream should have options encoding(octet) and type(binary), either as options to read_file_to_codes/3 or by calling set_stream/2 on the stream to read_stream_to_codes/2.)
MessageType- Fully qualified message name (from the .proto file's package and message) The initial '.' on the message type name is optional. For example, if the package is google.protobuf and the message is FileDescriptorSet, then you would use '.google.protobuf.FileDescriptorSet' or 'google.protobuf.FileDescriptorSet'. You can see the message names by looking at protobufs:proto_meta_field_name('.google.protobuf.FileDescriptorSet', FieldNumber, FieldName, FqnName) (the initial '.' is not optional for these facts).
Term- The generated term, as nested dicts.
See also
- library(protobufs): Google's Protocol Buffers
bug
- Doesn't fill in "default" values (note that this behavior is different for proto2 and proto3; and the default information is available in proto_meta_field_default_value/2, which is generated from the .proto files).
- Ignores .proto extensions.
- Doesn't do anything special for oneof or map.
- Generates fields in a different order from the C++, Python, Java implementations, which use the field number to determine field order whereas currently this implementation uses field name. (This isn't stricly speaking a bug, because it's allowed by the specification; but it might cause some surprise.)
To be done
- document the generated terms (see library(http/json) and json_read_dict/3)
- add options such as true and value_string_as (similar to json_read_dict/3)
- add option for form of the dict tags (fully qualified or not)
- add option for outputting fields in the C++/Python/Java order (by field number rather than by field name).
- add proto2/proto3 options for processing default values.
Source protobuf_serialize_to_codes(+Term:dict, -MessageType:atom, -WireCodes:list(int)) is det
Process a Prolog term into bytes (list of int) that is the serialized form of a message (designated by MessageType).

Fails if the term isn't of an appropriate form or if the appropriate meta-data from protoc hasn't been loaded.

Arguments:
Term- The Prolog form of the data, as nested dicts.
MessageType- Fully qualified message name (from the .proto file's package and message). For example, if the package is google.protobuf and the message is FileDescriptorSet, then you would use '.google.protobuf.FileDescriptorSet' or 'google.protobuf.FileDescriptorSet'. You can see the message names by looking at protobufs:proto_meta_field_name('.google.protobuf.FileDescriptorSet', FieldNumber, FieldName, FqnName). The initial '.' on the message type name is optional.
WireCodes- Wire format of the message, which can be output using format('~s', [WireCodes]).
See also
- library(protobufs): Google's Protocol Buffers
Source protobuf_message(?Template, ?WireStream) is semidet
Source protobuf_message(?Template, ?WireStream, ?Rest) is nondet
Marshals and unmarshals byte streams encoded using Google's Protobuf grammars. protobuf_message/2 provides a bi-directional parser that marshals a Prolog structure to WireStream, according to rules specified by Template. It can also unmarshal WireStream into a Prolog structure according to the same grammar. protobuf_message/3 provides a difference list version.
Arguments:
Template- is a protobuf grammar specification. On decode, unbound variables in the Template are unified with their respective values in the WireStream. On encode, Template must be ground.
WireStream- is a code list that was generated by a protobuf encoder using an equivalent template.
bug
- The protobuf specification states that the wire-stream can have the fields in any order and that unknown fields are to be ignored. This implementation assumes that the fields are in the exact order of the definition and match exactly. If you use protobuf_parse_from_codes/3, you can avoid this problem.o

Undocumented predicates

The following predicates are exported, but not or incorrectly documented.

Source protobuf_message(Arg1, Arg2, Arg3)