On FaceBook's Thrift semantics, code generation, and OCaml

Date: 2010-July-18 @ 23:42
Tags: asn.1, ocaml, thrift

Note: The obligatory TL;DR section is at the very end of this text.

The ASN.1 rant

I have my own personal score with the ASN.1 world. The standard is laden with design-by-committee complexity, no doubt evolved to “address the real world demands”. Its mind-numbing standardese and semantics effectively prohibit any newcomer from ever entering this field and producing a decent new compiler. So, we're stuck with something like 2.5 alive ASN.1 compilers covering some 3 mainstream languages (C++, C#, Java). The commercial products are often cost prohibitive: they're squarely aimed at rich telecom market. Where would a small Ruby or Python startup go? (While I am at it, Erlang has a free decent ASN.1 compiler, you know).

Yet, there's an opposite side to this complexity. Many things you struggle with or “invent” for the purpose of better data serialization have already been invented in the ASN.1 world. Things like broiled-to-perfection TLV-based encodings (BER/DER/CER), bitwise Packed Encoding Rules (competing with gzip'ing your serialized binary blob), Information Object Classes (think of SNMP MIB macros on steroids), or Encoding Control Notation have a lot to offer and learn from.

But I should stop kicking that dead horse. Let's try Thrift for a change.

Thrift

0. Selecting Thrift for OCaml

Oleg Sharov selected Thrift for our next project based on language support matrix. We needed safe serialization mechanism for OCaml and it seemed that Thrift would offer us some help with that.

Why would we need Thrift instead of the built-in Marshal functionality? The format produced by the OCaml's Marshal module lacks versioning and its implementation is unsafe. Marshal.from_channel crashes the process when the consumer attempts to read a serialized message created by a different version of the producer.

So, off we went and created a module for our OCaml serializing needs. Let's name it test.thrift for the purposes of this article (Point 1):

// test.thrift
struct MyStruct {
  1:           string myNoReq,
  2:           string myNoReqDef = "noreqdef",
  3:  optional string myOpt,
  4:  optional string myOptDef = "optdef",
  5:  required string myReq,
  6:  required string myReqDef = "reqdef",
  7:     list<string> myListDefEmpty = [],
  8:     list<string> myListDef = ["listElement"],
  9:  required double myDblDef0 = 0.0,
 10:  required double myDblDefPi = 3.1415
}

1. Problems with requiredness

1.1. Treatment of absent (default) requiredness

 // __isset member:
bool myNoType;
// comparison operator:
if (!(myNoType == rhs.myNoType))
  return false;
// write: happens always

// __isset member:
bool myOpt;
// comparison operator:
if (__isset.myOpt != rhs.__isset.myOpt)
  return false;
else if (__isset.myOpt && !(myOpt == rhs.myOpt))
  return false;
// write:
if (this->__isset.myOpt) { … }

// Not part of __isset.
// comparison operator:
if (!(myReq == rhs.myReq))
  return false;
// write: happens always

// write():
if (this.myNoReq != null) {
  oprot.writeFieldBegin(MY_NO_REQ_FIELD_DESC);
  oprot.writeString(this.myNoReq);
  oprot.writeFieldEnd();
}

// write():
if (this.myOpt != null) {
  // Yes, they do it twice:
  if (this.myOpt != null) {
    oprot.writeFieldBegin(MY_OPT_FIELD_DESC);
    oprot.writeString(this.myOpt);
    oprot.writeFieldEnd();
  }
}

// write():
if (this.myReq != null) {
  oprot.writeFieldBegin(MY_REQ_FIELD_DESC);
  oprot.writeString(this.myReq);
  oprot.writeFieldEnd();
}
// validate(): check for required fields
if (myReq == null) {
  throw new TProtocolException(
  "Required field 'myReq' was not present!"
  + " Struct: " + toString());
}

public $myNoReq = null;

public $myOpt = null;

public $myReq = null;

The experiment shows that the absent requiredness flag:

• means neither “optional” nor “required” for C++,
• does not do much for Java, except removing some code duplication compared to “optional”, and
• does not affect the code at all for PHP.

The Thrift compiler source code specifies the third kind of requiredness, namely T_OPT_IN_REQ_OUT, which is not documented properly, but used in a fancy way at least for C++ code generation. In fact, the best description of the field comes from the JIRA-455 issue, still open after a year:

Jonathan Ellis added a comment - 14/Aug/09 11:21 AM I have no idea what this means, but having fields that are neither required nor optional and get special cased differently in different places of the code is broken. If you really want to have 3 states then 2 of them should not be called "required" and "optional" because common sense implies that if a field is not one it must be the other.

David Reiss added a comment - 14/Aug/09 01:29 PM That is not going to happen. In C++, optional fields require applications to manually set __isset fields, otherwise they are not serialized on writing. Required fields throw an exception if they are absent. The default has neither of these issues. It is the most sensible choice when working with C++. […]

I believe that coming up with, maintaining, and defending this kind of poor naming and semantic discrepancy is the first step to perdition. If Thrift continues to evolve this way, it will turn into another ASN.1-like nightmare, mark my words.

1.2. Relating requiredness semantics to problem domains

struct targetWebSystem {
  required string host_name,
  optional int32  host_port = 80,
}

I believe the requiredness flag should be useful in modeling the aspects of a problem domain, whereas right now it appears to be too protocol-centric (describing what should be on the wire), inconsistent in treatment by the target language generators, and thus confusing. At the very least, it should be properly described in the documentation rather than JIRA issue.

2. There is a problem with default values

2.1. Semantics of default values is non-existent or worse

The purpose is not described in the whitepaper! In fact, the default values are introduced only by example, and are never formally defined.

One could think that specifying a default value tells the receiver that in the absence of the field value in the received blob (if omitted during encoding), the receiver should assume the field as having the default value.

There is not much stretch in this assumption. In fact, this is how ASN.1 describes the default values: the “OPTIONAL” values turning into default when there's no corresponding data on the wire. For example, consider the following ASN.1 structure:

MyStruct ::= SEQUENCE {
  fldOpt    String OPTIONAL,         -- If nothing is on the wire, asnOpt is not defined (NULL?)
  fldOptDef String DEFAULT "optdef", -- If nothing is on the wire, the asnOptDef is "optdef"
  fldReq    String                   -- This one must always be present on the wire.
}

What is the purpose of specifying a default value for a “required” field?

To allow the structure to be fully valid even if deserialization didn't find the corresponding value on the wire? The C++ code is not consistent with it: if you try to deserialize a blob, which does not contain the value of a required field, the decoder will throw INVALID_DATA exception.

Perhaps, allow the structure to be serializable even if the structure owner didn't manually set these required fields ((new MyStruct)->write())? Nice, but hold that thought for a moment (Point 2).

What is the purpose of specifying a default value for an “optional” field?

To allow the structure to think that it's got a value of an optional field off the wire if there was no value for that field on the wire? The C++ code is not consistent with this: it would not treat the optional field as set if the value didn't specifically came off the wire. So, default value set in the class constructor for an optional field is semantically very different from the default value taken off the wire. What's the point in this very fine distinction? Perhaps, this distinction is used to allow for a round-trip: if the value is never decoded off the wire, it won't be placed on the wire during encoding. The new->read()->write() sequence would not put the default value of an optional field to the output stream (Point 3).

But hey, remember I told you to hold the thought (see Point 2)? The discrepancy is that for “required” fields the default value is going to appear on the wire during new->write() sequence, whereas for “optional” field it won't.

That's not a subtle mess. It very much looks like that the semantics of the requiredness is not orthogonal to the semantics of a default value specification. The ASN.1 kept this mess a bit more honestly exposed, at least, by lumping these things into a three-way toggle (required, optional, and optional-with-default-value)!

2.2. Default values cannot exercise their power

You might argue that in Thrift this goes the same way: an “optional” field having a default value is never encoded (see Point 3). Well, this is not true at least for Java — an “optional” field having a default value is always encoded. Even in C++, if you set the value of a field manually, the implementation will encode it ignoring the fact that it might bear the default value. So, this is not even close to the ASN.1 DEFAULT semantics for DER.

The main difference in the approach is that ASN.1 specified semantics for the syntax, whereas in Thrift the default values are introduced to the definition language with no apparent or mandated semantics. The ASN.1 specifying tight semantics for any language construct fosters interoperability between conforming implementations. In the absence of formal definition, the Thrift generators choose to treat default values as largely pointless comments, and certainly do not realize their full potential.

The right approach here would be to:

• Require that “optional” fields with default values are treated as set by default, unless explicitly unset by the user. This would be consistent with current Java and PHP semantics.
• Allow skipping encoding of an actual value of an “optional” field with a default value if the actual value is structurally the same as the default value.

3. The OCaml generators are lagging behind

3.1. There is no account for requiredness flag or default values

class myStruct =
object (self)
  val mutable _myNoType : string option = None
  val mutable _myNoTypeDef : string option = None
  val mutable _myOpt : string option = None
  val mutable _myOptDef : string option = None
  val mutable _myReq : string option = None
  val mutable _myReqDef : string option = None
  val mutable _myEmptyListDef : string list option = None
  val mutable _myListDef : string list option = None
  val mutable _myDblDef0 : float option = None
  val mutable _myDblDefPi : float option = None
  method grab_myNoReq = match _myNoReq with
    | None -> raise (Field_empty "myStruct.myNoReq")
    | Some _x -> _x
  method set_myNoReq _x = _myNoType <- Some _x
  …
end

The “optional” and/or “required” fields with default values, on the other hand, need to be defined without that option/Maybe wrapping. Whereas I advocate changing the types of fields, I see a merit in retaining the uniform access to them through get_x/grab_x methods with conveniently uniform signatures.

But which requiredness should be transformed into a wrapper-less field, addressable without Some/None constructors and statically guaranteed to be exception-free? If we make a “required” field with specified default value wrapper-less, then we lose ability to detect whether the wire contained the value for such field. Thus, we lose protocol-centric semantics for the “required” field (that is, semantics which requires that the received wire encoding contains the value for a specific field). However, we gain the problem-domain-centric semantics, which is "make sure this field is never unset, for a peace of mind".

With the problem-domain-centric semantics we don't really need to always make sure that the field value is always present on the wire: if it is not present, it will be assigned the default value. See where I am going with that? There is no difference between “required”, “optional” and absent requiredness in case the field has a default value and we drop the notion of protocol-centric semantics for the requiredness. This should have been thought through before being put in the Thrift…

For the sake of clarity, we could have defined the field format as in the following simplified BNF:

  Field ::= RequiredField | OptionalField | DefaultValuedField
  RequiredField ::= "required" FieldType
  OptionalField ::= "optional" FieldType
  DefaultValuedField ::= FieldType "=" DefaultValue

I think that “required” and “optional” fields with default values do not have meaningful difference, at least for OCaml code, if we don't consider backward compatibility with existing C++ code base.

To maintain compatibility with C++, the requiredness flag can be used to distinguish between always sending the value (for “required” fields) and avoiding sending actual values equal to default (for “optional” fields).

3.2. Patch for the OCaml generator

• makes unwrapped types for all fields for which default values are defined,
• honors the protocol-centric semantics for the “required” field: such fields are always serialized,
• throws exception instead of skipping “required” fields which do not have a set value,
• avoids encoding values of “optional” and absent-requiredness fields if the field value is structurally equivalent to the field's default value,
• does not break compatibility with C++ receivers,
• is drop-in compatibe with OCaml code generated by the OCaml generator in the unmodified trunk

class myStruct =
object (self)
  val mutable _myNoType : string option = None
  val mutable _myNoTypeDef : string option = None
  val mutable _myOpt : string option = None
  val mutable _myOptDef : string option = None
  val mutable _myReq : string option = None
  val mutable _myReqDef : string option = None
  val mutable _myEmptyListDef : string list option = None
  val mutable _myListDef : string list option = None
  val mutable _myDblDef0 : float option = None
  val mutable _myDblDefPi : float option = None

  method grab_myNoReq = match _myNoReq with
    | None -> raise (Field_empty "myStruct.myNoReq")
    | Some _x -> _x
  method set_myNoReq _x = _myNoType <- Some _x

  method get_myOpt = _myOpt
  method grab_myOpt = match _myOpt with
    | None->raise (Field_empty "myStruct.myOpt")
    | Some _x -> _x
  method set_myOpt _x = _myOpt <- Some _x

  // Serializing
  (match _myOptDef with None -> () | Some _v -> 
    oprot#writeFieldBegin("myOptDef",Protocol.T_STRING,4);
    oprot#writeString(_v);
    oprot#writeFieldEnd
  );
  // Unsafe if myReq is not set!
  (match _myReq with None -> () | Some _v -> 
    oprot#writeFieldBegin("myReq",Protocol.T_STRING,5);
    oprot#writeString(_v);
    oprot#writeFieldEnd
  );
  // Unsafe if myReqDef is not set!
  (match _myReqDef with None -> () | Some _v -> 
    oprot#writeFieldBegin("myReqDef",Protocol.T_STRING,6);
    oprot#writeString(_v);
    oprot#writeFieldEnd
  );

  …

class myStruct =
object (self)
  val mutable _myNoReq : string option = None
  val mutable _myNoReqDef : string = "noreqdef"
  val mutable _myOpt : string option = None
  val mutable _myOptDef : string = "optdef"
  val mutable _myReq : string option = None
  val mutable _myReqDef : string = "reqdef"
  val mutable _myListDefEmpty : string list = []
  val mutable _myListDef : string list = ["listElement"]
  val mutable _myDblDef0 : float = 0.0
  val mutable _myDblDefPi : float = 3.1415

  method grab_myNoReq = match _myNoReq with
    | None -> raise (Field_empty "myStruct.myNoReq")
    | Some _x -> _x
  method set_myNoReq _x = _myNoType <- Some _x

  method get_myOptDef = Some _myOptDef
  method grab_myOptDef = _myOptDef
  method set_myOptDef _x = _myOptDef <- _x


  // Serializing
  (match _myOptDef with "optdef" -> () | _v -> 
    oprot#writeFieldBegin("myOptDef",Protocol.T_STRING,4);
    oprot#writeString(_v);
    oprot#writeFieldEnd
  );
  (match _myReq with 
  | None -> raise (Field_empty "myStruct._myReq")
  | Some _v -> 
    oprot#writeFieldBegin("myReq",Protocol.T_STRING,5);
    oprot#writeString(_v);
    oprot#writeFieldEnd
  );
  (
    oprot#writeFieldBegin("myReqDef",Protocol.T_STRING,6);
    oprot#writeString(_myReqDef);
    oprot#writeFieldEnd
  );

  …

You can download the patch here: http://lionet.info/patches/thrift-trunk-962854.patch.

4. Obligatory TL;DR section

https://issues.apache.org/jira/browse/THRIFT-827
https://issues.apache.org/jira/browse/THRIFT-860

P.S. The above patches have since been merged into Thrift.

P.P.S. This text first appeared in my LiveJournal post http://lionet.livejournal.com/66899.html.