ASN.1 Exposed — latest articles

Question: How to use streaming?

Date: 2010-September-22 @ 20:42
Tags: streaming, support

A Forum post raises a question of the practical usage of streaming in the ASN.1 code generated by asn1c:

Hello, I want to use the protocol to send a lot of data, but the device isn't able to hold the whole amount of received data in RAM. It has to work only with parts of the structure at a time.

For example,

BigMessage ::= SEQUENCE {
    time-stamp UTCTime,
    messages SEQUENCE OF SmallMessage
}

If I get only the first half of the Big Message, I want to work with all received messages and then I have to to free my buffer due to limited resources. When I get the second part, I want to work on it and I want to know, if the message was valid...

On the encoder site, I also have to send "half" messages.

How can I do this?

The ASN.1 streaming is easily the most exciting thing the asn1c has to offer. It's not something that's needed too often, but when it does, it's invaluable. Let's split the problem into two parts: how to decode in constant space and how to encode in constant space.

(Continue reading…)

Question: Extensibility and dropping fields

Date: 2010-September-21 @ 16:42
Tags: extensibility, support

Today's email was asking me about an interesting case of ASN.1 extensibility. Usually, extensibility means adding new members to existing structures. In this case, the user was asking whether ASN.1 supports removing structure members.

Hello, Mr. Walkin.

I am using the ASN.1 compiler and for an application I am developing. I am using this to store the datastructure in a file, for persistence. I am trying to find out whether forward compatibility is support. By this I mean the following.

In the version 1 I have the following structure:

Struct a {
	Int x;
	Int y;
	Int z;
}

I store the data x = 111; y = 222 and z = 333;

Now I upgrade the software to the version 2 where in the new struct is:

Struct a {
	Int x;
	// int y is removed
	Int z;
}

Now when I read the existing file (generated with the older version of ASN.1 specification) with the newer version of the software, I get the following data back: x = 111, and z = 222.

Can you please tell me whether it is possible to get x = 111 and z = 333. If yes, then what ASN syntax should I follow? I am using der_encode and ber_decode.

(Continue reading…)

On FaceBook's Thrift semantics, code generation, and OCaml

Date: 2010-July-18 @ 23:42
Tags: asn.1, ocaml, thrift

Note: The obligatory TL;DR section is at the very end of this text.

The ASN.1 rant

After co-founding Echo, I had to put the asn1c's development on hold, for the sheer lack of time. (If you don't know what my asn1c is, think of it as the most evolved open source ASN.1 compiler.) Despite suspending development, I've been tracking the ASN.1 evolution, as well as the emergence of some newer technologies competing with what ASN.1 has to offer. I am referring to FaceBook's Thrift, Google's Protocol Buffers, Cisco's Etch, and the likes. Yet to this day I had no opportunity to actually use any of those in production.

I have my own personal score with the ASN.1 world. The standard is laden with design-by-committee complexity, no doubt evolved to “address the real world demands”. Its mind-numbing standardese and semantics effectively prohibit any newcomer from ever entering this field and producing a decent new compiler. So, we're stuck with something like 2.5 alive ASN.1 compilers covering some 3 mainstream languages (C++, C#, Java). The commercial products are often cost prohibitive: they're squarely aimed at rich telecom market. Where would a small Ruby or Python startup go? (While I am at it, Erlang has a free decent ASN.1 compiler, you know).

Yet, there's an opposite side to this complexity. Many things you struggle with or “invent” for the purpose of better data serialization have already been invented in the ASN.1 world. Things like broiled-to-perfection TLV-based encodings (BER/DER/CER), bitwise Packed Encoding Rules (competing with gzip'ing your serialized binary blob), Information Object Classes (think of SNMP MIB macros on steroids), or Encoding Control Notation have a lot to offer and learn from.

But I should stop kicking that dead horse. Let's try Thrift for a change.

(Continue reading…)