# Binary Parser Parse fixed-format binary payloads into structured data. Binary Parser is a function for Soracom Air devices that parses binary data with a fixed format and converts the data into easy-to-use JSON data, which can then be passed to Soracom Beam, Funnel, Funk, and Harvest Data. Binary Parser is available for Air for Cellular, Air for Sigfox, and Air for LoRaWAN. When using Binary Parser, you must select either a **Predefined Format** or define a **Custom Format**. ## How Data Is Parsed With the Binary Parser option enabled, the binary payload sent from a device to Beam, Funnel, Funk, Harvest Data, or Unified Endpoint is automatically parsed into a JSON object, and the resulting JSON data is sent to the configured destination. There is no need to modify the Beam, Funnel, Funk, or Harvest Data destination configuration. ![Binary Parser Mechanism](https://docs.soracom.io/_astro/mechanism.CA85itS1_Z1a47ub.webp) When binary data is converted to JSON, a `binaryParserEnabled: true` attribute will be added to the JSON data. ## Configuration The Binary Parser option is found within Soracom Air Group settings. > [!NOTE] > > Binary Parser settings are applied to all device types in the same group. Therefore, configuring Binary Parser for cellular devices in a group will also apply the same Binary Parser configuration for Sigfox and LoRa devices in the same group. If you require multiple Binary Parser settings, create separate groups. 1. Sign in to the **[User Console](https://console.soracom.io/?coverage_type=g)**. From the **☰ Menu**, open the **Groups** screen. 2. From the list of groups, click the **Name** of the group you want to configure to open its settings page. 3. From the **Basic Settings** tab, click the **SORACOM Air for Cellular**, **SORACOM Air for Sigfox**, or **SORACOM Air for LoRaWAN** panel to expand its settings. ![Binary Parser](https://docs.soracom.io/_astro/binary-parser-option.D7CVH3Mb_2eu5oC.webp) 4. Enable the **Binary Parser** option by switching the option to **ON**. 5. Enter the Binary Parser **Format** to use. 6. Click **Save** at the bottom of the panel. ### Options - **Format** - A **Predefined Format** or **Custom Format** string that defines how Binary Parser should parse binary data. ## Predefined Format Predefined formats are available for select devices and libraries. When using a supported device or library, simply enter the **Format** string when configuring Binary Parser settings. Supported devices: | Device | Format | Notes | | - | - | - | | [Soracom LTE-M Button](https://docs.soracom.io/en/devices/soracom-lte-m-button) | `@soracom.button` | Supports Soracom LTE-M Button | | [Sigfox Sens'it v2](https://soracom.jp/products/sigfox/sensit/) | `@sensit` | Supports Sens'it v2 | | [Sigfox Sens'it v3 (Discovery)](https://support.sigfox.com/docs/sens'it-datasheet) | `@sensit.v3` | Supports Sens'it v3 | | [Sigfox Shield for Arduino](https://soracom.jp/products/sigfox/sigfox_shield_for_arduino/) | `@unashield` | Supports data sent using the [unabiz-arduino](https://github.com/UnaBiz/unabiz-arduino) library; other data formats not supported | | (any) | `@msgpack` | Supports data that is compressed using [MessagePack](https://msgpack.org/index.html); data must be sent to [Unified Endpoint](https://docs.soracom.io/en/services/unified-endpoint) in order for Binary Parser to decompress | | (any) | `@json` | Supports data that is in JSON format | | (any) | `@cbor` | Supports data that is in CBOR format, as specified in [RFC 8949 Concise Binary Object Representation](https://cbor.io/); data containing CBOR Tags extensions used for representing datetime types and large integers (Big Integer) cannot be processed and will return an error | Refer to the [Soracom blog](https://blog.soracom.jp/blog/2017/09/08/binary-parser-predefined-format/) for an example of using Binary Parser with a supported device. ## Custom Format When defining a custom format, individual values can be parsed from the payload using the following definition: `[name]:[byte index]:[type]:[options]` Multiple values can be parsed by creating multiple definitions, each separated with a space. For example, the following **Format** definition will produce a JSON object that looks like this: ``` flag:0:bool:7 temp:1:int:13:/10 humid:3:uint:8:/100 lat::float:32 long::float:32 ``` ```json { "flag": true, "temp": 18.0, "humid": 80.0, "lat": 39.8283, "long": 98.5795 } ``` ### Name The JSON object **key** where the parsed value should be stored. Available characters: letters, numbers, `-`, `_`, `#`, and `$`. Note that the following words cannot be used, as they are reserved for use in Sigfox and LoRaWAN paylods: - **Sigfox** reserved words: `device`, `time`, `station`, `lat`, `lng`, `rssi`, `data`, `seqNumber`, and `data` - **LoRaWAN** reserved words: `date`, `gatewayData`, `data`, and `deveui` #### Byte Index Defines which byte of the payload should be read in order to parse the value, with `0` corresponding to the first byte of the payload. #### Type Defines the type to use for the value when parsing data. The following value types are available: - `bool` - returns `true` or `false` - `int` - returns a signed integer - `uint` - returns an unsigned integer - `float` - returns a decimal value - `char` - returns an ASCII character #### Options Available options vary depending on the defined type. For types that support multiple options, each option should be separated using a colon `:`. #### **bool** The **bool** type will read one bit within the specified byte, and return `true` if the bit is 1, or `false` if the bit is 0. ##### Options - **offset** (_required_, min: `0`, max: `7`) - indicates which bit should be used to parse the boolean value, where: - `7` will read the left-most bit of the byte - `0` will read the right-most bit of the byte For example, with a byte value of `0x9A` or `10011010b`, the Binary Parser format `flag:0:bool:7` will: - read the first byte of the payload (byte index: `0`), - read the left-most bit of the byte (offset: `7`), - convert the bit value to a boolean (type: `bool`), - assign the value to a key named `flag` (name: `flag`) ![Binary Parser Format Offset](https://docs.soracom.io/_astro/offset.Csd-cOcj_2tgQUO.webp) ##### **int, uint** The **int** and **uint** types will read a specified number of bits and convert the signed or unsigned binary value to an integer. When specifying `int`, the first (left-most) bit of the bit sequence is used as the sign of the integer value, with `0` returning a positive number, and `1` returning a negative number. ##### Options (int, uint) - **length** (_required_, min: `1`, max: `32`) - the number of bits to read when parsing the integer value. If the number of bits (including any offset) exceeds the number of bits in the byte, the bit sequence will continue to the next byte. - **offset** (_optional_, min: `0`, max: `7`, default: `7`) - indicates which bit should be used as the start of the bit sequence, where: - `7` will begin reading the bit sequence from the left-most bit of the byte - `0` will begin reading the bit sequence from the right-most bit of the byte Examples: - Read the last four bits of a byte by specifying a **length** of `4` with an **offset** of `3`: `temp:0:uint:4:3`. - Read the last ten bits of a 2-byte sequence by specifying a **length** of `10` with an offset of `1`: `temp:0:int:10:1`. - **operations** (_optional_, limit: 5 operations) - a simple arithematic expression to evaluate on the parsed integer value, supporting: - `+` to add a number to the previous value - `-` to subtract a number from the previous value - `*` to multiply the previous value by a number - `/` to divide the previous value by a number - (any number) to use in the operation Expressions are evaluated left-to-right, and order of operations (such as paranthetical evaluation) is not supported. For example, an expression `+10*2` will first parse a binary value to an integer value, then add `10` to the parsed value, then multiply the result by `2`. When dividing, the resulting value is converted to a decimal value. ##### **float** The **float** type will read a specified number of bits and convert the single- or double-precision float value to a decimal using the corresponding IEEE 754 standard. **Options** - **length** (_required_, `32` or `64`) - the number of bits to read when parsing the float value - **offset** (_optional_, min: `0`, max: `7`, default: `7`) - same as the **int** and **uint** offset parameter - **operations** (_optional_, limit: 5 operations) - same as the **int** and **uint** operations parameter ##### **char** The **char** type will convert a bit value to the corresponding ASCII value. **Options** - **length** (_required_, min: `1`, max: `8`) - the number of bits to read when parsing the ASCII value ## Advanced Formatting ### Endianness By default, Binary Parser will parse numerical values using **big-endian** format, with most-significant bit first and least-significant bit last. You can override this behavior to parse the bit sequence using **little-endian** by appending `:little-endian` to the end of the format definition. Example: `value:0:float:64:little-endian` ### Byte Index Shorthand Normally, the byte index defines which byte in the payload contains the value to be parsed. However, the byte index can be left blank using `::`, which will cause the Binary Parser to use the next byte as the byte index, following the previously-used byte index. For example, with 2-byte payload and a definition `value1:0:int:6 value2::int:8`: - **value1** will be parsed using the first `6` bits of the first byte (byte index: `0`), and - **value2** will be parsed using all `8` bits of the second byte; - the last two bits of the first byte are ignored However, when using the byte index shorthand with multiple consecutive definitions that are type `bool`, Binary Parser will _re-use_ the previously-used byte index. Any non-`bool` definition after a sequence of `bool` definitions will resume the normal byte index shorthand behavior of using the next byte index. This allows you to read each of the individual bits of a byte as separate boolean values, using a definition such as: ``` flagA:0:bool:7 flagB::bool:6 flagC::bool:5 value:0:int:5:4 ``` This format will parse the following values: - **flagA** will be parsed using the first bit (byte offset: `7`) of the first byte (byte index: `0`); - **flagB** will be parsed using the second bit (byte offset: `6`) of the same byte (byte index shorthand); - **flagC** will be parsed using the third bit (byte offset: `5`) of the same byte (byte index shorthand); - **value** will be parsed using the last five bits (byte offset: `4`, length: `5`) of the first byte (byte index: `0`) Note that when using the byte index shorthand with multiple type `bool` definitions, Binary Parser _will not_ keep track of bit offsets, and will continue to re-use the same byte index for all consecutive type `bool` definitions. For example, to parse the first ten bits of a 2-byte sequence as boolean values, you can use the byte index shorthand for the 2nd-8th definitions. However, you must re-specify the byte index in the 9th definition in order to move the byte index forward. Afterwards, you can use the byte index shorthand again for the 10th definition. ## Limitations - When using Binary Parser with Beam, the following protocol conversions are supported: - TCP to HTTP/HTTPS entry point - UDP to HTTP/HTTPS entry point - When using Binary Parser with Funnel or Harvest Data, the following forwarding protocols are supported: - TCP - UDP - While there is no forwarding byte size limit, network MTU may affect parsing. - When using [Unified Endpoint](https://docs.soracom.io/en/services/unified-endpoint), it can be used together with [Soracom Binary Format v1](https://docs.soracom.io/en/services/binary-format-v1). - When not using [Unified Endpoint](https://docs.soracom.io/en/services/unified-endpoint), only Beam, Funk, and Harvest Data can be used together with [Soracom Binary Format v1](https://docs.soracom.io/en/services/binary-format-v1). ## Error Handling Any parse errors are stored in an **errors** key within the JSON object. Soracom does not provide any tools for inspecting or debugging binary data. ## Advanced Configuration Binary Parser settings can also be configured through the Soracom API or CLI by using the **SoracomAir** namespace. > [!NOTE] > > Binary Parser settings are shared among all device types (cellular, Sigfox, and LoRaWAN) within the same group. If you require different Binary Parser settings, create separate groups for each device type. ### Configuration Structure ```json "binaryParserEnabled": true|false, "binaryParserFormat": "", ``` ### Parameters Enable or disable Binary Parser: - **key** - `binaryParserEnabled` - **value** (_boolean_, default: `false`) - Enables or disables the Binary Parser option. Modify the Binary Parser format: - **key** - `binaryParserFormat` - **value** (_string_, default: `""`) - Defines the parser format to use when Binary Parser is enabled. ### Example The following configuration will set and enable Binary Parser: ```json [ { "key": "binaryParserEnabled", "value": true }, { "key": "binaryParserFormat", "value": "flag:0:bool:7 temp:1:int:13:/10 humid:3:uint:8:/100 lat::float:32 long::float:32" } ] ``` ## Programmatic Usage You can use the advanced configuration structure to configure the Binary Parser option programmatically using the Soracom API and Soracom CLI. ### Soracom API To access the Soracom API, first use the [**auth**](https://docs.soracom.io/en/api#!/Auth/auth) API to obtain an API Key and Token. Refer to the [**API Usage Guide**](https://docs.soracom.io/en/developers/soracom-api) for instructions on how to use the API Key and Token in API requests. Then, use the [putConfigurationParameters](https://docs.soracom.io/en/api#!/Group/putConfigurationParameters) API with the `SoracomAir` namespace to set the Binary Parser option: **Global** ```bash curl -X PUT \ -H 'X-Soracom-API-Key: ' \ -H 'X-Soracom-Token: ' \ -H 'Content-Type: application/json' \ -d '[ { "key": "binaryParserEnabled", "value": true }, { "key": "binaryParserFormat", "value": "flag:0:bool:7 temp:1:int:13:/10 humid:3:uint:8:/100 lat::float:32 long::float:32" } ]' \ https://g.api.soracom.io/v1/groups//configuration/SoracomAir ``` **Japan** ```bash curl -X PUT \ -H 'X-Soracom-API-Key: ' \ -H 'X-Soracom-Token: ' \ -H 'Content-Type: application/json' \ -d '[ { "key": "binaryParserEnabled", "value": true }, { "key": "binaryParserFormat", "value": "flag:0:bool:7 temp:1:int:13:/10 humid:3:uint:8:/100 lat::float:32 long::float:32" } ]' \ https://jp.api.soracom.io/v1/groups//configuration/SoracomAir ``` ### Soracom CLI To use the Soracom CLI, you must first configure it to authenticate with your account information, authorization key, or SAM user credentials. Then, run the following command to set the Binary Parser option: **Global** ```bash soracom groups put-config --group-id "" --namespace "SoracomAir" --body "" --coverage-type g ``` **Japan** ```bash soracom groups put-config --group-id "" --namespace "SoracomAir" --body "" --coverage-type jp ``` Here, the JSON array from the API example above should be passed in to the `--body` parameter.