Global Forge Computing, LLC InFIX Protocol Transformer

FIX Message Transformer

Find it on GitHub!

Table of Contents
  1. Introduction
  2. Features
  3. Terminology
  4. Examples
  5. Assignment
  6. Literals
  7. Templates
  8. Concatenation
  9. Arithmetic Operations
  10. Field Exchange
  11. Expressions
  12. Unary Operations
  13. Conditional Operations
  14. User Defined Behavior
  15. Repeating Groups
  16. Complex Operations
  17. Automatic Source Code Generation
  18. Custom Dictionaries
  19. Caveats

Introduction

Skip directly to examples

InFIX is a FIX Protocol Message transformer and powerful tool written in Java that allows applications to transform FIX message fields by applying actions defined in an easy to use text language. A rich set of operations including literal assignment, concatenation, math and boolean expressions, unary operations, templates, conditionals and user-defined behavior are supported.

InFIX is designed in a way that makes it possible to greatly extend it's supported features. Users will inevitably think of ways to create new behaviors and find it easy to build in support for their particular domain. As new challenges arrive the product will rapidly grow to meet those challenges. InFIX comes with a rich set of over 500 unit tests ready to be plugged into any continuous integration environment.

Features

InFIX allows FIX connectivity hubs to define sets of field and message transformations outside of application code and apply them at runtime. Data and actions are independent. Applications can define new actions statically and apply them without a code release.

InFIX greatly reduces the number of code releases needed when on-boarding new FIX clients and reduces the on-boarding time considerably.

InFIX can transform messages between different FIX versions and message types making it easy for connectivity hubs to transform messages between trading venues and acknowledge or reject orders from the edge effortlessly if application defined conditions are not met.

InFIX is built using Antlr. Antlr is a powerful parser generator and is the engine behind InFIX. It allows support teams to configure actions using an easy to understand syntax and apply them when necessary without developer involvement. Developers can extend or modify the language to define new actions and features and Antlr will generate the code.

InFIX can transform messages between different FIX versions and message types making it easy for connectivity hubs to transform messages between trading venues and acknowledge or reject orders from the edge effortlessly if application defined conditions are not met.

InFIX generates an understanding of FIX automatically by building it's data dictionaries directly from the FIX specification provided by the FIX Consortium. A two stage pipeline can be executed at any time to build java code from the FIX specification. In the first stage, XML data dictionaries are generated by parsing the FIX specification. Once the XML is generated, developers can modify them or create new custom dictionaries for in-house FIX version. In the second stage Java code is produced from the XML data dictionaries making inFIX aware of any custom or standard repeating groups at runtime.

Terminology

FIX message transformations are accomplished by parsing a FIX message, applying actions, and returning a new FIX message containing the changes specified by the actions.

Actions are written in a text language called, not surprisingly, "InFIX" and applied at runtime.

Actions belong to 1 of 4 categories: assignment , unary, conditional, and user defined operations.

Actions can be combined within an expression where they make sense and can be chained together as a series by separating them using a semicolon (;).

InFIX or "action syntax" is easy to understand and can be understood as follows:

InFIX actions operate on FIX Fields and always involve two concepts:

#1 - Referencing the field you wish to transform.
#2 - Specifying the action that defines the transformation.

Let's look at some examples.

Introduction to Actions and Examples

All of the below example actions can be plugged into the Sample Application. Try them out!

Assignment

<field reference> = <expression>

Fields are referenced in a FIX message by prefixing the tag number with an ampersand ("&").

&44 (reference to tag 44)
&60 (reference to tag 60)
&382[1]->&375 (reference to the second occurrence of tag 375 within a repeating group)

See Repeating Groups for help with repeating groups.

<expression> = Literal | Template | Concatenation | Addition | Subtraction | Multiplication | Division |
User Defined Behavior | <expression>

Examples:

Literals
&44 = 152.0500 Assign a number to tag 44
&72 = "CAD" Assign a string to tag 44. Alphanumerics must be quoted
&115 = &116 Assign the value of tag 116 to the value of tag 115
&46 = {com.globalforge.infix.example.ExampleUserAssignment} Assign tag 46 to a value returned by a custom class **
&110 = (int) &110 Truncate a floating point value

** See User Defined Behavior for help with user defined values.

Templates
&60 = <DATETIME> 60=current time stamp (YYYYMMDD-HH:MM:SS.sss)
&75 = <DATE> 75=current date (YYYYMMDD)

Concatenation Operator ( | )
&46 = &46 | 1 46 = value of tag 46 concatenated with "1"
&46 = &46 | "FOO" 46 = value of tag 46 concatenated with "FOO"
&46 = "FOO" | "BAR" 46 = "FOOBAR"
&10016 = &55 | "-" | &46 10016 = value of tag 55 + "-" + value of 46

Field Exchange Operator ( <-> )
&49 <-> &56 Exchange values between tags 49 and 56

Arithmetic Operators ( + - * / )
&44 = 1 + 3 44 = 4
&44 = &44 + 3 44 = value of tag 44 + 3
&44 = &44 * 3 44 = value of tag 44 * 3
&44 = &44 / &43 44 = value of tag 44 / value of tag 43
&44 = &44 - {com.globalforge.infix.example.ExampleUserNumberAssignment} 44 = value of tag 44 - runtime discount
&44 = &44 * (1 + 3) 44 = value of 44 * 4
&44 = (&44 * 1) + 3 44 = value of 44 + 3

Combined Expression
&44 = (((((&44 * 1) + 3) - 0.0500) + 10) * 2) / &43 | "0000" 44 = -1.00000000 **

** Demonstrates the recursive attribute of <expression>.

Precedence of Operators

Multiplication and Division ( *, / )
Addition and Subtraction ( +, - )
Concatenation ( | )

Unary Operations

~ <field reference>
~ &[tag number, tag number, tag number,...]
+ &[tag number, tag number, tag number,...]

The delete operator ( ~ ) deletes a field reference or a set of given tag numbers from a FIX message.
The keep operator ( + ) deletes all fields from a FIX message except the tag numbers specified.

Examples:

Unary Operators **
~&43 Removes tag 43 from a FIX message.
~&[43,44,115,116] Removes tags 43,44,115,116 from a FIX message.
+&[43,44,115,116] Removes all tags from a FIX message except 8,9,43,44,115,116, and 10.

** Delete will never remove tag 8. Tags 9 and 10 will always be recalculated regardless of action.

Conditional Operators

if <condition> [ AND | OR <condition> ]* then <actions> else <actions>

Conditions
== is equal
!= not equal
< less than
> greater than
<= less than or equal to
>= greater than or equal to
! not
^ is
&& AND
|| OR
? IF/THEN
: ELSE

Examples:

Conditional Operators and Operation Sets
&43 == -100 ? &43 = 3 : &207 = "CAD" IF tag 43 equals -100 then set 43=3 ELSE set 207=CAD
&43 != -100 ? &43 = 3 : &207 = "CAD" IF tag 43 not equal to -100 then set 43=3 ELSE set 207=CAD
!&102 ? &102 = "FOO" IF tag 102 IS NOT present then add field 102=FOO
^&43 ? &207 = "AUS" IF tag 43 IS present then set field 207=AUS
&45 == 0 || &43 == -100 && &207 == "USA" ? &45=1 : &48=1 IF tag (45 == 0 OR 43 == -100) AND 207=USA THEN set 45=1 ELSE set 48=1
&43 < 0 || &44 >= 37.0000 ? &45 = 1.3214 : &48 = 1 IF tag value of 43 less than 0 or value of 44 greater than or equal to 37 THEN set 45=1.3214 ELSE set 48=1
&43 == -100 ? [&43 = 3; &207 = "CAD"] : [&43 = 4; &207 = "USA"] IF tag 43 equals -100 then do a set of actions ELSE do a different set of actions

User Defined Behavior

User defined behaviors come in two flavors: Stand-alone actions and actions embedded in expressions.

Stand-alone actions are simply method calls that don't exist as part of any expression but, nevertheless, hand you control as actions are being parsed.

In the below example, tag 207 is assigned a value and then control is passed to your class, once your class returns, transformations continue where tag 115 is assigned a value of "COMPID". The FIX message is given to your class and whatever changes you make to it are present before the next action starts.

&207="USA" ; {com.globalforge.infix.ExampleUserContext} ; &115="COMPID"

In the Sample Application, the above will assign field 44 to the natural logarithm (e) but there is no limit to how many fields you can affect with a user defined action. User defined actions allow you to execute custom code and do whatever you want to a FIX message as it is being transformed. There is an associated API that comes with user defined actions that you conform to. We'll peek at that later.

User defined actions embedded in expressions always return a literal.

&44={com.globalforge.infix.ExampleUserAssignment}

Lets take a look at the code for ExampleUserAssignment first.

public class ExampleUserAssignment implements InfixUserTerminal {
    @Override
    public String visitTerminal( InfixAPI infixApi ) {
        return Double.toString( Math.PI );
    }
}

As you can see it simply returns a value. How it generates a value is up to the implementer.

public interface InfixUserTerminal {
    String visitTerminal( InfixAPI infixApi ) ;
}

Price algorithms may be swapped at runtime (provided there is a means to reload actions) and differ according to client or other business requirements. All that is needed is to implement the interface beforehand and then you just specify the action where needed.

InfixUserTerminal may be used in assignment, conditional, or unary expressions.

Now lets take a look at the code for ExampleUserContext. In the below example there are two methods implemented. Each method demonstrates a unique way of transforming a FIX message. The first method, visitMessage(), assigns tag 44 a value of PI. It accomplishes this by parsing the entire FIX message. This technique is useful when much custom code is needed say for a particular client. The second technique, visitInfixApi(), assigns tag 44 a value of E. This technique is the preferred way as it performs in constant time. Notice that tag 44 starts with '&'. When referencing tags using the InfixAPI, you must use action syntax; the same syntax you use when specifying actions. This holds true when referring to individual tags in repeating groups.

InfixUserContext is meant for stand-alone actions. Neither of the methods return an individual value so they can't be used in assignment, unary, or conditional expressions the way InfixUserTerminal can but they provide a valuable hook into the runtime state of the action parser allowing for complex user defined transformations.

/**
 * An example implementation showing how to use user defined behavior
 * @author Michael C. Starkie
 */
public class ExampleUserContext implements InfixUserContext {
    @Override
    /**
     * User defined but the message returned must be legal for the fix version
     * otherwise the behavior is undefined.
     * @param fixMessage The message to manipulate with java code
     * @return String The new message to return to the parser.
     */
    public String visitMessage(String fixMessage) {
        return calculatePrice(fixMessage);
    }

    /**
     * Demonstrate how to change the state of a fix message as it resides in
     * memory before final transformation.
     * @param InfixMappingAPI A handle into the infix internals.
     * @see InfixAPI
     */
    @Override
    public void visitInfixAPI(InfixAPI infixApi) {
        infixApi.putContext("&44", Double.toString(Math.E));
    }

    /**
     * Demonstrate that you can write java to be called during a rule parse.
     * @param baseMsg The message to manipulate with java code
     * @return String The new message to return to the parser.
     */
    private String calculatePrice(String baseMsg) {
        double price = Math.PI;
        String[] msgArray =
            Pattern.compile(Character.toString((char) 0x01), Pattern.LITERAL).split(baseMsg);
        StringBuilder newMsg = new StringBuilder();
        for (String field : msgArray) {
            int tagNum = Integer.parseInt(field.substring(0, field.indexOf("=")));
            if (tagNum == 44) {
                newMsg.append("44=" + price);
            } else {
                newMsg.append(field);
            }
            newMsg.append((char) 0x01);
        }
        return newMsg.toString();
    }
}
		

Let's take a peek at InfixUsercContext

public interface InfixUserContext {
    String visitMessage(String fixMessage);
    void visitInfixAPI(InfixAPI infixApi);
}
		

How does the runtime system know which method to call?

visitInfixAPI() is the default. To call visitMessage() you must explicitly call it in an action using the '#' char as shown below.

Try them out using the Sample Application.

User Defined Behavior Summary
&44={com.globalforge.infix.example.ExampleUserAssignment} calls visitTerminal() which assigns tag 44 = E
{com.globalforge.infix.example.ExampleUserContext} calls visitInfixAPI() which assigns tag 44 = E
{com.globalforge.infix.example.ExampleUserContext#visitMessage} calls visitMessage() which assigns tag 44 = PI

Finally let's look at InfixAPI to get a understanding of what options are available at runtime.

package com.globalforge.infix.api;

package com.globalforge.infix.api;

import java.util.LinkedHashMap;
import java.util.Map;


/**
 * An interface that provides a limited number of operations on the in-memory
 * state of a FIX message as actions are being parsed. This class should never
 * be called by an application thread or any thread other than the engine thread
 * making the callback. 
* No control should be passed to any other thread during any operation defined * in this interface. Any deviation from this policy will produce undefined * results and may lead to data corruption or runtime exceptions.
* THREAD SAFETY: NOT SAFE. * @author Michael C. Starkie */ public interface InfixAPI { /** * Removes a FIX field from the in-memory message. * @param ctx The tag number in action syntax (e.g., &44). */ public void removeContext(String ctx); /** * Inserts a FIX field. * @param key The tag number in Infix syntax (e.g., &44) * @param value The tag value (e.g., 42.0000) */ public void putContext(String key, String value); /** * Returns an object containing the Field field, including tag number and * value as well as it's relative order within the FIX message * @param ctx The tag number in action syntax (e.g., &44). * @return InfixField Tag number and tag value. * @see InfixField */ public InfixFieldInfo getContext(String ctx); /** * Insert FIX fields into the parsed message. Keys are tag numbers in action * syntax and values are the tag values associated with the keys. This * method will replace any fields already parsed. Insert order must be * preserved for the integrity of repeating groups but it's up to the caller * to ensure order of keys. * @param msgDict LinkedHashMap The FIX fields to insert in * the form of a dictionary of tag numbers in action syntax to tag values. */ public void putMessageDict(LinkedHashMap msgDict); /** * Get the runtime data dictionary * @return Map */ public Map getMessageDict(); /** * Obtain a fully formatted FIX message representing the memory state as it * currently exists during a action parse. * @return String a fully formatted FIX message. */ public String getMessage(); }

Example Use Case

For a particular customer, always copy tag 115(OnBehalfOfCompID) onto tag 116(OnBehalfOfSubID) and tag 49(SenderCompID) onto tag 115(OnBehalfOfCompID) before retransmitting a message. You can test the below using the Sample Application.

String actionsStr = "&116=&115 ; &115=&49";
InfixActions actions = new InfixActions(actionsStr);
String outputFIXMessage = actions.transformFIXMsg(inputFIXMessage);
            

That's pretty much it. Ideally you would read the sampleRule from a persistent data store and initialize the FixRulesEngine at application start-up then simply call transformMsg() after applying your business logic and just prior to retransmitting the message.

Repeating Groups

Every repeating group in FIX Protocol starts with a field that indicates the number of repeating groups that follow. This is referred to as the "GroupID" in InFIX. Members of a group are always referenced using the GroupId, an array index indicating the nesting level and a pointer to the member field reference within the group.

For example, ContraGrp is a repeating group found in Execution Report. ContraGroup is always preceded by 382 (NoContraGrps) and followed by the member fields which can repeat. Therefore 382 is considered the GroupID of ContraGrp and serves as the array indicating the nesting level in which the member tag being referenced is to be found. Members of ContraGrp like Field 375 can repeat whereas the GroupId 382 can not.

Assume 382=2 in the input FIX message. This means two repeating ContraGrps follow immediately after tag 382.

Example 1:
8=FIX.4.4|9=10|35=8|382=2|375=FOO|655=2|375=BAR|655=3|10=004

To reference the first occurrence of 375 use the following syntax:
&382[0]->&375="FOOBAR" (assign "FOOBAR" to the first occurrence of tag 375)

Result:
8=FIX.4.4|9=10|35=8|382=2|375=FOOBAR|655=2|375=BAR|655=3|10=004";

To reference the second occurrence of 375 use the following syntax:
&382[1]->&375="FOOBAR" (assign "FOOBAR" to the second occurrence of tag 375)

Result:
8=FIX.4.4|9=10|35=8|382=2|375=FOO|655=2|375=FOOBAR|655=3 |10=004";

&382[1] (The second ContraGrp)
&382[1]->&375 (tag 375 inside the second ContraGrp).

&382[2]->&375 would be an error since there are only two ContraGrps indicated by 382=2 and indices start at 0.

Example 2 (Nested Groups):
NestedParties is a repeating group that can be nested inside the InstrmtLegExecGrp repeating group.

[InstrmtLegExecGrp], id = [555], members = [600|601|602|...]
[NestedParties], id = [539], members = [524|525|538]

555=2|600=FOO|601=2|539=2|524=STR|525=8|538=-33|524=MCS|525=22|538=33|600=FOO1|601=3|539=1|524=STR1|525=0|538=-34

Suppose you wanted to change 524=STR to 524=XYZ?

555=2|600=FOO|601=2|539=2|524=STR|525=8|538=-33|524=MCS|525=22|538=33|600=FOO1|601=3|539=1|524=STR1|525=0|538=-34

You would specify the action like this: &555[0]->&539[0]->&524="XYZ"

Applying the action yields the following result:

555=2|600=FOO|601=2|539=2|524=XYZ|525=8|538=-33|524=MCS|525=22|538=33|600=FOO1|601=3|539=1|524=STR1|525=0|538=-34

&555[0] = fist instance of InstrmtLegExecGrp
&539[0] = first nesting of NestedParties inside 555
&555[0]->&539[0]->&524 = tag 524 in the first nesting of NestedParties inside first nesting of InstrmtLegExecGrp.

Example 3 (Nesting at Multiple Levels):
NstdPtysSubGrp is a repeating group that can be nested inside NestedParties which can be nested inside InstrmtLegExecGrp.

[InstrmtLegExecGrp], id = [555], members = [600|601|602|...]
[NestedParties], id = [539], members = [524|525|538]
[NstdPtysSubGrp], id = [804], members = [545|805]

555=2|600=FOO|601=2|539=2|524=STR|525=8|538=-33|524=MCS|525=22|538=33|804=2|545=THEIR_ACCT1|805=1|545=THEIR_ACCT2|805=1

Suppose you wanted to change 545=THEIR_ACCT1 to 545=MY_ACCT1?

You would specify the action like this: &555[0]->&539[1]->&804[0]->&545="MY_ACCT1"

&555[0] = fist instance of InstrmtLegExecGrp
&539[0] = first nesting of NestedParties inside 555
&804[0] = first nesting of NstdPtysSubGrp inside 539
&555[0]->&539[0]->&804[0]->&545 = tag 545 in the first instance of 804 inside first instance of 539 inside the first instance of 555

Applying the action yields the following result:

555=2|600=FOO|601=2|539=2|524=STR|525=8|538=-33|524=MCS|525=22|538=33|804=2|545=MY_ACCT1|805=1|545=THEIR_ACCT2|805=1

See you if can change 545="THEIR_ACCT2" to 545="MY_ACCT2" using the Sample Application.

Now you know how to reference and transform any field within a FIX protocol message.

Complex Operations

Adding a Repeating Block:
The following action adds a repeating group (Underlying Instrument) when transforming a message. It assumes we're sure the repeating group will not exist on the inbound FIX message. See next example for the case where not sure to see how to test for existence first.

Add a Repeating Group To a Message
&711=1;&711[0]->&311="AMZN";&711[0]->&309=3672345

Result:
711=1 | 311=AMZN | 309=3672345

Adding a Hop:
The following action states that if HopGrp is received on a FIX message then append our own Firm's information on the Hop.

Add a Hop to an existing Hop Group
^&627 ? [&627=&627+1; &627[&627]->&628="NEWCOMPID"; &627[&627]->&629=<DATETIME>;&627[&627]->&630=&34]

Let's break it down:

[HopGrp], id = [627], members = [628|629|630]

^&627 ? (IF we see the Group Id for HopGrp on the inbound message THEN do the following)
&627=&627+1 (increment the Group Id (NoHop) because we're appending a block to the group)
&627[&627] (we can use the value of &627 as an index after we increment it as a pointer to tags in the new block)
&627[&627]->&628="NEWCOMPID" (add HopCompID to the new block)
&627[&627]->&629=<DATETIME> (add HopSendingTime and give it a timestamp)
&627[&627]->&630=&34 (set HopRefID to our sequence number).

To summarize: if we receive this:
35=8 | 627=1 | 628=SenderCompID | 629=20141013-08:13:50.836 | 630=5656 | 43=-100

Then we'll transform it into this:
35=8 | 627=2 | 628=SenderCompID | 629=20141013-08:13:50.836 | 630=5656 | 628=NEWCOMPID | 629=20141013-08:18:51.909 | 630=100 | 43=-100

Otherwise we won't add a Hop.

Here is the rule which say's "if we receive a Hop then add to it otherwise start the group"

Add a Hop to an existing Hop Group otherwise start one
^&627 ? [&627=&627+1; &627[&627]->&628="NEWCOMPID"; &627[&627]->&629=<DATETIME>; &627[&627]->&630=&34] : [&627=1; &627[0]->&628="NEWCOMPID"; &627[0]->&629=<DATETIME>; &627[0]->&630=&34;]

To summarize: if we don't receive a Hop we add it:
627=1 | 628=NEWCOMPID | 629=20141014-07:28:39.134 | 630=99

It's left as an excersize for the reader to try it out.

Source Code Generation

InFIX builds it's knowledge of FIX directly from the FIX specification through a series of build steps.
The final build step produces a java library that you link with your application.
The below diagram illustrates this process.

FIX Message Transformer

Infixes uses standard FIX data dictionaries offered by Quick FIX to build it's Java source code. The Quick FIX data dictionaries are actively used by the Open Source community and serve as the most reliable XML source of the FIX specification. Each data dictionary file is parsed by the Dictionary Parser into Java source code. Adding a custom data dictionary for a proprietary FIX version is as simple as copying one of the standard Quick FIX dictionary files and adding your custom data. Running the Dictionary Parser against your custom dictionary file will produce the necessary Java code to support your unique FIX version.

Custom Dictionaries

Custom data dictionaries for proprietary FIX versions are fully supported by InFIX as described in the section above.

Caveats

If a field reference is used in an action other than on the left side of an assignment and the field reference does not exist in the original FIX message, then the entire action is ignored. If you are unsure if a field will be present in an input message, test for existence first using IF/THEN and one of the existence operators ( '^' | '!' ). See next caveat.

If a field that does not exist is referenced on either side of a conditional operation the result is false.
Example: &39 != 0 will be false when tag 39 does not exist in the input FIX message. The proper way to handle
this is to check for existence first: !&39 || &39 != 0 ? &41=1 : &41=2. In this case the expression is true when tag 39 is not present.

For repeating groups, if a nested field reference is used on the left side of an assignment and the group for which the field does not exist, the field is added to the message just like a non repeating field. However the field will exist independently (not a part of a group) and will most likely result in an invalid message. If the intention is to start a group then the GroupID must be added first and then each tag belonging to the group can be assigned using the GroupID.