Choosing backend based on tcp payload

@gagan_goku wrote:

My company has multiple development environments based on multiple features being tested. Instead of everyone maintain and point to each other’s respective feature environment, I am thinking of setting up a whitelisted user based routing setup. For example:

Screen Shot 2017-09-22 at 8.17.42 PM.jpg1150x1142 88.7 KB

This is easier to do for http requests because haproxy / nginx easily support routing http requests to backends based on the value of say a user_id header.
Things get messy when you want to do this for binary protocols like thrift.

Have written a custom router config in haproxy which I wanted to share.

frontend thriftrouter
  bind *:10090
  mode tcp
  tcp-request inspect-delay 100ms
  option tcplog
  log global
  log-format "%ci:%cp -> %fi:%fp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq captured_user:%{+Q}[capture.req.hdr(0)] req.len:%[capture.req.hdr(1)]"

  tcp-request content capture req.payload(52,10) len 10
  tcp-request content capture req.len len 10
  tcp-request content accept if WAIT_END

  acl acl_thrift_call    req.payload(2,2)  -m bin 0001		# Thrift CALL method
  acl acl_magic_field_id req.payload(30,2) -m bin 270f		# Magic field number 9999

  # Define access control list for each user
  acl acl_user_u1 req.payload(0,0),hex -m sub 7573657231  # user1
  acl acl_user_u2 req.payload(0,0),hex -m sub 7573657232  # user2

  # Route based on the user. No default backend so that one always has to set it
  use_backend backend_1 if acl_user_u1 acl_magic_field_id acl_thrift_call
  use_backend backend_2 if acl_user_u2 acl_magic_field_id acl_thrift_call


backend backend_1
  mode tcp
  option tcp-check
  server server1 127.0.0.1:9090
backend backend_2
  mode tcp
  option tcp-check
  server server2 107.0.0.1:9091

So now the only missing piece is the java code to generate custom metadata into thrift.

/**
 * Injects custom data into the egress thrift call.
 *
 * @author Gagandeep Singh
 */
@Slf4j
public class CustomInjectorProtocol extends TProtocolDecorator {
    private static final short FIELD_ID = 9999; // Magic number - this will be field id of custom data

    private final Function<Void, Map<String, String>> function;
    public CustomInjectorProtocol(TProtocol protocol, Function<Void, Map<String, String>> function) {
        super(protocol);
        this.function = function;
    }

    @Override
    public void writeMessageBegin(TMessage tMessage) throws TException {
        super.writeMessageBegin(tMessage);

        Map<String, String> map;
        try {
            map = function.apply(null);
            log.info("Injecting custom data into egress thrift call");
        } catch (Exception e) {
            log.error("Error in applying function: {}", e.getMessage());
            log.debug("Exception: ", e);
            map = new HashMap<>();
        }

        super.writeFieldBegin(new TField("custom_data", TType.MAP, FIELD_ID));
        super.writeMapBegin(new TMap(TType.STRING, TType.STRING, map.size()));
        for (Map.Entry<String, String> entry : map.entrySet()) {
            super.writeString(entry.getKey());
            super.writeString(entry.getValue());
        }
        super.writeMapEnd();
        super.writeFieldEnd();
    }
}

Set the user_id when making the thrift call.

TTransport transport = new TSocket("localhost", 10090, 1000000);
transport.open();

TProtocol protocol = new TBinaryProtocol(transport);
TProtocol spanProtocol = new CustomInjectorProtocol(protocol, (v) -> {
    Map<String, String> map = Maps.newHashMap();
    map.put("userid", "|user1|");
    return map;
});
client = new Service.Client(spanProtocol);

Posts: 1

Participants: 1

Read full topic