Search This Blog

Friday, August 11, 2017

Shared Client Libraries - A Microservices Anti-Pattern, a study into why?


As I continue my adventures with demystifying Microservices for myself, I keep hearing of this anti-pattern, 'Shared Service Client' or 'Shared Binary Client' or even 'Client Library'. Sam Newman in his book about Microservices says, "I've spoken to more than one team who has insisted that creating libraries for your services is an essential part of creating services in the first place". I must admit that I have seen and implemented this pattern right through my career. The driver for generating a client has been DRY (Don't repeat yourself).  The creation of Service Clients and making them available for consumption by other applications and services is a prevalent pattern. An example is shown below where a shareable Notes-client is created by a Team as part of the project for the Notes Web Service. They might also provide a DTO (Data Transfer Object) library representing exchanged payload.

This BLOG is my investigation into why this concept of a Service Client or a Client library causes heart burn to Microservice advocates.

Service Clients over time

Feel free to skip this section totally, it's primarily me reminiscing.

In my university days, when we used to work on network assignments, the interaction between a server and client would primarily involve creating a Socket, connecting to the server and sending/receiving data. There really was no explicit contract that someone wishing to communicate with the server could use to generate a client. The remote invocation however was portable across language/platform. Any client that can open a socket could invoke the server method.

I subsequently worked with Java and RMI. One would take a stub, then generate client and server code from it. Callers higher in the stack would invoke 'methods' using the client which would then run across the wire and be invoked on the server. From a client's perspective, it was as if the call was happening in the local JVM. Magical! A contract of sorts was established. Heterogeneous clients did not really have a place in this universe unless you got dirty with JNI. Another model existed at roughly the same time with CORBA and ORBs. The IDL or Interface Definition Language served to provide a contract by which client and server could be generated for a language and platform that could generate corresponding code from the IDL. Heterogeneous Clients could thrive in this ecosystem. Later down the chain came WS*, SOAP, Axis, Auto WSDL to Whatever generation tools like RAD (Rational Application Developer) that created a Client for a SOAP service.I had used a tool called Rational Application Developer which was a 4 C/D install which would fail often but when it worked, you could click on a WSDL and see it generate these magical code snippets for the client.

The WS* bubble was followed by the emergence of an Architecture style known as REST and its poster child, HTTP. Multiple representation types were a major selling point. To define Resources and their supported operations, there was WADL. Think of WADL as a poor cousin of WSDL that even the W3C preferred to not standardize. REST like it not, many a time is used for making remote procedure calls than walking a Hypermedia graph via Links as prescribed by Richardson Maturity model and HATEOS. Many developers of a REST service create a Client library that communicates via HTTP to the REST resources and due to the simplicity of REST, writing the client code was not arduous and did not require generation tools. REST itself was a contract. WADL if you want to stretch it and HATEOS if you want to argue about it made their ways to provide something like an IDL. Heterogeneous clients thrived with REST. The point to note is IDL seems to have taken a back seat here. Enter GRPC, a high performance open source framework from google. Has an IDL via .proto files. Supports Heterogeneous/Polyglot clients. It's just fascinating how IDL's are here to stay.

Reasons why Service Clients get a bad name

To facilitate the discussion of Service Clients and why they do not work, consider the following Customer Service Application written in Java that uses Service clients for Notes, Order management, Product Search and Credit Card. These Clients are provided by individual service maintainers.

The Not-So-Thin Service Client or The Very "Opinionated Client"

Service Clients many times start serving a higher purpose than what they were originally designed for. Consider for example the following hypothetical Notes Client:

The Notes Client seems to do a lot of things:
  • Determining strategy around whether to call the Notes Service or fall back to the Database
  • Determining whether to invoke calls in parallel or serial
  • Providing a Request Cache and caching data
  • Translating the response from the service into something the client will understand and absorb
  • Providing functionality to Audit note creation (stretching here:-))
  • Providing Configuration defaults around Connection pools, Read Timeouts and other client properties
The argument is "Who but the service owner knows best how to interact with the service and the controls to provide?". There are a few things with this design that are happening:
  • It does not account for non-Java clients
  • Bundles in a bunch of logic into the client around Notes (yes stretching it here with a Notes example :-))
  • Imposes a Request Cache on the Consumer. Resource imposition.
  • Assumes that it knows best regarding how you would like to invoke the service from a serial/parallel perspective
  • Makes resource decisions for you regarding threads etc
  • Provides defaults that it thinks are best without making you even think of your SLAs

Logic Creep into the Client

I have seen this happen a few times where business logic creeps into the binary client. This is just bad design that really negates the benefits of the services architecture. Sometimes the logic is around things like fall backs, counters, request caching etc. While these make sense to abstract away from a DRY perspective, they introduce a coupling or dependency on the client from the consumer's perspective that making changes might could require all consumers having to update. There are some clients that have 'modes' of operation and logic around what will be done in a given mode. These can be painful if they needed to change.

Resource Usage

The library provides a Request Cache to cache request data. It will use threads, memory, connection pools and many other resources while performing its job. The Service consumer might not need a multi-threaded connection pool, it might only need to obtain Notes every once in a while and does not even need to keep the connection alive. The Service consumer might not even care about paralleling calls to obtain data, it might be fine with serial. The consumer is however automatically opted-in to these invading resource usages.

Black Box

The Service Client is not something the Service Consumer wrote. On one hand service consumers care grateful that they have a library that takes away the effort around remote communication  but on the other hand, they have absolutely no control over what the library is doing. When something goes wrong, the consumer team needs to start debugging the Service Client logic and  other details, something they never owned or have in depth knowledge about. Phew!

Overloaded Responsibility

The Service owners perspective is really to generate a service that fulfills the business purpose. Their job is not to provide clients for the multitude of consumers that might want to talk to them. Writing client code to provide resiliency, monitoring and logging on communicating with the service just does not appear to be the responsibility for the service owner.

Hydra Client and Stack Invasion

Consider the following figure for this discussion:
A service client often brings along with it dependencies on libraries that the service maintainers have chosen for their stack. A consumer of the Service client are often forced to deal with managing out dependencies that would probably have never wanted in their stack but are forced to absorb them. As an example, consider a team that is invested in Spring and using Spring MVC and RestTemplate as their stack but have to suffer dependency management hell because they need to interface with a team whose Service Client is written using Jersey and need to deal with dependencies that might not be compatible. Multiply this with other Service Clients and the dependencies they bring in or go ahead and run with a Jersey 1 and Jersey 2 Client dependent libraries in your classpath to experience first hand how it would feel :-).

Heterogeneity and Polyglot Clients

"First you lose true technology heterogeneity. The library typically has to be in the same language, or at the very least run the same platform" - Sam Newman

Expecting the team generating a service to have to maintain service clients for different languages and platforms is a maintenance hell and simply not scalable. If they do not provide a client, then the ability to support a polyglot environment is stifled. When service clients start having business logic in them, it becomes really hard for service consumers not using the service team provided client to generate their own due to the complexity of logic involved. This can be a serious problem as it curbs innovation and technological advances.

Strong Coupling

The Service Client when done wrong can lead to a strong coupling between the service and its consumer. This coupling would mean that making even the smallest of changes to the service would involve significant upgrade effort from the consumers. What this does is stymie innovation on the service and taxes the service consumers into some upgrade cycle advocated by the service providers.

Benefits of the Service Client

The primary benefits that Service clients provide is DRY (Don't repeat yourself). A team that develops a service might quite well develop a client to test their service.  They might also provide integration tests with multiple versions of binary clients to ensure backward compatibility of their APIs. Their responsibility though ends there really, they really should have no obligation to distribute this client to others but the question we find a service consumer asking is; If this client is already created by the service owners and is available for free, why should I need to build one?

Remember that the Service creators are knowledgeable around the expected SLA of their service end points and the client they provide has meaningful defaults and optimizations for the same. A Service owner might augment the client library with things like Circuit Breakers, Bulkheading, Metrics, Caching and Defaults that will make the Client work really well with the service.

If a companies stack is very opinionated, for example, a non polyglot java and Spring Boot stack where every team uses RestTemplate for HTTP invocation with some discipline around approved versions of third party jars considering backward compatibility etc, then this model could very well hum along.

Client Generation

To facilitate the choice for service consumers to generate their own client, an IDL needs to be made available.  Tools like Swagger Code Gen and GRPC gen are great examples where these could be made available.  The auto-generated clients are good but as a word of caution, they aren't always optimized for high load and/or might need customization for sending custom company specific data anyways, diminishing their value as generated clients. Examples of company specific data might be things like custom trace identifiers, security info etc etc. Generated client code also will not have Circuit Breaking, Bulkheading and Monitoring features available on it effectively requiring each service consumer to write their own and with the tools available for them.

Parting Thoughts and Resources

We want to promote a loosely coupled architecture with distributed systems, i.e., between services and their consumers. This enables changes on the service to evolve without impacting consumers. A Service itself must NOT be platform/language biased and support a polyglot consumer ecosystem.

If Service owners were to design their services with the mentality that they have no control around the stack of the consumer apart from a transport protocol, it promotes better design of the service where logic does not creep into the client.

Service Clients, if created by Service owners must be 'thin' libraries devoid of any business logic that can safely be absorbed by consumers. If the client libraries are doing too much, you ought to re-think your service responsibilities and if needed consider introducing an intermediary service. I would also recommend that if a Service client is created for consumption by consumers, then limit the third party dependencies used by the client to prevent the Hydra Client effect.

It is also important to note that if a Services team creates a Client, that they do not force the consumers to use it. If the choice to use a different stack exists, then the Consumer is empowered to use the client generated by the Service maintainer or develop one independently. To a great degree service consumer teams maintaining their own clients keeps the service maintainers in check regarding compatibility and prevents logic creeping into client tier.
  • Ben Christiansen  has a great presentation on the Distributed Monolith that is must see for anyone interested in this subject. 
  • Read the Microservices book by Sam Newman. Pretty good data!
Keep following these postings as we walk into the next generation of services and tools to enable them!

If you have thoughts for or against Service Clients, please do share in the comments. 

Sunday, July 16, 2017

Interviewing -The undiscovered and maybe never found guide for job changers


I recently changed employers. As an individual who had spent a long time in one organization (9+ years), I was very unsure of how best to prepare for the interview process and what to expect. I was employed in a role that required me to conduct interviews for many people from different backgrounds like Developers, Testers and Software Directors. These experiences should have made me feel quite knowledgeable and confident entering an interview but truth be said...not really. I wanted to share my thoughts around this process with you and figured, should you find yourself in a similar boat, you might find this a tad useful.

I have some links to video's as I feel they describe my sentiments as I stepped through the process.

Please note I am an NOT an authority on interviewing, just someone sharing his experience and thoughts around leaving a job and what to expect in interviews.

The Sphere of Workplace Quan

If you are currently employed, before you consider going through interview processes, you might want to evaluate your Sphere of Workplace Quan (defined here!)

As an employee of any organization, you will face some forces that are influencing your decision to stay or look else where. These forces are not independent but tightly tug at each other. It is very important that you see the below video to understand what I am saying about Quan:

Now that you are well educated on the Quan or simply Quan, lets delve into the factors that affect your Quan.


If the market is offering considerably more for what you bring to the table or you see your counterparts exceeding your income, you need to either get your current employer to own up and bring your compensation to market or you should consider market options, your Quan is disturbed!

Work Duties and Responsibilities

When you take on a role, you sign on to a profile of what is expected of that role. Many a time, due to different forces, these start to diverge. Every one of us would like to step in and help and are ready to wear different hats as team players. If the duties and responsibilities grow and you see potential for your extra contributions to be rewarded, you should definitely take the added responsibilities and contribute. If on the other hand, you find the job responsibilities have somehow morphed into something your are not comfortable with, not interested in, you should bring it up with your manager to help address, and if that is not fruitful, you have a case for looking elsewhere, you Quan is disturbed!

Career Trajectory and Growth

Having a career trajectory is very important. An upward trend is healthy both from your own growth as well as that of the company that invests in its employees. Avenues for learning and knowing your skill set and experience are evolving with the industry is really important.  If you desire to grow and do not see a trajectory or opportunity ahead (assuming you have done the needful), your Quan is disturbed

The Mission or Vision

Working in an organization is not like when you go at something alone in your free time. It is team work. In order to do something, you need to believe in the mission. You need to commit, when you start to have doubts and are unable to control the direction, frustration has a way of creeping in. This frustration can be debilitating. For this reason, if you do not believe in the Mission as presented by your leaders, you should consider trying to impact your leaders to consider changing else your Quan is disturbed!

Relationship with Your Boss

Many a website rank relationship with your boss as being the primary reason someone leaves. A boss that is unreasonable, not appreciative, not invested in you, not having the ability to articulate a vision, not consistent with their decisions, not honest or just difficult to work with can negatively impact an employee's morale. Start by trying to make it work with your boss, have a conversation and if that does not work, try to find another position that would work for you in the same organization, else your Quan is disturbed and you should be in the market, its not short of good bosses...

Relationship with Peers

Sometimes in a work environment, one meets difficult peers or colleagues. If there is a peer who is difficult, try to understand why the relationship is strained and try to work it out with them. You might find, many a time, it is some inconsequential misunderstanding that could be causing the unwanted tensions. Spend time making your peer understand your viewpoint and come to some consensus.  If this does not work, escalate to your boss and try other avenues like HR. Hopefully your boss can diffuse this problem but if he is unable to and you have exhausted all avenues and the environment stresses you enough to compromise you health and work, your Quan is disturbed....the market awaits. While this sounds like a cop out, I have seen people quit for this very reason and find a better home.

Work/Life Balance

Some jobs are close to satisfying all the above but just seem to push the boundaries of your time. You are unable to balance your personal life with your work life. Long work hours, dealing with production issues, dealing with unreasonable deadlines, all these take away your personal moments with family or things you want to do. An occasional late night or late week is totally reasonable, one always wants to help move the needle, but when it becomes a pattern, frustration creeps in and your Quan is disturbed...

Valued and Appreciated

Feeling valued is what drives armies that are hopeless of success to fight a battle. They know their contributions matter. They feel valued and appreciated by their superior and peers for what they bring to the table. Appreciation does not have to be compensation related. A pat on your back, a calling you out explicitly in email, a gift card for lunch all can be quite morale boosting. If you are in an environment where you don't see this happening, it can be the most deadliest killers of your Quan and you need to address this soon...your Quan is disturbed...

The above are some of the forces that affect your Quan requiring you to look elsewhere. It does not have be a single item in the mentioned list but could be a combination of factors that affect your Quan.

The Push-Pull Principle When Changing Jobs

Someone I once knew would always ask employees when they turned in their resignation, "Are you running away from this job or running to your new job?" It is a very valid question and can put someone into retrospective. I have used that same question many a time, and most answers have been that they are running to a new job. In my naivety, I always assumed it was a binary answer, which I have come to realize was a flawed assumption. An existing job might be just great but a part of it might be disturbing the Quan which a new position might be fulfilling.

Are you running TO something or running AWAY from something? The Acid Test that helps you answer is really below...

As you get to work every day, ask yourself one question, are you dancing or close to it? You should be! If you are not,  your Quan is disturbed. Every day that you go to work, it has to be one where you are motivated and happy in your sphere of Quan, if that is not the case, you are doing yourself and the company you are working for an injustice.  A disturbed Quan bites!

Please note, that sometimes its all about putting bread on the table and/or circumstances that might require you to suck it up with hope the situation will change or it is the best you got. You are a survivor and you will fight to "Die another Day":


If you are here, your Quan is sufficiently disturbed that you are looking elsewhere or you simply are someone who wants to take your Quan to the next level. Welcome!
Below are some thoughts I am sharing. If you choose to use it, it is at your own risk!


Interviews are hard. Yes they are, don't let some someone convince you with: "Just be you and you'll be fine". It's you making a case about your desire to join an organization and how you will be the best fit for a job profile. The interviewers are judging you vs. all other candidates they are working with to fill the role. Prepare, even Rocky did!

It is very important that you prepare the right things though. There is no use building your biceps for a spelling bee competition right?

So how do you prepare?

Understand what is expected of your targeted  new job and assess whether you have the chops to do it or you have it in you to pick up the skills to deliver. It's very important that you understand what will be expected of you at your target job and whether you feel you have the aptitude to learn and deliver and most importantly will it make you dance?

Now that you know what is expected of you in the new job and you want to do the job, start your preparation:
  • Read through the Job Description in Detail. Determine compensation, trajectory etc etc associated with the job.
  • Learn about your target organization. It's is quite important that you know why you want you want to join wherever you are applying. Wikipedia it, google it, LinkedIn it,  see youtube videos, call a friend who works there...etc etc.
  • Glassdoor and other company focused sites - Never say no to a gift horse. There are sites like Glassdoor etc that provide information on interviews that you must familiarize yourself with.  Sometimes Glassdoor interviews and reviews might be enough to act as deterrent. 
  • Understand what the job skills are and that they match your interest. Hone them or refresh them. For example, if the job requires strong Data Structures and Algorithms knowledge and you want to be in that area, you should definitely spend some time refreshing those skills.
Don't target a job profile but target what you want to be doing.

Your preparation is complete. In Rocky terms, you can sprint up the Philadelphia museum of art without breaking a sweat. 

The Interview

You will face many challengers along the way that you need to overcome like the 36th Chamber of Shaolin:

Chamber One - The Recruiter

You have been reached out to. Your first contact is usually a recruiter assigned with filling the position. These folk can range from just wanting to know your basics to deep divers who want to know:
  • What your passions are?
  • What you bring to the table?
  • Why you want to join the organization?
  • Why are you looking to leave?
  • Do you meet the checklist expected of this position?
The smarter requiters will put you in a conversation rather than run you through a bullet point question list. These folk are the sentry into the interview process. They are very good at what they do. Take this time to understand the role, the company, the benefits, any asks from your end, like 'relocation', 'Work from Home', 'Commute' etc. This would be a great place for you to get some basic questions answered that are important to determine whether you want to move ahead with this job.

Chamber Two - The Phone Screeners

This is where you will arguably meet the some folk who might well be the people you are directly working with. These folk will be evaluating your chops to see whether or not you are ringing true to your resume and how well you would fit the job and team. These are also the folk who usually determine whether or not you are invited to a face to face interview. The screens are usually under an hour long and are either over the phone or via video chat. While their goal is to determine whether you are a potential fit and whether or not to proceed to the next level, you should also consider that this is where you get to chat with folk who you might be working with and to gleam whether or not to invest more time with this opportunity. As with every step in an interview process, knowledge is half the battle. If you are able to get information on your interviewers, take some time in researching them. Linked In, google search etc. The simple steps to approach this are:
  • Answer questions to the point without meandering too much. Let them drive.
  • Be honest. These are people who know their job and it's in your best interest to stay true. If you don't know something, it might be better to say, "I am not very familiar in that area but..."
  • Demonstrate why you will be a fit for the job. Highlight something you feel is pertinent from your experience.
  • They are looking to see how you would fit into the team(s). Inject your personality here and gauge how they work on a day to day basis and whether its a place you want to be at.
  • Get your list of questions answered. It might range from company/team life, culture, asking interviewers of their tenure at the company, etc etc

Chamber X - The interview

Preparing for the Interview

You have passed the most important gate now. You have an invitation into the building. Take a moment to congratulate yourself on your accomplishment. Your goals, drivers and skills seem to be in line with what your target company desires
As part of your preparation, ask your recruiter for the interview schedule and interviewers you will be interviewing with. It is very important that you understand your audience when presenting your stance in an interview. For example explaining the details of a core dump file to a Business person might not be a good use of interview time. Researching the persons you will be interviewing with ahead of the interview on LinkedIn, Facebook, google etc will arm you with information to assist during your interviews.
Lets to get to the site. Ask ahead about the attire, people have been denied just because they looked too stiff. Not sure I agree with the rationale behind that decision but there is usually nothing wrong with putting your best foot forward. Remember that one of the most important areas that interviewers will be evaluating you is on is how much the job would mean to you. 
You are dressed to kill now, go tiger!!!!! Don't forget a copy of your resume...


It's almost always down to this. If your preparation was good, this should be about translating that into a days work. As you enter into your day take these thoughts with you...
  • These are people you will be working with. While they are judging you, remember you are also  judging them to see if you will be a fit into their team and culture.
  • Relax. This goes without saying and is the hardest to do but stress and nervousness can impair your ability to project your full potential. Understand that these are just people like you and most likely will be the folk you will have your morning coffee with in a few weeks. So chill!
  • Be honest on what you bring to the table
    • What you are good at
    • What you want to do
    • What you can do
    • What is expected of you
  • Look for a...holes, they pop up like pimples and are obvious to spot...count the number, it matters, less is better. 

  • Ask questions about the work, the company, the culture, the vision, the target. These will help you understand more about the people and whether you want to work for the company.
  • Signs - Keep you radar up at all times. Be attentive to the environment around you, see the signs. A boss shouting at an employee, terminals unlocked, you being left alone in the interview process, interviewers not prepared...while there are negative, there are many positive signs as well that you can witness like courteous people, collaborative work, ease of access, quality time for employees, prepared interviewers etc etc
  • Honesty - As mentioned right through this process, your best bet is being honest and up front.
  • Pace of Interview - Pace of the interview is really the employers show.  However, you do have an influence on the pace of your the interview Don't fear to ask and assert appropriately. 
  • The Boss - Hopefully you are able to get some time with your would be boss. Ask about his management style, try to get an idea of how it would be working for him. Try to determine if he receptive to inputs and ideas. Look for data around his interaction with his reports.
  • The Team - If you are able to meet the team you will be working with, that is great as you will be able to visualize how you would fit with them and get some idea of the dynamics as they might play out.
  • Rock on - Be open, be free, be focused,  Answer questions to the point, don't wind into to unwanted areas. 
This is where you get a chance to decide whether or not you want to be part of this organization if an offer were to be made. For this reason, you should take your time, ask what you need to, get as much data as you can. 

The Offer

You are here because previous chambers have been positive. You have made a positive impression and are ready to receive that call.  If you have the luxury of refusing an offer that is lower than what you are making, you should, unless your Quan is significantly disturbed that would need you to opt out of your current job,
Things to focus on the offer;
  • The Base: Your base salary is what you take home. That is the one that puts the bread on the table for sure.  Bonuses and other promises are nice but not guaranteed unless contractual.
  • The Benefits: Very important perspective that can either make or break. You should consider health, dental, retirement contribution etc etc.
  • The Bonus: Bonuses are promises that might pay out well or never materialize. For this reason, its very important to get your 'Base' where you are comfortable with. That said, in some cases, companies are known to go lower on the base but have a large variable bonus component. If you see that model working for you, get some historical data of how that worked for others to help your decision making process.
  • The Perks: Perks are things a company goes above and beyond with. These matter a lot to some folk. Understand what these are and see how they impact you.
  • The RSU: If your target company can offer RSU's, ensure you negotiate a good settlement. Working for your employer is a mutually beneficial relationship. You want the share price to climb and they do as well, RSU's are a mechanism that make you feel like an invested owner.
  • The Relo: If you need to move to join the new gig, ask about the relocation expenses your employer would bear. 
  • Sign On: Many employers will give you a sign-on that kinda represents a good-faith from their end. Ask for this else you might be leaving money on the table.

The Commitment

This is where you say "Yes or No!!!!". It's a major decision. You need to be as sure as you can, you need to understand the change involved for you and yours and make a call. Hopefully its like below:

If you are now here, are entering your 'Honeymoon period' with your new company with a hope that it lasts as long as possible ;-)...

As mentioned at the beginning, this is me sharing an experience. I am at my new employer with whom I am having a blast but I thoroughly enjoyed working at my former employer. I leave you with some fond  memories of my time with my former employer.

Some Good References

Sunday, December 6, 2015

Maven Integration Testing of Spring Boot Cloud Netflix Eureka Services with Docker


My previous blog was about using Spring Cloud Netflix and my experiences with it.  In this BLOG, I will share how one could perform a maven integration test that involves multiple Spring Boot services that use Netflix Eureka as a service registry. I have utilized a similar example as the one I used in my BLOG about Reactive Programming with Jersey and RxJava where a product is assembled from different JAX-RS microservices. For this BLOG though, each of the microservices are written in Spring Boot using Spring Cloud Netflix Eureka for service discovery. A service which I am calling the Product Gateway, assembles data from the independent product based microservices to hydrate a Product.

Product Gateway

A mention of the Product Gateway is warranted for completeness. The following represents the ProductGateway classes where the ObservableProductResource uses a ProductService which in turn utilizes the REST clients of the different services and RxJava to hydrate a Product.

The ObservableProductResource Spring Controller class is shown below which uses a DeferredResult for Asynchronous processing:
public class ObservableProductResource {  
  private ProductService productService;
  public DeferredResult<Product> get(@PathVariable Long productId) {
    DeferredResult<Product> deferredResult = new DeferredResult<Product>();
    Observable<Product> productObservable = productService.getProduct(productId);

    .subscribe(productToSet -> deferredResult.setResult(productToSet), t1 -> deferredResult.setErrorResult(t1));
    return deferredResult;
Each Micro Service client is declaratively created using Netflix Feign. As an example of one such client, the BaseProductClient, is shown below:
public interface BaseProductClient {
  @RequestMapping(method = RequestMethod.GET, value = "/baseProduct/{id}", consumes = "application/json")
  BaseProduct getBaseProduct(@PathVariable("id") Long id);

What does the Integration Test do?

The primary purpose is to test the actual end to end integration of the Product Gateway Service. As a maven integration test, the expectancy is that it would entail:
  • Starting an instance of Eureka
  • Starting Product related services registering them with Eureka
  • Starting Product Gateway Service and registering it with Eureka
  • Issuing a call to Product Gateway Service to obtain said Product
  • Product Gateway Service discovering instances of Product microservices like Inventory, Reviews and Price from Eureka
  • Product Gateway issuing calls to each of the services using RxJava and hydrating a Product
  • Asserting the retrieval of the Product and shutting down the different services
The test itself would be a maven JUnit integration test.
As the services are bundled as JARs with embedded containers, they present a challenge to start up and tear down during an integration test. 
One option is to create equivalent WAR based artifacts for testing purposes only and use the maven-cargo plugin to deploy each of them under a separate context of the container and test the gateway. That however does mean creating a WAR that might never really be used apart from testing purposes.
Another option to start the different services is using the exec maven plugin and/or some flavor(hack) to launch external JVMs.
Yet another option is write custom class loader logic [to prevent stomping of properties and classes of individual microservices] and launch the different services in the same integration test JVM.
All these are options but what appealed to me was to use Docker containers to start each of these microservice JVMs and run the integration test.  So why Docker? Docker seems a natural fit to compose an application and distribute it across a development environment as a consistent artifact. The benefits during micro service based integration testing where one can simply pull in different docker images such as services, data stores etc of specific versions without dealing with environment based conflicts is what I find appealing.

Creating Docker Images 

As part of building each of the web services, it would be ideal to create a Docker image. There are many maven plugins out there to create Docker images [actually too many]. In the example, we have used the one from Spotify.  The building of the Docker image using the Spotify plugin for Spring Boot applications is nicely explained in the BLOG from, Spring Boot with Docker.
What I would see happening is that as part of the build process, the Docker image would be published to a docker repository which is internal to an organization and then made available for other consumers.

Integration Test

As part of the pre-integration test phase of maven, we would like to start up the Docker containers representing the different services. In order for the gateway container to work with the other service containers, we need to be able to link the Docker containers. I was not able to find a way to do that using the Spotify plugin. What I instead found myself doing is utilizing another maven plugin for Docker by Roland HuB which has much better documentation and more features. Shown below is the plugin configuration for the integration test.

        <!-- Eureka Server -->
            <color>green</color> <!-- Color the output green -->
            <port>8761:8761</port> <!-- Local to container port mapping -->
            <eureka.instance.hostname>eureka</eureka.instance.hostname> <!-- Override host name property -->
            <link>eureka</link> <!-- Link to Eureka Docker image -->
            <!-- Notice the system property overriding of the eureka service Url -->
      <!--....Other service containers like price, review, inventory-->
            <!-- Links to all other containers -->
            <!-- Setting this property to prefer ip address, else Integration will fail as it does not know host name of product-gateway container-->
One of the nice features is the syntax color prefix of each containers messages, this gives one a sense of visual separation among the multitude of containers that are started. The Integration Test itself is shown below:
@ContextConfiguration(classes = { ProductGatewayIntegrationTest.IntegrationTestConfig.class })
public class ProductGatewayIntegrationTest {
  private static final Logger LOGGER = Logger.getLogger(ProductGatewayIntegrationTest.class);
   * A Feign Client to obtain a Product
  public static interface ProductClient {
    @RequestMapping(method = RequestMethod.GET, value = "/products/{productId}", consumes = "application/json")
    Product getProduct(@PathVariable("productId") Long productId);

  public static class IntegrationTestConfig {}

  // Ribbon Load Balancer Client used for testing to ensure an instance is available before invoking call
  LoadBalancerClient loadBalancerClient;

  private ProductClient productClient;

  static final Long PRODUCT_ID = 9310301L;

  @Test(timeout = 30000)
  public void getProduct() throws InterruptedException {
    Product product = productClient.getProduct(PRODUCT_ID);

   * Waits for the product gateway service to register with Eureka
   * and be available on the client.
  private void waitForGatewayDiscovery() {
    while (!Thread.currentThread().isInterrupted()) {
      LOGGER.debug("Checking to see if an instance of product-gateway is available..");
      ServiceInstance choose = loadBalancerClient.choose("product-gateway");
      if (choose != null) {
        LOGGER.debug("An instance of product-gateway was found. Test can proceed.");
      try {
        LOGGER.debug("Sleeping for a second waiting for service discovery to catch up");
      catch (InterruptedException e) {
The test uses the LoadBalancerClient client from Ribbon to ensure an instance of 'product-gateway' can be discovered from Eureka prior to using the Product client to invoke the gateway service to obtain back a product.

Running the Example

The first thing you need to do is make sure you have Docker installed on your machine. Once you have Docker installed, clone the example from github ( and then execute a mvn install from the root level of the project. This will result in the creation of Docker images and the running of the Docker based integration tests of the product gateway. Cheers!

Friday, November 6, 2015

Spring Cloud Netflix Eureka Example...It's so Bootiful


Been looking at Netflix Eureka for Service Registry/Discovery and thought I'd share some thoughts via an example. My interest was more around Eureka 2.0 and I was eagerly awaiting their release in early 2015 per swags(?) but that did not materialize with the team stating its better to adopt Eureka 1.0. Not very thrilled I must say :-(.

Looking at Eureka 1.0, there appears two routes that one could go:
  1. Use Stock Netflix Eureka
  2. Use  Spring Cloud Netflix which is a Netflix Eureka and garnished with other Netflix OSS components, heated to a Spring temperature and presented as a Spring Boot Dish.
This Blog will utilize Spring Cloud Netflix and my favorite Notes Web Service to demonstrate a workable project that one can check out play with to understand Service Discovery with Eureka. The example will register an instance of  the Notes Service with Eureka Server (the registry) and a Notes Client will discover the service instance and issue a request to it simulating a full integration. Additional Technologies used:
  • RestAssured - Rest Assured is used a fluent testing of the service API   
  • Spring Fox - Used to expose Swagger UI for service documentation and testing
  • Netlflix Ocelli - Client Side Load Balancing
For the Client side of the application, I choose to use Netflix Ocelli for Client side load balancing along with Spring Frameworks RestTemplate. Why Ocelli over Ribbon for Client Side Load Balancing? Well,  Netflix seems to be looking at Ocelli as the new Ribbon, at least that is what this email chain seems to indicate.

Eureka Components

The following are some components one deals with when working with Eureka


EurekaClientConfig provides the configuration required by eureka clients to register an instance with Eureka. Netflix Eureka provides an implementation DefaultEurekaClientConfig that is configured via a property file eureka.client.props. Spring Cloud Netflix extends EurekaClientConfig and provides a EurekaClientConfigBean which implements and EurekaClientConfig and works with Spring properties. Properties prefixed by eureka.client will map into the EurekaClientConfigBean via Project Lombok magic.


Follows a similar approach like the ClientConfig but this is used for Eureka Server properties


Provides the configuration required by an instance to register with Eureka server. If building a service, you might want to tweak some of  knobs here. You can control properties like:
  • Name of Service
  • Secure/Non-secure ports of operation
  • Lease renewal and expiration
  • Home page URL and Status page URL
If working with vanilla Netflix Eureka, you could use MyDataCenterInstanceConfig (for non-AWS data center or the CloudInstanceConfig (AWS). Spring Cloud Netflix users get a EurekaInstanceConfigBean spruced with Lombok magic to control their properties. The properties are prefixed by eureka.instance.

Discovery Client

Discovery Client uses a 'jersey 1' client to communicate with Eureka Server to:
  • Register a service instance
  • Renew the lease of a Service instance
  • Cancel of a lease of service instance
  • Query Eureka Server for registered instances
DiscoveryClient is configured by the DiscoveryManager with has information of the EurekaClientConfig and the EurekaInstanceConfig

The Notes Example

The Notes example defines a Server side component that will register with Netflix Eureka, a Client side library, the Notes Client which discovers a  Notes Service Instance from Eureka and an integration test that demonstrates this lifecycle.

Notes Service Code

The Heart of the Service code is the declaration of the Spring Boot Application as shown below:

@SpringBootApplication // Spring Boot 
@EnableEurekaClient    // Enable Eureka Client
@EnableSwagger2        // Swagger  
public class Application {
  public String home() {
    return "This is the Notes App";

  public Docket swaggerSpringMvcPlugin() { // Minimalistic setting for Swagger Support
    return new Docket(DocumentationType.SWAGGER_2)

  public static void main(String args[]) {, args);

@Api(basePath = "/notes", value = "Notes", description = "Note Creation and Retrieval", produces = "application/xml")
public class NotesController {
  private NotesService notesService;

  @RequestMapping(value = "/notes", method = RequestMethod.POST)
  public Long create(@RequestBody Note note) {
    return notesService.createNote(note);

  @RequestMapping(value = "/notes/{id}", method = RequestMethod.GET, produces = MediaType.APPLICATION_XML_VALUE)
  public Note get(@PathVariable Long id) {
    return notesService.getNote(id);
In the above Boot application, we enabled Eureka Client via the @EnableEurekaClient annotation and also set up swagger support by creating a Docket and adding the @EnableSwagger2 annotation. The JSON Swagger resource is available at http://localhost:9090/v2/api-docs and the Swagger HTML is available at http://localhost:9090/swagger-ui.html.  The configuration for the application is driven by a simple YAML file as shown below where the port, location of Eureka server instance and a custom instance Id are defined:
    name: Notes

server.port: 9090

          defaultZone: http://localhost:8761/eureka/v2/
      instanceId: ${}:${spring.application.instance_id:${random.value}}
If one wished to integration test (exercise DB and other calls) in the Service application without having to worry about the Eureka registration and discovery, you could simply turn of the eureka client and use, say RestAssured to validate your service as shown below.
@SpringApplicationConfiguration(classes = Application.class)
@IntegrationTest({"server.port=0", // Pick an ephemeral port
                  "eureka.client.enabled=false" // Prevent registering/discovering with Eureka
public class NotesControllerTest {

 private  int port;

  public void setUp() {
    RestAssured.port = port; 

  public void createAndGetNote() {
           .body(new Note("Test"))
            .post("http://localhost:" + port + "/notes")

    when().get("/notes/{id}", 1).then()
             .body("note.content", equalTo("Test"));

Notes Client Code

For the NoteClient, as mentioned, Ocelli is used as a Load Balancer with RestTemplate used to make the REST call.

public class NotesClientImpl implements NotesClient, Closeable {
  private final RestTemplate restTemplate;
  private final Subscription subscription;
  private final AtomicReference<List<InstanceInfo>> result;
  private final RoundRobinLoadBalancer<InstanceInfo> instanceInfoLoadBalancer;

  public NotesClientImpl(RestTemplate restTemplate, DiscoveryClient discoveryClient,
                         long refreshInterval, TimeUnit refreshIntervalTimeUnit) {
    this.restTemplate = restTemplate;
    this.result = new AtomicReference<List<InstanceInfo>>(new ArrayList<>());
    this.subscription = new EurekaInterestManager(discoveryClient).newInterest()
            .forApplication("Notes") // Notes Service
            .withRefreshInterval(refreshInterval, refreshIntervalTimeUnit) // Ocelli Refresh Interval
    instanceInfoLoadBalancer = RoundRobinLoadBalancer.<InstanceInfo>create(); // Round Robin

  private String getServiceInstanceUrl() {
    InstanceInfo instanceInfo = instanceInfoLoadBalancer.choose(result.get()); // Choose an instance
    if (instanceInfo != null) {
      return "http://" + instanceInfo.getIPAddr() + ":" + instanceInfo.getPort();
    throw new RuntimeException("Service Not available");

  public Note getNote(Long noteId) {
    return restTemplate.getForEntity(getServiceInstanceUrl() + "/notes/{id}", Note.class, noteId).getBody(); 
Ocelli uses an 'interest' manager to register a subscription for changes to Notes Service. A round robin load balancer is used in the example. The NotesClient is set up in the NotesClientConfig.

Notes Integration Test

NotesIntegrationTest  is where things get interesting. The integration test will -
  • Start a Eureka Server in the pre-integration-test phase of maven using Cargo. Note that we are using the stock Netflix eureka WAR for this. One could also use the Spring Cloud version via an exec maven plugin.
  • Have Notes Service start on a random port and register with the Eureka Server
  • Have the Notes Client create and obtain a 'Note' by discovering the Notes Service from Eureka
Eureka operations are usually set in range of 20 to 30 seconds for things like registration, heartbeart, renewal and registry fetching. That works in a production environment but in an integration test waiting for so long is not an option. Thankfully, the Spring Boot test framework allows for easy overriding of these properties.
@SpringApplicationConfiguration(classes = {Application.class, NotesClientConfig.class}) // Note the Import of Server and Client
@WebIntegrationTest(value = {"eureka.instance.leaseRenewalIntervalInSeconds=10", // Lease Interval
        "eureka.client.serviceUrl.defaultZone=http://localhost:9000/eureka/v2/"}, // Overriding eureka location
        randomPort =  true)
public class NotesIntegrationTest {

  private NotesClient notesClient; // Notes Client injected

  private Long ocelliRefreshIntervalSeconds; // How often Ocelli Client has been set up to refresh
  public void createAndGetNote() throws InterruptedException {
    // Wait for Notes App to register
    waitForNotesRegistration(); // Waits for Notes Service to register before proceeding
    Thread.sleep((ocelliRefreshIntervalSeconds * 1000) + 1000L); // Ocelli might not be in sync
    Note note = new Note("Test");
    Long noteId = notesClient.createNote(note); // Create a note
    assertEquals(note, notesClient.getNote(noteId)); // Read the node

  private DiscoveryClient discoveryClient;

  private void waitForNotesRegistration() throws InterruptedException {
   /* Discovery client is used to make sure service is registered before the Notes Client gets a chance to discover and exercise API */
The important thing here to note is that I am using the 'war' version of the Netflix Eureka server to start up before the integration test runs. It is equally possible to launch the Spring Boot 'jar' version of the eureka server via the exec maven plugin prior to the start of the integration test.

Parting Thoughts

For the curious, this Notes Spring Cloud Netflix Eureka example can be found at sleeplessinslc github.

I would also like to share the below as my personal 'opinion' only

Netflix Eureka

The good - 

Awesome project, a Service Registry for the Web. Apps in any language can participate. REST + Jersey, preaching to the choir. More importantly, developed with a profound understanding of failure scenarios around a Microservice 'service registry'. Nitesh Kants presentation is a must see.

Good starter documentation and responsive folks on the mailing list. 

The bad - 

I understand Eureka was designed for AWS usage primarily by Netflix.  Maybe a more modular (plugin) approach would have been nicer ensuring a clean separation between AWS and non-AWS code, especially considering it is a fine candidate for private cloud.

Discovery client fetches entire registry rather than information about specific services. Eureka 2 architecture looks better in that regard where one only gets data about services they are interested in.

Jersey 1 - Lets update to Jersey 2.X folks! 

Eureka UI looks pretty horrid and that is coming from a 'server side' only programmer :-). 

There is an issue where shutting down a service instance leads to shutting down all other services on that host which is concerning if hosting multiple services on a node.

Ocelli vs. Ribbon. Can we have clear direction? Ocelli has not had a release since July 10th, Ribbon though has a more recent release. So is Ocelli really the new Ribbon? 

Spring Cloud Netflix

The good -

If using a Spring ecosystem, a no brainer to adopt. Meshes really well with Spring properties and enables Hystrix and other Netflix OSS.

Spring Boot is pretty sweet but you need to work in a 'controlled' environment in the 'Spring Boot Way' to enjoy full benefit. 

Spring Cloud Netflix re-bundled Eureka server is much more visually pleasing than the core Eureka UI. I believe they also have a workaround with reference to the Shutdown bug I mentioned.

The bad - 

You sold yourself to the Bootiful devil :-)
  • Custom extension classes of Netflix interfaces (EurekaClientConfigBean..etc), rather than bridging
  • Version of Eureka is older
  • Documentation does not match up with implementation. Case in point the property "" defined in their documentation is not even part of current release. I spent some time chasing this
  • Do we really need Lombok?
So, what I want to do next?

Leaning back on my reactive example,  lets say I have an aggregator Product Service that I want to test that utilizes a Base Product Service, a Price Service and an Inventory Service, all developed with Spring Cloud Eureka, how would I write a maven integration test for this? Off to Docker land my friends, onward and upward!

Tuesday, September 29, 2015

RxJava Presentation

I have been an employee of for close to 8 years now. Every year Overstock has a Technology Day where employees present on technology and simply have fun. This year I presented on RxJava with a colleague of mine. We had a lot of fun preparing for the presentation while learning as well. We also won a small reward for being a being a popular presentation :-)

The following are the slides that we are sharing from our presentation. I would recommend you check the RxAir Demo by my colleague, it's pretty sweet.


Friday, May 8, 2015

RxNetty and Jersey 2.0


I have been interested in reactive programming and micro-services of recent and a few things have caught my attention:
  • Netty - An asynchronous event-driven framework for high performance protocol servers and clients 
  • Netflix RxNetty - Reactive Extension Adapter for Netty
  • Netflix Karyon - A blueprint for a cloud ready application
Netty became my primary interest from a performance perspective and some reading demonstrated that Netflix had done some benchmarks between RxNetty and Tomcat and found results in favor of RxNetty. Take a look at the following links -

So I wanted to see how I could use RxNetty for HTTP. One problem that surfaced is when using RxNetty and HTTP, one is limited to a basic  HTTP framework, i.e, something not as rich as Jersey JAX-RS. The same led me to Karyon which has 'blocking' Jersey support. However, Karyon uses Jersey 1.X and the Karyon team did not look like they would be making a move to Jersey 2.X anytime soon . A post from Netflix on the same

Searching to see if Jersey 2.X had support for Netty, I found the following two items -
The above project is a Netty-Jersey bridge but does not use RxNetty. So I set out to make my own RxNetty Jersey Container to see how it would work.


The following code demonstrates a RxNettyHttpContainer. It has been stripped of some methods. The container has been modeled after GrizzlyHttpContainer and code from Karyon.

public class RxNettyHttpContainer implements Container, RequestHandler<ByteBuf, ByteBuf> {

  private volatile ApplicationHandler appHandler;

  private final Type RequestTYPE = (new TypeLiteral<Ref<HttpServerRequest<ByteBuf>>>() {
  private final Type ResponseTYPE = (new TypeLiteral<Ref<HttpServerResponse<ByteBuf>>>() {
  private volatile ContainerLifecycleListener containerListener;
   * Referencing factory for RxNetty HttpServerRequest.
  private static class RxNettyRequestReferencingFactory extends ReferencingFactory<HttpServerRequest<ByteBuf>> {

   * Referencing factory for RxNetty HttpServerResponse.
  private static class RxNettyResponseReferencingFactory extends ReferencingFactory<HttpServerResponse<ByteBuf>> {
   * An internal binder to enable RxNetty HTTP container specific types injection.
  static class RxNettyBinder extends AbstractBinder {

    protected void configure() {
       // Bind RxNetty HttpServerRequest and HttpServerResponse so that they are available for injection...
  public RxNettyHttpContainer(Application application) {
    this.appHandler = new ApplicationHandler(application , new RxNettyBinder());
    this.containerListener = ConfigHelper.getContainerLifecycleListener(appHandler);

  public void reload(ResourceConfig configuration) {
    appHandler = new ApplicationHandler(configuration, new RxNettyBinder());
    containerListener = ConfigHelper.getContainerLifecycleListener(appHandler);

  // This is the meat of the code where a request is handled
  public Observable<Void> handle(HttpServerRequest<ByteBuf> request,
    HttpServerResponse<ByteBuf> response) {
    InputStream requestData = new HttpContentInputStream(response.getAllocator(),
    URI baseUri = toUri("/");
    URI requestUri = toUri(request.getUri());

    ContainerRequest containerRequest = new ContainerRequest(baseUri, requestUri, request
        .getHttpMethod().name(), getSecurityContext(requestUri, request),
        new MapPropertiesDelegate());


    for (String headerName : request.getHeaders().names()) {
      containerRequest.headers(headerName, request.getHeaders().getAll(headerName));

    containerRequest.setWriter(new RxNettyContainerResponseWriter(response));

    containerRequest.setRequestScopedInitializer(new RequestScopedInitializer() {

        public void initialize(final ServiceLocator locator) {

   return Observable.<Void> create(subscriber -> {
      try {
      finally {
    }).doOnTerminate(() -> response.close(true)).subscribeOn(;

  private SecurityContext getSecurityContext(URI requestUri, HttpServerRequest<ByteBuf> request) {
     // Not yet handling security

  // A custom ContainerResponseWriter to write to Netty response
  public static class RxNettyContainerResponseWriter implements ContainerResponseWriter {
    private final HttpServerResponse<ByteBuf> serverResponse;

    private final ByteBuf contentBuffer;

    public RxNettyContainerResponseWriter(HttpServerResponse<ByteBuf> serverResponse) {
      this.serverResponse = serverResponse;
      this.contentBuffer = serverResponse.getChannel().alloc().buffer();

    public void commit() {
      if (!serverResponse.isCloseIssued()) {

    public void failure(Throwable throwable) {
      LOGGER.error("Failure Servicing Request", throwable);
      if (!serverResponse.isCloseIssued()) {
        serverResponse.writeString("Request Failed...");     

    public OutputStream writeResponseStatusAndHeaders(long contentLength,
      ContainerResponse containerResponse) throws ContainerException {
      HttpResponseHeaders responseHeaders = serverResponse.getHeaders();
      if (!containerResponse.getHeaders().containsKey(HttpHeaders.Names.CONTENT_LENGTH)) {
        responseHeaders.setHeader(HttpHeaders.Names.CONTENT_LENGTH, contentLength);
      for (Map.Entry<String, List<String>> header : containerResponse.getStringHeaders().entrySet()) {
        responseHeaders.setHeader(header.getKey(), header.getValue());
      return new ByteBufOutputStream(contentBuffer);
As mentioned, the above is  heavily influenced by the code from Karyon's Jersey 1.X JerseyBasedRouter and NettyToJerseyBridge. It has a few additional enhancements in being able to inject RxNetty's HttpServerRequest and HttpServerResponse into a Jersey Resource. One thing to note is that I could not find a way to inject the parameterized equivalents of HttpServerRequest and HttpServerResponse into a Jersey Resource and unless greatly mistaken it is limitation of HK2 that prevents the same. An example of the injected resource looks like:
public class SomeResource {
  private final HttpServerRequest<ByteBuf> request;

  private final HttpServerResponse<ByteBuf> response;

  public SomeResource(@SuppressWarnings("rawtypes") HttpServerRequest request,
      @SuppressWarnings("rawtypes") HttpServerResponse response) {
    this.request = request;
    this.response = response;

Using the Container

In line with other containers provided by Jersey, there is a factory class and methods to start the RxNetty Jersey 2.X container.
public class RxNettyHttpServerFactory {

  private static final int DEFAULT_HTTP_PORT = 80;

  public static HttpServer<<ByteBuf, ByteBuf> createHttpServer(Application application) {
    return createHttpServer(DEFAULT_HTTP_PORT, application, false);

  public static HttpServer<ByteBuf, ByteBuf> createHttpServer(Application application, boolean start) {
    return createHttpServer(DEFAULT_HTTP_PORT, application, start);

  public static HttpServer<ByteBuf, ByteBuf> createHttpServer(int port, Application application) {
    return createHttpServer(port, application, false);

  public static HttpServer<ByteBuf, ByteBuf> createHttpServer(int port, Application application,
    boolean start) {
    HttpServer<ByteBuf, ByteBuf> httpServer = RxNetty.createHttpServer(port,
      new RxNettyHttpContainer(application));
    if (start) {
    return httpServer;
As an example:
public class Main {
 public static void main(String args[]) {
   ResourceConfig resourceConfig = ResourceConfig.forApplication(new NotesApplication());
   RxNettyHttpServerFactory.createHttpServer(9090, resourceConfig, true);; // Wait for user input to shutdown
With the above, you have a RxNetty based Jersey 2.0 Container that you can have fun with. Wait! What about an example?

Example Using RxNetty Jersey 2.0 Container

As always, I lean back to my Notes example using Jersey 2.0 where a Note can be CRUDded! In addition, my example would not be complete without a touch of Spring in it, so we use Spring for DI. One of the sadness on the Jersey Spring integration from the artifact jersey-spring3 is that one needs to have javax.servlet as compile time dependency even if one does not use it. The Notes client is developed using RxJersey and the example has Spring for injection. and can be downloaded from HERE or accessed at github at

Performance of the RxNetty Jersey 2 Container

A simple JMeter Load Tests exists in the Notes integration test project that tests the performance of the above RxNetty Container and Grizzly.
Now as a disclaimer, I am not a proclaimed JMeter expert so I could have got some stuff wrong. That said the following are some of the results contrasting the performance of the Grizzly container with the RxNetty Jersey 2 Container.

User Threads - 50 - Loop Count 25

User Threads - 500 - Loop Count 10

The two tests runs are to create notes and to obtain back notes. It appears that from both the tests, the throughput and responsiveness of the RxNettyContainer on the Notes Service is far better than the Grizzly equivalent. Note that GC metrics have not been looked into. There is one gotcha that I have not mentioned to the above. When a large payload is requested, the RxNetty Jersey 2.0 container chokes. As it is based of the Karyon Jersey 1.X implementation, I need to check and see if Karyon Jersey has the same problem. I believe it has to do with buffer copy between the HttpServerResponse and the content buffer (ByteBuf). To witness the problem in action, add the get all notes call to the JMeter script. There are two launchers for the Grizzly and RxNetty Containers respectively. Both of these need to be started before the JMeter test is launched.


RxNetty looks pretty sweet at least from my simple tests. Using it as a Micro-container looks interesting with Jersey 2. The container I have created does not support @Aysnc on a Jersey Resource, similar to the jdk-http container and the Karyon jersey bridge, and to be honest, I am not even sure it makes sense to do so.

The RxNetty Jersey 2.X container shown above -
  • Is only a proof of concept. It does not have the necessary unit-tests and or validation. Something that could be easily done though.
  • Utilizes a class from Karyon - HttpContentInputStream so has a dependency on a Karyon jar
  • Karyon Jersey 1.X could easily be migrated to Jersey 2.X based of the above code it appears
  • If you wanted to use Jersey 2 with Netty, I would look into the Netty Jersey 2.X Container I mentioned earlier.
If anyone has thoughts around the container, load test, or example, ping me.