Implementing Enterprise Integration Patterns using Apache Camel

About Me

1st Career

About Me

2nd Career

About Me

3rd Career?

Overview

Enterprise Integration & Camel concepts
Developing Camel applications
Miscellany

Caveat: This is a brief introduction to Camel. Camel is very simple to use, but offers so many components and capabilities, I probably couldn't cover them all in three full days of training.

However, you don't need to know everything before doing anything, so let's get started...

Enterprise Integration with Camel

Enterprise Integration - Before

Fortunately*, the picture's changing...

*Fortunately for software developers and consultants, that is...

Enterprise Integration - After

Enterprise Integration - Soon

Text

https://olimould.files.wordpress.com/2009/01/network-diagram-1.jpg

Enterprise Integration Patterns

http://camel.apache.org/manual/camel-manual-2.16.2.html#chapter-enterprise-integration-patterns

http://www.enterpriseintegrationpatterns.com/

EIP topics we'll cover using Camel

Messages
Channels & Endpoints
Filters
Routing
Translation
Error, logging, more...

Apache Camel implements many of these out of the box, with customization and extension points enabling us to do any kind of integration we need

What is Camel?

Modular and extensible message routing framework
Lightweight core with a pluggable architecture
Easy DSL configuration
Automatic, but extensible, type conversion
Built-in components allowing routing from/to almost anything
Built-in logging, testing and monitoring support
Plays well with other frameworks, like Spring
Large and vibrant community

Now for some terminology...

Message

An interface representing a single message
Messages have a String messageId
Headers Map<String, Object>
Attachments Map<String, Object>
A body of type Object

Endpoint

The point where a message is produced or consumed
Examples
- Camel Processor
- POJO bean
- JMS Queue
- Web service
- File, FTP server, AWS S3
- Email address
- Logger
- Twitter, Facebook, SalesForce, more...

Exchange

An interface abstracting the exchange of messages
Exchange ID (useful as correlation id)
Exceptions during routing stored in Exchange
Properties are Map<String,Object>
Exchange Pattern is In-Only or In-Out

URL, URI, URN, IRI

URL: Uniform Resource Locator
URN: Uniform Resource Name
URI: Uniform Resource Indicator
- A URL or URN
IRI: Internationalized Resource Identifier
- an i18n of URI
Camel route configurations use URI Strings
- URL style: file:///someDirectory?options
- URN style: file:/someDirectory?options

Component

A singleton endpoint factory
Camel comes with dozens of different components
- Bean, File, FTP, HTTP, Jetty, JMS, Log, SMTP
You can create and register custom components

// The component used in this route is a File
// There is one File component instantiated
from("file:/orders/inbound")        // File creates inbound endpoint
    .process(someProcessor)
    .to("file:/shipping/outbound");  // File creates outbound endpoint

// Note: inbound and outbound are directories. The file name is in a header

Route

A step by step movement of a message from one endpoint to the next
- Often with data conversion in between
- Often with processing between
To configure routes, we use a DSL (domain specific language)

Spring (XML)
Java
Groovy
Scala
Rest
Annotations
OSGi Blueprint

Route

// Spring XML DSL

<bean id="someProcessor" class="com.foo.SomeProcessor" />
<bean id="someBean" class="com.foo.PojoBean" />

<camelContext xmlns="http://camel.apache.org/schema/spring">
   <route>
      <from uri="someComponentX:abc?options"/>
      <process ref="someProcessor"/>
      <bean ref="someBean"/>
      <to uri="someComponentY:abc?options"/>
   </route>
</camelContext>

Configure routes with Spring XML DSL

Route

// Using Java DSL
from("someComponentX:abc?options")     // Consume a message here
    .process(someProcessor)            // Process using Camel Processor
    .to("bean:someBean")               // Process using bean from registry
    .to("someComponentY:abc?options"); // Send it to some endpoint

Configure routes with Java DSL

Route

// Using Groovy DSL which is a lot like Java DSL
from('someComponentX:abc?options')       // Consume a message here
    .process( { Exchange ex -> ... } )   // Process using Closure
    .to("bean:someBean")                 // Process using bean from registry
    .to("someComponentY:abc?${options}") // Send it to some endpoint

Configure routes with Groovy DSL

CamelContext

Represents the Camel runtime system
Similar in concept to the Spring ApplicationContext
We usually just have one of these in an application

CamelContext

Using the CamelContext

We create a CamelContext object (Let's call it ctx)
Add endpoints and maybe components
Add our routes via ctx.addRoutes(RouteBuilder)
On startup, we invoke ctx.start()
On shutdown, we invoke ctx.stop()

Camel is to message routing as Spring is to object collaboration

It's really an Inversion of Control framework where we wire up our message routing, endpoints, and processors in configuration outside our domain classes

Developing Camel Applications

Project setup
Creating routes
Data conversion and processing
Testing routes and processors
Fun with components

Project setup

I'm using Gradle, but Maven is well supported

...

dependencies {
    //It needs to be one of the first dependencies to work
    compile 'ch.qos.logback:logback-classic:1.1.6'

    compile 'org.codehaus.groovy:groovy-all:2.3.11'

    // Camel dependencies
    def camelVersion = '2.16.2'
    compile "org.apache.camel:camel-core:$camelVersion"
    compile "org.apache.camel:camel-groovy:$camelVersion"

    testCompile 'junit:junit:4.11'
    testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'
    testCompile 'cglib:cglib-nodep:2.2'
}







group 'com.jackfrosch.camel'
version '1.0-SNAPSHOT'

apply plugin: 'groovy'
apply plugin: 'java'
apply plugin: 'idea'

sourceCompatibility = 1.8 // Camel 2.14 onwards requires JDK 1.7 or better

repositories {
    jcenter()
    mavenCentral()
}


...

build.gradle

...

sourceSets {
    integrationTest {
        groovy {
            compileClasspath += main.output + test.output
            runtimeClasspath += main.output + test.output
            srcDir file('src/integration-test/groovy')
        }
        resources.srcDir file('src/integration-test/resources')
    }
}

configurations {
    integrationTestCompile.extendsFrom testCompile
    integrationTestRuntime.extendsFrom testRuntime

    all*.exclude group: "org.slf4j", module: "slf4j-simple"
    all*.exclude group: "log4j", module: "log4j"
}

idea {
    module {
        //add integration test source dirs to IDEA
        testSourceDirs += file('src/integration-test/groovy')
        scopes.TEST.plus += [ configurations.integrationTestCompile ]
        downloadJavadoc = false
        downloadSources = false
    }
}

// dependencies shown earlier

build.gradle

Creating routes

I'll be using the Java and (mostly) Groovy DSLs
- They're easier on the eyes than XML
- If you like XML, the Spring XML DSL is for you
The easiest way to create routes with a DSL is using a RouteBuilder

CamelContext ctx = new DefaultCamelContext();

// Use a timer component to generate message events
ctx.addRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
        from("timer:sayHello?period=5s")
            .log("Hello");
    }
});

Component options

from("timer:sayHello?period=5s")

http://camel.apache.org/timer.html

Demo

Hello World

package com.jackfrosch.camel;

import org.apache.camel.CamelContext;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.impl.DefaultCamelContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class HelloWorld {
    final static Logger logger = LoggerFactory.getLogger(HelloWorld.class);

    public static void main(String[] args) throws Exception {

        CamelContext ctx = new DefaultCamelContext();

        ctx.addRoutes(new RouteBuilder() {
            @Override
            public void configure() throws Exception {
                from("timer:sayHello?period=1s")
                    .log("Hello");
            }
        });

        ctx.start();
        Thread.sleep(10000);
        ctx.stop();
    }

}

Creating routes

A route is defined with the from method
From there, you can route the message to processors, logger, and other endpoints

CamelContext ctx = new DefaultCamelContext();

ctx.addRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
        from("direct:startWorkflow")
            .process(processFirstStep)
            .process(processSecondStep)
            .process(processThirdStep)
            .to("someComponent:finalDestination");
    }
});

Creating routes

I recommend defining new routes for every step in the workflow

CamelContext ctx = new DefaultCamelContext();

ctx.addRoutes(new RouteBuilder() {
    @Override
    public void configure() throws Exception {
        from("direct:startWorkflow")
            .routeId("startWorkflow")
            .process(processFirstStep)
            .to("direct:secondStep");

        from("direct:secondStep")
            .routeId("secondStep")
            .process(processSecondStep)
            .to("direct:thirdStep");

        from("direct:thirdStep")
            .routeId("thirdStep")
            .process(processThirdStep)
            .to("someComponent:finalDestination");
    }
});

Why use fine grained routes?

Testability
- Easier to test
Maintainability
- Camel route logging makes it easier to spot where things went wrong
Extensibility
- Easier to find clean insertion points for new routes
Modularity:
- Routes can be aligned in bounded contexts for dividing monolith into microservices

Downside: App feels more complex with more routes

Demo

Inventory Reporting Workflow

Input & Output

SKU,Qty,Price,Taxable
 101,30,20.00,Y
 102,40,10.00,N
 103,100,0.50,Y

Inventory Report - 04 / 03 / 2016

Summary
----------------------------------------------

Total SKUs in inventory: 3
Total Taxable Value in inventory: 650.00
Total Non-taxable Value in inventory: 400.00
----------------------------------------------

Inventory Details
----------------------------------------------
Item #        SKU   Qty   Price   Value   Tax?
------ ---------- ----- ------- -------   ----
     1        101    30   20.00  600.00      Y
     2        102    40   10.00  400.00      N
     3        103   100    0.50   50.00      Y
----------------------------------------------

--End Report--

Store123_Inventory_2016-04-01_01-35-45.csv

Store123_Inventory_2016-04-01_01-35-45_Report.txt

Our main app

package com.jackfrosch.camel.inventory;

import org.apache.camel.CamelContext;
import org.apache.camel.impl.DefaultCamelContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class InventoryReportingApp {
    final static Logger logger = LoggerFactory.getLogger(InventoryReportingApp.class);

    public static void main(String[] args) throws Exception {

        CamelContext ctx = new DefaultCamelContext();

        ctx.addRoutes(new InventoryReporterRouteBuilder());

        ctx.start();
        Thread.sleep(20000);
        ctx.stop();
    }
}

The RouteBuilder

// package and import statements omitted

/**
 * These routes are a bit contrived, but illustrate some fundamental Camel
 * routing and processing ideas
 */
class InventoryReporterRouteBuilder extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        from("file:/tmp/camel-demo/inventory/in?delay=1s&move=../archive") // consumer
            .convertBodyTo(String.class)
            .process(new InventoryProcessor())
            .to("direct:valueInventory");

        from("direct:valueInventory")
            .process(new InventoryValuator())
            .to("direct:reportInventory");

        from("direct:reportInventory")
            .process(new InventoryReportGenerator())
            .to("file:///tmp/camel-demo/inventory/out");                  //  producer

    }
}

InventoryProcessor

// package and import statements omitted

public class InventoryProcessor implements Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
        Message in = exchange.getIn();
        String input = (String) in.getBody(); // in.getBody(String.class) & no cast

        List<StockItem> items = parseInput(input);
        in.setBody(items);
        in.setHeader("INVENTORY_COUNT", items.size());
    }

    private List<StockItem> parseInput(String input) {
        List<StockItem> items = new ArrayList<>();

        String[] lines = input.split("\n");
        for(int i = 1; i < lines.length; i++) {
            items.add(createStockItem(lines[i].split(",")));
        }

        return items;
    }

    private StockItem createStockItem(String[] fields) {
        return new StockItem(fields[0], Integer.valueOf(fields[1]), 
                             new BigDecimal(fields[2]), "Y".equals(fields[3]));
    }
}

StockItem

// package and import statements omitted

public class StockItem {
    private final String itemSku;
    private final int quantityOnHand;
    private final BigDecimal markedPrice;
    private final boolean taxable;
    private BigDecimal itemValuation;

    public StockItem(String itemSku, int quantityOnHand, BigDecimal markedPrice, boolean taxable) {
        this.itemSku = itemSku;
        this.quantityOnHand = quantityOnHand;
        this.markedPrice = markedPrice;
        this.taxable = taxable;
        this.itemValuation = new BigDecimal(quantityOnHand).multiply(markedPrice);
    }
    @Override public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        StockItem stockItem = (StockItem) o;
        return itemSku.equals(stockItem.itemSku);
    }
    public String getItemSku() { return itemSku; }
    public int getQuantityOnHand() { return quantityOnHand; }
    public BigDecimal getMarkedPrice() { return markedPrice; }
    @Override public int hashCode() { return itemSku.hashCode(); }
    public boolean isTaxable() { return taxable; }
    public boolean isNotTaxable() { return !taxable; }
    public BigDecimal getItemValuation() { return itemValuation; }
}

InventoryValuator

// package and import statements omitted

public class InventoryValuator implements Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
        Message in = exchange.getIn();

        @SuppressWarnings("unchecked")
        List<StockItem> items = (List<StockItem>) in.getBody();

        in.setHeader("INVENTORY_TAXABLE_VALUATION", 
                        calculateInventoryValuation(items, StockItem::isTaxable));

        in.setHeader("INVENTORY_NONTAXABLE_VALUATION", 
                        calculateInventoryValuation(items, StockItem::isNotTaxable));
    }

    // Note: The Predicate here is a Java 8 Predicate functional interface, not a Camel Predicate!
    private BigDecimal calculateInventoryValuation(List<StockItem> items, 
                                                   Predicate<StockItem> selector) {

        return items.stream()  // we could use a parallelStream() here
                    .filter(selector)
                    .map(StockItem::getItemValuation)
                    .reduce(BigDecimal.ZERO, BigDecimal::add);
    }
}

InventoryReportGenerator

// package and import statements omitted

public class InventoryReportGenerator implements Processor {
    @Override
    public void process(Exchange exchange) throws Exception {
        Message in = exchange.getIn();
        in.setHeader(Exchange.FILE_NAME, 
                        createReportFileName(in.getHeader(Exchange.FILE_NAME, String.class)));
        in.setBody(buildReport(in));
    }

    private String createReportFileName(String inputFileName) {
        return inputFileName.substring(0, inputFileName.indexOf('.')) + "_Report.txt";
    }

    ...

InventoryReportGenerator

    ...

    private String buildReport(Message in) {
        StringBuilder sb = new StringBuilder();
        sb.append("Inventory Report - ")
            .append(LocalDate.now().format(DateTimeFormatter.ofPattern("MM / dd / yyyy")))
            .append("\n\n")
            .append("Summary")
            .append("\n----------------------------------------------\n\n")
            .append("Total SKUs in inventory: ").append(in.getHeader("INVENTORY_COUNT"))
            .append("\n")
            .append("Total Taxable Value in inventory: ")
            .append(in.getHeader("INVENTORY_TAXABLE_VALUATION"))
            .append("\n")
            .append("Total Non-taxable Value in inventory: ")
            .append(in.getHeader("INVENTORY_NONTAXABLE_VALUATION"))
            .append("\n----------------------------------------------\n\n")
            .append("Inventory Details")
            .append("\n----------------------------------------------\n")
            .append(String.format("%6s %10s %5s %7s %7s %6s", 
                                    "Item #", "SKU", "Qty", "Price", "Value", "Tax?"))
            .append("\n")
            .append(String.format("%6s %10s %5s %7s %7s %6s", 
                                    "------", "----------", "-----", "-------", "-------", "----"))
            .append("\n");


            // ... more

InventoryReportGenerator

    ...

        // @formatter:off
        @SuppressWarnings("unchecked")
        List<StockItem> items = (List<StockItem>) in.getBody();
        int index = 1;
        for (StockItem item : items) {
            sb.append(String.format("%6d %10s %5d %7.2f %7.2f %6s", 
                                    index, item.getItemSku(), item.getQuantityOnHand(),
                                    item.getMarkedPrice(), item.getItemValuation(), 
                                    item.isTaxable() ? "Y" : "N"))
              .append("\n");

            index++;
        }
        sb.append("----------------------------------------------\n\n--End Report--");

        return sb.toString();
    }
}

What I like

Well defined responsibilities
Clear flow in and out
It demonstrates basic Camel routing

What I hate

We're doing a lot of work doing low level stuff like parsing CSV file
StringBuilder for report
No tests

The Inventory Reporting app...

The worst thing is, no automated tests.

I'd be afraid to change anything without a lot of manual testing!

Camel Testing

Testing processors and beans should be unit tested
Testing routing within the app should be system tested
Testing routing and processing end-to-end including to/from endpoints outside the app should be integration tested

Camel Testing

I prefer Spock for unit testing
- JUnit or TestNG with mocking works well
Camel provides excellent testing support for integration testing, but it only supports JUnit and TestNG
- I have used Spock for this, but often find myself missing the Camel testing goodies
For system testing I've used Cucumber and Spock
- I am not a big fan of Cucumber, but if it's your company standard, have at it

Preparing for testing

dependencies {
    //It needs to be one of the first dependencies to work
    compile 'ch.qos.logback:logback-classic:1.1.6'

    compile 'org.codehaus.groovy:groovy-all:2.3.11'

    // Camel dependencies
    def camelVersion = '2.16.2'
    compile "org.apache.camel:camel-core:$camelVersion"
    compile "org.apache.camel:camel-groovy:$camelVersion"
    testCompile "org.apache.camel:camel-test:$camelVersion"

    testCompile 'junit:junit:4.11'
    testCompile 'org.spockframework:spock-core:1.0-groovy-2.4'
    testCompile 'cglib:cglib-nodep:2.2'
}

Route testing template

// Route testing is integration testing. We'll use CameltestSupport here
// I'm using Groovy syntax, but Java's is similar - just more verbose
class MyTest extends CamelTestSupport {

    // Register our routes with the test CamelContext
    @Override
    protected RouteBuilder createRouteBuilder() {
        new InventoryReporterRouteBuilder()
    }

    @Test
    void "verify Route A will send correct Message to Route B"() {
        // Mock out routing to route B so we can evaluate what route A sends
        context.getRouteDefinition('routeA').adviceWith(context, 
                                                new AdviceWithRouteBuilder() {
            void configure() throws Exception {
                interceptSendToEndpoint('direct:routeB')
                        .skipSendToOriginalEndpoint()
                        .to('mock:routeB')
            }
        })

        // create a Notifier so tell us when the routing happened
        NotifyBuilder notifier = new NotifyBuilder(context).fromRoute('routeA')
                                                           .wereSentTo('mock:routeB')
                                                           .whenCompleted(1)
                                                           .create()

Route testing in general

    @Test
    void "verify Route A will send correct Message to Route B"() {
        //... we previously mocked next route endpoint and configure a notifier

        // now set up expectations for the mock
        MockEndpoint mockEp = getMockEndpoint('mock:routeB');
        mockEp.expectedHeaderReceived('headerName', headerValue);
        mockEp.expectedBodiesReceived("The answer is 42"); // one or more bodies
        mockEp.resultMinimumWaitTime = 500;

        // now send the inbound message to the route we're testing
        // template is a ProducerTemplate for sending messages, usually in tests
        template.sendBodyAndHeaders('direct:routeA', theBody,
                                    [headerName : headerValue])


        // Now do the assertions. Suppose we expect route A to send
        // In Java use JUnit assertMethods. With Groovy we can use power assert
        assert notifier.matchesMockWaitTime() // Groovy power assert
        assertMockEndpointsSatisfied()
    }

Demo

Testing Inventory Reporter

Routes and Processors

Prepare routes for testing

public class InventoryReporterRouteBuilder extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        getContext().setTracing(true);          // Camel Tracer logs routing at INFO level

        from("file:/tmp/camel-demo/inventory/in?delay=1s&move=../archive")  // consumer
            .routeId("inventoryReporterEntry")
            .convertBodyTo(String.class)
            .process(new InventoryProcessor())
            .to("direct:valueInventory");

        from("direct:valueInventory")
            .routeId("valueInventory")          // routeId gives us a handle to the route
            .process(new InventoryValuator())
            .to("direct:reportInventory");

        from("direct:reportInventory")
            .routeId("reportInventory")
            .process(new InventoryReportGenerator())
            .to("file:///tmp/camel-demo/inventory/out");                    //  producer

    }
}

Testing our RouteBuilder

class InventoryRouteBuilderTest extends CamelTestSupport {
    private static final String TEST_INPUT_FILE_NAME = 'Store123_Inventory_2016-04-01_01-35-45.csv'
    private static final String TEST_OUTPUT_FILE_NAME = 'Store123_Inventory_2016-04-01_01-35-45_Report.txt'

    List<StockItem> items

    @Before
    public void setUp() {
        super.setUp()
        items = createStockItems()
    }

    @Test
    void "test inventoryReporterEntry route"() {
        mockToRoute('inventoryReporterEntry', 'direct:valueInventory', 'mock:valueInventory');
        NotifyBuilder notifier = createNotifier('inventoryReporterEntry', 'mock:valueInventory')

        MockEndpoint mockEp = getMockEndpoint('mock:valueInventory')
        mockEp.expectedHeaderReceived(Exchange.FILE_NAME, TEST_INPUT_FILE_NAME)
        mockEp.expectedHeaderReceived('INVENTORY_COUNT', items.size())
        mockEp.expectedBodiesReceived([items])  // This looks strange, but I'll explain
        mockEp.resultMinimumWaitTime = 500

        // This will actually write the file out to the inbound directory
        template.sendBodyAndHeader('file:/tmp/camel-demo/inventory/in?delay=1s&move=../archive', 
                                    createInput(), Exchange.FILE_NAME, TEST_INPUT_FILE_NAME)

        assert notifier.matchesMockWaitTime()
        assertMockEndpointsSatisfied()

        List skus = mockEp.exchanges[0].in.body.collect { StockItem item -> item.itemSku }
        assert skus == ['101', '102', '103']
    }

    // more...

Testing

    // ... more

    @Test
    void "test valueInventory route"() {
        mockToRoute('valueInventory', 'direct:reportInventory', 'mock:reportInventory');
        NotifyBuilder notifier = createNotifier('valueInventory', 'mock:reportInventory')

        MockEndpoint mockEp = getMockEndpoint('mock:reportInventory')
        mockEp.expectedHeaderReceived(Exchange.FILE_NAME, TEST_INPUT_FILE_NAME)
        mockEp.expectedHeaderReceived('INVENTORY_COUNT', this.items.size())
        mockEp.expectedHeaderReceived('INVENTORY_TAXABLE_VALUATION', new BigDecimal("650.00"))
        mockEp.expectedHeaderReceived('INVENTORY_NONTAXABLE_VALUATION', new BigDecimal("400.00"))
        mockEp.expectedBodiesReceived([this.items])
        mockEp.resultMinimumWaitTime = 500

        // This will actually write the file out to the inbound directory
        template.sendBodyAndHeaders('direct:valueInventory', this.items,
                                    [(Exchange.FILE_NAME) : TEST_INPUT_FILE_NAME, 
                                     INVENTORY_COUNT : this.items.size()])

        assert notifier.matchesMockWaitTime()
        assertMockEndpointsSatisfied()

        List skus = mockEp.exchanges[0].in.body.collect { StockItem item -> item.itemSku }
        assert skus == ['101', '102', '103']
    }

    // more ...

Testing

    // ... more

    @Test
    void "test reportInventory route"() {
        mockToRoute('reportInventory', 'file:/tmp/camel-demo/inventory/out', 
                    'mock:reportDir')

        NotifyBuilder notifier = createNotifier('reportInventory', 'mock:reportDir')

        MockEndpoint mockEp = getMockEndpoint('mock:reportDir')
        mockEp.expectedHeaderReceived(Exchange.FILE_NAME, TEST_OUTPUT_FILE_NAME)
        mockEp.expectedBodiesReceived(createExpectedReport())
        mockEp.resultMinimumWaitTime = 500

        // This will actually write the file out to the inbound directory
        template.sendBodyAndHeaders('direct:reportInventory', this.items,
                [(Exchange.FILE_NAME) : TEST_INPUT_FILE_NAME,
                 'INVENTORY_COUNT' : this.items.size(),
                 'INVENTORY_TAXABLE_VALUATION' :new BigDecimal("650.00"),
                 'INVENTORY_NONTAXABLE_VALUATION' : new BigDecimal("400.00")])

        assert notifier.matchesMockWaitTime()
        assertMockEndpointsSatisfied()
    }

    // more ...

Testing

    // ... more

    private String createInput() {
        '''SKU,Qty,Price,Taxable
101,30,20.00,Y
102,40,10.00,N
103,100,0.50,Y'''
    }

    private List<StockItem> createStockItems() {
        [
                new StockItem('101', 30, new BigDecimal("20.00"), true),
                new StockItem('102', 40, new BigDecimal("10.00"), false),
                new StockItem('103', 100, new BigDecimal("0.50"), true)
        ]
    }

    // more ...

Testing

    // ... more

    private String createExpectedReport() {
"""Inventory Report - ${InventoryReportGenerator.createReportDate()}

Summary
----------------------------------------------

Total SKUs in inventory: 3
Total Taxable Value in inventory: 650.00
Total Non-taxable Value in inventory: 400.00
----------------------------------------------

Inventory Details
----------------------------------------------
Item #        SKU   Qty   Price   Value   Tax?
------ ---------- ----- ------- -------   ----
     1        101    30   20.00  600.00      Y
     2        102    40   10.00  400.00      N
     3        103   100    0.50   50.00      Y
----------------------------------------------

--End Report--"""
    }

    // more ...

Testing

    // ... more

    @Override
    protected RouteBuilder createRouteBuilder() {
        new InventoryReporterRouteBuilder()
    }

    private NotifyBuilder createNotifier(String fromRouteId, String toRouteUri) {
        new NotifyBuilder(context).fromRoute(fromRouteId)
                .wereSentTo(toRouteUri)
                .whenCompleted(1)
                .create()
    }

    private RouteDefinition mockToRoute(final String fromRouteId, 
                                        final String oldToRouteUri, 
                                        final String mockToRouteUri) {
        context.getRouteDefinition(fromRouteId).adviceWith(context, 
                                                    new AdviceWithRouteBuilder() {
            void configure() throws Exception {
                interceptSendToEndpoint(oldToRouteUri)
                        .skipSendToOriginalEndpoint()
                        .to(mockToRouteUri)
            }
        })
    }
}

Type Conversion

As we move through routes, we often need to convert types
Sometimes, we do the conversion ourselves
Often, we can let Camel do it using its built-in converters which allow conversion to/from these types:
- File
- String
- byte[] and ByteBuffer
- InputStream and OutputStream
- Reader and Writer
- Document and Source

Data Formats

Data formats are a related concept to type conversion
We often need to marshal and unmarshal from one format to another
Some common formats
- String
- CSV
- JSON-XML
- JAXB
- See this page for a list of built-in data formats
  - http://camel.apache.org/data-format.html
We'll look at three in detail...

Data Formats

String

from("file:/var/inbox/data.csv")
    .marshal().string("UTF-8")
    .to("jms:queue:stage")

from("jms:queue:stage")
    .process(orderPreparation)
    .to("jms:queue:order")

from("jms:queue:order")
    .unmarshal().string("UTF-8")
    .process(newOrderProcessor)

Data Formats

JSON

// lets turn Object messages into json - component: camel-xstream
// then send to MQSeries
from("activemq:My.Queue1")
    .marshal().json()
    .to("mqseries:Another.Queue")

// lets use Jackson - component: camel-jackson
from("activemq:My.Queue2")
    .marshal().json(JsonLibrary.Jackson)
    .to("mqseries:Another.Queue")


// let's use GSON - component: camel-gson
from("activemq:My.Queue3")
    .marshal().json(JsonLibrary.Gson)
    .to("mqseries:Another.Queue")

// and back from JSON to POJO
from("direct:backToPOJO")
    .unmarshal().json(JsonLibrary.Jackson, MyPojo)
    .process(pogoProcessor)

Data Formats

CSV

class InventoryReporterRouteBuilder extends RouteBuilder {
    @Override
    void configure() throws Exception {
        getContext().setTracing(true);

        CsvDataFormat csv = new CsvDataFormat()
        csv.skipHeaderRecord = true

        // file here creates a polling consumer
        from("file:/tmp/camel-demo/inventory/in?delay=1s&move=../archive") 
                .routeId("inventoryReporterEntry")
                .unmarshal(csv)
                .process(new InventoryProcessor())
                .to("direct:valueInventory");

        from("direct:valueInventory")
                .routeId("valueInventory")
                .process(new InventoryValuator())
                .to("direct:reportInventory");

        from("direct:reportInventory")
                .routeId("reportInventory")
                .process(new InventoryReportGenerator())
                .to("file:///tmp/camel-demo/inventory/out"); // a producer

    }
}

Data Formats

CSV

class InventoryProcessor implements Processor {
    @Override
    void process(Exchange exchange) throws Exception {
        Message inMsg = exchange.getIn()
        List<List<String>> records = inMsg.body as List<List<String>>

        List<StockItem> items = parseInput(records)
        inMsg.setBody(items)
        inMsg.setHeader("INVENTORY_COUNT", items.size())
    }

    protected List<StockItem> parseInput(List<List<String>> records) {
        records.collect { List<String> fields -> createStockItem(fields) }
    }

    private StockItem createStockItem(List<String> fields) {
        new StockItem(fields[0], Integer.valueOf(fields[1]), 
                        new BigDecimal(fields[2]), "Y".equals(fields[3]))
    }
}

Processors & Beans

All along, I've been using processors

from("direct:valueInventory")
    .routeId("valueInventory")
    .process(new InventoryValuator())
    .to("direct:reportInventory");

class InventoryValuator implements Processor {
    @Override
    void process(Exchange exchange) throws Exception {
        ...
    }
}

Processors & Beans

Using the Groovy DSL, closures can be coerced into processors for simple processing

from("direct:enterHere")
    .process( { Exchange ex -> ex.in.body = 'The answer is 42' } )
    .to("direct:nextStop");

Processors & Beans

Besides process, you can use simple beans

// One POGO bean, with id = serviceBean, has multiple methods
// If method takes 
from("direct:enterHere")
    .to("bean:serviceBean?method=filter")    // bean uri is one way
    .to(serviceBean, "map")                  // object ref is another way
    .beanRef("serviceBean", "reduce")        // beanRef is another way
    .to("direct:nextStop");

// Camel will supply / convert the type as needed
// Camel will try to bind the body of the Exchange to the first parameter; e.g.

/*
The second argument can be:
org.apache.camel.Exchange
org.apache.camel.Message
org.apache.camel.CamelContext
org.apache.camel.TypeConverter
org.apache.camel.spi.Registry
java.lang.Exception
*/
public String doSomething(String body, Exchange exchange) { ... }

Camel Registry

Camel uses a registry for its beans
By default it's a JNDI implementation
However, Camel plays very well with Spring
If Spring is used, Camel will rely on the Spring ApplicationContext as its registry

Other Spring Goodies

Camel uses Spring Transactions as the default transaction handling in components like JMS and JPA
Camel works with Spring 2 XML processing with the Xml Configuration
Camel Spring XML Schema's is defined at Xml Reference
Camel supports a powerful version of Spring Remoting which can use powerful routing between the client and server side along with using all of the available Components for the transport
Camel provides powerful Bean Integration with any bean defined in a Spring ApplicationContext
Camel integrates with various Spring helper classes; such as providing Type Converter support for Spring Resources etc
Allows Spring to dependency inject Component instances or the CamelContext instance itself and auto-expose Spring beans as components and endpoints.
Allows you to reuse the Spring Testing framework to simplify your unit and integration testing using Enterprise Integration Patterns and Camel's powerful Mock and Test endpoints
From Camel 2.15 onwards Camel supports Spring Boot using the camel-spring-boot component.

Miscellany

The common experience of enterprise integration can be summed up in one word

Failure!

Error Handling

Not this kind of failure...

Error Handling

This kind...

Error Handling

We must ~~anticipate~~ expect failure

http://bit.ly/1SMvowA

Error Handling

Why is integration so hard?

We are generally optimists and think about sunny day solutions
We do an okay job of handling errors in our code (think: try/catch), but rest of world is a black box
Tight schedules squeeze out error handling stories
We lose control of messages once they start routing
We embrace most or all of the 8 fallacies of distributed computing

Error Handling

8 Fallacies of Distributed Computing

The network is reliable.
Latency is zero.
Bandwidth is infinite.
The network is secure.
Topology doesn't change.
There is one administrator.
Transport cost is zero.
The network is homogeneous.

https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

Camel Error Handling

Camel treats errors (exceptions) as
- Recoverable
- Irrecoverable
Irrecoverable examples
- NullPointerException
- SQLException from missing driver
Recoverable examples
- ConnectionException
- IOException
Errors assumed to be recoverable unless the fault flag set on the Exchange Message

Error Handling

class OrderProcessor implements Processor {
    OrderDao orderDao

    @Override
    void process(Exchange exchange) {
        String orderId = exchange.in.body

        try {
            Order order = orderDao.load(orderId)
        } catch(DataException e) {
            Message msg = exchange.in
            msg.setFault(true)
            msg.body = "Missing Order for order id = ${orderId}"
            msg.exception = e
        }

    }

    void setOrderDao(OrderDao orderDao) {
        this.orderDao = orderDao
    }   

}

Error Handling

Camel error handling doesn't handle the edges of the app

From Camel in Action, 1st Ed, Manning

However, components often implement some error handling

Error Handling

DefaultErrorHandler
- default error handler in Camel 2.x
- by default, exceptions passed back to caller
Dead Letter Channel
- after retries fail send msg to a dead letter ep
TransactionErrorHandler
LoggingErrorHandler
NoErrorHandler

Camel provides multiple error handlers

Error Handling

Camel provides decent error handling out of the box

// By default, we get defaultErrorHandler with no retries, but we want retries
// so we need to declare at RouteBuilder level in Java/Groovy DSL
// Can also declare a route level which will take precedence over the RB level

errorHandler(defaultErrorHandler()     // defaultErrorHandler doesn't do much
    .useOriginalMessage()              // processors xform msg; retry w/ original
    .maximumRedeliveries(5)
    .redeliveryDelay(5000)
    .retryAttemptedLogLevel(LoggingLevel.WARN))

Error Handling

Dead Letter Channel

http://camel.apache.org/dead-letter-channel.html

Error Handling

Camel "Dead Letter" handling

// After retries exhausted route to dead queue
errorHandler(deadLetterChannel("jms:queue:dead") 
    .useOriginalMessage()
    .maximumRedeliveries(5)
    .redeliverDelay(5000)

Error Handling

Exception Handlers

// Exception handlers can deal with specific exceptions that take precedence 
// over the error handler. Here, we retry more times, but at longer intervals
// if an IOException is thrown

onException(IOException)
    .maximumRedeliveries(10);
    .redeliveryDelay(30000)

errorHandler(defaultErrorHandler()
    .useOriginalMessage()
    .maximumRedeliveries(5)
    .redeliveryDelay(5000)
    .retryAttemptedLogLevel(LoggingLevel.WARN))

...

Error Handling

Error handling per route

// Error handlers and exception handlers can be on the routes, too

.onException(ConnectionException)
    .maximumRedeliveries(7)
    .redeliveryDelay(60000)

errorHandler(defaultErrorHandler()
    .useOriginalMessage()
    .maximumRedeliveries(5)
    .redeliveryDelay(5000)
    .retryAttemptedLogLevel(LoggingLevel.WARN))

from("jms:queue:inbox")
    .process(decryptProcessor)
    .to(validateProcessor)
    .to(enrichProcessor)
    .to("direct:queue:order")

from("jms:queue:order?concurrentConsumers=10")
    .onException(IOException)
        .maximumRedeliveries(10)
        .redeliveryDelay(30000)
    .to("http://com.supplier.com/ws/submitOrder")

Camel Components

There are dozens of Camel components
- Recall a component is an endpoint factory
We've seen file and timer
Let's see a few more...

Camel Components

camel-jetty

from("jetty:http://localhost:8080/myapp/myservice")
    .convertTo(String)
    .process(wsProcessor);

<== component name

Camel Components

camel-ftp

from("ftp://username@acme.com/inventory/in?password=foobar&delay=1s&move=../archive")
    .convertBodyTo(String.class)
    .process(new InventoryProcessor())
    .to("direct:valueInventory")

from("direct:valueInventory")
    .process(new InventoryValuator())
    .to("direct:reportInventory")

from("direct:reportInventory")
    .process(new InventoryReportGenerator())
    .to("sftp://username@acme.com/inventory/reports?password=foobaz")

Camel Components

camel-mail

from("jms://queue:subscription")
    .to("smtp://jack@mymailserver.com?password=secret")

from("pop3://jack@mymailserver.com?password=foobar&consumer.delay=60000")
    .to("log:email")

from("imap://jack@mymailserver.com?password=foobaz&consumer.delay=30000")
    .to("log:email")

Camel Components

camel-quartz

// fire a message every five minutes starting at 12pm (noon) to 6pm on weekdays
from("quartz://monitors/MonitoringTimer?cron=0+0/5+12-18+?+*+MON-FRI")
    .to("direct:monitoringService")

// fire a message on the 2nd Monday every month at 2:15 am
from("quartz://housekeeping/MonthlyHousekeepingTimer?cron=0 15 2 ? 1/1 MON#2 *")
    .to("direct:housekeepingService")

/* Cron Expressions

   Each field from left to right:

    1. Seconds
    2. Minutes
    3. Hours
    4. Day-of-Month
    5. Month
    6. Day-of-Week
    7. Year (optional field)
*/

http://www.quartz-scheduler.org/documentation/quartz-2.x/tutorials/tutorial-lesson-06

http://camel.apache.org/quartz.html

Logging

class InventoryReporterRouteBuilder extends RouteBuilder {
    @Override
    void configure() throws Exception {
        getContext().setTracing(true)                     // logs route tracing at INFO level

        CsvDataFormat csv = new CsvDataFormat()
        csv.skipHeaderRecord = true

        from("file:/tmp/camel-demo/inventory/in?delay=1s&move=../archive")
            .routeId("inventoryReporterEntry")
            .to("log:com.jackfrosch.camel")               // Logs the message exchange
            .unmarshal(csv)
            .log('Hey look! Unmarshalled csv: ${body}")   // Simple Expression Language
            .process(new InventoryProcessor())
            .to("direct:valueInventory");

    }
}

http://camel.apache.org/logeip.html

http://camel.apache.org/log.html

Simple EL: http://camel.apache.org/simple.html

Other EIP Implementations

Message Filters
Content Based Routing
Message Splitting
Message Aggregation

Message Filter

errorHandler(deadLetterChannel("file:/var/dead_letters"));
 
from("direct:quotes")
    .filter(header("type").isEqualTo("widget"))
        .to("direct:widgetsOnly");

http://camel.apache.org/message-filter.html

Content Based Router

errorHandler(deadLetterChannel("file:/var/dead_letters"));
  
from("direct:a")
    .choice()
        .when(header("type").isEqualTo("widget"))
            .to("direct:Widgets_R_Us")
        .when(header("type").isEqualTo("gadget"))
            .to("direct:Gadgets_R_Us")
        .otherwise()
            .log(LoggingLevel.WARN, "Unsupoorted type: ${header.type}");

http://camel.apache.org/content-based-router.html

Message Splitting

errorHandler(deadLetterChannel("file:/var/dead_letters"));
 
from("direct:orderEntry")
    .split(body())    // if body is a Collection, iterator or array
    .to("direct:lineItemHandler")


from("direct:csvOrderEntry")
    .split(body(String.class).tokenize("\n"))    // split lines
    .process(csvToLineItems)
    .to("direct:lineItemHandler")

http://camel.apache.org/splitter.html

http://javarticles.com/2015/07/apache-camel-splitter-using-tokenizer-expression.html

Message Aggregation

from("direct:start")
    // aggregate all exchanges correlated by the id header.
    // Aggregate them using the BodyInAggregatingStrategy strategy which
    // and after 3 seconds of inactivity them timeout and complete the aggregation
    // and send it to mock:aggregated
    .aggregate(header("id"), new BodyInAggregatingStrategy())
        .completionTimeout(3000)
    .to("direct:aggregated");

http://camel.apache.org/aggregator2.html

Message Aggregation

// simply combines Exchange body values into an List (done the Groovy way)
class ListAggregationStrategy implements AggregationStrategy {
 
   Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
        Object newBody = newExchange.in.body
        if (oldExchange == null) {
            newExchange.in.body = [newBody]
            return newExchange
        } else {
            List list = oldExchange.in.body as List
            list += newBody
            return oldExchange
        }
    } 
}

http://camel.apache.org/aggregator2.html

Message Aggregation

//simply combines Exchange String body values using '+' as a delimiter
class StringAggregationStrategy implements AggregationStrategy {
 
    public Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
        if (oldExchange == null) { 
            return newExchange;  
        }
 
        String oldBody = oldExchange.getIn().getBody(String.class);
        String newBody = newExchange.getIn().getBody(String.class);
        oldExchange.getIn().setBody(oldBody + "+" + newBody);
        return oldExchange;
    }
}

http://camel.apache.org/aggregator2.html

Message Aggregation

http://camel.apache.org/aggregator2.html

completionTimeout
completionInterval
completionSize
completionPredicate
completionFromBatchConsumer
forceCompletionOnStop
Using a AggregateController

How will we know when we're done aggregating?

Demo

Message Splitting and Aggregation

Orders and LineItems

@ToString(includeNames = true, includePackage = false, 
          ignoreNulls = true)
class LineItem {
    int itemNo
    String productNo
    BigDecimal price = 0.00
    int qty = 1
    BigDecimal taxRate = 0.00
}

@ToString(includeNames = true, includePackage = false, ignoreNulls = true, 
          excludes = 'lineItems')
class Order {
    String orderId                  // this will be our "correlation id"
    String postalCode               // drives tax on LineItems
    List<LineItem> lineItems = []

    void addLineItem(LineItem item) { lineItems << item  }

    BigDecimal getOrderTotal() {
        lineItems.collect { LineItem item ->  item.price * item.qty 
                                                * (1.00 + item.taxRate) }
                 .inject(0.00) { sum, value -> sum + value }
    }

    List<Integer> getLineItemIndexes() { lineItems*.itemNo }
}

SplitterAggregatorRouteBuilder

class SplitterAggregatorRouteBuilder extends RouteBuilder {
    OrderHandler orderHandler = new OrderHandler()

    ExecutorService cachedThreadPool = Executors.newCachedThreadPool()
    ExecutorService fixedThreadPool = Executors.newFixedThreadPool(100) // # of line items

    void configure() {
        from("direct:orderEntry")
                .routeId("orderEntry")
                .bean(orderHandler, 'prepareForSplit')
                .split(body())                                        // one thread
//                .split(body()).parallelProcessing()                 // 10 threads
//                .split(body()).executorService(cachedThreadPool)    // variable threads
//                .split(body()).executorService(fixedThreadPool)     // thread pool
                .to('direct:calculateTax')

        from("direct:calculateTax")
                .routeId("calculateTax")
                .log('Received at calculateTax: ${body}')
                .bean(orderHandler, 'calculateTax')
                .to("direct:aggregator")

        // ...
    }
}

SplitterAggregatorRouteBuilder cont'd

class SplitterAggregatorRouteBuilder extends RouteBuilder {
    //... 
    void configure() {
        // ... 

        from("direct:aggregator")
                .routeId("aggregator")
                .log('Received at aggregator: ${body}')
                .aggregate(header('orderId'), new LineItemAggregatorStrategy())
                    .completionSize(header('lineItemCount'))
                .bean(orderHandler, 'rebuildOrder')
                .to('direct:finished')

        from("direct:finished")
                .routeId("finished")
                .log('Received at finished: Order: ${body} ...')
                .log('LineItem order: ${body.itemNumbers}')

    }
}

LineItemAggregatorStrategy

class LineItemAggregatorStrategy implements AggregationStrategy {
    @Override
    Exchange aggregate(Exchange oldExchange, Exchange newExchange) {
        LineItem item = newExchange.in.body as LineItem

        if (oldExchange == null) {
            newExchange.in.body = [item]
            newExchange
        } else {
            List<LineItem> list = oldExchange.in.body as List<LineItem>
            list << item
            return oldExchange
        }
    }
}

OrderHandler

class OrderHandler {

    void prepareForSplit(Exchange exchange) {
        Message msg = exchange.in
        Order order = msg.body as Order

        msg.headers.orderId = order.orderId
        msg.headers.postalCode = order.postalCode
        msg.headers.lineItemCount = order.lineItems.size()

        exchange.in.body = order.lineItems
    }

    // ...
}

OrderHandler cont'd

class OrderHandler {

    // ...

    void calculateTax(Exchange exchange) {
        String postalCode = exchange.in.headers.postalCode
        LineItem item = exchange.in.body as LineItem

        BigDecimal taxRate
        switch(postalCode) {
            case '12345':
                taxRate = 0.05
                break
            case '67890':
                taxRate = 0.08
                break
            default:
                taxRate = 0.06
        }
        item.taxRate = taxRate

        Thread.sleep(250) // simulating time to calculate it
    }
}

OrderHandler cont'd

class OrderHandler {

    // ...

    void rebuildOrder(Exchange exchange) {
        Message msg = exchange.in
        List<LineItem> items = msg.body as List<LineItem>
        items.sort { it.itemNo }

        Order order = new Order(orderId: msg.headers.orderId,
                postalCode: msg.headers.postalCode,
                lineItems: items)

        msg.body = order
    }
}

Whew!

That's a lot of information

Start small

Experiment with routes

Experiment with components

In no time, you'll get the hang of it!

Summary

Camel Concepts
Developing Camel Applications
Miscellany

Questions?

Resources

Manning Camel in Action
- 2nd Ed in MEAP
camel.apache.org
- /manual/camel-manual-2.16.2.html
- /discussion-forums.html
- /mailing-lists.html
www.enterpriseintegrationpatterns.com
Feel free to reach out to me

Code can be found at: https://github.com/jfrosch/intro_to_camel

Thank you!

Please complete surveys
Connect with me
- LinkedIn: jackfrosch
- Twitter: @jackfrosch
- Email: jackfrosch@gmail.com