Category: java

  • Metrics in Hoplin.io

    Hoplin does not have any dependencies on any existing metrics libraries but rather it provides a way to hook into the underlying metrics via MetricsPublisher interface. Metrics expose a number of key/value pairs that are updated and send to metrics consumers.

    Depending on which client we use the metrics key will be different and it is up to the consumer to normalize the name. Data is packed into a Map<String, Map<String,String>> structure, this allows us to add metrics easily without breaking any API.

    Sample Payload
    {exchange.rpc.logs-rpc.request.log={received.size=211, sent.size=204, received.count=1, sent.count=1}}

    Metrics Key = exchange.rpc.logs-rpc.request.log
    received.size = Amount of data received by this client
    received.count = Number of messages received

    sent.size = Amount of data sent by this client
    sent.count = Number of messages sent

    Instantiating metrics consumer

    FunctionMetricsPublisher
    	.consumer(EmitLogTopic::metrics)
    	.withInterval(1, TimeUnit.SECONDS)
    	.withResetOnReporting(false)
    	.build()
    	.start();
    
    private static void metrics(final Map<String, Map<String, String>> stat)
    {
      System.out.println("Metrics Info : " + stat);
    }

  • RabittMQ batch message processing

    There are times when we want to fire set of jobs and be notified when all of them complete. This can be easily accomplished with the latest version of Hoplin.io RabbitMQ client.

    A use case for using batch messages would be partitioning a document and processing each partition via separate client.

    As always we need our Publisher and Subscriber we will start with publisher first.

    We start by creating a new client and then enqueuing number of jobs to process, upon completion we display the time it took to complete all jobs. Client will attempt to use Direct-Reply queue if available.

       public static void main(final String... args) throws IOException, InterruptedException
        {
            final BatchClient client = new DefaultBatchClient(options(), bind());
    
            client.startNew(context ->
            {
                for(int i = 0; i < 1000; ++i)
                {
                    context.enque(() -> new LogDetail("Msg >> " + System.nanoTime(), "info"));
                    context.enque(() -> new LogDetail("Msg >> " + System.nanoTime(), "warn"));
                }
            })
            .whenComplete((context, throwable)->
            {
                    log.info("Batch completed in : {}", context.duration());
            });
    
            Thread.currentThread().join();
        }
    
        private static Binding bind()
        {
            return BindingBuilder
                    .bind("batch.documents")
                    .to(new DirectExchange("exchange.batch"))
                    .build()
                    ;
        }

    The subscriber is little bit more involved, this is the part of API that needs to be simplified.

    
    /**
     * Batch Job receiver
     */
    public class ReceiveBatchJob extends BaseExample
    {
        private static final Logger log = LoggerFactory.getLogger(ReceiveBatchJob.class);
    
        private static final String EXCHANGE = "exchange.batch";
    
        private static RabbitMQClient mqClient;
    
        public static void main(final String... args) throws InterruptedException
        {
            final ExchangeClient client = DirectExchangeClient.create(options(), EXCHANGE);
            mqClient = client.getMqClient();
    
            client.subscribe("test", LogDetail.class, ReceiveBatchJob::handle);
            Thread.currentThread().join();
        }
    
        private static void handle(final LogDetail msg, final MessageContext context)
        {
            final AMQP.BasicProperties properties = context.getProperties();
            final String replyTo = properties.getReplyTo();
            final String correlationId = properties.getCorrelationId();
            final Map<String, Object> headers = properties.getHeaders();
            final Object batchId = headers.get("x-batch-id");
            headers.put("x-batch-correlationId", correlationId);
    
            log.info("Incoming context        >  {}", context);
            log.info("Incoming replyTo        >  {}", replyTo);
            log.info("Incoming msg            >  {}", msg);
            log.info("Incoming correlationId  >  {}", correlationId);
            log.info("Incoming batchId        >  {}", batchId);
    
            final LogDetail reply = new LogDetail("Reply Message", "WARN");
            mqClient.basicPublish("", replyTo, reply, headers);
        }
    }
    

    There at two important properties that need to be looked at x-batch-id and x-batch-correlationId, currently they need to be copied directly form message and then reply need to be published via the underlying client.

    Not really best API design at the moment as it mixes concerns and exposed the underlying RattitMQ client but the initial release will try to address this and simplify usage.

  • hoplin.io A lightweight RabbitMQ client for Java (built on top of rabittmq java client)

    A lightweight RabbitMQ client for Java (built on top of rabittmq java client)

    Documentation and project available at GitHub repo
    https://github.com/gregbugaj/hoplin.io

    To make working with RabbitMQ as simple as possible with minimum dependencies.

    Minimal dependencies, simple configuration and API.

    • Subscriber client
    • Publisher client
    • Async RPC Client

    Creating simple RabbitMQ client can be done in couple different ways. 

    The simplest way with minimal configuration 

    ExchangeClient client = ExchangeClient.topic(RabbitMQOptions.from("host=localhost")) 

    This creates new Exchange client bound to a Topic exchange.

    We can also specify which queue and which routing key we want to handle.

    final RabbitMQOptions options = RabbitMQOptions.from("host=localhost");
    final ExchangeClient client = ExchangeClient.topic(options, "my.exchange", "log.critical", "log.critical.*")
    

    For complete control we can use the Exchange to Queue Binding builder.

    ExchangeClient clientFromBinding(final String exchange, final String queue, final String routingKey)
        {
            final Binding binding = BindingBuilder
                    .bind(queue)
                    .to(new TopicExchange(exchange))
                    .withAutoAck(true)
                    .withPrefetchCount(1)
                    .withPublisherConfirms(true)
                    .with(routingKey)
                    .build();
    
            return ExchangeClient.topic(options(), binding);
        }

    This is the most flexible method as it allows us to control all the aspect of how messages are handled.

    Publishing and receiving messages is simple as well. Both methods provide number of overloaded methods to provide different levels of flexibility.

    // Publish message
    client.publish(new LogDetail("Msg : " + System.nanoTime()));
    
    // Consume message
     client.subscribe(LogDetail.class, msg-> log.info("Message received [{}]", msg));

    Here is example that includes both the Publisher and Subscriber

    public class SamePublisherConsumerExample extends BaseExample
    {
        private static final Logger log = LoggerFactory.getLogger(SamePublisherConsumerExample.class);
    
        private static final String EXCHANGE = "topic_logs";
    
        public static void main(final String... args) throws InterruptedException
        {
            log.info("Starting producer/consumer for exchange : {}", EXCHANGE);
            final ExchangeClient client = ExchangeClient.topic(options(), EXCHANGE);
            client.subscribe(LogDetail.class, SamePublisherConsumerExample::handle);
    
            for(int i = 0; i < 5; ++i)
            {
                client.publish(createMessage("info"), "log.info.info");
                client.publish(createMessage("debug"), "log.info.debug");
    
                Thread.sleep(1000L);
            }
        }
    
        private static void handle(final LogDetail msg)
        {
            log.info("Incoming msg : {}", msg);
        }
    
        private static LogDetail createMessage(final String level)
        {
          return new LogDetail("Msg : " + System.nanoTime(), level);
        }
    
    }

  • Data retrieval service with exponential backoff

    Here we will create Data retrieval service with exponential backoff that we covered in the previous post.

    Implementation

    
    package com.rms.blueprint.data;
    
    import java.util.Date;
    import java.util.Objects;
    import java.util.concurrent.TimeUnit;
    import java.util.function.Function;
    import java.util.function.ObjLongConsumer;
    import java.util.function.Supplier;
    
    import org.slf4j.Logger;
    import org.slf4j.LoggerFactory;
    
    public class DataRetrievalWithBackoff implements Runnable
    {
        public final Logger LOGGER = LoggerFactory.getLogger(DataRetrievalWithBackoff.class);
    
        private final Supplier pendingSupplier;
    
        private final ObjLongConsumer readyConsumer;
    
        private final Function capacitySupplier;
    
        private final long minLoadDurationInSeconds;
    
        private final long capacity;
    
        private final long maxBackoffDelayInSeconds;
    
        /**
         * @param config
         *            Configuration properties
         * @param pendingSupplier
         *            Supplier that tells us how many items is being processed at
         *            this time
         * @param readyConsumer
         *            Consumer that will be called when data is ready to load
         * @param capacitySupplier
         *            Function to calculate current capacity
         */
        private DataRetrievalWithBackoff(final long capacity, final long minLoadDurationInSeconds,
                final long maxBackoffDelayInSeconds, final Supplier pendingSupplier,
                final ObjLongConsumer readyConsumer, final Function capacitySupplier)
        {
            Objects.requireNonNull(pendingSupplier);
            Objects.requireNonNull(readyConsumer);
            Objects.requireNonNull(capacitySupplier);
    
            this.capacity = capacity;
            this.minLoadDurationInSeconds = minLoadDurationInSeconds;
            this.maxBackoffDelayInSeconds = maxBackoffDelayInSeconds;
    
            this.pendingSupplier = pendingSupplier;
            this.readyConsumer = readyConsumer;
            this.capacitySupplier = capacitySupplier;
    
            LOGGER.info(String.format("capacity ", capacity));
            LOGGER.info(String.format("minLoadDurationInSeconds ", minLoadDurationInSeconds));
            LOGGER.info(String.format("maxBackoffDelayInSeconds ", maxBackoffDelayInSeconds));
    
        }
    
        @Override
        public void run()
        {
            LOGGER.info("Running Data Retrieval");
    
            long lastLoadedTime = 0l;
            int attempt = 0;
    
            while (true)
            {
                if (Thread.currentThread().isInterrupted())
                {
                    LOGGER.trace("Interrupted stopping [while]");
                    break;
                }
    
                final long delta = TimeUnit.MILLISECONDS.toSeconds(System.currentTimeMillis() - lastLoadedTime);
                final long pending = pendingSupplier.get();
    
                final long backoffTime = DataUtil.backoff(attempt,
                                                          maxBackoffDelayInSeconds,
                                                          minLoadDurationInSeconds / 2.0);
    
                LOGGER.trace("Loading : lastLoaded : {} > {}  delta(s) {} pending : {} backoffTime  = {}",
                             new Object[] { lastLoadedTime, new Date(lastLoadedTime), delta, pending, backoffTime });
    
                if (delta >= minLoadDurationInSeconds && pending <= capacitySupplier.apply(capacity))
                {
                    LOGGER.info("Loading : lastLoaded :  {} >  {}  delta(s) {} pending : {}",
                                new Object[] { lastLoadedTime, new Date(lastLoadedTime), delta, pending });
    
                    // let the consumer know that we are ready
                    readyConsumer.accept(backoffTime, attempt);
    
                    if (pending == 0)
                        ++attempt;
                    else
                        attempt = 0;
    
                    lastLoadedTime = System.currentTimeMillis();
                }
                else
                {
                    ++attempt;
                }
    
                try
                {
    
                    Thread.sleep(TimeUnit.SECONDS.toMillis(backoffTime));
                }
                catch (final InterruptedException e)
                {
                    LOGGER.trace("Interrupted stopping  [sleep]");
                    Thread.currentThread().interrupt();
                    break;
                }
            }
        }
    
        public static class Builder
        {
            private final Function DEFAULT_CAPACITY_SUPPLIER = (capacity) -> capacity / 2;
    
            private long minLoadDurationInSeconds = 60;
    
            private long capacity = 100;
    
            private long maxBackoffDelayInSeconds = 120;
    
            private Supplier pendingSupplier;
    
            private ObjLongConsumer readyConsumer;
    
            private Function capacitySupplier;
    
            public Builder capacity(final long capacity)
            {
                this.capacity = capacity;
                return this;
            }
    
            public Builder maxBackoffDelay(final long duration, final TimeUnit unit)
            {
                Objects.requireNonNull(unit);
                this.maxBackoffDelayInSeconds = unit.toSeconds(duration);
                return this;
            }
    
            public Builder minLoadDuration(final long duration, final TimeUnit unit)
            {
                Objects.requireNonNull(unit);
                this.minLoadDurationInSeconds = unit.toSeconds(duration);
                return this;
            }
    
            public Builder readyConsumer(final ObjLongConsumer readyConsumer)
            {
                Objects.requireNonNull(readyConsumer);
                this.readyConsumer = readyConsumer;
                return this;
            }
    
            public Builder capacitySupplier(final Function capacitySupplier)
            {
                Objects.requireNonNull(capacitySupplier);
                this.capacitySupplier = capacitySupplier;
                return this;
            }
    
            public Builder pendingSupplier(final Supplier pendingSupplier)
            {
                Objects.requireNonNull(capacitySupplier);
                this.pendingSupplier = pendingSupplier;
                return this;
            }
    
            public DataRetrievalWithBackoff build()
            {
                // check invariant
                Objects.requireNonNull(pendingSupplier, "Pening items supplier not provided");
                Objects.requireNonNull(readyConsumer, "Ready Consumer not provided");
    
                if (capacitySupplier == null)
                    capacitySupplier = DEFAULT_CAPACITY_SUPPLIER;
    
                return new DataRetrievalWithBackoff(capacity,
                                                    minLoadDurationInSeconds,
                                                    maxBackoffDelayInSeconds,
                                                    pendingSupplier,
                                                    readyConsumer,
                                                    capacitySupplier == null ? DEFAULT_CAPACITY_SUPPLIER
                                                        : capacitySupplier);
            }
        }
    }
    
    
    

    Usage

    
       //@formatter:off
                final DataRetrievalWithBackoff service = new DataRetrievalWithBackoff.Builder()
                        .capacity(1000)
                        .maxBackoffDelay(100, TimeUnit.SECONDS)
                        .minLoadDuration(10, TimeUnit.SECONDS)
                        .pendingSupplier(() -> getNumberOfPendingItemsToProcess())
                        .readyConsumer((time, attempt) -> fire(new DataLoadEvent()))
                    .build(); 
               //@formatter:on
    
  • PhantomSQL 2.0

    Since the beginning of this project, I have learned many lessons.

    First of all the syntax of the language was to closely related to SQL, as this was the language that I wanted to mimic and fit into this framework. Main reason for that was that I wanted something familiar to the end user, I was hoping for easier adoption rate. Secondly, the lack of Web 2.0+ futures. They were not thought out in the beginning and shoehorning them in just made no sense and even more complicated the language.

    Example : Grab data from woot (v1)

    foreach @srcUrl : [http://www.woot.com/, http://home.woot.com/, http://shirt.woot.com/, http://kids.woot.com/]
    begin
    	set @doc = select 
    			xpaths("//*[@class='amount']/text()"),
    			xpaths("//h2[@class='fn']/text()"),
    			xpaths("//*[@class='lightBox' and @rel='sale']/img/@src"),			
    			xpaths("//*[@class='wootOffProgressBarValue']/@style")
    		 from @srcUrl
    		
    	set @images = @document.xpaths("//*[@class='lightBox' and @rel='sale']/@href")
    	set @first = @doc[0]
    	set @price = @first[0]	
    	set @title = @first[1]
    	set @image = @first[2]	
    	set @wootoff = @first[3].split(":")
    	set @wootoff = @wootoff[1].replace("%", "")
    
    	insert  products (@srcUrl, @title, @price, @wootoff, @image, @images)
    end
    

    Couple problems that this introduced, first of all, we were constrained by the original language structure. Let’s take simple snippet

    set @price = @first[0]

    It might not look like much but this type of declaration is very declarative and it could easily be removed from the syntax. 

    Another drawback was the lack of extensibility.  Many of the basic functions have been hardcoded into the parser, thus adding any new language extension was dependent on recreating the AST. Only later and ‘dynamic’ methods have been added but again they were an afterthought and seem more like a hack.

    There was no way to reuse existing libraries, basically, this means I had to reimplement all the features that someone already did. This only became apparent to me when I needed to run a md5 checksum on one of the downloaded files. 

    Another issue for me was that there was no good support for collections (List, Set, Dictionary). There was the basic support for ‘list’ and then later ‘dictionary ‘ but again it was not thought out and missed many features.

    As we can see there were quite a few design issues that were not thought out in the initial version.  Many of this thing have been learned though running the system on real word problems. As long as we are able to take this and adapt I think we will be in good shape.

    Roadmap

    • Reworking language syntax
      • Based on modern languages / frameworks (EcmaScript / Python / Rubby / SQL/ Node.js)
      • Web 2.0 (Support for extracting data from dynamic websites)
    • Distributed Processing(Will run on a distributed clustering framework)
    • Package Manager (PhantomSQL Package Manager(PPM))

     

  • Generate ID from UUID

    This is a method to generate a long id in the positive space.

    There are few issues to consider with this method
    – UUID is 16 bytes / 128 bits
    – Long is 8 bytes / 64 bits

    This means that we will loose some information, if we don’t want to lose that we could use a BigInteger but In this case we are dealing with longs.

    
        /**
         * Gnereate unique ID from UUID in positive space
         * @return long value representing UUID
         */
        private Long generateUniqueId()
        {
            long val = -1;
            do
            {
                final UUID uid = UUID.randomUUID();
                final ByteBuffer buffer = ByteBuffer.wrap(new byte[16]);
                buffer.putLong(uid.getLeastSignificantBits());
                buffer.putLong(uid.getMostSignificantBits());
                final BigInteger bi = new BigInteger(buffer.array());
                val = bi.longValue();
            } while (val < 0);
            return val;
        }
    

    This works simply by creating new BigInteger from parts of UUID object, and then getting the longValue. We also make sure that the ID is in positive space, if its not we simply repeat the process. During testing most cases completed in one iteration but it did encounter few runs that reached four iterations.

  • Starting jetty via command line an nohup

    Somehow I am getting problems starting Jetty via

    service jetty start
    

    We will be using unix command called nohup
    “Nohup is a unix command, used to start another program, in such a way that it does not terminate when the parent process is terminated.”

    I have opted out for using this

    nohup java -jar start.jar -Djetty.port=8085
    

    while this works it shown an message

    nohup: ignoring input and appending output to `nohup.out'
    

    to fix that up we need to redirect in put and output to /dev/null

     nohup java -jar start.jar -Djetty.port=8085  /dev/null &
    
  • Taking heap dump of java process on linux and windows

    Taking a heap dump from console when Java VisualVM and JMX is not available to us.
    We will use following tools

      • jmap
      • jps
      • ps

    Dumping heap requires two steps
    1) Obtaining target process id
    2) Dumping heap for given pid

    First we need to obtain the target process id we would like to dump, here I will show couple ways I like to use.

    ps aux | grep 'java'
    -----
    userx     29901  6.7 47.0 25418812 3848276 ?    Sl   Mar23  85:42 /opt/java/bin/java -Djava.util.logging.config.
    

    Here second column indicates our process id (pid)

    Second method that is quite useful to obtain pid for java processes

    uxserx@WS4:/opt/java/bin# ./jps -l
    4281 sun.tools.jps.Jps
    29901 org.apache.catalina.startup.Bootstrap
    

    As we see both methods returned us pid of 29901
    Npw to perform the dump we issue our second command

    userx@WS4:/opt/java/bin# ./jmap -dump:format=b,file=/tmp/heapdump-001.hprof 29901
    Dumping heap to /tmp/heapdump-001.hprof ...
    

    At this point we have our heap dump that is ready to be analyzed, for my analysis I use two tools. Eclipse Memory Analyzer (MAT) and Java Visual VM

  • Calculate centroid of 2D non crossing polygon

    Calculate centroid of 2D non crossing polygon,
    To accommodate that points are correct using Gift wrapping algorithm(Finding Convex Hull)

    Test case

    import static org.junit.Assert.assertEquals;
    import static org.junit.Assert.assertNotNull;
    
    import java.awt.Point;
    import java.util.ArrayList;
    import java.util.Arrays;
    import java.util.List;
    
    import org.junit.Test;
    
    public class MathUtilTest
    {
    
        @Test
        public void computeCentroidWithHull()
        {
            Point p1 = new Point(1, 1);
            Point p2 = new Point(2, 2);
            Point p3 = new Point(3, 1);
            Point p4 = new Point(1, 0);
            Point p5 = new Point(0, 1);
            Point p6 = new Point(5, 5);
    
            Point centroid2d = MathUtil.centroid2D(Arrays.asList(p1, p2, p3, p4, p5, p6));
            assertEquals(new Point(2, 1), centroid2d);
        }
    
        @Test
        public void computeCentroid()
        {
            Point p1 = new Point(1, 1);
            Point p2 = new Point(2, 2);
            Point p3 = new Point(3, 1);
    
            Point centroid2d = MathUtil.centroid2D(Arrays.asList(p1, p2, p3));
            assertEquals(new Point2D(2, 1), centroid2d);
        }
    }
    

    Implementation

        /**
         * Calculate centroid of 2D non crossing polygon, To accommodate that points
         * are correct using Gift wrapping algorithm(Finding Convex Hull)
         * 
         * @ref http://en.wikipedia.org/wiki/Centroid#Centroid_of_polygon
         * @param vertices
         * @return
         */
        public static Point centroid2D(final List vertices)
        {
            if (vertices == null)
                return new Point(0, 0);
    
            List hull = null;
            if (vertices.size() < 2)
                hull = new ArrayList(vertices);
            else
                hull = findConvexHull(vertices);
    
            // Now we can calculate the centroid of polygon using standard mean
            final int len = hull.size();
            final double xy[] = new double[] { 0, 0 };
            for (int i = 0; i < len; ++i)
            {
                final Point p = hull.get(i);
                xy[0] += p.getX();
                xy[1] += p.getY();
            }
    
            final int x = (int) (xy[0] / len);
            final int y = (int) (xy[1] / len);
    
            return new Point(x, y);
        }
    
  • Find Convex hull of given points using Gift wrapping algorithm

    Find Convex hull of given points using Gift wrapping algorithm

    This is implementation of Grift wrapping algorithm for finding convex hull.

    
      private static final Integer ZERO = new Integer(0);
    
    
    /**
         * Find Convex hull of given points
         * 
         * @ref http://en.wikipedia.org/wiki/Gift_wrapping_algorithm
         * @param vertices
         * @return
         */
        private static List findConvexHull(final List vertices)
        {
            if (vertices == null)
                return Collections.emptyList();
    
            if (vertices.size() < 3)
                return vertices;
    
            final List points = new ArrayList(vertices);
            final List hull = new ArrayList();
            Point pointOnHull = getExtremePoint(points, true);
            Point endpoint = null;
            do
            {
                hull.add(pointOnHull);
                endpoint = points.get(0);
    
                for (final Point r : points)
                {
                    // Distance is used to find the outermost point -
                    final int turn = findTurn(pointOnHull, endpoint, r);
                    if (endpoint.equals(pointOnHull) || turn == -1 || turn == 0
                        && dist(pointOnHull, r) > dist(endpoint, pointOnHull))
                    {
                        endpoint = r;
                    }
                }
                pointOnHull = endpoint;
            } while (!endpoint.equals(hull.get(0))); // we are back at the start
    
            return hull;
        }
    
    
        private static double dist(final Point p, final Point q)
        {
            final double dx = (q.x - p.x);
            final double dy = (q.y - p.y);
            return dx * dx + dy * dy;
        }
    
    
       /**
         * Returns -1, 0, 1 if p,q,r forms a right, straight, or left turn. 
         * 1 = left, -1 = right, 0 = none
         * 
         * @ref http://www-ma2.upc.es/geoc/mat1q1112/OrientationTests.pdf
         * @param p
         * @param q
         * @param r
         * @return 1 = left, -1 = right, 0 = none
         */
        private static int findTurn(final Point p, final Point q, final Point r)
        {
            final int x1 = (q.x - p.x) * (r.y - p.y);
            final int x2 = (r.x - p.x) * (q.y - p.y);
            final int anotherInteger = x1 - x2;
            return ZERO.compareTo(anotherInteger);
        }