domingo, 23 de marzo de 2014

Date Range Query Medicine for CQ AEM

One of the most annoying features of xpath queries is the performance problem of date range queries. QueryBuilder tends to perform badly. You can see Marcel's presentation in AdaptTo if you want to know the details. Thus, when faced with this problem, there are several alternatives:
  • modifying the granularity of the dates,so that you select up to the day and exclude hours/min/secs
  • using a node structure in which the date is saved as the path; for example,
    • yyyy/mm/dd/my_content_node
  • or converting the date property into a long 
I am going to focus on the third one: ways to convert a date property into a long property.

Approach
The basic idea is to leverage the JCR Observation offered in AEM to add a property of type long each time a property of type date is created or modified.

First I created my own namespace (thanks this notes) by just going to

http://<host>:<port>/crx/explorer/nodetypes/index.jsp

I selected my own namespace netval so that there will be no conflicts with existing properties. 

Then I registered my class DatePropertyEventListener that implements the EventListener interface. The class needs to use the @Component annotation. You have to register for the event listener in the @Activate method. Make sure you get the ComponentContext in the activate method in order to register for the event. 

In my onEvent I used the EventIterator to get the property. I then filtered those properties that are of type date. Conveniently the Property class can return almost any type of value being String, long, Calendar, etc. In my case I needed the value as a long. 

Once I have the value, I can set it in the new property that has netval as the namespace and the name of the property in order to avoid collisions. For example, if the attribute name was activationDate the new attribute would be:

netval:activationDate

Code
The first thing is to register and de-register the event listener in the @Activate and @Deactivate methods. 

A quick look at my DatePropertyEventListener class reveals the following. The observationManager and admin session are class attributes. They will be used in the @Deactivate  nd Activate methods. It is in the @Activate method where we register the event listener. 

observationManager = adminSession.getWorkspace().getObservationManager();
observationManager.addEventListener(this,
Event.PROPERTY_ADDED|Event.PROPERTY_CHANGED,
                "/content/news/", // absolute path
                true, // isDeep
                null, // uuid
                null, // nodeTypeName
                true // no local
        );

Note that the absolute path is restrictive to the root path of your content. I highly recommend being as specific as possible in order not to receive too much noise.

The onEvent it is easy to get the date property

        while( eventIterator.hasNext() ){
            final Event event = eventIterator.nextEvent();
            if(PropertyType.DATE == getPropertyType(event) ){

Caveats and Warnings
This procedure should be disabled for operations that modify many nodes, like during a migration of content.

A suggestion is to have a producer-consumer data structure (like LinkedBlockedQueue) that will allow you to process the events more efficiently.

lunes, 18 de marzo de 2013

Heavy duty ETL: Extraction Transformation and Load

This is a cross post blog from Cloud Data Viz project

These days, many people are talking about Big Data. However, very few talk about how big is Big Data and about all the different components that need to be considered prior, during, and after running a system based on Big Data. Many don't know the time it takes to extract the data. Others forget that they need to validate it. And only a few mention the loading tools they use to speed up the process of loading millions of rows into a system (not to mention normalizing the data while you are loading it). Therefore, I thought our "little" project could be of interest. This post will explain all the different issues you need to consider before implementing a system with Big Data.

Data Volumes
The amount of data that we use for the Climate Viz project is astronomical. Just to give an idea, we are collecting temperature data from satellites at a 0.25 resolution, which means that there are 864,000 data points being collected every 3 hours. At the end of just one day, we get close to 7 million data points. Just for one day! And we haven't mentioned transformation and other data aggregation statistics that we need for the project.

Extraction
Every project always comes with dirty work. And extracting and loading the files is the dirty part of this project. Visualizing the data in maps and writing the UI controls is the easy part. But getting all the data in and validating it, that's where the pain begins.

We use a two-step process for extracting the files from NASA Giovanni. Setting up a wget script that downloads the GRIB files was easy. Then we process them with pygrib and slice them in order to have them ready for GAE.

For data is never going to change, we generate master tables. In our case, latitude and longitude are fixed for every resolution and can be generated programatically. Two for loops ( x and  y ) and you think you are done but, when you are dealing with Big Data, that translates into several troubles. First consider that the loops could take so long that you could reach timeouts as part of the request and post methods of the web layer. Then, the same thing could happen even if you make a dummy get method to generate all the different tasks using the queue.

Validation
Once you start dealing with data collected from different sources, the first need is to validate them. Most likely you would like to have a visualization tool that compares and contrasts your data. Unfortunately, you can't use excel because it has a limit of 65,535 rows (far too limited for big data sources).

Thus, we are left with building our own tools to validate our data. Think about it because it makes a big difference. Google Maps was a useful interface for us. Also, Google drive is another option for loading big files.

Transfer 
Of course, transferring data from your source to the destination is another issue. If you have a normal ETL tool, then your problem is solved. However, the rest of us have to deal with several issues.  For each POST, there is a limit in both the size of the file we are sending as well as the time it takes to process it. I had to play around with how fast GAE was processing the files. I started with 15000 lines per file and had to go down to 200 lines. Otherwise, I received a timeout error in the operation for 5K, 2K, and 800. Luckily, I was generating files with a test script but at times is not easy to generate data.

I also generated a script to scan the files and post them automatically (using multi-part). Even with a queue of 40 tasks a second, the time to process the files is agonisingly slow (5 seconds per 100 lines) and it could probably take several months to load. :(

Another trick that I used was to slice the files to upload.  Each file contained a line order that will help me aggregate the data later.

Conclusion
I hope the ideas exposed in this post will help you in your Big Data projects.

miércoles, 31 de octubre de 2012

Delegando Tareas con App Engine Task Queues

Contenido de la presentación para el Google Dev Fest en Barcelona (9/11/2012)

Ver otras ponencias

Título:
Dejando tareas a Google App Engine: Introducción a Task Queues

Presentacion: https://www.box.com/files/0/item/f_3909761486

Contenido de la Presentación:

Introducción
  • Concepto General: herramienta para los procesos "background"
    • Creado por el limite de 30 segundos por peticion a 10 minutos
  • Visión general del API: los dos tipos de queues: push vs pull
  •  
  • Casos de Uso
    • Migración de Datos
    • Procesos Batch
    • Sincronización Avanzada
    • Integración con crons
Uso / Desarrollo / Cómo Dividir Tareas
  • Conceptos generales para dividir y conquistar (Divide and conquer)
  • Framework de Google y su Parametrización de las tareas
  • Cómo llamar a los "workers" y consideraciones especiales
    • idempotency: qué es
    • consideración para el entorno "cloud"
Configuración y Control
  • Técnicas de afinamiento
  • Repasando los atributos de una queue
    • frecuency/rate
    • bucket size
    • etc
  • Tu amigo el "control panel"
  • Versionado
Mejores Prácticas
  • Cómo diseñar para parar en "masa"
  • Control de tareas
Otros Frameworks y Herramientas
  • Map Reduce
  • appengine-pipeline

sábado, 18 de agosto de 2012

Analyzing Web Performance: love my RUM

It was news to me the other day while watching my favorite tv channel (Google Developers on YouTube) that I ran into this presentation about web performance. Previously I had no idea that modern browsers could measure metrics like "dns lookup", "server connection time", "server response time". With the awesome work from the W3C Performance Working Group, now this is a reality.

Added to the new and latests trends in continuous build cycle and automated testing, now there is the fact that you can get your RUM (Real User Metrics) as part of Google Analytics. There is always the need to do your performance testing but you can also see how the site evolves and how you could even monitor in real time what is going on. So I went I checked the Google Analytics and indeed, they are providing with all this data.

I remember a long while ago when a client had some problems with performance and we were working in the dark until we found out it was the dns lookup time! Imagine how much easy it would have been had we used this data.

For the curious I am posting the image of the lifecycle for a request. All this data could be obtained from the "Developers tool" from most modern browsers. Safari is still lagging.


domingo, 22 de julio de 2012

Google App Engine Research Award

I am happy to inform that the project that I am collaborating with Dr. Enrique Vivoni and Giuseppe Mascaro was selected as a finalist for the Google App Engine Research Award.

Objective
The project proposes new ways to visualize and accesss data set for global climate data. Our first step is providing access to land surface temperature.

Background Information
Progress has been made through ground and space-based observing networks and the development of sophisticated numerical models. A class of these models, known as Land Surface Models or LSMs, simulates terrestrial water and energy processes and their two-way interaction with the Earth's atmosphere.

We plan to base our analysis on the Global Land Data Assimilation System (GLDAS), jointly developed by NASA and NOAA, and uses a series of four LSMs to provide global, 0.25 and 1 degree, 3-hourly fields of a wide range of geophysical variables over the period of 1979 to present.

Procedures for downloading, visualizing and interpreting the data requires advanced technical skills typically possessed by scientist involved in geophysical disciplines. We try to make it easy for normal users.

Execution
Our initial step is to focus only on land surface temperature (LST) at 3-hourly resolution from the period 1980 to 2010 as a target variable. We expect the query tools to allow a range of inquiry into the land surface temperature distribution at:
  1. Specific location
  2. Across different time periods
  3. For memorable events (for example, Hurricane Katrina)
Stay on for future post on how we are going to develop a set of data visualization tools.

viernes, 6 de julio de 2012

How-to: Preparando Presentaciones para GTUG Barcelona

Si vas a presentar en el GTUG, vale la pena repasar los siguientes puntos que te ayudaran a prepararte para la presentación.

Q&A: Test Run!
Igual que en software, vale la pena realizar un "acceptance test" o quizás una prueba unitaria acerca del contenido de tu presentación. Es mi experiencia que siempre tiendo a mezclar más temas de los que debería, así que mantenerlo simple es a veces mas claro a la audiencia. Si alguien te puede escuchar o si puedes contrastar la tabla de contenido con algún colega, te dará una buena perspectiva de la relevancia.

Resolución 800 x 600
Recordar que cuando se presenta la resolución en pantalla es menor que la que usualmente tenemos en el escritorio. Vale la pena practicar tener eclipse en una resolución de 800x600 para tener una idea de como agrupar los elementos en pantalla.

Doble Monitor
Para los que usamos dos monitores en la oficina y usamos Windows, recordar que por defecto, se tiene la proyección de Microsoft Powerpoint en el monitor principal. Esto provoca que no se muestre en pantalla al momento de proyectar. Antes de ir a la presentación, habría que cambiar la configuración y practicar que cuando ejecutamos la presentación en Powerpoint, se proyecta en el monitor secundario.

Visualización de Código Fuente
Intentar cambiar de pantalla con lentitud para no marear al publico. Aunque normalmente cambiamos de pantalla con facilidad, cuando proyectamos es mejor hacerlo con mas lentitud y lo menos posible. De esta manera guardamos el enfoque.

Basado en la experiencia de expertos (como la peña de Google I/O) se debería limitar al máximo el código que se muestra en Eclipse y decantarse por usar más las diapos para mostrar solo extractos de código. Si estás pensando en mostrar una clase entera, vamos mal. Debería ser mucho más concreto y enfocarte a nivel de funciones y bloques en particular. 

Interactividad
No olvidar hacer preguntas al publico. Puedes pedir sugerencias, lecciones aprendidas, pasadas experiencias o simplemente si tienen algo que compartir -- bueno o malo.

Audiencia
Recordar mantener el enfoque en la audiencia y no en el monitor local. Aunque es difícil, tratar siempre de ver al publico para ver si todavía están ahí :)

Código Fuente
Falta hace que el código presentado este probado y que compile. En caso de algún error, indicarlo como "known issues". Es una buena práctica colgarlo en algún sitio. (Ver Google code o github ).

Get Social
Recordar tener abierto tu cuenta de Twitter para poder recibir preguntas y aceptar nuevas amistades.

Update
Recordar que la audiencia de GTUG es una de las más motivadas por las nuevas tecnologías, por lo que si vas a presentar acerca de algún producto, conocer las ultimas versiones, etc.

Cualquier otro punto que se les ocurra, es bienvenida.

sábado, 19 de mayo de 2012

Presentación en Barcelona GTUG

Paso a detallar los puntos que sugiero presentar para el GTUG de Barcelona en la próxima reunion. Cualquier sugerencia es bienvenida, así que si tienen alguna, solo basta que me envíen un email o que comenten en el blog.

Título: Implementando RESTful en GAE
Cuando: Miércoles 20 de Junio 2012
Hora: 18:15h
Lugar:
FIB - UPC Campus Nord - Sala de actos B6
Calle Jordi Girona, 31, 08034 Barcelona, Spain
08034

Temario

  • Repaso a configurar un proyecto Java en GAE
  • Uso de persistencia con JDO
  • Gestion de las instancias y servicios CRON
  • Configuración del plugin de maven para GAE y JAVA
  • Libreria RESTlet para Android y GAE
  • Ejemplos de implementacion en un sitio web real
  • Discutir patrones de interacción/sincronización con servicios RESTful
  • Manejo de Errores en la capa de comunicación 
  • Uso de threads para mejor respuesta
  • Implementación RESTful en Android tomando en cuenta el ciclo de vida de una actividad