Perl6 CRO and Data Warehouse streaming services

What do you do when you do not have anything to do on a Sunday afternoon?

Why not implement Streaming services for a Data Warehouse? The company’s tool fleet can send performance stats over the net. I get questions about and  requests for Internet of Things, if not daily at least on a monthly basis. Often from apprehensive or perplexed persons asking for bongoDB or Big Data Hadraloop solutions. For some reason they all seem to think they need some new more or less esoteric software to gather and store data from net-connected gadgets. Sunday after lunch I stumbled on this presentation. I liked the idea about building a streaming service of stacked micro services so I had a go with Perl6 Cro. I sat up a streaming HTTP server using Cro, this server receives GET and POST calls and write them to disk as text files. Then there is another service ( a PHP program) picking up these textfiles and inserts them in a MySQL database.

The Data Warehouse streaming services

This is not by far an optimal performant or final solution, it is just a PoC, still my first test indicated it can easily handle 10 transactions a second (nothing in the chain seemed saturated), and it is a persistent solution, all incoming transactions are written to disk and not erased until they are safely stored in the database.

Not bad for a Sunday afternoon. I had some help from Andreas with POST messages and Postman for testing.

This solution is not without problems, CRO is in Beta stage, and I have problems to take care of POST and GET messages at the same time. I do not know Perl6, so I do not know if my Perl6 knowledge is too low, the CRO documentation is not correct or there are bugs in CRO first Beta version. But I’m confident my issues will be ironed out in near future. There are other more mature frameworks, but CRO looks both cool and promising, so I’ll try to stick to it.

I have said many times SQL databases can be used for so called Big Data applications, and I always eat my own dog food. If for some reason it shows SQL databases is not up to the task I switch to something else. The same goes for CRO, but it’s not fair to ditch a Beta software just because you do not understand it, and so far it looks great. (Andreas thinks I should go for a mature production ready alternative though.)