If you are working in a decent sized company, then you likely have a ton of log data. And if you are the one made responsible to analyse it, then times are tough for you!
Does this sound like your regular workday?
Logs have structure, yet no structure
22.214.171.124 - - [07/Mar/2004:16:47:12 -0800] "GET /robots.txt HTTP/1.1" 200 68 126.96.36.199 - - [07/Mar/2004:16:47:46 -0800] "GET /twiki/bin/rdiff/Know/ReadmeFirst?rev1=1.5&rev2=1.4 HTTP/1.1" 200 5724 188.8.131.52 - - [07/Mar/2004:16:49:04 -0800] "GET /twiki/bin/view/Main/TWikiGroups?rev=1.2 HTTP/1.1" 200 5162 184.108.40.206 - - [07/Mar/2004:16:50:54 -0800] "GET /twiki/bin/rdiff/Main/ConfigurationVariables HTTP/1.1" 200 59679
If you are responsible for analysing your companies log data, then this probably looks very familiar to you.
There are two ways to look at this data. You can either say that this log has a structure, or you can say that it does not. Most standard BI & Analytics tools cannot make any sense of this data. It just appears as textual lines, and you basically can get nothing out of loading this data into a self-service data discovery tool like Tableau, Qlik or Power BI.
The right tool is all you need
The log data actually does have structure. The problem is that common BI tools are not comfortable with its structure. If you had a tool that could understand this structure, things would work just fine.
Do all logs have the same structure? Well No! Then?
Logs from different softwares or hardware come with different structures. What you really need is a tool where you can input the structure of your log, and make the tool interpret your custom logging format.
Splunk: These guys are literally market leaders. If you can afford their services, Splunk would be the way to go for even the most pesky log management and log analytics requirements.
Logstash: A popular open source tool for log management. You can store log events and retrieve them for future analysis. It uses Elasticsearch and Kibana underneath.
Loggy: Cloud based tool for log analytics. Very friendly for log management in DevOps, SysOps and several engineering requirements.
BlobCity: Offers cloud and on-premise solution for integrated analytics of log and structured data. If you need to analyse your log data along side your structured relational or NoSQL data, then BlobCity is a good choice.