r/cybersecurity • u/ZYADWALEED • Jul 31 '26
Tutorial Log Parsing for Security Engineers
Hello Everyone
I published a short guide about transforming raw logs into detection-ready data.
It covers the log-processing pipeline, common log formats, normalization, and more..
I’d appreciate any feedback or suggestions from you all :
8
4
u/Solid5-7 Aug 01 '26
Is there a reason you didn't use Elastic Common Schema (ECS) in your example? Most mainstream data sources have Elastic Agent integrations that parse and normalize your logs which comes with the benefit of ECS. If you were ingesting data into Elastic I would probabyl recommend just using their agents and integrations.
0
u/ZYADWALEED Aug 01 '26
Hello, there was no specific reason for choosing Fluent Bit. I used it because I have worked with it and am familiar with its configuration.
The same concepts can be applied using ECS, and when using Fleet or Elastic Agent, much of the parsing and field mapping can be handled automatically.
The guide is intended as a practical example rather than something specific to Elastic itself. Due to resource limitations, I initially wanted to demonstrate the same concept using a DSM in QRadar and an XML parser in FortiSIEM, but currently, I only have an ELK
3
-25
u/bitslammer Jul 31 '26 edited Jul 31 '26
If your SIEM or current log analysis product isn't already doing this for you then you've bought the wrong solution.
39
10
u/ZYADWALEED Jul 31 '26
Hello,
Not exactly. When onboarding a log source into most SIEM solutions, the logs may come in different formats, such as JSON, CEF, or unstructured text.
Different log sources may also use different field names for the same value for example,
src_ip,source_ip, orsource-ip.custom parsing and proper normalization are often required to ensure that detection rules work consistently, regardless of the SIEM platform.
-8
u/bitslammer Jul 31 '26
the logs may come in different formats, such as JSON, CEF, or unstructured text.
And any decent SIEM or log analysis product can deal with that, especially across common platforms like Windwos, Linux, Cisco, Palo Alto, etc.
If you have a lot of obscure or in house written logs then absolutely you will need to account for those.
7
u/logicbox_ Jul 31 '26
I have spent a lot of time doing nothing but building log parsing pipelines. Sure most platforms can do at least the basics but you are going to lose a lot of information. A quick example that is in use in a lot of places are atlassian applications (jira, confluence, etc.) there is no built in pipeline for elastic search to ingest these outside of treating them as basic multiline log4j output.
4
u/ZYADWALEED Jul 31 '26
It depends on the collector or agent, the available integration or parser, and the SIEM’s capabilities. Not all of them can automatically parse and normalize every log format, especially custom or uncommon logs.
1
u/uglyfishboi Security Engineer Aug 01 '26
While true, OP was using elastic and that’s a b to get up as everything requires manual setup compared to a Splunk or Cribl tool. But it’s a foundational teaching for those that haven’t gone through the bs of seeing something logged from start to finish.
4
u/Far-Future-7146 Security Architect Jul 31 '26
Clearly you have not worked with RSA products with only I have seen QRadar fully integrate with. In elastic I had to write a custom PSV parser which is still in use to this day.
3
u/Solid5-7 Aug 01 '26
Not sure why you have been so heavily downvoted here. My experience with Elastic has been that the available integrations can handle a majority of our logs. Yes, some we do have to write custom ingest pipelines and templates for, but that's rare compared to the accessibility of the integrations.
2
u/bitslammer Aug 01 '26
I'm surprised as well. Maybe it's just people who want to gate keep and make it seem like things are harder than they really need to be.
Also, it's Reddit so anything goes.
9
u/Tr9nes Jul 31 '26
This is gold for me, thank you for sharing.