feature request: logstash friendly S3 log files

[sarkortrantor]

Hello guys,

I'm trying to read the S3 files written by the AWS Serverless Application from logstash, and I'm facing issues spanning from the fact that the files

• are containing a JSON list as the root object
• the JSON events are spanning multiple lines (indent=2)

So in order to read them from logstash I would have to do one of :

• create a custom logstash input/codec or retrieve and preprocess the files before logstash using some other tool / jq etc.
• try to leverage the multiline codec to split on some well chosen separator pattern and postprocess to reconstruct valid JSON before decode
• try to leverage the multiline codec to read the entire files in single event and decode then split later in the pipeline

This is definitely possible but maybe not the best user experience.

Would you consider providing a second serverless app that emmits line delimited JSON (logstash friendly) in the files written in S3 ?

(so flat, no JSON list, and one event per line)

Best regards,

[kmaranionjc]

Hello, you can edit the code and make your desired changes to the json format then manually publish the serverless app to your AWS org instead of going through the AWS Serverless App Repo.

https://github.com/TheJumpCloud/JumpCloud-Serverless/tree/master/AWS/DirectoryInsights#readme