So you have a K8S cluster. Its got a lovely Ingress controller courtesy of helm install stable/nginx-ingress. You’ve spent the last hours getting fluent-bit + elastic + kibana going (the EFK stack). Now you are confident, you slide the user-story to completed and tell all and sundry “well at least when you’re crappy code gets hacked, my logging will let us audit who did it”.
Shortly afterwards l33t hackerz come in and steal all your infos. And your logs are empty. What happened? As you sit on the unemployment line pondering this, it hits you. Your regex. You parsed the nginx ingress controller logs with this beauty:
^(?<host>[^ ]*) - \[(?<real_ip>)[^ ]*\] - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<request_length>[^ ]*) (?<request_time>[^ ]*) \[(?<proxy_upstream_name>[^ ]*)\] (?<upstream_addr>[^ ]*) (?<upstream_response_length>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_status>[^ ]*) (?<last>[^$]*)
And why not? The format is documented. But, you and little bobby tables both forgot the same thing. Your hackers were smart, they put a ” in the name of the user-agent.
So nginx dutifully logged “hacker”agent-name”, and, your regex didn’t hit that of course, so no message was logged.
Red team only needs to get it right once. Blue team needs to be ever vigilant.
Leave a Reply