The agony and the ecstasy of the read-only

So earlier today I counselled to run your container filesystem read-only.

• Its higher security (something can’t weasel in as easily)
• You want to be able to dynamically dispose and restart containers somewhere else, how can you do this if they are stateful
• The overlay fileystem is not hgh performance

Now, this last one. Lets say you have a container, oh, say, elasticsearch for the sake of argument.  Its already a pain to schedule under Kubernetes since it is Java (it wants all the memory and then allocate it locally, fighting the scheduler), and its stateful (needing a StatefulSet). But you might not also realise that it logs locally to it (naughty naughty). And you only find this when you make it read-only. You get this lovely error:

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[0.000s][error][logging] Error opening log file 'logs/gc.log': Read-only file system
Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.

But, now you know. And the solution is simple. Add an ’emptyDir’ with ‘medium = Memory’ and mount it on /usr/share/elasticsearch/logs (and mount one on /tmp while you are at it). Presto.