The agony and the ecstasy of the read-only
So earlier today I counselled to run your container filesystem read-only.
- Its higher security (something can't weasel in as easily)
- You want to be able to dynamically dispose and restart containers somewhere else, how can you do this if they are stateful
- The overlay fileystem is not hgh performance
Now, this last one. Lets say you have a container, oh, say, elasticsearch for the sake of argument. Its already a pain to schedule under Kubernetes since it is Java (it wants all the memory and then allocate it locally, fighting the scheduler), and its stateful (needing a StatefulSet). But you might not also realise that it logs locally to it (naughty naughty). And you only find this when you make it read-only. You get this lovely error:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m' Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. [0.000s][error][logging] Error opening log file 'logs/gc.log': Read-only file system Initialization of output 'file=logs/gc.log' using options 'filecount=32,filesize=64m' failed.
But, now you know. And the solution is simple. Add an 'emptyDir' with 'medium = Memory' and mount it on /usr/share/elasticsearch/logs (and mount one on /tmp while you are at it). Presto.