Today we had an issue with a Linux server that filled (100%) it's entire root (/) partition due to a misconfiguration in postfix that led to a huge /var/log/syslog file and yesterday we started using AWS' EFS (NFS service) for other use case.
Based on these two events we had an internal talk about the possible workarounds for avoiding server/disk outages due to log files going crazy and, it seems to us like using a single NFS mount for all servers' log files might be a good idea as AWS' EFS service provides a virtually limitless (8 Exabytes reported by df) and consolidating all logs in a single drive might even ease debugging failures.
Given the above facts and idea, the question is quite obvious: is this suggested approach, of using NFS mount for all Linux logs, a good one? Pros / Cons?
I'm aware that this might be an oppinionated question, but that's not the feedback I need, but actual possible drawbacks that we might encounter based on real life facts / issues / measurements.
1 Answer
1) It of course depends on the amount of logging, but network logging is slow, and this can drastically slow down parts of your system, compared to logging on local disk. NFS can also use significant CPU. I have seen people chasing performance problems for weeks on a server until they figured out that it came from logs moved to a network FS due to lack of disk space (and this was a NFS in the same machine room).
2) Better to 1) move the logs to their own partition, 2) have a rather aggressive rolling policy, and 3) then perhaps move the compressed logs to AWS. Logs can reveal a lot if compromised, so you better encrypt them before exporting them or have very tight security around your AWS storage.
4
