[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: openshift mcollective agent log rotation



+++ Aaron Knister [17/12/13 12:45 -0500]:
Thanks Brenton,

I implemented that feature (
https://github.com/aaronknister/origin-server/commit/227e592f433de716d7631ecd7061f138feed8e4f).
I think it should go hand in hand with the other patch (
https://github.com/aaronknister/origin-server/commit/c409a272fa67643367ba0b5d3b0e3d5d64d597aa)
that closes log devices when reinitialize is called. I was concerned about
race conditions during log rotation arising from both the node agent and
commands like oo-admin-ctl-gears that open logs but I don't think it should
be an issue at least not with rotatelogs. rotatelogs doesn't move old files
out of the way, it just creates new ones.

Is there interest in accepting these patches? Any chance they could be
applied to the origin 3 branch?

Mind submitting a pull request for these in github?  That will be the
best way for the code review to begin.


Thanks,
Aaron


On Mon, Dec 16, 2013 at 11:55 PM, Brenton Leanhardt <bleanhar redhat com>wrote:

+++ Aaron Knister [16/12/13 22:57 -0500]:

 Hi Everyone,

I'm noticing that the openshift node platform and platform-trace logs can
grow fairly large, at least with Origin 2. I haven't tried 3 yet. The
split
trace logger instantiates its Logger instance with args that enable log
rotation, however because the logging device is passed in instead of
having
the logger instance open the file itself, log rotation never happens. How
are others handling platform log rotation?

There are a couple approaches I've come up with:

1. I can use logrotate with a postrotate action that restarts mcollective
however I'm concerned that this could cause gear operations to fail. It
seemed to do so in testing.

2. I also tried modifying the split trace logger and the openshift agent
such that a re-initialize of the mcollective agents would trigger a close
and re-open the log file. This seemed much more resilient in testing,
however it did require code modification (
https://github.com/aaronknister/origin-server/commit/
c409a272fa67643367ba0b5d3b0e3d5d64d597aa).
That and there's always the risk that arises of log file corruption when
gziping with logrotate because of the asynchronous nature of signals.

3. The solution I'm working with now is to make the platform and platform
trace logs fifo's and send the logs from a fifo through apache's
rotatelogs
command. This doesn't require any code changes.


+1.  I think we could support a syntax similar to Apache in that if
you specify a file name we would use it, but if the value starts with
"|" we would treat it as a fifo.



4. It would be nice if I could specify a command to run for the platform
and platform trace logs, similar to what one can do with apache and the
rotatelogs command. This would require code modification of course.

Is any one of these approaches preferable?

Thanks,
Aaron


 _______________________________________________
dev mailing list
dev lists openshift redhat com
http://lists.openshift.redhat.com/openshiftmm/listinfo/dev





[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]