From time to time Continua gets to this strange state that server sees agents as offline, although they are operational, pingable from server/agent and Continua service is running on agent.
Sometimes it’s all agents, sometimes part of agents. Restarting agents does not help, only reboot of server helps.
I suspect the agent registration issues are a side effect of a database connectivity issue.
Looking at the logs, it looks like you are using Postgresql, is that the bundled version or your own postgresql install?
What is the load on the server, cpu usage, memory usage, i/o etc.
You say only rebooting the server works, did you try restarting the ContinuaCIPostgreSQL and ContinuaCI server service and the ContinuaCIPostgreSQL (in that order)?
I used the bundled version of Postgres.
Usually load is not high, but the host resides on backup heavy virtualization and I suspect those problems occur in such moments, but I am not able to verify when each backup runs.
I usually just restart the whole machine as it’s faster for me, but next time I can try to restart only the services.
Yeah virtualization backups can cause problems with i/o - I don’t really have an answer for that. We run our vms on proxmox servers, and backups do not cause us these issues.
We’ll look into this, but I doubt there is going to be a quick fix for it.
