Cleanup continua CI server workspace?

(Brenden Walker) #1

We’ve got over 600gigs of space used in Continua\WS folders, which I believe is the server workspace?

Our cleanup policy is pretty tight with a 2 week build age, 1 min build and 2 max builds. All options are checked. There are folders in continua\ws that are months old and as far as I can tell should have been cleaned up.

It’s entirely possible that we’re doing something wrong… I’ve asked developers to stop builds that are pending promotion when no longer needed… as I think that would keep stuff around.

I’ve reviewed projects/configs and the ‘recently completed builds’ list seems to reflect cleanup policy.

What situation(s) can keep server workspace from being cleaned up? Any handy queries I could do on the DB to help identify the problem?

Thanks…

(Dave Sparks) #2

Hi Brendan,

What version of ContinuaCI are you using? Earlier versions did have some timeout issues with build cleanup which have been fixed in later versions. Check if there are any errors on the Event Log page in the Administration section.

Note that pinned builds and builds which have not finished, including those pending promotion, are not cleaned up. Also note that the global cleanup policy may be overridden at the project and configuration level. Check that the relevant workspace items are ticked under What to cleanup.

If you are using PostgreSQL for your database you can run the SQL query in the attached file to get a report of the cleanup policies for each configuration. CleanedUpBuildsQuery.sql.txt (7.5 KB)

This will also show various build counts for each configuration. The last column Number Of Non Pinned Terminated Builds Over Maximum shows how many builds are left which should be cleaned up but haven’t. You would expect there to be a few builds depending on the time since the last scheduled cleanup.

Tools such as WizTree, TreeSize and WinDirStat that show the size of each folder are useful to identify which workspace folders are taking up the most space. The workspace folders are named by project_slug and build_id, so once you have identified a bloated folder you can go to http://(server_host_name)/(project_slug)/ci/builds/view/(build_id) to view the build details.

(Brenden Walker) #3

1.9.0.374. I’ve already hunted down the projects/configs that overrode cleanup and turned that off. Cleanup policy has everything checked.

I’ve already narrowed it down to a specific project. The query you provided sounds like just what I need, AFAIK we are not pinning anything however getting devs to stop un-needed builds that are pending promotion has been a challenge.

I’ll reply to this thread when I know more, thanks!

(Brenden Walker) #4

I think I found the problem. We have one project/config reporting 253 'Number of Builds Pending Promotion".

Just took a look and the history has 11 pages of builds in pending promotion. Used the ‘Stop All’ except for most recent 50 builds (I’ll clean those up manually). Kicked of a cleanup and pretty quickly recovered a bunch of free space.

Thanks again!

(Brenden Walker) #5

I noticed that the stage we stop at has promote options set to ‘Enable stage promotion timeout’ with 72 hours in the duration. Auto promote is not checked, and there are no conditions specified. I would think that should be terminating the build and not leave it in a waiting for promotion state?

(Dave Sparks) #6

Hi Brenden,

The stage promotion timeout is working in our tests. Note that the relevant stage promotion options are those on the stage before the one marked pending promotion, rather than those on the stage marked pending promotion which has not yet run.

Are there any errors related to promotion or cleanup on the Event Log page in the Administration section? Otherwise, can you generate and download a copy of the diagnostics report using the links at the top of the Event Log page. Send it to me via direct message or email to support at finalbuilder.com.