After unathourising all agents, updating Windows (2 updates) then rebooting CI server, I noticed that all of the licenses are unavailable, even the one that is free.
It looks like this:
Update.
After service restart, all is OK.
Before I rebooted the whole machine at least twice and it did not help…
Hi Michal,
We’ve not seen this before and have just tried in vain to reproduce by unauthorising, updating and rebooting our server.
Are there any relevant errors listed in your Continua event log or the Windows event log on the server?
Hello,
nothing that I could match to missing license, there were a lot of errors as I was also doing git server maintanance.
One more thing to add, that I unathourized agents before the last running build has finished (so no new one would start), maybe that’s a clue.
As to errors I see couple interesting:
An exception was encountered while executing build expiration policies: Exception: DataException Message: Exception while reading from stream Stack Trace: at Dapper.SqlMapper.ThrowDataException(Exception ex, Int32 index, IDataReader reader, Object value) in /_/Dapper/SqlMapper.cs:line 3928 at Deserializec8ebd766-a386-46ff-9f82-c42fbdb5dc2d(DbDataReader ) at Dapper.SqlMapper.<QueryImpl>d__144`1.MoveNext() in /_/Dapper/SqlMapper.cs:line 1203 at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType) in /_/Dapper/SqlMapper.cs:line 815 at Continua.Modules.Builds.Data.Dapper.ConfigurationData.GetMinimalConfigurations(Nullable`1 projectId, Nullable`1 configurationId) at Continua.Modules.Builds.BuildExpirationPolicyManager.ExecutePolicies(Boolean runNow, Boolean includeBuildsAlreadyCleanedUp, Nullable`1 projectId, Nullable`1 configurationId) Exception: NpgsqlException Message: Exception while reading from stream Stack Trace: at Npgsql.NpgsqlReadBuffer.<<Ensure>g__EnsureLong|40_0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlDataReader.<SeekToColumnSequential>d__128.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlDataReader.GetValue(Int32 ordinal) at Deserializec8ebd766-a386-46ff-9f82-c42fbdb5dc2d(DbDataReader ) Exception: TimeoutException Message: Timeout during reading attempt Stack Trace:
An error occurred while getting build statistics data for the UI. Details: Exception: NpgsqlException Message: Exception while reading from stream Stack Trace: at Npgsql.NpgsqlReadBuffer.<<Ensure>g__EnsureLong|40_0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlConnector.<<ReadMessage>g__ReadMessageLong|194_0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlDataReader.<NextResult>d__44.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlDataReader.NextResult() at Npgsql.NpgsqlCommand.<ExecuteReader>d__100.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Npgsql.NpgsqlCommand.<ExecuteReader>d__100.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior) at Dapper.SqlMapper.ExecuteReaderWithFlagsFallback(IDbCommand cmd, Boolean wasClosed, CommandBehavior behavior) in /_/Dapper/SqlMapper.cs:line 1156 at Dapper.SqlMapper.<QueryImpl>d__144`1.MoveNext() in /_/Dapper/SqlMapper.cs:line 1184 at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection) at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) at Dapper.SqlMapper.Query[T](IDbConnection cnn, String sql, Object param, IDbTransaction transaction, Boolean buffered, Nullable`1 commandTimeout, Nullable`1 commandType) in /_/Dapper/SqlMapper.cs:line 815 at Continua.Modules.Builds.Data.Dapper.BuildData.GetSummaryBuildsForConfigurationList(IEnumerable`1 configIds, Boolean includePendingPromotion) at Continua.Modules.Builds.BuildSummaryManager.GetSummaryBuildsForConfigurationList(IEnumerable`1 configIds, Boolean withStages) at Continua.Modules.Builds.BuildUI.Cache.BuildUIDataCache.GetNonCachedConfigs(Dictionary`2 configCacheItemDict, List`1 nonCachedViewableConfigs) at Continua.Modules.Builds.BuildUI.Cache.BuildUIDataCache.GetConfigs(IEnumerable`1 viewableConfigs) at Continua.Modules.Builds.Services.BuildUIService.GetBuildStatisticsData(Boolean showDisabledConfigs) Exception: TimeoutException Message: Timeout during reading attempt Stack Trace:
Error while starting build queue: Exception: AggregateException Message: One or more errors occurred. Stack Trace: Exception: GenericADOException Message: could not execute query [ select build0_.Id as id1_39_, build0_.BuildNumber as buildnumber2_39_, build0_.Version as version3_39_, build0_.Created as created4_39_, build0_.LastContinued as lastcontinued5_39_, build0_.Started as started6_39_, build0_.Finished as finished7_39_, build0_.Queued as queued8_39_, build0_.Pinned as pinned9_39_, build0_.PinComment as pincomment10_39_, build0_.Priority as priority11_39_, build0_.StartedFromBuildId as startedfrombuildid12_39_, build0_.SummaryState as summarystate13_39_, build0_.BuildSuccess as buildsuccess14_39_, build0_.OnlyNotifyTriggeringUsers as onlynotifytriggeringusers15_39_, build0_.QuietPeriodLength as quietperiodlength16_39_, build0_.QuietPeriodExpiry as quietperiodexpiry17_39_, build0_.TimeOnQueue as timeonqueue18_39_, build0_.AccumulateChanges as accumulatechanges19_39_, build0_.IsFeatureBranchBuild as isfeaturebranchbuild20_39_, build0_.UsesDefaultBranch as usesdefaultbranch21_39_, build0_.CleanedUp as cleanedup22_39_, build0_.Counted as counted23_39_, build0_.HasNewChanges as hasnewchanges24_39_, build0_.Branches as branches25_39_, build0_.State as state26_39_, build0_.status_message as status27_39_, build0_.status_datetime as status28_39_, build0_.runningactionid as runningactionid29_39_, build0_.runningactionname as runningactionname30_39_, build0_.runningactiontype as runningactiontype31_39_, build0_.runningactionmessage as runningactionmessage32_39_, build0_.ConfigurationVersionId as configurationversionid33_39_, build0_.ConfigurationId as configurationid34_39_, build0_.WorkspaceId as workspaceid35_39_, build0_.UserId as userid36_39_, build0_.StartedFromIPAddress as startedfromipaddress37_39_, build0_.PinUserId as pinuserid38_39_, build0_.TriggerId as triggerid39_39_, build0_.TriggeringRepositoryId as triggeringrepositoryid40_39_, build0_.VariableNamespaceId as variablenamespaceid41_39_, build0_.RepoBranchHashCode as repobranchhashcode42_39_ from builds_build build0_ where build0_.State in (:p0 , :p1 , :p2) ] Name:p1_0_ - Value:BuildStopping [Type: NHibernate.Type.EnumType`1[Continua.Modules.Builds.Common.BuildState] (SqlType: Int32)] Name:p1_1_ - Value:BuildCompleting [Type: NHibernate.Type.EnumType`1[Continua.Modules.Builds.Common.BuildState] (SqlType: Int32)] Name:p1_2_ - Value:BuildCancellingPromotion [Type: NHibernate.Type.EnumType`1[Continua.Modules.Builds.Common.BuildState] (SqlType: Int32)] [SQL: select build0_.Id as id1_39_, build0_.BuildNumber as buildnumber2_39_, build0_.Version as version3_39_, build0_.Created as created4_39_, build0_.LastContinued as lastcontinued5_39_, build0_.Started as started6_39_, build0_.Finished as finished7_39_, build0_.Queued as queued8_39_, build0_.Pinned as pinned9_39_, build0_.PinComment as pincomment10_39_, build0_.Priority as priority11_39_, build0_.StartedFromBuildId as startedfrombuildid12_39_, build0_.SummaryState as summarystate13_39_, build0_.BuildSuccess as buildsuccess14_39_, build0_.OnlyNotifyTriggeringUsers as onlynotifytriggeringusers15_39_, build0_.QuietPeriodLength as quietperiodlength16_39_, build0_.QuietPeriodExpiry as quietperiodexpiry17_39_, build0_.TimeOnQueue as timeonqueue18_39_, build0_.AccumulateChanges as accumulatechanges19_39_, build0_.IsFeatureBranchBuild as isfeaturebranchbuild20_39_, build0_.UsesDefaultBranch as usesdefaultbranch21_39_, build0_.CleanedUp as cleanedup22_39_, build0_.Counted as counted23_39_, build0_.HasNewChanges as hasnewchanges24_39_, build0_.Branches as branches25_39_, build0_.State as state26_39_, build0_.status_message as status27_39_, build0_.status_datetime as status28_39_, build0_.runningactionid as runningactionid29_39_, build0_.runningactionname as runningactionname30_39_, build0_.runningactiontype as runningactiontype31_39_, build0_.runningactionmessage as runningactionmessage32_39_, build0_.ConfigurationVersionId as configurationversionid33_39_, build0_.ConfigurationId as configurationid34_39_, build0_.WorkspaceId as workspaceid35_39_, build0_.UserId as userid36_39_, build0_.StartedFromIPAddress as startedfromipaddress37_39_, build0_.PinUserId as pinuserid38_39_, build0_.TriggerId as triggerid39_39_, build0_.TriggeringRepositoryId as triggeringrepositoryid40_39_, build0_.VariableNamespaceId as variablenamespaceid41_39_, build0_.RepoBranchHashCode as repobranchhashcode42_39_ from builds_build build0_ where build0_.State in (:p0 , :p1 , :p2)] Stack Trace: at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder) at NHibernate.Loader.Loader.ListUsingQueryCache(ISessionImplementor session, QueryParameters queryParameters, ISet`1 querySpaces) at NHibernate.Loader.Hql.QueryLoader.List(ISessionImplementor session, QueryParameters queryParameters) at NHibernate.Hql.Ast.ANTLR.QueryTranslatorImpl.List(ISessionImplementor session, QueryParameters queryParameters) at NHibernate.Engine.Query.HQLQueryPlan.PerformList(QueryParameters queryParameters, ISessionImplementor session, IList results) at NHibernate.Impl.SessionImpl.List(IQueryExpression queryExpression, QueryParameters queryParameters, IList results, Object filterConnection) at NHibernate.Impl.SessionImpl.List(IQueryExpression queryExpression, QueryParameters queryParameters, IList results) at NHibernate.Impl.AbstractSessionImpl.List[T](IQueryExpression query, QueryParameters parameters) at NHibernate.Impl.AbstractQueryImpl2.List[T]() at NHibernate.Linq.DefaultQueryProvider.ExecuteList[TResult](Expression expression) at NHibernate.Linq.NhQueryable`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator() at System.Linq.Buffer`1..ctor(IEnumerable`1 source) at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source) at Continua.Modules.Builds.BuildQueue.StopAnyStoppingBuilds() at Continua.Modules.Builds.BuildQueue.<OnApplicationStarting>b__28_0() at System.Threading.Tasks.Task.Execute() Exception: NpgsqlException Message: Exception while reading from stream Stack Trace: at Npgsql.NpgsqlConnector.<<ReadMessage>g__ReadMessageLong|194_0>d.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlDataReader.<NextResult>d__44.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlDataReader.NextResult() at Npgsql.NpgsqlCommand.<ExecuteReader>d__100.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Npgsql.NpgsqlCommand.<ExecuteReader>d__100.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior) at NHibernate.AdoNet.AbstractBatcher.DoExecuteReader(DbCommand cmd) at NHibernate.AdoNet.AbstractBatcher.ExecuteReader(DbCommand cmd) at NHibernate.Loader.Loader.GetResultSet(DbCommand st, QueryParameters queryParameters, ISessionImplementor session, IResultTransformer forcedResultTransformer) at NHibernate.Loader.Loader.DoQuery(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder) at NHibernate.Loader.Loader.DoQueryAndInitializeNonLazyCollections(ISessionImplementor session, QueryParameters queryParameters, Boolean returnProxies, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder) at NHibernate.Loader.Loader.DoList(ISessionImplementor session, QueryParameters queryParameters, IResultTransformer forcedResultTransformer, QueryCacheResultBuilder queryCacheResultBuilder) Exception: TimeoutException Message: Timeout during reading attempt Stack Trace:
Hi Michal,
We’ve done some more testing and have still not managed to reproduce this issue. We have however put a small check in the code for the next version to ensure that a zero license count is not cached. If you are able to reproduce this with debug logging, we’ll investigate further.
The errors that you posted are all database timeouts. Is it possible that these occurred while the server was being restarted? Maybe the database service hadn’t started up yet? If these errors are happening frequently, then we would suggest some database maintenance.