-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Add SeederApi PlayData delete scheduled job #7281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
MGibson1
wants to merge
5
commits into
main
Choose a base branch
from
arch/add-hosted-service-to-clear-old-play-data
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
ccec83e
Use Quartz-based hosted service to clear old play data
MGibson1 036268a
Trigger play data delete frequently enough for dev servers
MGibson1 bf73bfa
Fixup sonarqube
MGibson1 ee5cce8
Fixup parallel test issues with jobs hosted services
MGibson1 0c10e4a
Remove alive job and unneeded fixme
MGibson1 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| using Bit.Core; | ||
| using Bit.Core.Jobs; | ||
| using Bit.SeederApi.Commands.Interfaces; | ||
| using Bit.SeederApi.Queries.Interfaces; | ||
| using Quartz; | ||
|
|
||
| namespace Bit.SeederApi.Jobs; | ||
|
|
||
| public class DeleteOldPlayDataJob : BaseJob | ||
| { | ||
| private readonly IGetAllPlayIdsQuery _getAllPlayIdsQuery; | ||
| private readonly IDestroyBatchScenesCommand _destroyBatchScenesCommand; | ||
|
|
||
| public DeleteOldPlayDataJob( | ||
| IGetAllPlayIdsQuery getAllPlayIdsQuery, | ||
| IDestroyBatchScenesCommand destroyBatchScenesCommand, | ||
| ILogger<DeleteOldPlayDataJob> logger) | ||
| : base(logger) | ||
| { | ||
| _getAllPlayIdsQuery = getAllPlayIdsQuery; | ||
| _destroyBatchScenesCommand = destroyBatchScenesCommand; | ||
| } | ||
|
|
||
| protected async override Task ExecuteJobAsync(IJobExecutionContext context) | ||
| { | ||
| _logger.LogInformation(Constants.BypassFiltersEventId, "Execute job task: DeleteOldPlayDataJob"); | ||
| var olderThan = DateTime.UtcNow.AddDays(-1); | ||
| var playIds = _getAllPlayIdsQuery.GetAllPlayIds(olderThan); | ||
| await _destroyBatchScenesCommand.DestroyAsync(playIds); | ||
| _logger.LogInformation(Constants.BypassFiltersEventId, "Finished job task: DeleteOldPlayDataJob. Deleted {PlayIdCount} root items", playIds.Count); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| using Bit.Core.Jobs; | ||
| using Bit.Core.Settings; | ||
| using Quartz; | ||
|
|
||
| namespace Bit.SeederApi.Jobs; | ||
|
|
||
| public class JobsHostedService : BaseJobsHostedService | ||
| { | ||
| public JobsHostedService( | ||
| GlobalSettings globalSettings, | ||
| IServiceProvider serviceProvider, | ||
| ILogger<JobsHostedService> logger, | ||
| ILogger<JobListener> listenerLogger) | ||
| : base(globalSettings, serviceProvider, logger, listenerLogger) { } | ||
|
|
||
| public override async Task StartAsync(CancellationToken cancellationToken) | ||
| { | ||
| var everyFifteenMinutesTrigger = TriggerBuilder.Create() | ||
| .WithIdentity("everyFifteenMinutesTrigger") | ||
| .StartNow() | ||
| .WithCronSchedule("0 */15 * ? * *") | ||
| .Build(); | ||
|
|
||
|
|
||
| var jobs = new List<Tuple<Type, ITrigger>> | ||
| { | ||
| new Tuple<Type, ITrigger>(typeof(DeleteOldPlayDataJob), everyFifteenMinutesTrigger), | ||
| }; | ||
|
|
||
| Jobs = jobs; | ||
| await base.StartAsync(cancellationToken); | ||
| } | ||
|
|
||
| public static void AddJobsServices(IServiceCollection services) | ||
| { | ||
| services.AddTransient<DeleteOldPlayDataJob>(); | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤔 Deleting data with a hosted service always give me a little heartburn. Have we implemented opt-in techniques using environment settings in other places, by chance? I have done that in the past where my teams implemented background services that mutated data and gave us a better feel of control (especially when developing) knowing that one had to explicitly set the setting to ON to mutate/delete data?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. Are you concerned with unintentional data loss? To some extent the goal of this job is to enforce that play data is ephemeral.
That can be a server setting if we want, I suppose, but maybe a better tweak would be a lifetime?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the ephemeral environments, I am not too concerned because I would not expect an ephemeral environment to live long enough to see the job run. Since the intention is really deleting this ephemeral data from long lived environments then I'm cool with it.
Only real issue is if you're developing tests locally or on one of the QA team VMs and you're expecting data to be there over a couple days (like a weekend) then you'd better be ready for the Seeder to clean itself up.
Did we add a README.md? (Sorry on the GH app so hard to go back n forth).