{"id":11722,"date":"2023-01-31T17:17:29","date_gmt":"2023-01-31T17:17:29","guid":{"rendered":"https:\/\/teamkinetic.co.uk\/blog\/?p=11722"},"modified":"2023-02-03T16:16:31","modified_gmt":"2023-02-03T16:16:31","slug":"major-service-loss-january-30th-2023-response-andecommendations","status":"publish","type":"post","link":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/","title":{"rendered":"Major Service Loss January 30th 2023: Response and Recommendations"},"content":{"rendered":"\n

What Happened?<\/span><\/h2>\n

Web servers and other public servers are always under attack. Our systems deflect and mitigate thousands of unauthorised attempts to gain access every day, both direct login attempts and denial of service attacks where volumes of junk traffic are thrown at servers.<\/span><\/p>\n

It is impossible to both have an accessible service and an inaccessible service and we are always trying to tread that line and this time we failed to get it right.<\/span><\/p>\n

We experienced intermittent connection issues across the weekend but some access was still available.<\/span><\/p>\n

All connection and service was lost during the early hours of Monday morning when the full attack was started.<\/span><\/p>\n

An attempted \u2018<\/span>ransomware attack<\/span><\/a>\u2019 on the main server encrypted some files before it was stopped by our anti-malware processes. This was an automated scripted bot attack. There was no unauthorised access to customer data or backup data which is separately encrypted and stored. There was no evidence of any data exfiltration (access to data stores, virtual servers, external storage, no increase in bandwidth consumption, and no FTP\/SFTP access).<\/span><\/p>\n

To be clear, no data was viewed, accessed, or removed during this attack.\u00a0<\/b><\/p>\n

As we had complete backups from shortly before the attack started it was deemed that the most secure response would be to wipe the main server and so negate any potential additional issues with lingering malware or potential trojans\/backdoor attacks.<\/span><\/p>\n

As per our standard operating procedures in such an event, the servers hard drives and RAID arrays were wiped and reinitialised and the process of reinstalling the base operating system started.<\/span><\/p>\n

Although the initial steps of installation were successful we experienced additional issues when the new operating system would not load. The server was troubleshooted for potential issues\/changes that were preventing the new operating system from loading.<\/span><\/p>\n

Working with our hosting providers, we established that the attack had also led to a hardware fault with the RAID controller. This additional issue meant our existing server hardware was no longer viable and a new server provision was initiated.<\/span><\/p>\n

The new server was brought online and the server operating system was installed. This initially failed due to the unavailability of Microsoft’s license approval servers.<\/span><\/p>\n

By Monday evening at 19:30, the new server was responsive and the restoration process could begin.<\/span><\/p>\n

Our established processes did kick in but we lost approximately 12 hours of user data when rolling back.\u00a0 However, this episode has provided a valuable learning experience and we have started a more in depth review of our response, looking at our successes and failures and how this might any future response.\u00a0\u00a0\u00a0<\/span><\/p>\n

We have been proud of our performance to date when it comes to cyber security, with this being our first ever full day outage in our twelve year history, but, as always, there are lessons we can learn and things we can do better.<\/span><\/p>\n

Below is the details timeline of events, actions taken, and lessons learned.<\/span><\/p>\n

Timeline<\/span><\/h2>\n

07:00 Initial investigations of the affected server begin<\/span><\/p>\n

08:00 Attack was identified and its severity assessed<\/span><\/p>\n

08:30 Total loss of our main server was suspected<\/span><\/p>\n

09:00 Decision was made to wipe main server and restore to earlier backups.<\/span><\/p>\n

10:30 The affected server was wiped and the new raid array of hard drives initialised<\/span><\/p>\n

10:50 Raid completed and new OS installation was started<\/span><\/p>\n

11:50 New OS installation fails to start correctly<\/span><\/p>\n

12:00 Second attempt to install new OS<\/span><\/p>\n

12:40 Second attempt also fails<\/span><\/p>\n

13:00 Troubleshooting starts on the hardware to try to get Windows to boot<\/span><\/p>\n

14:30 New server is provisioned<\/span><\/p>\n

15:40 New server built and brought online<\/span><\/p>\n

15:40 New OS installation started on new hardware<\/span><\/p>\n

16:15 New OS installation unsuccessful due to licesne server unavailability from Microsoft<\/span><\/p>\n

18:00 New OS configuration complete<\/span><\/p>\n

18:30 Required software and utility installation<\/span><\/p>\n

19:30 Begin to restore backups<\/span><\/p>\n

20:20 Backups restored<\/span><\/p>\n

20:45 Applications start to be restored and access gained<\/span><\/p>\n

22:30 All services responding normally<\/span><\/p>\n

\u00a0<\/p>\n

Review of the Disaster Recovery Procedure<\/span><\/h2>\n

Our full recovery plans can be viewed on our website;<\/span><\/p>\n

https:\/\/teamkinetic.co.uk\/policies\/Contingency%20and%20Continuity%20Planning%20Policy<\/span><\/a><\/p>\n

https:\/\/teamkinetic.co.uk\/policies\/Data%20Asset%20Protection%20and%20Resilience<\/span><\/a><\/p>\n

In summary, we failed to meet our recovery time objective (RTO) of 2 hours because of the continued knock-on effects of hardware issues. The actual recovery process from downloading, extracting and installing the most recent backups was close to 2 hours once the hardware and operating system platform was stable.<\/span><\/p>\n

Incidence Reporting, Communication, and Support<\/span><\/h2>\n

Once the outage was affecting our customers we started to send out regular updates to keep customers informed.\u00a0<\/span><\/p>\n

These were sent via email as all internal messaging systems were affected. We also posted on our Facebook page and volunteer manager groups with the current status.<\/span><\/p>\n

We had multiple members of staff available all day on the phone to take calls and requests for support and believe we did a satisfactory job of keeping people up to date.<\/span><\/p>\n

This was a major and long-lasting outage and all our affected customers are entitled to a month’s service credit that is redeemable at the next invoicing period. We know this doesn’t make up for lost time and the frustration of not having access to your applications.<\/span><\/p>\n

What Did We Learn?<\/span><\/h2>\n

Our notification system for monitoring server health failed and was not able to cope with the specific complexity of this attack. We had a situation where our network accessible servers and systems were alive but not working correctly.<\/span><\/p>\n

Our response times during the weekend exacerbated the monitoring issues.<\/span><\/p>\n

Our hypervisor server is our most critical single point of failure.<\/span><\/p>\n

It takes longer to download and extract backups now than it did as they are considerably larger and so our RTO needs to be updated.<\/span><\/p>\n

Our transactional database backups (which fill in the gaps between full backups) need to be available from off-site backups to further limit the data loss in total failure events like this.<\/span><\/p>\n

Our hardware provider was too slow to respond and made mistakes in provisioning that were made worse by lower staff numbers over the weekend, changes in shifts, and lack of communication between those shifts.<\/span><\/p>\n

Almost 70% of the time to restoration of services was spent waiting for our hardware providers to execute their responsibilities.<\/span><\/p>\n

Our customers are incredibly understanding and supportive, thank you!<\/span><\/p>\n

Mitigations and Improvements<\/span><\/h2>\n

Add in more sensitive monitoring and also include positive monitoring that tells us that things are OK not just negative monitoring.<\/span><\/p>\n

Mandate two factor authentication for UAC as well as login.<\/span><\/p>\n

Switch to a new hardware provider with better response times and procedures for dealing with issues.<\/span><\/p>\n

Recalculate our RTO bearing in mind the increase in size of our systems.<\/span><\/p>\n

Move transactional data logs to temp off-site storage at regular intervals within a 24 hour period. Retain these logs transactions for 48 hours.<\/span><\/p>\n

Provision a duplicate server for quicker server reinstatment. If we get a total failure\/loss of the mainserver we can rollback to the most recent backup within the RTO period.<\/span><\/p>\n

Look at a double daily complete backup of virtual servers. Would need to test the impact of backups on serve performance during regular accessing hours (right now the backup is performed at our quietest time). This would half our potential data loss in the case of complete failure.<\/span><\/p>\n

Follow up<\/span><\/h2>\n

An attack of this type can cause anxiety for our users, and it is important to us here at TeamKinetic, that you feel confident in our response to this incident and trust that we have taken away the important lessons from this experience.<\/span><\/p>\n

If you would like to speak to a member of the team, we would be only too happy to spend some time answering your questions. Feel free to use <\/span>this link<\/span><\/a> to arrange a call with the team.<\/span><\/p>\n

You can also subscribe to service status updates <\/span>here<\/span><\/a>.<\/span><\/p>\n



<\/p>\n

\u00a0<\/p>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[292,5],"tags":[],"class_list":["post-11722","post","type-post","status-publish","format-standard","hentry","category-securitygdpr","category-technical-updates"],"jetpack_publicize_connections":[],"yoast_head":"\r\nMajor Service Loss January 30th 2023: Response and Recommendations - TeamKinetic<\/title>\r\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\r\n<link rel=\"canonical\" href=\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/\" \/>\r\n<meta property=\"og:locale\" content=\"en_GB\" \/>\r\n<meta property=\"og:type\" content=\"article\" \/>\r\n<meta property=\"og:title\" content=\"Major Service Loss January 30th 2023: Response and Recommendations - TeamKinetic\" \/>\r\n<meta property=\"og:url\" content=\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/\" \/>\r\n<meta property=\"og:site_name\" content=\"TeamKinetic\" \/>\r\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/TeamKineticUK\" \/>\r\n<meta property=\"article:published_time\" content=\"2023-01-31T17:17:29+00:00\" \/>\r\n<meta property=\"article:modified_time\" content=\"2023-02-03T16:16:31+00:00\" \/>\r\n<meta name=\"author\" content=\"Rolf Herbert\" \/>\r\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\r\n<meta name=\"twitter:creator\" content=\"@TeamKineticUK\" \/>\r\n<meta name=\"twitter:site\" content=\"@TeamKineticUK\" \/>\r\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Rolf Herbert\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\r\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/\"},\"author\":{\"name\":\"Rolf Herbert\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/person\/bf564ec6a7b141b3fd5b42b422977775\"},\"headline\":\"Major Service Loss January 30th 2023: Response and Recommendations\",\"datePublished\":\"2023-01-31T17:17:29+00:00\",\"dateModified\":\"2023-02-03T16:16:31+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/\"},\"wordCount\":1282,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#organization\"},\"articleSection\":[\"Security and GDPR\",\"Technical Updates\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/\",\"url\":\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/\",\"name\":\"Major Service Loss January 30th 2023: Response and Recommendations - TeamKinetic\",\"isPartOf\":{\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#website\"},\"datePublished\":\"2023-01-31T17:17:29+00:00\",\"dateModified\":\"2023-02-03T16:16:31+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/teamkinetic.co.uk\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Major Service Loss January 30th 2023: Response and Recommendations\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#website\",\"url\":\"https:\/\/teamkinetic.co.uk\/blog\/\",\"name\":\"TeamKinetic\",\"description\":\"Connect, with great value volunteer management software\",\"publisher\":{\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/teamkinetic.co.uk\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#organization\",\"name\":\"TeamKinetic\",\"url\":\"https:\/\/teamkinetic.co.uk\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2023\/06\/logo-TeamK.png\",\"contentUrl\":\"https:\/\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2023\/06\/logo-TeamK.png\",\"width\":1850,\"height\":498,\"caption\":\"TeamKinetic\"},\"image\":{\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/TeamKineticUK\",\"https:\/\/twitter.com\/TeamKineticUK\",\"https:\/\/www.instagram.com\/teamkineticuk\/\",\"https:\/\/www.linkedin.com\/company\/teamkinetic\/\",\"https:\/\/www.youtube.com\/c\/TeamKinetic\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/person\/bf564ec6a7b141b3fd5b42b422977775\",\"name\":\"Rolf Herbert\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c2bcd1f933804170bd2b84b2048457a8?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c2bcd1f933804170bd2b84b2048457a8?s=96&d=mm&r=g\",\"caption\":\"Rolf Herbert\"},\"url\":\"https:\/\/teamkinetic.co.uk\/blog\/author\/rolf-herbert\/\"}]}<\/script>\r\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Major Service Loss January 30th 2023: Response and Recommendations - TeamKinetic","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/","og_locale":"en_GB","og_type":"article","og_title":"Major Service Loss January 30th 2023: Response and Recommendations - TeamKinetic","og_url":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/","og_site_name":"TeamKinetic","article_publisher":"https:\/\/www.facebook.com\/TeamKineticUK","article_published_time":"2023-01-31T17:17:29+00:00","article_modified_time":"2023-02-03T16:16:31+00:00","author":"Rolf Herbert","twitter_card":"summary_large_image","twitter_creator":"@TeamKineticUK","twitter_site":"@TeamKineticUK","twitter_misc":{"Written by":"Rolf Herbert","Estimated reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/#article","isPartOf":{"@id":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/"},"author":{"name":"Rolf Herbert","@id":"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/person\/bf564ec6a7b141b3fd5b42b422977775"},"headline":"Major Service Loss January 30th 2023: Response and Recommendations","datePublished":"2023-01-31T17:17:29+00:00","dateModified":"2023-02-03T16:16:31+00:00","mainEntityOfPage":{"@id":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/"},"wordCount":1282,"commentCount":0,"publisher":{"@id":"https:\/\/teamkinetic.co.uk\/blog\/#organization"},"articleSection":["Security and GDPR","Technical Updates"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/","url":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/","name":"Major Service Loss January 30th 2023: Response and Recommendations - TeamKinetic","isPartOf":{"@id":"https:\/\/teamkinetic.co.uk\/blog\/#website"},"datePublished":"2023-01-31T17:17:29+00:00","dateModified":"2023-02-03T16:16:31+00:00","breadcrumb":{"@id":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/teamkinetic.co.uk\/blog\/2023\/01\/major-service-loss-january-30th-2023-response-andecommendations\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/teamkinetic.co.uk\/blog\/"},{"@type":"ListItem","position":2,"name":"Major Service Loss January 30th 2023: Response and Recommendations"}]},{"@type":"WebSite","@id":"https:\/\/teamkinetic.co.uk\/blog\/#website","url":"https:\/\/teamkinetic.co.uk\/blog\/","name":"TeamKinetic","description":"Connect, with great value volunteer management software","publisher":{"@id":"https:\/\/teamkinetic.co.uk\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/teamkinetic.co.uk\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-GB"},{"@type":"Organization","@id":"https:\/\/teamkinetic.co.uk\/blog\/#organization","name":"TeamKinetic","url":"https:\/\/teamkinetic.co.uk\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2023\/06\/logo-TeamK.png","contentUrl":"https:\/\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2023\/06\/logo-TeamK.png","width":1850,"height":498,"caption":"TeamKinetic"},"image":{"@id":"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/TeamKineticUK","https:\/\/twitter.com\/TeamKineticUK","https:\/\/www.instagram.com\/teamkineticuk\/","https:\/\/www.linkedin.com\/company\/teamkinetic\/","https:\/\/www.youtube.com\/c\/TeamKinetic"]},{"@type":"Person","@id":"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/person\/bf564ec6a7b141b3fd5b42b422977775","name":"Rolf Herbert","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/teamkinetic.co.uk\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c2bcd1f933804170bd2b84b2048457a8?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c2bcd1f933804170bd2b84b2048457a8?s=96&d=mm&r=g","caption":"Rolf Herbert"},"url":"https:\/\/teamkinetic.co.uk\/blog\/author\/rolf-herbert\/"}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p8AVXm-334","jetpack-related-posts":[{"id":8539,"url":"https:\/\/teamkinetic.co.uk\/blog\/2021\/12\/network-outage-thursday-30th-december-2021\/","url_meta":{"origin":11722,"position":0},"title":"Network Outage Thursday 30th December 2021","author":"Rolf Herbert","date":"30th December 2021","format":false,"excerpt":"Duration - 4 Hours Our sincere apologies for the length of the outage reported today, we always strive to give our customers the best possible service and we failed today to identify and resolve the issue quickly enough. What Happened? From approximately 08:30 on Thursday 30th December our services started\u2026","rel":"","context":"In "Technical Updates"","block_context":{"text":"Technical Updates","link":"https:\/\/teamkinetic.co.uk\/blog\/category\/technical-updates\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2021\/12\/image-8.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2021\/12\/image-8.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2021\/12\/image-8.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2021\/12\/image-8.png?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":4575,"url":"https:\/\/teamkinetic.co.uk\/blog\/2019\/10\/contingency-and-continuity-planning\/","url_meta":{"origin":11722,"position":1},"title":"Contingency and Continuity Planning","author":"Rolf Herbert","date":"17th October 2019","format":false,"excerpt":"Contingency and continuity planning policy \u00a0 Identified Potential Risks The following identified risks will trigger a potential contingency event, each event is independently risk assessed and will be managed in accordance with this policy. Natural disaster Fire Flood Sudden cessation of trade Sale of company Hardware failure Software failure Malicious\u2026","rel":"","context":"In "Security and GDPR"","block_context":{"text":"Security and GDPR","link":"https:\/\/teamkinetic.co.uk\/blog\/category\/securitygdpr\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2989,"url":"https:\/\/teamkinetic.co.uk\/blog\/2019\/04\/teamkinetic-and-software-as-a-service-saas-cutting-through-the-jargon\/","url_meta":{"origin":11722,"position":2},"title":"TeamKinetic and Software as a Service (SaaS) cutting through the jargon.","author":"Chris Martin","date":"5th April 2019","format":false,"excerpt":"Our ambition was to build the very best online volunteer management application in the world. Part of our definition of the very best means easiest to use, as well as the most useful features. Unfortunately when you build software there is a real danger of \"JARGON\"! We know volunteer managers\u2026","rel":"","context":"In "Application Features and Tutorials"","block_context":{"text":"Application Features and Tutorials","link":"https:\/\/teamkinetic.co.uk\/blog\/category\/application-features-and-tutorials\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":16909,"url":"https:\/\/teamkinetic.co.uk\/blog\/2025\/09\/teamkinetic-pricing-why-it-increases\/","url_meta":{"origin":11722,"position":3},"title":"Why Our Pricing Grows with Us","author":"Chris Martin","date":"26th September 2025","format":false,"excerpt":"Here at TeamKinetic, we love what we do: helping you make volunteer management simple. So you can focus on building stronger communities. It's a partnership we truly value, and part of a great partnership is open and honest communication, especially when it comes to our pricing. You might be wondering\u2026","rel":"","context":"In "Policy and Operations"","block_context":{"text":"Policy and Operations","link":"https:\/\/teamkinetic.co.uk\/blog\/category\/policy-and-operations\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":4577,"url":"https:\/\/teamkinetic.co.uk\/blog\/2019\/10\/it-and-information-governance-faqs\/","url_meta":{"origin":11722,"position":4},"title":"IT and Information Governance FAQs","author":"Rolf Herbert","date":"17th October 2019","format":false,"excerpt":"Have security-related job responsibilities, including oversight and accountability, been clearly defined and documented? Yes, all jobs have detailed JD in place with attention to Information Governance and responsibility.\u00a0 Chris Martin is currently accountable for Information Governance. Are policies for information handling and labelling in place? Yes.\u00a0 TeamKinetic have policies for\u2026","rel":"","context":"In "Security and GDPR"","block_context":{"text":"Security and GDPR","link":"https:\/\/teamkinetic.co.uk\/blog\/category\/securitygdpr\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1357,"url":"https:\/\/teamkinetic.co.uk\/blog\/2017\/11\/introducing-teamkinetic-rolf-herbet\/","url_meta":{"origin":11722,"position":5},"title":"Introducing TeamKinetic: Rolf Herbert -The man behind the application","author":"Rolf Herbert","date":"10th November 2017","format":false,"excerpt":"I'm one of the founders of TeamKinetic and have been responsible for developing and maintaining our technical assets for the last decade. Wow that's a weighty word....a decade. As anyone that has been involved in a startup or building up their own business will know, you have to be adaptable\u2026","rel":"","context":"In "News & Views"","block_context":{"text":"News & Views","link":"https:\/\/teamkinetic.co.uk\/blog\/category\/news-views\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rolf-sea.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rolf-sea.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rolf-sea.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/teamkinetic.co.uk\/blog\/wp-content\/uploads\/2017\/11\/rolf-sea.jpg?resize=1050%2C600&ssl=1 3x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/posts\/11722","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=11722"}],"version-history":[{"count":4,"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/posts\/11722\/revisions"}],"predecessor-version":[{"id":11783,"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/posts\/11722\/revisions\/11783"}],"wp:attachment":[{"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=11722"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=11722"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/teamkinetic.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=11722"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}