ThousandEyes Status

All Systems Operational

Uptime over the past 90 days. View historical uptime.

Agent Configuration and Data Collection Operational

90 days ago

99.96 % uptime

Today

Cloud and Enterprise Agents: Registration controller Operational

90 days ago

100.0 % uptime

Today

Cloud and Enterprise Agents: Test assignment and configuration controller Operational

90 days ago

99.99 % uptime

Today

Cloud and Enterprise Agents: Data ingress Operational

90 days ago

99.99 % uptime

Today

Endpoint Agents: Test assignment and configuration controller Operational

90 days ago

99.84 % uptime

Today

Endpoint Agents: Data submission collector Operational

90 days ago

99.96 % uptime

Today

ThousandEyes Platform and API Operational

90 days ago

99.99 % uptime

Today

Platform Availability Operational

90 days ago

99.99 % uptime

Today

API Availability Operational

90 days ago

99.99 % uptime

Today

Test data availability Operational

90 days ago

99.99 % uptime

Today

Reports and Dashboards Operational

90 days ago

99.99 % uptime

Today

Snapshots and Share Links Operational

90 days ago

100.0 % uptime

Today

SAML/SSO Operational

90 days ago

100.0 % uptime

Today

Configuration and Management Operational

90 days ago

100.0 % uptime

Today

Embedded Widgets Operational

90 days ago

100.0 % uptime

Today

Customer Success Chat Operational

90 days ago

100.0 % uptime

Today

Alerts and Notifications Operational

90 days ago

100.0 % uptime

Today

Alert processing Operational

90 days ago

100.0 % uptime

Today

Notification dispatching Operational

90 days ago

100.0 % uptime

Today

Webhook and Integrations Operational

90 days ago

100.0 % uptime

Today

Email notifications Operational

90 days ago

100.0 % uptime

Today

Internet Insights collection and processing Operational

90 days ago

100.0 % uptime

Today

BGP Operational

90 days ago

99.99 % uptime

Today

Public BGP collection and processing Operational

90 days ago

99.99 % uptime

Today

Private BGP Collection & Processing Operational

90 days ago

99.99 % uptime

Today

Agent Repositories and Downloads Operational

90 days ago

100.0 % uptime

Today

Ubuntu repository Operational

90 days ago

100.0 % uptime

Today

RHEL / CentOS repository Operational

90 days ago

100.0 % uptime

Today

Appliance downloads Operational

90 days ago

100.0 % uptime

Today

Custom Appliance Downloads Operational

90 days ago

100.0 % uptime

Today

Endpoint Agent downloads Operational

90 days ago

100.0 % uptime

Today

Cloud Agents: Americas Operational

Data Centers: AMER Operational

Broadband Providers: AMER Operational

Mobile Providers: AMER Operational

Azure: AMER Operational

AWS: AMER Operational

Google Cloud: AMER Operational

Alibaba Cloud: AMER Operational

Cloud Agents: Europe, Middle East and Africa Operational

Data Centers: EMEA Operational

Azure: EMEA Operational

AWS: EMEA Operational

Google Cloud: EMEA Operational

Alibaba Cloud: EMEA Operational

Cloud Agents: Asia Pacific Operational

Data Centers: APAC Operational

Azure: APAC Operational

AWS: APAC Operational

Google Cloud: APAC Operational

Alibaba Cloud: APAC Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

Scheduled Maintenance

Scheduled maintenance for database systems Oct 25, 16:00-19:00 UTC

Update - Maintenance postponed to 25th October 16:00 to 19:00 UTC
Oct 16, 22:48 UTC

Scheduled - The existing maintenance have been postponed to 25th October, please find the details as below:

Our Operations team will be performing maintenance of our database infrastructure.

Maintenance period:

Start: 16:00 UTC

End: 19:00 UTC

Expected impact:

At some point during this time, the web platform and API will briefly be in an immutable state. This could result in configuration related delays to test updates, agent registration, and the like. Alerts may also be slightly delayed.
Our Customer Success team is available at support@thousandeyes.com to answer any questions.

Oct 16, 22:45 UTC

Past Incidents

Oct 20, 2020

No incidents reported today.

Oct 19, 2020

ThousandEyes Platform degraded performance

Resolved - This incident has been resolved.
Oct 19, 21:29 UTC

Monitoring - Endpoint agents are now submitting data and catching up. Our Operations team is monitoring the situation.
Oct 19, 18:47 UTC

Identified - Summary:
Endpoint Agent data services are partially unavailable. The ThousandEyes operations team is working towards a resolution.

Impact:
Endpoint Agents data reads are impacted by this issue. As a result, endpoint agent data is partially unavailable from dashboards, views, scheduled tests, and the API.

The Endpoint Agent controller (c1.eb.thousandeyes.com) is also impacted. During this outage Endpoint Agents will not be able to receive new scheduled test, label, or monitored domain assignments. The platform will report Agents as unseen during this outage, affecting the reported "last seen" time. New Endpoint Agents will not be able to complete the registration process during this time.

Endpoint Agent data submission is also affected by this service degradation. Agent data will be submitted to the platform as connectivity is re-established.

We do not anticipate any loss of Endpoint Agent data.
Oct 19, 17:12 UTC

Update - We are continuing to investigate this issue.
Oct 19, 16:52 UTC

Update - We are continuing to investigate this issue alongside the MongoDB team.
Oct 19, 16:40 UTC

Update - We are continuing to investigate this issue.
Oct 19, 15:38 UTC

Update - We are continuing to investigate this issue.
Oct 19, 15:36 UTC

Investigating - ThousandEyes Platform is currently experiencing degraded performance.

Affected scope
Due to degraded performance of ThousandEyes web servers, customers might notice increased page load times. Customers might see intermittent errors and delays on some app pages, and 500 response codes via API. Endpoint data and alerts may be delayed. Usage related requests are also affected and may not be available from the previous billing cycle.
Oct 19, 15:00 UTC

Oct 18, 2020

Service degradation impacting ThousandEyes Application and API

Resolved - This incident has been resolved.

Between 2020-10-18 05:02 UTC and 2020-10-18 05:12 UTC requests to app.thousandeyes.com and api.thousandeyes.com may have resulted in an error response. Between 2020-10-18 05:04 UTC and 2020-10-18 05:20 UTC test result data ingested into the platform may have been delayed by up to 7 minutes.
Oct 18, 07:23 UTC

Monitoring - The issue has been understood and resolved, and should not repeat. We're still monitoring the situation, just in case.
Oct 18, 06:57 UTC

Update - We are continuing to investigate this issue.
Oct 18, 05:24 UTC

Update - We are continuing to investigate this issue.
Oct 18, 05:15 UTC

Investigating - Starting at 4:58 UTC today we are observing degraded performance for requests to app.thousandeyes.com and api.thousandeyes.com. Requests may time out or fail with 5XX response codes.
Oct 18, 05:12 UTC

Oct 17, 2020

ThousandEyes App Availability

Resolved - This incident has been resolved.
Oct 17, 00:13 UTC

Monitoring - Customers may experience some slow responses using the Web App or API. We are currently monitoring the situation and working on a fix.
Oct 16, 18:27 UTC

Investigating - ThousandEyes Platform experienced a brief availability issue earlier today at 12:20 UTC. Currently the issue is resolved, but could return.

Affected scope
Due to degraded performance of ThousandEyes web servers, customers might notice increased page load times or 500 response codes. The issue only affects web portal and API availability. All customer tests and alerts are working as expected.
Oct 16, 14:42 UTC

Oct 16, 2020

ThousandEyes Platform degraded performance

Resolved - This incident has been resolved.
Oct 16, 10:47 UTC

Update - Engineering team deployed the fix.
Oct 16, 04:46 UTC

Update - As Operations identified the root cause, they are applying the fix shortly. We confirmed the scope of impact was the partial service degradation between 16:28 - 16:32 UTC affected following services:

app.thousandeyes.com (webapps-app)

embed.thousandeyes.com

*.share.thousandeyes.com

Oct 16, 00:15 UTC

Monitoring - A fix has been implemented and we keep on investigating the root cause.
Oct 15, 20:49 UTC

Investigating - ThousandEyes Platform is currently experiencing degraded performance.

Affected scope
Due to degraded performance of ThousandEyes web servers, customers might notice increased page load times. The issue is only affecting web portal rendering times. All customer tests and alerts are working as expected.
Oct 15, 20:38 UTC

Oct 15, 2020

ThousandEyes Platform degraded performance

Resolved - Due to an unexpected outage of ThousandEyes primary ISP, some customers experienced issues with the Platform reachability. Agent Controller Infrastructure reachability, as well as BGP Private peering infrastructure, were affected. Customers with ThousandEyes agents and private peering in Europe, Asia, and the US East Coast were affected the most.

Affected scope
Platform reachability, Agent Controller Infrastructure reachability

Status
The issue is currently mitigated.
Our Operation Team was able to identify the root cause and mitigate the issue.

Event timeline
2020-10-13 06:24 UTC: Event detected by ThousandEyes service reliability team, an investigation started.
2020-10-13 06:39 UTC: Issue identified and mitigated.
Oct 15, 06:45 UTC

Investigating - Due to an unexpected outage of ThousandEyes primary ISP, some customers experienced issues with the Platform reachability. Agent Controller Infrastructure reachability, as well as BGP Private peering infrastructure, were affected. Customers with ThousandEyes agents and private peering in Europe, Asia, and the US East Coast were affected the most.

Affected scope
Platform reachability, Agent Controller Infrastructure reachability

Status
Our Operation Team is currently investigating the issue

Event timeline
2020-10-13 06:24 UTC: Event detected by ThousandEyes service reliability team, an investigation started.
Oct 15, 06:30 UTC

Oct 14, 2020

Service degradation affecting BGP data availability

Resolved - The incident has been identified and resolved.

Between 2020-10-14 03:02 UTC and 2020-10-14 03:07 UTC, delays loading BGP test data views in the web application, or error responses from API requests retrieving BGP data may have been observed.
Oct 14, 03:17 UTC

Investigating - Operations team have been alerted to delays in the platform potentially impacting BGP data availability and are currently investigating.
Oct 14, 03:07 UTC

Oct 13, 2020

Service degradation affecting alert notification

Resolved - This issue has been resolved now.
Oct 13, 22:13 UTC

Monitoring - Engineering team mitigated the issue and we observed the alerts notification dispatched now. We will keep on monitoring this issue.
Oct 13, 21:50 UTC

Investigating - Service degradation affecting alert notification dispatching

Affected scope
We are investigating possible delays in alert notification dispatching. An increase in notification delivery has been observed as of 19:30 UTC.
Oct 13, 21:39 UTC

(Mitigated) Service Provider Outage Impacting Platform reachability, Agent Controller Infrastructure reachability, and BGP Private peering from some geographical locations

Resolved - This incident has been resolved by the service provider.
Oct 13, 18:30 UTC

Monitoring - Due to an unexpected outage of ThousandEyes primary ISP, some customers experienced issues with the Platform reachability. Agent Controller Infrastructure reachability, as well as BGP Private peering infrastructure, were affected. Customers with ThousandEyes agents and private peering in Europe, Asia, and the US East Coast were affected the most.

Affected scope
Platform reachability, Agent Controller Infrastructure reachability, and BGP Private peering were affected.

Status
The issue is currently mitigated.
Our Operation Team was able to identify the root cause and mitigate the issue.
An investigation is currently ongoing.

Event timeline
2020-10-13 07:35 UTC: Event detected by ThousandEyes service reliability team, an investigation started.
2020-10-13 08:03 UTC: Issue identified and mitigated.
Oct 13, 09:32 UTC

Service degradation affecting Endpoint Agent data availability

Resolved - This incident has been resolved.
Oct 13, 02:36 UTC

Update - Instant tests services have been restored while the scheduled tests are still experiencing a delay in Endpoint Agent data availability.

Data availability of the scheduled tests is anticipated to return to normal by 02:10 UTC 2020-10-13
Oct 13, 01:27 UTC

Update - Since the changes have been applied, we are observing the delay decreasing.
Data availability is anticipated to return to normal by 01:30 UTC 2020-10-13.
Oct 12, 23:40 UTC

Update - Endpoint Agent services have been restored but we are still observing a delay in Endpoint Agent data availability. As a result, Endpoint Agent data is partially unavailable for use in dashboards, views, scheduled tests, and alerts.

Data availability is anticipated to return to normal by 00:30 UTC 2020-10-13
Oct 12, 22:39 UTC

Update - We are still observing a delay in Endpoint Agent data availability of up to 150 minutes.
Operations applied the additional changes to the Endpoint services and continue to investigate the issues closely.
Oct 12, 22:29 UTC

Update - We are still observing a delay in Endpoint Agent data availability of up to 120 minutes.
Operations have scaled up a few services in the pipeline and continue to investigate the issues closely.
Oct 12, 19:48 UTC

Update - Endpoint Agent services have been restored but we are still observing a delay in Endpoint Agent data availability of up to 90 minutes.

As a result, Endpoint Agent data is partially unavailable for use in dashboards, views, scheduled tests, and alerts.

Operations is closely monitoring Endpoint Agent service performance.
Oct 12, 17:32 UTC

Update - Endpoint Agents services have been restored but users may still observe delay in Endpoint Agent data availability.

Operations is closely monitoring Endpoint Agent service performance.
Oct 12, 17:02 UTC

Monitoring - Endpoint Agent controller service has been restored. Endpoint Agents may experience delays when receiving new scheduled test, label, or monitored domain assignments.

Endpoint Agent data services are still in the process of being restored.
Oct 12, 16:24 UTC

Update - Users may have experienced availability issues when attempting to access app.thousandeyes.com or api.thousandeyes.com between 15:36 and 15:45 UTC on 2020-10-12. Operations has applied a fix and we are currently monitoring performance.
Oct 12, 15:51 UTC

Update - We are continuing to investigate this issue.
Oct 12, 14:20 UTC

Investigating - Affected scope
Starting at 13:53 UTC, an issue was detected with the ThousandEyes platform affecting availability of the Endpoint Data collection service and the Reporting & Usage service.

Status
The ThousandEyes Operations team is actively investigating the issue.
Oct 12, 14:18 UTC

Oct 12, 2020

Cloud Agent 'Dublin, Ireland - IPv6' is currently unavailable

Resolved - Service to cloud Agent 'Dublin, Ireland - IPv6' has been successfully restored.

Due to the change in service provider, customers with tests assigned to Cloud Agent ‘Dublin, Ireland - IPv6’ will observe a corresponding change in path-visualization and source IP.

Customers who utilize an IP based ACL to protect targeted services should update their ACL to reflect the following IP addresses:
2001:4d40:4003::8:1/112
Oct 12, 23:25 UTC

Update - We are continuing to monitor for any further issues.
Oct 12, 22:20 UTC

Update - Services for Dublin, Ireland (IPv6) are scheduled to be restored today, October 12th 2020, before end of day.
Oct 12, 17:38 UTC

Monitoring - Summary
Due to service provider related issues, Cloud Agent 'Dublin, Ireland - IPv6' is unavailable to perform tests. This agent was first reported as unavailable on Wednesday, 2020-10-07, at 17:55 UTC

Impact
Tests assigned to 'Dublin, Ireland - IPv6' will report this agent as experiencing local problems. Test data from this agent will not be included in reports or dashboards while local problems are reported. No units will be consumed by this agent while it is unavailable.

We are working to provide the most reliable data possible and will restore services once agent performance has been verified with our new service provider.

Resolution
The ThousandEyes operations team will return services to Cloud Agent 'Dublin, Ireland - IPv6' with a new service provider. The Agent ID associated with this agent will remain unchanged.

Services are expected to return on or before October 9th, 2020.

Due to the change in service provider, customers with tests assigned to Cloud Agent ‘Dublin, Ireland - IPv6’ will observe a corresponding change in path-visualization and source IP.

Customer action required
Customers who utilize an IP based ACL to protect targeted services should update their ACL to reflect the change in service provider. We will announce these new IP addresses once they are available.
Oct 7, 20:29 UTC

Oct 11, 2020

No incidents reported.

Oct 10, 2020

No incidents reported.

Oct 9, 2020

Service degradation impacting ThousandEyes application and API

Resolved - As of 19:42 UTC, all services have fully recovered. The incident is now resolved. More details can be found here: https://success.thousandeyes.com/PublicArticlePage?articleIdParam=kA02R000000HpYlSAK_Monitoring-Service-degradation-impacting-platform-and-API-availability
Oct 9, 20:08 UTC

Monitoring - ThousandEyes operations team released a fix at 19:40UTC and the performance metrics since seem to have stabilized. We are still monitoring the issue closely.
Oct 9, 19:58 UTC

Update - We are continuing to investigate this issue.
Oct 9, 19:47 UTC

Investigating - Affected scope
Starting at 19:30 UTC today we are observing degraded performance for requests to app.thousandeyes.com and api.thousandeyes.com. Requests may time out or fail with 5XX response codes.

Status
The ThousandEyes Operations team is actively investigating the issue.
Oct 9, 19:46 UTC

Service degradation affecting BGP test data

Resolved - The incident has been resolved.
Oct 9, 05:49 UTC

Monitoring - The incident has been resolved.
Oct 9, 05:07 UTC

Identified - Engineering have identified the issue and are currently implementing a fix.
Oct 9, 05:05 UTC

Investigating - Service degradation impacting BGP test data retrieval occurred between 2020-10-09 04:42 UTC - 2020-10-09 05:07 UTC.

Affected scope
Attempting to retrieve BGP test result data either from ThousandEyes App or via API may have been delayed or resulted in an error response during this time. No BGP test data is missing as a result of this incident.
Oct 9, 04:42 UTC

Scheduled Maintenance of private BGP monitoring infrastructure

Completed - The scheduled maintenance has been completed.
Oct 9, 02:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 8, 22:00 UTC

Scheduled - Summary
Our operations team will perform maintenance on private BGP monitoring infrastructure

Impact
Only connections to the private BGP collectors assigned to IP below are anticipated to be disrupted. We will make further announcements if the scope of maintenance increases.

192.150.160.138
192.150.160.135
192.150.160.136
192.150.160.137
192.150.160.139
192.150.160.140
192.150.160.144
192.150.160.145
192.150.160.146
192.150.160.147
192.150.160.148
192.150.160.149
192.150.160.150

Timeline
2020-10-08 22:00 UTC: Beginning of maintenance
2020-10-09 02:00 UTC: End of maintenance
Oct 5, 22:38 UTC

Oct 8, 2020

Emergency maintenance of alerts infrastructure

Completed - The scheduled maintenance has been completed.
Oct 8, 20:00 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 8, 16:00 UTC

Update - We will be undergoing scheduled maintenance during this time.
Oct 7, 23:48 UTC

Scheduled - Summary
Our operations team will perform maintenance on alerts infrastructure

Impact
Assignment of FTP and BGP alert rules created or modified during this time will be delayed by approximately 2-4 hours. This would not impact any existing tests/rules setup for all tests/alert types.

Timeline
2020-10-08 16:00 UTC: Beginning of maintenance
2020-10-08 20:00 UTC: End of maintenance
Oct 7, 22:12 UTC

Degraded performance on Endpoint Agent controller infrastructure

Resolved - This incident has been resolved.
Oct 8, 09:10 UTC

Monitoring - We have increased the capacity of the Endpoint check-in service and will continue to monitor the service.
Oct 8, 07:17 UTC

Identified - Endpoint Agent check-in and configuration update service has been fully restored. The root cause of the issue was due to scale and load.
Oct 8, 07:05 UTC

Update - We are continuing to investigate this issue.
Oct 8, 06:55 UTC

Update - Affected scope
Due to a capacity issue, Endpoint Agents will experience failures when attempting to check-in with the controller.
All existing Endpoint Agents tests will continue to run as normal. Any new tests or Endpoint settings are affected. We have identified the issue and are working to resolve the issue as quickly as possible.
Oct 8, 06:51 UTC

Investigating - We are currently experiencing degraded performance on our Endpoint Agent controller infrastructure.

Affected scope
Endpoint Agents may experience a failure when attempting to check-in with the controller.
Oct 8, 06:27 UTC

Oct 7, 2020

2020-10-07: Scheduled maintenance of Cloud Agent "Paris, France"

Completed - The scheduled maintenance has been completed.
Oct 7, 23:46 UTC

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Oct 7, 00:00 UTC

Scheduled - Due to a change in Service Provider, customers with tests assigned to Cloud Agent ‘Paris, France’ will observe a corresponding change in path-visualization and source IP.

This maintenance is scheduled to take place on Wednesday, October 7th, between 20:00 and 22:00 UTC.

We do not anticipate downtime or impact on test performance. The Cloud Agent ID used within the ThousandEyes API will remain unchanged.

Customers who utilize an IP based ACL to protect targeted services should update their ACL to reflect the following IP addresses:

185.255.85.224/27
185.255.86.32/27
2A02:04BA:0100:0000:0008::/124
2A02:04BA:0100:0000:0009::/124
2A02:04BA:0100:0000:0010::/124
2A02:04BA:0100:0000:0011::/124

ThousandEyes support is available via in-app chat to provide assistance 24 hours a day.

Instructions for contacting support may be found in the following article:
https://docs.thousandeyes.com/product-documentation/getting-started/getting-support-from-thousandeyes
Oct 2, 23:10 UTC

Service degradation impacting ThousandEyes application and API

Resolved - The incident has been resolved.
Oct 7, 18:25 UTC

Monitoring - ThousandEyes operations team implemented a fix and response times are returning to normal baselines. We are actively monitoring the issue.
Oct 7, 18:08 UTC

Investigating - Affected scope

Requests to https://api.thousandeyes.com and https://app.thousandeys.com are currently suffering higher than usual response times and may fail with 5XX errors.

Status
The operations team is actively investigating the issue.
Oct 7, 17:55 UTC

Oct 6, 2020

No incidents reported.

Related