All Systems Operational
Agent Configuration and Data Collection Operational
90 days ago
99.96 % uptime
Today
Cloud and Enterprise Agents: Registration controller ? Operational
90 days ago
100.0 % uptime
Today
Cloud and Enterprise Agents: Test assignment and configuration controller ? Operational
90 days ago
99.99 % uptime
Today
Cloud and Enterprise Agents: Data ingress ? Operational
90 days ago
99.99 % uptime
Today
Endpoint Agents: Test assignment and configuration controller ? Operational
90 days ago
99.84 % uptime
Today
Endpoint Agents: Data submission collector ? Operational
90 days ago
99.96 % uptime
Today
ThousandEyes Platform and API Operational
90 days ago
99.99 % uptime
Today
Platform Availability ? Operational
90 days ago
99.99 % uptime
Today
API Availability ? Operational
90 days ago
99.99 % uptime
Today
Test data availability ? Operational
90 days ago
99.99 % uptime
Today
Reports and Dashboards ? Operational
90 days ago
99.99 % uptime
Today
Snapshots and Share Links ? Operational
90 days ago
100.0 % uptime
Today
SAML/SSO ? Operational
90 days ago
100.0 % uptime
Today
Configuration and Management ? Operational
90 days ago
100.0 % uptime
Today
Embedded Widgets ? Operational
90 days ago
100.0 % uptime
Today
Customer Success Chat ? Operational
90 days ago
100.0 % uptime
Today
Alerts and Notifications Operational
90 days ago
100.0 % uptime
Today
Alert processing ? Operational
90 days ago
100.0 % uptime
Today
Notification dispatching ? Operational
90 days ago
100.0 % uptime
Today
Webhook and Integrations ? Operational
90 days ago
100.0 % uptime
Today
Email notifications ? Operational
90 days ago
100.0 % uptime
Today
Internet Insights collection and processing Operational
90 days ago
100.0 % uptime
Today
BGP Operational
90 days ago
99.99 % uptime
Today
Public BGP collection and processing ? Operational
90 days ago
99.99 % uptime
Today
Private BGP Collection & Processing ? Operational
90 days ago
99.99 % uptime
Today
Agent Repositories and Downloads Operational
90 days ago
100.0 % uptime
Today
Ubuntu repository ? Operational
90 days ago
100.0 % uptime
Today
RHEL / CentOS repository ? Operational
90 days ago
100.0 % uptime
Today
Appliance downloads ? Operational
90 days ago
100.0 % uptime
Today
Custom Appliance Downloads ? Operational
90 days ago
100.0 % uptime
Today
Endpoint Agent downloads ? Operational
90 days ago
100.0 % uptime
Today
Cloud Agents: Americas Operational
Data Centers: AMER ? Operational
Broadband Providers: AMER ? Operational
Mobile Providers: AMER ? Operational
Azure: AMER ? Operational
AWS: AMER ? Operational
Google Cloud: AMER ? Operational
Alibaba Cloud: AMER ? Operational
Cloud Agents: Europe, Middle East and Africa Operational
Data Centers: EMEA ? Operational
Azure: EMEA ? Operational
AWS: EMEA ? Operational
Google Cloud: EMEA ? Operational
Alibaba Cloud: EMEA ? Operational
Cloud Agents: Asia Pacific Operational
Data Centers: APAC ? Operational
Azure: APAC ? Operational
AWS: APAC ? Operational
Google Cloud: APAC ? Operational
Alibaba Cloud: APAC ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Scheduled Maintenance
Update - Maintenance postponed to 25th October 16:00 to 19:00 UTC
Oct 16, 22:48 UTC
Scheduled - The existing maintenance have been postponed to 25th October, please find the details as below:

Our Operations team will be performing maintenance of our database infrastructure.



Maintenance period:

Start: 16:00 UTC

End: 19:00 UTC

Expected impact:

At some point during this time, the web platform and API will briefly be in an immutable state. This could result in configuration related delays to test updates, agent registration, and the like. Alerts may also be slightly delayed.
Our Customer Success team is available at support@thousandeyes.com to answer any questions.

Oct 16, 22:45 UTC
Past Incidents
Oct 20, 2020

No incidents reported today.

Oct 19, 2020
Resolved - This incident has been resolved.
Oct 19, 21:29 UTC
Monitoring - Endpoint agents are now submitting data and catching up. Our Operations team is monitoring the situation.
Oct 19, 18:47 UTC
Identified - Summary:
Endpoint Agent data services are partially unavailable. The ThousandEyes operations team is working towards a resolution.

Impact:
Endpoint Agents data reads are impacted by this issue. As a result, endpoint agent data is partially unavailable from dashboards, views, scheduled tests, and the API.

The Endpoint Agent controller (c1.eb.thousandeyes.com) is also impacted. During this outage Endpoint Agents will not be able to receive new scheduled test, label, or monitored domain assignments. The platform will report Agents as unseen during this outage, affecting the reported "last seen" time. New Endpoint Agents will not be able to complete the registration process during this time.

Endpoint Agent data submission is also affected by this service degradation. Agent data will be submitted to the platform as connectivity is re-established.

We do not anticipate any loss of Endpoint Agent data.
Oct 19, 17:12 UTC
Update - We are continuing to investigate this issue.
Oct 19, 16:52 UTC
Update - We are continuing to investigate this issue alongside the MongoDB team.
Oct 19, 16:40 UTC
Update - We are continuing to investigate this issue.
Oct 19, 15:38 UTC
Update - We are continuing to investigate this issue.
Oct 19, 15:36 UTC
Investigating - ThousandEyes Platform is currently experiencing degraded performance.

Affected scope
Due to degraded performance of ThousandEyes web servers, customers might notice increased page load times. Customers might see intermittent errors and delays on some app pages, and 500 response codes via API. Endpoint data and alerts may be delayed. Usage related requests are also affected and may not be available from the previous billing cycle.
Oct 19, 15:00 UTC
Oct 18, 2020
Resolved - This incident has been resolved.

Between 2020-10-18 05:02 UTC and 2020-10-18 05:12 UTC requests to app.thousandeyes.com and api.thousandeyes.com may have resulted in an error response. Between 2020-10-18 05:04 UTC and 2020-10-18 05:20 UTC test result data ingested into the platform may have been delayed by up to 7 minutes.
Oct 18, 07:23 UTC
Monitoring - The issue has been understood and resolved, and should not repeat. We're still monitoring the situation, just in case.
Oct 18, 06:57 UTC
Update - We are continuing to investigate this issue.
Oct 18, 05:24 UTC
Update - We are continuing to investigate this issue.
Oct 18, 05:15 UTC
Investigating - Starting at 4:58 UTC today we are observing degraded performance for requests to app.thousandeyes.com and api.thousandeyes.com. Requests may time out or fail with 5XX response codes.
Oct 18, 05:12 UTC
Oct 17, 2020
Resolved - This incident has been resolved.
Oct 17, 00:13 UTC
Monitoring - Customers may experience some slow responses using the Web App or API. We are currently monitoring the situation and working on a fix.
Oct 16, 18:27 UTC
Investigating - ThousandEyes Platform experienced a brief availability issue earlier today at 12:20 UTC. Currently the issue is resolved, but could return.

Affected scope
Due to degraded performance of ThousandEyes web servers, customers might notice increased page load times or 500 response codes. The issue only affects web portal and API availability. All customer tests and alerts are working as expected.
Oct 16, 14:42 UTC
Oct 16, 2020
Resolved - This incident has been resolved.
Oct 16, 10:47 UTC
Update - Engineering team deployed the fix.
Oct 16, 04:46 UTC
Update - As Operations identified the root cause, they are applying the fix shortly. We confirmed the scope of impact was the partial service degradation between 16:28 - 16:32 UTC affected following services:
  • app.thousandeyes.com (webapps-app)

  • embed.thousandeyes.com

  • *.share.thousandeyes.com

  • Oct 16, 00:15 UTC
    Monitoring - A fix has been implemented and we keep on investigating the root cause.
    Oct 15, 20:49 UTC
    Investigating - ThousandEyes Platform is currently experiencing degraded performance.

    Affected scope
    Due to degraded performance of ThousandEyes web servers, customers might notice increased page load times. The issue is only affecting web portal rendering times. All customer tests and alerts are working as expected.
    Oct 15, 20:38 UTC
    Oct 15, 2020
    Resolved - Due to an unexpected outage of ThousandEyes primary ISP, some customers experienced issues with the Platform reachability. Agent Controller Infrastructure reachability, as well as BGP Private peering infrastructure, were affected. Customers with ThousandEyes agents and private peering in Europe, Asia, and the US East Coast were affected the most.

    Affected scope
    Platform reachability, Agent Controller Infrastructure reachability

    Status
    The issue is currently mitigated.
    Our Operation Team was able to identify the root cause and mitigate the issue.

    Event timeline
    2020-10-13 06:24 UTC: Event detected by ThousandEyes service reliability team, an investigation started.
    2020-10-13 06:39 UTC: Issue identified and mitigated.
    Oct 15, 06:45 UTC
    Investigating - Due to an unexpected outage of ThousandEyes primary ISP, some customers experienced issues with the Platform reachability. Agent Controller Infrastructure reachability, as well as BGP Private peering infrastructure, were affected. Customers with ThousandEyes agents and private peering in Europe, Asia, and the US East Coast were affected the most.

    Affected scope
    Platform reachability, Agent Controller Infrastructure reachability

    Status
    Our Operation Team is currently investigating the issue

    Event timeline
    2020-10-13 06:24 UTC: Event detected by ThousandEyes service reliability team, an investigation started.
    Oct 15, 06:30 UTC
    Oct 14, 2020
    Resolved - The incident has been identified and resolved.

    Between 2020-10-14 03:02 UTC and 2020-10-14 03:07 UTC, delays loading BGP test data views in the web application, or error responses from API requests retrieving BGP data may have been observed.
    Oct 14, 03:17 UTC
    Investigating - Operations team have been alerted to delays in the platform potentially impacting BGP data availability and are currently investigating.
    Oct 14, 03:07 UTC
    Oct 13, 2020
    Resolved - This issue has been resolved now.
    Oct 13, 22:13 UTC
    Monitoring - Engineering team mitigated the issue and we observed the alerts notification dispatched now. We will keep on monitoring this issue.
    Oct 13, 21:50 UTC
    Investigating - Service degradation affecting alert notification dispatching

    Affected scope
    We are investigating possible delays in alert notification dispatching. An increase in notification delivery has been observed as of 19:30 UTC.
    Oct 13, 21:39 UTC
    Resolved - This incident has been resolved by the service provider.
    Oct 13, 18:30 UTC
    Monitoring - Due to an unexpected outage of ThousandEyes primary ISP, some customers experienced issues with the Platform reachability. Agent Controller Infrastructure reachability, as well as BGP Private peering infrastructure, were affected. Customers with ThousandEyes agents and private peering in Europe, Asia, and the US East Coast were affected the most.

    Affected scope
    Platform reachability, Agent Controller Infrastructure reachability, and BGP Private peering were affected.

    Status
    The issue is currently mitigated.
    Our Operation Team was able to identify the root cause and mitigate the issue.
    An investigation is currently ongoing.

    Event timeline
    2020-10-13 07:35 UTC: Event detected by ThousandEyes service reliability team, an investigation started.
    2020-10-13 08:03 UTC: Issue identified and mitigated.
    Oct 13, 09:32 UTC
    Resolved - This incident has been resolved.
    Oct 13, 02:36 UTC
    Update - Instant tests services have been restored while the scheduled tests are still experiencing a delay in Endpoint Agent data availability.

    Data availability of the scheduled tests is anticipated to return to normal by 02:10 UTC 2020-10-13
    Oct 13, 01:27 UTC
    Update - Since the changes have been applied, we are observing the delay decreasing.
    Data availability is anticipated to return to normal by 01:30 UTC 2020-10-13.
    Oct 12, 23:40 UTC
    Update - Endpoint Agent services have been restored but we are still observing a delay in Endpoint Agent data availability. As a result, Endpoint Agent data is partially unavailable for use in dashboards, views, scheduled tests, and alerts.

    Data availability is anticipated to return to normal by 00:30 UTC 2020-10-13
    Oct 12, 22:39 UTC
    Update - We are still observing a delay in Endpoint Agent data availability of up to 150 minutes.
    Operations applied the additional changes to the Endpoint services and continue to investigate the issues closely.
    Oct 12, 22:29 UTC
    Update - We are still observing a delay in Endpoint Agent data availability of up to 120 minutes.
    Operations have scaled up a few services in the pipeline and continue to investigate the issues closely.
    Oct 12, 19:48 UTC
    Update - Endpoint Agent services have been restored but we are still observing a delay in Endpoint Agent data availability of up to 90 minutes.

    As a result, Endpoint Agent data is partially unavailable for use in dashboards, views, scheduled tests, and alerts.

    Operations is closely monitoring Endpoint Agent service performance.
    Oct 12, 17:32 UTC
    Update - Endpoint Agents services have been restored but users may still observe delay in Endpoint Agent data availability.

    Operations is closely monitoring Endpoint Agent service performance.
    Oct 12, 17:02 UTC
    Monitoring - Endpoint Agent controller service has been restored. Endpoint Agents may experience delays when receiving new scheduled test, label, or monitored domain assignments.

    Endpoint Agent data services are still in the process of being restored.
    Oct 12, 16:24 UTC
    Update - Users may have experienced availability issues when attempting to access app.thousandeyes.com or api.thousandeyes.com between 15:36 and 15:45 UTC on 2020-10-12. Operations has applied a fix and we are currently monitoring performance.
    Oct 12, 15:51 UTC
    Identified - Summary:
    Endpoint Agent data services are partially unavailable. The ThousandEyes operations team is working towards a resolution.

    Impact:
    Endpoint Agents data reads are impacted by this issue. As a result, endpoint agent data is partially unavailable from dashboards, views, scheduled tests, and the API.

    The Endpoint Agent controller (c1.eb.thousandeyes.com) is also impacted. During this outage Endpoint Agents will not be able to receive new scheduled test, label, or monitored domain assignments. The platform will report Agents as unseen during this outage, affecting the reported "last seen" time. New Endpoint Agents will not be able to complete the registration process during this time.

    Endpoint Agent data submission is not affected by this outage.
    Oct 12, 15:29 UTC
    Update - We are continuing to investigate this issue.
    Oct 12, 14:20 UTC
    Investigating - Affected scope
    Starting at 13:53 UTC, an issue was detected with the ThousandEyes platform affecting availability of the Endpoint Data collection service and the Reporting & Usage service.

    Status
    The ThousandEyes Operations team is actively investigating the issue.
    Oct 12, 14:18 UTC
    Oct 12, 2020
    Resolved - Service to cloud Agent 'Dublin, Ireland - IPv6' has been successfully restored.

    Due to the change in service provider, customers with tests assigned to Cloud Agent ‘Dublin, Ireland - IPv6’ will observe a corresponding change in path-visualization and source IP.

    Customers who utilize an IP based ACL to protect targeted services should update their ACL to reflect the following IP addresses:
    2001:4d40:4003::8:1/112
    Oct 12, 23:25 UTC
    Update - We are continuing to monitor for any further issues.
    Oct 12, 22:20 UTC
    Update - Services for Dublin, Ireland (IPv6) are scheduled to be restored today, October 12th 2020, before end of day.
    Oct 12, 17:38 UTC
    Monitoring - Summary
    Due to service provider related issues, Cloud Agent 'Dublin, Ireland - IPv6' is unavailable to perform tests. This agent was first reported as unavailable on Wednesday, 2020-10-07, at 17:55 UTC

    Impact
    Tests assigned to 'Dublin, Ireland - IPv6' will report this agent as experiencing local problems. Test data from this agent will not be included in reports or dashboards while local problems are reported. No units will be consumed by this agent while it is unavailable.

    We are working to provide the most reliable data possible and will restore services once agent performance has been verified with our new service provider.

    Resolution
    The ThousandEyes operations team will return services to Cloud Agent 'Dublin, Ireland - IPv6' with a new service provider. The Agent ID associated with this agent will remain unchanged.

    Services are expected to return on or before October 9th, 2020.

    Due to the change in service provider, customers with tests assigned to Cloud Agent ‘Dublin, Ireland - IPv6’ will observe a corresponding change in path-visualization and source IP.

    Customer action required
    Customers who utilize an IP based ACL to protect targeted services should update their ACL to reflect the change in service provider. We will announce these new IP addresses once they are available.
    Oct 7, 20:29 UTC
    Oct 11, 2020

    No incidents reported.

    Oct 10, 2020

    No incidents reported.

    Oct 9, 2020
    Resolved - As of 19:42 UTC, all services have fully recovered. The incident is now resolved. More details can be found here: https://success.thousandeyes.com/PublicArticlePage?articleIdParam=kA02R000000HpYlSAK_Monitoring-Service-degradation-impacting-platform-and-API-availability
    Oct 9, 20:08 UTC
    Monitoring - ThousandEyes operations team released a fix at 19:40UTC and the performance metrics since seem to have stabilized. We are still monitoring the issue closely.
    Oct 9, 19:58 UTC
    Update - We are continuing to investigate this issue.
    Oct 9, 19:47 UTC
    Investigating - Affected scope
    Starting at 19:30 UTC today we are observing degraded performance for requests to app.thousandeyes.com and api.thousandeyes.com. Requests may time out or fail with 5XX response codes.

    Status
    The ThousandEyes Operations team is actively investigating the issue.
    Oct 9, 19:46 UTC
    Resolved - The incident has been resolved.
    Oct 9, 05:49 UTC
    Monitoring - The incident has been resolved.
    Oct 9, 05:07 UTC
    Identified - Engineering have identified the issue and are currently implementing a fix.
    Oct 9, 05:05 UTC
    Investigating - Service degradation impacting BGP test data retrieval occurred between 2020-10-09 04:42 UTC - 2020-10-09 05:07 UTC.

    Affected scope
    Attempting to retrieve BGP test result data either from ThousandEyes App or via API may have been delayed or resulted in an error response during this time. No BGP test data is missing as a result of this incident.
    Oct 9, 04:42 UTC
    Completed - The scheduled maintenance has been completed.
    Oct 9, 02:00 UTC
    In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
    Oct 8, 22:00 UTC
    Scheduled - Summary
    Our operations team will perform maintenance on private BGP monitoring infrastructure

    Impact
    Only connections to the private BGP collectors assigned to IP below are anticipated to be disrupted. We will make further announcements if the scope of maintenance increases.

    192.150.160.138
    192.150.160.135
    192.150.160.136
    192.150.160.137
    192.150.160.139
    192.150.160.140
    192.150.160.144
    192.150.160.145
    192.150.160.146
    192.150.160.147
    192.150.160.148
    192.150.160.149
    192.150.160.150

    Timeline
    2020-10-08 22:00 UTC: Beginning of maintenance
    2020-10-09 02:00 UTC: End of maintenance
    Oct 5, 22:38 UTC
    Oct 8, 2020
    Completed - The scheduled maintenance has been completed.
    Oct 8, 20:00 UTC
    In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
    Oct 8, 16:00 UTC
    Update - We will be undergoing scheduled maintenance during this time.
    Oct 7, 23:48 UTC
    Scheduled - Summary
    Our operations team will perform maintenance on alerts infrastructure

    Impact
    Assignment of FTP and BGP alert rules created or modified during this time will be delayed by approximately 2-4 hours. This would not impact any existing tests/rules setup for all tests/alert types.

    Timeline
    2020-10-08 16:00 UTC: Beginning of maintenance
    2020-10-08 20:00 UTC: End of maintenance
    Oct 7, 22:12 UTC
    Resolved - This incident has been resolved.
    Oct 8, 09:10 UTC
    Monitoring - We have increased the capacity of the Endpoint check-in service and will continue to monitor the service.
    Oct 8, 07:17 UTC
    Identified - Endpoint Agent check-in and configuration update service has been fully restored. The root cause of the issue was due to scale and load.
    Oct 8, 07:05 UTC
    Update - We are continuing to investigate this issue.
    Oct 8, 06:55 UTC
    Update - Affected scope
    Due to a capacity issue, Endpoint Agents will experience failures when attempting to check-in with the controller.
    All existing Endpoint Agents tests will continue to run as normal. Any new tests or Endpoint settings are affected. We have identified the issue and are working to resolve the issue as quickly as possible.
    Oct 8, 06:51 UTC
    Investigating - We are currently experiencing degraded performance on our Endpoint Agent controller infrastructure.

    Affected scope
    Endpoint Agents may experience a failure when attempting to check-in with the controller.
    Oct 8, 06:27 UTC
    Oct 7, 2020
    Completed - The scheduled maintenance has been completed.
    Oct 7, 23:46 UTC
    In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
    Oct 7, 00:00 UTC
    Scheduled - Due to a change in Service Provider, customers with tests assigned to Cloud Agent ‘Paris, France’ will observe a corresponding change in path-visualization and source IP.

    This maintenance is scheduled to take place on Wednesday, October 7th, between 20:00 and 22:00 UTC.

    We do not anticipate downtime or impact on test performance. The Cloud Agent ID used within the ThousandEyes API will remain unchanged.

    Customers who utilize an IP based ACL to protect targeted services should update their ACL to reflect the following IP addresses:

    185.255.85.224/27
    185.255.86.32/27
    2A02:04BA:0100:0000:0008::/124
    2A02:04BA:0100:0000:0009::/124
    2A02:04BA:0100:0000:0010::/124
    2A02:04BA:0100:0000:0011::/124

    ThousandEyes support is available via in-app chat to provide assistance 24 hours a day.

    Instructions for contacting support may be found in the following article:
    https://docs.thousandeyes.com/product-documentation/getting-started/getting-support-from-thousandeyes
    Oct 2, 23:10 UTC
    Resolved - The incident has been resolved.
    Oct 7, 18:25 UTC
    Monitoring - ThousandEyes operations team implemented a fix and response times are returning to normal baselines. We are actively monitoring the issue.
    Oct 7, 18:08 UTC
    Investigating - Affected scope

    Requests to https://api.thousandeyes.com and https://app.thousandeys.com are currently suffering higher than usual response times and may fail with 5XX errors.

    Status
    The operations team is actively investigating the issue.
    Oct 7, 17:55 UTC
    Oct 6, 2020

    No incidents reported.