Potential Rebilly service degradation due to AWS outage

Resolved
Resolved

Webhook activity resumed. There is a queue of webhooks to process. There still seems to be some latency with that delivery. We believe the service is fully functional now.

Avatar for
Identified

AWS has fixed the underlying network hardware and services are recovering. What does this mean to you?

Our observations are:

  • Rebilly's API had increased latency from ~17:20 UTC to ~20:20 UTC. However, it was functional. Some requests may reach our 60 second time out.
  • Rebilly's webhook requests dropped dramatically at ~15:40 UTC. And then dropped almost completely at ~17:20 UTC. Webhook requests have NOT start to recover yet. Webhooks infrastructure is connected to our main infrastructure through a service which is still not functional.

What is the impact?

If you rely on webhooks to complete transaction processing, your transaction processing will be delayed (until webhooks are delivered). Transactions are being processed though, and you can log into app.rebilly.com and see reports of transactions, including export transactions and other data.

Avatar for
Identified

Looks like we are having some issues with our main service provider (AWS). We have raised an issue.

Avatar for
Began at:

Affected components
  • API
  • Web App (Admin Panel)
  • Checkout App
  • Background Processes