On New Years’ Day, I woke up to 40 emails from Twilio, each with the subject line of, “Your Twilio account has been recharged.” In total, they added up to $4,600 in charges in the span of an hour.
Like most apps, we verify users by sending them a text message. We chose to use Twilio Verify for many reasons, but specifically they say, “Reduce fraud at scale without the workload.” Most startups gladly offload big tasks so they can focus on their core product, us included.
So, when I emailed Twilio, I figured they’d just say, no worries, our bad, here’s a refund. Instead they immediately pointed to their Terms of Service and asked us not to dispute the charges. If we agree, they’ll consider a “partial refund.” They pasted this back to me:
(a) You’re responsible for all use of our services under your account;
e) You will prevent unauthorized access to or use of our services;
I thought this was a really weird response, especially since Twilio spent so many years building up credibility with the startup community. I was a super early user of their products and even won their contest by making an SMS door lock 10 years ago.
They also sent “helpful” ways to ensure we could avoid this in the future. Many of their tips were steps we already took, like throttling requests based on IP addresses, geo-blocking, and limiting the number of SMS calls to a specific phone number.
This was my way of learning that “toll fraud” is a thing and it’s a growing problem that is hitting more and more startups. Talking with other founders, I’ve heard a growing chorus of complaints and Twilio has been reluctant to do anything about it. Our team didn’t even get the worst of it and I was told we, “should feel lucky.”
A quick twitter search shows a few of those people.
When we analyzed the traffic, it was clear it was from a botnet that carefully used both VPN and non-VPN IPs from around the world. They were also careful to only make a few calls from each IP. It was a sophisticated attack that even when Twilio stepped in and blocked a few countries, it still drained the remaining $100 in our account (I turned off auto-reload at this point, which saved us from an even larger drain).
Twilio made a few key errors.
- They added a toggle for “fraud guard” which is apparently a new option, but they did not turn it on by default. So since we started using their Verify product before they added a toggle, we didn’t even see it as an option (and it was off).
2. Their auto-recharge toggle has no way to set a maximum spend or to stop if there is unusual activity. It will happily charge your credit card hundreds of times an hour.
3. Their customer service tried to pass the blame entirely to us and coerce us to not dispute the charge. They said the refund was up to the Finance Team and could take up to 10 days for a decision.
Ultimately, they refunded us $1,700, about 37% of the bill and left us with a $2,900 charge.