What happened?
On Sunday 9th September our monitoring noticed that the Gateway site had become unavailable. Analysis provided by monitoring narrowed the problem down to the DNS not resolving correctly; our entire lncn.eu domain had been rendered inaccessible. The root cause of this was a DDoS attack against our DNS providers for the lncn.eu domain, PointHQ, causing their upstream provider to temporarily remove all their servers from the routing pool to mitigate damage.
What was affected?
PointHQ provides DNS services for the lncn.eu and lncd.org domains, meaning that any services using these domains were potentially unavailable. The most visible of these services are the lncn.eu address shortener, and the University’s Gateway service, although other services were affected.
As DNS caches expired any services using the affected domains were left unable to be resolved by end users, meaning that the services were inaccessible. Since gateway.lincoln.ac.uk implements a redirect to a lncn.eu subdomain, accessing Gateway using this domain was also affected.
What was done to fix the problem?
PointHQ were working to mitigate the problem throughout its duration, and DNS servers were restored later in the day. Essential records from the lncn.eu domain were also duplicated on the Rackspace Cloud DNS service, with the Rackspace DNS servers being added to the lncn.eu domain record to serve as a backup in the event of PointHQ becoming unavailable again later.
What is being done to stop this happening again?
- The lncn.eu domain will retain at least one backup DNS server in its record, protecting essential services against a single failure.