Server Status

Summary:

  • A Bell Canada DSL line issue has taken the network offline. A ticket has been opened with the ISP and is awaiting resolution.
  • Partial access to some services is currently being established. Some services will begin to slowly come online via a cellular backup link.
  • Received emails during this period may have been returned to sender due to the extended outage -- please request emails to be resent once service is restored.

Overall status:

  • Note that most services are SLOWER than usual since they are running over a cellular link. Your patience is appreciated -- we're working on the problem.
  • Websites: Most are offline. In progress.
  • Outbound email: Now functional. If you have a domain name hosted you may need to add mx2.justinho.com as a MX DNS record if it is not already there. Email me if you need help.
  • Inbound email: Online.
  • Mobile push email: Mostly online, depending on client configuration. Not all phones will be able to connect. Add JUPITERSTATION as the domain or as a prefix to your login name (i.e. JUPITERSTATION\your-user-name)
  • Web access: Online at https://mail.justinho.com/owa . Authenicate using JUPITERSTATION\your-user-name.
  • Blackberry BES Push email: Online.

Log:

  • DEC-29-2011 16:21PST -- ~5% packet loss being reported from monitoring, TekSavvy discovers lantern not mapped. Opened ticket with Bell.
  • DEC-31-2011 07:55PST -- Site goes offline. Another ticket opened shortly thereafter.
  • DEC-31-2011 11:20PST -- No update. Still waiting.
  • DEC-31-2011 13:34PST -- Bell confirms that line was incorrectly modified.
  • DEC-31-2011 22:48PST -- No update.
  • JAN-01-2012 06:05PST -- Still no service.
  • JAN-01-2012 14:23PST -- A backup cellular link is now being established to temporarily connect services. No ETA at this point, but hopefully things will be back online (but slow!) in the next few hours.
  • JAN-01-2012 23:43PST -- We weren't able to provision service on a long weekend, but will get this backup link established in the future.
  • JAN-02-2012 09:25PST -- A technician is onsite now to swap the modem to 'resolve' the first lantern not mapped problem. Site remains offline.
  • JAN-02-2012 21:20PST -- The backup cellular link is now routing some traffic. We are gradually re-routing some services via this (slow) link to bring email and some websites online at this time.
  • JAN-03-2012 06:05PST -- Ticket being routed to technician again.
  • JAN-03-2012 07:30PST -- Bell ticket is being worked on by tech, expect fix between 11am - 3pm today
  • JAN-03-2012 15:05PST -- Problem still not fixed. No update.
  • JAN-04-2012 06:01PST -- Update from Bell: Line mapping incorrect on ATM link, being fixed, ETA 08:00PST (11:00EST).
  • JAN-04-2012 07:56PST -- DSLAM reconfigured, ATM group believes everything should be working. Still no connectivity.
  • JAN-04-2012 08:21PST -- Still no connectivity. Power cycled modem again, and called ISP again to escalate the issue. DSL was moved to gigE port on Bell's side, off ATM links.
  • JAN-04-2012 09:25PST -- Service has been restored. DNS entries are now being updated to point back to the main DSL link -- expect full service restoration in the next 20-30 minutes while all routing entries are restored to normal configuration.