Dead Man's Snitch

Posts About System Administration

Public API + Change Snitch Intervals Anytime

Public API

Our API is now open to the public! Documentation is available here and your API key is on your Account page. We’d love to know how you’re using it!

Change Snitch Intervals Anytime

Until now, there wasn’t a way to change a snitch’s interval. You had to delete your snitch and create a new one with the new interval.

Now it’s super easy:

Don’t forget—it matters when you ping your snitch for the first time!

Take me to my dashboard

Enjoy!

Posted on 2014-11-07 in api, intervals, sysadmin, updates


Snitch Check-In & Alert Overview

Two questions we often get are:

“I know my job failed but I haven’t received an alert yet—what’s going on?”
“I received an alert that said my job hasn’t checked in, then, ten minutes later I received a second email saying my job checked in. Why is this?”

Both have to do with what Dead Man’s Snitch refers to as periods and their relationship with snitch intervals. Each interval has a period start and end time.

At the end of each period Dead Man’s Snitch looks back to see if the job checked-in during the period. If it did, great. If it didn’t, an alert email is sent.

It Matters When You Ping Your Snitch

The closer a snitch is pinged to the end of a period, the sooner you will be alerted after a job fails. Take a look at this daily snitch. Notice how the email is sent soon after the missed check-in. 

If you ping your snitch after a new period starts and your snitch fails to check-in, you won’t receive an alert email until after the next period has ended. Notice the extra time between the snitch failure and the alert email. 

Follow-Up Emails

After the first alert email, DMS will continue to send one email per failed period until the snitch is paused or pinged again. As soon as the snitch is pinged an email notification will be sent to let you know the snitch is reporting again. 

Check-In History

Once your snitch has been pinged for the first time you’ll be able to view its history on the Activity page. The Activity page shows whether or not the snitch checked-in during each period and the exact time. The most recent period is shown first. For demonstration purposes, the screenshot below has additional notes on the right-hand side to explain the timestamps. For your convenience, we also display check-in times in your current timezone in a tooltip when you hover over the timestamps.

More Control Over Check-In Times

We recognize the current intervals and periods aren’t great for everyone. As we’ve grown we’ve received great feedback and requests for more control over snitch check-in times and intervals. We’re listening and couldn’t agree more. We plan to release both of these updates in the future. 

Take-aways:

  • Dead Man’s Snitch sends alert emails at the end of each period if a job hasn’t checked-in during that period.
  • Ping a snitch closer to the end of a period to receive alerts faster.
  • One email alert is sent per failed period until a snitch is paused or pinged again.
  • An email notification is sent immediately after a snitch begins reporting again.
  • Extra intervals and greater control over snitch check-in times will be released in the future.

Posted on 2014-03-31 in faq, intervals, sysadmin


Creating a Test Snitch

The first thing a lot of users do is create an hourly test Snitch to see how Dead Man’s Snitch works. Don't worry, you can delete it at any time.

1. Name your Snitch and set the interval to Hour. You can set a Snitch-specific email address if it's different than your account email. Click Save.


2. The next page displays your unique Snitch URL and methods for installing it: cURL, email, and Ruby. To kick off the checking process request the URL using any of these three methods. The most common and basic method is using cURL and would look something like:

$ run_backups_or_something && curl https://nosnch.in/c2354d53d2



For the purpose of the test Snitch, cURL only the Snitch URL in terminal or paste the Snitch URL in your browser.


3. Head back to your dashboard. Your Snitch status light should now be a green checkmark. Since you're checking in your Snitch manually it won't be pinged in the next hour (unless you do it manually) and you’ll receive an alert email.


Note: Hourly snitches check in on the hour, every hour. Keep this in mind when you hit the Snitch URL since the first check-in may take longer than you expected. For example: If you hit the URL for the first time at 1:55 PM and your snitch doesn’t check in between 2-3 PM, you’ll receive an alert around 3:01 PM. However, if you hit the URL for the first time just after the hour, say, 2:10 PM and your snitch doesn’t check-in (from 3-4 PM), you won’t receive an alert until 4:01 PM. Since the URL was hit after the hour, the first full period is from 3-4 PM. Your snitch will continue to check-in every hour, on the hour thereafter. This blog post goes into more detail on snitch check-ins and alerts.

Posted on 2014-03-18 in crontab, devops, faq, sysadmin, testing


Agile Planner's Fail-Proof Backups

Over the past couple of weeks I’ve had the pleasure of getting to know the founder of Agile Planner, Graham Ashton. Graham has built a fail-proof database backup routine to protect his customers’ data. Like most sysadmins, Graham takes the security of customer data very seriously, and he’s using Dead Man’s Snitch to keep tabs on his automated backups. In fact, backups are one of the top snitched-on tasks by DMS users.

Here’s Graham’s story:

Agile Planner is a planning tool for teams using iterative development, such as XP or Scrum. Agile Planner helps users turn a stack of estimated cards into a plan they can deliver on. I’m using Dead Man’s Snitch to guarantee that customer data is securely backed up off site, multiple times each day.
The site is hosted on Heroku, which runs on Amazon’s cloud services. While we benefit from the Heroku Postgres team’s expertise to keep our databases running smoothly, there’s a small risk that Agile Planner would be affected by downtime at Amazon. I wanted to ensure my customers would still be able to access their data even if Amazon went down.
My solution was to copy a backup of the Agile Planner database to a dedicated VPS, several times a day. Storing backups on multiple service providers reduces Agile Planner’s dependence on third parties, whose downtime is outside of my control.
Ensuring that the backup job runs successfully multiple times per day is where Dead Man’s Snitch comes in. DMS alerts me promptly if the backup script fails to run successfully. I’m confident that I can quickly move the site to an alternative hosting company should the unthinkable happen over at Heroku.
If you implement something similar on your site, don’t forget to regularly test your restoration procedure so you’ll know your backups will actually work when you need them!

To learn more about Agile Planner, visit www.agileplannerapp.com.

Happy Snitching!

Spencer

Posted on 2013-11-13 in backups, DMS in Practice, heroku, sysadmin