25 Nov 2013

A Proper Server Naming Scheme 

By - devops 4 Comments

Here at MNX Solutions, we’re busy setting up a brand new data center for our hosted services – MNX.io. A large part of our business involves consulting, which means we’re exposed to a ton of different customer environments and an equal number of schemes for naming equipment…not all of them good. It’s a problem that goes back as far as computers have existed, and everyone has their own opinion on the “best” way to name hosts. Most methods start out fine at the beginning, but quickly become unwieldy as infrastructure expands and adapts over time. Read more

09 Oct 2013

Risks of Running a Blog and How 2FA Can Help 

By - 2fa No Comments

This is a guest post from Mark Stanislav of Duo Security — At MNX we use Duo everywhere we can, and we think you should too.  If your not using 2FA, talk to Duo and start securing all of your password logins!


The conversation always starts the same way, “I think my web site was hacked.” Recently, a friend of mine brought this topic up and I immediately went into incident response mode. After checking out his evidence of the alleged breach, I quickly noted multiple known vulnerabilities afflicting his various WordPress deployments hosted on his server. In his case, he had entrusted some of the administrative duties to the people running those sites. Unfortunately, they weren’t quite as vigilant as he was hoping for in terms of adequate and timely security patching.

Read more

08 May 2013

Golang: Determine a device major/minor number 

By - golang No Comments

While working on a monitoring agent for a service we will be introducing soon. I needed a way to determine the Major/Minor numbers for a device (e.g. /dev/mapper/datavg-srvlv).

It is actually much easier than I though. My first version as a gist.

But a much simpler solution exists, without using cgo.


stat := syscall.Stat_t{}
_ := syscall.Stat("/dev/sda", &stat)
fmt.Println("Major:",uint64(stat.Rdev/256), "Minor:",uint64(stat.Rdev%256))

17 Apr 2013

Amusing a Sysadmin 

By - amusings 2 Comments

As most geek discussions go.. we like to push the envelope — to see what is possible!

As we discussed using chattr to protect a file from being overwritten, it occurred to me that we might perform a weak form of “Inception” on chattr!

So I ran:
Read more

09 Apr 2013

Blocking WordPress Brute Force Attacks against wp-login.php 

By - Apache 11 Comments

We had a support request recently that indicated a server was under heavy load due to a wordpress wp-login.php brute force login attack. This attack was impacting 3 customer servers from more than 500 different IP addresses. We needed a solution fast.

Others on the internet have reported tens of thousands of unique IPs involved in this attack.

There is currently a significant attack being launched at a large number of WordPress blogs across the Internet. The attacker is brute force attacking the WordPress administrative portals, using the username “admin” and trying thousands of passwords. It appears a botnet is being used to launch the attack and more than tens of thousands of unique IP addresses have been recorded attempting to hack WordPress installs.[1]

mod_security seemed like the perfect fit for this type of problem, and we came across a great blog post: http://www.frameloss.org/2011/07/29/stopping-brute-force-logins-against-wordpress/.

A few minor tweaks were required based on different mod_security versions, this seemed to work unless the attacker is only using a single IP address every few hours:
Read more

21 Mar 2012

Develop your Puppet Modules Inside of a Sandbox 

By - puppet No Comments

When you want to develop a new Puppet module, it’s best to work outside of your production environment. You could use Puppet’s built-in environments feature to slice up your site, but there are a handful of caveats and it’s not always the most convenient. Enter Puppet Sandbox…

Puppet Sandbox is a multi-VM Vagrant-based Puppet development environment that can be used for creating and testing new modules independent from your production servers. It’s also a great way to get your feet wet if you’re new to the whole world of Puppet and automated configuration management software. While you could manually configure this type of development environment yourself, Puppet Sandbox makes the whole process trivial, reproducible, and fast!

Ideally, you would first develop your modules using Puppet Sandbox, then move them to a staging environment where they would be applied to a limited number of real hosts, and then finally you would put the modules into production. This type of development cycle will provide confidence that your modules will work as expected, when you need them to.

Here’s a short screencast to give you an overview of how the project works:

Puppet Sandbox is open source, and available on GitHub: https://github.com/elasticdog/puppet-sandbox

29 Jan 2012

Postfix Dovecot User Admin Script 

By - scripts 2 Comments

While working on a postfix/dovecot non-mysql virtual domain configuration, I needed a simple admin script to add new users. The script below is what I came up with after about 30 minutes.

I was unable to find something that met my needs on the interweb, so I hope this post finds someone else in need!

Please submit bugs/suggestions to: nick.wilkens@mnxsolutions.com

Read more

11 Jan 2012

Terabytes on a budget – 106TB for $8788 

By - storage 1 Comment

When working with a client, we were discussing storage for an application they have developed. Storage price was a key component for success of this product. I read the BackBlaze posts here, and here — but I wanted a mix of cost savings, off the shelf parts, and hot-swappable drives.

Read more

13 May 2011

Two Factor SSH Authentication with Duo Security 

By - Linux, puppet, Security 1 Comment

As is probably obvious from our blog posts on Yubikey SSH/WordPress two factor authentication and Google Authenticator SSH two factor authentication, we at MNX Solutions are big proponents of multi-factor authentication schemes to add security to infrastructure. On the other side of things, though, we have to be agile system administrators in order to offer customers efficient and timely service when a problem occurs. While we still really enjoy our Yubikeys (especially for WordPress), we’ve found that the offering from Ann Arbor, MI based Duo Security is the one for us.
Read more

21 Apr 2011

Designing for failure with Amazon Web Services 

By - Amazon 1 Comment

Avoid single points of failure. You can and should assume everything will fail.

Start by listing all major points of your architecture, then break it down further, and then maybe one more level. Now review each of these points and consider what would happen if any of these failed.
Read more