Before joining OmniTI, I worked for Blackboard, Inc., as a Senior
DevOps/Systems Administrator within their K-12 Content Management Services Division, with my primary focus on five
legacy products housed in a separate datacenter. At Blackboard, I learned a great deal and been taught how crucial
up-time is to a business’s image and overall success.
The environments I worked with supported thousands of daily visitors and users. This was a huge environment with many
complexities, non-standard oddities, all with the constant threat of being a high visibility DDoS target.
In my pursuit of keeping with Blackboard’s goal of near perfect up-time, I turned a hard situation into a great
learning experience for me. There was barely a day that went by that we are not fixing some new issue, it can be very
overwhelming at times as we were in charge of everything from server layer to storage to network and virtualization,
but also extremely rewarding.
I tell you all this because I want you to understand that I had taken this environment as a personal challenge to
improve my own skills. To that end, I built Ansible, Python, and Bash scripts to automate the environment such as
setting up LDAP on over 200 Linux hosts. I wrote a Python script that self-heals a flaw in one of our High-Availability
file storing solutions. When I joined in July of 2017, I discovered their Nagios instance that had over 600 critical
errors, I fixed all those issues so that Nagios was down to only 2 outstanding problems. When we repeatedly were
taken down by DDoS and had no logging infrastructure to review what occurred, I took it upon myself to install
Logstash and record all traffic, which led to us being able to defend ourselves much more effectively, while shutting
down a lot of dangerous internal communication that did not need to be occurring. Within a few months of working in
this position, I had become a primary source of knowledge and taken on a leadership role within our small group.
There are many people that would look at a situation like this and run away, or bury their head in the sand and just
wait until the product is shut down, but I reject that line of thinking. The only way to make my job easier was to fix
the problems, nothing else would suffice and nothing else would meet the goal of providing as close to perfect up-time
as possible. I look forward to the next challenge out there on my life’s journey.