IT Automation – What You Should Know

The digital transformation that continues to reshape business and industry practices has resulted in an exponential increase in the amount of digital data that needs to be managed. Information Technology in particular has been dramatically affected and has resulted in a critical need for automation. IT Automation refers to removing the manual execution of various IT related tasks by IT professionals and replacing those processes with sophisticated software tools and suites that can perform the tasks automatically. This automation was usually accomplished via a programmatic backend and more frequently today is imbued with artificial intelligence and/or machine learning algorithms.

There are many advantages to automating IT tasks as long as it is done properly. Repeatable workflows, well documented and strictly adhered to business processes, fewer errors, and more time for skilled IT professionals to focus on more sophisticated tasks that directly drive increased efficiency and productivity are just a few.

Let’s take a look at IT automation, some of the major tasks that are currently being automated, and finish with presenting several of the major players in this space.

IT Automation in Pre-Historic Times

Imagine a time when you couldn’t store every single digital text, email, picture, video, favorite restaurant and music library in multiple clouds and retrieve any one of them virtually instantly. Also imagine a vast dystopian wasteland, devoid of any semblance of society or sophistication. Yes, I’m talking about the early 1980’s. Perhaps a bit overstated, but the truth is, computers used to need to be backed up onto physical magnetic tape – reel-to-reel and later mini-cartridges. These backups took a very long time to run, often required a large number of tapes per backup, needed manual supervision, and were error prone.

Automating this process using a simple shell script and perhaps an automatic tape-loader was certainly one of the earliest, and simplest forms of IT automation.

IT Automation and Orchestration in Modern Times

IT automation has come a long way from those early days and today there are numerous sophisticated tools to help automate IT tasks and workflows. In fact, the term “IT automation” has morphed (informally) into a catch-all phrase referring to all types of business process automation and “IT orchestration”. Perhaps a reasonable distinction would be that IT automation relates to fairly low-level automation of relatively simple tasks, while IT orchestration can be thought of as weaving many IT automation tasks into a larger framework. This allows for the management of hundreds or thousands of servers, many running different operating systems to perform very different tasks (database servers vs web servers vs file servers, etc.), and often spread out among multiple data centers, public, private, and hybrid clouds. Without a coherent, comprehensive high-level orchestrator to automate much of this work, the modern internet could not be managed.

Let’s look at a few examples of modern day IT automation and/or orchestration. In particular, let’s look at the following IT automation sub-classes:

• Workflow Automation / Business Process Automation
• IT Operations Automation / IT Process Automation
• Infrastructure Automation
• Big Data / Hadoop Automation

Workflow Automation / Business Process Automation

Workflow/Business Process Automation is a bit more general than IT automation but is a great place to start our discussion because it deals with general business tasks that are probably more familiar to most of us than querying an SQL database. Most of the tools in this space are based on relatively simple logic flows of the type IFTTT (If this, then that). They are also often designed for non-technical people to use, so they have front-ends/GUIs that allow you to drag-and-drop and fill in simple forms to automate your workflows. No coding needed!

There are many companies that provide workflow automation software, with three of the biggest being Nintex, Kissflow, and Zapier. What sort of tasks might we want to automate? Virtually any task that is clearly defined with pre-determined steps and logic.

For example, Zapier “moves info between your web apps automatically, so you can focus on your most important work.” Zapier has APIs that connect over 1500 common web apps including, Google Docs, Calendar and Gmail, Trello, QuickBooks, Salesforce, Facebook Pages and Lead Ads, Slack, Dropbox, Microsoft Outlook, Exchange and Office 365 and over 1487 more! You can easily connect these web apps and automate tasks using triggers and actions. So you might set up a trigger on receiving an email with any attachments from a certain group of people, and Zapier could automatically download the attachment, move it to your Dropbox account, set up a meeting on Google Calendar, update your social media page, or any other custom action you might need.

IT Operations Automation / IT Process Automation

These two terms act as high-level references to IT automation in general. For the purposes of this article, they are considered to be interchangeable. They refer to taking any IT-related task, and automating it using scripting tools and comprehensive software suites designed specifically to help automate IT tasks. The following represents a sampling of the types of activities IT automation is concerned with:

• Password Reset – Thankfully, this was one of the first IT tasks to be automated. If a user forgets a password, it can be reset with a few clicks with no human intervention needed.

• Automate Service Restart – This task constantly monitors the systems for Windows, Linux, or Unix applications, services and/or processes and intervenes if there are any issues, such as restarting stalled processes.

• Automate SQL Query – Simplifies the process of requesting, retrieving, and processing data from a database server.

• Freeing up server space – Monitors multiple physical and virtual servers for disk space and performs pre-determined actions such as deleting or archiving files or directories or notifying IT personnel.

• Shutdown remote computer – Sounds trivial, but when managing thousands of individual machines, power consumption costs real money. Allows for monitoring computer use and enforcing sleep/hibernation/shutdown schedules.

• Automate VMware Snapshots – Allows for automating maintenance and configuration of multiple virtual machines by simplifying the process of documenting machine state via system snapshots.

• File Monitoring & Automation – Facilitates file and directory management over distributed systems – physical, virtual, data centers, cloud-based. Uses pre-defined and custom scripts to perform repeatable ftp and secure ftp operations and other file transfers, modifications, or processing.

• File and Event Log Monitoring – Event log (and other file) monitoring is a particularly interesting area of IT automation. It removes the tedious and error-prone manual activity from an IT administrator and replaces it with a repeatable, customizable process that can evolve over time. Specific actions can be performed depending on what events are recorded. This allow for effective network configuration management by reacting to problems in real time. An interesting development in this area is the increased use of machine learning for tasks like this. A machine learning algorithm can observe and monitor the event log over time and learn what a “normal, healthy” log looks like. It can also alert IT and/or respond if it notices events in the log that are abnormal.

Infrastructure Automation

Infrastructure as a Service (IaaS) is a relatively new phenomenon and allows companies to offload their network infrastructure configuration management (servers, databases, filesystems, disk storage, computational capability, firewalls, load balancers) to a hosting agent in the cloud. Obviously, this ability to rapidly and accurately duplicate and/or modify hundreds or thousands of cloud-based infrastructures is absolutely essential for any company trying to manage such resources.

Another recently adopted term for this type of automation is “Infrastructure as Code” (IaC) which Wikipedia defines as “the process of managing and provisioning computer data centers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools. The IT infrastructure managed by this comprises both physical equipment such as bare-metal servers as well as virtual machines and associated configuration resources. The definitions may be in a version control system. It can use either scripts or declarative definitions, rather than manual processes, but the term is more often used to promote declarative approaches.”

While the two terms IaaS and IaC are closely related, Infrastructure as Code refers more to the idea of creating/maintaining/modifying cloud-based infrastructures based on well documented pieces of code or scripts. Infrastructure as a Service refers more to the overall cloud-based hosting of a company’s network infrastructure.

Automation of Services and Workloads Running on IaaS

Two related but distinct types of IaaS automation are service automation and workload automation. Automating the IaaS services refers to the setting up of virtual machines (VMs) or cloud instances. Amazon Web Services Elastic Compute Cloud (AWS EC2) has become one of the most popular providers. Their portal allows users to spin up VMs relatively easily and reliably by providing a clear framework for creating, launching, configuring, establishing security policies, and connecting the VMs. A nice step-by-step procedure is outlined here by AWS EC2.

Workload automation for tasks running on these VMs refers to the ability to rapidly (or instantly) ramp up or down resources depending on the requirements of the running tasks. If a particular task finishes and no longer needs access to RAM or disk memory, these resources can be shifted to other tasks when they require additional RAM, disk memory, or even CPU time.

These types of automations are critical for both the companies that provide IaaS services as well as for the companies using the services.

Big Data / Hadoop Automation

Big Data refers to data that is so large and distributed over many computer systems that it simply cannot be processed by traditional data analysis software and techniques. These huge datasets often grow exponentially over time and may incorporate many different kinds and formats of data.

Although Big Data may seem a bit esoteric, it is a natural outgrowth of the digital transformation that has dramatically changed nearly every aspect of our lives. One very clear example of Big Data that has exploded recently is the tracking and management of our individual, personal, (and once private) data. Think about the processing, AI and machine learning that goes into showing you an ad in Facebook about a baldness-treating cream (sorry about that!) you had googled the day before.

In fact, think about all the data that Google, Facebook and others have about you – and it’s only growing larger by the minute. If you’re reading this blog on a Chrome browser, Google already knows about it! Years of every google search, website you’ve visited, hobbies and locations of friends and friends of friends, people you’ve texted, phoned, or face-timed, every single address you’ve ever looked up, every Yelp review, DoorDash delivery and Uber trip and so much more are constantly being mined and updated. As an advertiser trying to optimize each and every ad sent to each and every potential customer, processing this Big Data set and getting actionable results is vital. And Big Data has swamped virtually every industry and business as part of the on-going digital transformation, and the data is only getting bigger! Big Data has already transformed much of the IT landscape and with the explosion of the Internet of Things (IoT), both consumer IoT and industrial IoT, Big Data is not going anywhere. Enter Hadoop.

Apache Hadoop software library was one of the first formalizations of Big Data processing and is still the major player. According to Apache, it is “a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.” While details of the Hadoop environment is beyond the scope of this article, (a great intro video can be found on YouTube here), the network configuration, data staging, retrieval, testing, and verification are best handled using an automation tool.

A few of the major players

There are many companies trying to compete in the IT automation, orchestration, and/or configuration management space with more appearing nearly daily. While your company is unique and must do its own research on which specific tools would benefit them, the following brief list contains a few of the more established companies and tools, with a quick summary from the company about their product:

Red Hat Ansible – Ansible is a powerful open source automation language. Uniquely, it’s also a deployment and orchestration tool. While Ansible provides more productive drop-in replacements for many core capabilities in other automation solutions, it also seeks to solve other major unsolved IT challenges.

Zapier – Zapier will “Connect Your Apps and Automate Workflows.” Easy automation for busy people. Zapier moves info between your web apps automatically, so you can focus on your most important work. Link your web apps with a few clicks, so they can share data. Pass info between your apps with workflows called Zaps. Build processes faster and get more done—no code required

Chef Infra and Automate – With Chef Infra, infrastructure is defined as code, ensuring that configuration policy is flexible, versionable, testable, and human readable. Servers managed by Chef Infra are continuously evaluated against their desired state, ensuring that configuration drift is automatically corrected, and configuration changes are universally applied. Chef Automate provides actionable insights with enterprise scale and performance across multiple data centers and cloud providers.

Puppet Enterprise – Puppet Enterprise combines remote agentless capabilities with robust agent-based solutions to give customers the advantage of automating anything, anywhere while extending changes across their infrastructure at scale.

And finally of course, all of the major cloud service providers, with the top three being Amazon Web Services, Microsoft Azure, and Google Cloud. All of these major providers have their own suite of software and automation tools, and many of the other ‘third-party’ tools integrate to varying degrees with these services.

Rob Stone, Mushroom Networks, Inc. 

Mushroom Networks is the provider of Broadband Bonding appliances that put your networks on auto-pilot. Application flows are intelligently routed around network problems such as latency, jitter and packet loss. Network problems are solved even before you can notice.

https://www.mushroomnetworks.com