Writing For Tech

As my career has progressed, I have discovered writing to be an invaluable skill to develop and polish as an engineer.  The skill of writing well translates to a number of areas outside of tech including things like writing good emails, networking and chat using real time collaboration tools (IRC, Slack, etc.), writing documentation, writing specs, or even just asking for help in online formats like message boards or communicating on social media sites.

For example, when asking for help in a technical public forum, e.g. GitHub issues or Stack Overflow, knowing exactly what the problem you are having and describing it in a way that makes sense to others (who often don’t speak English as a first language) is much more difficult than it looks.  It takes time and practice to learn how to craft questions well and to frame technical problems in easy to understand ways.  In my own experience, people are almost always happy to help but I’ve seen so many bad questions on Stack Overflow.

There are two books that I recently read that have had a tremendous impact on how I think about and approach writing, which has helped me grow as a writer, engineer, and technical collaborator, which I’d like to share with readers today.  These books have been around for a long time so if you’ve already heard of them or it has been some time since reading them, I encourage you to reread or at least skim through them again.

The first book, On Writing Well: The Classic Guide to Writing Nonfiction is a great book and really forced me think a lot about my writing and what I could be doing better.  Instead of focusing on a lot of the mechanics and building blocks of writing (it does touch on these a little bit), On Writing Well focuses mainly on the style and how to make your writing better by making it more interesting and less wasteful.  The book teaches readers that often times, more is less in writing, and it focuses on teaching lessons of simplicity as well brevity, boiling things down to their simplest forms and avoiding certain traps and pitfalls.

The second book is called The Elements of Style, Fourth Edition.  There are some really good tricks and tidbits in this book that 100% improved my writing fundamentals and mechanics, even without much practice outside of reading the book.  I would highly recommend this book for anybody that is interesting in improving the foundations of their writing, from things like improving vocabulary to improving the structure and overall quality of their writing.  The book is fairly short so doesn’t take long to work through and is a great tool for improvement and you will more than likely find some tips that are immediately useful in your own writing style.

Getting better at writing is a process, just like learning any other skill.  The more time you spend thinking about it and practicing, the better you will get.  Obviously in my own personal experience, having this blog has been a great way for me to learn and grow my writing skills.  Not every blog post is a success in my eyes but I have learned lessons from doing things over and over again and discovered things that work or don’t work.

One lesson from On Writing Well that has really stuck with me is the idea that your writing should be written for yourself.  Instead of thinking about things that other people want, or what you think they want, just write about things that are interesting or that have personal meaning and the writing process will be much more rewarding.  Applying this concept makes the process of writing much more enjoyable and keeps the gears turning.

Another idea from the book that stuck with me is that everybody has their own style of writing and none of them are bad.  So if you feel pressure to write or create a certain way, don’t.  Your writing process works best for you and that is fine, you just need to find it if you don’t know what it is already.  One of the most important lessons in writing that I have discovered over the years is that I’m not really interested in writing my blog posts according to any set of formulas or criteria.

In my own writing process, I usually like to find a problem that is interesting or challenging to me, sit down and just start writing.  This process helps me internalize and understand the problem I am attempting to solve better, as well gives me a platform to help others.  I attribute my own process and writing style to a lot of practice and just using the lessons I have learned to eventually build up my own style, which works for me.

Good luck and happy writing.

Read More

Templated Nginx configuration with Bash and Docker

Shoutout to @shakefu for his Nginx and Bash wizardry in figuring a lot of this stuff out.  I’d like to take credit for this, but he’s the one who got a lot of it working originally.

Sometimes it can be useful to template Nginx files to use environment variables to fine tune and adjust control for various aspects of Nginx.  A recent example of this idea that I recently worked on was a scenario where I setup an Nginx proxy with a very bare bones configuration.  As part of the project, I wanted a quick and easy way to update some of the major Nginx configurations like the port it uses to listen for traffic, the server name, upstream servers, etc.

It turns out that there is a quick and dirty way to template basic Nginx configurations using Bash, which ended up being really useful so I thought I would share it.  There are a few caveats to this method but it is definitely worth the effort if you have a simple setup or a setup that requires some changes periodically.  I stuck the configuration into a Dockerfile so that it can be easily be updated and ported around – by using the nginx:alpine image as the base image the total size all said and done is around 16MB.  If you’re not interested in the Docker bits, feel free to skip them.

The first part of using this method is to create a simple configuration file that will be used to substitute in some environment variables.  Here is a simple template that is useful for changing a few Nginx settings.  I called it nginx.tmpl, which will be important for how the template gets rendered later.

events {}

http {
  error_log stderr;
  access_log /dev/stdout;

  upstream upstream_servers {
    server ${UPSTREAM};
  }

  server {
    listen ${LISTEN_PORT};
    server_name ${SERVER_NAME};
    resolver ${RESOLVER};
    set ${ESC}upstream ${UPSTREAM};

    # Allow injecting extra configuration into the server block
    ${SERVER_EXTRA_CONF}

    location / {
       proxy_pass ${ESC}upstream;
    }
  }
}

The configuration is mostly straight forward.  We are basically just using this configuration file and inserting a few templated variables denoted by the ${VARIABLE} syntax, which are just environment variables that get inserted into the configuration when it gets bootstrapped.  There are a few “tricks” that you may need to use if your configuration starts to get more complicated.  The first is the use of the ${ESC} variable.  Nginx uses the ‘$’ for its variables, which also is used by the template.  The extra ${ESC} basically just gives us a way to escape that $ so that we can use Nginx variables as well as templated variables.

The other interesting thing that we discovered (props to shakefu for this magic) was that you can basically jam arbitrary server block level configurations into an environment variable.  We do this with the ${SERVER_EXTRA_CONF} in the above configuration and I will show an example of how to use that environment variable later.

Next, I created a simple Dockerfile that provides some default values for some of the various templated variables.  The Dockerfile aslso copies the templated configuration into the image, and does some Bash magic for rendering the template.

FROM nginx:alpine

ENV LISTEN_PORT=8080 \
  SERVER_NAME=_ \
  RESOLVER=8.8.8.8 \
  UPSTREAM=icanhazip.com:80 \
  UPSTREAM_PROTO=http \
  ESC='$'

COPY nginx.tmpl /etc/nginx/nginx.tmpl

CMD /bin/sh -c "envsubst < /etc/nginx/nginx.tmpl > /etc/nginx/nginx.conf && nginx -g 'daemon off;' || cat /etc/nginx/nginx.conf"

There are some things to note.  First, not all of the variables in the template need to be declared in the Dockerfile, which means that if the variable isn’t set it will be blank in the rendered template and just won’t do anything.  There are some variables that need defaults, so if you ever run across that scenario you can just add them to the Dockerfile and rebuild.

The other interesting thing is how the template gets rendered.  There is a tool built into the shell called envsubst that substitutes the values of environment variables into files.  In the Dockerfile, this tool gets executed as part of the default command, taking the template as the input and creating the final configuration.

/bin/sh -c "envsubst < /etc/nginx/nginx.tmpl > /etc/nginx/nginx.conf

Nginx gets started in a slightly silly way so that daemon mode can be disabled (we want Nginx running in the foreground) and if that fails, the rendered template gets read to help look for errors in the rendered configuration.

&& nginx -g 'daemon off;' || cat /etc/nginx/nginx.conf"

To quickly test the configuration, you can create a simple docker-compose.yml file with a few of the desired environment variables, like I have below.

version: '3'
services:
  nginx_proxy:
    build:
      context: .
      dockerfile: Dockerfile
    # Only test the configuration
    #command: /bin/sh -c "envsubst < /etc/nginx/nginx.tmpl > /etc/nginx/nginx.conf && cat /etc/nginx/nginx.conf"
    volumes:
      - "./nginx.tmpl:/etc/nginx/nginx.tmpl"
    ports:
      - 80:80
    environment:
    - SERVER_NAME=_
    - LISTEN_PORT=80
    - UPSTREAM=test1.com
    - UPSTREAM_PROTO=https
    # Override the resolver
    - RESOLVER=4.2.2.2
    # The following would add an escape if it isn't in the Dockerfile
    # - ESC=$$

Then you can bring up Nginx server.

docker-compose up

The configuration doesn’t get rendered until the container is run, so to test the configuration only, you could add in a command in the docker-compose file that renders the configuration and then another command that spits out the rendered configuration to make sure it looks right.

If you are interested in adding additional configuration you can use the ${SERVER_EXTRA_CONF} as eluded to above.  An example of this extra configuration can be assigned to the environment variable.  Below is an arbitrary snippet that allows for connections to do long polling to Nginx, which basically means that Nginx will try to hold the connection open for existing connections for longer.

error_page 420 = @longpoll;
if ($arg_wait = "true") { return 420; }
}
location @longpoll {
# Proxy requests to upstream
proxy_pass $upstream;
# Allow long lived connections
proxy_buffering off;
proxy_read_timeout 900s;
keepalive_timeout 160s;
keepalive_requests 100000;

The above snipped would be a perfectly valid environment variable as far as the container is concerned, it will just look a little bit weird to the eye.

nginx proxy environment variables

That’s all I’ve got for now.  This minimal templated Nginx configuration is handy for testing out simple web servers, especially for proxies and is also nice to port around using Docker.

Read More

Top Five Reasons to Use a Hybrid Cloud

Hybrid cloud

Guest post by Aventis Systems

Cloud computing is increasing in popularity as business users have become more comfortable with cloud capabilities. According to a recent VMware report, some 15% of workloads currently reside in the public cloud, and 50% are projected to be running in the public cloud by 2030.

Some business customers will move completely to the public cloud, drawn by its ability to help them respond to changing business needs, align costs and stay on the cutting edge of innovation. But a complete move isn’t the best option for all customers. Some processes still simply run more efficiently and securely on on-premise hardware.

For many businesses, there is no one-size-fits-all solution. Ultimately, using a mix of public clouds and private infrastructures is the best way for many companies to make the most of their resources while optimizing performance and productivity.

What Is a Hybrid Cloud?
A hybrid cloud is a combination of a private cloud platform designed for use by a specific organization and a public cloud provider like Amazon Web Services (AWS) or Google Cloud. These public clouds are shared by customers all over the world and are a cheaper alternative to buying physical servers.

Though the public and private cloud platforms operate independently from one another, they can communicate over an encrypted connection.

A hybrid approach enables data and applications to move between the public and private infrastructures. These are independent platforms, so businesses can store protected data on the private cloud while still leveraging applications that rely on that protected data on the public cloud.

In other words, your sensitive data stays out of the public cloud and on the private platform. The challenge is integrating the different public and private clouds and technologies in a way that is seamless for business users.

Here are some of the top benefits of a hybrid cloud approach, according to the experts at Aventis Systems:

1. Workload Flexibility
With hybrid cloud technology, your IT team has the flexibility to match resources with the infrastructure that best serves the needs of your business.

For example, an integrated hybrid cloud approach with VMware Cloud on AWS enables you to decide where to most effectively run workloads based on cost, risk and changing business needs. With the flexibility to move workloads onsite or offsite as needed, IT is able to better serve the business as a whole.

The ability for organizations to easily transition applications without having to re-platform them — along with the ability to effortlessly access and leverage native cloud services — enables businesses to create a flexible infrastructure in a constantly evolving IT landscape. When new technology becomes available or new trends emerge, businesses are agile enough to take advantage of them quickly.

2. Consistency and Scalability
VMware Cloud on AWS enables companies to leverage operational consistency, along with scalability, on one streamlined platform. By maintaining security and networking policies, along with consistent resource utilization both on- and off-premise, businesses can benefit most from a hybrid infrastructure.

Customers can strategically leverage and allocate company resources to get the most out of system functionality, while becoming better positioned for growth. As your business and capacity needs grow, a hybrid cloud infrastructure offers an easy way to scale to fit these complex needs.

3. Improved Security
Maintaining secure customer transaction data and personal information with a hybrid cloud infrastructure also offers a major benefit over an exclusively public platform. The hybrid approach enables specified servers to be isolated from specific security threats by allowing devices to be configured to communicate with them on a private network.

Where some compliance requirements prevent businesses from running payments in the cloud, for example, a hybrid cloud platform allows you to house secure customer data on a dedicated server, while maintaining the flexibility and convenience of online transactions.

4. Maximized Skillsets and Cost Optimization
Not only are hybrid clouds less expensive to manage, with VMware Cloud on AWS, business customers can also reap the benefits of utilizing their existing IT investments.

Hybrid cloud offerings integrate with your existing IT and use many of the same tools as those used on-premise. You can leverage the resources you already have without having to adopt new tools or acquire new hardware.

Additionally, an integrated hybrid cloud approach enables customers to better align their costs to business needs. Upfront costs can be balanced with recurring expenses, depending on the requirements.

5. Innovation
With a hybrid cloud approach, your business will have access to all of the resources on the public cloud without the burden of big upfront investments. With access to all the newest technologies and innovations, you can stay on the forefront of the latest capabilities.

As businesses become more comfortable and reliant on cloud capabilities, more and more companies will look for the right mix of public cloud and on-premise infrastructure models to increase efficiency and performance.

Read More

Remote Jenkins builds using Github auth

Having the ability to call Jenkins jobs remotely is pretty slick and adds some extra flexibility and allows for some interesting applications.  For example, you could use remote builds to call a script from a chat app or from some other web application.  I have chosen to write a quick bash script as a proof of concept, but this could easily be extended or written in a different language via one of its language specific libraries.

The instructions for the method I am using assume that you are using the Jenkins freestyle build as I haven’t experimented much yet with pipelines for remote builds yet.

The first step is to enable remote builds for the Jenkins job that will be triggered.  There is an option in the job for “Trigger builds remotely” which allows the job to be called from a script.

trigger remote builds

The authentication token can be any arbitrary string you choose.  Also note the URL below, you will need that later as part of the script to call this job.

With the authentication token configured and the Jenkins URL recorded, you can begin writing the script.  The first step is to populate some variables for kicking off the job.  Below is an example of how you might do this.

jenkins_url="https://jenkins.example.com"
job_name="my-jenkins-job"
job_token="xxxxx"
auth="username:token"
my_repo="some_git_repo"
git_tag="abcd123"

Be sure to fill in these variables with the correctly corresponding values.  job_token should correspond to the string you entered above in the Jenkins job, auth should correspond to your github username/token combination.  If you are not familiar with Github tokens you can find more information about setting them up here.

As part of the script, you will want to create a Jenkins “crumb” using your Github credentials that will be used to prevent cross-site scripting attacks.  Here’s what the creation of the crumb looks like (borrowed from this Stackoverflow post).

crumb=$(curl -s 'https://'auth'@jenkins.example.com/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')

Once you have your variables configured and your crumb all set up, you can test out the Jenkins job.

curl -X POST -H "$crumb" $jenkins_url/job/$job_name/build?token=$job_token \
  --user $auth \
  --data-urlencode json='
  {"parameter":
    [
      {"name":"parameter1", "value":"test1"},
      {"name":"parameter2", "value":"test2"},
      {"name":"git_repo", "value":"'$my_repo'"},
      {"name":"git_tag", "value":"'$git_tag'"}
    ]
  }'

In the example job above, I am using several Jenkins parameters as part of the build.  The json name values correspond to the parameters.  Notice that I am using variables for a few of the values above, make sure those variables are wrapped in singe quotes to correctly escape the json.  The syntax for variables is slightly different but allows for some additional flexibility in the job configuration and also allows the script to be called with dynamic values.

If you call this script now, it should kick off a Jenkins job for you with all of the values you have provided.

Read More

Copy text to clipboard using Powershell

In day-to-day life as an administrator, it is very common to copy and paste text between files, editors, etc.  There are a few features built into Windows and Powershell that can help make the copy pasta process easier.

The “clip” utility is a tool that has been built into Windows for a while now (I think since Windows 7), which allows you to redirect command output to the Windows system clipboard.  Pairing the clip tool with the capabilities of Powershell piping and you have a very nice way of getting text out of the shell and into a text editor, or wherever else you may want.

The following commands are simple and quick examples of how to pipe output text of a Powershell command to the clipboard.  You can use this method to output text from a file or from another Powershell command.

cat <file> | clip
Get-NetAdapater | clip

Then just hit CTRL+V in notepad (or your favorite text tool) and you will have the output of the previously clipped command.  In the above example, cat is just an alias for Get-Content in Powershell, carried over from the Linux and Unix shells.  Use the Get-Alias command to find at all of the various shortcuts for Powershell commands.

I tend to grab text out of files quite often, so writing out text to the clipboard and pasting elsewhere has saved me countless hours over the years of clicking around in GUI’s with a few keystrokes.  Shortcuts like this are often missed or over looked by admins that are used to point and clicking, so make sure you try it out if you haven’t discovered this trick.

Read More