Bash tricks

bash

Update 2/18/18 – add some handy alt shortcuts

Bash is great.  As I have discovered over the years, Bash contains many different layers, like a good movie or a fine wine.  It is fun to explore and expose these different layers and find uses for them.  As my experience level has increased, I have (slowly) uncovered a number of these features of Bash that make life easier and worked to incorporate them in different ways into my own workflows and use them within my own style.

The great thing about fine arts, Bash included, is that there are so many nuances and for Bash, a huge number of features and uses, which makes the learning process that much more fun.

It does take a lot of time and practice to get used to the syntax and to become effective with these shortcuts.  I use this page as a reference whenever I think of something that sounds like it would be useful and could save time in a script or a command.  At first, it may take more time to look up how to use these shortcuts, but eventually, with practice and drilling will become second nature and become real time savers.

Shell shortcuts

Navigating the Bash shell is easy to do.  But it takes time to learn how to do well.  Below are a number of shortcuts that make the navigation process much more efficient.  I use nearly all of the shortcuts daily (except Ctrl + t and Ctrl + xx, which I only recently discovered).  In a similar vein, I wrote a separate post long ago about setting up CLI shortcuts on iterm that can further augment the capabilities of the CLI.

This is a nice reference with more examples and features

  • Ctrl + a => Return to the start of the command you’re typing
  • Ctrl + e => Go to the end of the command you’re typing
  • Ctrl + u => Cut everything before the cursor to a special clipboard
  • Ctrl + k => Cut everything after the cursor to a special clipboard
  • Ctrl + y => Paste from the special clipboard that Ctrl + u and Ctrl + k save their data to
  • Ctrl + t => Swap the two characters before the cursor (you can actually use this to transport a character from the left to the right, try it!)
  • Ctrl + w => Delete the word / argument left of the cursor
  • Ctrl + l => Clear the screen
  • Ctrl + _ => Undo previous key press
  • Ctrl + xx => Toggle between current position and the start of the line

There are some nice Alt key shortcuts in Linux as well.  You can map the alt key in OSX pretty easily to unlock these shortcuts.

  • Alt + l => Uncapitalize the next word that the cursor is under (If the cursor is in the middle of the the word it will capitalize the last half of the word).
  • Alt + u => Capitalize the word that the cursor is under
  • Alt + t => Swap words or arguments that the cursor is under with the previous
  • Alt + . => Paste the last word of the previous command
  • Alt + b => Move backward one word
  • Alt + f => Move forward one word
  • Alt + r => Undo any changes that have been done to the current command

Argument tricks

Argument tricks can help to grow the navigation capabilities that Bash shortcuts provide and can even further speed up your effectiveness in the terminal.  Below is a list of special arguments that can be passed to any command that can be expanded into various commands.

Repeating

  • !! => Repeat the previous (full) command
  • !foo => Repeat the most recent command that starts with ‘foo‘ (e.g. !ls)
  • !^ => Repeat the first argument of the previous command
  • !$ => Repeat the last argument of the previous command
  • !* => Repeat all arguments of last command
  • !:<number> => Repeat a specifically positioned argument
  • !:1-2 => Repeat a range of arguments

Printing

  • !$:p => Print out the word that !$ would substitute
  • !*:p => Print out the previous command except for the last word
  • !foo:p =>Print out the command that !foo would run

Special parameters

When writing scripts , there are a number of special parameters you can feed into the shell.  This can be convenient for doing lots of different things in scripts.  Part of the fun of writing scripts and automating things is discovering creative ways to fit together the various pieces of the puzzle in elegant ways.  The “special” parameters listed below can be seen as pieces of the puzzle, and can be very powerful building blocks in your scripts.

Here is a full reference from the Bash documentation

  • $* => Expand parameters. Expands to a single word for each parameter separated by IFS delimeter – think spaces
  • [email protected] => Expand parameters. Each parameter expand to a separate word, enclosed by “” –  think arrays
  • $# => Expand the number of parameters of a command
  • $? => Expand the exit status of the previous command
  • $$ => Expand the pid of the shell
  • $! => Expand the pid of the most recent command
  • $0 => Expand the name of the shell or script
  • $_ => Expand the last previous argument

Conclusion

There are some many crevices and cracks of Bash to explore, I keep finding new and interesting things about Bash that lead down new paths and help my skills grow.  I hope some of these tricks give you some ideas that can help and improve your own Bash style and workflows in the future.

Read More

Add Office 365 Calendars to Google Calendar

I ran into a scenario recently where I wanted to be able to combine both my work (Office 365) and personal (Google Calendar) calendars, which I found to be a really painful process.  Likewise, the information on the web turned out to be equally painful.  The main motivation for doing this is so that I can sync all of my calendars to my (Android phone), so that when I’m away from the computer I can easily see what is on both my personal schedule as well as the work calendar.

Since I started using Office 365 I have found a number of small annoyances, and this post servers to illustrate a good example of one.  Office 365 is a good product, there are just certain things that are needlessly confusing and complicated as well as a few things that could just work better out of the box.  My opinion is probably also skewed, due to the fact that I haven’t used any of the Microsoft Office online offerings for a while now.

The first step is to publish/export the calendar on the Office 365 side of things.  The setting to export the calendar is completely buried in the O365 web app but can be located by navigating to Settings -> Your app settings -> Calendar

office 365 calendar settings

This should open a navigation menu on the left with more options.  From here you can find Shared Calendars -> Publish Calendars settings.

calendar publishing

From here you can select which Office 365 calendar you’d like to publish and also the permissions to share.  Once those options have been chosen, you can save and a link will be generated for both the HTML and ICS versions of the calendar.

calendar publishing

It turns out that only the ICS link will work on the Google Calendar side of things.  I did have some trouble when I was playing around with the HTML version, which turns out is because Google doesn’t cooperate with HTML calendars, only ICS.

Now you can import the ICS calendar on the Google Calendar side of things.  There are a few ways to do this.  Pull open the settings in Google Calendar and either use the Import & Export feature to upload the ICS file generated above or add the ICS URL to the shared calendar using the Add Calendar -> From URL option.  Note again that the HTML version of the URL won’t work.

import ics file

Using either the ICS web link or importing the calendar using the ICS file should work.  And after adding the calendar in Google you should see all of the events pop up and if you look at your calendar on your phone, the events should be present.  The sync can sometimes take a few minutes, so if they don’t show up right away just wait a bit (or sync them manually) and if you created a new calendar you might need to turn it on.

NOTE: If you add the calendar using the ICS import method, you will not have any way to delete “unmerge” the calendars.  Basically, you will have to manually delete the items that were imported from the ICS file if you want to get rid of them permanently.  If you create a work calendar in Google Calendars and import the ICS there, it will probably be a lot more manageable.

This solutions feels a little bit clunky but it is good enough for me, and there doesn’t seem to be a good work around for sharing the Office 365 calendar with Google.  If there are any good Android apps that can add calendars from different providers let me know, that would probably work just as well but I didn’t really need to look after putting this workaround together.

Read More

Writing For Tech

As my career has progressed, I have discovered writing to be an invaluable skill to develop and polish as an engineer.  The skill of writing well translates to a number of areas outside of tech including things like writing good emails, networking and chat using real time collaboration tools (IRC, Slack, etc.), writing documentation, writing specs, or even just asking for help in online formats like message boards or communicating on social media sites.

For example, when asking for help in a technical public forum, e.g. GitHub issues or Stack Overflow, knowing exactly what the problem you are having and describing it in a way that makes sense to others (who often don’t speak English as a first language) is much more difficult than it looks.  It takes time and practice to learn how to craft questions well and to frame technical problems in easy to understand ways.  In my own experience, people are almost always happy to help but I’ve seen so many bad questions on Stack Overflow.

There are two books that I recently read that have had a tremendous impact on how I think about and approach writing, which has helped me grow as a writer, engineer, and technical collaborator, which I’d like to share with readers today.  These books have been around for a long time so if you’ve already heard of them or it has been some time since reading them, I encourage you to reread or at least skim through them again.

The first book, On Writing Well: The Classic Guide to Writing Nonfiction is a great book and really forced me think a lot about my writing and what I could be doing better.  Instead of focusing on a lot of the mechanics and building blocks of writing (it does touch on these a little bit), On Writing Well focuses mainly on the style and how to make your writing better by making it more interesting and less wasteful.  The book teaches readers that often times, more is less in writing, and it focuses on teaching lessons of simplicity as well brevity, boiling things down to their simplest forms and avoiding certain traps and pitfalls.

The second book is called The Elements of Style, Fourth Edition.  There are some really good tricks and tidbits in this book that 100% improved my writing fundamentals and mechanics, even without much practice outside of reading the book.  I would highly recommend this book for anybody that is interesting in improving the foundations of their writing, from things like improving vocabulary to improving the structure and overall quality of their writing.  The book is fairly short so doesn’t take long to work through and is a great tool for improvement and you will more than likely find some tips that are immediately useful in your own writing style.

Getting better at writing is a process, just like learning any other skill.  The more time you spend thinking about it and practicing, the better you will get.  Obviously in my own personal experience, having this blog has been a great way for me to learn and grow my writing skills.  Not every blog post is a success in my eyes but I have learned lessons from doing things over and over again and discovered things that work or don’t work.

One lesson from On Writing Well that has really stuck with me is the idea that your writing should be written for yourself.  Instead of thinking about things that other people want, or what you think they want, just write about things that are interesting or that have personal meaning and the writing process will be much more rewarding.  Applying this concept makes the process of writing much more enjoyable and keeps the gears turning.

Another idea from the book that stuck with me is that everybody has their own style of writing and none of them are bad.  So if you feel pressure to write or create a certain way, don’t.  Your writing process works best for you and that is fine, you just need to find it if you don’t know what it is already.  One of the most important lessons in writing that I have discovered over the years is that I’m not really interested in writing my blog posts according to any set of formulas or criteria.

In my own writing process, I usually like to find a problem that is interesting or challenging to me, sit down and just start writing.  This process helps me internalize and understand the problem I am attempting to solve better, as well gives me a platform to help others.  I attribute my own process and writing style to a lot of practice and just using the lessons I have learned to eventually build up my own style, which works for me.

Good luck and happy writing.

Read More

Fix Google Analytics search queries in WordPress

I embarrassingly discovered the other day that I was not receiving metrics or analytics about keyword queries in the Google Analytics console.  It turns out that problem was twofold.  First, I didn’t have the SSL version of my site enabled in the Google webmaster tools and second, I was serving a cached version of my sitemap that was several months out of date.

To give you an idea of how this issue manifested itself, and how I discovered that there were issues in the first place – here’s what my search keywords were showing as in the Google Analytics console.

search queries

Clearly the data is less than useful.  The solution to this problem is pretty easy to fix at least.

Fixing the webmaster properties

Open up the Google webmasters site (you should have this setup already, if not go ahead and get signed up and add your WordPress site).  If you have recently updated your site to use https, make sure you add a new property in the webmaster tools for the https version of your site that matches your http version.

Doing this will tell google to keep track of search queries for the https version of your site, which should be the default after swtiching.  After adding the new https property and indexing it, give it a day or two to start collecting metrics, and check back.  Now when you check your search query traffic in the webmaster tools you should start to see all of the search results.

search queries

Also be sure to update properties to use https in both the Webmaster console as well as the Analytics console.  For example, in the Analytics console under Admin -> Property settings -> default URL, there is an option to use http or https.  Likewise, in the webmaster console there is an option for defaulting to http, which is buried in the Google Analytics interface under Admin -> Property settings -> Search Console.  Make sure you update ALL of the site settings to use https.

NOTE: It can take some time for queries to begin showing up in the Google Analytics console (it took about two days for them to start showing up for my site after fixing all the https references).

Fixing sitemaps issues

If you find that Google isn’t indexing and using all of your posts and pages, the next thing to look at the sitemaps.  A quick way to know if you your sitemaps file is doing its job is to pull open the sitemaps, which can be found under the Crawl -> Sitemaps menu.

webmaster tools sitemap

The above shows what a healthy sitemap index looks like (after I corrected the problem).  There is a button located in the top left of this view that can help you test your sitemap while you are updating your settings.  First check for any items in the “issues” column.  Also, if the “processed” date here isn’t recent then there is probably an issue.  One last thing to check – if there are either no entries in this view or fewer then you expect, something is probably not working.

There are many more knobs and dials you can adjust in the webmaster tools, so if you haven’t played with them much I would recommend spending some time and poking around.

I should quickly mention that my solution assumes that you are using the Google XML Sitemaps plugin.  If you’re not using this plugin, and you are either 1) new to WordPress or 2) don’t want to manage your sitemaps file manually, I suggest you enable it.  It makes sitemap management so much easier.

After you have the plugin turned on, navigate to your blog settings for sitemaps, which can be located in Dashboard -> Settings -> XML-Sitemap.  Clicking the popup should bring up a page similar to the following.

xml sitemaps

First, make sure everything looks correct in the settings.  If you are setting this up for the first time you might need to configure some of the settings.  For example, make sure the site name matches the listing, and the options to notify search indexers are all turned on.

When I was troubleshooting the search queries not getting set, I navigated to this menu and immediately noticed that the plugin was showing a warning about using a cached version of my sitemaps.xml file.  To fix this warning, there should be an option to remove the cached versions.

Next, there should be an option near the top of the sitemap settings to “notify search engines about your sitemap”.  After you have adjusted all the sitemap settings and cleared the cached sitemaps file, click that link to trigger a ping to the search indexers.

Be aware that the crawling process may take up to a few days to index and update so be patient.

Read More

Defaulting Google search results to the past year

If you spend as much time looking around the internet for answers to obscure questions as I do, you quickly realize that often times, Google will happily present you with results that are many years old and out of date.  This is especially frustrating when you eventually find an answer after spending quite some time searching, only to realize the answer you’re looking for is 5 years old.  For more reading on efficient searching check out this post on the 10 tab rule, which has some useful ideas for better searching in general.

xkcd

There is a trick that will allow you to customize the Google default results.  The key to mapping the Google search terms is really just the URL.  Google uses search parameters for querying, so you can do some really cool things.  Obviously this is a powerful concept, so a lot of useful searches can be mapped.  This idea can be taken further to map keywords to other searches, like YouTube, Google maps, Stack Overflow, etc, or basically any site that provides a search interface.

I have only tested these key/search mappings for Google search results on Google Chrome, so if you use another browser there might be a similar trick, I just haven’t attempted it.  Open Chrome settings and navigate to the “Search” section.

This will pull up a dialog box with a list of default search engines.  Scroll to the bottom of the list and add the following values to the corresponding fields from the screenshot.

The Keyword is just a “>” symbol, and it can be anything really.  I chose the > because it is quick and easy to get to.  The rest is pretty self explanatory.  After the entry has been added, scroll through the search engines and find the new “Google recent” entry – there is a button that says “Make default” if you hover over the search engine entry.  Click that and then click done.

Now when you do a Google search from your search bar it will default to items from the past year.

Bonus

You can extend this trick further to map keys in your Google search bar to do other searches.  For example, you can map a key (or word) to search for image results.  In the below example, I am using “I” as the keyword.

After adding the above snippet into the search settings you can navigate to the search bar, type in I (followed by a tab) and the term to search for and it will automatically do an image search.  The key to making the mapping being a tab completion in the search bar is the q=%s part in the URL.

The last bonus search that has worked for me is the “feeling lucky” search.

That keyword (I used a tilde) can once again be anything, but preferably should be fast to get reach to make searching easier.

One final note

Sometimes you actually do want to search for results that are over a year old.  This is true of information that doesn’t really change often.  So if you are having trouble finding a website you think should be at the top of the search, make sure that the default search result is set to any time.  Ideally you would make another key mapping to handle this searching behavior.

Read More