XMPP to Spark Bot - Presence Upgrade and Simplification!

Last September I first published my XMPP to Spark bot based on Nodejs that I created to consolidate my collaboration tools. Over the holiday break I made some upgrades and also simplified or I should say reduced the amount of code and modules required. The official Cisco Spark SDK is pretty large and bulky for the purposes of this bot so in the new version I have created a couple of simpler methods to enable its removal. This bot doesn't use webhooks and a bunch of other Spark API's so it was much easier to just create a couple of new methods. This bot is about getting others to use Spark versus a migration or integration tool but with expanded development it could be made to serve both of those purposes as well.

Below is the main server file. I haven't done a lot of commenting inside my code but you will notice I have a bot monitoring service for some experimenting. I am just monitoring and posting my cloud containers CPU up-time to a bot account in Spark. Eventually I am thinking about sending these updates to another monitoring service which posts updates and alerts on bot issues. You will also notice that I have added the ability to translate presence from Spark into XMPP. It is simple and readable. Presence only updates if my Spark presence/activity changes.It is defaulted on startup to "inactive".

This is a simple and fun personal bot to experiment with. Presence is new inside the Spark API and if your building a personal Spark bot the easiest way to get access is through the "get my own details". The "status" label gives your current presence.

Have fun coding!

Getting Started with Spark API's for Non-Coders

Once considered the domain of the technical elite, application programming interface or API's are now more open and available than ever but it's not the API's themselves that are increasing viability. It is an ever increasing ecosystem of platforms that are helping us plug API's into, together and around us that have changed the way we look at how we can get stuff done and allowing the average user access to the forbidden fruit.

When I recently presented at a conference in Seattle I wanted a group of non-coders to realize that API's are something worth taking a closer look at. I thought what makes more sense than to categorize their use into ever increasing levels of complexity. I come up with four levels.

Level 1 - Easy peasy lemon squeezy (that is British for, "yes, its f@#$%^ easy")

Level 2 - A little harder but my 10 year could still work it out

Level 3 - I might need some help, please standby

Level 4 - Please help me I just had a brain hemorrhage

Level 1 is base level, can't get any simpler out of the box. While most of you might overlook the fact this is using API integrations the Cisco Spark Depot is simple easy to use click and go API and bot integration. There is no coding, you can access it straight from the Spark application from within a room and there is not too much to think about.

Level 2 is getting a little harder by using a API broker. Now when I say harder I mean you have to go outside the application and visit a website. So in reality not too hard. But there are some steps to get you integration working but sites like IFTTT and Zapier make it pretty easy to get your applet/zap/whatever you want to name it working.
  • IFTTT - This is the easier of the two examples to work through. If This Then That makes setting up integrations pretty simple with click and go work flows.
  • Zapier -  Along the same lines as IFTTT but allows more complex work flows and trigger scenario's.
Level 3 is more complex but not beyond the skill level of most IT pro's. While the works flows at level 3 are a mix of drag and drop and there is some coding to create more complex scenario's and integrations. If you want to take a look at some try:
  • Gupshup - This site along with hosting your chatbot across multiple platforms has the ability to also host Javascript code to perform more complex API interactions. While the hosting and initial integrations are taken care of most of the hard work will be done with your hosted code.
  • api.ai - Now part of Google this cool NLP platform has a lot of prebuilt NLP domains for you to use to get you started. It also has the ability to host the integration to Spark to leave you to do the business logic which it has the ability to integrate with using Webhooks. This is close to a level 4 but seeing as it does some of the hosting work for you it's making your life easier getting it a level 3.
  • Built.io - This is an interesting platform that does drag and drop integrations along with the ability to host custom code. It makes your life easier by doing the hosting for you and providing some nice drag and drop integrations. 
Level 4 is custom code. The most complex of the 4 levels that requires a in depth knowledge of programming although you would be surprised what you can write with less than expert knowledge. Some of the interactions with technical non-programmers over the last year or so have highlighted the fact that people just don't know where to start when it comes to this level. If you have never tackled programming before it is a challenging endeavour but not impossible. The fact there is so many paths you can take can put some people off but sticking with it will in the end bring results. Some of benefits and challenges are:
  • Ultimate in business logic customization
  • Ability to use open source frameworks and platforms (Nodejs, Node-Red, Spark Flint, etc.)
  • Access to all sorts of API providers (Google, Facebook, Salesforce, etc.)
  • Language and platform flexibility (Javascript, Python, .net, Java, etc.)
  • Lots of hosting choices (AWS, Heroku, Azure, etc.)
  • Too much choice leading to analysis paralysis.
Having choice of course is a good thing and while level 1 and 2 are much simpler you would be surprised what sort of integrations you can create. Give it a go and it may inspire you to move on to levels 3 and 4.


Using IBM's Watson Conversation API with Cisco Spark in Node.js

IBM's Watson has a number of APIs but the one I am dealing with in this post is the Conversation API which has some interesting features. Even though IBM are clearly still adding features to this fairly new API it is pretty easy to work with once you understand how context can be used. I am not going to go over the basics of the API which if you have never worked with Watson before there are a million You Tube videos on the basic's. Seems even a 13 year old can put a video together on how to use Watson. I did watch this video BTW and its a handy starting place for the basics but one thing that is missing from all the videos is how to use the context data to get information back and forth from your bot to Watson. The video's mention it and the Watson documentation talks about it but it is a little confusing to start with.

What is Context?

Context is a pretty commonly used phrase in natural language processing(NLP) systems, it allows you to pass information back and forth between your bot and the NLP processor, in our case Watson. It does this using a JSON format which can include basically any attribute you want to put into it. Watson does have its own tracking with the conversation_id but tracking of the user and the conversation the user is currently in is up to you. I like to think of context as a cookie that gets passed back and forth. If you are working with a chat app in the browser the browser can take care of the cookie for you. But in our case with Cisco Spark you have no such luck. You as the developer have to track context which is included with every response from Watson. Watson will update the context with information from your users and information Watson uses to track the conversation. Without the mechanism to track the conversation context Watson assumes every transaction is a new conversation and the conversation_id is reset to a new value.

Building a Cisco Spark Bot with Watson

I recently built a Spark bot that started out using an array to track context but I quickly moved to MongoDB database (I wanted some persistence on restarts of node) . Seeing as Watson's context is already formatted in JSON it is easy to slip this straight into Mongo. Below is an example of some of the attributes I add to the context parameter before I send my message to Watson. Email, roomId, user and orderComplete (it is a pizza bot) are all created and added before the first message is sent to Watson. The combination of email and roomId from a Spark point of view makes every conversation unique as well as giving me some handy attributes. I, as a person can have multiple conversations with Watson in either a one-on-one room or multiple participant room so having a way to track that is important.

{ _id: '',
  email: '@cisco.com',
  roomId: '
  user: 'Christopher Norman',
  orderComplete: 'true',
  conversation_id: 'af4487c4-5543-4bf8-ab9c-f4611b3498bd',
   { dialog_stack: [ 'node_9_1475525825835' ],
     dialog_turn_counter: 9,
     dialog_request_counter: 9 },
  pizzaType: 'vegetarian',
  hackerNumber: 5 }

PizzaType and hackerNumber are appended during the conversation by Watson, which I will cover in just a bit. Obviously there are some other attributes Watson will add like what node it is up to in your conversation with node_9 as an example. This is its own tracking system and as long as you relay this back to Watson during your conversation it will keep your place in the dialog exchange. The exception to the this is a flat dialog structure where every node in the Watson dialog is a conversation entry point. In this case you don't care where the conversation is because the dialog has no depth. A FAQ bot where every question is the start and end and there is no follow up question as an example. I was going to say data collection as well but you can still make context updates even in this more simple dialog structure.

Below is a sample of what I used to build the context request using the Node.js Watson module. You may notice this is different than the example from the module. That is because the module example doesn't show you how to pass context through your conversation, this does. Watson's Node.js module is pretty useless without a way to collect and pass context. I worked this out by looking at this code, go look its worth the time. Stefania is an IBM dev evangelist, she has written some great examples over on Github that helped me a great deal. Just remember you have to save the returned context and retrieve what you have saved to send back a reply when your user responds to Watson. Watsons NLP engine will append entity information to context as you dictate but your still need to save it.

If you are interested in what I did to write my context to a MongoDB see my gist below. I am not going to cover it blow by blow but some might find it handy so I have included it for completeness. There is nothing to complicated in there as I replace the previous context Watson sends me rather than update the one in the database. There are quite few fields that require updating in every transaction so replacing context was just simpler than updating fields in the stored JSON record. If you prefer another database go for it this is just a quick solution I put it together in a couple of days. I didn't have weeks to explore the perfect database or why one option may be better than another or even put much thought into the data structure. I am just throwing JSON data chunks into the MongoDB and retrieving them later. I only mention this after reading this article while doing some research. It is not so much the article, which is interesting (although not sure I agree with), but some of the comments. The blogger should have mentioned to check your ego at the door before writing your comment. Is it just me or are people on the Internet snarky? lol.

So now we have covered how to build the context but what else can we do with it? Obviously I am going to use the classic pizza bot example, doesn't everyone. As you build out your dialog you can use Watson to pick out and update information in the context as well as create some more complex logic for dialog decision making. In the first dialog box even though you can not quite see it, I am checking the content of an attribute in my context using $orderComplete.contains('new'). This allows me to see if this person has ordered yet. If they had that same attribute "orderComplete" would have read true and I could have put them in a different thread.

In the first dialog box we ask the user what Pizza type they want. In the second box we are accepting their response, relaying it back and prompting for more info. But first we must record their response in our context. In our case it is the pizza selection using NLP we identify with entities(sorry I never covered this earlier. Now is the time to go read about intents and entities on IBM's Watson documentation site if you don't know WTF I am talking about).

Quick tip. If you need access to info in your context at any stage to respond to your user, using $ will get it for you. Below I added $user to grab the user name attribute from my context to add a personal touch to the interaction but this can be anything stored in the context.

Watson has a very interesting NLP engine that is actually pretty easy to work with once you get past the initial knowledge gap. Not to say there are not a load of other possible alternatives such as api.ai etc. or a node module that does NLP this is just one engine. If you have tried others feel free to comment about it. Always interested to check out new stuff especially if you have done NLP using a node module.

Thanks for reading.


Stop Bugging Me: My life Hack Continues

It has been nearly a month now since I first started my journey and I thought it was time for some reflection on my progress or lack there of (if you would like to skip all this and go down to my summary of what is working go for it). It is an interesting journey that has seen some ups and some downs to be sure but it is mostly positive. Just making a cold turkey change and ignoring all my distractions was a stupid thought I had. I gave up smoking that way but this is far more challenging. Its more akin to getting back into physically working out than giving up smoking.

The Mental Workout

I spent a number of years in the military as a much younger me where I was very physically active. I competed in triathlons, cycled competitively at least at a club level and worked out generally around 12-14 hours a week. I kept a workout journal so I know this is pretty accurate. This matched my military role where I was a electronics technician. Keeping fit is expected and encouraged. I managed to maintain that for a number of years after leaving the service but over time it dwindled. Now I probably do somewhere in the range of 3 hours a week, not including walking for golf etc. Could I go back to 12 hours a week, probably. But it's not something I could do rapidly. It would take time to build up to that level especially with any intensity. (Funny side note at the time I followed a training program put together by disgraced cyclist Lance Armstrong to build up to my 12 hours a week minus the performance enhancing substances. Ah memories.)

Why is this important? It is the same sort of buildup that I am realizing that I need for my mental work outs/focus/deep work (you choose the term). Short bursts of focus or long bursts of focus only provide marginal productivity if I slide back into my old habits which I find myself doing. Its a constant struggle of self-correction. I have been doing a lot of reading on various techniques to improve though and one which seems to be helping me is writing. Writing this blog, writing notes while reading material, writing down ideas etc. seems to be helping build focus. It is such a great creative outlet I really forgot what it can do for me.

Evolving Life Style Challenges

No screen Sundays as I have termed it have been a great success. I am rededicating myself to activities I had pretty much halted like gardening. I really look forward to Sundays and planning what I am going to do now. Whereas before it was nothing special. I think it has helped me feel more refreshed as well for the upcoming week.

Using Alex to wake me, remind me of upcoming meetings and also track my focus period lengths has proven to be very useful. Right now 35 minutes is optimal for focus periods before giving in to the distractions. Who knew 35 minutes could be so long. While I celebrate the 35 minutes which is a 15- minute improvement from my original 20, it's still pathetic. This needs to be much higher with back to back focus sessions. I am incrementally moving the needle 5 minutes at a time and its working. I recently read an interesting post by Cal Newport's Study hacks blog on taking deep breaks, something else I am looking at incorporating into how I extend my focus.

Having a variety of reading material on hand has been very helpful. I have a pretty good selection of reading material and am reading around 4 different books at the same time. I slip focus sessions in between meetings on different material as often as I can. I find studying a broad set of topics is keeping me more motivated than just focusing on one. This may be in part because of how I have previously managed my time. I do find activities such as building a new bot where I can stay focused for hours which is great but that is not going to expand my base knowledge to enjoy greater success.

My mind still wonders but I am finding it easier to pull it back on track. Seems like this has been my whole existence throughout my life I just didn't realize it. I have read a lot of different material on why things bore us. I have come down to two reasons for myself. One which comes from my reading is not challenging the creative side of our minds, obvious. Secondly is my own theory of, am I really bored by the material or task or has a distracted life style persuaded me to think anything that takes extended concentration as boring. Classifying it as boring makes it easy for me to justify not doing it and continue the skimming the web lifestyle. Truly challenging your intellect becomes a tiresome task which is easy to classify as boring. Producing something worthwhile also takes time and focus. It would be easy for me not to triple read my blog post for errors but no one wants to read something carelessly put together (I really had to make sure that there were no errors in that last sentence, it could have been super embarrasing (yes the spelling mistake in embarrassing was on purpose, it is a joke (please don't point out other errors in this post I am a work in progress ( fuck, I am lost in all my brackets)))) .

I found the biggest hurdle to overcoming the boredom challenge is finding where to start. So many of us feel overwhelmed by the volume of a topic that it's so hard to find where to start you just don't start at all. Funny thing is its not until you engross yourself in the environment that you find the right paths to follow. It is okay to not know where to start. Its the acting of beginning the endeavor that will lead you along to find the right path. BTW the person that said follow your passion is full of horse shit. No one ever got their dream job because of passion. Most people don't even know what their dream job is untill they are in it and are experts at said job.

Things I Need to Change

Planned focus time on the morning. Seems most people want to meet earlier rather than later. I just cannot escape that morning routine to clear my inbox either. I do not find this overwhelmingly tiresome but it is not a good habit either. Having filters on my email has certainly helped a great deal. Anything CC'd is in a separate folder and in my mind I have already categorized this as less important so therefor spend less time looking at it. I glance at it and move on typically rarely responding. Probably this is one of the best changes I have made. I spend less than 25% of my time responding to email and messages. This is down by at least 20%.

 I have found my early evening time works best (note - I have no kids) for free time but it is also the lowest energy I have during the day. Afternoon naps are certainly not out of the question but then I find I lose to much time getting functional again after the nap if its to long. So I haven't found the right solution yet.

Realizations and Improvement -

Thinking I could just become some mind ninja overnight - It is a constant struggle to do what is not easy.

Finding the right time to focus - Still working on this part. Is it early morning, late at night or a combo? I haven't found my rhythm yet.

Linking my success to lifestyle change - I have made all these changes when do I see the pay off.

Better planning - I started off doing a good job of this but it seems to have fallen off. I need to better plan my daily schedule. I have fallen back into some old habits of checking email etc. when I could be pursuing something more worthwhile.

Successes -

Writing - I have a bunch of note pads around I am continually scratching in but I have a special note pad for ideas and planning. I just feel good writing. Side effect, my spelling and grammar is improving immensely in general correspondence.

Alexa time management - Now I understand why having a personal assistant is so cool. Thank you Amazon.

Email and messaging management - Getting this under control is key.

Screen free Sunday - Turning off all screens for a day is something to enjoy.

Blogging - Although related to writing, its helping me formalize my plan and ensure continued commitment. Hopefully you will see me add some interesting topics in upcoming posts, that is not to say you will read them but interesting to me at least.

Thanks for reading about my life hack.