Author Archives: Mike Chaliy

My article on Medium: Simple serverless data pipeline using AWS Kinesis and AWS Lambda

Example implementation of data pipeline based on serverless offerings by AWS. Data arrive to AWS Kinesis, processed by AWS Lambda and then stored in AWS DynamoDb. Check out “Simple serverless data pipeline using AWS Kinesis and AWS Lambda“.

Advertisements

My article on Medium: Your very own private Docker registry for Kubernetes cluster on Azure (ACR)

Another article about building applications using Kubernetes on Azure. This time about your private payloads: Your very own private Docker registry for Kubernetes cluster on Azure (ACR). Article describes how to create your private Docker registry using ACR, configure it and then provides an example of deploying simple payload.

My article on Medium: Your very own Kubernetes cluster on Azure (ACS)

In the article Your very own Kubernetes cluster on Azure (ACS) I am going to describe how to deploy Kubernetes cluster to Azure using Azure Container Services (ACS). It turns to be quite easy, so don’t hesitate to read and try on your own!

Interesting Findings #3

Python

Quite old news, but I added miniconda to chocolatey. If you need Python on Windows and do not want waste your time to setup wired stuff to compile packages like NumPy, you probably better to use Anaconda or Miniconda which is the same but with smaller number of packages pre installed. You however, always can install pre-built packages with ‘conda install’. So.. to install Miniconda for Python 2.x, use:

choco install miniconda

for Python 3.x

choco install miniconda3

Do not forget, after installation issue

conda update conda

NodeJS

Few weeks ago I completed nano project using NodeJS (express, mongodb, browserify, gulp).

  1. Feels quite good for small fire-and-forget style projects.
  2. Lol, but NodeJS still uses callbacks! Promices are almost here, but… really they are not. Every lib invents own Promices, styles.. and yet they are actually like C# from 2005.
  3. Deployment is a joy.
    1. For development npm start.
    2. For win server, iisnode works really nice. Handles restarts in case of crash. Runs node per processor by default. Nice experience.
    3. For Amazon, Beanstalk allows load balanced, autoscaled, rolling deployment with few command line commands. Super nice.
  4. gulp.watch and Co are great. Browserify, ReactJS JSX, unit tests.. all work when you hit Ctrl+S. Page auto reload when something changed are also very handy.
  5. Time to start from zero knowledge is amazing. Actually after few hours, you have comfortable workflow and running application.
  6. Node.js Tools for Visual Studio – well, I cannot find any need of it. But it works and now it support Visual Studio 2015 (It works with free Community Edition too).

Amazon Beanstalk

May be half a year ago, Amazon eventually implemented something like Web and Worker Roles from Azure.

  1. Looks quite good for NodeJS and Python, but not so good for .NET. For example, there is no Worker Environment for .NET.
  2. Works very well with Docker, I managed to run my small app with Docker on Beanstalk in just few commands.
  3. Supports integration to Visual Studio and have nice command line interface. Quite easy to integrate into build process.

Microsoft

This year Build 2015 is exceptionally interesting.

  1. Docker on Windows. I am not sure how this will change Windows development, but at least Mark Russinovich already made a demo of Docker for Windows. It works. It is really containers on windows. Demo was from non Server SKU. Windows and Linux Docker images will not be compatible.
  2. Docker is nicely integrated in ASP.NET development. New DNX infrastructure is fully supported on Linux, Visual Studio can now package DNX sites as Docker containers and… Visual Studio can debug code remotely on Linux in Docker – sounds crazy, but that was also demoed on Keynote.
  3. CoreCLR is open source. DNX stuff is also open source. Everything works on Linux and OSx.
  4. Visual Studio Code (yet another JS editor, based on Electron), well works quite nice, except I do not understand why not just a plugin for Atom. Marketing BS probably.

Misc

  1. Playing with Splunk. It is super easy to start with and it provides results immediately. The only problem for me is price. I just do not get their pricing model and it looks like quite expensive.  I still hope that I will be able to try it in real environment.
  2. Nice site with ML resources http://www.startup.ml/resources , especially I love their twitter.
Tagged , , ,

Interesting Findings #2

Kinect

Huge disappointment here, because of circumstances that is not under my control, I didn’t got v2. So only v1 real experience for the moment.

  1. As I mentioned before we need some gestures. And we actually did this with some heuristic algorithms. What I can say. People are very different. It is very hard to extract unique characteristics of the gesture. Surprisely, too many gestures are actually conflict with each other. Next time I will probably try to apply some learning or tree based decision making technics.

Kinect 2.0

There were chances to get v2, so I downloaded SDK 2.0. Few interesting findings here:

  1. Skeletons now bodies. Also bodies now include interaction information. This is quite good.
  2. For some reasons there is no prebuilt gestures. Probably MS will include them next time.
  3. SDK now distributed as one package. Developer Toolkit bits are now included in main package.

D3

I tried to use D3. Well… I am not sure that I like it’s API. Some strange intersections with JQuery and AngularJS. I was using D3 to visualize joint points in Kinect. Looks quite nice, but also feels a bit slow. Probably this is by design.

Example snipped that I came up to ensure that all additions/removes/updates are rendered,

SolidWorks

I am on the final stage of my labs project for Lviv Polytechnic. I still hate API. But results looks much better.

[youtube https://www.youtube.com/watch?v=Q46NjJcPZjk%5D

Courses

I am doing https://courses.edx.org/courses/DelftX/FP101x/3T2014/info from edX. It does not take too much time and looks quite promising.

Tagged , , , ,

Interesting Findings #1

Kinect

Last few weeks at ELEKS I am playing with Kinect to help our UX team to implement exciting stuff they need. So my finding here:

  1. It is very easy to work with. No magic, just works. Easy to setup, easy to start.
  2. Kinect SDK is very nice. It has super abstraction of user skeleton as well as raw data. With skeleton it is very easy to implement cool gestures and funny interactions. It took me few days to get comfortable and implement basic stuff with WPF. Microsoft.Kinect.Toolkit and Microsoft.Kinect.Toolkit.Controls provide usable stuff like handling connection to the Kinect, showing skeleton and interaction.
  3. There are many libs that actually work with Kinect. OpenCV, OpenNI and others. Many cool experiments. I have not tried this yet, but it definitely on my todo list.
  4. I was quite disappointed not to find good control sets for WPF and Kinect. There are some controls in Microsoft.Kinect.Toolkit.Controls library, but they are not super polished. Animations, feedback effects, transitions are boring. What is even more disappointing is lack of high quality entertainment controls even just for WPF. For example carousel control. for WPF it is https://wpfcarousel.codeplex.com/ – damn, it looks 80th… Compare to dozens of HTMLcarousels.. And this moves us to next point.
  5. It might look funny, but it is quite easy to use Kinect with HTML. Idea is simple, you have application that actually connects to Kinect and you have static HTML page that talk though WebSocket to that application. You can implement logic on application side and just notify HTML app with changes or you can stream everything form Kinect to HTML page and implement logic in JavaScript. Anybody who concerned about performance of such solution, do not worry, it just works, even raw video(color) stream works fine without any lags. In Kinect SDK there is kinect.js and sample server that actually implement this, but it looks a bit overcomplicated, so I implemented my own, took me few hours of work.

Kinect 2.0

On next week I will have Kinect 2.0 on my hands. I will probably have much more interesting things, but for the moment here is it:

  1. Kinect 1.0 libs were only distributed though SKD download. For Kinect 2.0 many libraries are actually distributed though NuGet. No more Microsoft.Kinect.Toolkit.dll in real projects. Yay.
  2. Kinect 2.0 has gestures support built-in. It provides tooling to teach it with new gestures. I do not know if SDK will have some gestures built in, but this is good start. It uses ML instead of heuristics so possible it will work for not so obvious gestures. Gestures support is actually what I will research next week. Wish me lack.
  3. More points to track. For example it now tracks your big thumb.
  4. Better quality of the depth stream. I hope this will boost implementations of small gesture detection like sign language.

WPF

  1. MVVM is still hot. Also I tried MVVM Light and well… do not feel that it provides more value then it requires to learn. So probably next time vanilla MVVM.
  2. Open source WPF controls in general are very low quality. Many of them is not even on NuGet. After tons of HTML controls and frameworks it looks disappointing.
  3. WebBrowser control does not work with WPF layouts. Disappointing. Awesomium has issues with rendering in full screen at least on my computer. Actually I was not able to find solution that will be able to show local HTML and feel natural to WPF.
  4. Grid is quite ok, but for now CSS grid systems found a way better expression of such layouts. Syntax of CSS grids is more simple and less verbose.

AngularJS

It is first time I work for production AngualrJS. My findings could be naїve, but anyways.

  1. I was able to start quite fast. Two way binding, templates in HTML, ng- attributes looks dirty but they work.
  2. It is more or less easy to learn when you have template or example. For me it is very hard to remember all this directives and assumptions that AngularJS put on you.
  3. Looks like nice and free introduction course for AngularJS on CodeSchool.
  4. Hope AngualJS 2.0 will do better.
  5. There is no EventEmmiter built. Quite sad. And because of lack of normal modules you cannot use something that does not made with AngularJS in mind.

Geograpty

  1. I found amusing where Adelaide located. I failed even with continent.

Misc

  1. I tried to build mock screens with actual HTML. It uses Twitter Bootstrap and few other JS frameworks(e.g. ChartJS). It uses real links for navigation. And result looks very good. You need bower install before user. It even have responsive design! Examples screens in docx(sic!) format.
  2. Solidworks has very bad API for automation for .NET. Full of unobvious states magic strings and so on. And for the end methods with 27 parameters without default values and names like dX1, rQur and so on.
Tagged , ,

Terminal access from Windows Phone to VM somewhere™ in cloud

Suppose you are in situation when you need to edit and then run some Python code, or C++ or whatever. And you need to this from your Windows Phone. What you will do?

Okay, most obvious solution is just to run one of the compliers that send your code somewhere and return result. You can find few of them in store. Most of them are quite low quality, but they work. Unfortunately, this options does not work for me. In my Python scripts I am using NumPy and none of the “compilers” have one preinstalled. So the question is custom environment.

Custom VM

Requirement of custom environment leads to custom VM somewhere™ in cloud. I already have had a few VMs on Azure, so I decided just to fire up another one. I created Ubuntu VM.

image

Through SSH I configured environment:

sudo apt-get update
sudo apt-get install git
sudo apt-get install python python-numpy python-scipy

Sidenote: this might be temporary issue, but. apt-get might fail with some wired 503 errors:

E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/s/suitesparse/libamd2.3.1_4.2.1-3ubuntu1_amd64.deb  503  Service Temporarily Unavailable

E: Unable to fetch some archives, maybe run apt-get update or try with –fix-missing?

well… Azure distribution of Ubuntu, preconfigured with Azure own mirror for packages archive and looks like some packages are missing. To fix this replace all “azure.archive.ubuntu.com” to something like “archive.ubuntu.com” or “ua.archive.ubuntu.com” in “/etc/apt/sources.list”. One-liner do this from askubuntu:

sudo sed 's@http://azure\.archive\.ubuntu\.com/@http://archive.ubuntu.com/@' -i /etc/apt/sources.list

Windows Phone Terminal

Now we have VM in somewhere™ in cloud. Next step is to connect from Windows Phone. Search gives few SSH clients. I tried most of them. They all actually looks the same(I think they use same terminal emulator) and only difference is keyboard. I selected Admin Shell and everything else is just self explanatory.

wp_ss_20140621_0001

Vim works fine too.

wp_ss_20140621_0002

And results:

wp_ss_20140621_0004

Enjoy! Happy cheating on exams!

Tagged , ,

Octave on Windows

For me this is third attempt to complete ML course on Coursera Посмішка, wish me lack so I have time to eventually complete it. In general I prefer to use Python for my studies, I like IPython, numpy and just Python, but for this course Andrew Ng decided to go with Octave, so I need Octave environment.

GNU Octave

Octave does not have Windows distribution on home site. So you need to grab one from Octave Forge. Installation is easy and after all you will have octave.exe on your PATH (if not, just add bin folder to your PATH).

image

There are few items, that I found important for daily use.

– By default Octave starts in non interactive mode. You can type your commands and do math, but first error will terminate process, effectively. You need interactive mode, to do so just add -i when you start octave. In interactive mode you will have more chances to survive Посмішка

– Octave outputs huge welcome screen, after you read it and can repeat it word by word at 3:00 AM, you can turn it off. Just add -q when you start octave. Personally I created cmd file that add both of this params for me.

Unfortunately, console Octave cannot plot graphs inline, this actually means that it shows plot in different window, with quite wired life time. Comparing to the other environments like IPython, wxMaxima or R Studio, I was sure that I will be able to find something like this for Octave. Actually I was wrong, there are nothing useful, but anyway, I spent some time to investigate and will put here aggregated information about Octave environments.

Octave Workshop

imageOctave Workshop is firs item that pops up when you search for Octave IDE. Project seems to be inactive, home site is down. If you will take a look at downloads section and find 5 downloads, it probably all y me.

In general it works. It has panel that shows variables(good), but actually without values(bad). It bundles quite old Octave 2.9.4. And does not plot inline.

Dead end. No value at all.

Xoctave

Commercial IDE for Octave, costs $49 for students.

image

Good things are: has variables view, can show actual values of variables. Uses Octave that installed on your system.

Bad things: on my system does not plot at all, when plot, still does not plot inline or at least to one of the tabs, it plots to separate window, that makes it quite unusable.

Well, my verdict. I would take one for free, $49 is over-over price(take R Studio as base of price judgment).

Qtoctave

It is not in Internet any more :(.

Octave inside IPython

imageYeh, that is true. There is magic that enables Octave in IPython. It uses system installed Octave. It can output variables, it can work almost interactively. And it can plot inline! And of course it can exchange variables with main Python session, so after doing something in Octave you can easily play with results in vanilla IPython.

For me most visible problem with IPython as environment for Octave is requirement to put “%%octave” on every cell that actually octave cell, but I can live with that.

Conclusion

It is obvious, there is not single IDE for Octave that actually meets my requirements. Neither free, nor commercial. Until I will find something useful, I will continue to use console Octave in interactive mode and will also give a try for IPython Octave magic.

Tagged

R Language – Newbie tricks

Disclaimer: I just completed Data Analysis course on Coursera, so this tricks could look very naїve.

First trick is to use RStudio.

Main reason is that RStudio makes it more visible of what are you doing now. Plot something and this plot will be right behind your eyes. Load data and you will see list of your variables. Invoke help and voilà.. help will not go away just because you typed few commands. And what was most important for me is data view, when you can see you data, you can feel it. You can also consider:

  1. You can type View(yourData) to show data view. You can always use something like View(yourData[1:200,]), if there is too much data.;
  2. Ctrl+1, Ctrl+2 will switch you between console and script view;
  3. There is window with history of the commands.

Second trick is not to use data name in models.

It is kind of unreadable when your model looks like:

lm(superData$AgeOfTheElephant ~ superData$SizeOfTheTail + superData$NumberOfLegs + superData$Location)

Better to use:

attach(superData)
lm(AgeOfTheElephant ~ SizeOfTheTail + NumberOfLegs + Location)

Use detach to detach data. Attach works good, you can use it in any operations, but I do not like such side effect heavy tricks. My favorite is just to use:

lm(AgeOfTheElephant ~ SizeOfTheTail + NumberOfLegs + Location, data=superData)

Third trick is to use help for anything.

In console just type “?lm” to get help for lm, if nothing appear try “??lm”, this is full text search for the help. If you load data packages, you can also use help. In most cases help provides valuable information on the structure of the data, its roots or even history.

Trick #4 use RMD.

You might know what Markdown is. If you don’t it is simple write and read friendly syntax for formatted text. RMD is Markdown with R extensions. Very useful to generate report of the analysis.

Trick #5, take a look but do not use ProjectTemplate.

This is project scaffolding system for R. Looks very good, but as for me too havy on newbie stage. Also RStudio have it is own project system, but honestly I do not understand how this two things play together. I hope to play more with this stuff later.

Trick #6, use str.

Most important function.

Trick #7, use http://stats.stackexchange.com/,

ask questions there or on Stackoverflow. There are plenty information about R language on internet.

Trick #8, complete some courses Посмішка,

just now it is Computing for Data Analysis on Coursera (deadline for first week is Mon 13 Jan). In two weeks Stanford will lunch related course: StatLearning: Statistical Learning.

Misc

This is just suggestion, but use some style of the code. In R there are many naming styles, looks quite dirty. So you can follow Google’s R language code style guide as a base.

Also you might need R .gitignore file here you are.

Tagged , , , ,

My Surface RT (Gen 1)

I own Surface RT. This post is all about my subjective opinions about it.

TL;DR;

What I like:

  1. Accounts. It was planned that it will be used as personal device. It does not. So now we have three account. My wife’s account, she writes some stuff, my account – almost for student work and what is more important my daughter’s account. My daughter’s account is most critical. She actually loves to make a mess on start screen, so now she have her own.
  2. Quite simple device. My 5yr old daughter has account, picture password, know how to install games. And finally she trained my 1yr old son, to play games.
  3. Hard to break device. Physically break. With two kids it is very important.
  4. Office and keyboard. Actually we bought this device because wife can write and I can do paper work for my studies.
  5. Also I love that unlike ultrabooks it does not shake when you touch the screen. This is actually what stopping me from buying ultrabook, and why I think that my next home computer will be Surface Pro.

What I would like to have:

  1. More space. I bought 32GB device. It was mistake. With 3 accounts, event with almost no personal files, there is not enough space. Only updates took about 2GB of space. If I will have next Surface it will be at least 64GB.
  2. 3G connectivity. May be quite temporary need, but about 1-2% of time I am spending with Surface RT there is no wi-fi around. And this 1-2% is quite important for me.
  3. More dots on my screen. I do not really need this, but it would be really nice to have.
  4. Cisco VPN client. Lack of Cisco VPN client limits me hardly. There is even PowerShell installed, but I cannot use to access to my working environment.
  5. Fully automatic updates for apps. Why I should click update button?
  6. More games for kids (1-3yro).

With just announced Surface 2, only #2 is fixed. Surface LTE will fix #3.

What I almost do not care:

  1. More apps. Actually there are apps for almost all I need.

What I do not like:

  1. Size of wasted space on my disk. I know technical details, but I do not care. 6GB at max with all Program Files will be acceptable.
  2. Rubbish web cameras, both of them.

Enjoy

Tagged