I missed the RSA Conference this week for various reasons but I did get to spend the better part of two days out of the office hanging with a group of CTO’s.  The group was a mix of early stage and later stage companies from various industries, which made for some interesting conversation.  One of the sessions I enjoyed being part of was focused around Offshoring.  I had very little to add in this session since we don’t have to deal with this at my company, directly anyway, but I was all ears.

Most of the folks in the room were offshoring at least some of their development work.  Here were some of the key takeaways that everyone seemed to be in agreement on:

  • The quality in India has gone down dramatically over the past 10 years.  Good for fixing small bugs but not for innovative ideas or dealing with big issues.  Most of this is attributed to how competitive the market is in India for good people.  The talented engineers typically jump from place to place for the $$$.
  • Vietnam seems to be ramping up as the new India.  One CTO described it as India 16 years ago with very eager and talented individuals.  The downfall there is language barrier is still high.
  • Greece, Russia, and Bulgaria also got high marks on the technical aptitude and ability to tackle the tougher projects.
  • For the most part, people are not offshoring to save money they are primarily doing it to follow-the-sun.
  • One company (who shall remain nameless) pays the same regardless of where the work is being done.  So if they would hire you to work from India, Russia, or Vietnam you would get paid the same as the engineer in Silicon Valley.
  • They recommend you keep the senior folks who are driving the architecture of the apps and systems local and supplement the lower level with offshoring.
  • Security of the code is not a big concern for most of the companies.  Recommend you work with reputable groups that have good references as opposed to just the lowest bidder.
  • Tight integration with your offshore teams is critical for retention.  Meet with these teams regularly either via video or by getting on a plane.
  • Communication is KING.

I thought there were some interesting points in there.  One of them that was most surprising to me was the “We don’t save money by doing this” comment.  So if you don’t save money, and you just want to follow-the-sun, why not hire some 2nd and 3rd shift developers local to do the job.  The problem there appears to be an educational issue.  Common theme among the CTO’s was that, similar to my last post, hiring is hard.  Kids are coming out of school with CS degrees wanting $100k+ salaries yet they can’t actually write useful code.  They also come out cocky as I heard multiple stores of interviews where recent graduates are basically demanding things of the companies prior to even getting a job offer.  I’m not saying CS grads are useless, there are obviously some talented people coming out of colleges and universities, but the bottom line is that they are having trouble finding talented and motivated individuals.  Often times when they compare the resume from someone local with someone in, lets say Russia, it is an easy decision because the person in Russia has more experience, more advanced degrees and more desire to work.  Too much theoretical teaching going on in the US schools seemed to be a common thread.

So if you are a software engineer, what do you do?  For one, I think you have to realize that you can’t expect to get oodles and oodles of money because you can write “Hello World”.  This is a much different world today than it was 20, 15, 10, hell even 5 years ago.  If I look back to when I graduated from college a few moons ago, if you could write code you could write yourself a paycheck.  I had friends coming out of CS who hated it but knew they would get paid.  I’m not sure that will work today.  It is an industry where you need to have a passion for what you do and how you do it.  From what I hear, too many people are still just looking to get paid.

What do you think?


Hiring is hard


When we had an opening on my staff for a system administrator I thought it was going to be a pretty easy hire.  If you watch the news there are lots of people out of work, that should help right?  It is a great company, that should help right?  It isn’t a super senior level position so should have a fairly wide audience, that should help right?  Pay is good and benefits are better, that should help right?  They would have an amazing boss, ok maybe not but 4 out of 5 isn’t bad right?  😉

Boy was I wrong!  I have been working with recruiting firms, placing ads, hitting up my professional and friend network which has resulted in me looking at 98 candidates.  I haven’t met with all 98 candidates but I have read all the resumes and interviewed about 20.  One guy almost got an offer but with two outs in the bottom of the ninth he got pulled.

I will admit that our interview process isn’t a walk in the park, but it isn’t so bad that people cry when finished (well most people anyway…more on that later).  We have had all kinds of candidates.  Some were smart but couldn’t make a complete sentence.  Some couldn’t make eye contact.  Some couldn’t spell, at all.  Some had NO background in tech!  The list goes on, you get the point but here are some of my favorites:

Candidate 1:  The breakdown

This guy had about 5 years experience in mostly support roles but was in a Junior SysAdmin role.  Resume was good, personality was good, what he wanted to do in the future was good.  I have some basic tech questions I ask all candidates in an interview just to make sure they know their ass from their elbow.  So I ask him to describe to me how DHCP works….couldn’t do it.  I ask him how DNS works…..couldn’t do it.  I asked him a simple logic/troubleshooting question….couldn’t do it.  He looked like he was ready to cry, at which point he asked if he should leave the interview.  Awkward!

Candidate 2:  The people person

This guy was doing some sys admin work at a fairly large company but was also the top-level support guy for the C-Level team.  We get near the end of the interview, which was going ok until he said he didn’t like to do anything that has to do with a computer once he leaves the office. So I ask him a question:  “If you could have any job in the world, what would it be?”  His response was: “Probably something in manual labor because I don’t have any other skills.”  So I changed it up a little, “Assuming you had all the skills, what job would you choose?”  After about 30 seconds of deep thought he then says, “Well I would probably still do manual labor, because if I took any other office type job I would likely need to work with people more than I already do in IT, and I don’t like dealing with people.”  WHAT!?  I thought I was being punked!

It has been an interesting road and I have learned a lot in the process.  My interviewing skills have improved dramatically which has probably been the best result from this whole thing.  I have also learned that there are a lot of people who are in IT that really should not be in IT.  It amazes me how many people have made a career out of this without really knowing anything.  I want to find someone who is hungry.  Hungry to learn, hungry to grow, hungry to be better.  They should want to know the unknown while being honest with themselves on what that means.  I’ve got the buffet, but can’t find anyone who wants to eat.

Thanksgiving is upon us and as I reflect upon all of the things in my life I have to be thankful for one thing is certain, Life is Good.  Sure, there are things that could be better, but when put into perspective I really don’t have much to complain about.  There are people out there with no jobs this holiday season, trying to provide for their families.  There are people who don’t know where their next meal will come from.  There are people, brave men and women, who are over seas fighting for our country who will not see their families and are just trying to make it home in one piece.  I could go on and on with the examples of people who have hardships, more hardships than myself, but lets stop there.  So where am I going this this feel good story?  I am going to the topic that no one can seem to avoid right now, the TSA.

Whether I am on Facebook, Twitter, blogs, around the water cooler, listening to the radio or watching the nightly news I can not seem to escape people bitching about the TSA.  I know this is not going to be a popular sentiment with the readers of my blog, or most of my friends: STFU about the TSA and in particular the TSA Agents.  I get that people don’t like being touched.  I get that people don’t like being seen “naked”.  You have options though, drive or take the train.  Do I think the TSA has gone too far?  To some extent yes, but on the other hand, whether you agree or disagree the goal is to keep us safe.  I’m not going to make this post about whether or not the new procedures are actually helping to keep us safe.  That is a battle for another day.  What I am going to make this post about is how all your bitching, whining, and harassment toward the TSA agents is misdirected.

If you didn’t like a movie that was playing at theaters across the country, would you harass the person selling tickets to the movie?  You know, the kid behind the counter who sells the actual tickets for 8 hours per day, do you harass him?  I hope not.  He is just doing his job.  Whether he agrees with the theater’s choice of showing the movie or not he is trained to stand there and collect money for tickets.  He does it or he loses his job.  I look at TSA Agents the same way, they are doing their jobs.  Don’t like that comparison, ok here is another one.

Do you agree with the war in Iraq?  If the answer is no, do you then harass soldiers when you see them?  What about veterans, do you harass them?  If you do, then you deserve a punch in the face…twice.  You should be thanking them for their service.  You should be picking up their tab at the bar.  You should be glad they are out there doing what we can’t or won’t.  At the end of the day though, these men and women are out there doing their jobs (though a very dangerous one).  Whether or not you agree with the policy that puts them there doesn’t change the fact that they are serving our country and doing what is asked of them.  While I do not put TSA Agents on par with soldiers, I think their goals are the same, to keep people safe.  They are being asked to keep the skies safe by screening thousands of people per day who pass by them.  For the most part, they encounter normal people just trying to get from point A to point B.  Now what about that one passenger who isn’t trying to get from point A to point B?  Would you want the pressure of finding that one in 10 million passenger who’s agenda is to hijack the plan, or worse…blow it up?  I wouldn’t.  There is a lot of pressure that goes along with that job, so for travelers to say TSA agents “take their job too seriously” I’m not sure they understand what it is they are actually tasked with doing.

Lets say the TSA agent doesn’t do his job and lets say he lets someone through who later blows up a plane.  Imagine the backlash from the American public.  Because one guy was not screened thoroughly enough things can be turned upside down.  One could argue that is how we got to where we are today.  What about the agent?  His life will be turned upside down as well, as will every TSA agent currently employed.  It is a thankless job, and it just got a lot harder because now you are harassing these guys and gals.  You refuse to get scanned, then you threaten them about giving you a pat down, then you video tape it, post it on YouTube, and post to Facebook or Twitter for all the world to see.  Thankless doesn’t even describe the position.  For the record, yes there are some TSA Agents who step over their boundaries, but that is true in every profession from cops, to firefighters, to IT folks.  There are always going to be some bad apples.

So I leave you with this:  If you don’t like the policies and want to see them changed, write to your representatives and offer up suggestions on making the policies better but let the TSA Agents do their job and while you are at it thank them for putting up with us.  They want to get in their hours, get home to their families, and collect a pay check like the rest of us.

Hope everyone have a save and happy Thanksgiving.



Once again I made the annual pilgrimage to Las Vegas for BlackHat and Decfon.  As expected it was another great week spent attending interesting talks and hanging out with some of my favorite people…doesn’t get much better than that.  My last day in Vegas somehow got me roped into a fitness challenge, sadly I can’t use alcohol as an excuse.  My buddy Ward decided we should have a little competition to see who can get in better shape for BlackHat 2011.  I’m not one to back away from a challenge so it’s on!  This is going to spread into more of an open challenge for all, but my goal is really just to beat Ward.  Good luck buddy, you are going to need it.

I really wanted to get this post out last night but for various reasons it didn’t happen.  Since a day late is better than nothing, here it goes.  Yesterday Kaspersky announced that the First SMS Trojan for Android has been found in the wild.  Usually this is not something that I would blog about, sms for profit on a mobile device is nothing new.  What I think stands out though is how much easier Android makes this type of attack.  There are very little controls in place to prevent users from installing anything they want, good or bad.  This is kind of the point of Android but I see it as a flaw in the current implementation.  I decided to re-tweet this announcement from Kaspersky and it raised some questions over why I and the security industry make a big deal out of these things.  I get where that attitude comes from, really I do.  From a technical standpoint it isn’t impressive, new, or surprising.  In Vegas, between BlackHat and Defcon, there were a lot of sessions related to Android, so it is expected that there would be malware out there.  However; from a general awareness standpoint, I think it is a valid story.  So why do I think it is valid from an awareness standpoint if it is expected?  Expected by a security or tech guy is different than expected by the masses.  My non techie friends have no idea this kind of stuff is possible unless I tell them or they see it on the news.  Taking that up a notch, I often have discussions about security risks with CxO folks who after the explanation ask “Has it happened?” They want real life examples.  I can talk until I am blue in the face about something that was demonstrated onstage at Defcon or in my lab, but until it happens to someone in the real world and it gets press, it is as if it can’t/won’t happen.  Mobile devices are still looked at simply as cell phones by many, but they are much more.

Now, I’ll admit that this is not something people should freak out about and vendors are going to milk this to try and profit (what else is new).  As I mentioned before, malware for a phone that allows someone to initiate SMS messages and profit from it isn’t new.  As an individual the worst that can happen is you get a big bill and then have the hassle of disputing the charges.  You would probably call Verizon and tell them you didn’t send those text messages they would work something out and you would not pay the full amount.  Now if the malware was smart, it would only initiate a few messages per month and hide on the phone, therefore not raising the eyebrows of most users.  Would you notice a few extra dollars on your personal mobile account or family plan?  What about companies that have corporate plans for employee phones?  They have hundreds if not thousands of phones.  The likelihood of a few premium text messages being caught is low.  I know that at our small company with less than 100 mobile lines that are paid for by the company the finance department would never notice an extra $5-$10 per line each month. It was pointed out that premium SMS isn’t really bad like giving away company secrets, so why are we talking so much about it.

My argument to that is the SMS vector is the quick hit for profit and just the tip of the iceberg.  It takes much less effort by an attacker to write code that will send a text and instantly make money than to invest the time to write much more complicated spy software, grab the data, look for company secrets, and then try to profit from it (more risk too).  That doesn’t mean it can’t be done right, it just isn’t being done yet.  There are a few companies selling this type of spy software today (Flexispy, MobiStealth, and Mobile-Spy to name a few), so it exists.  It requires you to have access to the phone as after installing the application you need to activate and configure it but it will log location data, sms messages, email messages, call logs (some record actual calls).  Sure, turning that into a piece of malware that self activates and configures is more work but I don’t think it is far off.

Make no mistake, I do not think installing an agent from Kaspersky, McAfee, Sophos, Symantec etc. is the answer.  It isn’t the answer on desktops and it will not be the answer on Android and other mobile devices.  We need to treat these mobile devices more like a computer and less like a phone.  A lot of the same protections we use on the laptop/desktop side should carry over, for example:

1.  Better protection for users.  That may take away some of the functionality but have a user mode and admin mode.  Just as you don’t need to run everything as root, users don’t need complete admin access to their mobile device at all times.
2.  Better controls for corporate IT departments.  Allow them to push policies for what can be installed and what can be accessed on the device.
3.  More user awareness is needed.  Many Android users do not understand that the device is really a mini computer that allows them to text and place calls.  They look at it like a cell phone with cool features.  They need to change how they think of the device.

Interested to hear what others think.


Do you remember the days when you would call technical support and actually get someone knowledgeable on the phone who could help you resolve a problem?  I remember a time when this was true, or maybe I just want to remember it that way…kind of like remembering how I used to walk to school, uphill, both ways, in the snow…..barefoot.  My latest support nightmare revolved around trying to get an IP security camera configured.  I took it out of the box, followed the directions….nothing.  I then verified via the DHCP server that it was grabbing an address and it was…good news.   Tried using the software that came with the camera to access it once again….nothing.  Tried accessing the web interface…nothing.  Port scanned the device…nothing.  Power cycle and repeat…nothing.  Break out the paper clip and reset the device…nothing.

Seems like the web server built into the camera is defective, but maybe, just maybe there is some secret piece I am missing.  The call to tech support begins.

Starts out simple enough, give out my name, email address, camera model, and serial number.  The woman, who has a heavy accent (India if I had to guess) is having a hard time understanding me as I spell out the information.  I explain the problem and what steps I have taken to troubleshoot the problem.

Her:  Did you install the software?
Me:  Yes
Her:  Did you search for the camera?
Me: Yes
Her:  Did it find the camera?
Me:  No, that is why I am calling you.
Her:  Is the camera powered on?
Me: Umm…yes it is powered on.
Her: Is it connected to your network?
Me:  Yes
Her:  With a cable?
Me:  Yes with a cable…it is blue in case that matters.
Her:  Lets uninstall and reinstall the software.
Me:  I already did that.  It didn’t make a difference.
Her:  Let me know when the software is uninstalled.
Me:  I already uninstalled and reinstalled, it didn’t make a difference.

Silence for about a minute.

Me:  Hello
Her:  Is the software uninstalled yet?
Me: ummm…sure…I mean yes, yes it is.
Her:  Ok please put the CD in and install the software again.
Me:  Reinstalled….same problem.
Her:  That was fast, what software did you install?
Me:  My computer is super fast.  I installed the config software.

At this point she has me repeat the search process using the software.  Still no dice.  I explain that I know it has an IP address on the network.

Me:  Is there a way to configure this camera without using the software?
Her:  Lets reinstall the software again.
Me:  Do we have to use this software for the initial config or is there a special admin page we can connect to on the device?
Her:  Lets do a reset of the device while you reinstall the software.
Her:  Yes, the software is required for initial setup.

30 minutes later after rebooting, resetting, changing cables

Her:  Ok, the software is not working.  Lets configure this manually.
Me:  You mean configure without using the shitty software?
Her:  Yes, please open Internet Explorer and type…
Me:  Grrrr…..

That didn’t work either.

Her:  What do you see for lights on the device?
Me:  I see a solid orange light.
Her:  Do you see a green light?
Me:  No, I see an orange light.
Her:  Is there a green flashing light?
Me:  All I see is a single light.  It is ORANGE, not GREEN, and it is not flashing it is solid.
Her:  Please unplug the power to the device, I will tell you when to plug it back in.
Me:  Ok
Her:  Did you unplug it?
Me:  Yes
Her:  Did you plug it back in?
Me:  No, I was waiting for you to tell me when to plug it back in.
Her:  Good
(wait 20 seconds)
Her:  Did you plug it back in yet?
Me:  Still waiting for you to tell me.
Her: Good
(wait 30 seconds)
Me:  Do you want me to plug this in yet?
Her:  Oh yes, please plug it in now.

The device goes through it’s boot process and when it settles down.

Her:  What do you see for lights?
Me:  I still only see an orange light.
Her:  Do you see a green light?
Me:  I only see an orange light.
Her:  Is the green light solid or flashing?
Me:  OMG

In the end I just told her to give me the case number and I would get a new camera because as much fun as this phone call has been I can’t take it anymore.

New camera arrived yesterday and appears to be working just fine….for now.