A Stupid Idea

“Is this a stupid idea?” the young software engineer asked.

His idea to build a more abstract testing framework was not stupid. It also wasn’t unique.

My experience is the desire to build highly abstract systems is very popular. As a Software Engineer specializing in Test Automation over the last 15 years, I’ve found it common. Avoiding mundane work (like writing test cases) is a keen focus for devs, especially for the smartest software engineers — and those earliest in their careers.

The problem in this case, is that I’ve seen exactly what he was suggesting – and I’ve tried it myself – at least a dozen times. I’ve seen it absorb at least 3 years of software engineering salary in the last decade alone. In fact, avoiding activities like this is exactly why companies pay me – to avoid wasting their money.

I had 2 options in responding to him:

1) Tell him it won’t work, likely demotivating him and encouraging him to ignore me from now on — he was going to run with this idea anyway OR

2) Find a way to make sure he had my experience in mind while attempting his idea

“Actually, I think it is a very smart idea,” I said — and I meant it. What I didn’t say is that sometimes the smartest ideas make the least business sense.

There are many, many smart people out there focused on abstracting test automation frameworks. Some have spent their entire careers on it. So what sense would it make to use the time and salary of an application developer to focus on creating yet another piece of software for testing?

Sure, he may create something that succeeds in its purpose — although I find it more likely that he’ll end up using the time to learn all the ways not to do it. Further, someone is going to have to maintain that software over time.

And during this effort of a smart idea, who will be writing the application code the company uses to pay his salary?

My experience is the best test cases are those that are concrete. If I can look at the name of a failing test case and know what part of the system is causing trouble, it saves me the time of having to debug my system, learn the test framework, and verify the environment is setup correctly.

I told him as much – my experience – the commodity his superiors were purchasing from me. I encouraged him to try it as a quick proof of concept and hoped it would force him toward experiencing the warning I raised.

I felt our relationship as teammates and his motivation were worth a day or two of creating a proof of concept. My experience tells me it’s not worth pursuing after that, but that’s not my call.

Smart ideas don’t always make business sense. They can be headed off quickly – but sometimes there’s as much of a hidden cost to mishandling a smart idea as there is to pursuing it when it doesn’t make business sense.

Thanks, Code Providence for the opportunity to guest blog. [Anytime, Paul! – editor]

Paul Merrill (@dpaulmerrill) is a Software Engineer who specializes in Test Automation. He likes writing code when it makes sense and then he uses Java, Javascript and Python – sometimes with a sprinkling of Ruby on top. He’s built the company he founded in 2009, Beaufort Fairmont, as a solution for companies that want help automating or learning to automate testing. He uses a variety of open source tools (like Cucumber & RobotFramework) to decrease the length of test cycles and provide quality feedback to devs and stakeholders. Learn more at http://beaufortfairmont.com or https://www.linkedin.com/in/dpaulmerrill

Beginner’s Guide to Workplace Safety – Rails 2.3 Cross Site Scripting

Brakeman generates a “Cross Site Scripting” warning when it detects unchecked, user supplied values into view templates. Attackers inject their content on your pages with Cross Site Scripting (often abbreviated XSS).

Perhaps your application does something like this in a view template:

<% entries.each do |e| %>
  <tr>
    <td><%= text_field_tag 'entry_value_' + e.id.to_s %></td>
    <td><%= e.some_attribute %></td>
    ...
  </td>
<% end %>

The problem is that your template allows whatever value e.some_attribute to be displayed. We’ll return to the potential for misuse in a minute.

There’s another possible scenario for XSS. What if your application relies on unchecked controller parameters?

This query returned <%= params[:entry_count] %> of <%= params[:entry_total] %> total items for this query

In both these cases it’s possible for a client with evil intent to put what they want on your application’s page!

This looks pretty tame:

<script type="text/javascript" >alert('Im in ur base, gettin ur dataz');</script>

An attacker may also use document.write inject HTML img tag to redirect to URL they control. It’s possible for them to sniff cookie data and assume control of user sessions on your application.

There are many avenues of abuse here. None of them are good for you.

HOW DO I FIX THIS?

You can strip (or “escape”) html code from strings with the h method.

This query returned <%= h(params[:entry_count]) %>

“Escaping” a string strips HTML tags. The string “<script>” becomes “&lt;script&gt;”, for example.

Another option (that required a little bit of work) is to install the rails/rails_xss plugin. The plugin will change the default behavior of strings:

Strings now have a notion of “html safe”, which is false by default. Whenever rails copies a string into the response body it checks whether or not the string is safe, safe strings are copied verbatim into the response body, but unsafe strings are escaped first.

Further reading:

Other “Workplace Safety” articles:

Beginner’s Guide to Workplace Safety – Rails 2.3 File Access

“File Access” is another warning category generated by Brakeman. “File Access” refers to a security vulnerability where attackers may gain access to files otherwise inaccessible.

Imagine your Rails application does this:

file = File.open(@somemodel.someattribute)

or perhaps

file = File.open(params[:somepath])

In the both cases you are opening a file based on the completely arbitrary value supplied by a client. Depending on what you do with the file, such as displaying it in a view, you might expose data to an attacker.

The danger here is that your application is not checking the validity of the input. An attacker can easily probe your system using your application just by experimenting with different inputs.

Maybe your Rails application does this:

url = URI.parse(params[:file])
http = Net::HTTP.new(url.host, url.port)
resp = http.start() { |http|
  # contact some third party service and pull it down
  http.get(url.path)
}

The important thing to note: the application opens an HTTP request to whatever value the attacker supplies as a parameter for params[:file].

In this case the params[:file] value is assumed to be a valid URL. Maybe it’s one that is supplied another controller in the same Rails application. Or the client is supposed to behave reasonably, sending a URL to a resource that the Rails application expects to work with.

Unfortunately, that’s not always going to be the case.

Let’s pretend that someone wants to make mischief for your application. What if they supply a URL for a very, very large file (perhaps digital elevation model data sets in 80MB range)?

If you application naively accepts any input, you are asking for trouble. In this scenario, an attacker could easily drown your application with requests that tie up your application’s request threads. Soon your web application will no longer be able to service real requests – they’ll be be busy fetching bogus URLs.

What if your application simply displays file contents? Or deletes files?

Whatever your application does, it certainly should not open files or URIs based on an unchecked input.

HOW DO I FIX THIS?
Instead trusting parameter hash values from clients, use only relative paths or paramter strings for well-known URLs.

Let’s look at the HTTP example with some changes:

uri = "http://somethirdparty.com/"
special_uri_path = "/special_files/" + params[:file]
url = URI.parse(special_uri_path)
...

While not ideal – the application still allows the client to influence the location of the URI, it’s better than blind trust.

Further reading:

Other “Workplace Safety” articles:

Beginner’s Guide to Workplace Safety – Rails 2.3 Mass Assignment

We’ve been using Brakeman to check our Rails 2.3 code for potential security problems. Let’s take a look at the output, this time at the results that look like:

High	UsersController  create  Mass Assignment	 Unprotected mass assignment near line 96: User.new(params[:user])

A “Mass Assignment” warning refers to the assignment of values to an object from a “naked” input. In this case, it looks like the input is actually a params hash value. What wrong with that? Isn’t that darned convenient, slapping together a record without explicitly assigning values for each field? The problem is the model itself. Let’s take a closer look at our model!

create_table "users", :force => true do |t|
    t.string   "email"
    t.string   "crypted_password",   :limit => 40
    t.string   "salt",               :limit => 40
    t.datetime "created_at"
    t.datetime "updated_at"
    t.datetime "deleted_at"
    t.boolean  "admin",          :default => false,     :null => false
    ...

You can see the “admin” field at the bottom. You probably have the admin field act as a flag to the system to indicate if a user can do, well, admin things, like create or delete users. Or look at orders. Or addresses and phone numbers. The sort of personally identifiable information that, if leaked, would be a problem. Use your imagination.

When you allow mass assignment of fields, an attacker may send you anything in the parameter hash.

 { "email" => "uptonogood@teehee.com", "phone" => "12345", "admin" => "1", ... }

Think of all the help you’ll get with new admins in the system!

HOW DO I FIX THIS?

In Rails 2.3 you can protect yourself from mass assignment by labeling model attributes that may be mass assigned with attr_accessible.

class User < ActiveRecord::Base
   ...
   attr_accessible :email
   attr_accessible :name
   attr_accessible :phone
   ...
end

This prevents mass assignment on the admin attribute and any other you didn’t explicitly label with attr_accessible.

You might choose to use attr_protected, instead. This indicates that every attribute except those labeled with attr_protected can be mass assigned.

The lesson here is: don’t trust unchecked input.

Further reading:

Other “Workplace Safety” articles:

Beginner’s Guide to Workplace Safety: Rails 2.3 SQL Injection

In a previous article, you read how random strangers were invited to run whatever command they wanted on your production systems.

Today, you will learn how you invite them to administer your database.

Look again in your Brakeman output and you might see:

High	OrdersController	show	SQL Injection	Possible SQL injection near line 87 ...

First things first – what does “SQL Injection” mean?

When an attacker uses “SQL Injection” they attempt run SQL statements or alter SQL statements in your code. They do this by manipulating input parameters that your code reads and then uses in ActiveRecord methods.

Maybe your code does this:

@user = User.find(:first, :conditions => 'id = ' + params[:item])

What’s wrong with this? This looks harmless, right?

Let’s take a look at what is generated when this is executed in Rails 2.3.18. (You are running at least 2.3.18, right? For that matter, if you are still using Rails 2.3 you should consider upgrading to a long term support version.)

The expected input appears to be an id, or an integer. Let’s say the value of params[:item] is “23”. The statement, when evaluated looks like:

User.find(:first, :conditions => 'id = 23')

An attacker (or any misinformed client) could issue a request that uses a weird parameter like “jones!” or “-12#2$%”, right?

User.find(:first, :conditions => 'id = jones!')

This may generate an ActiveRecord exception. But the potential for trouble is much, much deeper.

Keep in mind that ActiveRecord translates the
find methods into SQL statements. In this case the SQL statement looks like:

SELECT * FROM `users` WHERE (id = 23) LIMIT 1

The record returned (if it exists) should have an id 23.

#<User id: 23, email: "someone@mycompany.org", crypted_password: "...", salt: "..."

The problem is that the use of the “naked” input parameter allows for any value. Why is that a problem? Let’s take a look at what happens when the

params[:item]

is “23 OR 1=1″.

The SQL ActiveRecord generates uses the same “naked” input as strings for the query. Your query is now

SELECT * FROM `users` WHERE (id = 23 OR 1=1) LIMIT 1

By injecting the OR clause into the statement the query now returns the the first record with no other conditions.

#=<User id: 1, email: "suzy.admin@mycompany.com", crypted_password: "...", salt: "..."

That should terrify you.

If you allow unchecked parameters into your ActiveRecord queries, you also allow anyone to commit untold horrors to and with your data.

 

HOW DO YOU FIX THIS?

A simple remedy is to use a conditional array. The Ruby Guide for 2.3.11 says
“Putting the variable directly into the conditions string will pass the variable to the database as-is.”
In a conditional array the “naked” parameter input will properly handle the value of params[:user], for example:

User.find(:first, :conditions => ["id = ?", params[:user]])

generates

SELECT * FROM `users` WHERE (id = '23 OR 1=1') LIMIT 1

In this case, we’re using MySQL, so the condition

(id = '23 OR 1=1')

does not evaluate the “OR 1=1″ as a WHERE condition, but as extraneous string data.

Further reading:

  1. Brakeman SQL Injection Warning
  2. Ruby Guide for Rails 2.3.11 – Conditional Arrays
  3. OWASP SQL Injection Prevention Cheat Sheet
  4. Rails SQL Injection Summary
  5. Avoiding Rails SQL Injection
  6. Rails SQL Injection vulnerability: hold your horses, here are the facts

Other “Workplace Safety” articles:

Beginner’s Guide to Workplace Safety: Rails 2.3 Command Injection

One of the best ways to reduce the number of business-destroying security defects in your Rails code is to review it. The second best way is to lean on automated checks.

Luckily, the Brakeman gem supports automated static code analysis.

To use brakeman install it and then run it in the root directory of your Rails application source code. It’s pretty simple.

 

gem install brakeman
brakeman

 

If you’ve run the Brakeman tool you probably have a list of security warnings. Some of your output might look like this:

| High       | CommentsController     | update | Command Injection   | Possible command injection near l>>
| High       | ConnectionsController  | create | Command Injection   | Possible command injection near l>>
| High       | ContestEntryController | update | Command Injection   | Possible command injection near l>>
| High       | CoordinatesController  | show   | Command Injection   | Possible command injection near l>>

 

If you are unlucky, there are hundreds of these warnings.

What does “Command Injection” mean, anyways? Command injection means that an attacker takes advantage of your Rails code to run commands they choose on your system.

If that doesn’t cause a shiver to run up your spine, you aren’t sufficiently paranoid. Maybe you are running commands in controller code that looks like

system(cmd)

or maybe

logger.info %x[#{cmd}]

 

At first glance this doesn’t look dangerous. You probably wrote the logger.info code so you can log the output of the “%x[…]” command (which returns the output as a string), something that system doesn’t do.

Your Rails application will attempt to run the cmd with the same permissions as the owner of the Rails process. (For Rails 2.3.x this is the owner of the config/environment.rb file. This is one reason why running Rails as a super-user privileged account – like root – is discouraged.)

The problem is that by running an arbitrary command, you make it possible that the string you pass to system or the %x won’t be what you think it will be.

What happens if you are building a command string from parts of a parameters hash?

 

cmd = "ls -al #{params[:user_id]}"
logger.info %x[#{cmd}]

 

What happens if params[:user_id] is something really sneaky?

Let’s pretend that an attacker provides the value “|rm -rf /” for the user_id parameter? The command will be executed as

ls -al | rm -rf /

Deleting all the files on your production host would be inconvenient, at the least. It could potentially end your business if you are not prepared for a total data loss.

What can you do to protect yourself?

One way is to not allow untrusted inputs as parameters to Ruby system calls. Instead of using

system(cmd)

try

system(cmd, arg1, arg2, ...)

The arguments after the command are not interpreted as arbitrary commands. The |rm -rf / is not executed.

What if you are using logger.info %x[...] you need to do a little more work.

If you are looking to record the output of a command you can use the open3 library.

 

stdin, stdout, stderr = Open3.popen3(cmd, args)
logger.info stdout

 

The point is don’t assume anything about what other clients will supply to your controller. If you must use parameters to build command strings, don’t open yourself to unlimited abuse with open-ended system shell commands.

Further reading:

Other “Workplace Safety” articles:

Look for the Squares

A while ago I looked around for software business opportunities. I considered what was “good” or “bad” differently than I do now. I knew that I did not know what to look for. I wanted to “figure out” that part of my life.

I reached out to someone I worked for several years ago. I’ll call this person “Steve”.  When I worked for Steve I wrote software. I repaired and enhanced an existing software product. The company was bought and sold a couple of times. The final owner decided to shutter the business, but most of the orignal employees enjoyed buy-outs of their shares.

Since that time Steve resurrected the product under a new corporation he owned as a partner. Steve sold the company to a well-known, privately owned corporation. Steve’s track record participating in software startups is quite unusual – almost as many “hits” as attempts. I value Steve’s insight and advice.

Steve and I ate lunch, and I began to explain what I was looking for. I was not looking for a job. I was far more interested in building a real business. Steve gave me a great piece of advice.

“If I were to do this over again, I would look for the squares.”

Steve meant look for the businesses that no one else considers.

Many startups that the technology press glorifies focus on hacker culture or consumer vanity. Pictures of your dinner, social media tools, computer games – the things that intoxicate the public. These business have one goal – grow fast and reach as many people as possible. They rely on network effects and customer acquisition models. The business of making money comes next, after they gorge themselves on millions of users.

On the other hand, there is what DHH called the “Fortune 5 Million” – the businesses providing a living for their owners. These are the “squares” – the unhip, real, businesses relentlessly centered on making money. Businesses like dry cleaners, doctors, financial planners, and gutter installers.

There is a sea of people who are in real pain who no one pays attention to. I called a dozen of “square” businesses over 2 days. Three called me back and agreed to talk to me. One owner offered to buy me lunch just because he wanted to talk about things in his work that frustrated him.

Because the market opportunity is not enough to sustain the interest of the “hip” software scene, most startups ignore these people. They won’t stoop down to pick up the dollar they see floating by in the street. But every moment there’s another dollar floating by.

 

How Do You Find Pain With People Who Have No Pain?

For the last 3 weeks I’ve sent out hundreds of emails to fee-only financial advisors.

A typical email looks like this:

I’m doing a research project on fee-only financial planners and advisors.

I’m interested in the struggles and pain points you have running your business.

I’m not selling anything and I’m not marketing anything. I really just want to talk to you about the most painful things that cost you the most time and money in your practice.

This particular iteration of my contact email is not very different from the first. The differences are minor. For example, I added information that answered frequently asked questions. I don’t want to eliminate every question. If someone is curious enough, they’ll write back.

The goal of the email contact is a scheduled phone call. First the recipient has to open the email. Only 40% to 50% of the recipients open the emails. 26% of the recipients click on links in the email to direct them to this blog and my personal LinkedIn account. The funnel narrows with each step.

5% of all emails I’ve sent result in a phone conversation. I’ve done almost 12 hours of interviews with advisors and planners from Florida to Wisconsin and Portland.

If it takes 10,000 hours of practice to truly master a skill, then I’m clearly just beginning to crawl down this path.

Most phone calls don’t go as well as I’d like. When I’m talking to someone about their business, I want to hear about the things that make them grind their teeth. I want to hear about frustration. I want to hear about rage or despair. I want to know the tedious, boring, and wasteful things they have to deal with every day.

I’ve noticed that more than once a business owner will proclaim that they have no pain. They will say that they love every part of their job. I don’t believe them.

I do believe that they enjoy the meaningful outcomes they give to their clients. It’s the reason I decided to start with fee-only financial advisors. They care about helping their clients safeguard and grow their wealth.

Even on days when software development sucked me into a hyper-productive mode I had pain. I had to deal with buggy tools. I had to deal with broken build machines. I had to configure security settings for source code control systems. I could rationalize the pain as a necessary hurdle to get to the real work of writing working software.

So, what do I do when someone tells me that they have no pain? I try to understand their world. I try to put myself in their shoes and understand why they feel that way.

Next, I ask them to get as detailed as possible about what they do every day, step by step. Sometimes people “discover” a pain they didn’t realize they had. One person realized they didn’t really enjoy preparing quarterly billing for their clients.

This tactic doesn’t always work. Sometimes the call feels like a dud.

I try to remind myself that I’m consciously unskilled. Developing a skill takes practice and reflection. Each stumble brings me further down the road.

 

Marching in the Pain Parade

It’s been a week of contacting hundreds of businesses from Wisconsin to Florida to Louisiana. It’s time to ask “What have I learned?”

Not everyone is happy to see you.

I’ve sent over 400 emails to fee-only financial planners. Most of the people I sent emails to did not respond. Only about a third of the people who saw the emails opened them. Less than 5% responded to the email.

I am not surprised I didn’t get more responses. Sales and marketing emails bombard these people every day. My message isn’t a sales or marketing email. Without anything more than a casual glance my email appears to be spam. Most of the emails were probably ignored or deleted.

Responses were not uniform. Here are some of the summarized responses:

“Are you a reporter?”

“If you looking for a job, send us your resume.”

“Are you selling something?”

“If you want my time, I charge $300/hour, with a minimum fee of $500.”

“Who are you working for? Who is this research for?”

“I just don’t have the time to talk. This is the busiest time of the year for me.”

“Please remove me from your list.”

Luckily, several people did agree to talk to me. Even after they agreed to talk, and agreed to spare time out of their day for a phone call, they were skeptical.

Don’t feed the trolls.

For every negative response I wanted to write back and explain myself. Suppressing the need to be right is a skill. I felt like I had to wipe away the misconceptions and make things right.

I don’t think it mattered what they thought. I was not going to convince someone who I really did want to help them with an email. I have a limited number of hours every day to talk to the people who did want to talk. I had to focus on those people and move on.

I did however attempt to apologize to any one who felt I was spamming them. I also told them I would never contact them again. If someone asks you to not email them be polite and move on.

It’s not about you.

The whole point of doing these interviews is to work out their pain. If you decide in the interview to talk about yourself, it’s very easy to lose focus. Let them talk. In fact, they should talk most of the time.

I will ask a few questions. My favorite is “What’s the most painful part of your day?” Occasionally I will ask “What else?” and sometimes I will prompt them with “Tell me more.”

When things are really on a roll, they will begin to dig into the pain themselves. It seems cathartic. I don’t need to “sell” them that they have frustrations. Only after they talk about the pain do I (sometimes) begin to ask explicit questions about the wasteful actions, frustrations, etc, of that pain.

Get used to rejection.

People ignore or reject 98% of my emails. If being ignored hurts, give up now. It will only get worse. The old fairy-tale aphorism “You have to kiss a lot of frogs to meet a prince” also fits here. People will reject you.

Don’t dwell on rejection. Keep looking for the people who want to talk to you. They exist. You must find them.

Be honest.

People want to know why you are trying to call them. If someone called me out of the blue, I’d be curious, too. And skeptical. And on my guard. Don’t complicate things by trying to “spin” your mission as something that it’s not.

I tell people I’m looking for pain. I say that because it’s true. I make no effort to hide my hope that I hope to make a product. Of course, I explain that without real pain there is no solution to worry about.

While I don’t people distracted by my goals, I feel that being open and honest is best. In some cases, people became more interested in what I was going. They seemed  genuinely supportive of my efforts.

I’ll take all the help I can get.