jump to navigation

Mobile App and Website Testing Roundup for 2015 April 5, 2015

Posted by willhlaw in Development, Mobile, Testing.
Tags: , , , , , , ,
3 comments

After researching testing for a startup, I thought I would share my findings.

Background on Testing

There are a few concerns that testing attempts to solve that is applicable to most mobile app product deployments.

  • There is functional testing (is anything broken?).
  • There is load testing (does app or website fold under pressure?).
  • And there is usability testing (do paid testers acting like users find the app or website easy to use?).
  • Then, there are the targets. Need to consider performing these tests against all of the combination of workflows, iOS and Android and the backend server and database as well as any websites.
  • iOS and Android app testing is different, see why
  • List of Testing Tools for mobile and others

Appium (site, review) – Free, required for services like AppThwack

Free, open source, raw foundation to create test scripts that can be written outside of the project’s code base. Has improved much over the past two years to become the tool of choice among QA shops. I chose this over other testing frameworks such as MonkeyTalk and  Robotium.

  • Free
  • Requires coding knowledge and uses black box testing Selenium style
  • It would be ideal if the development team uses this framework or one of the aforementioned, because other testing websites like AppThwack expect to run these test scripts.

AppThwack (site, review) – Cheap, requires test scripts like Appium

If you have test scripts running for your project, like Appium or others, then AppThwack can automate the running of those tests on 100s of real devices.

  • $20/month for 200 device test minutes up to $500/year for 7,500 device minutes
  • Requires suite of test scripts to be already written

TestElf (site, review) – Cheap

Covers functional testing and has 2 day turn-around

  • $50 signup offer, $200 for 1 test, $1,000 for 6 tests, $2,000 per month

UserTesting (site, review) – Variety

Shows videos of test users using the app or website

  • $49/video or $3,000/year

Offers a free version called Peek, but the app needs to be in the app store.

  • Free

Applause (site, review) – Expensive

Following estimates (see and modify actual quote) are for 4 Apps (iOS App and Android count separately, and there is Consumer and Merchant app):

  • 4 Apps Functional Testing costs $4,500 – $7,499*
    • *With annual subscription, and salesman said these autoquote numbers are high and can come down
    • Custom team of testers and allows up to 10 test case hours
  • 4 Apps Load Testing costs $18,000 – $30,000*
    • *Salesman said these autoquote numbers are high and can come down
    • Led by a performance engineer and hand creates tests and will create reports and improvement recommendations

Recommendations

  • Try TestElf’s functional testing by paying $50 special offer
  • Try Peek’s usability testing for free
  • Depending on seriousness and budget, start to negotiate with Applause and/or require development team to create a suite of tests to cover each major requirement and workflow
Advertisements

Notes on Agile Product Ownership in a Nutshell March 9, 2015

Posted by willhlaw in Administrivia, Productivity.
Tags: ,
add a comment

Illustration

Above is an illustration from the video ‘Agile Product Ownership in a Nutshell‘ that uses the RSA animation technique. Below are the notes highlighting key points made during the video as well as some other points drawn from additional resources. This post will be a good read for Product Owners, both new and experienced, as well as any team member on an Agile Scrum team that wants to revisit the basic principles and possibly realign their team.

Roles in Scrum
PO – Product Owner carries the vision, says no or yes to customer requests, prioritizes, and responsible for building the right thing.
SM – Scrum Master is the coach, responsible for building it fast and fast feedback cycles with the users.
Other Roles
TL – Technical Lead is responsible for building the thing right, talks closely with customers and other teams, but still encourages self organization.
CPO – Chief Product Owner organizes multiple POs and interdependencies.
Development Manager hires, mentors engineers, creates culture, and knows when to step in and lead discussion on branching strategy or versioning [1]
Key Points
Product backlog becomes Team backlog when working on multiple products (new products, old products, O&M, etc)
Value = knowledge value + customer value.
Knowledge value is gained early to reduce risk. Knowledge user stories are UI mock ups, trade study, spikes, prototypes, etc.
Stories have an estimate for effort & value so priority = value / effort. This should make it easier for PO to prioritize Team backlog.
Velocity goes down overtime due to technical debt, architecture decisions, getting behind on automated testing. It is the team’s job to correct. However, how is this investment effort tracked? This question is asked and a lot of the sources below were found here.
  1. Be transparent by explaining benefits of paying down technical debt so that PO can prioritize. [4]
  2. Have a separate Improvement backlog that is internal that is adds a tax to each sprint (e.g. 10-20% of velocity). [5] [6]
  3. Do not obsess over it, basically just pick one and fix the broken stuff already [7]. Also, consider adding to the definition of done (DoD) that the work is done only if it does not add any technical debt.
Charts [2]
Time vs customer value curve = knowledge value, customer value focus, trim the tail
Time vs delivered stories
– fixed scope
– fixed time
– fixed scope and time -> no, let’s decrease scope (b/c can always extend time and not the other way around)
Reasons for Scrum:
– team motivation (not overworked, pressure from above, lack of input or control of march)
– deliver value in sweet spot of the triple constraints (time, cost, quality) Venn diagram
– more accurate predictions and expectation management
– standard metrics to evaluate team improvement and tech choices
Points measure effort [3]
Effort should translate to time, and is influenced by uncertainty and complexity.
  1. How much effort to get to that building? Answer for runner and cripple is one.
  2. How much effort to get to farther away bldg? Answer for both is two, since it looks twice as far.
  3. How much effort to get to farther away bldg where there is a chasm of lava and a small walkway? Answer for both finally agree that it is a 4, since they will have to be extra careful and may drastically slow their progress.
  4. How much effort to get to close bldg while singing Gangnam Style? Answer for both finally agree that it is still a 1, as the extra complexity doesn’t really have an effect on the effort that causes a slowdown.
[1] Development managers vs scrum masters, Dan Radigan at Atlasian, https://www.atlassian.com/agile/effective-management-across-agile
[2] Agile Product Ownership in a Nutshell, Henrik Kniberg, Youtube video – https://www.youtube.com/watch?v=502ILHjX9EE, Transcription – http://blog.crisp.se/2012/10/25/henrikkniberg/agile-product-ownership-in-a-nutshell
[3] Story Points Are Still About Effort, Mike Cohn at Mountain Goat Software, http://www.mountaingoatsoftware.com/blog/story-points-are-still-about-effort
[4] How to translate “business value” of things that are technically important, Matthias Marschall at Agile Web Development & Operations, http://www.agileweboperations.com/how-to-translate-business-value-of-things-that-are-technically-important
[5] Effective Steps to reduce technical debt: An agile approach, Bastian Buch at Codovation, http://www.codovation.com/2012/06/effective-steps-to-reduce-technical-debt-an-agile-approach/
[6] Scrum Strategy – The Dev Team Improvement Backlog, Professional Scrum Trainer at Scrum Crazy, http://www.scrumcrazy.com/Scrum+Strategy+-+The+Dev+Team+Improvement+Backlog
[7] Do agile right – Delivery – Technical Debt, Atlassian, http://www.scrumcrazy.com/Scrum+Strategy+-+The+Dev+Team+Improvement+Backlog

Famo.us Easter Egg April 2, 2014

Posted by willhlaw in API, Javascript, Mobile.
Tags: ,
1 comment so far

In Steve Newcomb’s FAQ: Tough Questions on Famo.us, he puts an Easter Egg for those patient enough to read to the end. I thought those that wanted to google for a quick translation might find this helpful.  Famo.us is a free and open source JavaScript development framework back by a host of cloud services.

Si vos vere postulo impetro in BETA mox steve@famo.us ad minim veniam. Sciam si vestrae res et faciam te in BETA possim.

April 9 Si ad res, in prima acie, simul te BETA.

— Using Google Translate from Latin to English (link) –>

If you really need to get the Beta as soon as steve@famo.us more information, I come. I am able to rest assured that if your situation and I will make of thee in beta.

April 9 On the real thing, in the front line, at the same time you beet.

How to save money by tethering your BlackBerry to laptop November 1, 2009

Posted by willhlaw in BlackBerry, Laptop, Mobile, Productivity, Verizon.
Tags: , , , , , ,
add a comment

I recently purchased the BlackBerry Tour from Verizon. The promotion at the time allowed me to get a subsidized Gateway netbook that is essentially free because of the rebates. However, you need to sign up for a 2-year contract for Verizon’s 3G plan. They only have two plans: $39.99/month for 250MB or $59.99/month for 5GB. I chose the $39.99/mo plan. And I ate up 236MB in the first 20 days. Not good. Note to self, do not watch YouTube videos or any videos with progressive loading while on 3G. It also did not help that I was surfing and blogging like a mad man during the biggest Sharepoint Conference of the year at Los Vegas (see all my posts at https://willhlaw.wordpress.com/category/sharepoint/spc09/). The thing is, I only watched one video and it was 9 minutes. I have been scared of going over (well, not that scared since it’s only $.10/MB once you go over), so I have turned it off. In the meantime, I have wanted to hook up my BlackBerry to the netbook. Using your mobile phone to act like a modem for your laptop, is called Tethering.

So, while watching football today, I decided to try it. From what all the forums were saying and from a previous foray into tethering with Verizon, it seemed that the only official way to tether is to pay Verizon $15/month. I think you can surf tethered as long as you want if you have the unlimited data plan with your phone plan. Comment #15 by blackberryontherun led me to http://tetherberry.com. These are the features they advertise on their website.

With Tetherberry, you can use your BlackBerry has a modem and not have to pay Verizon’s $15/month tethering fees. The Tetherberry product is a one time cost of $49.99, they have a free trial which is nice. I was able to install it on my pc quickly, send the link to the tetherberry.jad file to my mobile, and install it on the BlackBerry in less than 5 minutes. I turned off the WiFi on my netbook and saw magic happen. Below are screen captures of speed tests taken while tethering with the BlackBerry on Verizon’s network, while using Verizon’s 3G built-in modem, and while connected to WIFi wireless router on Verizon FIOS.

A) Speedtest with netbook tethered with the BlackBerry on Verizon’s network

 

B) Speedtest with netbook connected using Verizon’s 3G built-in adapter

 

C) Speedtest with netbook connected to wireless router on Verizon FIOS using speakeasy.com/speedtest

 

The tests were using http://speakeasy.net/speedtest. Tethering was decent at around 1Mb/second. I could watch this YouTube video Introducing the jQuery Sharepoint project, jPoint, without a problem. The 3G plan with Verizon was about 50% faster. The wireless router on Verizon FIOS was 15-20X faster, but that was expected. The purpose of the latter speed test is to give a baseline for some of the readers who may not have an idea on what 1249 kbps means. It means 15-20X slower than your network at home if you have Verizon FIOS or high-speed broadband internet through your cable provider.

Connection

Cost

Speed

Required

BB Tethering w/ Verizon

$15/mo

0.9-1.2Mbps

Mobile data plan & VZAccess Manager

BB Tethering w/ TetherBerry

$49.99

0.9-1.2Mbps

Mobile data plan & Tetherberry app

3G w/ Verizon (250MB)

$39.99/mo

1.7-1.9Mbps

3G modem built-into laptop or USB

3G w/ Verizon (5MB)

$49.99/mo

1.7-2.9Mbps

3G modem built-into laptop or USB

WiFi w/ Verizon FIOS

(Used for speed comparison)

15-19Mbps

Be within 100 ft. of wireless router

 

In conclusion, I am going to save a lot of money by going with a specific application like Tetherberry in order to surf on my netbook from anywhere within Verizon’s network (shown below).