Using Augmented Reality Code for a Better Stretch Gesture

Loose Leaf is more about photos and imports than it is about drawing or sketching, and I wanted to make sure it was easy not only to cut and create new scraps, but also to manipulate and duplicate existing scraps. To make a copy of a photo or scrap, I thought through numerous gesture options, menus, long press popups, buttons, and more, and in the end I settled on a simple pull-it-apart-gesture.

So how’d it turn out? Here’s what it looks like to duplicate any photo scrap in Loose Leaf, just pull it apart into two identical pieces.

It’s a simple gesture to pull a photo into two copies. The stretch animation as you pull makes it obvious what’s happening, and then — snap! — It’s two copies!

The difficult piece of this gesture isn’t the copy itself, it’s the stretch animation as you pull the image apart. For a number of reasons, I needed to do this without additional OpenGL rendering, I needed to keep the code solidly in the normal UIKit stuff. To make it work, I borrowed some technology that’s fairly common in Augmented Reality apps: it’s called homography.

The short description is that any convex quadrilateral can appear to be any other convex quadrilateral by just rotating and translating it- It basically lets me turn any four sided shape into any other four sided shape. Perfect! scraps, even non-rectangle one’s cut by scissors, are still modeled as simple rectangle UIViews. As I stretch, I can transform that rectangle into a parallelogram for that stretched effect.

Here’s the simple test application I used to fine-tune the animation with the above strategy:

The algorithm visible above goes through the following steps:

  1. create a 4 sided quadrilateral from the four touch points on the scrap
  2. as the fingers move, create a 2nd quadrilateral from the new locations of those same touches
  3. compute the homography between those two quadrilaterals to find the transform between them
  4. apply that transform to the scrap itself

Note that I’m not trying to calculate the transform from the UIView’s bounds to its new parallelogram – instead i’m transforming the quad of finger positions to thew new quad of finger positions, then using that transform on the view.

The red lines in the above image is the quadrilateral between the touch points. The green line is an “average” quadrilateral, which averages the red quad into a parallelogram. You can see the stretch is very dramatic, and it causes a problem if the touch points form ever form a concave instead of convex polygon. You can see an example of what goes wrong below:

But if I use the average quad instead, and treat those points as transforms between parallelograms, then the same gesture gives smoother results:

My last experiment for the stretch gesture was how to keep the orientation of the image as it stretched. I thought about trying to maintain the “up”ness of the image regardless of the stretch vs. having the gesture also rotate the image during the stretch. You can see what I mean with the two videos below.


In the end, I decided to rotate the image during the gesture, and not try to maintain “up” orientation. It gave a cool effect, especially when stretching, but felt just a bit off since it also caused the fingers to slide more over the image.

To try it for yourself, download Loose Leaf on the app store.

You can also find all of the code on GitHub for each one of the versions talked about here, and be sure to check out the rest of Loose Leaf’s open source contributions as well.

Improving UIBezierPath Performance and API

I shared two weeks ago how I built the scissors feature in Loose Leaf; short story, lots and lots of UIBezierPath. As I worked through the algorithm for slicing paths together, it didn’t take me long to realize that default performance of UIBezierPath was… lacking. The only way to introspect the path itself is to create a custom function – not block mind you, function – and use CGPathApply() to iterate the path and calculate what you’re looking for. Every time.

For a path of size N, finding the first point? O(N). Finding points or tangents along the path? O(N). Is the path closed? O(N).

Enter PerformanceBezier.

PerformanceBezier improves many of these simple tasks with operations that average to O(1) for any length bezier. The first calculation may still iterate the path, but afterwards the answer will be cached for far faster access, and for some operations a full path iteration is never needed. It also brings UIBezierPath a bit closer to its NSBezierPath counterpart.

And the best part? It works with all UIBezierPath objects – no custom subclass, no extra code, it “just works.”

How ’bout an example?

Scissors isn’t the only feature in Loose Leaf that requires finding intersections and clipping Bezier paths – drawing over images is essentially the same algorithm. With each stroke, I need to clip the pen’s path to each scrap. Even drawing over multiple images will clip the pen’s ink across each image.


With every stroke, the scissor algorithm needs to be run for each image that the pen draws over. To do this in 60FPS means it needs to be done quick. Each image’s border path only changes when it’s cut with scissors, so it makes sense to cache as much information about that path as possible to reduce the cost of future clips with the pen.

So How Much Does It Help?

I’ve included a macro in PerformanceBezier so you can compare times as well. Turning this flag on will simulate running CGPathApply to retrieve the same information. So [path firstPoint] becomes O(N) instead of O(1), etc.

When I run PerformenceBezier without caching on a first gen iPad mini, the FPS drops dramatically. I used instruments to measure how much time was spent in the Main Thread in total, and how much of that time was wasted with uncached Bezier operations.

The Test: I dropped 4 irregularly cut images onto a page, and scribble over them with the pen. This stresses the bezier clipping algorithm as each new stroke segment is clipped to multiple paths.

The results were dramatic. Out of 10s spent on the Main Thread during drawing, 5s were spent wasting time in methods that could’ve been cached. Put another way, this doubled performance in Loose Leaf. After optimizations, I can get twice the clipping done in the same amount of time. Some method calls saw 10x speed improvement. Obviously, your milage will vary depending on your algorithm and how heavily you use UIBezierPath in your code, but Loose Leaf would simply not be possible without these optimizations.

Get the Code

Download and browse the code: This isn’t the first repo I’m open sourcing – there’s lots more code from Loose Leaf, and more to come

I’m working to open source all of path clipping code for the scissors, and the first step starts today. Stay tuned!

Step by Step App Store Optimization: An Objective Measure to Find the Best Keywords

When I launched Loose Leaf, I’d heard about how bad app discovery was in the App Store, so I’d assumed any search optimization I did for the App Store would be a wasted effort. If quality Twitter apps are outranked by unrelated apps, it seems like a crap shoot to even try, so I didn’t.

Big mistake. It turns out that roughly half of users are finding and downloading apps through App Store search. Put another way, by not optimizing for App Store search, I’ve effectively cut my sales in half.

Over the past month, I’ve started diving into ASO – App Store Optimization – and finding the best practices, and most of all, finding exact step-by-steps that I can follow. Knowing that “keywords are important” is only half the battle. How do i decide which keywords to use and why? How can I turn this black art into a spreadsheet, and how can I do it for free?

Step 1: Get the Free Spreadsheet

I’ve setup a free spreadsheet with some example data that you can use. You can easily copy it to your own Google Drive to start editing and swapping in your own targeted keywords.

Step 2: Finding Keywords with Sensor Tower

Screen Shot 2015-01-30 at 4.16.08 PMA great resource for finding relevant keywords for your app is This site can track your App Store search rankings for any of your keywords, and it also helps with finding new keywords that you may want to use – it’s the 2nd case we’re worried about here.

Screen Shot 2015-01-30 at 4.17.22 PMCreate a free account, then login and add your app to their system. Next, head to the Track Competitors section – you’ll likely see it pre-populated with probable competitors, or you can also look up specific apps. Once you find a close competitor, click the Keyword Spy button. This will take you to a page that’ll show your vs their keywords.

Add each of these keywords to your spreadsheet, and repeat the process for each competitor.

Step 3: Finding Keyword Value

By now, you should have a long list of potential keywords, with very little idea of which keywords are actually valuable. This is our next step: let’s track down which of these keywords are actually worthwhile.

Screen Shot 2015-01-30 at 4.33.00 PMNext up: the Keyword Research tool in SensorTower. For each keyword you’ve added to your spreadsheet, search for that word in this research tool. It’ll give you the Traffic, iPhone/iPad Difficulty, and number of apps for each keyword. Add each of these values into the spreadsheet.

Note: To make this process quicker, you can start the free trial of Sensor Tower and track those keywords in your account. This’ll let you download a single spreadsheet that you can then import into the Google Spreadsheet – but again, it’s only a 2 week trial.

As you add each of these words into the spreadsheet, you should see the rest of the columns fill in automatically.

Step 4: Choosing the Right Keywords

And now you have all the information you need to choose valuable and meaningful keywords for your App Store Optimization. Let’s dig into the spreadsheet and find out what’s going on.

The Sensor Tower data tells you 3 pieces of very important data: How much search traffic that keyword generates and how difficult (0-10) it is to rank for that keyword. So to determine the value of a keyword, we first find how easy it is to rank for a particular word (10-difficulty), then we average the iPad/iPhone values and multiply by the total search traffic to estimate our Traffic Return if we ranked for that keyword.

Screen Shot 2015-01-30 at 4.49.27 PMApple only gives us 100 characters to work with in the keywords field in iTunesConnect, so not all keywords are equal. If two keywords would pull in the same estimated traffic, it only makes sense to choose the shorter word, right? That’s what that last column does, it divides that estimated traffic by the length of the word so you can more equally compare any two keywords.

Once you’ve entered all your keywords and data, just sort by Traffic/Letter to see the most valuable keywords at the top of the list!

Step 5: Enjoy a Coffee!

Now you have an objective measure of which keywords to use in your App Name and Keyword fields, and most importantly you know why they’re valuable – that’s enough to make any engineer smile! 🙂

Google Author link
Page 7 of 51« First...56789...203040...Last »