UX for natural digital handwriting
Advanced AI brings handwriting into the digital world, but inspired UX makes it one of the most versatile tools
The importance of UX
UX is short for ‘user experience’ and refers to the experience of interacting with an app, product or system. In software development, UX is typically the focus of two closely related disciplines: UX research and UX design.
UX research
At MyScript, we perform UX research to understand why and in what contexts people prefer to write by hand. We also study people’s expectations, needs and behaviors when writing by hand on paper or a digital device.
Using a range of statistical and qualitative techniques (from interviews and surveys to diary studies, benchmarking and usability tests) our UX researchers gather, sort and analyze huge quantities of data. They then develop detailed personas (models of different types of user) and journey maps (showing the routes users take when performing tasks in an application) covering the most important needs in a range of enterprises.
UX design
Our UX designers use the personas and journeys identified by UX research to innovate solutions to user needs. They’re responsible for workshopping, proposing and prototyping potential design solutions, while helping to ensure that design updates also meet legal, business and other requirements (like localization).
While this sounds straightforward, often user needs or problems don’t have simple solutions – particularly when working with AI. This is where things get more challenging (and interesting!) for UX teams.
To illustrate those challenges, let’s take a deeper dive into how UX at MyScript has helped to shape and define our core products.
Making digital ink work harder
When we began to develop our AI-driven digital ink in 1998, our primary focus was the accuracy and speed of its handwriting recognition. But we knew that this wasn’t the end of the story.
There are many digital inks on the market, but most are somewhat restricted in their capabilities: they capture the strokes made by users as static images. These can be resized or moved around the page, but not much more. The result is that handwritten inputs exist separately from other inputs on the page – most notably typed text.
So while we were developing an AI engine that could recognize and convert handwriting with astonishing accuracy, we were also asking ourselves: what more could digital ink be and do? In what other ways could it benefit its users and out-perform traditional paper and ink?
The existing literature helped us to understand why people choose to write or draw by hand instead of typing, dictating or using other digital inputs. So we began to experiment with expanding digital ink’s horizons beyond handwriting recognition. We broadened our focus to include capabilities such as responsiveness and editability via pen gestures – and while we made good progress, it soon became clear that these were challenges AI could not overcome alone. We needed a parallel focus on inspired, intelligent UX design.
A naturally effective input
We were working amid an exponential rise in the adoption and diversification of digital devices, when the status and future of handwriting was uncertain. So we intensified our research, interviewing people from diverse social and professional backgrounds over extended periods, to understand how they used handwriting in their daily lives.
We studied thousands of real notes (paper and digital) and organized detailed surveys involving thousands of respondents. And remarkably, despite the explosive proliferation of digital devices, our starting hypothesis was confirmed again and again: creating content by hand brings remarkable and unique benefits, from increased freedom of expression to enhanced memorization.
It’s much easier to record non-linear ideas or to brainstorm relationships with pen and paper than it is with a keyboard; this is also true of writing mathematical equations or musical notation. And it’s often easier to create diagrams by hand than with a mouse. Writing is often about much more than just creating finished content: it’s a way of thinking. When we write or draw by hand, our ideas can evolve more freely and organically.
The advantages of digital content
But creating content by hand also brings limitations and frustrations.
Our research showed that paper notes often failed to integrate cleanly with digital tools and resources. People were losing track of information scribbled down on scraps of paper. They were losing valuable time to transcription, as they typed up handwritten minutes or reformulated sketched diagrams in PowerPoint or Keynote. Their notes often started well before becoming increasingly chaotic and hard to understand, as they ran out of space or made errors they couldn’t erase. Digital alternatives to handwriting were generally more effective for information retrieval, organization, portability, editing and connectivity.
Why should people have to choose between two distinct input methods, each offering equally distinct productivity and usability benefits? We saw an opportunity to create a solution that offered the best of both worlds.
Understanding use-cases
During our research, we had also examined contexts where traditional digital input methods seemed unwieldy, and where writing by hand (even with just a finger) might prove beneficial.
In-car displays and infotainment systems are a good example: buttons, dials and touchscreen keyboards can distract drivers and pose a threat to safety. Even voice recognition brings significant drawbacks – cars are often full of background noise and the tech isn’t accurate enough to avoid frequent errors. If drivers have to divide their attention to resolve recognition errors, the consequences could be dire.
What if drivers could simply write their instructions with a finger while keeping their eyes on the road? This would pose challenges, for sure – the software would need to be able to separate and understand superimposed characters, for a start. But the benefits and potential applications of such a solution would be significant – not least for smart watches with very small screens.
Another example is diagram creation. Drawing diagrams for reports or presentations typically requires users either to master a separate application or to gain specialized expertise in an application with which they’re already familiar (like Microsoft Word). What if our tech could enable them to create a diagram with their pen, then convert it to perfect shapes, straight lines and typed text in just a couple of taps?
Moreover, what if that converted diagram was responsive, automatically resizing connectors when elements were moved around? Even better, what if the hand-drawn diagram could auto-convert when copy-pasted into a popular presentation app, like PowerPoint or Keynote?
We were forming a picture of a new kind of digital ink – one that offered the user more than handwriting recognition and write-to-text conversion.
From understanding to vision
After identifying user needs through rigorous UX research, we now had a clear objective: to ensure our digital ink acted as the ‘missing link’ between the natural benefits of handwriting and the power of digital content. Combining the two could seriously boost people’s productivity – but it would take more than just AI. We needed intelligent, considered and innovative UX design.
Interactivity and productivity
The first step we took was to define the ways in which our digital ink would go further than others on the market. Working closely with our dev teams and product managers, our UX team envisioned an ‘interactive ink’ that would offer three major innovations: multimodality, editability and responsiveness.
Multimodality
We wanted our digital ink to operate as the equal of typed text – not a separate form of content. So we designed our software to treat different kinds of text input in the same way, allowing users to mix handwriting and type in the same document, paragraph, sentence or word. You can see this for yourself in Nebo , our digital note-taking app.
Nebo’s regular page was designed for the creation of structured, linear notes and documents, where handwritten and typed content could be mixed, merged and converted as needed. Users can add a variety of ‘objects’ (diagrams, math, sketches and so on) to enrich their notes without impacting the accuracy of the handwriting recognition.
In fact, the use of objects touches on one of the greatest challenges we currently face: how to enable users to mix text and non-text content on a digital page with complete freedom and no loss of functionality for either. Our emphasis on achieving this goal is most visible in the introduction and evolution of Nebo’s freeform page, which launched in late 2020.
The freeform page is an infinite canvas that allows users to write and draw freely with no constraints on the placement of their content – and without the potential distraction that automatic digital processing can cause. This makes it a great solution for use cases like brainstorming sessions or free-ranging lecture notes, where users focus less on linear structure and text-only content.
We’re currently working on creating the next iteration of the freeform page, which will empower users to type, dictate and even convert handwriting to typed text – further closing the gap between analog and digital inputs.
Editability
We also wanted our ‘interactive ink’ to bring full digital flexibility to handwritten content. Handwriting had to remain fully editable right down to the level of individual strokes – and it had to be editable with your pen.
We conducted a huge amount of research and, with the help of our AI team, implemented a set of pen gestures so intuitive they can be learned and employed in moments: scratch-out to erase, an upwards stroke to join and a downwards one to break, etc. And because our ink is multimodal, these gestures work on typed text as well as handwriting – one more way to increase our users’ productivity.
Responsiveness
In some contexts – such as when creating lecture notes that mix text with sketches, diagrams and/or math equations – it’s vital to preserve the original layout of your content.
In others – such as when writing meeting minutes that will be converted to typed text and shared with colleagues – it’s more productive for content to reflow automatically when edits are made, or when you view synced content on a different device.
Responsiveness is therefore a context-specific requirement – so our solutions had to offer responsiveness when needed, automatically accommodating changes in layout for all forms of content (even mixed handwriting and type).
Nebo: showcase, testing ground and solution
Nebo is our digital notebook app. It’s driven by the MyScript SDK AI engine and shaped by our emphasis on a superior handwriting experience. It’s also an ideal testing ground for our UX solutions – a place where we can implement and iterate on features that support and enhance our core handwriting-recognition tech.
We use Nebo to gather insights from a wide range of data points, revealing how real-world users experience our new features and improvements. The feedback we gain through analytics, support and feature requests, app store reviews, social media engagement and our Insider program is invaluable to shaping the future development of our products. And of course, we conduct extensive research on Nebo use patterns, allowing us to determine what works and what needs to be improved, as well as to identify new use cases and new expectations.
Looking at Nebo today gives a clearer picture of what this means in practice. The app offers super accurate handwriting recognition and real-time write-to-text conversion in over 60 languages. But its UX design transforms this already-impressive tool into a far more holistic and productivity-boosting experience.
With Nebo, for example, users can edit handwriting effortlessly using natural pen gestures. Handwritten content is also responsive – so, for example, handwritten meeting notes will automatically reflow for easier viewing on a range of different devices; perfect for when sharing them via email or web link with colleagues likely to view them on a smartphone. Nebo also enables formatting through text ‘decoration’ – like underlining text to emphasize it, or boxing-in text to highlight it. And users can export their notes to a range of popular formats.
This is by no means an exhaustive list and Nebo’s feature set is growing all the time. But it gives a clear idea of the many ways our UX team has sought to amplify and extend the power of MyScript SDK. What’s more, every feature is backed by rigorous research and evolves through constant testing and review: for the UX team at MyScript, innovation and iteration are as important as implementation.
