Project Process
Brainstorm & Define problems
The user problem of this project just stem from my design system experience. 2018 Inspired by the AirBnB’s DLS and the Material 2.0 of Google, I started using the design library. Early 2020, I moved to the company’s HQ to manage the LINE design system. During the pandemic, I should come back to Korea but started this open source project to help people like me who have a struggle with the design system.
The design system is a co-visual language for both designers and engineers alike, and it is extremely common when it comes to creating new user interfaces for large-scale products and services to ensure consistency and productivity. However, they still have major obstacles - one of which is the increasing complexity with the increasing number of components and patterns included in the one single library.
Because of those kinds of complexities, as you can see above, even if designers shape the same designs…
- Some uses the library exactly
- but another one has broken symbols,
- Some of them even has detached components due to small modifications from the anonymous designer,
In fact, those artboards have been such a mess…
In the LINE design system, I worked super hard to prevent those kinda mistakes via guidance, cartoon manuals, and design QA. But human resources were always limited. So I started to think of something differently.
Ideate & Prototype
Idea? Computer Vision!
The basic idea of this project is to detect various UI elements in the browser using a custom trained object detection model via TensorFlow.js live in the plugin.
I started this idea because we already have been living with many services using machine learning such as face recognition, mask detection, and something like this.
If I can apply this computer vision technology into the design system world, I can make a computer learn the numerous design elements instead of human resources, and also autonomously recommends the design grammar to designers if it finds something that does not conform to the design system.
This will not only be convenient for the design system users but also it will be the freedom on people like me who should manage this system.
Prototyping
After pointing out this idea, I started making a prototype to see if my hypothesis was possible or not. I googled and downloaded 200 button images and made the button detector ML model by using TensorFlow.
Fortunately, it successfully worked! I got this working prototype that could detect the button UI elements from the screen in real-time. I was super excited when my hypothesis came true. And tried to move forward.
You can find my first prototype story in more detail at this Medium post.
Machine Learning
Although my first prototype was very successful, there were many challenges for creating a useful design system plugin. A much larger dataset, and more optimized ML model, and I had to find the best way to make my designers feel this AI power seamlessly. From 2021, I have been solving those problems one by one.
Raw data? Partnership!
For data, I met a Singapore-based startup called Mobbin. They had over 50,000 mobile screenshots for product designers. So contacted them, presented this project, and got over 10,000 high-quality UI screenshots data for free!
Data Labeling? Sponsored!
I got the large amount of data, but the data-preparation doesn’t end there. I must data-label all of them in order to use in machine learning. This process is very important, but it took a lot of resources and time, so it was huge difficult for me to solve.
So I applied to a sponsorship program hosted by a Korean data labeling company, SelectStar, and luckily I was selected and eligible for free labeling support from there!
You can download some samples of this open dataset that I created with Mobbin, and SelectStar through this link. Moreover, if you are interested in the whole dataset, please contact me.
Training with TensorFlow
To be honest, none of this was easy for me because I just started to study TensorFlow at that time. No background, No tutoring. So I just had tried to study AI/ML by myself via Google code labs, blog posts, and Youtube channels. The result is not perfect, but by my standards, I have successfully made TensorFlow models with enough performance level. You can find more details from my Colab notebooks below:
Make plugins
And the last challenge was how to make it easy for designers to use this AI feature?
I thought it would be much more natural to check the spelling while designing it rather than separated services. So I decided to focus on the handoff moment by making a plugin with TensorFlow JS not TensorFlow.
I designed some frames on Figma and made a simple single-page web application using Figma Plugin API, Zeplin API, React, and TensorFlow JS. The best way to illustrate this is to see it in action together!
Zeplin Extension
- Demo: http://zeplin-ml.herokuapp.com
- Github: https://github.com/dusskapark/zeplin-ml
Figma Plugin
- Install Link: https://www.figma.com/community/plugin/1049316538308913961/Figma-ML
- Github: https://github.com/dusskapark/figma-ml
Reactions
After I released Zeplin ML in August and shared my stories via my blog because I’d like to show off and not forget this journey. And… its reaction was awesome!
I choose some and attach them below!
Machine learning meets design systems? 🤯 Learn how one of our users built a nifty app with the Zeplin API and @TensorFlow JS that helps designers and engineers utilize their design library. 👉 https://t.co/mwiHhpZSkl
— Zeplin (@zeplin) October 4, 2021
Thanks @dusskapark for sharing your project with the world! pic.twitter.com/VsPrXXetI9
Super cool use case for #TensorFlowJS by @dusskapark who led Global #Design at #Line to create a system that acts like a spell check for designers. Learn how he made it below. Excited to see how it evolves. Read / watch more: https://t.co/Vk1F5Crohi#MadeWithTFJS #HCI #GUI #Figma pic.twitter.com/9xIrU3mpP8
— Jason Mayes (@jason_mayes) August 12, 2021
🤯So cool—tell me more! I looked at the code and have no idea how the model is working. Excited to see where ML design linting can take us. @sidahuj @jsngr 👀
— Anthony DiSpezio (@adispezio) December 7, 2021
For a moment I thought you built a ML spell-checker for Figma but I get it now :) I'm desperately seeking spellcheck.
And… DEVIEW 2021!
FINALLY, I was invited to DEVIEW 2021, which is one of the biggest Developer conference in Korea, and shared my experiences with the title “Using TensorFlow JS to build an AI-powered design system”. This was a truly honorable experience for me. I will take a step further based on this experience.
https://deview.kr/2021/sessions/460
