The Jewelry Market

The overall market

For thousands of years, humans have adorned themselves with jewelry. It is one of the oldest and most durable modes of self-expression. Whether it’s a spectacular diamond necklace or a modest charm bracelet, jewelry can quickly boost an individual’s confidence and sense of style. The worldwide luxury jewelry market is expected to reach approximately 22 billion euros in 2020. China accounts for the lion’s share of jewelry and watches market revenue, followed by the United States and India. The jewelry market is predicted to grow in value from approximately 230 billion US dollars in 2020 to approximately 307 billion dollars in 2026. One of the most exciting new developments in the jewelry business is the development of lab-created diamonds, which are frequently indistinguishable from genuine diamonds and are likely to become much more widespread and accessible in the near future. [1]

E-commerce in jewelry

One of the tabloid press’s favorite topics in recent years has been to accuse millennials of destroying the diamond industry. Without wealth, but with new investing ideals, today’s youth are disinterested in diamonds and emeralds.

Because every cloud has a silver lining, other demographic groups increased demand for luxury goods, that is, as the first effects of the financial crisis subsided. This can be attributed to the rising middle and upper classes in nations such as China, which are eager to invest in jewelry and western brands.

Currently, the jewelry sector’s annual global sales are 148 billion US dollars, are growing at a 6% annual rate, and will reach 250 billion dollars by 2020. In terms of online jewelry sales data, digital channels continue to represent a modest portion of the business, accounting for just 5% of the market, which is expected to grow to 10% by 2020 (Gave, S. (n.d.), 2021). The success of ecommerce in the jewelry industry. Sales Layer Blog. Retrieved March 28, 2022, from https://blog.saleslayer.com/ecommerce-jewelry-industry 

). [2] However, this does not mean that the scene has not begun to shift.

AR’s promising future in Jewel

What enable ecommerce

To assist users in purchasing things online, we require a strong e-commerce platform, such as an online marketplace, a website, or a mobile application. Additionally, an increasing number of people nowadays use smartphones, tablets, and laptops to access the internet. These devices are another means of facilitating e-commerce. We are entering a new era, the era of the metaverse since Facebook announced “Meta”. AR and VR technologies are gaining popularity and are being further developed. This technique has the potential to generate excitement for gem products.

Benefits of AR in marketing

The widespread adoption of smartphones, the decrease in cost, the growth in mobility, and the capacity of augmented reality to provide experience value all contributed to AR’s transition from the laboratory to the commercial retail realm [3]. Due to its influence on engagement and decision-making, as well as its ability to enhance consumer experiences, augmented reality offers significant benefits to businesses and customers [4]. AR helps merchants to deliver extensive information to their customers to aid in their decision-making [5]. Additional benefits include enhanced product evaluation and the opportunity to try alternatives that consumers would not have considered in the absence of AR. AR greatly boosts a consumer’s proclivity to purchase [6].

The success of IKEA

IKEA, a well-known “Ready-to-Assemble” furniture company with Swedish origins, has recently seized opportunities to improve service through the use of modern technology. The corporation, founded in 1943, has grown into a global behemoth. Although IKEA is the most well-known Scandinavian furniture firm, the services it provides to its consumers also account for a sizable portion of its marketing efforts. In this vein, the business launched an augmented reality software, dubbed IKEA Place, in September 2017 with the goal of resolving practical issues associated with furniture shopping. 

Figure 1: Ikea app placing a virtual chair (Tech crunch)

Customers would be able to try out furniture in their homes before purchasing it via the free downloadable app. The software would utilize augmented reality to enable users to imagine how furniture might appear in their own homes. Not only would this alleviate the inconvenience of shopping for furniture, but it would also remove the pressure of returning any furniture that does not fit. IKEA establishes a service-centered value with this free app by demonstrating that it recognizes the challenges associated with the furniture purchase process and offers assistance. Not only does the IKEA Place App assist clients in deciding which furniture to purchase, it also protects them against unwanted unsuitable consequences. To accomplish this, the app enables furniture shoppers to digitally arrange their rooms with over 2,000 IKEA goods and accessories. It’s as simple as snapping the space around the item and then choosing it. The software scales the selected product to the appropriate size based on the shopper’s room’s dimensions with an accuracy of approximately 98 percent. Additionally, the ability to perceive the fabric’s texture and the simulation of light and shadows are included as features. Additionally, IKEA Place wanted to transform the process of selecting a piece of furniture into a rather enjoyable digital engagement.

The augmented reality app’s service extension garnered significant marketing buzz. Google searches for “IKEA AR” increased significantly (Figure 2), and the company received major news coverage for being named one of the 50 most creative companies (FastCompany, 2018). [7]

Figure 2: Google search trends: interest over time (keyword: IKEA AR)/1 September 2017–1 November 2019.

The overall idea of the project

As previously expressed, augmented reality is a highly successful technique for increasing jewelry product marketing and sales, as well as improving the customer experience. We demonstrate how to employ augmented reality (AR) technology to enable consumers to try on jewelry ring products on their hand in this article.

Method

Problem solving Pipeline

To ensure the project’s success, we must address the following issues. To begin, we must monitor the user’s hands and then the ring finger placement in order to fit the ring in. The finger’s position must have both an XY coordinate and a depth coordinate. After the ring is fitted into position, the second difficulty is to load and attach the ring model to the user’s finger utilizing computer graphic techniques. To make the ring model appear realistic, we must use certain special effects such as shaders; this is the final issue.

Figure 3: Problem solving pipeline

Hand-tracking methodology (MediaPipe)

The ability to perceive the shape and motion of hands can be a vital component in improving the user experience across a variety of technological domains and platforms. For example, it can form the basis for sign language understanding and hand gesture control, and can also enable the overlay of digital content and information on top of the physical world in augmented reality. While coming naturally to people, robust real-time hand perception is a decidedly challenging computer vision task, as hands often occlude themselves or each other (e.g. finger/palm occlusions and hand shakes) and lack high contrast patterns.

MediaPipe Hands is a high-fidelity hand and finger tracking solution. It employs machine learning (ML) to infer 21 3D landmarks of a hand from just a single frame. Whereas current state-of-the-art approaches rely primarily on powerful desktop environments for inference, our method achieves real-time performance on a mobile phone, and even scales to multiple hands.

ML pipeline

MediaPipe Hands utilizes an ML pipeline consisting of multiple models working together: A palm detection model that operates on the full image and returns an oriented hand bounding box. A hand landmark model that operates on the cropped image region defined by the palm detector and returns high-fidelity 3D hand keypoints

The pipeline is implemented as a MediaPipe graph that uses a hand landmark tracking subgraph from the hand landmark module, and renders using a dedicated hand renderer subgraph. The hand landmark tracking subgraph internally uses a hand landmark subgraph from the same module and a palm detection subgraph from the palm detection module. 

Models

Palm Detection Model

To detect initial hand locations, Mediapipe designed a single-shot detector model optimized for mobile real-time uses in a manner similar to the face detection model in MediaPipe Face Mesh. Detecting hands is a decidedly complex task: our lite model and full model have to work across a variety of hand sizes with a large scale span (~20x) relative to the image frame and be able to detect occluded and self-occluded hands. Whereas faces have high contrast patterns, e.g., in the eye and mouth region, the lack of such features in hands makes it comparatively difficult to detect them reliably from their visual features alone. Instead, providing additional context, like arm, body, or person features, aids accurate hand localization.

With the above techniques, mediapipe achieve an average precision of 95.7% in palm detection. Using a regular cross entropy loss and no decoder gives a baseline of just 86.22%. (Lugaresi et al., 2019) [8]

Hand Landmark Model

After the palm detection over the whole image our subsequent hand landmark model performs precise keypoint localization of 21 3D hand-knuckle coordinates inside the detected hand regions via regression, that is direct coordinate prediction. The model learns a consistent internal hand pose representation and is robust even to partially visible hands and self-occlusions

Figure 4: Hand landmark model

Choosing 3D engine

Globally, there are currently 6.4 billion smartphone users (Statista, 2021). With a global population of 7.9 billion, this equates to a smartphone penetration rate of more than 80%. To reach the majority of customers that use smartphones, we chose Unity as the 3D development framework. This game engine enables us to easily create cross-platform 3D mobile applications.

Loading ring into right position

From the Hand Landmark there will be total 21 key points and the Finger where most people use to where ring it’s fourth finger next to little finger 

Figure 5: Code get keypoints 

The position for render the ring will be between Key point 13 and Key point 14

Figure 6: Code get keypoints 

At this point get the upper and lower position where put the ring render the gameObject which it’s the Ring Model 

Figure 7: Ring model

After tracking the point in the three-dimensional environment, we load the demo ring model into the Unity 3D engine. The ring just needs to be rendered in real time according to the position of the ring point.

Shader

This package contains shaders that render realistic diamonds, at the same time they work even on mobile devices, they do not require installation of additional scripts; But in LWRP/ URP it works in a slightly different way, first of all it is made for standard render, it does not work on HDRP; For better performance – easy post processes were made that replace the standard one but it works several times faster, which allows you to work on mobile devices

Link to Asset: Diamond Realistic Shaders

Result

Real-time rendering with hand-tracking

FPS

We’ve reached 30 frames per second on a Samsung A7, a fairly standard Android phone, which is rather promising. The ring was successfully rendered and instantaneously affixed to the user’s finger. However, due to picture and model prediction noise, the ring may occasionally be shown in the incorrect place.

Future work

We’re looking forward to increasing the algorithm’s accuracy. Additionally, the vibration or noise in the 3D ring model should be improved. Another future upgrade that can be considered is reading illumination information from the environment to increase reflection.

Contribution & Bibliography

Contribution

Tran Ngoc Hieu Nam, Le Duy Son (YITEC): Research, integrate, and improve MediaPipe model, load 3D model into scene

Minh Dinh, Nguyen Ngoc Anh (YITEC): Debug, test flight Unity 3D app on Xcode

References

[1] Smith, P. (2022, February 18). Topic: Jewelry market worldwide. Statista. Retrieved March 27, 2022, from https://www.statista.com/topics/5163/jewelry-market-worldwide/#dossierKeyfigures 

[2] Gave, S. (n.d.). The success of ecommerce in the jewelry industry. Sales Layer Blog. Retrieved March 28, 2022, from https://blog.saleslayer.com/ecommerce-jewelry-industry

[3] A, Baier, D, Geyer-Schulz, A, et al. (2016) How augmented reality apps are accepted by consumers: A comparative analysis using scales and opinions. Technological Forecasting and Social Change 124: 306–319.

[4] Huang, T-L, Liao, S (2014) A model of acceptance of augmented-reality interactive technology: The moderating role of cognitive innovativeness. Electronic Commerce Research 15(2): 269–295.

[5] Wakim, RS, Drak, Al, Sebai, L, Miladinovic, M, et al. (2018) A study of Swedish eyewear retailer’s smartphone-based augmented reality application. In: Paper presented at the Engineering Technology Management Summit, Istanbul, 4–5 October.

[6] Poushneh, A, Vasquez-Parraga, A (2017) Discernible impact of augmented reality on retail customer’s experience, satisfaction and willingness to buy. Journal of Retailing and Consumer Services 34(C): 229–234.

[7] Ozturkcan, S. (2020). Service innovation: Using augmented … – sage journals. Service innovation: Using augmented reality in the IKEA Place app. Retrieved March 27, 2022, from https://journals.sagepub.com/doi/10.1177/2043886920947110?icid=int.sj-abstract.citing-articles.1 [8] Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., Zhang, F., Chang, C.-L., Yong, M. G., Lee, J., Chang, W.-T., Hua, W., Georg, M., & Grundmann, M. (2019, June 14). MediaPipe: A framework for building perception pipelines. arXiv.org. Retrieved March 28, 2022, from https://arxiv.org/abs/1906.08172