3D USDZ Models with ARKit & RealityKit : AR with iOS (Part-III)

3D USDZ Models

what is usdz : USDZ is a file format developed by Pixar and Apple for sharing and displaying 3D content in AR and VR environments. It’s based on Pixar’s Universal Scene Description (USD), and is designed to be lightweight and efficient. USDZ files can contain 3D models, textures, animations, and other assets needed to render a complete 3D scene.

ARKit is a framework developed by Apple for building AR apps on iOS devices. While USDZ is the recommended format for creating and sharing 3D models in ARKit, the framework also supports other common 3D file formats such as OBJ, Collada (DAE), and Alembic (ABC).

Using USDZ format can have advantages over other formats, such as improved performance, better support for materials and lighting, and easier integration with Apple’s Quick Look and Reality Composer tools. Additionally, USDZ models can be easily previewed in AR using the Quick Look feature on iOS devices.

Some Resources for free USDZ models

1. Apple’s USDZ Sample Gallery: https://developer.apple.com/augmented-reality/quick-look/
2. Sketchfab: https://sketchfab.com/
3. TurboSquid: https://www.turbosquid.com/
4. CGTrader: https://www.cgtrader.com/
5. Free3D: https://free3d.com/
6. Or one can convert other 3D model formats to USDZ using Reality Converter

Adding 3D Model in project

1. First create Assets folder in project
2. Download add few USDZ models from Apple’s official site or any other source
3. Add these USDZ models in Assets with “File” > “Add Files to …” > select USDZ files > “Add”
4. Similarly, we can add images in Assests.xcassets as icons for these USDZ models

Few AR Concepts

• ARViewContainer: This is a UIViewRepresentable that wraps the ARView from the RealityKit framework. The ARView is the main view for displaying augmented reality content, and the ARViewContainer is responsible for managing the creation and updating of the view.
• ModelEntity: This is a type from the RealityKit framework that represents a 3D model that can be added to the augmented reality scene. It is loaded from a file using the load(named:) method.
• AnchorEntity: This is a type from the RealityKit framework that represents a position and orientation in the augmented reality scene. A ModelEntity can be added as a child of an AnchorEntity to place it in the scene. The AnchorEntity is added to the ARView's scene using the addAnchor(_:) method.

Code Explanation

overview

When the user selects a model, the corresponding 3D model is loaded using ModelEntity and added to an AnchorEntity. The AnchorEntity is then positioned 1 meter in front of the camera and added to the AR view scene.

import SwiftUI
import RealityKit

struct ContentView : View {
    
    let modelNames = ["toy_car", "robot_walk_idle", "toy_biplane_idle", "toy_drummer_idle"]
    
    // State variable for selected model index
    @State private var selectedModelIndex = 0
    
    var body: some View {
        VStack {
            // AR view container with selected model
            ARViewContainer(modelName: modelNames[selectedModelIndex])
                .frame(height: 400)
            
            // Picker for selecting the model
            Picker("Select Model", selection: $selectedModelIndex) {
                ForEach(0 ..< modelNames.count) { index in
                    Image(modelNames[index])
                        .resizable()
                        .frame(width: 50, height: 50)
                        .tag(index)
                }
            }
            .pickerStyle(SegmentedPickerStyle())
        }
    }
}

struct ARViewContainer: UIViewRepresentable {
    let modelName: String
    
    func makeUIView(context: Context) -> ARView {
        let arView = ARView(frame: .zero)
        return arView
    }
    
    func updateUIView(_ uiView: ARView, context: Context) {
        // Remove any existing anchor entities from the scene
        uiView.scene.anchors.removeAll()
        
        // Load the model from the app's asset catalog
        let modelEntity = try! ModelEntity.load(named: modelName + ".usdz")
        
        // Create an anchor entity and add the model to it
        let anchorEntity = AnchorEntity()
        anchorEntity.addChild(modelEntity)
        
        // Set the position of the anchor entity to 1 meter in front of the camera
        anchorEntity.position = [0, 0, -1]
        
        // Add the anchor entity to the scene
        uiView.scene.addAnchor(anchorEntity)
    }
}

The ARView is initially empty when the app is launched, and the user needs to select a model from the Picker for it to be displayed in the ARView. The selected model is displayed at the position of the AnchorEntity which is 1 meter in front of the camera, and the user can move the camera to view the model from different angles and distances.

The ARViewContainer struct creates an instance of ARView, which is a UIView subclass provided by RealityKit. The makeUIView method returns the ARView instance when the view is created, and the updateUIView method is called to update the view. This method removes any existing anchor entities from the scene, loads the model from the app’s asset catalog, creates an anchor entity, adds the model to the anchor entity, sets the position of the anchor entity to 1 meter in front of the camera, and adds the anchor entity to the scene.