Occlusion, Material with ARKit & RealityKit: AR with iOS (Part-VI)

In this article, we will explore how to implement occlusion and material in AR using SwiftUI and RealityKit, two powerful and user-friendly frameworks provided by Apple. We will start with a brief introduction to the main concepts and tools, and then we will dive into the code and build a simple AR app that showcases the capabilities of occlusion and material. So let’s get started and unleash the full potential of AR!

Occlusion

Occlusion refers to the ability of virtual objects to hide behind real-world objects in the scene, as if they were behind a physical barrier. Occlusion helps to create a sense of depth and realism, and it requires precise tracking and depth sensing, as well as accurate geometry and lighting of the virtual content.

Occlusion

// occlusion
self.environment.sceneUnderstanding.options.insert(.occlusion) 

import SwiftUI
import RealityKit
import ARKit
import FocusEntity
import Combine

struct ContentView: View {
    
    var body: some View {
        ZStack(alignment: .bottom) {
            CustomARViewContainer()
            
            Button(action: {
                ActionManager.shared.actionStream.send(.place3DModel)
            }, label: {
                Text("Place 3D Model")
                    .font(.headline)
                    .foregroundColor(.white)
                    .padding()
                    .background(Color.blue)
                    .cornerRadius(10)
            })
            .padding(.bottom, 50)
        }
    }
}

struct CustomARViewContainer: UIViewRepresentable {
    
    func makeUIView(context: Context) -> CustomARView {
        return CustomARView()
    }
    
    func updateUIView(_ uiView: CustomARView, context: Context) {}
}



class CustomARView: ARView {
    
    var focusEntity: FocusEntity?
    var cancellables: Set<AnyCancellable> = []
    
    init() {
        super.init(frame: .zero)
        
        // ActionStrean
        subscribeToActionStream()
        
        // FocusEntity
        self.focusEntity = FocusEntity(on: self, style: .classic(color: .yellow))
        
        // Configuration
        let config = ARWorldTrackingConfiguration()
        config.planeDetection = [.horizontal]
        config.environmentTexturing = .automatic
        
        if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) {
            config.sceneReconstruction = .meshWithClassification
        }
        
        // occlusion
        self.environment.sceneUnderstanding.options.insert(.occlusion)
        
        self.session.run(config)
    }

    
    func place3DModel() {
        guard let focusEntity = self.focusEntity else { return }

        let modelEntity = try! ModelEntity.load(named: "toy_car.usdz")
        let anchorEntity = AnchorEntity(world: focusEntity.position)
        anchorEntity.addChild(modelEntity)
        self.scene.addAnchor(anchorEntity)
    }
    
    
    func subscribeToActionStream() {
        ActionManager.shared
            .actionStream
            .sink { [weak self] action in
                
                switch action {
                    
                case .place3DModel:
                    self?.place3DModel()
                    
                case .remove3DModel:
                    print("Removeing 3D model: has not been implemented")
                }
            }
            .store(in: &cancellables)
    }
    
    @MainActor required dynamic init?(coder decoder: NSCoder) {
        fatalError("init(coder:) has not been implemented")
    }
    
    @MainActor required dynamic init(frame frameRect: CGRect) {
        fatalError("init(frame:) has not been implemented")
    }
}

enum Actions {
    case place3DModel
    case remove3DModel
}

class ActionManager {
    static let shared = ActionManager()
    
    private init() { }
    
    var actionStream = PassthroughSubject<Actions, Never>()
}

Functionality: This code sets up an AR view using SwiftUI and RealityKit that allows the user to place a 3D model in their real-world environment. The CustomARView class is a subclass of ARView that sets up the AR session and adds a yellow focus indicator for the user to interact with. When the user taps the "Place 3D Model" button, it triggers the place3DModel() function which loads a 3D model of a toy car and places it in the real-world environment at the position of the focus indicator.

The occlusion feature is implemented by adding the .occlusion option to the sceneUnderstanding.options of the CustomARView. This tells the ARKit to recognize and track the depth of the real-world objects and use that information to hide the virtual objects. The end result is a more realistic augmented reality experience where virtual objects interact seamlessly with the real world.

The ActionManager class is used to manage the user's actions and communicates them between different parts of the code. It uses a PassthroughSubject to send and receive actions. The subscribeToActionStream() function is called to listen for new actions being sent and performs the appropriate action, either placing or removing a 3D model.

Occlusion

Material

Material refers to the appearance and properties of the virtual objects, such as their color, texture, reflectivity, and transparency. Material is crucial for making the virtual content blend seamlessly with the real-world environment, and it can also convey meaning and emotion through visual cues.

Metallic Material

    func place3DModel() {
        guard let focusEntity = self.focusEntity else { return }

        let modelEntity = try! ModelEntity.load(named: "toy_drummer_idle.usdz")
        
        // adding material
        var metal = SimpleMaterial(color: .clear, isMetallic: true)
        metal.metallic = 1.0
        modelEntity.model?.materials.append(metal)

        let anchorEntity = AnchorEntity(world: focusEntity.position)
        anchorEntity.addChild(modelEntity)
        self.scene.addAnchor(anchorEntity)
    }