Occlusion, Material with ARKit & RealityKit: AR with iOS (Part-VI)

Shiru99
4 min readJun 21, 2023

--

In this article, we will explore how to implement occlusion and material in AR using SwiftUI and RealityKit, two powerful and user-friendly frameworks provided by Apple. We will start with a brief introduction to the main concepts and tools, and then we will dive into the code and build a simple AR app that showcases the capabilities of occlusion and material. So let’s get started and unleash the full potential of AR!

Occlusion

Occlusion refers to the ability of virtual objects to hide behind real-world objects in the scene, as if they were behind a physical barrier. Occlusion helps to create a sense of depth and realism, and it requires precise tracking and depth sensing, as well as accurate geometry and lighting of the virtual content.

Occlusion
// occlusion
self.environment.sceneUnderstanding.options.insert(.occlusion)
import SwiftUI
import RealityKit
import ARKit
import FocusEntity
import Combine

struct ContentView: View {

var body: some View {
ZStack(alignment: .bottom) {
CustomARViewContainer()

Button(action: {
ActionManager.shared.actionStream.send(.place3DModel)
}, label: {
Text("Place 3D Model")
.font(.headline)
.foregroundColor(.white)
.padding()
.background(Color.blue)
.cornerRadius(10)
})
.padding(.bottom, 50)
}
}
}

struct CustomARViewContainer: UIViewRepresentable {

func makeUIView(context: Context) -> CustomARView {
return CustomARView()
}

func updateUIView(_ uiView: CustomARView, context: Context) {}
}



class CustomARView: ARView {

var focusEntity: FocusEntity?
var cancellables: Set<AnyCancellable> = []

init() {
super.init(frame: .zero)

// ActionStrean
subscribeToActionStream()

// FocusEntity
self.focusEntity = FocusEntity(on: self, style: .classic(color: .yellow))

// Configuration
let config = ARWorldTrackingConfiguration()
config.planeDetection = [.horizontal]
config.environmentTexturing = .automatic

if ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) {
config.sceneReconstruction = .meshWithClassification
}

// occlusion
self.environment.sceneUnderstanding.options.insert(.occlusion)

self.session.run(config)
}


func place3DModel() {
guard let focusEntity = self.focusEntity else { return }

let modelEntity = try! ModelEntity.load(named: "toy_car.usdz")
let anchorEntity = AnchorEntity(world: focusEntity.position)
anchorEntity.addChild(modelEntity)
self.scene.addAnchor(anchorEntity)
}


func subscribeToActionStream() {
ActionManager.shared
.actionStream
.sink { [weak self] action in

switch action {

case .place3DModel:
self?.place3DModel()

case .remove3DModel:
print("Removeing 3D model: has not been implemented")
}
}
.store(in: &cancellables)
}

@MainActor required dynamic init?(coder decoder: NSCoder) {
fatalError("init(coder:) has not been implemented")
}

@MainActor required dynamic init(frame frameRect: CGRect) {
fatalError("init(frame:) has not been implemented")
}
}

enum Actions {
case place3DModel
case remove3DModel
}

class ActionManager {
static let shared = ActionManager()

private init() { }

var actionStream = PassthroughSubject<Actions, Never>()
}

Functionality: This code sets up an AR view using SwiftUI and RealityKit that allows the user to place a 3D model in their real-world environment. The CustomARView class is a subclass of ARView that sets up the AR session and adds a yellow focus indicator for the user to interact with. When the user taps the "Place 3D Model" button, it triggers the place3DModel() function which loads a 3D model of a toy car and places it in the real-world environment at the position of the focus indicator.

The occlusion feature is implemented by adding the .occlusion option to the sceneUnderstanding.options of the CustomARView. This tells the ARKit to recognize and track the depth of the real-world objects and use that information to hide the virtual objects. The end result is a more realistic augmented reality experience where virtual objects interact seamlessly with the real world.

The ActionManager class is used to manage the user's actions and communicates them between different parts of the code. It uses a PassthroughSubject to send and receive actions. The subscribeToActionStream() function is called to listen for new actions being sent and performs the appropriate action, either placing or removing a 3D model.

Occlusion

Material

Material refers to the appearance and properties of the virtual objects, such as their color, texture, reflectivity, and transparency. Material is crucial for making the virtual content blend seamlessly with the real-world environment, and it can also convey meaning and emotion through visual cues.

Metallic Material
    func place3DModel() {
guard let focusEntity = self.focusEntity else { return }

let modelEntity = try! ModelEntity.load(named: "toy_drummer_idle.usdz")

// adding material
var metal = SimpleMaterial(color: .clear, isMetallic: true)
metal.metallic = 1.0
modelEntity.model?.materials.append(metal)

let anchorEntity = AnchorEntity(world: focusEntity.position)
anchorEntity.addChild(modelEntity)
self.scene.addAnchor(anchorEntity)
}

--

--