AR Persistence with ARKit & RealityKit [in Details]: AR with iOS (Part-VIII)

Refer Part VII for intro of AR Persistence & ARWorldMap. We’ll explore functionalities of ARPersistence package here.

AR Persistence

Save AR Functionality

public func save(_ uiView: ARView) {
    var modelEntityDetailsDict: [String: [String]] = [:]
    
    // Iterate through AR anchors
    if let arAnchors = uiView.session.currentFrame?.anchors {
        for arAnchor in arAnchors {
            if let modelName = arAnchor.name,
               let identifier = arAnchor.identifier {
                
                // Retrieve model entity and transform details
                if let anchorEntity = uiView.scene.findEntity(named: identifier) as? AnchorEntity,
                   let modelEntity = anchorEntity.children.first as? ModelEntity {
                    let transformDetails = [
                        modelEntity.transform.translation.debugDescription,
                        modelEntity.transform.rotation.debugDescription,
                        modelEntity.transform.scale.debugDescription
                    ]
                    modelEntityDetailsDict["\(modelName)@\(identifier.uuidString)"] = transformDetails
                }
            }
        }
    }
    
    // Save AR Anchors as WorldMap
    let anchorDataURL = self.worldMapDir.appendingPathComponent("ARExperience.map")
    uiView.session.getCurrentWorldMap { worldMap, error in
        guard let worldMap = worldMap else { return }
        do {
            let anchorData = try NSKeyedArchiver.archivedData(withRootObject: worldMap, requiringSecureCoding: true)
            try anchorData.write(to: anchorDataURL, options: [.atomic])
        } catch {
            fatalError("Debug: Can't save map: \(error.localizedDescription)")
        }
    }
    
    // Save modelEntity transforms as JSON
    let modelEntityDataURL = self.worldMapDir.appendingPathComponent("ARExperience.json")
    do {
        let jsonData = try JSONSerialization.data(withJSONObject: modelEntityDetailsDict, options: [])
        try jsonData.write(to: modelEntityDataURL)
    } catch {
        print("Debug: \(error.localizedDescription)")
    }
}

Summary: The save method is responsible for saving the AR anchors and model entity transforms for AR persistence. It iterates through the AR anchors, retrieves the model entity and its transform details, and stores them in a dictionary. Then, it saves the AR anchors as a WorldMap using the getCurrentWorldMap method. Finally, it saves the model entity transforms as JSON. This method allows the user to save the positions and transformations of virtual objects placed in the scene for future retrieval and AR experiences.

Detailed Explanation:

Let’s go through the save function step by step:

1. Create a dictionary modelEntityDetailsDict to store the details of the model entities' transformations. The keys of the dictionary will be a combination of the model name and the anchor identifier.
2. Iterate through the AR anchors present in the current AR frame.
3. For each anchor, check if it has a name and an identifier. If both are available, proceed to retrieve the model entity and its transformation details.
4. Using the anchor’s identifier, find the corresponding AnchorEntity in the AR scene using the findEntity(named:) method. Then, access the first child of the AnchorEntity, assuming it is the model entity.
5. Retrieve the translation, rotation, and scale values of the model entity’s transformation and store them as an array of strings in the transformDetails variable.
6. Add an entry to the modelEntityDetailsDict dictionary using the combination of the model name and the anchor identifier as the key and the transformDetails as the value.
7. Define the URL for storing the AR anchors as a World Map. The file will be named “ARExperience.map” and stored in the worldMapDir directory.
8. Call getCurrentWorldMap on the AR session to obtain the current world map.
9. Inside the completion block, check if the worldMap is not nil. If it exists, archive the world map data using NSKeyedArchiver and write it to the anchorDataURL.
10. If an error occurs during the archiving and writing process, handle it by reporting a fatal error.
11. Define the URL for storing the model entity transforms as JSON. The file will be named “ARExperience.json” and stored in the worldMapDir directory.
12. Serialize the modelEntityDetailsDict dictionary into JSON data using JSONSerialization.data(withJSONObject:options:).
13. Write the JSON data to the modelEntityDataURL.
14. If an error occurs during the serialization and writing process, print the error description for debugging purposes.

In summary, the save function iterates through the AR anchors in the current frame, retrieves the transformation details of the associated model entities, and stores both the AR anchors as a World Map and the model entity transforms as a JSON file. This enables saving the state of the AR experience, allowing it to be reloaded and restored at a later time with the same spatial alignment and placed virtual objects.

Load AR Functionality

public func load(_ uiView: ARView) {
    // Load AR Anchors
    let anchorDataURL = self.worldMapDir.appendingPathComponent("ARExperience.map")
    
    do {
        let anchorData = try Data(contentsOf: anchorDataURL)
        if let worldMap = try NSKeyedUnarchiver.unarchivedObject(ofClass: ARWorldMap.self, from: anchorData) {
            let configuration = ARWorldTrackingConfiguration()
            configuration.initialWorldMap = worldMap
            uiView.session.run(configuration, options: [.resetTracking, .removeExistingAnchors])
        }
    } catch {
        print("Debug: Can't load map: \(error.localizedDescription)")
    }
    
    // Load modelEntity transforms
    let modelEntityDataURL = self.worldMapDir.appendingPathComponent("ARExperience.json")
    
    do {
        let jsonData = try Data(contentsOf: modelEntityDataURL)
        if let modelEntityDetailsDict = try JSONSerialization.jsonObject(with: jsonData, options: []) as? [String: [String]] {
            for (key, transformDetails) in modelEntityDetailsDict {
                let components = key.components(separatedBy: "@")
                guard components.count == 2 else { continue }
                
                let modelName = components[0]
                let identifier = components[1]
                
                let anchorEntity = AnchorEntity()
                anchorEntity.name = identifier
                uiView.scene.addAnchor(anchorEntity)
                
                let modelEntity = try ModelEntity.loadModel(named: modelName + ".usdz")
                modelEntity.transform.translation = parseVector3(transformDetails[0])
                modelEntity.transform.rotation = parseQuaternion(transformDetails[1])
                modelEntity.transform.scale = parseVector3(transformDetails[2])
                anchorEntity.addChild(modelEntity)
                
                for animation in modelEntity.availableAnimations {
                    modelEntity.playAnimation(animation.repeat())
                }
            }
        } else {
            print("Debug: Can't load map")
        }
    } catch {
        print("Debug: \(error.localizedDescription)")
    }
}

Summary: The load method in the AR persistence code loads saved AR anchors and model entity transforms. It retrieves the ARWorldMap from the saved data and uses it to initialize the ARWorldTrackingConfiguration, which is then used to run the AR session. It also retrieves the model entity transform details from the JSON data and creates corresponding anchor entities with model entities in the AR scene. The method enables animations for the loaded model entities.

Detailed Explanation:

Let’s break down the load functionality in detail:

1. Define the URL for the stored AR anchors as a World Map. The file is named “ARExperience.map” and located in the worldMapDir directory.
2. Attempt to read the data from the anchorDataURL using Data(contentsOf:).
3. If the data is successfully loaded, try to unarchive it using NSKeyedUnarchiver.unarchivedObject(ofClass:from:) as an instance of ARWorldMap.
4. If the unarchiving is successful and the worldMap exists, create a new ARWorldTrackingConfiguration and set its initialWorldMap property to the loaded worldMap.
5. Run the AR session with the new configuration, specifying options to reset the tracking and remove any existing anchors.
6. If an error occurs during the loading process, print the error description for debugging purposes.
7. Define the URL for the stored model entity transforms as a JSON file. The file is named “ARExperience.json” and located in the worldMapDir directory.
8. Attempt to read the data from the modelEntityDataURL using Data(contentsOf:).
9. If the data is successfully loaded, try to deserialize it using JSONSerialization.jsonObject(with:options:) as a dictionary with string keys and array values.
10. If the deserialization is successful and the modelEntityDetailsDict exists, iterate through its key-value pairs.
11. Split each key into components using the “@” separator. Ensure that there are exactly two components; otherwise, skip the current iteration.
12. Extract the model name and identifier from the components.
13. Create a new AnchorEntity and set its name to the identifier.
14. Add the AnchorEntity to the AR scene using uiView.scene.addAnchor(_:).
15. Load the model entity using ModelEntity.loadModel(named:), appending the model name with the ".usdz" extension.
16. Set the translation, rotation, and scale of the model entity’s transformation using the parsed vector and quaternion values from the transformDetails array.
17. Add the model entity as a child of the AnchorEntity.
18. Play all available animations of the model entity in a repeating manner.
19. If the JSON deserialization fails or the modelEntityDetailsDict does not exist, print a debug message indicating the failure.
20. If an error occurs during the loading process, print the error description for debugging purposes.

In summary, the load function loads the previously saved AR anchors from a World Map file and restores the corresponding model entity transforms from a JSON file. It sets up the AR session with the loaded world map, creates anchor entities, loads model entities, applies their transformations, and plays animations if available.

Share AR Experience

To share the AR experience with others, the generated .map and .json files can be shared. These files contain the necessary information to recreate the saved AR anchors and model entity transforms. By sharing these files, other users can load the same AR scene with the exact positions and transformations of virtual objects. This allows for a consistent and synchronized AR experience across multiple devices, enabling users to collaborate, explore, and interact with the same augmented environment. Whether it's for educational purposes, gaming, or remote assistance, sharing these files empowers users to share their AR creations and immerse others in their augmented reality world.