Table of Content:

Extended Reality for Everybody The XR Process can be broken down into four major steps: (1) needfinding & brainstorming, (2) storyboarding & prototyping (both (1) and (2) are the focus of the second course in the XR series), (3) development & testing (the focus of this course), and (4) deployment & analytics.

1. Intro to AR/VR/MR/XR.
2. Design: Design Thinking, Design Ethics, Design Guidelines, Design Jams, Design Critiques.
3. Development: The development of XR applications with WebXR, Unity & Unreal.
4. Some Ideas.

1. Intro to AR/VR/MR/XR

Coursera - Intro to AR/VR/MR/XR: Technologies, Applications & Issues, Certification link.

• Virtual Reality, Augmented Reality, Mixed Reality.
• Four classes of XR technology: devices, platforms, applications, and tools.
• People – Tasks – Technology: the three main angles used to explore the question of how to use our understanding of people, their tasks, and technical capabilities to design better systems?
• Key issues: social and ethical concerns, accessibility and equity, privacy and security.

XR strategy : knowledge, team, equipment, and users.

• Knowledge: XR applications, technologies, key issues, design principles, and development tools.
• Team: designers, developers, managers, artists, researchers, and entrepreneurs.
• Equipment: smartphones and exploring ways to turn them into AR or VR devices.
• Users: growing.

Cases Study :

• Niantic Case Study.
• Oculus Case Studies.
• Microsoft Case Studies.

2. User Experience & Interaction Design for AR/VR/MR/XR

Coursera - User Experience & Interaction Design for AR/VR/MR/XR, Certification link.

• XR Experiences : (1) Environment (Real; Virtual); (2) 3D Characters (Avatars; 3D models; Data); (3) Interactions (Explicit, Implicit).
• Four major steps: (1) needfinding & brainstorming, (2) storyboarding & prototyping, (3) development & testing, and (4) deployment & analytics.

2.1 Design

Design Process:

graph TD
PS["Problem Statement"]
CA["Competitive Analysis"]
DS["Design Space Analysis"]

subgraph PT["Prototypes"]
Paper
Physical
Digital
end

PS --> PT
CA --> PT
DS --> PT

PT --> MVP["Minimum Viable Product (MVP)"]

Design Thinking : Building empathy with your users.

• Challenge : Build empathy with limited users.
• (1) Empathize; (2) Define (the problem); (3) Ideate; (4) Prototype; (5) Test; (6) Implement.
• Design Keys:
  • Problem Statement (Scenarios - Setting, Agents, Goals, Actions, Events).
  • “The User” (Personas).
  • Competitive Analysis (SWOT)..
  • Design Space Analysis (QOC - Questions & Options & Criterion).
• Additional video : 💡 XR Design Theory and Practice for Digital Eyewear 2020

Design Ethics : Ethical and responsible design. About the acquisition and use of personal data.

• Autonomy, Transparency, Safety.
• Ethics Review:
  1. Define purpose, procedure, and duration.
  2. Maximize benefits / limit trade-offs.
  3. Minimize risks.
  4. Empower participants.
• Design Ethics Review:
  1. Context & Situation : where is the XR device being used? Who is around the user?
  2. Sensory Information & Data : What does the XR device see? What kind of data is acquired?
  3. Processing Pipeline : What happens on-device? What happens in the cloud?
  4. Data Ownership & Governance : Who owns and manages the data? How is the user informed?
• Design impact:
  • Before : set expectations of users for new reality.
  • During : balance realism, comfort, safety.
  • After : help users transition back to old reality.

Design Guidelines : Best practices and patterns of design. By Vendors (platform driven), By Designers (user oriented), By Practitioners (experience based), By Researchers (empirically derived).

Design Jams : Solving design challenges collaboratively. Mini project plan; Scenarios; Personas; Storyboards; Physical Prototypes; Digital Prototypes.

Design Critiques : Learning from existing interfaces.

• I like … I wish … What if ?
• Evaluation : Usability; Usefulness; Utility; User Acceptance.

2.2 Brainstorming

Brainstorming & Problem framing. Seven Techniques to Help You Innovate

1. Ideate–Implement–Inspect–Iterate Design Cycle.
2. “Killer-App” Problem / Novelty ≠ Creativity.
3. Problem–Promise–Premise Approach. (why important? what is the value proposition? what are your assumptions?)
4. Write the Intro First Approach. (problem, obstacle, solution in essence, contribution scientifically)
5. Idea Hexagon: a conceptual framework that to navigate through the multifaceted world of invention. How to Think Like an MIT Media Lab Inventor
6. Object-Control Design Matrix.
7. Threshold & Ceiling in Tool Design. (lower-bound and upper-bound of different tools)

2.3 Prototyping

(1) Organizing Sketches into Storyboards. Storyboards organize sketches into sequences to illustrate user flow. Wireframing: blueprint of screen and VR scenes. video playlist

(2) Physical Prototyping, video playlist.

1. Paper prototyping: good for thinking through interactions.
2. 3D modeling with clay : good for creating the main 3d characters.
3. Diorama : good for thinking through proxemics.
4. 360 paper prototyping : good for thinking spatially around the user.

(3) Digital Prototyping digital authoring tools to mock up the XR experience with interactions happening in VR or AR. (1) Digital & Immersive Authoring; (2) Web-based development; (3) Cross-platform development; (4) Native development.

Testing Physical and Digital Prototypes

3. Developing AR/VR/MR/XR Apps with WebXR, Unity & Unreal

Coursera - Developing AR/VR/MR/XR Apps with WebXR, Unity & Unreal, Certification link.

1. Development Approaches - WebXR, Unity, and Unreal.
2. VR Application Development - VR interaction; Object placement. Layers.
3. AR Application Development.
   • Navigating social and ethical concerns when collecting and processing data about the user’s surroundings.
   • Adapting the interactions to a broad range of AR devices including less powerful devices.
   • Making the virtual content accessible to users with special needs.
4. Special Topics.

4. Some Ideas

1. 有时候想要记住一个场景，将它重建并且可以通过VR“回去”。（尤其是曾经的家）
2. AR（HUD和眼镜）的对焦问题。