How to Send Sensor Data to TouchDesigner Using ESP32: A Step-by-Step Guide

Approximately 75% of interactive art installations now depend on real-time sensor data.

The ability to transmit sensor information from microcontrollers to software platforms such as TouchDesigner facilitates a range of dynamic media experiences.

This guide details the process of sending sensor data to TouchDesigner using an ESP32, specifically the LilyGO T-Display S3 and an MPU6050 gyroscope and accelerometer sensor.

The instructions cover fundamental setup procedures, software requirements, and advanced optimization strategies to help bridge the physical and digital worlds for interactive art, immersive environments, interactive performances or just Internet of Things (IoT) prototypes.

This project is designed with a modular architecture, enabling adaptation to diverse use cases.

I will cover the following:

• Detailed instructions for configuring the hardware components.
• Methods for acquiring and processing raw sensor data to ensure accuracy and reliability.
• Implementation of a Node.js relay server for bidirectional data streaming.
• Configuration to receive and process incoming messages for real-time visualization.
• Demonstration of project adaptability through a browser-based interface for monitoring incoming messages.

Why Use ESP32 with TouchDesigner?

The ESP32’s built-in wireless communication capabilities are highly suitable for interactive media projects that require fast, bidirectional data transfer.

When paired with TouchDesigner, which excels at real-time visual programming, the ESP32 can stream motion and environmental data for visualization or audio generation.

In particular:

The ESP32 supports the WebSocket protocol for low-latency, full-duplex data flow, ensuring near-instantaneous feedback crucial for creative installations.

The MPU6050 gyroscope provides acceleration and rotational readings, enabling real-time 3D motion mapping, particle simulations, or other interactive visuals in TouchDesigner.

The ESP32 can control a vibration motor to deliver tactile responses synchronized with visual or auditory events.

Using Wi-Fi and Bluetooth, multiple ESP32 boards can communicate sensor data concurrently, supporting larger-scale interactive systems.

System Architecture and Components

Hardware Requirements

To build this system, the following components are required:

1. Lilygo T-Display S3 (ESP32-S3 Development Board): Features a dual-core processor and native Wi-Fi/Bluetooth and includes an integrated display.
2. MPU6050 Gyroscope: Provides three-axis acceleration and gyroscopic data, critical for motion-sensitive installations.
3. Push Button (with LED): This allowing you to quickly switch between different functionalities, such as calibration mode and active motion capture.
4. Haptic Feedback Module: Comprises a small vibration motor controlled by a PWM pin on the ESP32 for tactile feedback.
5. 1200mAh LiPo Battery: Ensures mobility. Properly rated batteries are essential for stable operation and untethered use.

I also designed a custom 3D enclosure to house all the components neatly.

While the design went through several iterations and isn’t perfect, it works well for this project.

You can download the STL files from the GitHub page and modify them to suit your needs.

Whether you want to polish it further or adapt it to your specific setup, feel free to make it your own!

Software Stack

The following software stack is essential:

1. Arduino IDE: Simplifies ESP32 programming, including libraries for JSON data transmission, gyroscope handling, and WebSockets.
2. Node.js: Capable of hosting a WebSocket relay server to distribute incoming sensor data to multiple clients.
3. TouchDesigner: Through the WebSocket DAT and CHOP operators, TouchDesigner can map incoming motion data to dynamic visuals and sound.
4. ReactJS (Optional): Provides a browser-based dashboard for real-time sensor data visualization, often useful for remote monitoring or live demonstrations.

To utilize this specific development board, it is essential to configure the required libraries prior to implementation.

Begin with my guide on setting up the LilyGo T-Display S3, which provides step-by-step instructions for proper configuration and initialization.

Communication Workflow: WebSockets + Serial Backup

WebSocket Protocol

The ESP32 WebSocket protocol supports real-time data streaming and control messages, suitable for installations demanding rapid feedback loops.

Serial Backup

Although WebSocket communication is prioritized, serial connections remain vital for debugging or fallback communication if wireless links drop.

Data Structure: JSON Format

A typical JSON payload may resemble the following:

{
  "ax": 0.0,
  "ay": 0.0,
  "az": 0.0,
  "gx": 0.0,
  "gy": 0.0,
  "gz": 0.0,
  "acc": 0.0,
  "btn": 0,
}

JSON is lightweight, straightforward to parse, and compatible across TouchDesigner, Node.js, and ReactJS or other IoT environments.

Coding the ESP32: Motion, Haptics, and Data Transmission

Step 1: Wiring the MPU6050 and Vibration Motor

Here’s how to connect the hardware components:

The MPU6050:

1. VCC → ESP32 3.3V
2. GND → ESP32 GND
3. SDA → ESP32 GPIO17
4. SCL → ESP32 GPIO18

The Vibration Motor:

1. Input terminal → ESP32 GPIO10

The Push Button & LED:

1. Button Pin A → ESP32 GPIO3
2. Button Pin B → ESP32 GND
3. LED Positive Pin → ESP32 GPIO2
4. LED Negative Pin → ESP32 GND

Step 2: Writing Arduino Code

A basic sketch involves:

1. Initializing and calibrating the MPU6050.
2. Acquiring and filtering accelerometer and gyroscope data.
3. Making the data available for network transmission or display output.
4. Sending the filtered data via WebSocket for real-time use in TouchDesigner or other clients.

An illustrative setup snippet is shown below:

void setup() {
    Serial.begin(115200);
    pinMode(DISPLAY_POWER_PIN, OUTPUT);
    digitalWrite(DISPLAY_POWER_PIN, HIGH); // Power the display

    // Initialize button and LED pins.
    pinMode(BUTTON_PIN, INPUT_PULLUP);
    pinMode(LED_PIN, OUTPUT);
    digitalWrite(LED_PIN, LOW);
    
    // Initialize modules
    Vibration::begin(10);
    tft.init();
    tft.setRotation(3);
    
    if (!MPU6050Sensor::begin(17, 18)) {
        tft.println("MPU6050 Error!");
        while(1);
    }

    tft.println("Calibrating...");
    MPU6050Sensor::calibrate(); // Calibrate for 3 seconds
    tft.fillScreen(TFT_BLACK);

    // Start WiFi Manager
    startWiFiManager();
    preferences.begin("wifi_config", true);
    ws_server = preferences.getString("server_ip", "");
    preferences.end();
    
    if (wifiConnected && !ws_server.isEmpty()) {
        initializeWebSocket(ws_server);
    }
}

Full project files are available in the GitHub Repository.

Building the Data Pipeline: Node.js, TouchDesigner, or ReactJS

Node.js WebSocket Relay Server

A Node.js server can relay messages from the ESP32 to all connected clients:

wss.on("connection", (ws) => {
	console.log("New client connected");

	ws.on("message", (message) => {
	   try {
	      const sensorData = JSON.parse(message);
              console.log("Sensor Data:", sensorData);

	      wss.clients.forEach((client) => {
		if (client.readyState === WebSocket.OPEN) {
	   	   client.send(JSON.stringify(sensorData));
		}
   	      });
	} catch (error) {
	   console.error("Error processing message:", error);
           console.log("Raw message that failed:", message);
        }
      });
      ws.on("close", () => {
         console.log("Client disconnected");
      });
});

TouchDesigner: Visualizing Motion Data

1. WebSocket DAT: Connect to the Node.js server to receive sensor data.
2. Data Selection: Use Select and Convert DATs to isolate relevant numerical values.
3. CHOP Integration: Employ a DAT to CHOP node for direct manipulation within TouchDesigner’s real-time environment.
4. Visual Mapping: Link CHOP channels to 3D objects or particle systems to generate dynamic motion-dependent effects.

ReactJS Dashboard & Visualization

React is a widely utilized JavaScript library designed for developing user interfaces. 

Applications created with React are recognized for their interactivity, responsiveness, and efficiency. 

This is achieved through the use of components—modular and reusable units of UI—and the virtual DOM, which ensures that only the necessary portions of the interface are updated when changes occur. 

A ReactJS-based interface can display sensor streams for rapid testing or for many cases, I use react to develop low-budget interactive art installations.

Use react-use-websocket to connect to the WebSocket server.

Advanced Tips for Integration

To optimize the functionality, performance, reliability, and interactivity of your project, consider implementing the following advanced strategies and best practices.

Power Management for ESP32-S3

To maximize battery life in mobile installations, it is crucial to implement efficient power management strategies, including ESP32 LiPo battery optimization and peripheral control.

Key tactics include:

• Deep Sleep Mode: Utilize the ESP32’s deep sleep functionality to deactivate unnecessary components during idle periods. For instance, the MPU6050 gyroscope can be configured to trigger a wake-up upon detecting motion:

esp_sleep_enable_ext0_wakeup(GPIO_NUM_21, 1); // Wake up on MPU6050 interrupt
esp_deep_sleep_start();

• Dynamic Frequency Scaling (DFS): Adjust the CPU clock speed dynamically to balance performance requirements and power consumption.
• Peripheral Power Management: Disable unused peripherals, such as Wi-Fi and Bluetooth, when not in active use. For example:

WiFi.mode(WIFI_OFF);
btStop();

Haptic Feedback Tuning

When implementing haptic feedback with the ESP32, pulse-width modulation (PWM) signals are used to modulate intensity (or digital control-depending on your module).

To prevent overheating and preserve battery power, consider the following optimizations:

• Intensity Modulation: Adjust the motor’s PWM signal based on motion magnitude. For instance, more intense motion triggers stronger vibrations.
• Feedback Timing: Avoid continuous motor activation by introducing cooldown intervals between activations. This prevents motor overheating and conserves energy.
• Directional Feedback: For systems with multiple motors, trigger specific motors based on the direction of detected movement (e.g., front, back, left, or right).

TouchDesigner Optimization

To ensure seamless performance in TouchDesigner, focus on maintaining efficiency in data handling and visualization:

• WebSocket Reconnection Handling: Use the Execute DAT to monitor WebSocket connection status and implement automatic reconnection if the connection is lost.
• CHOP Optimization: Reduce the number of active CHOP channels by consolidating related data streams or lowering sampling rates where high precision is unnecessary.
• GPU Acceleration: Leverage GPU-accelerated nodes for computationally intensive tasks, such as particle simulations and 3D rendering, to minimize CPU overhead.

Data Compression

Efficient bandwidth usage is critical for real-time systems.

Optimize data payloads by:

• Shortening JSON Keys: Replace verbose keys such as "x_axis", "y_axis", and "z_axis" with shorter alternatives like "x", "y", and "z" to reduce data size.
• Quantizing Data: Round sensor readings to a practical level of precision (e.g., two decimal places) to reduce payload size without significant loss of accuracy.

Modular Code Design

Maintain a modular code structure to enhance scalability, readability, and ease of debugging:

• Separate Functions: Isolate key functionalities such as MPU6050 data acquisition (e.g. MPU6050Sensor.h & MPU6050Sensor.cpp), WebSocket communication, and motor control into distinct modules.
• Configuration Files: Use a centralized configuration file to define hardware pin mappings, thresholds, and other constants. This approach simplifies updates and facilitates reuse in future projects.

By adhering to these strategies, your system will achieve improved reliability, efficiency, and adaptability, enabling seamless integration into diverse use cases.

Troubleshooting Common Issues

Even with great planning, challenges may arise during development.

Below is a guide to address some of the most frequent issues:

WebSocket Connection Timeouts

WebSocket connections may unexpectedly timeout or disconnect due to network instability or configuration errors.

• Increase Ping Interval: Configure the WebSocket server and client to send periodic ping messages to maintain the connection:

const server = new WebSocket.Server({ port: 8080, keepAliveTimeout: 30000 });

• Handle Reconnection: Add logic in the WebSocket client to detect disconnections and automatically reconnect:

if (!client.connected()) {
  client.connect("ws://your-server-ip:8080");
}

JSON Parsing Errors

Errors in JSON formatting can disrupt data handling in TouchDesigner or ReactJS.

• Debugging Raw Data: Use the Text DAT in TouchDesigner to inspect incoming messages before parsing.
• Validation in Arduino: Implement checks in your Arduino code to ensure valid JSON formatting:

if (isnan(x) || isnan(y) || isnan(z)) return; // Skip invalid data

ReactJS CORS Errors

Cross-Origin Resource Sharing (CORS) restrictions may prevent a ReactJS app from connecting to a WebSocket server hosted on a different domain or port.

• Enable CORS in Node.js: Modify the server to include the appropriate headers:

const server = new WebSocket.Server({ port: 8080 });
server.on('headers', (headers) => {
  headers.push("Access-Control-Allow-Origin: *");
});

• Use a Proxy: Configure a development proxy in React to bypass CORS restrictions during testing.

MPU6050 Calibration Issues

Inaccurate or erratic motion data from the MPU6050 gyroscope may indicate calibration problems.

• Static Offset Calibration: Measure and subtract baseline offset values for both accelerometer and gyroscope readings:

int16_t ax_offset = -500; // Example offset value
ax = ax - ax_offset;

• Dynamic Calibration: Implement a startup calibration routine to adjust offsets based on initial readings dynamically.

TouchDesigner Performance Lag

Large data streams or complex visualizations can degrade performance in TouchDesigner.

• Reduce Update Rates: Throttle data updates from the ESP32 to transmit only when a significant change is detected.
• Use Null CHOPs: Insert Null CHOP nodes to optimize processing by caching data and reducing unnecessary computations.

LiPo Battery Drain

Rapid battery depletion can hinder portability and system uptime.

• Battery Health Check: Monitor the battery voltage using the ESP32’s ADC and trigger low-battery warnings as needed:

int batteryLevel = analogRead(A0);
if (batteryLevel < 3.2) {
  // Trigger low battery alert
}

• Power-Saving Features: Disable unused peripherals or enable sleep modes when the device is idle to conserve power.

By addressing these common challenges and employing proactive strategies, you can enhance the reliability, efficiency, and performance of your ESP32-S3 and TouchDesigner projects.

These techniques ensure a smooth development process and empower you to deliver robust and interactive installations with confidence.

Conclusion

Streaming sensor data from an ESP32 to TouchDesigner provides a robust framework for interactive installations, IoT prototypes, and responsive media experiences.

By adhering to best practices in hardware setup, WebSocket networking, data structuring, and software integration, creators can seamlessly connect physical motion or environmental data with real-time visual and auditory outputs.

This step-by-step methodology enables developers, artists, and researchers to generate immersive, motion-responsive environments that enrich audience engagement.

Got questions, project ideas, or just want to say hi?

Drop a comment under the tutorial video on YouTube—I’d love to hear from you!

Frequently Asked Questions (FAQ)