tobi/esp8266-weather-clock-opensource
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: trim README, extract architecture notes to docs/ARCHITECTURE.md

Browse Source 
README went from 1094 to 671 lines (-39%) by removing:
- "Technical Deep Dive" section (had stale NTPState enum showing removed values;
  useful content moved to docs/ARCHITECTURE.md)
- "The Journey: v1.7 → v1.9.8" — ~250 lines duplicating CHANGELOG;
  replaced with one-paragraph summary and CHANGELOG link
- "Memory Evolution" table — already in CHANGELOG entries
- "What's Next: Home Assistant Integration" — year-old vaporware promise
- "Lessons Learned" section — editorial blog filler
- "Final Thoughts" + "P.S." footer — same

Added docs/ARCHITECTURE.md (~110 lines) with the useful technical content:
async model, timer correctness, memory budget, IRAM discipline,
heap fragmentation pattern, EEPROM layout, factory reset, web server.

TOC restructured into "Story" (history) and "Use it" (practical) sections.
This commit is contained in:
Alex Petrochenko
2026-05-19 16:24:35 +01:00
parent 259eda6663
commit 75b6a21f3c
2 changed files with 138 additions and 443 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+20 -443
View File
@@ -22,19 +22,25 @@ I bought a cute weather clock kit from AliExpress ([TJ-56-654](https://pt.aliexp
## Table of Contents
**Story**
- [The Discovery: When "Smart" Means "Insecure"](#the-discovery-when-smart-means-insecure)
- [The Device](#the-device)
- [The Investigation](#the-investigation)
- [The Solution: Custom Firmware](#the-solution-custom-firmware)
- [Technical Deep Dive](#technical-deep-dive)
- [The Journey: v1.7 → v1.9.8](#the-journey-v17--v198)
- [What's Next: Home Assistant Integration](#whats-next-home-assistant-integration)
- [Hardware Quirk: Swapped I2C Pins](#hardware-quirk-swapped-i2c-pins)
- [Version History](#version-history)
**Use it**
- [How to Flash This Firmware](#how-to-flash-this-firmware)
- [Web Interface](#web-interface)
- [API Documentation](#api-documentation)
- [Testing](#testing)
- [Security Improvements](#security-improvements)
- [Lessons Learned](#lessons-learned)
- [Credits](#credits)
- [Project Structure](#project-structure)
For internal architecture notes (state machines, memory, EEPROM layout), see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
---
@@ -246,396 +252,23 @@ All endpoints return JSON:
---
## Technical Deep Dive
## Hardware Quirk: Swapped I2C Pins
### Architecture: Fully Async State Machines
The firmware uses **zero blocking operations** in the main loop. Everything is state-machine-based:
#### Weather State Machine
ESP-01S exposes only GPIO0 and GPIO2. The TJ-56-654 board designer used them as I2C — but **swapped from typical breakouts**:
```cpp
enum WeatherState { IDLE, REQUESTING, SUCCESS, FAILED };
Wire.begin(0, 2); // SDA=GPIO0, SCL=GPIO2 (non-standard!)
```
Uses `AsyncHTTPRequest` library:
Standard ESP8266 boards use GPIO4=SDA, GPIO5=SCL. Easy to miss if you're porting code from a different ESP8266 setup. The display is a GM009605v4.3 OLED (SSD1306-compatible) at I2C address 0x3C.
- Non-blocking HTTP requests
- Callback-based response handling
- Exponential backoff on failures (1s → 2s → 4s)
- Maximum 3 retries before giving up
For architecture details (async state machines, memory budget, EEPROM layout, factory reset), see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
#### NTP State Machine
## Version History
```cpp
enum NTPState { IDLE, REQUEST_SENT, WAITING, SUCCESS, FAILED };
```
The clock has gone through many iterations — display hardware discovery (v1.5–v1.7), stability and security fixes (v1.8), full async refactor (v1.9.0), and a long series of bug fixes informed by community reports and AI-assisted code review (v1.9.1–v1.9.9).
Custom manual NTP implementation:
- Builds raw UDP packets (48 bytes)
- Non-blocking `parsePacket()` checks
- 5-second timeout
- Independent epoch tracking for accuracy between syncs
#### WiFi State Machine
```cpp
enum WiFiConnectionState { IDLE, CONNECTING, CONNECTED, FAILED };
```
**Hybrid model** (this was critical!):
- **Setup phase**: Synchronous connection (waits up to 10 seconds)
- Why? OTA, web server, NTP all need WiFi ready
- Without this, device shows blank display for 10+ seconds
- **Loop phase**: Async reconnection (checks every 5 seconds)
- Why? Don't freeze the entire system if WiFi drops
### Memory Optimization
ESP8266 has strict memory limits:
| Memory Type | Total | Used | Usage | Status |
| ----------- | --------- | ------- | ------- | ----------- |
| **Flash** | 1,048,576 | 408,844 | 38% | ✅ Plenty |
| **RAM** | 80,192 | 37,644 | 46% | ✅ Safe |
| **IRAM** | 65,536 | 61,987 | **94%** | ⚠️ Critical |
**IRAM Crisis Solution:**
IRAM (Instruction RAM) is limited and fills fast. The solution: `ICACHE_FLASH_ATTR` macro.
```cpp
void ICACHE_FLASH_ATTR handleConfig() {
// This function's code lives in Flash, not IRAM
// Saves precious IRAM at cost of slightly slower execution
}
```
Applied to 26 functions (web handlers, display, config utilities), reducing IRAM pressure from **overflow risk** to **sustainable 94%**.
**String Safety:**
Avoid String concatenation in loops (causes heap fragmentation):
```cpp
// ❌ BAD - 140+ concatenations
String html = "";
html += F("<!DOCTYPE html>");
html += F("<head>..."); // x138 more times
// ✅ GOOD - Chunked responses
server.setContentLength(CONTENT_LENGTH_UNKNOWN);
server.send(200, "text/html", "");
server.sendContent_P(HTML_HEADER);
server.sendContent_P(HTML_FOOTER);
server.sendContent(""); // End
```
### Configuration Storage
26-field struct stored in EEPROM (512 bytes):
```cpp
struct Config {
char ssid[32];
char password[64];
int timezone_offset;
bool dst_enabled;
uint8_t brightness;
char ntp_server[64];
unsigned long ntp_interval;
bool hour_format_24;
char hostname[32];
float latitude;
float longitude;
char city_name[32];
bool weather_enabled;
unsigned long weather_interval;
unsigned long display_rotation_sec;
bool show_weather;
bool show_sunrise_sunset;
uint8_t display_orientation;
uint32_t magic; // 0xC10CC10C - validation
};
```
**EEPROM validation**: Magic number check prevents loading corrupted data. On validation failure, gracefully defaults to safe config.
### Display Hardware Discovery
This took **3 firmware iterations** to get right:
**v1.5**: Assumed TM1637 (7-segment LED driver)
- ❌ Wrong - device has OLED, not 7-segment LEDs
**v1.6**: Tried TM1650 (another LED driver)
- ❌ Wrong - I2C addresses didn't match
**v1.7**: Identified GM009605v4.3 (SSD1306-compatible OLED)
- ✅ Correct! Used Adafruit_SSD1306 library
- ✅ Discovered swapped pins: SDA on GPIO0, SCL on GPIO2
**Pin mapping quirk:**
Standard ESP8266 I2C uses GPIO4 (SDA) and GPIO5 (SCL), but ESP-01S only exposes GPIO0 and GPIO2. The board designer mapped:
- GPIO0 → SDA (unusual)
- GPIO2 → SCL (unusual)
This is **backwards** from typical breakout boards, but works perfectly once configured:
```cpp
Wire.begin(0, 2); // SDA=GPIO0, SCL=GPIO2
```
---
## The Journey: v1.7 → v1.9.8
### v1.7: Display Discovery ✅
- Identified correct display hardware
- Basic time display working
- WiFiManager integration
- First OTA deployment
### v1.8: Stability & Security 🔒
**Goals**: Fix memory issues, eliminate security holes
**Changes:**
- IRAM optimization (added `ICACHE_FLASH_ATTR` to 26 functions)
- Removed hardcoded WiFi credentials
- Fixed NTP interval bug (config value was ignored)
- Fixed boolean parsing in JSON import
- Added input validation (buffer overflow protection)
- Chunked HTTP responses (eliminated 140+ String concatenations)
**Result**: IRAM usage 94% → 70%, no memory leaks, secure config storage
### v1.9.0: Full Async Refactoring ⚡
**Goals**: Eliminate all blocking operations
**Changes:**
- Async HTTP weather fetch (AsyncHTTPRequest library)
- Custom async NTP implementation (manual UDP packets)
- Async WiFi connection (state machine)
- Removed all `delay()` calls from `loop()`
- Exponential backoff retry logic
**Performance:**
| Operation | Before (v1.8) | After (v1.9.0) | Improvement |
| -------------- | -------------- | -------------- | ------------------- |
| Weather fetch | 1-10s blocking | 0ms | ✅ Async callback |
| NTP sync | 5-20s blocking | 0ms | ✅ Non-blocking UDP |
| WiFi reconnect | 15s blocking | 0ms | ✅ State machine |
| Loop time | 10ms minimum | <1ms | ✅ 10x faster |
**Result**: Device stays responsive during OTA updates while weather is fetching!
### v1.9.1: Hybrid Fix 🎯
**Problem Discovered:**
After deploying v1.9.0, the display showed **blank screen for 10 seconds** after boot, with `DNS resolution failed` errors in logs.
**Root Cause:**
Making WiFi fully async broke the **initialization order**:
```cpp
void setup() {
setupWiFi(); // Returns immediately (async)
setupOTA(); // WiFi NOT ready! ❌
setupWebServer(); // WiFi NOT ready! ❌
testInternetConnectivity(); // WiFi NOT ready! → DNS error
}
```
**Solution: Hybrid Model**
| Phase | WiFi Mode | Blocking? | Why? |
| --------- | ------------ | --------- | --------------------------- |
| `setup()` | Synchronous | 10s max | OTA/web/NTP need WiFi ready |
| `loop()` | Asynchronous | 0s | Don't freeze on reconnect |
**Results:**
- ✅ Display shows time immediately after WiFi connects (~15 sec boot)
- ✅ No "DNS resolution failed" errors
- ✅ Proper initialization order guaranteed
- ✅ Device never freezes on WiFi loss during operation
### v1.9.2: WiFi Resilience 🛡️
**Problem Discovered:**
After WiFi outages, the device would **clear stored credentials** and enter AP mode, requiring manual reconfiguration every time the router restarted.
**Root Cause:**
Aggressive credential clearing on connection failure:
```cpp
if (wifiRetry.currentRetry >= wifiRetry.maxRetries) {
memset(config.ssid, 0, sizeof(config.ssid)); // ❌ Clears credentials!
memset(config.password, 0, sizeof(config.password));
saveConfig();
// Enter AP mode...
}
```
**Solution: Resilient WiFi**
| Feature | Before (v1.9.1) | After (v1.9.2) |
| ------------------- | -------------------------- | ------------------------------- |
| Credential clearing | After 5 failed attempts | Never |
| Retry strategy | Give up after 5 tries | Infinite with backoff |
| Max retry interval | N/A | 5 minutes |
| Fallback AP | After clearing credentials | After ~5 min (dual STA+AP mode) |
| Clock during outage | Blank display | Shows last synced time |
**Key Changes:**
- **Never clear credentials** on connection failure
- **Exponential backoff**: 5s → 10s → 20s → ... → 5min max
- **Fallback AP** ("TJ56654-Setup") enabled after ~5 min, while still retrying
- **Dual STA+AP mode**: Device continues reconnect attempts while AP is active
- **SDK credentials**: Used only on first boot (no saved SSID); subsequent boots go straight to saved credentials
- **"No WiFi" display**: Shows retry countdown instead of cryptic numbers
- **"!" indicator**: Shown in date line when WiFi disconnected
**Network Activity Summary:**
| Service | Interval | Endpoint | Protocol |
| ------- | ---------- | ------------------ | ------------- |
| NTP | 1 hour | pool.ntp.org:123 | UDP |
| Weather | 30 min | api.open-meteo.com | HTTP |
| mDNS | continuous | 224.0.0.251 | UDP multicast |
~50 requests/day total.
**Results:**
- ✅ Credentials persist through WiFi outages
- ✅ Device automatically reconnects when WiFi returns
- ✅ Clock continues running with last synced time
- ✅ User can reconfigure via fallback AP if needed
**Startup Timeline:**
```
[0-5s] Display init, startup animation
[5-15s] WiFi connection (SYNCHRONOUS in setup())
✅ WiFi connected! IP assigned
[15-20s] OTA init, web server start, NTP client ready
✅ Internet test: PASSED
[20-30s] First async NTP sync
✅ Time synced and displayed
```
### v1.9.3: Modular Architecture 🗂️
Split monolithic 2,100-line `.ino` into focused modules:
| File | Responsibility |
| ------------------- | --------------------------------------- |
| `weather_clock.ino` | Entry point: `setup()` and `loop()` |
| `config.h` | Config struct, EEPROM layout, constants |
| `globals.h` | Shared state and extern declarations |
| `display.cpp` | OLED rendering |
| `ntp_client.cpp` | Async NTP sync |
| `weather.cpp` | Open-Meteo API fetch |
| `web_server.cpp` | Web UI and REST API |
| `wifi_manager.cpp` | WiFi connection and resilience |
### v1.9.4: Community Bug Fixes 🐛
Community-reported bugs fixed:
- **Date timezone** (#5): Date now changes at local midnight, not UTC midnight
- **Weather refresh** (#7): Periodic weather updates no longer blocked after first fetch
- **WiFi hotspot** (#3): Device no longer connects to open SDK-cached networks (e.g. public hotspots) when a saved SSID exists, preventing config corruption
- **ArduinoJson v7**: Updated `StaticJsonDocument` → `JsonDocument` for library compatibility
- **Compiler warnings**: Removed unused variable, fixed sprintf buffer size
### v1.9.5: Weather Recovery 🌦
- **Permanent weather lockup after 3 API failures** (#9): state machine no longer
deadlocks; resets to IDLE after max retries so periodic refresh resumes
### v1.9.6: Long-Running Stability ⏱
Six bugs found via dual AI code review (Sonnet + Gemini):
- **WEATHER_REQUESTING timeout watchdog** (C1): 15s timeout prevents TCP-hang deadlock
- **millis() rollover at 49.7 days** (C2): all retry timers now use subtraction-safe comparison
- **DST formula mathematically wrong** (H1): now uses tm_wday-based last-Sunday calculation
- **String heap fragmentation** (H2): API handlers use `snprintf+sendContent` instead of `String +=`
- **Volatile shared state** (H3): `weatherState`/`ntpState` written from callbacks, read in loop
- **Triple power-cycle factory reset**: 3 quick power cycles within 10s clears WiFi creds (no FTDI needed)
### v1.9.7: Config Validation 🛡
- **`/config` accepted any input, could brick device**: now rejects empty/all-same/oversized
SSIDs (HTTP 400), clamps numeric fields to safe ranges. Closes M1 from internal backlog.
### v1.9.8: Live Config Updates ⚡ (Current)
- **Restart only on WiFi/network changes**: brightness, timezone, intervals, coordinates,
display options apply live. Eliminates display blackouts on minor config tweaks.
- **73/73 automated tests pass** on hardware including full fuzz suite
### Memory Evolution
| Version | RAM Usage | IRAM Usage | Flash Usage | Notes |
| ------- | ------------ | ---------------- | ------------- | --------------------- |
| v1.7 | 34,980 (43%) | **61,987 (94%)** | 407,500 (38%) | IRAM crisis |
| v1.8 | 36,980 (46%) | **45,120 (68%)** | 407,800 (38%) | ICACHE_FLASH_ATTR fix |
| v1.9.0 | 37,516 (46%) | **61,987 (94%)** | 408,540 (38%) | Async libs added |
| v1.9.1 | 37,644 (46%) | **61,987 (94%)** | 408,844 (38%) | Hybrid WiFi fix |
| v1.9.2 | 37,800 (47%) | **61,987 (94%)** | 409,100 (39%) | WiFi resilience |
| v1.9.6 | 37,268 (46%) | **61,987 (94%)** | 410,436 (39%) | String → snprintf |
| v1.9.8 | 37,664 (46%) | **61,987 (94%)** | 411,380 (39%) | Live config updates |
**Verdict**: Stable memory usage, no leaks detected after 24h+ uptime tests.
---
## What's Next: Home Assistant Integration
The firmware is designed to be extensible. Next planned features:
### Custom Display Screens
Pull data from Home Assistant via REST API:
- **Smart home stats**: Energy usage, room temperatures
- **Sensor data**: Air quality, CO2 levels
- **Automation states**: Alarm status, door locks
### MQTT Integration
- Publish time/weather data to MQTT broker
- Subscribe to topics for display content
- Enable automation triggers (e.g., display alert when door opens)
### WebSocket Live Updates
Replace polling with WebSocket for:
- Real-time config changes without page refresh
- Live display preview in web UI
- Push notifications for firmware updates
See [CHANGELOG.md](CHANGELOG.md) for the full version history with technical details.
---
@@ -961,39 +594,6 @@ Device reboots immediately.
---
## Lessons Learned
### Hardware
1. **Always check pinouts**: Don't assume standard pin mappings - this device swaps SDA/SCL
2. **FTDI is your friend**: A $2 adapter unlocks any ESP8266 device
3. **Transparent cases are great**: Made debugging and identification trivial
4. **Read the PCB silk screen**: Model numbers and pin labels save hours of guessing
### Software
1. **Async is hard but worth it**: Fully non-blocking architecture eliminates user-facing freezes
2. **IRAM is precious**: On ESP8266, use `ICACHE_FLASH_ATTR` liberally
3. **Hybrid approaches work**: Don't be dogmatic - synchronous WiFi in setup() solved a critical UX issue
4. **State machines scale**: Better than callback hell for complex async operations
5. **Test on real hardware**: Emulators can't catch pin mapping errors or memory constraints
### Security
1. **IoT security is often terrible**: Always audit devices before trusting them on your network
2. **Open source is safer**: Closed firmware is a black box - you have no idea what it's doing
3. **Defaults matter**: Insecure defaults (open AP, plaintext passwords) lead to real vulnerabilities
4. **Defense in depth**: Multiple layers (WiFiManager timeout, password protection, validation) catch mistakes
### Development
1. **OTA from day 1**: Flashing via FTDI gets old fast - build OTA support early
2. **Version your work**: Backup files (.bak, .bak2) saved me multiple times
3. **Document as you go**: Release notes and architecture docs prevent "what was I thinking?" moments
4. **Incremental improvements**: v1.7 → v1.8 → v1.9.x made debugging manageable
---
## Credits
**Hardware**: TJ-56-654 Weather Clock Kit ([AliExpress](https://pt.aliexpress.com/item/1005008333782531.html))
@@ -1066,29 +666,6 @@ This project is released into the public domain. Do whatever you want with it. I
---
## Final Thoughts
This project started as "I don't trust this device" and ended as "I built something better."
The original firmware had security holes you could drive a truck through. The custom replacement:
- ✅ Doesn't leak WiFi passwords
- ✅ Uses free, open APIs
- ✅ Updates over WiFi
- ✅ Runs fully async (no freezing)
- ✅ Integrates with Home Assistant (coming soon)
- ✅ Is completely auditable (you're reading the source)
Total cost: €5 hardware + a weekend of tinkering.
If you have one of these devices, **flash this firmware**. If you're buying IoT gadgets, **always audit them first**. And if something seems insecure, **fix it yourself** - that's the hacker spirit.
Now go make something cool. 🚀
---
**P.S.**: If you found this useful, consider starring the repo. If you found a bug, open an issue. If you want to add Home Assistant screens, let's collaborate - I'm planning that next!
**Author**: apetrochenko
**Date**: 2026-01-06
**Firmware Version**: v1.9.8 (Production Ready)
**Author**: apetrochenko · **License**: MIT · **Firmware**: v1.9.9