Google Code Setup

Well I finally made the jump and got and my code setup on Google code to start sharing my HomeNet Code. You can also find some documentation on how it all works 1800 lines of code and growing...

Google Code: HomeNet

Even More Modular

My last code post simplified things, but many things were hard coded in. These weekend I modularized things so you can choose which network adapters you needs as well as what devices are connected.

#include <ports.>h>
//#include <portslcd.h>
#include <rf12.>h>
#include <homenet.>h>
//#include <homenetdevices.h>
#include <homenetdevicelcd.>h>

//Start HomeNet Packet Stack
HomeNet stack(0x04);//0x01 is RF12 base station //0xFF is PC uplink

//Setup Network Adapters
//HomeNetPortSerial portSerial(PORT_SERIAL, stack);  //PORT_SERIAL tells the adpater which serial serial port to use (for future support of the arduino mega)
HomeNetPortRF12   portRF12(stack);

//Setup attached devices
HomeNetDeviceStatusLights statusLights(stack);
//HomeNetDeviceTMP37 tmp37(stack);
//HomeNetDeviceLight light(stack);
HomeNetDeviceLCD lcd(stack);

//package the setup info in a nice neat arrays
 HomeNetPort*  ports[] = {&portRF12};//,&portSerial}; 
 HomeNetDevice*  devices[] = {&statusLights,  &lcd};// &light , &tmp37};

MilliTimer sendTimer;

void setup() {
  //initize home net with the setup info
  stack.init(ports, 1, devices, 2); 
  stack.registerStatusLights(statusLights); //setup status lights
}

void loop() {
  //receive incoming packets
  stack.receive();
  
  if (sendTimer.poll(10000)){
   // PayloadBuffer buffer;
   // buffer.print("TEST NODE 3");
   // stack.addUdpPacket(0, 2, 0, CMD_STRING, buffer.payload());
    //stack.schedule(2,255,0); //type, toNode, toDevice
     
  }
  
  //process packets in the stack- send packets that ready
  stack.process();
  delay(1);

}

More Code

So the code finally reached a tipping it's time to package the code up into a library. I'm calling it HomeNet because it describes its perfectly it's a new home net work and amazingly enough, there isn't anything going by the name yet.

Compared to the long mess of code posted before, Moving most of the code to a library really simplified things:

#include <ports.h>
#include <rf12.h>
#include <homenet.h>

HomeNet stack(0x04);//start stack with a node id
MilliTimer sendTimer;

void setup() {
  // put your setup code here, to run once:
  stack.init();

}

void loop() {
  // put your main code here, to run repeatedly: 
  stack.receive();
  if (sendTimer.poll(2000)){
    PayloadBuffer buffer;
    buffer.print(123,DEC);
    stack.addUdpPacket(0, 255, 2, CMD_BYTE, buffer.payload());
  }
  stack.process();
}

and the processing sketch using the library

import processing.serial.*;
import xmlrpclib.*;
import org.apache.commons.codec.binary.Base64; //part of the xmlrpclib

XmlrpcClient homeNetXmlrpcClient = new XmlrpcClient("http://matthew.sgcs.usf.edu/mcdportal/public/xmlrpc.php?api_key=dfgdfgdgdgd");
XmlrpcServer homeNetXmlrpcServer = new XmlrpcServer(8081);



final int  NODE_ID = 255;
final int PACKET_TIMEOUT = 150;
PacketStack stack = new PacketStack(NODE_ID);

Serial myPort;  // The serial port


XMLElement xmlConfig;



XMLElement commands[];
XMLElement devices[];

String findXMLid(XMLElement[] xml , int value, String rAttribute){
  String rValue = ""; 
  for(int a = 0; aif(unhex(commands[a].getAttribute("id")) == value){
      return commands[a].getAttribute(rAttribute);
    }
  }
  return rValue;
} 


void setup() {
  size(600, 400);  // Stage size
  //noStroke();      // No border on the next thing drawn
  homeNetXmlrpcServer.add("testserver", this); //setup server
  println("HomeNet XML-RPC Server started at " + homeNetXmlrpcServer.ip()+":8081/RPC2"); // Note that /RPC2 is hard-coded to URI in the Apache Library.
  
  textFont(createFont("Consolas", 14));
  //fixedFont);
  
  xmlConfig = new XMLElement(this, "config.txt");
  devices = xmlConfig.getChildren("devices/device");
  commands = xmlConfig.getChildren("commands/command");
  
  //load the above as hash table from my sevrer.
  
  // Print a list of the serial ports, for debugging purposes:
  //println(Serial.list());
  
  String portName = Serial.list()[1];
  myPort = new Serial(this, portName, 115200);
}

boolean arduinoBoot = false;
int count = 0;
long sendTimer;
void draw() {
  
  if(arduinoBoot == false){
    delay(2500);
    sendTimer = millis();
    arduinoBoot = true;;
  }
  
  if((millis() - sendTimer) > 2000){
    //DataPacket packet = build_udp_packet(NODE_ID,0,3,2,CMD_STRING,str_to_payload("Hello "+count));
    //packet.status = STATUS_READY;
    sendTimer=millis();
    count++;
  }

  stack.process();
    
  stack.tempSerialCheck(myPort); //this is temp until I add support for multiple serial ports, and pass a list of ports to the packetStack
}
void packetEvent(DataPacket packet){
  packet.status = STATUS_CLEAR;
  Hashtable send = new Hashtable();
  String packetBase64 = new String(Base64.encodeBase64(packet.payload));
  send.put("received",packet.received);
  send.put("packet", packetBase64);
  println("Server Reply: "+homeNetXmlrpcClient.execute("HomeNet.addPacket", send));
}  

void serialEvent(Serial port) {
  stack.buildSerialPacket(port);
}

Much easier to use now!

First Tests

I finally completed enough of the packet code that I finally test to see if it actually works! and Amazingly enough ti works. Packets can now be sent from a remote wireless node to a receiver and passed to the computer. It also works the other way, data from the computer can be sent to a remote node. I recorded a video but I don't have it up yet.

Code after the jump

XML Config File

Its starting to be pain to setup on the constants in both Arduino and processing and in the future php so it's time to create a configuration file that all of these different languages can use (sort of)

And here it is:

<?xml version="1.0"?>
<configuration>

<devices>
  <device id="0x01" code="DVC_TMP37">Temperature Sensor TMP37</device>
  <device id="0x02" code="DVC_LIGHT">Light Sensor LDR</device>
  <device id="0x03" code="DVC_SHT21">Temperature and Humidy Sensor SHT21</device>
  <device id="0x04" code="DVC_DOORCONTACT">Door Contact Sensor</device>
  <device id="0x05" code="DVC_WINDOWCONTACT">Window Contact Sensor</device>
  <device id="0x06" code="DVC_MOTION">Motion Sensor</device>
  <device id="0x07" code="DVC_2LINEDISPLAY">2 Line Character Display</device>
  <device id="0x08" code="DVC_POWER30">30A Current Sensor</device>
  <device id="0x09" code="DVC_POWER100">100A Current Sensor</device>
  <device id="0x10" code="DVC_SMARTOUTLET">Smart Outlet</device>
  <device id="0x11" code="DVC_RGBLIGHT">RGN LED Light</device>
  <device id="0x12" code="DVC_sound">Sound Sensor</device>
  <device id="0x13" code="DVC_SWITCH">Simple Switch</device>
  <device id="0x14" code="DVC_STATUSLED">Simple Status LEDS</device>
  <device id="0x15" code="DVC_BUZZER">Alert Buzzer</device>
</devices>

<commands>

  <!-- System Commands Port 0 -->
  <command id="0x00" code="CMD_ERROR"  payload="message" return="CMD_ACK"></command>

  <command id="0x01" code="CMD_VERSION" payload="" return="CMD_FLOAT">
    Get current version of firmware on the node
  </command>
  <command id="0x03" code="CMD_BATTERYLEVEL" payload="" return="">
    Get current battery level
  </command>
  <command id="0x3E" code="CMD_PING" payload="varies" return="CMD_PONG">
    Send Test Pack, node should reply back pong
  </command>
  <command id="0xE3" code="CMD_PONG" payload="varies" return="CMD_ACK">
    Response to a Ping Command
  </command>
  <command id="0x11" code="CMD_ACK" payload="packetid" return="CMD_BYTE">
    Acknowledge Packet, letting the sender of a packet know that data arriced safely
  </command>
  <command id="0x21" code="CMD_GETNODEID" payload="" return="CMD_BYTE">
    Get Node's current ID (node id 0 is for broadcast)
  </command>
  <command id="0x22" code="CMD_SETNODEID" payload="passcode" return="CMD_ACK">
    Changes NODE ID, used for initial setup
  </command>
  <command id="0x23" code="CMD_GETPLUG" payload="byte port" return="CMD_BYTE">
    Get the device code for what is attached on the node
  </command>
  <command id="0x24" code="CMD_SETPLUG" payload="passcode" return="CMD_ACK">
    Changes Device ID, used for initial setup
  </command>

  <!-- Plug commands Ports 1-15 -->
  <command id="0xC1" code="CMD_AUTOSENDSTART" payload="node to, int" return="CMD_ACK">
    Set up the node to automatically send sensor data to a particular node 
  </command>
  <command id="0xC2" code="CMD_AUTOSENDSTOP" payload="node to" return="CMD_ACK">
    Stop auto sending sensor data
  </command>
  <command id="0xD0" code="CMD_GETVALUE" payload="" return="Value">
    Get a value from a plug
  </command>
  <command id="0xD1" code="CMD_SETVALUE" payload="" return="Value">
    Send a value to a plug
  </command>

  <command id="0xE0" code="CMD_ON" payload="" return="CMD_ACK">
    Simple Turn on
  </command>
  <command id="0xE1" code="CMD_OFF" payload="" return="CMD_ACK">
    Simple Turn Off
  </command>
  <command id="0xE2" code="CMD_LEVEL" payload="byte" return="CMD_ACK">
    Set a light to a level 0-255
  </command>

  <!-- Value commands -->
  <command id="0xF0" code="CMD_BYTE"   payload="byte" return="CMD_ACK"></command>
  <command id="0xF1" code="CMD_STRING" payload="string" return="CMD_ACK"></command>
  <command id="0xF2" code="CMD_INT"    payload="int" return="CMD_ACK"></command>
  <command id="0xF3" code="CMD_FLOAT"  payload="float" return="CMD_ACK"></command>
  <command id="0xF4" code="CMD_LONG"   payload="long" return="CMD_ACK"></command>
</commands>

<!-- One day more stuff will be setup too  -->
<pachube>
  <apikey></apikey>
  <feeds>
    <feed name="Bedroom">
      <datastream id="0" node="" plug="" device="" />
    </feed>
  </feeds>
</pachube>

<webserver>
  <address></address>
</webserver>

</configuration>

Processing has built in tools for reading XML and this quick sketch reads the XML file a translates it in to list of #defines for the Arduino. this way as the lists of devices and commands evolve, it's very easy to translate it from one language to another.

XMLElement xml;

PFont myFont;
PrintWriter file;

int i;
String spaces(int count){
 String spaces = "";
  for(int a = 0;a<count;a++){
   
   spaces += " ";
  } 
  return spaces;
}

void setup() {
  size(200, 200);
  
  myFont = createFont("Consolas", 14);
  textFont(myFont);
  
  xml = new XMLElement(this, "config.txt"); //in txt
  file = createWriter("define.txt");
  
  XMLElement commands[] = xml.getChildren("commands/command");
  
  for(i = 0; i<commands.length;i++){
    file.println("#define "+ commands[i].getAttribute("code")+spaces(20 - commands[i].getAttribute("code").length())+commands[i].getAttribute("id"));
  }
  file.println(""); //blank line
  XMLElement devices[] = xml.getChildren("devices/device");
  
  for(i = 0; i<devices.length;i++){
    file.println("#define "+ devices[i].getAttribute("code")+spaces(20 - devices[i].getAttribute("code").length())+devices[i].getAttribute("id"));
  }
  file.flush(); // Write the remaining data
  file.close();
  exit();
}

And here is the Arduino Code the script above generated for me

#define CMD_ERROR           0x00
#define CMD_VERSION         0x01
#define CMD_BATTERYLEVEL    0x03
#define CMD_PING            0x3E
#define CMD_PONG            0xE3
#define CMD_ACK             0x11
#define CMD_SETNODEID       0x21
#define CMD_GETNODEID       0x21
#define CMD_SETNODEID       0x22
#define CMD_GETPLUG         0x23
#define CMD_SETPLUG         0x24
#define CMD_AUTOSENDSTART   0xC1
#define CMD_AUTOSENDSTOP    0xC2
#define CMD_GETVALUE        0xD0
#define CMD_SETVALUE        0xD1
#define CMD_ON              0xE0
#define CMD_OFF             0xE1
#define CMD_LEVEL           0xE2
#define CMD_BYTE            0xF0
#define CMD_STRING          0xF1
#define CMD_INT             0xF2
#define CMD_FLOAT           0xF3
#define CMD_LONG            0xF4

#define DVC_TMP37           0x01
#define DVC_LIGHT           0x02
#define DVC_SHT21           0x03
#define DVC_DOORCONTACT     0x04
#define DVC_WINDOWCONTACT   0x05
#define DVC_MOTION          0x06
#define DVC_2LINEDISPLAY    0x07
#define DVC_POWER30         0x08
#define DVC_POWER100        0x09
#define DVC_SMARTOUTLET     0x10
#define DVC_RGBLIGHT        0x11
#define DVC_sound           0x12
#define DVC_SWITCH          0x13
#define DVC_STATUSLED       0x14
#define DVC_BUZZER          0x15

Ping Pong Packets

After another week of work on my packet code, I finally have a working example of call and response working. Processing sends a Ping packet to the Arduino and the Arduino replies back Pong along with the payload of the original packet. Next step is to expand this to connect to wireless nodes.

Code after the jump

Goals and Issues

Below is current look at some of goals and issues I'm currently dealing with-

So far my project has three main components:

Sensor Network <-> Base Station  <-> Web Interface

They are interconnected and pass data between each other.

Sensor Network

The sensor network is gathering the sensor data is currently based on Jeenodes which are Arduino based nodes.

Status: Hardware has been solder together, sensors have been purchased and the code for the protocol has been written.

Goals

• Create robust adaptable nodes that can easily be reconfigured
• Create Wired and Wireless Links
• Create a Mesh Network

 Current Issues

• How to design so an average user can configure a node? Pre programed ROM? Dip switches? Web interface?
• Solution: Processing sketch that directly communicates with node to set address and sensor configuration to the Arduino ROM
• Current Wireless Chip doesn't do mesh networking: cost prohibitive

Base Station

A Server program running on a PC reads packets of data from the sensor network and passes them on to a web interface. Currently using a Processing.org Java Program

Status: Protocol code has been ported to java. Actual interface still needs to be built.

Goals

• Be user configurable
• Easy to deploy/setup

Current Issues

• How to Connect to Web server?
• XML-RPC seems like the simplest solution right now and offers a lot of flexibility. 
• Using EEML like pachube might work too but would require a lot more code.
• How to save settings? flat file, SQL database, remotel?
• I haven't decided how to solve this yet. Some settings must be saved locally, ie web server address, Pachube api key. Using an XML file is the best soultion for this. 
• Other settings like node and sensors names would be better stored in a better database if it is to scale to thousands of sensors. Can it call a remote server to get this info or should it be local?
• How to store data if Internet Connection goes down?
• Store it memory is the easiest, SQL would be better

Web Interface

Status: In planning.

The web interface right now is going to be using LAMP Linux, Apache, mySQL and PHP.  The code is going to based on the Zend Framework and a UI library I started writng a while back. The front end is going to use jQuery and jQueryUI. Graphs will use the Google Graph API.

Goals

• Work on a varety of platforms and screen sizes
• Use Web 2.0 Ajax etc. Jquery
• Create an interactive Remote for the network

Current Issues

• Server local or remote?
• A good webserver is hard for an average user to setup. A remote server can also handle more users. User can also access the remote to get info about power/data outages

XML-RPC A better way to connect Processing and the Web

After yesterdays post about how to connect the webserver to the processing sketch, I thought that there had to be a better way and there is.

There is a nice XML-RPC Library for Processing that someone at MIT wrote. XML-RPC is very common on the web so there are multiple ways to implement in php, the best way is probably with the built in XML-RPC Class. After reading the documentation the built in funtions aren't well documented. It looks like I will be using the ZendFramework XML-RPC Module

First Packet Test

The Arduino sends test packets to  processing which decodes them on screen



Code after the jump

From Processing to the Web

An Arduino is connected to a PC that is running a program created in Processing. It reads the data stream from the arduino and then it can publish the data to an online service like pachube or my own server.

The main user interface of my project is going to be web based so it can be used on all sorts of devices from laptops to cellphones. Processing doesn't make a very good webserver to build an interface around however there is some really good webserver software - Apache, PHP and MySQL.

Linking the webserver to the Processing program is a bit of challenge. A webserver doesn't typically have a program always running, rather the program runs only when generating a web page. To send data to the webserver involved calling a webpage with get/post data. For the webserver to send data to the Processing sketch involves using a simple socket server. I haven't found any example of this online but I think I can combine some of this processing code and some of this php code and make it work.

Update: there is a better way to connect things- XML RPC

Communication Protocol

One of the big steps is developing a language for how all the nodes talk to each other.

There are several existing protocols like BACnet and LonWorks. These a both full featured but almost too complex for what I need right now. I looked to see if anyone had used an arduino with either of these protocols but my search turned up blank. So I decided to write my own.

Data is sent in packets. Each packet is like a letter that has a to and from address and carries a payload inside.

There are several types of packets based on what sort of data is being carried

The first three I plan on implementing are

• TCP Packet - based on the Transmission Control Protocol, well a very simplified version but when a node receives one of these packets it sends a ack (acknowledgment) packet letting the sender know that the packet reached it destination.
• UDP Packet - Based on the User Datagram Protocol, again a simplified version. These are packets that don't need acknowledgment. These is used when data is constantly being sent and lost packets would be outdated by the time they were resent
• Broadcast - this is variant of UDP that doesn't have a set destination, rather it has a max number of hops that the packet will take and all nodes will take a look at it's data

Each packet has a 8 byte header and 2 byte footer. this a lot smaller than the 20 byte header of a standard TCP packet.

8 bits = 1 byte

So far my protocol looks like this

1. 8 bits - Length of Packet (could be shorter if we don't need big packets
2. 8 bits - Settings (could be smaller only using 2 bits right now
3. 12 bits - From node, allows for 4096 nodes
4. 4 bits - From ports (allows 16 ports on a node)
5. 12 bits - To Node (or ttl Time to live, if broadcast)
6. 4 bits - From Port
7. 8 bits - Packet ID; So that packets can be put back in the right order
8. 8 bits - Command; read, write, etc  (could be shorter)
9. 56 bytes Payload
10. 16 bits - Checksum (verifies data didn't get corrupted in transmission)