Streaming Frames out of Unity into OpenCV

This is what I have so far for my simulation. I take frames out of Unity and send them to a local server that does thresholding on it. I dig it. The simulation is here too if you wanna check it out. I still have to work out the steering and motor physics, so more on that later. 


At some point later in time, I'll post some of the code I used for this both on the Unity side and on the image processing side. It's nothing innovative, but when I was trying to set this up, it was hard to find stuff on the internet. 


Quick recap of the video:


In Unity, I have 2 threads. The main thread runs in the update() function. A separate thread sets up communication between the game engine and a server running on localhost. This is a simple client-server model, only that I transfer image data instead of any other data. In the main thread using C#, I grab a frame by reading pixels into a Texture2D object. Luckily, Unity will encode it to PNG for you and the EncodeToPNG() function will return you a byte[], perfect for sending over a socket. I add this byte[] to a Queue (with Monitors for thread-safety).


On the client thread, it opens up communication with localhost server, and as long as there are items in the Queue, it'll send the byte[] over the socket. 


Outside of Unity, I have a server in C++ that accepts the connection and receives the image, thresholds it, and displays it in a window. 


I'm going to start implementing this using JavaCV which supports Android so that the final transfer of the program should be a relatively simple copy/paste effort to get it to run on the phone instead of on the desktop. Of course, this is barring any drastic performance issues. In which case, I'll just get a different phone instead of the G1 and just bite the bullet with the increased cost. G1 retails (as of now 2/22/12) ~$120 (when I actually bought it in Washington Heights for $15. =))  While I could get another phone for a similar dirt cheap price, I have to use the actual retail cost for the competition.