python 爬取实时数据django显示,在Django / Python中实时数据处理

We are working on a project involving realtime data processing. We plan to use Django/Python. The actual process is:

Tens of thousands of devices take 4 samples per seconds (0, 0.25, 0.5, 0.75) and continuously send back to our Django server, basically they are time series with timestamp and value

We need to align samples from all devices according to the timestamp (need to have milliseconds precision) and do a simple average of all the time series

All these needs to be done in realtime (maximum 1 second delay) and send away using another thread

We are looking into RRDTool and scikits.timeseries, but they don't have the precision of milliseconds, so they couldn't align our time series.

Just wondering is there any tools/data structure we can use with Django/Python for this type of realtime data processing. And thread safe is important, as sending the result away will be done in another thread.

Thanks in advance.

解决方案

Your options for real time web services in python are: Twisted, Tornado and Eventlet

You can integrate all this to work with Python/Django. Tutorial on that.