At work we have a rest API that receives a lot of information from devices that are placed externally. The amount of data received has grown and the single API can no longer manage the workload and must be redesigned.
The setup is:
We have around 1700 external units that sends data every ten seconds in xml-format via http. This data must be parsed on the server before being saved.
The number of units will grow for every year so the system must be able to scale. Right now the parsing is put on a queue for later processing, but it’s still very slow.
Any ideas on architecture for something like this? We would like to keep sending via http. Any good patterns?