The Arduino serial monitor is usable when you want to watch data from an Arduino. However, it does not have a built-in method for saving the data. Here are some ideas if you want to build an Arduino data logger with or without a PC.
Important note on Arduino Data Logger examples
With all of these examples, please remember that whenever you open the Arduino’s serial port,
the board will reset. So if your log file shows “Initializing SD card…” with a few data lines in between, it is because there is a reset happening.
Initializing SD card…initialized.
Temp: 34, Time: 03:24:44
Temp: 33, Time: 03:24:45
Temp: 34, Time: 03:24:46
Tem
Initializing SD card…initialized.
Temp: 34, Time: 03:24:50
Temp: 34, Time: 03:24:51
Temp: 33, Time: 03:24:52
Temp: 34, Time: 03:24:53
In that code you can see data logging started and then restarted. What happened is that after programming, the board starts logging. Then when you open the Serial Monitor, the data logger restarts.
To solve this issue, either disable auto-reset, add a 3-4 second delay at the start of setup(), wait for a character to be received, or wait for a button press. That will give you time to open the Serial Monitor.
SD Card
If you can add an SD card to your project, that is an obvious option to create an Arduino data logger. Some Ethernet shields come with an SD Card reader. Alternatively, you can get
an adapter to add SD Cards.
SD Cards use a form of the SPI protocol, so it is easy to interact with them from an Arduino. Just make sure it is a slower class. I use Class 4 cards with my projects. And I make sure then are 16GB or less. Newer or faster cards use a different protocol and won’t work.
I will not provide a code example here. The IDE comes with an SD Example called “
Datalogger.” Guess what it does.
Better Serial Terminal
In the past, I mentioned some
Arduino serial monitor alternatives. I find these tools useful for real-time interaction. For Windows there is
PuTTY, and for all operating systems there is the command-line tool ‘
screen.’
Most terminal apps will offer a “save buffer.” Others provide controls for keeping serial output. Personally, I prefer the simplicity of “
screen.”
For an Arduino data logger, just use the -L command line option. This switch will save a file called “screenlog.0″, where the 0 is the number of the screen. In most cases, it will be 0.
MacMan:~ james$ screen -L /dev/tty.usbserial-AE01AX15 > text.txt
6:41:23 PM 4/3/2017,1481
6:41:24 PM 4/3/2017,1481
BinBoo Release 0.3 - James Lewis ([email protected] )
Use date +%s to get current time in seconds
RTC has set the system time
6:41:27 PM 4/3/2017,1481
6:41:28 PM 4/3/2017,1481
6:41:29 PM 4/3/2017,1481
6:41:30 PM 4/3/2017,1481
[screen exited]
MacMan:~ james$ more screenlog.0
6:41:23 PM 4/3/2017,1481
6:41:24 PM 4/3/2017,1481
BinBoo Release 0.3 - James Lewis ([email protected] )
Use date +%s to get current time in seconds
RTC has set the system time
6:41:27 PM 4/3/2017,1481
6:41:28 PM 4/3/2017,1481
6:41:29 PM 4/3/2017,1481
6:41:30 PM 4/3/2017,1481
MQTT
If you have WiFi, like with an ESP8266, you could use
MQTT. Set up a broker on your PC (or Raspberry Pi). Then publish the messages to the
MQTT broker. When I use this method, I use the topic “debug.” Why debug? Because “Arduino Data Logger” is too long! Of course, you will need something to capture the content. You could use mosquitto_sub on the broker to capture it directly to a file.
MacMan:~ james$ mosquitto_sub -h rasppib -t debug >> mqtt-example.txt
^C
MacMan:~ james$ more mqtt-example.txt
181, Lights Turning On (Receive: 1)
182, Lights Turning Off (Receive: 0)
183, Lights Turning On (Receive: 1)
184, Lights Turning Off (Receive: 0)
185, Lights Turning Off (Receive: 0)
186, Warning: Lights Alright Off, not doing that again!
187, Lights Turning Off (Receive: 0)
188, Warning: Lights Alright Off, not doing that again!
189, Lights Turning Off (Receive: 0)
190, Warning: Lights Alright Off, not doing that again!
MacMan:~ james$
The downside to this approach is that you will not get to see the saved data in real-time. If you want to see the messages on-screen while saving to a file, you could run screen -L from above and then mosquitto_sub.
Python (and other scripting languages)
Another option for an Arduino data logger is to use a scripting language. Open up the serial port and save the contents to a file. This approach has the added benefit that you can also act on the incoming data. For example, you could use PyQt, like from the
Raspberry Pi GUI tutorial, to create a trend graph of that data.
You may need to install “pyserial.” Run pip or pip3, depending on your Python version. Or to be safe, run both.
ModuleNotFoundError: No module named 'serial'
MacMan:~ james$ pip install pyserial
MacMan:~ james$ pip3 install pyserial
Here’s a bit of code that will capture the serial port, print to the screen and write to a file. (Overwrites if the file exists.) When you are done logging, hit ctrl-c to exit the script.
#!/usr/local/bin/python3
import serial, io
device = '/dev/tty.usbserial-AE01AX15' # serial port
baud = 9600 # baud rate
filename = 'bald-log.txt' # log file to save data in
with serial.Serial(device,baud) as serialPort, open(filename,'wb') as outFile:
while(1):
line = serialPort.readline() # must send \n! get a line of log
print (line) # show line on screen
outFile.write(line) # write line of text to file
outFile.flush() # make sure it actually gets written out
Python will complain, like in the output below, but that is okay. outFile.flush() makes sure the file’s contents are saved.
MacMan:2017-04-05 datalogger options james$ ./python-serial-logger.py
b'BinBoo Release 0.3 - James Lewis ([email protected] )\r\n'
b'Use date +%s to get current time in seconds\r\n'
b'RTC has set the system time\r\n'
b'7:16:53 PM 4/3/2017,6191\r\n'
b'7:16:54 PM 4/3/2017,6191\r\n'
b'7:16:55 PM 4/3/2017,6191\r\n'
b'7:16:56 PM 4/3/2017,6191\r\n'
b'7:16:57 PM 4/3/2017,6191\r\n'
^CTraceback (most recent call last):
File "./python-serial-logger.py", line 11, in <module>
line = serialPort.readline() # must send \n! get a line of log
File "/usr/local/lib/python3.6/site-packages/serial/serialposix.py", line 483, in read
ready, _, _ = select.select([self.fd, self.pipe_abort_read_r], [], [], timeout.time_left())
KeyboardInterrupt
MacMan:2017-04-05 datalogger options james$ more bald-log.txt
BinBoo Release 0.3 - James Lewis ([email protected] )
Use date +%s to get current time in seconds
RTC has set the system time
7:16:53 PM 4/3/2017,6191
7:16:54 PM 4/3/2017,6191
7:16:55 PM 4/3/2017,6191
7:16:56 PM 4/3/2017,6191
7:16:57 PM 4/3/2017,6191
I think compared to the other options, using Python is a bit of overkill. However, if you want to extend the script to respond to the logged values, Python creates an excellent one-stop solution.
Conclusion
Whether you want an Arduino data logger with or without a PC, you have several options. Personally, MQTT and screen are my favorite two methods. MQTT on an ESP8266 does not require a tethered PC.
I am curious about how difficult it would be to make a PyQt based GUI that logs and graphs the data it is receiving. Any takers?
11 Comments
In the example with ‘screen’ how were you able to pass ‘date +%s’ so that the time stamps are included?
Following your example I was able to log the serial output to a file using ‘screen’. In order to specify the name of the log file, the syntax provided in the example didn’t work for me using screen v 4.0.3 (default version on macOS Sierra). However by updating to v 4.6.2 using Homebrew I was able to utilize the solution suggested here: https://stackoverflow.com/questions/14208001/save-screen-program-output-to-a-file/37559327#37559327 to specify the filename using the -Logfile name option.
Thanks for the helpful post
I didn’t. The device I attached to was my binary clock, which outputs the time (and the current value of millis).
The date +%s reference was a reminder for me on how to get the number of seconds, so I can re-sync the clock after it drifts.
Ah, that makes sense.
Btw, when you use screen as a serial monitor do you know a way to initiate local echo so you can see what commands are being typed? I have been able to establish a serial connection using screen and have successfully passed messages to it but only in a blinded fashion. I have gone through the entire screen manual and have not found a way to do this.
Thanks
This page has some options to try: https://superuser.com/questions/752098/local-echo-using-screen-to-connect-to-a-serial-terminal
I think logging can be classified as: Standalone; or Wired to PC; or Radio Link.
In Standalone, SD card (including micro SD card) is one option; the other main one would be USB drive. There are add-on modules for both on the market, including the one for micro SD you link in your article. There are libraries available for both also. Both SD card and USB libraries are pretty big pieces of software, and so may limit the complexity of the software that generates the logging data. Here is a link to a collection of micro SD data loggers on Instructables that provide some practical examples: https://www.instructables.com/id/Temperature-Loggers/
In Wired to PC, I can’t add to your list.
Under Radio Link, I think there are configurations using WiFi (you mention ESP8266, which is certainly one option). I believe Bluetooth is another option, which I have seen mentioned in several articles although I have no experience with using it. I think other radio technologies using simple radio modules, are also available. Also, there are several web sites that offer support for data logging and presentation. Thingspeak comes to mind; again I have no practical experience with using them.
Thanks for a stimulating article.
Maybe instead of Python you could use Processing. It’s more “Arduino-like” and its output is intrinsically graphical
Good point. I actually wrote a processing program to graph data from a serial port (in my case, my reflow oven) a couple of years ago: https://www.baldengineer.com/revision-2-of-the-open-source-reflow-solder-oven-is-done.html
This made me think about a slick little board I bought a few years ago to do some data logging. It uses an ATMega328P and has a micro-SD card slot on the back. It’s smaller than a quarter. I think they’ve made some improvements since I bought it but as I recall it worked right out of the box with a minimal amount of configuration.
https://www.sparkfun.com/products/13712
I am in no way affiliated with SparkFun.
Very slick looking board. Probably an ideal solution for most logging applications. I wonder why they have a note about IDE compatibility. Wonder what is causing that.
I recommend visiting the fine folks at http://www.angstromdesigns.com/software/faq/58-labview-arduino-driver-larva They have a sweet GUI and a simple interface to a PC/laptop. I love simple + powerful. Enjoy!
Interesting. I don’t have windows to test it, do you have to have LavView or is it standalone?